MiniMax M3.1 Flash 1.59×
Harness: Codex · switches: YOCO_OPT_REPLICATED_ADAM=1 YOCO_OPT_SKIP_MASTER_ZERO=1 YOCO_OPT_FUSED_GRAD_SCALE=1 YOCO_OPT_FUSED_WEIGHT_CAST=1 YOCO_OPT_FUSED_ADAM=1 YOCO_OPT_FAST_LINEAR=1 YOCO_OPT_FAST_TRITON=1 YOCO_OPT_GRAD_BUCKET_VIEW=1 YOCO_OPT_TRITON_RMSNORM=1 YOCO_OPT_CACHED_PTR_TABLE=1 YOCO_OPT_DDP_BUCKET_MB=100 YOCO_OPT_ADAM_BF16_GRAD=1 YOCO_OPT_FUSED_L2NORM=1 YOCO_OPT_ADAM_PLAN=1 YOCO_OPT_FAST_ADAM=1
- Judged
- 1.59×
- Tier
- N1
- Self-best
- 1.59×
- Rounds
- 29
- Hours
- 10.0
- API cost
- $6.03
- Input tokens · 96.4% cached
- 235 M
- Output tokens
- 2.3 M
- Requests
- 1,820
§ 02Key levers
- Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
- ✗ no exchange:
YOCO_OPT_REPLICATED_ADAMhas both ranks run Adam on all params (1.00× → 1.11×); grad all-reduce still NCCL - Forward / backward CUDA graph
- ✗ judged infeasible, no measurement in the log
- Newer cuBLAS
- ✗
- Retention / attention kernel swap
- ✗ only rotary and recurrence kernels switched to direct launch (1.41× → 1.45×)
- Other main levers
- Triton fused "cast + scale" 1.18× → 1.30×; Triton Adam also writing bf16 weights 1.34× → 1.40×; Triton RMSNorm with two-level reduction 1.48× → 1.53×
§ ARaw trace
Raw work log31 sections
0.2 hRound 1 — baseline profile (unchanged reference tree)
Start 2026-11-07 05:24, end 05:27.bench.py profile --tree /submission/yoco --config /submission/switches.json --out /tmp/r1/profile(0.2891 s/update under the profiler; the reference itself is 0.2336). Kernel time by category, rank 0, 5 updates (profile.md): | category | ms | ms/update | |---|---|---| | nccl | 435.7 | 87.1 | | gemm | 330.8 | 66.2 | | memcpy/cast | 219.2 | 43.8 | | elementwise | 153.8 | 30.8 | | optimizer (apex fused adam) | 91.4 | 18.3 | | flash-attention | 29.8 | 6.0 | | reduce | 19.2 | 3.8 | | triton (rotary/recurrence/inductor) | 13.9 | 2.8 | | jiterator (swiglu) | 11.1 | 2.2 | Findings (from the Chrome trace, one update = 266 ms under the profiler): * The GPU is essentially never idle (largest gap 2.2 ms) -> the update is GPU-bound, not data- or launch-bound. * NCCL is 41% of the kernel time. 595AllReduce_Sum_bf16(DDP gradient all-reduce, 119 calls/update, 47.5 ms/update) and 10Broadcast_RING_LL(fairscale ZeRO-1 master-weight broadcast, 2 calls/update, 191.7 ms total = 38.3 ms/update). * The ZeRO-1 broadcast is fully serial:fairscale::oss::refresh_trainable(23.5 ms/update) is preceded by acudaDeviceSynchronize(23.3 ms) and is followed by a 44.7 mscudaStreamSynchronizeat the end of the update -> the broadcast is on the critical path. * The copy traffic of the fp32 master weights: memset of the master grads (13.8 GiB), bf16->fp32 grad copy (20.7 GiB), clip-coefficient multiply of the master grads (27.6 GiB), fp32->bf16 weight copy (20.7 GiB) = ~83 GiB of memory traffic per update, all of it avoidable or fusable. * The GEMMs run at ~575 TFLOP/s (the output projection at 787 TFLOP/s) -> there is little to win in the model's matmuls. *aten::itemsyncs: 4 per update, all of them waiting for the GPU (the longest is the 44.7 ms one at the end of the update). Conclusion: attack (1) the optimizer's inter-rank communication, (2) the master-weight copy traffic, (3) the gradient all-reduce's efficiency. The model's matmuls and attention are fine.
0.2 hRound 2 — replicated Adam (no ZeRO-1 broadcast)
yoco/models/optim_overrides.py(new, imported bymodels/__init__.py), three switches, all off by default: *YOCO_OPT_REPLICATED_ADAM=1: replaces fairscale's ZeRO-1 wrapper with a variant that keeps the Adam state of every parameter on every rank and drops the per-step broadcast (and the device synchronize in front of it). DDP has already reduced the gradients, so both ranks compute the same update from the same inputs; the per-parameter arithmetic (apex fused Adam, fp32 master weights, fp32 gradients) is unchanged. Tier N0. *YOCO_OPT_DDP_BUCKET_MB=<int>: DDP gradient bucket size (torch's default is 25 MiB). *YOCO_OPT_SKIP_MASTER_ZERO=1: skip the memset of the fp32 master gradients (the copy that follows overwrites them completely). Tier N0. Results (judgetime, 2 x Hopper, each run ~1.5 min): | configuration | s/update | speedup vs frozen reference | peak mem | |---|---|---|---| | reference (frozen) | 0.2336 | 1.000 | 61.75 GiB | |YOCO_OPT_REPLICATED_ADAM=1| 0.2106 | 1.1092 | 74.51 GiB | |YOCO_OPT_REPLICATED_ADAM=1+YOCO_OPT_SKIP_MASTER_ZERO=1| 0.2097 | 1.1140 | 74.51 GiB | |..._REPLICATED_ADAM+YOCO_OPT_DDP_BUCKET_MB=256| 0.2107 | 1.1087 | 74.51 GiB |check --quick(gates 0+2,/tmp/r3/quick/check.json): PASS -> the change is bitwise identical to the reference (tier N0). The DDP bucket size has no effect (torch's bucket is already large enough / the all-reduce is bandwidth bound, not latency bound) -> drop it. Kept in/submission/switches.json:YOCO_OPT_REPLICATED_ADAM=1,YOCO_OPT_SKIP_MASTER_ZERO=1(1.114x). Cost: +12.8 GiB of Adam state per rank (replicated instead of sharded); the machine has 139.8 GiB per GPU, so there is room, but the headroom is smaller than the reference's. Commit: b264b25 "Add environment-switched overrides for the fp32 optimizer step and DDP buckets (N0, off by default)".
0.9 hRound 3 — screening the remaining stages (`/tmp/r5`, `/tmp/r6`)
The profile of the current best (/tmp/r4/profile) showed (5 updates, profiler inflated, rank 0): gemm 67 ms/update, nccl 58, memcpy/cast 44, optimizer 33, elementwise 26, flash 6, triton 14. Two findings from the trace: the GPU is idle 24 % of the window (the CPU cannot keep 6200 kernel launches/update fed) and the NCCL work is 58 ms of kernel time but only ~17 ms of it is not already overlapped with compute. Screening runs (judgetime, one run each; these switches *change what is computed*, they only size the opportunities): | screen | s/update | speedup | what it measures | |---|---|---|---| | base (round 2) | 0.2095 | 1.115 | - | |YOCO_SCREEN_NO_MASTER2BF16=1| 0.1996 | 1.170 | the fp32 master -> bf16 weight copy costs 9.9 ms | |YOCO_SCREEN_NO_ADAM=1| 0.1814 | 1.288 | apex's fused Adam costs 28.1 ms | |YOCO_SCREEN_NO_ALLREDUCE=1| 0.1874 | 1.247 | DDP's gradient all-reduce costs 22.1 ms on the critical path | |YOCO_OPT_FOREACH_OPS=1| 0.2319 | 1.007 | regression: torch's_foreach_copy_has no fast path for mixed dtypes and falls back to a per-tensor loop -> reverted | Failed direction:gradient_as_bucket_view=Truefor DDP (would have removed DDP's 240copy_bucket_to_gradcopies) hangs with fairseq, becauseFP16Optimizer.zero_gradsetsp.grad = Noneand the reducer then waits for gradients that never arrive. Abandoned. Also checked and rejected: the twostate["exp_avg"].to(...)calls per parameter in apex'sFusedAdamV2.stepare *not* copies (verified with a standalone probe: the state tensors keep their data pointer), so there is nothing to win there.
0.9 hRound 4 — fused RMSNorm (`torch.compile`), N1
yoco/models/decoder/rms_norm.py: the eager normalization materializes three full-size fp32 temporaries per call (x.float(),x.pow(2), the fp32 product) and the model calls it ~130 times per update. WithYOCO_OPT_COMPILE_RMSNORM=1the same arithmetic is compiled into one kernel per shape. The reduction order of the mean of squares changes -> tier N1. *time(/tmp/r7/time_compile): 0.1987 s/update, speedup 1.1756 (from 1.115). *check --n1(/tmp/r7/n1/check.json): PASS, gate 3 "equivalent"; every one of the six endpoints is within 0.81 sigma of the reference (the bound is 3 sigma). * Notorch.compilerecompile-limit hit and no "skipping cudagraphs" message in the run log; the compilation happens before the first update (startup 66 s, same as the reference's).
1.4 hRound 5 — fused gradient cast-and-scale (Triton), N0
Profile of the current best (/tmp/r8/profile, 1.1756x) put the two memory-bound optimizer kernels at the top of the update: | kernel | calls | ms/update | |---|---|---| |unrolled_elementwise_kernel<direct_copy>(bf16 grad -> fp32 master grad) | 1200 | 12.9 | |elementwise_kernel<BinaryFunctor<float>>(master grad *= scale) | 1200 | 12.3 | |multi_tensor_apply<AdamFunctor>(apex Adam) | 825 | 28.7 | |ncclDevKernel_AllReduce_Sum_bf16| 595 | 60.9 (mostly overlapped) | The cast reads 2 bytes and writes 4 bytes per element, the scale reads 4 and writes 4: 40.6 GB of HBM traffic per update for two passes over the model, in 480 small kernels (one per parameter) that reach only ~1.0-1.9 TB/s.yoco/models/fused_grad.py(YOCO_OPT_FUSED_GRAD_SCALE=1) does both in one Triton kernel: it reads the bf16 gradient, converts it to fp32, multiplies it by the scale and writes the fp32 master gradient. All the parameters are handled by **one launch** (the kernel walks a table of the parameter addresses, built once from the shapes), so the launch overhead is one kernel per update instead of 480. The scale (the data-parallel size over the sample size, times the clipping coefficient) is only known after the gradient norm, so the norm is computed from the bf16 gradients instead of from the fp32 master gradients. apex's fused L2 norm was probed to return the *same* value for a bf16 tensor and for its fp32 image (/tmp/probe_l2.py, bitwise equal on four tensor lists), so the clipping coefficient is unchanged. The cast-and-scale is bitwise equal tocopy_+mul_(probed on four tensor sizes,/tmp/probe_engine.py). *time(/tmp/r5/time_fused): 0.1803 s/update, speedup 1.2956 (from 1.1756). *check --quickwith the same switches but without the RMSNorm compile (/tmp/r5/quick_nofuse): PASS, tier N0 - 9616 recorded tensors bitwise identical to the reference, 0 exact failures. So this change is bitwise neutral; the only non-bitwise change in the tree is the compiled RMSNorm (N1, already validated bycheck --n1).
2.3 hRound 6 — fused fp32-master -> bf16-weight copy (Triton), N0
The fp32 master weights have to be copied into the bf16 model parameters after every optimizer step: 12.6 GiB read + 6.3 GiB written per update, in 240 kernels that reach only about 2 TB/s.yoco/models/fused_grad.py(YOCO_OPT_FUSED_WEIGHT_CAST=1) does the copy for every parameter in one Triton launch (the kernel walks a table of the parameter addresses, built once from the shapes). *time(/tmp/r6/time_wc): 0.1747 s/update, speedup 1.3371 (from 1.2956). *check --quick(/tmp/r6/quick_n0): PASS, tier N0 (9616 tensors bitwise).
2.3 hRound 7 — fused Adam (Triton), N0
The apex fused Adam was the largest single kernel of the update: 825multi_tensor_applylaunches per update, 28.7 ms of kernel time at about 2.8 TB/s. apex'sFusedAdamV2runs inADAM_MODE_1(decoupled weight decay): m = beta1*m + (1-beta1)*g ; v = beta2*v + (1-beta2)*g*g m_hat = m/bc1 ; v_hat = v/bc2 ; denom = sqrt(v_hat) + eps update = m_hat/denom + decay*p ; p = p - lr*update withbc = 1 - pow((double)float(beta), step)— the beta is rounded to fp32 first, which is what apex does (float beta = ...thenstd::pow((double)float_beta, step)); using the Python double beta gives a different bias correction and a different result.yoco/models/fused_grad.py(YOCO_OPT_FUSED_ADAM=1) replacesfairseq.optim.fused_adam.FusedAdamV2.stepwith one Triton launch for all the parameters (4.1 TB/s, measured over 26 M elements). The kernel usestl.math.fma(beta1, m, (1-beta1)*g)(nvcc contracts the same way), plain division andtl.sqrt(notdiv_rn/sqrt_rn), and it writes the **bf16 model weights in the same pass**, so the separate master -> model copy of round 6 is no longer needed (the weight-cast override then skips its copy). *time(/tmp/r8/time_adam): 0.1667 s/update, speedup 1.4013 (from 1.3371). *check --quickwithout the RMSNorm compile (/tmp/r10/quick_n0): PASS, tier N0 — 9616 recorded tensors bitwise identical to the reference, 0 exact failures. * Fullcheckwith the submitted switch file (/tmp/r11/check): PASS, speedup 1.4013, tier N1 (the only non-bitwise change is the compiled RMSNorm); gate 3: every one of the six endpoints within 0.81 sigma (bound 3 sigma).
2.3 hRound 8 — where the time goes now (profile of the current best)
The trace of the reference (/tmp/r8/profile) says the update is **launch bound in the forward pass**: per update the GPU is busy 175 ms (profiled) out of a 225 ms window and idle 50-67 ms, and most of that idle is inside the model forward (## self.mixer19 ms/update,self_decoder3.8,cross_decoder3.2). The backward is GPU bound (its compute is longer than the CPU work that drives it). Two causes: * the forward issues about 3300 kernels per update (780 of them GEMMs) and the Python overhead of megatron's parallel linear layers (twoautograd.Functionnodes per layer and per direction, for a copy/reduce that is the identity with one model-parallel rank) leaves the GPU waiting; * 119 NCCL all-reduce calls per update (the DDP buckets are 25 MiB, but a single FFN weight is 50 MB, so every bucket holds about one FFN weight): 59 ms of NCCL kernel time at about 200 GB/s, well below the NVLink bandwidth.
1.4 hRound 5b — failed direction: flat parameter layout
yoco/models/flat_layout.py(now removed) laid the model parameters, the fp32 master weights and the master gradients out in one buffer each, so that the master -> bf16 copy and the gradient scaling became single kernels. Measured 0.2180 s/update (a regression of 19 ms). The profile of that run (/tmp/r9/profile) explains why: the optimizer phase ran 720 gradient copies per update instead of 240 (the extra ones are in theoptimizerrange, i.e. in the master -> bf16 copy, and their source was not identified), which cost more than the fused kernels saved. The same run showed the DDP all-reduce kernel time dropping from 61 to 36 ms/update, which is not explained either; the effect could not be separated from the regression, so the direction was dropped. The file is kept at /tmp/flat_layout.py.bak in case it is worth revisiting.
3.1 hRound 9 — plain `F.linear` for the megatron parallel layers (N0)
Profile of round 8 (/tmp/r8/profile): the forward of the model issues about 3300 kernels per update and the CPU (the python overhead of the megatron linear layers) is the bottleneck — the forward's wall time is 50.9 ms against about 35 ms of GPU work. Every linear layer of the model is aColumnParallelLinearor aRowParallelLinear; with--model-parallel-size 1their copy/reduce is the identity, but the code still goes through twoautograd.Functionnodes per layer and per direction, plus thereduce/copyplumbing.yoco/models/fast_layers.py(YOCO_OPT_FAST_LINEAR=1) patchesColumnParallelLinear.forwardandRowParallelLinear.forwardto a plainF.linearwhen the model-parallel world size is 1. The bias is still added separately (not fused into the GEMM) so that the accumulation order of the addition is unchanged — bitwise identical results. *time(/tmp/r14/time): 0.1653 s/update, speedup 1.4132 (from 1.4013).
3.1 hRound 10 — direct launch of the rotary Triton kernel (N0)
Triton's python launcher costs about 105 us per launch for the rotary kernel (20 arguments); the kernel itself is about 20 us.yoco/models/fast_triton.py(YOCO_OPT_FAST_TRITON=1) keeps theCompiledKernelobject and calls it directly (compiled.run(...)), skipping the JIT bookkeeping, the argument specialization and the cache lookup on every call. The launch key contains the constexpr values, the dtypes, the alignment of the pointers, the exact values of the integer arguments and(num_warps, num_stages). Used bydecoder/kernel/rotary.py; the kernel is verified bitwise equal to the reference launch by a probe. *time(/tmp/r15/time): 0.1631 s/update, speedup 1.4323 (from 1.4132).
3.1 hRound 11 — direct launch of the chunked-retention kernels (N0)
The same treatment fordecoder/kernel/gate_recurrent.py(the chunked recurrence of the gated retention): about 90 us of launch overhead per call for a kernel of a few tens of microseconds. *time(/tmp/r16/time_rec): 0.1611 s/update, speedup 1.4500 (from 1.4323). *check --quick(/tmp/r15/quick): PASS (0 errors, 0 exact failures).
3.1 hRound 12 — failed direction: a bigger DDP bucket
The DDP buckets are 25 MiB while a single FFN weight is 50 MB, so every bucket
holds about one FFN weight and the all-reduce is issued 119 times per update at
about 200 GB/s, well below the NVLink bandwidth. Overriding the bucket size
(YOCO_OPT_DDP_BUCKET_MB=100) was measured worse: 0.1638 s/update
(1.4261), because the bigger buckets serialize the all-reduce against the
gradient computation instead of overlapping it. The switch stays off (the
override itself was a no-op until commit a85d6ce, which fixed the bug but kept
the default off).3.1 hRound 13 — profile of the current best (1.4500)
bench.py profile --tree /submission/yoco --config /submission/switches.json --out /tmp/r13/profilePer update: the GPU is busy 142 ms of a 187 ms profiled window (about 25% idle, mostly in the model forward — the issue rate is CPU bound). GEMM 67 ms, NCCL 32 ms (fully overlapped), the fused Adam 25 ms, the memcpy/cast 15 ms, triton 9 ms, elementwise 8 ms, flash 6 ms.aten::mm/linear/matmuldispatch is about 25 ms of profiled CPU, the compiled RMSNorm about 12 ms. Five pageable HtoD copies per update cause about 2.2 ms of GPU idle; their source is not yet identified.
3.6 hRound 14 — the gradient buckets: no copy back into the parameters (N0)
Profile of the current best (/tmp/r17/profile, 1.4500 tree): 34.9 GiB of device-to-device copies per 5 updates, i.e. 7 GiB per update, in 240 copies of the size of the parameters. The trace shows where they come from: right after the lastncclDevKernel_AllReduce_Sum_bf16_RING_LLof the backward there is a run of 240 consecutiveMemcpy DtoDkernels (6.89 GiB), one per trainable parameter. That is torch's DDP copying the reduced buckets back intoparam.grad.YOCO_OPT_GRAD_BUCKET_VIEW=1setsgradient_as_bucket_view=Trueon the DDP (yoco/models/optim_overrides.py), so the gradient of a parameter *is* a view into the communication bucket: the reduced values are already where the parameter expects them and the copy disappears. The reduced gradients are the same values (the sum over the two ranks does not depend on where the buffer lives). *time(/tmp/r17/time_gbv): 0.1577 s/update, speedup 1.4813 (from 1.4500). * The loss of the timing run is the reference's:loss=16.399at update 10 andloss=14.135at update 20 (reference: 16.399 / 14.135). *check --quick(/tmp/r17/quick): PASS, 0 errors, 0 exact failures. The 1478 tensors that differ from the reference (the compiled RMSNorm) have exactly the same relative errors as before the change (0 of 1478 changed), so this change is numerically neutral.
4.3 hRound 15 — a fused RMSNorm of our own, in Triton (N1)
The compiled RMSNorm cost about 320 us of CPU time per call (146 us forward, 174 us backward) and is called 54 times per update: about 17 ms of the host time of an update, and the forward pass of the update is GPU bound, so the host time shows up directly in the wall clock.YOCO_OPT_TRITON_RMSNORM=1replaces it with one Triton kernel per direction (yoco/models/triton_rmsnorm.py), keeping the rounding of the reference: the normalized value is rounded to bf16 *before* it is multiplied by the weight, and the gradient of the weight is the sum of the bf16 products of the incoming gradient and the rounded normalized value. Against the eager reference, 29 of 6.29M elements of the output and 67 of 6.29M elements of the input gradient differ by one bf16 ulp, and 6 of 3072 elements of the weight gradient differ by one ulp — that is the order of the reduction of the mean of the squares, and nothing else. This is *much* closer to the reference than the compiled version, which differed on almost every element. *time(/tmp/r22/time_triton): 0.1600 s/update, 1.4600 — *worse* than the compiled version. The profile (/tmp/r22/profile) showed why: the gradient of the weight was reduced by three programs only, and that kernel took 188 us per call (33 GB/s). The host time did not change (0.171 s per update): the GPU was the bottleneck, not the host.
4.3 hRound 16 — the gradient of the weight, reduced in two stages (N1)
The gradient of the weight is the sum over the rows of the products, so it is reduced over 2048 rows. The first version reduced it with one program per column block, of which there were only 48 for 3072 columns: 188 us per call. Now the rows are split in two stages (a partial sum per part, 16 parts, then the sum of the parts), which is 16 times more parallelism. *time(/tmp/r23/time_triton): 0.1528 s/update, 1.5288 (from 1.4813). *check --quick(/tmp/r23/quick): PASS, 0 errors, 0 exact failures. * The loss of the timing run is the reference's, and the two windows are within 0.06 % of each other (window_range0.0006).
4.9 hRound 17 — the table of the addresses of the fused kernels (N0)
The profile of the current best (/tmp/r24/profile) shows where the update spends its time: the GPU is busy 136 ms of a 164 ms window (about 16 ms of idle), the kernels are 67 ms of GEMM (at 620 TFLOP/s, which is 85 % of the 733 TFLOP/s a big square GEMM reaches on this machine), 32 ms of NCCL (87 % overlapped with the backward), 25 ms of the fused Adam (at 4.1 TB/s, which is 95 % of the 4.3 TB/s a copy reaches) and 45 ms of everything else. The host side: the tail of the update costs 2.3 ms of host time in the fused Adam alone, and the profile attributes 1.7 ms of it to Python that is not a tensor operation: writing the addresses of the 165 parameters (five pointers each) into the table of the kernel, 825 assignments of a numpy buffer, every update. The addresses do not move.YOCO_OPT_CACHED_PTR_TABLE=1writes the table only when an address changed. The values are the same, so this is N0. *time(/tmp/r26/time): 0.1540 s/update, 1.5169. *timeof the previous best right after (/tmp/r26/time_prev): 0.1529 s/update, 1.5278 — the previous best measured 1.5358 earlier, so the run to run spread of the candidate runs is about 0.5 %, twice the spread of the reference runs. Changes of less than 1 % cannot be told apart from the noise with a single run.
5.4 hRound 18 — the size of the DDP gradient buckets (N0)
With 25 MiB buckets (torch's default) the reduction of the 6.3 GiB of bf16 gradients is 119 small NCCL calls per update; every one of them has a fixed cost and the two ranks have to meet at each of them.YOCO_OPT_DDP_BUCKET_MBsets the size of the buckets (fairseq passesbucket_cap_mbto torch's DDP explicitly, so the value has to be overridden, not defaulted — the first version of the switch did nothing, see the commita85d6ce). * 50 MiB: 0.1508 s/update, 1.5491 * 100 MiB: 0.1505 / 0.1506 s/update, 1.5522 / 1.5511 * 200 MiB: 0.1506 s/update, 1.5511 The size is saturated between 100 and 200 MiB, so 100 MiB is kept. The values of the reduced gradients do not depend on the size of the buckets, so this is N0.
5.4 hRound 19 — the full check of the current tree (`/tmp/r29/check`)
check --tree /submission/yoco --config /submission/switches.json: * PASS,performance.speedup= 1.5532, tier N1. * Gate 2 (bitwise): 0 errors, 0 exact failures; 1271 of 9616 recorded tensors are bitwise identical, the rest are the ones the compiled RMSNorm touches. * Gate 3 (statistical): all six endpoints within the bound, the worst one (loss_u050_100) at 0.96 σ of the 3 σ limit. * The timing run of the check: 0.1505 s/update over the two windows.
6.2 hRound 21 — the order of the gradient buckets (N0, no gain)
Profile. The trace of the last update of/tmp/r24/profile(164 ms profiled, 150 ms real) gives the timeline of the update: | phase | window (ms) | note | |---|---|---| | forward | 2.8 – 38.9 | 835 kernels, the CPU issues them faster than the GPU runs them | | backward | 38.9 – 112.5 | the gradient reduction runs from 50 to 124 ms | | *gap* | 112.5 – 124.8 | 12.3 ms with nothing to do: the reduction of the last buckets | | clip + cast | 124.8 – 135.1 | the L2 norm (3 ms) and the fused cast and scale (5.2 ms) | | Adam | 135.1 – 162 | the fused Adam, 25 ms, at 4.1 TB/s | | tail | 162 – 164.3 | the event based update timing of the bench | The 122 NCCL kernels of the reduction are 0.2-0.35 ms each (100 MiB at the full 400 GB/s) with 0.2-1 ms gaps: the reduction is not bandwidth limited but *latency* limited, it waits for the gradients. The gradients of the last 1.2 GiB (the output projection and the embedding) are only ready at the end of the backward, so their reduction is not hidden: 12.3 ms of the update are the reduction of gradients that were ready long before. Change. torch's DDP fills the buckets in the order of the registration of the parameters and then reverses the list of the buckets, so the bucket that is reduced first holds the gradients that the backward produces first. The YOCO model registers its parameters in the order of the forward pass, so the gradient of the output projection -- the first one the backward produces -- is registered second and lands in one of the last buckets.YOCO_OPT_DDP_BUCKET_ORDER=1(yoco/models/optim_overrides.py) moves the parameters of the output projection to the end of the list that the reducer sees, so that their gradient is reduced first. The reduced values do not depend on the grouping, so this is N0. *time(/tmp/r21/time): 0.1503 s/update, speedup 1.5542 (the current best is 0.1505 / 1.5532): no gain, the difference is inside the noise (0.5 %). * The switch stays off. So the 12.3 ms are not the order of the buckets: the reduction of the last buckets is exposed because it cannot start before the gradients are ready, and the gradients of the output projection are ready *late* in the backward as well (the wgrad of the output projection is the first GEMM of the backward, but the reduction of the buckets is a chain: a bucket is only reduced when the previous one is done).
6.7 hRound 22 — where the time goes now (the profile of `/tmp/r24/profile`)
Profile. The trace of the last update of/tmp/r24/profile(164 ms profiled, 150 ms real) gives the timeline of the update: | phase | window (ms) | note | |---|---|---| | forward | 2.8 – 38.9 | 835 kernels, the CPU issues them at the same rate as the GPU runs them | | backward | 38.9 – 112.5 | the reduction runs from 50 to 124 ms | | tail of the reduction | 112.5 – 124.8 | the last buckets of the reduction, overlapped with the backward | | clip + cast | 124.8 – 135.1 | the L2 norm (3 ms) and the fused cast and scale (5.2 ms) | | Adam | 135.1 – 162 | the fused Adam, 25 ms, at 4.1 TB/s | | tail | 162 – 164.3 | the event based update timing of the bench | The GPU is busy 136 ms of the 164 ms of the window: 28 ms of idle, in 2496 gaps (2394 of them shorter than 50 us). Twelve of those milliseconds are in the forward, which is exactly at the limit of the host: 24.4 ms of host time for 23.0 ms of GPU work. The kernels of the update (rank 0): 67 ms of GEMM (575 TFLOP/s, 78 % of the 733 TFLOP/s a big square GEMM reaches on this machine), 34 ms of NCCL (the reduction of 6.9 GiB of bf16 gradients, 119 kernels of 0.2-0.35 ms, 87 % overlapped with the backward), 25 ms of the fused Adam, 15 ms of elementwise (411 multiplications, 268 copies, 231 additions), 5.6 ms of flash-attention, 5.2 ms of the fused cast, 3 ms of the L2 norm of apex, 1.7 ms of softmax. The elementwise work is the largest remaining item that is not a GEMM: it is what the retention of the self decoder does with the (1, 24, 8, 256, 256) tensors (the mask of the gates), 13 times in the forward and 13 times in the backward of an update. * A diagnostic (YOCO_SCREEN_PHASE_TIMING=1,yoco/models/screening.py) prints the host time of the forward of the criterion: 24.4 ms against 23.0 ms of GPU work, so the forward is balanced and the 13 ms of idle that the profiler shows there is an artefact of the profiler.
6.7 hRound 23 — the fused Adam reads the bf16 gradients (N0, +1 %)
Change. The fused cast writes the master gradient asfloat(bf16 gradient) * scale, so the Adam can read the bf16 gradient and multiply it by the same scale inside its kernel: the value is the same, bit for bit, and the kernel reads two bytes per element instead of four (28 instead of 30 bytes per element, 96 instead of 103 GiB per update). * The kernel has aBF16_GRADbranch (yoco/models/fused_grad.py), the scale is left in a device scalar by the fused cast and consumed by the Adam. * A unit test of the two paths of the kernel (/tmp/t3/t_adam.py) agrees bit for bit on every parameter, moment and weight. *time(/tmp/r32/time_bf16grad): 0.1491 s/update, speedup 1.5667 (the previous best was 0.1505 / 1.5532). The same session's run of the previous best was noisy (0.1531, window range 2.9 %), its median was 0.1509. *check(/tmp/r33/check): PASS, speedup 1.5646, tier N1 (the trajectory's worst endpoint is 0.09 sigma of the 3 sigma bound). The switchYOCO_OPT_ADAM_BF16_GRAD=1is now in/submission/switches.json.
6.7 hRound 24 — no copy of the Adam states that do not move (N0, no gain)
The copy of apex (state["exp_avg"].to(device=..., dtype=...)) returns the same tensor when the device and the dtype do not change, but its dispatch costs a few microseconds per parameter and per moment. The check avoids it. *time(/tmp/r32/time_bf16grad2): 0.1493 s/update, speedup 1.5646: **no measurable gain**. The 2.2 ms of host time that the profile attributes to the step of the optimizer is an artefact of the profiler (it inflates the host time of every operation). The change is kept: it is bit for bit the same code path and it cannot hurt.
6.7 hCurrent summary (after round 24)
Switch file (/submission/switches.json): ```json {"environment": {"YOCO_OPT_REPLICATED_ADAM": "1", "YOCO_OPT_SKIP_MASTER_ZERO": "1", "YOCO_OPT_FUSED_GRAD_SCALE": "1", "YOCO_OPT_FUSED_WEIGHT_CAST": "1", "YOCO_OPT_FUSED_ADAM": "1", "YOCO_OPT_FAST_LINEAR": "1", "YOCO_OPT_FAST_TRITON": "1", "YOCO_OPT_GRAD_BUCKET_VIEW": "1", "YOCO_OPT_TRITON_RMSNORM": "1", "YOCO_OPT_CACHED_PTR_TABLE": "1", "YOCO_OPT_DDP_BUCKET_MB": "100", "YOCO_OPT_ADAM_BF16_GRAD": "1"}} ``` Latest full check:/tmp/r33/check/check.json— PASS, speedup 1.5646, tier N1 (gate 3: the worst endpoint is 0.09 σ of the 3 σ bound). Where the gain comes from (frozen reference: 0.2336 s/update): | change | s/update | speedup | tier | |---|---|---|---| | reference | 0.2336 | 1.000 | – | | replicated Adam (no ZeRO broadcast) + no master zero | 0.2095 | 1.115 | N0 | | compiled RMSNorm | 0.1987 | 1.176 | N1 | | fused gradient cast and scale | 0.1803 | 1.296 | N0 | | fused master → bf16 weight copy | 0.1747 | 1.337 | N0 | | fused Adam (also writes the bf16 weights) | 0.1667 | 1.401 | N0 | | plainF.linearfor the megatron linears | 0.1653 | 1.413 | N0 | | direct launch of the rotary kernel | 0.1631 | 1.432 | N0 | | direct launch of the recurrence kernels | 0.1611 | 1.450 | N0 | | DDPgradient_as_bucket_view| 0.1577 | 1.481 | N0 | | fused Triton RMSNorm | 0.1528 | 1.529 | N1 | | DDP buckets of 100 MiB | 0.1505 | 1.553 | N0 | | the fused Adam reads the bf16 gradients | 0.1491 | 1.567 | N0 | Directions that did not work (and why): * CUDA graphs: the DDP reducer's host side logic and the autograd graph cannot be captured; a graph of the forward only would keep the activations of the previous iteration alive and the peak memory (73 GiB) is already close to the limit. * A Triton flash kernel forinner_chunk: analysed, see round 20. * Eager instead of compiledinner_chunk: 228 us of GPU per call instead of 68 us. *NCCL_BUFFSIZE=32 MB: no gain (0.1531 s/update). * A bigger DDP bucket (200 MiB) or a smaller one (50 MiB): no gain. * The table of the addresses of the fused kernels (round 17): no measurable gain, the run to run noise of the candidate (0.5 %) is larger than the effect; the change is kept because it cannot hurt. * The order of the DDP gradient buckets (round 21): no gain. * The copy of the Adam states that do not move (round 24): no gain. Where the time goes now (profile/tmp/r24/profile, 5 updates, rank 0): GPU busy 136 ms of a 164 ms window — 28 ms of idle, mostly in the forward, which is exactly at the limit of the host (24.4 ms of host time for 23.0 ms of GPU work). Kernels: 67 ms of GEMM (575 TFLOP/s, 78 % of the 733 TFLOP/s a big square GEMM reaches on this machine), 34 ms of NCCL (87 % overlapped with the backward), 25 ms of the fused Adam (4.1 TB/s, 95 % of the 4.3 TB/s a copy reaches), 15 ms of elementwise (the retention of the self decoder and its mask), 5.6 ms of flash-attention, 5.2 ms of the fused gradient cast, 3 ms of the L2 norm of apex, 1.7 ms of softmax.
7.6 hRound 25 — where the time goes now, and three experiments that did not pay
Profile of the current best (/tmp/r35/profile, 5 updates, rank 0). The main stream (the one the model runs on) has 128.2 ms of kernels in a 160.5 ms window; the update is therefore bound by the main stream and by the time the gradients are reduced: | phase | window | note | |---|---|---| | forward | 3.0 – 40.0 ms | 37 ms, the host is the limit here (24 ms of host time for 23 ms of GPU work) | | backward | 40.0 – 106.8 ms | 67 ms | | *idle* | 106.8 – 123.5 ms | 16.7 ms: the host waits for the reduction of the gradients | | clip + cast | 123.5 – 133.6 ms | the L2 norm (3.0 ms) and the fused cast (5.2 ms) | | Adam | 133.6 – 160.3 ms | the fused Adam, 23.6 ms at 4.1 TB/s | The kernels of the main stream per update: 71.7 ms of GEMM (78 % of the peak), 23.6 ms of fused Adam, 14.1 ms of elementwise (**4.7 ms of it is the division that DDP does on every gradient before the reduction**:mul_outis called 240 times per update), 5.2 ms of fused cast, 3.0 ms of L2 norm, 2.8 ms of Triton, 2.1 ms of swiglu, 1.7 ms of softmax. The reduction of the gradients: 47 NCCL calls of 100 MiB, from 51.5 ms to 122.7 ms. The pure NCCL cost of the same reduction is 33 ms (measured with a standalone benchmark: 47 x 100 MiB, 296 GB/s), so the reduction is not bandwidth bound but *gated by the production of the gradients*: the last bucket (the embedding, 616 MiB) is only ready at the end of the backward, and the chain of the buckets that follows it adds 16 ms to the update. Experiment 1:YOCO_OPT_NO_SETITEM_SYNC=1(N0, no gain). The backward of the chunked recurrence writes two slices withDI[:, :, -1] = 0, which goes throughindex_put_and reads the value of the mask from the device: a synchronization, twice per decoder layer. The switch (already in the tree, committed inb6cce5c) replaces them withzero_(). *time(/tmp/r36/time_setitem): 0.1494 s/update, 1.5636 — the same as the previous best (0.1491 / 1.5667). The switch stays off. Experiment 2:YOCO_OPT_FAST_EMBEDDING=1(N0, no gain). fairseq's megatronVocabParallelEmbedding.forwardmasks the tokens withmasked_input[input_mask] = 0, andindex_put_reads the value of the mask from the device: a synchronization in the middle of the forward pass. The new module (yoco/models/fast_embedding.py) writes the same zeros withmasked_fill_, which is whatindex_put_does internally, without the synchronization (a unit test: the values and the gradients are bit for bit the same). *time(/tmp/r36/time_emb): 0.1495 s/update, 1.5625 — no gain. The switch stays off. Experiment 3: the Triton inner chunk (already in the tree,c68fe59). The profile withYOCO_OPT_TRITON_INNER_CHUNK=1(/tmp/r36/profile_inner) shows that the elementwise work of the update does not change (14.1 ms -> 15.1 ms): the mask of the gates is not what the elementwise kernels do. The triton category shrinks by 0.4 ms (the compiled inner chunk is replaced by the Triton one) and the window by 3 ms, but the measured time does not move.
7.6 hRound 26 — the fused Adam writes the master gradients (N0, +1.4 %)
Change. The fused cast writes the master gradient of every parameter asfloat(bf16 gradient) * scale, and the fused Adam reads the bf16 gradient and multiplies it by the same scale, so the master gradient is written but never read: 12.4 GiB of writes and 6.9 GiB of reads that the update does not need.YOCO_OPT_FUSED_MASTER_GRAD=1lets the fused Adam write the master gradient itself, in the same kernel that reads the bf16 gradient: the gradients are read once instead of twice, and the master gradient is written once. The value is the same expression (float(bf16 gradient) * scale, in fp32, with the same device scalar), so it is bit for bit the value the fused cast wrote. * A unit test (/tmp/t9/t_fused.py) compares the two paths on random gradients: the master gradients, the master weights, the two moments and the bf16 weights are all bit for bit equal. *time(/tmp/r37/time_mastergrad2): 0.1471 s/update, 1.5880 (the previous best was 0.1491 / 1.5667). The first run of the same tree (/tmp/r37/time_mastergrad) gave 0.1502 / 1.5553, but its two windows differ by 2.8 %, so it is not a valid measurement. * The profile (/tmp/r37/profile) shows what the change does: the fused cast is gone (5.24 ms), the fused Adam grows from 23.61 ms to 27.13 ms (it writes the 12.4 GiB of master gradients), and the main stream has 126.2 ms of kernels instead of 128.2 ms. * Result: INCORRECT (check --quick,/tmp/r37/quick/check.json). The judge records the fp32 master gradient *before*optimizer.step(), so a master gradient that is only written inside the Adam kernel is never seen: everygrad/adam#0/master.*tensor had a relative error of about 1.0 and thecross_rank_patternexact item failed (the reference's master gradients are rank-local slices of the flat fp32 buffer, the candidate's became replicated zeros). **Lesson: the fp32 master gradient has to be materialized before the optimizer step.** The change was reverted (bb9e076is the last commit that does not contain it).
8.2 hRound 27 — the gradient norm in one kernel (N1, +0.5 %)
Change.clip_grad_normasked apex for the norm of the 240 gradients withmulti_tensor_total_norm, which is 165 launches of a small kernel (3.0 ms of GPU time per update) and a lot of host time in the middle of the update.YOCO_OPT_FUSED_L2NORM=1computes the norm with the table of the addresses that the fused cast already keeps: one Triton kernel sums the squares of every chunk of every gradient into a partial sum and a second one sums the partials (two launches for the whole model). The sum is accumulated in a different order than apex's, so the norm differs in the last bits of the fp32 mantissa and so does the clipping coefficient: the tier is N1 (the deviation is about 1e-7 relative, against the 1e-2 of the Triton RMSNorm that the tree already has). * A unit test (/tmp/t10/t_l2.py) compares the norm with apex's on random gradients and with the real ones: the values are bit for bit equal in the test, and the cast that shares the table is unchanged. *time(/tmp/r38/time_l2): 0.1481 s/update, 1.5773; the same tree without the switch (/tmp/r38/time_base): 0.1488 s/update, 1.5699. *check --quick(/tmp/r39/quick/check.json): PASS, tier N1, no exact failure (1478 tensors beyond the per-tensor tolerances, which are diagnostic only).
8.5 hRound 28 — the plan of the Adam update is kept between the updates (N0, +1.7 %)
Profile. The profile of the current tree (/tmp/r37/profile) shows that the gap between the end of the backward and the start of the Adam kernel is pure Python: the fused Adam rebuilt, at every update, five lists of 240 tensors, collected 1200 addresses withdata_ptr(), rebuilt the chunk table of the parameters and went through the bookkeeping of the Triton JIT (which re-binds the arguments and rebuilds the specialization key at every call). The host is the critical path there: the GPU waits for the plan before it can run the kernel. Change. The plan (the tensors, the table of their addresses, the chunk table) is built once and reused as long as the gradients and the Adam states keep their buffers; the kernel is launched directly, without the JIT bookkeeping. Triton does not specialize on the value of a floating point argument, so the compiled kernel is the same even though the learning rate and the bias corrections change at every update. * A unit test (/tmp/t11/t_plan.py) compares the rebuild and the cached plan over five updates: the weights, the gradients, the two moments and the bf16 weights are bit for bit equal. The plan is rebuilt when a gradient or an Adam state moves. *check --quick(/tmp/r41/quick/check.json): PASS, and the summary of the numerics is *identical* to the one of the run without the change (9616 tensors checked, 1272 bitwise identical, 1478 beyond the per-tensor tolerances, 0 exact failure): the change is bitwise neutral. *time(/tmp/r40/time_new): 0.1468 s/update, 1.5913; the same tree without the three switches (/tmp/r40/time_base2): 0.1493 / 1.5646. A trap. The first version of the change rebuilt the chunk table at every update. The table is uploaded with a pageable buffer, so the copy blocks the host until the stream has caught up: the update went from 0.1493 to 0.1682 s/update. The chunk table is now built once, with the plan. *check(full,/tmp/r42/check/check.json): PASS, speedup 1.5491 (the candidate run of the check measured 0.1508 s/update), tier N1. The switch file of the submission now has the fifteen switches of the tree (the twelve of the previous rounds plusYOCO_OPT_FUSED_L2NORM,YOCO_OPT_ADAM_PLANandYOCO_OPT_FAST_ADAM).
8.9 hRound 29 — the statistics of the update, without reading them back (N0, no gain)
Profile (/tmp/r43/profile, the current tree, 5 updates, 187 ms/update under the profiler): the GPU is busy 133 ms of the 160 ms of the update; 160 ms of kernels run in that window, so 27 ms of the NCCL reduction is hidden behind the computation and about 14 ms of the update (in the unprofiled run) is GPU idle. The end of the update holds a burst of 16 device-to-host copies: the meters of the statistics are accumulated on the device and read back on the host, once per meter, at the end of every update (AverageMeter.smoothed_valuerounds the value on the host; the average of the meter that is weighted by the number of tokens tests its count on the host;torch.isfinite(grad_norm)reads the norm). Change (yoco/models/fast_metrics.py,YOCO_OPT_FAST_METRICS=1): the values of the meters stay on the device. The average of a meter is selected on the device (torch.where(count > 0, sum / count, val)) instead of testing the count on the host -- the value is the same tensor as fairseq's -- and the logging output holds the values as they are on the device. The training loop only tests that the output is not None; the numbers that are written to the log everylog_intervalupdates come from the "train_inner" aggregator, whose meters are accumulated exactly as before (the logged loss and perplexity of a run are the same as the reference's, up to the run-to-run noise of the loss itself). *time(/tmp/r43/time_fast): 0.1471 s/update, speedup 1.5880; the same tree without the switch measured 0.1468 / 1.5913 in the same session (/tmp/r40/time_new) and 0.1508 / 1.5491 in the last full check. No gain: the syncs of the statistics are not on the critical path, the host has work to do while the device finishes the Adam anyway. * A unit test (/tmp/t_fastmetrics.py): the average of a meter is the same tensor as fairseq's, and taking it does not synchronize the device (torch.cuda.set_sync_debug_mode("error")). Conclusion. The 2.7 ms that the screening switchYOCO_SCREEN_NO_METRICS=1showed in round 40 was the noise of the run, not the cost of the statistics: with the syncs removed and the accumulation kept, the update time does not move. The change is kept in the tree, off by default (it is bitwise neutral and cannot hurt, but it buys nothing).
9.3 hRound 30 — the batch of the loader in pinned memory (N0, no gain) and the phases of the update
Profile (/tmp/r43/profile, the current tree): 128.5 ms of kernels on the compute stream in an update of 146.7 ms: 66 ms of GEMM, 31.5 ms of NCCL (86 % overlapped), 23.6 ms of the fused Adam (at the memory roofline), 15 ms of elementwise (about 1500 launches), 6.4 ms of flash attention, 5.2 ms of the fused cast of the master gradients, 2.8 ms of Triton, 2.1 ms of the L2 norm, 2.1 ms of SwiGLU and 1.8 ms of the loss. Change 1 (yoco/tasks/data/basic_loader.py,YOCO_OPT_PINNED_BATCH=1): the tensors of the batch are allocated in pinned host memory, so the copy of the sample to the device (non_blocking=True) does not make the host wait for the device. The bytes are the same, so the data of the run is the same. *time(/tmp/r44/time_pin): 0.1471 s/update, 1.5880; the same tree without the switch in the same session (/tmp/r44/time_base): **0.1466 / 1.5935. No gain**: the copy is already hidden behind the work of the device. Change 2 (yoco/models/screening.py): the diagnosticYOCO_SCREEN_PHASE_TIMING=1now prints the host time of the phases of the update (train_step, prepare, forward, backward, optimizer, statistics) and the newYOCO_SCREEN_TRACE_EVENTS=1prints a Python stack for everytorch.cuda.Event.elapsed_timecall. * The phases of an update (host,YOCO_SCREEN_PHASE_TIMING=1): `train_step 146.67 ms = prepare 0.21 + forward 22.6 + backward 59.3 + optim 0.72 + stats 23.9` and about 40 ms of waiting for the device (the tail of the reduction, the clip, the norm and the cast of the master gradients). * The trace of the events (/tmp/r44/time_events): the 9387 calls ofcudaEventElapsedTimeall come from the autotuning of inductor (torch/_inductor/triton_heuristics.py:benchmark_all_configs) during the compilation, not from the update. The 3.4-5.6 ms gaps that the profile shows between the kernels are therefore artefacts of the profiler and of the compilation, not stalls of the steady state. Conclusion. Two directions closed: the pinned memory of the loader buys nothing, and the "gaps" of the profile are not worth chasing. The pinned loader is kept in the tree, off by default.