MiniMax M3.1 Flash 1.59×

Harness: Codex · switches: YOCO_OPT_REPLICATED_ADAM=1 YOCO_OPT_SKIP_MASTER_ZERO=1 YOCO_OPT_FUSED_GRAD_SCALE=1 YOCO_OPT_FUSED_WEIGHT_CAST=1 YOCO_OPT_FUSED_ADAM=1 YOCO_OPT_FAST_LINEAR=1 YOCO_OPT_FAST_TRITON=1 YOCO_OPT_GRAD_BUCKET_VIEW=1 YOCO_OPT_TRITON_RMSNORM=1 YOCO_OPT_CACHED_PTR_TABLE=1 YOCO_OPT_DDP_BUCKET_MB=100 YOCO_OPT_ADAM_BF16_GRAD=1 YOCO_OPT_FUSED_L2NORM=1 YOCO_OPT_ADAM_PLAN=1 YOCO_OPT_FAST_ADAM=1

Judged
1.59×
Tier
N1
Self-best
1.59×
Rounds
29
Hours
10.0
API cost
$6.03
Input tokens · 96.4% cached
235 M
Output tokens
2.3 M
Requests
1,820
Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
✗ no exchange: YOCO_OPT_REPLICATED_ADAM has both ranks run Adam on all params (1.00× → 1.11×); grad all-reduce still NCCL
Forward / backward CUDA graph
✗ judged infeasible, no measurement in the log
Newer cuBLAS
✗
Retention / attention kernel swap
✗ only rotary and recurrence kernels switched to direct launch (1.41× → 1.45×)
Other main levers
Triton fused "cast + scale" 1.18× → 1.30×; Triton Adam also writing bf16 weights 1.34× → 1.40×; Triton RMSNorm with two-level reduction 1.48× → 1.53×
Raw work log31 sections
0.2 hRound 1 — baseline profile (unchanged reference tree)
Start 2026-11-07 05:24, end 05:27.

bench.py profile --tree /submission/yoco --config /submission/switches.json --out /tmp/r1/profile
(0.2891 s/update under the profiler; the reference itself is 0.2336).

Kernel time by category, rank 0, 5 updates (profile.md):

| category | ms | ms/update |
|---|---|---|
| nccl | 435.7 | 87.1 |
| gemm | 330.8 | 66.2 |
| memcpy/cast | 219.2 | 43.8 |
| elementwise | 153.8 | 30.8 |
| optimizer (apex fused adam) | 91.4 | 18.3 |
| flash-attention | 29.8 | 6.0 |
| reduce | 19.2 | 3.8 |
| triton (rotary/recurrence/inductor) | 13.9 | 2.8 |
| jiterator (swiglu) | 11.1 | 2.2 |

Findings (from the Chrome trace, one update = 266 ms under the profiler):

* The GPU is essentially never idle (largest gap 2.2 ms) -> the update is GPU-bound, not
  data- or launch-bound.
* NCCL is 41% of the kernel time. 595 AllReduce_Sum_bf16 (DDP gradient all-reduce,
  119 calls/update, 47.5 ms/update) and 10 Broadcast_RING_LL (fairscale ZeRO-1 master-weight
  broadcast, 2 calls/update, 191.7 ms total = 38.3 ms/update).
* The ZeRO-1 broadcast is fully serial: fairscale::oss::refresh_trainable (23.5 ms/update) is
  preceded by a cudaDeviceSynchronize (23.3 ms) and is followed by a 44.7 ms
  cudaStreamSynchronize at the end of the update -> the broadcast is on the critical path.
* The copy traffic of the fp32 master weights: memset of the master grads (13.8 GiB), bf16->fp32
  grad copy (20.7 GiB), clip-coefficient multiply of the master grads (27.6 GiB), fp32->bf16
  weight copy (20.7 GiB) = ~83 GiB of memory traffic per update, all of it avoidable or fusable.
* The GEMMs run at ~575 TFLOP/s (the output projection at 787 TFLOP/s) -> there is little to win
  in the model's matmuls.
* aten::item syncs: 4 per update, all of them waiting for the GPU (the longest is the 44.7 ms
  one at the end of the update).

Conclusion: attack (1) the optimizer's inter-rank communication, (2) the master-weight copy
traffic, (3) the gradient all-reduce's efficiency. The model's matmuls and attention are fine.
0.2 hRound 2 — replicated Adam (no ZeRO-1 broadcast)
yoco/models/optim_overrides.py (new, imported by models/__init__.py), three switches, all off
by default:

* YOCO_OPT_REPLICATED_ADAM=1: replaces fairscale's ZeRO-1 wrapper with a variant that keeps the
  Adam state of every parameter on every rank and drops the per-step broadcast (and the device
  synchronize in front of it). DDP has already reduced the gradients, so both ranks compute the
  same update from the same inputs; the per-parameter arithmetic (apex fused Adam, fp32 master
  weights, fp32 gradients) is unchanged. Tier N0.
* YOCO_OPT_DDP_BUCKET_MB=<int>: DDP gradient bucket size (torch's default is 25 MiB).
* YOCO_OPT_SKIP_MASTER_ZERO=1: skip the memset of the fp32 master gradients (the copy that
  follows overwrites them completely). Tier N0.

Results (judge time, 2 x Hopper, each run ~1.5 min):

| configuration | s/update | speedup vs frozen reference | peak mem |
|---|---|---|---|
| reference (frozen) | 0.2336 | 1.000 | 61.75 GiB |
| YOCO_OPT_REPLICATED_ADAM=1 | 0.2106 | 1.1092 | 74.51 GiB |
| YOCO_OPT_REPLICATED_ADAM=1 + YOCO_OPT_SKIP_MASTER_ZERO=1 | 0.2097 | 1.1140 | 74.51 GiB |
| ..._REPLICATED_ADAM + YOCO_OPT_DDP_BUCKET_MB=256 | 0.2107 | 1.1087 | 74.51 GiB |

check --quick (gates 0+2, /tmp/r3/quick/check.json): PASS -> the change is bitwise identical to the
reference (tier N0). The DDP bucket size has no effect (torch's bucket is already large enough / the
all-reduce is bandwidth bound, not latency bound) -> drop it.

Kept in /submission/switches.json: YOCO_OPT_REPLICATED_ADAM=1, YOCO_OPT_SKIP_MASTER_ZERO=1
(1.114x). Cost: +12.8 GiB of Adam state per rank (replicated instead of sharded); the machine has
139.8 GiB per GPU, so there is room, but the headroom is smaller than the reference's.

Commit: b264b25 "Add environment-switched overrides for the fp32 optimizer step and DDP buckets (N0, off by default)".
0.9 hRound 3 — screening the remaining stages (`/tmp/r5`, `/tmp/r6`)
The profile of the current best (/tmp/r4/profile) showed (5 updates, profiler inflated, rank 0):
gemm 67 ms/update, nccl 58, memcpy/cast 44, optimizer 33, elementwise 26, flash 6, triton 14.
Two findings from the trace: the GPU is idle 24 % of the window (the CPU cannot keep 6200
kernel launches/update fed) and the NCCL work is 58 ms of kernel time but only ~17 ms of it
is not already overlapped with compute.

Screening runs (judge time, one run each; these switches *change what is computed*, they only
size the opportunities):

| screen | s/update | speedup | what it measures |
|---|---|---|---|
| base (round 2) | 0.2095 | 1.115 | - |
| YOCO_SCREEN_NO_MASTER2BF16=1 | 0.1996 | 1.170 | the fp32 master -> bf16 weight copy costs 9.9 ms |
| YOCO_SCREEN_NO_ADAM=1 | 0.1814 | 1.288 | apex's fused Adam costs 28.1 ms |
| YOCO_SCREEN_NO_ALLREDUCE=1 | 0.1874 | 1.247 | DDP's gradient all-reduce costs 22.1 ms on the critical path |
| YOCO_OPT_FOREACH_OPS=1 | 0.2319 | 1.007 | regression: torch's _foreach_copy_ has no fast path for mixed dtypes and falls back to a per-tensor loop -> reverted |

Failed direction: gradient_as_bucket_view=True for DDP (would have removed DDP's 240
copy_bucket_to_grad copies) hangs with fairseq, because FP16Optimizer.zero_grad sets
p.grad = None and the reducer then waits for gradients that never arrive. Abandoned.

Also checked and rejected: the two state["exp_avg"].to(...) calls per parameter in apex's
FusedAdamV2.step are *not* copies (verified with a standalone probe: the state tensors keep
their data pointer), so there is nothing to win there.
0.9 hRound 4 — fused RMSNorm (`torch.compile`), N1
yoco/models/decoder/rms_norm.py: the eager normalization materializes three full-size fp32
temporaries per call (x.float(), x.pow(2), the fp32 product) and the model calls it ~130
times per update.  With YOCO_OPT_COMPILE_RMSNORM=1 the same arithmetic is compiled into one
kernel per shape.  The reduction order of the mean of squares changes -> tier N1.

* time (/tmp/r7/time_compile): 0.1987 s/update, speedup 1.1756 (from 1.115).
* check --n1 (/tmp/r7/n1/check.json): PASS, gate 3 "equivalent"; every one of the six
  endpoints is within 0.81 sigma of the reference (the bound is 3 sigma).
* No torch.compile recompile-limit hit and no "skipping cudagraphs" message in the run log;
  the compilation happens before the first update (startup 66 s, same as the reference's).
1.4 hRound 5 — fused gradient cast-and-scale (Triton), N0
Profile of the current best (/tmp/r8/profile, 1.1756x) put the two memory-bound
optimizer kernels at the top of the update:

| kernel | calls | ms/update |
|---|---|---|
| unrolled_elementwise_kernel<direct_copy> (bf16 grad -> fp32 master grad) | 1200 | 12.9 |
| elementwise_kernel<BinaryFunctor<float>> (master grad *= scale) | 1200 | 12.3 |
| multi_tensor_apply<AdamFunctor> (apex Adam) | 825 | 28.7 |
| ncclDevKernel_AllReduce_Sum_bf16 | 595 | 60.9 (mostly overlapped) |

The cast reads 2 bytes and writes 4 bytes per element, the scale reads 4 and writes
4: 40.6 GB of HBM traffic per update for two passes over the model, in 480 small
kernels (one per parameter) that reach only ~1.0-1.9 TB/s.

yoco/models/fused_grad.py (YOCO_OPT_FUSED_GRAD_SCALE=1) does both in one Triton
kernel: it reads the bf16 gradient, converts it to fp32, multiplies it by the scale
and writes the fp32 master gradient.  All the parameters are handled by **one
launch** (the kernel walks a table of the parameter addresses, built once from the
shapes), so the launch overhead is one kernel per update instead of 480.

The scale (the data-parallel size over the sample size, times the clipping
coefficient) is only known after the gradient norm, so the norm is computed from the
bf16 gradients instead of from the fp32 master gradients.  apex's fused L2 norm was
probed to return the *same* value for a bf16 tensor and for its fp32 image
(/tmp/probe_l2.py, bitwise equal on four tensor lists), so the clipping
coefficient is unchanged.  The cast-and-scale is bitwise equal to
copy_ + mul_ (probed on four tensor sizes, /tmp/probe_engine.py).

* time (/tmp/r5/time_fused): 0.1803 s/update, speedup 1.2956 (from 1.1756).
* check --quick with the same switches but without the RMSNorm compile
  (/tmp/r5/quick_nofuse): PASS, tier N0 - 9616 recorded tensors bitwise
  identical to the reference, 0 exact failures.  So this change is bitwise
  neutral; the only non-bitwise change in the tree is the compiled RMSNorm (N1,
  already validated by check --n1).
2.3 hRound 6 — fused fp32-master -> bf16-weight copy (Triton), N0
The fp32 master weights have to be copied into the bf16 model parameters after every
optimizer step: 12.6 GiB read + 6.3 GiB written per update, in 240 kernels that
reach only about 2 TB/s.  yoco/models/fused_grad.py (YOCO_OPT_FUSED_WEIGHT_CAST=1)
does the copy for every parameter in one Triton launch (the kernel walks a table
of the parameter addresses, built once from the shapes).

* time (/tmp/r6/time_wc): 0.1747 s/update, speedup 1.3371 (from 1.2956).
* check --quick (/tmp/r6/quick_n0): PASS, tier N0 (9616 tensors bitwise).
2.3 hRound 7 — fused Adam (Triton), N0
The apex fused Adam was the largest single kernel of the update: 825
multi_tensor_apply launches per update, 28.7 ms of kernel time at about 2.8 TB/s.
apex's FusedAdamV2 runs in ADAM_MODE_1 (decoupled weight decay):

    m = beta1*m + (1-beta1)*g ; v = beta2*v + (1-beta2)*g*g
    m_hat = m/bc1 ; v_hat = v/bc2 ; denom = sqrt(v_hat) + eps
    update = m_hat/denom + decay*p ; p = p - lr*update

with bc = 1 - pow((double)float(beta), step) — the beta is rounded to fp32 first,
which is what apex does (float beta = ... then std::pow((double)float_beta, step));
using the Python double beta gives a different bias correction and a different result.

yoco/models/fused_grad.py (YOCO_OPT_FUSED_ADAM=1) replaces
fairseq.optim.fused_adam.FusedAdamV2.step with one Triton launch for all the
parameters (4.1 TB/s, measured over 26 M elements).  The kernel uses
tl.math.fma(beta1, m, (1-beta1)*g) (nvcc contracts the same way), plain division
and tl.sqrt (not div_rn/sqrt_rn), and it writes the **bf16 model weights in the
same pass**, so the separate master -> model copy of round 6 is no longer needed
(the weight-cast override then skips its copy).

* time (/tmp/r8/time_adam): 0.1667 s/update, speedup 1.4013 (from 1.3371).
* check --quick without the RMSNorm compile (/tmp/r10/quick_n0): PASS, tier
  N0 — 9616 recorded tensors bitwise identical to the reference, 0 exact failures.
* Full check with the submitted switch file (/tmp/r11/check): PASS,
  speedup 1.4013, tier N1 (the only non-bitwise change is the compiled
  RMSNorm); gate 3: every one of the six endpoints within 0.81 sigma (bound 3 sigma).
2.3 hRound 8 — where the time goes now (profile of the current best)
The trace of the reference (/tmp/r8/profile) says the update is **launch bound in
the forward pass**: per update the GPU is busy 175 ms (profiled) out of a 225 ms
window and idle 50-67 ms, and most of that idle is inside the model forward
(## self.mixer 19 ms/update, self_decoder 3.8, cross_decoder 3.2).  The
backward is GPU bound (its compute is longer than the CPU work that drives it).

Two causes:

* the forward issues about 3300 kernels per update (780 of them GEMMs) and the
  Python overhead of megatron's parallel linear layers (two autograd.Function
  nodes per layer and per direction, for a copy/reduce that is the identity with
  one model-parallel rank) leaves the GPU waiting;
* 119 NCCL all-reduce calls per update (the DDP buckets are 25 MiB, but a single
  FFN weight is 50 MB, so every bucket holds about one FFN weight): 59 ms of
  NCCL kernel time at about 200 GB/s, well below the NVLink bandwidth.
1.4 hRound 5b — failed direction: flat parameter layout
yoco/models/flat_layout.py (now removed) laid the model parameters, the fp32
master weights and the master gradients out in one buffer each, so that the
master -> bf16 copy and the gradient scaling became single kernels.  Measured
0.2180 s/update (a regression of 19 ms).  The profile of that run
(/tmp/r9/profile) explains why: the optimizer phase ran 720 gradient copies
per update instead of 240 (the extra ones are in the optimizer range, i.e. in
the master -> bf16 copy, and their source was not identified), which cost more
than the fused kernels saved.  The same run showed the DDP all-reduce kernel time
dropping from 61 to 36 ms/update, which is not explained either; the effect could
not be separated from the regression, so the direction was dropped.  The file is
kept at /tmp/flat_layout.py.bak in case it is worth revisiting.
3.1 hRound 9 — plain `F.linear` for the megatron parallel layers (N0)
Profile of round 8 (/tmp/r8/profile): the forward of the model issues about 3300
kernels per update and the CPU (the python overhead of the megatron linear layers)
is the bottleneck — the forward's wall time is 50.9 ms against about 35 ms of GPU
work.  Every linear layer of the model is a ColumnParallelLinear or a
RowParallelLinear; with --model-parallel-size 1 their copy/reduce is the
identity, but the code still goes through two autograd.Function nodes per layer
and per direction, plus the reduce/copy plumbing.

yoco/models/fast_layers.py (YOCO_OPT_FAST_LINEAR=1) patches
ColumnParallelLinear.forward and RowParallelLinear.forward to a plain
F.linear when the model-parallel world size is 1.  The bias is still added
separately (not fused into the GEMM) so that the accumulation order of the
addition is unchanged — bitwise identical results.

* time (/tmp/r14/time): 0.1653 s/update, speedup 1.4132 (from 1.4013).
3.1 hRound 10 — direct launch of the rotary Triton kernel (N0)
Triton's python launcher costs about 105 us per launch for the rotary kernel
(20 arguments); the kernel itself is about 20 us.  yoco/models/fast_triton.py
(YOCO_OPT_FAST_TRITON=1) keeps the CompiledKernel object and calls it
directly (compiled.run(...)), skipping the JIT bookkeeping, the argument
specialization and the cache lookup on every call.  The launch key contains the
constexpr values, the dtypes, the alignment of the pointers, the exact values of
the integer arguments and (num_warps, num_stages).

Used by decoder/kernel/rotary.py; the kernel is verified bitwise equal to the
reference launch by a probe.

* time (/tmp/r15/time): 0.1631 s/update, speedup 1.4323 (from 1.4132).
3.1 hRound 11 — direct launch of the chunked-retention kernels (N0)
The same treatment for decoder/kernel/gate_recurrent.py (the chunked
recurrence of the gated retention): about 90 us of launch overhead per call for a
kernel of a few tens of microseconds.

* time (/tmp/r16/time_rec): 0.1611 s/update, speedup 1.4500 (from 1.4323).
* check --quick (/tmp/r15/quick): PASS (0 errors, 0 exact failures).
3.1 hRound 12 — failed direction: a bigger DDP bucket
The DDP buckets are 25 MiB while a single FFN weight is 50 MB, so every bucket
holds about one FFN weight and the all-reduce is issued 119 times per update at
about 200 GB/s, well below the NVLink bandwidth.  Overriding the bucket size
(YOCO_OPT_DDP_BUCKET_MB=100) was measured worse: 0.1638 s/update
(1.4261), because the bigger buckets serialize the all-reduce against the
gradient computation instead of overlapping it.  The switch stays off (the
override itself was a no-op until commit a85d6ce, which fixed the bug but kept
the default off).
3.1 hRound 13 — profile of the current best (1.4500)
bench.py profile --tree /submission/yoco --config /submission/switches.json --out /tmp/r13/profile

Per update: the GPU is busy 142 ms of a 187 ms profiled window (about 25% idle,
mostly in the model forward — the issue rate is CPU bound).  GEMM 67 ms, NCCL 32 ms
(fully overlapped), the fused Adam 25 ms, the memcpy/cast 15 ms, triton 9 ms,
elementwise 8 ms, flash 6 ms.  aten::mm/linear/matmul dispatch is about
25 ms of profiled CPU, the compiled RMSNorm about 12 ms.  Five pageable HtoD
copies per update cause about 2.2 ms of GPU idle; their source is not yet
identified.
3.6 hRound 14 — the gradient buckets: no copy back into the parameters (N0)
Profile of the current best (/tmp/r17/profile, 1.4500 tree): 34.9 GiB of
device-to-device copies per 5 updates, i.e. 7 GiB per update, in 240 copies of
the size of the parameters.  The trace shows where they come from: right after the
last ncclDevKernel_AllReduce_Sum_bf16_RING_LL of the backward there is a run of
240 consecutive Memcpy DtoD kernels (6.89 GiB), one per trainable parameter.
That is torch's DDP copying the reduced buckets back into param.grad.

YOCO_OPT_GRAD_BUCKET_VIEW=1 sets gradient_as_bucket_view=True on the DDP
(yoco/models/optim_overrides.py), so the gradient of a parameter *is* a view into
the communication bucket: the reduced values are already where the parameter
expects them and the copy disappears.  The reduced gradients are the same values
(the sum over the two ranks does not depend on where the buffer lives).

* time (/tmp/r17/time_gbv): 0.1577 s/update, speedup 1.4813 (from 1.4500).
* The loss of the timing run is the reference's: loss=16.399 at update 10 and
  loss=14.135 at update 20 (reference: 16.399 / 14.135).
* check --quick (/tmp/r17/quick): PASS, 0 errors, 0 exact failures.  The
  1478 tensors that differ from the reference (the compiled RMSNorm) have exactly
  the same relative errors as before the change (0 of 1478 changed), so this
  change is numerically neutral.
4.3 hRound 15 — a fused RMSNorm of our own, in Triton (N1)
The compiled RMSNorm cost about 320 us of CPU time per call (146 us forward,
174 us backward) and is called 54 times per update: about 17 ms of the host
time of an update, and the forward pass of the update is GPU bound, so the
host time shows up directly in the wall clock.

YOCO_OPT_TRITON_RMSNORM=1 replaces it with one Triton kernel per direction
(yoco/models/triton_rmsnorm.py), keeping the rounding of the reference: the
normalized value is rounded to bf16 *before* it is multiplied by the weight,
and the gradient of the weight is the sum of the bf16 products of the incoming
gradient and the rounded normalized value.  Against the eager reference, 29 of
6.29M elements of the output and 67 of 6.29M elements of the input gradient
differ by one bf16 ulp, and 6 of 3072 elements of the weight gradient differ by
one ulp — that is the order of the reduction of the mean of the squares, and
nothing else.  This is *much* closer to the reference than the compiled
version, which differed on almost every element.

* time (/tmp/r22/time_triton): 0.1600 s/update, 1.4600 — *worse* than
  the compiled version.  The profile (/tmp/r22/profile) showed why: the
  gradient of the weight was reduced by three programs only, and that kernel
  took 188 us per call (33 GB/s).  The host time did not change (0.171 s per
  update): the GPU was the bottleneck, not the host.
4.3 hRound 16 — the gradient of the weight, reduced in two stages (N1)
The gradient of the weight is the sum over the rows of the products, so it is
reduced over 2048 rows.  The first version reduced it with one program per
column block, of which there were only 48 for 3072 columns: 188 us per call.
Now the rows are split in two stages (a partial sum per part, 16 parts, then the
sum of the parts), which is 16 times more parallelism.

* time (/tmp/r23/time_triton): 0.1528 s/update, 1.5288 (from 1.4813).
* check --quick (/tmp/r23/quick): PASS, 0 errors, 0 exact failures.
* The loss of the timing run is the reference's, and the two windows are within
  0.06 % of each other (window_range 0.0006).
4.9 hRound 17 — the table of the addresses of the fused kernels (N0)
The profile of the current best (/tmp/r24/profile) shows where the update
spends its time: the GPU is busy 136 ms of a 164 ms window (about 16 ms of
idle), the kernels are 67 ms of GEMM (at 620 TFLOP/s, which is 85 % of the
733 TFLOP/s a big square GEMM reaches on this machine), 32 ms of NCCL (87 %
overlapped with the backward), 25 ms of the fused Adam (at 4.1 TB/s, which is
95 % of the 4.3 TB/s a copy reaches) and 45 ms of everything else.

The host side: the tail of the update costs 2.3 ms of host time in the fused
Adam alone, and the profile attributes 1.7 ms of it to Python that is not a
tensor operation: writing the addresses of the 165 parameters (five pointers
each) into the table of the kernel, 825 assignments of a numpy buffer, every
update.  The addresses do not move.

YOCO_OPT_CACHED_PTR_TABLE=1 writes the table only when an address changed.
The values are the same, so this is N0.

* time (/tmp/r26/time): 0.1540 s/update, 1.5169.
* time of the previous best right after (/tmp/r26/time_prev): 0.1529
  s/update, 1.5278 — the previous best measured 1.5358 earlier, so the
  run to run spread of the candidate runs is about 0.5 %, twice the spread of
  the reference runs.  Changes of less than 1 % cannot be told apart from the
  noise with a single run.
5.4 hRound 18 — the size of the DDP gradient buckets (N0)
With 25 MiB buckets (torch's default) the reduction of the 6.3 GiB of bf16
gradients is 119 small NCCL calls per update; every one of them has a fixed
cost and the two ranks have to meet at each of them.  YOCO_OPT_DDP_BUCKET_MB
sets the size of the buckets (fairseq passes bucket_cap_mb to torch's DDP
explicitly, so the value has to be overridden, not defaulted — the first
version of the switch did nothing, see the commit a85d6ce).

* 50 MiB: 0.1508 s/update, 1.5491
* 100 MiB: 0.1505 / 0.1506 s/update, 1.5522 / 1.5511
* 200 MiB: 0.1506 s/update, 1.5511

The size is saturated between 100 and 200 MiB, so 100 MiB is kept.  The values
of the reduced gradients do not depend on the size of the buckets, so this is
N0.
5.4 hRound 19 — the full check of the current tree (`/tmp/r29/check`)
check --tree /submission/yoco --config /submission/switches.json:

* PASS, performance.speedup = 1.5532, tier N1.
* Gate 2 (bitwise): 0 errors, 0 exact failures; 1271 of 9616 recorded tensors
  are bitwise identical, the rest are the ones the compiled RMSNorm touches.
* Gate 3 (statistical): all six endpoints within the bound, the worst one
  (loss_u050_100) at 0.96 σ of the 3 σ limit.
* The timing run of the check: 0.1505 s/update over the two windows.
6.2 hRound 21 — the order of the gradient buckets (N0, no gain)
Profile.  The trace of the last update of /tmp/r24/profile (164 ms profiled,
150 ms real) gives the timeline of the update:

| phase | window (ms) | note |
|---|---|---|
| forward | 2.8 – 38.9 | 835 kernels, the CPU issues them faster than the GPU runs them |
| backward | 38.9 – 112.5 | the gradient reduction runs from 50 to 124 ms |
| *gap* | 112.5 – 124.8 | 12.3 ms with nothing to do: the reduction of the last buckets |
| clip + cast | 124.8 – 135.1 | the L2 norm (3 ms) and the fused cast and scale (5.2 ms) |
| Adam | 135.1 – 162 | the fused Adam, 25 ms, at 4.1 TB/s |
| tail | 162 – 164.3 | the event based update timing of the bench |

The 122 NCCL kernels of the reduction are 0.2-0.35 ms each (100 MiB at the full
400 GB/s) with 0.2-1 ms gaps: the reduction is not bandwidth limited but *latency*
limited, it waits for the gradients.  The gradients of the last 1.2 GiB (the output
projection and the embedding) are only ready at the end of the backward, so their
reduction is not hidden: 12.3 ms of the update are the reduction of gradients that
were ready long before.

Change.  torch's DDP fills the buckets in the order of the registration of the
parameters and then reverses the list of the buckets, so the bucket that is reduced
first holds the gradients that the backward produces first.  The YOCO model
registers its parameters in the order of the forward pass, so the gradient of the
output projection -- the first one the backward produces -- is registered second
and lands in one of the last buckets.  YOCO_OPT_DDP_BUCKET_ORDER=1
(yoco/models/optim_overrides.py) moves the parameters of the output projection to
the end of the list that the reducer sees, so that their gradient is reduced first.
The reduced values do not depend on the grouping, so this is N0.

* time (/tmp/r21/time): 0.1503 s/update, speedup 1.5542 (the current best is
  0.1505 / 1.5532): no gain, the difference is inside the noise (0.5 %).
* The switch stays off.

So the 12.3 ms are not the order of the buckets: the reduction of the last buckets
is exposed because it cannot start before the gradients are ready, and the
gradients of the output projection are ready *late* in the backward as well (the
wgrad of the output projection is the first GEMM of the backward, but the reduction
of the buckets is a chain: a bucket is only reduced when the previous one is done).
6.7 hRound 22 — where the time goes now (the profile of `/tmp/r24/profile`)
Profile.  The trace of the last update of /tmp/r24/profile (164 ms profiled,
150 ms real) gives the timeline of the update:

| phase | window (ms) | note |
|---|---|---|
| forward | 2.8 – 38.9 | 835 kernels, the CPU issues them at the same rate as the GPU runs them |
| backward | 38.9 – 112.5 | the reduction runs from 50 to 124 ms |
| tail of the reduction | 112.5 – 124.8 | the last buckets of the reduction, overlapped with the backward |
| clip + cast | 124.8 – 135.1 | the L2 norm (3 ms) and the fused cast and scale (5.2 ms) |
| Adam | 135.1 – 162 | the fused Adam, 25 ms, at 4.1 TB/s |
| tail | 162 – 164.3 | the event based update timing of the bench |

The GPU is busy 136 ms of the 164 ms of the window: 28 ms of idle, in 2496 gaps
(2394 of them shorter than 50 us).  Twelve of those milliseconds are in the
forward, which is exactly at the limit of the host: 24.4 ms of host time for
23.0 ms of GPU work.

The kernels of the update (rank 0): 67 ms of GEMM (575 TFLOP/s, 78 % of the
733 TFLOP/s a big square GEMM reaches on this machine), 34 ms of NCCL (the
reduction of 6.9 GiB of bf16 gradients, 119 kernels of 0.2-0.35 ms, 87 %
overlapped with the backward), 25 ms of the fused Adam, 15 ms of elementwise
(411 multiplications, 268 copies, 231 additions), 5.6 ms of flash-attention,
5.2 ms of the fused cast, 3 ms of the L2 norm of apex, 1.7 ms of softmax.

The elementwise work is the largest remaining item that is not a GEMM: it is
what the retention of the self decoder does with the (1, 24, 8, 256, 256)
tensors (the mask of the gates), 13 times in the forward and 13 times in the
backward of an update.

* A diagnostic (YOCO_SCREEN_PHASE_TIMING=1, yoco/models/screening.py) prints
  the host time of the forward of the criterion: 24.4 ms against 23.0 ms of GPU
  work, so the forward is balanced and the 13 ms of idle that the profiler
  shows there is an artefact of the profiler.
6.7 hRound 23 — the fused Adam reads the bf16 gradients (N0, +1 %)
Change.  The fused cast writes the master gradient as
float(bf16 gradient) * scale, so the Adam can read the bf16 gradient and
multiply it by the same scale inside its kernel: the value is the same, bit
for bit, and the kernel reads two bytes per element instead of four (28 instead
of 30 bytes per element, 96 instead of 103 GiB per update).

* The kernel has a BF16_GRAD branch (yoco/models/fused_grad.py), the scale
  is left in a device scalar by the fused cast and consumed by the Adam.
* A unit test of the two paths of the kernel (/tmp/t3/t_adam.py) agrees bit
  for bit on every parameter, moment and weight.
* time (/tmp/r32/time_bf16grad): 0.1491 s/update, speedup 1.5667 (the
  previous best was 0.1505 / 1.5532).  The same session's run of the previous
  best was noisy (0.1531, window range 2.9 %), its median was 0.1509.
* check (/tmp/r33/check): PASS, speedup 1.5646, tier N1 (the
  trajectory's worst endpoint is 0.09 sigma of the 3 sigma bound).

The switch YOCO_OPT_ADAM_BF16_GRAD=1 is now in /submission/switches.json.
6.7 hRound 24 — no copy of the Adam states that do not move (N0, no gain)
The copy of apex (state["exp_avg"].to(device=..., dtype=...)) returns the same
tensor when the device and the dtype do not change, but its dispatch costs a
few microseconds per parameter and per moment.  The check avoids it.

* time (/tmp/r32/time_bf16grad2): 0.1493 s/update, speedup 1.5646: **no
  measurable gain**.  The 2.2 ms of host time that the profile attributes to
  the step of the optimizer is an artefact of the profiler (it inflates the
  host time of every operation).

The change is kept: it is bit for bit the same code path and it cannot hurt.
6.7 hCurrent summary (after round 24)
Switch file (/submission/switches.json):

```json
{"environment": {"YOCO_OPT_REPLICATED_ADAM": "1", "YOCO_OPT_SKIP_MASTER_ZERO": "1",
 "YOCO_OPT_FUSED_GRAD_SCALE": "1", "YOCO_OPT_FUSED_WEIGHT_CAST": "1",
 "YOCO_OPT_FUSED_ADAM": "1", "YOCO_OPT_FAST_LINEAR": "1", "YOCO_OPT_FAST_TRITON": "1",
 "YOCO_OPT_GRAD_BUCKET_VIEW": "1", "YOCO_OPT_TRITON_RMSNORM": "1",
 "YOCO_OPT_CACHED_PTR_TABLE": "1", "YOCO_OPT_DDP_BUCKET_MB": "100",
 "YOCO_OPT_ADAM_BF16_GRAD": "1"}}
```

Latest full check: /tmp/r33/check/check.json — PASS, speedup
1.5646, tier N1 (gate 3: the worst endpoint is 0.09 σ of the 3 σ bound).

Where the gain comes from (frozen reference: 0.2336 s/update):

| change | s/update | speedup | tier |
|---|---|---|---|
| reference | 0.2336 | 1.000 | – |
| replicated Adam (no ZeRO broadcast) + no master zero | 0.2095 | 1.115 | N0 |
| compiled RMSNorm | 0.1987 | 1.176 | N1 |
| fused gradient cast and scale | 0.1803 | 1.296 | N0 |
| fused master → bf16 weight copy | 0.1747 | 1.337 | N0 |
| fused Adam (also writes the bf16 weights) | 0.1667 | 1.401 | N0 |
| plain F.linear for the megatron linears | 0.1653 | 1.413 | N0 |
| direct launch of the rotary kernel | 0.1631 | 1.432 | N0 |
| direct launch of the recurrence kernels | 0.1611 | 1.450 | N0 |
| DDP gradient_as_bucket_view | 0.1577 | 1.481 | N0 |
| fused Triton RMSNorm | 0.1528 | 1.529 | N1 |
| DDP buckets of 100 MiB | 0.1505 | 1.553 | N0 |
| the fused Adam reads the bf16 gradients | 0.1491 | 1.567 | N0 |

Directions that did not work (and why):

* CUDA graphs: the DDP reducer's host side logic and the autograd graph cannot
  be captured; a graph of the forward only would keep the activations of the
  previous iteration alive and the peak memory (73 GiB) is already close to
  the limit.
* A Triton flash kernel for inner_chunk: analysed, see round 20.
* Eager instead of compiled inner_chunk: 228 us of GPU per call instead of
  68 us.
* NCCL_BUFFSIZE=32 MB: no gain (0.1531 s/update).
* A bigger DDP bucket (200 MiB) or a smaller one (50 MiB): no gain.
* The table of the addresses of the fused kernels (round 17): no measurable
  gain, the run to run noise of the candidate (0.5 %) is larger than the
  effect; the change is kept because it cannot hurt.
* The order of the DDP gradient buckets (round 21): no gain.
* The copy of the Adam states that do not move (round 24): no gain.

Where the time goes now (profile /tmp/r24/profile, 5 updates, rank 0):
GPU busy 136 ms of a 164 ms window — 28 ms of idle, mostly in the forward,
which is exactly at the limit of the host (24.4 ms of host time for 23.0 ms of
GPU work).  Kernels: 67 ms of GEMM (575 TFLOP/s, 78 % of the 733 TFLOP/s a
big square GEMM reaches on this machine), 34 ms of NCCL (87 % overlapped with
the backward), 25 ms of the fused Adam (4.1 TB/s, 95 % of the 4.3 TB/s a copy
reaches), 15 ms of elementwise (the retention of the self decoder and its
mask), 5.6 ms of flash-attention, 5.2 ms of the fused gradient cast, 3 ms of
the L2 norm of apex, 1.7 ms of softmax.
7.6 hRound 25 — where the time goes now, and three experiments that did not pay
Profile of the current best (/tmp/r35/profile, 5 updates, rank 0). The
main stream (the one the model runs on) has 128.2 ms of kernels in a
160.5 ms window; the update is therefore bound by the main stream and by the
time the gradients are reduced:

| phase | window | note |
|---|---|---|
| forward | 3.0 – 40.0 ms | 37 ms, the host is the limit here (24 ms of host time for 23 ms of GPU work) |
| backward | 40.0 – 106.8 ms | 67 ms |
| *idle* | 106.8 – 123.5 ms | 16.7 ms: the host waits for the reduction of the gradients |
| clip + cast | 123.5 – 133.6 ms | the L2 norm (3.0 ms) and the fused cast (5.2 ms) |
| Adam | 133.6 – 160.3 ms | the fused Adam, 23.6 ms at 4.1 TB/s |

The kernels of the main stream per update: 71.7 ms of GEMM (78 % of the peak),
23.6 ms of fused Adam, 14.1 ms of elementwise (**4.7 ms of it is the division
that DDP does on every gradient before the reduction**: mul_out is called
240 times per update), 5.2 ms of fused cast, 3.0 ms of L2 norm, 2.8 ms of
Triton, 2.1 ms of swiglu, 1.7 ms of softmax.

The reduction of the gradients: 47 NCCL calls of 100 MiB, from 51.5 ms to
122.7 ms.  The pure NCCL cost of the same reduction is 33 ms (measured with a
standalone benchmark: 47 x 100 MiB, 296 GB/s), so the reduction is not
bandwidth bound but *gated by the production of the gradients*: the last
bucket (the embedding, 616 MiB) is only ready at the end of the backward, and
the chain of the buckets that follows it adds 16 ms to the update.

Experiment 1: YOCO_OPT_NO_SETITEM_SYNC=1 (N0, no gain).  The backward of
the chunked recurrence writes two slices with DI[:, :, -1] = 0, which goes
through index_put_ and reads the value of the mask from the device: a
synchronization, twice per decoder layer.  The switch (already in the tree,
committed in b6cce5c) replaces them with zero_().

* time (/tmp/r36/time_setitem): 0.1494 s/update, 1.5636 — the same as
  the previous best (0.1491 / 1.5667).  The switch stays off.

Experiment 2: YOCO_OPT_FAST_EMBEDDING=1 (N0, no gain).  fairseq's
megatron VocabParallelEmbedding.forward masks the tokens with
masked_input[input_mask] = 0, and index_put_ reads the value of the mask
from the device: a synchronization in the middle of the forward pass.  The new
module (yoco/models/fast_embedding.py) writes the same zeros with
masked_fill_, which is what index_put_ does internally, without the
synchronization (a unit test: the values and the gradients are bit for bit the
same).

* time (/tmp/r36/time_emb): 0.1495 s/update, 1.5625 — no gain.  The
  switch stays off.

Experiment 3: the Triton inner chunk (already in the tree, c68fe59).
The profile with YOCO_OPT_TRITON_INNER_CHUNK=1 (/tmp/r36/profile_inner)
shows that the elementwise work of the update does not change (14.1 ms ->
15.1 ms): the mask of the gates is not what the elementwise kernels do.  The
triton category shrinks by 0.4 ms (the compiled inner chunk is replaced by the
Triton one) and the window by 3 ms, but the measured time does not move.
7.6 hRound 26 — the fused Adam writes the master gradients (N0, +1.4 %)
Change.  The fused cast writes the master gradient of every parameter as
float(bf16 gradient) * scale, and the fused Adam reads the bf16 gradient and
multiplies it by the same scale, so the master gradient is written but never
read: 12.4 GiB of writes and 6.9 GiB of reads that the update does not need.

YOCO_OPT_FUSED_MASTER_GRAD=1 lets the fused Adam write the master gradient
itself, in the same kernel that reads the bf16 gradient: the gradients are
read once instead of twice, and the master gradient is written once.  The
value is the same expression (float(bf16 gradient) * scale, in fp32, with
the same device scalar), so it is bit for bit the value the fused cast wrote.

* A unit test (/tmp/t9/t_fused.py) compares the two paths on random
  gradients: the master gradients, the master weights, the two moments and the
  bf16 weights are all bit for bit equal.
* time (/tmp/r37/time_mastergrad2): 0.1471 s/update, 1.5880 (the
  previous best was 0.1491 / 1.5667).  The first run of the same tree
  (/tmp/r37/time_mastergrad) gave 0.1502 / 1.5553, but its two windows
  differ by 2.8 %, so it is not a valid measurement.
* The profile (/tmp/r37/profile) shows what the change does: the fused cast
  is gone (5.24 ms), the fused Adam grows from 23.61 ms to 27.13 ms (it writes
  the 12.4 GiB of master gradients), and the main stream has 126.2 ms of
  kernels instead of 128.2 ms.

* Result: INCORRECT (check --quick, /tmp/r37/quick/check.json).  The
  judge records the fp32 master gradient *before* optimizer.step(), so a
  master gradient that is only written inside the Adam kernel is never seen:
  every grad/adam#0/master.* tensor had a relative error of about 1.0 and
  the cross_rank_pattern exact item failed (the reference's master
  gradients are rank-local slices of the flat fp32 buffer, the candidate's
  became replicated zeros).  **Lesson: the fp32 master gradient has to be
  materialized before the optimizer step.**  The change was reverted
  (bb9e076 is the last commit that does not contain it).
8.2 hRound 27 — the gradient norm in one kernel (N1, +0.5 %)
Change.  clip_grad_norm asked apex for the norm of the 240 gradients
with multi_tensor_total_norm, which is 165 launches of a small kernel (3.0
ms of GPU time per update) and a lot of host time in the middle of the update.

YOCO_OPT_FUSED_L2NORM=1 computes the norm with the table of the addresses
that the fused cast already keeps: one Triton kernel sums the squares of every
chunk of every gradient into a partial sum and a second one sums the partials
(two launches for the whole model).

The sum is accumulated in a different order than apex's, so the norm differs
in the last bits of the fp32 mantissa and so does the clipping coefficient:
the tier is N1 (the deviation is about 1e-7 relative, against the 1e-2 of
the Triton RMSNorm that the tree already has).

* A unit test (/tmp/t10/t_l2.py) compares the norm with apex's on random
  gradients and with the real ones: the values are bit for bit equal in the
  test, and the cast that shares the table is unchanged.
* time (/tmp/r38/time_l2): 0.1481 s/update, 1.5773; the same tree
  without the switch (/tmp/r38/time_base): 0.1488 s/update, 1.5699.
* check --quick (/tmp/r39/quick/check.json): PASS, tier N1, no exact
  failure (1478 tensors beyond the per-tensor tolerances, which are
  diagnostic only).
8.5 hRound 28 — the plan of the Adam update is kept between the updates (N0, +1.7 %)
Profile.  The profile of the current tree (/tmp/r37/profile) shows that the
gap between the end of the backward and the start of the Adam kernel is pure
Python: the fused Adam rebuilt, at every update, five lists of 240 tensors,
collected 1200 addresses with data_ptr(), rebuilt the chunk table of the
parameters and went through the bookkeeping of the Triton JIT (which re-binds
the arguments and rebuilds the specialization key at every call).  The host is
the critical path there: the GPU waits for the plan before it can run the
kernel.

Change.  The plan (the tensors, the table of their addresses, the chunk
table) is built once and reused as long as the gradients and the Adam states
keep their buffers; the kernel is launched directly, without the JIT
bookkeeping.  Triton does not specialize on the value of a floating point
argument, so the compiled kernel is the same even though the learning rate and
the bias corrections change at every update.

* A unit test (/tmp/t11/t_plan.py) compares the rebuild and the cached plan
  over five updates: the weights, the gradients, the two moments and the bf16
  weights are bit for bit equal.  The plan is rebuilt when a gradient or an
  Adam state moves.
* check --quick (/tmp/r41/quick/check.json): PASS, and the summary of
  the numerics is *identical* to the one of the run without the change
  (9616 tensors checked, 1272 bitwise identical, 1478 beyond the per-tensor
  tolerances, 0 exact failure): the change is bitwise neutral.
* time (/tmp/r40/time_new): 0.1468 s/update, 1.5913; the same tree
  without the three switches (/tmp/r40/time_base2): 0.1493 / 1.5646.

A trap.  The first version of the change rebuilt the chunk table at every
update.  The table is uploaded with a pageable buffer, so the copy blocks the
host until the stream has caught up: the update went from 0.1493 to 0.1682
s/update.  The chunk table is now built once, with the plan.

* check (full, /tmp/r42/check/check.json): PASS, speedup 1.5491
  (the candidate run of the check measured 0.1508 s/update), tier N1.

The switch file of the submission now has the fifteen switches of the
tree (the twelve of the previous rounds plus YOCO_OPT_FUSED_L2NORM,
YOCO_OPT_ADAM_PLAN and YOCO_OPT_FAST_ADAM).
8.9 hRound 29 — the statistics of the update, without reading them back (N0, no gain)
Profile (/tmp/r43/profile, the current tree, 5 updates, 187 ms/update under the
profiler): the GPU is busy 133 ms of the 160 ms of the update; 160 ms of kernels
run in that window, so 27 ms of the NCCL reduction is hidden behind the
computation and about 14 ms of the update (in the unprofiled run) is GPU idle.
The end of the update holds a burst of 16 device-to-host copies: the meters of
the statistics are accumulated on the device and read back on the host, once per
meter, at the end of every update (AverageMeter.smoothed_value rounds the value
on the host; the average of the meter that is weighted by the number of tokens
tests its count on the host; torch.isfinite(grad_norm) reads the norm).

Change (yoco/models/fast_metrics.py, YOCO_OPT_FAST_METRICS=1): the values
of the meters stay on the device.  The average of a meter is selected on the
device (torch.where(count > 0, sum / count, val)) instead of testing the count
on the host -- the value is the same tensor as fairseq's -- and the logging
output holds the values as they are on the device.  The training loop only tests
that the output is not None; the numbers that are written to the log every
log_interval updates come from the "train_inner" aggregator, whose meters are
accumulated exactly as before (the logged loss and perplexity of a run are the
same as the reference's, up to the run-to-run noise of the loss itself).

* time (/tmp/r43/time_fast): 0.1471 s/update, speedup 1.5880; the same
  tree without the switch measured 0.1468 / 1.5913 in the same session
  (/tmp/r40/time_new) and 0.1508 / 1.5491 in the last full check.  No gain:
  the syncs of the statistics are not on the critical path, the host has work to
  do while the device finishes the Adam anyway.
* A unit test (/tmp/t_fastmetrics.py): the average of a meter is the same
  tensor as fairseq's, and taking it does not synchronize the device
  (torch.cuda.set_sync_debug_mode("error")).

Conclusion.  The 2.7 ms that the screening switch YOCO_SCREEN_NO_METRICS=1
showed in round 40 was the noise of the run, not the cost of the statistics: with
the syncs removed and the accumulation kept, the update time does not move.  The
change is kept in the tree, off by default (it is bitwise neutral and cannot
hurt, but it buys nothing).
9.3 hRound 30 — the batch of the loader in pinned memory (N0, no gain) and the phases of the update
Profile (/tmp/r43/profile, the current tree): 128.5 ms of kernels on the
compute stream in an update of 146.7 ms: 66 ms of GEMM, 31.5 ms of NCCL (86 %
overlapped), 23.6 ms of the fused Adam (at the memory roofline), 15 ms of
elementwise (about 1500 launches), 6.4 ms of flash attention, 5.2 ms of the
fused cast of the master gradients, 2.8 ms of Triton, 2.1 ms of the L2 norm,
2.1 ms of SwiGLU and 1.8 ms of the loss.

Change 1 (yoco/tasks/data/basic_loader.py, YOCO_OPT_PINNED_BATCH=1):
the tensors of the batch are allocated in pinned host memory, so the copy of
the sample to the device (non_blocking=True) does not make the host wait for
the device.  The bytes are the same, so the data of the run is the same.

* time (/tmp/r44/time_pin): 0.1471 s/update, 1.5880; the same tree
  without the switch in the same session (/tmp/r44/time_base): **0.1466 /
  1.5935.  No gain**: the copy is already hidden behind the work of the
  device.

Change 2 (yoco/models/screening.py): the diagnostic
YOCO_SCREEN_PHASE_TIMING=1 now prints the host time of the phases of the
update (train_step, prepare, forward, backward, optimizer, statistics) and the
new YOCO_SCREEN_TRACE_EVENTS=1 prints a Python stack for every
torch.cuda.Event.elapsed_time call.

* The phases of an update (host, YOCO_SCREEN_PHASE_TIMING=1):
  `train_step 146.67 ms = prepare 0.21 + forward 22.6 + backward 59.3 +
  optim 0.72 + stats 23.9` and about 40 ms of waiting for the device (the tail
  of the reduction, the clip, the norm and the cast of the master gradients).
* The trace of the events (/tmp/r44/time_events): the 9387 calls of
  cudaEventElapsedTime all come from the autotuning of inductor
  (torch/_inductor/triton_heuristics.py:benchmark_all_configs) during the
  compilation, not from the update.  The 3.4-5.6 ms gaps that the profile
  shows between the kernels are therefore artefacts of the profiler and of the
  compilation, not stalls of the steady state.

Conclusion.  Two directions closed: the pinned memory of the loader buys
nothing, and the "gaps" of the profile are not worth chasing.  The pinned
loader is kept in the tree, off by default.
↑ YOCO