GLM-5.3 1.60×

Harness: Claude Code · switches: YOCO_FAST_SYNC=1 YOCO_FAST_BCAST=1 YOCO_FAST_NOZERO=1 YOCO_FAST_PIPE=8 YOCO_FAST_SHARDGRADS=1 YOCO_FAST_RMSNORM=1 YOCO_FAST_CASTCOPY=1 YOCO_FAST_WREFRESH=1 YOCO_FAST_SHARDMUL=1 YOCO_FAST_DDPGRADVIEW=1 YOCO_FAST_DDPMEAN=1 YOCO_FAST_BF16X=1 YOCO_FAST_TLAUNCH=1 YOCO_FAST_SCALESYNC=1 YOCO_FAST_MPU=1 YOCO_FAST_GRAPH=1 YOCO_FAST_ROTARY=1

Judged
1.60×
Tier
N1
Self-best
1.60×
Rounds
16
Hours
10.0
API cost
$37.2
Input tokens · 99.4% cached
123 M
Output tokens
1.0 M
Requests
931
Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
✗ still NCCL (all_to_all + 8-stage pipeline); exchanged payload switched to bf16 (BF16X, 1.30× → 1.38×)
Forward / backward CUDA graph
✓ but forward only, and only in round 21: 1.51× → 1.60×; earlier judged "the judge's criterion hook doesn't allow it"
Newer cuBLAS
✗
Retention / attention kernel swap
✗
Other main levers
fp32 master grads materialized only for this rank's shard; Triton kernels launched directly after caching (1.38× → 1.45×); bitwise-identical Triton RMSNorm; DDP reduction switched to AVG
Raw work log20 sections
1.6 hRound 2 (2026-11-06 18:10–18:50): pipelining the Adam step with the shard exchange (YOCO_FAST_PIPE)
Profile after round 1 (/tmp/r2/profile, 5 updates, rank 0): per-update kernel time ≈ 1.11 s
instrumented; gemm 65.7 ms, nccl 50.8 ms (DDP all-reduce 31.7 ms mostly overlapped + OSS exchange
19.1 ms fully serialized after the Adam step), memcpy/cast 43.8 ms, elementwise 26.3 ms, optimizer
multi_tensor (Adam) 18.3 ms. The optimizer section is still Adam (18.3) + exchange (18.6) back to
back = ~37 ms with zero overlap — the round's hotspot.

Change: YOCO_FAST_PIPE=K in yoco/models/_fast_optim.py — pipelined OSS.step. The local
shard is split into K=8 near-equal param-aligned chunks; the inner fused Adam takes one step
over K param groups (fairseq's FusedAdamV2.step launches one multi_tensor_adam multi-tensor
kernel set per group), and each chunk's all_to_all exchange is issued right after that group's
kernels are enqueued — from a wrapper around fairseq's module-global multi_tensor_applier — so
exchange k overlaps Adam chunk k+1 (NCCL waits at enqueue for the current stream's pending writes
to the exchange input; the K exchanges serialize on NCCL's stream and the last .wait() covers
all). Tier N0: same per-element Adam math over the same tensors (bitwise whatever the grouping),
same bytes exchanged, group step advances by exactly 1.

Two failures found and fixed before any valid timing:
1. First design stepped the inner Adam K times per update. The judge's timing harness counts
   updates with torch's global optimizer-step post hook — "the first optimizer object that ever
   steps defines the iteration (in the reference the shard-local FusedAdam inside fairscale's
   OSS; one step per update)" (bench/timing.py). With 8 inner steps/update the timer counted 8
   "iterations" per update: time returned ERROR ("the random-number stream differs from the
   reference from update 1 on"), bogus 7.24× / 0.0323 s-per-update, windows of 10 chunks ≈ 1.25
   real updates, StopTiming after 3 updates (other_optimizer_steps [3], window_range 0.62).
   The judge defines an update as one FusedAdam step, so the redesign above keeps exactly one
   inner step per update. (torch.optim.Optimizer._patch_step_function wraps the class's step
   at first instantiation, so the patched OSS.step and the inner FusedAdamV2.step each still
   fire the hook once per update, in the reference order: other_optimizer_steps [24] again.)
2. Rank-1 split sizes flipped in the rewrite (`RuntimeError: Split sizes doesn't match total
   dim 0 size`): the output tensor is the *other* rank's bucket slice (n_other elements), so
   output_split_sizes must be [n_other, 0], input_split_sizes [n_mine, 0]. Fixed.

Correctness evidence:
- A/B (real 2-rank torchrun entry, deterministic mode, 8 updates, all four switches incl.
  PIPE=8): per-update loss and gnorm bitwise identical to the reference OFF run
  (/tmp/r2/ab_pipe2.log vs /tmp/r1/ab_off3.log; reference self-determinism was shown in
  round 1).

time result (/tmp/r2/time2): speedup_vs_frozen_reference 1.1041 (0.2337 → 0.2117
s/update), status ok, window range 0.0031, no problems (gate 1's RNG-stream check passed:
other_optimizer_steps [24] like the reference), beyond_reference_noise true. PIPE adds ≈ +2.4%
over round 1 (Adam+exchange ~37 ms → ~33 ms; the overlap is limited by HBM bandwidth contention
between the Adam kernels (~2.6 TB/s) and the NCCL exchange (~0.7 TB/s), not by ordering).
0.9 hRound 1 (2026-11-06 17:10–18:05): optimizer section (zero-sharding exchange + per-parameter sync loops)
Profile of the unchanged reference (/tmp/r1/profile, 5 updates, rank 0):
per-update kernel time ≈ 260 ms (instrumented; real update = 233.7 ms): nccl 84 ms
(OSS fp32 master broadcast 37 ms = two sequential 18.5 ms Broadcast_RING_LL of 6.9 GB each,
fully serialized; DDP bf16 grad all-reduce 39 ms of which only ~9 ms exposed, rest overlaps
backward), gemm 66 ms, memcpy/cast 44 ms (~850 copy kernels/update: per-parameter bf16↔fp32
sync loops, ~240 params), elementwise 31 ms, optimizer multi_tensor 18 ms. GPU idle 15.7% in
the instrumented window; biggest gaps ~2 ms/update at the optimizer boundary (OSS's
torch.cuda.synchronize + CPU-blocking waits + per-param Python loops).

Changes (all in yoco/models/_fast_optim.py, installed at import models, each behind its
own environment switch in /submission/switches.json, off by default; all N0 — same elementwise
math / same bytes moved):

1. YOCO_FAST_SYNC: fairseq _FP16OptimizerMixin._sync_fp16_grads_to_fp32,
   _sync_fp32_params_to_fp16, zero_grad and FairseqOptimizer.multiply_grads replaced by
   torch._foreach_copy_/_foreach_mul_/_foreach_zero_ batches (identical elementwise ops,
   ~4×240 individual kernel launches → a handful of batched ones).
2. YOCO_FAST_BCAST: fairscale OSS._broadcast_params → one dist.all_to_all_single exchange
   of the two flat bucket buffers (grouped ncclSend/ncclRecv overlap both directions:
   microbench 37.2 ms → 18.6 ms), with async_op=True + .wait() like the reference's
   per-handle waits.
3. YOCO_FAST_NOZERO: skip zeroing the fp32 master grads in zero_grad (every element is
   overwritten by the next update's grad sync before anything reads them).

Bugs found on the way (evidence in /tmp/r1):
- First BCAST version issued the exchange fire-and-forget. NOT sound: NCCL runs the kernel on
  its own stream, not the current stream (proved by /tmp/r1/test_stream.py,
  /tmp/r1/test_wait.py: a same-stream fill after a fire-and-forget exchange sometimes wins,
  sometimes loses). The A/B loss comparison diverged from update 3 (16.824/16.825 vs 16.823) and
  differed between two runs of the same config — a race, not a value change. async_op=True +
  .wait() (and the plain sync call) are properly ordered (20/20 and 6/6 at 6.9 GB); NCCL also
  waits for pending current-stream writes to the exchange *input* (/tmp/r1/test_input.py), so
  no pre-sync is needed.
- My first "synchronous variant" test never took effect (_on() returns a bool compared to
  '2'), so both BCAST variants were the racy one. Fixed the guard, then the A/B matched.

Correctness evidence:
- A/B (real torchrun 2-rank entry, deterministic mode like the bench's numerics runner:
  CUBLAS_WORKSPACE_CONFIG=:4096:8, torch deterministic warn-only, cuDNN deterministic,
  flash-attn deterministic backward; /tmp/r1/det/sitecustomize.py, /tmp/r1/ab_run.sh):
  reference is perfectly self-deterministic (two identical OFF runs); SYNC-only, NOZERO-only,
  fixed-BCAST-only and all-three match the reference's per-update loss and gnorm exactly for
  updates 1–8 (/tmp/r1/ab_*.log).
- all_to_all vs two broadcasts on OSS-like buffers with unequal shard sizes: bitwise equal
  (/tmp/r1/test_a2a.py).
- check --quick: PASS (gate 0 structure, gate 2 bitwise at world 1 — covers SYNC and
  NOZERO at full tensor precision). BCAST only engages at world 2, where the gates do not
  record; its evidence is the A/B runs above plus gate 1's RNG check.

time result (/tmp/r1/time): speedup_vs_frozen_reference 1.0851 (0.2337 → 0.2154
s/update), window range 0.0008, peak memory 61.75 GiB (unchanged), beyond_reference_noise true.
Per-config screening from A/B runs (ups, not a verdict): SYNC ≈ 3.85, NOZERO ≈ 3.85 vs OFF
≈ 3.75 (small); BCAST ≈ 4.34 (the big one).

Full check (/tmp/r1/check): PASS, speedup 1.0746 (all gates; numerics bitwise PASS,
gate 3 skipped).

Committed: yoco/models/_fast_optim.py + the models/__init__.py hook; switches.json =
all three on. Tier N0 (bitwise per gate 2 and A/B).
2.0 hRound 3 (2026-11-06 18:55–19:25): fp32 master gradients only for the optimizer shard (YOCO_FAST_SHARDGRADS)
Profile after round 2 (/tmp/r3/profile, 5 updates): gemm 335 ms, nccl 386 ms (DDP
all-reduce + the 8 chunked exchanges, instrumented), memcpy/cast 218 ms, elementwise 132 ms,
optimizer 104 ms per 5 updates. The clip-grads phase is 55.8 GPU ms per 5 updates and reads:
bf16→fp32 grad sync (all 240 params), the l2 norm over all fp32 grads, the scale multiply over
all fp32 grads. Host time per update 0.245 s; GPU busy 71%.

Change: YOCO_FAST_SHARDGRADS=1 — with --zero-sharding os, only the local shard's fp32
master gradients are ever consumed (Adam steps the shard), but the reference syncs *every*
parameter's bf16 grad to an fp32 copy and multiplies all of them. Two pieces:
1. the gradient norm is computed from the bf16 gradients directly (they exist for all
   parameters on every rank after the all-reduce), via fairseq's own
   multi_tensor_total_norm — bitwise-equal to the reference's fp32 norm because the fp32
   copies are exact float() upcasts and apex's multi-tensor l2norm reduces both dtypes
   identically (verified bitwise standalone: /tmp/r3/test_l2norm.py, several shapes/orders/
   draws — torch.equal True);
2. _sync_fp16_grads_to_fp32 skips the parameters outside the shard; their p32.grad stays
   None, so the scale multiply (_unscale_grads) skips them too.

Failure found on the way: my first version skipped the sync but kept the reference's norm
over the fp32 grads — WRONG: the norm reads *all* fp32 grads (fairseq's
utils.clip_grad_norm_ over the full param list, no aggregation: every rank norms the whole
set and trainer._check_grad_norms requires the per-rank norms to agree). The A/B run failed
immediately with "gradients are inconsistent between workers" (rank 0 = 1.673 vs rank 1 =
1.989 — each rank had only its own shard synced). The bf16 norm fixes it: every rank norms the
same complete set again. Tier N0.

Correctness evidence: A/B (2-rank torchrun, deterministic, 8 updates, all five switches):
per-update loss and gnorm bitwise identical to the reference OFF run
(/tmp/r3/ab2.log vs /tmp/r1/ab_off3.log).

time result (/tmp/r3/time): speedup_vs_frozen_reference 1.1441 (0.2337 → 0.2043
s/update), status ok, window range 0.0006, no problems. +3.6% over round 2 (the sync, the
norm and the multiply each lose the other rank's half ≈ 31 GB of traffic per update).

Full check (/tmp/r3/check): PASS, speedup 1.1396. Gate 2 is NOT_BITWISE — the 908
violations are exactly the non-shard master grads (rel_err ≈ 1: the adapter records them as
zeros where the reference has the dead copies; exact_failures: 0); every tensor that is
actually computed — loss, delta, Adam states, weights — is bitwise identical. Tier N1;
gate 3 (statistical, 200 updates, frozen seed) ran and returned "equivalent" with **all six
endpoint diffs exactly 0.0** — the training trajectory is bitwise the reference's, so the
verdict has maximal margin and reproduces deterministically in the final evaluation.
3.2 hRound 5 — fused multi-tensor cast copy (YOCO_FAST_CASTCOPY)
Start 2026-11-06 20:06, end 2026-11-06 20:25. Hotspot from the fresh profile
(/tmp/r5/profile): memcpy/cast 25.4 ms/update — the fp32↔bf16 gradient/weight syncs.
torch._foreach_copy_ has no fast path for mixed dtypes: it silently falls back to one
unvectorized per-tensor kernel per parameter (measured: identical time and launch count to
an explicit per-tensor loop, ~33x off the HBM roofline for the cast traffic).

Change (_fast_optim.py, switch YOCO_FAST_CASTCOPY=1): one fused Triton kernel per
copy list. Program pid copies BLOCK contiguous elements of segment block_seg[pid] at
offset block_off[pid] (block layout cached per size list); tensors are passed as int64
pointer tables (tl.load(...).to(tl.pointer_type(...)), verified supported in triton 2.3).
bf16→fp32 is exact; fp32→bf16 is cvt.rn like torch's copy_ (proven bitwise in the RMSNorm
work). Elementwise → deterministic; guards fall back to foreach for anything but
contiguous CUDA tensors of the expected dtypes.

Rejected in-round: YOCO_FAST_SKIPWSYNC (skip the fp32→bf16 refresh of the non-owned
half). The A/B caught it: loss matched updates 1–2 then diverged. My invariant was wrong —
each rank's fp32 masters are ALL live: own shard ← Adam, other shard ← the OSS exchange
every update, so the reference's full refresh is semantically required (only the non-shard
fp32 *gradients* are dead, which is what round 3's SHARDGRADS correctly skips). Removed.

Correctness evidence: unit bitwise suite /tmp/r5/castcopy_test.py (mixed sizes, odd
tails for block masking, single tensors, 240 same-size, non-contiguous rejected) — all
bitwise; A/B (/tmp/r5/ab_cast2.log, 2 ranks, deterministic, 8 updates, all seven
switches): loss and gnorm bitwise identical to /tmp/r1/ab_off3.txt.

time result (/tmp/r5/time): speedup_vs_frozen_reference 1.2067 (0.23373 →
0.1937 s/update), beyond_reference_noise true. −5.9 ms/update vs round 4 (0.1996 → 0.1937).

check --quick (/tmp/r5/quick): PASS (gates 0 + 2), measured_tier N1, 908
violations (unchanged, all SHARDGRADS's), exact_failures 0.
3.6 hRound 6 — NCCL protocol / chunking (no gain)
Start 2026-11-06 20:33, end 2026-11-06 20:55. Target: the nccl category (55 ms/update
after round 5 — 28.8 DDP all-reduce + 26.4 pipelined exchange).

Findings: the DDP all-reduce kernels are named ..._RING_LL even for 25-37 MB buckets,
which suggested LL's halved bandwidth — but NCCL_PROTO=Simple/LL128 changed nothing at
all (minimal 2-rank test: same kernel name for every setting; the name is not protocol-
truthful in NCCL 2.20.5). Per-bucket arithmetic from the profile (0.24 ms per 37.5 MB
bucket, ring moves 2x size per rank) gives ~310 GB/s — already ~70% of NVLink, so the
all-reduce is near-optimal and mostly overlapped with backward. Judge time with
NCCL_PROTO=Simple (1.2123), LL128 (1.2011), PIPE=16 (1.2104) vs baseline 1.2067 — all
within the 0.006 noise band; no reproducible gain. The remaining exchange inefficiency
(26.4 ms for 4.46 GB, ~170 GB/s effective, vs 18.6 ms unoverlapped in round 1) is SM
contention with the concurrent Adam kernels, not a protocol choice.

Conclusion: communication config is exhausted; switches restored to the round-5 set
(PIPE=8, no NCCL vars). No code change, nothing to commit. Next: hide optimizer-section
work under the exchange tail (weight refresh) and tune the fused cast kernel, then the
gate-retention fp32 mask chains (12.2 ms/update of latency-bound elementwise).
3.8 hRound 7 — weight-refresh overlap + cast-kernel tuning
Start 2026-11-06 20:56, end 2026-11-06 21:15. Hotspot: the optimizer section's exposed tail —
the fp32→bf16 weight refresh (10.3 ms/update measured) runs after the exchange is waited on.

Change 1 (kept): YOCO_FAST_WREFRESH=1 — the pipelined step (_oss_step) enqueues the
own-shard half of the weight refresh right after the Adam kernels, before the exchange tail
is waited on (those copies read only locally-stepped masters, current stream after Adam, so
they execute concurrently with the NCCL exchange); the outer sync then refreshes only the
other rank's half. A/B bitwise (/tmp/r7/ab_wr.log vs /tmp/r1/ab_off3.txt); quick check
PASS (/tmp/r7/quick, 908 known violations, 0 exact failures). time: 1.2085
(/tmp/r7/time_wr, 0.1934 s/update) vs 1.2067 — the gain is ~+0.3 ms, within the noise
band: the refresh's HBM traffic contends with the concurrent Adam+exchange kernels, eating
most of the hiding. Kept (verified, ≥ 0), but weak.

Change 2 (reverted): tl.multiple_of alignment hints on the cast kernel's table-loaded
pointers, hoping to enable 128-bit vectorization. Measured the OPPOSITE: 1.1357–1.1418
(hints made the kernel ~12 ms/update SLOWER — the hint on runtime-loaded scalar pointers
pessimizes codegen). Reverted; the round-5 kernel stands. The fused kernel remains ~1.7x
off roofline (10.3 vs ~6.1 ms) — a hand-vectorized variant (explicit 128-bit loads/stores
per thread, e.g. via inline PTX ld.global.v4/st.global.v4) is a possible later step.

Current best: 8 switches, time 1.2085, quick check PASS. Committed next.
4.9 hRound 8 — the fp32 master-grad scale multiply: shard-only + one fused kernel
Start 2026-11-06 21:20, end 2026-11-06 22:10. Hotspot found by attribution, not the
profile row alone: elementwise_kernel<128,4,BinaryFunctor<float...MulFunctor> was
60.9 ms per 5 updates (12.2 ms/update, 1200 launches) — one kernel *per parameter*.

Attribution (record_shapes trace /tmp/r8/shapes_trace.json, 2 updates, + 3
dispatch/monkeypatch probe runs /tmp/r8/probe*.log):
- Every one of the 240 fp32 master grads is multiplied once per update by the
  _multiply_factor ((2/2048) x clip_coef, a 0-dim fp32 CUDA tensor): fairseq's
  FairseqOptimizer.step takes the scale != 1.0 -> multiply_grads(1/scale) path
  (FusedAdamV2 ignores scale for fp32 grads), i.e. _unscale_grads in
  FP16Optimizer.step.
- My earlier multiply_grads patch already batched it as torch._foreach_mul_, but
  foreach has no fast path for a 0-dim CUDA tensor scalar — it silently falls back
  to one unvectorized mul_ kernel per tensor (visible as 240 aten::mul_ cpu_ops
  nested inside a single aten::_foreach_mul_ span, 1.8 ms of CPU launch time).
- 240, not the ~120-param shard, because build_fp32_params gives every fp32
  master a zeros() grad at construction — p.grad is not None is always true, so the
  non-shard half (dead: never written by the shard-filtered sync, never read — Adam
  steps the shard, the norm runs on bf16 grads) was scaled too. 0 * c == 0 keeps the
  skipped ones bit-identical.
- (The bf16 mul.out rows at param shapes are DDP's div-by-world-size write-back —
  C++, out of reach; the [25165824]/[9437184] copy_ rows are one-time Adam state init.)

Change: YOCO_FAST_SHARDMUL=1 in yoco/models/_fast_optim.py:
- multiply_grads filters the grad list to the optimizer shard (same
  _shard_param_ids set the sync uses; only when the _FP16OPT chain matches).
- New _mulsc_kernel (Triton): all fp32 grads scaled in ONE launch (block layout
  reuses the castcopy copier); the scalar is read from device memory — no D2H sync.
  Guards mirror _cast_copy (fp32/contiguous/CUDA/16-byte aligned), else foreach.
- Lone v * s multiply, no add — no FMA-contraction pitfall.

Verification: unit bitwise suite /tmp/r8/mulsc_test.py PASS (random/denormal/
zero/overflow x 6 scalars, odd sizes, fallback guards); A/B 8-update run bitwise
identical to the reference (/tmp/r8/ab_on.log vs /tmp/r1/ab_off3.txt: loss and
gnorm match all 8 updates).

Results: time 1.2655 (/tmp/r8/time, 0.1847 s/update) vs 1.2085 — +4.7%,
far beyond the noise band (removes ~6 ms of dead-tensor traffic and ~1.8 ms of CPU
launch time, and the fused kernel is faster than the fallback on the kept half).
check --quick PASS (/tmp/r8/quick).

Tier: N1 (same class as SHARDGRADS: the dead non-shard fp32 grads keep their zeros —
the 908 known violations stay, no new ones; every read value is bitwise the
reference's). Committed next.
4.9 hCurrent summary (after round 8)
- Switch file /submission/switches.json: 9 switches — YOCO_FAST_SYNC=1,
  YOCO_FAST_BCAST=1, YOCO_FAST_NOZERO=1, YOCO_FAST_PIPE=8,
  YOCO_FAST_SHARDGRADS=1, YOCO_FAST_RMSNORM=1, YOCO_FAST_CASTCOPY=1,
  YOCO_FAST_WREFRESH=1, YOCO_FAST_SHARDMUL=1.
- Latest full check: PASS, speedup 1.2573 (/tmp/r8/check/check.json,
  2026-11-06 22:14). Latest time: 1.2655 (/tmp/r8/time). Quick check PASS
  (/tmp/r8/quick). HEAD 5c567b5, tree clean.
- Gain breakdown (speedup over the 0.23373 s/update frozen reference):
  rounds 1–3 (foreach sync/bcast/nozero, PIPE=8): 1.147 → ~1.17;
  round 4 YOCO_FAST_RMSNORM (fused Triton RMSNorm, bitwise): → 1.1763 (full check);
  round 5 YOCO_FAST_CASTCOPY (fused multi-tensor cast sync): → 1.1998 (full check);
  round 7 YOCO_FAST_WREFRESH (own-shard refresh under the exchange tail): +~0.3 ms;
  round 8 YOCO_FAST_SHARDMUL (shard-only scale multiply, one fused kernel): → 1.2573.
- Dead ends: NCCL_PROTO/NCCL_ALGO (inert in NCCL 2.20.5), PIPE=16 (noise),
  skipping the non-shard weight refresh (wrong: the exchange writes those masters),
  tl.multiple_of hints on table-loaded pointers (−12 ms/update), tl.exp (not
  bitwise vs torch), SKIPWSYNC (diverged).
- Next candidates: fold the remaining shard scale multiply into the grad-sync
  kernel (fused fp32 = float(bf16) x c — same rounding, saves the second pass);
  hand-vectorized (128-bit) cast kernel; gate-retention fp32 mask chains.
5.2 hRound 9 — fused scale-into-sync (YOCO_FAST_SCALESYNC): no gain, switch off
Start 2026-11-06 22:15, end 2026-11-06 22:30. Follow-up on round 8's hotspot: after
SHARDMUL the scale multiply still costs a separate pass over the shard's fp32 grads
(~4.5 ms, one fused kernel). Fusing it into the bf16→fp32 grad-sync kernel
(p32.grad = float(p.grad) * c, the same single fp32 multiply as copy-then-mul) removes
that pass entirely.

Change: YOCO_FAST_SCALESYNC=1 — the fast clip (where _multiply_factor is final)
calls a scaled sync instead of syncing before the norm; the later _unscale_grads
multiply is skipped via a flag. Unit bitwise suite /tmp/r9/scalesync_test.py PASS
(7 scalars × bf16 patterns incl. denormals/huge/zeros, unscaled path unchanged, guards);
A/B 8 updates bitwise identical (/tmp/r9/ab_on.log).

time: 1.2593 (/tmp/r9/time, 0.1856 s/update) vs 1.2655 — slightly WORSE
(delta 0.0062, just outside the 0.006 noise band). Why: the unscaled sync used to run
*before* the clip's cross-rank norm allreduce and overlapped it; the scaled sync must run
*after* the factor is final, so the copy fell out of that overlap, and the multiply it
removed was already hiding under the PIPE exchange tail (the optimizer section is
exchange-bound: 27 ms exchange vs 16.8 Adam + 4.5 multiply). Net ≈ 0.

Decision: switch removed from /submission/switches.json (back to the round-8 set,
the best-evidence state: full check PASS 1.2573); the code stays, off by default.
Committed. Lesson: in this pipeline an elementwise pass is only worth fusing away if it
sits on the critical path *after* the last overlap opportunity — measure, don't assume.
5.5 hRound 10 — DDP wrapper overhead: bucket cap (no gain) + gradients as bucket views (kept)
Start 2026-11-06 22:30, end 2026-11-06 22:45. Hotspot from the fresh profile
(/tmp/r9/profile): the DDP allreduce's own overhead — ~119 buckets per update (cap
25 MB), ~240 grad→bucket Memcpy DtoD copies per update (4.2 ms) plus the div-by-2
write-back (4.8 ms, C++), 55.7 ms of AllReduce kernel time per update.

Change 1 (not adopted): YOCO_FAST_DDPBUCKET=<MB> — raise the bucket cap via a
DistributedFairseqModel wrapper. Values bitwise (A/B /tmp/r9/ab_b256.log); timing
256 MB: 1.2675, 64 MB: 1.2710 then 1.2573 on repeat — straddles the baseline, inside
the 0.006 noise band. No reproducible gain (per-element allreduce was already
~310 GB/s on the big buckets, per round 6). Code kept, switch off.

Change 2 (kept): YOCO_FAST_DDPGRADVIEW=1 — gradient_as_bucket_view=True on
torch DDP (via a DistributedDataParallel.__init__ wrapper; fairseq's wrapper passes
no such kwarg). The autograd engine then accumulates straight into the bucket storage:
the ~240 grad→bucket DtoD copies per update disappear. Per-element values identical
(allreduce sums and the div-by-2 act on the same elements in the same order).
A/B 8 updates bitwise identical (/tmp/r9/ab_gv.log); check --quick PASS
(/tmp/r9/quick, tier N1). time: 1.2828 then 1.2814 on repeat
(/tmp/r9/time_gv, /tmp/r9/time_gv2; 0.1822/0.1824 s/update) vs 1.2655 — a
reproducible +2.4 ms/update, beyond the noise band.

Committed. Switch set now 10 on.
7.9 hCurrent summary (after round 16)
- Submission state: 14 switches on in /submission/switches.json: SYNC, BCAST,
  NOZERO, PIPE=8, SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, BF16X,
  TLAUNCH, SCALESYNC, MPU. Latest full check PASS 1.4737 (/tmp/r16/check, tree HEAD
  d69d0ab); standalone time with the same tree: 1.4964 (/tmp/r16/time) and 1.4992
  (/tmp/r16/time2), s/update 0.1559-0.1562 vs frozen reference 0.23373. Gate 2 exact
  failures 0, tier N1 (the standing dead-tensor class from SHARDGRADS+BF16X: 1591
  unresolved, 7719 bitwise — unchanged by MPU), gate 3 decides and passed.
- Gain breakdown: rounds 1-3 ~1.15; RMSNorm -> 1.1763; CASTCOPY -> 1.1998; SHARDMUL ->
  1.2573; DDPGRADVIEW -> 1.2835; fp64 scalar fix -> 1.2992; BF16X -> 1.3806; TLAUNCH ->
  1.4526; SCALESYNC -> 1.4859; MPU -> ~1.49 (mean of 1.4964/1.4992/1.4737).
- Wall structure now (r16 profile, instrumented): forward CPU ~51 ms vs ~35.5 GPU;
  backward CPU ~93 ms vs ~40 GPU; optimizer CPU ~25 ms vs ~20 GPU (Adam 16.4 at HBM
  roofline + bf16 exchange ~17 ms contention-bound, PIPE-overlapped); DDP allreduce 41.6 ms
  GPU off the wall (protocol inert, verified). Top elementwise GPU: aten::mul ~6.8 ms
  (~400 kernels), copies ~3.9 ms, adds ~3.7 ms, sums ~2.7 ms.
- CUDA graphs are OUT for forward/backward: the judge counts criterion calls with a
  module forward hook and raises if it does not fire (bench/yoco_adapter.py:622); a graph
  replay never calls the module. The fp32-optimizer-step count is a Python wrapper on
  fp32_optimizer.step, so optimizer-internal rewrites stay viable.
- Dead ends: NCCL_PROTO inert; PIPE 4-16 within noise; CUBLASLT_WORKSPACE_SIZE no gain;
  SKIPWSYNC wrong; multiple_of hints pessimize; tl.exp/tl.log not bitwise vs aten (blocks
  swiglu/retention/logsigmoid Triton fusions); DDP divide fold drops a bf16 rounding
  (rejected); apex fp32 Adam has no scale argument; lr is a baked-in kernel argument in
  apex's Adam (kills CUDA-graphing the optimizer step without a custom kernel).
- Next candidates (priority): (1) cached Adam call — skip fairseq FusedAdamV2.step's
  per-param list building + .to() no-ops, re-issue the identical multi_tensor_applier
  call (N0, ~2-4 ms CPU); (2) Triton Adam with device-pointer lr (enables later graphing;
  bitwise needs fma/div_rn/sqrt_rn care, apex op order extracted in
  /tmp/r17/multi_tensor_adam.cu); (3) RMSNorm residual-add fusion (~104 kernels, N0);
  (4) rotary wrapper trim; (5) F.linear -> torch.mm dispatch trim.
8.2 hRound 17 (2026-11-07 01:13–01:21): YOCO_FAST_ADAMCALL — cached Adam lists: NO GAIN
- Hotspot: optimizer-phase CPU 25 ms/update (profiled) — fairseq's FusedAdamV2.step
  rebuilds its 4 multi-tensor lists every step: per param a grad check, sparse check,
  state-dict lookup, two .to() no-ops and four appends (~350 params/update, 8 PIPE
  chunks).
- Change: yoco/models/_fast_optim.py _patch_adamcall() behind
  YOCO_FAST_ADAMCALL (off): p/m/v lists cached per params-tuple key, grad list rebuilt
  per step, full identity validation (p.data, state dict, exp_avg/exp_avg_sq objects);
  any surprise falls back to the reference step for that call; identical
  multi_tensor_applier call through the same module global (PIPE keeps firing).
- Evidence: A/B 2-rank deterministic update-8 line byte-identical to reference
  (/tmp/r16/ab_adamcall: loss 16.655 / ppl 103209 / gnorm 4.973 / clip 100).
- time: 1.4973 (/tmp/r17/time, 0.1561 s/update) vs 1.4964/1.4992 without —
  no gain. Conclusion: the removed Python was already hidden under the GPU Adam
  kernels; the optimizer phase's CPU number is dominated by the exchange .wait() (real
  GPU wait), not addressable Python. Switch committed but left off in switches.json.
- Committed e227faf. Submission switches back to the 14 evidenced ones.

### Lesson (recorded for future rounds)
The profiled phase CPU times include host-blocking GPU waits; the addressable CPU in the
optimizer phase was ~2-3 ms and already overlapped. Treat "phase CPU" as an upper bound
on addressable work; check what fraction is .wait()/sync before attacking it.
8.6 hRound 18 (2026-11-07 01:22–01:50): rotary trim landed; DDP per-param division identified
- Probe first (/tmp/r18/rotary_bm_probe2.py, CUDA-graph replay for true GPU time):
  the reference rotary tiling (BLOCK_M=4/8, nw=4) is already the FASTEST config (10.1/5.9
  us); bigger blocks are 16-30 us AND bitwise-unstable across tilings/codegen (fma
  contraction) — BLOCK_M bump abandoned on both counts.
- Change (commit 13474ef): YOCO_FAST_ROTARY — rotary_tail_kernel (reference
  kernel + in-launch bf16 tail copy, verified bitwise in all 4 branches), backward
  do.clone() dropped. A/B update-8 byte-identical; time 1.4954 (/tmp/r18/time) —
  within noise, strictly less GPU work, kept on.
- Attribution dig (the round's real finding): the top elementwise row (~250 bf16
  scalar mul(grad, 1/world, out=grad) on weight-shaped tensors, 4.8 ms GPU + ~250
  launches/update) has NO Python caller — it is **c10d DDP's per-parameter gradient
  division** in the C++ reducer before the allreduce. Also confirmed the current
  config's scaled sync engages in real 2-rank runs (det and non-det; diagnostics in
  /tmp/r18/ab_nodet): no mulsc, sync fused. The r16 profile's mulsc row did not
  reproduce (today's fresh profile /tmp/r18/profile: elementwise 48.1 ms/5, no mulsc).
- Next round target: DDP register_comm_hook with per-bucket
  all_reduce(op=MEAN). With a hook registered DDP skips its division; NCCL MEAN
  computes fl((a+b)/2), bitwise equal to DDP's fl(a/2+b/2) because power-of-2 scaling
  commutes with rounding (subnormal edge to be probed over all 65536 bf16 patterns).
  Expected ~3-5 ms/update.
9.2 hRound 19 (2026-11-07 01:52–02:25): YOCO_FAST_DDPMEAN — per-bucket ncclMean, landed
- Basis: round 18's attribution — ~250 per-param bf16 mul(grad, 1/world) kernels
  (4.8 ms GPU) are c10d DDP's C++-reducer pre-division before the SUM allreduce.
- Bitwise proof first (/tmp/r19/mean_probe.py, 2 ranks): DDP's fl(a/2 + b/2) vs
  ncclMean's fl((a+b) x 0.5) over ALL 65536 bf16 bit patterns x 4 pairings
  (same/rotated/negated/x3): 0/65536 mismatches (ws=2; scaling by a power of 2
  commutes with the single bf16 rounding). dist.ReduceOp.AVG (-> ncclMean) exists in
  torch 2.3.1; MEAN does not.
- Change: YOCO_FAST_DDPMEAN in _fast_optim.py — patches
  DistributedDataParallel.__init__ (chained after DDPGRADVIEW's) to
  register_comm_hook(pg, hook) at power-of-2 world size; the hook runs one
  `dist.all_reduce(bucket.buffer(), op=ReduceOp.AVG, async_op=True).get_future()
  .then(lambda f: f.value()[0])` per bucket. With a hook registered the reducer skips its
  per-parameter division (torch documents this), so the x1/2 is fused into the allreduce
  kernel itself.
- Failure debugged en route: first A/B hung >420 s with no output (grep pipe had
  buffered everything). Rerun with output to a file + NCCL_DEBUG=WARN showed the real
  error instantly: `result.isPyObject() INTERNAL ASSERT FAILED
  python_comm_hook.cpp:43` — a comm hook must return a future completing with a Python
  object; torch 2.3.1's canonical allreduce_hook chains `.then(lambda fut:
  fut.value()[0])`. Mine returned the raw Work future. Added the same chain -> clean run.
  (AVG itself was never the problem; the probe had bypassed the hook machinery.)
- A/B (8 updates, 16 switches): update-8 line byte-identical to reference
  (loss 16.655 | ppl 103209 | gnorm 4.973 | clip 100), /tmp/r19/ab_mean8.log.
- time: 1.5021 (/tmp/r19/time, s/update 0.1556, beyond_reference_noise true) —
  best single time so far (previous standalone: 1.4954/1.4964/1.4992, mean ~1.497).
  Gain over them ~+0.3%, i.e. within single-run noise: the 4.8 ms of division kernels
  were mostly hidden under comm/backward overlap, so eliminating them buys launch/gap
  overhead, not the full 4.8 ms.
- check --quick: PASS, gate-2 fingerprint unchanged (1591 unresolved / 7719 bitwise,
  exact_failures 0, dtype_changes 0, N1) — DDPMEAN contributes nothing to the standing
  comparison (/tmp/r19/quick).
- Kept on: bitwise-safe (proven + byte-identical A/B), strictly less GPU work, best
  time yet. Tier N0 on its own.
- Committed; switches.json now 16 switches.
9.2 hCurrent summary (after round 19)
- Switch file: /submission/switches.json — SYNC, BCAST, NOZERO, PIPE=8, SHARDGRADS,
  RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH, SCALESYNC,
  MPU, ROTARY (16 switches).
- Latest full check: PASS 1.4737 at 14 switches (/tmp/r16/check, tree 13474ef minus
  MPU? no — full check predates MPU/ROTARY/DDPMEAN). DDPMEAN/MPU/ROTARY passed
  check --quick with the identical standing N1 fingerprint (1591 unresolved, 0 exact
  failures). A full check with all 16 is the next milestone once one more gain lands.
- Best time evidence: 1.5021 (r19, 16 switches) vs 1.4970 mean of r16-r18 runs.
- Gain breakdown (vs frozen reference 0.23373 s/update): fused/trimmed elementwise sync
  + cast paths (SYNC/CASTCOPY/SCALESYNC/SHARDGRADS/SHARDMUL ~+13%), PIPE+Adam-call
  trimming (~+8%), BF16X weight-exchange (~+8%), RMSNORM fused (~+4%), misc
  (NOZERO/BCAST/DDPGRADVIEW/TLAUNCH/MPU/ROTARY/DDPMEAN, each ~0-1%).
- Dead ends: CUDA-graphing fwd/bwd (adapter's criterion forward-hook contract),
  ADAMCALL list caching (no wall gain — Python overlapped), rotary BLOCK_M retuning
  (reference fastest + bitwise-unstable), DDPMEAN's 4.8 ms (hidden under overlap).
- Next candidates: RMSNorm residual-add fusion (~104 add kernels, 2.6 ms GPU, N0);
  F.linear->matmul dispatch trim in the MPU fast path; Triton Adam with device-pointer lr
  (bitwise risk: fma/div_rn/sqrt_rn must match apex's op order exactly).
9.4 hRound 20 (2026-11-07 02:15–02:36): full check PASS at 16 switches; F.linear dispatch trim backfired
- Full check (all 16 switches, tree 353e7b2): PASS 1.4719, gate-2 fingerprint
  unchanged (1591 unresolved / 0 exact failures / N1) (/tmp/r20/check). This is the
  submission's evidenced state.
- Profile analysis (r18 trace, 15-switch config): per update the GPU is busy ~139 ms,
  idle ~17 ms. The idle concentrates in ONE ~45 ms window per cycle whose host ops are
  FORWARD dispatches: 138 aten::linear -> 164 aten::matmul -> 138 aten::mm chains
  (~17 ms host per profiled forward) plus view/reshape/empty_like churn from the retention
  layers. Optimizer tail (Adam 16.5 + cast 6.9 + SendRecv 16.3) is already pipelined
  (SendRecv hides under Adam; HBM-bound ~23 ms, 57 GB) — my earlier "exposed bcast"
  reading was wrong. DDP AR (40 ms kernel time, RING_LL) is bucket-overlapped under
  backward; NCCL_PROTO=Simple was already shown inert (r13).
- Change attempted (YOCO_FAST_LIN): collapse the linear->matmul->mm dispatch chain
  in the MPU fast path to a direct torch.mm(x, w.t()) (2-D) / torch.matmul (n-D) with
  a cached transposed view. Byte-identical A/B (update-8 line identical) — but judge
  time regressed to 1.3789 (0.1695 s/update; window_range 0.0028 — steady, real),
  while the 16-switch control right after measured 1.5128 (0.1545).
- Why (understood after the fact): w.t() created in forward is a differentiable op,
  so every projection gains a TBackward node and AccumulateGrad receives the grad as a
  NON-CONTIGUOUS transposed view — autograd then materializes full-size contiguous grad
  copies every update (~170 params, ~12 GB traffic) and the DDP bucket-view/grad
  relationship degrades. F.linear's fused backward writes w.grad directly; its extra
  dispatch layers are the price of that. Reverted (uncommitted; nothing to keep).
- Lesson: a dispatch-count trim that changes the AUTOGRAD NODE SHAPE can cost far more
  than the dispatches it saves. Any forward-path rewrite must keep one backward node per
  projection (or be CUDA-graphed, which fixes node shapes by construction).
- Files: /tmp/r20/switches_lin.json (17-switch test config, not the submission's),
  /tmp/r20/time_lin deleted; submission remains 16 switches + 353e7b2.
9.4 hCurrent summary (after round 20)
- Switch file: /submission/switches.json (16 switches: SYNC, BCAST, NOZERO, PIPE=8,
  SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH,
  SCALESYNC, MPU, ROTARY). Tree 353e7b2.
- Latest full check: PASS 1.4719 (/tmp/r20/check). Best time evidence: 1.5128
  (control, /tmp/r20/time_ctl deleted after reading; value recorded here) and 1.5021
  (r19).
- Next candidates: (1) Triton fused Adam (integrate the fp32->bf16 refresh cast, read bf16
  grads directly): saves ~6-9 GB/update of the HBM-bound optimizer tail (~4 ms, 2.5%);
  bitwise risk mitigated by disassembling apex's SASS for fma contraction + dual-check
  harness. (2) CUDA-graph the model forward (+criterion eager, backward via
  make_graphed_callables-style split): kills the forward host cost + its idle (~8-12 ms),
  highest remaining gain, highest risk — the adapter allows it (criterion hook fires
  outside the graph; no RNG in forward). (3) RMSNorm residual-add fusion (~1 ms, N0).
- Dead ends so far: CUDA-graph fwd/bwd via adapter hooks (criterion hook raises), ADAMCALL
  (no wall gain), rotary BLOCK_M (reference fastest + bitwise unstable), DDPMEAN's full
  4.8 ms (hidden under overlap), F.linear->mm dispatch trim (TBackward + non-contiguous
  AccumulateGrad, -9%).
9.9 hRound 21 (2026-11-07 02:38–03:02): YOCO_FAST_GRAPH — CUDA-graphed decoder forward, +6.7%
- Hotspot (from the r18 trace): the forward is host-bound — 138 aten::linear ->
  164 aten::matmul -> 138 aten::mm dispatch chains plus view/reshape churn per update;
  ~17 ms/update of GPU idle concentrated in the forward window. Round 20's dispatch trim
  backfired, so the remaining lever for the same bottleneck class was eliminating host
  work wholesale.
- Probes first:
  - apex capturable_master Adam (fused bf16 grad read + bf16 weight write): m/v bitwise
    identical to the reference kernel, fp32 masters differ in ~1% of elements at 1-2 ulp
    (different contraction in the p-update), bf16 weights identical — viable N1/N2 path
    worth ~2.3 ms, NOT taken (yet).
  - my castcopy table kernel already runs at 2969 GB/s (faster than aten copy_ 2232):
    the 6.88 ms castcopy row is refresh ~3 ms + grad sync ~3 ms, both near HBM peak —
    no bitwise-safe slack in the optimizer tail (it is HBM-bound at ~60 GB/update).
- Change (YOCO_FAST_GRAPH): CUDA-graph YOCOModel.forward (the yoco_3b decoder).
  The forward is fixed-shape (1,2048) int64, RNG-free (no dropout in the model) and
  collective-free, so its kernel sequence captures once (call 3, inside the judge's
  warmup) and replays bitwise identically at ~zero host cost. Criterion + backward stay
  eager (adapter hooks fire; DDP's per-bucket reduction hooks keep firing); the optimizer
  backward passes retain_graph=True so the captured autograd graph is reused (the
  patch is otherwise an exact copy of the reference's, including _needs_sync = True).
  Capture failures or unexpected args/shapes fall back to the eager forward permanently.
- Two bugs found by the A/B before landing (both fixed): (1) I first patched
  models.transformer.LLaMA — the arch in use is yoco (YOCOModel in models/yoco.py);
  (2) my backward copy dropped the reference's self._needs_sync = True — the grad sync
  never ran and the model froze at update 3 (losses plateaued at 16.82). The A/B
  per-update diff caught both; after the fix all 8 updates + the summary line are
  byte-identical to the reference (/tmp/r21/ab_graph2.log).
- time: 1.602 (/tmp/r21/time_graph, s/update 0.1459) vs 1.5128 control without
  the graph — +6.7%.
- check --quick: PASS, standing fingerprint unchanged (1591 unresolved / 0 exact
  failures / 0 dtype changes / N1) (/tmp/r21/quick_graph).
- Adopted: switches.json now 17 switches. Full check started (/tmp/r22/check).
9.9 hCurrent summary (after round 21)
- Switch file: /submission/switches.json — 17 switches (SYNC, BCAST, NOZERO, PIPE=8,
  SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH,
  SCALESYNC, MPU, GRAPH, ROTARY).
- Latest full check: PASS 1.4719 at 16 switches (/tmp/r20/check); GRAPH passed quick +
  A/B byte-identical; full check at 17 switches pending (/tmp/r22/check).
- Best time: 1.602 (r21, 17 switches).
- Gain breakdown: elementwise/cast fusions ~+13%, PIPE/Adam-call trims ~+8%, BF16X ~+8%,
  RMSNORM ~+4%, GRAPH ~+6.7%, DDPMEAN/MPU/ROTARY/misc ~+1-2% total.
- Dead ends: LIN dispatch trim (-9%, TBackward + non-contiguous AccumulateGrad), rotary
  BLOCK_M, ADAMCALL, DDPMEAN's full 4.8 ms (hidden), capturable-master Adam (2.3 ms at
  1-ulp master drift — available as a future N1 lever).
- Next candidates: graph the criterion+backward? (needs manual DDP reduction — probably
  negative); capturable-master Adam (~2.3 ms, N1); RMSNorm residual fusion (~1 ms, N0);
  forward-idle leftovers (retention view/reshape churn now inside the graph — gone).
—Round 22 (started 02:38) — full-check result + YOCO_FAST_TADAM
- Full check at 17 switches: PASS, speedup 1.5976 (/tmp/r22/check, gate-1 timing
  0.1463 s/update, window_range 0.0013). Gate 3: all 6 endpoints within 3σ (worst
  |d|/σ = 1.09 at loss_u100_200); exact_failures 0, unresolved 1591, bitwise_identical
  7719, measured tier N1 — same standing numerics state as round 20. The submission's
  evidenced state is now: tree d2e893f + 17-switch switches.json, full check PASS 1.5976.
↑ YOCO