GLM-5.3 1.60×
Harness: Claude Code · switches: YOCO_FAST_SYNC=1 YOCO_FAST_BCAST=1 YOCO_FAST_NOZERO=1 YOCO_FAST_PIPE=8 YOCO_FAST_SHARDGRADS=1 YOCO_FAST_RMSNORM=1 YOCO_FAST_CASTCOPY=1 YOCO_FAST_WREFRESH=1 YOCO_FAST_SHARDMUL=1 YOCO_FAST_DDPGRADVIEW=1 YOCO_FAST_DDPMEAN=1 YOCO_FAST_BF16X=1 YOCO_FAST_TLAUNCH=1 YOCO_FAST_SCALESYNC=1 YOCO_FAST_MPU=1 YOCO_FAST_GRAPH=1 YOCO_FAST_ROTARY=1
- Judged
- 1.60×
- Tier
- N1
- Self-best
- 1.60×
- Rounds
- 16
- Hours
- 10.0
- API cost
- $37.2
- Input tokens · 99.4% cached
- 123 M
- Output tokens
- 1.0 M
- Requests
- 931
§ 02Key levers
- Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
- ✗ still NCCL (
all_to_all+ 8-stage pipeline); exchanged payload switched to bf16 (BF16X, 1.30× → 1.38×) - Forward / backward CUDA graph
- ✓ but forward only, and only in round 21: 1.51× → 1.60×; earlier judged "the judge's criterion hook doesn't allow it"
- Newer cuBLAS
- ✗
- Retention / attention kernel swap
- ✗
- Other main levers
- fp32 master grads materialized only for this rank's shard; Triton kernels launched directly after caching (1.38× → 1.45×); bitwise-identical Triton RMSNorm; DDP reduction switched to AVG
§ ARaw trace
Raw work log20 sections
1.6 hRound 2 (2026-11-06 18:10–18:50): pipelining the Adam step with the shard exchange (YOCO_FAST_PIPE)
Profile after round 1 (/tmp/r2/profile, 5 updates, rank 0): per-update kernel time ≈ 1.11 s instrumented; gemm 65.7 ms, nccl 50.8 ms (DDP all-reduce 31.7 ms mostly overlapped + OSS exchange 19.1 ms fully serialized after the Adam step), memcpy/cast 43.8 ms, elementwise 26.3 ms, optimizer multi_tensor (Adam) 18.3 ms. The optimizer section is still Adam (18.3) + exchange (18.6) back to back = ~37 ms with zero overlap — the round's hotspot. Change:YOCO_FAST_PIPE=Kinyoco/models/_fast_optim.py— pipelinedOSS.step. The local shard is split into K=8 near-equal param-aligned chunks; the inner fused Adam takes one step over K param groups (fairseq'sFusedAdamV2.steplaunches onemulti_tensor_adammulti-tensor kernel set per group), and each chunk'sall_to_allexchange is issued right after that group's kernels are enqueued — from a wrapper around fairseq's module-globalmulti_tensor_applier— so exchange k overlaps Adam chunk k+1 (NCCL waits at enqueue for the current stream's pending writes to the exchange input; the K exchanges serialize on NCCL's stream and the last.wait()covers all). Tier N0: same per-element Adam math over the same tensors (bitwise whatever the grouping), same bytes exchanged, groupstepadvances by exactly 1. Two failures found and fixed before any valid timing: 1. First design stepped the inner Adam K times per update. The judge's timing harness counts updates with torch's global optimizer-step post hook — "the first optimizer object that ever steps defines the iteration (in the reference the shard-local FusedAdam inside fairscale's OSS; one step per update)" (bench/timing.py). With 8 inner steps/update the timer counted 8 "iterations" per update:timereturned ERROR ("the random-number stream differs from the reference from update 1 on"), bogus 7.24× / 0.0323 s-per-update, windows of 10 chunks ≈ 1.25 real updates, StopTiming after 3 updates (other_optimizer_steps [3], window_range 0.62). The judge defines an update as one FusedAdam step, so the redesign above keeps exactly one inner step per update. (torch.optim.Optimizer._patch_step_functionwraps the class'sstepat first instantiation, so the patchedOSS.stepand the innerFusedAdamV2.stepeach still fire the hook once per update, in the reference order:other_optimizer_steps [24]again.) 2. Rank-1 split sizes flipped in the rewrite (`RuntimeError: Split sizes doesn't match total dim 0 size`): the output tensor is the *other* rank's bucket slice (n_other elements), sooutput_split_sizesmust be[n_other, 0],input_split_sizes[n_mine, 0]. Fixed. Correctness evidence: - A/B (real 2-rank torchrun entry, deterministic mode, 8 updates, all four switches incl. PIPE=8): per-update loss and gnorm bitwise identical to the reference OFF run (/tmp/r2/ab_pipe2.logvs/tmp/r1/ab_off3.log; reference self-determinism was shown in round 1).timeresult (/tmp/r2/time2): speedup_vs_frozen_reference 1.1041 (0.2337 → 0.2117 s/update), status ok, window range 0.0031, no problems (gate 1's RNG-stream check passed:other_optimizer_steps [24]like the reference), beyond_reference_noise true. PIPE adds ≈ +2.4% over round 1 (Adam+exchange ~37 ms → ~33 ms; the overlap is limited by HBM bandwidth contention between the Adam kernels (~2.6 TB/s) and the NCCL exchange (~0.7 TB/s), not by ordering).
0.9 hRound 1 (2026-11-06 17:10–18:05): optimizer section (zero-sharding exchange + per-parameter sync loops)
Profile of the unchanged reference (/tmp/r1/profile, 5 updates, rank 0): per-update kernel time ≈ 260 ms (instrumented; real update = 233.7 ms): nccl 84 ms (OSS fp32 master broadcast 37 ms = two sequential 18.5 msBroadcast_RING_LLof 6.9 GB each, fully serialized; DDP bf16 grad all-reduce 39 ms of which only ~9 ms exposed, rest overlaps backward), gemm 66 ms, memcpy/cast 44 ms (~850 copy kernels/update: per-parameter bf16↔fp32 sync loops, ~240 params), elementwise 31 ms, optimizer multi_tensor 18 ms. GPU idle 15.7% in the instrumented window; biggest gaps ~2 ms/update at the optimizer boundary (OSS'storch.cuda.synchronize+ CPU-blocking waits + per-param Python loops). Changes (all inyoco/models/_fast_optim.py, installed atimport models, each behind its own environment switch in/submission/switches.json, off by default; all N0 — same elementwise math / same bytes moved): 1.YOCO_FAST_SYNC: fairseq_FP16OptimizerMixin._sync_fp16_grads_to_fp32,_sync_fp32_params_to_fp16,zero_gradandFairseqOptimizer.multiply_gradsreplaced bytorch._foreach_copy_/_foreach_mul_/_foreach_zero_batches (identical elementwise ops, ~4×240 individual kernel launches → a handful of batched ones). 2.YOCO_FAST_BCAST: fairscaleOSS._broadcast_params→ onedist.all_to_all_singleexchange of the two flat bucket buffers (grouped ncclSend/ncclRecv overlap both directions: microbench 37.2 ms → 18.6 ms), withasync_op=True+.wait()like the reference's per-handle waits. 3.YOCO_FAST_NOZERO: skip zeroing the fp32 master grads inzero_grad(every element is overwritten by the next update's grad sync before anything reads them). Bugs found on the way (evidence in/tmp/r1): - First BCAST version issued the exchange fire-and-forget. NOT sound: NCCL runs the kernel on its own stream, not the current stream (proved by/tmp/r1/test_stream.py,/tmp/r1/test_wait.py: a same-stream fill after a fire-and-forget exchange sometimes wins, sometimes loses). The A/B loss comparison diverged from update 3 (16.824/16.825 vs 16.823) and differed between two runs of the same config — a race, not a value change.async_op=True+.wait()(and the plain sync call) are properly ordered (20/20 and 6/6 at 6.9 GB); NCCL also waits for pending current-stream writes to the exchange *input* (/tmp/r1/test_input.py), so no pre-sync is needed. - My first "synchronous variant" test never took effect (_on()returns a bool compared to '2'), so both BCAST variants were the racy one. Fixed the guard, then the A/B matched. Correctness evidence: - A/B (real torchrun 2-rank entry, deterministic mode like the bench's numerics runner:CUBLAS_WORKSPACE_CONFIG=:4096:8, torch deterministic warn-only, cuDNN deterministic, flash-attn deterministic backward;/tmp/r1/det/sitecustomize.py,/tmp/r1/ab_run.sh): reference is perfectly self-deterministic (two identical OFF runs); SYNC-only, NOZERO-only, fixed-BCAST-only and all-three match the reference's per-update loss and gnorm exactly for updates 1–8 (/tmp/r1/ab_*.log). -all_to_allvs two broadcasts on OSS-like buffers with unequal shard sizes: bitwise equal (/tmp/r1/test_a2a.py). -check --quick: PASS (gate 0 structure, gate 2 bitwise at world 1 — covers SYNC and NOZERO at full tensor precision). BCAST only engages at world 2, where the gates do not record; its evidence is the A/B runs above plus gate 1's RNG check.timeresult (/tmp/r1/time): speedup_vs_frozen_reference 1.0851 (0.2337 → 0.2154 s/update), window range 0.0008, peak memory 61.75 GiB (unchanged), beyond_reference_noise true. Per-config screening from A/B runs (ups, not a verdict): SYNC ≈ 3.85, NOZERO ≈ 3.85 vs OFF ≈ 3.75 (small); BCAST ≈ 4.34 (the big one). Full check (/tmp/r1/check): PASS, speedup 1.0746 (all gates; numerics bitwise PASS, gate 3 skipped). Committed:yoco/models/_fast_optim.py+ themodels/__init__.pyhook; switches.json = all three on. Tier N0 (bitwise per gate 2 and A/B).
2.0 hRound 3 (2026-11-06 18:55–19:25): fp32 master gradients only for the optimizer shard (YOCO_FAST_SHARDGRADS)
Profile after round 2 (/tmp/r3/profile, 5 updates): gemm 335 ms, nccl 386 ms (DDP all-reduce + the 8 chunked exchanges, instrumented), memcpy/cast 218 ms, elementwise 132 ms, optimizer 104 ms per 5 updates. The clip-grads phase is 55.8 GPU ms per 5 updates and reads: bf16→fp32 grad sync (all 240 params), the l2 norm over all fp32 grads, the scale multiply over all fp32 grads. Host time per update 0.245 s; GPU busy 71%. Change:YOCO_FAST_SHARDGRADS=1— with--zero-sharding os, only the local shard's fp32 master gradients are ever consumed (Adam steps the shard), but the reference syncs *every* parameter's bf16 grad to an fp32 copy and multiplies all of them. Two pieces: 1. the gradient norm is computed from the bf16 gradients directly (they exist for all parameters on every rank after the all-reduce), via fairseq's ownmulti_tensor_total_norm— bitwise-equal to the reference's fp32 norm because the fp32 copies are exact float() upcasts and apex's multi-tensor l2norm reduces both dtypes identically (verified bitwise standalone:/tmp/r3/test_l2norm.py, several shapes/orders/ draws —torch.equalTrue); 2._sync_fp16_grads_to_fp32skips the parameters outside the shard; theirp32.gradstaysNone, so the scale multiply (_unscale_grads) skips them too. Failure found on the way: my first version skipped the sync but kept the reference's norm over the fp32 grads — WRONG: the norm reads *all* fp32 grads (fairseq'sutils.clip_grad_norm_over the full param list, no aggregation: every rank norms the whole set andtrainer._check_grad_normsrequires the per-rank norms to agree). The A/B run failed immediately with "gradients are inconsistent between workers" (rank 0 = 1.673 vs rank 1 = 1.989 — each rank had only its own shard synced). The bf16 norm fixes it: every rank norms the same complete set again. Tier N0. Correctness evidence: A/B (2-rank torchrun, deterministic, 8 updates, all five switches): per-update loss and gnorm bitwise identical to the reference OFF run (/tmp/r3/ab2.logvs/tmp/r1/ab_off3.log).timeresult (/tmp/r3/time): speedup_vs_frozen_reference 1.1441 (0.2337 → 0.2043 s/update), status ok, window range 0.0006, no problems. +3.6% over round 2 (the sync, the norm and the multiply each lose the other rank's half ≈ 31 GB of traffic per update). Full check (/tmp/r3/check): PASS, speedup 1.1396. Gate 2 is NOT_BITWISE — the 908 violations are exactly the non-shard master grads (rel_err ≈ 1: the adapter records them as zeros where the reference has the dead copies;exact_failures: 0); every tensor that is actually computed — loss, delta, Adam states, weights — is bitwise identical. Tier N1; gate 3 (statistical, 200 updates, frozen seed) ran and returned "equivalent" with **all six endpoint diffs exactly 0.0** — the training trajectory is bitwise the reference's, so the verdict has maximal margin and reproduces deterministically in the final evaluation.
3.2 hRound 5 — fused multi-tensor cast copy (YOCO_FAST_CASTCOPY)
Start 2026-11-06 20:06, end 2026-11-06 20:25. Hotspot from the fresh profile (/tmp/r5/profile): memcpy/cast 25.4 ms/update — the fp32↔bf16 gradient/weight syncs.torch._foreach_copy_has no fast path for mixed dtypes: it silently falls back to one unvectorized per-tensor kernel per parameter (measured: identical time and launch count to an explicit per-tensor loop, ~33x off the HBM roofline for the cast traffic). Change (_fast_optim.py, switchYOCO_FAST_CASTCOPY=1): one fused Triton kernel per copy list. Programpidcopies BLOCK contiguous elements of segmentblock_seg[pid]at offsetblock_off[pid](block layout cached per size list); tensors are passed as int64 pointer tables (tl.load(...).to(tl.pointer_type(...)), verified supported in triton 2.3). bf16→fp32 is exact; fp32→bf16 is cvt.rn like torch'scopy_(proven bitwise in the RMSNorm work). Elementwise → deterministic; guards fall back to foreach for anything but contiguous CUDA tensors of the expected dtypes. Rejected in-round: YOCO_FAST_SKIPWSYNC (skip the fp32→bf16 refresh of the non-owned half). The A/B caught it: loss matched updates 1–2 then diverged. My invariant was wrong — each rank's fp32 masters are ALL live: own shard ← Adam, other shard ← the OSS exchange every update, so the reference's full refresh is semantically required (only the non-shard fp32 *gradients* are dead, which is what round 3's SHARDGRADS correctly skips). Removed. Correctness evidence: unit bitwise suite/tmp/r5/castcopy_test.py(mixed sizes, odd tails for block masking, single tensors, 240 same-size, non-contiguous rejected) — all bitwise; A/B (/tmp/r5/ab_cast2.log, 2 ranks, deterministic, 8 updates, all seven switches): loss and gnorm bitwise identical to/tmp/r1/ab_off3.txt.timeresult (/tmp/r5/time): speedup_vs_frozen_reference 1.2067 (0.23373 → 0.1937 s/update), beyond_reference_noise true. −5.9 ms/update vs round 4 (0.1996 → 0.1937).check --quick(/tmp/r5/quick): PASS (gates 0 + 2), measured_tier N1, 908 violations (unchanged, all SHARDGRADS's), exact_failures 0.
3.6 hRound 6 — NCCL protocol / chunking (no gain)
Start 2026-11-06 20:33, end 2026-11-06 20:55. Target: the nccl category (55 ms/update after round 5 — 28.8 DDP all-reduce + 26.4 pipelined exchange). Findings: the DDP all-reduce kernels are named..._RING_LLeven for 25-37 MB buckets, which suggested LL's halved bandwidth — butNCCL_PROTO=Simple/LL128changed nothing at all (minimal 2-rank test: same kernel name for every setting; the name is not protocol- truthful in NCCL 2.20.5). Per-bucket arithmetic from the profile (0.24 ms per 37.5 MB bucket, ring moves 2x size per rank) gives ~310 GB/s — already ~70% of NVLink, so the all-reduce is near-optimal and mostly overlapped with backward. Judgetimewith NCCL_PROTO=Simple (1.2123), LL128 (1.2011), PIPE=16 (1.2104) vs baseline 1.2067 — all within the 0.006 noise band; no reproducible gain. The remaining exchange inefficiency (26.4 ms for 4.46 GB, ~170 GB/s effective, vs 18.6 ms unoverlapped in round 1) is SM contention with the concurrent Adam kernels, not a protocol choice. Conclusion: communication config is exhausted; switches restored to the round-5 set (PIPE=8, no NCCL vars). No code change, nothing to commit. Next: hide optimizer-section work under the exchange tail (weight refresh) and tune the fused cast kernel, then the gate-retention fp32 mask chains (12.2 ms/update of latency-bound elementwise).
3.8 hRound 7 — weight-refresh overlap + cast-kernel tuning
Start 2026-11-06 20:56, end 2026-11-06 21:15. Hotspot: the optimizer section's exposed tail — the fp32→bf16 weight refresh (10.3 ms/update measured) runs after the exchange is waited on. Change 1 (kept):YOCO_FAST_WREFRESH=1— the pipelined step (_oss_step) enqueues the own-shard half of the weight refresh right after the Adam kernels, before the exchange tail is waited on (those copies read only locally-stepped masters, current stream after Adam, so they execute concurrently with the NCCL exchange); the outer sync then refreshes only the other rank's half. A/B bitwise (/tmp/r7/ab_wr.logvs/tmp/r1/ab_off3.txt); quick check PASS (/tmp/r7/quick, 908 known violations, 0 exact failures).time: 1.2085 (/tmp/r7/time_wr, 0.1934 s/update) vs 1.2067 — the gain is ~+0.3 ms, within the noise band: the refresh's HBM traffic contends with the concurrent Adam+exchange kernels, eating most of the hiding. Kept (verified, ≥ 0), but weak. Change 2 (reverted):tl.multiple_ofalignment hints on the cast kernel's table-loaded pointers, hoping to enable 128-bit vectorization. Measured the OPPOSITE: 1.1357–1.1418 (hints made the kernel ~12 ms/update SLOWER — the hint on runtime-loaded scalar pointers pessimizes codegen). Reverted; the round-5 kernel stands. The fused kernel remains ~1.7x off roofline (10.3 vs ~6.1 ms) — a hand-vectorized variant (explicit 128-bit loads/stores per thread, e.g. via inline PTXld.global.v4/st.global.v4) is a possible later step. Current best: 8 switches,time1.2085, quick check PASS. Committed next.
4.9 hRound 8 — the fp32 master-grad scale multiply: shard-only + one fused kernel
Start 2026-11-06 21:20, end 2026-11-06 22:10. Hotspot found by attribution, not the profile row alone:elementwise_kernel<128,4,BinaryFunctor<float...MulFunctor>was 60.9 ms per 5 updates (12.2 ms/update, 1200 launches) — one kernel *per parameter*. Attribution (record_shapes trace/tmp/r8/shapes_trace.json, 2 updates, + 3 dispatch/monkeypatch probe runs/tmp/r8/probe*.log): - Every one of the 240 fp32 master grads is multiplied once per update by the_multiply_factor((2/2048) x clip_coef, a 0-dim fp32 CUDA tensor): fairseq'sFairseqOptimizer.steptakes thescale != 1.0 -> multiply_grads(1/scale)path (FusedAdamV2 ignoresscalefor fp32 grads), i.e._unscale_gradsinFP16Optimizer.step. - My earliermultiply_gradspatch already batched it astorch._foreach_mul_, but foreach has no fast path for a 0-dim CUDA tensor scalar — it silently falls back to one unvectorizedmul_kernel per tensor (visible as 240aten::mul_cpu_ops nested inside a singleaten::_foreach_mul_span, 1.8 ms of CPU launch time). - 240, not the ~120-param shard, becausebuild_fp32_paramsgives every fp32 master azeros()grad at construction —p.grad is not Noneis always true, so the non-shard half (dead: never written by the shard-filtered sync, never read — Adam steps the shard, the norm runs on bf16 grads) was scaled too. 0 * c == 0 keeps the skipped ones bit-identical. - (The bf16mul.outrows at param shapes are DDP's div-by-world-size write-back — C++, out of reach; the [25165824]/[9437184]copy_rows are one-time Adam state init.) Change:YOCO_FAST_SHARDMUL=1inyoco/models/_fast_optim.py: -multiply_gradsfilters the grad list to the optimizer shard (same_shard_param_idsset the sync uses; only when the_FP16OPTchain matches). - New_mulsc_kernel(Triton): all fp32 grads scaled in ONE launch (block layout reuses the castcopy copier); the scalar is read from device memory — no D2H sync. Guards mirror_cast_copy(fp32/contiguous/CUDA/16-byte aligned), else foreach. - Lonev * smultiply, no add — no FMA-contraction pitfall. Verification: unit bitwise suite/tmp/r8/mulsc_test.pyPASS (random/denormal/ zero/overflow x 6 scalars, odd sizes, fallback guards); A/B 8-update run bitwise identical to the reference (/tmp/r8/ab_on.logvs/tmp/r1/ab_off3.txt: loss and gnorm match all 8 updates). Results:time1.2655 (/tmp/r8/time, 0.1847 s/update) vs 1.2085 — +4.7%, far beyond the noise band (removes ~6 ms of dead-tensor traffic and ~1.8 ms of CPU launch time, and the fused kernel is faster than the fallback on the kept half).check --quickPASS (/tmp/r8/quick). Tier: N1 (same class as SHARDGRADS: the dead non-shard fp32 grads keep their zeros — the 908 known violations stay, no new ones; every read value is bitwise the reference's). Committed next.
4.9 hCurrent summary (after round 8)
- Switch file/submission/switches.json: 9 switches —YOCO_FAST_SYNC=1,YOCO_FAST_BCAST=1,YOCO_FAST_NOZERO=1,YOCO_FAST_PIPE=8,YOCO_FAST_SHARDGRADS=1,YOCO_FAST_RMSNORM=1,YOCO_FAST_CASTCOPY=1,YOCO_FAST_WREFRESH=1,YOCO_FAST_SHARDMUL=1. - Latest full check: PASS, speedup 1.2573 (/tmp/r8/check/check.json, 2026-11-06 22:14). Latesttime: 1.2655 (/tmp/r8/time). Quick check PASS (/tmp/r8/quick). HEAD5c567b5, tree clean. - Gain breakdown (speedup over the 0.23373 s/update frozen reference): rounds 1–3 (foreach sync/bcast/nozero, PIPE=8): 1.147 → ~1.17; round 4YOCO_FAST_RMSNORM(fused Triton RMSNorm, bitwise): → 1.1763 (full check); round 5YOCO_FAST_CASTCOPY(fused multi-tensor cast sync): → 1.1998 (full check); round 7YOCO_FAST_WREFRESH(own-shard refresh under the exchange tail): +~0.3 ms; round 8YOCO_FAST_SHARDMUL(shard-only scale multiply, one fused kernel): → 1.2573. - Dead ends: NCCL_PROTO/NCCL_ALGO (inert in NCCL 2.20.5), PIPE=16 (noise), skipping the non-shard weight refresh (wrong: the exchange writes those masters),tl.multiple_ofhints on table-loaded pointers (−12 ms/update),tl.exp(not bitwise vs torch), SKIPWSYNC (diverged). - Next candidates: fold the remaining shard scale multiply into the grad-sync kernel (fused fp32 = float(bf16) x c — same rounding, saves the second pass); hand-vectorized (128-bit) cast kernel; gate-retention fp32 mask chains.
5.2 hRound 9 — fused scale-into-sync (YOCO_FAST_SCALESYNC): no gain, switch off
Start 2026-11-06 22:15, end 2026-11-06 22:30. Follow-up on round 8's hotspot: after SHARDMUL the scale multiply still costs a separate pass over the shard's fp32 grads (~4.5 ms, one fused kernel). Fusing it into the bf16→fp32 grad-sync kernel (p32.grad = float(p.grad) * c, the same single fp32 multiply as copy-then-mul) removes that pass entirely. Change:YOCO_FAST_SCALESYNC=1— the fast clip (where_multiply_factoris final) calls a scaled sync instead of syncing before the norm; the later_unscale_gradsmultiply is skipped via a flag. Unit bitwise suite/tmp/r9/scalesync_test.pyPASS (7 scalars × bf16 patterns incl. denormals/huge/zeros, unscaled path unchanged, guards); A/B 8 updates bitwise identical (/tmp/r9/ab_on.log).time: 1.2593 (/tmp/r9/time, 0.1856 s/update) vs 1.2655 — slightly WORSE (delta 0.0062, just outside the 0.006 noise band). Why: the unscaled sync used to run *before* the clip's cross-rank norm allreduce and overlapped it; the scaled sync must run *after* the factor is final, so the copy fell out of that overlap, and the multiply it removed was already hiding under the PIPE exchange tail (the optimizer section is exchange-bound: 27 ms exchange vs 16.8 Adam + 4.5 multiply). Net ≈ 0. Decision: switch removed from/submission/switches.json(back to the round-8 set, the best-evidence state: full check PASS 1.2573); the code stays, off by default. Committed. Lesson: in this pipeline an elementwise pass is only worth fusing away if it sits on the critical path *after* the last overlap opportunity — measure, don't assume.
5.5 hRound 10 — DDP wrapper overhead: bucket cap (no gain) + gradients as bucket views (kept)
Start 2026-11-06 22:30, end 2026-11-06 22:45. Hotspot from the fresh profile (/tmp/r9/profile): the DDP allreduce's own overhead — ~119 buckets per update (cap 25 MB), ~240 grad→bucketMemcpy DtoDcopies per update (4.2 ms) plus the div-by-2 write-back (4.8 ms, C++), 55.7 ms of AllReduce kernel time per update. Change 1 (not adopted):YOCO_FAST_DDPBUCKET=<MB>— raise the bucket cap via aDistributedFairseqModelwrapper. Values bitwise (A/B/tmp/r9/ab_b256.log); timing 256 MB: 1.2675, 64 MB: 1.2710 then 1.2573 on repeat — straddles the baseline, inside the 0.006 noise band. No reproducible gain (per-element allreduce was already ~310 GB/s on the big buckets, per round 6). Code kept, switch off. Change 2 (kept):YOCO_FAST_DDPGRADVIEW=1—gradient_as_bucket_view=Trueon torch DDP (via aDistributedDataParallel.__init__wrapper; fairseq's wrapper passes no such kwarg). The autograd engine then accumulates straight into the bucket storage: the ~240 grad→bucket DtoD copies per update disappear. Per-element values identical (allreduce sums and the div-by-2 act on the same elements in the same order). A/B 8 updates bitwise identical (/tmp/r9/ab_gv.log);check --quickPASS (/tmp/r9/quick, tier N1).time: 1.2828 then 1.2814 on repeat (/tmp/r9/time_gv,/tmp/r9/time_gv2; 0.1822/0.1824 s/update) vs 1.2655 — a reproducible +2.4 ms/update, beyond the noise band. Committed. Switch set now 10 on.
7.9 hCurrent summary (after round 16)
- Submission state: 14 switches on in/submission/switches.json: SYNC, BCAST, NOZERO, PIPE=8, SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, BF16X, TLAUNCH, SCALESYNC, MPU. Latest full check PASS 1.4737 (/tmp/r16/check, tree HEADd69d0ab); standalonetimewith the same tree: 1.4964 (/tmp/r16/time) and 1.4992 (/tmp/r16/time2), s/update 0.1559-0.1562 vs frozen reference 0.23373. Gate 2 exact failures 0, tier N1 (the standing dead-tensor class from SHARDGRADS+BF16X: 1591 unresolved, 7719 bitwise — unchanged by MPU), gate 3 decides and passed. - Gain breakdown: rounds 1-3 ~1.15; RMSNorm -> 1.1763; CASTCOPY -> 1.1998; SHARDMUL -> 1.2573; DDPGRADVIEW -> 1.2835; fp64 scalar fix -> 1.2992; BF16X -> 1.3806; TLAUNCH -> 1.4526; SCALESYNC -> 1.4859; MPU -> ~1.49 (mean of 1.4964/1.4992/1.4737). - Wall structure now (r16 profile, instrumented): forward CPU ~51 ms vs ~35.5 GPU; backward CPU ~93 ms vs ~40 GPU; optimizer CPU ~25 ms vs ~20 GPU (Adam 16.4 at HBM roofline + bf16 exchange ~17 ms contention-bound, PIPE-overlapped); DDP allreduce 41.6 ms GPU off the wall (protocol inert, verified). Top elementwise GPU: aten::mul ~6.8 ms (~400 kernels), copies ~3.9 ms, adds ~3.7 ms, sums ~2.7 ms. - CUDA graphs are OUT for forward/backward: the judge counts criterion calls with a module forward hook and raises if it does not fire (bench/yoco_adapter.py:622); a graph replay never calls the module. The fp32-optimizer-step count is a Python wrapper onfp32_optimizer.step, so optimizer-internal rewrites stay viable. - Dead ends: NCCL_PROTO inert; PIPE 4-16 within noise; CUBLASLT_WORKSPACE_SIZE no gain; SKIPWSYNC wrong; multiple_of hints pessimize; tl.exp/tl.log not bitwise vs aten (blocks swiglu/retention/logsigmoid Triton fusions); DDP divide fold drops a bf16 rounding (rejected); apex fp32 Adam has no scale argument; lr is a baked-in kernel argument in apex's Adam (kills CUDA-graphing the optimizer step without a custom kernel). - Next candidates (priority): (1) cached Adam call — skip fairseq FusedAdamV2.step's per-param list building +.to()no-ops, re-issue the identical multi_tensor_applier call (N0, ~2-4 ms CPU); (2) Triton Adam with device-pointer lr (enables later graphing; bitwise needs fma/div_rn/sqrt_rn care, apex op order extracted in/tmp/r17/multi_tensor_adam.cu); (3) RMSNorm residual-add fusion (~104 kernels, N0); (4) rotary wrapper trim; (5) F.linear -> torch.mm dispatch trim.
8.2 hRound 17 (2026-11-07 01:13–01:21): YOCO_FAST_ADAMCALL — cached Adam lists: NO GAIN
- Hotspot: optimizer-phase CPU 25 ms/update (profiled) — fairseq'sFusedAdamV2.steprebuilds its 4 multi-tensor lists every step: per param a grad check, sparse check, state-dict lookup, two.to()no-ops and four appends (~350 params/update, 8 PIPE chunks). - Change:yoco/models/_fast_optim.py_patch_adamcall()behind YOCO_FAST_ADAMCALL (off): p/m/v lists cached per params-tuple key, grad list rebuilt per step, full identity validation (p.data, state dict, exp_avg/exp_avg_sq objects); any surprise falls back to the reference step for that call; identicalmulti_tensor_appliercall through the same module global (PIPE keeps firing). - Evidence: A/B 2-rank deterministic update-8 line byte-identical to reference (/tmp/r16/ab_adamcall: loss 16.655 / ppl 103209 / gnorm 4.973 / clip 100). - time: 1.4973 (/tmp/r17/time, 0.1561 s/update) vs 1.4964/1.4992 without — no gain. Conclusion: the removed Python was already hidden under the GPU Adam kernels; the optimizer phase's CPU number is dominated by the exchange.wait()(real GPU wait), not addressable Python. Switch committed but left off in switches.json. - Committede227faf. Submission switches back to the 14 evidenced ones. ### Lesson (recorded for future rounds) The profiled phase CPU times include host-blocking GPU waits; the addressable CPU in the optimizer phase was ~2-3 ms and already overlapped. Treat "phase CPU" as an upper bound on addressable work; check what fraction is.wait()/sync before attacking it.
8.6 hRound 18 (2026-11-07 01:22–01:50): rotary trim landed; DDP per-param division identified
- Probe first (/tmp/r18/rotary_bm_probe2.py, CUDA-graph replay for true GPU time): the reference rotary tiling (BLOCK_M=4/8, nw=4) is already the FASTEST config (10.1/5.9 us); bigger blocks are 16-30 us AND bitwise-unstable across tilings/codegen (fma contraction) — BLOCK_M bump abandoned on both counts. - Change (commit13474ef):YOCO_FAST_ROTARY—rotary_tail_kernel(reference kernel + in-launch bf16 tail copy, verified bitwise in all 4 branches), backwarddo.clone()dropped. A/B update-8 byte-identical;time1.4954 (/tmp/r18/time) — within noise, strictly less GPU work, kept on. - Attribution dig (the round's real finding): the top elementwise row (~250 bf16 scalarmul(grad, 1/world, out=grad)on weight-shaped tensors, 4.8 ms GPU + ~250 launches/update) has NO Python caller — it is **c10d DDP's per-parameter gradient division** in the C++ reducer before the allreduce. Also confirmed the current config's scaled sync engages in real 2-rank runs (det and non-det; diagnostics in/tmp/r18/ab_nodet): no mulsc, sync fused. The r16 profile's mulsc row did not reproduce (today's fresh profile/tmp/r18/profile: elementwise 48.1 ms/5, no mulsc). - Next round target: DDPregister_comm_hookwith per-bucketall_reduce(op=MEAN). With a hook registered DDP skips its division; NCCL MEAN computes fl((a+b)/2), bitwise equal to DDP's fl(a/2+b/2) because power-of-2 scaling commutes with rounding (subnormal edge to be probed over all 65536 bf16 patterns). Expected ~3-5 ms/update.
9.2 hRound 19 (2026-11-07 01:52–02:25): YOCO_FAST_DDPMEAN — per-bucket ncclMean, landed
- Basis: round 18's attribution — ~250 per-param bf16mul(grad, 1/world)kernels (4.8 ms GPU) are c10d DDP's C++-reducer pre-division before the SUM allreduce. - Bitwise proof first (/tmp/r19/mean_probe.py, 2 ranks): DDP's fl(a/2 + b/2) vs ncclMean's fl((a+b) x 0.5) over ALL 65536 bf16 bit patterns x 4 pairings (same/rotated/negated/x3): 0/65536 mismatches (ws=2; scaling by a power of 2 commutes with the single bf16 rounding).dist.ReduceOp.AVG(-> ncclMean) exists in torch 2.3.1; MEAN does not. - Change:YOCO_FAST_DDPMEANin_fast_optim.py— patchesDistributedDataParallel.__init__(chained after DDPGRADVIEW's) toregister_comm_hook(pg, hook)at power-of-2 world size; the hook runs one `dist.all_reduce(bucket.buffer(), op=ReduceOp.AVG, async_op=True).get_future() .then(lambda f: f.value()[0])` per bucket. With a hook registered the reducer skips its per-parameter division (torch documents this), so the x1/2 is fused into the allreduce kernel itself. - Failure debugged en route: first A/B hung >420 s with no output (grep pipe had buffered everything). Rerun with output to a file + NCCL_DEBUG=WARN showed the real error instantly: `result.isPyObject() INTERNAL ASSERT FAILED python_comm_hook.cpp:43` — a comm hook must return a future completing with a Python object; torch 2.3.1's canonicalallreduce_hookchains `.then(lambda fut: fut.value()[0])`. Mine returned the raw Work future. Added the same chain -> clean run. (AVG itself was never the problem; the probe had bypassed the hook machinery.) - A/B (8 updates, 16 switches): update-8 line byte-identical to reference (loss 16.655 | ppl 103209 | gnorm 4.973 | clip 100),/tmp/r19/ab_mean8.log. - time: 1.5021 (/tmp/r19/time, s/update 0.1556, beyond_reference_noise true) — best singletimeso far (previous standalone: 1.4954/1.4964/1.4992, mean ~1.497). Gain over them ~+0.3%, i.e. within single-run noise: the 4.8 ms of division kernels were mostly hidden under comm/backward overlap, so eliminating them buys launch/gap overhead, not the full 4.8 ms. - check --quick: PASS, gate-2 fingerprint unchanged (1591 unresolved / 7719 bitwise, exact_failures 0, dtype_changes 0, N1) — DDPMEAN contributes nothing to the standing comparison (/tmp/r19/quick). - Kept on: bitwise-safe (proven + byte-identical A/B), strictly less GPU work, best time yet. Tier N0 on its own. - Committed; switches.json now 16 switches.
9.2 hCurrent summary (after round 19)
- Switch file:/submission/switches.json— SYNC, BCAST, NOZERO, PIPE=8, SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH, SCALESYNC, MPU, ROTARY (16 switches). - Latest full check: PASS 1.4737 at 14 switches (/tmp/r16/check, tree 13474ef minus MPU? no — full check predates MPU/ROTARY/DDPMEAN). DDPMEAN/MPU/ROTARY passedcheck --quickwith the identical standing N1 fingerprint (1591 unresolved, 0 exact failures). A full check with all 16 is the next milestone once one more gain lands. - Besttimeevidence: 1.5021 (r19, 16 switches) vs 1.4970 mean of r16-r18 runs. - Gain breakdown (vs frozen reference 0.23373 s/update): fused/trimmed elementwise sync + cast paths (SYNC/CASTCOPY/SCALESYNC/SHARDGRADS/SHARDMUL ~+13%), PIPE+Adam-call trimming (~+8%), BF16X weight-exchange (~+8%), RMSNORM fused (~+4%), misc (NOZERO/BCAST/DDPGRADVIEW/TLAUNCH/MPU/ROTARY/DDPMEAN, each ~0-1%). - Dead ends: CUDA-graphing fwd/bwd (adapter's criterion forward-hook contract), ADAMCALL list caching (no wall gain — Python overlapped), rotary BLOCK_M retuning (reference fastest + bitwise-unstable), DDPMEAN's 4.8 ms (hidden under overlap). - Next candidates: RMSNorm residual-add fusion (~104 add kernels, 2.6 ms GPU, N0); F.linear->matmul dispatch trim in the MPU fast path; Triton Adam with device-pointer lr (bitwise risk: fma/div_rn/sqrt_rn must match apex's op order exactly).
9.4 hRound 20 (2026-11-07 02:15–02:36): full check PASS at 16 switches; F.linear dispatch trim backfired
- Full check (all 16 switches, tree353e7b2): PASS 1.4719, gate-2 fingerprint unchanged (1591 unresolved / 0 exact failures / N1) (/tmp/r20/check). This is the submission's evidenced state. - Profile analysis (r18 trace, 15-switch config): per update the GPU is busy ~139 ms, idle ~17 ms. The idle concentrates in ONE ~45 ms window per cycle whose host ops are FORWARD dispatches: 138aten::linear-> 164aten::matmul-> 138aten::mmchains (~17 ms host per profiled forward) plus view/reshape/empty_like churn from the retention layers. Optimizer tail (Adam 16.5 + cast 6.9 + SendRecv 16.3) is already pipelined (SendRecv hides under Adam; HBM-bound ~23 ms, 57 GB) — my earlier "exposed bcast" reading was wrong. DDP AR (40 ms kernel time, RING_LL) is bucket-overlapped under backward; NCCL_PROTO=Simple was already shown inert (r13). - Change attempted (YOCO_FAST_LIN): collapse the linear->matmul->mm dispatch chain in the MPU fast path to a directtorch.mm(x, w.t())(2-D) /torch.matmul(n-D) with a cached transposed view. Byte-identical A/B (update-8 line identical) — but judgetimeregressed to 1.3789 (0.1695 s/update; window_range 0.0028 — steady, real), while the 16-switch control right after measured 1.5128 (0.1545). - Why (understood after the fact):w.t()created in forward is a differentiable op, so every projection gains aTBackwardnode and AccumulateGrad receives the grad as a NON-CONTIGUOUS transposed view — autograd then materializes full-size contiguous grad copies every update (~170 params, ~12 GB traffic) and the DDP bucket-view/grad relationship degrades.F.linear's fused backward writesw.graddirectly; its extra dispatch layers are the price of that. Reverted (uncommitted; nothing to keep). - Lesson: a dispatch-count trim that changes the AUTOGRAD NODE SHAPE can cost far more than the dispatches it saves. Any forward-path rewrite must keep one backward node per projection (or be CUDA-graphed, which fixes node shapes by construction). - Files:/tmp/r20/switches_lin.json(17-switch test config, not the submission's),/tmp/r20/time_lindeleted; submission remains 16 switches +353e7b2.
9.4 hCurrent summary (after round 20)
- Switch file:/submission/switches.json(16 switches: SYNC, BCAST, NOZERO, PIPE=8, SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH, SCALESYNC, MPU, ROTARY). Tree353e7b2. - Latest full check: PASS 1.4719 (/tmp/r20/check). Besttimeevidence: 1.5128 (control,/tmp/r20/time_ctldeleted after reading; value recorded here) and 1.5021 (r19). - Next candidates: (1) Triton fused Adam (integrate the fp32->bf16 refresh cast, read bf16 grads directly): saves ~6-9 GB/update of the HBM-bound optimizer tail (~4 ms, 2.5%); bitwise risk mitigated by disassembling apex's SASS for fma contraction + dual-check harness. (2) CUDA-graph the model forward (+criterion eager, backward via make_graphed_callables-style split): kills the forward host cost + its idle (~8-12 ms), highest remaining gain, highest risk — the adapter allows it (criterion hook fires outside the graph; no RNG in forward). (3) RMSNorm residual-add fusion (~1 ms, N0). - Dead ends so far: CUDA-graph fwd/bwd via adapter hooks (criterion hook raises), ADAMCALL (no wall gain), rotary BLOCK_M (reference fastest + bitwise unstable), DDPMEAN's full 4.8 ms (hidden under overlap), F.linear->mm dispatch trim (TBackward + non-contiguous AccumulateGrad, -9%).
9.9 hRound 21 (2026-11-07 02:38–03:02): YOCO_FAST_GRAPH — CUDA-graphed decoder forward, +6.7%
- Hotspot (from the r18 trace): the forward is host-bound — 138 aten::linear ->
164 aten::matmul -> 138 aten::mm dispatch chains plus view/reshape churn per update;
~17 ms/update of GPU idle concentrated in the forward window. Round 20's dispatch trim
backfired, so the remaining lever for the same bottleneck class was eliminating host
work wholesale.
- Probes first:
- apex capturable_master Adam (fused bf16 grad read + bf16 weight write): m/v bitwise
identical to the reference kernel, fp32 masters differ in ~1% of elements at 1-2 ulp
(different contraction in the p-update), bf16 weights identical — viable N1/N2 path
worth ~2.3 ms, NOT taken (yet).
- my castcopy table kernel already runs at 2969 GB/s (faster than aten copy_ 2232):
the 6.88 ms castcopy row is refresh ~3 ms + grad sync ~3 ms, both near HBM peak —
no bitwise-safe slack in the optimizer tail (it is HBM-bound at ~60 GB/update).
- Change (YOCO_FAST_GRAPH): CUDA-graph YOCOModel.forward (the yoco_3b decoder).
The forward is fixed-shape (1,2048) int64, RNG-free (no dropout in the model) and
collective-free, so its kernel sequence captures once (call 3, inside the judge's
warmup) and replays bitwise identically at ~zero host cost. Criterion + backward stay
eager (adapter hooks fire; DDP's per-bucket reduction hooks keep firing); the optimizer
backward passes retain_graph=True so the captured autograd graph is reused (the
patch is otherwise an exact copy of the reference's, including _needs_sync = True).
Capture failures or unexpected args/shapes fall back to the eager forward permanently.
- Two bugs found by the A/B before landing (both fixed): (1) I first patched
models.transformer.LLaMA — the arch in use is yoco (YOCOModel in models/yoco.py);
(2) my backward copy dropped the reference's self._needs_sync = True — the grad sync
never ran and the model froze at update 3 (losses plateaued at 16.82). The A/B
per-update diff caught both; after the fix all 8 updates + the summary line are
byte-identical to the reference (/tmp/r21/ab_graph2.log).
- time: 1.602 (/tmp/r21/time_graph, s/update 0.1459) vs 1.5128 control without
the graph — +6.7%.
- check --quick: PASS, standing fingerprint unchanged (1591 unresolved / 0 exact
failures / 0 dtype changes / N1) (/tmp/r21/quick_graph).
- Adopted: switches.json now 17 switches. Full check started (/tmp/r22/check).9.9 hCurrent summary (after round 21)
- Switch file:/submission/switches.json— 17 switches (SYNC, BCAST, NOZERO, PIPE=8, SHARDGRADS, RMSNORM, CASTCOPY, WREFRESH, SHARDMUL, DDPGRADVIEW, DDPMEAN, BF16X, TLAUNCH, SCALESYNC, MPU, GRAPH, ROTARY). - Latest full check: PASS 1.4719 at 16 switches (/tmp/r20/check); GRAPH passed quick + A/B byte-identical; full check at 17 switches pending (/tmp/r22/check). - Besttime: 1.602 (r21, 17 switches). - Gain breakdown: elementwise/cast fusions ~+13%, PIPE/Adam-call trims ~+8%, BF16X ~+8%, RMSNORM ~+4%, GRAPH ~+6.7%, DDPMEAN/MPU/ROTARY/misc ~+1-2% total. - Dead ends: LIN dispatch trim (-9%, TBackward + non-contiguous AccumulateGrad), rotary BLOCK_M, ADAMCALL, DDPMEAN's full 4.8 ms (hidden), capturable-master Adam (2.3 ms at 1-ulp master drift — available as a future N1 lever). - Next candidates: graph the criterion+backward? (needs manual DDP reduction — probably negative); capturable-master Adam (~2.3 ms, N1); RMSNorm residual fusion (~1 ms, N0); forward-idle leftovers (retention view/reshape churn now inside the graph — gone).
—Round 22 (started 02:38) — full-check result + YOCO_FAST_TADAM
- Full check at 17 switches: PASS, speedup 1.5976 (/tmp/r22/check, gate-1 timing
0.1463 s/update, window_range 0.0013). Gate 3: all 6 endpoints within 3σ (worst
|d|/σ = 1.09 at loss_u100_200); exact_failures 0, unresolved 1591, bitwise_identical
7719, measured tier N1 — same standing numerics state as round 20. The submission's
evidenced state is now: tree d2e893f + 17-switch switches.json, full check PASS 1.5976.