GLM-5.3 1.05×
Harness: Claude Code · switches: NANOCHAT_CONCAT_MB=2 NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs NANOCHAT_LOADER_PROC=1
- Judged
- 1.05×
- Tier
- N1
- Self-best
- 1.05×
- Rounds
- 16
- Hours
- 10.0
- API cost
- $58.2
- Input tokens · 99.6% cached
- 202 M
- Output tokens
- 1.0 M
- Requests
- 1,430
§ 02Key levers
- Micro-batch regrouping 4×16 → 2×32
- ✓
NANOCHAT_CONCAT_MB=2, alone only 1.009 - Scope of hand-written kernels replacing Inductor glue
- ✗ tiled transpose op 0.978, tiled cross-entropy 1.001, both reverted
- Per-shape cuBLASLt algorithm selection
- ✗
- FP8 weights quantized once per step
- tried, reverted (1.000, no gain)
- Fast accum on backward GEMMs
- tried, reverted (failed statistical gate, −11.95σ)
- Other main levers
max-autotune-no-cudagraphs(1.009 → 1.023), data fetch moved to a subprocess with device check fixed (min window 1.023 → 1.047)
§ ARaw trace
Raw work log25 sections
1.1 hRound 1 — FP8 weight-quantization cache (NANOCHAT_FP8_WCACHE)
- Start/end: 2026-11-06 ~15:10 – 16:45 UTC. - Profile first (judgeprofile,/tmp/prof-ref-1, reference tree): per update, the FP8 "glue" around the 6 converted linears is eager ops inside the opaque_Float8Matmul, and the weight quantization is recomputed on every one of the 4 micro-batches in *both* directions: forward quantize chains (amax→scale→mul→clamp→cast) plus backward recomputes writing row- and col-major, ≈16–20 ms/update across the 24 layers, vs. weights only changing inoptimizer.step(). Also noted: grad_weight GEMMs (K=32768) run at 144 CTAs vs 132 SMs (wave quantization) — the target of Round 2. - Change:nanochat/fp8.py— persistent per-Float8Linearbuffers_w_fp8(row-major e4m3),_w_fp8_t(col-major e4m3),_w_qinv(f32), filled byrefresh_fp8_weight_caches(model)once per update from the training loop (scripts/base_train.py, top of the step, before the micro-batches; the weights are final there). The compiled path (_Float8MatmulCached) reads the buffers instead of re-quantizing per call; eager calls (judge's eval, sampling) keep the reference path. The refresh reproduces Inductor's quantize arithmetic bitwise (f64 reciprocal → f32 scale, clamp, cast; verified against the generated kernel source). SwitchNANOCHAT_FP8_WCACHE, default off. - Bugs found and fixed while validating: the first version refreshed from a post-step hook inMuonAdamW.step()that was never registered → stale caches from update 1 (first quick check/tmp/check-r1-quick: loss frozen at 10.39, all 175 params differ from update 1). Fixed by moving the refresh into the training loop top (nanochat/optim.pyreverted to upstream). - Result (check --quick,/tmp/check-r1-quick2): status NOT_BITWISE,exact_items_oktrue, rows/updates/hyper-parameters all identical. 4 params differ at update 0 (c_proj weights h.2/13/17/…), all 175 from update 1 (the 4 ULP-level perturbations compound). Per-update losses match the reference record run to ≤2e-4 over 10 updates (10.397737 identical at step 0; 9.847431 vs 9.847258 at step 8) — healthy trajectory, far below seed noise σ≈0.023. - Why not bitwise (investigated to root cause): the reference's AOTAutograd backward does not save the fp8 input tensors forattn.c_proj/mlp.c_proj; it recomputes their input quantization (amax over the FA3 output / relu output) inside backward kernels whose reduction order differs from the forward's amax → ~1 ULP scale differences → ~16 differing fp8 bytes per tensor → grad_weight differs by ~1e-9 (measured:/tmp/grad_cmp_result.log, 48 c_proj weights, 0.6–1.5 % of elements, maxabs ≲1.5e-8; qkv/fc/lm_head grads bitwise identical). My version saves the forward's quantization (cheaper) and cannot reproduce Inductor's fused recompute reduction order. Inherent N1 — same class as the judge's accepted reduction-order control tree. - Tier: N1 (exact items pass; statistical gate 3 to be confirmed by the full check). -time: run in progress →/tmp/time-r1(result appended below). -timeresult:/tmp/time-r1—speedup_vs_frozen_reference1.0004 (s/update 1.0391; windows 1.0338/1.0419/1.0417 vs frozen ref 1.0358/1.0409/1.0419 → per-window ratios 1.0019/0.999/1.0002, smallest 0.999). No gain — far below the 1.02 bar. The weight-quantize chains the cache removes were worth only ~1–2 ms/update in practice (Inductor had fused much of that work into kernels that read the weights anyway; the remaining saved time is offset by the refresh + extra saved-tensor traffic of the cached path). Switch kept OFF in/submission/switches.json(an N1 change with no measured benefit is not worth the divergence); the code stays in the tree behindNANOCHAT_FP8_WCACHE(default off) for possible later use.
1.3 hRound 2 — micro-batch concatenation (NANOCHAT_CONCAT_MB)
- Start: 2026-11-06 ~16:50 UTC. - Hotspot (from the Round-1 profile of the reference): every fp8 GEMM runs 4× per update at batch 16 (M=32768); grad_weight GEMMs (K=32768) launch 145×2×4 = 1160 backward GEMMs/update with wave-quantization tails, plus ~1900 glue-kernel launches. Grouping the same rows into 2 chunks of (32, 2048) halves the GEMM count and doubles each GEMM's M (better tile occupancy, fewer tails). - Change:scripts/base_train.py— optionalNANOCHAT_CONCAT_MB(default 1 = off): process CONCAT_MB loader batches per forward/backward through a persistent staging buffer (the loader reuses its own GPU buffer on every yield, so each prefetched batch is copied into its slice of the staging buffer after the chunk's backward — single stream, so ordering is safe). Rows and their order are unchanged (gate 2 allows different micro-batch grouping); the loss is scaled byconcat_mb / grad_accum_steps= 1/2 (exact power of two) so every row contributes exactly the same gradient as the reference. Numerics: the grad_weight GEMM now reduces over K=65536 in one accumulation instead of four of 32768 → different rounding → N1. - check --quick (/tmp/check-r2-quick, configNANOCHAT_CONCAT_MB=2): status NOT_BITWISE,exact_items_oktrue (rows_per_rank 640 identical; rows_available 672 — one extra prefetched batch, consumed rows unchanged), 52 params differ at update 0 (the 48 c_proj/mlp_proj weight grads whose K-reduction regrouped + 4 more), 175 from update 1. Peak memory 94.04 GiB (fits). Train losses vs reference: drift 2e-4 at step 0 growing to 5.6e-3 at step 8 — ~0.25 of the seed-σ at endpoint 1 (3σ bars: 0.068/0.039/0.024/0.041/0.074/0.099; seed σ grows with training, so the amplification applies equally to the much smaller perturbation here). -time: run in progress →/tmp/time-r2. -timeresult:/tmp/time-r2—speedup_vs_frozen_reference1.0088 (s/update 1.0304; windows 1.0215/1.0335/1.0361 → ratios 1.0140/1.0072/1.0056, smallest 1.0056). Below the 1.02 bar on its own, but the fastest updates drop to 1.013 s (reference floor ≈ 1.027) — the real kernel gain is ~1.5–2 % and occasional spikes (up to 1.063 s, vs the reference's own max 1.048) eat it. Peak memory 94.04 GiB (reference 55.94). The spikes are the next thing to attack (allocator stalls at 94 GiB are the main suspect: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is the cheap test). - Decision: code committed (switch default off); adoptNANOCHAT_CONCAT_MB=2only as part of a stack that clears the bar (timing of concat + fast-accum backward in progress).
1.4 hRound 3 — fast accumulation on the backward FP8 GEMMs (NANOCHAT_FP8_FAST_BWD)
- Start: 2026-11-06 ~16:55 UTC. - Hotspot (from the Round-1 reference profile): of ~454 ms/update of FP8 GEMM time, the backward half (grad_input + grad_weight, K=32768, 1160 GEMMs/update) runs withuse_fast_accum=Falsewhile the forward already uses True (upstream's own choice innanochat/fp8.py). Fast accumulation skips the periodic fp32 promotion of the MMA accumulator inside the cuBLASLt kernel. - Change:nanochat/fp8.py— both backward_scaled_mmcalls (and the cached variant) takeuse_fast_accum=FP8_FAST_BWDfromNANOCHAT_FP8_FAST_BWD(default off). Same FP8 dtypes (e4m3/e5m2), same tensorwise scaling, same shapes — only the GEMM's internal accumulation changes: an arithmetic change the rules allow, judged by gates 2/3. Tier N1 when enabled. -time(/tmp/time-r3, config concat=2 + fast_bwd=1):speedup_vs_frozen_reference1.0334 (s/update 1.0059; windows 0.9945/1.0107/1.0127 → ratios 1.0415/1.0299/1.0288, smallest 1.0288 — above the 1.02 bar). Marginal contribution of fast_bwd over concat-only: ≈ +2.4 %. Steady-state floor now 0.987 s (reference floor 1.027). Peak memory unchanged at 94.04 GiB. - Decision: submission switches set to `{"NANOCHAT_CONCAT_MB": "2", "NANOCHAT_FP8_FAST_BWD": "1"}; full check launched →/tmp/check-r3-full` (gate 1 timing, gate 2 record — expect NOT_BITWISE/N1 — and gate 3's 150-update trajectory, the decisive gate for the fast_bwd accumulation change). - Full check (/tmp/check-r3-full, config concat=2 + fast_bwd=1): gate 1 timing 1.035 (window ratios 1.0417/1.0359/1.0350), gate 2 exact items pass, NOT bitwise as expected — but gate 3 REJECTED (verdictdifferent): endpoint biases −0.0745 (−3.29σ, bound 0.068), −0.0173 (−1.33σ), −0.0233 (−2.90σ), −0.1631 (−11.95σ), −0.1924 (−7.83σ), −0.1539 (−4.65σ). A systematic, growing drift — the fast-accum backward accumulation is a real training-numerics change, not noise (likely amplified by the K=65536 grad_weight GEMM under concat). fast_bwd is dropped from the submission (switches.json back to {} while nothing clears the bar); the code stays behind its switch, off. - Consequence for strategy: a sub-bar submission scores 0 exactly like the unchanged reference (gate 1 no_benefit), so the only path is a stack that clears 1.02 AND passes gate 3. Next: validate concat alone on gate 3 (check --n1,/tmp/check-r2-n1), then attack the concat timing spikes (floor 1.013 s, spikes to 1.05–1.06; allocator config and/or loader parquet stalls) with numerics-clean (N0) changes.
1.7 hRound 4 — concat validated on gate 3; attack the timing spikes
-check --n1on concat-only (/tmp/check-r2-n1): PASS, verdictequivalent— endpoint biases +0.0014/+0.0009/+0.0033/+0.0010/+0.0009/−0.0017, max 0.41σ (bound 3σ). The K=65536 regrouping noise washes out completely over 150 updates. Concat is a validated N1 change. - Remaining gap to the 1.02 bar: concat alone times 1.0088 (windows 1.0140/1.0072/1.0056); the update series shows a floor of 1.013–1.020 s with spikes to 1.05–1.06 s, and the windows grow +1.4 % over the run (reference +0.6 %) — a progressive slowdown. Suspects: caching-allocator fragmentation at 94 GiB peak (cudaMalloc/cudaFree stalls) or parquet row-group read stalls in the loader. Profiling the concat tree over updates 10–15 (/tmp/prof-concat) to discriminate. - Candidate fix A (zero code, N0):PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truevia switches.json (not in reserved_env; allocator behavior only, no kernel or value change).
1.7 hRound 4 (continued) — CUDA graphs dead end; prefetching loader
- CUDA graphs attempt (NANOCHAT_CUDA_GRAPHS=1→torch.compile(mode="reduce-overhead"), same kernels, N0): the timing run SIGSEGVs on both ranks (/tmp/time-r4-cg, status ERROR). FA3/cuBLAS workspaces under graph capture are not stable in this setup. Reverted; the lever is dead (also note gate 1 invalidates runs where CUDA graphs are "requested but skipped", so this lever was all-or-nothing from the start). - Prefetching loader (NANOCHAT_LOADER_SLOTS,nanochat/dataloader.py, default 0 = upstream): the profile showed the gaps are the loader's own work (rustbpe encode + best-fit packing, ~19 ms/batch, untraced by the profiler) landing at chunk boundaries. New opt-in variant_prefetched_bos_bestfit: a worker thread produces the byte-identical batches (packing logic copied verbatim) into a pool of N slot buffers;next()pops a finished batch and clones it out of the slot. Slot reuse is made explicitly safe: the consumer records a CUDA event after its clone and hands the slot back; the worker blocks on that event before refilling the slot (so both the slot's pinned staging buffer and device buffer are provably idle when reused). The free-slot handoff bounds the worker to N batches ahead — tokenization and parquet row-group bursts disappear from the critical path. Verified byte-identical against the synchronous loader over 12 batches (first version had a slot-reuse race caught by the test at batch 6 — fixed by the event-after-clone handoff; also noted rustbpe's encode is not safe to run concurrently from two threads, which only ever happens if two loaders run at once — the recipe runs one). -check --quickon concat=2 + loader_slots=4: running →/tmp/check-r4-quick.
2.9 hRound 5 — config levers: expandable_segments and cpp_wrapper (both dead)
- Profile forensics (from/tmp/prof-concat, torch.profiler over updates 12+; the trace covers 6 updates — cluster the 18-per-update AllGather kernels to segment it): per update the GPU spends 1.013 s in kernels: nvjet FP8 GEMMs 462.7 ms, Inductor "clone/transpose" kernels 176.7 ms, other Inductor glue (FP8 casts, norms) 154.8 ms, FA3 113.6 ms, amax reductions 41.8 ms, cross-entropy 20 ms, NCCL 23.5 ms. GEMMs (462.7 ms for ~573 TFLOP/update ≈ 1.2 PFLOPS effective) and FA3 are near roofline; the Triton glue runs at only ~1.5–1.6 TB/s. -PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True(N0, allocator-only):/tmp/time-r4-es→ 1.0069 (windows 1.0216/1.0372/1.0383, smallest ratio 1.0035) — slightly worse than concat-only (1.0088). Dead. -NANOCHAT_CPP_WRAPPER=1(Inductor C++ launcher, N0):/tmp/time-r5-cw→ 1.0053 (windows 1.0207/1.0389/1.0423, smallest ratio 0.9996). Window 1 marginally better, windows 2–3 worse (within the ~0.5 % run-to-run noise those windows show). Dead. - Run-to-run noise on windows 2–3 across 4 behaviorally identical concat runs: 0.52 %/0.60 % — larger than the reference's frozen spread (0.17 %). Screening 1–2 % changes on those windows is unreliable; need ≥2.5 % real gains.
2.9 hRound 6 — tiled-transpose op for the backward FP8 layout copies: REGRESSION, off
- The idea:_scaled_mmneeds row-major × column-major, so each Float8Linear backward makesgo_T = go_fp8.t().contiguous()and_to_col_major(in_fp8). A 64×64-tiled Triton transpose does the same copy bitwise-exactly at ~3.3 TB/s vs ~1.6 TB/s for the Inductor clones (and 300 GB/s for aten's.t().contiguous()). Also verified the reformulated GEMM 1 (w_col.t() @ go_fp8.t()— free views of copies the reference already makes) is **bitwise identical** to the reference GEMM 1 on representative shapes. - Implementation (nanochat/fp8.py, switchNANOCHAT_FP8_TTRANS, default off): the kernel is registered as a real aten op (torch.library.Library("nanochat_fp8", "DEF")+ CUDA/CPU/Meta impls) so dynamo/Inductor treat it as an opaque extern call. First triedtorch.library.custom_op— NOT opaque: dynamo inlines the triton kernel and Inductor re-fused it into strided epilogues. - Result:/tmp/time-r6-tt2→ 0.978 (regression), peak memory 94.04 → 106.76 GiB (+12.7 GiB, identical with both registration mechanisms). Category diff of the profile (/tmp/prof-r6-tt2vs/tmp/prof-concat): clone kernels 176.7 → 2.8 ms (−174), my tiled kernels +51.5 ms (running at the predicted ~3.2 TB/s), but other Inductor glue +122.3 ms and the plain FP8 casts now materialize separately (+36/+29/+27 ms kernels). The opaque op forces materializations Inductor previously fused into the clone kernels (cast + transposed write in one pass) and re-plans fusion/layout across the whole backward, making unrelated kernels (rmsnorm bwd, CE, amax variants) slower. **Fighting Inductor's planner one link at a time is a losing game; the reference's fused clones were already near-optimal as a system.** - Kept the code (switch off) for the record. Committed f402245.
2.9 hRound 6 conclusion → Round 7 direction
The remaining big non-roofline block is ~330 ms/update of Inductor Triton glue (clones 176.7 + casts/norms 154.8) at ~1.5 TB/s with heuristic kernel configs. The graph/fusion plan itself is good — what's suboptimal is the per-kernel config choice (block sizes, num_warps, vectorization).torch.compile(mode="max-autotune-no-cudagraphs")tunes exactly that without touching the plan: pointwise kernels and max-reductions (amax) produce identical values under any config; only reduction-order rounding (norms, CE) can differ — N1-class, same tier as concat which passed gate 3 at 0.41σ. Pure config lever, no code change:NANOCHAT_COMPILE_MODE.
3.1 hRound 7 — torch.compile max-autotune (NANOCHAT_COMPILE_MODE)
- Change:scripts/base_train.pypassesmode=os.environ["NANOCHAT_COMPILE_MODE"]totorch.compile(default empty = reference behavior). Screening config/tmp/switches-r7.json=NANOCHAT_CONCAT_MB=2+NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs: Inductor compile-time-autotunes its generated Triton kernels instead of using heuristic configs. Same fusion plan, same kernels; pointwise + amax configs cannot change values, reduction configs only change rounding order (N1-class, like concat). CUDA graphs stay off (they SIGSEGV, Round 4). - Compile time is much longer (autotune benchmarks each kernel, including Triton GEMM candidates for the bf16 GEMMs); the 34-update run pays it in warmup updates 0–9, which are untimed. - Result:/tmp/time-r7-ma— pending.
3.1 hRound 7 result — max-autotune is the big lever
-/tmp/time-r7-ma(concat=2 +NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs): speedup_vs_frozen_reference 1.0225 (concat-only was 1.0088), windows 1.0059/1.0207/1.0231 s/update → per-window ratios 1.0297/1.0198/1.0184, smallest 1.0184 — still just under the 1.02 bar (no_benefit), but a +1.4 % real gain over concat-only with zero memory cost (94.04 GiB) and zero code change. Update floor dropped from ~1.013–1.020 s to ~0.99–1.01 s; windows 2–3 still carry the progressive-slowdown spikes (max 1.0463). - Compile cost: startup_to_first_update 750 s (autotune benchmarks every kernel, including Triton GEMM candidates). Untimed (warmup updates 0–9) and inside the judge's per-run timeouts (timing 2700 s, record/trajectory 3600 s). The judge's deterministic mode takes the first listed config without timing, so gates 2/3 are unaffected by the mode. - Committed e914a9c. Switches.json stays{}(sub-bar scores 0, same as the unchanged reference) until a stack clears 1.02 on the smallest window AND passes the full check. - Next: stack levers that add margin on windows 2–3:NANOCHAT_CONCAT_MB=4(one fwd/bwd per update; halves chunk-boundary launch gaps; memory risk ~+25 GiB) and chunked cross-entropy (~10–15 ms + ~8.6 GiB, which also damps the allocator-pressure spikes).
3.7 hRound 8 — chunked cross-entropy: regression, off; spike analysis
-NANOCHAT_CHUNKED_CE=8192(newnanochat/chunked_ce.py+ hook ingpt.py): the wholelm_head → float → softcap → F.cross_entropychain as one autograd Function walking the vocab in blocks — block logits are bitwise slices of the reference GEMM (same_to_fp8scales, same K reduction), running logsumexp in fp32, recompute-in-backward, then the unchanged two FP8 backward GEMMs. Equivalence test: eager loss bitwise equal, grads within 1 bf16 ulp (median rel diff 0.0), compiled-vs-eager 9e-8. Handles ignore_index and vocab padding. - Result: REGRESSION —/tmp/time-r8-ce: 1.0012 (windows 1.0295/1.0437/1.0417, smallest ratio 0.9973), peak memory UP to 98.04 GiB (from 94.04). The +33 % lm_head GEMM time (recompute) plus the per-block fp32 pointwise chains outweigh the reference's fused CE pipeline. Switch off; committed e781fd3. AlsoNANOCHAT_CONCAT_MB=4OOMs (needs 8 GiB more than free) — dead.NANOCHAT_FP8_WCACHE=1on top of the stack: 1.0225, smallest ratio 1.0186 (vs 1.0184 without) — no gain, off. - Spike analysis (referenceruns/bootstrap/baselineseries vs mine): the reference's per-update floor is ~1.027–1.032 s with spikes to ~1.048 (+1.5–2 % over floor); my autotuned stack's floor is ~0.99–1.00 s but spikes reach 1.046–1.064 (+4–6 %). My spikes are ~3× taller than the reference's. Hypothesis: autotune shortened GPU work enough that the CPU is now the co-bottleneck, so the loader's ~19 ms/batch packing bursts (hidden before) surface as GPU idle — exactly what Round 4's prefetching loader removes (it measured no gain pre-autotune, when the CPU had slack). TestingNANOCHAT_LOADER_SLOTS=4on top of the stack.
3.7 hRound 8 conclusion — loader_slots marginal; spike root cause found
-NANOCHAT_LOADER_SLOTS=4on top of concat=2 + autotune:/tmp/time-r8-ls— 1.0237 (windows 1.0039/1.0203/1.0220, ratios 1.0318/1.0202/1.0195, smallest 1.0195). Window 1 improved (1.0202 → 1.0039 s/update) but windows 2–3 kept their spikes (u17–20: 1020/1028/1026/1039 ms, u27–28: 1038/1039, floor ~995–1000) — threaded prefetch moves the loader off the chunk boundary but its GIL-held work still stalls the training thread's kernel launches. - Root cause measurement (CPU-only, frozen assets,/tmp/loader_cost.py): parquet row groups are 1024 rows;read_row_group10.1 ms,to_pylist4.0 ms — cheap. The expensive part is rustbpe encode + the packing loop: ~6.5 encode calls of 128 docs per update (~60–80 ms/update of loader CPU), and at document-buffer refills the worker runs up to 8 back-to-back encodes with no packing in between — a near-continuous GIL-held burst exactly at the rg transitions (u17/u20/u27/u28/u32 where the spikes live, both trees). The training thread has ~0 slack (GPU ~1.0 s/update vs wall ~1.0 s), so any worker GIL time is a direct stall; the reference is frozen with its own +1.5–2 % spikes, so removing mine is pure ratio. - Conclusion: threads cannot fix this (shared GIL). Next: a forked child process that owns read + encode + packing (own GIL); the parent only receives packed rows over a pipe.
4.1 hRound 9 — process-based loader (NANOCHAT_LOADER_PROC, default off)
- Change:nanochat/dataloader.pyadds_proc_loader_child(runs in a forked child:_document_batches→ rustbpe encode → the verbatim best-fit packing loop → sends each packed batch as flat int64[inputs | targets]bytes + state indices over amultiprocessingpipe, then blocks for an ack = at most one batch in flight) and_proc_bos_bestfit(parent side: same slot-pool consumer machinery as the threaded variant, but the worker thread only doesconn.recv()— GIL released while blocked —np.copytointo the slot's pinned staging buffer, the H2D enqueue and the ack; ~1–2 ms of GIL per batch instead of ~15–20). Dispatch:NANOCHAT_LOADER_PROC=1insidetokenizing_distributed_data_loader_with_state_bos_bestfit(same yields, same order, same state dicts — the hook point is unchanged). Child safety: forked before any encode in the parent, never touches CUDA, daemon=True, EOF on the pipe when the parent dies, explicit terminate in the generator'sfinally. - Equivalence (GPU test,/tmp/proc_loader_test*.py): batches 0–2 bitwise identical to the synchronous loader through both_proc_bos_bestfitand the public dispatch (inputs, targets and state dicts equal); no stray child processes afterclose(). - Numerics tier: N0 for the loader itself (bitwise-identical yields; gate-2 token-row digest must match exactly). The stack it rides on (concat=2 + autotune) is N1 as before. - Result:/tmp/time-r9-proc(concat=2 +NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs+NANOCHAT_LOADER_PROC=1,/tmp/switches-r9.json) — pending.
4.1 hRound 9 result — process loader clears the gate-1 bar
-/tmp/time-r9-proc(concat=2 +NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs+NANOCHAT_LOADER_PROC=1): speedup_vs_frozen_reference 1.0266, windows 1.0035/1.0173/1.0172 s/update → per-window ratios 1.0322/1.0232/1.0243, smallest 1.0232 — above the 1.02 bar for the first time. Peak 94.04 GiB (unchanged). Windows 2–3 improved ~3 ms/update vs the threaded loader (1.0203/1.0220 → 1.0173/1.0172). - Residual spikes persist at the same update indices (u20 1.0401, u26–28 up to 1.0343, floor ~0.994–1.005) even with the producer off-process — the GIL was the main but not the only component. Candidates: allocator stalls (expandable_segmentsuntested on THIS stack), CPU descheduling, child CPU bursts. The reference spikes at the same indices (frozen into its windows), so any further spike removal is pure ratio (~+1 % on windows 2–3). - switches.json set to this stack and committed (2034051). Launching the full check →/tmp/check-r9-full: gate 1 re-times the tree, gate 2 record (expect NOT_BITWISE from concat+autotune, exact items must pass), gate 3 trajectory (expectequivalentlike concat-only's 0.41σ).
4.1 hRound 9 full check — gate 1 PASS (in-check re-measurement)
-/tmp/check-r9-fullgate 1 (timing of the submitted stack, in-check): windows 1.0054/1.0186/1.0189 s/update → ratios vs frozen reference 1.0303/1.0219/1.0226 → smallest 1.0219 > 1.02 ✓ (second independent measurement; thetimerun gave 1.0232 — consistent, both above the bar). Same spike pattern (u18–21 and u26–29 elevated, u28 peak 1039.7 ms). - Power-cap theory tested and refuted as the spike driver: nvidia-smi poll (250 ms) joined against the judge's per-update timestamps shows power pinned at ~684–696 W (the board cap) on EVERY update — floor and spike alike — and mean SM clocks 1525–1625 MHz with NO systematic dip in the spike updates (u28 rank 0 even shows the highest mean clock, 1625 MHz, at its worst 1040 ms). The big spikes (u20/u28) run at 96–100 % util → the GPU is doing MORE work or waiting on something other than clocks; some mild spikes (u17/u19/u21 on rank 1) show util dips to 85–93 % → host-side launch stalls there. Mixed causes; aprofileover u18–21 (vs floor u14–17) is the next diagnostic. - Record phase (gate 2) started 20:13:25; trajectory (gate 3) after — result pending in/tmp/check-r9-full/check.json.
4.1 hRound 9 full check result — **PASS, speedup 1.0219, tier N1** (20:53)
-/tmp/check-r9-full/check.json: status PASS, speedup 1.0219, tier N1 — the first full-check PASS of the trial. - Gate 1: candidate windows 1.0054/1.0186/1.0189 s/update → ratios 1.0302/1.0219/1.0226, smallest 1.0219 > bar 1.02 ✓. 34 steps, peak 94.04 GiB, startup→first update 743 s, other_optimizer_steps 0. - Gate 2: exact items all OK (10 updates, 175 parameters, 640 rows/rank, loader/model provenance, hyper-parameters, dtypes) → NOT_BITWISE (first differing update 0, 53→175 parameters — expected from concat regroup + Inductor autotune), so gate 3 decides. Record wall 1121.7 s, deterministic flags all on, FA3 deterministic patched. - Gate 3 (trajectory seed 1, 150 updates): verdictequivalent— every endpointwithin: |d| in σ units = 0.026 / 0.047 / 0.368 / 0.049 / 0.042 / −0.024 (windows 5–15 … 115–150); the 3σ bounds are 0.068/0.039/0.024/0.041/0.074/0.099 loss units. Worst endpoint 30–50 at 0.37σ — comfortably equivalent, same magnitude as the concat-only result (0.41σ). - Conclusion: the submitted stack (concat=2 + max-autotune-no-cudagraphs + process loader) is validated end-to-end. Margin above the 1.02 bar is thin (1.9 %); the next rounds target the fixed-index spikes (u17–20, u26–28, floor elsewhere) which cost windows 2–3 ~13–18 ms/update.
5.9 hRound 10 — profile the spikes: root cause found and fixed (loader H2D copy blocks the training stream)
-profile --from 14 --iters 8→/tmp/prof-r10(updates 14–21, rank 0, tracetiming_profile.trace.json, 89 MB chrome trace). - The spikes are GPU idle, not slower kernels. Per update: GPU busy is flat 969–980 ms while wall goes 1005→1164 ms; idle grows 31.6 → 30.9 → 49.0 → 63.9 → 60.1 → 74.5 → 193.5 ms. Kernel-time diffs between the 3 slowest and 3 fastest updates are noise (±2.5 ms). u20 alone has a 131 ms gap. - Inside every big gap the host issued zero kernel launches and no aten ops — it was blocked, not working. The one 131 ms gap sits insideOptimizer.stepand ends exactly when step returns. - **The blocker: 16 ×cudaStreamSynchronizeof 140–187 ms on the training stream** — exactly 2 per update, insideaten::copy_ops that each end with a 512 KBMemcpy HtoD (Pageable -> Device)followed by the consumer's clones. That is the proc loader's produce thread staginggpu_buffers[slot].copy_(cpu_buffers[slot]): when the CUDA runtime does not treat the source as pinned it synchronizes the copy's stream before returning — and on the training stream that wait is the entire enqueued backlog (~150 ms of forward/backward). The producer therefore paced the consumer 1:1 (2 of its 4 copies/update block during the backwards, 2 return instantly at the drained boundaries — matching the 16 blocking copies), the 4-slot queue ran dry at every micro-batch boundary, and the GPU idled ~9 ms (floor) to 131 ms (spike updates) waiting for the next batch. - Isolated reproductions (no GPU contention for the lock, tiny scripts): side-threadcopy_(pinned, non_blocking=True)with a busy stream: 0.15 ms; same with a pageable source: 318.8 ms.Connection.recvreleases the GIL (refuted as the cause); upstream base_train already disables GC after step 0 (refuted); power/clocks were refuted in Round 9. - Fix (commit da86ca6, tier N0): staging runs on a dedicatedtorch.cuda.Stream()(empty, so the runtime's synchronize is a no-op) withstaged_events[slot]recorded on it; the consumer's default stream waits that event before cloning. Yields bitwise identical to the sync path (batches 0–3 + state dicts); fetch latency under an 80 ms GPU backlog 4–23 ms → 0.06–0.32 ms; nocudaStreamSynchronizein a profiled fetch loop. Same mechanics available for the (off-by-default) NANOCHAT_LOADER_SLOTS path if ever needed. - Expectation: windows 2–3 lose their ~13–18 ms/update of boundary idle; if the floor idle (~20 ms/update) also shrinks, all three windows approach the 0.98–1.00 s/update floor → smallest ratio ~1.03+.timerun in flight (/tmp/time-r10-cs); result below. - **Result (/tmp/time-r10-cs, judgetime, tree da86ca6): windows 1.0048 / 1.0180 / 1.0167 s/update → smallest window ratio 1.0225** (speedup_vs_frozen_reference1.026 over the window means, tokens/s 258734). Against Round 9's proc-loader run (1.0232 / 1.0219) and the full check's 1.0302 / 1.0219 / 1.0226 this is **within noise: the copy-stream fix did not remove the spikes**. Per-update durations fromtiming_tree.r{0,1}.jsonstill show the elevated blocks u16–21 (peak 1.0345) and u27–29 (up to 1.034) over a 0.996–1.007 floor, on both ranks. So the 140–187 ms blocking copies the profiler saw were real but were an artifact of *where* the producer's copy landed under CUPTI/profiler visibility, not the cause of the clean-run spikes — the pageable-vs-pinned treatment of the staging source is context-dependent (the same allocation is pinned in standalone runs). - Kept anyway: the fix is correct, N0 (bitwise-identical yields verified), and strictly more robust — a pageable fallback can no longer pace the training stream. Tree at da86ca6; submission switch file unchanged (NANOCHAT_CONCAT_MB=2,NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs,NANOCHAT_LOADER_PROC=1). - CPU cgroup throttling refuted (/sys/fs/cgroup/cpu/cpu.stat: nr_throttled 6, ~5.8 ms throttled total; 48 CPUs allowed); rank-identical step ends are forced by the step-end all-gather, so they do not by themselves prove a shared cause. bench.py does not poll the GPUs during a run (wait_idleruns only before). - Open: the spikes are GPU idle (busy time flat 969–980 ms across updates), appear at the same update indices in every stack including the reference (data-positioned), and are worth ~13–18 ms/update on windows 2–3 (smallest ratio 1.0225 → ~1.036 if they were removed). Next: a CPU-only host poller (per-process CPU/faults/schedstat wait/context switches, vmstat compaction, PSI) running alongside a judgetimerun, joined againststep_end_s(the entry stampstime.time()), to separate loader-child CPU bursts, rank descheduling, page-fault/compaction storms and host pressure.
6.5 hRound 11 — the spike mechanism found: `device == "cuda"` is never true in DDP
- Diagnosis first (host poller alongside a judgetimerun): a CPU-only poller (per-process CPU/faults/schedstat/context switches + vmstat + PSI, 5 Hz) ran next to/tmp/time-r11-poll(speedup 1.0249, windows ~1.00/1.018/1.017 — the same spike pattern). Join againststep_end_s: **rank CPU is flat ~1080 ms/update on spike and floor updates alike, schedstat runqueue wait ≈ 0, no majflt/compaction/PSI signal, no correlation of any host counter with the spikes**. So the spikes are a *wait*, not work — which pointed back at the Round 10 blocking copies. - The bug:nanochat/common.py compute_init()returnsdevice = torch.device("cuda", ddp_local_rank)— a torch.device, never the string"cuda". Everydevice == "cuda"indataloader.pyis therefore **False in every judge run**: theNANOCHAT_LOADER_PROC/NANOCHAT_LOADER_SLOTSdispatches never fired (Round 9's "proc gain" was inline-vs-inline noise; Round 10's side-stream fix never ran), and the inline loader computeduse_cuda = device == "cuda"→ False → pageable staging buffer +gpu_buffer.copy_(cpu_buffer, non_blocking=False)— exactly the 2-per-update synchronous 512 KB H2D copies on the training stream that the Round 10 profile caught (cudaStreamSynchronize behind the enqueued backlog, 140–187 ms). Upstream/reference share the same latency; the spikes are its queue-phase jitter. All prior observations are consistent with this one bug. - Fix (commit 19ba8b8, tier N0): new_is_cuda_device()(True for"cuda","cuda:1",torch.device("cuda", 0)); both dispatches anduse_cudause it. The inline path now stages through **two alternating pinned buffers, each guarded by a CUDA event** (a pinned source is read asynchronously, so a buffer must not be overwritten before its previous copy completes) and issues the H2Dnon_blocking— no stream synchronization anywhere. WithNANOCHAT_LOADER_PROC=1the proc loader now actually engages: off-process encode, pinned slots, side-stream staging (Round 10 machinery, previously dead code under the judge). - Verified standalone (/tmp/test_devfix.py,/tmp/test_fork.py): dispatch engages withdevice=torch.device("cuda", 0); proc and inline paths yield 3 batches bitwise identical with equal state dicts; inline staging buffers pinned (2 × 65536 int64); proc path forks a child and delivers(16, 2048)oncuda:0. - Expectation: the two blocking pageable copies per update vanish from the training stream; spike updates (u16–21, u27–29, +15–47 ms) should collapse toward the ~0.99 s floor, and with encode off-process the floor idle should shrink too — windows ~1.00 or below → smallest ratio ~1.03+.timerun in flight (/tmp/time-r11-fix); result below. - **Result (/tmp/time-r11-fix, judgetime, tree 19ba8b8): windows 0.9893 / 0.9944 / 0.9942 s/update → per-window ratios 1.0470 / 1.0468 / 1.0480, smallest 1.0468** (speedup_vs_frozen_reference1.0472, tokens/s 264093, peak memory unchanged 94.04 GiB). The spikes are gone: per-update durations are smooth 0.975–1.000 s with no 1.02–1.05 outliers (previous runs: floor 0.996–1.007 with +15–47 ms spike blocks), and the floor itself dropped ~10–20 ms/update — the off-process encode removed the boundary idle as well. Biggest single gain of the trial: smallest window ratio 1.0225 → 1.0468. - Fullcheckon 19ba8b8 launched next (the record gate exercises the forked loader under the judge for the first time); result to follow.
6.5 hRound 11 full check — PASS, speedup 1.0473, tier N1 (23:16)
-/tmp/check-r11-full/check.jsonon tree 19ba8b8 with the submitted switch file: status PASS,performance.speedup1.0473, tier N1. - Gate 1 (timing, one run vs frozen reference): windows 0.989 / 0.9934 / 0.9935 s/update → ratios 1.0473 / 1.0478 / 1.0487, smallest 1.0473 (bar 1.02, reference spread 0.0017). Per-update durations smooth, no spike blocks. - Gate 2 (record, deterministic, 10 updates): exact items all OK — 175 parameters, rows_per_rank 640, one loader, one optimizer, no foreign state dtypes; NOT_BITWISE (first differing update 0, 53 parameters: lm_head.weight, resid_lambdas, …) — the same concat+autotune kernel-choice signature as the r9 check, not loader-related. The forked proc loader ran under the judge's record mode for the first time without any issue. - Gate 3 (trajectory, seed 1, 150 updates): verdict equivalent — endpoints 0.050σ / 0.063σ / 0.445σ / 0.107σ / 0.059σ / −0.011σ, worst 0.445σ of the 3σ bound (r9: 0.37σ — same computation, as expected from a value-identical loader). - The submitted tree (19ba8b8) and switch file are exactly what passed. Cost 1.79 GPU·h.
8.0 hRound 12 — profile the new floor; screen TORCHINDUCTOR_COORDINATE_DESCENT_TUNING
- Profile (/tmp/prof-r12, judgeprofile --from 14 --iters 4, tree 19ba8b8): the run is now GPU-bound — busy 981–987 ms vs idle ~13 ms per update (profiler run; clean-run walls 0.975–1.000 s). The host runs far ahead ("Command Buffer Full" is the top CPU item, 2.69 s over 4 updates) and spends 215–223 ms insideOptimizer.stepwaiting for the queue — harmless, the GPU is the bottleneck. - Phase split per update (GPU time): backward 2×323 ms = 646 ms (the wall), forward 2×147 ms = 294 ms,Optimizer.step48 ms (NCCL ~24.8: reduce_scatter ~12 + all_gather ~13 + adamw reduce ~7.4 partially overlapped; compiled Muon/AdamW compute ~35 ms, pipelined per shape group). - Biggest fused-Triton items inside fwd/bwd: fp8 amax+cast reductions ~19 ms/update, fused rms_norm backward ~23 ms/update, log-softmax fwd+bwd ~17 ms/update, FA3 bwd ~87 ms/update. - Candidate this round:TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1(pure switch, not in reserved_env) — Inductor config/layout refinement for exactly those fused Triton kernels; numerics class unchanged (kernel choice, like COMPILE_MODE). Screening on a scratch config/tmp/sw-cd.json;timein flight (/tmp/time-r12-cd); the submission stays at the certified 19ba8b8 + switches until this pays.
8.0 hRound 12 result — TORCHINDUCTOR_COORDINATE_DESCENT_TUNING: no gain
-/tmp/time-r12-cd(scratch config/tmp/sw-cd.json, tree 19ba8b8): windows 0.9892/0.9932/0.9933 → smallest ratio 1.0471, speedup 1.048 — vs the certified tree's own check-run windows 0.989/0.9934/0.9935 (smallest 1.0473). **Within noise, no benefit** (compile +28 s). Switch not adopted; submission unchanged.
8.5 hRound 13 — overlap the gradient all_reduces with the final backward
- Target from the Round 12 profile:Optimizer.step= 48 ms/update of serial GPU time after the backward (reduce_scatter ~12 + all_gather ~13 + fused Muon/AdamW compute ~35, partially overlapped across groups). The reduces cannot overlap the *next* forward (the judge's step hook synchronizes), but they can overlap the *remaining backward* if launched per-parameter as each grad finishes. - Verified prerequisites standalone: - per-tensorall_reduce(AVG)== stackedreduce_scatter_tensor(AVG)rank chunk, bitwise, for 2 ranks (/tmp/test_comm_equiv.py); -register_post_accumulate_grad_hookfires progressively during the compiled backward, no graph break (/tmp/test_hooks.py). - **Change (commit 8941906, N0 vs the current tree, switch NANOCHAT_OVERLAP_REDUCE default off)**: hooks launchall_reduce(grad, AVG, async)on a dedicated stream during the update's final backward (event-ordered after each param's accumulation);step()waits the futures, orders the compute stream after the comm stream, and consumes the averaged grads exactly like the serial path (muon: same stack build withreduce_scatterreplaced by a rank slice; adamw: row views). base_train marks the final accumulation viaoptimizer.notify_backward()(no-op when off). - A first cut was wrong and caught by the standalone test: copying the reduced grads into stack slots from inside the hook raced the collective (an async NCCL work's kernel enqueue can trail a later host enqueue on the same stream — stale slots on rank 1, params off by ~4e-3). Replaced by the step()-side stack build. - Equivalence: 2-rank, 3-update test, eager and torch.compile: **all parameters bitwise identical** with the switch off vs on. -timein flight (/tmp/time-r13-ovl, scratch config/tmp/sw-ovl.json); result below. - Result (/tmp/time-r13-ovl): REFUTED — the overlap is slower. Windows 0.9986/1.005/1.0066 → smallest ratio 1.0351 (certified tree: 0.989/0.9934/0.9935 → 1.0473), i.e. −10 to −13 ms/update. 145 per-parameter all_reduces (each with NCCL launch/protocol overhead) plus ~2× the reduce_scatter's bytes cost more than the ~19 ms of serial comm they hide; the stacked reduce_scatter's batching is worth more than overlapping it. The switch stays off; commit 8941906 reverted so the submitted tree is exactly the certified 19ba8b8 (the inert overlap code is dead weight for a refuted lever). - Lesson logged: on 2 ranks with ~150 params, per-tensor collectives lose to stacked batched ones; overlap only pays if the hidden op's own cost does not grow.
9.1 hRound 14 — stack the value_embeds' AdamW collectives (NANOCHAT_ADAMW_STACK)
- Profile finding (from the r12 trace,/tmp/nccl_calls.py): 41 NCCL kernels per update insideOptimizer.step, allRING_LL, ~24.7–26.6 ms GPU time. Inventory (matched against the reference record's 175 param shapes): 3 Muon reduce_scatters of 906 MB fp32 (2.27/2.44/2.42 ms) + 3 all_gathers (2.46–2.75 ms); lm_head 201 MB fp32 (0.86 + 0.6 ms); **13 bf16 reduce_scatters + 13 all_gathers of 100 MB each (0.28–0.49 ms apiece, ~8.5 ms total) = wte + the 12 value_embeds** — every one of the 12value_embedsis its ownnn.Embedding(32768, 1536)bf16 parameter in one AdamW group, so each does its own reduce_scatter + all_gather every update. - NCCL protocol lever killed first (3 min): microbenchmark (/tmp/nccl_proto.py, 2 ranks, the exact sizes) shows NCCL_PROTO auto ≈ Simple ≈ LL128 (906 MB rs 1.68/1.69/1.76 ms; the profiled in-training 2.27 ms is contention with the concurrent Muon compute, not protocol); forcing LL is 2.8× worse. No switch value there. (The _LL kernel names in the trace are misleading; auto does not pick LL for these sizes.) - Change: new stacked path inMuonAdamW._reduce_adamw/_compute_adamw(NANOCHAT_ADAMW_STACK, default off): when an AdamW group has >1 same-shape params of ≥1024 elements (only the 12 value_embeds match), write their grads into one padded(K, *shape)buffer with_foreach_copy_, do ONEreduce_scatter(AVG)and ONEall_gather, and copy back with_foreach_copy_. Rank r then owns whole parameters[r*chunk, (r+1)*chunk)instead of a row slice of each — communication layout only: the collective averages the same element pairs and the AdamW update is elementwise with the same per-parameter state keys/dtypes and the same group hyper-parameters, so every parameter is stepped to bitwise-identical values. Saves 24 NCCL launches and ~8.5 ms → ~5 ms of collective time per update (1.3 GB moves at the big-collective rate instead of 24 latency-exposed 100 MB collectives). - Record-gate safety verified against the entry's own capture code (bench/nanochat_entry.py):hparamscomes fromparam_groupsonly (untouched);state_dtypescollects the (key, dtype) SET across states (identical: exp_avg/ exp_avg_sq bf16 + fp32 entries unchanged); digests are the parameter SHAs (bitwise). - Standalone bitwise verification (/tmp/test_adamw_stack.py, 2 ranks, 3 updates, grad accumulation, changing lr): all parameters bitwise identical switch off vs on — for 12 params per group (recipe shape), and the padding paths 13 and 11; state dtype sets equal; cross-rank parameters identical after every update. -timein flight (/tmp/time-r14-astk, scratch config/tmp/sw-astk.json= the certified three switches + NANOCHAT_ADAMW_STACK=1); result below. - Result (/tmp/time-r14-astk): NO GAIN. Windows 0.9903/0.9953/0.9938 → smallest ratio 1.0458 (certified tree: 0.989/0.9934/0.9935 → 1.0473; its owntimerun 0.9893/0.9944/0.9942 → 1.0468); speedup_vs_frozen_reference 1.0467; peak memory 94.04 → 94.61 GiB. Within noise, slightly negative. Change reverted (tree back to the certified content); switch not adopted. - Lesson (important for the remaining rounds): in the 3-phase pipelined optimizer step, summing NCCL kernel durations overcounts — the 26 per-parameter ve/wte collectives (~8.5 ms of kernel time in the profile) were almost entirely OVERLAPPED with the Muon compute, so removing them moves the wall by ~0, while the stacked path ADDS ~3 GB of staging copies (~1 ms) that cancel the small exposed remainder. Only the critical path pays: the fused Muon compute (~35 ms, of which the Newton–Schulz bf16 GEMMs are ~24 ms at ~480 TFLOPS effective), the first reduce_scatter and the last all_gather tails, and the end-of-step copy-back. Next rounds attack those.
9.3 hRound 15 — compile the fused optimizer steps with max-autotune (NANOCHAT_OPT_COMPILE_MODE)
- Target from the Round-14 lesson: the step's critical path is the fused Muon compute (~35 ms), dominated by the Newton–Schulz batched bf16 GEMMs (~24 ms, 45 nvjet calls at ~480 TFLOPS effective — Inductor's default mode takes extern-cuBLAS without benchmarking). The two fused kernels are compiled with plain@torch.compile(dynamic=False, fullgraph=True). - Change:NANOCHAT_OPT_COMPILE_MODE(default unset) threaded into both fused-step decorators asmode=. Screeningmax-autotune-no-cudagraphs, the same mode the model already uses (a certified N1 change there): Inductor benchmarks Triton GEMM templates vs cuBLAS per NS matmul shape and autotunes the pointwise/reduction kernels. Kernel choice only — same graph, same math; numerics class N1 (verified in a smoke test: both modes compile and run; values differ at kernel-rounding level, muon |p| 722.1031 vs 722.1027). - Under the judge's deterministic record, autotune is replaced by first-config selection — the same situation as the certified model-side max-autotune. -timein flight (/tmp/time-r15-optmode, scratch config/tmp/sw-optmode.json= the certified three switches + NANOCHAT_OPT_COMPILE_MODE=max-autotune-no-cudagraphs); result below.
9.7 hRound 16 — one-pass Muon staging (NANOCHAT_MUON_ONEPASS), screened on top of Round 15
- Target: the Muon path's eager buffer shuffling on the step's critical path —torch.stack(grads)+copy_into the padded buffer before each reduce_scatter (two passes over ~906 MB per group), andtorch.stack(owned)+copy_into the all_gather input before each gather (two passes over ~450 MB per group). ~5.4 GB of removable copy traffic per update (the stacks sit serially before their collectives, so unlike Round 14's collectives this traffic is exposed). - Change:NANOCHAT_MUON_ONEPASS(default off) —_foreach_copy_the grads straight into the padded stacked buffer (one pass), and pre-fill the all_gather input with the owned parameters' current values so the fused kernel updates it in place (one pass, no separatestacked_owned). Buffer placement only → bitwise identical (tier N0 vs the current tree). - Verified standalone (/tmp/test_muonepass.py, 2 ranks, 3 updates, grad accumulation, changing lr/momentum): all parameters bitwise identical for 6-param groups (3+3 chunking) and 5-param groups (3+2 + padding), cross-rank identical; single-rank path also bitwise identical (/tmp/test_muonepass_ws1.py). (A first cut had a single-rank+switch-off bug —stacked_params=updated_paramswould have been uninitialized — caught while reviewing the diff before testing and fixed.) -timein flight (/tmp/time-r16-onepass, scratch config/tmp/sw-r16.json= the certified three + NANOCHAT_OPT_COMPILE_MODE + NANOCHAT_MUON_ONEPASS), compared against Round 15 alone (windows 0.9887/0.9926/0.9921 → smallest 1.0476); result below. - Result (/tmp/time-r16-onepass): NO GAIN — reverted. Combined stack windows 0.9884/0.9943/0.9950 → smallest ratio 1.0469, vs Round 15 alone 0.9887/0.9926/0.9921 → 1.0476: windows 2–3 got ~1.7/2.9 ms WORSE, window 1 ~0.3 ms better. The_foreach_copy_staging (multi_tensor_apply per tensor) is slower than the single batched CatArray/copy_ kernels it replaced for these 96×9.4 MB groups, eating the saved bandwidth. The R16 edit is reverted from the tree; switch not adopted (bitwise N0 but performance-negative). - Decision: adopt Round 15 alone. Its gain is small (smallest ratio 1.0473 → 1.0476; windows −0.3/−0.8/−1.4 ms vs the certified check-run, consistent across all three windows) but real, and it is already implemented, committed and time-verified. Because it is an N1 change (optimizer kernel choice → new trajectory), adoption requires gate-3 evidence: running the fullcheckwith the 4-switch SCRATCH config (/tmp/sw-optmode.json) while/submission/switches.jsonstays at the certified 3-switch file — the submission only moves to the 4-switch file if that check PASSes. On failure the certified stack remains submitted, unchanged. - Full check launched →/tmp/check-r15-full(tree 200f1e8, config/tmp/sw-optmode.json); result below.