GLM-5.3 1.05×

Harness: Claude Code · switches: NANOCHAT_CONCAT_MB=2 NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs NANOCHAT_LOADER_PROC=1

Judged
1.05×
Tier
N1
Self-best
1.05×
Rounds
16
Hours
10.0
API cost
$58.2
Input tokens · 99.6% cached
202 M
Output tokens
1.0 M
Requests
1,430
Micro-batch regrouping 4×16 → 2×32
✓ NANOCHAT_CONCAT_MB=2, alone only 1.009
Scope of hand-written kernels replacing Inductor glue
✗ tiled transpose op 0.978, tiled cross-entropy 1.001, both reverted
Per-shape cuBLASLt algorithm selection
✗
FP8 weights quantized once per step
tried, reverted (1.000, no gain)
Fast accum on backward GEMMs
tried, reverted (failed statistical gate, −11.95σ)
Other main levers
max-autotune-no-cudagraphs (1.009 → 1.023), data fetch moved to a subprocess with device check fixed (min window 1.023 → 1.047)
Raw work log25 sections
1.1 hRound 1 — FP8 weight-quantization cache (NANOCHAT_FP8_WCACHE)
- Start/end: 2026-11-06 ~15:10 – 16:45 UTC.
- Profile first (judge profile, /tmp/prof-ref-1, reference tree): per update, the FP8
  "glue" around the 6 converted linears is eager ops inside the opaque _Float8Matmul, and the
  weight quantization is recomputed on every one of the 4 micro-batches in *both* directions:
  forward quantize chains (amax→scale→mul→clamp→cast) plus backward recomputes writing row- and
  col-major, ≈16–20 ms/update across the 24 layers, vs. weights only changing in optimizer.step().
  Also noted: grad_weight GEMMs (K=32768) run at 144 CTAs vs 132 SMs (wave quantization) — the
  target of Round 2.
- Change: nanochat/fp8.py — persistent per-Float8Linear buffers _w_fp8 (row-major e4m3),
  _w_fp8_t (col-major e4m3), _w_qinv (f32), filled by refresh_fp8_weight_caches(model) once
  per update from the training loop (scripts/base_train.py, top of the step, before the
  micro-batches; the weights are final there). The compiled path (_Float8MatmulCached) reads the
  buffers instead of re-quantizing per call; eager calls (judge's eval, sampling) keep the
  reference path. The refresh reproduces Inductor's quantize arithmetic bitwise (f64 reciprocal →
  f32 scale, clamp, cast; verified against the generated kernel source). Switch
  NANOCHAT_FP8_WCACHE, default off.
- Bugs found and fixed while validating: the first version refreshed from a post-step hook in
  MuonAdamW.step() that was never registered → stale caches from update 1 (first quick check
  /tmp/check-r1-quick: loss frozen at 10.39, all 175 params differ from update 1). Fixed by
  moving the refresh into the training loop top (nanochat/optim.py reverted to upstream).
- Result (check --quick, /tmp/check-r1-quick2): status NOT_BITWISE, exact_items_ok
  true, rows/updates/hyper-parameters all identical. 4 params differ at update 0 (c_proj weights
  h.2/13/17/…), all 175 from update 1 (the 4 ULP-level perturbations compound). Per-update losses
  match the reference record run to ≤2e-4 over 10 updates (10.397737 identical at step 0;
  9.847431 vs 9.847258 at step 8) — healthy trajectory, far below seed noise σ≈0.023.
- Why not bitwise (investigated to root cause): the reference's AOTAutograd backward does not
  save the fp8 input tensors for attn.c_proj/mlp.c_proj; it recomputes their input
  quantization (amax over the FA3 output / relu output) inside backward kernels whose reduction
  order differs from the forward's amax → ~1 ULP scale differences → ~16 differing fp8 bytes per
  tensor → grad_weight differs by ~1e-9 (measured: /tmp/grad_cmp_result.log, 48 c_proj weights,
  0.6–1.5 % of elements, maxabs ≲1.5e-8; qkv/fc/lm_head grads bitwise identical). My version saves
  the forward's quantization (cheaper) and cannot reproduce Inductor's fused recompute reduction
  order. Inherent N1 — same class as the judge's accepted reduction-order control tree.
- Tier: N1 (exact items pass; statistical gate 3 to be confirmed by the full check).
- time: run in progress → /tmp/time-r1 (result appended below).
- time result: /tmp/time-r1 — speedup_vs_frozen_reference 1.0004 (s/update 1.0391; windows
  1.0338/1.0419/1.0417 vs frozen ref 1.0358/1.0409/1.0419 → per-window ratios 1.0019/0.999/1.0002, smallest 0.999).
  No gain — far below the 1.02 bar. The weight-quantize chains the cache removes were worth only ~1–2 ms/update
  in practice (Inductor had fused much of that work into kernels that read the weights anyway; the remaining saved
  time is offset by the refresh + extra saved-tensor traffic of the cached path). Switch kept OFF in
  /submission/switches.json (an N1 change with no measured benefit is not worth the divergence); the code stays in
  the tree behind NANOCHAT_FP8_WCACHE (default off) for possible later use.
1.3 hRound 2 — micro-batch concatenation (NANOCHAT_CONCAT_MB)
- Start: 2026-11-06 ~16:50 UTC.
- Hotspot (from the Round-1 profile of the reference): every fp8 GEMM runs 4× per update at
  batch 16 (M=32768); grad_weight GEMMs (K=32768) launch 145×2×4 = 1160 backward GEMMs/update with
  wave-quantization tails, plus ~1900 glue-kernel launches. Grouping the same rows into 2 chunks of
  (32, 2048) halves the GEMM count and doubles each GEMM's M (better tile occupancy, fewer tails).
- Change: scripts/base_train.py — optional NANOCHAT_CONCAT_MB (default 1 = off): process
  CONCAT_MB loader batches per forward/backward through a persistent staging buffer (the loader
  reuses its own GPU buffer on every yield, so each prefetched batch is copied into its slice of
  the staging buffer after the chunk's backward — single stream, so ordering is safe). Rows and
  their order are unchanged (gate 2 allows different micro-batch grouping); the loss is scaled by
  concat_mb / grad_accum_steps = 1/2 (exact power of two) so every row contributes exactly the
  same gradient as the reference. Numerics: the grad_weight GEMM now reduces over K=65536 in one
  accumulation instead of four of 32768 → different rounding → N1.
- check --quick (/tmp/check-r2-quick, config NANOCHAT_CONCAT_MB=2): status NOT_BITWISE,
  exact_items_ok true (rows_per_rank 640 identical; rows_available 672 — one extra prefetched
  batch, consumed rows unchanged), 52 params differ at update 0 (the 48 c_proj/mlp_proj weight grads
  whose K-reduction regrouped + 4 more), 175 from update 1. Peak memory 94.04 GiB (fits).
  Train losses vs reference: drift 2e-4 at step 0 growing to 5.6e-3 at step 8 — ~0.25 of the
  seed-σ at endpoint 1 (3σ bars: 0.068/0.039/0.024/0.041/0.074/0.099; seed σ grows with training,
  so the amplification applies equally to the much smaller perturbation here).
- time: run in progress → /tmp/time-r2.
- time result: /tmp/time-r2 — speedup_vs_frozen_reference 1.0088 (s/update 1.0304;
  windows 1.0215/1.0335/1.0361 → ratios 1.0140/1.0072/1.0056, smallest 1.0056). Below the 1.02 bar
  on its own, but the fastest updates drop to 1.013 s (reference floor ≈ 1.027) — the real kernel
  gain is ~1.5–2 % and occasional spikes (up to 1.063 s, vs the reference's own max 1.048) eat it.
  Peak memory 94.04 GiB (reference 55.94). The spikes are the next thing to attack (allocator
  stalls at 94 GiB are the main suspect: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is the
  cheap test).
- Decision: code committed (switch default off); adopt NANOCHAT_CONCAT_MB=2 only as part of a
  stack that clears the bar (timing of concat + fast-accum backward in progress).
1.4 hRound 3 — fast accumulation on the backward FP8 GEMMs (NANOCHAT_FP8_FAST_BWD)
- Start: 2026-11-06 ~16:55 UTC.
- Hotspot (from the Round-1 reference profile): of ~454 ms/update of FP8 GEMM time, the
  backward half (grad_input + grad_weight, K=32768, 1160 GEMMs/update) runs with
  use_fast_accum=False while the forward already uses True (upstream's own choice in
  nanochat/fp8.py). Fast accumulation skips the periodic fp32 promotion of the MMA accumulator
  inside the cuBLASLt kernel.
- Change: nanochat/fp8.py — both backward _scaled_mm calls (and the cached variant) take
  use_fast_accum=FP8_FAST_BWD from NANOCHAT_FP8_FAST_BWD (default off). Same FP8 dtypes
  (e4m3/e5m2), same tensorwise scaling, same shapes — only the GEMM's internal accumulation
  changes: an arithmetic change the rules allow, judged by gates 2/3. Tier N1 when enabled.
- time (/tmp/time-r3, config concat=2 + fast_bwd=1): speedup_vs_frozen_reference
  1.0334 (s/update 1.0059; windows 0.9945/1.0107/1.0127 → ratios 1.0415/1.0299/1.0288,
  smallest 1.0288 — above the 1.02 bar). Marginal contribution of fast_bwd over
  concat-only: ≈ +2.4 %. Steady-state floor now 0.987 s (reference floor 1.027). Peak memory
  unchanged at 94.04 GiB.
- Decision: submission switches set to `{"NANOCHAT_CONCAT_MB": "2",
  "NANOCHAT_FP8_FAST_BWD": "1"}; full check launched → /tmp/check-r3-full` (gate 1 timing,
  gate 2 record — expect NOT_BITWISE/N1 — and gate 3's 150-update trajectory, the decisive gate
  for the fast_bwd accumulation change).
- Full check (/tmp/check-r3-full, config concat=2 + fast_bwd=1): gate 1 timing 1.035
  (window ratios 1.0417/1.0359/1.0350), gate 2 exact items pass, NOT bitwise as expected — but
  gate 3 REJECTED (verdict different): endpoint biases −0.0745 (−3.29σ, bound 0.068),
  −0.0173 (−1.33σ), −0.0233 (−2.90σ), −0.1631 (−11.95σ), −0.1924 (−7.83σ), −0.1539 (−4.65σ).
  A systematic, growing drift — the fast-accum backward accumulation is a real training-numerics
  change, not noise (likely amplified by the K=65536 grad_weight GEMM under concat).
  fast_bwd is dropped from the submission (switches.json back to {} while nothing clears the
  bar); the code stays behind its switch, off.
- Consequence for strategy: a sub-bar submission scores 0 exactly like the unchanged reference
  (gate 1 no_benefit), so the only path is a stack that clears 1.02 AND passes gate 3. Next:
  validate concat alone on gate 3 (check --n1, /tmp/check-r2-n1), then attack the concat
  timing spikes (floor 1.013 s, spikes to 1.05–1.06; allocator config and/or loader parquet
  stalls) with numerics-clean (N0) changes.
1.7 hRound 4 — concat validated on gate 3; attack the timing spikes
- check --n1 on concat-only (/tmp/check-r2-n1): PASS, verdict equivalent — endpoint
  biases +0.0014/+0.0009/+0.0033/+0.0010/+0.0009/−0.0017, max 0.41σ (bound 3σ). The K=65536
  regrouping noise washes out completely over 150 updates. Concat is a validated N1 change.
- Remaining gap to the 1.02 bar: concat alone times 1.0088 (windows 1.0140/1.0072/1.0056); the
  update series shows a floor of 1.013–1.020 s with spikes to 1.05–1.06 s, and the windows grow
  +1.4 % over the run (reference +0.6 %) — a progressive slowdown. Suspects: caching-allocator
  fragmentation at 94 GiB peak (cudaMalloc/cudaFree stalls) or parquet row-group read stalls in
  the loader. Profiling the concat tree over updates 10–15 (/tmp/prof-concat) to discriminate.
- Candidate fix A (zero code, N0): PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True via
  switches.json (not in reserved_env; allocator behavior only, no kernel or value change).
1.7 hRound 4 (continued) — CUDA graphs dead end; prefetching loader
- CUDA graphs attempt (NANOCHAT_CUDA_GRAPHS=1 → torch.compile(mode="reduce-overhead"),
  same kernels, N0): the timing run SIGSEGVs on both ranks (/tmp/time-r4-cg, status ERROR).
  FA3/cuBLAS workspaces under graph capture are not stable in this setup. Reverted; the lever is
  dead (also note gate 1 invalidates runs where CUDA graphs are "requested but skipped", so this
  lever was all-or-nothing from the start).
- Prefetching loader (NANOCHAT_LOADER_SLOTS, nanochat/dataloader.py, default 0 = upstream):
  the profile showed the gaps are the loader's own work (rustbpe encode + best-fit packing,
  ~19 ms/batch, untraced by the profiler) landing at chunk boundaries. New opt-in variant
  _prefetched_bos_bestfit: a worker thread produces the byte-identical batches (packing
  logic copied verbatim) into a pool of N slot buffers; next() pops a finished batch and clones
  it out of the slot. Slot reuse is made explicitly safe: the consumer records a CUDA event after
  its clone and hands the slot back; the worker blocks on that event before refilling the slot
  (so both the slot's pinned staging buffer and device buffer are provably idle when reused).
  The free-slot handoff bounds the worker to N batches ahead — tokenization and parquet
  row-group bursts disappear from the critical path. Verified byte-identical against the
  synchronous loader over 12 batches (first version had a slot-reuse race caught by the test at
  batch 6 — fixed by the event-after-clone handoff; also noted rustbpe's encode is not safe to
  run concurrently from two threads, which only ever happens if two loaders run at once — the
  recipe runs one).
- check --quick on concat=2 + loader_slots=4: running → /tmp/check-r4-quick.
2.9 hRound 5 — config levers: expandable_segments and cpp_wrapper (both dead)
- Profile forensics (from /tmp/prof-concat, torch.profiler over updates 12+; the trace
  covers 6 updates — cluster the 18-per-update AllGather kernels to segment it): per update the
  GPU spends 1.013 s in kernels: nvjet FP8 GEMMs 462.7 ms, Inductor "clone/transpose" kernels
  176.7 ms, other Inductor glue (FP8 casts, norms) 154.8 ms, FA3 113.6 ms, amax reductions
  41.8 ms, cross-entropy 20 ms, NCCL 23.5 ms. GEMMs (462.7 ms for ~573 TFLOP/update ≈ 1.2
  PFLOPS effective) and FA3 are near roofline; the Triton glue runs at only ~1.5–1.6 TB/s.
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (N0, allocator-only):
  /tmp/time-r4-es → 1.0069 (windows 1.0216/1.0372/1.0383, smallest ratio 1.0035) —
  slightly worse than concat-only (1.0088). Dead.
- NANOCHAT_CPP_WRAPPER=1 (Inductor C++ launcher, N0): /tmp/time-r5-cw → 1.0053
  (windows 1.0207/1.0389/1.0423, smallest ratio 0.9996). Window 1 marginally better, windows
  2–3 worse (within the ~0.5 % run-to-run noise those windows show). Dead.
- Run-to-run noise on windows 2–3 across 4 behaviorally identical concat runs: 0.52 %/0.60 % —
  larger than the reference's frozen spread (0.17 %). Screening 1–2 % changes on those windows
  is unreliable; need ≥2.5 % real gains.
2.9 hRound 6 — tiled-transpose op for the backward FP8 layout copies: REGRESSION, off
- The idea: _scaled_mm needs row-major × column-major, so each Float8Linear backward makes
  go_T = go_fp8.t().contiguous() and _to_col_major(in_fp8). A 64×64-tiled Triton transpose
  does the same copy bitwise-exactly at ~3.3 TB/s vs ~1.6 TB/s for the Inductor clones (and
  300 GB/s for aten's .t().contiguous()). Also verified the reformulated GEMM 1
  (w_col.t() @ go_fp8.t() — free views of copies the reference already makes) is **bitwise
  identical** to the reference GEMM 1 on representative shapes.
- Implementation (nanochat/fp8.py, switch NANOCHAT_FP8_TTRANS, default off): the kernel is
  registered as a real aten op (torch.library.Library("nanochat_fp8", "DEF") + CUDA/CPU/Meta
  impls) so dynamo/Inductor treat it as an opaque extern call. First tried
  torch.library.custom_op — NOT opaque: dynamo inlines the triton kernel and Inductor re-fused
  it into strided epilogues.
- Result: /tmp/time-r6-tt2 → 0.978 (regression), peak memory 94.04 → 106.76 GiB (+12.7 GiB,
  identical with both registration mechanisms). Category diff of the profile
  (/tmp/prof-r6-tt2 vs /tmp/prof-concat): clone kernels 176.7 → 2.8 ms (−174), my tiled
  kernels +51.5 ms (running at the predicted ~3.2 TB/s), but other Inductor glue +122.3 ms
  and the plain FP8 casts now materialize separately (+36/+29/+27 ms kernels). The opaque op
  forces materializations Inductor previously fused into the clone kernels (cast + transposed
  write in one pass) and re-plans fusion/layout across the whole backward, making unrelated
  kernels (rmsnorm bwd, CE, amax variants) slower. **Fighting Inductor's planner one link at a
  time is a losing game; the reference's fused clones were already near-optimal as a system.**
- Kept the code (switch off) for the record. Committed f402245.
2.9 hRound 6 conclusion → Round 7 direction
The remaining big non-roofline block is ~330 ms/update of Inductor Triton glue (clones 176.7 +
casts/norms 154.8) at ~1.5 TB/s with heuristic kernel configs. The graph/fusion plan itself
is good — what's suboptimal is the per-kernel config choice (block sizes, num_warps,
vectorization). torch.compile(mode="max-autotune-no-cudagraphs") tunes exactly that without
touching the plan: pointwise kernels and max-reductions (amax) produce identical values under
any config; only reduction-order rounding (norms, CE) can differ — N1-class, same tier as
concat which passed gate 3 at 0.41σ. Pure config lever, no code change: NANOCHAT_COMPILE_MODE.
3.1 hRound 7 — torch.compile max-autotune (NANOCHAT_COMPILE_MODE)
- Change: scripts/base_train.py passes mode=os.environ["NANOCHAT_COMPILE_MODE"] to
  torch.compile (default empty = reference behavior). Screening config
  /tmp/switches-r7.json = NANOCHAT_CONCAT_MB=2 +
  NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs: Inductor compile-time-autotunes its
  generated Triton kernels instead of using heuristic configs. Same fusion plan, same kernels;
  pointwise + amax configs cannot change values, reduction configs only change rounding order
  (N1-class, like concat). CUDA graphs stay off (they SIGSEGV, Round 4).
- Compile time is much longer (autotune benchmarks each kernel, including Triton GEMM candidates
  for the bf16 GEMMs); the 34-update run pays it in warmup updates 0–9, which are untimed.
- Result: /tmp/time-r7-ma — pending.
3.1 hRound 7 result — max-autotune is the big lever
- /tmp/time-r7-ma (concat=2 + NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs):
  speedup_vs_frozen_reference 1.0225 (concat-only was 1.0088), windows
  1.0059/1.0207/1.0231 s/update → per-window ratios 1.0297/1.0198/1.0184,
  smallest 1.0184 — still just under the 1.02 bar (no_benefit), but a +1.4 %
  real gain over concat-only with zero memory cost (94.04 GiB) and zero code
  change. Update floor dropped from ~1.013–1.020 s to ~0.99–1.01 s; windows 2–3
  still carry the progressive-slowdown spikes (max 1.0463).
- Compile cost: startup_to_first_update 750 s (autotune benchmarks every kernel,
  including Triton GEMM candidates). Untimed (warmup updates 0–9) and inside the
  judge's per-run timeouts (timing 2700 s, record/trajectory 3600 s). The
  judge's deterministic mode takes the first listed config without timing, so
  gates 2/3 are unaffected by the mode.
- Committed e914a9c. Switches.json stays {} (sub-bar scores 0, same as the
  unchanged reference) until a stack clears 1.02 on the smallest window AND
  passes the full check.
- Next: stack levers that add margin on windows 2–3: NANOCHAT_CONCAT_MB=4
  (one fwd/bwd per update; halves chunk-boundary launch gaps; memory risk
  ~+25 GiB) and chunked cross-entropy (~10–15 ms + ~8.6 GiB, which also damps
  the allocator-pressure spikes).
3.7 hRound 8 — chunked cross-entropy: regression, off; spike analysis
- NANOCHAT_CHUNKED_CE=8192 (new nanochat/chunked_ce.py + hook in gpt.py): the whole
  lm_head → float → softcap → F.cross_entropy chain as one autograd Function walking the vocab
  in blocks — block logits are bitwise slices of the reference GEMM (same _to_fp8 scales, same
  K reduction), running logsumexp in fp32, recompute-in-backward, then the unchanged two FP8
  backward GEMMs. Equivalence test: eager loss bitwise equal, grads within 1 bf16 ulp
  (median rel diff 0.0), compiled-vs-eager 9e-8. Handles ignore_index and vocab padding.
- Result: REGRESSION — /tmp/time-r8-ce: 1.0012 (windows 1.0295/1.0437/1.0417, smallest
  ratio 0.9973), peak memory UP to 98.04 GiB (from 94.04). The +33 % lm_head GEMM time
  (recompute) plus the per-block fp32 pointwise chains outweigh the reference's fused CE
  pipeline. Switch off; committed e781fd3. Also NANOCHAT_CONCAT_MB=4 OOMs (needs 8 GiB more
  than free) — dead. NANOCHAT_FP8_WCACHE=1 on top of the stack: 1.0225, smallest ratio 1.0186
  (vs 1.0184 without) — no gain, off.
- Spike analysis (reference runs/bootstrap/baseline series vs mine): the reference's
  per-update floor is ~1.027–1.032 s with spikes to ~1.048 (+1.5–2 % over floor); my autotuned
  stack's floor is ~0.99–1.00 s but spikes reach 1.046–1.064 (+4–6 %). My spikes are ~3× taller
  than the reference's. Hypothesis: autotune shortened GPU work enough that the CPU is now the
  co-bottleneck, so the loader's ~19 ms/batch packing bursts (hidden before) surface as GPU idle
  — exactly what Round 4's prefetching loader removes (it measured no gain pre-autotune, when
  the CPU had slack). Testing NANOCHAT_LOADER_SLOTS=4 on top of the stack.
3.7 hRound 8 conclusion — loader_slots marginal; spike root cause found
- NANOCHAT_LOADER_SLOTS=4 on top of concat=2 + autotune: /tmp/time-r8-ls —
  1.0237 (windows 1.0039/1.0203/1.0220, ratios 1.0318/1.0202/1.0195, smallest
  1.0195). Window 1 improved (1.0202 → 1.0039 s/update) but windows 2–3 kept their spikes
  (u17–20: 1020/1028/1026/1039 ms, u27–28: 1038/1039, floor ~995–1000) — threaded prefetch
  moves the loader off the chunk boundary but its GIL-held work still stalls the training
  thread's kernel launches.
- Root cause measurement (CPU-only, frozen assets, /tmp/loader_cost.py): parquet row
  groups are 1024 rows; read_row_group 10.1 ms, to_pylist 4.0 ms — cheap. The expensive
  part is rustbpe encode + the packing loop: ~6.5 encode calls of 128 docs per update
  (~60–80 ms/update of loader CPU), and at document-buffer refills the worker runs up to 8
  back-to-back encodes with no packing in between — a near-continuous GIL-held burst exactly
  at the rg transitions (u17/u20/u27/u28/u32 where the spikes live, both trees). The
  training thread has ~0 slack (GPU ~1.0 s/update vs wall ~1.0 s), so any worker GIL time is
  a direct stall; the reference is frozen with its own +1.5–2 % spikes, so removing mine is
  pure ratio.
- Conclusion: threads cannot fix this (shared GIL). Next: a forked child process that owns
  read + encode + packing (own GIL); the parent only receives packed rows over a pipe.
4.1 hRound 9 — process-based loader (NANOCHAT_LOADER_PROC, default off)
- Change: nanochat/dataloader.py adds _proc_loader_child (runs in a forked child:
  _document_batches → rustbpe encode → the verbatim best-fit packing loop → sends each
  packed batch as flat int64 [inputs | targets] bytes + state indices over a
  multiprocessing pipe, then blocks for an ack = at most one batch in flight) and
  _proc_bos_bestfit (parent side: same slot-pool consumer machinery as the threaded
  variant, but the worker thread only does conn.recv() — GIL released while blocked —
  np.copyto into the slot's pinned staging buffer, the H2D enqueue and the ack; ~1–2 ms of
  GIL per batch instead of ~15–20). Dispatch: NANOCHAT_LOADER_PROC=1 inside
  tokenizing_distributed_data_loader_with_state_bos_bestfit (same yields, same order, same
  state dicts — the hook point is unchanged). Child safety: forked before any encode in the
  parent, never touches CUDA, daemon=True, EOF on the pipe when the parent dies, explicit
  terminate in the generator's finally.
- Equivalence (GPU test, /tmp/proc_loader_test*.py): batches 0–2 bitwise identical to
  the synchronous loader through both _proc_bos_bestfit and the public dispatch
  (inputs, targets and state dicts equal); no stray child processes after close().
- Numerics tier: N0 for the loader itself (bitwise-identical yields; gate-2 token-row
  digest must match exactly). The stack it rides on (concat=2 + autotune) is N1 as before.
- Result: /tmp/time-r9-proc (concat=2 + NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs
  + NANOCHAT_LOADER_PROC=1, /tmp/switches-r9.json) — pending.
4.1 hRound 9 result — process loader clears the gate-1 bar
- /tmp/time-r9-proc (concat=2 + NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs +
  NANOCHAT_LOADER_PROC=1): speedup_vs_frozen_reference 1.0266, windows
  1.0035/1.0173/1.0172 s/update → per-window ratios 1.0322/1.0232/1.0243,
  smallest 1.0232 — above the 1.02 bar for the first time. Peak 94.04 GiB
  (unchanged). Windows 2–3 improved ~3 ms/update vs the threaded loader
  (1.0203/1.0220 → 1.0173/1.0172).
- Residual spikes persist at the same update indices (u20 1.0401, u26–28 up to
  1.0343, floor ~0.994–1.005) even with the producer off-process — the GIL was
  the main but not the only component. Candidates: allocator stalls
  (expandable_segments untested on THIS stack), CPU descheduling, child CPU
  bursts. The reference spikes at the same indices (frozen into its windows), so
  any further spike removal is pure ratio (~+1 % on windows 2–3).
- switches.json set to this stack and committed (2034051). Launching the full
  check → /tmp/check-r9-full: gate 1 re-times the tree, gate 2 record (expect
  NOT_BITWISE from concat+autotune, exact items must pass), gate 3 trajectory
  (expect equivalent like concat-only's 0.41σ).
4.1 hRound 9 full check — gate 1 PASS (in-check re-measurement)
- /tmp/check-r9-full gate 1 (timing of the submitted stack, in-check): windows
  1.0054/1.0186/1.0189 s/update → ratios vs frozen reference
  1.0303/1.0219/1.0226 → smallest 1.0219 > 1.02 ✓ (second independent
  measurement; the time run gave 1.0232 — consistent, both above the bar).
  Same spike pattern (u18–21 and u26–29 elevated, u28 peak 1039.7 ms).
- Power-cap theory tested and refuted as the spike driver: nvidia-smi poll
  (250 ms) joined against the judge's per-update timestamps shows power pinned
  at ~684–696 W (the board cap) on EVERY update — floor and spike alike — and
  mean SM clocks 1525–1625 MHz with NO systematic dip in the spike updates
  (u28 rank 0 even shows the highest mean clock, 1625 MHz, at its worst 1040 ms).
  The big spikes (u20/u28) run at 96–100 % util → the GPU is doing MORE work or
  waiting on something other than clocks; some mild spikes (u17/u19/u21 on rank 1)
  show util dips to 85–93 % → host-side launch stalls there. Mixed causes; a
  profile over u18–21 (vs floor u14–17) is the next diagnostic.
- Record phase (gate 2) started 20:13:25; trajectory (gate 3) after — result
  pending in /tmp/check-r9-full/check.json.
4.1 hRound 9 full check result — **PASS, speedup 1.0219, tier N1** (20:53)
- /tmp/check-r9-full/check.json: status PASS, speedup 1.0219, tier N1 — the first
  full-check PASS of the trial.
  - Gate 1: candidate windows 1.0054/1.0186/1.0189 s/update → ratios
    1.0302/1.0219/1.0226, smallest 1.0219 > bar 1.02 ✓. 34 steps, peak 94.04 GiB,
    startup→first update 743 s, other_optimizer_steps 0.
  - Gate 2: exact items all OK (10 updates, 175 parameters, 640 rows/rank, loader/model
    provenance, hyper-parameters, dtypes) → NOT_BITWISE (first differing update 0,
    53→175 parameters — expected from concat regroup + Inductor autotune), so gate 3
    decides. Record wall 1121.7 s, deterministic flags all on, FA3 deterministic patched.
  - Gate 3 (trajectory seed 1, 150 updates): verdict equivalent — every endpoint
    within: |d| in σ units = 0.026 / 0.047 / 0.368 / 0.049 / 0.042 / −0.024
    (windows 5–15 … 115–150); the 3σ bounds are 0.068/0.039/0.024/0.041/0.074/0.099 loss
    units. Worst endpoint 30–50 at 0.37σ — comfortably equivalent, same magnitude as the
    concat-only result (0.41σ).
- Conclusion: the submitted stack (concat=2 + max-autotune-no-cudagraphs + process loader)
  is validated end-to-end. Margin above the 1.02 bar is thin (1.9 %); the next rounds
  target the fixed-index spikes (u17–20, u26–28, floor elsewhere) which cost windows 2–3
  ~13–18 ms/update.
5.9 hRound 10 — profile the spikes: root cause found and fixed (loader H2D copy blocks the training stream)
- profile --from 14 --iters 8 → /tmp/prof-r10 (updates 14–21, rank 0, trace
  timing_profile.trace.json, 89 MB chrome trace).
- The spikes are GPU idle, not slower kernels. Per update: GPU busy is flat
  969–980 ms while wall goes 1005→1164 ms; idle grows 31.6 → 30.9 → 49.0 → 63.9 →
  60.1 → 74.5 → 193.5 ms. Kernel-time diffs between the 3 slowest and 3 fastest
  updates are noise (±2.5 ms). u20 alone has a 131 ms gap.
- Inside every big gap the host issued zero kernel launches and no aten ops —
  it was blocked, not working. The one 131 ms gap sits inside Optimizer.step and
  ends exactly when step returns.
- **The blocker: 16 × cudaStreamSynchronize of 140–187 ms on the training
  stream** — exactly 2 per update, inside aten::copy_ ops that each end with a
  512 KB Memcpy HtoD (Pageable -> Device) followed by the consumer's clones.
  That is the proc loader's produce thread staging gpu_buffers[slot].copy_(cpu_buffers[slot]):
  when the CUDA runtime does not treat the source as pinned it synchronizes the
  copy's stream before returning — and on the training stream that wait is the
  entire enqueued backlog (~150 ms of forward/backward). The producer therefore
  paced the consumer 1:1 (2 of its 4 copies/update block during the backwards,
  2 return instantly at the drained boundaries — matching the 16 blocking copies),
  the 4-slot queue ran dry at every micro-batch boundary, and the GPU idled
  ~9 ms (floor) to 131 ms (spike updates) waiting for the next batch.
- Isolated reproductions (no GPU contention for the lock, tiny scripts):
  side-thread copy_(pinned, non_blocking=True) with a busy stream: 0.15 ms;
  same with a pageable source: 318.8 ms. Connection.recv releases the GIL
  (refuted as the cause); upstream base_train already disables GC after step 0
  (refuted); power/clocks were refuted in Round 9.
- Fix (commit da86ca6, tier N0): staging runs on a dedicated torch.cuda.Stream()
  (empty, so the runtime's synchronize is a no-op) with staged_events[slot]
  recorded on it; the consumer's default stream waits that event before cloning.
  Yields bitwise identical to the sync path (batches 0–3 + state dicts); fetch
  latency under an 80 ms GPU backlog 4–23 ms → 0.06–0.32 ms; no
  cudaStreamSynchronize in a profiled fetch loop. Same mechanics available for
  the (off-by-default) NANOCHAT_LOADER_SLOTS path if ever needed.
- Expectation: windows 2–3 lose their ~13–18 ms/update of boundary idle; if the
  floor idle (~20 ms/update) also shrinks, all three windows approach the
  0.98–1.00 s/update floor → smallest ratio ~1.03+. time run in flight
  (/tmp/time-r10-cs); result below.

- **Result (/tmp/time-r10-cs, judge time, tree da86ca6): windows
  1.0048 / 1.0180 / 1.0167 s/update → smallest window ratio 1.0225**
  (speedup_vs_frozen_reference 1.026 over the window means, tokens/s 258734).
  Against Round 9's proc-loader run (1.0232 / 1.0219) and the full check's
  1.0302 / 1.0219 / 1.0226 this is **within noise: the copy-stream fix did not
  remove the spikes**. Per-update durations from timing_tree.r{0,1}.json
  still show the elevated blocks u16–21 (peak 1.0345) and u27–29 (up to 1.034)
  over a 0.996–1.007 floor, on both ranks. So the 140–187 ms blocking copies the
  profiler saw were real but were an artifact of *where* the producer's copy
  landed under CUPTI/profiler visibility, not the cause of the clean-run spikes —
  the pageable-vs-pinned treatment of the staging source is context-dependent
  (the same allocation is pinned in standalone runs).
- Kept anyway: the fix is correct, N0 (bitwise-identical yields verified),
  and strictly more robust — a pageable fallback can no longer pace the training
  stream. Tree at da86ca6; submission switch file unchanged
  (NANOCHAT_CONCAT_MB=2, NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs,
  NANOCHAT_LOADER_PROC=1).
- CPU cgroup throttling refuted (/sys/fs/cgroup/cpu/cpu.stat: nr_throttled 6,
  ~5.8 ms throttled total; 48 CPUs allowed); rank-identical step ends are forced
  by the step-end all-gather, so they do not by themselves prove a shared cause.
  bench.py does not poll the GPUs during a run (wait_idle runs only before).
- Open: the spikes are GPU idle (busy time flat 969–980 ms across updates),
  appear at the same update indices in every stack including the reference
  (data-positioned), and are worth ~13–18 ms/update on windows 2–3
  (smallest ratio 1.0225 → ~1.036 if they were removed). Next: a CPU-only host
  poller (per-process CPU/faults/schedstat wait/context switches, vmstat
  compaction, PSI) running alongside a judge time run, joined against
  step_end_s (the entry stamps time.time()), to separate loader-child CPU
  bursts, rank descheduling, page-fault/compaction storms and host pressure.
6.5 hRound 11 — the spike mechanism found: `device == "cuda"` is never true in DDP
- Diagnosis first (host poller alongside a judge time run): a CPU-only poller
  (per-process CPU/faults/schedstat/context switches + vmstat + PSI, 5 Hz) ran next to
  /tmp/time-r11-poll (speedup 1.0249, windows ~1.00/1.018/1.017 — the same spike
  pattern). Join against step_end_s: **rank CPU is flat ~1080 ms/update on spike and
  floor updates alike, schedstat runqueue wait ≈ 0, no majflt/compaction/PSI signal, no
  correlation of any host counter with the spikes**. So the spikes are a *wait*, not
  work — which pointed back at the Round 10 blocking copies.
- The bug: nanochat/common.py compute_init() returns
  device = torch.device("cuda", ddp_local_rank) — a torch.device, never the string
  "cuda". Every device == "cuda" in dataloader.py is therefore **False in every
  judge run**: the NANOCHAT_LOADER_PROC / NANOCHAT_LOADER_SLOTS dispatches never
  fired (Round 9's "proc gain" was inline-vs-inline noise; Round 10's side-stream fix
  never ran), and the inline loader computed use_cuda = device == "cuda" → False →
  pageable staging buffer + gpu_buffer.copy_(cpu_buffer, non_blocking=False) —
  exactly the 2-per-update synchronous 512 KB H2D copies on the training stream that
  the Round 10 profile caught (cudaStreamSynchronize behind the enqueued backlog,
  140–187 ms). Upstream/reference share the same latency; the spikes are its queue-phase
  jitter. All prior observations are consistent with this one bug.
- Fix (commit 19ba8b8, tier N0): new _is_cuda_device() (True for "cuda",
  "cuda:1", torch.device("cuda", 0)); both dispatches and use_cuda use it. The
  inline path now stages through **two alternating pinned buffers, each guarded by a
  CUDA event** (a pinned source is read asynchronously, so a buffer must not be
  overwritten before its previous copy completes) and issues the H2D non_blocking —
  no stream synchronization anywhere. With NANOCHAT_LOADER_PROC=1 the proc loader now
  actually engages: off-process encode, pinned slots, side-stream staging (Round 10
  machinery, previously dead code under the judge).
- Verified standalone (/tmp/test_devfix.py, /tmp/test_fork.py): dispatch engages
  with device=torch.device("cuda", 0); proc and inline paths yield 3 batches bitwise
  identical with equal state dicts; inline staging buffers pinned (2 × 65536 int64);
  proc path forks a child and delivers (16, 2048) on cuda:0.
- Expectation: the two blocking pageable copies per update vanish from the training
  stream; spike updates (u16–21, u27–29, +15–47 ms) should collapse toward the
  ~0.99 s floor, and with encode off-process the floor idle should shrink too —
  windows ~1.00 or below → smallest ratio ~1.03+. time run in flight
  (/tmp/time-r11-fix); result below.
- **Result (/tmp/time-r11-fix, judge time, tree 19ba8b8): windows
  0.9893 / 0.9944 / 0.9942 s/update → per-window ratios 1.0470 / 1.0468 / 1.0480,
  smallest 1.0468** (speedup_vs_frozen_reference 1.0472, tokens/s 264093,
  peak memory unchanged 94.04 GiB). The spikes are gone: per-update durations
  are smooth 0.975–1.000 s with no 1.02–1.05 outliers (previous runs: floor
  0.996–1.007 with +15–47 ms spike blocks), and the floor itself dropped
  ~10–20 ms/update — the off-process encode removed the boundary idle as well.
  Biggest single gain of the trial: smallest window ratio 1.0225 → 1.0468.
- Full check on 19ba8b8 launched next (the record gate exercises the forked
  loader under the judge for the first time); result to follow.
6.5 hRound 11 full check — PASS, speedup 1.0473, tier N1 (23:16)
- /tmp/check-r11-full/check.json on tree 19ba8b8 with the submitted switch file:
  status PASS, performance.speedup 1.0473, tier N1.
- Gate 1 (timing, one run vs frozen reference): windows 0.989 / 0.9934 / 0.9935
  s/update → ratios 1.0473 / 1.0478 / 1.0487, smallest 1.0473 (bar 1.02,
  reference spread 0.0017). Per-update durations smooth, no spike blocks.
- Gate 2 (record, deterministic, 10 updates): exact items all OK — 175 parameters,
  rows_per_rank 640, one loader, one optimizer, no foreign state dtypes; NOT_BITWISE
  (first differing update 0, 53 parameters: lm_head.weight, resid_lambdas, …) — the
  same concat+autotune kernel-choice signature as the r9 check, not loader-related.
  The forked proc loader ran under the judge's record mode for the first time without
  any issue.
- Gate 3 (trajectory, seed 1, 150 updates): verdict equivalent — endpoints
  0.050σ / 0.063σ / 0.445σ / 0.107σ / 0.059σ / −0.011σ, worst 0.445σ of the 3σ bound
  (r9: 0.37σ — same computation, as expected from a value-identical loader).
- The submitted tree (19ba8b8) and switch file are exactly what passed. Cost 1.79 GPU·h.
8.0 hRound 12 — profile the new floor; screen TORCHINDUCTOR_COORDINATE_DESCENT_TUNING
- Profile (/tmp/prof-r12, judge profile --from 14 --iters 4, tree 19ba8b8): the
  run is now GPU-bound — busy 981–987 ms vs idle ~13 ms per update (profiler run;
  clean-run walls 0.975–1.000 s). The host runs far ahead ("Command Buffer Full" is the
  top CPU item, 2.69 s over 4 updates) and spends 215–223 ms inside Optimizer.step
  waiting for the queue — harmless, the GPU is the bottleneck.
- Phase split per update (GPU time): backward 2×323 ms = 646 ms (the wall), forward
  2×147 ms = 294 ms, Optimizer.step 48 ms (NCCL ~24.8: reduce_scatter ~12 +
  all_gather ~13 + adamw reduce ~7.4 partially overlapped; compiled Muon/AdamW compute
  ~35 ms, pipelined per shape group).
- Biggest fused-Triton items inside fwd/bwd: fp8 amax+cast reductions ~19 ms/update,
  fused rms_norm backward ~23 ms/update, log-softmax fwd+bwd ~17 ms/update, FA3 bwd
  ~87 ms/update.
- Candidate this round: TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1 (pure switch,
  not in reserved_env) — Inductor config/layout refinement for exactly those fused
  Triton kernels; numerics class unchanged (kernel choice, like COMPILE_MODE).
  Screening on a scratch config /tmp/sw-cd.json; time in flight
  (/tmp/time-r12-cd); the submission stays at the certified 19ba8b8 + switches
  until this pays.
8.0 hRound 12 result — TORCHINDUCTOR_COORDINATE_DESCENT_TUNING: no gain
- /tmp/time-r12-cd (scratch config /tmp/sw-cd.json, tree 19ba8b8): windows
  0.9892/0.9932/0.9933 → smallest ratio 1.0471, speedup 1.048 — vs the certified
  tree's own check-run windows 0.989/0.9934/0.9935 (smallest 1.0473). **Within noise,
  no benefit** (compile +28 s). Switch not adopted; submission unchanged.
8.5 hRound 13 — overlap the gradient all_reduces with the final backward
- Target from the Round 12 profile: Optimizer.step = 48 ms/update of serial GPU time
  after the backward (reduce_scatter ~12 + all_gather ~13 + fused Muon/AdamW compute
  ~35, partially overlapped across groups). The reduces cannot overlap the *next*
  forward (the judge's step hook synchronizes), but they can overlap the *remaining
  backward* if launched per-parameter as each grad finishes.
- Verified prerequisites standalone:
  - per-tensor all_reduce(AVG) == stacked reduce_scatter_tensor(AVG) rank chunk,
    bitwise, for 2 ranks (/tmp/test_comm_equiv.py);
  - register_post_accumulate_grad_hook fires progressively during the compiled
    backward, no graph break (/tmp/test_hooks.py).
- **Change (commit 8941906, N0 vs the current tree, switch NANOCHAT_OVERLAP_REDUCE
  default off)**: hooks launch all_reduce(grad, AVG, async) on a dedicated stream
  during the update's final backward (event-ordered after each param's accumulation);
  step() waits the futures, orders the compute stream after the comm stream, and
  consumes the averaged grads exactly like the serial path (muon: same stack build
  with reduce_scatter replaced by a rank slice; adamw: row views). base_train marks
  the final accumulation via optimizer.notify_backward() (no-op when off).
- A first cut was wrong and caught by the standalone test: copying the reduced
  grads into stack slots from inside the hook raced the collective (an async NCCL
  work's kernel enqueue can trail a later host enqueue on the same stream — stale
  slots on rank 1, params off by ~4e-3). Replaced by the step()-side stack build.
- Equivalence: 2-rank, 3-update test, eager and torch.compile: **all parameters
  bitwise identical** with the switch off vs on.
- time in flight (/tmp/time-r13-ovl, scratch config /tmp/sw-ovl.json); result
  below.
- Result (/tmp/time-r13-ovl): REFUTED — the overlap is slower. Windows
  0.9986/1.005/1.0066 → smallest ratio 1.0351 (certified tree: 0.989/0.9934/0.9935 →
  1.0473), i.e. −10 to −13 ms/update. 145 per-parameter all_reduces (each with NCCL
  launch/protocol overhead) plus ~2× the reduce_scatter's bytes cost more than the
  ~19 ms of serial comm they hide; the stacked reduce_scatter's batching is worth more
  than overlapping it. The switch stays off; commit 8941906 reverted so the submitted
  tree is exactly the certified 19ba8b8 (the inert overlap code is dead weight for a
  refuted lever).
- Lesson logged: on 2 ranks with ~150 params, per-tensor collectives lose to stacked
  batched ones; overlap only pays if the hidden op's own cost does not grow.
9.1 hRound 14 — stack the value_embeds' AdamW collectives (NANOCHAT_ADAMW_STACK)
- Profile finding (from the r12 trace, /tmp/nccl_calls.py): 41 NCCL kernels per
  update inside Optimizer.step, all RING_LL, ~24.7–26.6 ms GPU time. Inventory
  (matched against the reference record's 175 param shapes): 3 Muon reduce_scatters of
  906 MB fp32 (2.27/2.44/2.42 ms) + 3 all_gathers (2.46–2.75 ms); lm_head 201 MB fp32
  (0.86 + 0.6 ms); **13 bf16 reduce_scatters + 13 all_gathers of 100 MB each
  (0.28–0.49 ms apiece, ~8.5 ms total) = wte + the 12 value_embeds** — every one of the
  12 value_embeds is its own nn.Embedding(32768, 1536) bf16 parameter in one AdamW
  group, so each does its own reduce_scatter + all_gather every update.
- NCCL protocol lever killed first (3 min): microbenchmark
  (/tmp/nccl_proto.py, 2 ranks, the exact sizes) shows NCCL_PROTO auto ≈ Simple ≈
  LL128 (906 MB rs 1.68/1.69/1.76 ms; the profiled in-training 2.27 ms is contention
  with the concurrent Muon compute, not protocol); forcing LL is 2.8× worse. No switch
  value there. (The _LL kernel names in the trace are misleading; auto does not pick LL
  for these sizes.)
- Change: new stacked path in MuonAdamW._reduce_adamw/_compute_adamw
  (NANOCHAT_ADAMW_STACK, default off): when an AdamW group has >1 same-shape params
  of ≥1024 elements (only the 12 value_embeds match), write their grads into one padded
  (K, *shape) buffer with _foreach_copy_, do ONE reduce_scatter(AVG) and ONE
  all_gather, and copy back with _foreach_copy_. Rank r then owns whole parameters
  [r*chunk, (r+1)*chunk) instead of a row slice of each — communication layout only:
  the collective averages the same element pairs and the AdamW update is elementwise
  with the same per-parameter state keys/dtypes and the same group hyper-parameters, so
  every parameter is stepped to bitwise-identical values. Saves 24 NCCL launches and
  ~8.5 ms → ~5 ms of collective time per update (1.3 GB moves at the big-collective
  rate instead of 24 latency-exposed 100 MB collectives).
- Record-gate safety verified against the entry's own capture code
  (bench/nanochat_entry.py): hparams comes from param_groups only (untouched);
  state_dtypes collects the (key, dtype) SET across states (identical: exp_avg/
  exp_avg_sq bf16 + fp32 entries unchanged); digests are the parameter SHAs (bitwise).
- Standalone bitwise verification (/tmp/test_adamw_stack.py, 2 ranks, 3 updates,
  grad accumulation, changing lr): all parameters bitwise identical switch off vs on —
  for 12 params per group (recipe shape), and the padding paths 13 and 11; state dtype
  sets equal; cross-rank parameters identical after every update.
- time in flight (/tmp/time-r14-astk, scratch config /tmp/sw-astk.json = the
  certified three switches + NANOCHAT_ADAMW_STACK=1); result below.
- Result (/tmp/time-r14-astk): NO GAIN. Windows 0.9903/0.9953/0.9938 → smallest
  ratio 1.0458 (certified tree: 0.989/0.9934/0.9935 → 1.0473; its own time run
  0.9893/0.9944/0.9942 → 1.0468); speedup_vs_frozen_reference 1.0467; peak memory
  94.04 → 94.61 GiB. Within noise, slightly negative. Change reverted (tree back to
  the certified content); switch not adopted.
- Lesson (important for the remaining rounds): in the 3-phase pipelined optimizer
  step, summing NCCL kernel durations overcounts — the 26 per-parameter ve/wte
  collectives (~8.5 ms of kernel time in the profile) were almost entirely OVERLAPPED
  with the Muon compute, so removing them moves the wall by ~0, while the stacked path
  ADDS ~3 GB of staging copies (~1 ms) that cancel the small exposed remainder. Only
  the critical path pays: the fused Muon compute (~35 ms, of which the Newton–Schulz
  bf16 GEMMs are ~24 ms at ~480 TFLOPS effective), the first reduce_scatter and the
  last all_gather tails, and the end-of-step copy-back. Next rounds attack those.
9.3 hRound 15 — compile the fused optimizer steps with max-autotune (NANOCHAT_OPT_COMPILE_MODE)
- Target from the Round-14 lesson: the step's critical path is the fused Muon compute
  (~35 ms), dominated by the Newton–Schulz batched bf16 GEMMs (~24 ms, 45 nvjet calls
  at ~480 TFLOPS effective — Inductor's default mode takes extern-cuBLAS without
  benchmarking). The two fused kernels are compiled with plain
  @torch.compile(dynamic=False, fullgraph=True).
- Change: NANOCHAT_OPT_COMPILE_MODE (default unset) threaded into both fused-step
  decorators as mode=. Screening max-autotune-no-cudagraphs, the same mode the
  model already uses (a certified N1 change there): Inductor benchmarks Triton GEMM
  templates vs cuBLAS per NS matmul shape and autotunes the pointwise/reduction
  kernels. Kernel choice only — same graph, same math; numerics class N1 (verified in
  a smoke test: both modes compile and run; values differ at kernel-rounding level,
  muon |p| 722.1031 vs 722.1027).
- Under the judge's deterministic record, autotune is replaced by first-config
  selection — the same situation as the certified model-side max-autotune.
- time in flight (/tmp/time-r15-optmode, scratch config /tmp/sw-optmode.json =
  the certified three switches + NANOCHAT_OPT_COMPILE_MODE=max-autotune-no-cudagraphs);
  result below.
9.7 hRound 16 — one-pass Muon staging (NANOCHAT_MUON_ONEPASS), screened on top of Round 15
- Target: the Muon path's eager buffer shuffling on the step's critical path —
  torch.stack(grads) + copy_ into the padded buffer before each reduce_scatter
  (two passes over ~906 MB per group), and torch.stack(owned) + copy_ into the
  all_gather input before each gather (two passes over ~450 MB per group). ~5.4 GB of
  removable copy traffic per update (the stacks sit serially before their
  collectives, so unlike Round 14's collectives this traffic is exposed).
- Change: NANOCHAT_MUON_ONEPASS (default off) — _foreach_copy_ the grads
  straight into the padded stacked buffer (one pass), and pre-fill the all_gather
  input with the owned parameters' current values so the fused kernel updates it in
  place (one pass, no separate stacked_owned). Buffer placement only → bitwise
  identical (tier N0 vs the current tree).
- Verified standalone (/tmp/test_muonepass.py, 2 ranks, 3 updates, grad
  accumulation, changing lr/momentum): all parameters bitwise identical for 6-param
  groups (3+3 chunking) and 5-param groups (3+2 + padding), cross-rank identical;
  single-rank path also bitwise identical (/tmp/test_muonepass_ws1.py). (A first
  cut had a single-rank+switch-off bug — stacked_params=updated_params would have
  been uninitialized — caught while reviewing the diff before testing and fixed.)
- time in flight (/tmp/time-r16-onepass, scratch config /tmp/sw-r16.json = the
  certified three + NANOCHAT_OPT_COMPILE_MODE + NANOCHAT_MUON_ONEPASS), compared
  against Round 15 alone (windows 0.9887/0.9926/0.9921 → smallest 1.0476); result
  below.
- Result (/tmp/time-r16-onepass): NO GAIN — reverted. Combined stack windows
  0.9884/0.9943/0.9950 → smallest ratio 1.0469, vs Round 15 alone
  0.9887/0.9926/0.9921 → 1.0476: windows 2–3 got ~1.7/2.9 ms WORSE, window 1 ~0.3 ms
  better. The _foreach_copy_ staging (multi_tensor_apply per tensor) is slower than
  the single batched CatArray/copy_ kernels it replaced for these 96×9.4 MB groups,
  eating the saved bandwidth. The R16 edit is reverted from the tree; switch not
  adopted (bitwise N0 but performance-negative).
- Decision: adopt Round 15 alone. Its gain is small (smallest ratio 1.0473 →
  1.0476; windows −0.3/−0.8/−1.4 ms vs the certified check-run, consistent across all
  three windows) but real, and it is already implemented, committed and time-verified.
  Because it is an N1 change (optimizer kernel choice → new trajectory), adoption
  requires gate-3 evidence: running the full check with the 4-switch SCRATCH config
  (/tmp/sw-optmode.json) while /submission/switches.json stays at the certified
  3-switch file — the submission only moves to the 4-switch file if that check PASSes.
  On failure the certified stack remains submitted, unchanged.
- Full check launched → /tmp/check-r15-full (tree 200f1e8, config
  /tmp/sw-optmode.json); result below.
↑ NanoChat