Claude Opus 5.5 2.68×

Harness: Claude Code · switches: YOCO_FAST_OPT=1 YOCO_FAST_GRAPH=1 YOCO_FAST_NORM=1 YOCO_FAST_RET=1 YOCO_FAST_LINEAR=1 YOCO_FAST_MERGE=1 YOCO_CUBLAS_DIR=/opt/candidate-env/lib/python3.10/site-packages/yoco_cublas128 YOCO_FAST_NORM2=1 YOCO_FAST_CE=1 YOCO_FAST_EMBED=1 YOCO_FAST_CHAIN=1 YOCO_FAST_ACC=1 YOCO_FAST_ATTN=1 YOCO_FAST_GEMM=1 YOCO_FAST_HOIST=1 YOCO_FAST_NORM3=1 YOCO_FAST_PITCH=1 YOCO_FAST_NORM4=1

Judged
2.68×
Tier
N1
Self-best
2.68×
Rounds
18
Hours
10.0
API cost
$57.6
Input tokens · 99.1% cached
144 M
Output tokens
0.9 M
Requests
286
Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
✓ Adam kernel writes bf16 weights straight to both GPUs, no more weight exchange; grads pushed via copy engine; this step 2.18× → 2.35×
Forward / backward CUDA graph
✓ YOCO_FAST_GRAPH, forward + loss + backward + grad reduction in one graph, 1.52× → 1.64×
Newer cuBLAS
✓ 12.8.4.1, 2.11× → 2.18×
Retention / attention kernel swap
✓ YOCO_FAST_RET custom Triton retention 1.985× → 2.08×; YOCO_FAST_ATTN switched to cuDNN SDPA 2.57× → 2.63×; retention kernel moved to Triton 3.8, 2.64× → 2.68×
Other main levers
round 2 replaced the whole optimizer path (flat buffers, custom bitwise-equivalent CUDA Adam, custom grad reduction replacing DDP and OSS, straight to 1.52×); merged q/k/v/g and fc1/gate GEMMs; dW written directly into the flat grad buffer; fused cross-entropy; CUDA RMSNorm; per-shape GEMM algorithm table
Raw work log19 sections
0.1 hRound 1: baseline profile and NCCL protocol switch
- Start: 2026-11-06 15:37 UTC.
- Profile of the unchanged reference: /tmp/r1/profile (5 updates, instrumented, 0.287 s/update; frozen reference timing 0.2344 s/update).
  Kernel time per update (rank 0): NCCL 67 ms (Broadcast RING_LL 37 ms fp32 master weights, AllReduce RING_LL 30 ms bf16 grads),
  GEMM 64 ms, memcpy/cast 44 ms, elementwise 31 ms, optimizer multi_tensor 18 ms (Adam 14, L2 norm 4), flash-attention 6 ms,
  reduce 4 ms, triton 3 ms, swiglu 2 ms, loss 2 ms. GPU busy 75 %. Host: about 260 ms per train_step under the profiler, so the
  CPU launch path is nearly as long as the GPU time.
- NCCL_PROTO=Simple (/tmp/sw/nccl_simple.json): time /tmp/r1/time_simple: speedup 1.0024, no gain (closed).
- End: 15:46 UTC. No code change.
0.5 hRound 2: fused optimizer path, no DDP/OSS, no host syncs in train_step
- Start 15:46 UTC. Hotspot from the round 1 profile: everything around the optimizer (per-parameter fp32 grad copies, multiply_grads,
  fill, fp32->bf16 copies, the fp32 OSS broadcast behind a full device sync): about 80 ms of the 234 ms, all on the critical path.
- Change (switch YOCO_FAST_OPT): yoco/fastrt: flat bf16 weights and gradients, Adam in one CUDA kernel that reads the bf16 reduced
  gradient and writes fp32 master/m/v and the bf16 weight (same instruction sequence as apex's AdamFunctor, read from its SASS,
  bit-equal in a unit test on 16M elements); gradient norm kernel with apex's chunking, in-block order and 320 slots; own
  gradient reduction replacing DDP (hooks, all-reduce per layer unit); bf16 all-gather of the weights instead of the fp32 OSS
  broadcast; Trainer.train_step without host syncs (pinned H2D, logging deferred to the log interval).
  fp32_params are MasterView objects: .data/.grad give the full fp32 master and clipped gradient when read (adapter contract: views).
- time /tmp/r2/time1: speedup_vs_frozen_reference 1.5238 (0.154 s/update).
- check --quick /tmp/r2/quick1: PASS, measured tier N0, bitwise on all tensors.
- End 16:07 UTC.
0.7 hRound 3: CUDA graph of forward+backward+reduction; fused RMSNorm
- Start 16:07 UTC. Profile /tmp/r3/profile (of the round 2 tree): host time 0.161 s/update, CPU 155 ms in forward+backward, GPU busy 75 %: host bound.
  Kernel time per update: GEMM 62.6, NCCL 39.6 (all-reduce 28.7, all-gather 10.9), memcpy/cast 19, elementwise 14.5, Adam 12.3, flash 5.9, reduce 3.7.
- Change 1 (switch YOCO_FAST_GRAPH): one CUDA graph for forward, loss, backward and gradient all-reduce, captured at the second update
  (fastrt/graph.py); criterion and model are still called as modules every update.
  time /tmp/r3/time1: 1.6388 (0.1432 s/update). check --quick /tmp/r3/quick1: PASS, N0.
- Change 2 (switch YOCO_FAST_NORM): RMSNorm forward/backward and add+RMSNorm as CUDA kernels (csrc/model_kernels.cu), bit-exact with the
  eager chain (the row sums reproduce torch's reduce_kernel order: warp per row, float4 input path above 128 elements).
  time /tmp/r3/time2: 1.801 (0.1303 s/update). check --quick /tmp/r3/quick2: PASS, N0.
- End 16:24 UTC.
1.0 hRound 4: fused gated retention; dW straight into the flat buffer; weights stored to both GPUs by Adam
- Start 16:24 UTC. Hotspot: non-GEMM work of gate retention (0.6 ms per layer forward), all-gather 12 ms exposed, gradient copy 4.7 ms.
- Profile /tmp/r4/profile (with retention). Kernel time per update: GEMM 61.6, NCCL 41 (all-reduce 28.9 overlapped, all-gather 12.1 exposed),
  Adam 12.35, flash 6.2, grad copy 4.65, dkv_bwd 1.9, chunk_sumsq 1.8, rotary 1.5, bf16 add 1.4, swiglu 2.
- Change 1 (YOCO_FAST_RET): chunked retention + subln + gate as Triton kernels (fastrt/retention.py), forward and backward, bf16 rounding
  points of the eager code kept. time /tmp/r4/time1: 1.8895. check --n1 /tmp/r4/n1: PASS (largest endpoint 0.73 sigma, bound 3 sigma).
- Change 2 (YOCO_FAST_LINEAR): autograd function for the bias-free linears, dW by the same cuBLAS call straight into the flat
  gradient buffer (no AccumulateGrad, no halving copy; kernels halve exactly instead).
- Change 3 (in YOCO_FAST_OPT): the Adam kernel stores the bf16 weight to both GPUs (CUDA IPC), no all-gather; gradient sum of squares per unit
  on a side stream as the reductions finish.
  Without retention: time /tmp/r4/time2: 1.9854 (0.1182). check --quick /tmp/r4/quick2: PASS, N0.
  With retention: time /tmp/r4/time3: 2.0786 (0.1129).
- Tried NCCL_PROTO=Simple again? no. pypi.org and github.com are unreachable; download.pytorch.org, pypi.nvidia.com (via redirect, curl -L) and
  conda.anaconda.org work.
- Full check /tmp/r4/check1 (switches: OPT, GRAPH, NORM, RET, LINEAR): PASS, speedup 2.0804, tier N1, trajectory equivalent (largest 0.73 sigma).
- End 16:50 UTC.
1.5 hRound 5: merged GEMMs, cuBLAS 12.8, gradient exchange by copy engine (dev tree /tmp/dev/yoco)
- Start 16:52 UTC. Profile /tmp/r5/profile: GEMM 59 ms (nvjet kernels with cuBLAS 12.8), NCCL all-reduce 29 ms overlapped, Adam 13.3.
- Sizing experiments (timing only, results not valid numerically): Adam concurrent with the next forward: 0.1068-0.1072 vs 0.1077 s/update, no gain
  (GEMMs fill the SMs), direction closed. No gradient all-reduce at all: 0.0971, so the all-reduce costs 10.6 ms (SM contention and tail).
- NCCL sweep on top: PROTO=Simple 2.195, Simple+8 channels 2.044, Simple+4 channels 1.591, LL128 2.145 (default 2.179).
- Merged q/k/v/g/gt and fc1/gate GEMMs (YOCO_FAST_MERGE): time /tmp/r5/time1: 2.1084. Plus cuBLAS <ip> preloaded from /opt/candidate-env (switch YOCO_CUBLAS_DIR):
  time /tmp/r5/time2: 2.1789. Gradient exchange by copy engine + owner-side fused sum/sum of squares: time /tmp/r5/time_ce2: 2.3467 (0.1000).
- check --quick /tmp/r5/quick1 of the copy-engine path without the N1 pieces (OPT, GRAPH, NORM, LINEAR): PASS, N0: the new gradient exchange is bitwise.
- Fused cross entropy (YOCO_FAST_CE) + RMSNorm kernels with vector loads and the weight gradient fused in the backward (YOCO_FAST_NORM2):
  time /tmp/r5/v6: 2.4394 (0.0961 s/update). cuBLAS 12.9.1: 2.4394, cuBLAS 12.6.4: 2.4343: no difference between versions, <ip> kept.
- Full check /tmp/r5/check1 on the dev tree (commit promoted to /submission/yoco): PASS, speedup 2.4394, tier N1, trajectory equivalent (largest 0.46 sigma).
- End 17:29 UTC.
2.1 hRound 6: tail of the backward, fewer tiny kernels (dev tree)
- Start 17:29 UTC. Profile /tmp/r6/profile: GEMM 53 ms now that NCCL is off the SMs, Adam 13.3, flash 6.4, retention 6, norm 3, swiglu 2, rotary 1.5;
  1412 graph nodes per update, 556 of them under 8 us; side-stream chunk_reduce floods the SMs and delays GEMM starts.
- Sparse exchange of the embedding gradient (YOCO_FAST_EMBED) + residual additions folded into the norms (YOCO_FAST_CHAIN): time /tmp/r6/v7: 2.4728.
- Adam variant with 8 elements per thread and streaming cache hints: 2.39-2.40, worse, dropped.
- Graph captured on a high-priority stream: 2.478. Gate activation and its backward as two Triton kernels: 2.4965. Retention tile sweep: 2.5018 (128,64,8 / 128,64,8 / 64,128,8).
- dW GEMMs on a parallel stream: 2.5045 but the loss drifted (allocator reuse race across streams in capture), dropped.
- Full check /tmp/r6/check1 (promoted to /submission/yoco): PASS, speedup 2.5018, tier N1, trajectory equivalent (largest 0.70 sigma).
- End 17:57 UTC.
2.6 hRound 7: rotary and norm kernels, gradient sum inside the GEMM (dev tree)
- Start 17:57 UTC. Profile /tmp/r7/profile: kernels are 98 % of the update now (gaps 2 %): GEMM 54.5, Adam 13.3, flash 7, retention 5.8, chunk_reduce 4, norm 2.3, swiglu 2.2.
  nvjet GEMM: 132 persistent CTAs (clusters of 2), 384 threads x 168 regs, 220 KB smem: no other kernel can share an SM with them, so side-stream kernels are not free.
- Rotary of q and k in one CUDA kernel (bit-exact with the Triton kernel) + RMSNorm backward split in a dx and a dW kernel: time /tmp/r7/t1: 2.526.
- chunk_reduce unrolled: 2.5342. Adam grid 16 blocks/SM: 2.5397 (4: 2.5315, 32: 2.537); Adam with 8 elements per thread: 2.47-2.49, dropped.
- Custom attention judged not worth it: FA2 backward runs at 190 TFLOP/s, a Triton version needs 40 % more FLOPs (recompute) at a similar rate.
- Owner's dW GEMM accumulates onto the pushed gradient (beta=1, YOCO_FAST_ACC), deferred by ACC_LAG units: lag 1: 2.5152 (stall on the 616 MB output projection push),
  lag 2: 2.5425, lag 4: 2.5591, lag 8: 2.5619 (0.0915 s/update, /tmp/r7/lag8).
- Full check /tmp/r7/check1 (promoted to /submission/yoco): PASS, speedup 2.5675, tier N1, trajectory equivalent (largest 0.74 sigma).
- SM budget experiment (timing only): forward GEMMs through cuBLASLt with CUBLASLT_MATMUL_DESC_SM_COUNT_TARGET=116: 2.5045; plus Adam on 16 or 24 persistent CTAs next to the forward:
  2.3995 / 2.4702. Net loss, direction closed.
- End 18:32 UTC.
3.0 hRound 8: cuDNN fused attention (dev tree)
- Start 18:32 UTC. flash-attention is the largest remaining kernel group besides GEMM and Adam (6.7 ms). pypi.nvidia.com has nvidia-cudnn-frontend 1.32.2 and cuDNN 9.28 (installed in /opt/candidate-env, torch keeps its cuDNN 8).
- fastrt/attention.py: SDPA graphs, GQA, causal, deterministic backward algorithm when torch's deterministic mode is on. Unit test: 0.2-0.4 % from flash-attn (rounding level), same distance to fp32 as flash-attn,
  deterministic backward bitwise reproducible, CUDA graph capture works.
- time /tmp/r8/attn (YOCO_FAST_ATTN): 2.6308 (0.0891 s/update).
- Full check /tmp/r8/check1 (promoted to /submission/yoco): PASS, speedup 2.6279, tier N1, trajectory equivalent (largest 0.77 sigma).
3.5 hRound 9: micro-optimizations and closed directions (dev tree)
- Start 18:44 UTC. Profile /tmp/r8/profile: kernels 84.5 ms of about 88.7 ms measured: GEMM 54, Adam 13.2, retention 6, attention 4, swiglu 2.1, norm 2.2, chunk_sumsq 1.3.
  Per GEMM 650-750 TFLOP/s. Remaining gap about 4 ms: inter-kernel latency, about 900 nodes on the main chain (158 of them cuBLAS memsets).
- Logging without host waits (numbers to pinned memory, meters updated one update late) + chunk_sumsq unrolled: time /tmp/r8/v9: 2.6397. Triton num_stages sweep: 2: 2.6457, 3: 2.6397, 4: 2.6367, kept 2.
- Adam memory patterns (2 elements per thread 2.6133, contiguous range per block 2.6397/2.6367): no gain, Adam stays at about 3.65 TB/s.
- GEMM algorithm search through cuBLASLt (judge profile, heuristic positions 0-5, 321 GEMMs per update): totals 54.6, 53.75, 55.45, 54.8, 58.0, 56.3 ms; best per GEMM 53.1 ms; time of positions 0 and 1: 2.6338 both.
  GEMM is at its floor with these libraries, path removed.
- Retention chunk size 128: 2.6546 (+0.3 %), 64: 2.6017; kernels are load bound, kept 256 (same rounding structure as the reference).
- Full check /tmp/r8/check2 (promoted): PASS, 2.6397, N1, trajectory equivalent (largest 0.77 sigma).
- Norm weight gradient on the side stream: 2.6367, no gain, reverted. One sum-of-squares pass at the end of the backward: 2.622, worse. swiglu with fast intrinsics: 2.6427, kernels unchanged (memory bound), reverted.
- nvidia-smi during a time run: 625-696 W of the 700 W cap, SW power cap active, SM clock 1350-1695 MHz of 1980: the GEMMs are power limited.
- End 19:27 UTC.
- Definitive full check of /submission/yoco with /submission/switches.json, /tmp/r9_check: PASS, speedup 2.6457, N1 (0.0887 s/update).
- All dW GEMMs on a parallel branch through cuBLASLt with an SM budget (inputs held until the join, so no allocator race): no budget 2.6308, 116 SMs 2.6279; see the summary for 100 SMs. Reverted.
- dW branch with 100 SMs: 2.6162. All three settings below the plain 2.64, direction closed.
- End 19:40 UTC.
4.3 hRound 10: last retention and attention checks (dev tree)
- Start 19:40 UTC. Scaled keys straight from the rotary kernel (Triton loads feed the dot directly, bit-identical): _kv_fwd 0.31 -> 0.22 ms, others unchanged; stage sweep with profile: o=3, q=1, k=2.
  time /tmp/r9/ks2: 2.6457; full check /tmp/r9/check1: PASS, 2.6427, N1. Within noise of the submission (2.6457); promoted to /submission/yoco (full check passed).
- dA stored by _dkv_bwd so that _dq_bwd needs one dot per tile: results match, time /tmp/r9/da: 2.6249, slower, reverted.
- cuDNN SDPA plans: two per direction only (engines 9/8 forward, 6/7 backward), not pursued.
- End 19:58 UTC.
4.6 hRound 11: newer Triton for the retention kernels (dev tree)
- Start 19:58 UTC. Why retention runs at about 70 TFLOP/s when cuDNN attention reaches 370: Triton 2.3.1 uses wgmma but with register spills (_dkv_bwd 200, _dq_bwd 40, _out_fwd 30) and many shared-memory
  layout conversions; the kernels are latency bound (one CTA per SM), not ALU or memory bound.
- User kernels do not depend on torch's Triton, and the fast path no longer uses torch.compile: Triton 3.8.0 and 3.4.0 from download.pytorch.org unpacked to /opt/candidate-env/yoco_triton_<v>,
  added to sys.path by yoco_cublas_preload.py when the switch YOCO_TRITON_DIR is set. retention.py got own copies of the recurrence kernels (the upstream signature's typing.Optional is refused by Triton 3) and libdevice names.
- time /tmp/r11/t38: 2.6758 (Triton 3.8), /tmp/r11/t34: 2.6637 (Triton 3.4). Tile and stage sweeps under 3.8 with profile (/tmp/r11/s*, m*): out 64x64 4 warps 3 stages 0.82 ms, dq 64x64 4 warps 2 stages 0.85 ms,
  dkv 64x64 8 warps 3 stages 1.39 ms: 3.1 ms in total, from 4.6 ms with Triton 2.3.1.
- Full check /tmp/r11/check1 (Triton 3.8 through the path switch): PASS, 2.6697, N1, trajectory equivalent (largest 0.16 sigma), all exact items hold.
- The judge's library_versions read 2.3.1 with the path switch (its version probe does not get the switches), so Triton 3.8.0 was installed normally with pip in /opt/candidate-env and the switch removed:
  time /tmp/r11/sub: 2.6789, fingerprint triton 3.8.0. Promoted to /submission/yoco (commit ac281df), __pycache__ no longer tracked.
- TMA descriptors and warp specialization for _dkv_bwd under 3.8: 1.38 / 1.41 ms against 1.39: no gain, reverted.
- End 20:30 UTC.
5.0 hRound 12: rank balance, residency, the power bound
- Start 20:30 UTC. Profile /tmp/r12/profile: kernels 84.7 ms, GEMM 53.8, Adam 13.2, attention 4, retention 3.9, norm 2.2, swiglu 2.1, chunk_sumsq 1.3, rotary 0.7; steady update period in the trace 85 ms.
- NCCL showed 2.1 ms per update, which looked like rank skew; the per-call list shows outliers at the profile start only, 7-14 us per call in steady state. Shares of the parameters for rank 0
  (time): 0.48: 2.6728, 0.50: 2.6789, 0.52: 2.6697, 0.54: 2.6576: balanced is best, change dropped.
- Residency: every nvjet kernel in the heuristic lists (also 128x128) has 384 threads x 168 registers (profile trace, /tmp/r12/gf*): 256 registers left per SM sub-partition, not one warp fits next to it. That explains the failed overlaps.
- Power: 625-696 W of 700 W in steady state, so time is bounded by energy / 700 W. Overlapping Adam (about 500 W) with GEMMs can only reclaim the headroom, 3-4 ms at most, before any GEMM slowdown. Overlap direction closed for good.
- Definitive full check of /submission with pip-installed Triton 3.8.0, /tmp/r12/check_sub: PASS, speedup 2.6819, N1, library_versions triton 3.8.0.
- End 20:50 UTC.
5.4 hRound 13: per-shape GEMM algorithms, host path
- Start 20:50 UTC. cuBLASLt path again, all GEMMs, heuristic positions 0-7 with the judge's profile (/tmp/r12/ga0-7); per-shape table counting 4 us for a memset node: 53.3 ms against 55.3 ms for position 0.
  time: table 2.6881, 2.6881; position 0: 2.6758; plain torch path: 2.6789. Far less than the kernel numbers suggest (GEMM phases are power limited), but repeatable.
- YOCO_FAST_GEMM: fastrt/gemm_table.py + csrc/gemm.cu. Host path: no model.train()/set_num_updates module walks per update (no module of this model uses the update count).
  time /tmp/r13/t1: 2.6912 (0.0871 s/update).
- Full check /tmp/r13/check1: PASS, 2.6912, N1; promoted (commit 53bde3a), switch YOCO_FAST_GEMM on.
- Gate/subln backward as a CUDA kernel: time 2.6912, no change, dropped. Adam with 32 registers (8 blocks per SM): 13.22 ms, no change. Adam in chunks with the weights pushed by the copy engine: 2.6728 (8 chunks), 2.6819 (16), worse than the fused remote stores.
- Accumulation lag: 10: 2.6912, 12: 2.7067 / 2.6943, 14: 2.7005, 16: 2.6974, 20: 2.7005: plateau from 12 on, fixed at 14.
- gate_recurrent.py: the typing.Optional annotation of _fwd_recurrence removed, so that the eager path works with Triton 3 as well (checked: recurrence forward and backward run under 3.8).
- End 21:30 UTC.
5.4 hRound 13 (continued) and round 14: nodes, memset nodes
- Round 13 close: full check /tmp/r13/check2 (commit feae98e): PASS 2.6912, N1, largest endpoint 0.156 sigma; interleaved A/B 0.0870, 0.0872, 0.0868 s/update.
  No gain: dq on a parallel branch 0.0873; CUDA_DEVICE_MAX_CONNECTIONS 1: 2.60-2.62, 32: 2.6943 (= default); L1 no_allocate loads in Adam; rotary tables cached together with
  rmsnorm_dw merged into its partial kernel with a last-block finish: /tmp/r13/nd1, nd2: 2.6789, 2.685, slower: reverted.
- Round 14, start 22:00 UTC. Kernel comparison tests added to the tree (fastrt/tests/test_kernels.py), all pass.
- Profile /tmp/r14/p0: GEMM 53.5 ms, Adam 13.2, attention 4.1, retention 4.2, swiglu 2.1, norm 2.3, sum of squares 1.2.
- Swiglu reading the GEMM output in reverse order (L2 reuse): /tmp/r14/p1 30.5 -> 31 us forward, 50 -> 48 us backward: no gain, reverted.
- Timeline (/tmp/h/tl.py, ms.py on the trace): 180 Memset nodes of 4 bytes per update, cuBLASLt and cuDNN tile counters in their workspace. Each costs 9-14 us between the
  neighbouring kernels instead of 1 us (1.9 ms in the trace).
- YOCO_FAST_HOIST: csrc/graphtool.cu. Every call of the captured graph gets a workspace of its own (GEMM: cursor in an arena sized by the algorithm's need; cuDNN: one tensor per call);
  at the end of the capture the graph is cloned (cuStreamGetCaptureInfo + cuGraphClone), memset nodes with small, pairwise disjoint targets are removed with predecessors
  wired to successors, and the launch runs one clear kernel for all the targets in front of cuGraphLaunch. Instantiation and allocation wait for the first launch (not allowed in capture).
  Graph: 1190 nodes, 180 memsets taken out.
- time /tmp/r14/h3: 2.8172 (0.0833 s/update); full check /tmp/r14/c1: PASS, 2.8037, N1. THESE TWO NUMBERS ARE INVALID, DO NOT USE THEM. The profile afterwards (/tmp/r14/p2) shows the cuDNN backward
  kernel at 2 us instead of 204 us and the forward at 40 us instead of 72 us, and the timed run's loss is wrong: 16.348 at update 20 against 14.312 for the correct code (/tmp/r13/t1). In the timed,
  non-deterministic mode the attention kernels did not do their work. The gates did not see it: gates 2 and 3 run in deterministic mode, with cuDNN's other backward kernel, and gate 1 does not look at the loss.
  The switch was on in /submission for about 10 minutes; /submission/switches.json was put back to v10 (HOIST off) as soon as this was seen. Commit b46c246's message quotes the invalid numbers.

- Non-deterministic validation (new, correctness only, /tmp/h/nd_run.py + nd_cmp.py): the recipe with --log-interval 1 for 30 updates, reference against the fast path: loss and gnorm agree update by update to the third decimal
  once the one-update log lag is taken into account (update 10: 14.592 = 14.592; update 20: 12.824 = 12.824, gnorm 3.839 / 3.84). So the timed path is correct, and the 16.6 / 14.31 against 16.40 / 14.14 in the timing logs is the lag only.
- New rule: every timed run goes through /tmp/h/guard.py (loss and gnorm at updates 10 and 20 against the good run): it flags /tmp/r14/c1 as BAD and passes /tmp/r13/check2 and /tmp/r14/h4, c2. A check PASS alone says nothing about the timed mode's numerics.
- Fix: memsets removed for the GEMM workspaces only (154 of 180), cuDNN back on its shared workspace (commit eb3f614). time /tmp/r14/h4: 2.7161 (0.0864 s/update), guard ok. Full check /tmp/r14/c2: PASS, 2.7192, N1, largest endpoint 0.156 sigma, guard ok. Promoted, v11.
  The true gain of removing the memset nodes is 0.6 ms, not 3.7 ms: the other 3 ms were the attention kernels not running.
- Root cause of the failure (/tmp/h/attn_hoist.py: hoisted cuDNN memsets work on fresh memory in both modes): the per-call cuDNN workspaces were allocated inside the capture, so they could reuse blocks freed by earlier temporaries of the same graph. At replay those temporaries write over the counter after the clear at graph start, and the kernel starts with garbage in its tile counter. Exclusive means no other use of the memory in the whole graph, before as well as after. The GEMM arena is allocated outside the capture and is not affected.

- Arena rule (commit after eb3f614): fastrt_graph_build only takes out memsets inside arenas allocated outside the capture. cuDNN workspaces in such an arena: all 180 memsets out, loss correct, kernel times normal (/tmp/r14/h5, p3: forward 72 us, backward 203 us), which confirms the root cause, but no gain (A/B 2.7036, 2.7098 against 2.7067, 2.7098), so cuDNN keeps its memsets.
- End round 14: 22:55 UTC.
7.7 hRound 15: the power cap, norm finish
- Start 22:55 UTC. Why time says 86.5 ms when the trace period is 83.9 ms and true idle (/tmp/h/idle.py, no kernel on any stream) is only 0.9 ms: the profile run idles the GPU about 20% of the time (slow host), the timed run does not.
- NVML over 300 updates (/tmp/h/nvml.py, telemetry only): 688 W mean of a 700 W cap, SM clock mostly 1470-1560 MHz of 1980, throttle reason SwPowerCap in 99% of the samples, on both GPUs. The update is energy limited, about 59 J per GPU. Consequences: gaps and low-power kernels pay about half of their nominal time (154 memsets: 1.2 ms of gaps, 0.6 ms in time); kernel times from profile overstate GEMM differences, because that run is not power limited.
- GEMM table under sustained load: big tiles for the large shapes time 2.7067, 2.7098 against 2.7067, 2.7067: no change although those kernels are about 5% slower in profile; heuristic position 0 for all: 2.6881, worse. Table unchanged. Recomputing it without the memset penalty changes 3 entries by 1-2 us: not worth it.
- Swiglu in reverse order: no L2 effect. Direct GEMM stores into the peer's gradient slot: rejected on paper, about 0.2 J saved but those GEMMs would become NVLink limited (100 MB per 280 us).
- Host path after a window edge: about 3 ms per 10 updates of many small Python calls (cProfile, /tmp/h/prof_train.py), no single item worth changing.
- YOCO_FAST_NORM3: the row-group sums of all 54 norms in one kernel at the end of the backward, same sums in the same order. A/B on the same tree: on 2.7161, 2.7098; off 2.7098, 2.7036.
- Full check /tmp/r15/c1 of c46c09e: PASS, 2.7129, N1, largest endpoint 0.156 sigma, timed-mode guard ok. Non-deterministic comparison with the reference over 28 updates: max |dloss| 0.001, max |dgnorm| 0.005. Promoted, v12.
- Promotion rule from now on: full check PASS, and timed-mode guard ok, and the non-deterministic comparison with the reference.
- End 23:15 UTC.
8.3 hRound 16: GEMM clusters, row pitch
- Start 23:15 UTC. Energy view: L2 to shared memory traffic is a share of GEMM energy, larger clusters multicast more. 2x4 / 4x2 cluster kernels for the FFN shapes: time 2.7036 (one shape), 2.6943, 2.6789 (three shapes) against 2.7129, 2.7161 (/tmp/r16/c2x4*): worse, closed.
- Adam's inner loop in SASS: about 30 instructions per element, FMUL/FFMA/MUFU only: nothing to save.
- Row pitch: the merged retention projection has 12312 columns, 24624 bytes per row, not a multiple of 32: every other row straddles an L2 sector, in three GEMM shapes (16% of the GEMM time) and in the v and gate reads of the Triton kernels. YOCO_FAST_PITCH: output and gradient allocated with pitch 12352, the real leading dimension goes to cuBLASLt, only the stride changes.
  A/B on the same tree: on 2.7287, 2.7224, 2.7192; off 2.7224, 2.7129, 2.7161: +0.23%, in every pair. Other pitches and flat-buffer offsets checked: all multiples of 128 bytes.
- GEMM table for the 12312 shapes at the new pitch: positions 0-4 of the forward shape 221-236 us against 209 us for the table's entry; positions 5-7 stop with the arena check (split-K variants, large workspaces, rc 300). The scan's other columns were misaligned (the GEMM order changed since the r12 logs) and are not used. Table unchanged.
- Full check /tmp/r16/c1: PASS, 2.7192, N1, largest endpoint 0.156 sigma, timed-mode guard ok, non-deterministic comparison with the reference: max |dloss| 0.001, max |dgnorm| 0.01, kernel tests ok. Promoted, v13.
- End 23:55 UTC.
8.7 hRound 17: conjugate rotary in the retention stores (no gain)
- Start 23:55 UTC. Conjugate rotary applied in the store stage of _dq_bwd and _dkv_bwd (tl.split / tl.join, libdevice fma), straight into the projection gradient: bit for bit equal to the separate kernel at the real shapes, 13 kernels and about 1.3 GB of traffic less per update.
- profile /tmp/r16/rf0, rf: rotary -0.239 ms, _dq_bwd +0.086, _dkv_bwd +0.086: 0.07 ms net. time A/B: on 2.7256, 2.7224, 2.7224; off 2.7287, 2.7224, 2.7192: means 2.7235 and 2.7234, no gain. Reverted, not committed.
- Full check on the real path, /tmp/r16/c_sub (/submission/yoco, /submission/switches.json): PASS, 2.7161, timed-mode guard ok.

- Non-deterministic comparison over 200 updates (the timed path, same horizon as gate 3), /tmp/h/nd/ref200.log, sub200.log, log values in bits per token: window means ref / fast: updates 5-15 15.6666 / 15.6666, 15-25 13.4903 / 13.4902, 25-35 12.0432 / 12.0430, 35-50 11.0262 / 11.0261, 50-100 11.0596 / 11.0598, 100-198 10.2942 / 10.2907.
  gnorm: the two runs separate gradually from about update 60 (first difference above 1%), largest at the loss spike of update 143 (12.7 / 18.1), mean gnorm over updates 100-197 2.798 / 2.805: rounding differences amplified by the spikes, no systematic deviation. One pair of runs, not a statistical result.
- Control for that comparison (a second run of each tree, /tmp/h/nd/ref200b.log, sub200b.log), window 100-198 in units of the frozen sigma: reference against reference -1.11, fast against fast -1.15: the noise floor of a single pair in non-deterministic mode is about 1 sigma. Fast against reference, the four pairs: -1.52, -1.57, -2.67, -0.42, mean -1.57 (lower loss on the fast side), about 2 standard errors with 2 + 2 runs. Window 50-100: +0.04 to +0.64 / -0.42, nothing. No sign of a defect, but with this few runs a small systematic difference of about 0.002 nats in that window cannot be excluded. Gate 3 (deterministic mode) has 0.156 sigma on the same window.
- End 00:30 UTC.
9.4 hRound 18: one-pass norm backward
- Start 00:35 UTC. Profile: rmsnorm3072_bwd 16.5 us and rmsnorm_dw_partial 9.4 us per norm, two passes over the same rows, about 1.4 ms per update.
- YOCO_FAST_NORM4, rmsnorm3072_bwd_dw_kernel: dx unchanged (one warp per row, bit for bit), the weight gradient summed in the same pass. Accumulators in registers spilled (748 bytes), so they live in a shared-memory row per warp, each lane touching only its own elements (no spills, 255 registers). 128 blocks: 28.8 us per call, slower than the pair (4 warps per SM). 256 blocks, 2 rows per warp: 1.10 ms against 1.39 ms per update (/tmp/r18/n4b).
- time, 5 interleaved pairs, order swapped in the last two: on 0.0862, 0.0858, 0.0859, 0.0858, 0.0860 (mean 0.08594); off 0.0861, 0.0866, 0.0862, 0.0862, 0.0863 (mean 0.08628): 0.4%, better in 4 of 5 pairs.
- Full check /tmp/r18/c1: PASS, 2.7383, N1, timed-mode guard ok, non-deterministic comparison with the reference: max |dloss| 0.001, max |dgnorm| 0.006. Promoted, v14 (commit 7dd0a8e).
- The trajectory differences were identical to 16 digits to those of the previous checks although the order of the sum changed, so this was checked before promoting: the gate runs do use the kernel (their graphs lost the same 54 nodes: 1966 -> 1912, timed graph 1137 -> 1083), and the reordered sum changes only 3 of 122880 bf16 weight-gradient elements (2.4e-5, /tmp/h/n4cmp.py), dx bitwise. That rarely reaches a bf16 weight, so nothing visible in 200 updates is expected.
- Note: gate 3 is not sensitive to changes of this size, and neither is its identical result evidence of more than that.
- End 01:05 UTC.
9.8 hRound 19: integer division in kernels
- Start 01:05 UTC. Reading the kernels instead of the profile: swiglu and rotary did a 64-bit integer division per 8 elements in the loop. Shift and mask for the power-of-two width (swiglu), 32-bit index math (rotary, range checked at the launch). Bit for bit the same (kernel tests). profile /tmp/r19/p1: rotary -20%, swiglu forward -6%, swiglu backward unchanged, -0.2 ms in total.
- time, 4 interleaved pairs against the promoted tree: on 0.0855, 0.0856, 0.0858, 0.0859 (mean 0.0857); off 0.0857, 0.0857, 0.0860, 0.0859 (mean 0.08582): 0.14%, equal or better in all 4.
- Also: row-count check in the norm backward (the one-pass kernel would silently skip rows otherwise; cannot happen with the recipe's 2048 rows).
- Fast sigmoid in swiglu (__expf, __frcp_rn), retried because the earlier test still had the divisions: kernel times -0.01 ms, time 2.7224, 2.7319, 2.7415: memory limited, no gain, reverted.
- Full check /tmp/r19/c1 of 1a14e1e: PASS, 2.7383, N1, timed-mode guard ok, non-deterministic comparison with the reference: max |dloss| 0.001, max |dgnorm| 0.006. Promoted.
- End 01:40 UTC.
↑ YOCO