Claude Haiku 5.5 1.70×

Harness: Claude Code · switches: YOCO_FAST_OPT=1 YOCO_FAST_ZERO=1 YOCO_FAST_DDPVIEW=1 YOCO_FAST_LOG=1 YOCO_FAST_PIPE=1 YOCO_PIPE_CHUNKS=8 YOCO_FAST_NORM=1 YOCO_FAST_CE=1 YOCO_FAST_RETPRE=1 YOCO_FAST_NORM2=1 YOCO_FAST_ZB=1 YOCO_FAST_FFN=1 YOCO_FAST_RPROJ=1 YOCO_FAST_EMBSP=1 YOCO_FAST_PPDDP=1

Judged
1.70×
Tier
N1
Self-best
1.71×
Rounds
23
Hours
10.0
API cost
$27.4
Input tokens · 99.6% cached
448 M
Output tokens
1.4 M
Requests
785
Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
✗ still NCCL: broadcast in both directions at once, 8-stage pipeline (1.31× → 1.38×), grads switched to per-parameter all-reduce
Forward / backward CUDA graph
✗ not in the log; instead torch.compile fuses pointwise ops of FFN, projections, retention into blocks (1.585× → 1.637×)
Newer cuBLAS
✗ no library swap; zero-bias addmm makes the bundled library take cuBLASLt, 1.577× → 1.585×
Retention / attention kernel swap
partial: only retention pointwise ops and readout compiled; recurrence kernel and attention not swapped
Other main levers
per-step logging sync removed (~11 ms); compiled RMSNorm 1.38× → 1.51×; sparse embedding grad reduction 1.637× → 1.662×; fused cross-entropy
Raw work log30 sections
0.2 hRound 1 (started 2026-11-06 15:31 UTC)
### Profile of the unchanged reference (judge profile, 10 updates, instrumented; /tmp/r1/profile_ref/profile.md)
Un-instrumented reference timing: 0.2327-0.2332 s/update (same as frozen 0.2332 s).
GPU busy 80% (2.19 s / 2.73 s window). Per update, rank 0, GPU ms (kernel sums; NCCL overlaps compute):
- NCCL 76: fp32 OSS shard broadcasts (Broadcast_RING_LL) 39 ms, not overlapped; bf16 DDP all-reduce 36 ms, mostly overlapped with backward.
- GEMM (sm90 cuBLAS bf16) 65 ms, about 600 TFLOP/s effective. Little headroom.
- memcpy/cast 44: bf16<->fp32 per-parameter copies about 26 ms (240 + 240 kernels per update), fill (zeroing fp32 grads) ~5, DtoD (DDP buckets) ~5.
- elementwise 31: per-parameter fp32 grad multiply (_unscale_grads) ~12 ms; the rest is model-side (RMSNorm fp32 casts, residuals, SwiGLU).
- Adam (apex multi_tensor, AdamFunctor) 14-18 ms; clip-grad L2 norm ~11 ms.
- flash-attn 6, reduce 4, triton/rotary/retention 3, jiterator swiglu 2, loss 2.
Optimizer phase (copies, multiply, norm, Adam, broadcast, param cast) is about half of the update.

### NCCL switches (no code): judge time, s/update, frozen reference 0.2332
- base (empty switches): 0.2327 (speedup 1.002)
- NCCL_PROTO=Simple: 0.2330 (no change)
- NCCL_MIN_NCHANNELS=16: 0.2321 (within noise)
- NCCL_BUFFSIZE=32MB: 0.2332 (no change)
Conclusion: NCCL knobs do not help; the two 6 GB broadcasts run back to back over one NVLink direction each.

### Bitwise checks of multi-tensor replacements (/tmp/r1/tests/bitwise_multitensor.py, GPU lock held)
- apex amp_C multi_tensor_l2norm on bf16 grads == on their fp32 copies (raw norm and torch.norm(stack)): equal.
- multi_tensor_scale bf16->fp32 with scale == copy then mul_(factor): equal for four factors.
- multi_tensor_scale fp32->bf16 with scale 1.0 == copy_: equal.

### Change 1 (switch YOCO_FAST_OPT=1; code yoco/models/fast_optim.py, default off)
Bitwise-intended restructuring of the optimizer phase:
- grad norm on the bf16 gradients (same kernel and chunking as fairseq's fp32 norm);
- fp32 master gradients materialized as bf16 * factor in one multi-tensor pass (replaces 240 copies + 240 mul_);
- fp32 -> bf16 parameter cast in one multi-tensor pass (replaces 240 copy_);
- OSS shard exchange: the two broadcasts issued on two process groups, one per owner rank, so NCCL runs them in opposite directions at once.
Status: timing in progress (see below).
0.2 hRound 1 results (ended 2026-11-06 ~15:55 UTC)
- Change 1 committed as 56b0e04 (switch YOCO_FAST_OPT=1). /submission/switches.json = {"environment": {"YOCO_FAST_OPT": "1"}}.
- judge time (/tmp/r1/time_fastopt): 0.1912 s/update, speedup_vs_frozen_reference 1.2195, beyond reference noise.
- judge check --quick (/tmp/r1/quick_fastopt/check.json): PASS, measured tier N0, 9616 of 9616 recorded tensors bitwise identical, 0 exact failures, structure gate PASS.
- Full check not run yet for this state (gate 1 timing only measured by time).
- Profile of the fast path (/tmp/r1/profile_fastopt/profile.md): GPU busy 177 ms/update (was 219). Copies+scale now ~12 ms (was ~38 with multiply). Adam 14 ms. DDP bucket DtoD copies ~5 ms. Host time ~217 ms/update profiled, so CPU launch overhead is close to binding.
- Round 1 pace: about 25 min of GPU work plus analysis.
0.3 hRound 2 plan
- YOCO_FAST_ZERO=1: skip zero_() of the fp32 master gradients in zero_grad (materialisation rewrites every one before the step; stale-flag guard keeps the no-backward fallback exact). Expected -4.6 ms GPU and ~240 fewer launches.
- YOCO_FAST_DDPVIEW=1: DistributedDataParallel(gradient_as_bucket_view=True) via a subclass patched into fairseq.models.distributed_fairseq_model. Expected -5 ms DtoD bucket copies; the all-reduce values are unchanged.
- Both drafts: /tmp/r2/fast_optim.py (not yet in the tree; the tree is frozen while judge runs use it).
0.3 hRound 2 results (15:50-16:03 UTC, judge `time` on the working tree, frozen reference 0.2332)
| config (switches) | s/update | speedup | window range |
|---|---|---|---|
| YOCO_FAST_OPT=1 (control) | 0.1913 | 1.2189 | 0.0026 |
| + YOCO_FAST_ZERO=1 | 0.1901 | 1.2266 | 0.0056 |
| + YOCO_FAST_DDPVIEW=1 | 0.1879 | 1.2409 | 0.0004 |
| + both | 0.1869 | 1.2476 | 0.0017 |
- ZERO: skips zero_() of the fp32 master gradients in zero_grad (rewritten before each step; stale flag guards the no-backward fallback).
- DDPVIEW: DistributedDataParallel(gradient_as_bucket_view=True) via a subclass patched into fairseq's distributed_fairseq_model (removes DtoD bucket copies).
- Quick bitwise check of the combined switches: see /tmp/r2/quick_both/check.json (result appended below when read).
0.5 hDiagnostics (round 3 prep)
- Sync-debug run (torch.cuda.set_sync_debug_mode('warn'), 8 updates, /tmp/r3/diag.log). Per-update host syncs in the step path:
  trainer.py:919 numer / (sample_size or 1.0) (tensor truthiness), :938 isfinite(grad_norm).all(), :1389/:1393 _check_grad_norms,
  fairseq/distributed/utils.py:669 (all_gather_list in _all_gather_list_sync, every update: the criterion's outputs are not summable),
  trainer.py:1413-1424 and meters.py (_reduce_and_log_stats runs on every update, not only at log_interval).
  Consequence: the CPU cannot queue the next forward/batch while the optimizer and exchange still run on the GPU.
- Measured: forward is roughly CPU-bound (profile: forward CPU ~54 ms vs GPU ~40 ms, profiled); ZERO saved less than its GPU time, consistent with launch-bound forward.
0.5 hRound 3 (16:01-16:25 UTC) status
- Control (same switches as fed013d: OPT+ZERO+DDPVIEW), judge time: 0.1882 s/update, speedup 1.2389 (window range 0.0091). Round-2 measurement of the same set was 0.1869 (1.2476); difference is session noise.
- YOCO_FAST_PIPE (chunked owned-partition Adam, each chunk's shard broadcast as soon as it is updated): NOT in the submission path (flag off, not in switches.json). Three attempts, all hung on the first update:
  1. receives issued before the local Adam; groups created lazily in opposite orders on the two ranks (rank 0 waited in group 1's communicator init while rank 1 waited in group 0's). Killed.
  2. eager ordered creation of the two groups with a warm-up collective per group. Still hung (GPU 0%, ranks spinning).
  3. reproduced in a minimal two-rank script (/tmp/r3/tests/nccl_pattern2.py, not timing): receives queued before the compute-then-send sequence deadlock with or without a host sync; sends-first (compute, send per chunk, then receives) completes. Restructured to sends-first, the real run still stalled (GPU 100% for 2.5 min, no window output; normal run 1.3 min). Killed by PID; root cause not found.
  Decision: drop the pipelined exchange for now (expected gain ~7 ms, about 3-4%); revisit only if cheaper levers run out.
- YOCO_FAST_LOG (no per-update trainer syncs: CPU gloo sum of sample size / OOM count / train time; logging reduction only on logged updates) and YOCO_FAST_MP1 (plain F.linear for single-rank column/row parallel linears): timing chain in progress at 16:25.
- Sync-debug localisation (/tmp/r3/diag.log) identified the per-update syncs listed in the round-2 entry.
0.5 hRound 3 results (judge `time`, frozen reference 0.2332)
| config | s/update | speedup | window range |
|---|---|---|---|
| OPT+ZERO+DDPVIEW (control, this session) | 0.1882 | 1.2389 | 0.0091 |
| + YOCO_FAST_MP1 (plain F.linear for single-rank parallel linears) | 0.1860 | 1.2536 | 0.0025 |
| + YOCO_FAST_LOG (no per-update trainer syncs) | 0.1767 | 1.3196 | 0.0006 |
| + LOG + MP1 | 0.1769 | 1.3181 | 0.0024 |
- LOG: the reference's _reduce_and_log_stats ran on every update and read tensors back to the host (about 47 aten::item per update in the fast-path profile); the all-gather of logging outputs synced every update too. Removing those syncs lets the CPU queue the next batch and forward while the optimizer and exchange still run on the GPU. Gain about 11 ms per update.
- First LOG run errored (ZeroDivisionError): the aggregation decided on the update counter before it moved, the reduce on the counter after it. Fixed (both name the same logged update); re-timed as above.
- MP1 is bitwise-neutral by construction (same F.linear, identity autograd wrappers removed); gain alone about 2 ms, none on top of LOG (within noise). Not included in the submission.
- PIPE (chunked Adam with exchange): parked (see above).

Round 4 (16:30 UTC): committed module rebuilt from fed013d plus the LOG section only (diff: additions only). Time and quick check on that exact module queued.
1.3 hRound 4 results (16:30-16:52 UTC)
- Committed 1cab840: YOCO_FAST_LOG now means "logging reduction only on logged updates". /submission/switches.json = OPT, ZERO, DDPVIEW, LOG.
- judge time on the committed module (/tmp/r4/time_final2): 0.1785 s/update (median 0.1773), speedup 1.3063, window range 0.0028.
- judge check --quick on the committed module (/tmp/r4/quick_final2): PASS, tier N0, 9616 of 9616 bitwise identical, 0 exact failures.
- Negative result that shaped the commit: an earlier LOG version also replaced fairseq's per-update all-gather of the logging outputs with a CPU gloo sum. That changed the training state slightly (quick check: 1672 of 9616 identical, tier N1, one tolerance violation at update 61; the quick check reports PASS, the tier is N1, gate 3 would decide). Diagnosis: keeping the reference aggregation with the reduction skip is bitwise (9616 of 9616). The gloo aggregation was removed. Not fully explained (the sum is the same value; likely a timing-dependent effect of removing the all-gather's ordering point); recorded as an open question.
- Timing of the bitwise diagnostic configuration before cleanup: 0.1781 (1.3092). Final module 0.1785 (1.3063): same within noise.
- Full check on the committed switches: running (started 16:50 UTC, output /tmp/r4/full_check).
- MP1 (plain F.linear for single-rank parallel linears): timed 0.1860 without LOG (vs 0.1882 control), no gain with LOG (0.1769 vs 0.1767). Not in the submission.
- PIPE: parked (hangs, see round 3).
1.3 hFull check on the committed submission (round 4, /tmp/r4/full_check/check.json)
- status PASS, tree /submission/yoco (commit 1cab840), switches /submission/switches.json = OPT, ZERO, DDPVIEW, LOG.
- Gate 2: bitwise, tier N0, 9616 of 9616 recorded tensors identical, 0 exact failures.
- Gate 1: candidate 0.1779 s/update (median 0.1769) against the frozen reference 0.23317 s/update; speedup 1.3107 (reference spread 0.0021).
- This is the accepted state. Later rounds must keep it passing.
1.6 hRound 5 (17:00-17:10 UTC)
- Profile of the accepted state (/tmp/r5/profile_accepted/profile.md): GPU busy 171 ms of about 177 ms per update; NCCL broadcast ~40 ms of kernel time (about 20 ms per owner direction, concurrent), NCCL all-reduce ~39 ms (mostly overlapped with backward), GEMM 65, optimizer multi-tensor ~29 (Adam 14, parameter casts about 12), elementwise ~14, memcpy/cast ~19, flash 6.
- Commit e51a4aa: cleanup. The module body had been duplicated by an earlier rebuild; the single-copy module verifies (quick PASS N0 9616/9616, time 0.1784, speedup 1.307). No behaviour change.
- Root cause of the earlier pipeline hang, found this round: the judge's timer calls torch.cuda.synchronize() in the optimizer's post-step hook (first iteration and window edges, /opt/researchboost/ws/bench/timing.py). The first pipelined version queued NCCL collectives inside the optimizer step, before that hook; a device-wide sync then waits on collectives that the peer cannot post until its own sync returns. Deadlock on both ranks. The reduced two-rank test passed only because it had no such hook.
- Pipeline v2 (YOCO_FAST_PIPE, scratch draft /tmp/r5/fast_optim_pipe2.py, installed in the tree, NOT committed): Adam in chunks with a CUDA event per chunk; after the optimizer step has returned (hooks done), receives are posted at once and each chunk's send after its event, on a side stream. No host waits. Timing test running at 17:07.
1.6 hRound 5 results: pipelined exchange (judge `time`, frozen reference 0.2332)
| config (all with OPT, ZERO, DDPVIEW, LOG) | s/update | speedup | window range |
|---|---|---|---|
| committed state (no pipe), this session | 0.1784 | 1.307 | 0.0002 |
| + PIPE, 2 chunks | 0.1725 | 1.3517 | 0.0002 |
| + PIPE, 4 chunks | 0.1707 | 1.366 | 0.0 |
| + PIPE, 8 chunks | 0.1696 | 1.3748 | 0.0006 |
| + PIPE, 16 chunks | 0.1694 | 1.3764 | 0.0010 |
- Bitwise quick check of PIPE (4 chunks) on the switch set: PASS, N0, 9616 of 9616 identical.
- Chosen: 8 chunks (16 is within noise of 8 and issues twice the collectives). Submission switches set to OPT, ZERO, DDPVIEW, LOG, PIPE, YOCO_PIPE_CHUNKS=8 at 17:15 UTC.
- Full check on that switch set: running (started 17:15, output /tmp/r5/full_pipe). Verdict pending.
- Previous accepted state for reference: commit 1cab840 / e51a4aa (four switches), full check PASS, speedup 1.3107.
1.6 hFull check on the accepted submission (round 5, /tmp/r5/full_pipe/check.json)
- status PASS. Submission: /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE, YOCO_PIPE_CHUNKS=8; tree commit e5369c7.
- Gate 2: bitwise, tier N0, 9616 of 9616 identical, 0 exact failures.
- Gate 1: candidate 0.1695 s/update against the frozen reference 0.23317; speedup 1.3756 (bar 1.02; reference spread 0.0021). Peak memory equal to the reference (61.75 GiB).
- This is the accepted state from round 5 on. Previous accepted: 1cab840/e51a4aa (four switches), speedup 1.3107.
1.9 hRound 7 (17:24-17:36 UTC): fused RMSNorm via torch.compile, switch YOCO_FAST_NORM (N1 expected)
- Change: yoco/models/decoder/rms_norm.py. With YOCO_FAST_NORM=1 the RMSNorm math (fp32 upcast, mean, rsqrt, downcast, weight product) is one torch.compile'd function; the eager path is unchanged when the switch is off. Numerics: inductor's reduction order, so not bitwise (N1); gate 3 decides.
- judge time (accepted switches + YOCO_FAST_NORM=1, /tmp/r6/time_norm): 0.1546 s/update (median 0.1522), speedup 1.5082, window range 0.0008. Log checked: no recompile-limit, cudagraph or overflow signatures, no dynamo/inductor warnings.
- Full check with this switch set (gates 0-3, /tmp/r6/full_norm): running (started 17:26). Not yet the submission switch set: the submission stays on the verified five-switch set until this verdict.
- Full check, round 7 (/tmp/r6/full_norm/check.json): status PASS, stage statistical, tier N1. Gate 2 not bitwise (1267 of 9616 identical; one tensor beyond its diagnostic tolerance, not an exact item). Gate 3: one seed, six endpoints, all within 3 sigma (diffs -0.0003, -0.0010, -0.0012, -0.0004, -0.00001, +0.0021; worst 1.32 sigma). Gate 1 in the same run: candidate 0.1544 s/update, speedup 1.5102.
- Accepted now (commit 4bbd036, /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE, YOCO_PIPE_CHUNKS=8, YOCO_FAST_NORM=1). Tier N1, passed the statistical gate; the previous accepted state (e5369c7, five switches, tier N0) is kept in history.
2.1 hRound 8 profile of the accepted state (4bbd036, /tmp/r8_profile/profile.md, 17:36 UTC)
- GPU busy 158 ms per update; host time per update 157-165 ms, so CPU and GPU are now nearly balanced.
- Per update, rank 0: GEMM ~68 ms (cuBLAS, unchanged); optimizer multi-tensor ~33 (Adam ~14, parameter copies ~12, norm ~3); inductor/triton ~9 (fused RMSNorm and retention kernels); elementwise ~9 (down from ~14); memcpy/cast ~6 (down from ~20); flash ~6; jiterator swiglu ~2; loss ~2; reduce ~1; NCCL exposed tail ~20 (exchange transfer beyond Adam, plus the all-reduce tail).
- Next levers are small (each about 1-3 ms) or need either a rewrite (merged projections: parameter names must stay, so a per-forward concatenation would cost what it saves) or further N1 changes (each needs gate 3 again). Decision: keep the accepted state, verify any further change with the full check before it is committed.
- Round 8 result (no gain, reverted): YOCO_FAST_MP1 re-tested on the accepted seven-switch set: judge time 0.1544 s/update (median 0.1521), speedup 1.5102, window range 0.0011, against 0.1544-0.1546 for the accepted state. No gain now that the logging syncs are gone; the module was restored to HEAD (4bbd036). Not committed.
- Gate-3 recheck of the submission configuration (check --n1, /tmp/r8/n1_recheck): status PASS, six of six endpoints within 3 sigma, worst 1.324 sigma. Identical to the verdict of the full check, so the statistical verdict is reproducible.
- Trace check of the exchange (round 8, /tmp/r8_profile): the shard transfer finishes at the same time as the local Adam (within about 0.1 ms in the sampled update), so the exchange is overlapped and the earlier estimate of a 20 ms exposed tail was too high. Remaining optimizer-phase work is at the bandwidth floor (Adam about 14 ms, parameter copies about 12 ms, norm about 3 ms). GEMM (about 68 ms) runs mostly on 128x128 cuBLAS tiles; no in-process knob to choose a 256-wide tile here.
- Status at 17:42 UTC: accepted 4bbd036 (switches: OPT, ZERO, DDPVIEW, LOG, PIPE with 8 chunks, NORM), full check PASS (speedup 1.5102, tier N1), gate 3 reproduced. Working tree equals HEAD. No judge or experiment processes running.
2.6 hRound 9 (17:45-18:10 UTC): localisation of host and GPU idle time (analysis only, no code change)
- Method: localisation trace of the real entry (torch.profiler with Python stacks, 12 updates, /tmp/r9/rank0_stack.json, 4.2 GB); host syncs and copies attributed to the enclosing Python frame; GPU idle gaps on the main stream attributed to the CPU frame active during each gap (/tmp/r9/gap_cpu*.txt).
- Parameter copies under Module.to / convert: once per run, in Trainer.__init__ (model construction). Startup cost only, not per update.
- Steady state (last 5 updates, after warm-up): GPU idle gaps are small (under 2 ms each). The one large gap is in the logged update (every 10 updates): about 15 ms of CPU-side logging work (fairseq _reduce_and_log_stats / _all_reduce_dict) that the GPU cannot overlap. Amortised about 1.5 ms per update in the judge's windows.
- Ranks are in lockstep: the window times of rank 0 and rank 1 agree to about 1 ms (/tmp/r6/time_norm), so there is no straggler to fix.
- Conclusion: the step is GPU-bound at about 150 ms. GEMM (about 68 ms) and the optimizer (about 33 ms, bandwidth-bound) are at their floors; the remaining non-GEMM work (about 40 ms) is spread over many small kernels. Further gains need custom fused kernels, each estimated at 1-3 percent, with the statistical gate for any numerics change. Submission unchanged: 4bbd036, full check PASS, speedup 1.5102.
2.7 hRound 10 (18:08-18:20 UTC): bf16 scalar-multiply kernels (analysis, no change)
- The AUnaryFunctor<bf16> MulFunctor kernels (about 268 per update, about 4.6 ms per update in the judge's profile) are launched inside torch::autograd::AccumulateGrad during backward, one per bf16 parameter (240 per update). The Python frames are only torch/autograd/graph.py and backward, so the multiply is in torch's C++ DDP reducer. The reference profile has the same count (2680 in 10 updates), so it is not caused by DDPVIEW or by our code.
- Removing it would mean changing DDP's averaging (a SUM comm hook plus a compensating factor). Exact in value, but the bf16 gradients that the judge records would then be sums rather than averages, which is a change to recorded state. Not pursued.
- No code change; submission unchanged (4bbd036, speedup 1.5102).
- Round 10, second check (bf16 add kernels, about 260 per update, about 3.4 ms per update in the judge's profile): mostly gradient accumulation in backward. The largest group is the sum of the five projection gradients that share one input (q, k, v, gate and gt projections of each self-decoder layer, via the model-parallel copy wrapper's backward); the rest are inductor's backward and the matmul/transpose backward nodes. Fusing them would need a bit-exact summation order inside autograd, roughly 2 ms at best. Not pursued. No code change.
- Round 10 conclusion: the remaining per-update cost is GEMM (about 68 ms, cuBLAS), optimizer (about 33 ms, bandwidth floor), gradient accumulation and DDP per-parameter kernels (about 8 ms, tied to autograd and the reducer), and about 40 ms of small kernels across attention, rotary, retention and the compiled norm. Each remaining candidate is 1–3 percent with a bitwise or statistical verification cost. Submission unchanged: 4bbd036, speedup 1.5102.
3.2 hRound 11 (18:16-18:46 UTC): NCCL protocol, DDP bucket multiply (analysis), fused cross-entropy (N1)
- Profile basis: round-8 profile of the accepted state (/tmp/r8_profile/profile.md). Per update: GEMM ~68 ms, optimizer ~33 ms (Adam ~18, scale casts ~12, norm ~3), inductor RMSNorm ~5, flash ~6, bf16 AUnaryFunctor (DDP bucket multiply) ~4.9, bf16 add ~2.7, loss (fp32 log_softmax) ~2.5, BinaryFunctor ~1.6, copies ~4.
- NCCL_PROTO=Simple on the accepted set (judge time, /tmp/r11/time_proto): 0.1550 s/update, speedup 1.5043, against 0.1544-0.1546 accepted. No gain. Not submitted.
- DDP per-parameter multiply: torch's reducer copies each bf16 gradient into its bucket with mul_out(bucket, grad, 1/world) (one kernel per parameter) when no comm hook is registered. A comm hook with ReduceOp.AVG would replace the multiply by a copy of the same size (the grad is not an alias of the bucket after zero_grad sets it to None), so no traffic is saved. Analysis only, not implemented.
- YOCO_FAST_CE=1 (commit 1c93cb8, tier N1): fairseq cross_entropy criterion's compute_loss replaced by a sum-NLL computed from the bf16 logits in one compiled forward, and a compiled backward that writes the bf16 logit gradient directly. Old path: fp32 copy of the logits, log_softmax, nll_loss, and their backward (about 7 GB of traffic per update).
  - Unit check on random logits at 2048x100352 (/tmp/r11/tests/ce_check.py): loss relative difference 6e-8; 0.01% of gradient elements differ by one bf16 step; padded rows zero in both.
  - judge time (/tmp/r11/time_ce): 0.1529 s/update, speedup 1.525.
  - judge check --n1 (/tmp/r11/n1_ce): PASS; six endpoints within 1.03 sigma (-0.18, -0.46, -0.80, -0.36, +0.12, -1.03); verdict equivalent.
- Adam chunk size (apex applier chunk for the Adam launches only, elementwise so no value changes; temporary switch, reverted, not committed): chunk 262144 gave 0.1541 s/update, chunk 16384 gave 0.1535, against 0.1529 at the default 65536. No gain.
- Full check on the submission switch set (OPT, ZERO, DDPVIEW, LOG, PIPE with 8 chunks, NORM, CE), /tmp/r11/full_ce/check.json: status PASS, speedup 1.528 (candidate 0.1526 s/update against the frozen reference 0.23317). Gate 2 not bitwise (tier N1); gate 3 equivalent, worst endpoint 1.03 sigma (limit 3).
- Accepted now: commit 1c93cb8, /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE (8 chunks), NORM, CE. Previous accepted: 4bbd036, speedup 1.5102.
4.0 hRound 12 (19:02-19:40 UTC): compiled retention pointwise (N1), NCCL channel tests, MP1 retest
- Attribution of the elementwise kernels in the round-9 localisation trace (/tmp/r11/analysis/copy_src2.txt, 12 updates): the chunked retention's broadcast products (v * decay, q * exp(g), the scale of k) and their autograd nodes were about 4.5 ms per update (cross_chunk forward 1.44, MulBackward0 1.03, ViewBackward0 1.2, ToCopyBackward0 0.6 ms per update).
- YOCO_FAST_RETPRE=1 (commit 48b1a52, tier N1): chunk_gate_retention runs those pointwise products as one compiled function (same operations and bf16 rounding points; the fp32 cumsum, matmuls and recurrence kernels stay eager).
  - Check against the eager code (/tmp/r11/tests/retpre_check.py, random inputs at model shapes): 1-ulp bf16 flips in about 7% of outputs and gradients (largest deviations exactly one bf16 step at those magnitudes).
  - judge time (/tmp/r11/time_retpre): 0.1508 s/update (speedup 1.546), against 0.1526-0.1529 without it.
  - judge full check (/tmp/r11/full_retpre/check.json): PASS, speedup 1.5472, candidate 0.1507 s/update; gate 2 not bitwise (N1); gate 3 equivalent, worst endpoint loss_u100_200 at 1.93 sigma (limit 3). The endpoint values of the CE set reproduced exactly on a rerun (1.029 both), so the statistical verdict is deterministic.
- Submission: commit 48b1a52 with /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE (8 chunks), NORM, CE, RETPRE.
- NCCL channel limits on the accepted set (judge time): NCCL_MAX_NCHANNELS=8: 0.1628 s/update; =4: 0.2213 (communication is on the critical path); NCCL_MIN_NCHANNELS=32: 0.1512 (no gain over the 0.1504-0.1508 default). Torch 2.3.1 has no environment switch for the NCCL stream priority (checked the library strings).
- MP1 retest (single-rank parallel linears as F.linear, bitwise by construction) on the new set: 0.1572 and 0.1559 s/update, both runs flagged for window interference (range 7.7-8%). Reverted (not committed). Removing ~185 identity autograd nodes per forward did not help; the CPU is not the binding limit for this change.
- Profile of the submission (/tmp/r12/profile, 5 updates, instrumented; not comparable with time): GPU kernels per update: GEMM 66.7 ms, optimizer 33 (Adam 18, scale casts 12, norm 3), triton/inductor 10.4 (RMSNorm fwd is the largest, ~5), elementwise 8.0 (the bf16 DDP bucket multiply ~4.6 and bf16 adds ~2.7), flash 6.1, memcpy/cast 3.2 (down from 6.2), swiglu 2.1. CPU per update, forward ~56 ms and backward ~86 ms in the profiler's ranges, which sum to about the wall time.
- Status: accepted 48b1a52 at speedup 1.5472 (full check). Remaining levers are each about 1% (RMSNorm kernels, rotary inside the compiled pre-function, residual add with the final norm); the DDP bucket multiply, the Adam bandwidth (chunk sizes), and the NCCL settings have been tried or ruled out.
4.1 hRound 13 (19:45-20:05 UTC): rotary inside the compiled retention pre-function (no gain, reverted)
- YOCO_FAST_ROPE (uncommitted, reverted): the interleaved rotary of q and k run inside the compiled pre-function. The rotary alone is bit-identical in eager, compiled and Triton forms (/tmp/r12/tests/rope_piece.py), and so is the q output. But inductor (torch 2.3.1) fuses the bf16 round trips after the rotary with the following multiplies: k, v_dec and q_dec differ in 26-35% of elements by up to 4 ulps (torch 2.3.1 has no emulate_precision_casts option). Gradients differ more.
- judge time (/tmp/r12/time_rope): 0.1507 s/update against 0.1504-0.1508 without it. No gain, so reverted.
- Lesson: the rounding points inside compiled pointwise chains are not preserved by inductor 2.3.1; the RETPRE set (7% one-ulp flips) passed gate 3 at 1.93 sigma; a larger deviation would be a larger gate-3 risk for no gain.
4.6 hRound 14 (19:55-20:30 UTC): RMSNorm backward with a two-level weight-gradient sum (YOCO_FAST_NORM2, N1, committed b34ebbe)
- Attribution (/tmp/r11/analysis/norm_src.txt, round-9 localisation trace): the inductor kernel family triton__0d1d2d3d4de5de ran about 5 ms per update, mostly inside the backward of the compiled graphs and the accumulation of the norm weight gradients (about 80-90 us per launch). Its signature is the column reduction over the 2048 rows of each norm, which inductor's autograd backward runs in a few blocks.
- YOCO_FAST_NORM2=1 (rms_norm.py): the RMSNorm backward written out (fp32 formulas as autograd), and the weight gradient summed over 32 row chunks first, then over the chunks. Forward unchanged; needs YOCO_FAST_NORM.
  - Check against eager autograd (/tmp/r12/tests/norm2_check.py): 1-ulp bf16 differences only (grad_x 66% identical with max 1 ulp; grad_w max 1 ulp).
  - judge time (/tmp/r12/time_norm2): 0.1478 s/update, speedup 1.5776, against 0.1504-0.1508 without it.
  - judge full check (/tmp/r12/full_norm2/check.json): PASS, speedup 1.5765 (candidate 0.1479 s/update), gate 2 not bitwise (N1); gate 3 equivalent, worst endpoint 1.77 sigma.
  - Committed b34ebbe; /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE (8 chunks), NORM, CE, RETPRE, NORM2.
- Profile after the change (/tmp/r13/profile, 5 updates, instrumented): the inductor family dropped from about 10.4 to 6.4 ms per update; GEMM 70; optimizer 33; elementwise 8.1 (bf16 DDP bucket multiply 4.6, bf16 adds 2.8); memcpy/cast 3.6; flash 6.4; swiglu 2.2; reduce 0.8.
- No-gain tests on this set (judge time): YOCO_FAST_VCONT (value tensor contiguous before the retention kernels; 0.1479, reverted).
- Triton multi-tensor Adam (YOCO_FAST_ADAMT, /tmp/r13): numerics agree with apex and with a PyTorch AdamW reference to about 1e-9 absolute, but the judge time was slower in every variant: 0.1528 (4096 block), 0.1524 (alignment hints, 8 warps), 0.1531 (16384 block). Apex's kernel stays. Reverted; the module was removed (not committed).
- Status: accepted b34ebbe at speedup 1.5765. The remaining levers are each at or below about 1%.
- Chunk count re-check on the accepted set (judge time): YOCO_PIPE_CHUNKS 4: 0.1486; 16: 0.1475; 8 (accepted): 0.1478. 16 is within noise of 8; kept 8.
- Weight-gradient chunk sum moved into the compiled RMSNorm backward (judge time on the accepted set, /tmp/r13/time_gwfold): 0.1483 s/update against 0.1478. No gain; reverted (tree equals b34ebbe).
5.0 hRound 15 (20:05-20:45 UTC): copies around the retention chunk function (no gain, reverted)
- Localisation trace of the accepted set (/tmp/r15/rank0_stack.json, 12 updates) with autograd and Python attribution (/tmp/r15/copy_src2.txt): elementwise copy kernels ~0.73 ms per update launched in chunk_gate_retention (the g.float() of the transposed gate), ~0.73 ms per update in the backward of the q/k/v views (3 per self-decoder layer), ~0.45 in compiled backwards, ~0.7 in the RMSNorm backward (input copies), ~0.2 in the retention output reshape. Total about 2.8 ms per update, roughly 2%.
- YOCO_FAST_CONT=1 (inputs of the retention chunk function made contiguous, training path): judge time 0.1475 s/update, against 0.1478 for the accepted set measured in the same session. Within noise, so reverted (tree equals b34ebbe).
- The copies sit at the view boundary of the head-major layout; removing them needs a different layout from the projection on (rotary writing head-major output), which is a larger change for under 1%.
5.2 hRound 16 (20:50-21:30 UTC): GPU idle attribution and a CUDA multi-tensor Adam (no gain, reverted)
- Localisation trace of the accepted set (/tmp/r15/rank0_stack.json) with stacks: the idle gaps on the compute stream are dominated by the C++ autograd engine during backward (about 16 ms per update in the profiled run, where the stack recording inflates CPU time), then Python-level autograd Functions, Triton launches and dynamo wrappers. The judge's unprofiled timing is smaller; the wall-time gains of the last changes were about 60-100% of the GPU time they removed, so the GPU still binds.
- YOCO_FAST_ADAMC=1 (fast_adam_cuda.py, removed; not committed): a CUDA multi-tensor Adam with float4 loads, one launch per tensor list. Agrees with apex to rounding (AdamW p max abs 7e-9; L2 mode m and v exact), misaligned views handled. Judge time (/tmp/r16/time_adamc): 0.1497 s/update against 0.1478 for apex. Both kernels reach the same bandwidth ceiling, so the optimizer phase stays at its floor.
- Reverted (tree equals b34ebbe). The ninja binary resolves from the system PATH (/usr/bin/ninja), and torch 2.3.1's load_inline has no use_ninja argument; both noted for any future extension.
5.4 hRound 17 (20:40-21:10 UTC): zero-bias forward GEMMs (YOCO_FAST_ZB, N1, committed cd32162)
- Hypothesis: the parallel linears call F.linear without a bias, which torch lowers to plain mm. With a bias, torch 2.3.1 lowers addmm to cuBLASLt, which may choose a different forward GEMM kernel. The bias here is a constant zero (not a parameter), so the values are the same.
- YOCO_FAST_ZB=1 (fast_optim.py): the ColumnParallelLinear and RowParallelLinear forwards call F.linear with a zero bias of the output size; identity autograd wrappers unchanged; falls back to fairseq's forward when gather or scatter options are set.
  - judge time (/tmp/r16/time_zb, time_zb2): 0.1469 and 0.1467 s/update (speedup 1.587-1.589), against 0.1477 and 0.1478 without it.
  - judge full check (/tmp/r16/full_zb/check.json): PASS, speedup 1.5851 (candidate 0.1471 s/update); gate 2 not bitwise (N1); gate 3 equivalent, worst endpoint 0.79 sigma.
  - Committed cd32162; /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE (8 chunks), NORM, CE, RETPRE, NORM2, ZB.
- Extension to the backward (YOCO_FAST_ZBB: input and weight gradients as addmm with a zero 1-D bias, custom autograd Function): 0.1463 and 0.1466 s/update, within noise of ZB. Reverted (not committed).
5.9 hRound 18 (21:00-21:30 UTC): compiled blocks for the feed-forward and the retention projections (N1, committed)
- Basis: the zero-bias forward GEMMs (cd32162, 1.5851) left the backward as many small Python-level autograd nodes per layer (identity wrappers of the parallel linears, the custom SwiGLU, the rotary functions). Compiling a whole block makes it one node and one compiled call.
- YOCO_FAST_FFN=1 (commit d79046a, N1): the feed-forward (two column linears with a zero bias, SwiGLU in the kernel's fp32 formula, row linear) is one compiled function on the training path with one model-parallel rank. Judge time (/tmp/r16/time_ffn, time_ffn2): 0.1458 and 0.1456 s/update, against 0.1471 without it. Full check (/tmp/r16/full_ffn): PASS, speedup 1.5971; gate 3 worst endpoint 1.47 sigma.
- YOCO_FAST_RPROJ=1 (commit a74ad9a, N1): the five input projections with the interleaved rotary of q and k and the gate preparation are one compiled function, and the SwiGLU with the output projection another (subln, retention kernels and reshape unchanged). Judge time (/tmp/r16/time_rproj, time_rproj2): 0.1440 and 0.1441 s/update. Full check (/tmp/r16/full_rproj): PASS, speedup 1.6204; gate 3 worst endpoint 0.74 sigma.
- Compiling the cross-attention query projection with its rotary and the output projection (flash attention eager between): judge time 0.1440, no gain over RPROJ alone. Reverted.
- Submission: /submission/switches.json = OPT, ZERO, DDPVIEW, LOG, PIPE (8 chunks), NORM, CE, RETPRE, NORM2, ZB, FFN, RPROJ. Accepted commit a74ad9a at speedup 1.6204.
- Retention chunk preparation (cumsum, decay products, cross decay, scale of k) moved into the compiled projection block, with a chunk function that runs only the matmuls, the recurrence and the inner chunk: committed 242ccbc. Alternating judge time in the same session (committed a74ad9a: 0.1441, 0.1443; working tree: 0.1423, 0.1427, 0.1420, 0.1441 for the first round of that sequence); full check (/tmp/r16/full_prep): PASS, speedup 1.6374, gate 3 worst endpoint 0.74 sigma.
- Subln (no weight) and the output reshape moved into the compiled output block: judge time 0.1425 and 0.1424 against 0.1420-0.1427 for 242ccbc. No gain; reverted.
- Accepted: 242ccbc, speedup 1.6374 (full check).
6.8 hRound 19 (21:30-22:10 UTC): profile of 242ccbc and the remaining structure (no change kept)
- Judge profile of 242ccbc (/tmp/r19/profile, 5 updates, instrumented): GEMM 65 ms per update, optimizer 33, triton (compiled blocks, norms) 9.2, elementwise 7.1 (bf16 DDP bucket multiply ~4.6, bf16 adds ~1.5), flash 6.7, copies 2.7, reduce 0.9. GPU busy 138 ms of 143 ms per update.
- Exposed NCCL (trace /tmp/r15, stacks on, so inflated): about 14 ms inside the backward pass and 13 ms after it. The post-optimizer timeline of one update: all-reduce tail to about 14 ms, norm and materialize 9-22 ms, Adam with broadcasts 19-42 ms, post-exchange cast 46-52 ms (all parameters; it cannot start before the last receive). Phases are HBM-bound: about 110 GB of traffic per update (materialize 21, norm 7, Adam 48, broadcast 14, cast 21) at roughly 3.3 TB/s.
- Tested, no gain (judge time, all on 242ccbc, same session):
  - Residual add of the self-decoder layers folded into the compiled output and feed-forward blocks (YOCO_FAST_RPROJ, DecoderLayer fast path): 0.1431, 0.1422 vs 0.1420-0.1427. Reverted.
  - NCCL_PROTO=^LL: 0.1425 (same). NCCL_PROTO=LL128: 0.1463 (worse). Reverted.
  - Train-flag skip on the DDP wrapper (the trainer's model.train() walk per update; YOCO_FAST_TRAINFLAG): 0.1426, 0.1425. The walk is cheap; the stack-recording profiler inflated it. Reverted.
- Accepted: 242ccbc, speedup 1.6374 (full check). The remaining structure is the bandwidth-bound optimizer phase, the all-reduce tail of the embedding gradient (its bucket is last in backward), and the GEMMs.
- NCCL_BUFFSIZE on 242ccbc (judge time): 16 MB 0.1426 (same), 64 MB 0.1435 (worse). No change kept.
- Considered and not pursued: a fused owned-partition Adam with the bf16 cast (saves about 2 ms of HBM traffic if its kernel reaches apex's bandwidth; the CUDA kernel I wrote was about 1.3% slower than apex's, and the fp32 gradient materialization must stay before the optimizer step for the judge's pre-step record); a DDP comm hook (the bucket copy remains); a custom RMSNorm kernel without the Python autograd layer (CPU overhead only matters if the CPU binds, and the judge profile shows the GPU about 96% busy).
- Accepted: 242ccbc, speedup 1.6374 (full check). Tree clean.
- TORCH_NCCL_AVOID_RECORD_STREAMS=1 on 242ccbc (judge time): 0.1425 and 0.1424 against 0.1420-0.1427. No gain; not kept.
8.1 hRound 20 (22:20-23:10 UTC): sparse reduction of the embedding gradient (YOCO_FAST_EMBSP, exact, committed a61fc55)
- Premise: the embedding weight's gradient is exactly zero in every row that neither rank looked up, and it is the last gradient backward produces, so its dense all-reduce (616 MB bf16) sat on the exposed tail. The LM head (output projection) is a separate weight (share_input_output_embed is False in the recipe), so only the embedding is sparse.
- Implementation (fast_optim.py): DDP's ignore list keeps the embedding out of the buckets (DDP's own mechanism, names set before construction); the weight is broadcast once from rank 0 at construction; a forward pre-hook records the token ids; after backward, FP16Optimizer.all_reduce_grads reduces the embedding gradient over the looked-up rows: all-gather of the ids, unique, pre-division by the world size (as DDP's bucket copy does), one all-reduce of the rows, written back.
- Exactness check (two GPUs, /tmp/r20/tests/embsp_check.py): the sparse result equals the dense bucket all-reduce bit for bit in three trials (union 620-870 of 5000 rows).
- First run errored: the trainer passes fairseq's ModuleProxyWrapper; the marker is on the DDP wrapper it holds (module.module). Found with a driver that printed the calls per rank (/tmp/r20/dbg/driver.py); fixed by walking the proxy's child 'module'.
- judge time (/tmp/r20/time_embsp*): 0.1414, 0.1401, 0.1401 against 0.1426 for the accepted set in the same session.
- judge full check (/tmp/r20/full_embsp/check.json): PASS, speedup 1.6619, candidate 0.1403 s/update; gate 3 endpoints identical to the previous full check to three decimals.
- Post-change timeline (/tmp/r21): the all-reduce tail ends about 8.6 ms after backward (13.8 ms before).
- Tested, no gain: 16 pipeline chunks (0.1406, 0.1401); DDP bucket size 10 MB (0.1408) and 100 MB (0.1403) via a bucket override (reverted).
8.1 hRound 21 (23:10-23:40 UTC): retention read-out and inner chunk in one compiled function (committed d21dfd3)
- The kv product moves into the compiled projection block; the read-out and the inner chunk become one compiled function (_combine_chunk), leaving the recurrence as the only custom kernel per layer.
- judge time (same tree): 0.1396, 0.1399 s/update against 0.1401-0.1414 before. Full check (/tmp/r21/full_comb/check.json): PASS, speedup 1.6715, candidate 0.1395; gate 3 worst endpoint 1.23 sigma.
- Sync-debug run (/tmp/r21/sync.log, six updates): the per-update host syncs are in the trainer (multiply-grads on the sample size, the finite check and the grad-norm consistency check) and in our materialize (factor.item()); the rest are one-time (position tables, the exchange-group probe). Each costs a host round trip of tens of microseconds; not pursued.
- Timeline of one update after backward (/tmp/r21/timeline_stream7.txt): the embedding reduction ops and the norm end by 8.6 ms; materialize 9-16 ms; Adam 17-36 ms overlapping the broadcasts (15.6-41 ms); post-exchange cast 41-47 ms; then a CPU-side gap of about 7 ms (profiled; trainer and task bookkeeping, mostly model.train() and omegaconf reads) before the next forward. The judge timing does not show the train() walk as significant (round 19).
8.2 hRound 22 (23:45-00:05 UTC): per-partition parameter cast queued after each receive (YOCO_FAST_CASTPIPE, no gain, reverted)
- Idea: the post-exchange cast (fp32 -> bf16 of all parameters, about 6 ms) starts only after the last receive. Queue the owned partition's cast after its Adam kernels and each received partition's cast after its own receive (device-side wait), so the cast overlaps the broadcast tail.
- Judge time: 0.1400 and 0.1393 s/update against 0.1395 for the committed state in the same session. No gain; the overlap does not shorten the optimizer phase (the broadcast tail is not HBM-idle as the byte model assumed). Reverted (tree equals d21dfd3).
8.9 hRound 23 (00:00-00:40 UTC): NCCL footprint and inductor autotuning (no gain, reverted)
- NCCL_NTHREADS=256 on d21dfd3 (judge time): 0.1435 s/update, worse than 0.1395.
- NCCL_P2P_USE_CUDA_MEMCPY=1 (collectives via copy engines, to free SMs for the GEMMs): no window after 15 minutes; the run was killed by the 900 s limit of the wrapper and its processes were stopped by PID. Not usable.
- YOCO_FAST_MAXTUNE=1 (inductor max_autotune for the GEMMs inside the compiled blocks, Triton templates against cuBLAS; judge timing limit is 1800 s per timing run): 0.1404 s/update against 0.1395. Triton's GEMM templates do not beat cuBLAS on these shapes. Reverted.
- Tree equals d21dfd3.
9.4 hRound 24 (00:40-01:20 UTC): per-parameter gradient all-reduce without DDP buckets (YOCO_FAST_PPDDP, exact, committed 1a629df)
- Premise (from the round-21 attribution): the 240 per-parameter bf16 multiplies of DDP's reducer (about 4.6 ms per update) are the copies of each gradient into its bucket with the 1/world factor.
- NCCL's average equals DDP's pre-divided sum bit for bit for bf16 at all sizes tested (two GPUs, /tmp/r21/avg_check.py).
- Implementation (fast_optim.py, _install_per_param_ddp): a module wrapper without buckets; the initial state is broadcast from rank 0; a post-accumulate hook on each parameter (except the embedding, which stays with EMBSP) starts an in-place all-reduce with ReduceOp.AVG; all_reduce_grads waits for the handles before the scaling and the norm.
- judge time (/tmp/r21/time_ppddp*): 0.1383, 0.1386 s/update against 0.1395, 0.1400 for the previous commit in the same session.
- judge full check (/tmp/r21/full_ppddp/check.json): PASS, speedup 1.712 (candidate 0.1362 s/update). The six gate-3 endpoints are identical to the previous full check (all printed digits), as expected for an exact change.
- Profile after the change (/tmp/r22/profile, instrumented): the elementwise category dropped from 7.1 to 1.9 ms per update; the all-reduce kernels are 240 per update (37 ms of NCCL time per update, overlapped).
- Exposed NCCL in the localisation trace (/tmp/r22, profiled and inflated): all-reduce 14.8 ms per update inside backward (mostly short segments: the compute stream idles between kernels while the per-parameter calls launch) and 2.8 ms after it; broadcast 5.5 ms.
- Accepted: 1a629df, speedup 1.712 (full check).
- The per-parameter hook calls group.allreduce directly (the call dist.all_reduce makes, without argument handling): committed 8349e37. A/B in alternating runs: committed 0.1379, 0.1367; direct 0.1360, 0.1364 s/update. Full check (/tmp/r22/full_pgdirect): PASS, speedup 1.7107, endpoints identical.
- NCCL_PROTO=Simple on the committed per-parameter state (judge time): 0.1365 s/update, same as the committed state. No change kept.
- NCCL_MAX_NCHANNELS on the committed per-parameter state (judge time): 16 gives 0.1392 (worse), 24 gives 0.1371 (same as the committed 0.1360-0.1379). Default kept.
- Phase ranges of the committed-state profile (/tmp/r22/profile, profiled): backward CPU about 78 ms per update against about 60-65 ms of backward GPU work, so the backward is host-heavy when profiled; the judge timing is not profiled. The 240 per-parameter collectives add host work in backward.
- YOCO_FAST_NRMBW=1 (the clip norm accumulated during backward, in batches of reduced gradients, so only the last few norms remain after backward): judge time 0.1362, 0.1367, 0.1371 against 0.1364, 0.1362 for the committed state in alternating runs. No gain; reverted (tree equals 8349e37). Note: the batched norm is the same sum of squares in another order; it was not pursued further.
↑ YOCO