GPT-6.2 Sol 1.86×

Harness: Codex · switches: YOCO_SHARD_EXCHANGE=1 YOCO_FUSED_GRADIENTS=1 YOCO_FUSED_RMS_NORM=1 YOCO_BATCH_WEIGHT_REFRESH=1 YOCO_OVERLAP_WEIGHT_REFRESH=1 YOCO_DDP_BUCKET_VIEWS=1 YOCO_DDP_BUCKET_MB=64 YOCO_FUSED_CROSS_ENTROPY=1 YOCO_RETENTION_GRAPH=1 YOCO_PIPELINED_SHARDS=24 YOCO_STAGED_REMOTE_REFRESH=1 YOCO_GRAD_SCALE_TILE=16384 YOCO_FUSED_RMS_AFFINE=1 YOCO_LOCAL_IPC_SHARDS=1 YOCO_FUSED_SHARD_ADAM=8192 YOCO_GRAD_NORM_TILE=8192 YOCO_ROTARY_PAIR_TILE=2048 YOCO_IPC_PRIORITY=token YOCO_FUSED_RESIDUAL_RMS=1 YOCO_FUSED_RETENTION_POST=1 YOCO_RETENTION_POST_TILE=token16 YOCO_FUSED_INNER_RETENTION=1 YOCO_INNER_RETENTION_TILE=64 YOCO_CHAIN_RESIDUAL_RMS=1 YOCO_PEER_DDP=1 YOCO_PEER_DDP_NORM=1 YOCO_FUSED_CROSS_PROJECTIONS=1

Judged
1.86×
Tier
N1
Self-best
1.88×
Rounds
45
Hours
10.0
API cost
$20.8
Input tokens · 97.6% cached
102 M
Output tokens
0.6 M
Requests
735
Raw work log45 sections
0.0 hRound 1
- Started 2026-11-18 07:22:35 UTC. Baseline switches: {"environment": {}}.
- Unchanged-reference profile: PID 48658, /tmp/r1/profile.log, output /tmp/r1/profile.
- Profile (5 updates, rank 0): NCCL 29.4% (FP32 weight broadcast 15.0%, BF16 gradient all-reduce 14.2%), GEMM 26.2%, copy/cast 17.8%, elementwise 12.5%, multi-tensor optimizer 7.4%. GPU busy 74%, with overlap. Started with the NCCL protocol switch since both large collectives used LL.
- NCCL Simple timing: PID 50323, /tmp/r1/time-simple.log, output /tmp/r1/time-simple.
- NCCL Simple was within noise: 1.0014× (0.2340 s/update), no promotion.
- Same hotspot, next action: exchange the two existing FP32 OSS buckets with grouped bidirectional NCCL P2P instead of two serial broadcasts. No changes to sharding, optimizer state or dtypes. Scratch tree /tmp/r1/tree; timing PID 51795, /tmp/r1/time-exchange.log, output /tmp/r1/time-exchange.
- P2P timing: 1.0844× (0.2161 s/update), beyond noise, RNG stream accepted. End of round measurements: 2026-11-18 07:28:57 UTC.
- Full check (outside round): PID 53305, /tmp/r1/check.log, /tmp/r1/check/check.json PASS N0 at 1.0839×. All 9,616 gate-2 records were bitwise identical; gate 3 passed. Finished 2026-11-18 07:37:50 UTC and promoted.
0.3 hRound 2
- Aborted the initial profile (PID 58537, /tmp/r2/profile.log) because it launched before the promotion commit completed. No measurements used.
- Restarted 2026-11-18 07:40:06 UTC on promoted commit 87e03f1: PID 60006, /tmp/r2/profile-promoted.log, output /tmp/r2/profile-promoted.

- Promoted profile (5 updates, rank 0, 07:40:22 UTC GPU lock): GEMM 28.8%, NCCL 22.2% (shard exchange 8.5%, gradient reduction 13.8%), copy/cast 19.6%, elementwise 13.7%, multi-tensor optimizer 8.2%. The grad clipping range launched 54 ms and both master-gradient copies and scaling were visible. Compared to R1, the FP32 weight collective dropped from 185 to 95 ms per profile.
- Test Apex FP32-accumulating norm on BF16 inputs plus a multi-tensor BF16-to-FP32 cast and scale. Reuses overwritten master gradient buffers rather than clearing them first; Adam and its norms, state and FP32 interface remain in place. Scratch tree /tmp/r2/tree; timing PID 61570, /tmp/r2/time.log, output /tmp/r2/time.
- Timing finished 2026-11-18 07:44:26 UTC: 1.1799× (0.1986 s/update), a 0.0175 s/update reduction from R1. Quick check PID 63062, /tmp/r2/quick.log, output /tmp/r2/quick.
- Ended 2026-11-18 07:49:36 UTC: /tmp/r2/quick/check.json PASS N0 (9,616/9,616 bitwise; no exact failures). Promoted with timing and N0 evidence; statistical gate pending for this stack.
0.5 hRound 3
- Started 2026-11-18 07:50:15 UTC. Profile of grouped exchange plus fused gradients: PID 65278, /tmp/r3/profile.log, output /tmp/r3/profile.
- Profile (5 updates, rank 0): GEMM 32.0%, NCCL 24.3%, copy/cast 16.6%, multi-tensor optimizer 11.7%, elementwise 7.0%, reduce 1.9%. BF16 Apex norm (14.8 ms) and fused cast+scale (31.0 ms) confirm the R2 fast path. This round targets decoder RMSNorm: casts and elementwise materializations around 335 FP32 mean kernels, plus their backward reductions.
- Try the Apex non-affine RMS norm (FP32 reduction, BF16 output), then keep the BF16 affine multiply as its own op, preserving the rounding boundary. Scratch tree /tmp/r3/tree; timing PID 66782, /tmp/r3/time.log, output /tmp/r3/time.
- Timing finished 2026-11-18 07:54:15 UTC: 1.2784× cumulative (0.1833 s/update), 0.0153 s/update less than R2. Quick check PID 68292, /tmp/r3/quick.log, output /tmp/r3/quick.
- Ended 2026-11-18 07:59:19 UTC: /tmp/r3/quick/check.json PASS with NOT_BITWISE tensor comparison (N1, exact invariants passed). Best submission remains R2 until statistical validation.
- Full check (outside round): PID 70470, /tmp/r3/check.log, output /tmp/r3/check.
- Full check finished 2026-11-18 08:07:38 UTC: /tmp/r3/check/check.json PASS N1 at 1.2798×, all six trajectory endpoints within 0.66σ. Promoted.
0.8 hRound 4
- Started 2026-11-18 08:08:20 UTC. Profile of the full-check-passing stack: PID 75521, /tmp/r4/profile.log, output /tmp/r4/profile.
- Profile (5 updates, rank 0): GEMM 34.3%, NCCL 27.7%, optimizer 12.2%, copy/cast 11.9%, elementwise 4.7%. The largest cast kernel family alone took 54 ms; this round targets FP32 → BF16 weight refresh, retaining FP32 weights and communication precision.
- Trial batches the existing refresh with Apex mixed-type multi-tensor conversion. Scratch tree /tmp/r4/tree; timing PID 77019, /tmp/r4/time-batch.log, output /tmp/r4/time-batch.
- Batched refresh timing: 1.3098× cumulative (0.1789 s/update), 0.0044 s/update less than R3.
- Second switch for the same hotspot: convert the locally owned weights after posting NCCL exchange, then wait and convert the remote shard. FP32 copies are still complete on return from OSS. Timing PID 78524, /tmp/r4/time-overlap.log, output /tmp/r4/time-overlap.
- Overlap finished 2026-11-18 08:13:59 UTC at 1.3307× cumulative (0.1761 s/update), 0.0028 s/update less than batching alone. Quick check with RMSNorm off to isolate N0 changes: PID 80070, /tmp/r4/quick.log, output /tmp/r4/quick.
- Ended 2026-11-18 08:24:57 UTC: /tmp/r4/quick/check.json PASS N0 (9,616/9,616 bitwise in the stand-in configuration). Promoted both weight refresh switches with the existing RMSNorm enabled.
1.1 hRound 5
- Started 2026-11-18 08:26:30 UTC. Profile including batched and overlapped refresh: PID 82228, /tmp/r5/profile.log, output /tmp/r5/profile.
- Profile (5 updates, rank 0): GEMM 34.6%, NCCL 28.8% (BF16 gradient all-reduce 18.2%), optimizer and fused conversions 15.9%, copy/cast 6.7%. D2D copies took 21.5 ms including copies back from gradient buckets. This round targets DDP bucket storage and grouping; all-reduce largely overlaps backward.
- Trial enables DDP gradient bucket views for CUDA BF16. Scratch tree /tmp/r5/tree; timing PID 83767, /tmp/r5/time-views.log, output /tmp/r5/time-views.
- Views: 1.3529× cumulative (0.1732 s/update), down 0.0029 s/update from R4. Also test a 128 MiB cap to group more of the same reductions: PID 85293, /tmp/r5/time-views128.log, output /tmp/r5/time-views128.
- 128 MiB cap: 1.3712× cumulative (0.1709 s/update). Screen 256 MiB as the final point in this grouping sweep: PID 86792, /tmp/r5/time-views256.log, output /tmp/r5/time-views256.
- 256 MiB cap: 1.3704× (0.1710 s/update), essentially tied with 128 MiB; use smaller buckets. Quick N0 stand-in check for the 128 MiB candidate: PID 88286, /tmp/r5/quick.log, output /tmp/r5/quick.
- Ended 2026-11-18 08:39:53 UTC: /tmp/r5/quick/check.json PASS N0 (9,616/9,616 bitwise in the stand-in configuration). Promoted the 128 MiB configuration with RMSNorm enabled.
- Full check (outside round), active submission: PID 90570, /tmp/r5/check.log, output /tmp/r5/check.
- Full check finished 2026-11-18 08:52:59 UTC: /tmp/r5/check/check.json PASS N1 at 1.3720×, with all six trajectory endpoints within 1.22σ. This is the active submission.
1.5 hRound 6
- Started 2026-11-18 08:53:36 UTC. Profile of the full-check-passing DDP stack: PID 95627, /tmp/r6/profile.log, output /tmp/r6/profile.

- Profile (5 updates, rank 0): GEMM 36.5%, NCCL 26.0%, optimizer and conversions 17.3%, elementwise 5.2%, copy/cast 4.7%. Feed-forward forward ranges used 66.3 ms. The 128 MiB DDP cap reduced BF16 all-reduce calls from 595 to 205 over the profile.
- Try grouping FFN input projections in one forward GEMM and one GEMM each for input/weight gradients, with shared backing storage and unchanged individual Parameter identities. Keep the existing SwiGLU first. Scratch tree /tmp/r6/tree; timing PID 97124, /tmp/r6/time.log, output /tmp/r6/time.
- Packed FFN with eager split and SwiGLU: 1.3624× (0.1720 s/update), slower than R5. Next trial uses a Triton SwiGLU that reads the packed layout and writes the two gradients directly into one buffer, avoiding backward concatenation. Timing PID 98623, /tmp/r6/time-swiglu.log, output /tmp/r6/time-swiglu.
- First Triton timing errored before an update: Triton 2.3 exposes FMA under tl.math, not tl. Fixed that call only. Retry PID 99961, /tmp/r6/time-swiglu-fixed.log, output /tmp/r6/time-swiglu-fixed.
- Ended 2026-11-18 09:02:41 UTC: fixed Triton variant was 1.3720× (0.1708 s/update), tied with the checked R5 stack. Dropped both packing variants; no numerics checks or changes to the active tree.
1.8 hRound 7
- Started 2026-11-18 09:08:03 UTC. Reprofile the checked submission: PID 101527, /tmp/r7/profile.log, output /tmp/r7/profile.

- Profile (5 updates, rank 0): GEMM 36.2%, NCCL 26.8%, optimizer/conversions 17.2%, elementwise 5.2%, copy/cast 4.7%. Loss kernels used 8.8 ms, exclusive of the large BF16 ↔ FP32 logits and gradient casts. Target that loss materialization with flash-attn's fused cross-entropy, preserving the criterion's reduction, padding mask and sample size.
- Scratch tree /tmp/r7/tree; timing PID 102993, /tmp/r7/time.log, output /tmp/r7/time.
- Timing ended 2026-11-18 09:11:43 UTC at 1.3915× cumulative (0.1684 s/update), 0.0024 s/update less than the checked R5 stack. Quick check PID 104525, /tmp/r7/quick.log, output /tmp/r7/quick.
- Quick check finished 2026-11-18 09:18:42 UTC: PASS, N1 / NOT_BITWISE, all exact invariants passed. Run the remaining trajectory gate via check --n1: PID 106736, /tmp/r7/n1.log, output /tmp/r7/n1.

- Ended 2026-11-18 09:22:07 UTC: /tmp/r7/n1/check.json PASS; maximum trajectory deviation 0.695σ. Promoted fused CE with gates 0–3 checked separately via time, quick and n1.
2.0 hRound 8
- Started 2026-11-18 09:22:35 UTC. Profile including fused vocabulary cross-entropy: PID 108812, /tmp/r8/profile.log, output /tmp/r8/profile.

- Profile (5 updates, rank 0): GEMM 36.1%, NCCL 28.7%, optimizer/conversions 16.8%, elementwise 5.0%, copy/cast 4.1%. Cross-entropy fwd/bwd total was 2.1 ms (old loss kernels 8.8 ms, with more cast traffic). This round screens a different GEMM backend for the model-parallel linear wrappers at MP=1.
- Try the existing addmm cuBLASLt path with a cached zero epilogue, preserving bias-free projections and FP32 accumulation. Optional custom autograd uses the same backend for dX and dW. Scratch tree /tmp/r8/tree; timing PID 110423, /tmp/r8/time.log, output /tmp/r8/time.
- Forward + backward cuBLASLt: 1.3768× (0.1702 s/update), worse than R7. Forward-only trial PID 111952, /tmp/r8/time-forward.log, output /tmp/r8/time-forward.
- Ended 2026-11-18 09:29:30 UTC: forward-only 1.3776× (0.1701 s/update), also slower. Dropped both variants without correctness runs; no active changes.
- Combined full check of the active fused-CE stack (outside round): PID 113403, /tmp/r7/check.log, output /tmp/r7/check.

- Full check finished 2026-11-18 09:37:58 UTC: /tmp/r7/check/check.json PASS N1 at 1.3874× on the active submission; six trajectory endpoints within 0.96σ.
2.3 hRound 9
- Started 2026-11-18 09:39:00 UTC. Profile of verified fused-CE stack: PID 118630, /tmp/r9/profile.log, output /tmp/r9/profile.

- Profile (5 updates, rank 0): GEMM 35.8%, NCCL 29.3%, optimizer/conversions 16.7%, elementwise 4.9%, copy/cast 4.0%. GPU busy 73% in instrumented window. Self-mixer forward range used 136 ms host vs 56 ms GPU; 2,775 matmul and 780 batched matmul CPU calls overall. This round targets launch overhead in gated retention.
- Capture forward and backward of retention only with PyTorch CUDA Graphs, warming once with synthetic tensors (no random draws) and keeping DDP, loss, and optimizer outside the capture. Scratch tree /tmp/r9/tree; timing PID 120090, /tmp/r9/time.log, output /tmp/r9/time.
- First capture attempt ended 2026-11-18 09:42:17 UTC with a legacy-stream dependency in backward (/tmp/r9/time/time.json, no timing). Retry with detached parameter aliases and one explicit capture stream for warmup and both graphs.
- Detached alias timing: PID 121732, /tmp/r9/time-alias.log, output /tmp/r9/time-alias.
- Detached capture succeeded at 1.4646× (0.1600 s/update). Add independent replay outputs and gradients so subsequent replays cannot overwrite tensors visible to callers; retime that version before any promotion.
- Buffer-safe graph timing: PID 123293, /tmp/r9/time-safe.log, output /tmp/r9/time-safe.
- Round timing ended 2026-11-18 09:52:01 UTC: buffer-safe graph screened at 1.4636× (0.1601 s/update, 0.06% window range), 0.0083 s/update less than R7. Full check also covers the quick gates: PID 124858, /tmp/r9/check.log, output /tmp/r9/check.
- Full check finished 2026-11-18 09:59:44 UTC: /tmp/r9/check/check.json PASS N1 at 1.4618×. Exact items passed; all six trajectory endpoints within 1.11σ. Promoted as 2e570a2.
2.7 hRound 10
- Started 2026-11-18 10:01:09 UTC. Profile of the retention-graph stack: PID 130161, /tmp/r10/profile.log, output /tmp/r10/profile.

- Profile (5 updates, rank 0): GEMM 36.2%, NCCL 27.5% (15.7% BF16 gradient all-reduce and 11.7% FP32 weight exchange), optimizer/conversions 17.1%, elementwise 5.0%, copy/cast 4.7%. GPU busy 79%; self-mixer host time dropped from 136 to 18 ms. The FP32 shard exchange is 103.7 ms across five updates and largely exposed.
- Test a single in-place NCCL all-gather into two padded views of a common FP32 allocation, retaining OSS ownership and overlapped local BF16 conversion. Scratch tree /tmp/r10/tree; timing PID 131683, /tmp/r10/time.log, output /tmp/r10/time.

- In-place all-gather finished 2026-11-18 10:04:35 UTC at 1.4394× (0.1628 s/update), 2.7 ms slower than grouped P2P. Not promoted.
- Same hotspot, next test: divide each FP32 shard into four contiguous regions. Retain one FusedAdam step and the same kernels and state, but post the grouped P2P operations for completed regions as soon as Adam has written them. Scratch tree /tmp/r10/pipeline-tree; timing PID 133173, /tmp/r10/time-pipeline.log, output /tmp/r10/time-pipeline.

- Four-stage pipeline: 1.5206× (0.1541 s/update), 6.0 ms less than R9. Screen eight stages: PID 134721, /tmp/r10/time-pipeline8.log, output /tmp/r10/time-pipeline8.

- Round timing ended 2026-11-18 10:12:05 UTC. Eight stages: 1.5286× (0.1533 s/update; identical window times), selected for full check. A separate two-rank, untimed spot check matched FP32 weights, first and second moments, and Adam step exactly against upstream OSS on uneven shards, including an update with no local gradients on one rank (/tmp/r10/verify_pipeline.log).
- Full check including exact and statistical gates: PID 136567, /tmp/r10/check.log, output /tmp/r10/check. R9 stays active pending the verdict.

- Full check finished 2026-11-18 10:19:53 UTC: /tmp/r10/check/check.json PASS N1 at 1.5266×, all trajectory endpoints within 2.43σ. The two-rank pipeline is bitwise in the spot check (N0 change; stack N1). Promoted as b5a1b55.
3.0 hRound 11
- Started 2026-11-18 10:24:32 UTC. Profile of the eight-stage optimizer pipeline: PID 141787, /tmp/r11/profile.log, output /tmp/r11/profile.

- Profile (5 updates, rank 0): GEMM 34.8%, NCCL 28.3%, optimizer/conversions 18.5%, elementwise 4.8%, copy/cast 4.5%. GPU busy 78%; cross-decoder forward was 63.1 ms host and 73.0 ms GPU, including 29.7 ms of mixer launches. This round targets the remaining decoder launch overhead.
- Reuse detached CUDA graph capture for whole decoder blocks, keeping their autograd inputs and shared cross-attention KV gradients. cross mode retains the existing retention graphs, while all switches to per-layer graphs throughout without nesting captures. Scratch tree /tmp/r11/tree; cross-only timing PID 143359, /tmp/r11/time-cross.log, output /tmp/r11/time-cross.

- Cross-only block graphs timed at 1.5177× (0.1544 s/update, 61.0 GiB allocated), 1.1 ms slower than R10. Test self-decoder layers only, leaving cross-attention eager: PID 144896, /tmp/r11/time-self.log, output /tmp/r11/time-self.

- Self-only block graphs timed at 1.5246× (0.1537 s/update), within noise of R10. Test both decoder halves together: PID 146469, /tmp/r11/time-all.log, output /tmp/r11/time-all.

- Ended 2026-11-18 10:32:49 UTC. All-layer replay: 1.5138× (0.1548 s/update); none of the three modes beat the active stack. No promotion or correctness check needed for the discarded trials.
3.2 hRound 12
- Started 2026-11-18 10:32:49 UTC. Reprofile the unchanged R10 stack: PID 147979, /tmp/r12/profile.log, output /tmp/r12/profile.

- Profile (5 updates, rank 0): GEMM 34.1%, NCCL 29.8%, optimizer/conversions 18.0%, elementwise 4.7%, copy/cast 4.4%. FP32→BF16 casts took 33.8 ms across the profile; the preceding trace showed about 3 ms of remote casts exposed at the end of each update.
- Convert each completed remote FP32 chunk on a separate stream while later chunks are in flight, then join the stream before the optimizer returns. Keep local conversion and all FP32 master weights as before. Scratch tree /tmp/r12/tree; timing PID 149465, /tmp/r12/time.log, output /tmp/r12/time.

- Round timing ended 2026-11-18 10:38:45 UTC: staged remote refresh screened at 1.5478× (0.1514 s/update), 1.9 ms faster than R10. An untimed two-rank spot check matched BF16 model weights as well as master weights and Adam state (/tmp/r12/verify_refresh.log). Full check PID 151322, /tmp/r12/check.log, output /tmp/r12/check. R10 remains active pending the verdict.

- Full check finished 2026-11-18 10:46:23 UTC: /tmp/r12/check/check.json PASS N1 at 1.5437×. All six trajectory endpoints within 0.77σ. Promoted as 2480d15 (N0 change).
3.4 hRound 13
- Started 2026-11-18 10:47:04 UTC. Profile the promoted staged-refresh stack: PID 156499, /tmp/r13/profile.log, output /tmp/r13/profile.

- Profile (5 updates, rank 0): GEMM 34.1%, NCCL 27.2% (gradient reductions 13.7%, P2P 13.4%), optimizer/conversions 20.6%, elementwise 4.7%, copy/cast 4.4%. Gradient all-reduce used the ring LL kernels over 205 calls. This round screens NCCL tuning for that traffic on the NV18 link.
- NCCL 2.30.7 download (PID 154217) stopped after more than six minutes and under 5 MiB transferred; no packages installed. The pinned NCCL source skips NVLS for two local GPUs, so test the existing ring transport first. Trial 32 minimum channels: PID 158051, /tmp/r13/time-channels32.log, output /tmp/r13/time-channels32.

- 32 channels regressed to 1.5316× (0.1530 s/update). NCCL logged version 2.20.5, 32 collective and P2P channels, and zero NVLS channels. Screen ring LL128 on the direct NVLink path: PID 159578, /tmp/r13/time-ll128.log, output /tmp/r13/time-ll128.

- Ended 2026-11-18 10:53:27 UTC. LL128: 1.5366× (0.1525 s/update), also slower. Default topology reported 24 collective and 32 P2P channels. No switch promoted.
3.5 hRound 14
- Started 2026-11-18 10:53:27 UTC. Profile the active R12 stack: PID 161157, /tmp/r14/profile.log, output /tmp/r14/profile.

- Profile (5 updates, rank 0): GEMM 33.0%, NCCL 30.0%, optimizer/conversions 19.7%, elementwise 4.5%, copy/cast 4.1%. The BF16 Apex L2 norm runs after backward and took 14.9 ms across five updates; target its placement on the critical path.
- Compute per-bucket FP32 L2 norms after each normal BF16 all-reduce, aggregate all of them before clipping, and keep the normalized FP32 gradient copy unchanged. Scratch tree /tmp/r14/tree; timing PID 162709, /tmp/r14/time.log, output /tmp/r14/time.

- Ended 2026-11-18 10:57:44 UTC: overlapped norms timed at 1.5079× (0.1554 s/update), 4.0 ms slower than R12. The callback and extra reduction work outweighed any overlap in this implementation. Not promoted.
3.6 hRound 15
- Started 2026-11-18 10:57:44 UTC. Profile the unchanged R12 stack: PID 164258, /tmp/r15/profile.log, output /tmp/r15/profile.

- Profile (5 updates, rank 0): GEMM 33.7%, NCCL 28.1%, optimizer/conversions 20.4%, elementwise 4.7%, copy/cast 4.3%. BF16→FP32 gradient scale was exposed before Adam, 31.1 ms in 825 Apex launches across the five updates.
- Test one Triton launch with a cached tile schedule and live CUDA addresses, emitting the same rounded FP32 product into every master gradient. Scratch tree /tmp/r15/tree; 16K-element tile timing PID 165806, /tmp/r15/time-16k.log, output /tmp/r15/time-16k.

- Initial 16K schedule timed at 1.5226× (0.1539 s/update), slower than Apex. Limit the kernel to aligned lengths so Triton can vectorize the masked loads and stores, and test wider tiles: PID 167361, /tmp/r15/time-64k-vector.log, output /tmp/r15/time-64k-vector.

- Vectorized 64K tile: 1.5612× (0.1501 s/update), 1.3 ms faster than R12. Retest the 16K tile with the same alignment guarantees: PID 168900, /tmp/r15/time-16k-vector.log, output /tmp/r15/time-16k-vector.

- Vectorized 16K tied 64K at 1.5612× (0.1501 s/update) with identical windows. Exact probe found Apex flushes FP32 subnormals; use its rounded FTZ multiply. Updated kernel matched Apex bits for every finite BF16 encoding at all tested scales (/tmp/r15/verify_scale_ftz.log). Retime the exact 16K version: PID 170874, /tmp/r15/time-ftz.log, output /tmp/r15/time-ftz.

- Round timing ended 2026-11-18 11:09:58 UTC: exact FTZ version screened at 1.5601× (0.1502 s/update), 1.2 ms faster than R12. Full check including quick exact invariants: PID 172442, /tmp/r15/check.log, output /tmp/r15/check. R12 is still active pending the verdict.

- Full check finished 2026-11-18 11:17:36 UTC: /tmp/r15/check/check.json PASS N1 at 1.5643×. All six trajectory endpoints within 0.23σ. Promoted the exact scaling kernel as d1bce7c (N0 change).
3.9 hRound 16
- Started 2026-11-18 11:18:31 UTC. Profile of the promoted exact gradient-scale kernel: PID 177692, /tmp/r16/profile.log, output /tmp/r16/profile.

- Profile (5 updates, rank 0): GEMM 34.1%, NCCL 27.8%, optimizer 17.3%, copy/cast 7.1%, elementwise 4.7%. The new gradient scaling kernel ran five times (25.4 ms vs 31.1 ms and 825 launches in R15). RMSNorm forward and input backward kernels totaled 16.1 ms, exclusive of the affine multiplies and weight reductions. Target the extra memory passes around affine RMSNorm.
- Screen native Apex affine RMSNorm on CUDA BF16, with the same weights and epsilon. Its fused backward can keep intermediates in FP32 longer (local precision change), so it needs a full check before promotion. Scratch tree /tmp/r16/tree; timing PID 179235, /tmp/r16/time.log, output /tmp/r16/time.

- Round timing ended 2026-11-18 11:22:05 UTC: affine Apex screened at 1.5822× (0.1481 s/update), 2.1 ms less than R15 screening. Full check is required for the precision change, PID 180772, /tmp/r16/check.log, output /tmp/r16/check; the validated R15 submission stays active.

- Full check finished 2026-11-18 11:29:53 UTC: /tmp/r16/check/check.json PASS, measured N1 at 1.5759×. All six endpoints within 0.94σ. Promoted as b096884.
4.1 hRound 17
- Started 2026-11-18 11:31:01 UTC. Profile of the affine-normalization stack: PID 186034, /tmp/r17/profile.log, output /tmp/r17/profile.

- Profile (5 updates, rank 0): GEMM 33.6%, NCCL 30.7% (FP32 P2P 12.9%, BF16 all-reduce 17.7%), optimizer 16.6%, copy/cast 5.7%, elementwise 4.3%. The exposed FP32 shard exchange uses 126.4 ms over five updates and competes with Adam/casts for SMs. Test copy-engine transport for that same traffic.
- Share FP32 shard allocations via CUDA IPC; coordinate each stage through an NCCL readiness token, copy the exact remote bytes with peer DMA, and acknowledge after all reads finish. An untimed spot check matched both ranks' FP32/BF16 weights and Adam state bitwise (/tmp/r17/verify_ipc.log). Scratch tree /tmp/r17/tree; timing PID 187959, /tmp/r17/time.log, output /tmp/r17/time.
- Source-device mapping screened at 1.6341× (0.1434 s/update). Test mapping the incoming shard on the receiving GPU to avoid a second CUDA context per GPU. The untimed spot check again matched every tensor (/tmp/r17/verify_ipc_local.log); timing PID 189791, /tmp/r17/time-local.log, output /tmp/r17/time-local.

- Ended round timing 2026-11-18 11:40:46 UTC. Receiver-device mapping: 1.6039× (0.1461 s/update), slower than source-device mapping. Restored the exact source-device trial and launched full check: PID 191350, /tmp/r17/check.log, output /tmp/r17/check. R16 remains active.

- Full check finished 2026-11-18 11:48:29 UTC: /tmp/r17/check/check.json PASS N1 at 1.6318×; all six trajectory endpoints within 0.36σ. Transport is exact in the two-rank spot check (N0 change); promoted as fb4843b.
4.5 hRound 18
- Started 2026-11-18 11:49:09 UTC. Profile the promoted CUDA peer-copy pipeline: PID 196595, /tmp/r18/profile.log, output /tmp/r18/profile.
- Profile (5 updates, rank 0): GPU busy 77%; GEMM 35.1%, optimizer 17.4%, NCCL 17.1%, copy/cast 16.2%. Forty peer DMA copies took 94.1 ms; 45 readiness and completion collectives took 10.3 ms. Screen four and sixteen pipelining stages to trade off overlap against synchronization. PID 198183, /tmp/r18/screen.log, outputs /tmp/r18/time-4 and /tmp/r18/time-16. The installed judge has no sweep subcommand, so this uses two sequential time runs.
- Ended 2026-11-18 11:54:56 UTC: four stages timed at 1.6250× (0.1442 s/update); sixteen at 1.6330× (0.1435 s/update). Neither improves on eight stages at 1.6341× (0.1434 s/update). No code or switch promoted; the full-check-passing submission is unchanged.
4.6 hRound 19
- Started 2026-11-18 11:55:22 UTC. New profile of the unchanged eight-stage pipeline: PID 201379, /tmp/r19/profile.log, output /tmp/r19/profile.
- Profile (5 updates, rank 0): GPU busy 77%; GEMM 35.6%, optimizer 17.6%, copy/cast 16.5%, NCCL 15.7%. Adam took 92.4 ms in 440 launches; BF16 conversion took 52.8 ms, roughly half for the local shard. Test updating FP32 weights and moments and writing their local BF16 copies in one vectorized Triton pass per stage, preserving the Apex FP32 rounding and scheduling the existing peer copies afterward. Scratch tree /tmp/r19/tree; verification and timing PID 203031, /tmp/r19/verify_adam.log, /tmp/r19/time-4096.log, output /tmp/r19/time-4096.
- Both ranks matched FP32 weights and Adam moments, BF16 weights, and the Adam step bitwise with and without deterministic algorithms. The 4096-element tile timed at 1.6502× (0.1420 s/update). Screen 2048 and 8192 tiles: PID 205000, /tmp/r19/screen.log, outputs /tmp/r19/time-2048 and /tmp/r19/time-8192.
- Round timing ended 2026-11-18 12:04:49 UTC: 2048 at 1.6502× (0.1420 s/update); 8192 at 1.6525× (0.1418 s/update), selected. Quick check then full check if it passes: PID 208136, /tmp/r19/quick.log and /tmp/r19/check.log, outputs /tmp/r19/quick and /tmp/r19/check. Keep R17 active until the full verdict.
- Quick check finished at 12:09:47 UTC: PASS with no exact failures. Full check finished 2026-11-18 12:17:25 UTC: PASS N1 at 1.6525×. Promoted as 9795b25 with the 8192-element tile.
- The largest trajectory endpoint difference was 0.281σ.
4.9 hRound 20
- Started 2026-11-18 12:18:19 UTC. Profile of the fused Adam stack: PID 215675, /tmp/r20/profile.log, output /tmp/r20/profile.
- Profile (5 updates, rank 0): GPU busy 76%; GEMM 37.5%, NCCL 18.3%, copy/cast 16.7%, optimizer 12.3%. The new AdamW/refresh kernel took 73.6 ms in 40 launches. BF16 DDP prescaling takes 24.3 ms in 1,340 per-parameter launches before the same 205 BF16 NCCL reductions. Test the built-in bucket prescale and NCCL pre-multiply SUM, checking raw BF16 gradient bits before screening each with the judge. Scratch tree /tmp/r20/tree, PID 217303, /tmp/r20/screen.log, outputs /tmp/r20/time-builtin and /tmp/r20/time-premul.
- The first fixture passed no inputs to DDP and raised before backward. Retried with a BF16 scalar input, unchanged candidate: PID 217739, /tmp/r20/screen-retry.log and /tmp/r20/verify-{builtin,premul}-retry.log.
- Ended 2026-11-18 12:23:18 UTC. Built-in bucket scaling matched every BF16 gradient bit in the two-rank fixture but regressed to 1.6150× (0.1451 s/update). The installed PyTorch NCCL binding rejected BF16 PreMulSum (/tmp/r20/verify-premul-retry.log); no timing or dtype change attempted. Neither promoted.
5.0 hRound 21
- Started 2026-11-18 12:23:18 UTC. Profile of the active submission: PID 219752, /tmp/r21/profile.log, output /tmp/r21/profile.
- Profile (5 updates, rank 0): GEMM 38.0%, copy/cast 17.1%, NCCL 16.9%, optimizer 12.6%; GPU busy 65% including an instrumented first-update outlier. Peer DMA takes 89.4 ms; 45 readiness and completion collectives take 12.1 ms. Try publishing IPC stage events in one host rendezvous and using explicit stream waits for each DMA, plus one completion rendezvous before any shard is overwritten. Scratch tree /tmp/r21/tree, verification and timing PID 221321, /tmp/r21/verify_events.log and /tmp/r21/screen.log, output /tmp/r21/time.
- The direct cross-device wait on the imported completion event segfaulted on both ranks (fault diagnostic PID 221731, /tmp/r21/verify_fault.log). Retry using a local-process stream event to relay the completion dependency across devices: PID 222094, /tmp/r21/verify_events_relay.log and /tmp/r21/screen-relay.log.
- Batched event handoff passes the two-rank bitwise fixture with host skew, but times at 1.5995× (0.1465 s/update). Test per-stage posting and DMA-only completion acknowledgement: PID 224019, /tmp/r21/verify_events_stage.log and /tmp/r21/screen-stage.log, output /tmp/r21/time-stage.
- Ended 2026-11-18 12:32:30 UTC. Per-stage handoff also matched bitwise, but regressed to 1.5844× (0.1479 s/update). Keep the validated NCCL token synchronization. No change promoted.
5.2 hRound 22
- Started 2026-11-18 12:33:24 UTC. Profile of the active submission: PID 226004, /tmp/r22/profile.log, output /tmp/r22/profile.
- Profile (5 updates, rank 0): GPU busy 71% including the first-update profiling overhead; GEMM 38.0%, copy/cast 17.1%, NCCL 16.9%, optimizer 12.5%. Peer DMA took 89.4 ms and the exact remote BF16 conversion took 18.0 ms in 440 launches. Test a receiver-side Triton pass that reads each ready IPC-mapped FP32 span once and writes both the exact FP32 replica and the BF16 model weights. The initial mapping needed an integer device index, rather than a torch.device (/tmp/r22/verify_peer_copy.log). Scratch tree /tmp/r22/tree, retry and timing PID 227962, /tmp/r22/verify_peer_copy_retry.log and /tmp/r22/screen-retry.log, output /tmp/r22/time-2048.
- Installed Triton rejects the .cv cache modifier. Retry with system-scope volatile peer loads: PID 228326, /tmp/r22/verify_peer_copy_volatile.log and /tmp/r22/screen-volatile.log.
- Direct peer reads matched FP32 weights, BF16 mirrors, and moments on both ranks, but regressed to 1.5306× (0.1531 s/update). Keep source-mapped DMA and test vectorized single-launch remote casts after each stage instead: PID 230280, /tmp/r22/verify_remote_cast.log, /tmp/r22/screen-cast.log, output /tmp/r22/time-cast16384.
- Ended 2026-11-18 12:43:26 UTC. The vectorized local cast matched bitwise and timed at 1.6560× (0.1415 s/update), only 0.2% faster than the fully checked stack and within reference noise. Do not promote.
5.4 hRound 23
- Started 2026-11-18 12:43:52 UTC. New profile of the active submission: PID 232225, /tmp/r23/profile.log, output /tmp/r23/profile.
- Profile (5 updates, rank 0): GPU busy 75%; GEMM 38.0%, NCCL 17.3%, copy/cast 17.0%, optimizer 12.4%. Reading BF16 gradients for their FP32 norm took 14.5 ms in 825 Apex launches. Compare larger 131072/262144 Apex chunks and a single Triton BF16 square-sum pass with a separate FP32 partial reduction. Clipping and the FP32 gradient copy stay unchanged. Scratch tree /tmp/r23/tree, spot check and timing PID 233831, /tmp/r23/verify_norm.log and /tmp/r23/screen.log, outputs /tmp/r23/time-tile8192, /tmp/r23/time-chunk131072, /tmp/r23/time-chunk262144.
- Ended round timing 2026-11-18 12:50:45 UTC. Triton tile 8192: 1.6738× (0.1400 s/update); both larger Apex chunks: 1.6607× (0.1411 s/update). The Triton fixture verifies deterministic repeatability, close FP32 reduction, and NaN/Inf propagation. Remove the trial chunk override and validate tile 8192: PID 238697, /tmp/r23/quick.log and /tmp/r23/check.log, outputs /tmp/r23/quick and /tmp/r23/check. R19 remains active until full verdict.
- Quick check finished 12:56:01 UTC: PASS, no exact failures. Full check finished 2026-11-18 13:03:55 UTC: PASS N1 at 1.6714×, maximum trajectory endpoint difference 1.040σ. Promoted as 1d72e8c.
5.7 hRound 24
- Started 2026-11-18 13:04:29 UTC. Profile after promoting the batched norm: PID 246308, /tmp/r24/profile.log, output /tmp/r24/profile.
- Profile (5 updates, rank 0): GPU busy 76%; GEMM 37.6%, NCCL 18.7%, copy/cast 16.9%, optimizer 10.7%. RoPE used 7.2 ms in 400 launches with four token rows per CTA on the interleaved path. Test sixteen and thirty-two token rows without changing the arithmetic or allocation layout. Scratch tree /tmp/r24/tree, verification and timing PID 247937, /tmp/r24/screen.log, outputs /tmp/r24/time-16 and /tmp/r24/time-32.
- Both tile sizes matched bitwise but regressed: 16 at 1.6514× (0.1419 s/update), 32 at 1.6228× (0.1444 s/update). Try one packed load/store per BF16 rotary pair while retaining the original FP32 FMA and rounding: PID 251539, /tmp/r24/verify-pair2048.log, /tmp/r24/screen-pair.log, output /tmp/r24/time-pair2048.
- Ended round timing 2026-11-18 13:12:28 UTC: packed pairs match bitwise on both contiguous and head-major forward/backward in the fixture and time at 1.6846× (0.1391 s/update). Drop the slower tile-only switch and run the full check (includes the exact gate): PID 253377, /tmp/r24/check.log, output /tmp/r24/check. R23 remains active pending verdict.
- Full check finished 2026-11-18 13:20:55 UTC: PASS N1 at 1.6822×; exact invariants passed and all trajectory endpoints were within 1.914σ. Promoted as 5dce5bb.
6.0 hRound 25
- Started 2026-11-18 13:21:48 UTC. Profile of the active packed RoPE stack: PID 258940, /tmp/r25/profile.log, output /tmp/r25/profile.
- Profile (5 updates, rank 0): GPU busy 77%; GEMM 38.0%, NCCL 18.3%, copy/cast 17.0%, optimizer 10.8%. Packed RoPE drops from 7.2 to 2.7 ms over five updates. Peer DMA (89.4 ms) is now longer than its concurrent fused AdamW producer (73.7 ms); stage readiness/completion takes 13.5 ms. Retune the stage count with the faster Adam kernel, screening 12 and 16 before considering uneven stages. PID 260618, /tmp/r25/screen.log, outputs /tmp/r25/time-12 and /tmp/r25/time-16.
- Twelve stages timed at 1.6980× (0.1380 s/update); sixteen at 1.6993× (0.1379 s/update). In the profile's last update, the first local stage completes after 1.560 ms, but the first DMA starts at 3.173 ms. Screen uniform 24 and a geometric eight-stage layout (smaller leading stage; each stage 1.25× the previous size), without changing the FP32 values. PID 263834, /tmp/r25/screen-more.log and /tmp/r25/verify_ramp.log, outputs /tmp/r25/time-24 and /tmp/r25/time-ramp8.
- Uniform 24 reached 1.7092× (0.1371 s/update), while the bitwise-verified ramped layout regressed to 1.6726× (0.1401 s/update). Screen 32 and 48 uniform stages to bound the readiness/launch cost: PID 267388, /tmp/r25/screen-fine.log, outputs /tmp/r25/time-32 and /tmp/r25/time-48.
- Ended round timing 2026-11-18 13:32:51 UTC. Thirty-two stages reached 1.7079× (0.1372 s/update); 48 reached 1.7017× (0.1377 s/update). Select standard 24 stages, no ramp code. Full check: PID 270798, /tmp/r25/check.log, output /tmp/r25/check, tree /tmp/r25/uniform-tree. R24 remains active until verdict.
- Full check finished 2026-11-18 13:41:01 UTC: PASS N1 at 1.7092×; all exact checks passed and the largest trajectory deviation was 1.209σ. Promoted the 24-stage setup as b4e178d.
6.3 hRound 26
- Started 2026-11-18 13:41:44 UTC. Profile of the active 24-stage stack: PID 276324, /tmp/r26/profile.log, output /tmp/r26/profile.
- Profile (5 updates, rank 0): GPU busy 76%; GEMM 37.3%, NCCL 19.2%, copy/cast 16.9%, optimizer 11.3%. The model-parallel identity autograd wrappers took 9.0 ms of host time in 1,070 calls at degree one. Try direct F.linear with identical weights and Megatron initialization, leaving actual partitioned projections unchanged. Scratch tree /tmp/r26/tree, fixture and timing PID 277980, /tmp/r26/verify.log and /tmp/r26/screen.log, output /tmp/r26/time.
- Ended 2026-11-18 13:45:10 UTC. The fixture matched initialization RNG, state, outputs and gradients bitwise, but judge time tied R25 at 1.7092× (0.1371 s/update). No change promoted.
6.4 hRound 27
- Started 2026-11-18 13:45:58 UTC. Profile of the validated 24-stage stack: PID 279855, /tmp/r27/profile.log, output /tmp/r27/profile.
- Profile (5 updates, rank 0): GPU busy 74%; GEMM 37.9%, NCCL 17.8%, copy/cast 17.1%, optimizer 11.5%. Peer DMA takes 90.2 ms, fused AdamW 77.0 ms; 125 readiness/completion collectives use 21.8 ms. Test a dedicated high-priority NCCL group for IPC tokens, and separately high-priority copy streams, preserving the same FP32 bytes and dependencies. Scratch tree /tmp/r27/tree, PID 281559, /tmp/r27/screen.log, outputs /tmp/r27/time-token and /tmp/r27/time-copy.
- Ended round timing 2026-11-18 13:51:02 UTC: both two-rank fixtures matched all weights and moments bitwise. Token priority reached 1.7332× (0.1352 s/update); copy priority regressed to 1.7005× (0.1378 s/update). Drop copy priority and run the full check for the token group (includes exact gate): PID 285571, /tmp/r27/check.log, output /tmp/r27/check. R25 remains active pending verdict.

- Full check finished 2026-11-18 13:59:25 UTC: PASS N1 at 1.7345×, with all exact checks passed and the largest trajectory deviation 0.695σ. Promoted the IPC token group as c2a71bd (N0 change).
6.6 hRound 28
- Started 2026-11-18 14:00:22 UTC. Profile of the promoted high-priority IPC token stack: PID 291286, /tmp/r28/profile.log, output /tmp/r28/profile.

- Profile (5 updates, rank 0): GPU busy 76%; GEMM 36.7%, NCCL 19.9%, copy/cast 16.7%, optimizer 11.4%. FFN input-gradient GEMMs alone account for roughly 70 ms (8% of kernel time); native backend and packed projections had no gain in earlier rounds. Try grouped 128×128×64 BF16 Triton tiles with FP32 accumulators, comparing FFN input-gradient-only and all FFN projections. Scratch tree /tmp/r28/tree; numerical fixture and judge time PID 293007, /tmp/r28/verify.log and /tmp/r28/screen.log, outputs /tmp/r28/time-dgrad and /tmp/r28/time-all.

- The fixture passed for 128×128. Input-gradient-only reached 1.7294× (0.1355 s/update), essentially tied with R27, and all FFN GEMMs regressed to 1.6655× (0.1407 s/update). Test 64×128 and 64×256 for input gradients only: PID 296698, /tmp/r28/screen-tiles.log, outputs /tmp/r28/time-dgrad-64x128 and /tmp/r28/time-dgrad-64x256.

- Ended 2026-11-18 14:10:28 UTC. Both smaller tiles also passed the numerical fixture, but 64×128 reached only 1.6895× (0.1387 s/update) and 64×256 reached 1.6907× (0.1386 s/update). No quick/full check or promotion without a timing gain. Scratch commit ef5bc33.
6.8 hRound 29
- Started 2026-11-18 14:11:41 UTC. Reprofile the validated R27 stack: PID 300455, /tmp/r29/profile.log, output /tmp/r29/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 37.8%, NCCL 18.0%, copy/cast 17.1%, optimizer 11.6%. Apex RMSNorm apply, input backward, and affine-gradient partials total 24.0 ms before separate residual additions (all BF16 add kernels total 13.1 ms). Screen the mixer residual-add plus FFN affine RMSNorm as one deterministic BF16/FP32 kernel pair with a fused weight-gradient partial and the original BF16 residual rounding. PID 302160, /tmp/r29/verify.log and /tmp/r29/screen.log, output /tmp/r29/time, scratch /tmp/r29/tree.

- First fixture stopped before timing: Triton 2.3.1 exposes rsqrt via tl.math. Corrected the API call and restarted the same fixture/timing, PID 302346, /tmp/r29/verify-fixed.log and /tmp/r29/screen-fixed.log.

- The API-fixed fixture rejected forward (relative L2 0.00281). Apex at the pinned commit rounds the normalized value to BF16 before multiplying by gamma. Preserved that rounding in the fused forward and kept the fixture tolerance unchanged. Retry PID 302521, /tmp/r29/verify-rounded.log and /tmp/r29/screen-rounded.log, output /tmp/r29/time-rounded.

- Ended timing 2026-11-18 14:19:16 UTC. Corrected fixture passed in deterministic and production settings, including exact BF16 residuals, and timing reached 1.7513× (0.1338 s/update), 1.4 ms faster than R27 screening. Full check (includes exact records and the production trajectory): PID 304400, /tmp/r29/check.log, output /tmp/r29/check.

- Full check finished 2026-11-18 14:28:00 UTC: PASS N1 at 1.7448×, all exact checks passed and all six endpoints within 0.481σ. Promoted as 0e767aa.
7.1 hRound 30
- Started 2026-11-18 14:28:35 UTC. Profile of the promoted mixer residual/RMSNorm stack: PID 310244, /tmp/r30/profile.log, output /tmp/r30/profile.

- Profile (5 updates, rank 0): GPU busy 76%; GEMM 37.3%, NCCL 19.6%, copy/cast 16.8%, optimizer 11.3%. The standalone SwiGLU kernels total 10.8 ms, RMSNorm apply 5.1 ms, layout-copy kernels 12.7 ms across the model. Fuse retention's head-wise RMSNorm, head-to-token transposition, and output gate in both directions. Keep BF16 rounding before the SwiGLU and before propagating its gradient through normalization. Scratch /tmp/r30/tree; numerical fixture and timing PID 312008, logs /tmp/r30/verify.log, /tmp/r30/screen.log, output /tmp/r30/time.

- The base 8-row head-major kernel passes the fixture but screened at 1.7487× (0.1340 s/update), tied with the active stack within noise. Screen 32-row head-major and 16-row token-major scheduling with identical arithmetic. PID 313982, /tmp/r30/screen-tiles.log, outputs /tmp/r30/time-head32 and /tmp/r30/time-token16.

- Ended timing 2026-11-18 14:39:53 UTC. Both layouts passed the fixture. Head32: 1.7672× (0.1326 s/update). Token16: 1.7685× (0.1325 s/update), 1.3 ms faster than the R29 screen. Full check of Token16: PID 317789, /tmp/r30/check.log, output /tmp/r30/check. Keep R29 active until the verdict.

- Full check finished 2026-11-18 14:48:04 UTC: PASS N1 at 1.7632× (0.1329 s/update vs 0.23433 frozen), all exact checks passed, all endpoints within 1.958σ. Promoted as 8d1abc5.
7.4 hRound 31
- Started 2026-11-18 14:48:56 UTC. Profile of the validated token-major retention stack: PID 323772, /tmp/r31/profile.log, output /tmp/r31/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 38.4%, NCCL 18.1%, copy/cast 17.1%, optimizer 11.8%. The 205 DDP BF16 Ring LL collectives use 137.6 ms, with 18.1 ms not overlapping other kernels. Test whether capping channels below the defaults reduces competition with backward kernels. Same validated tree; only NCCL_MAX_NCHANNELS changes. Screen 16 then 8, PID 325565, /tmp/r31/screen.log, outputs /tmp/r31/time-channels16 and /tmp/r31/time-channels8.

- Ended 2026-11-18 14:53:18 UTC. Sixteen channels reached 1.7592× (0.1332 s/update), tied or slower than the active baseline. Eight regressed to 1.7142× (0.1367 s/update). Both timing windows were stable, so keep the defaults. No check or code change.
7.5 hRound 32
- Started 2026-11-18 14:53:51 UTC. Reprofile the unchanged validated R30 stack: PID 328970, /tmp/r32/profile.log, output /tmp/r32/profile.

- Profile (5 updates, rank 0): GPU busy 77%; GEMM 38.4%, NCCL 18.3%, copy/cast 17.1%, optimizer 11.8%. The 780 small CUTLASS BMM launches in retention consume 14.1 ms, before score/decay materialization. Stream causal tiles for the inner-chunk half of this work, keeping BF16 rounding at both matmuls and decay products and computing both prefix-gradient contributions in FP32. No atomics. Test 32- and 64-token tiles, scratch /tmp/r32/tree, PID 330752, /tmp/r32/verify-32.log, /tmp/r32/verify-64.log, /tmp/r32/screen.log, outputs /tmp/r32/time-tile32 and /tmp/r32/time-tile64.

- Fixture setup failed before invoking the kernel: PyTorch 2.3.1 rejects CUDA cumsum under strict deterministic algorithms. Generate the synthetic FP32 prefix inputs on CPU instead; no training code change. Retry PID 330916, /tmp/r32/verify-cpu-32.log, /tmp/r32/verify-cpu-64.log, /tmp/r32/screen-cpu.log.

- Triton 2.3.1 requires unwrapping the constexpr stride tuple before subscripting. Fixed that compile error and restarted the same fixtures and timings, PID 331119, /tmp/r32/verify-stride-32.log, /tmp/r32/verify-stride-64.log, /tmp/r32/screen-stride.log.

- Ended timing 2026-11-18 15:03:13 UTC. Both layouts passed deterministic and production fixtures on real chunk/width/head dimensions. Tile32: 1.7739× (0.1321 s/update); Tile64: 1.7766× (0.1319 s/update). Run the full check for Tile64, PID 334420, /tmp/r32/check.log, output /tmp/r32/check. R30 stays active until verdict.

- Full check finished 2026-11-18 15:11:30 UTC: PASS N1 at 1.7779× (0.1318 s/update vs 0.23433 frozen), all exact checks passed, and all six endpoints within 0.592σ. Promoted Tile64 as c79134e.
7.8 hRound 33
- Started 2026-11-18 15:11:59 UTC. Profile of the promoted causal retention tiles: PID 339526, /tmp/r33/profile.log, output /tmp/r33/profile.

- Profile (5 updates, rank 0): GPU busy 77%; GEMM 36.7%, NCCL 20.0%, copy/cast 16.8%, optimizer 11.5%. Standard BF16 adds take 9.4 ms, and the remaining separate Apex RMSNorm apply/input-backward/affine partials take 8.9 ms. Extend R29's checked residual/RMSNorm fusion to the 24 boundaries between consecutive self or cross layers. Keep separate decoder-stage outputs, checkpointed/incremental paths, and all parameter names. Scratch /tmp/r33/tree; fixture and timing PID 340954, /tmp/r33/verify.log, /tmp/r33/screen.log, output /tmp/r33/time.

- Ended timing 2026-11-18 15:15:38 UTC. Three-layer self and cross stacks passed the output and all-gradient fixture. Judge screened at 1.7901× (0.1309 s/update), 1.0 ms faster than the R32 screening run. Full check PID 342643, /tmp/r33/check.log, output /tmp/r33/check. R32 remains active until verdict.

- Full check finished 2026-11-18 15:23:41 UTC: PASS N1 at 1.7915× (0.1308 s/update vs 0.23433 frozen), all exact checks passed and all six endpoints within 0.438σ. Promoted as caa659b.
8.0 hRound 34
- Started 2026-11-18 15:24:14 UTC. Profile of the promoted decoder-layer chaining stack: PID 347746, /tmp/r34/profile.log, output /tmp/r34/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 38.2%, NCCL 17.8%, copy/cast 17.3%, optimizer 11.9%. Fused shard Adam takes 77.8 ms in 120 launches; the four-warp kernel uses 255 registers per thread, estimated occupancy 13%. Screen 8 and 16 warps at the same 8192-element tile and 24 stages. No arithmetic changes. Scratch /tmp/r34/tree; two-rank fixture and judge times PID 349187, /tmp/r34/screen.log, /tmp/r34/verify-8.log, /tmp/r34/verify-16.log, outputs /tmp/r34/time-warps8 and /tmp/r34/time-warps16.

- Ended 2026-11-18 15:29:01 UTC. Both two-rank fixtures matched master weights, BF16 weights, moments, and step exactly. Eight warps reached 1.7901× (0.1309 s/update) and sixteen reached 1.7915× (0.1308 s/update), tied with the active baseline. Keep the default launch. Scratch commit c98eee2.
8.1 hRound 35
- Started 2026-11-18 15:30:24 UTC. Fresh profile of the validated R33 stack: PID 352737, /tmp/r35/profile.log, output /tmp/r35/profile.

- Profile (5 updates, rank 0): GPU busy 76%; GEMM 38.1%, NCCL 18.2%, copy/cast 17.3%, optimizer 11.9%. BF16 DDP all-reduce alone uses 137.0 ms across 205 calls. Try peer DMA on stable rebuilt DDP buckets, with high-priority readiness/completion tokens and one BF16 predivide-plus-add pass. Keep the initial native reduction while bucket storage can still rebuild. Scratch /tmp/r35/tree; exact BF16 and two-rank DDP fixture + judge timing PID 354166, logs /tmp/r35/verify.log and /tmp/r35/screen.log, output /tmp/r35/time.

- Ended timing 2026-11-18 15:35:06 UTC. The two-rank fixture matched predivision and sum across all finite BF16 encodings and matched all gradients from multiple rebuilt buckets bitwise. Judge screened at 1.8109× (0.1294 s/update), 1.5 ms faster than R33 screening. Full check PID 355984, /tmp/r35/check.log, output /tmp/r35/check. R33 stays active until verdict.

- Full check finished 2026-11-18 15:43:37 UTC: PASS N1 at 1.8109× (0.1294 s/update vs 0.23433 frozen), all exact checks passed, all six endpoints within 1.222σ. Promoted as 34dbd2e (N0 transport change).
8.4 hRound 36
- Started 2026-11-18 15:44:24 UTC. Profile of the promoted peer-DMA DDP stack: PID 361207, /tmp/r36/profile.log, output /tmp/r36/profile.

- Profile (5 updates, rank 0): GPU busy 74%; GEMM 34.3%, copy/cast 29.4%, optimizer 11.6%, NCCL 10.1%. Peer DMA now accounts for 325 calls and 179.1 ms; 205 peer averaging kernels take 29.3 ms. Token collectives account for 535 calls / 85.8 ms, of which 410 are DDP readiness/completion. Group completion acknowledgments for 4 or 8 buckets and use a separate averaging stream once both peers have completed the reads. Keep in-place results and exact BF16 math. Scratch /tmp/r36/tree; 11-bucket fixture and timing PID 362659, /tmp/r36/screen.log, /tmp/r36/verify-4.log, /tmp/r36/verify-8.log, outputs /tmp/r36/time-batch4 and /tmp/r36/time-batch8.

- Ended 2026-11-18 15:49:25 UTC. The 11-bucket fixture passed bitwise on both ranks for both group sizes. Four-bucket acknowledgments reached 1.8123× (0.1293 s/update), eight reached 1.8095× (0.1295 s/update), both effectively tied. Keep per-bucket acknowledgments. Scratch commit 7e05b04.
8.5 hRound 37
- Started 2026-11-18 15:50:05 UTC. Reprofile the unchanged peer-DMA DDP stack: PID 366223, /tmp/r37/profile.log, output /tmp/r37/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 36.8%, copy/cast 31.8%, optimizer 12.6%, NCCL 2.6%. The serial full-gradient FP32 sum-of-squares pass takes 7.5 ms over five updates, in addition to 30.2 ms averaging the peer buckets. Fuse FP32 squared partials into that already-required averaging pass; consume them only once and only when the live optimizer gradients exactly cover the completed buckets. Fallback to the existing norm otherwise. Scratch /tmp/r37/tree; fixture and time PID 367698, logs /tmp/r37/verify.log, /tmp/r37/screen.log, output /tmp/r37/time.

- Ended timing 2026-11-18 15:54:13 UTC. Two-rank DDP fixture passed with bitwise gradients and FP32 norms within 5e-7 relative, and checked that the cached norm cannot be consumed twice. Judge reached 1.8236× (0.1285 s/update), 0.9 ms faster than R35. Full check PID 369489, /tmp/r37/check.log, output /tmp/r37/check. R35 remains active until the verdict.

- Full check finished 2026-11-18 16:03:08 UTC: PASS N1 at 1.8179× (0.1289 s/update vs 0.23433 frozen), all exact checks passed and the six endpoints were within 0.419σ. Promoted as 678304c.
8.7 hRound 38
- Started 2026-11-18 16:05:26 UTC. Profile of the promoted peer averaging and gradient norm stack: PID 374735, /tmp/r38/profile.log, output /tmp/r38/profile.

- Profile (5 updates, rank 0): GPU busy 66% including the first profiled update's host pause; GEMM 36.6%, copy/cast 31.6%, optimizer 12.5%, NCCL 4.4%. Peer DMA is 179.2 ms; 205 BF16 averaging/norm launches use 30.8 ms; readiness/completion tokens use most of the 34.3 ms of small NCCL all-reduces. Retune DDP buckets for the peer-DMA path to trade overlap for fewer handoffs. Test 64, 192, 256 and 512 MiB on the same tree; PID 376240, /tmp/r38/screen.log, outputs /tmp/r38/time-64, /tmp/r38/time-192, /tmp/r38/time-256, /tmp/r38/time-512.

- First sweep: 64 MiB 1.8278× (0.1282 s/update), 192 MiB 1.8137×, 256 MiB 1.8151×, and 512 MiB 1.8151×. Screen 96 and 32 MiB and the 128 MiB control. PID 382010, /tmp/r38/screen-fine.log, outputs /tmp/r38/time-96, /tmp/r38/time-32 and /tmp/r38/time-128.

- Ended 2026-11-18 16:17:47 UTC. Fine sweep: 96 MiB 1.8222× (0.1286 s/update), 32 MiB 1.8264× (0.1283 s/update), and the 128 MiB control 1.8193× (0.1288 s/update). The 64 MiB result is promising but the gain alone is below the frozen-reference noise margin; do not promote or run a separate check. Carry 64 MiB as an independent switch trial for the next full check.
8.9 hRound 39
- Started 2026-11-18 16:17:47 UTC. Profile of the active R37 stack: PID 386234, /tmp/r39/profile.log, output /tmp/r39/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 34.8%, copy/cast 30.1%, optimizer 11.8%, NCCL 9.3%. Retention uses 390 strided BF16 multiplies (7.5 ms), associated layout copies, and separate exponentials and channel reductions. Fuse gated query/value/decay preparation for the cross-chunk BMMs and its backward, writing contiguous operands and preserving BF16 rounding before products, FP32 gradient reductions, and no atomics. Scratch /tmp/r39/tree; fixture and timing PID 387770, logs /tmp/r39/verify.log and /tmp/r39/screen.log, output /tmp/r39/time.

- The deterministic and production fixtures both passed at 24 heads, 8 chunks, 256 tokens and width 128. The fusion alone screened at 1.8509× (0.1266 s/update), 2.2 ms faster than the R38 128 MiB control. Screen it with R38's best 64 MiB switch before full validation; PID 389488, /tmp/r39/screen-bucket64.log, output /tmp/r39/time-bucket64.

- Timing ended 2026-11-18 16:24:18 UTC. The combined 64 MiB and cross-projection stack reached 1.8746× (0.1250 s/update); the 64 MiB change contributes 1.6 ms/update on this stack. Detached quick then full check for the combined stack: PID 390970, /tmp/r39/quick.log and /tmp/r39/check.log, outputs /tmp/r39/quick and /tmp/r39/check. R37 remains active pending verdict.

- Quick check finished 2026-11-18 16:29:06 UTC: PASS, including all exact invariants (/tmp/r39/quick/check.json). The full check is running.

- Full check finished 2026-11-18 16:36:44 UTC: PASS N1 at 1.8716× (0.1252 s/update vs 0.23433 frozen). All exact checks passed and all six endpoints were within 0.618σ. Promoted as dd1e7b6 with 64 MiB DDP buckets.
9.3 hRound 40
- Started 2026-11-18 16:37:42 UTC. Profile of the promoted cross-projection and 64 MiB DDP stack: PID 398680, /tmp/r40/profile.log, output /tmp/r40/profile.

- Profile (5 updates, rank 0): GPU busy 74%; GEMM 33.1%, copy/cast 27.4%, NCCL 14.3%, optimizer 11.3%. The 335 peer BF16 average/norm launches take 35.1 ms. Try native packed two-lane BF16 predivision and addition, using the same BF16 rounding points, then decompose pairs into FP32 for the norm. Scratch /tmp/r40/tree; two-rank fixture and time PID 400178, logs /tmp/r40/verify.log and /tmp/r40/screen.log, output /tmp/r40/time.

- Fixture launcher retry: the candidate environment has no standalone torchrun script. Use its Python with -m torch.distributed.run, PID 400190, logs /tmp/r40/verify-launch.log and /tmp/r40/screen-launch.log; no code change.

- Ended 2026-11-18 16:41:57 UTC. Both ranks matched BF16 predivision and gradients bitwise, and FP32 norms within 5e-7 relative. The packed trial screened at 1.8746× (0.1250 s/update), identical to R39's screening. No judge check or promotion. Scratch commit 6a2ffd4.
9.3 hRound 41
- Started 2026-11-18 16:42:20 UTC. Reprofile the unchanged validated R39 stack: PID 402082, /tmp/r41/profile.log, output /tmp/r41/profile.

- Profile (5 updates, rank 0): GPU busy 74%; GEMM 34.9%, copy/cast 29.0%, optimizer 12.0%, NCCL 9.3%. Peer DMA uses 179.6 ms; 795 NCCL token launches use 76.0 ms, with 670 from DDP. Push BF16 buckets into peer-owned receive buffers instead of pulling from live gradient storage. One reuse fence before the first peer bucket protects the previous backward; each copy has a single completion fence before BF16 averaging. Scratch /tmp/r41/tree; rank-skewed 12-update fixture and time PID 403598, logs /tmp/r41/verify.log and /tmp/r41/screen.log, output /tmp/r41/time.

- Twelve rank-skewed backward passes matched gradients bitwise and FP32 norms within 5e-7 relative. Push timing regressed to 1.0471× (0.2238 s/update); keep R39 active. Profile the rejected variant to distinguish DMA transport from synchronization, PID 405449, /tmp/r41/profile-push.log, output /tmp/r41/profile-push.

- Push profile: DMA stayed near 181 ms, but the 465 token kernels rose to 374 ms total (median 268 μs, versus 11 μs on R39), consistent with exposed completion waits. Preserve the first trial as scratch commit 4b92b40. Try queueing copies independently of earlier token waits and averaging while retaining a reuse wait at the next backward. PID 406988; /tmp/r41/verify-split.log, /tmp/r41/screen-split.log, output /tmp/r41/time-split.

- Ended 2026-11-18 16:51:20 UTC. Split-stream push also matched all gradients bitwise and norms in the 12-update fixture, but screened at 1.0503× (0.2231 s/update). Scratch commit bd7ef03. No check or promotion; keep the validated pull transport.
9.5 hRound 42
- Started 2026-11-18 16:51:59 UTC. Fresh profile of the unchanged R39 stack: PID 408837, /tmp/r42/profile.log, output /tmp/r42/profile.

- Profile (5 updates, rank 0): GPU busy 75%; GEMM 32.4%, copy/cast 26.4%, NCCL 16.9%, optimizer 10.9%. After R39, 260 strided BF16 layout copies still take 6.1 ms. The scaled retention key has token-major strides, so cross-chunk BMMs cannot flatten its head/chunk batch dimensions without copying. Try compiling the scalar multiply and layout pack together while preserving BF16 rounding. Scratch /tmp/r42/tree; fixture and timing PID 410357, logs /tmp/r42/verify.log and /tmp/r42/screen.log, output /tmp/r42/time.

- Compiled scaling/packing matched bitwise both directions but screened at 1.8687× (0.1254 s/update), no gain; scratch commit 02ea607. Fuse the same key pack and BF16 scale into the existing rotary pair kernel, rounding before the scale in forward and before inverse rotation in backward. Scratch /tmp/r42/rotary-tree, PID 412168, logs /tmp/r42/verify-rotary.log and /tmp/r42/screen-rotary.log, output /tmp/r42/time-rotary.

- Ended 2026-11-18 16:59:05 UTC. Fused rotary produced bitwise outputs and key gradients in both modes, but screened at 1.8776× (0.1248 s/update), only 0.2 ms better than the R39 screen. Scratch commit 8eaf0f5; no separate check or promotion for a gain within noise.
9.7 hRound 43
- Started 2026-11-18 17:02:40 UTC. Reprofile the validated R39 stack: PID 413942, /tmp/r43/profile.log, output /tmp/r43/profile.

- Profile (5 updates, rank 0): GPU busy 76%; GEMM 32.0%, copy/cast 26.5%, NCCL 17.4%, optimizer 10.9%. The final gradient bucket transfers 635.0 MB/update and takes about 1.6 ms in peer DMA before its averaging; it carries the embedding gradient. Try a lossless zero-block mask on large final buckets, copying only blocks with any nonzero bits (including signed zeros), then performing the unchanged dense BF16 average and FP32 norm. Dense inputs remain supported. Scratch /tmp/r43/tree; two-rank alternating dense/sparse fixture and timing PID 415462, logs /tmp/r43/verify.log and /tmp/r43/screen.log, output /tmp/r43/time.

- The direct-read fixture matched alternating sparse/dense gradients bitwise, but timing aborted: Triton could not address the IPC gradient buffer in the receiver context. Replace the direct peer loads with explicitly packed local BF16 buffers and use ordinary source-mapped peer DMA. A count exchange selects full-bucket fallback on both ranks when either pack overflows. PID 417477; /tmp/r43/verify-packed.log, /tmp/r43/screen-packed.log, output /tmp/r43/time-packed.

- Ended 2026-11-18 17:11:20 UTC. Packed DMA matched alternating sparse and dense-overflow gradients bitwise on both ranks and FP32 norms within 5e-7. Timing reached 1.8746× (0.1250 s/update), tying R39. The pack/metadata work and count synchronization offset the lower transfer volume. Scratch commit e76eb67; no check or promotion.
9.8 hRound 44
- Started 2026-11-18 17:12:02 UTC. Fresh profile of the active R39 stack: PID 419425, /tmp/r44/profile.log, output /tmp/r44/profile.

- Profile (5 updates, rank 0): GPU busy 74%; GEMM 34.1%, copy/cast 28.3%, optimizer 11.7%, NCCL 11.6%. Fused Adam refresh takes 77.7 ms across 120 launches. Compare streaming moment stores alone and with L2-only gradient/moment reads; leave FP32 arithmetic, weight cache behavior and 24-stage schedule unchanged. Scratch /tmp/r44/tree; two-rank optimizer fixture and times PID 420970, /tmp/r44/screen.log, /tmp/r44/verify-stream.log, /tmp/r44/verify-l2.log, outputs /tmp/r44/time-stream and /tmp/r44/time-l2.

- Ended 2026-11-18 17:17:08 UTC. Both cache settings matched master/model weights, moments and step bitwise on both ranks. Streaming stores alone reached 1.8657× (0.1256 s/update); adding L2-only reads reached 1.8672× (0.1255 s/update). Neither is faster; keep default caches. Scratch commit bcb8288.
9.9 hRound 45
- Started 2026-11-18 17:17:35 UTC. Fresh profile of the active R39 stack: PID 424656, /tmp/r45/profile.log, output /tmp/r45/profile.

- Profile (5 updates, rank 0): GPU busy 72%; GEMM 36.0%, copy/cast 30.2%, optimizer 12.5%, NCCL 5.6%. Steady training queues compute, peer copies, token collectives and averaging on distinct CUDA streams. Screen driver connection limits 32 and 4 for potential overlap changes. Same tree and math; PID 426184, /tmp/r45/screen.log, outputs /tmp/r45/time-32 and /tmp/r45/time-4.
↑ YOCO