Kimi k3.1-pro 1.37×

Harness: Claude Code · switches: YOCO_DDP_BUCKET_MB=100 YOCO_OPT_OVERLAP_CAST=1 YOCO_OPT_FUSED_GRAD=1 YOCO_OPT_APEX_RMS_NORM=1 YOCO_OPT_BROADCAST_CHUNKS=2 YOCO_OPT_DEFER_STATS=1 YOCO_OPT_PACKED_SELF=2 YOCO_OPT_FLASH_CE=2 YOCO_OPT_NORM_CHUNK=262144 YOCO_OPT_FUSED_GATE=1 YOCO_OPT_ADAM_BROADCAST=6 YOCO_OPT_GRAD_BUCKET_VIEW=1 YOCO_OPT_FLAT_GRAD=1

Judged
1.37×
Tier
N1
Self-best
1.38×
Rounds
33
Hours
10.0
API cost
$33.5
Input tokens · 99.2% cached
138 M
Output tokens
0.4 M
Requests
939
Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
✗ fp32 broadcast kept as-is, only chunked and pipelined with Adam (1.24× → 1.29×)
Forward / backward CUDA graph
✗ only listed as a next step at the end
Newer cuBLAS
✗ tried cuBLASLt workspace, no gain
Retention / attention kernel swap
✗ instead merged five projections into one GEMM (1.196× → 1.213×)
Other main levers
Apex RMSNorm 1.09× → 1.16×; master-grad cast batched per DDP bucket 1.31× → 1.37×; fused cross-entropy; DDP bucket views (round 30)
Raw work log33 sections
0.1 hRound 1 — baseline profile and foreach optimizer data movement
- Start (GPU lock): 2026-11-06 17:12:29 UTC.
- Reference profile: /tmp/r1/profile/profile.json and profile.md; status ok, instrumented 0.2903 s/update (not comparable with time). GPU busy 75%. Per 5 profiled updates: NCCL 399.6 ms (including 185.2 ms of ten fp32 broadcasts), GEMM 327.6 ms, memcpy/cast 218.1 ms, elementwise 153.6 ms, Adam multi-tensor 91.6 ms. Optimizer range 241.4 ms over four recorded steps; ZeRO refresh range 187.5 ms over ten entries; clip range 54.5 ms over five steps. Hotspot class: optimizer memory traffic and launch overhead (missed existing foreach fast path).
- Unchanged-tree judge timing: /tmp/r1/time/time.json, status ok, 0.2340 s/update, speedup_vs_frozen_reference=0.9988, windows 2.341/2.338 s, range 0.0012. Frozen reference 0.23373 s/update.
- Change being screened: YOCO_OPT_FOREACH=1 replaces per-parameter bf16↔fp32 copies and fp32 gradient scaling with PyTorch foreach operations. Same tensors, dtypes, scaling points, and optimizer layout; intended N0. A small live functional probe showed mixed foreach copies and scaling bitwise equal to the original operations.
- Detached jobs: profile launcher PID 12710 (actual bench PID 12711), log /tmp/r1/profile-launch.log; timing launcher PID 14534 (actual bench PID 14535), log /tmp/r1/time-launch.log. Both completed successfully.
- Candidate timing: /tmp/r1/foreach-time/time.json, status ok, 0.2346 s/update, speedup_vs_frozen_reference=0.9963, windows 2.353/2.340 s, range 0.0055. No gain; the change was disabled in the submitted switch file.
- Exactness: /tmp/r1/foreach-quick/check.json, status PASS, bitwise N0 (9616/9616 recorded items identical, zero exact failures).
- End: 2026-11-06 17:26:45 UTC. Conclusion: foreach optimizer copies/scaling are exact but slightly slower in end-to-end timing; do not enable. The file remains behind an off-by-default switch as a screened fast-path inventory entry.
0.5 hRound 2 — NCCL protocol sweep
- Start (GPU lock): 2026-11-06 17:27:52 UTC; end: 2026-11-06 17:38:40 UTC.
- Profile of current best (baseline behavior): /tmp/r2/profile/profile.json, status ok, instrumented 0.2824 s/update. NCCL remained the largest kernel category at 417.6 ms/5 updates: 224.7 ms bf16 gradient allreduce (595 calls) and 185.4 ms fp32 broadcast (10 calls), both RING_LL. Optimizer range 241.2 ms/4 recorded steps. Hotspot class: communication configuration/backend.
- NCCL_PROTO=Simple: /tmp/r2/simple-time/time.json, status ok, 0.2337 s/update, speedup_vs_frozen_reference=1.0001, windows 2.336/2.338 s, range 0.0006. No resolved gain.
- NCCL_PROTO=LL128: /tmp/r2/ll128-time/time.json, status ok, 0.2358 s/update, speedup_vs_frozen_reference=0.9912, windows 2.357/2.360 s, range 0.0014. Slower.
- Simple exactness: /tmp/r2/simple-quick/check.json, status PASS, bitwise N0 (9616/9616). No protocol was enabled in the submission.
- Detached jobs: profile launcher 20150 (actual 20151); Simple timing 21638 (actual 21639); LL128 timing 23124 (actual 23125); Simple quick 24667 (actual 24668). Logs are the corresponding /tmp/r2/*-launch.log files. No code change was made for this sweep, so there was nothing to commit.
0.7 hRound 3 — DDP gradient bucket sizing
- Start (GPU lock): 2026-11-06 17:39:36 UTC; end: 2026-11-06 17:51:52 UTC.
- Profile: /tmp/r3/profile/profile.json, status ok, instrumented 0.2889 s/update. NCCL 344.0 ms/5 updates: 191.9 ms in ten fp32 broadcasts and 151.1 ms in 595 bf16 allreduces (119/update). Gradient bucketing was the selected communication/launch hotspot.
- Added off-by-default YOCO_DDP_BUCKET_MB; it overrides only the DDP constructor argument and leaves the parsed recipe configuration unchanged.
- 50 MiB: /tmp/r3/ddp50-time/time.json, 0.2332 s/update, speedup_vs_frozen_reference=1.0023, windows 2.333/2.330 s.
- 100 MiB: /tmp/r3/ddp100-time/time.json, 0.2308 s/update, speedup_vs_frozen_reference=1.0127, windows 2.311/2.306 s, range 0.0022; best of the sweep.
- 250 MiB: /tmp/r3/ddp250-time/time.json, 0.2316 s/update, speedup_vs_frozen_reference=1.0092, windows 2.314/2.318 s.
- 100 MiB exactness: /tmp/r3/ddp100-quick/check.json, status PASS, bitwise N0 (9616/9616). Enabled 100 MiB in /submission/switches.json as the current best validated setting; alone it remains below the full check's 2% benefit threshold.
- Detached jobs: profile launcher 26912 (actual 26913); timing launchers 28419/29934/31431 (actual 28420/29935/31432); quick launcher 32864 (actual 32865). Corresponding logs under /tmp/r3/.
0.9 hRound 4 — overlap local weight casting with ZeRO broadcasts
- Start (GPU lock): 2026-11-06 17:53:23 UTC; end: 2026-11-06 18:04:26 UTC (including the full check).
- Profile of the 100 MiB bucket candidate: /tmp/r4/profile/profile.json, status ok, instrumented 0.2769 s/update. DDP allreduce calls fell from 119 to 47 per update. Ten fp32 broadcasts still cost 185.5 ms/5 updates; optimizer range 240.2 ms/4 steps. Selected hotspot: exposed fp32→bf16 weight conversion around the broadcasts.
- Added off-by-default YOCO_OPT_OVERLAP_CAST. It launches the unchanged fp32 broadcasts asynchronously, performs the reference per-parameter fp32→bf16 copies for the rank-owned shard while they run, waits, then performs only the remaining remote-shard copies. Same dtypes, values, collectives, and optimizer layout; intended and measured N0.
- Screening time: /tmp/r4/overlap-time/time.json, status ok, 0.2272 s/update, speedup_vs_frozen_reference=1.0287, windows 2.273/2.271 s, range 0.0009.
- Full verdict: /tmp/r4/check/check.json, status PASS, speedup 1.0296, bitwise N0 (candidate 0.2270 s/update, windows 2.272/2.269 s; 9616/9616 recorded items identical, zero exact failures). Enabled both switches in the submission and committed the overlap change.
- Detached jobs: profile launcher 35087 (actual 35088), overlap timing launcher 36599 (actual 36600), full-check launcher 38119 (actual 38120). Logs /tmp/r4/profile-launch.log, overlap-time-launch.log, and check-launch.log.
1.2 hRound 5 — fused gradient norm, cast, scale, and redundant-fill removal
- Start (GPU lock): 2026-11-06 18:05:57 UTC; end: 2026-11-06 18:20:39 UTC (including the full check).
- Profile of the validated N0 stack: /tmp/r5/profile/profile.json, status ok, instrumented 0.2725 s/update. memcpy/cast 225.1 ms and elementwise 153.4 ms per five updates; clip range 55.7 ms/5 and optimizer range 249.0 ms/4. Selected hotspot: master-gradient memory passes and fills.
- Functional probes: Apex multi-tensor norm on bf16 was bitwise equal to the norm after fp32 conversion on production shapes. /tmp/r5/probe_cast_scale.py showed the Triton bf16→fp32 cast-scale bitwise equal to separate PyTorch operations on all tested shapes.
- Added off-by-default YOCO_OPT_FUSED_GRAD: norm directly from bf16 gradients, Triton cast with the final clipping/sample-size scale, and no zero-fill of master gradients that are completely overwritten. Missing-gradient and other-layout fallbacks are retained.
- Screening time: /tmp/r5/fused-grad-time/time.json, status ok, 0.2197 s/update, speedup_vs_frozen_reference=1.0639, windows 2.195/2.198 s, range 0.0015.
- Full verdict: /tmp/r5/check/check.json, status PASS, speedup 1.0907, tier N1. Candidate 0.2143 s/update, windows 2.151/2.135 s, range 0.0073 (valid). Gate 2 was NOT_BITWISE with zero exact-item failures; gate 3 passed all six endpoints (largest |difference|/bound about 0.3975). Promoted the switch and committed the change. This supersedes the N0 full check as the best validated submission.
- Detached jobs: profile launcher 41812 (actual 41813), timing launcher 43535 (actual 43536), full-check launcher 45060 (actual 45061). Logs /tmp/r5/profile-launch.log, fused-grad-time-launch.log, and check-launch.log.
1.5 hRound 6 — packed FFN input projections
- Start (GPU lock): 2026-11-06 18:22:47 UTC; end: 18:38:21 UTC (including the full check and comparison timing).
- Profile of the current best: /tmp/r6/profile/profile.json, status ok, instrumented 0.2626 s/update. GEMMs consumed 326.2 ms/5 updates; the self and cross FFN ranges consumed 36.39 and 32.41 ms/5 updates. Selected hotspot: pairs of same-input fc1/gate GEMMs and their launch overhead.
- Added off-by-default YOCO_OPT_PACKED_FFN: concatenate the two existing weight tensors, run one F.linear, split the result, and preserve the original parameters, names, shapes, optimizer entries, and autograd gradient routing. Merged-GEMM arithmetic makes the enabled path N1.
- Screening time: /tmp/r6/packed-ffn-time/time.json, status ok, 0.2180 s/update, speedup_vs_frozen_reference=1.0722, windows 2.183/2.177 s, range 0.0024. Peak allocated memory increased from 61.75 to 64.19 GiB because of the temporary packed weights.
- Full verdict: /tmp/r6/check/check.json, status PASS, speedup 1.0687, tier N1. Candidate 0.2187 s/update, windows 2.194/2.181 s. Gate 2 had zero exact failures; gate 3 passed all six endpoints (largest |difference|/bound about 1.4968). This was slower than the round-5 full check at 1.0907×.
- Same-tree comparison with packing disabled: /tmp/r6/unpacked-time/time.json, status ok, 0.2151 s/update, speedup_vs_frozen_reference=1.0866, windows 2.151/2.152 s, range 0.0004. Packing is therefore not promoted; the submitted switches remain unchanged and the implementation is committed disabled as screened fast-path inventory.
- Detached jobs: logs /tmp/r6/profile-launch.log, packed-ffn-time-launch.log, check-launch.log (launcher 53155, actual 53156), and unpacked-time-launch.log (launcher 58167, actual 58168). All completed successfully.
1.8 hRound 7 — Apex fused RMSNorm
- Start (GPU lock): 2026-11-06 18:42:24 UTC; end: 18:56:33 UTC (including the full check).
- Profile of the round-5 validated stack: /tmp/r7/profile/profile.json, status ok, instrumented 0.3033 s/update. NCCL 472.8 ms, GEMM 322.1 ms, elementwise 71.3 ms, and reductions 18.8 ms per five updates. Selected hotspot: RMSNorm's separate fp32 reduction, conversion, affine, and backward kernels.
- /tmp/r7/probe_rms.py checked production shapes 128 and 3072 with and without affine weights. Apex retained bf16 outputs, input gradients, and weight gradients. Affine forward outputs were bitwise equal in this probe; non-affine outputs and backward reductions were not, so the enabled path is N1.
- Added off-by-default YOCO_OPT_APEX_RMS_NORM, selecting Apex's fused RMSNorm/RMSNorm-affine implementation for CUDA tensors and retaining the original formula as the fallback.
- Screening time: /tmp/r7/apex-rms-time/time.json, status ok, 0.1974 s/update, speedup_vs_frozen_reference=1.1840, windows 1.973/1.975 s, range 0.0011. Peak allocated memory 60.32 GiB.
- Full verdict: /tmp/r7/check/check.json, status PASS, speedup 1.1559, tier N1. Candidate 0.2022 s/update, windows 2.024/2.021 s, range 0.0014. Gate 2 had zero exact failures; gate 3 passed all six endpoints (largest |difference|/bound about 0.6607). Promoted the switch; this is the new best validated submission.
- Detached jobs: profile launcher 59783 (actual bench 59785), log /tmp/r7/profile-launch.log; timing launcher 61399 (actual 61400), log apex-rms-time-launch.log; full-check launcher 62937 (actual 62938), log check-launch.log. All completed successfully.
2.1 hRound 8 — chunk-pipelined ZeRO broadcasts and remote casts
- Start (GPU lock): 2026-11-06 18:59:51 UTC; end: 19:17:03 UTC (including the full check).
- Profile of the Apex RMSNorm stack: /tmp/r8/profile/profile.json, status ok, instrumented 0.2424 s/update. Ten fp32 broadcasts consumed 191.5 ms and weight-cast copies 63.6 ms per five updates; refresh remained the largest optimizer-side range. Selected hotspot: remote casts left exposed after the two full-buffer broadcasts.
- Added off-by-default YOCO_OPT_BROADCAST_CHUNKS. For values above one, each OSS bucket broadcast is split into interleaved chunks. The unchanged local casts overlap the launched collectives; after each remote chunk completes, its existing bf16 parameters are copied from the corresponding fp32 master-view slices while later broadcasts remain in flight. Parameter names, values, dtypes, optimizer state, and RNG are unchanged; this increment is N0. Noncontiguous and broadcast-fp16 layouts retain the prior path.
- Two chunks: /tmp/r8/chunks2-time/time.json, status ok, 0.1951 s/update, speedup_vs_frozen_reference=1.1980, windows 1.950/1.953 s, range 0.0019.
- Four chunks: /tmp/r8/chunks4-time/time.json, status ok, 0.1952 s/update, speedup_vs_frozen_reference=1.1974, windows 1.952/1.952 s. Neutral versus two, so two was selected to minimize collective count.
- Full verdict for two chunks: /tmp/r8/check/check.json, status PASS, speedup 1.1943, stack tier N1. Candidate 0.1957 s/update, windows 1.952/1.963 s, range 0.0059. Gate 2 had zero exact failures and the same 1436 unresolved tensor differences as round 7; all six trajectory endpoints were identical to round 7, confirming that this increment did not alter the recorded numerics. Promoted YOCO_OPT_BROADCAST_CHUNKS=2; this is the new best validated submission.
- Detached jobs: profile launcher 67989 (actual 67990); two-chunk timing launcher 69549 (actual 69550); four-chunk timing launcher 71113 (actual 71114); full-check launcher 72625 (actual 72626). Logs under /tmp/r8/. All completed successfully.
2.2 hRound 9 — persistent packed-FFN weight views
- Start (GPU lock): 2026-11-06 19:19:08 UTC; end: 19:24:12 UTC.
- Profile of the validated two-chunk stack: /tmp/r9/profile/profile.json, status ok, instrumented 0.2339 s/update. GEMMs consumed 325.7 ms/5 updates in 925 NT, 855 NN, and 860 TN main-kernel calls; FFN ranges consumed 67.4 ms/5. Selected hotspot: repeated same-input FFN GEMMs, this time without round 6's per-forward concatenation.
- Revised the disabled YOCO_OPT_PACKED_FFN path to alias the original fc1 and gate parameters into adjacent shared storage after initialization and device/dtype conversion. A custom autograd view spans both weights and splits the packed gradient back to the original parameters. No packed buffer is registered and no concatenation occurs in the training loop.
- /tmp/r9/probe_packed_view.py confirmed shared adjacent storage, unchanged parameter names/shapes, finite bf16 parameter gradients, and correct forward/backward behavior. The small-case forward was bitwise; backward differed slightly, consistent with N1 merged-GEMM arithmetic.
- Timing: /tmp/r9/packed-view-time/time.json, status ok, 0.1980 s/update, speedup_vs_frozen_reference=1.1805, windows 1.963/1.998 s, range 0.0173, peak allocated memory 60.32 GiB. This is slower than the current 1.1943× full-check candidate and shows no memory benefit. No full check was run; the path remains disabled and the submitted switches are unchanged.
- Detached jobs: profile launcher 77567 (actual 77568), log /tmp/r9/profile-launch.log; timing launcher 79372 (actual 79373), log packed-view-time-launch.log. Both completed successfully.
2.6 hRound 10 — defer unused per-update statistics
- Start (GPU lock): 2026-11-06 19:27:20 UTC; end: 19:46:58 UTC (including the full check).
- Profile: /tmp/r10/profile/profile.json, status ok, instrumented 0.2478 s/update. Apex Adam was already multi-tensor fused (71.4 ms/4 recorded steps), so no standard optimizer substitution was indicated. The trace instead exposed 225 aten::item calls with 36.1 ms host time; the longest calls followed the optimizer and came from redundant finite checking and metric rounding.
- The fixed train loop only tests log_output is not None and independently computes actual interval logs from train_inner. Added off-by-default YOCO_OPT_DEFER_STATS: _reduce_and_log_stats now returns a lazy dictionary that performs the original smoothed-value computation on any access, while normal training leaves it unconsumed. The second gradient finite check is omitted because train_step has already raised for non-finite norms. Metric aggregation and actual printed statistics are unchanged; this increment is N0 for the valid finite training path.
- Screening time: /tmp/r10/defer-stats-time/time.json, status ok, 0.1946 s/update, speedup_vs_frozen_reference=1.2011, windows 1.942/1.949 s, range 0.0037.
- Full verdict: /tmp/r10/check/check.json, status PASS, speedup 1.1955, stack tier N1. Candidate 0.1955 s/update, windows 1.958/1.953 s, range 0.0026. Gate 2 had zero exact failures and the same 1436 differences; every trajectory endpoint was identical to rounds 7–8. Promoted the switch as the best validated configuration, a small gain over round 8's 1.1943×.
- Detached jobs: profile launcher 80982 (actual 80983), timing launcher 82716 (actual 82717), full-check launcher 84241 (actual 84242). Logs under /tmp/r10/. All completed successfully.
2.8 hRound 11 — cuBLASLt workspace sweep
- Start (GPU lock): 2026-11-06 19:48:59 UTC; end: 19:57:21 UTC.
- Profile of the deferred-statistics stack: /tmp/r11/profile/profile.json, status ok, instrumented 0.2648 s/update. GEMMs consumed 322.1 ms/5 updates and were the selected library-configuration hotspot. The previous round's long aten::item entries were absent.
- 32768 KiB workspace: /tmp/r11/cublaslt32-time/time.json, status ok, 0.1998 s/update, speedup_vs_frozen_reference=1.1698, windows 1.999/1.996 s. Clearly slower.
- 1024 KiB workspace: /tmp/r11/cublaslt1-time/time.json, status ok, 0.1942 s/update, speedup_vs_frozen_reference=1.2036, windows 1.943/1.941 s. This matches default-path variation and is consistent with the usual 1 MiB default.
- 512 KiB workspace: /tmp/r11/cublaslt05-time/time.json, status ok, 0.1944 s/update, speedup_vs_frozen_reference=1.2023, windows 1.942/1.946 s. Neutral. No setting showed a resolved gain over the current default path, so no workspace override was added and no full check was run.
- Detached jobs: profile launcher 89219 (actual 89220); timing launchers 90745/92237/93758 (actual 90746/92238/93759). Logs under /tmp/r11/. All completed successfully. No code changed, so there was nothing to commit.
3.1 hRound 12 — packed self-decoder projections
- Start (GPU lock): 2026-11-06 19:59:55 UTC; end: 20:18:28 UTC (including the full check).
- Profile: /tmp/r12/profile/profile.json, status ok, instrumented 0.2339 s/update. GEMMs consumed 322.8 ms/5 updates; self-mixer ranges consumed 72.67 ms GPU and 133.22 ms CPU per five updates. Selected hotspot: four large same-input q/k/v/g projections plus the small same-input gt projection.
- Added off-by-default YOCO_OPT_PACKED_SELF. Original projection parameters are aliased into adjacent storage after initialization and device/dtype conversion; a custom autograd view splits packed weight gradients back to the original parameters. Mode 1 packs q/k/v/g and leaves gt separate; mode 2 packs all five. No fused buffer is registered and no concatenation occurs during updates. Merged-GEMM arithmetic is N1.
- /tmp/r12/probe_packed_self.py confirmed shared storage, unchanged parameter names/shapes, and finite bf16 forward/backward tensors for mode 1. Both modes subsequently completed judge timing with normal optimizer-step counts.
- Four-way timing: /tmp/r12/packed-self-time/time.json, status ok, 0.1925 s/update, speedup_vs_frozen_reference=1.2142, windows 1.922/1.927 s. Five-way timing: /tmp/r12/packed-self5-time/time.json, status ok, 0.1919 s/update, speedup_vs_frozen_reference=1.2180, windows 1.916/1.923 s. Mode 2 was slightly better.
- Full verdict for mode 2: /tmp/r12/check/check.json, status PASS, speedup 1.2129, tier N1. Candidate 0.1927 s/update, windows 1.920/1.934 s, range 0.0069; peak allocated memory 60.63 GiB. Gate 2 had zero exact failures; gate 3 passed all endpoints (largest |difference|/bound about 1.7947). Promoted YOCO_OPT_PACKED_SELF=2 as the new best validated submission.
- Detached jobs: profile launcher 95359 (actual 95360); timing launchers 97372/98937 (actual 97373/98938); full-check launcher 100521 (actual 100522). Logs under /tmp/r12/. All completed successfully.
3.4 hRound 13 — Adam-only multi-tensor chunk sizes
- Start (GPU lock): 2026-11-06 20:24:42 UTC; end: 20:34:11 UTC.
- Profile of the packed-self stack: /tmp/r13/profile/profile.json, status ok, instrumented 0.2304 s/update. Apex Adam consumed 71.34 ms of kernel time in 410 calls over five updates (82/update), 7.0% of kernel time. Selected hotspot: Adam launch chunking; gradient-norm reductions were explicitly left unchanged.
- Added off-by-default YOCO_OPT_ADAM_CHUNK, temporarily setting Apex's multi_tensor_applier.chunk_size only around the original FusedAdamV2.step and restoring it in finally. The Adam kernel, elementwise formula, states, and optimizer-step count are unchanged.
- 131072 elements: /tmp/r13/adam131072-time/time.json, status ok, 0.1948 s/update, speedup_vs_frozen_reference=1.1998, windows 1.941/1.955 s. 262144 elements: /tmp/r13/adam262144-time/time.json, status ok, 0.1940 s/update, speedup_vs_frozen_reference=1.2048, windows 1.945/1.936 s. Both are slower than the promoted 1.2129× full-check result, so neither was promoted and no full check was run.
- /tmp/r13/probe_adam_chunk.py compared default and 262144-element chunks for two steps on varied tensor sizes, including cross-chunk shapes: master weights, exp_avg, and exp_avg_sq were bitwise identical after both steps. This establishes the isolated override as N0 despite its negative timing.
- Detached jobs: profile launcher 105720 (actual 105721); timing launchers 107403/109019 (actual 107404/109020). Logs under /tmp/r13/. All completed successfully. The experimental switch remains disabled.
3.5 hRound 14 — foreach remote master-weight casts
- Start (GPU lock): 2026-11-06 20:36:54 UTC; end: 20:44:01 UTC.
- Profile: /tmp/r14/profile/profile.json, status ok, instrumented 0.2433 s/update. Copy/cast kernels consumed 152.1 ms and NCCL 324.1 ms over five updates; the bf16 direct-copy kernel alone consumed 72.93 ms in 1490 calls. Selected hotspot: the per-parameter remote fp32→bf16 casts after chunked ZeRO broadcasts.
- Added off-by-default YOCO_OPT_REMOTE_FOREACH, collecting the same source/destination slices and using torch._foreach_copy_ instead of one copy_ launch per parameter. Broadcast chunking and all other cast paths are unchanged.
- Candidate timing: /tmp/r14/remote-foreach-time/time.json, status ok, 0.1932 s/update, speedup_vs_frozen_reference=1.2098, windows 1.930/1.933 s. Same-session control with the switch off: /tmp/r14/control-time/time.json, status ok, 0.1927 s/update, speedup_vs_frozen_reference=1.2129, windows 1.931/1.922 s. The candidate showed no gain and was marginally slower, so it remains disabled and no full check was run.
- /tmp/r14/probe_remote_foreach.py verified bitwise-equal fp32→bf16 outputs for 200000, 65537, 527, and 1 elements. The isolated path is N0 but not beneficial.
- Detached jobs: profile launcher 110729 (actual 110730); timing launchers 112267/113841 (actual 112268/113842). Logs under /tmp/r14/. All completed successfully. The promoted switch file is unchanged.
3.7 hRound 15 — rotary tile-size sweep
- Start (GPU lock): 2026-11-06 20:44:46 UTC; end: 20:54:55 UTC.
- Profile: /tmp/r15/profile/profile.json, status ok, instrumented 0.2305 s/update. Two rotary kernel variants consumed 4.886 ms (270 calls) and 2.196 ms (130 calls) over five updates; self-mixer ranges consumed 64.73 ms GPU and 120.32 ms CPU. Selected hotspot: fixed four-row tiles in the existing interleaved rotary kernel.
- Added off-by-default YOCO_OPT_ROTARY_BLOCK_M, retaining the original per-element Triton formula, q/k layouts, autograd path, and backward while allowing larger sequence tiles. /tmp/r15/probe_rotary_tiles.py verified bitwise-identical production-shape forward outputs and input gradients for the default and 8/16/32-row tiles, establishing the isolated configuration as N0.
- Eight rows: /tmp/r15/rotary8-time/time.json, status ok, 0.1939 s/update, speedup_vs_frozen_reference=1.2054, windows 1.943/1.934 s. Sixteen rows: /tmp/r15/rotary16-time/time.json, status ok, 0.1947 s/update, 1.2005×, windows 1.937/1.957 s. Thirty-two rows: /tmp/r15/rotary32-time/time.json, status ok, 0.1965 s/update, 1.1895×, windows 1.963/1.967 s. All were slower than the 1.2129× control; larger tiles monotonically worsened timing. No setting was promoted and no full check was run.
- Detached jobs: profile launcher 115456 (actual profiler completed before a separate wait); timing launchers 117205/118774/120332 (actual 117206/118775/120334). Logs under /tmp/r15/. All completed successfully. The experimental override remains disabled.
4.5 hRound 16 — flat ZeRO bucket-to-model casts
- Start (GPU lock): 2026-11-06 21:00:35 UTC; end: 21:38:02 UTC (including the full check).
- Profile: /tmp/r16/profile/profile.json, status ok, instrumented 0.2377 s/update. The preceding profile measured 72.9 ms in 1490 rank-0 remote-cast kernels over five updates. Selected hotspot: per-parameter fp32-bucket→bf16-model casts.
- Added off-by-default YOCO_OPT_FLAT_CAST. After the first optimizer step, it mirrors trainable bf16 parameters into one persistent allocation without changing parameter objects, names, shapes, values, dtypes, or fp32 masters. It sorts metadata by actual ZeRO bucket offsets, validates complete coverage, and merges only source/destination-contiguous runs. Later local and remote chunk casts use those runs; unsupported layouts log a fallback reason and use the original path.
- The first contiguous-rank-span version fell back because greedy ZeRO ownership is not contiguous (/tmp/r16/flat-cast-time, 1.2104×). A run-based version also fell back because fairseq parameter order differs from bucket order (/tmp/r16/flat-runs-time, 1.2180×; profile still showed 1490 casts). An intentional diagnostic failure identified the offset mismatch in /tmp/r16/flat-debug-time/time.json. Sorting by bucket offset fixed activation; these intermediate timings were not accepted as fast-path evidence.
- Active bucket-sorted timing: /tmp/r16/flat-sorted-time/time.json, status ok, 0.1915 s/update, speedup_vs_frozen_reference=1.2205. Quick check /tmp/r16/quick/check.json: PASS, zero exact failures, same 1463 unresolved entries as round 12 and no new dtype/precision changes.
- Full verdict: /tmp/r16/check/check.json, status PASS, speedup 1.2117, stack tier N1. Candidate 0.1929 s/update, windows 1.932/1.927 s, range 0.0026; peak memory 60.63 GiB. All six trajectory endpoints were numerically identical to round 12, confirming the increment as N0. The full timing is slightly worse than round 12's 1.2129×, so the switch remains disabled and the promoted configuration is unchanged.
- Detached jobs: profile 122003; initial timing 123648; fallback profiles 125236/128399; guard diagnostic 130015; active timing 131636; quick check 133234; full check 135492 (actual PIDs launcher+1). Logs under /tmp/r16/. All intended runs completed; the diagnostic alone returned its planned ERROR with the guard reason.
4.8 hRound 17 — FlashAttention fused cross-entropy
- Start (GPU lock): 2026-11-06 21:40:16 UTC; end: 21:57:39 UTC (including the full check).
- Profile: /tmp/r17/profile/profile.json, status ok, instrumented 0.2303 s/update. Log-softmax/NLL consumed 8.8 ms over five updates and materialized fp32 log-probabilities; RMSNorm forward consumed 7.4 ms and residual adds were a smaller ceiling. Selected the existing FlashAttention fused-loss fast path rather than writing a residual+norm kernel.
- Added off-by-default YOCO_OPT_FLASH_CE in criterions/fast_cross_entropy.py. It replaces log-softmax + NLL with FlashAttention's Triton cross-entropy, preserving fp32 accumulation and scalar loss, target selection, ignore index, reduction, and fairseq's outer sample-size normalization. Mode 1 uses a separate gradient allocation; mode 2 uses the library's supported in-place logits backward. Other inputs fall back to the original criterion. The changed reduction/kernel rounding is N1.
- /tmp/r17/probe_flash_ce.py at 2048×50257 found finite fp32 loss and bf16 input gradients, absolute summed-loss difference 0.001953125, maximum gradient difference 1.91e-6, and relative gradient L2 difference 2.32e-7.
- Out-of-place timing: /tmp/r17/flash-ce-time/time.json, status ok, 0.1895 s/update, speedup_vs_frozen_reference=1.2334, windows 1.892/1.898 s, peak 59.30 GiB. In-place timing: /tmp/r17/flash-ce-inplace-time/time.json, status ok, 0.1891 s/update, 1.2360×, windows 1.887/1.896 s, peak 59.30 GiB. Mode 2 was slightly faster and was selected for validation.
- Full verdict for mode 2: /tmp/r17/check/check.json, status PASS, speedup 1.2250, stack tier N1. Candidate 0.1908 s/update, windows 1.904/1.913 s, range 0.0046. Gate 2 had zero exact failures, no dtype/precision-mode changes, and 1467 unresolved entries. Gate 3 passed every endpoint; largest |difference|/bound about 1.1961. Promoted YOCO_OPT_FLASH_CE=2 as the new best validated submission.
- Detached jobs: profile 140572; timing 142330/143940; full check 145557 (actual PIDs launcher+1). Logs under /tmp/r17/. All completed successfully.
4.9 hRound 18 — packed shared cross-decoder K/V projections
- Start (GPU lock): 2026-11-06 21:59:30 UTC; end: 22:05:51 UTC.
- Profile of the fused-loss stack: /tmp/r18/profile/profile.json, status ok, instrumented 0.2312 s/update. Selected hotspot: the two same-input 1024-wide K/V projections that build the shared cross-decoder cache.
- Added off-by-default YOCO_OPT_PACKED_KV, using persistent adjacent K/V weight storage and a custom autograd view that returns the original two gradient slices. Parameter objects, names, shapes, and optimizer entries remain unchanged; merged-GEMM arithmetic is N1.
- /tmp/r18/probe_packed_kv.py verified shared storage, unchanged parameter contract, finite output and all gradients. Its forward was bitwise and input-gradient relative L2 difference was 2.70e-4. The first probe incorrectly changed the global switch before the reference forward; fixing the probe to restore mode 0 for the reference resolved the unintended packed-view error.
- Timing: /tmp/r18/packed-kv-time/time.json, status ok, 0.1955 s/update, speedup_vs_frozen_reference=1.1955, windows 1.966/1.944 s, range 0.0117, peak 59.30 GiB. This is clearly slower than the 1.2250× full-check stack, so the path remains disabled and no correctness check was run.
- Detached jobs: profile 150792 (actual 150793); timing 152721 (actual 152722). Logs under /tmp/r18/. Both completed successfully. The promoted switch file is unchanged.
5.2 hRound 19 — gradient-norm chunk-size sweep
- Start (GPU lock): 2026-11-06 22:07:45 UTC; end: 22:25:45 UTC (including the full check).
- Profile: /tmp/r19/profile/profile.json, status ok. The preceding traces showed about 165 bf16 L2Norm kernels per update. Selected hotspot: fairseq's fixed 65536-element gradient-norm chunks, independently of the rejected Adam-only chunk override.
- Added off-by-default YOCO_OPT_NORM_CHUNK in fast_grad.py. For multiple ordinary gradients it calls fairseq's existing multi_tensor_total_norm with the requested chunk size and preserves aggregate_norm_fn; single, expert, base-expert, and sharded layouts retain the original wrapper. Adam chunking is untouched. Changed reduction partitioning is N1.
- 32768 elements: /tmp/r19/norm32768-time/time.json, status ok, 0.1939 s/update, speedup_vs_frozen_reference=1.2054 (slower). 131072: /tmp/r19/norm131072-time/time.json, status ok, 0.1896 s/update, 1.2328×. 262144: /tmp/r19/norm262144-time/time.json, status ok, 0.1895 s/update, 1.2334×. The two larger settings screened above the 1.2250× full-check stack; 262144 was selected.
- /tmp/r19/probe_norm_chunks.py found finite norms; 32768 and 131072 were bitwise in that case, while 262144 differed by 6.10e-5 in the total norm.
- Full verdict for 262144: /tmp/r19/check/check.json, status PASS, speedup 1.2367, stack tier N1. Candidate 0.1890 s/update, windows 1.887/1.893 s, range 0.0027, peak 59.30 GiB. Gate 2 had zero exact failures and no dtype/precision-mode changes; all six trajectory endpoints passed, with largest |difference|/bound about 1.5183. Promoted as the new best validated submission.
- Detached jobs: profile 154370; timing 156028/157623/159239; full check 160916 (actual PIDs launcher+1). Logs under /tmp/r19/. All completed successfully.
5.5 hRound 20 — cast-scale block-size sweep
- Start (GPU lock): 2026-11-06 22:27:46 UTC; end: 22:40:30 UTC.
- Profile of the 1.2367× stack: /tmp/r20/profile/profile.json, status ok, instrumented 0.2343 s/update. _cast_scale_kernel consumed 26.376 ms in 1200 calls over five updates. The accepted norm chunking reduced L2Norm launches from 825 to 210 and their time to 12.498 ms. Selected hotspot: the fixed 1024-element cast-scale block.
- Added off-by-default YOCO_OPT_CAST_BLOCK, changing only the Triton grid and BLOCK constant. /tmp/r20/probe_cast_blocks.py verified bitwise-equal fp32 outputs for 1024/2048/4096/8192 at odd and production-like sizes, establishing the isolated configuration as N0.
- Timing results: 512 /tmp/r20/cast512-time/time.json 0.1904 s/update, 1.2276×; 2048 /tmp/r20/cast2048-time/time.json 0.1892 s/update, 1.2354×; 4096 /tmp/r20/cast4096-time/time.json 0.1908 s/update, 1.2250×; 8192 /tmp/r20/cast8192-time/time.json 0.1896 s/update, 1.2328×. None exceeded the 1.2367× best full result; no override was promoted and no full check was run.
- Detached jobs: profile 166136; timing 172758/167972/169536/171127 (actual PIDs launcher+1). Logs under /tmp/r20/. All completed successfully. The submitted switches are unchanged.
5.8 hRound 21 — fused mixer residual + RMSNorm
- Start (GPU lock): 2026-11-06 22:41:40 UTC; end: 22:58:00 UTC.
- Profile of the promoted stack: /tmp/r21/profile/profile.json, status ok, instrumented 0.2354 s/update. Over five updates NCCL consumed 331.1 ms, GEMMs 315.5 ms, copies/casts 146.4 ms, and optimizer multi-tensor kernels 83.9 ms. Apex RMSNorm forward/backward kernels consumed 24.07 ms and bf16 adds 10.18 ms; selected the 26 mixer-residual/final-RMSNorm pairs as one memory-bound fusion class.
- The compiled FlashAttention dropout_layer_norm extension is not installed, so the candidate uses the existing flash_attn.ops.triton.layer_norm.rms_norm_fn residual+prenorm path. Added off-by-default YOCO_OPT_ADD_RMS_NORM; the original RMSNorm module is still called once at the same site, and its optional residual argument returns both normalized output and the residual sum. Dropout remains zero and no random draws are added.
- /tmp/r21/probe_add_rms.py at [1,2048,3072] verified bitwise-identical residual sums, unchanged CUDA RNG state, finite bf16 gradients, and deterministic-mode execution. Normalized outputs and gradients were not bitwise (output relative L2 difference 0.00339), so the enabled increment is N1.
- Timing: /tmp/r21/add-rms-time/time.json, status ok, 0.1925 s/update, speedup_vs_frozen_reference=1.2142, windows 1.921/1.930 s, range 0.0046. This is slower than the promoted 1.2367× full-check result, so the switch remains off and no full check was run.
- Exactness despite rejection: /tmp/r21/quick/check.json, status PASS, N1/NOT_BITWISE with zero exact failures, 1572 unresolved entries, and no dtype or precision-mode changes.
- Detached jobs: profile launcher 174352 (actual 174353); timing actual PID 176329; quick actual PID 178084. Logs /tmp/r21/profile-launch.log, add-rms-time-launch.log, and quick-launch.log. All completed successfully. The promoted switch file is unchanged.
6.3 hRound 22 — retune DDP buckets on the promoted stack
- Start (GPU lock): 2026-11-06 22:59:04 UTC; end: 23:27:10 UTC (including the full check and paired remeasurement).
- Profile: /tmp/r22/profile/profile.json, status ok, instrumented 0.2519 s/update. NCCL remained the largest kernel category at 333.7 ms/5 updates: 189.5 ms in 20 fp32 broadcasts and 141.8 ms in 235 bf16 gradient allreduces. GEMMs consumed 317.7 ms. Selected the communication-configuration hotspot to retest the round-3 bucket choice after all later changes.
- Existing-switch screening, with no code change: 75 MiB /tmp/r22/ddp75-time/time.json 0.1894 s/update, 1.2341×; 150 MiB /tmp/r22/ddp150-time/time.json 0.1879 s/update, 1.2439× (windows 1.877/1.881 s, range 0.0018); 200 MiB /tmp/r22/ddp200-time/time.json 0.1883 s/update, 1.2413×. All runs had status ok and peak allocated memory 59.30 GiB.
- 150 MiB quick check: /tmp/r22/quick/check.json, PASS, zero exact failures, the same 1466 unresolved entries and 1268 bitwise-identical entries as round 19, and no dtype/precision-mode changes. This supports an N0 increment over the promoted stack.
- 150 MiB full verdict: /tmp/r22/check/check.json, PASS, N1 stack, speedup 1.2067, candidate 0.1937 s/update, windows 1.949/1.925 s, range 0.0124. Gate 3 passed all endpoints with values identical to round 19 (largest |difference|/bound about 1.5183). Despite passing, this timing is slower than the retained 1.2367× full result, so it was not promoted.
- Paired follow-up: /tmp/r22/paired150/time.json gave 0.1885 s/update, 1.2399× versus frozen and 1.2366× versus its paired reference, windows 1.885/1.886 s. /tmp/r22/paired100/time.json gave 0.1893 s/update, 1.2347× versus frozen and 1.2340× paired, but its windows were noisier at 1.878/1.907 s. The 0.0026 paired advantage is unresolved against the contradictory full run; retain 100 MiB as the best validated setting. No submission or code change was made, so there was nothing to commit.
- Detached jobs: profile actual PID 180538; three-run sweep 182151; quick 186901; full check 189184; paired 150 194365; paired 100 197417. Logs under /tmp/r22/. All completed successfully.
6.5 hRound 23 — restore vectorized SwiGLU for the packed self gate
- Start (GPU lock): 2026-11-06 23:29:38 UTC; end: 23:38:58 UTC.
- Profile: /tmp/r23/profile/profile.json, status ok, instrumented 0.2295 s/update. The FFNs used vectorized SwiGLU (3.409 ms forward and 5.436 ms backward over five updates), while the 13 self-mixer gates/update used non-vectorized fallback kernels (1.593 ms forward and 2.143 ms backward). The packed projection's last-dimension gate view is noncontiguous; selected this missed existing fast path.
- Added off-by-default YOCO_OPT_CONTIGUOUS_G, applying .contiguous() only to the self-mixer gate immediately before its existing SwiGLU operation. This preserves the projection packing and all arithmetic; the enabled increment is N0.
- /tmp/r23/probe_contiguous_g.py used the production [1,2048,12312] packed layout and [1,2048,3072] gate/other inputs in deterministic mode. Forward output, packed input gradients, and the other input gradient were all bitwise identical and finite.
- Timing: /tmp/r23/contiguous-g-time/time.json, status ok, 0.1889 s/update, speedup_vs_frozen_reference=1.2373, windows 1.883/1.896 s, range 0.0066. Peak memory rose from 59.30 to 59.45 GiB because the contiguous gate is saved for backward. This is effectively tied with the 1.2367× best full result, so no full check was run and the switch remains off.
- Quick check: /tmp/r23/quick/check.json, PASS, zero exact failures, the same 1466 unresolved and 1268 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. This confirms the N0 increment at the judge's recorded sites.
- Detached jobs: profile actual PID 200513; timing 202181; quick 203823. Logs under /tmp/r23/. All completed successfully. The promoted switch file is unchanged.
6.9 hRound 24 — fused self-mixer RMSNorm + SwiGLU layout
- Start (GPU lock): 2026-11-06 23:40:47 UTC; end: 2026-11-07 00:02:39 UTC (including the full check).
- Profile: /tmp/r24/profile/profile.json, status ok, instrumented 0.2306 s/update. The self-mixer still used non-vectorized SwiGLU (1.596 ms forward and 2.128 ms backward over five updates) around Apex per-head RMSNorm and transpose/reshape layout work. Round 23 showed that a separate contiguous copy only breaks even; selected the whole norm/gate/layout region as one memory-bound fusion class.
- Added off-by-default YOCO_OPT_FUSED_GATE and kernel/fused_gate.py. A fixed-tile Triton forward computes the same non-affine per-head RMSNorm in fp32, rounds the normalized value to bf16, applies the existing SwiGLU formula, and writes directly in [batch,sequence,heads,dim] layout. The backward recomputes the normalized value from the saved bf16 input and fp32 reciprocal standard deviation, rounds the SwiGLU norm gradient to bf16 before the norm backward, and returns both input gradients. No atomics, RNG, or autotuning are used. The existing RMSNorm module call and output contract are retained; unsupported layouts/dtypes use the original path.
- /tmp/r24/probe_fused_gate.py checked small and production [1,24,2048,128] cases with packed, offset gate views. Outputs and both gradients were finite; the small case was bitwise, while production relative L2 differences were 1.88e-5 forward and at most 2.04e-5 for gradients. Enabled increment: N1. An initial compatibility error (tl.rsqrt unavailable in Triton 2.3) was fixed with fp32 1 / tl.sqrt before the successful probe.
- Screening: /tmp/r24/fused-gate-time/time.json, status ok, 0.1882 s/update, speedup_vs_frozen_reference=1.2419, windows 1.885/1.879 s, range 0.0028, peak 59.15 GiB (down 0.15 GiB). Quick check /tmp/r24/quick/check.json: PASS, zero exact failures, 1475 unresolved entries, no dtype/precision-mode changes.
- Full verdict: /tmp/r24/check/check.json, status PASS, speedup 1.2373, tier N1. Candidate 0.1889 s/update, windows 1.888/1.889 s, range 0.0010, peak 59.15 GiB. Gate 2 had zero exact failures; gate 3 passed all endpoints with largest |difference|/bound about 0.1960. Promoted YOCO_OPT_FUSED_GATE=1 as the new best validated submission.
- Detached jobs: profile actual PID 206159; timing 208026; quick 209633; full check 211986. Logs under /tmp/r24/. All completed successfully.
7.0 hRound 25 — fused-gate warp-count sweep
- Start (GPU lock): 2026-11-07 00:04:29 UTC; end: 00:09:49 UTC.
- Profile of the promoted fused-gate stack: /tmp/r25/profile/profile.json, status ok, instrumented 0.2296 s/update. _fused_gate_fwd consumed 2.292 ms and _fused_gate_bwd 2.510 ms over five updates; the removed non-vectorized SwiGLU kernels were absent. Selected the fixed four-warp launch configuration for these 128-wide rows.
- Added off-by-default YOCO_OPT_FUSED_GATE_WARPS, using one value for the existing forward and backward launches and retaining four as the default. No formula, layout, or precision change was made; reduction partitioning keeps the enabled variants at N1.
- /tmp/r24/probe_fused_gate.py was rerun with one and two warps. Both produced finite outputs and gradients at small and production shapes; production relative L2 differences versus the original Apex+jiterator path remained about 2e-5.
- Timing: one warp /tmp/r25/warps1-time/time.json 0.1897 s/update, 1.2321×; two warps /tmp/r25/warps2-time/time.json 0.1893 s/update, 1.2347×. Both are slower than the four-warp 1.2373× full result and 1.2419× screening result. No quick/full check was needed for the rejected variants, no override was promoted, and the default remains four.
- Detached jobs: profile actual PID 217328; two-run timing sweep 219170. Logs /tmp/r25/profile-launch.log and sweep-launch.log. All completed successfully.
7.4 hRound 26 — pipeline bucket-aligned Adam with ZeRO broadcasts
- Start (GPU lock): 2026-11-07 00:10:42 UTC; end: 00:33:01 UTC (including the full check).
- Profile: /tmp/r26/profile/profile.json, status ok, instrumented 0.2320 s/update. NCCL consumed 337.0 ms/5 updates, including 193.3 ms of fp32 broadcasts; Apex Adam consumed 71.46 ms in 410 calls over four recorded optimizer steps. Selected serialization between the full-shard Adam call and subsequent broadcasts.
- Added off-by-default YOCO_OPT_ADAM_BROADCAST=2 in fast_adam_pipeline.py, integrated with fast_overlap.py. The existing FusedAdamV2.step() is still entered once and increments each group's step once. Its fp32 parameter, gradient, exp_avg, and exp_avg_sq lists are split into bucket-aligned views; after each existing Apex multi-tensor call, matching source-rank broadcast chunks are launched asynchronously while later Adam chunks execute. Views are cached while the persistent base tensors are unchanged. All parameters and states are updated exactly once, and the original fp32 objects and optimizer layout are retained.
- The refresh path waits for the prelaunched broadcasts and reuses the existing local/remote bf16 casts and converted-parameter tracking. Guards retain the original path for multiple groups/devices, missing or incomplete buckets, non-fp32/noncontiguous tensors, or unexpected tensor lists. Collective chunk order is identical on all ranks; no RNG or new precision mode is used.
- Screening: /tmp/r26/adam-broadcast2-time/time.json, status ok, 0.1867 s/update, speedup_vs_frozen_reference=1.2519, windows 1.870/1.863 s, range 0.0039, peak 59.15 GiB. No pipeline fallback was logged. Quick check /tmp/r26/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as round 24.
- Full verdict: /tmp/r26/check/check.json, status PASS, speedup 1.2614, stack tier N1; increment N0. Candidate 0.1853 s/update, windows 1.850/1.856 s, range 0.0034, peak 59.15 GiB. Gate 2 had zero exact failures, and all six trajectory values were exactly identical to round 24 (largest |difference|/bound about 0.1960), establishing that the pipeline did not change recorded numerics. Promoted as the new best validated submission.
- Detached jobs: profile actual PID 222438; timing 224194; quick 225839; full check 228189. Logs under /tmp/r26/. All completed successfully.
7.8 hRound 27 — Adam-broadcast pipeline chunk-count sweep
- Start (GPU lock): 2026-11-07 00:34:33 UTC; end: 00:57:47 UTC (including the full check).
- Profile of the two-chunk pipeline: /tmp/r27/profile/profile.json, status ok, instrumented 0.2307 s/update. Broadcasts consumed 194.1 ms and Adam 73.26 ms over five updates. Optimizer-step range CPU time was 25.26 ms/4 steps and refresh range CPU time 24.44 ms, versus 5.44 and 81.48 ms before pipelining; the Apex kernel count remained 410. Selected pipeline granularity.
- Existing-switch timings, with no code change: three chunks /tmp/r27/chunks3-time/time.json 0.1845 s/update, 1.2668×; four /tmp/r27/chunks4-time/time.json 0.1818 s/update, 1.2856×; five /tmp/r27/chunks5-time/time.json 0.1940 s/update, 1.2048×; six /tmp/r27/chunks6-time/time.json 0.1813 s/update, 1.2892×; eight /tmp/r27/chunks8-time/time.json 0.1860 s/update, 1.2566×. Six was selected. Five's stable but pathological windows are consistent with the alignment sensitivity of bucket slices; eight also loses efficiency.
- Six-chunk quick check /tmp/r27/quick6/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the two-chunk pipeline, and no dtype/precision-mode changes.
- Full verdict: /tmp/r27/check6/check.json, status PASS, speedup 1.2928, stack tier N1; increment N0. Candidate 0.1808 s/update, windows 1.808/1.807 s, range 0.0006, peak 59.15 GiB. Gate 2 had zero exact failures and every trajectory value was exactly identical to rounds 24 and 26 (largest |difference|/bound about 0.1960). Promoted YOCO_OPT_ADAM_BROADCAST=6; the submitted switch file is updated and the git tree needed no new commit.
- Detached jobs: profile actual PID 233501; three/four-chunk sweep 235140; six/eight-chunk sweep 238428; five-chunk timing 241684; quick 243317; full check 245627. Logs under /tmp/r27/. All completed successfully.
8.1 hRound 28 — retest flat ZeRO-to-model casts after Adam pipelining
- Start (GPU lock): 2026-11-07 00:59:25 UTC; end: 01:16:04 UTC (including the full check).
- Profile of the promoted six-chunk pipeline: /tmp/r28/profile/profile.json, status ok, instrumented 0.2317 s/update. Over five updates, broadcasts consumed 201.095 ms in 60 calls, Apex Adam 80.240 ms in 420 calls, and direct bf16 casts 65.988 ms in 1505 calls. Selected the existing flat-cast path for retesting because the new optimizer overlap changed its exposure.
- Enabled the already committed, off-by-default YOCO_OPT_FLAT_CAST=1 in /tmp/r28/flat-cast.json; no source change was made. It allocates persistent flat bf16 storage, aliases the existing parameter objects without changing names/shapes, sorts copy metadata by actual bucket offset, and preserves the original fallback.
- Screening /tmp/r28/flat-cast-time/time.json: status ok, 0.1804 s/update, speedup_vs_frozen_reference=1.2956, windows 1.805/1.804 s, range 0.0009, peak 59.15 GiB. No flat-cast or Adam-pipeline fallback appeared. Quick /tmp/r28/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as round 27, and no dtype/precision-mode changes.
- Full verdict /tmp/r28/check/check.json: PASS, N1 stack, but speedup regressed to 1.2641. Candidate 0.1849 s/update, windows 1.846/1.851 s, range 0.0024; all trajectory values again exactly matched the promoted stack. Despite the favorable screening, this is slower than /tmp/r27/check6 at 1.2928×, so flat casting remains disabled and /submission/switches.json is unchanged. No source commit was needed.
- Detached jobs: profile actual PID 250945; timing 252574; quick 254218; full check 256533. Logs under /tmp/r28/. All completed successfully.
8.3 hRound 29 — compile self-retention cross-chunk helpers
- Start (GPU lock): 2026-11-07 01:18:34 UTC; end: 01:28:28 UTC.
- Profile of the promoted submission: /tmp/r29/profile/profile.json, status ok, instrumented 0.2256 s/update; GPU busy fraction 69.56%. GEMMs consumed 325.2 ms/5 updates, NCCL 394.2 ms, memcpy/cast 140.8 ms, and elementwise 39.3 ms. The self-retention inner_chunk was already compiled, but the sibling cross-chunk gate exponentials, dtype conversions, scaling, and surrounding matrix products remained eager. Selected that contained host/memory-bound region.
- Added off-by-default YOCO_OPT_CROSS_COMPILE=1 in gate_recurrent.py, compiling two pure helpers for the K/V decay product and the query/state product. The custom recurrence, formulas, explicit bf16 conversion points, call counts, RNG, and all other decoder paths remain unchanged. Inductor partitioning makes the enabled increment N1.
- Timing /tmp/r29/time/time.json: status ok, 0.1813 s/update, speedup_vs_frozen_reference=1.2892, windows 1.809/1.817 s, range 0.0044, startup 69.9 s, peak 59.14 GiB. This exactly matched the earlier six-chunk screening result and did not improve it. No compile/recompilation signature invalidated the windows.
- Quick /tmp/r29/quick/check.json: PASS, zero exact failures, 1482 unresolved and 1271 bitwise-identical entries, no dtype/precision-mode changes. Since there was no timing gain, no full check was run and the switch remains off. Commit 85d7a53 (Compile cross-chunk gate helpers (N1; time 1.2892x, disabled)).
- Existing-trace diagnosis for the next hotspot: 594 copy_bucket_to_grad device copies over five updates (about 119/update), carrying 15.878 GB and taking 9.839 ms, plus many host launches. DDP gradient bucket views have not yet been tested.
- Detached jobs: profile actual PID 261937; timing 263611; quick 265672. Logs under /tmp/r29/. All completed successfully.
8.6 hRound 30 — DDP gradient bucket views
- Start (GPU lock): 2026-11-07 01:29:50 UTC; end: 01:49:32 UTC (including the full check).
- Profile of the promoted submission: /tmp/r30/profile/profile.json, status ok, instrumented 0.2275 s/update and 72.13% GPU busy. NCCL consumed 436.3 ms/5 updates, GEMMs 325.5 ms, and memcpy/cast 141.2 ms. The previous trace's 594 copy_bucket_to_grad calls over five updates (about 119/update) selected DDP's existing gradient bucket-view path.
- Added off-by-default YOCO_OPT_GRAD_BUCKET_VIEW=1 in fast_ddp.py, passing gradient_as_bucket_view=True to the same PyTorch DDP constructor. Parameter gradients become views of the same reduction buckets, eliminating bucket-to-gradient copies without changing reduction math, gradient values, optimizer state, parameter objects, or RNG. The increment is N0 by unchanged downstream records.
- Screening /tmp/r30/time/time.json: status ok, 0.1790 s/update, speedup_vs_frozen_reference=1.3058, windows 1.804/1.776 s, range 0.0156. Quick /tmp/r30/quick/check.json: PASS, zero exact failures, exactly the promoted stack's 1475 unresolved and 1264 bitwise-identical entries, no dtype/precision-mode changes.
- Repeated paired timing /tmp/r30/paired/time.json: 0.1784 s/update, 1.3101× versus frozen and 1.3122× versus the adjacent reference (reference 0.2341 s/update), windows 1.790/1.778 s, range 0.0066. This resolved the initial small-gain screening.
- Full verdict /tmp/r30/check/check.json: status PASS, speedup 1.3065, stack N1; increment N0. Candidate 0.1789 s/update, windows 1.787/1.792 s, range 0.0028, peak 59.15 GiB. Gate 2 counts were unchanged and every trajectory value exactly matched rounds 24–28. Promoted YOCO_OPT_GRAD_BUCKET_VIEW=1; commit 679084e (Use DDP gradient bucket views (N0; full check 1.3065x)).
- Detached jobs: profile actual PID 268255; timing 269908; quick 271545; paired timing 273859; full check 276924. Logs under /tmp/r30/. All completed successfully.
8.9 hRound 31 — DDP static graph
- Start (GPU lock): 2026-11-07 01:51:59 UTC; end: 02:05:25 UTC.
- Profile of the promoted bucket-view stack: /tmp/r31/profile/profile.json, status ok, instrumented 0.2189 s/update. copy_bucket_to_grad calls fell from 594 to 0 and GPU memcpy time from 21.743 ms to 0.709 ms over five updates, confirming round 30's intended path. NCCL consumed 346.1 ms, GEMMs 317.1 ms, memcpy/cast 119.6 ms, and Apex Adam 92.0 ms. Selected the remaining DDP per-step graph management.
- Added off-by-default YOCO_OPT_DDP_STATIC_GRAPH=1, passing static_graph=True to the existing DDP constructor alongside gradient bucket views. The model graph and all parameters are static and used; no math or recipe setting changes. The enabled increment is N0 by unchanged records.
- Three valid timings did not resolve a gain: /tmp/r31/time/time.json 0.1773 s/update, 1.3183×, windows 1.773/1.773 s; /tmp/r31/paired/time.json 0.1800 s/update, 1.2985× frozen and 1.3022× adjacent-reference, windows 1.802/1.797 s; /tmp/r31/repeat/time.json 0.1785 s/update, 1.3094×, windows 1.789/1.781 s. Their mean is effectively tied with bucket views alone. No full check was justified and the switch remains off.
- Quick /tmp/r31/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. Commit 4b10a8d (Add optional DDP static graph (N0; time 1.2985-1.3183x, disabled)).
- Detached jobs: profile actual PID 282277; timing 283938; quick 285579; paired timing 287884; repeat 291014. Logs under /tmp/r31/. All completed successfully.
9.4 hRound 32 — bucket-aligned fp32 master-gradient casts
- Start (GPU lock): 2026-11-07 02:06:59 UTC; end: 02:36:04 UTC (including the full check).
- Profile of the promoted bucket-view stack: /tmp/r32/profile/profile.json, status ok, instrumented 0.2180 s/update. _cast_scale_kernel still consumed 26.375 ms in 1200 calls over five updates. NCCL consumed 359.7 ms, GEMMs 321.0 ms, memcpy/cast 119.5 ms, and Adam 92.6 ms. Selected per-parameter master-gradient conversion and launch overhead.
- Added off-by-default YOCO_OPT_FLAT_GRAD=1. The first sync groups contiguous p.grad views by actual DDP storage and offset, allocates flat fp32 destinations, and rebinds each existing p32.grad tensor object with set_; names, shapes, dtypes, optimizer keys, and values are preserved. Cast-scale work then runs once per contiguous bucket run. Later source signatures reuse cached views; a changed finalized layout rebuilds them and explicitly invalidates cached Adam chunk views before rebinding. Unsupported layouts retain the per-parameter path with a visible fallback.
- A CPU metadata probe verified gap splitting, exact offsets/values, tensor identities, cache reuse, and changed-layout handling. The first GPU timing /tmp/r32/time/time.json was ERROR, not a valid result: the initial single storage covers over 2³¹ elements and overflowed 32-bit Triton offsets, causing an illegal memory access. Added a 64-bit-offset kernel used only above that boundary. A locked functional probe crossed 2³¹ with correct boundary values; ordinary casts retain the original kernel.
- /tmp/r32/time64/time.json then ran validly at 1.3094×, but logged a layout change after update 0 and fell back, so it was not promotion evidence. Allowing rebuilds produced the expected two-stage layout: one initial storage, then 47 buckets for 240 parameters, with no later fallback. /tmp/r32/rebuild-time/time.json: 0.1732 s/update, 1.3495×, windows 1.748/1.715 s, range 0.0191, peak 70.74 GiB.
- Quick /tmp/r32/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. Repeat /tmp/r32/paired/time.json: 0.1730 s/update and 1.3510× versus frozen, confirming the gain. Its adjacent reference was slower (0.2379 s/update), so the inflated 1.3751× paired ratio is not used as the evidence.
- Full verdict /tmp/r32/check/check.json: status PASS, speedup 1.3733, stack N1; increment N0. Candidate 0.1702 s/update, windows 1.703/1.702 s, range 0.0008, peak allocated 70.74 GiB and reserved 77.98 GiB. Gate 2 counts and every trajectory value exactly matched the promoted stack. Promoted YOCO_OPT_FLAT_GRAD=1; commit 2b642fc (Batch master-gradient casts by DDP bucket (N0; full check 1.3733x)). Probe-generated __pycache__ was removed after the commit and the tree is clean.
- Detached jobs: profile actual PID 292686; failed timing 294474; 64-bit timing 296209; rebuild timing 297885; quick 299579; paired timing 301916; full check 305004. Logs under /tmp/r32/. All launched jobs terminated; the failed run was followed by the explicit functional probe and successful validation above.
9.8 hRound 33 — foreach local and remote weight casts
- Start (GPU lock): 2026-11-07 02:38:52 UTC; end: 02:58:22 UTC (including the full check).
- Profile of the promoted flat-gradient stack: /tmp/r33/profile/profile.json, status ok, instrumented 0.2135 s/update. Cast-scale calls fell from 1200 to 235 over five updates while their GPU time remained about 25 ms, confirming round 32 removed launch/host overhead. The remaining direct weight casts consumed 66.293 ms in 1505 calls. Selected local and remote fp32→bf16 weight refresh launches.
- Added off-by-default YOCO_OPT_WEIGHT_FOREACH=1 in fast_overlap.py, using torch._foreach_copy_ for the existing local owner casts and each already pipelined remote chunk. The older independent remote-only switch is preserved. No storage layout, parameter view, arithmetic, broadcast order, or optimizer behavior changes; the increment is N0.
- Two screenings were identical: /tmp/r33/time/time.json and /tmp/r33/paired/time.json, both 0.1710 s/update and speedup_vs_frozen_reference=1.3668; the paired ratio was 1.3661× with a normal 0.2336 s/update reference. Window ranges were 0.0038 and 0.0093. Quick /tmp/r33/quick/check.json: PASS, zero exact failures, unchanged 1475 unresolved and 1264 bitwise-identical entries, and no dtype/precision-mode changes.
- Full verdict /tmp/r33/check/check.json: PASS, N1 stack with an N0 increment, but speedup 1.3700, slightly below the promoted /tmp/r32/check at 1.3733×. Candidate 0.1706 s/update, windows 1.706/1.706 s. Every trajectory value was unchanged. The change is neutral-to-slower and remains off; /submission/switches.json is unchanged. Commit d706fa6 (Add optional foreach weight casts (N0; full check 1.3700x, disabled)).
- Detached jobs: profile actual PID 310358; timing 312028; quick 313684; paired timing 315994; full check 319088. Logs under /tmp/r33/. All completed successfully.
↑ YOCO