Kimi k3.1-pro 1.37×
Harness: Claude Code · switches: YOCO_DDP_BUCKET_MB=100 YOCO_OPT_OVERLAP_CAST=1 YOCO_OPT_FUSED_GRAD=1 YOCO_OPT_APEX_RMS_NORM=1 YOCO_OPT_BROADCAST_CHUNKS=2 YOCO_OPT_DEFER_STATS=1 YOCO_OPT_PACKED_SELF=2 YOCO_OPT_FLASH_CE=2 YOCO_OPT_NORM_CHUNK=262144 YOCO_OPT_FUSED_GATE=1 YOCO_OPT_ADAM_BROADCAST=6 YOCO_OPT_GRAD_BUCKET_VIEW=1 YOCO_OPT_FLAT_GRAD=1
- Judged
- 1.37×
- Tier
- N1
- Self-best
- 1.38×
- Rounds
- 33
- Hours
- 10.0
- API cost
- $33.5
- Input tokens · 99.2% cached
- 138 M
- Output tokens
- 0.4 M
- Requests
- 939
§ 02Key levers
- Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
- ✗ fp32 broadcast kept as-is, only chunked and pipelined with Adam (1.24× → 1.29×)
- Forward / backward CUDA graph
- ✗ only listed as a next step at the end
- Newer cuBLAS
- ✗ tried cuBLASLt workspace, no gain
- Retention / attention kernel swap
- ✗ instead merged five projections into one GEMM (1.196× → 1.213×)
- Other main levers
- Apex RMSNorm 1.09× → 1.16×; master-grad cast batched per DDP bucket 1.31× → 1.37×; fused cross-entropy; DDP bucket views (round 30)
§ ARaw trace
Raw work log33 sections
0.1 hRound 1 — baseline profile and foreach optimizer data movement
- Start (GPU lock): 2026-11-06 17:12:29 UTC. - Reference profile:/tmp/r1/profile/profile.jsonandprofile.md; statusok, instrumented 0.2903 s/update (not comparable withtime). GPU busy 75%. Per 5 profiled updates: NCCL 399.6 ms (including 185.2 ms of ten fp32 broadcasts), GEMM 327.6 ms, memcpy/cast 218.1 ms, elementwise 153.6 ms, Adam multi-tensor 91.6 ms. Optimizer range 241.4 ms over four recorded steps; ZeRO refresh range 187.5 ms over ten entries; clip range 54.5 ms over five steps. Hotspot class: optimizer memory traffic and launch overhead (missed existing foreach fast path). - Unchanged-tree judge timing:/tmp/r1/time/time.json, statusok, 0.2340 s/update,speedup_vs_frozen_reference=0.9988, windows 2.341/2.338 s, range 0.0012. Frozen reference 0.23373 s/update. - Change being screened:YOCO_OPT_FOREACH=1replaces per-parameter bf16↔fp32 copies and fp32 gradient scaling with PyTorch foreach operations. Same tensors, dtypes, scaling points, and optimizer layout; intended N0. A small live functional probe showed mixed foreach copies and scaling bitwise equal to the original operations. - Detached jobs: profile launcher PID 12710 (actual bench PID 12711), log/tmp/r1/profile-launch.log; timing launcher PID 14534 (actual bench PID 14535), log/tmp/r1/time-launch.log. Both completed successfully. - Candidate timing:/tmp/r1/foreach-time/time.json, statusok, 0.2346 s/update,speedup_vs_frozen_reference=0.9963, windows 2.353/2.340 s, range 0.0055. No gain; the change was disabled in the submitted switch file. - Exactness:/tmp/r1/foreach-quick/check.json, statusPASS, bitwise N0 (9616/9616 recorded items identical, zero exact failures). - End: 2026-11-06 17:26:45 UTC. Conclusion: foreach optimizer copies/scaling are exact but slightly slower in end-to-end timing; do not enable. The file remains behind an off-by-default switch as a screened fast-path inventory entry.
0.5 hRound 2 — NCCL protocol sweep
- Start (GPU lock): 2026-11-06 17:27:52 UTC; end: 2026-11-06 17:38:40 UTC. - Profile of current best (baseline behavior):/tmp/r2/profile/profile.json, statusok, instrumented 0.2824 s/update. NCCL remained the largest kernel category at 417.6 ms/5 updates: 224.7 ms bf16 gradient allreduce (595 calls) and 185.4 ms fp32 broadcast (10 calls), both RING_LL. Optimizer range 241.2 ms/4 recorded steps. Hotspot class: communication configuration/backend. -NCCL_PROTO=Simple:/tmp/r2/simple-time/time.json, statusok, 0.2337 s/update,speedup_vs_frozen_reference=1.0001, windows 2.336/2.338 s, range 0.0006. No resolved gain. -NCCL_PROTO=LL128:/tmp/r2/ll128-time/time.json, statusok, 0.2358 s/update,speedup_vs_frozen_reference=0.9912, windows 2.357/2.360 s, range 0.0014. Slower. - Simple exactness:/tmp/r2/simple-quick/check.json, statusPASS, bitwise N0 (9616/9616). No protocol was enabled in the submission. - Detached jobs: profile launcher 20150 (actual 20151); Simple timing 21638 (actual 21639); LL128 timing 23124 (actual 23125); Simple quick 24667 (actual 24668). Logs are the corresponding/tmp/r2/*-launch.logfiles. No code change was made for this sweep, so there was nothing to commit.
0.7 hRound 3 — DDP gradient bucket sizing
- Start (GPU lock): 2026-11-06 17:39:36 UTC; end: 2026-11-06 17:51:52 UTC. - Profile:/tmp/r3/profile/profile.json, statusok, instrumented 0.2889 s/update. NCCL 344.0 ms/5 updates: 191.9 ms in ten fp32 broadcasts and 151.1 ms in 595 bf16 allreduces (119/update). Gradient bucketing was the selected communication/launch hotspot. - Added off-by-defaultYOCO_DDP_BUCKET_MB; it overrides only the DDP constructor argument and leaves the parsed recipe configuration unchanged. - 50 MiB:/tmp/r3/ddp50-time/time.json, 0.2332 s/update,speedup_vs_frozen_reference=1.0023, windows 2.333/2.330 s. - 100 MiB:/tmp/r3/ddp100-time/time.json, 0.2308 s/update,speedup_vs_frozen_reference=1.0127, windows 2.311/2.306 s, range 0.0022; best of the sweep. - 250 MiB:/tmp/r3/ddp250-time/time.json, 0.2316 s/update,speedup_vs_frozen_reference=1.0092, windows 2.314/2.318 s. - 100 MiB exactness:/tmp/r3/ddp100-quick/check.json, statusPASS, bitwise N0 (9616/9616). Enabled 100 MiB in/submission/switches.jsonas the current best validated setting; alone it remains below the full check's 2% benefit threshold. - Detached jobs: profile launcher 26912 (actual 26913); timing launchers 28419/29934/31431 (actual 28420/29935/31432); quick launcher 32864 (actual 32865). Corresponding logs under/tmp/r3/.
0.9 hRound 4 — overlap local weight casting with ZeRO broadcasts
- Start (GPU lock): 2026-11-06 17:53:23 UTC; end: 2026-11-06 18:04:26 UTC (including the full check). - Profile of the 100 MiB bucket candidate:/tmp/r4/profile/profile.json, statusok, instrumented 0.2769 s/update. DDP allreduce calls fell from 119 to 47 per update. Ten fp32 broadcasts still cost 185.5 ms/5 updates; optimizer range 240.2 ms/4 steps. Selected hotspot: exposed fp32→bf16 weight conversion around the broadcasts. - Added off-by-defaultYOCO_OPT_OVERLAP_CAST. It launches the unchanged fp32 broadcasts asynchronously, performs the reference per-parameter fp32→bf16 copies for the rank-owned shard while they run, waits, then performs only the remaining remote-shard copies. Same dtypes, values, collectives, and optimizer layout; intended and measured N0. - Screening time:/tmp/r4/overlap-time/time.json, statusok, 0.2272 s/update,speedup_vs_frozen_reference=1.0287, windows 2.273/2.271 s, range 0.0009. - Full verdict:/tmp/r4/check/check.json, status PASS, speedup 1.0296, bitwise N0 (candidate 0.2270 s/update, windows 2.272/2.269 s; 9616/9616 recorded items identical, zero exact failures). Enabled both switches in the submission and committed the overlap change. - Detached jobs: profile launcher 35087 (actual 35088), overlap timing launcher 36599 (actual 36600), full-check launcher 38119 (actual 38120). Logs/tmp/r4/profile-launch.log,overlap-time-launch.log, andcheck-launch.log.
1.2 hRound 5 — fused gradient norm, cast, scale, and redundant-fill removal
- Start (GPU lock): 2026-11-06 18:05:57 UTC; end: 2026-11-06 18:20:39 UTC (including the full check). - Profile of the validated N0 stack:/tmp/r5/profile/profile.json, statusok, instrumented 0.2725 s/update. memcpy/cast 225.1 ms and elementwise 153.4 ms per five updates; clip range 55.7 ms/5 and optimizer range 249.0 ms/4. Selected hotspot: master-gradient memory passes and fills. - Functional probes: Apex multi-tensor norm on bf16 was bitwise equal to the norm after fp32 conversion on production shapes./tmp/r5/probe_cast_scale.pyshowed the Triton bf16→fp32 cast-scale bitwise equal to separate PyTorch operations on all tested shapes. - Added off-by-defaultYOCO_OPT_FUSED_GRAD: norm directly from bf16 gradients, Triton cast with the final clipping/sample-size scale, and no zero-fill of master gradients that are completely overwritten. Missing-gradient and other-layout fallbacks are retained. - Screening time:/tmp/r5/fused-grad-time/time.json, statusok, 0.2197 s/update,speedup_vs_frozen_reference=1.0639, windows 2.195/2.198 s, range 0.0015. - Full verdict:/tmp/r5/check/check.json, status PASS, speedup 1.0907, tier N1. Candidate 0.2143 s/update, windows 2.151/2.135 s, range 0.0073 (valid). Gate 2 wasNOT_BITWISEwith zero exact-item failures; gate 3 passed all six endpoints (largest |difference|/bound about 0.3975). Promoted the switch and committed the change. This supersedes the N0 full check as the best validated submission. - Detached jobs: profile launcher 41812 (actual 41813), timing launcher 43535 (actual 43536), full-check launcher 45060 (actual 45061). Logs/tmp/r5/profile-launch.log,fused-grad-time-launch.log, andcheck-launch.log.
1.5 hRound 6 — packed FFN input projections
- Start (GPU lock): 2026-11-06 18:22:47 UTC; end: 18:38:21 UTC (including the full check and comparison timing). - Profile of the current best:/tmp/r6/profile/profile.json, statusok, instrumented 0.2626 s/update. GEMMs consumed 326.2 ms/5 updates; the self and cross FFN ranges consumed 36.39 and 32.41 ms/5 updates. Selected hotspot: pairs of same-inputfc1/gateGEMMs and their launch overhead. - Added off-by-defaultYOCO_OPT_PACKED_FFN: concatenate the two existing weight tensors, run oneF.linear, split the result, and preserve the original parameters, names, shapes, optimizer entries, and autograd gradient routing. Merged-GEMM arithmetic makes the enabled path N1. - Screening time:/tmp/r6/packed-ffn-time/time.json, statusok, 0.2180 s/update,speedup_vs_frozen_reference=1.0722, windows 2.183/2.177 s, range 0.0024. Peak allocated memory increased from 61.75 to 64.19 GiB because of the temporary packed weights. - Full verdict:/tmp/r6/check/check.json, status PASS, speedup 1.0687, tier N1. Candidate 0.2187 s/update, windows 2.194/2.181 s. Gate 2 had zero exact failures; gate 3 passed all six endpoints (largest |difference|/bound about 1.4968). This was slower than the round-5 full check at 1.0907×. - Same-tree comparison with packing disabled:/tmp/r6/unpacked-time/time.json, statusok, 0.2151 s/update,speedup_vs_frozen_reference=1.0866, windows 2.151/2.152 s, range 0.0004. Packing is therefore not promoted; the submitted switches remain unchanged and the implementation is committed disabled as screened fast-path inventory. - Detached jobs: logs/tmp/r6/profile-launch.log,packed-ffn-time-launch.log,check-launch.log(launcher 53155, actual 53156), andunpacked-time-launch.log(launcher 58167, actual 58168). All completed successfully.
1.8 hRound 7 — Apex fused RMSNorm
- Start (GPU lock): 2026-11-06 18:42:24 UTC; end: 18:56:33 UTC (including the full check). - Profile of the round-5 validated stack:/tmp/r7/profile/profile.json, statusok, instrumented 0.3033 s/update. NCCL 472.8 ms, GEMM 322.1 ms, elementwise 71.3 ms, and reductions 18.8 ms per five updates. Selected hotspot: RMSNorm's separate fp32 reduction, conversion, affine, and backward kernels. -/tmp/r7/probe_rms.pychecked production shapes 128 and 3072 with and without affine weights. Apex retained bf16 outputs, input gradients, and weight gradients. Affine forward outputs were bitwise equal in this probe; non-affine outputs and backward reductions were not, so the enabled path is N1. - Added off-by-defaultYOCO_OPT_APEX_RMS_NORM, selecting Apex's fused RMSNorm/RMSNorm-affine implementation for CUDA tensors and retaining the original formula as the fallback. - Screening time:/tmp/r7/apex-rms-time/time.json, statusok, 0.1974 s/update,speedup_vs_frozen_reference=1.1840, windows 1.973/1.975 s, range 0.0011. Peak allocated memory 60.32 GiB. - Full verdict:/tmp/r7/check/check.json, status PASS, speedup 1.1559, tier N1. Candidate 0.2022 s/update, windows 2.024/2.021 s, range 0.0014. Gate 2 had zero exact failures; gate 3 passed all six endpoints (largest |difference|/bound about 0.6607). Promoted the switch; this is the new best validated submission. - Detached jobs: profile launcher 59783 (actual bench 59785), log/tmp/r7/profile-launch.log; timing launcher 61399 (actual 61400), logapex-rms-time-launch.log; full-check launcher 62937 (actual 62938), logcheck-launch.log. All completed successfully.
2.1 hRound 8 — chunk-pipelined ZeRO broadcasts and remote casts
- Start (GPU lock): 2026-11-06 18:59:51 UTC; end: 19:17:03 UTC (including the full check). - Profile of the Apex RMSNorm stack:/tmp/r8/profile/profile.json, statusok, instrumented 0.2424 s/update. Ten fp32 broadcasts consumed 191.5 ms and weight-cast copies 63.6 ms per five updates; refresh remained the largest optimizer-side range. Selected hotspot: remote casts left exposed after the two full-buffer broadcasts. - Added off-by-defaultYOCO_OPT_BROADCAST_CHUNKS. For values above one, each OSS bucket broadcast is split into interleaved chunks. The unchanged local casts overlap the launched collectives; after each remote chunk completes, its existing bf16 parameters are copied from the corresponding fp32 master-view slices while later broadcasts remain in flight. Parameter names, values, dtypes, optimizer state, and RNG are unchanged; this increment is N0. Noncontiguous and broadcast-fp16 layouts retain the prior path. - Two chunks:/tmp/r8/chunks2-time/time.json, statusok, 0.1951 s/update,speedup_vs_frozen_reference=1.1980, windows 1.950/1.953 s, range 0.0019. - Four chunks:/tmp/r8/chunks4-time/time.json, statusok, 0.1952 s/update,speedup_vs_frozen_reference=1.1974, windows 1.952/1.952 s. Neutral versus two, so two was selected to minimize collective count. - Full verdict for two chunks:/tmp/r8/check/check.json, status PASS, speedup 1.1943, stack tier N1. Candidate 0.1957 s/update, windows 1.952/1.963 s, range 0.0059. Gate 2 had zero exact failures and the same 1436 unresolved tensor differences as round 7; all six trajectory endpoints were identical to round 7, confirming that this increment did not alter the recorded numerics. PromotedYOCO_OPT_BROADCAST_CHUNKS=2; this is the new best validated submission. - Detached jobs: profile launcher 67989 (actual 67990); two-chunk timing launcher 69549 (actual 69550); four-chunk timing launcher 71113 (actual 71114); full-check launcher 72625 (actual 72626). Logs under/tmp/r8/. All completed successfully.
2.2 hRound 9 — persistent packed-FFN weight views
- Start (GPU lock): 2026-11-06 19:19:08 UTC; end: 19:24:12 UTC. - Profile of the validated two-chunk stack:/tmp/r9/profile/profile.json, statusok, instrumented 0.2339 s/update. GEMMs consumed 325.7 ms/5 updates in 925 NT, 855 NN, and 860 TN main-kernel calls; FFN ranges consumed 67.4 ms/5. Selected hotspot: repeated same-input FFN GEMMs, this time without round 6's per-forward concatenation. - Revised the disabledYOCO_OPT_PACKED_FFNpath to alias the originalfc1andgateparameters into adjacent shared storage after initialization and device/dtype conversion. A custom autograd view spans both weights and splits the packed gradient back to the original parameters. No packed buffer is registered and no concatenation occurs in the training loop. -/tmp/r9/probe_packed_view.pyconfirmed shared adjacent storage, unchanged parameter names/shapes, finite bf16 parameter gradients, and correct forward/backward behavior. The small-case forward was bitwise; backward differed slightly, consistent with N1 merged-GEMM arithmetic. - Timing:/tmp/r9/packed-view-time/time.json, statusok, 0.1980 s/update,speedup_vs_frozen_reference=1.1805, windows 1.963/1.998 s, range 0.0173, peak allocated memory 60.32 GiB. This is slower than the current 1.1943× full-check candidate and shows no memory benefit. No full check was run; the path remains disabled and the submitted switches are unchanged. - Detached jobs: profile launcher 77567 (actual 77568), log/tmp/r9/profile-launch.log; timing launcher 79372 (actual 79373), logpacked-view-time-launch.log. Both completed successfully.
2.6 hRound 10 — defer unused per-update statistics
- Start (GPU lock): 2026-11-06 19:27:20 UTC; end: 19:46:58 UTC (including the full check). - Profile:/tmp/r10/profile/profile.json, statusok, instrumented 0.2478 s/update. Apex Adam was already multi-tensor fused (71.4 ms/4 recorded steps), so no standard optimizer substitution was indicated. The trace instead exposed 225aten::itemcalls with 36.1 ms host time; the longest calls followed the optimizer and came from redundant finite checking and metric rounding. - The fixed train loop only testslog_output is not Noneand independently computes actual interval logs fromtrain_inner. Added off-by-defaultYOCO_OPT_DEFER_STATS:_reduce_and_log_statsnow returns a lazy dictionary that performs the original smoothed-value computation on any access, while normal training leaves it unconsumed. The second gradient finite check is omitted becausetrain_stephas already raised for non-finite norms. Metric aggregation and actual printed statistics are unchanged; this increment is N0 for the valid finite training path. - Screening time:/tmp/r10/defer-stats-time/time.json, statusok, 0.1946 s/update,speedup_vs_frozen_reference=1.2011, windows 1.942/1.949 s, range 0.0037. - Full verdict:/tmp/r10/check/check.json, status PASS, speedup 1.1955, stack tier N1. Candidate 0.1955 s/update, windows 1.958/1.953 s, range 0.0026. Gate 2 had zero exact failures and the same 1436 differences; every trajectory endpoint was identical to rounds 7–8. Promoted the switch as the best validated configuration, a small gain over round 8's 1.1943×. - Detached jobs: profile launcher 80982 (actual 80983), timing launcher 82716 (actual 82717), full-check launcher 84241 (actual 84242). Logs under/tmp/r10/. All completed successfully.
2.8 hRound 11 — cuBLASLt workspace sweep
- Start (GPU lock): 2026-11-06 19:48:59 UTC; end: 19:57:21 UTC. - Profile of the deferred-statistics stack:/tmp/r11/profile/profile.json, statusok, instrumented 0.2648 s/update. GEMMs consumed 322.1 ms/5 updates and were the selected library-configuration hotspot. The previous round's longaten::itementries were absent. - 32768 KiB workspace:/tmp/r11/cublaslt32-time/time.json, statusok, 0.1998 s/update,speedup_vs_frozen_reference=1.1698, windows 1.999/1.996 s. Clearly slower. - 1024 KiB workspace:/tmp/r11/cublaslt1-time/time.json, statusok, 0.1942 s/update,speedup_vs_frozen_reference=1.2036, windows 1.943/1.941 s. This matches default-path variation and is consistent with the usual 1 MiB default. - 512 KiB workspace:/tmp/r11/cublaslt05-time/time.json, statusok, 0.1944 s/update,speedup_vs_frozen_reference=1.2023, windows 1.942/1.946 s. Neutral. No setting showed a resolved gain over the current default path, so no workspace override was added and no full check was run. - Detached jobs: profile launcher 89219 (actual 89220); timing launchers 90745/92237/93758 (actual 90746/92238/93759). Logs under/tmp/r11/. All completed successfully. No code changed, so there was nothing to commit.
3.1 hRound 12 — packed self-decoder projections
- Start (GPU lock): 2026-11-06 19:59:55 UTC; end: 20:18:28 UTC (including the full check). - Profile:/tmp/r12/profile/profile.json, statusok, instrumented 0.2339 s/update. GEMMs consumed 322.8 ms/5 updates; self-mixer ranges consumed 72.67 ms GPU and 133.22 ms CPU per five updates. Selected hotspot: four large same-input q/k/v/g projections plus the small same-inputgtprojection. - Added off-by-defaultYOCO_OPT_PACKED_SELF. Original projection parameters are aliased into adjacent storage after initialization and device/dtype conversion; a custom autograd view splits packed weight gradients back to the original parameters. Mode 1 packs q/k/v/g and leavesgtseparate; mode 2 packs all five. No fused buffer is registered and no concatenation occurs during updates. Merged-GEMM arithmetic is N1. -/tmp/r12/probe_packed_self.pyconfirmed shared storage, unchanged parameter names/shapes, and finite bf16 forward/backward tensors for mode 1. Both modes subsequently completed judge timing with normal optimizer-step counts. - Four-way timing:/tmp/r12/packed-self-time/time.json, statusok, 0.1925 s/update,speedup_vs_frozen_reference=1.2142, windows 1.922/1.927 s. Five-way timing:/tmp/r12/packed-self5-time/time.json, statusok, 0.1919 s/update,speedup_vs_frozen_reference=1.2180, windows 1.916/1.923 s. Mode 2 was slightly better. - Full verdict for mode 2:/tmp/r12/check/check.json, status PASS, speedup 1.2129, tier N1. Candidate 0.1927 s/update, windows 1.920/1.934 s, range 0.0069; peak allocated memory 60.63 GiB. Gate 2 had zero exact failures; gate 3 passed all endpoints (largest |difference|/bound about 1.7947). PromotedYOCO_OPT_PACKED_SELF=2as the new best validated submission. - Detached jobs: profile launcher 95359 (actual 95360); timing launchers 97372/98937 (actual 97373/98938); full-check launcher 100521 (actual 100522). Logs under/tmp/r12/. All completed successfully.
3.4 hRound 13 — Adam-only multi-tensor chunk sizes
- Start (GPU lock): 2026-11-06 20:24:42 UTC; end: 20:34:11 UTC. - Profile of the packed-self stack:/tmp/r13/profile/profile.json, statusok, instrumented 0.2304 s/update. Apex Adam consumed 71.34 ms of kernel time in 410 calls over five updates (82/update), 7.0% of kernel time. Selected hotspot: Adam launch chunking; gradient-norm reductions were explicitly left unchanged. - Added off-by-defaultYOCO_OPT_ADAM_CHUNK, temporarily setting Apex'smulti_tensor_applier.chunk_sizeonly around the originalFusedAdamV2.stepand restoring it infinally. The Adam kernel, elementwise formula, states, and optimizer-step count are unchanged. - 131072 elements:/tmp/r13/adam131072-time/time.json, statusok, 0.1948 s/update,speedup_vs_frozen_reference=1.1998, windows 1.941/1.955 s. 262144 elements:/tmp/r13/adam262144-time/time.json, statusok, 0.1940 s/update,speedup_vs_frozen_reference=1.2048, windows 1.945/1.936 s. Both are slower than the promoted 1.2129× full-check result, so neither was promoted and no full check was run. -/tmp/r13/probe_adam_chunk.pycompared default and 262144-element chunks for two steps on varied tensor sizes, including cross-chunk shapes: master weights,exp_avg, andexp_avg_sqwere bitwise identical after both steps. This establishes the isolated override as N0 despite its negative timing. - Detached jobs: profile launcher 105720 (actual 105721); timing launchers 107403/109019 (actual 107404/109020). Logs under/tmp/r13/. All completed successfully. The experimental switch remains disabled.
3.5 hRound 14 — foreach remote master-weight casts
- Start (GPU lock): 2026-11-06 20:36:54 UTC; end: 20:44:01 UTC. - Profile:/tmp/r14/profile/profile.json, statusok, instrumented 0.2433 s/update. Copy/cast kernels consumed 152.1 ms and NCCL 324.1 ms over five updates; the bf16 direct-copy kernel alone consumed 72.93 ms in 1490 calls. Selected hotspot: the per-parameter remote fp32→bf16 casts after chunked ZeRO broadcasts. - Added off-by-defaultYOCO_OPT_REMOTE_FOREACH, collecting the same source/destination slices and usingtorch._foreach_copy_instead of onecopy_launch per parameter. Broadcast chunking and all other cast paths are unchanged. - Candidate timing:/tmp/r14/remote-foreach-time/time.json, statusok, 0.1932 s/update,speedup_vs_frozen_reference=1.2098, windows 1.930/1.933 s. Same-session control with the switch off:/tmp/r14/control-time/time.json, statusok, 0.1927 s/update,speedup_vs_frozen_reference=1.2129, windows 1.931/1.922 s. The candidate showed no gain and was marginally slower, so it remains disabled and no full check was run. -/tmp/r14/probe_remote_foreach.pyverified bitwise-equal fp32→bf16 outputs for 200000, 65537, 527, and 1 elements. The isolated path is N0 but not beneficial. - Detached jobs: profile launcher 110729 (actual 110730); timing launchers 112267/113841 (actual 112268/113842). Logs under/tmp/r14/. All completed successfully. The promoted switch file is unchanged.
3.7 hRound 15 — rotary tile-size sweep
- Start (GPU lock): 2026-11-06 20:44:46 UTC; end: 20:54:55 UTC. - Profile:/tmp/r15/profile/profile.json, statusok, instrumented 0.2305 s/update. Two rotary kernel variants consumed 4.886 ms (270 calls) and 2.196 ms (130 calls) over five updates; self-mixer ranges consumed 64.73 ms GPU and 120.32 ms CPU. Selected hotspot: fixed four-row tiles in the existing interleaved rotary kernel. - Added off-by-defaultYOCO_OPT_ROTARY_BLOCK_M, retaining the original per-element Triton formula, q/k layouts, autograd path, and backward while allowing larger sequence tiles./tmp/r15/probe_rotary_tiles.pyverified bitwise-identical production-shape forward outputs and input gradients for the default and 8/16/32-row tiles, establishing the isolated configuration as N0. - Eight rows:/tmp/r15/rotary8-time/time.json, statusok, 0.1939 s/update,speedup_vs_frozen_reference=1.2054, windows 1.943/1.934 s. Sixteen rows:/tmp/r15/rotary16-time/time.json, statusok, 0.1947 s/update, 1.2005×, windows 1.937/1.957 s. Thirty-two rows:/tmp/r15/rotary32-time/time.json, statusok, 0.1965 s/update, 1.1895×, windows 1.963/1.967 s. All were slower than the 1.2129× control; larger tiles monotonically worsened timing. No setting was promoted and no full check was run. - Detached jobs: profile launcher 115456 (actual profiler completed before a separate wait); timing launchers 117205/118774/120332 (actual 117206/118775/120334). Logs under/tmp/r15/. All completed successfully. The experimental override remains disabled.
4.5 hRound 16 — flat ZeRO bucket-to-model casts
- Start (GPU lock): 2026-11-06 21:00:35 UTC; end: 21:38:02 UTC (including the full check). - Profile:/tmp/r16/profile/profile.json, statusok, instrumented 0.2377 s/update. The preceding profile measured 72.9 ms in 1490 rank-0 remote-cast kernels over five updates. Selected hotspot: per-parameter fp32-bucket→bf16-model casts. - Added off-by-defaultYOCO_OPT_FLAT_CAST. After the first optimizer step, it mirrors trainable bf16 parameters into one persistent allocation without changing parameter objects, names, shapes, values, dtypes, or fp32 masters. It sorts metadata by actual ZeRO bucket offsets, validates complete coverage, and merges only source/destination-contiguous runs. Later local and remote chunk casts use those runs; unsupported layouts log a fallback reason and use the original path. - The first contiguous-rank-span version fell back because greedy ZeRO ownership is not contiguous (/tmp/r16/flat-cast-time, 1.2104×). A run-based version also fell back because fairseq parameter order differs from bucket order (/tmp/r16/flat-runs-time, 1.2180×; profile still showed 1490 casts). An intentional diagnostic failure identified the offset mismatch in/tmp/r16/flat-debug-time/time.json. Sorting by bucket offset fixed activation; these intermediate timings were not accepted as fast-path evidence. - Active bucket-sorted timing:/tmp/r16/flat-sorted-time/time.json, statusok, 0.1915 s/update,speedup_vs_frozen_reference=1.2205. Quick check/tmp/r16/quick/check.json: PASS, zero exact failures, same 1463 unresolved entries as round 12 and no new dtype/precision changes. - Full verdict:/tmp/r16/check/check.json, status PASS, speedup 1.2117, stack tier N1. Candidate 0.1929 s/update, windows 1.932/1.927 s, range 0.0026; peak memory 60.63 GiB. All six trajectory endpoints were numerically identical to round 12, confirming the increment as N0. The full timing is slightly worse than round 12's 1.2129×, so the switch remains disabled and the promoted configuration is unchanged. - Detached jobs: profile 122003; initial timing 123648; fallback profiles 125236/128399; guard diagnostic 130015; active timing 131636; quick check 133234; full check 135492 (actual PIDs launcher+1). Logs under/tmp/r16/. All intended runs completed; the diagnostic alone returned its planned ERROR with the guard reason.
4.8 hRound 17 — FlashAttention fused cross-entropy
- Start (GPU lock): 2026-11-06 21:40:16 UTC; end: 21:57:39 UTC (including the full check). - Profile:/tmp/r17/profile/profile.json, statusok, instrumented 0.2303 s/update. Log-softmax/NLL consumed 8.8 ms over five updates and materialized fp32 log-probabilities; RMSNorm forward consumed 7.4 ms and residual adds were a smaller ceiling. Selected the existing FlashAttention fused-loss fast path rather than writing a residual+norm kernel. - Added off-by-defaultYOCO_OPT_FLASH_CEincriterions/fast_cross_entropy.py. It replaces log-softmax + NLL with FlashAttention's Triton cross-entropy, preserving fp32 accumulation and scalar loss, target selection, ignore index, reduction, and fairseq's outer sample-size normalization. Mode 1 uses a separate gradient allocation; mode 2 uses the library's supported in-place logits backward. Other inputs fall back to the original criterion. The changed reduction/kernel rounding is N1. -/tmp/r17/probe_flash_ce.pyat 2048×50257 found finite fp32 loss and bf16 input gradients, absolute summed-loss difference 0.001953125, maximum gradient difference 1.91e-6, and relative gradient L2 difference 2.32e-7. - Out-of-place timing:/tmp/r17/flash-ce-time/time.json, statusok, 0.1895 s/update,speedup_vs_frozen_reference=1.2334, windows 1.892/1.898 s, peak 59.30 GiB. In-place timing:/tmp/r17/flash-ce-inplace-time/time.json, statusok, 0.1891 s/update, 1.2360×, windows 1.887/1.896 s, peak 59.30 GiB. Mode 2 was slightly faster and was selected for validation. - Full verdict for mode 2:/tmp/r17/check/check.json, status PASS, speedup 1.2250, stack tier N1. Candidate 0.1908 s/update, windows 1.904/1.913 s, range 0.0046. Gate 2 had zero exact failures, no dtype/precision-mode changes, and 1467 unresolved entries. Gate 3 passed every endpoint; largest |difference|/bound about 1.1961. PromotedYOCO_OPT_FLASH_CE=2as the new best validated submission. - Detached jobs: profile 140572; timing 142330/143940; full check 145557 (actual PIDs launcher+1). Logs under/tmp/r17/. All completed successfully.
4.9 hRound 18 — packed shared cross-decoder K/V projections
- Start (GPU lock): 2026-11-06 21:59:30 UTC; end: 22:05:51 UTC. - Profile of the fused-loss stack:/tmp/r18/profile/profile.json, statusok, instrumented 0.2312 s/update. Selected hotspot: the two same-input 1024-wide K/V projections that build the shared cross-decoder cache. - Added off-by-defaultYOCO_OPT_PACKED_KV, using persistent adjacent K/V weight storage and a custom autograd view that returns the original two gradient slices. Parameter objects, names, shapes, and optimizer entries remain unchanged; merged-GEMM arithmetic is N1. -/tmp/r18/probe_packed_kv.pyverified shared storage, unchanged parameter contract, finite output and all gradients. Its forward was bitwise and input-gradient relative L2 difference was 2.70e-4. The first probe incorrectly changed the global switch before the reference forward; fixing the probe to restore mode 0 for the reference resolved the unintended packed-view error. - Timing:/tmp/r18/packed-kv-time/time.json, statusok, 0.1955 s/update,speedup_vs_frozen_reference=1.1955, windows 1.966/1.944 s, range 0.0117, peak 59.30 GiB. This is clearly slower than the 1.2250× full-check stack, so the path remains disabled and no correctness check was run. - Detached jobs: profile 150792 (actual 150793); timing 152721 (actual 152722). Logs under/tmp/r18/. Both completed successfully. The promoted switch file is unchanged.
5.2 hRound 19 — gradient-norm chunk-size sweep
- Start (GPU lock): 2026-11-06 22:07:45 UTC; end: 22:25:45 UTC (including the full check). - Profile:/tmp/r19/profile/profile.json, statusok. The preceding traces showed about 165 bf16 L2Norm kernels per update. Selected hotspot: fairseq's fixed 65536-element gradient-norm chunks, independently of the rejected Adam-only chunk override. - Added off-by-defaultYOCO_OPT_NORM_CHUNKinfast_grad.py. For multiple ordinary gradients it calls fairseq's existingmulti_tensor_total_normwith the requested chunk size and preservesaggregate_norm_fn; single, expert, base-expert, and sharded layouts retain the original wrapper. Adam chunking is untouched. Changed reduction partitioning is N1. - 32768 elements:/tmp/r19/norm32768-time/time.json, statusok, 0.1939 s/update,speedup_vs_frozen_reference=1.2054(slower). 131072:/tmp/r19/norm131072-time/time.json, statusok, 0.1896 s/update, 1.2328×. 262144:/tmp/r19/norm262144-time/time.json, statusok, 0.1895 s/update, 1.2334×. The two larger settings screened above the 1.2250× full-check stack; 262144 was selected. -/tmp/r19/probe_norm_chunks.pyfound finite norms; 32768 and 131072 were bitwise in that case, while 262144 differed by 6.10e-5 in the total norm. - Full verdict for 262144:/tmp/r19/check/check.json, status PASS, speedup 1.2367, stack tier N1. Candidate 0.1890 s/update, windows 1.887/1.893 s, range 0.0027, peak 59.30 GiB. Gate 2 had zero exact failures and no dtype/precision-mode changes; all six trajectory endpoints passed, with largest |difference|/bound about 1.5183. Promoted as the new best validated submission. - Detached jobs: profile 154370; timing 156028/157623/159239; full check 160916 (actual PIDs launcher+1). Logs under/tmp/r19/. All completed successfully.
5.5 hRound 20 — cast-scale block-size sweep
- Start (GPU lock): 2026-11-06 22:27:46 UTC; end: 22:40:30 UTC. - Profile of the 1.2367× stack:/tmp/r20/profile/profile.json, statusok, instrumented 0.2343 s/update._cast_scale_kernelconsumed 26.376 ms in 1200 calls over five updates. The accepted norm chunking reduced L2Norm launches from 825 to 210 and their time to 12.498 ms. Selected hotspot: the fixed 1024-element cast-scale block. - Added off-by-defaultYOCO_OPT_CAST_BLOCK, changing only the Triton grid andBLOCKconstant./tmp/r20/probe_cast_blocks.pyverified bitwise-equal fp32 outputs for 1024/2048/4096/8192 at odd and production-like sizes, establishing the isolated configuration as N0. - Timing results: 512/tmp/r20/cast512-time/time.json0.1904 s/update, 1.2276×; 2048/tmp/r20/cast2048-time/time.json0.1892 s/update, 1.2354×; 4096/tmp/r20/cast4096-time/time.json0.1908 s/update, 1.2250×; 8192/tmp/r20/cast8192-time/time.json0.1896 s/update, 1.2328×. None exceeded the 1.2367× best full result; no override was promoted and no full check was run. - Detached jobs: profile 166136; timing 172758/167972/169536/171127 (actual PIDs launcher+1). Logs under/tmp/r20/. All completed successfully. The submitted switches are unchanged.
5.8 hRound 21 — fused mixer residual + RMSNorm
- Start (GPU lock): 2026-11-06 22:41:40 UTC; end: 22:58:00 UTC. - Profile of the promoted stack:/tmp/r21/profile/profile.json, statusok, instrumented 0.2354 s/update. Over five updates NCCL consumed 331.1 ms, GEMMs 315.5 ms, copies/casts 146.4 ms, and optimizer multi-tensor kernels 83.9 ms. Apex RMSNorm forward/backward kernels consumed 24.07 ms and bf16 adds 10.18 ms; selected the 26 mixer-residual/final-RMSNorm pairs as one memory-bound fusion class. - The compiled FlashAttentiondropout_layer_normextension is not installed, so the candidate uses the existingflash_attn.ops.triton.layer_norm.rms_norm_fnresidual+prenorm path. Added off-by-defaultYOCO_OPT_ADD_RMS_NORM; the originalRMSNormmodule is still called once at the same site, and its optional residual argument returns both normalized output and the residual sum. Dropout remains zero and no random draws are added. -/tmp/r21/probe_add_rms.pyat[1,2048,3072]verified bitwise-identical residual sums, unchanged CUDA RNG state, finite bf16 gradients, and deterministic-mode execution. Normalized outputs and gradients were not bitwise (output relative L2 difference 0.00339), so the enabled increment is N1. - Timing:/tmp/r21/add-rms-time/time.json, statusok, 0.1925 s/update,speedup_vs_frozen_reference=1.2142, windows 1.921/1.930 s, range 0.0046. This is slower than the promoted 1.2367× full-check result, so the switch remains off and no full check was run. - Exactness despite rejection:/tmp/r21/quick/check.json, status PASS, N1/NOT_BITWISE with zero exact failures, 1572 unresolved entries, and no dtype or precision-mode changes. - Detached jobs: profile launcher 174352 (actual 174353); timing actual PID 176329; quick actual PID 178084. Logs/tmp/r21/profile-launch.log,add-rms-time-launch.log, andquick-launch.log. All completed successfully. The promoted switch file is unchanged.
6.3 hRound 22 — retune DDP buckets on the promoted stack
- Start (GPU lock): 2026-11-06 22:59:04 UTC; end: 23:27:10 UTC (including the full check and paired remeasurement). - Profile:/tmp/r22/profile/profile.json, statusok, instrumented 0.2519 s/update. NCCL remained the largest kernel category at 333.7 ms/5 updates: 189.5 ms in 20 fp32 broadcasts and 141.8 ms in 235 bf16 gradient allreduces. GEMMs consumed 317.7 ms. Selected the communication-configuration hotspot to retest the round-3 bucket choice after all later changes. - Existing-switch screening, with no code change: 75 MiB/tmp/r22/ddp75-time/time.json0.1894 s/update, 1.2341×; 150 MiB/tmp/r22/ddp150-time/time.json0.1879 s/update, 1.2439× (windows 1.877/1.881 s, range 0.0018); 200 MiB/tmp/r22/ddp200-time/time.json0.1883 s/update, 1.2413×. All runs had statusokand peak allocated memory 59.30 GiB. - 150 MiB quick check:/tmp/r22/quick/check.json, PASS, zero exact failures, the same 1466 unresolved entries and 1268 bitwise-identical entries as round 19, and no dtype/precision-mode changes. This supports an N0 increment over the promoted stack. - 150 MiB full verdict:/tmp/r22/check/check.json, PASS, N1 stack, speedup 1.2067, candidate 0.1937 s/update, windows 1.949/1.925 s, range 0.0124. Gate 3 passed all endpoints with values identical to round 19 (largest |difference|/bound about 1.5183). Despite passing, this timing is slower than the retained 1.2367× full result, so it was not promoted. - Paired follow-up:/tmp/r22/paired150/time.jsongave 0.1885 s/update, 1.2399× versus frozen and 1.2366× versus its paired reference, windows 1.885/1.886 s./tmp/r22/paired100/time.jsongave 0.1893 s/update, 1.2347× versus frozen and 1.2340× paired, but its windows were noisier at 1.878/1.907 s. The 0.0026 paired advantage is unresolved against the contradictory full run; retain 100 MiB as the best validated setting. No submission or code change was made, so there was nothing to commit. - Detached jobs: profile actual PID 180538; three-run sweep 182151; quick 186901; full check 189184; paired 150 194365; paired 100 197417. Logs under/tmp/r22/. All completed successfully.
6.5 hRound 23 — restore vectorized SwiGLU for the packed self gate
- Start (GPU lock): 2026-11-06 23:29:38 UTC; end: 23:38:58 UTC. - Profile:/tmp/r23/profile/profile.json, statusok, instrumented 0.2295 s/update. The FFNs used vectorized SwiGLU (3.409 ms forward and 5.436 ms backward over five updates), while the 13 self-mixer gates/update used non-vectorized fallback kernels (1.593 ms forward and 2.143 ms backward). The packed projection's last-dimension gate view is noncontiguous; selected this missed existing fast path. - Added off-by-defaultYOCO_OPT_CONTIGUOUS_G, applying.contiguous()only to the self-mixer gate immediately before its existing SwiGLU operation. This preserves the projection packing and all arithmetic; the enabled increment is N0. -/tmp/r23/probe_contiguous_g.pyused the production[1,2048,12312]packed layout and[1,2048,3072]gate/other inputs in deterministic mode. Forward output, packed input gradients, and the other input gradient were all bitwise identical and finite. - Timing:/tmp/r23/contiguous-g-time/time.json, statusok, 0.1889 s/update,speedup_vs_frozen_reference=1.2373, windows 1.883/1.896 s, range 0.0066. Peak memory rose from 59.30 to 59.45 GiB because the contiguous gate is saved for backward. This is effectively tied with the 1.2367× best full result, so no full check was run and the switch remains off. - Quick check:/tmp/r23/quick/check.json, PASS, zero exact failures, the same 1466 unresolved and 1268 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. This confirms the N0 increment at the judge's recorded sites. - Detached jobs: profile actual PID 200513; timing 202181; quick 203823. Logs under/tmp/r23/. All completed successfully. The promoted switch file is unchanged.
6.9 hRound 24 — fused self-mixer RMSNorm + SwiGLU layout
- Start (GPU lock): 2026-11-06 23:40:47 UTC; end: 2026-11-07 00:02:39 UTC (including the full check). - Profile:/tmp/r24/profile/profile.json, statusok, instrumented 0.2306 s/update. The self-mixer still used non-vectorized SwiGLU (1.596 ms forward and 2.128 ms backward over five updates) around Apex per-head RMSNorm and transpose/reshape layout work. Round 23 showed that a separate contiguous copy only breaks even; selected the whole norm/gate/layout region as one memory-bound fusion class. - Added off-by-defaultYOCO_OPT_FUSED_GATEandkernel/fused_gate.py. A fixed-tile Triton forward computes the same non-affine per-head RMSNorm in fp32, rounds the normalized value to bf16, applies the existing SwiGLU formula, and writes directly in[batch,sequence,heads,dim]layout. The backward recomputes the normalized value from the saved bf16 input and fp32 reciprocal standard deviation, rounds the SwiGLU norm gradient to bf16 before the norm backward, and returns both input gradients. No atomics, RNG, or autotuning are used. The existingRMSNormmodule call and output contract are retained; unsupported layouts/dtypes use the original path. -/tmp/r24/probe_fused_gate.pychecked small and production[1,24,2048,128]cases with packed, offset gate views. Outputs and both gradients were finite; the small case was bitwise, while production relative L2 differences were 1.88e-5 forward and at most 2.04e-5 for gradients. Enabled increment: N1. An initial compatibility error (tl.rsqrtunavailable in Triton 2.3) was fixed with fp321 / tl.sqrtbefore the successful probe. - Screening:/tmp/r24/fused-gate-time/time.json, statusok, 0.1882 s/update,speedup_vs_frozen_reference=1.2419, windows 1.885/1.879 s, range 0.0028, peak 59.15 GiB (down 0.15 GiB). Quick check/tmp/r24/quick/check.json: PASS, zero exact failures, 1475 unresolved entries, no dtype/precision-mode changes. - Full verdict:/tmp/r24/check/check.json, status PASS, speedup 1.2373, tier N1. Candidate 0.1889 s/update, windows 1.888/1.889 s, range 0.0010, peak 59.15 GiB. Gate 2 had zero exact failures; gate 3 passed all endpoints with largest |difference|/bound about 0.1960. PromotedYOCO_OPT_FUSED_GATE=1as the new best validated submission. - Detached jobs: profile actual PID 206159; timing 208026; quick 209633; full check 211986. Logs under/tmp/r24/. All completed successfully.
7.0 hRound 25 — fused-gate warp-count sweep
- Start (GPU lock): 2026-11-07 00:04:29 UTC; end: 00:09:49 UTC. - Profile of the promoted fused-gate stack:/tmp/r25/profile/profile.json, statusok, instrumented 0.2296 s/update._fused_gate_fwdconsumed 2.292 ms and_fused_gate_bwd2.510 ms over five updates; the removed non-vectorized SwiGLU kernels were absent. Selected the fixed four-warp launch configuration for these 128-wide rows. - Added off-by-defaultYOCO_OPT_FUSED_GATE_WARPS, using one value for the existing forward and backward launches and retaining four as the default. No formula, layout, or precision change was made; reduction partitioning keeps the enabled variants at N1. -/tmp/r24/probe_fused_gate.pywas rerun with one and two warps. Both produced finite outputs and gradients at small and production shapes; production relative L2 differences versus the original Apex+jiterator path remained about 2e-5. - Timing: one warp/tmp/r25/warps1-time/time.json0.1897 s/update, 1.2321×; two warps/tmp/r25/warps2-time/time.json0.1893 s/update, 1.2347×. Both are slower than the four-warp 1.2373× full result and 1.2419× screening result. No quick/full check was needed for the rejected variants, no override was promoted, and the default remains four. - Detached jobs: profile actual PID 217328; two-run timing sweep 219170. Logs/tmp/r25/profile-launch.logandsweep-launch.log. All completed successfully.
7.4 hRound 26 — pipeline bucket-aligned Adam with ZeRO broadcasts
- Start (GPU lock): 2026-11-07 00:10:42 UTC; end: 00:33:01 UTC (including the full check). - Profile:/tmp/r26/profile/profile.json, statusok, instrumented 0.2320 s/update. NCCL consumed 337.0 ms/5 updates, including 193.3 ms of fp32 broadcasts; Apex Adam consumed 71.46 ms in 410 calls over four recorded optimizer steps. Selected serialization between the full-shard Adam call and subsequent broadcasts. - Added off-by-defaultYOCO_OPT_ADAM_BROADCAST=2infast_adam_pipeline.py, integrated withfast_overlap.py. The existingFusedAdamV2.step()is still entered once and increments each group's step once. Its fp32 parameter, gradient,exp_avg, andexp_avg_sqlists are split into bucket-aligned views; after each existing Apex multi-tensor call, matching source-rank broadcast chunks are launched asynchronously while later Adam chunks execute. Views are cached while the persistent base tensors are unchanged. All parameters and states are updated exactly once, and the original fp32 objects and optimizer layout are retained. - The refresh path waits for the prelaunched broadcasts and reuses the existing local/remote bf16 casts and converted-parameter tracking. Guards retain the original path for multiple groups/devices, missing or incomplete buckets, non-fp32/noncontiguous tensors, or unexpected tensor lists. Collective chunk order is identical on all ranks; no RNG or new precision mode is used. - Screening:/tmp/r26/adam-broadcast2-time/time.json, statusok, 0.1867 s/update,speedup_vs_frozen_reference=1.2519, windows 1.870/1.863 s, range 0.0039, peak 59.15 GiB. No pipeline fallback was logged. Quick check/tmp/r26/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as round 24. - Full verdict:/tmp/r26/check/check.json, status PASS, speedup 1.2614, stack tier N1; increment N0. Candidate 0.1853 s/update, windows 1.850/1.856 s, range 0.0034, peak 59.15 GiB. Gate 2 had zero exact failures, and all six trajectory values were exactly identical to round 24 (largest |difference|/bound about 0.1960), establishing that the pipeline did not change recorded numerics. Promoted as the new best validated submission. - Detached jobs: profile actual PID 222438; timing 224194; quick 225839; full check 228189. Logs under/tmp/r26/. All completed successfully.
7.8 hRound 27 — Adam-broadcast pipeline chunk-count sweep
- Start (GPU lock): 2026-11-07 00:34:33 UTC; end: 00:57:47 UTC (including the full check). - Profile of the two-chunk pipeline:/tmp/r27/profile/profile.json, statusok, instrumented 0.2307 s/update. Broadcasts consumed 194.1 ms and Adam 73.26 ms over five updates. Optimizer-step range CPU time was 25.26 ms/4 steps and refresh range CPU time 24.44 ms, versus 5.44 and 81.48 ms before pipelining; the Apex kernel count remained 410. Selected pipeline granularity. - Existing-switch timings, with no code change: three chunks/tmp/r27/chunks3-time/time.json0.1845 s/update, 1.2668×; four/tmp/r27/chunks4-time/time.json0.1818 s/update, 1.2856×; five/tmp/r27/chunks5-time/time.json0.1940 s/update, 1.2048×; six/tmp/r27/chunks6-time/time.json0.1813 s/update, 1.2892×; eight/tmp/r27/chunks8-time/time.json0.1860 s/update, 1.2566×. Six was selected. Five's stable but pathological windows are consistent with the alignment sensitivity of bucket slices; eight also loses efficiency. - Six-chunk quick check/tmp/r27/quick6/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the two-chunk pipeline, and no dtype/precision-mode changes. - Full verdict:/tmp/r27/check6/check.json, status PASS, speedup 1.2928, stack tier N1; increment N0. Candidate 0.1808 s/update, windows 1.808/1.807 s, range 0.0006, peak 59.15 GiB. Gate 2 had zero exact failures and every trajectory value was exactly identical to rounds 24 and 26 (largest |difference|/bound about 0.1960). PromotedYOCO_OPT_ADAM_BROADCAST=6; the submitted switch file is updated and the git tree needed no new commit. - Detached jobs: profile actual PID 233501; three/four-chunk sweep 235140; six/eight-chunk sweep 238428; five-chunk timing 241684; quick 243317; full check 245627. Logs under/tmp/r27/. All completed successfully.
8.1 hRound 28 — retest flat ZeRO-to-model casts after Adam pipelining
- Start (GPU lock): 2026-11-07 00:59:25 UTC; end: 01:16:04 UTC (including the full check). - Profile of the promoted six-chunk pipeline:/tmp/r28/profile/profile.json, statusok, instrumented 0.2317 s/update. Over five updates, broadcasts consumed 201.095 ms in 60 calls, Apex Adam 80.240 ms in 420 calls, and direct bf16 casts 65.988 ms in 1505 calls. Selected the existing flat-cast path for retesting because the new optimizer overlap changed its exposure. - Enabled the already committed, off-by-defaultYOCO_OPT_FLAT_CAST=1in/tmp/r28/flat-cast.json; no source change was made. It allocates persistent flat bf16 storage, aliases the existing parameter objects without changing names/shapes, sorts copy metadata by actual bucket offset, and preserves the original fallback. - Screening/tmp/r28/flat-cast-time/time.json: statusok, 0.1804 s/update,speedup_vs_frozen_reference=1.2956, windows 1.805/1.804 s, range 0.0009, peak 59.15 GiB. No flat-cast or Adam-pipeline fallback appeared. Quick/tmp/r28/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as round 27, and no dtype/precision-mode changes. - Full verdict/tmp/r28/check/check.json: PASS, N1 stack, but speedup regressed to 1.2641. Candidate 0.1849 s/update, windows 1.846/1.851 s, range 0.0024; all trajectory values again exactly matched the promoted stack. Despite the favorable screening, this is slower than/tmp/r27/check6at 1.2928×, so flat casting remains disabled and/submission/switches.jsonis unchanged. No source commit was needed. - Detached jobs: profile actual PID 250945; timing 252574; quick 254218; full check 256533. Logs under/tmp/r28/. All completed successfully.
8.3 hRound 29 — compile self-retention cross-chunk helpers
- Start (GPU lock): 2026-11-07 01:18:34 UTC; end: 01:28:28 UTC. - Profile of the promoted submission:/tmp/r29/profile/profile.json, statusok, instrumented 0.2256 s/update; GPU busy fraction 69.56%. GEMMs consumed 325.2 ms/5 updates, NCCL 394.2 ms, memcpy/cast 140.8 ms, and elementwise 39.3 ms. The self-retentioninner_chunkwas already compiled, but the sibling cross-chunk gate exponentials, dtype conversions, scaling, and surrounding matrix products remained eager. Selected that contained host/memory-bound region. - Added off-by-defaultYOCO_OPT_CROSS_COMPILE=1ingate_recurrent.py, compiling two pure helpers for the K/V decay product and the query/state product. The custom recurrence, formulas, explicit bf16 conversion points, call counts, RNG, and all other decoder paths remain unchanged. Inductor partitioning makes the enabled increment N1. - Timing/tmp/r29/time/time.json: statusok, 0.1813 s/update,speedup_vs_frozen_reference=1.2892, windows 1.809/1.817 s, range 0.0044, startup 69.9 s, peak 59.14 GiB. This exactly matched the earlier six-chunk screening result and did not improve it. No compile/recompilation signature invalidated the windows. - Quick/tmp/r29/quick/check.json: PASS, zero exact failures, 1482 unresolved and 1271 bitwise-identical entries, no dtype/precision-mode changes. Since there was no timing gain, no full check was run and the switch remains off. Commit85d7a53(Compile cross-chunk gate helpers (N1; time 1.2892x, disabled)). - Existing-trace diagnosis for the next hotspot: 594copy_bucket_to_graddevice copies over five updates (about 119/update), carrying 15.878 GB and taking 9.839 ms, plus many host launches. DDP gradient bucket views have not yet been tested. - Detached jobs: profile actual PID 261937; timing 263611; quick 265672. Logs under/tmp/r29/. All completed successfully.
8.6 hRound 30 — DDP gradient bucket views
- Start (GPU lock): 2026-11-07 01:29:50 UTC; end: 01:49:32 UTC (including the full check). - Profile of the promoted submission:/tmp/r30/profile/profile.json, statusok, instrumented 0.2275 s/update and 72.13% GPU busy. NCCL consumed 436.3 ms/5 updates, GEMMs 325.5 ms, and memcpy/cast 141.2 ms. The previous trace's 594copy_bucket_to_gradcalls over five updates (about 119/update) selected DDP's existing gradient bucket-view path. - Added off-by-defaultYOCO_OPT_GRAD_BUCKET_VIEW=1infast_ddp.py, passinggradient_as_bucket_view=Trueto the same PyTorch DDP constructor. Parameter gradients become views of the same reduction buckets, eliminating bucket-to-gradient copies without changing reduction math, gradient values, optimizer state, parameter objects, or RNG. The increment is N0 by unchanged downstream records. - Screening/tmp/r30/time/time.json: statusok, 0.1790 s/update,speedup_vs_frozen_reference=1.3058, windows 1.804/1.776 s, range 0.0156. Quick/tmp/r30/quick/check.json: PASS, zero exact failures, exactly the promoted stack's 1475 unresolved and 1264 bitwise-identical entries, no dtype/precision-mode changes. - Repeated paired timing/tmp/r30/paired/time.json: 0.1784 s/update, 1.3101× versus frozen and 1.3122× versus the adjacent reference (reference 0.2341 s/update), windows 1.790/1.778 s, range 0.0066. This resolved the initial small-gain screening. - Full verdict/tmp/r30/check/check.json: status PASS, speedup 1.3065, stack N1; increment N0. Candidate 0.1789 s/update, windows 1.787/1.792 s, range 0.0028, peak 59.15 GiB. Gate 2 counts were unchanged and every trajectory value exactly matched rounds 24–28. PromotedYOCO_OPT_GRAD_BUCKET_VIEW=1; commit679084e(Use DDP gradient bucket views (N0; full check 1.3065x)). - Detached jobs: profile actual PID 268255; timing 269908; quick 271545; paired timing 273859; full check 276924. Logs under/tmp/r30/. All completed successfully.
8.9 hRound 31 — DDP static graph
- Start (GPU lock): 2026-11-07 01:51:59 UTC; end: 02:05:25 UTC. - Profile of the promoted bucket-view stack:/tmp/r31/profile/profile.json, statusok, instrumented 0.2189 s/update.copy_bucket_to_gradcalls fell from 594 to 0 and GPU memcpy time from 21.743 ms to 0.709 ms over five updates, confirming round 30's intended path. NCCL consumed 346.1 ms, GEMMs 317.1 ms, memcpy/cast 119.6 ms, and Apex Adam 92.0 ms. Selected the remaining DDP per-step graph management. - Added off-by-defaultYOCO_OPT_DDP_STATIC_GRAPH=1, passingstatic_graph=Trueto the existing DDP constructor alongside gradient bucket views. The model graph and all parameters are static and used; no math or recipe setting changes. The enabled increment is N0 by unchanged records. - Three valid timings did not resolve a gain:/tmp/r31/time/time.json0.1773 s/update, 1.3183×, windows 1.773/1.773 s;/tmp/r31/paired/time.json0.1800 s/update, 1.2985× frozen and 1.3022× adjacent-reference, windows 1.802/1.797 s;/tmp/r31/repeat/time.json0.1785 s/update, 1.3094×, windows 1.789/1.781 s. Their mean is effectively tied with bucket views alone. No full check was justified and the switch remains off. - Quick/tmp/r31/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. Commit4b10a8d(Add optional DDP static graph (N0; time 1.2985-1.3183x, disabled)). - Detached jobs: profile actual PID 282277; timing 283938; quick 285579; paired timing 287884; repeat 291014. Logs under/tmp/r31/. All completed successfully.
9.4 hRound 32 — bucket-aligned fp32 master-gradient casts
- Start (GPU lock): 2026-11-07 02:06:59 UTC; end: 02:36:04 UTC (including the full check). - Profile of the promoted bucket-view stack:/tmp/r32/profile/profile.json, statusok, instrumented 0.2180 s/update._cast_scale_kernelstill consumed 26.375 ms in 1200 calls over five updates. NCCL consumed 359.7 ms, GEMMs 321.0 ms, memcpy/cast 119.5 ms, and Adam 92.6 ms. Selected per-parameter master-gradient conversion and launch overhead. - Added off-by-defaultYOCO_OPT_FLAT_GRAD=1. The first sync groups contiguousp.gradviews by actual DDP storage and offset, allocates flat fp32 destinations, and rebinds each existingp32.gradtensor object withset_; names, shapes, dtypes, optimizer keys, and values are preserved. Cast-scale work then runs once per contiguous bucket run. Later source signatures reuse cached views; a changed finalized layout rebuilds them and explicitly invalidates cached Adam chunk views before rebinding. Unsupported layouts retain the per-parameter path with a visible fallback. - A CPU metadata probe verified gap splitting, exact offsets/values, tensor identities, cache reuse, and changed-layout handling. The first GPU timing/tmp/r32/time/time.jsonwas ERROR, not a valid result: the initial single storage covers over 2³¹ elements and overflowed 32-bit Triton offsets, causing an illegal memory access. Added a 64-bit-offset kernel used only above that boundary. A locked functional probe crossed 2³¹ with correct boundary values; ordinary casts retain the original kernel. -/tmp/r32/time64/time.jsonthen ran validly at 1.3094×, but logged a layout change after update 0 and fell back, so it was not promotion evidence. Allowing rebuilds produced the expected two-stage layout: one initial storage, then 47 buckets for 240 parameters, with no later fallback./tmp/r32/rebuild-time/time.json: 0.1732 s/update, 1.3495×, windows 1.748/1.715 s, range 0.0191, peak 70.74 GiB. - Quick/tmp/r32/quick/check.json: PASS, zero exact failures, the same 1475 unresolved and 1264 bitwise-identical entries as the promoted stack, and no dtype/precision-mode changes. Repeat/tmp/r32/paired/time.json: 0.1730 s/update and 1.3510× versus frozen, confirming the gain. Its adjacent reference was slower (0.2379 s/update), so the inflated 1.3751× paired ratio is not used as the evidence. - Full verdict/tmp/r32/check/check.json: status PASS, speedup 1.3733, stack N1; increment N0. Candidate 0.1702 s/update, windows 1.703/1.702 s, range 0.0008, peak allocated 70.74 GiB and reserved 77.98 GiB. Gate 2 counts and every trajectory value exactly matched the promoted stack. PromotedYOCO_OPT_FLAT_GRAD=1; commit2b642fc(Batch master-gradient casts by DDP bucket (N0; full check 1.3733x)). Probe-generated__pycache__was removed after the commit and the tree is clean. - Detached jobs: profile actual PID 292686; failed timing 294474; 64-bit timing 296209; rebuild timing 297885; quick 299579; paired timing 301916; full check 305004. Logs under/tmp/r32/. All launched jobs terminated; the failed run was followed by the explicit functional probe and successful validation above.
9.8 hRound 33 — foreach local and remote weight casts
- Start (GPU lock): 2026-11-07 02:38:52 UTC; end: 02:58:22 UTC (including the full check). - Profile of the promoted flat-gradient stack:/tmp/r33/profile/profile.json, statusok, instrumented 0.2135 s/update. Cast-scale calls fell from 1200 to 235 over five updates while their GPU time remained about 25 ms, confirming round 32 removed launch/host overhead. The remaining direct weight casts consumed 66.293 ms in 1505 calls. Selected local and remote fp32→bf16 weight refresh launches. - Added off-by-defaultYOCO_OPT_WEIGHT_FOREACH=1infast_overlap.py, usingtorch._foreach_copy_for the existing local owner casts and each already pipelined remote chunk. The older independent remote-only switch is preserved. No storage layout, parameter view, arithmetic, broadcast order, or optimizer behavior changes; the increment is N0. - Two screenings were identical:/tmp/r33/time/time.jsonand/tmp/r33/paired/time.json, both 0.1710 s/update andspeedup_vs_frozen_reference=1.3668; the paired ratio was 1.3661× with a normal 0.2336 s/update reference. Window ranges were 0.0038 and 0.0093. Quick/tmp/r33/quick/check.json: PASS, zero exact failures, unchanged 1475 unresolved and 1264 bitwise-identical entries, and no dtype/precision-mode changes. - Full verdict/tmp/r33/check/check.json: PASS, N1 stack with an N0 increment, but speedup 1.3700, slightly below the promoted/tmp/r32/checkat 1.3733×. Candidate 0.1706 s/update, windows 1.706/1.706 s. Every trajectory value was unchanged. The change is neutral-to-slower and remains off;/submission/switches.jsonis unchanged. Commitd706fa6(Add optional foreach weight casts (N0; full check 1.3700x, disabled)). - Detached jobs: profile actual PID 310358; timing 312028; quick 313684; paired timing 315994; full check 319088. Logs under/tmp/r33/. All completed successfully.