GPT-6 Astrasuspected nerf 1.79×
Harness: Codex · switches: YOCO_RMSNORM_IMPL=apex YOCO_OSS_OVERLAP=1 YOCO_OSS_CHUNK_MIB=256 YOCO_FUSED_GRAD_PREP=1 YOCO_FUSED_MASTER_REFRESH=1 YOCO_GRAPH_DECODER=1 YOCO_DDP_BUCKET_MIB=64 YOCO_DDP_BUCKET_VIEWS=1 YOCO_OPTIMIZER_CHUNK_ELEMS=524288 YOCO_OPTIMIZER_CHUNK_SCOPE=adam YOCO_PIPELINED_MASTER_REFRESH=1 YOCO_MASTER_GRAD_CLEAR=defer YOCO_FUSED_CROSS_ENTROPY=1 YOCO_OSS_P2P=1 YOCO_FUSED_RMS_AFFINE=1 YOCO_OSS_IPC_DMA=1 YOCO_DDP_IPC_REDUCE=1 YOCO_ADAM_LOCAL_REFRESH=1 YOCO_CUBLAS_LT_LIBRARY=/opt/candidate-env/cublas-12.8/nvidia/cublas/lib/libcublasLt.so.12 YOCO_IPC_STREAM_SIGNALS=1
- Judged
- 1.79×
- Tier
- N1
- Self-best
- 1.79×
- Rounds
- 25
- Hours
- 10.0
- API cost
- $99.7
- Input tokens · 95.5% cached
- 47 M
- Output tokens
- 0.6 M
- Requests
- 396
§ 02Key levers
- Cross-GPU exchange via CUDA IPC + copy engine (no SMs)
- ✓
YOCO_OSS_IPC_DMA, 1.52× → 1.65×; gradsYOCO_DDP_IPC_REDUCE→ 1.68× (both full check) - Forward / backward CUDA graph
- ✓
YOCO_GRAPH_DECODER, per layer, 1.23× → 1.31× - Newer cuBLAS
- ✓ cuBLASLt from 12.8.4.1, loaded explicitly, 1.72× → 1.78×
- Retention / attention kernel swap
- ✗ retention untouched; started compiling FlashAttention-3 in round 26, log ends there
- Other main levers
- Apex RMSNorm and its fused affine variant; fused cross-entropy; Adam overlapped with broadcast chunks; custom CUDA Adam + bf16 writeback
§ ARaw trace
Raw work log26 sections
0.0 hRound 1 — baseline profile and first hotspot
- Start/profile launch: 2026-11-06T15:16:36Z. - Detached profile PID: 13132; log:/tmp/r1/profile.log. - Command: `bench.py profile --tree /submission/yoco --config /submission/switches.json --out /tmp/r1/profile --iters 4`. - Baseline profile finished 2026-11-06T15:17:53Z. Kernel-time shares: communication 31.5% (all-reduce 16.5%, broadcast 14.5%), GEMM 25.6%, copies/casts 17.2%, elementwise 12.1%, fused Adam/norm 7.2%. GPU union busy 72%; profile CPU 2.26 busy CPUs of 48 allowed. The large OSS master-weight broadcasts use NCCL LL; no NCCL override is configured. - Hotspot class: communication/backend configuration. ScreenNCCL_PROTOSimple and LL128 before changing communication code. Both preserve dtypes and operations; two-rank sums have only two operands. N0 hypothesis must be confirmed by the judge. Official NCCL environment-variable docs reviewed. - Experiment configs:/tmp/r1/simple.json,/tmp/r1/ll128.json; submission remains unchanged while they are screened. - Protocol screens completed 2026-11-06T15:22:05Z: -/tmp/r1/time-simple: valid, 0.2367 s/update, speedup 0.9838, windows 2.368/2.365 s, range 0.13%; rejected (slower). -/tmp/r1/time-ll128: valid, 0.2356 s/update, speedup 0.9884, windows 2.358/2.353 s, range 0.20%; rejected (slower). - Continuing the communication class in the same round: OSS performs a device-wide synchronization before every weight broadcast. ProcessGroupNCCL already orders its stream after the caller's stream andWork.wait()orders subsequent consumer work. Test removing only this redundant host barrier, retaining every fp32 broadcast and wait, with the original fallback for CPU orbroadcast_fp16. Experiment/tmp/r1/tree, switchYOCO_ASYNC_OSS_BROADCAST=1; baseline submission remains unchanged. -/tmp/r1/time-asyncfinished 2026-11-06T15:24:06Z: valid, 0.2330 s/update, speedup 0.9994, no gain beyond reference noise. - Conclusion/end 2026-11-06T15:24:20Z: reject all three changes on performance. No correctness check or tier claim for rejected experiments. Summed NCCL kernel duration included waiting/overlap, so it overstated the amount of exposed latency removable by these settings.
0.2 hRound 2 — fresh profile of unchanged best
- Profile launch: 2026-11-06T15:25:03Z; profile/tmp/r2/profile. Initial launcher PID 19114 used the wrong working directory and exited before acquiring GPUs; restarted from the judge workspace. - Profile finished 2026-11-06T15:26:26Z: NCCL 31.1%, GEMM 25.7%, copies/casts 17.3%, elementwise 12.1%, multi-tensor optimizer 7.2%. GPU union busy 75%. Copy/cast kernels include 3432 conversion calls in four updates (106 ms); float fill has 1388 calls (18.6 ms). - Hotspot: optimizer-side gradient/parameter memory traffic and launch overhead (missed existing foreach paths). Adam is already fused; the surrounding conversions, gradient scaling, and master zeroing are not. - Experiment copy/tmp/r2/treeadds three independent, default-off switches:YOCO_FOREACH_MASTERbatches dtype-preserving copies;YOCO_MASTER_GRAD_SET_TO_NONEavoids clearing gradients immediately overwritten by the next BF16-to-fp32 sync;YOCO_FOREACH_GRAD_SCALEbatches the existing scalar multiplies. Flat-parameter mode keeps the reference implementation. N0 hypotheses, not yet judged. - Individual screens completed 2026-11-06T15:31:00Z (all valid, no gain):/tmp/r2/time-copy0.2346 s/update, 0.9926x;/tmp/r2/time-zero0.2347 s/update, 0.9922x;/tmp/r2/time-scale0.2339 s/update, 0.9956x. - CPU setup confirms all three overrides are installed. A combined profile will verify whether PyTorch actually uses multi-tensor CUDA fast paths for the cross-dtype copies, rather than silently falling back. - Combined profile/tmp/r2/profile-combinedfinished 15:34:32Z: copy/cast time and conversion kernel counts are unchanged (3432 calls, 106 ms), confirming the cross-dtype foreach path falls back to individual copies. Float fills decrease, but this does not improve the update time. -/tmp/r2/time-combinedfinished 2026-11-06T15:35:44Z: valid, 0.2385 s/update, 0.9764x. Reject the combination and all individual screens. - End 2026-11-06T15:36:30Z. No numerical verdict requested for rejected performance directions. Submission still has unchanged training code.
0.4 hRound 3 — model normalization
- Start/profile launch: 2026-11-06T15:36:30Z;/tmp/r3/profile. - Fresh best-version profile precedes choosing the normalization rewrite. - Profile actually launched 2026-11-06T15:37:07Z and finished 15:38:24Z: GEMM 26.1%, copies/casts 17.7%, elementwise 12.4%, reductions 1.5%. GPU busy 75%. Sixty-seven RMSNorm calls per update use eager FP32 square/mean/rsqrt/multiply, conversion, and affine operations; backward repeats several memory passes. No fused norm path exists in the tree. - Hotspot class: normalization memory/launch overhead. Test the installed Apex non-affine RMSNorm (keep the separate BF16 affine multiply to preserve that rounding boundary), then regionaltorch.compileof the original formula including its casts. Both use the same precision and real-valued operation; expect N1 reduction-order changes, subject to exact gates and trajectory validation. CPU keeps the original path. -/tmp/r3/treehas default-offYOCO_RMSNORM_IMPLvaluesapexandcompile; baseline submission unchanged. - Screens finished 2026-11-06T15:41:43Z: -/tmp/r3/time-apex: valid, 0.2190 s/update, 1.0633x, windows 2.188/2.192 s (range 0.18%), peak allocated 60.96 GiB. -/tmp/r3/time-compile: valid, 0.2227 s/update, 1.0457x, windows 2.237/2.216 s (range 0.95%), peak allocated 60.32 GiB. - Selected Apex./tmp/r3/apex-treeremoves the unused compile alternative before validation. The normalization remains separate from the affine multiply. Deterministic operator comparisons precede the judge quick check; a successful quick check is followed by the full check. The full check is outside the round budget. - Deterministic operator probe passed shape/dtype/finite assertions at widths 3072 and 128 and scales 0.1/1/10. Maximum input-gradient relative L2 2.03e-5, maximum affine-gradient relative L2 1.03e-4 (diagnostic only). -/tmp/r3/quick/check.json: PASS, completed 15:47:34Z; exact items pass, tensors not bitwise, so statistical validation required. -/tmp/r3/check/check.json: full PASS/N1, completed 2026-11-06T15:55:26Z. Timing 0.2185 s/update, 1.0658x, windows 2.185/2.184 s (0.04% range). RNG, structural/loader, exact-item, and six trajectory endpoint gates pass. - End/promotion 2026-11-06T15:56:00Z: promote only the validated Apex path; keep compile alternative out. Switch mirrored in the git tree.
0.7 hRound 4 — profile the validated Apex version
- Profile launch: 2026-11-06T15:56:30Z, PID 39949,/tmp/r4/profile.log, output/tmp/r4/profile. - Actual profile start 15:57:13Z, finish 15:58:30Z. Summed kernel shares: broadcast 16.1%, gradient all-reduce 14.0%, GEMM 28.6%, copies/casts 14.4%, elementwise 11.1%, Adam 6.2%. Fused RMSNorm reduces the previous conversion/reduction work. The OSS optimizer still runs Adam (~14 ms) followed by full fp32 broadcasts (~37 ms), with no overlap between them. - Hotspot: exposed optimizer communication. Earlier protocol/barrier-only switches had no gain. This round instead chunks the *existing Apex Adam dispatch* and asynchronously broadcasts each completed chunk while later chunks update. The same kernel, fp32 states/masters, scalar arguments, single optimizer step and broadcasts are retained. No precision change. - Experiment/tmp/r4/tree, default-offYOCO_OSS_OVERLAP; screen 256 and 1024 MiB. A two-rank deterministic optimizer probe checks exact weights, states and steps with split parameters, partial chunks and odd sizes. Unsupported optimizers/layouts retain the original method and are counted. - Initial probe launcher exited immediately: the candidate environment has no standalonetorchrunscript. Restarted with its Python and-m torch.distributed.run; no training measurement was produced by the failed launcher. - Two-rank deterministic probe: all four updates on both ranks have bitwise-equal fp32 weights, Adam moments and step counters; fast-path count is four, with odd sizes, partial chunks and split parameters. - Screens finished 2026-11-06T16:04:13Z:/tmp/r4/time256: valid, 0.2095 s/update, 1.1116x;/tmp/r4/time1024: valid, 0.2101 s/update, 1.1084x. Both beat the Apex-only version. The small difference between chunk settings is not resolved; select 256 MiB and do not claim it is optimal. - Cache key also includes all remote bucket addresses/sizes, so changes to the optimizer's bucket storage invalidate every retained view. - Next: profile selected overlap; quick check with eager RMSNorm as an N0 stand-in, then full validation of Apex+overlap. Submission remains the previously validated Apex-only version. - Selected profile completed 16:06:59Z; overlapping Adam/broadcast kernels appear underfairscale::oss::overlap_adam_broadcast. -/tmp/r4/quick-standin/check.jsoncompleted 16:11:28Z: PASS/N0, all upstream tensors bitwise identical with original eager RMSNorm. -/tmp/r4/check/check.jsoncompleted 2026-11-06T16:19:26Z: full PASS/N1, 1.0892x, 0.2138 s/update. All six trajectory endpoints pass. This timing is slower than the 1.1116x screening run; report both rather than substituting the screening result for the full-check verdict. - End/promotion 2026-11-06T16:20:00Z: add validated overlap to the Apex baseline, keep 256 MiB chunks. The result-summary helper had an error reading the quick check's null performance field *after* both checks completed; the actual judge reports both PASS and is unaffected.
1.1 hRound 5 — profile the validated overlap stack
- Profile launch: 2026-11-06T16:21:00Z, PID 52864,/tmp/r5/profile.log, output/tmp/r5/profile. - Actual profile start 16:21:21Z, finish 16:22:38Z. Summed kernel time: GEMM 273 ms, copies/casts 133 ms, elementwise 103 ms, fused optimizer 81 ms over four updates. NCCL time includes overlap/waiting and is not counted as entirely exposed. Master conversion kernels still account for about 87 ms; separate fp32 gradient multiplication about 49 ms. - Hotspot class: optimizer memory traffic. The installed Apex binary has BF16-input, fp32-output L2-norm and cast/scale kernels. Unlike the failed torch foreach path, these are actual fused CUDA implementations. First compare BF16-input norm against widening then the same norm, and fused widening/scaling against the existing two operations. - Primitive probe:/tmp/r5/probe_primitives.py. Source fetch from GitHub failed (connection reset); installed binary exports confirm BF16 support, and numerical probes will verify arithmetic rather than assume it. - Primitive probe passes bitwise for all comparisons: norm from BF16 equals norm after fp32 widening; widening plus scaling equals the fused kernel; master refresh equals the original cast. Production FFN shape, small/odd lengths, three input magnitudes and three coefficients tested. - Experiment/tmp/r5/tree:YOCO_FUSED_GRAD_PREPcomputes the same fp32 norm directly from BF16 gradients, computes the unchanged clipping factor, then widens and scales into the fp32 gradients in one pass. It keeps the fp32 master gradients/state and original Adam call intact. A first-call live equality check compares the BF16-input norm with the original fp32-input norm before enabling this path. Unsupported layouts, scaling modes, and non-BF16 models use the original implementation. - IndependentYOCO_FUSED_MASTER_REFRESHuses Apex's batched fp32-to-BF16 scale-by-one conversion. Both default off; counters/logs identify use. - Screens completed 2026-11-06T16:29:05Z:/tmp/r5/time-grad: valid, 0.2106 s/update, 1.1057x (no resolved gain over the earlier 0.2095 overlap screen; faster than its 0.2138 full run)./tmp/r5/time-refresh: valid, 0.2060 s/update, 1.1304x. - Next screen the combination and profile it to check whether the expected memory passes actually disappear. Judge checks and round end pending. -/tmp/r5/time-combined(16:30:45Z): 0.2060 s/update, 1.1304x. Profile/tmp/r5/profile-combinedconfirms only master refresh was active: copies fall from 133 to 93 ms, but separate multiplication remains. - Found the missed guard: fairseq's logging-statistics all-reduce converts sample size to float64, so the scaling factor is a float64 scalar. The original multiply of an N-dimensional fp32 gradient by that scalar uses fp32 arithmetic (confirmed by the profile'sMulFunctor<float>and PyTorch promotion rules). Permit zero-dimensional float64 factors while preserving their original float64 norm/clipping arithmetic, and convert to float32 only at the same point as the original gradient multiply. Exclude scalar gradients to retain that exact promotion rule. -/tmp/r5/probe_double_scale.pyverifies this path, including rounding halfway cases, before re-timing. Fallbacks now log their reason once. - Corrected screen/tmp/r5/time-fixedcompleted 2026-11-06T16:34:02Z: valid, 0.1900 s/update, 1.2256x. Both activation logs are present and the live norm equality check passed. The float64-scalar probe is bitwise, including both tie-rounding cases. Add a guard for noncontiguous/sparse master gradients before the formal checks. - Next: quick check of overlap+both fusions with eager RMSNorm (N0 target), followed by the full check of the complete stack. Submission stays at R4. - While that check runs, prepare (CPU only) a separate, disabled GEMM backend draft under/tmp/r6/tree. It uses an exact zero-bias addmm identity to access PyTorch 2.3's cuBLASLt route without changing parameters or RNG. No GPU test, timing, or result is claimed yet; a fresh profile of the accepted version is required before screening it. -/tmp/r5/quick-standin/check.jsoncompleted 16:40:07Z: PASS/N0 for the entire optimizer stack with original eager RMSNorm; every recorded tensor and exact item matches upstream. -/tmp/r5/check/check.jsoncompleted 2026-11-06T16:47:53Z: full PASS/N1, 1.2276x, 0.1897 s/update; all six trajectory endpoints pass. - End/promotion 2026-11-06T16:48:00Z: promote both default-off fusions, committing gradient preparation and master refresh separately. Preserve the round-4 overlap and round-3 RMSNorm settings.
1.6 hRound 6 — GEMM backend screening
- Profile launch: 2026-11-06T16:49:00Z, PID 68902,/tmp/r6/profile.log, output/tmp/r6/profile. - CPU-only draft prepared while R5 validated; no GPU measurements yet. - Actual profile start 16:54:13Z, finish 16:55:25Z. The optimizer fusions remove the expected passes: elementwise drops to 53 ms and copies/casts to 51 ms over four updates. GEMM is 261 ms (30.8% of summed kernel time), now the largest non-communication category. GPU union busy is 71% in the instrumented run; it includes profiler launch gaps. - Screen the prepared default-offYOCO_LINEAR_BACKEND: zero-biasaddmmidentity in forward, and then forward/backward. This selects a different installed GEMM-library route without changing trainable parameters, input sizes, accumulation precision, or RNG. Tensor-parallel sizes other than one and non-BF16 inputs keep the reference implementation. - A deterministic production-shape forward/gradient probe runs before both timings. Submission remains the fully validated R5 stack. - Primitive probe: bitwise forward, input and weight gradients for both 3072-to-8192 and 8192-to-3072 production projections. -/tmp/r6/time-forwardcompleted 16:58:06Z: valid, 0.1888 s/update, 1.2334x./tmp/r6/time-backwardcompleted 16:59:18Z: valid, 0.1897 s/update, 1.2276x. Less than 1% improvement over R5 and not resolved from noise. Reject both as added complexity without clear gain. - End 2026-11-06T17:00:00Z. No whole-model numerical verdict claimed for these rejected alternatives; operator probes alone are not a verdict.
1.8 hRound 7 — launch overhead / regional CUDA graphs
- A fresh profile of the validated R5 stack precedes implementation. - Profile start 17:01:47Z, finish 17:03:04Z. GPU union busy is 75% in the instrumented run; GEMM 270 ms, elementwise 54 ms, copies 51 ms and multi-tensor optimizer work 125 ms over four updates. Standard GEMM substitution did not materially help in R6. Test whether repeated Python/autograd launch overhead is exposed using per-layer CUDA graphs. - Experiment/tmp/r7/tree, default-offYOCO_GRAPH_DECODER=1. A proxy captures each original decoder layer's forward and backward with PyTorchmake_graphed_callables. It shares the existing parameters, does not register new parameters/buffers in the training model, and uses cloned sample activations rather than aliasing a prior layer's output. Replay supplies current inputs and uses the same live model-weight addresses. Training, full-sequence, dtype, shape/stride and parameter-address guards retain the original path for unsupported calls; counters log capture/use. - Capture warmup performs no optimizer steps. The judge will verify the per-update RNG stream and all call-count/numerical contracts; no verdict is claimed from the design alone. Timing/checks/end pending. - First screen/tmp/r7/time/time.jsonended 17:06:18Z: ERROR during backward graph capture (legacy stream depends on a capturing stream). No time or speedup was produced; PID 74858 has exited. Try constructing DDP on a side stream to avoid default-stream accumulator ownership. This is still an isolated experiment; the validated submission is intact. - Side-stream construction fixed capture./tmp/r7/time-side/time.jsoncompleted 17:13:33Z: valid, 0.1882 s/update, 1.2374x, window spread 1.2%. Only the 13 cross layers captured: upstream calls the self decoder withis_prefilling=Trueeven during training. Since incremental state is absent this uses the same full-sequence math. Capture the actual boolean and include it in the signature, rather than rejecting those self layers. -/tmp/r7/time-all/time.jsoncompleted 17:15:58Z: valid, 0.1787 s/update, 1.3031x, windows 1.786/1.788 s (0.11% spread), peak 62.21 GiB. All layers captured before timing. This is enough incremental gain to justify numerical checks, starting with eager RMSNorm as an N0 stand-in. -/tmp/r7/quick-standin/check.jsoncompleted 17:21:05Z: PASS/N0, all 9,616 recorded tensors bitwise identical and all exact items pass. The graph change plus optimizer stack is bitwise with eager RMSNorm. Add CPU/unsupported-DDP-backend guards (unchanged CUDA recipe path), then run the full check with Apex RMSNorm. Submission remains R5. - Full/tmp/r7/check/check.jsoncompleted 17:29:53Z: PASS/N1, 1.3075x, 0.1781 s/update, windows 1.779/1.784 s, peak 62.21 GiB. All exact/RNG checks pass; all six trajectory endpoints are identical to the R5 candidate and within the frozen bounds (largest 0.7303 sigma). - End/promotion 2026-11-06T17:30:00Z. Full-check time is excluded from the round budget; active profile/implementation/screens took ~20 minutes. Promote the exact checked graph implementation and enabling switch.
2.3 hRound 8 — profiling the accepted graph stack
- A fresh profile precedes hotspot selection. The submission remains the
fully checked R7 stack while the next experiment is prepared separately.
- Profile started 2026-11-06T17:30:44Z, PID 86155,
log "/tmp/r8/profile.log", output "/tmp/r8/profile".
- Profile finished 17:32:14Z: five instrumented updates, GPU union busy
80%. Summed kernel time: NCCL 421.8 ms (36%), GEMM 335.1 ms (28.5%),
optimizer 155.9 ms (13.3%), elementwise 106.8 ms and copies 64.3 ms.
BF16 gradient all-reduce has 595 calls (119 per update), 211 ms.
Select communication scheduling/copy overhead: screen larger DDP buckets
(64/256 MiB) and bucket gradient views as separate default-off switches.
No parsed recipe, gradient dtype, reduction operation or optimizer changes.
- Sequential judge screens started 2026-11-06T17:33:14Z,
PID 87636, log "/tmp/r8/screens.log", outputs
"/tmp/r8/time-bucket64", "/tmp/r8/time-bucket256", "/tmp/r8/time-views".
- Screens finished 17:37:06Z, all valid: 64 MiB 0.1768 s/update,
1.3171x; 256 MiB 0.1763, 1.3209x; views only 0.1755, 1.3269x.
Screen the 256 MiB+views combination and compare two-rank BF16 DDP
gradients/updates against the original layout, over four updates.
- Combination time plus non-timing probe started 2026-11-06T17:37:58Z,
PID 92025, log "/tmp/r8/combined.log", probe log "/tmp/r8/probe_ddp.log".
- Combined /tmp/r8/time-combined/time.json finished 17:39:15Z:
valid, 0.1732 s/update, 1.3445x, windows 1.732/1.732 s (0.04% spread).
The two-rank production-shape DDP probe is bitwise for every parameter's
gradient and update across four updates, including bucket rebuilds.
Validate with quick eager-RMSNorm stand-in, then full Apex stack check.
- Validation queue started 2026-11-06T17:39:59Z, PID 93707,
log "/tmp/r8/validation.log"; quick output "/tmp/r8/quick-standin",
full output "/tmp/r8/check". Full-check time excluded from round budget.
- Quick finished 17:44:29Z: PASS/N0. Full check finished 17:52:26Z:
PASS/N1, 1.3437x, 0.1733 s/update. The independent two-rank probe
additionally covers the DDP-only path not exercised by single-rank records.
- End/promotion 2026-11-06T17:53:00Z: promote the exact checked DDP
module and 256 MiB+views configuration. Active round duration excluding
full check is about 14 minutes; no other hotspot was changed.2.6 hRound 9 — profiling the accepted DDP stack
- Profile current best before selecting the next hotspot. The passing R8
submission remains unchanged while experiments use a separate tree.
- Profile started 2026-11-06T17:53:34Z, PID 100628,
log "/tmp/r9/profile.log", output "/tmp/r9/profile".
- Profile completed 17:55:04Z. Five updates: NCCL 353.6 ms (33.1%),
GEMM 321.8 ms (30.1%), optimizer 156.7 ms (14.7%), elementwise 103.4 ms.
DDP tuning reduces all-reduce calls from 119 to 21 per update and copies
from 64.3 to 43.1 ms. BF16 scalar multiplications still take 61.4 ms:
390 calls at the 25M-element FFN weight size and 455 at the 9M projection
size, confirming per-parameter pre-reduction gradient averaging.
- Select gradient averaging: screen PyTorch's standard bucket all-reduce
hook and NCCL's averaging operation, replacing separate per-parameter
scaling. Both retain BF16 gradients and average across the same group.
NCCL AVG is guarded to two CUDA/NCCL ranks with BF16 buffers; unsupported
cases use the standard pre-divide bucket hook. Switch defaults off.
Run two-rank production-shape bitwise probes before timing either mode.
- Probe/screen queue started 2026-11-06T17:57:20Z, PID 102129,
log "/tmp/r9/screens.log"; outputs "/tmp/r9/time-bucket" and
"/tmp/r9/time-nccl"; probe logs "/tmp/r9/probe-{bucket,nccl}.log".
- Both two-rank probes are bitwise across four updates. Screens valid:
/tmp/r9/time-bucket completed 17:58:48Z, 0.1768 s/update, 1.3171x;
/tmp/r9/time-nccl completed 18:00:17Z, 0.1733 s/update, 1.3437x.
Standard bucket hook regresses; NCCL AVG exactly matches the previous
full-check time and supplies no gain. Reject both without promoting.
- End 2026-11-06T18:01:00Z. N0 operator probes only; no quick/full
whole-model verdict claimed for rejected code. Submission remains R8.2.8 hRound 10 — next hotspot profile
- Re-profile current best before selecting a new class of optimization.
- Profile started 2026-11-06T18:01:05Z, PID 105508,
log "/tmp/r10/profile.log", output "/tmp/r10/profile".
- Profile finished 18:02:35Z: five updates, NCCL 357.2 ms (33.4%), GEMM
319.2 ms (29.9%), multi-tensor optimizer work 156.8 ms (14.7%). Adam
dispatch uses 510 kernels; each conversion uses 825 kernels (165/update).
Screen launch granularity in these elementwise optimizer kernels: 256K
and 1M elements per block chunk instead of 64K. Norm reductions retain
their original chunk size and reduction tree. The original Apex formulas,
dtypes, states, and optimizer steps remain unchanged. Default-off switch.
- Production-shape cast/scale/refresh and Adam state probes precede each
timing, covering partial chunks and four step/bias-correction values.
- Probe/screen queue started 2026-11-06T18:04:16Z, PID 107067,
log "/tmp/r10/screens.log", outputs "/tmp/r10/time-{chunk256k,chunk1m}".
- Both primitive probes are bitwise. Valid screens: 256K chunks regress
to 0.1749 s/update (1.3314x); 1M chunks give 0.1712 (1.3602x), about
1.2% beyond R8. Isolate scale versus Adam dispatch to avoid retaining a
regressive half of the same launch-granularity change.
- Isolated screens started 2026-11-06T18:08:02Z, PID 110195,
log "/tmp/r10/isolation.log", outputs "/tmp/r10/time-{scale1m,adam1m}".
- Isolation screens completed 18:10:37Z, valid: scale-only 1M gives
0.1722 s/update, 1.3523x; Adam-only 1M gives 0.1713, 1.3594x.
Most gain comes from Adam; combined 1M is indistinguishable from Adam
alone. Finish the bounded search with neighboring Adam-only 512K/2M.
- Neighbor screens started 2026-11-06T18:11:52Z, PID 113147,
log "/tmp/r10/neighbors.log", outputs "/tmp/r10/time-{adam512k,adam2m}".
- Neighbor screens finished 18:14:26Z, valid: Adam 512K regresses to
0.1766 s/update (1.3186x); Adam 2M gives 0.1710 (1.3618x), windows
1.710/1.709 s (0.07% spread). 1M and 2M are nearly tied; select 2M.
Conversion and norm kernels keep their existing chunk sizes. Re-probe
selected setting, then quick stand-in and full checks before promotion.
- Validation queue started 2026-11-06T18:15:14Z, PID 116087,
log "/tmp/r10/validation.log", outputs "/tmp/r10/quick-standin" and
"/tmp/r10/check". Full-check time is excluded from round budget.
- Selected primitive probe is bitwise. Quick completed 18:19:47Z:
PASS/N0. Full /tmp/r10/check/check.json completed 18:27:34Z:
PASS/N1, 1.3602x, 0.1712 s/update. All exact/RNG/trajectory gates pass.
- End/promotion 2026-11-06T18:28:00Z. Promote the exact checked dispatch
override with Adam-only 2M chunks. Excluding the full check, the round
takes about 19 minutes. Norm and conversion granularity remain unchanged.3.2 hRound 11 — current-best critical-path profile
- Profile the accepted stack before selecting the next rewrite. All GPU tests and edits will use a separate experiment tree until validated. - Profile started 2026-11-06T18:28:30Z, PID 123131, log "/tmp/r11/profile.log", output "/tmp/r11/profile". - Profile finished 18:30:00Z: five updates, NCCL 348.2 ms (30.7%), GEMM 321.7 ms (28.4%), multi-tensor optimizer 226.7 ms (20.0%). Larger Adam chunks use fewer kernels (135 vs 510) and overlap longer kernels with broadcasts. Master refresh remains an exposed 6.20–6.23 ms after the final broadcast on every profiled update (165 conversion kernels). - Select that tail: pipeline unchanged scale-by-one fp32-to-BF16 refresh on a side stream as each OSS chunk broadcast completes. Keep every original broadcast/wait, and wait for refresh before the step returns. Cache plans by live master/model pointers; validate complete coverage. No optimizer formula, dtype, tensor, step, or RNG changes. The switch is default off; single-rank and unsupported layouts use the existing path. - A two-rank four-update probe covers partial chunks, split parameters, fp32 masters/states and the BF16 model refresh before judge timing. - Probe/time queue started 2026-11-06T18:31:50Z, PID 124621, log "/tmp/r11/screen.log", probe "/tmp/r11/probe_pipeline.log", timing output "/tmp/r11/time". - The two-rank probe is bitwise for masters, BF16 model weights and Adam states across all four updates./tmp/r11/time/time.jsoncompleted 18:33:18Z: valid, 0.1686 s/update, 1.3812x, 0.03% window spread. - Screen one scheduling refinement behindYOCO_PIPELINE_EARLY_LOCAL: locally owned chunks can refresh after their Adam kernel, while NCCL reads them, without waiting for their broadcast to finish. Remote chunks still wait for receipt. Both transfers and refresh complete before return. - Early-local probe/time started 2026-11-06T18:34:38Z, PID 126320, log "/tmp/r11/early-local.log", output "/tmp/r11/time-early-local". - Early-local probe is bitwise; timing finished 18:36:06Z, valid, 0.1686 s/update, 1.3812x: no extra gain on one refresh stream. Local refresh still queues behind earlier remote waits on that stream. Try independent local/remote refresh streams with separate overflow buffers, preserving dependencies and waiting for both before return. - Two-stream probe/time started 2026-11-06T18:36:37Z, PID 128008, log "/tmp/r11/two-streams.log", output "/tmp/r11/time-two-streams". - Two-stream probe is bitwise, but its valid time is 0.1687 s/update, 1.3804x (18:38:06Z), no gain over the simpler 0.1686/1.3812x path. Remove both early-local variants and their switch from the experiment. Validate the simpler pipeline with quick eager-RMSNorm stand-in and full check before promotion. The main submission remains the checked R10 stack. - Validation queue started 2026-11-06T18:38:59Z, PID 129688, log "/tmp/r11/validation.log", outputs "/tmp/r11/quick-standin" and "/tmp/r11/check". Full-check time excluded from the round budget. - Quick completed 18:43:30Z: PASS/N0. Full completed 18:51:21Z: PASS/N1, but timing is 0.1712 s/update, 1.3602x, equal to R10 rather than the three 0.1686–0.1687 pipeline screens. Before promotion, repeat current submitted R10 and the pipeline back-to-back with judgetime. No new change is made while resolving this measurement difference. - Repeat queue started 2026-11-06T18:52:14Z, PID 136559, log/tmp/r11/repeat.log, outputs/tmp/r11/time-{control,repeat}. - Controlled repeat completed 18:54:59Z, both valid: submitted R10 0.1712 s/update (1.3602x); pipeline 0.1686 (1.3812x). This reproduces the incremental gain. The full check is PASS despite its slower timing sample; promote with that distinction explicitly recorded. - End/promotion 2026-11-06T18:56:00Z. Same checked simple pipeline; no early-local switch or extra stream is retained. Active round time is about 20 minutes excluding the full check.
3.7 hRound 12 — current-best memory-work profile
- Profile the accepted pipeline stack before selecting the next hotspot.
- Profile started 2026-11-06T18:56:26Z, PID 139511,
log "/tmp/r12/profile.log", output "/tmp/r12/profile".
- Profile finished 18:57:56Z: five updates, NCCL 345.3 ms (29.8%), GEMM
317.8 ms (27.5%), optimizer 259.8 ms (22.4%). Pipelined refresh overlaps
transfers; fp32 fills still contribute 22.9 ms in 1,735 kernels.
Select master-gradient clearing, not a new optimizer formula: compare
batched foreach zeroing and deferring master initialization to the next
BF16-to-fp32 synchronization, which overwrites every used gradient and
explicitly writes zeros for missing model gradients in the reference.
- Both modes clear BF16 model gradients exactly as before, preserve the
scaling/pending-sync bookkeeping, and guard unsupported layouts. Probe
missing-gradient cases before time; the complete stack must pass checks.
- Probe/time queue started 2026-11-06T18:59:16Z, PID 140992,
log "/tmp/r12/screens.log", outputs "/tmp/r12/time-{foreach,defer}".
- Both missing-gradient probes pass bitwise. Screens valid: foreach
0.1688 s/update, 1.3796x (19:00:37Z), no gain over R11; deferred
initialization 0.1666, 1.3978x (19:01:58Z), 0.32% window spread.
Select defer and run quick stand-in plus full check before promotion.
- Validation queue started 2026-11-06T19:02:47Z, PID 144068,
log "/tmp/r12/validation.log", outputs "/tmp/r12/quick-standin" and
"/tmp/r12/check". Full-check time excluded from round budget.
- Quick completed 19:07:15Z: PASS/N0. Full completed 19:15:08Z:
PASS/N1, 1.3986x, 0.1665 s/update. Every exact, RNG, and trajectory
gate passes. Promote the exact checked deferred-clearing implementation.
- End/promotion 2026-11-06T19:15:30Z. Active round time excluding the
full check is about 11 minutes. No other hotspot changed.4.0 hRound 13 — current-best profile and loss-path investigation
- Profile the accepted R12 stack before selecting the next hotspot.
Experiments remain isolated from the submitted, fully passing version.
- Profile started 2026-11-06T19:16:15Z, PID 151016,
log "/tmp/r13/profile.log", output "/tmp/r13/profile".
- Profile finished 19:17:45Z. Five updates: NCCL 345.6 ms (30.4%), GEMM
319.8 ms (28.1%), optimizer 259.8 ms (22.8%), elementwise 80.8 ms.
Deferred clearing removes the expected ~23 ms of fills. Loss kernels
total 8.8 ms, in addition to logit widening and full-size temporary
tensors. Screen the installed FlashAttention fused cross-entropy, a
standard memory-pass fusion on the 100,352-wide vocabulary loss.
- Preserve the criterion's module forward, token normalization and ignored
labels; only replace its softmax/NLL computation. FP32 loss/accumulation
and BF16 logit gradients remain. The deterministic Triton kernels have
no atomic reductions. Guard to the YOCO single-model-parallel CUDA BF16
path, default off. This reduction/order change is an N1 candidate.
- Probe production vocabulary sizes, ignored labels, three magnitudes,
output/gradient dtypes, and loss error versus fp64 before judge timing.
- Probe/time queue started 2026-11-06T19:19:20Z, PID 152508,
log "/tmp/r13/screen.log", probe "/tmp/r13/probe_loss.log", timing
output "/tmp/r13/time".
- Primitive probe: FP32 losses/BF16 gradients and ignored-token zeros are
correct. Maximum per-token loss difference is 3.81e-6 across scales;
gradient relative L2 is at most 2.74e-7. Reference/candidate loss errors
versus fp64 are comparable. These are diagnostics, not a verdict.
- /tmp/r13/time/time.json completed 19:20:41Z: valid, 0.1639 s/update,
1.4208x, windows 1.639/1.639 s (0.01% spread). Fast-path log is present.
Run quick exact-invariant gates and the full statistical check with the
real fused loss and Apex norm. No bitwise claim is made for this kernel.
- Validation queue started 2026-11-06T19:21:47Z, PID 154118,
log "/tmp/r13/validation.log", outputs "/tmp/r13/quick" and
"/tmp/r13/check". Full-check time excluded from round budget.
- Quick completed 19:26:17Z: PASS exact gates, N1. Full completed
19:34:21Z: PASS/N1, all six endpoints within 0.637 sigma of reference.
The full timing is 0.1666 s/update (1.3978x), slower than the 0.1639
screen. As in R11, confirm the incremental gain with a controlled
current-submission versus candidate judge-time pair before promotion.
- Controlled repeat started 2026-11-06T19:35:48Z, PID 161173,
log "/tmp/r13/repeat.log", outputs "/tmp/r13/time-{control,repeat}".
- Controlled repeat completed 19:38:23Z, valid: R12 0.1662 s/update
(1.4011x), fused loss 0.1645 (1.4156x). This supports a modest incremental
gain. Preserve the distinction from the full check's 1.3978x timing.
- End/promotion 2026-11-06T19:39:00Z: promote the exact checked fused loss.
Full PASS/N1; all six endpoints within 0.637 sigma. Future N0 stand-ins
must disable both fused RMSNorm and fused cross-entropy.4.4 hRound 14 — communication critical-path profile
- Profile the accepted R13 stack before selecting the next rewrite.
Submission remains fully checked while experiments use a separate tree.
- Profile started 2026-11-06T19:40:45Z, PID 164208,
log "/tmp/r14/profile.log", output "/tmp/r14/profile".
- Profile finished 19:42:15Z. Five updates: NCCL 351.7 ms (31.1%), GEMM
318.2 ms (28.2%), optimizer 260.6 ms (23.1%). Owner-by-owner broadcasts
take 203.0 ms in 260 kernels; Adam takes 168.9 ms in 135 kernels.
Select exchange scheduling: batch both directions with NCCL point-to-
point operations for exactly two ranks, rather than serial broadcasts.
All fp32 bytes still reach their original master-weight destinations.
- Keep Adam arithmetic/states, complete transfer waits, and pipelined BF16
refresh unchanged. PyTorch coalescing can return one shared work handle;
a completion wrapper waits every returned handle before either refresh.
Other world sizes/backends retain the broadcast path. Default-off switch.
- The communication/compute balance can change, so screen the accepted 2M
Adam chunk and previously tested 64K/512K settings within this exchange
stage. A two-rank split/partial-chunk bitwise probe precedes the screens.
- Probe/screen queue started 2026-11-06T19:43:23Z, PID 165744,
log "/tmp/r14/screens.log", probe "/tmp/r14/probe_p2p.log", outputs
"/tmp/r14/time-p2p-{2m,64k,512k}".
- Two-rank probe: bitwise masters, BF16 weights and Adam states for four
updates, with fast-path counters verified. All three screens valid:
2M Adam 0.1564 s/update (1.4889x); 64K 0.1558 (1.4947x); 512K 0.1555
(1.4976x). Last screen finished 19:47:36Z. The 64K/512K gap is small;
select 512K, whose window spread is 0.09%.
- Re-probe the selected setting, then N0 stand-in check with both RMSNorm
and loss eager, followed by the full real-stack check. Submission stays
at the checked R13 version throughout validation.
- Validation queue started 2026-11-06T19:50:38Z, PID 170485,
log "/tmp/r14/validation.log", outputs "/tmp/r14/quick-standin" and
"/tmp/r14/check". Full-check time excluded from round budget.
- Selected two-rank probe is bitwise. Quick completed 19:55:19Z:
PASS/N0 with eager norm/loss. Full completed 20:03:06Z: PASS/N1,
1.4985x, 0.1554 s/update. All exact/RNG/statistical gates pass.
- End/promotion 2026-11-06T20:04:00Z: promote the exact checked exchange
and select 512K Adam chunks. Round active time is about 15 minutes,
excluding the full check. No precision change in the exchange.4.8 hRound 15 — current-best communication profile
- Profile the accepted R14 stack before selecting another optimization. - Profile started 2026-11-06T20:04:45Z, PID 177669, log "/tmp/r15/profile.log", output "/tmp/r15/profile". - Profile finished 20:06:15Z: five updates, NCCL 339.0 ms (32.0%), GEMM 319.5 ms (30.2%), optimizer 197.9 ms (18.7%). Weight send/receive totals 159.9 ms; BF16 gradient all-reduce is now the largest single kernel category, 178.3 ms across 105 calls. - Test direct two-rank gradient exchange: retain BF16 pre-division by two, exchange matching buffers, then add the peer buffer in BF16. CUDA-aware Futures are completed on the postprocessing stream so DDP consumers wait for the addition. Per-bucket receive buffers avoid reuse races. Other worlds/backends/dtypes retain the standard reduction. Default-off switch. - Check the installed Torch Future stream contract and compare two-rank gradients/updates over bucket rebuilds before judge timing. No reduced precision, loss, optimizer, or RNG change is intended. - Probe/time queue started 2026-11-06T20:08:21Z, PID 179204, log "/tmp/r15/screen.log", probe "/tmp/r15/probe_pair.log", timing output "/tmp/r15/time". - Two-rank probe is bitwise for gradients and parameter updates, with active-hook/fallback counters checked. Judge screen completed 20:09:50Z: valid, 0.1648 s/update, 1.4130x, 0.18% window spread. This regresses substantially from R14; reject the direct gradient reduction. - End 2026-11-06T20:12:00Z. Native DDP all-reduce remains in the submission. Only operator probes, not a full numerical verdict, are claimed here.
5.0 hRound 16 — current-best profile for further fusion
- Re-profile the accepted R14 stack before choosing another hotspot. - Profile started 2026-11-06T20:13:51Z, PID 180996, log "/tmp/r16/profile.log", output "/tmp/r16/profile". - Profile finished 20:15:21Z: five instrumented updates, GEMM 319.4 ms (31.2%), NCCL 307.2 ms (30.0%), optimizer kernels 197.8 ms (19.3%). Adam is 91.5 ms and master-weight BF16 refresh is 60.2 ms. Fusion of locally owned refresh into Adam may remove a master-weight read pass. - Experiment only in /tmp/r16/tree: custom CUDA Adam with the pinned Apex 70018365 FP32 formula/compiler flags and one additional BF16 store. Remote chunks retain existing pipelined refresh; transfer waits remain. Default-off YOCO_ADAM_LOCAL_REFRESH, guarded with first-use equality check. Submitted R14 code/config remain unchanged during investigation. - Custom-kernel probe / two-rank probe / judge-time queue started 2026-11-06T20:22:13Z, PID 182739, log /tmp/r16/screen.log. Probe logs: /tmp/r16/probe_adam_refresh.log and /tmp/r16/probe_p2p.log; timing output /tmp/r16/time. Wait for completion before other GPU work. - Primitive CUDA probes pass bitwise over both Adam modes, five step indices, three data scales, two block chunk sizes, odd/partial slices, and metadata overflow boundaries. Initial two-rank probe found stale cached gradient pointers on the second update. Fix plan invalidation to include gradients and states, not just master-weight pointers. No judge timing ran before that correction. - Corrected two-rank probe and timing started 2026-11-06T20:23:40Z, PID 183129, log /tmp/r16/screen-fixed.log; probe output /tmp/r16/probe_p2p_fixed.log and timing output /tmp/r16/time. - Corrected two-rank probe passes all masters, BF16 model weights, Adam states and steps bitwise for four updates. /tmp/r16/time/time.json: valid 0.1550 s/update, 1.5024x, essentially tied with R14's 1.4985x. Further test vectorized FP32 memory accesses within the same fused kernel; unaligned/tail elements retain scalar accesses and identical math. - Vectorized-kernel probes and time started 2026-11-06T20:26:27Z, PID 184884, log /tmp/r16/vector-screen.log; timing /tmp/r16/time-vector. - Vectorized path remains bitwise in primitive/two-rank probes. Its judge time is valid 0.1552 s/update, 1.5005x, 0.82% window spread, again unresolved versus R14. Screen 64K and 256K kernel chunks, then run the stand-in quick check; no promotion justified yet. - Tuning/stand-in validation started 2026-11-06T20:29:57Z, PID 186867, log /tmp/r16/tune.log. Outputs /tmp/r16/time-64k, /tmp/r16/time-256k, /tmp/r16/quick-standin. - 64K chunks: /tmp/r16/time-64k/time.json, valid 0.1545 s/update, 1.5072x. 256K chunks: /tmp/r16/time-256k/time.json, valid 0.1552, 1.5005x. Stand-in /tmp/r16/quick-standin/check.json passes N0. The new multi-rank path is covered by separate bitwise probes. The apparent gain remains under 1%; run current submission then 64K again with the judge to distinguish a repeatable gain from drift. - Confirmation queue started 2026-11-06T20:38:52Z, PID 192141, log /tmp/r16/confirm.log; outputs /tmp/r16/time-control and /tmp/r16/time-repeat. Submission remains checked R14 throughout. - Confirmation completed 20:41:41Z: control valid 0.1553 s/update, 1.4995x, 0.03% window spread; candidate valid 0.1549, 1.5034x, 0.35% spread. Only a 0.26% incremental gain in the controlled pair. Reject the extra custom-kernel complexity: no resolved benefit and no full check justified. End 2026-11-06T20:42:15Z, under 30 minutes. Submitted implementation and configuration remain R14.
5.5 hRound 17 — current-best profile for remaining model work
- Profile the unchanged accepted R14 stack before selecting another hotspot. Started 2026-11-06T20:42:39Z, PID 195202, log /tmp/r17/profile.log, output /tmp/r17/profile. - Profile finished 20:44:09Z: NCCL 353.9 ms (32.8%), GEMM 323.1 ms (30.0%), optimizer 197.4 ms (18.3%). RMSNorm forward/backward totals 15.9 ms, with surrounding broadcast multiply and reduction kernels. Per-parameter scalar BF16 scaling remains 61.5 ms, but its alternative communication routes were already rejected. Target affine RMSNorm memory passes instead, in /tmp/r17/tree only. - Default-off YOCO_FUSED_RMS_AFFINE combines norm and affine forward, then affine/input-gradient backward. Preserve explicit BF16 rounding of normalized activations, normalized gradients and affine-gradient products. Keep PyTorch's weight-gradient reduction. FP32 reductions are reordered (intended N1); deterministic path is the same kernel. Existing non-affine head norms and unsupported inputs retain Apex. - Probe, timing and quick-check queue started 2026-11-06T20:48:14Z, PID 196831, log /tmp/r17/screen.log; outputs /tmp/r17/probe_rms.log, /tmp/r17/time and /tmp/r17/quick. - First probe stopped before GPU execution: this pinned Triton exposes rsqrt in triton.language.math, not directly in triton.language. Correct the import; include non-unit affine weights in the operator probe. No timing or quick check ran before this compatibility correction. - Corrected probe/time/quick started 2026-11-06T20:49:30Z, PID 197018, log /tmp/r17/screen-fixed.log; probe /tmp/r17/probe_rms_fixed.log. - Operator probes pass at widths 128/3072/4096, rows 7/2048, three activation scales and non-unit weights. Maximum errors are small; explicit BF16 boundaries and gradient dtypes are preserved. - /tmp/r17/time/time.json: valid 0.1529 s/update, 1.5230x. /tmp/r17/quick/check.json passes all exact items; not bitwise (N1). About 1.6% incremental gain merits full validation. Add first-use enabled/fallback logging and device/empty-input guards before the full check. Submission remains R14 until the experimental tree passes. - Full check started 2026-11-06T20:56:10Z, PID 200930, log /tmp/r17/check.log, output /tmp/r17/check. Full-check time is excluded from the round budget. No code/config promotion yet. - Full check completed 21:04:17Z: /tmp/r17/check/check.json is PASS/N1, 1.5230x, 0.1529 s/update. Every structure, RNG, exact-item and statistical gate passes. Promote the exact checked kernel/module and switch file. End/promotion 2026-11-06T21:05:00Z; active round time about 14 minutes, excluding the full check.
5.9 hRound 18 — profile the accepted fused-normalization stack
- Profile started 2026-11-06T21:08:04Z, PID 205988, log /tmp/r18/profile.log, output /tmp/r18/profile. The submission is the fully validated R17 stack; select the next hotspot from this trace. - Profile finished 21:09:34Z: five updates, GEMM 321.6 ms (31.8%), NCCL 307.5 ms (30.4%), optimizer 195.8 ms (19.4%). Weight exchange remains the largest kernel, 161.3 ms/130 calls. Fused affine RMSNorm forward/backward now totals 5.2 ms; more local norm fusion has a low ceiling. Hardware topology reports NV18 between the two devices. - Experiment /tmp/r18/tree: direct CUDA IPC DMA copies of each updated FP32 master chunk into the peer's existing replica. A tiny NCCL all-reduce after each rank's copy establishes completion of both DMA transfers before either rank refreshes BF16 weights. All final waits, master buffers and optimizer arithmetic remain. Default-off YOCO_OSS_IPC_DMA, same-host/two-rank/static-allocation guards; no math or RNG change intended. Probe exactness before timing. - IPC probe and judge timing started 2026-11-06T21:12:41Z, PID 207573, log /tmp/r18/screen.log; /tmp/r18/probe_ipc.log and /tmp/r18/time. - Initial two-rank probe passes every master, BF16 model weight, Adam moment and step bitwise across four updates. /tmp/r18/time/time.json: valid 0.1414 s/update, 1.6469x, a substantial improvement over R17. - Before validation, add CUDA device guards and remove an optimizer reference cycle. Extend probes to twelve updates with asynchronous rank skew and changing learning rates, plus offsets spanning 4 GiB. Queue stand-in quick and full validation; no promotion until PASS. - Extended probes, stand-in quick and full check started 2026-11-06T21:16:40Z, PID 209369, log /tmp/r18/validation.log. Outputs /tmp/r18/probe_stress.log, /tmp/r18/probe_large.log, /tmp/r18/quick-standin and /tmp/r18/check. Full portion is excluded from the round budget. The submitted tree remains validated R17. - Extended two-rank and 4-GiB-offset probes pass bitwise. Stand-in quick completed 21:22:05Z, PASS/N0. Full check ran 21:22:05Z–21:30:34Z: /tmp/r18/check/check.json PASS/N1, 1.6469x, 0.1414 s/update; all exact, RNG and statistical gates pass. Screening window spread 0.24%. - Promote the exact checked IPC implementation and configuration. End/promotion 2026-11-06T21:31:30Z; active round time about 15 minutes, excluding the full check. No custom Adam arithmetic is submitted.
6.3 hRound 19 — current-best communication profile
- Profile accepted R18 before choosing another hotspot. Started 2026-11-06T21:33:43Z, PID 217094, log /tmp/r19/profile.log, output /tmp/r19/profile. - Profile finished 21:35:43Z: GEMM 324.0 ms, optimizer 242.6 ms, NCCL 172.7 ms, copies 119.4 ms across five instrumented updates. BF16 gradient all-reduce is the largest kernel at 145.1 ms, plus 61.9 ms in per-parameter BF16 pre-scaling. IPC weight copies are 91.7 ms and their small handshakes 27.3 ms, overlapped with Adam. - Test IPC DMA for gradient buckets in /tmp/r19/tree only. Exchange unscaled BF16 gradients, wait for both copies with an integer NCCL handshake, then fuse separate BF16 pre-divisions and the BF16 sum. CUDA-aware Futures preserve DDP stream dependencies. Receive storage is retained across bucket rebuilds; the reduction has 64-bit offsets and a first-use bitwise check. Default-off YOCO_DDP_IPC_REDUCE. - Averaging/DDP probes and time started 2026-11-06T21:37:51Z, PID 218676, log /tmp/r19/screen.log; /tmp/r19/probe_average.log, /tmp/r19/probe_ddp.log and /tmp/r19/time. Submitted R18 is unchanged. - Primitive averaging passes bitwise for odd/large sizes and scales from 1e-38 to 1e35. Twelve-update, two-rank DDP gradient/parameter probes are bitwise across bucket rebuilds. /tmp/r19/time/time.json is valid, 0.1389 s/update, 1.6765x. Check graph compatibility and 64-bit indexing, then screen 64/128 MiB DDP buckets and run the stand-in quick check. - Graph/large-index probes, tuning and quick queue started 2026-11-06T21:42:45Z, PID 220682, log /tmp/r19/tune.log. Outputs /tmp/r19/probe_graphs.log, /tmp/r19/probe_large_average.log, /tmp/r19/time-64, /tmp/r19/time-128, /tmp/r19/quick-standin. - Graph/DDP probe passes all gradients and weights bitwise for twelve updates. Large averaging probe passes across the signed 32-bit index boundary. Stand-in /tmp/r19/quick-standin/check.json passes N0. - 64 MiB DDP buckets: /tmp/r19/time-64/time.json, valid 0.1378 s/update, 1.6899x. 128 MiB: /tmp/r19/time-128/time.json, valid 0.1384, 1.6826x. Select 64 MiB for the full check; its extra gain over 256 MiB is small, but the complete IPC gradient path improves on submitted R18. - Full check started 2026-11-06T21:52:14Z, PID 226384, log /tmp/r19/check.log, output /tmp/r19/check. Full-check time is excluded from the round budget. Submission remains checked R18. - Full check completed 22:00:47Z: /tmp/r19/check/check.json PASS/N1, 1.6838x, 0.1383 s/update. All exact, RNG and trajectory gates pass. Promote the exact checked gradient hook/kernel and 64 MiB bucket configuration. End/promotion 2026-11-06T22:02:00Z; active round time about 20 minutes, excluding the full check.
6.8 hRound 20 — current-best optimizer/communication balance
- Re-profile accepted R19 before selecting another hotspot. Started 2026-11-06T22:03:25Z, PID 231554, log /tmp/r20/profile.log, output /tmp/r20/profile. - Profile finished 22:05:25Z: GEMM 307.1 ms, copies 260.8 ms, optimizer 242.0 ms (22.8%) across five updates. Adam totals 105.7 ms and master refresh 90.5 ms, up from 91.5/60.2 ms in R16 before IPC. The new communication overlap changes the bandwidth balance enough to re-screen the previously unresolved local Adam/refresh fusion. - Experiment /tmp/r20/tree reuses the corrected, bitwise-probed R16 CUDA kernel and gradient/state-aware cache. Integrate it with the IPC transfer branch, preserving all handshakes and remote refresh. Keep submitted R19 unchanged; screen 64K and 512K work chunks. - Primitive/two-rank probes and timing queue started 2026-11-06T22:06:57Z, PID 233178, log /tmp/r20/screen.log. Outputs /tmp/r20/probe_adam.log, /tmp/r20/probe_fused.log, /tmp/r20/time-64k, /tmp/r20/time-512k. - Primitive Adam/refresh and twelve-update skewed IPC probes pass bitwise. 64K: /tmp/r20/time-64k/time.json, valid 0.1361 s/update, 1.7110x. 512K: /tmp/r20/time-512k/time.json, valid 0.1357, 1.7161x. Unlike R16, the fusion now has a measurable gain. Select 512K (the existing launch setting), add unsupported-path logging, and validate. - Selected probe, stand-in quick and full validation started 2026-11-06T22:13:19Z, PID 236757, log /tmp/r20/validation.log. Outputs /tmp/r20/probe_selected.log, /tmp/r20/quick-standin and /tmp/r20/check. Full portion excluded from round budget. - Selected two-rank probe passes. Stand-in quick completed 22:18:42Z, PASS/N0. Full check completed 22:28:06Z: /tmp/r20/check/check.json PASS/N1, 1.7211x, 0.1353 s/update; all gates pass. Promote the exact checked custom Adam/refresh kernel and 512K configuration. - End/promotion 2026-11-06T22:29:00Z. Active round time about 16 minutes, excluding the full check. The R16 direction is accepted only after the changed IPC balance and fresh probes/timing/full validation in R20.
7.2 hRound 21 — profile the accepted fused optimizer stack
- Profile started 2026-11-06T22:29:21Z, PID 244412, log /tmp/r21/profile.log, output /tmp/r21/profile. The submission remains the fully checked R20 stack. - Profile finished 22:31:21Z: GEMM 307.7 ms (30.5%), copies 260.2 ms, optimizer 184.4 ms, small NCCL handshakes 123.6 ms across five updates. Target the remaining dense GEMMs with a newer library backend. - Loader override variables are reserved, so do not use LD_PRELOAD or LD_LIBRARY_PATH. Instead investigate a direct cuBLASLt API backend, loading an isolated library explicitly without changing ambient Torch or the submitted path. Download CUDA-12.8 cuBLAS into candidate-env isolation; choose deterministic non-split-K algorithms if viable. - Isolated library install started 2026-11-06T22:36:58Z, PID 246084, log /tmp/r21/install.log. No submitted library path or code is changed. - Isolated cuBLAS <ip> is installed under /opt/candidate-env/cublas-12.8, outside the active Python package path. Its Lt library has no CUDA-library DT_NEEDED dependencies. Implement a direct API backend in /tmp/r21/tree with explicit local dlopen, BF16 inputs/outputs and FP32 accumulation. Non-split-K/no-reduction algorithms are selected for deterministic execution in every mode; the old Torch backend and all reserved environment values are untouched. Default-off YOCO_CUBLAS_LT_LIBRARY selects the explicit backend. - Lt primitive probe and judge time started 2026-11-06T22:40:47Z, PID 246125, log /tmp/r21/screen.log; /tmp/r21/probe_lt.log and /tmp/r21/time. Submitted R20 remains unchanged.
7.2 hRound 21 conclusion — 2026-11-06T23:08:30Z
- Start 22:29:21Z; end 23:08:30Z, excluding the full check from 22:52:38Z through 23:02:42Z. Active round time is about 29 minutes. - Profile /tmp/r21/profile: GEMMs 307.7 ms/5 updates (30.5% of summed kernel durations), copies 260.2 ms, optimizer 184.4 ms, small NCCL handshakes 123.6 ms. Target: GEMM library backend, N1. - Isolated cuBLASLt <ip>, explicit dlopen with local symbol scope, BF16 inputs/outputs and FP32 accumulation, cached non-split-K plans. All six production shapes pass finite eager and repeated graph probes; only the 24-wide primitive differs from native Torch bitwise. - /tmp/r21/time/time.json: valid 1.7776x, 0.1310 s/update, 0.28% spread. - Excluding narrow projections caused NaN gradients before update zero (/tmp/r21/time-filtered/time.json, ERROR); that variant was discarded. - /tmp/r21/check-all/check.json: PASS, N1, 1.7763x. Exact items and RNG match, with no dtype or math-mode changes; the statistical gate passes. /tmp/r21/quick-all-standin/check.json passes exact checks, not bitwise. - Validation PID 249801 completed; all R21 jobs have ended.
8.4 hRound 22 conclusion — 2026-11-06T23:42:20Z
- Start 23:09:08Z; end 23:42:20Z, excluding the full check from 23:31:37Z through 23:41:40Z. Active round time: about 23 minutes. - Profile /tmp/r22/profile: GEMMs 29.7%, copies 25.6%, optimizer 20.1%, NCCL 10.9% of summed kernel durations. Target: IPC completion costs. - Add default-off YOCO_IPC_STREAM_SIGNALS for driver sequence-counter handshakes with local publication, remote waits and CUDA event ordering. Keep NCCL as the collectively selected unsupported-device fallback. - Initial serialized-copy-stream version: /tmp/r22/time/time.json, valid 1.7831x at 0.1306 s/update, insufficiently resolved incremental gain. Restore full lookahead by waiting on consumer streams instead. - /tmp/r22/time-async/time.json and /tmp/r22/time-repeat/time.json both measure 1.7899x, 0.1301 s/update. No skipped updates or RNG failures. - Two-rank payload, pipeline/wraparound, eager/graphed DDP and Adam probes are bitwise. /tmp/r22/quick-standin/check.json passes N0. - /tmp/r22/check/check.json: PASS/N1, 1.7858x, 0.1304 s/update. The complete stack retains the previously accepted norm/loss/Lt rounding differences. All exact, RNG and trajectory gates pass. - Validation PID 264092 and all other round-22 jobs have completed.
8.8 hRound 23 conclusion — 2026-11-07T00:04:40Z
- Start 2026-11-06T23:42:41Z; end 2026-11-07T00:04:40Z. About 22 minutes; no full-check exclusion is needed. - Target: paired FFN GEMMs, selected from /tmp/r23/profile with GEMMs 32.6%, copies 28.8%, optimizer 23.2% of summed kernel durations. - Packed forward/wgrad: /tmp/r23/time/time.json, valid 1.7443x, 0.1335 s/update. Forward-only: /tmp/r23/time-forward/time.json, valid 1.7602x, 0.1323 s/update. Contiguous activation outputs with independent backward nodes: /tmp/r23/time-independent/time.json, valid 1.7628x, 0.1321 s/update. All regress relative to R22. - /tmp/r23/profile-packed shows almost unchanged aggregate GEMM time; extra copying and missed vectorized SwiGLU paths explain the loss. Restoring those paths still fails to reach the baseline. - Primitive and repeated graph probes pass bitwise; the quick check /tmp/r23/quick-independent/check.json passes exact checks, N1 stack. No full check is run because performance rejects the whole direction. - All R23 jobs, including PID 280364, have completed. Only this negative result is committed; experimental code and switches remain outside the submission.
9.3 hRound 24 conclusion — 2026-11-07T00:34:30Z
- Start 00:05:00Z; end 00:34:30Z, about 29.5 minutes. No full check. - Profile /tmp/r24/profile: GEMMs 32.7%, copies 29.1%, optimizer 22.9% of summed kernel durations. Target optimizer/transfer chunk granularity. - Screened seven settings plus a repeat and one refinement, all through the judge. Results are /tmp/r24/time-*/time.json and the table above. - Best initial 128 MiB/256K: 1.7927x, 0.1299 s/update. Its repeat is 1.7886x, 0.1302, indistinguishable from the 1.7872x, 0.1303 baseline. Refinement 128 MiB/128K regresses to 1.7844x, 0.1305. - /tmp/r24/quick-standin/check.json passes N0. No full check is run because no incremental gain is resolved. Keep current settings. - All jobs, including confirmation PID 297692, are complete. Commit only the negative-result inventory; no code/configuration is changed.
9.8 hRound 25 conclusion — 2026-11-07T01:04:20Z
- Start 00:34:57Z; end 01:04:20Z, about 29.4 minutes. No full check. - Profile /tmp/r25/profile: GEMMs 32.7%, copies 29.1%, optimizer 22.9% of summed durations. Target the fused Adam kernel's launch geometry. - Tested threads/vector widths 256/4, 768/4, 1024/4, 1024/2, 512/8, 256/8. All primitive checks are bitwise. Times range from 1.7340x to 1.7886x initially; see /tmp/r25/time-*/time.json and the table above. - Leading 768/4 repeats at 1.7927x, 0.1299 s/update, versus its initial 1.7886x, 0.1302. This is not resolved against the existing 512/4 baseline. Larger vectors and most block sizes clearly regress. - Twelve skewed two-rank optimizer updates are bitwise. The final /tmp/r25/quick-standin/check.json passes N0. No full check is justified. - All jobs, including final confirmation PID 315723, completed. Only the negative-result inventory is committed; source and switches stay at the previously passing implementation.