GPT-6.2 Astra 2.01×

Harness: Codex · switches: YOCO_DUPLEX_OSS=1 YOCO_BULK_MASTER=1 YOCO_DDP_VIEW=1 YOCO_OSS_SLICES=24 YOCO_EXACT_RMS=1 YOCO_RETENTION_MODE=graph YOCO_LAYER_GRAPHS=all YOCO_STREAM_REFRESH=1 YOCO_COMPACT_CE=exact YOCO_PEER_COPY=push YOCO_DDP_BUCKET_MB=128 YOCO_LT_LINEAR=128 YOCO_LT_CHOICE=0 YOCO_DDP_TRANSPORT=peer YOCO_FUSED_ADAM_TILE=1024 YOCO_CROSS_SDPA=cudnn-b YOCO_AFFINE_RMS=residual YOCO_AFFINE_WARPS=8 YOCO_DDP_NORM=fused YOCO_NORM_CTA=512 YOCO_EMBED_TILES=3072 YOCO_DIRECT_WGRAD=layers YOCO_RETENTION_PACK=fused4 YOCO_LT_SHAPE_CHOICES={"W-2048-3072-8192":7} YOCO_MODULE_STATE=both

Judged
2.01×
Tier
N1
Self-best
2.02×
Rounds
40
Hours
10.0
API cost
$81.4
Input tokens · 97.4% cached
95 M
Output tokens
0.6 M
Requests
668
Raw work log40 sections
0.0 hRound 1 — baseline profile
- Started 2026-11-17 19:57 UTC on the unchanged d1ed474 tree. Detached judge profile PID 48387, log /tmp/r1/profile.log, output /tmp/r1/profile.
- Frozen reference: 0.2342 s/update, three-run spread 0.38%.
- Rank 0 profile (5 updates): NCCL 28.4% of kernel time (OSS broadcasts 15.3%, bf16 DDP all-reduce 13.1%), GEMM 26.6%, copies/casts 18.0%, elementwise 12.6%, multi-tensor optimizer 7.5%. The broadcasts were serialized on the critical path, unlike much of the all-reduce.
- NCCL_PROTO=Simple alone: PID 49968, /tmp/r1/time.log, /tmp/r1/time, windows 2.374/2.352 s, 0.9911x. Not retained.
- Replaced the two serialized fp32 weight broadcasts in two-rank OSS by a grouped bidirectional exchange. Same master weight buffers on each device, no casts; upstream fallback outside the supported configuration. PID 50912, /tmp/r1/duplex-time.log, /tmp/r1/duplex-time, windows 2.167/2.156 s, 0.2162 s/update, 1.0833x. Promoted 6e67912, YOCO_DUPLEX_OSS=1. Round ended 20:01 UTC. Full check queued (PID 53958, /tmp/r1/check.log, /tmp/r1/check).
0.1 hRound 2 — optimizer traffic
- Profile queued, PID 53957, log /tmp/r2/duplex-profile.log, output /tmp/r2/duplex-profile. Disregard /tmp/r2/profile (PID 52995), a repeat baseline started before the promotion completed.
- Profile started 20:03:19 UTC: duplex took 103.0 ms/5 updates, down from 185.3 ms of broadcasts. New kernel shares: GEMM 28.6%, NCCL 22.7%, copies/casts 19.4%, elementwise 13.6%, multi-tensor optimizer 8.1%. The two unrolled conversion kernels took 11.8% by themselves. The fp32 multiply and zeroing passes add further master-gradient traffic.
- Trial only: reuse backing storage with grad=None between steps, compute the existing Apex fp32 L2 reduction directly from the bf16 inputs, fuse cast/normalization into one Apex multi-tensor pass, and bulk-refresh bf16 weights. The fp32 Adam implementation and state are unchanged. YOCO_BULK_MASTER=1.
- Queued judge timing PID 58928, /tmp/r2/time.log and /tmp/r2/time, after the full check releases the GPUs. PID 59593 then runs the next profile if faster, and /tmp/r2/quick (/tmp/r2/quick.log).
- Non-timing local equivalence probe PID 60679, log /tmp/r2/compare_buffers.log, serialized with the GPU lock. Round 1 full check ended 20:12:57 UTC: PASS, 1.0858x, bitwise PASS and trajectory PASS.
- Timing /tmp/r2/time: windows 1.958/2.004 s, 0.1981 s/update, 1.1822x vs frozen (1.091x incremental). Apex probe: bitwise match of norm, both clipped and unclipped gradients, and refreshed weights. Commit 70e8e4c in trial worktree. Gate 2 pending; previous full PASS remains submitted.
- Ended 20:20:11 UTC: /tmp/r2/quick/check.json PASS, N0, including full-model determinism. Promoted 70e8e4c and its switches.
0.3 hRound 3 — DDP gradient buckets
- Started profile 20:14:15 UTC, /tmp/r3/profile and /tmp/r3/profile.log (PID 59593). GEMM 30.8%, NCCL 30.0%, optimizer 13.6%, copies 11.1%, elementwise 6.6%. At 595 bf16 all-reduces/5 updates (19.9% summed kernel time; overlaps backward), try avoiding the gradient bucket unpack and independent per-parameter division. Test gradient-as-bucket-view, then that plus a native two-rank AVG collective.
- Timings queued after the round 2 quick gate: PID 65655, /tmp/r3/view-time.log, /tmp/r3/avg-time.log; output /tmp/r3/view-time, /tmp/r3/avg-time.
- Ended 20:28 UTC. View alone: 1.915/1.917 s windows, 0.1916 s/update, 1.2223x (1.034x incremental). View+AVG: 1.904/1.907 s, 0.1906 s/update, 1.2288x. The difference is small; prefer view alone, which changes no arithmetic. Commit 8c2d777; queued full check (includes exactness gates), PID 69488, /tmp/r3/check.log, /tmp/r3/check.
- Full check completed: /tmp/r3/check/check.json PASS 1.2217x, N0, trajectory passed. Promoted.
0.5 hRound 4 — overlap inside the optimizer step
- Profile started 20:29:08 UTC, PID 69487, /tmp/r4/profile.log, /tmp/r4/profile on the bucket-view trial.
- GEMM 31.4%, NCCL 30.5%, optimizer kernels 13.9%, copies 9.3%. Sequential master-weight exchange alone took 102.7 ms/5 updates (9.8%) after 71 ms of fused Adam. Pipeline 12 disjoint intervals of the same local fp32 Apex update and duplex exchange. Original optimizer still owns the single step, bias correction, state and hyperparameters. PID 75471, /tmp/r4/time.log, /tmp/r4/time queued after full check; faster path queued for reprofile and quick validation.
- /tmp/r4/time: 1.848/1.849 s windows, 0.1848 s/update, 1.2673x (1.037x incremental). PID 75513 runs the next profile then /tmp/r4/quick, log /tmp/r4/quick.log.
- Ended 20:45:51 UTC: /tmp/r4/quick/check.json PASS N0. Promoted 87d32c8.
0.7 hRound 5 — decoder normalization
- Profile started 20:39 UTC, PID 75513, /tmp/r5/profile.log, /tmp/r5/profile.
- GEMM 31.7%, NCCL 27.5%, optimizer kernels 16.2%, copies 9.4%, elementwise 7.0%. Forward-op sequence IDs around rsqrt and their backward nodes account for ~18.6 + 32.1 ms/5 updates (~5% of kernel time). Tested non-affine Apex RMSNorm with the bf16 weight multiply outside, plus an independent CUDA implementation folding the PyTorch lane/vector reduction and its backward into two kernels. The CUDA build uses nvcc -std=c++17 -arch=sm_90 -O3 --fmad=false -shared -Xcompiler=-fPIC in the candidate environment.
- Apex timing PID 79596, /tmp/r5/time.log; exact op probe PID 81087, /tmp/r5/compare_norm.log. PID 81303 runs /tmp/r5/exact-time (/tmp/r5/exact-time.log), next profile, then full check of the preferred candidate /tmp/r5/check (/tmp/r5/check.log). Replaced the earlier queued job 79598 before it ran.
- Ended 20:49 UTC (full check excluded). Apex 556cebf: 1.699/1.694 s windows, 1.3801x. Custom 8792fb1: 1.700/1.698 s, 1.3785x (1.088x incremental), and zero differing fp32 inverse-RMS values, bf16 outputs or bf16 gradients on all four probe shapes and three input scales. Prefer custom at this unresolved timing difference. Full gate sequence in progress.
- Full check /tmp/r5/check/check.json PASS, 1.3720x, N0 (all 9,616 bitwise and all six trajectory endpoints passed).
1.0 hRound 6 — retention dispatch
- Profile 20:48:25 UTC, /tmp/r6/profile (PID 81303). NCCL 37.9%, GEMM 31.0%, multi-tensor optimizer 14.9%, elementwise 4.1%, copies 3.9%; NCCL overlaps computation. The self mixer accounted for 142 ms host vs 62.9 ms launched GPU work in 65 forwards. Compare CUDA graph replay of the pure differentiable training chunk vs compiling the whole chunk, rather than compiling only its quadratic inner term.
- PID 87951 runs /tmp/r6/graph-time and /tmp/r6/compile-time, logs with matching .log names, after the round 5 full check.
- PID 89513 selects a timing winner for /tmp/r7/profile and /tmp/r6/quick, logs /tmp/r7/profile.log and /tmp/r6/quick.log. Round 5 gate 2 /tmp/r5/check/compare.json: 9,616/9,616 records bitwise identical. Promoted 8792fb1; gate 3 in progress.
- CUDA graph: /tmp/r6/graph-time, 1.626/1.618 s windows, 1.4439x (1.047x incremental), 0.1622 s/update. Full-chunk torch.compile: /tmp/r6/compile-time, ERROR from Dynamo DDP's unsupported higher-order op; not retained, no suppression/fallback. Graph selected for the quick gate.
- Ended 21:06:08 UTC. /tmp/r6/quick/check.json PASS N0. Promoted graph-only cca1db2.
1.1 hRound 7 — projection dispatch and GEMMs
- Profile started 21:00:10 UTC, PID 89513, /tmp/r7/profile.log, /tmp/r7/profile.
- GEMM 35.0%, NCCL 28.6%, multi-tensor optimizer 17.5%, elementwise 4.8%, copies 4.6%; self.mixer host time fell from 142 to 85.5 ms/5 updates. Screen removal of identity model-parallel autograd functions at TP=1, then wider sibling projections sharing a persistent dense parameter storage. The wider GEMM reorders input-gradient accumulation; named parameters and fp32 Adam states remain separate.
- PID 96694: /tmp/r7/local-time, /tmp/r7/packed-time (corresponding .log); then /tmp/r8/profile and /tmp/r7/check if selected.
- Ended 21:08:46 UTC. Local only: 1.639/1.642 s windows, 1.4280x, slower than graph baseline. Wider projections: 1.611/1.607 s, 1.4556x, only 0.8% over baseline despite changing the gradient accumulation tree. Trial commit 1b753c4, neither enabled. PID 96998 then ran /tmp/r8/profile and /tmp/r7/check on the retained graph-only tree: full PASS 1.4475x N0.
1.4 hRound 8 — decoder layer graph replay
- Started 21:08:48 UTC: /tmp/r8/profile, log /tmp/r8/profile.log, PID 96998. GEMM 34.3%, NCCL 30.4%, optimizer 17.0%, elementwise/copies 9.2%. Eager phase forward queued 183.9 ms of kernels in 224.5 host ms; self/cross decoder forward host 140.4/76.0 ms vs GPU 104.0/63.5 ms. Try one graph per decoder layer to cut launch overhead while leaving communication outside. Detached weight aliases are read live each replay; parameter gradients still go through DDP.
- PID 103480 runs /tmp/r8/all-time, /tmp/r8/self-time with matching logs. The first all snapshot had a guard excluding prefilling, which upstream also sets for self-decoder training: only the cross layers were captured, with eager self chunks. Corrected guard passes the actual flag through capture; a distinct /tmp/r8/fixed-time retimes all layers, then selected reprofile and quick check.
- PID 108699 ran the corrected all-layer time and selected reprofile/quick. Initial run 1.3858x (not retained); self-only 1.579/1.577 s 1.4842x; corrected all-layer 1.572/1.576 s 1.4879x (1.030x incremental). Trial 4554657. /tmp/r8/quick in progress.
- Ended 21:28:08 UTC: /tmp/r8/quick/check.json PASS, N0. Promoted 4554657.
1.5 hRound 9 — streamed bf16 refresh
- Started profile at 21:22 UTC, /tmp/r9/profile, /tmp/r9/profile.log, PID 108699. GEMM 34.1%, NCCL 30.3%, optimizer multi-tensor operations 17.3%, elementwise and copies 9.2%. The full fp32→bf16 multi-tensor sweep still takes ~29 ms/5 updates after the exchange. Issue exactly that conversion in slices on a separate stream, waiting for the corresponding NCCL receive rather than for all slices. Keep the ordinary refresh for initialization and synchronize streams before using the weights.
- Timing PID 115087, /tmp/r9/time, /tmp/r9/time.log; selection/reprofile and full check queued, /tmp/r10/profile, /tmp/r9/check, logs /tmp/r10/profile.log, /tmp/r9/check.log.
- Ended 21:29:28 UTC, /tmp/r9/time: 1.533/1.534 s windows, 1.5267x (1.026x incremental). Trial c2a631a. PID 115192: profile selected refresh, full check in progress.
- Full gate 2 /tmp/r9/check/compare.json: 9,616/9,616 bitwise identical. Promoted c2a631a, trajectory pending.
- Final local full result /tmp/r9/check/check.json: PASS 1.5090x N0, including production trajectory.
1.6 hRound 10 — compact vocabulary loss
- Started 21:29:31 UTC, /tmp/r10/profile, log /tmp/r10/profile.log, PID 115192. GEMM ~32%, NCCL ~31%, overlapped multi-tensor optimizer ~21%. Loss category 8.8 ms/5 updates excludes casts and fill for dense NLL gradients. CUDA trace shows 512-thread rowwise float log-softmax and 32-thread NLL sum. Try an existing flash-attn loss and a custom kernel: retain fp32 max and log-sum per row, evaluate just the selected log-probability, let PyTorch do the original NLL scalar reduction, then generate the bf16 logit gradient directly.
- CUDA equivalence probe PID 121554, /tmp/r10/compare_ce.log. Judge time PID 121555, /tmp/r10/flash-time.log, /tmp/r10/exact-time.log and matching output dirs. Winner queued for /tmp/r11/profile and /tmp/r10/quick.
- Probe: zero differing per-token and reduced losses or bitwise logit gradients at all three shapes, including 2048×100288, padding and non-unit upstream gradients. Next profile/quick job PID 121705.
- Ended 21:47:14 UTC. Flash loss 1.5438x (1.524/1.511 s); exact compact loss 1.5510x (0.1510 s/update, 1.016x incremental). /tmp/r10/quick/check.json PASS N0. Promoted a5e3e69; full check PID 130439, /tmp/r10/check, log /tmp/r10/check.log.
- /tmp/r10/check/check.json PASS 1.5287x N0, including production trajectory. Exact timing windows 1.511/1.509 s.
2.0 hRound 11 — peer copy engines
- Profile began 21:41:24 UTC, PID 121705, /tmp/r11/profile.log, /tmp/r11/profile. NCCL bulk copies took 130.5 ms/5 updates (60 SendRecv kernels with 32 CTAs each); bf16 DDP all-reduce 160.0 ms (595 kernels with 24 CTAs). Both overlap tensor work. Test reading the other rank's complete fp32 updates with cudaMemcpyPeerAsync on a copy stream. PyTorch IPC shares the owned bucket with reference-counted lifetime. Two same-host synchronization schemes: NCCL ready markers plus a completion fence, and IPC events with CPU ordering barriers. Both preserve full replicated fp32 masters and bf16 refresh.
- Timing PIDs 131867, 133320; logs /tmp/r11/peer-time.log, /tmp/r11/events-time.log, output directories of the same name without suffix.
- Source-context peer memcpy was slow (0.6844x); IPC events with a Gloo ordering barrier crashed during initialization. Neither is enabled. Mapping the allocation into the consumer CUDA context and issuing an ordinary D2D copy on a dedicated stream achieved 1.5760x, windows 1.486/1.486 s, /tmp/r11/mapped-time, PID 138845; NCCL ready markers and the completion fence retain per-slice ordering.
- Ended 22:07:44 UTC, /tmp/r11/quick/check.json PASS N0, PID 138859; promoted 7f1189c with YOCO_PEER_COPY=mapped (1.016x incremental).
2.2 hRound 12 — DDP collective granularity
- Profile began 22:01:54 UTC (PID 138859), /tmp/r12/profile, /tmp/r12/profile.log. GEMM 325.1 ms/5 updates (33%), NCCL 189.7 (19%, including 162.8 ms/595 bf16 DDP kernels), multi-tensor 188.5 (19%), copy/cast 155.7 (16%). Vary bucket size and limit thread blocks used by the DDP communicator to reduce collective dispatch/competition with backward GEMMs. Also screen the native two-rank AVG reduction. The optimizer communicator is unaffected.
- Timing PID 143318, /tmp/r12/{large,half,small,avg}-time.log and matching directories; 128 MB, 128 MB/12 CTAs, 48 MB/8 CTAs, 128 MB/12 CTAs/AVG. PID 145793 selects the best variant for /tmp/r13/profile and the full check (/tmp/r12/check for a gain, /tmp/r11/check otherwise), logs alongside the output directories.
- Ended 22:13:01 UTC. Large buckets 1.5932x (1.471/1.469 s, 1.011x incremental); capped-CTA variants 1.5645x, 1.5418x, 1.5531x respectively. Keep just the bucket capacity switch (a66cff4). Full check /tmp/r12/check in progress; the newly selected profile shows 205 vs 595 bf16 all-reduces/5 updates, 133 vs 163 ms.
- Gate 2 /tmp/r12/check/compare.json PASS bitwise at 22:20 UTC. Promoted a66cff4; the full check's trajectory is running.
- Full check completed 22:22:23 UTC: /tmp/r12/check/check.json PASS 1.5932x N0 on the submitted configuration, including all 9,616 bitwise records and all six trajectory endpoints.
2.3 hRound 13 — bf16 GEMM backend
- Profile started 22:13:06 UTC, PID 145793, /tmp/r13/profile and /tmp/r13/profile.log: GEMM 321.1 ms/5 updates (34%), multi-tensor 188.6 (20%), copy/cast 155.9 (17%), NCCL 153.1 (16%). Three sm90 128×128×64 GEMM kernels alone total 297 ms. Screen cached cuBLASLt heuristic plans for the three linear products, with bf16 inputs/outputs and fp32 accumulation, filtering out split-K. Compare system Lt, private NVIDIA <ip> (library install PID 149528, /tmp/r13/library-install.log), and its second heuristic. The private library does not replace the training environment or any default library.
- PID 154569, /tmp/r13/times.log: numerical probe for each backend's forward, dX and dW, followed by judge time; logs /tmp/r13/{system,new,second}-{probe,time}.log and corresponding timing directories. PID 154570 queues /tmp/r14/profile and (only for a material gain) /tmp/r13/check on the winning backend, with matching logs. These run following the round 12 full check.
- Ended 22:26:32 UTC. System Lt 1.5856x, private Lt 1.6298x (1.437/1.436 s, 1.023x incremental), second heuristic 1.5986x. All 36 op comparisons, including gradients for workloads of 2048×3072×8192, were bitwise identical. 21feca6 remains in the trial tree pending the full /tmp/r13/check; the selected profile reduces GEMM time from 321 to 303 ms and uses newer nvjet Hopper tiles.
- /tmp/r13/check/check.json PASS 1.6321x N0 at 22:35:45 UTC. All 9,616 compared tensors were bitwise identical and the production trajectory passed. Promoted 21feca6 and both backend settings.
2.6 hRound 14 — fuse shard Adam and local refresh
- Profile 22:26:32 UTC, PID 154570, /tmp/r14/profile, /tmp/r14/profile.log. GEMM 303.3 ms/5 updates (32%), multi-tensor 188.4 (20%), NCCL 162 (17%), copies 155.7 (17%). The optimizer launches 420 fp32 Apex Adam kernels (88.8 ms) and 840 fp32→bf16 refresh kernels (53.5 ms). Build a cached per-slice GPU tile schedule and use one vectorized CUDA kernel for the exact AdamW state update and local bf16 conversion; apply the original refresh only to incoming fp32 weights. Preserve master grad/state objects and the fairseq/OSS step calls. Test 1024 and 4096 elements per block.
- PID 166540 queues /tmp/r14/probe.log (fp32 master, both states and bf16 copy versus Apex on five sizes/five steps), followed by /tmp/r14/{1024,4096}-time with matching logs. PID 166541 selects the candidate for /tmp/r15/profile and /tmp/r14/quick. The test switch files are lt{1024,4096}.json or old{1024,4096}.json in /tmp/r14, depending on the preceding full result.
- Ended 22:38:35 UTC. The probe had zero differing elements for either tile size, all steps and all four output types. 1024: 1.6378x (1.431/1.429 s); 4096: 1.6366x (1.434/1.428 s). Marginal gain only 0.5%, smaller than the frozen noise band. Committed trial 8373c28 separately and kept the previous submitted version. Quick check /tmp/r14/quick pending.
- /tmp/r14/quick/check.json PASS N0 at 22:44:40 UTC. Not promoted for speed.
2.8 hRound 15 — directional peer transport
- Profile began 22:38:36 UTC, /tmp/r15/profile, /tmp/r15/profile.log, PID 166541. GEMM 305.9 ms (32%), multi-tensor 189.3 (20%), NCCL 164.8 (17%), copy/cast 155.5 (17%). The single P2P copy-engine operation totals 119.9 ms/5 updates, more than the 89.0 ms of Adam. On the critical path the source's Adam completes before sending and the receiver must have the full fp32 slice before converting it. Compare source-initiated writes with receiver-initiated reads, using the copy engine or exact vector loads/stores on CUDA SMs. On push paths exchange ready markers after transport and allow later copies to issue before earlier markers finish; the consumer still waits before refresh.
- PID 174383, /tmp/r15/times.log, queues the four push, write32, write96, read64 timings in /tmp/r15 with corresponding -time.log files. PID 174384 selects a version for /tmp/r16/profile and a full /tmp/r15/check on a gain or /tmp/r15/quick otherwise.
- Added two copy-engine variants with round-robin scheduling on three streams to test available copy-engine concurrency. PID 176165, /tmp/r15/multi.log, runs /tmp/r15/{pull3,push3}-time after the first four. PID 176166 replaces the waiting selection job 174384 and considers all six results.
- Ended 22:52:38 UTC. Single-stream push 1.6634x (1.410/1.406 s, 1.021x incremental), 32-/96-CTA write 1.5287x/1.5697x, 64-CTA read 1.5500x, three-stream pull/push 1.6264x/1.6598x. Keep only the copy-engine push path, e96261a. Reprofile: P2P copies 92.2 vs 119.9 ms/5 updates. The full check /tmp/r15/check is in progress; main remains the previous full PASS.
- Full check /tmp/r15/check/check.json PASS 1.6493x N0, 0.1420 s/update, including 9,616 bitwise records and the production trajectory. Promoted e96261a with single-stream YOCO_PEER_COPY=push.
3.0 hRound 16 — master conversion sweeps
- Profile started 22:52:39 UTC, /tmp/r16/profile, /tmp/r16/profile.log, PID 176166. GEMM 304.9 ms/5 updates (33%), multi-tensor 213.9 (23%), NCCL 159.7 (17%), copy/cast 127.8 (14%). Apex launches 825 grad→master scale kernels (31.1 ms) and 840 master→bf16 kernels (68.5 ms). Each is a segmented pointwise conversion of aligned contiguous tensors. Reuse a GPU table of source/destination spans, one CUDA launch per sweep (per slice on refresh); keep the original Apex global norm reduction and pass the dynamic fp32 clip factor directly on device. Optionally capture the unchanged norm arithmetic and conversion in a graph once the DDP gradient addresses stabilize.
- PID 188658, /tmp/r16/times.log, runs an Apex-equivalence probe for both directions (/tmp/r16/probe.log), then the grad, refresh, both, graph, gradgraph timings (/tmp/r16/*-time.log and matching directories). PID 188659 queues /tmp/r17/profile and /tmp/r16/quick on the selected candidate after the tests.
- Probe passed both directions, five repeated norm/scale evaluations for each gradient mode, zero differing fp32 bits and bitwise-identical norm results, with a real graph replay.
- Also compare the existing Apex conversion with 262,144-element instead of 65,536-element chunks (without changing the norm or Adam kernel). PID 196582 runs /tmp/r16/wide-time and /tmp/r16/wide-time.log; PID 196583 replaces the waiting selector and includes this sixth variant.
- Ended 23:10:13 UTC. Grad sweep 1.6610x, refresh 1.6129x, both 1.6152x, both with graph 1.6551x, gradient graph alone 1.6563x, larger Apex chunk 1.6435x. Against the selected 1.6634x, none improved end-to-end time. The refresh traversal increased competition with the overlapping optimizer; eliminating the scalar read did not materially change throughput. Keep the submitted Apex path. Best trial /tmp/r16/grad-time had 1.407/1.413 s windows; /tmp/r16/quick queued.
- /tmp/r16/quick/check.json: PASS N0, 9,616/9,616 bitwise. Trial b519949 remains outside the submitted branch.
3.3 hRound 17 — bf16 gradient exchange
- Profile began 23:10:15 UTC, PID 196583, /tmp/r17/profile, /tmp/r17/profile.log. GEMM 305.0 ms/5 updates (~33%), multi-tensor 214.2 (23%), NCCL 161.8 (17%), copy/cast 127.7 (14%). The single largest kernel is the bf16 DDP SUM ring/LL, 139.9 ms (15%) over 205 buckets. Screen LL128, the resident NCCL's native pre-multiply/SUM, an isolated NCCL 2.27.7 communicator (library install PID 202210, /tmp/r17/library-install.log), and peer pushes into dedicated bf16 receive buffers followed by exact two-input scaling and addition. The peer path fences mailbox reuse after the last bucket; GPU-aware futures order each result on DDP's stream.
- PID 203693, /tmp/r17/times.log, queues the five ll128, premul, new, peer, large timings in /tmp/r17 with matching -time.log names. Ahead of custom-transport timings, compare both ranks' reduced buffers bitwise against divide-then-NCCL-SUM, including changed bucket layouts; logs premul-probe.log, new-probe.log, peer-probe.log. PID 203694 selects the next profile /tmp/r18/profile and a full check /tmp/r17/check for a gain (or /tmp/r17/quick otherwise).
- Ended 23:21:54 UTC. LL128 1.6309x, native PyTorch PREMUL_SUM rejected bf16, private NCCL 1.6447x, peer copy 1.6910x (1.385/1.384 s) and 256 MB peer 1.6885x. Both direct-library and GPU peer op probes were bitwise. Selected single-stream peer reduction with 128 MB buckets; reprofile shows GEMM 276 vs 305 ms/5 updates with less collective interference.
- /tmp/r17/check/check.json PASS 1.6910x N0 at 23:31:09 UTC. Promoted b857eba, including exact 9,616/9,616 records and full trajectory.
3.7 hRound 18 — lossless embedding-gradient exchange
- Profile 23:21:54 UTC, PID 203694, /tmp/r18/profile, /tmp/r18/profile.log. GEMM 276.1 ms/5 updates (31%), memcpy 235.9 (26%), multi-tensor 212.4 (24%), reductions 42.9 (5%). DDP pushes two isolated 616,169,472-byte matrices per update: input and output vocabulary weights. Only the input embedding has structurally sparse local gradients. Detect actual nonzero rows, form the union over ranks, average those rows, and preserve the full dense gradient for the ordinary all-parameter optimizer.
- PID 216983, /tmp/r18/times.log: compare row compaction inside the existing DDP bucket and a dedicated leaf reduction immediately after gradient accumulation (/tmp/r18/{bucket,leaf}-time, matching logs). Both scan actual bf16 bit patterns, include any non-positive-zero bit pattern, agree the row set with a MAX collective, and fall back to a full mean if dense. The leaf variant broadcasts the parameter at setup, respects DDP no_sync, and uses a separate communicator. PID 218699 runs /tmp/r19/profile and /tmp/r18/quick after timing.
- Ended 23:40:51 UTC. Isolated bucket 1.7008x (1.377/1.376 s), post-accumulate leaf 1.6983x (1.381/1.376 s). Despite the lower communication volume, detecting and packing active rows offset much of the benefit: the marginal speedup was under 0.6%. Trial 3669140 not promoted, /tmp/r18/quick queued on bucket mode.
- /tmp/r18/quick/check.json: PASS N0 on all 9,616 bitwise records. No change to the selected configuration.
3.8 hRound 19 — optimizer pipeline balance
- Profile 23:40:54 UTC, PID 218699, /tmp/r19/profile, /tmp/r19/profile.log. GEMM 275.0 ms/5 updates (30%), copy/cast 235.6 (25%), multi-tensor 212.1 (23%), NCCL 66.1 (7%). The fp32 Adam kernels alone take 99.3 ms and refresh takes 66.8 ms. Since switching to pushes, shard copies take about 92 ms, less than Adam (previously 120 ms, more than Adam). Retest the bitwise R14 Adam/refresh fusion in this changed bottleneck and vary the number of independent slices to balance computation and transport.
- PID 221965, /tmp/r19/times.log, runs apex4, apex8, apex24, fused4, fused12, fused24 in /tmp/r19, each with a -time directory and -time.log. PID 221966 selects /tmp/r20/profile, then /tmp/r19/check (on a gain) or /tmp/r19/quick (otherwise). The CUDA source and binary used by fused variants are the ones already checked bitwise in R14.
- Capability-only query for a possible later synchronization experiment (no timings): PID 226230, /tmp/r20/capabilities.log, under the GPU lock between runs. Removed the now-unused raw profiler traces from R11–R16, retaining their reports.
- Ended 23:54:57 UTC. Apex 4/8/24 slices 1.6610x/1.6873x/1.6934x; fused Adam/local refresh 4/12/24 slices 1.6825x/1.7107x/1.7491x. The 24-slice fused configuration took 1.339/1.339 s for its two windows (1.034x incremental). Its full check /tmp/r19/check follows reprofile; retain the previous full PASS until validated.
- Full check /tmp/r19/check/check.json finished November 18 at 00:04 UTC: PASS 1.7183x, N0, 0.1363 s/update. All 9,616 records were bitwise identical and the production trajectory passed. Promoted f313d2a with YOCO_OSS_SLICES=24 and YOCO_FUSED_ADAM_TILE=1024.
4.0 hRound 20 — peer readiness notifications
- Profile began 23:54:57 UTC on November 17, PID 221966, /tmp/r20/profile, /tmp/r20/profile.log. GEMM 276.4 ms/5 updates (31%), copy/cast 232.8 (26%), multi-tensor 145.1 (16%), NCCL 94.6 (11%). The scalar readiness/fence collectives now run 335 kernels, 94.5 ms; some of this is overlapped waiting. Both devices reported native owner visibility for remote writes, and the CUDA driver v2 stream write/wait probe observed the correct value. Replace per-copy scalar collectives with peer-published 32-bit sequence numbers; wait for both the CUDA-visible local copy event and remote ticket. Keep the collective retirement/reuse fences, use cyclic comparisons across wraparound, and retain NCCL on unsupported devices.
- PID 235813 (/tmp/r20/times.log) queues a two-rank opaque-payload correctness probe with multiple in-flight copies and sequence wraparound (/tmp/r20/probe.log), then independent judge timings for signals on optimizer only, DDP only, both, and both with 48 slices (oss, ddp, all, all48 output prefixes). PID 235814 selects /tmp/r21/profile and /tmp/r20/quick after timing.
- For later library-backend experiments, CPU-only dependency staging PID 239012 (/tmp/yoco-libraries/install.log) installs optional cuDNN and cuBLASLt versions in isolated opt-in directories, not on the Python or loader search path. No submitted code uses them.
- Replaced that slow serial download with PID 242593 (/tmp/yoco-libraries/stage.log), which checks each official wheel's SHA-256 before installation. The stream readiness probe passed on both GPUs, including the 32-bit counter wrap.
- Ended 00:09:50 UTC November 18: oss 1.7400x (1.346/1.345 s), ddp 1.7400x (1.347/1.346 s), all 1.7426x (1.341/1.348 s), all48 1.7387x (1.350/1.345 s). No improvement over 1.7491x: the time charged to NCCL markers was mostly overlapped waiting. Trial fb34b34 kept separately; /tmp/r20/quick checks all.
- /tmp/r20/quick/check.json PASS N0, 9,616/9,616 bitwise, at 00:15:59 UTC.
4.3 hRound 21 — dense projection plans
- Profile started 00:09:50 UTC November 18, PID 235814, /tmp/r21/profile, /tmp/r21/profile.log. GEMM 274.6 ms/5 updates (33%), cast/copy 232.6 (28%), multi-tensor 144.6 (17%), NCCL 49.4 (6%). Three dense GEMM kernels alone take 163.7 ms; backward projections execute concurrently with DDP transport. Screen an isolated cuBLASLt <ip> library, a 116-SM backward heuristic hint, and optional split-K with exclusively fp32 intermediate storage. Do not change PyTorch's resident library or model/accumulator types.
- PID 248850 (/tmp/r21/times.log) queues an output, input-gradient and weight-gradient comparison at six shapes, followed by the judge time for each of reserve128, new, reserve129, split129, second129. Each has a -probe.log, -time.log and -time directory in /tmp/r21. PID 248851 (/tmp/r21/finish.log) then profiles the selected best and runs the round's full check on a clear gain, or quick check otherwise.
- Ended 00:23:07 UTC: all sampled forward and gradient outputs equaled PyTorch. 12.8 with the 116-SM hint 1.6934x; 12.9 first plan 1.7387x, 116-SM hint 1.6777x, fp32 split-K permitted 1.7195x, second plan 1.7348x. None improved over 1.7491x. Trial 72dba2b not promoted. /tmp/r21/quick runs on the first 12.9 plan.
- /tmp/r21/quick/check.json: PASS N0, 9,616/9,616 bitwise, at 00:29:12 UTC.
4.5 hRound 22 — disjoint-owner peer gradient reduction
- Profile started 00:23:07 UTC, PID 248851, /tmp/r22/profile, /tmp/r22/profile.log. GEMM 276.8 ms/5 updates (31%), copy/cast 232.8 (26%), multi-tensor 145.1 (16%), NCCL 93.7 (11%). Peer memcpy takes 178.1 ms, and the 205 full-bucket gradient averages take 34.1 ms. Currently both GPUs read and average every gradient. Partition each bucket in half and have each device compute the mean for its half only, publishing the identical bf16 result to both replicas. Try direct peer operands and copy-engine input staging, with different thread-block limits. Both inputs are still rounded before addition. Keep all DDP futures pending until one final collective orders all system-visible peer writes.
- PID 260654 (/tmp/r22/times.log) runs a two-rank bit-pattern probe on fresh/reallocated buffers and odd tails (direct-probe.log, staged-probe.log), then judge time for direct96, direct384, direct1536, staged128, staged640 in /tmp/r22. PID 260655 (/tmp/r22/finish.log) reprofiles and selects a full or quick check based on the speed outcome.
- Ended 00:35:58 UTC. Direct 96/384/1536 CTA limits: 1.6634x/1.6645x/1.5634x. Staged 128/640 CTAs: 1.7183x/1.7020x. Both probes were bitwise for all four iterations on both GPUs, including finite bf16 bit patterns, odd tails, changed shapes and changed storage. Avoiding a duplicate sum did not pay for SM-driven peer traffic; staged reads improved it but still lost to full-bucket copy-engine sends. Trial bd02bf6 not promoted. /tmp/r22/quick on staged128.
- /tmp/r22/quick/check.json passed N0, 9,616/9,616 bitwise, at 00:42:02 UTC.
4.8 hRound 23 — causal grouped-query attention backend
- Profile began 00:35:58 UTC, PID 260655, /tmp/r23/profile, /tmp/r23/profile.log. GEMM 275.0 ms/5 updates (31%), copy/cast 232.3 (26%), multi-tensor 144.6 (16%), NCCL 102.4 (11%). FlashAttention takes 30.7 ms, primarily 20.2 ms in the 65 backward main kernels and 8.3 ms in forward. Test an isolated cuDNN <ip> SDPA graph on the same BF16 tensors, grouped query and KV heads, causal mask and fp32 intermediates/accumulation, preserving the existing projected KV cache. Compare two heuristic modes and the library's deterministic backward. Retain the original fallback where a plan is unsupported.
- PID 272231 (/tmp/r23/times.log) compares forward and all three derivatives with FlashAttention, checks RNG and graph replay, then runs judge timings for cudnn, cudnn-b, and cudnn-ordered with matching probe and time logs. PID 272232 (/tmp/r23/finish.log) profiles the winner and launches the appropriate gate check. The private cuDNN installation is not discoverable by PyTorch on its normal search path.
- Initial probes stopped before execution: the extension imports its public cudnn package when constructing backward graphs, so importing only the extension under a private name was insufficient. Used the public frontend and changed only its isolated wheel's fallback loader to explicitly name its private cuDNN 9 library. Retry PIDs 274249/274250 (/tmp/r23/retry.log, /tmp/r23/finish-retry.log), with -retry-probe.log files and /tmp/r24/selected-profile for the actual next-round profile. /tmp/r24/profile was an early repeat of the passing baseline, not evidence for this change.
- Those probes executed eager SDPA but reused eager-autograd inputs for graph capture, introducing a forbidden legacy-stream dependency. The actual model supplies fresh detached clones. With fresh samples, PID 275388 (/tmp/r23/fresh-probe.log) verified graph forward/backward, unchanged RNG and repeated bitwise deterministic cuDNN backward, max relative L2 0.00385 versus FlashAttention. Screening PIDs 276017/276018 (/tmp/r23/final-times.log, /tmp/r23/finish-final.log) time the three backends and then make the actual next profile at /tmp/r24/after-attention; both earlier R24 profile directories are premature baseline repeats.
- Ended 00:50:10 UTC: heuristic A 1.7702x (1.321/1.324 s), heuristic B 1.7783x (1.318/1.317 s), ordered backward 1.7689x (1.327/1.320 s). The fastest configuration improved 1.017x incrementally and reduced the profiled attention kernels from 30.7 to 17.8 ms/5 updates. Committed trial 5e87641; full check /tmp/r23/check in progress. Main remains the previous full PASS.
- Full check /tmp/r23/check/check.json PASS 1.7742x, N1 at 00:59 UTC: exact items passed, 1,268 of 9,616 tensors were bitwise identical, no dtype/math-mode changes, and all six production endpoints passed. Largest absolute difference was 1.370 seed standard deviations (3 allowed). Promoted 5e87641 with YOCO_CROSS_SDPA=cudnn-b.
5.0 hRound 24 — learned RMS and residual fusion
- Profile began 00:50:10 UTC, PID 276018, /tmp/r24/after-attention, /tmp/r24/after-attention.log. GEMM 276.0 ms/5 updates (33%), copy/cast 234.0 (28%), multi-tensor 145.1 (17%), NCCL 53.9 (6%). RMS kernels took 7.4 ms backward and 5.2 ms forward; bf16 binary kernels add 18.7 ms and residual additions 12.4 ms. The 3072-wide norm still materializes normalized activations and learned-scale intermediates. Fuse the affine operations and optionally the add before FFN normalization, mirroring the previously verified row-reduction order. Keep separate bf16 rounding for the residual sum, normalized value, norm input gradient and weight-gradient products; PyTorch still sums the latter across tokens.
- PID 285038 (/tmp/r24/times.log) selects the R23 full-passing baseline, checks forward/residual/inputs/weights bit patterns at three shapes and four scales, then times scale4, residual4, residual2, residual8 with matching -time.log files. PID 285039 (/tmp/r24/finish.log) profiles the selected best and queues a full or quick check. Separate library libyoco_affine_rms.so does not overwrite the passing RMS backend.
- Ended 01:05:09 UTC. All forward, residual and gradient probes were bitwise identical, including the weight gradients. Scale alone (4 rows/block) 1.8127x; residual fusion at 2/4/8 rows 1.8071x/1.8057x/1.8169x. The best windows were 1.291/1.287 s, 1.022x incremental. Trial aaa2263, full check /tmp/r24/check in progress; keep R23 in main until confirmed.
- Full check /tmp/r24/check/check.json PASS 1.7851x, N1, 0.1312 s/update, at 01:14 UTC. All exact items passed, 1,268/9,616 tensors matched the original reference bitwise, and all six production trajectory endpoints were within 0.977σ. Promoted aaa2263 with YOCO_AFFINE_RMS=residual, YOCO_AFFINE_WARPS=8.
5.2 hRound 25 — overlap clipping-norm reduction with DDP
- Profile started 01:05:09 UTC, PID 285039, /tmp/r25/profile, /tmp/r25/profile.log. GEMM 276.9 ms/5 updates (34%), copy/cast 225.9 (28%), multi-tensor 144.6 (18%), reduce 42.2 (5%). The 205 peer bf16 mean kernels take 36.7 ms, and 825 Apex norm kernels subsequently read the same gradients for another 14.6 ms. Compare running per-bucket fp32 Apex norms on a separate stream versus producing fp32 sums of squares inside the peer average. The final norm still covers every clipped gradient; only fp32 summation order changes. Incomplete coverage, replacement or mutation invalidates the cached norm.
- PID 297897 (/tmp/r25/times.log) checks actual two-rank DDP gradient bits, norm error and mutation fallback, then times overlap, fused4096, fused512, fused128 on the R24 full-passing baseline. PID 297898 (/tmp/r25/finish.log) selects the next profile and either the full or quick check when a probe/timing succeeds.
- Both two-rank probes passed, including rejection of mutated gradients. The first fused result was near the marginal noise band, so replaced the waiting selection task with PID 305924 (/tmp/r25/finish-confirmed.log): repeat the current baseline and fastest configuration back-to-back (/tmp/r25/repeat-base, /tmp/r25/repeat-best, matching logs) before committing to a full check.
- Ended 01:23:07 UTC. Separate Apex norm 1.7974x; fused norm at 4096/512/128 CTAs 1.8311x/1.8617x/1.8099x. On the repeat, baseline 1.8113x (1.297/1.289 s) versus fused512 1.8383x (1.268/1.280 s), 1.015x incremental. Kernel produced the exact same reduced gradients on both GPUs and probe norm relative error was at most 7.5e-8. Actual training reported complete coverage of all 240 trainable gradients. Trial b2e2954, /tmp/r25/check running.
- Full check /tmp/r25/check/check.json PASS 1.8602x, N1 at 01:32 UTC, 0.1259 s/update. All exact items passed, 1,269/9,616 records matched bitwise, and all six statistical endpoints were within 0.099σ. Promoted b2e2954 with YOCO_DDP_NORM=fused and YOCO_NORM_CTA=512.
5.5 hRound 26 — zero-aware peer reduction for embedding buckets
- Profile began 01:23:07 UTC, PID 305924, /tmp/r26/profile, /tmp/r26/profile.log. GEMM 293.5 ms/5 updates (36%), cast/copy 226.5 (27%), multi-tensor 130.3 (16%), reduction 53.0 (6%). Copy engines still move complete buckets (178.3 ms total peer traffic), with the input embedding contributing a vocabulary-sized gradient for a single 2048-token sample. Earlier R18's active-row packing needed dynamic host-visible indices; R22's direct peer reduction moved every tile. Combine actual on-device bit scans with alternating owner tiles: union the masks, compute only nonzero tiles, and publish identical bf16 results to both replicas. No host-side nonzero extraction, no assumption about token IDs or sparsity, and negative zero counts as active. When enabled, both owners' fp32 squared norms also cover every gradient element.
- PID 314101 (/tmp/r26/times.log) probes sparse/dense, signed-zero, mixed and changing-storage buckets on two ranks, with and without norm collection, then times tile widths 3072/1024/6144/24576. PID 314102 (/tmp/r26/finish.log) confirms near/positive results against the then-current baseline and queues the next profile plus gate check. Main stays at R24 until the preceding full check has a verdict.
- Two-rank comparison passed all six cases, including all-zero, sparse, dense, negative-zero and split parameter views. The first four training launches (*-time/time.json) stopped during DDP setup with NameError: torch in the new embedding eligibility check; no update timing from those runs. Added the missing import and restarted the judge screen under PIDs 320112/320113 (/tmp/r26/fixed.log, /tmp/r26/finish-fixed.log) with distinct *-fixed-time output directories. Main is the full-passing R25 revision.

- Ended 01:42:42 UTC. Corrected time runs in /tmp/r26/{3072,1024,6144,24576}-fixed-time screened at 1.8872x / 1.8872x / 1.8573x / 1.8558x; 3072 windows 1.242/1.240 s. Fresh /tmp/r26/repeat-base 1.8587x (1.259/1.260 s) vs /tmp/r26/repeat-best 1.8872x (1.243/1.240 s), 1.015x incremental. Both ranks' six probe cases passed including norm collection; the actual embedding bucket was 308,084,736 elements, or 100,288 tiles. Peer traffic fell from 178.3 to 170.1 ms/5 in the reprofile. Trial 2bbc78f selects YOCO_EMBED_TILES=3072. PID 320113 started the full /tmp/r26/check at 01:44:04 UTC (/tmp/r26/check.log); main remains the last full PASS pending its verdict.

- Full judge /tmp/r26/check/check.json PASS 1.8781x, N1, 0.1247 s/update at 01:52:18 UTC. All exact items passed; 1,269/9,616 bitwise, no dtype or math-mode changes. Endpoint differences in σ units: -0.030, +0.001, +0.033, -0.045, -0.560, +1.232. Promoted 2bbc78f and its exact 3072-element configuration.
5.9 hRound 27 — publish part of each master shard inside Adam
- Profile 01:42:42–01:44:04 UTC, PID 320113, /tmp/r27/profile/profile.md: GEMM 292.5 ms/5 (36%), cast/copy 218.1 (27%), optimizer 130.4 (16%), reductions 49.0 (6%). Peer transfers took 170.1 ms, Adam/refresh 75.9 ms, and remote fp32-to-bf16 conversion 23.3 ms. Test a persistent Adam grid (512 or 2048 CTAs) publishing 0%, 25%, 50%, or 100% of each contiguous outgoing master slice straight to peer memory, with copy engines sending only the disjoint remainder. Both master replicas still receive actual fp32 optimizer results. Require complete slice coverage and native owner visibility; preserve normal exchange otherwise.
- /tmp/trial/r27 builds an independent library, leaving the submitted library untouched. PID 333294 (/tmp/r27/times.log) waits for the R26 verdict, selects that or the previous passing configuration, probes both ranks' remote replicas and all four local Adam/model tensors across three steps, then times local512, quarter512, half512, full512 and half2048. PID 333295 (/tmp/r27/finish.log) confirms promising timings against baseline, profiles the selection at /tmp/r28/profile and runs full or quick gates according to the gain. No integration before a full PASS.

- Ended 01:59:05 UTC. Both ranks matched Apex bitwise for fp32 master, both fp32 moments, local bf16 weights and remote fp32 replicas at three rates and in every mode. Actual training published all 1,711,678,464 local fp32 values for full512. /tmp/r27/{local512,quarter512,half512,full512,half2048}-time: 1.8426x / 1.7919x / 1.7756x / 1.8499x / 1.8212x, respectively; full512 windows 1.268/1.265 s. Direct stores added SM work and lost to the retained copy-engine path (1.8872x). Trial 4bc6a8f remains outside the submission. PID 333295 now runs /tmp/r27/quick on full512 (/tmp/r27/quick.log).

- /tmp/r27/quick/check.json completed at 02:08:49 UTC: PASS for structure and exact invariants, with non-bitwise results on the existing N1 base. No full check was warranted by throughput.
6.1 hRound 28 — gradient bucket size on the peer-reduction path
- Profile 01:59:05–02:00:22 UTC, /tmp/r28/profile, PID 333295, reprofiled the fastest passing (R26) configuration. GEMM 294.1 ms/5 (36%), cast/copy 218.0 (27%), optimizer 130.2 (16%), reductions 49.9 (6%) and NCCL 45.0 (5.5%). There are still 200 dense peer reduction kernels (43.7 ms) and 320 total peer copies (170.2 ms); graph backward and these transfers overlap. R12's 128 MB granularity was selected on native NCCL before the transport, norm and embedding changes. Test 32, 64, 96, 192, 256 and 384 MB vs the existing 128 MB baseline without changing reduction math, precision or communication protocol.
- PID 345217 (/tmp/r28/run.log) launches the seven judge time runs at /tmp/r28/cap<SIZE>-time, confirms a promising winner against baseline, profiles the selected size at /tmp/r29/profile, then runs the full check for a confirmed gain or quick validation otherwise. The installed judge has no sweep subcommand, so every option is measured individually using time.

- Ended 02:14:34 UTC. Judge time at /tmp/r28/cap<SIZE>-time: 32 1.8661x, 64 1.8857x, 96 1.8857x, 128 1.8872x, 192 1.8887x, 256 1.8857x, 384 1.8426x. The 192 MB windows (1.242/1.238 s) overlap the 128 MB baseline (1.241/1.242 s); no reliable gain. No training code change. Documented the sweep separately in f7a0382; kept 128 MB. PID 345217 launched /tmp/r28/quick on 64 MB, log /tmp/r28/quick.log.

- /tmp/r28/quick/check.json: PASS, N1 at 02:20:39 UTC; all exact items passed. Timing did not favor a switch change.
6.4 hRound 29 — put weight-gradient GEMM outputs into their final DDP views
- Profile 02:14:34–02:15:50 UTC, PID 345217, /tmp/r29/profile. GEMM 295.2 ms/5 (36%), cast/copy 217.3 (26%), optimizer 130.4 (16%) and reductions 51.2 (6%). Correlating the actual CUDA memcpy calls with their CPU ranges identified 1,200 torch::distributed::reducer::copy_ operations taking 19.257 ms/5. Of those, 390 copies of 50,331,648 bytes took 10.791 ms and 455 copies of 18,874,368 bytes took 5.011 ms: copying already-computed dense weight derivatives into unit-stride bucket views. Try using those views as the original cuBLASLt output, not changing the GEMM plan, the DDP reduction or the master gradients.
- Trial /tmp/trial/r29 only binds unshared, aligned bf16 weights in an actual DDP bucket after two matching layouts. Re-capture the layer's graph if its destinations change; preserve prior accumulated gradients if they share the destination. ffn restricts to large matrices and layers covers all eligible replayed projections. PID 360541 (/tmp/r29/run.log) first probes two-rank DDP graph gradients, bucket rebuild and accumulation vs an unmodified graph, then runs /tmp/r29/{layers,ffn}-time, fresh baseline/best repeats, /tmp/r30/profile and full or quick validation. Main remains the passing R26 commit.

- Ended 02:26:26 UTC. /tmp/r29/layers-time 1.9355x (1.211/1.209 s); /tmp/r29/ffn-time 1.9212x (1.221/1.216 s); fresh /tmp/r29/repeat-base 1.8558x vs /tmp/r29/repeat-best 1.9056x (1.263/1.261 vs 1.227/1.232 s, 1.027x incremental). Both ranks' probes matched bitwise for the forward, input gradient and all weight gradients through eight steps, including existing gradients and multiple outstanding forwards; input hooks confirmed final-storage pointers before reduction. Training bound 169 gradients across all 26 decoder layers. The reprofile shows exactly 845 fewer DtoD copies (1905→1060) and their time fell from 22.0 to 6.0 ms/5. Committed fe93cbf in the trial tree. PID 360541 started the full /tmp/r29/check at 02:27:48 UTC (/tmp/r29/check.log), pending promotion.

- Full /tmp/r29/check/check.json at 02:36:08 UTC: PASS 1.9103x, N1, 0.1226 s/update. All exact items passed, 1,269/9,616 records bitwise, unchanged dtypes/math modes. Endpoint differences in σ units: +0.004, +0.010, +0.035, -0.157, +0.082, +0.358. Promoted fe93cbf with YOCO_DIRECT_WGRAD=layers.
6.6 hRound 30 — strided tensor-core chunk products
- Profile 02:26:26–02:27:48 UTC, /tmp/r30/profile, PID 360541. GEMM 300.6 ms/5 (37%), cast/copy 201.8 (25%), optimizer 129.9 (16%), reductions 56.6 (7%). Retention still uses three sm80 batched-GEMM kernels totaling 14.0 ms/780 calls, and the separate strided layout-copy kernel takes 12.76 ms/585 calls. Use bf16 tensor-core GEMMs that index the actual batch, head and chunk strides (including token-major and transposed operands) rather than having to flatten or repack the head and chunk axes. Each chunk has one producer per output tile, fp32 accumulation and a bf16 store. The existing gate recurrence, scale, causal mask and bf16 rounding points remain.
- Trial /tmp/trial/r30 compares the two cross-chunk products alone or all four products including the inner-chunk products; compare 64x64x32 and 64x128x64 tiles. PID 370547 (/tmp/r30/run.log) waits for R29's full verdict, then probes both layouts, scopes, the full forward and four gradients, RNG and repeated determinism (/tmp/r30/probe.log), and launches /tmp/r30/{cross64,all64,cross128,all128}-time. A promising option receives fresh baseline and winner repeats, /tmp/r31/profile, and full rather than quick validation.

- Ended 02:45:36 UTC. Both head counts at 512/2048 tokens, both scopes and both tile shapes matched the original for every forward value and every bf16 query/key/value/gate derivative (40/40 whole compare tensors bitwise); RNG and identical repeated results also passed. /tmp/r30/{cross64,all64,cross128,all128}-time: 1.9228x / 1.9308x / 1.9566x / 1.9468x. Cross128 looked promising (1.197/1.197 s) but /tmp/r30/repeat-best 1.9212x (1.219/1.218 s) did not beat /tmp/r30/repeat-base 1.9260x (1.218/1.213 s). No switch change. Trial 27bf8f6. PID 370547 started /tmp/r30/quick for cross128 at 02:46:58 UTC (/tmp/r30/quick.log).

- /tmp/r30/quick/check.json: PASS, N1 at 02:51:49 UTC. No promotion given the repeated timing comparison.
6.9 hRound 31 — produce retention projections in head-major order
- Profile 02:45:36–02:46:58 UTC, /tmp/r31/profile, PID 370547, on the retained R29 tree. GEMM 298.3 ms/5 (38%), cast/copy 201.5 (26%), optimizer 130.1 (16%), reductions 57.3 (7%). The strided copy kernel still takes 12.64 ms/585 calls, rotary 7.42 ms/400 calls. Adjacent-kernel attribution in the preceding trace showed repacks between rotary/scaling and native BMM and immediately before/after retention normalization. Rather than replace the matmuls, write head-major Q/K directly from rotation and return token-major projection derivatives. Compare leaving V as a view, copying it once, and packing Q/K/V together using one rotary kernel (4 or 16 tokens per tile). Native chunk matmuls/recurrence and cross attention are unchanged.
- Trial /tmp/trial/r31 on full-passing fe93cbf. PID 387402 (/tmp/r31/run.log) probes odd token counts, multiple batches and head counts, different/expanded upstream strides, conjugate gradients and complete retention forward/backward against the original (/tmp/r31/probe.log). Then /tmp/r31/{rotary,contiguous,fused4,fused16}-time, confirmation repeats on a promising result, /tmp/r32/profile and the appropriate full/quick check. The submitted configuration remains R29.

- Initial probe passed both nonfused variants but stopped before the fused kernel launched: this Triton frontend could not subscript a tuple-valued constexpr in the nested rotary helper. Supplied its stride components as individual constexprs; retry PID 387822 (/tmp/r31/retry.log, /tmp/r31/probe-scalar-strides.log). No judge time ran on the failed implementation.

- Ended 03:01:45 UTC. Corrected kernel probe matched every rotation and complete retention forward/derivative tensor bitwise, including expanded upstream strides and odd token counts. /tmp/r31/{rotary,contiguous,fused4,fused16}-time: 1.9582x / 1.9648x / 1.9730x / 1.9615x. Fresh /tmp/r31/repeat-base 1.9403x vs /tmp/r31/repeat-best 1.9814x (1.208/1.207 s vs 1.182/1.182 s; 1.021x incremental). Strided-copy time fell from 12.64 to 4.49 ms and separate rotary kernels from 7.42 to 2.60 ms/5. Trial 9b336d8; PID 387822 started /tmp/r31/check on fused4 at 03:03:07 UTC (/tmp/r31/check.log). A separate capability-only query, PID 393977 under the GPU lock, reported multicast support on both devices (/tmp/research-capabilities/multicast.log); no benchmark was run by the query.

- Full /tmp/r31/check/check.json: PASS 1.9714x, N1, 0.1188 s/update at 03:11 UTC. All exact items passed; 1,269/9,616 bitwise records, no dtype/math-mode changes. Endpoint differences in σ units: -0.011, +0.041, +0.076, -0.028, -0.758, +2.159 (limit 3). Promoted 9b336d8 and the exact fused4 switch file.
7.2 hRound 32 — switch-assisted two-rank gradient reduction
- Profile 03:01:45–03:03:07 UTC, /tmp/r32/profile, PID 387822. GEMM 297.0 ms/5 (37%), cast/copy 190.6 (24%), optimizer 130.0 (16%), NCCL 59.9 (7%), reductions 54.7 (7%). Peer copies take 170.56 ms/320 calls, and each dense gradient bucket also requires a consumer reduction/norm. Explore using the switch's actual multicast add rather than sending unmodified input across the link and reading both replicas. Stage exact bf16 half-values through a unicast mapping, then read the hardware two-rank sum through its multicast alias, write each full local result and form fp32 norm partials. Preserve readiness and retirement ordering and the masked embedding path. The dataset, optimizer, tensor dtypes and update stream are unchanged.
- Trial /tmp/trial/r32 uses a CUDA VMM allocation with a shared POSIX descriptor sent over a same-host Unix socket; no global CuPy/Torch allocator or installation is modified. Native source builds into the separate candidate-environment libyoco_multicast_grad.so. The CUDA 12.1 assembler rejected the optional acc::f32 PTX modifier; basic two-input bf16 packed add compiled. PID 402844 (/tmp/r32/run.log) waits for the R31 full result, runs a two-rank probe on random, extreme, subnormal and signed-zero inputs against the submitted bf16 mean and norm, then times 512/128/2048/8192 blocks at /tmp/r32/<blocks>-time only after numerical success. Promising results get repeats, /tmp/r33/profile, then full or quick verification.

- First probe could not map the backing storage; diagnostic PID 403416 (/tmp/r32/diagnostic.log) localized CUDA status 1 to the bind/map stage on both ranks. The driver header requires externally shareable backing for an externally shareable multicast group. Requested a POSIX-shareable backing allocation and rebuilt; retry PID 403512 (/tmp/r32/shareable.log, /tmp/r32/shareable-probe.log). No timing of an inactive fast path.

- Ended 03:19:29 UTC. Shareable backing passed both ranks' random/exact probes (both full local replicas, multiple simultaneous ranges and reuse, fp32 norm, extreme finite/subnormal/signed-zero inputs). Training confirmed a 6,979,321,856-byte fabric mapping and norm coverage of 240 parameters. /tmp/r32/{512,128,2048,8192}-time: 1.7120x / 1.7649x / 1.5900x / 1.5459x. Even 128 blocks (1.327/1.326 s windows) lost clearly to the selected peer path: the SM staging and fabric reads contend with backward. Trial b24f03e not retained. PID 403512 started /tmp/r32/quick on 128 blocks, log /tmp/r32/quick.log.

- /tmp/r32/quick/check.json completed at 03:25:35 UTC: PASS, N1, no exact-item failures. Slower multicast transport stays disabled.
7.5 hRound 33 — contiguous fp32 master-gradient conversion
- Profile /tmp/r33/profile, 03:19:29–03:20:56 UTC, PID 403512: GEMM 300.0 ms/5 (38%), cast/copy 190.6 (24%), multi-tensor 129.9 (16%) and reductions 56.0 (7%). After clipping, bf16→fp32 gradient normalization still launches 825 segmented Apex kernels (31.28 ms/5). The reducer already exposes dense contiguous spans. After the DDP storage repeats, mirror each actual contiguous run in an fp32 buffer and expose its original named/reshaped gradients to the masters. Compare Apex over the flattened runs, a vectorized per-span conversion, and a single persistent device-scale kernel with proportional CTA assignments. Overlapping or unsupported layouts keep the original path; neither norm math nor optimizer state changes.
- PID 412908 (/tmp/r33/run.log) first probes all output bits and overflow flags, odd tails, dynamic factors, reordered spans/storage changes and overlapping-storage fallback (/tmp/r33/probe.log), then times apex, flat, p512, p2048, p8192 under /tmp/r33, including confirmation repeats for a prospective winner, /tmp/r34/profile and corresponding full/quick gates. CUDA extension is separate (libyoco_packed_grads.so). An unavailable BF16-pair bitcast intrinsic in CUDA 12.1 was replaced by a packed integer read before the probe. Main remains the full-passing R31 revision.

- Ended 03:32:32 UTC. The probe matched all fp32 output bits and Apex flags for every mode, including infinities, finite values producing overflow under an extreme multiplier, changed storage, odd tails and overlap fallback. Actual training placed all 240 master gradients in 41 contiguous spans. /tmp/r33/{apex,flat,p512,p2048,p8192}-time: 1.9339x / 1.9648x / 1.9631x / 1.9664x / 1.9714x, respectively. Best windows 1.189/1.188 s, no improvement on the already passing R31 path. Trial 5789461 kept outside main; PID 412908 started /tmp/r33/quick for p8192 at 03:33:55 UTC.

- /tmp/r33/quick/check.json: PASS, N1 at 03:38:55 UTC. No reliable timing benefit, so the existing gradient path stays selected.
7.7 hRound 34 — lossless raw-word transport for fp32 masters
- Profile 03:32:32–03:33:54 UTC, /tmp/r34/profile, PID 412908: GEMM 300.9 ms/5 (38%), cast/copy 190.4 (24%), optimizer 130.0 (16%), reductions 55.8 (7%). Peer copies take 170.5 ms, the Adam/local refresh 75.7 ms and incoming conversion 22.9 ms/5. Reduce link traffic without changing any stored or computed fp32 or bf16 value: within Adam, pack the low bits of four updated fp32 words for bulk transport and send any changed high words to a separately maintained sideband. Reconstruct the complete 32 bits and refresh the bf16 replica together on receipt, independently of the prior replica contents. Unshared, private zero-initialized sidebands and explicit publication/retirement ordering make this lossless even when signs/exponents or externally modified masters change abruptly.
- Trial /tmp/trial/r34 and separate libyoco_wire_adam.so. Two ranks negotiate complete coverage of every slice; other layouts keep full-word exchange. PID 423434 (/tmp/r34/run.log) probes actual local/remote fp32 words, all bf16 outputs and both fp32 moments under changing signs, exponents and deliberately clobbered receiver replicas (/tmp/r34/probe.log). It then times low-24/low-16 bulk planes on 24 slices, plus 24 bits/12 slices and 16 bits/48 slices. No numeric precision reduction or alternative Adam is used. Fresh repeats/profile/check follow any promising result; /submission stays at the full-passing R31 configuration.

- The first wire probe detected seven one-ULP differences in the owned fp32 shard on each rank at step two. Diagnostic PID 423695 (/tmp/r34/diagnostic.log) showed precisely the same differences at the corresponding transmitted replica; the wire itself preserved the producer's words. The first trial library had been compiled with the wrong CUDA math flags. Rebuilt using the existing Adam/Apex backend's documented --use_fast_math rather than --fmad=false; retry PID 423813 (/tmp/r34/matched-math.log, /tmp/r34/matched-math-probe.log). No judge timings were started from the failed probe.

- Ended 03:46:49 UTC. Matched-math probe passed every fp32 master/moment and every bf16 weight bit on both ranks, for both wire formats, both slices, all four rates/steps, sign/exponent changes and clobbered receiver copies. Real training reported all 24 slices using the 24-bit bulk/8-bit sideband path. /tmp/r34/{raw24,raw16,raw24s12,raw16s48}-time: 1.9103x / 1.8558x / 1.9165x / 1.8455x. Best windows 1.223/1.222 s. The extra memory traffic and sideband publications outweighed the smaller bulk transfer. Trial 161e3c5 is not promoted. PID 423813 started /tmp/r34/quick for raw24s12 at 03:48:12 UTC.

- /tmp/r34/quick/check.json: PASS, N1 at 03:53:04 UTC, with no exact-item failures. The full-word path remains selected on throughput.
7.9 hRound 35 — optimizer pipeline depth with full-word peer copies
- Profile 03:46:49–03:48:11 UTC /tmp/r35/profile, PID 423813, retained R31 path: GEMM 299.2 ms/5 (38%), cast/copy 190.3 (24%), multi-tensor 130.0 (16%), reductions 53.9 (7%). The existing schedule has 24 Adam/communication intervals per update; its full-word peer transfers, fused Adam and remote casts still account for 170.23, 75.73 and 22.95 ms/5 respectively. R19 swept only through 24 native slices. Smaller intervals may allow the first peer transfer and last receiver conversion to overlap Adam more closely, at the cost of more host and collective dispatch. Compare 16, 32, 48, 64, 96 and 144 slices against the selected 24; do not change precision or the communication implementation.
- PID 432625 (/tmp/r35/run.log) runs seven separate judge timings /tmp/r35/slices<N>-time, then fresh winner/baseline confirmations on any prospective gain, /tmp/r36/profile and full or quick gates. Main remains 9b336d8 with 24 full-word slices until evidence warrants a change.

- Ended 04:03:05 UTC. /tmp/r35/slices<N>-time: 16 1.9436x, 24 1.9780x, 32 1.9797x, 48 1.9764x, 64 1.9747x, 96 1.9387x, 144 1.9664x. The 32-slice windows 1.185/1.182 s overlap the fresh 24-slice windows 1.184/1.185 s. Keep 24; no training-code change. Trial documentation 91845c9; PID 432625 started /tmp/r35/quick for 32 slices at 04:04:37 UTC (/tmp/r35/quick.log).

- /tmp/r35/quick/check.json finished PASS at 04:09:15 UTC for 32 slices; this did not change the timing decision.
8.2 hRound 36 — graph the complete sharded update pipeline
- Profile 04:03:05–04:04:37 UTC, /tmp/r36/profile, PID 432625: GEMM 296.1 ms/5 (37%), cast/copy 190.8 (24%), multi-tensor 130.0 (16%), NCCL 56.3 (7%), reductions 55.3 (7%). The optimizer phase took 24.34 host ms for four attributed calls, dispatching 24 individual Adam kernels, peer sends/readiness markers, and receiver casts per update. R35 showed shrinking intervals alone stops helping. Capture that complete existing multistream dependency graph to avoid reissuing its commands, while supplying the current learning rate, betas/bias corrections, epsilon and decay from a device buffer every update. The single fp32 optimizer step and all model/master/state writes remain in place.
- Trial /tmp/trial/r36, separate math-matched libyoco_graph_adam.so. Eligibility is agreed on both ranks and a changed pointer layout discards the graph. PID 446283 (/tmp/r36/run.log) compares dynamic eager dispatch and actual multi-rank CUDA graph replay to Apex for both master replicas, both moments, bf16 weights and RNG through four changing steps, then times dynamic24, graph24, graph48, graph96, graph144 at /tmp/r36 if the probe passes. Winner confirmation, /tmp/r37/profile, and full/quick gates follow; the submitted path remains R31.

- The first R36 probe matched all recorded bits on both ranks for all eight updates, then timed out before successful exit; no timing was started by PID 446283. PID 447013 (/tmp/r36/probe-teardown.log) instruments the fence, graph reset, control-group and NCCL destruction to identify the required resource lifetime.

- Explicit graph reset after stream synchronization resolved the shutdown hang (PID 447013, exit 0). The trial now releases captured collectives before normal process-group destruction and at interpreter exit. PID 447100 (/tmp/r36/run-auto.log) repeated the probe with automatic cleanup: both ranks matched every value and RNG for all eight updates and exited normally (/tmp/r36/probe-auto.log), then started the five judge timings. The graph24 real-training log confirms capture of all 24 slices.

- Ended 04:22:52 UTC. /tmp/r36/{dynamic24,graph24,graph48,graph96,graph144}-time: 1.9714x / 1.9681x / 1.9730x / 1.9681x / 1.9582x. Graph48's 1.187/1.186 s windows overlap eager dynamic24's 1.190/1.186 s; neither improves materially on the passing R31 path. Trial fe8f339 is not promoted. PID 447100 launched /tmp/r36/quick for graph48 after the next incumbent profile.

- /tmp/r36/quick/check.json finished PASS, N1 at 04:28:53 UTC with no exact-item errors; the graph trial remains disabled on throughput.
8.5 hRound 37 — shape-specific dense projection plans
- Profile 04:22:52–04:24:14 UTC, /tmp/r37/profile, PID 447100: GEMM 298.0 ms/5 updates (38%), copy/cast 190.3 (24%), optimizer 129.9 (16%), reductions 56.2 (7%). Two dense backward kernels alone take 144.6 ms/5. R13 and R21 used global plan choices. Account for the three dominant layer shapes separately for forward, input-gradient and weight-gradient products in the existing judge traces; each occurs 130–455 times per 5 updates. Compare the eight cached non-split-K heuristics using the judge profiler, then test mixed shape-specific choices with the judge time. No change to the precision, shape, accumulation or tensor bindings. Unlisted shapes keep the global default.

- PID 457639 (/tmp/r37/run.log) queues seven judge profiles (/tmp/r37/choice{1..7}-profile) behind R36's quick check, attributes the recorded dense kernels by graph and projection role, and writes mixed plan choices. For every distinct mix it compares output, dX, dW, repeatability and RNG to the incumbent at four shapes before its judge timing. A promising time gets fresh candidate/incumbent repeats, /tmp/r38/profile and the full check; otherwise the quick gate runs on the fastest mix. Main remains R31.

- All seven global-plan diagnostic profiles completed. The traces favor choices 3/7/3 for FFN-up forward/dW and square dX. All four tested shapes × three products were bitwise, repeatable and RNG-neutral in each mixed-plan probe, but mixed, strong and ffn all failed real judge time with NaN gradient norms on both ranks in the first update (no timing result). None can be submitted. A checkpoint incumbent profile ran at /tmp/r38/profile (04:43:19–04:44:46 UTC). The isolated forward3, dx3 and w7 alternatives test the shared selections separately; PID 473381 started the first, then PID 474906 (/tmp/r37/isolate-rest.log) queues the remaining two behind it and uses a fresh next-round profile directory /tmp/r38/start-profile. No new precision path or relaxed checks.

- Ended 04:52:03 UTC (29m11s from the first profile). Isolated /tmp/r37/{forward3,dx3,w7}-time: 1.9780x / 1.9681x / 1.9881x. All 36 output/gradient comparisons for the three alternatives were bitwise to the incumbent and repeated identically. Isolating the FFN-up weight gradient (w7) avoided the numerical failure of the mixes and repeated at 1.9566x vs an immediately preceding 1.9420x baseline (1.0075x incremental, nonoverlapping windows). Trial 1bf4799. PID 474906 profiled that candidate at /tmp/r38/start-profile, then queues its full check at /tmp/r37/check; do not promote mixed settings or promote the isolated choice before the full verdict.
8.9 hRound 38 — unchanged training-mode and update-count propagation
- /tmp/r38/profile inspected the R31 incumbent (04:43:19–04:44:46 UTC): GEMM 297.7 ms/5 (37%), copy/cast 190.2 (24%), optimizer 130.1 (16%), reductions 56.0 (7%). In steady updates there are distinct idle intervals of 2.26–2.33 ms before input H2D and 2.16–2.20 ms after H2D before embedding. The trainer and task each recursively set every module's training flag; fairseq separately walks the whole model for optional update listeners. Round start 04:52:03 UTC with reprofile /tmp/r38/start-profile on the R37 timing winner. Compare shortening only the no-effect calls: verify the actual tree's mode or absence of listeners, guard all instance dictionaries, registrations and class definitions for mutations, and otherwise call the original PyTorch/fairseq methods. Do not skip optimizer, scheduler or model work.
- A CPU-only actual-LanguageModel probe, PID 480522 (/tmp/r38/probe.log), passed all three modes, changed and mixed child flags, model copy, submodule replacement, custom callbacks, class/instance modification and invalid input. Native CPython helper compiled into a separate libyoco_module_guard.so; unsupported trees/ABIs fall back. PID 482160 (/tmp/r38/run.log) waits for R37's verdict and uses only its passing parent, then screens train, updates, both under /tmp/r38, repeats a promising winner, and runs the next profile and appropriate gates.

- The selected-parent profile /tmp/r38/start-profile finished 04:53:25 UTC: GEMM 294.6 ms/5 (37%), copy/cast 190.4 (24%), optimizer 130.0 (16%), reductions 57.4 (7%), NCCL 50.8 (6%). It independently reproduces idle gaps of 1.71–1.75 ms before H2D and 1.98–2.02 ms before embedding. R37 full check started 04:53:25 UTC (/tmp/r37/check.log).

- R37 full /tmp/r37/check/check.json completed at 05:01:39 UTC: PASS 1.9533x N1 (0.1199 s/update, all exact items matched, 1,269/9,616 bitwise, no dtype or math-mode changes). The six errors in σ were −0.011, −0.012, −0.112, −0.064, −0.146, +0.879. This unpaired full-check speed is below R31's older 1.9714x full check, but both subsequent R37 timings beat the more recent 1.9420x R31 confirmation and the original screen was 1.9881x. Promoted 1bf4799 and only the isolated plan-7 switch; the three failing combinations remain disabled. R38 starts its timings on this new fully checked parent.

- Ended 05:08:54 UTC (prior full check excluded). Real runs recognized 338 ordinary modules and no update listeners. /tmp/r38/{train,updates,both}-time: 1.9814x / 2.0017x / 2.0155x. Both walks guarded repeated 2.0103x (1.164/1.166 s windows) vs fresh 1.9533x R37 (1.199/1.199 s), 1.0292x incremental. Trial b223e01. PID 482160 launched full /tmp/r38/check at 05:10:22 UTC after profiling the winner; main stays R37 until that verdict.
9.3 hRound 39 — multicast stores into physical fp32 master replicas
- Profile 05:08:54–05:10:21 UTC, /tmp/r39/profile, PID 482160: GEMM 294.5 ms/5 (37%), copy/cast 190.3 (24%), optimizer 130.1 (16%), reductions 56.3 and NCCL 56.1 (7% each). Peer DMA takes 170.1 ms/5, the Adam/local refresh 75.7 ms, and incoming conversions 23.1 ms. R27's manual peer stores and R32's staged multicast gradient sums did not improve the respective incumbents. Here, allocate a real independent fp32 master buffer in each GPU's HBM, expose both through the same multicast mapping, and publish each updated master word to both destinations using an SM90 fabric store instead of a local write followed by a dedicated peer copy. Preserve and recast all weights; no master precision, gradient, moment or optimizer step changes. Compare separate versus integrated publication and several receiver readiness intervals.
- Native source and CPU-side mapping in /tmp/trial/r39; the separate libyoco_fabric_adam.so first built with the incumbent --use_fast_math flags (PID 498231, /tmp/r39/build.log). Trials use a real two-rank probe, including ordinary unicast reads/writes to both physical replicas and changed-storage fallback, before judge timings. Main remains unchanged during this experiment.

- Final native build PID 499676 (/tmp/r39/build-final.log, exit 0) includes synchronized mapping teardown. PID 499869 (/tmp/r39/run.log) waits for the GPU lock and probes stage/store publication at variable rates with changed storage, unequal rank shapes, a forced Apex slice, clobbered old replicas and full bit/RNG checks. It then times stage4, store1, store4, store24, store4s48 using the last fully passing parent. The probe log is /tmp/r39/probe.log; fresh repeats and full/quick judge gates follow a successful screen.

- R38 full /tmp/r38/check/check.json completed PASS 2.0155x N1 at 05:18:46 UTC (0.1162 s/update). All exact items passed; 1,269/9,616 tensors bitwise and no dtype or math-mode change. Endpoint differences were −0.029, +0.041, +0.031, −0.158, +0.669 and +1.256σ. Promoted b223e01 and the exact both switch file.
- The R39 two-rank probe passed all 12 actual Adam updates per rank (three protocols, four steps) including full fp32/bf16 word and RNG comparisons. Full-scale stage4 and store1 successfully allocated and exposed 13,958,643,712 bytes of physical multicast-backed storage per rank but failed the very first backward pass, before any Adam step, with NaN gradient norms. These runs are errors, not performance results, and are not submitted. Investigating the lifetime/ordering of storage changes; all finite checks remain in force.

- All five construction-time configurations failed before the first optimizer step (no timings); trial e84ad75. In a separate worktree /tmp/trial/r39bound, bind the physical replicas only when the real Adam call supplies its gradient, parameter and state lists. Validate every master against the live owning parameters, copy their current values, and rebuild that call's views before applying precisely the same update. This avoids rebinding storage during trainer/model construction and also handles checkpoint-initialized masters. PID 508148 (/tmp/r39/bound-run.log) times four such protocols in fresh bound-*-time directories. The first stage run has completed its training windows with normal gradients; no finite checks were disabled.

- Ended 05:31:11 UTC. All four first-use configurations completed real training with both physical fp32 replicas. /tmp/r39/{bound-stage4,bound-store1,bound-store4,bound-store24}-time: 1.6107x / 1.7132x / 1.7221x / 1.7107x. Integrated stores beat staged publication, but even the best (1.360/1.360 s) is much slower than copy-engine exchange. Neither e84ad75 nor first-use fix c650c4d is promoted. /tmp/r39/quick/check.json for bound-store4 completed PASS at 05:37:25 UTC; the selected branch remains R38.
9.7 hRound 40 — local Adam memory access and independent vector loads
- Profile 05:31:11–05:32:38 UTC, /tmp/r40/profile, PID 508148: GEMM 296.7 ms/5 (38%), copy/cast 190.6 (25%), optimizer 130.0 (17%), reductions 59.3 (8%). On the ordinary copy-engine path the last update's 24 local Adam kernels occupy 15.16 ms across a 15.32 ms span; 24 peer copies start at +0.51 ms, finish at +18.59 ms; remote casts end at +19.02 ms. The local kernel loads four streams and writes both moments, the master and bf16 output while peer DMA reads recently updated masters. Change independent vectors prefetched per thread, threads per block and streaming cache hints for state/gradients; leave master writes cached for the copy engine. No change to transfers, arithmetic, precision or clipping.
- Trial /tmp/trial/r40 factors out the original scalar vector update and host bias correction. Build PID 517727 (/tmp/r40/build.log, exit 0), separate libyoco_wide_adam.so. PID 518042 (/tmp/r40/run.log) verified all fp32 master/first/second moment words, bf16 model words, signed zeros, tail/short spans, RNG and unsupported mode through five Adam settings against Apex for all eight variants (/tmp/r40/probe.log). Eight separate judge timings follow, then conditional confirmation, /tmp/r41/profile and a full or quick check. Main remains fully verified R38.
↑ YOCO