GPT-6.2 Astra 2.74×
Harness: Codex · switches: none
- Judged
- 2.74×
- Tier
- N1
- Self-best
- 2.77×
- Rounds
- 65
- Hours
- 10.0
- API cost
- $114
- Input tokens · 97.5% cached
- 140 M
- Output tokens
- 0.8 M
- Requests
- 975
§ ARaw trace
Raw work log65 sections
0.0 hRound 1 — avoid padding work in the decoder
- Started 2026-11-17 19:42 UTC. Clean treec121f04, two GPUs, empty switches. Judge detail profile (two steps), PID 21265,/tmp/r1/profile.log, results/tmp/r1/profile. - Frozen reference: 2747.2 ms/step; reference spread 0.0006. Profile: cuBLAS BF16 GEMMs account for at least 50.4% of device time; NCCL all-gather 9.7%, reduce-scatter 3.0%; BF16 elementwise multiply and add kernels also prominent.aten::nonzero(used to unpad each decoder attention) 872 ms/step inclusive CPU time. Plan: unpad once and execute all decoder layers on real tokens. - Experiment in detached worktree/tmp/llava-opt:LLAVA_PACK_DECODER=1uses the same per-sample positions, causal varlen flash attention and the original individual parameter modules. Restore the padded output before the LM head. Timing followed by a quick check, PID 21868, log/tmp/r1/validate.log, started 19:46 UTC. - Round ended 19:51 UTC./tmp/r1/time/time_tree.json: 1921.4 ms, 1.4298x, 68.17 GiB peak allocated, within the window check./tmp/r1/quick/check.json: PASS, exact items match, tensors not bitwise (N1).
0.2 hRound 2 — profile packed decoder
- PID 23804 waits for round 1 quick check, runs/tmp/r2/profile(started 19:51 UTC) and then the round 1 full check (/tmp/r1/check). Combined log/tmp/r2/profile-and-r1check.log. - Packed trace: GEMMs 50.9%, ZeRO-3 all-gather 11.9%, reduce-scatter 2.4%; after eliminating attention's repeated unpadding, the parameter gathers are prominent. Test resident BF16 model parameters with optimizer/gradient partitioning using upstreamscripts/zero2.json.LLAVA_DEEPSPEED_CONFIGselects the config in the entry point; default unchanged. Trial/tmp/r2/time,/tmp/r2/quick, PID 25347,/tmp/r2/test.log(queued behind full check). - End 20:01 UTC./tmp/r2/time/time_tree.jsonERROR: 1848.2 ms/step (1.4864x, not valid); first loss differs by 0.585% (>0.5%) with unchanged sample assignment. Peak 82.58 GiB, higher than ZeRO-3. Do not promote, do not relax the gate. No precision claim; stopped at the timing sanity check.
0.3 hRound 3 — profile best valid candidate
- New trace of packing + ZeRO-3, PID 27970,/tmp/r3/profile, log/tmp/r3/profile.log, submitted 20:01 UTC. Round 2 quick check still running (the time command reports rejection in JSON rather than a nonzero exit). Stage-2 experiment commitb3cc908is isolated in/tmp/llava-opt. Future experiment uses/tmp/llava-r3. - Round 2 quick result at 20:04:/tmp/r2/quick/check.jsonPASS exact items, NOT_BITWISE; does not override its gate-1 rejection. - Profile: leading GEMMs total 50.8%; all-gather 12.1%; CheckpointFunction forward 388 ms/step inclusive CPU, backward including replay 1234 ms. R1 peak allocated only 68.17 GiB. Allow a configurable budget for saving decoder activations instead of redoing all forward GEMMs. Retain a suffix of layers selected from packed token count and estimated per-layer activation bytes; synchronize the count across ranks to keep ZeRO gathers ordered.LLAVA_ACTIVATION_GIB=58, timing then quick check, PID 29545, log/tmp/r3/test.log, submitted 20:05 UTC. -/tmp/r3/time/time_tree.json: ERROR, out of memory at step 2 (117.15 GiB allocated, 20.56 GiB reserved but not allocated). Conditional next-profile job PID 30122/tmp/r4/profile.logexited without taking the GPUs. Retrying memory configuration in the same round with 41 GiB and the allocator's expandable segments (variable activation sizes). PID 30938,/tmp/r3b/test.log, outputs/tmp/r3b/time, then/tmp/r4/profileand/tmp/r3b/quickon timing success. - End 20:16 UTC. The 58 GiB variant's quick run passed, but production timing OOMed. The 41 GiB configuration: 1735.3 ms/step, 1.5832x, 112.20 GiB allocated, 122.41 GiB reserved./tmp/r3b/quick/check.jsonPASS exact items. Commitd380a8c(N1 stack).
0.6 hRound 4 — fuse decoder elementwise chains around GEMMs
- Profile of 41 GiB configuration at/tmp/r4/profile(PID 30938, 20:12–20:13 UTC). GEMMs 53.2%, collectives 9.6%. Attributing device ops through CUDA launch correlation and host tensor shapes: hidden fp32 RMSNorm chains 9.0% (casting and mul most prominent), head layout/RoPE 6.1%, MLP activations 2.5%. Try installed flash-attn library fusions of these bandwidth-bound chains; independently switchable. No learned parameter replacement. All three enabled fortime+check --quick, PID 34153,/tmp/r4/test.log, started 20:17 UTC, experiment/tmp/llava-r4. -/tmp/r4/time/time_tree.json: 1503.4 ms, 1.8273x, peak 93.18 GiB (19.02 GiB less than R3). Fusions change intermediate rounding (local N2). Quick check pending; do not claim a full PASS yet. - Round ended 20:22 UTC./tmp/r4/quick/check.json: PASS exact items, NOT_BITWISE. Commit9700d7f.
0.7 hRound 5 — profile fused stack
- PID 34878,/tmp/r5/profile.log,/tmp/r5/profile, started 20:19 UTC after successful R4 timing. - Fused profile: leading GEMMs 65.2% of GPU kernel time, all-gather 8.8%, reduce-scatter 3.2%; checkpoint function still accounts for 313 ms inclusive CPU forward, 521 ms inclusive backward. Update estimated activation footprint for the fused implementations (RMSNorm no longer keeps fp32 streams; SwiGLU no longer saves a separate SiLU result), and test a 63 GiB budget. PID 36717/tmp/r5/test.log, started 20:21,/tmp/r5/timethen quick + full on timing success. On OOM, full check the fused R4 version instead. - End 20:28 UTC./tmp/r5/time/time_tree.json: 1326.6 ms/step, 2.0709x, 122.79 GiB allocated;/tmp/r5/quick/check.json: PASS exact items. Full/tmp/r5/checkis running. Commit58a324a. - Full check completed 20:34 UTC:/tmp/r5/check/check.jsonPASS, 2.0661x. Canonical training endpoint difference −0.020σ and probe difference −0.068σ.
0.8 hRound 6 — ZeRO-3 communication
- Profile of R5:/tmp/r6/profile, PID 37403,/tmp/r6/profile.log, 20:24–20:25 UTC. Top GEMMs 59.3%, parameter all-gathers 13.4%, gradient reduce-scatter 4.5%. Screen two independent approaches: use the normal ZeRO-3 optimizer with gathered params retained between uses, or select NCCL's Simple protocol instead of LL. Not changing the ZeRO stage or precision. - Judgetimesequence PID 42511,/tmp/r6/test.log, submitted 20:30, outputs/tmp/r6/time-residentand/tmp/r6/time-simple(GPU lock follows R5 full check). - End 20:37 UTC: persistent weights 1322.5 ms / 2.0773x, only 0.3% over R5 at 133.82 GiB (11 GiB extra); Simple protocol 1336.8 ms / 2.0551x, slower. Not promoted. Kept trial commit22ac318isolated.
1.0 hRound 7 — GEMM library
- Reprofiled retained R5 in/tmp/r7/profile, PID 44868/tmp/r7/profile-and-r6quick.log, 20:37–20:38 UTC. Leading SM90 cuBLAS kernels >60% of GPU time. Test NVIDIA cuBLAS <ip>, using the same PyTorch BF16 GEMMs and unmodified flash-attn on the CUDA 12.8-capable driver. Wheel staged by PID 44085 (/tmp/llava-wheels/download.log) and installed isolated in/opt/candidate-env/optional/cublas128; only the experiment preloads its two cuBLAS libraries. PID 46477/tmp/r7/test.log: timing + quick, started 20:39 UTC. Conditional next-round profile PID 46693/tmp/r8/profile.logchooses the faster timing configuration. - End 20:45 UTC./tmp/r7/time/time_tree.json: 1303.1 ms/step, 2.1083x;/tmp/r7/quick/check.json: PASS exact items.f33ef43documents the optional library and switches. Keep full-passing submission in place pending combined full validation.
1.1 hRound 8 — AdamW memory traffic
-/tmp/r8/profileof cuBLAS 12.8 variant 20:41–20:42 UTC: new NVJet GEMMs lead, collectives 22% attributed GPU kernel time, six multi-tensor Adam update passes take 65 ms (~4.3%). Use native PyTorch fused AdamW while DeepSpeed still owns the FP32 master tensors, both moments and the optimizer step. Test PID 48973/tmp/r8/test.log, submitted 20:43,/tmp/r8/time,/tmp/r8/quick. Next-profile PID 49415/tmp/r9/profile.logselects its parent by judge timing. - InitialtimeandquickERROR: torch's constructor rejectsfused=Trueon ZeRO's unmaterialized CPU placeholders. The actual Adam tensors are CUDA FP32 shards. Retry in/tmp/llava-r8b, selecting the native kernel in an optimizer pre-step hook on the actual parameter group. PID 51648/tmp/r8b/test.log, submitted 20:48,/tmp/r8b/time, then/tmp/r9b/profile+/tmp/r8b/quickif faster. - Deferred selection:/tmp/r8b/time/time_tree.json1289.6 ms, 2.1302x, peak unchanged. Commit04b1e4c. - Round ended 20:55 UTC./tmp/r8b/quick/check.json: PASS exact items. Full R8b check in progress at/tmp/r8b/check. - Full check 21:02 UTC:/tmp/r8b/check/check.jsonPASS, 2.132x. Canonical training +0.003σ, held-out probe +0.073σ; promoted code and switches.
1.1 hRound 9 — grouped projections
-/tmp/r9/profile(20:46–20:47 UTC, best R7), repeated on the fused-optimizer stack if R8b succeeds. NVJet GEMMs dominate; the packed trace has 256 calls for token-by-4096 × 4096-by-4096 alone, and 96 each for two MLP shapes. Test single grouped multiplications and direct packed-QKV flash attention. Only temporary weights are concatenated; save and differentiate the original individual parameters. Use ZeRO's public external-parameter mechanism to collect them at their owning attention/MLP call. - R8b re-profile/tmp/r9b/profile(20:49–20:50) confirms the same GEMM families and collectives dominate. CPU fp64 comparison of grouped linear forward and every gradient passed for three equal/unequal split sizes. Trial including R8b optimizer PID 52867/tmp/r9/test.log,/tmp/r9/time, then quick/full if faster, otherwise full R8b. Next profile PID 53660/tmp/r10/profile.log. - Round ended 20:52 UTC./tmp/r9/time/time_tree.json: 1363.6 ms (2.0147x), slower than 1289.6 ms, memory also increased to 125.81 GiB. Larger GEMMs / packed-QKV did not offset weight assembly and layout/collective costs. Discarded; experiment retained as0f92ab6. Did not spend another full validation on this slowdown.
1.3 hRound 10 — supervised LM head backward
-/tmp/r10/profile, best R8b, 20:56–20:57 UTC. GEMMs ~56%; vocabulary-shaped tensors account for 20.8 ms of copies/fills/softmax excluding the LM head's three dense GEMMs. Dense logits still have to be returned. Retain full logits, but make a second view of just the valid shifted rows for FP32 cross-entropy and compute only their nonzero contributions in the two backward GEMMs. Support dense-logit consumers/gradients too. CPU fp64 checks passed for both loss-only and joint loss+logits objectives.LLAVA_SUPERVISED_HEAD=1, PID 58390/tmp/r10/test.log,/tmp/r10/time,/tmp/r10/quick, submitted 20:58 UTC. -/tmp/r10/time/time_tree.json: 1269.2 ms, 2.1644x, peak 120.01 GiB. Commitbec61ca. - End 21:07 UTC./tmp/r10/quick/check.json: PASS exact items.
1.4 hRound 11 — batched multimodal embedding
- PID 59376/tmp/r11/profile.log,/tmp/r11/profile(21:03–21:04 UTC); 17 embedding ops (16 separate text embedding calls),nonzero35.7 ms anditem18.3 ms inclusive CPU, plus fills/gradient adds on the large embedding table. Replace the row loop for the common 0/1 image-per-sample case with a batch layout computed once from the IDs and mask. Embed all used text IDs together; scatter text/vision features and labels to the original padded positions. Multi-image/spatial-list formats fall back. PID 62445/tmp/r11/test.log,/tmp/r11/time,/tmp/r11/quick, started 21:05 UTC. - 24 CPU reference comparisons (both padding directions, mixed modalities, interior gaps and truncation through text or image patches): outputs bitwise equal and gradients close in fp64. Judge time 1254.1 ms, 2.1905x, peak 120.12 GiB. Commit36f517b. PID 65142/tmp/r12/profile-and-check.logqueues next profile then full/tmp/r11/check. - End 21:12 UTC./tmp/r11/quick/check.jsonPASS exact items. Full validation underway. - Full 21:20 UTC:/tmp/r11/check/check.jsonPASS, 2.1425x. Canonical +0.010σ, held-out +0.026σ. Promoted this stack (its standalonetimewas 2.1905x).
1.6 hRound 12 — frozen vision tower
-/tmp/r12/profile, 21:12–21:13 UTC. Decoder GEMMs and collectives still dominate; the fixed-size frozen CLIP tower makes 48 explicit batched attention matmuls and 24 attention softmaxes and runs one block past the selected feature. Use the installed flash-attn library for unmasked zero-dropout BF16 CLIP attention (retain CLIP's pre-scaled query), run exactly through the selected feature, and coalesce its frozen parameter fetch via ZeRO's external-parameter API. No caching of image features.LLAVA_FAST_CLIP=1,LLAVA_GATHER_CLIP=1, PID 67803/tmp/r12/test.log, submitted 21:14 UTC,time+quick. - CPU fallback test matched all positive and negative feature-layer selections against HF exactly. Next-profile PID 69481/tmp/r13/profile.log. - End 21:25 UTC./tmp/r12/time/time_tree.json: 1223.8 ms, 2.2447x./tmp/r12/quick/check.json: PASS exact items. Commitf5d5a2ais isolated pending a full check.
1.8 hRound 13 — NCCL channel occupancy
- Best R12 profile/tmp/r13/profile21:21–21:22 UTC. All-gather 14.8%, reduce-scatter 10.2%. The preceding trace showed 438 large all-gathers and 228 large reduce-scatters, using 24 CTAs each while decoder matmuls also run. Test caps of 8 and 16 channels independently; no changes to the reduction operation or ZeRO stage. PID 72902/tmp/r13/test.log, submitted 21:23,/tmp/r13/time8and/tmp/r13/time16. PID 72907/tmp/r14/profile-and-check.logprofiles the winner and runs full/tmp/r13/check(with quick first for a new channel cap). - End 21:28 UTC. 8 channels: 2.2525x (1219.6 ms), only +0.35%; 16 channels: 2.2406x, no gain. Retain NCCL's automatic channel count. Kernel interval union: 63.2 ms NCCL alone, 225.9 ms overlapping GEMMs, 56.5 ms overlapping other kernels, rather than all attributed NCCL duration exposed. - Full check of R12 with auto channels completed 21:36:/tmp/r13/check/check.jsonPASS, 2.2062x, endpoints −0.029σ and −0.107σ. Promoted.
1.9 hRound 14 — coalesce decoder parameter fetches
-/tmp/r14/profile(R12 auto channels, 21:28–21:29 UTC): all-gather and reduce-scatter remain top kernels, including hundreds of individual parameter fetches. Try ZeRO-3 external parameter groups at individual attention/MLP boundaries (unit) or once per entire decoder layer (block). Unlike R9, all original GEMM calls and gradient formulas stay untouched. PID 79311/tmp/r14/test.log, submitted 21:35, results/tmp/r14/time-block,/tmp/r14/time-unit. - Next profile / optional quick job PID 79478, log/tmp/r15/profile-and-quick.log. - End 21:39 UTC. Block 1230.5 ms / 2.2326x, unit 1223.2 ms / 2.2459x, effectively unchanged/slower than 1223.8 ms. Do not promote; experimentcd46817.
2.1 hRound 15 — shard-specialized AdamW
-/tmp/r15/profile(current R12, 21:39–21:40): the stock fused AdamW kernel alone takes 57 ms, 3.8% of device time. Its source computes per-step bias corrections in the element loop. Screen a Triton FP32 update on contiguous ZeRO master shards with the bias corrections computed once per shard, plus the DeepSpeed fused AdamW implementation. PyTorch initializes the Triton backend's original FP32 states; all other layouts/options fall back. PID 81883/tmp/r15/test.log, started 21:40 UTC,/tmp/r15/time-flatand/tmp/r15/time-deepspeed. - Initial Triton time ERROR: pinned Triton 2.1 lackstl.fma. Corrected multiply/add variant (85753f7, PID 83900/tmp/r15b/test.log):/tmp/r15b/time1200.8 ms / 2.2878x. DeepSpeed fused AdamW was simpler and faster: 1194.5 ms / 2.2999x, no custom update kernel needed. PID 83224/tmp/r16/profile-and-quick.logprofiles and quick-checks DS. Only DS backend in the selected candidate. - End 21:53 UTC./tmp/r15/quick/check.jsonPASS exact items.
2.2 hRound 16 — ZeRO gradient staging
-/tmp/r16/profileon R15 DS backend, 21:48–21:49 UTC: the FP64 per-parameter norm reduction (8.6 ms) follows widening copies; ZeRO separately flattens/casts the same gradients to FP32 and multiplies by the loss scale even when it equals one. Use the actual contiguous gradient partitions, simultaneously sum their squares in FP64 and write FP32 master gradients, keeping the global norm reductions and FP32 Adam state. Avoid only the identity unscale when clipping is disabled and scale is one. Fall back for noncontiguous/mismatched layouts or offload/model parallel. PID 86028/tmp/r16/test.log, started 21:51 UTC, timing then quick/full if faster, otherwise full R15. -/tmp/r16/time/time_tree.json: 1158.9 ms, 2.3705x. Commit04cf6c8. The_stagekernel appears at 4.9 ms in the next report; fast path is active. - Round ended 21:59 UTC:/tmp/r16/quick/check.jsonPASS, all exact items, non-bitwise tensors as expected. The full/tmp/r16/checkthen started. - Full check 22:06 UTC:/tmp/r16/check/check.jsonPASS, 2.3841x. Canonical training endpoint −0.037σ, held-out probe −0.071σ. Promoted04cf6c8and the identical switch values.
2.3 hRound 17 — token dimension alignment
-/tmp/r17/profile, PID 87035/tmp/r17/profile.log, two steps 21:55–21:56. Dominant GEMMs use noticeably different NVJet tilings across ragged sizes (13,600 and 14,724 real token rows on rank 0). Test aligning the token dimension in the three MLP linear operations to 16 or 128; immediately trim output, with no padding added to attention, labels or the data loader. PID 89729/tmp/r17/test.log, submitted 21:57,/tmp/r17/time16,/tmp/r17/time128. - Alignment 16: 1175.6 ms / 2.3368x, slower than R16. PID 90009,/tmp/r18/profile-and-quick.log, profiles the faster parent after both timings and queues the quick check for a winning alignment. - End 22:07 UTC. Alignment 128: 1177.7 ms / 2.3328x. Extra copies and padded work do not pay off. Commitf1328a4isolated; no new quick check of a losing configuration.
2.5 hRound 18 — direct gradient reduce-scatter
-/tmp/r18/profile(R16, 22:07–22:08 UTC): all-gather 15.2%, reduce-scatter 9.2%, BF16 pre-division 1.3% of device time. Correlating launches in the two-step R16 trace: 17.4 ms ofaten::div_and 6.8 ms copying 8M/22M-element shards. DeepSpeed normally pre-divides a full gradient, reduces into a slice of it and copies the result to the optimizer buffer. Test NCCL AVG directly into the actual local BF16 gradient partition for single contiguous divisible gradients on micro-step zero; retain the original path for accumulation and other layouts. Avoid the unnecessary contiguous-buffer copy for a full single-parameter input bucket.LLAVA_DIRECT_SCATTER=1, PID 95986/tmp/r18/test.log,/tmp/r18/time,/tmp/r18/quick. -/tmp/r18/time/time_tree.json: 1130.0 ms, 2.4312x, peak unchanged, first loss relative deviation 0.051%. Commit010d079. Next profile PID 97075/tmp/r19/profile.log. - End 22:15 UTC:/tmp/r18/quick/check.jsonPASS exact items.
2.5 hRound 19 — lazy reuse of gathered weights
-/tmp/r19/profileon R18, 22:10–22:11 UTC: all-gather 188.8 ms (14.8%) versus gradient reduce-scatter 61.6 ms (4.8%). R6's persistence eagerly gathered all parameters after Adam, even before the next forward needed them; moreover the upstream coordinator releases persistent weights when its dynamic trace is incomplete. Here leave newly gathered trainable Linear/Embedding weights available after forward within an element budget, release normally during backward and before the optimizer updates shards. Do not gather any weight eagerly. Increase the coordinator's allowed live count by the retention budget to keep normal forward prefetch available. Trials 7.2B and 3.4B elements, PID 98709/tmp/r19/test.log, started 22:12 UTC. Next-profile/quick/full PID 98787/tmp/r20/profile-and-check.log. -/tmp/r19/time7200: 1123.8 ms / 2.4445x, 130.58 GiB vs. R18's 120.12 GiB./tmp/r19/time3400: 1126.8 ms / 2.4381x, 126.47 GiB. Small incremental gain; full judge queued on 7.2B. Commit5f2a6ba. - End 22:23 UTC./tmp/r19/quick/check.jsonPASS exact items. Full running at/tmp/r19/check. - 22:31 UTC full verdict/tmp/r19/check/check.json: PASS, 2.4429x, canonical +0.003σ, held-out +0.018σ. Promoted5f2a6baand the checked switches.
2.7 hRound 20 — NCCL library
-/tmp/r20/profile, 22:17–22:18 UTC, confirms lazy weight reuse reduced all-gather attribution from 188.8 to 114.4 ms (9.6%); reduce-scatter is 67.8 ms (5.7%). cuBLAS kernels still dominate the step. Compare newer vendor NCCL runtimes with the bundled 2.18 implementation, with unchanged math, GPU count, collectives and default NCCL tuning. The torch binary's public NCCL symbols have dynamic jump slots.LD_PRELOADselects either 2.27.7 or 2.29.7 from isolated wheel directories; confirmed the loaded 2.27 runtime reports 22707. Dependency PIDs 99691 and 101993,/tmp/nccl-wheels/install*.log; times PID 102138/tmp/r20/test.log; next profile and optional quick PID 103305/tmp/r21/profile-and-quick.log. - 2.27.7:/tmp/r20/time227/time_tree.json1132.2 ms / 2.4265x (no gain). Optional cuDNN 9.10 + frontend dependencies staged separately for an attention trial, PID 103302/tmp/cudnn-wheels/install.log, not installed on the normal import path. - Correctness-only cuDNN ragged-attention API smoke (no timing) queued under the GPU lock, PID 105900/tmp/r21/smoke.log; compares forward and three backwards against FA2 before integrating a new backend. - End 22:24 UTC./tmp/r20/time229/time_tree.json1140.2 ms / 2.4093x, also slower. Keep bundled NCCL, no new winning variant to quick-check. Experiment documentation725f006isolated.
2.8 hRound 21 — cuDNN ragged attention
-/tmp/r21/profileon R19 with bundled NCCL, 22:28–22:29 UTC: tensor-core GEMMs dominate, but the existing FA2 backward main kernel is 46.4 ms (4.1%) and its forward 15.6 ms (1.4%), plus backward preprocessing/conversion. Test cuDNN's Hopper backend in BF16 with FP32 compute, sample-specific packed offsets and sequence lengths, causal masking and deterministic backward. Cache graph plans by device, stream and the model's sequence-length bound rather than compiling every different token count. Optional dependencies live only in/opt/candidate-env/optional. - First two correctness-only API smokes could not build a plan. PID 107348/tmp/r21/smoke2.logwith vendor diagnostics/tmp/r21/cudnn.logestablished the image's NVRTC 12.1 is too old for ragged offsets (cuDNN requires ≥12.3). Retry with isolated NVRTC 12.8.93, PID 109903/tmp/r21/test.log, correctness/tmp/r21/smoke3.log, then judge/tmp/r21/timeand/tmp/r21/quickonly if all forward/backward comparisons pass. - Adding only NVRTC to the search path was insufficient. Preloading the matching CUDA 12.8 runtime and NVRTC with the backend enabled it. PID 110046/tmp/r21/retry.log, config/tmp/r21/runtime.json,/tmp/r21/smoke4.log: output and every Q/K/V gradient matched FA2 within 0.004 relative L2 on short and long ragged sequences. Judge time and quick now running. - End 22:38 UTC./tmp/r21/time/time_tree.json: 1128.4 ms, 2.4345x, slight regression from 1123.8 ms, 4.85% relative window spread./tmp/r21/quick/check.jsonPASS exact items. Do not promote. Experiment8380647.
3.0 hRound 22 — dense projection backend
-/tmp/r22/profile(R19; PID 110563/tmp/r22/profile.log, 22:34–22:35 UTC): top four NVJet GEMM kernels alone account for 58.3% of device time. Rank 0 spent 804 ms in GEMMs. Test PyTorch's cuBLASLt entry point with a cached BF16 zero bias and 32 MiB of Lt scratch, replacing only bias-free q/k/v/o and gate/up/down projections. Screen forward-only and all three GEMMs including input and weight gradients. CPU fp64 comparisons of the custom forward and both gradients passed for 2D and 3D inputs. Timing PID 112651/tmp/r22/test.log,/tmp/r22/time-all,/tmp/r22/time-forward; next profile/optional quick PID 112656/tmp/r23/profile-and-quick.log. -/tmp/r22/time-all: 1143.1 ms / 2.4033x, forward-only/tmp/r22/time-forward: 1166.1 ms / 2.3559x. Both slower than the standard GEMM route. Isolated commit952c66c; separate quick check PID 115678/tmp/r22/quick-all.log. - End 22:45 UTC./tmp/r22/quick-all/check.jsonPASS exact items.
3.0 hRound 23 — within-step token balancing
-/tmp/r23/profile(R19; 22:41–22:42 UTC): top four GEMMs 54.7%, gradient and parameter collectives 13.2%. The preceding profile of the same stack shows 13,600 vs 13,254 packed token rows on the two ranks, with substantial difference in collective wait time. Starting from the original randomized order, one pair of samples may swap between the two original 16-example batches to minimize the work difference, estimated as a linear + small quadratic function of the actual post-image-expansion length. Skip incomplete and mixed-modality steps. Reuse the actual v1 prompt and tokenizer; no image loading for cost estimates. CPU check: 16 row costs matched full preprocessing, every step multiset unchanged, mean packed token-count difference reduced from 358.8 to 14.0 tokens in one independently sampled order. PID 117091/tmp/r23/test.log(time,quick); next-profile and conditional full check/tmp/r24/profile-and-check.log. - PID 117404 next profile./tmp/r23/time/time_tree.json: data gate confirmed reassignment with identical step samples; 1171.4 ms / 2.3452x, slower despite smaller token imbalance. Peak 126.57 GiB (lower), CPU busy in windows 2.49 vs 2.53 cores. Balancing token count alone does not control shape-dependent GEMM throughput. Isolated commit9432104. - End 22:50 UTC:/tmp/r23/quick/check.jsonPASS exact items. Kept the original sampler.
3.1 hRound 24 — residual addition and normalization
-/tmp/r24/profileon R19, 22:46–22:47 UTC: the remaining BF16 additions, RMS forward/backward and their memory traffic surround every decoder block. Test summing the attention output and skip input directly in flash-attn's RMS kernel, which also combines both gradient contributions. Compare this within a block with carrying the unadded MLP branch into the next block's norm (including across checkpointed layers), without changing any parameters. FP64 CPU comparison using three layers matched both layouts for output, input gradient and every trainable gradient. Timing PID 120724/tmp/r24/test.log,/tmp/r24/time-carryand/tmp/r24/time-within. - Optional CPU-only FA3 build for possible further attention work: PID 119878/tmp/fa3-build/build.log. Used official v2.8.3 sources, pinned CUTLASS revision and hash-checked CUDA 12.8.1 redistributables, configured only SM90 BF16 head size 128. The pinned torch lacksCUDAContextLight.h; adjusted the header/optional bindings to torch 2.1, rebuild PID 121616/tmp/fa3-build/rebuild.log. No change to the running submission. - Build succeeded and installed in/opt/candidate-env/optional/fa3. Correctness-only smoke PID 122910/tmp/fa3-build/check.log: both irregular batches matched FA2 for forward and dQ/dK/dV, largest relative L2 0.00113. R25 profile/quick/conditional full PID 121831/tmp/r25/profile-and-check.log. - Cross-layer carry/tmp/r24/time-carry: 1145.2 ms / 2.3989x. Does not improve the best time. - Within-layer fusion/tmp/r24/time-within: 1145.9 ms / 2.3973x. Also slower, no memory reduction at the step level. Isolated commit9133da8. - End 23:00 UTC./tmp/r24/quick/check.jsonPASS exact items; keep unfused residual routing.
3.3 hRound 25 — Hopper FlashAttention-3
-/tmp/r25/profile, 22:53–22:55 UTC: on R19 the FA2 BWD main kernel remains 45.5 ms (3.5%), forward 15.8 ms (1.2%). The optional SM90 BF16 library built during R24 switches only packed causal self-attention with equal head counts, width 128 and no dropout. Other shapes use the original path. Deterministic backward in all runs, no eval/timing detection. Screen normal SM count and a margin of 8 for concurrent NCCL. PID 124669/tmp/r25/test.log,/tmp/r25/time0,/tmp/r25/time8; next profile/quick/full PID 124974/tmp/r26/profile-and-check.log. Optional cuBLAS versions staged separately by PID 124971/tmp/cublas-versions/install.log, inactive. - Full SM count:/tmp/r25/time0/time_tree.json1100.1 ms, 2.4971x; exact and statistical checks not yet done. -/tmp/r25/time8/time_tree.json: 1101.7 ms / 2.4936x; no benefit from reserving 8 SMs. Selected the 0-margin path, commit908f4e2./tmp/r26/profileconfirms FA3's SM90 backward kernel at 23.9 ms, down from FA2's 45.5 ms. - End 23:07 UTC:/tmp/r25/quick/check.jsonPASS exact items. Full/tmp/r25/checkstarted; no promotion yet. - Full/tmp/r25/check/check.jsonat 23:14 UTC: PASS, 2.4987x, canonical −0.009σ and held-out −0.068σ. Promoted908f4e2with the checked switch values.
3.4 hRound 26 — dense GEMM library versions
-/tmp/r26/profileon R25, 23:01–23:02 UTC: NVJet tensor-core GEMMs remain the leading kernels after switching attention. Compare the same PyTorch BF16 projection calls with vendor cuBLAS <ip> and <ip> against the selected <ip>; no epilogue or graph rewrite. Both pairs of shared libraries are isolated under/opt/candidate-env/optionaland selected by preload. PID 129268/tmp/r26/test.log,/tmp/r26/time129,/tmp/r26/time126, followed by profile and quick/conditional full/tmp/r27/profile-and-check.log. -/tmp/r26/time129: 1103.1 ms / 2.4905x, no gain. Next-profile PID 130704. While the validation runs, staged an optimizer I/O trial: the current profile still shows the DS Adam kernel at 28.5 ms, then separate master-to-BF16 copies. Correctness-only test PID 132041/tmp/r27/smoke.logpassed for two odd shard sizes, four steps including zero LR, FP32 master + both moments, and bitwise BF16 cast of the updated master. -/tmp/r26/time126: 1123.1 ms / 2.4462x. Retain cuBLAS <ip>. Experiment documentation0d81c78. End 23:20 UTC: 12.9's/tmp/r26/quick/check.jsonPASS exact items, NOT_BITWISE.
3.6 hRound 27 — fuse Adam shard I/O
-/tmp/r27/profile(R25, 23:14–23:15 UTC) still shows the FP32 Adam update and subsequent conversion copies; the latter bandwidth kernels account for 10 ms/step, in addition to the gradient staging. Unlike the R15 arithmetic-only trial,writeissues both FP32 master and BF16 model-partition stores from one Adam kernel;readwritealso converts the real BF16 gradient to FP32 in registers rather than writing a full intermediate FP32 gradient. Retain FP64 pre-update gradient norm, both FP32 moments, bias correction and the DeepSpeed optimizer entry point. Original path for incompatible groups, offload, clipping/scaling when a raw gradient cannot be used, or unsupported optimizer states. CPU-prepared implementation plus the R26-mentioned GPU correctness probe. PID 134110/tmp/r27/test.log,/tmp/r27/time-readwrite,/tmp/r27/time-write; next-profile, quick and conditional full PID 134310/tmp/r28/profile-and-check.log. - Other sizing observation: over three shuffled CPU sample orders (240 complete steps), 239 steps already retained all decoder layers under the current 63 GiB budget. Raising it further is unlikely to recover much forward replay and costs OOM margin. -/tmp/r27/time-readwrite/time_tree.json: ERROR, rank 1 reported DataLoader workers segfaulting;timing_tree.r1.jsonrecords zero completed steps and no losses. No valid time or speed claim; free shared memory checked after exit (16 GiB device, 17 MiB in use). The independentwritetrial continues. Do not promote the failed path. - End 23:21 UTC./tmp/r27/time-write/time_tree.json: also ERROR, rank 1 DataLoader worker segfault before any step. Neither mode has a valid speed result. Experiment91c972cisolated, no change to the checked optimizer. The R25 parent subsequently completed a detailed profile normally. - Follow-up exact check, PID 143360/tmp/r27/quick-write.log:/tmp/r27/quick-write/check.jsonPASS, NOT_BITWISE at 23:34 UTC. This does not override the timing failures.
3.7 hRound 28 — align the packed decoder once
-/tmp/r28/profile(R25, 23:21–23:22 UTC): the first six dense NVJet GEMM kernels account for 68.1% of device time. Try a single internal zero extension to an aligned row count across the whole decoder instead of R17's padding per MLP projection. Append rows only to the final causal sequence, with no dropout; its original queries cannot see future rows. Strip them before restoring real rows and keep the original rotary length. No change to dataset rows or labels. CPU FP64 comparison matched outputs, every parameter and input gradient for irregular masks. Judge timings PID 141297,/tmp/r28/test.log,/tmp/r28/time16,/tmp/r28/time128; next profile and quick/conditional full PID 141298/tmp/r29/profile-and-check.log. -/tmp/r28/time16: 1103.8 ms / 2.4889x vs 1100.1 ms on R25./tmp/r28/time128: 1116.2 ms / 2.4612x. Neither helped. End 23:31 UTC:/tmp/r28/quick/check.jsonPASS exact items. Isolated6faffe0.
4.0 hRound 29 — frozen CLIP projection bias epilogues
-/tmp/r29/profile(R25, 23:26–23:27 UTC): six leading decoder GEMMs 74%; separate CLIP projection bias additions take 2.7 ms for 115 width-1024 calls, 2.4 ms for 23 width-4096 calls (matched CUDA launch correlations to enclosing CPU ops and shapes). For contiguous frozen 3D CLIP projections flatten just the leading batch and patch axes so the original DeepSpeed Linear takes its 2Daddmmbias-epilogue path; reshape the result back. Try every projection and just the two MLP projections. Original parameters and ZeRO hooks unchanged. Time PID 147494 (started 23:36),/tmp/r29/test.log, next profile/quick/conditional full PID 147925/tmp/r30/profile-and-check.log. - All projections/tmp/r29/time-all: 1098.5 ms / 2.5010x, only 0.15% faster than R25. No promotion on that difference. - MLP-only/tmp/r29/time-mlp: 1101.7 ms / 2.4937x, slower. Experiment commit6629d42remains isolated. The all-projection quick check is/tmp/r29/quick. - End 23:45 UTC./tmp/r29/quick/check.jsonPASS, exact items preserved. Keep R25.
4.0 hRound 30 — bias-free vendor GEMM plan selection
-/tmp/r30/profile(best R25, 23:40–23:41 UTC): the six leading NVJet GEMMs consume 68.2% of kernel time. Unlike R22's zero-bias emulation, use cuBLASLt directly without an epilogue for the original individual BF16 decoder projections. FP32 compute/reduction, 32 MiB of scratch per CUDA stream; cache vendor heuristic descriptors by shape/transposes, without runtime benchmarking or changing the original parameter set. The optional shim is built fromllava/model/language_model/lt_dispatch.cppinto/opt/candidate-env/optional/llava_lt. Compare first and second vendor heuristic, and first only on weight gradients. PID 150969/tmp/r30/test.log(23:42 UTC): locked numerical smoke/tmp/r30/smoke.log, followed by/tmp/r30/time-first,/tmp/r30/time-second,/tmp/r30/time-wgrad. PID 151788/tmp/r31/profile-and-check.logqueues the next profile and quick/conditional full. - Locked Cartesian BF16 numerical smoke passed: relative L2 below 0.0017 against FP32 for all transpose modes and both algorithms; autograd output/dX/dW passed separately./tmp/r30/time-first: 1105.0 ms / 2.4862x;/tmp/r30/time-second: 1106.0 ms / 2.4840x;/tmp/r30/time-wgrad: 1100.9 ms / 2.4954x. None beats the standard cuBLAS calls. Commit4527762isolated, quick check underway. - End 23:54 UTC./tmp/r30/quick/check.jsonPASS exact items. Retained R25.
4.2 hRound 31 — dispatch frozen blocks at the gather boundary
-/tmp/r31/profileon unchanged R25, 23:49–23:50 UTC: the six largest GEMMs account for 68.3% of kernel time, NCCL all-gather 10.2%. Theselected_hiddenCPU range alone spans 123.9 ms under the profiler; every individual norm, linear and activation goes through its own distributed module wrappers even though R12 has already gathered every frozen weight on the outer tower. Call the same per-layer operations and BF16 flash attention directly only inside this externally gathered, no-grad frozen block path. Ordinary towers and non-BF16 paths remain unchanged. Correctness smoke and judge timing PID 155749,/tmp/r31/smoke.log,/tmp/r31/test.log,/tmp/r31/time. - Numerical smoke passed bitwise for all three successive block outputs at two different batch sizes. In the parent profile, vision operations spanned 121.4 ms of GPU timeline on the main stream, but used only 26.8 ms of that interval; the separate communication stream overlapped much of it./tmp/r31/time/time_tree.json: 1075.2 ms / 2.5551x (+2.3% over R25). Next profile and quick/full PID 156215/tmp/r32/profile-and-check.log. Do not promote until full verdict. -d351d9aarchived. Guarded promotion PID 159968/tmp/r31/promote.logselects that commit and matching config only after the faster full PASS (including source/config checks). - Quick PASS 2026-11-18 00:00 UTC. Full/tmp/r31/check/check.json: PASS, 2.5529x at 00:07 UTC, canonical −0.063σ and held-out probe −0.152σ. Promotedd351d9aand the matching switches.
4.3 hRound 32 — weight prefetch depth with outer vision gather
-/tmp/r32/profile(R31, 23:55–23:56 UTC): CLIP's selected feature path spans just 27.9 ms on the CPU and 23.8 ms on the device (20.2 ms of kernels) after grouping; graph-capturing the remaining gaps has little headroom. The six largest GEMMs account for 71.9% of device time; all-gather 67.0 ms (6.1%) and reduce-scatter 49.5 ms (4.5%). This time no separate communication stream overlapped the CLIP blocks. Test the ordinary ZeRO coordinator's prefetch bucket with R19 reuse enabled: 64M, 256M or zero elements vs HF's auto ~15M. No weight, gradient or optimizer math changes. PID 159661/tmp/r32/test.log23:57 UTC, outputs/tmp/r32/time64,/tmp/r32/time256,/tmp/r32/time0. Next profile/quick/conditional full PID 159969/tmp/r33/profile-and-check.log. -/tmp/r32/time64: 1096.7 ms / 2.5050x;/tmp/r32/time256: 1079.6 ms / 2.5447x;/tmp/r32/time0: 1078.3 ms / 2.5478x. None beats R31. End of timing 2026-11-18 00:10 UTC; isolatedd2edd3b, zero-prefetch quick check underway. Retain auto depth. - End 00:16 UTC:/tmp/r32/quick/check.jsonPASS exact items.
4.5 hRound 33 — vectorized Adam shard stores without replacing the optimizer
-/tmp/r33/profile(R31, 00:10–00:11 UTC) shows 28.4 ms (2.4%) for FP32 Adam itself, plus 10.0 ms (0.9%) in the main conversion-copy kernel. The first two GEMM kernels account for 33.7%. Unlike R27, preserve the actual FusedAdam step, its normal FP32 gradient inputs, its own state initialization and counter increments. At the first real gradient norm call, install a per-instance low-level update operation that writes the FP32 weights, FP32 moments and BF16 ZeRO model shard together; skip only the separate cast/copy after a successful fused update. Precompiled CUDA code, not Triton JIT; no new optimizer imports/library load before DataLoader workers launch. Fallback on incompatible tensors, devices or offload. CPU compilation PID 163628/tmp/r33/build.logduring R31 full; source and binary in the experiment tree and/opt/candidate-env/optional/llava_adamrespectively. Correctness-only PID 164336/tmp/r33/smoke.logPASS for aligned, offset and odd-length shards over four steps and both vector/scalar kernels. Times PID 165833/tmp/r33/test.log,/tmp/r33/time-vector,/tmp/r33/time-scalar. -/tmp/r33/time-vector: 1063.4 ms / 2.5834x;/tmp/r33/time-scalar: 1063.9 ms / 2.5823x. Both production timings completed with the real DataLoader, unlike R27. Commit28a29e8, optional binary built with CUDA 12.1nvcc -std=c++17 -O3 --use_fast_math -gencode=arch=compute_90,code=sm_90 -Xcompiler -fPIC --shared. Next profile/quick/full PID 168729/tmp/r34/profile-and-check.log. Guarded promotion is queued at/tmp/r33/promote.log; the submitted tree stays on R31 until a faster full PASS. - Quick PASS 00:22 UTC; full/tmp/r33/check/check.jsonPASS, 2.5872x at 00:30 UTC, canonical −0.065σ and held-out −0.103σ. Promotion job selected28a29e8and its exact vector configuration.
4.7 hRound 34 — replicate immutable CLIP weights, not image features
-/tmp/r34/profile(R33 vector, 00:17–00:18 UTC): top six GEMMs ~72%, all-gather 7.4%, gradient scatter 4.6%. The new CUDA Adam writer is active. The outer frozen vision tower took 47.3 ms inclusive on CPU, while its selected-feature body took 16.0 ms; coalesced ZeRO still repeatedly fetches the frozen parameters. Try keeping a private frozen CLIP weight replica: initialize the identical transformer structure on meta without RNG draws, load the actual fully gathered tensors and position-ID buffer, rebuild on weight/storage changes (collectively if sharded). No cache of input images, features or projector outputs; all image inputs are processed every time. Frozen original parameters remain in checkpoints and the trainable projector/LLM remain untouched. Numerical smoke matched features and CPU/CUDA RNG, reuse, and invalidation after an in-place weight edit. PID 172517/tmp/r34/test.log, smoke/tmp/r34/smoke.log, timing/tmp/r34/time. Next profile/quick/full PID 172923/tmp/r35/profile-and-check.log. -/tmp/r34/time: 1066.9 ms / 2.5748x, slightly slower than R33, peak up to 131.09 GiB. A separate copy and cross-rank cache validity check do not pay for themselves here. Keep original gathered weights. Isolated39537c6, quick check pending. - End 00:32 UTC./tmp/r34/quick/check.jsonPASS exact items. No change to the submitted weights or cache behavior.
4.8 hRound 35 — make decoder modules true ZeRO gather regions
-/tmp/r35/profileon R33 (00:27–00:28 UTC): leading GEMMs ~71%, gather 7.8%, DeepSpeed's pre/post-backward autograd nodes cost 38.1 and 53.5 ms inclusive CPU. R14 grouped weights but still registered all nested sharding callbacks. Register the entire region's descendant parameters on its owner and have only that owner manage their ZeRO fetch/release and backward lifetime. Test decoder layer and attention/MLP region sizes. Only coalesce DeepSpeed's own parameter-lifetime registrations; all model operators, user hooks, optimizer gradient reduction hooks and trainable weights remain. Preserve module IDs and R19 forward retention on region owners. CPU ownership regression/tmp/r35/ownership.logpassed (including an independent child forward hook). Drafted without GPUs during R33 full; time job waited for the new R35 profile. PID 176755/tmp/r35/test.log,/tmp/r35/time-block,/tmp/r35/time-unit; next profile/quick/full PID 177507/tmp/r36/profile-and-check.log. - Block/tmp/r35/time-block: 1070.5 ms / 2.5662x. Attention/MLP units/tmp/r35/time-unit: 1067.9 ms / 2.5726x. Both slower than R33; less frequent fetch boundaries did not pay for the larger collective groups. Isolated7308f28. A separate locked feasibility check for ordinary NCCL coalesced all-gather (PID 180845/tmp/r36/nccl-smoke.log) passed on both ranks with different-sized flat output views. It did not time anything. - End 00:40 UTC./tmp/r35/quick/check.jsonPASS all exact items for the better unit layout. Keep individual module ownership.
5.0 hRound 36 — direct coalesced weight gather
-/tmp/r36/profileon R33 00:35–00:36 UTC: all-gather 88.7 ms (7.9%), top six GEMMs 71.2%,c10d::_allgather_base_27.0 ms inclusive CPU. Ordinary ZeRO gathers many parameters by first packing all local shards, then reassembling individual full weights with concatenations. Test NCCL's coalesced all-gather into parameter-sized slices of one final allocation instead. Wait on the ordinary communication work before exposing the original shape; preserve ZeRO fetch order and state transitions. Fallback for offload, quantization, secondary partitions, mixed dtype, single weights, or explicit extra rank validation. Screen all multi-param groups or just groups of up to 16. Two-rank smoke/tmp/r36/nccl-smoke.logpassed including unequal sizes, padded/unaligned flat views and a separate communication stream. Time PID 184273/tmp/r36/test.log,/tmp/r36/time-all,/tmp/r36/time-small_groups; next profile and quick/conditional full PID 184468/tmp/r37/profile-and-check.log. -/tmp/r36/time-all: 1066.0 ms / 2.5772x, slightly slower than 1063.4 ms. The small-group limit runs separately. - End 00:48 UTC./tmp/r36/time-small_groups: 1061.8 ms / 2.5874x;/tmp/r36/quick/check.jsonPASS exact items. The small gain did not justify changing the gather implementation. Isolated9c4565e.
5.2 hRound 37 — compact structurally zero rows for final-block backward
-/tmp/r37/profileon R33, 00:43–00:44 UTC: top six GEMMs ~72%; only 1,407 LM-head gradient rows out of 13,600 packed tokens. Use all rows in the forward and compact actually nonzero gradient rows in the last block's MLP backward GEMMs, optionally also the attention output/query projections. Preserve dense fallback, including nonfinite inputs/weights. FP64 smoke passed dense, sparse, zero and nonfinite cases (/tmp/r37/smoke.log). Timing job PID 188195/tmp/r37/test.log. - End 00:53 UTC./tmp/r37/time-final: 1062.7 ms / 2.5851x;/tmp/r37/time-mlp: 1059.7 ms / 2.5925x. Repeated compaction and finite checks in each projection leave <0.4% improvement over R33. Keep checked R33; experiment998435d. Quick check queued after R38 profile. -/tmp/r37/quick/check.jsonPASS exact items, NOT_BITWISE, 00:58 UTC.
5.2 hRound 38 — packed decoder to vocabulary projection
- Profile of the checked submission, then R37 quick check: PID 191231,/tmp/r38/profile-and-check.log, launched 00:53 UTC. No change to the live submission. - Six main GEMMs ~71.4%, parameter gather 7.8%. Shape attribution: the decoder actually executes 13,600 rows but the vocabulary projection takes 17,808 x 4,096; two dense copies of 16 x 1,113 x 32,000 cost 1.79 ms, in addition to the head GEMM. Pass compact hidden states and an inverse row mapping internally; project the zero-padding state once, then gather and cast all public logits to FP32. Preserve arbitrary dense-logit backward and labels whose preceding hidden state was padding. CPU smoke passed all output and gradient comparisons (/tmp/r38/smoke.log). - End 01:05 UTC. Timing PID 192987/tmp/r38/test.log;/tmp/r38/time: 1056.9 ms / 2.5993x, 129.60 GiB peak. Isolated commit91ec259./tmp/r38/quick/check.jsonPASS exact items. Next profile, quick and full PID 194736/tmp/r39/profile-and-check.log; full/tmp/r38/checkin progress. Submission still R33 pending verdict. - Full check completed 01:13 UTC:/tmp/r38/check/check.jsonPASS, 2.5746x (canonical −0.092σ, probe −0.149σ). This did not reproduce the standalone gain over the submitted full PASS, so no promotion.
5.4 hRound 39 — eliminate the staged FP32 gradient write and reread
-/tmp/r39/profileon R38 (01:00–01:01 UTC): six main GEMMs ~72.3%; the Adam shard writer costs 26.7 ms (2.4%) and gradient staging 5.5 ms (0.5%), excluding the separate FP64 norm collectives. Extend R33's working precompiled kernel to read the original BF16 gradient directly into FP32 registers. Keep the FP64 pre-update norm, DeepSpeed optimizer wrapper and one update per subgroup, both FP32 moments and FP32 master parameters. The optional FusedAdam subclass handles the no-clipping, unit-scale contiguous case and falls back otherwise. Also test combining the independent norm reductions into a single collective. Build PID 196403/tmp/r39/build.log(/opt/candidate-env/optional/llava_adam_raw/reader.so); locked numerical smoke passed scalar/vector, unaligned, three sizes, actual steps, both moments and model/master outputs (/tmp/r39/smoke.log). Timing PID 197209/tmp/r39/test.log,/tmp/r39/time-raw(combined) and/tmp/r39/time-norm(just the norm collective). - End 01:11 UTC./tmp/r39/time-raw: 1059.0 ms / 2.5941x;/tmp/r39/time-norm: 1064.2 ms / 2.5816x. Neither beats R38 at 1056.9 ms; no promotion. Next profile and exact check PID 200568,/tmp/r40/profile-and-check.log. R38 conditional full-pass promotion PID 198834/tmp/r38/promote.log. - Isolated commite33cb0d./tmp/r39/quick-raw/check.jsonPASS exact items, NOT_BITWISE, 01:18 UTC.
5.6 hRound 40 — gather pre-flattened decoder weight slabs
-/tmp/r40/profileon R33, 01:13–01:14 UTC: six dominant GEMMs 71.6%, all-gather 87.1 ms / 7.8%. Coalesce only trainable matrix weights at decoder layer or attention/MLP unit boundaries. If the input shards are contiguous portions of ZeRO's own flat backing, gather that slice without packing. Permute the gathered rank-major buffer by runs of equal size, rather than rebuilding every parameter separately. Use the normal ZeRO registry, active-module tracking and backward releases. Layout test/tmp/r40/layout_smoke.logpassed unequal sizes, offsets, 2–4 ranks and noncontiguous backing. PID 204494/tmp/r40/test.log; next profile/quick/full PID 205418/tmp/r41/profile-and-check.log. - Block/tmp/r40/time-block: 1061.9 ms / 2.5870x, not a decisive gain over the 1063.4 ms parent. Unit layout still running. A separate locked feasibility probe for host-local CUDA peer transfers (not a timing measurement), PID 206703/tmp/r41/peer-smoke.log, will use the GPUs when free. - End 01:26 UTC./tmp/r40/time-unit: 1067.4 ms / 2.5737x;/tmp/r40/quick/check.jsonPASS exact items with the better block layout. Isolated6790f02; keep R33.
5.7 hRound 41 — two-rank peer-copy all-gather
-/tmp/r41/profileon R33 01:21–01:22 UTC: top six GEMMs 72.3%, all-gather 71.7 ms / 6.6%, reduce-scatter 53.7 ms / 5.0%. Direct numerical peer-copy feasibility on actual devices passed for both ranks (/tmp/r41/peer-smoke.log). Test a bounded CUDA IPC scratch buffer per GPU to exchange eligible BF16 weight shards instead of using a NCCL kernel. Only the scratch buffer is exported; normal ZeRO holds original model/optimizer storage. Ready and completion signals order the peer transfers on a single CUDA stream; a CUDA event orders consumption. Fallback for unsupported groups/devices/precisions, explicit extra validation, offload, quantization, small and oversize groups. Test single-parameter collectives vs arbitrary size-compatible groups. Sources and compiled peer transport in the experiment and/opt/candidate-env/optional/llava_peer/exchange.so. - PID 209657/tmp/r41/test.log: initial scratch-exchange smoke selected the unavailable fallback before any GPU tensor had been allocated in the smoke. Initialize the CUDA context first; retry via/tmp/r41/test-2.logand/tmp/r41/exchange-smoke.log. Next profile and quick check PID 209906/tmp/r42/profile-and-check.log. - Retry PID 209925/tmp/r41/test-2.log: two-rank scratch exchange passed, including alternating slice groups, changing values and dynamic source allocations. Single/tmp/r41/time-single: 1059.7 ms / 2.5925x. All eligible groups/tmp/r41/time-all: 1054.9 ms / 2.6041x, window comparison within limits. End standalone timing 01:30 UTC. Full check queued with the exact all-groups configuration, not yet a verdict. - Isolateda2cd676, conditional pass-only promotion PID 212279/tmp/r41/promote.log. R42's profile confirms the large weight gather NCCL kernels disappear when using this path. -/tmp/r41/quick/check.jsonPASS exact items at 01:35 UTC; the full trajectory is proceeding. - Full verdict/tmp/r41/check/check.jsonPASS, 2.6165x at 01:42 UTC, N1, canonical −0.053σ and held-out −0.216σ. The promotion job fast-forwardedopttoa2cd676and selected/tmp/r41/all.json;cmpconfirms identical live switches. The ordinary model/Adam storage is not exposed over CUDA IPC.
5.9 hRound 42 — peer exchange for averaged ZeRO gradient shards
-/tmp/r42/profileon R41, 01:30–01:31 UTC: top six GEMMs 76.9%; gradient reduce-scatter is the largest remaining communication kernel, 56.7 ms / 5.6%. Test an independent peer scratch lane for two-rank BF16 gradient partitions: one peer copy of the remote half and a local, vectorized FP32 average into the actual ZeRO BF16 gradient partition. Keep the normal optimizer and original NCCL path for all other batches, sizes or precision modes. As with R41, stream events protect allocation lifetime and consumption. Screen 1 MiB vs 32 MiB minimum shard sizes, avoiding latency-heavy tiny transfers. Build PID 213216/tmp/r42/build.log, numerical smoke and timing PID 213495/tmp/r42/test.log; actual BF16 outputs matched NCCL exactly on two ranks for random gradients, odd sizes/misalignment and nonfinite cases (/tmp/r42/smoke.log). Timings/tmp/r42/time-1,/tmp/r42/time-32; next profile/quick/possible full PID 214492/tmp/r43/profile-and-check.log. -/tmp/r42/time-1: 1049.5 ms / 2.6177x. A second numerical test on 2,500 pairs including near-overflow and BF16 subnormals exposed 542 differences per rank vs NCCL: NCCL rounds both halves to BF16 before adding (0 differences for the pre-scaled BF16 model,/tmp/r42/extreme.log). Therefore do not select the original FP32 average (fb66f37) over R41, despite the timing. Test the identical arithmetic order with CUDA BF16 round-to-nearest multiply and add, for both scalar and paired lanes, in/tmp/llava-r42b; build PID 218916/tmp/r42/build-accurate.log. The third judge timing is/tmp/r42/time-accurateand requires the revised extreme/random-bit test to pass. Revised next-profile/quick/full waiter PID 218273/tmp/r43/profile-and-check-v2.log. - Original/tmp/r42/time-32: 1046.8 ms / 2.6243x. The revised BF16 prescaling matched NCCL for all 68,036 Cartesian and random raw-bit cases on both ranks (/tmp/r42/extreme-accurate.log). Timing the revised implementation at the faster 32 MiB cutoff (PID 218976,/tmp/r42/test-accurate.log). R43's joint-projection FP64 feasibility test, prepared on CPU during the R41 full check, matched every input/weight gradient, missing-output branches and double backward; no R43 timing yet. - End standalone time 01:45 UTC. Corrected/tmp/r42/time-accurate: 1045.7 ms / 2.6271x; selected for full validation. Isolatedea4666e; pass-only promotion PID 221554/tmp/r42/promote.log. R43's profile shows the custom peer-average kernel, with NCCL gradient scatter dropping from 56.7 ms to 40.0 ms on the profiled step. R41 remains live until the new full verdict. -/tmp/r42/quick/check.jsonPASS exact items, NOT_BITWISE. Full check running on the corrected implementation. - Full/tmp/r42/check/check.jsonPASS, 2.5885x, 01:58 UTC; N1, canonical −0.055σ and held-out −0.157σ. The standalone speedup did not repeat in the full timing. Not promoted; the submitted R41 full check is faster.
6.2 hRound 43 — fuse projection input-gradient accumulation into GEMMs
-/tmp/r43/profileon R42b at 01:45–01:46 UTC: six GEMMs ~77.9%; 159aten::add_calls on the 1 x 13,600 x 4,096 hidden tensor cost 12.94 ms. Along independent projection branches, accumulate each input-gradient product with GEMM beta=1 directly rather than writing full branch results then adding them. Separate forward and weight-gradient GEMMs, original trainable parameters retained; nothing concatenates weights or alters the public attention/MLP results. Screen MLP gate/up vs Q/K/V vs both. Register only the branch weights on their owner for ZeRO, retaining the ordinary forward path for custom hooks and unsupported types. FP64 smoke on distinct shapes, partial output use and double backward/tmp/r43/branch_smoke.log. Timing PID 222286/tmp/r43/test.log; next profile/quick/conditional full/tmp/r44/profile-and-check.log. - Both branch families/tmp/r43/time-both: 1058.6 ms / 2.5951x. MLP only/tmp/r43/time-gate: 1046.8 ms / 2.6244x. Neither improves on R42b so far. Attention-only trial outstanding; next six-step profile/quick/conditional full PID 224454. - End standalone time 01:59 UTC: Q/K/V/tmp/r43/time-qkv: 1051.2 ms / 2.6134x. Separate BF16 products plus adds were competitive; ownership changes and the beta=1 products did not repay the saved accumulation. No mode improves on its profiled R42b parent. Kept isolated; quick check of gate/up follows the next profile. - Isolated commit2db4284;/tmp/r43/quick/check.jsonPASS exact items at 02:05 UTC.
6.3 hRound 44 — phase-aware frozen image work
- Six-step detailed profile of the fastest verified stack (R41) requested after R43 screening, then R43 quick; PID 224454/tmp/r44/profile-and-check.log. The longer trace captures multiple image/text mixtures before deciding whether redundant frozen-encoder work is worth addressing. -/tmp/r44/profileon R41, 01:59–02:02 UTC, 6 steps. NCCL gradient scatter took 16.9% of aggregate device-kernel time on rank 0; per-rank breakdowns/tmp/r44/phases-r0.logandphases-r1.logdistinguish rank skew from arithmetic. Both ranks ran CLIP on 16 image slots on every profiled step, including a text-only step that consumed zero image patches (13.6–13.7 ms attributed CLIP kernels on that step, excluding projector). For unreferenced rows only, check each image's actual bytes against its representative. With duplicates, run the deterministic frozen tower on one copy and restore every row to the original output layout. Compare expanding before the projector vs after the deterministic rowwise Linear/GELU projector. No persistent input/feature cache, no change for different pixel data or different models. FP64 output and all projector gradients matched mixed and all-duplicate examples, including arbitrary upstream gradients and empty selections; bitwise inequality (including opposite signed zero) takes the normal path,/tmp/r44/smoke.log. Timing PID 231416/tmp/r44/test.log; next six-step profile and quick/conditional full PID 231575/tmp/r45/profile-and-check.log. - End standalone timing 02:08 UTC./tmp/r44/time-both: 1049.3 ms / 2.6181x;/tmp/r44/time-vision: 1053.7 ms / 2.6073x./tmp/r44/quick/check.jsonPASS exact items, NOT_BITWISE. Full/tmp/r44/check/check.jsonPASS, 2.6193x (1048.8 ms, canonical −0.004σ, probe −0.121σ). Promoted77ccab4and/tmp/r44/both.jsonat 02:22 UTC, PID 234045/tmp/r44/promote.log.
6.6 hRound 45 — balance only materially skewed text steps
-/tmp/r45/profileon R44, 02:08–02:10 UTC, six steps. Gradient reduce-scatter accounts for 18.4% of rank 0 device-kernel time. R44 phase attribution quantified the issue: on a text step 10,912 vs 14,915 live tokens, 632 vs 879 ms of GEMMs; on that step the earlier rank spent 647 ms in gradient collectives vs 47 ms on the other. Image steps were much nearer in packed compute. Unlike R23 which tokenized and swapped for every modality, preserve all image/mixed steps; only for complete all-text steps choose up to two swaps if estimated token/attention work differs by >2.5% of total. Test actual lengths from batched tokenization with the original prompt and truncation, or a cheap clipped byte/word estimate. No additional random draws and no changes to the original sampler's sign, permutation or step membership. Independent CPU feasibility on three other orders (/tmp/r45/feasibility.log): average absolute real text-token difference 979→175 (exact) or 768 (estimate); all 24 optimizer-step multisets and batch sizes preserved. - First attempt PID 235105/tmp/r45/test.log: CPU loader construction failed for models lacking.config(/tmp/r45/time-tokens/loader_tree_partition.log), stopped before timing; no speed claim. Use a fixed model-independent work estimate (the same 32,768 token-work coefficient on every invocation), without reading model metadata at all. Retry PID 236059/tmp/r45/test-v2.log, official/tmp/r45/time-tokens-v2and/tmp/r45/time-estimate-v2. Next profile/quick/full/tmp/r46/profile-and-check.log. - End standalone timing 02:20 UTC. Exact-token target/tmp/r45/time-tokens-v2: 1041.1 ms / 2.6388x; estimates/tmp/r45/time-estimate-v2: 1052.4 ms / 2.6104x. Both pass the data and loss checks. The estimate is not competitive. Commit110cacd; exact/full checks for tokens under/tmp/r45, guarded promotion PID 241301/tmp/r45/promote.log. R46 profile started 02:22 UTC (PID 238421,/tmp/r46/profile-and-check.log); draft rotary kernel compiled, PID 239387/tmp/r46/build.log, not yet measured. -/tmp/r45/quick/check.jsonPASS exact items, NOT_BITWISE at 02:27 UTC. Full/tmp/r45/check/check.jsonPASS, 2.6632x, N1, canonical −0.003σ and probe −0.168σ. The guarded promotion movedoptto110cacdand applied the exact-token switch file at 02:34 UTC.
6.8 hRound 46 — combine rotary lookups and rotations for Q and K
-/tmp/r46/profile(R45 tokens, 02:22–02:23 UTC): six dense GEMMs ~76.6% of device time, gradient reduce-scatter 55.1 ms / 5.3%; rotary kernels 12.8 ms / 1.2%, plus table lookups and the upstream rotary backward's extra gradient clones. Each Q and K uses the same packed positions. Rotate both together with one vectorized CUDA kernel reading the original BF16 cos/sin tables at those positions, then use the inverse operation for both gradients. Preserve cached values, full outputs and input dtype. Original path for other head widths, unequal head counts, trainable tables, unsupported devices or strides. Standalone library compiled from the experiment'srotary_pair.cuinto/opt/candidate-env/optional/llava_rotary/pair.so; locked smoke/tmp/r46/smoke.logmatched every BF16 bit for both directions, irregular positions, both vector widths, 1/7/32/33 heads, noncontiguous gradients and unused outputs. PID 243002/tmp/r46/test.log; eight BF16 lanes/tmp/r46/time-w8: 1032.2 ms / 2.6615x. Two-lane timing/tmp/r46/time-w2pending. Next profile, exact and conditional full PID 244517/tmp/r47/profile-and-check.log. - End time screening 02:35 UTC./tmp/r46/time-w2: 1028.3 ms / 2.6716x, better than width 8; commitfc1e81d. The next profile confirms the rotary kernels disappeared from the largest operations./tmp/r46/quick/check.jsonPASS, NOT_BITWISE; full/tmp/r46/check/check.jsonPASS, 2.6763x (1026.5 ms). N1, canonical +0.019σ and held-out −0.166σ. Guarded promotion PID 248412/tmp/r46/promote.logmoved the submission to the two-lane variant at 02:48 UTC.
6.9 hRound 47 — defer NCCL until both gradient buckets are ready
-/tmp/r47/profileon R46 width 2 at 02:35–02:36 UTC: six leading GEMMs ~77.8%; 51.3 ms (5.0%) in gradient reduce-scatter, the largest communication kernel. CPU attribution on the previous two-rank trace (/tmp/r47/queue-r{0,1}.log, PID 244102) found 293 scatters per rank: p50/p90/max = 0.117/0.272/1.466 ms vs 1.012/2.217/12.644 ms. Neither trace shows event-synchronize stalls in the gradient queue, so increasing DeepSpeed's queue cap is not supported. Instead, signal bucket arrival over an independent CUDA IPC control word before entering the existing NCCL reduction. The early rank waits on a stream semaphore rather than a resident NCCL CTA. Identical gradients, NCCL operator, destination and ZeRO partitioning; decide to fence using rank-invariant model sizes, independently of the actual gradient contents or contiguity. Screen 1 MiB and 64 MiB cutoffs. Keep the R41 binary intact; optional new driver transport is/opt/candidate-env/optional/llava_peer_fence/exchange.so. Locked alternating-stream smoke and judge timing PID 250303,/tmp/r47/test.logand/tmp/r47/smoke.log. - Both ranks passed the locked smoke, comparing actual NCCL results at the bit level while interleaving peer weight gathers on two streams./tmp/r47/time-m1: 1044.3 ms / 2.6308x;/tmp/r47/time-m64: 1026.4 ms / 2.6766x (timing ended 02:44 UTC). The extra waits slowed an aggressively fenced pipeline; the less frequent cutoff is only 0.2% over R46's standalone. Do not promote on this small difference. Experimentd1fa65d;/tmp/r47/quickis the separate exact check. Next profile PID 251677/tmp/r48/profile-and-check.log. -/tmp/r47/quick/check.jsonPASS exact items, NOT_BITWISE at 02:53 UTC; leave the fence off.
7.2 hRound 48 — directly read the remote trainable shard
-/tmp/r48/profileon R46, 02:47–02:48 UTC: leading six GEMMs ~77.7%, gradient scatter 5.1%; the weight gather stream copies 6.58 GiB P2P (374 transfers, 25.62 ms) and 13.15 GiB locally (748 transfers, 8.51 ms). Scratch gathering copies each received weight twice. Give ZeRO's actual BF16 trainable partition one CUDA-IPC-shareable contiguous backing allocation with DLPack ownership; master/Adam and parameter names remain unchanged. The ranks compare the full offset, size and parameter-ID layouts before mapping. With supported parameters, read the peer shard directly into the final full-weight view while preserving ready/completion handshakes. Original scratch path for frozen or nonmatching groups. Independent locked feasibility, PID 254396/tmp/r48/storage-smoke.log: both ranks matched NCCL on changing weights through offset views with expandable segments enabled, including lifetime after deleting the original tensor/capsule. Build with g++ fromflat_exchange.cppandshareable_storage.cppinto/opt/candidate-env/optional/llava_flat/transport.so; only the shareable BF16 allocation is outside the torch caching allocator's memory counters. Screen 1 and 64 MiB minimums with PID 258689/tmp/r48/test.log,/tmp/r48/time-m1,/tmp/r48/time-m64. Next profile/quick/conditional full PID 259656/tmp/r49/profile-and-check.log, promotion guard PID 259657/tmp/r48/promote.log. - End standalone screening 02:55 UTC./tmp/r48/time-m1: 1037.6 ms / 2.6477x;/tmp/r48/time-m64: 1025.9 ms / 2.6777x. Both report 120.13 GiB torch allocations (about 6.4 GiB of BF16 weights live separately). Small/mid-sized remote reads were slower than scratch writes, and the large-only variant gave <0.25% over R46 standalone. Do not promote; isolated8188407. Large-only quick check/tmp/r48/quick. -/tmp/r48/quick/check.jsonPASS exact items, NOT_BITWISE at about 03:01 UTC. The full-passing submitted storage remains unchanged.
7.3 hRound 49 — increase parallelism in the RMSNorm backward reduction
-/tmp/r49/profileon the checked R46, 02:55–02:56 UTC: six leading GEMMs ~77.2%, gradient reduce-scatter 5.0%, RMSNorm's main backward kernel 12.8 ms / 1.2% and ~3.8 ms for forward. Flash-attn's existing backward assigns only one program per SM; each loops sequentially over roughly 100 token rows of 4,096 columns. Retain its forward, saved reciprocal standard deviations, and backward kernel but split the rows into 2 or 4 teams per SM; combine the FP32 weight-gradient partials after every row is processed exactly once. The order of the weight-gradient reduction can change (N1). Fallback for unsupported dtypes/strides/shapes. Locked numerical smoke, then judge timing, PID 261713/tmp/r49/test.log,/tmp/r49/rms-smoke.log,/tmp/r49/time-t4and/tmp/r49/time-t2. Next profile/quick/conditional full PID 262254/tmp/r50/profile-and-check.log; guarded promotion PID 262255/tmp/r49/promote.log. - End timing 03:03 UTC. Numerical smoke passed 7–13,254 rows with strided gradients and both settings: exactly equal forward; dX and dW within 0.1% relative L2 of flash-attn./tmp/r49/time-t4: 1031.6 ms / 2.6630x./tmp/r49/time-t2: 1027.0 ms / 2.6749x. Neither sufficiently exceeds R46; larger FP32 partials offset the increased parallelism. Isolatedd4809ca; two-team quick check/tmp/r49/quick. -/tmp/r49/quick/check.jsonPASS exact items, NOT_BITWISE at about 03:09 UTC.
7.4 hRound 50 — push shards straight into their final weight views
-/tmp/r50/profileon R46, 03:03–03:04 UTC: six top GEMMs ~76.2%, reduce-scatter 5.8%. The fetch stream made 374 P2P writes (6.58 GiB, 25.75 ms) and 748 local copies (13.15 GiB, 8.47 ms). Unlike R48's remote reads, keep the more efficient peer writes, but target each parameter's disjoint final output rather than intermediate scratch. Allocate persistent output storage from CUDA with DLPack ownership; refresh both halves on every actual fetch, including after any changes to the source shards. Independent ready and completion signals protect writes and consumption. Only use the arena when both ranks agree on all parameter IDs, sizes and offsets; ordinary scratch or NCCL otherwise. Do not change the original sharded model, gradient or optimizer storage. Two-rank locked feasibility PID 263909/tmp/r50/arena-smoke.logmatched NCCL for repeated acquisitions with changing sources/group order, and verified stable, disjoint destinations. 1 and 32 MiB minimum local-shard sizes; optional output arena storage is outside torch's allocation counters. Judge PID 266384/tmp/r50/test.log,/tmp/r50/time-m1,/tmp/r50/time-m32; next profile/quick/conditional full PID 267191/tmp/r51/profile-and-check.log, guarded promotion PID 267192/tmp/r50/promote.log. - End standalone 03:11 UTC. All qualifying sizes/tmp/r50/time-m1: 1029.5 ms / 2.6684x, 113.83 GiB tracked by torch. Large local shards (32 MiB+)/tmp/r50/time-m32: 1022.3 ms / 2.6874x, 117.87 GiB torch-tracked, 1.38% relative window spread. Keep the large variant in full validation (after quick); isolated84e9ddduntil a faster full PASS. The live weights still use R46. -/tmp/r50/quick/check.jsonPASS exact items, NOT_BITWISE. The R51 trace of the large arena confirms 98 fewer local copies (650 vs 748), 8.88 vs 13.15 GiB locally and 19.62 vs 25.75 ms in the 374 P2P writes. Full/tmp/r50/check/check.jsonPASS, 2.6826x (N1, canonical +0.032σ, probe −0.090σ). Guarded promotion fast-forwardedoptto84e9dddat 03:25 UTC and applied/tmp/r50/m32.json.
7.6 hRound 51 — test physical layouts of individual decoder GEMMs
-/tmp/r51/profileon R50 large shards, 03:10–03:11 UTC: six cuBLAS kernel families account for 77.2% of device time, gradient scatter 5.5%. In the preceding CPU shape trace/tmp/r51/cpu-shapes.log, 256 multiplies are[13600,4096] @ [4096,4096], 128 are[4096,13600] @ [13600,4096], and two rectangular MLP shapes occur 96 times each. Vendor kernel names differed by matrix orientation; this trace does not correlate CUDA 12.8's individual GEMM launches with hostaten::mm, so no per-shape time is assumed. Retain each original projection, but physically transpose its weight for either forward or dX, or materialize the transposed dY operand for dW. For the latter, align the leading stride without changing the true token count. No persistent replicas or new parameters. Numerical smoke PID 268090/tmp/r51/layout-smoke.logpassed on FP64 CPU and BF16 GPU for all three, including strided inputs and gradients and irregular row counts. Official times PID 269714/tmp/r51/test.login/tmp/r51/time-{input,forward,weight}. Input gradient variant: 1056.8 ms / 2.5995x, slower. Next profile and quick/conditional full PID 270167/tmp/r52/profile-and-check.logand promotion guard PID 270168/tmp/r51/promote.log. -/tmp/r51/time-forward: 1077.7 ms / 2.5492x. Both transposed-weight cases were slower than using the original layout; the dW operand case runs separately. CPU-prepared R52 feasibility (/tmp/r52/compact-smoke.log, PID 272345) preserved full outputs and all four gradients for a grouped last-MLP backward, including sparse, dense, all-zero and nonfinite cases. No R52 speed claim yet. - End screening 03:23 UTC./tmp/r51/time-weight: 1983.4 ms / 1.3851x, much slower with a physically transposed token-gradient operand. Best was dX at 2.5995x, still below the 2.6874x parent. Isolated8ee1860; quick check/tmp/r51/quick. No layout changes in the submission. -/tmp/r51/quick/check.jsonPASS exact items, NOT_BITWISE at about 03:30 UTC, for the least-slow dX layout.
7.8 hRound 52 — compact the entire last MLP backward once
-/tmp/r52/profileon R50, 03:24–03:25 UTC. Six major GEMMs consume ~77.5% of GPU kernel time and gradient reduce-scatter 5.3%. The last MLP still runs backward GEMMs over all 13,600 packed rows while the loss reaches only about 1,400 of them in the profiled shape. Unlike R37's independent per-projection scans, run one full, unchanged gated MLP forward and detect actual nonzero upstream rows once in its backward. Use one shared row selection across the gate, up, and down weight/input products, and scatter the input gradient only once. Test individual live rows or complete aligned 32-row tiles. Nonfinite inputs, intermediates or weights and dense gradients use full GEMMs; arbitrary objectives on public logits still work. Existing trainable matrices are individually registered with ZeRO. Locked GPU smoke PID 272345/tmp/r52/compact-smoke.logpassed full-output and all-four-gradient checks for dense, sparse, block, zero and nonfinite cases. Judge PID 278038/tmp/r52/test.log,/tmp/r52/time-t1,/tmp/r52/time-t32; next profile/quick/conditional full PID 277118/tmp/r53/profile-and-check.log; guarded promotion PID 277119/tmp/r52/promote.log. - End standalone 03:32 UTC:/tmp/r52/time-t11017.5 ms / 2.7001x,/tmp/r52/time-t321018.6 ms / 2.6971x. Both pass data/loss checks. Single live rows selected at/tmp/r52/t1.json; this is not yet a full verdict. Before it acquired the lock, replaced the waiting R53 script with PID 281298/tmp/r53/priority.log: fresh R53 profile, then the justified R52 full check before its additional quick check, rather than delaying the full result. No judge run was interrupted or duplicated. Pending promotion still requires the full check and a speedup above live R50. - Full /tmp/r52/check/check.json: PASS, 2.7032x, 1016.3 ms, N1; canonical -0.022σ, held-out -0.161σ. Promoted bfaa043; /submission/switches.json matches /tmp/r52/t1.json. Recorded 03:40 UTC.
7.9 hRound 53 — segment ZeRO's traced weight lookahead
-/tmp/r53/profileon R52 individual rows, 03:32–03:33 UTC: six leading GEMMs ~77%, gradient scatter 58.0 ms / 5.6%. The fetch stream still submits hundreds of peer transfers using ZeRO's small auto lookahead. R51's parent trace verified 6.58 GiB of P2P writes and 8.88 GiB local copies. Rather than making one large handle block every parameter in a prefetched group (R32), let ZeRO trace farther ahead while splitting each forward submission into independent ready handles. Preserve the original__all_gather_params_for every segment and all of its parameter lifetimes. Use model parameter order, a 64M-element segment cap, and separate persistent-arena vs ordinary destinations. Leave entirely frozen groups, quantized/offloaded storage and backward group submissions untouched. CPU splitting test PID 276525/tmp/r53/split-smoke.logpreserved complete parameter coverage, disjoint groups, capacity bounds and transport categories. Test 128M and 384M lookahead; PID 283083/tmp/r53/test.logwaits for the R52 full verdict before timings/tmp/r53/time-a128,/tmp/r53/time-a384, to avoid delaying it. Next profile/quick/conditional full PID 283657/tmp/r54/profile-and-check.log, promotion guard PID 283658/tmp/r53/promote.log. R54 CPU preparation only: optional CUDA transport comparing actual frozen BF16 words, compile PID 284086/tmp/r54/build.log, not timed. - End standalone 03:46 UTC./tmp/r53/time-a128: 1017.2 ms / 2.7008x, 118.63 GiB torch memory./tmp/r53/time-a384: 1023.6 ms / 2.6837x. Broader lookahead did not improve on R52 (2.7001x standalone / 2.7032x full); do not promote. Smaller-window exact check/tmp/r53/quick./tmp/r52/quick/check.jsonalso PASS, NOT_BITWISE, after its full verdict. Frozen-buffer feasibility/tmp/r54/frozen-smoke.logpassed on both GPUs for conditional and unconditional refresh, arbitrary BF16 bit patterns, asymmetric source edits and overwritten public output views. CPU transfer-size attribution PID 286408/tmp/r54/copy-sizes.log.
8.2 hRound 54 — refresh frozen weight words without trusting version counters
-/tmp/r54/profileon the checked R52, 03:46–03:47 UTC: large GEMMs ~77%, gradient reduce-scatter 5.3%. The recent two-rank fetch trace (/tmp/r54/copy-sizes.log) includes 96 one-MiB peer writes (CLIP attention weights) and 49 four-MiB writes (mostly CLIP MLP), totaling 292 MiB in many transfers even though CLIP stays frozen. Keep an internal BF16 snapshot for each eligible frozen shard. At every gather a multi-slice kernel reads actual source bits, updating both GPUs' private copies where they differ, then materializes fresh ordinary full-weight outputs after the peer completion barrier. No dependence on PyTorch version counters; no memoization of model outputs or image features. Test bitwise comparison (changed) and unconditional multi-tensor delivery (always). The regular trainable arena and unsupported gather paths remain as before. New compiled library/opt/candidate-env/optional/llava_frozen/transport.so, build PID 284086/tmp/r54/build.log. Both ranks matched NCCL in/tmp/r54/frozen-smoke.log(PID 284899) for misaligned and odd sizes, arbitrary raw BF16 including nonfinite values, edits on only one rank, different/reordered groups, and deliberate overwrites of public output views. Judge timing PID 289574/tmp/r54/test.log,/tmp/r54/time-changed,/tmp/r54/time-always; profile, quick and conditional full PID 290552/tmp/r55/profile-and-check.log, guarded promotion PID 290553/tmp/r54/promote.log. R55 small real DeepSpeed optimizer correctness probe (not timing) queued PID 291255/tmp/r55/engine-smoke.log. - End timing 03:54 UTC,/tmp/r54/time-changed: 1019.1 ms / 2.6957x;/tmp/r54/time-always: 1018.3 ms / 2.6978x. Neither beats R52. The faster unconditional mode passed/tmp/r54/quick/check.json(NOT_BITWISE, exact gates pass) at 04:00 UTC. Commitb224363remains isolated. R53 also passed its exact quick check but its best timing was 2.7008x.
8.3 hRound 55 — prepare current full weights at the Adam boundary
-/tmp/r55/profileon R52, 03:54–03:55 UTC, rank 0: the six largest GEMMs account for 76.6% of kernel time, BF16 gradient reduce-scatter 5.5%, shard Adam writer 2.5%. Large trainable weights still travel to both GPUs on the forward path. Rather than fetching stale values or predicting optimizer results, issue regular ZeRO gathers of the newly updated shards when Adam releases a subgroup (subgroup) or immediately before its post-step (tail). Only parameters with output buffers already in the arena are eligible. Keep the normal engine, waits and gradients. Three-step two-rank DeepSpeed correctness checks matched every prepared weight to the current FP32 master after every optimizer step in both modes (/tmp/r55/engine-subgroup.log,/tmp/r55/engine-tail.log, PID 291255). Timing PID 294702/tmp/r55/test.log; next profile and conditional checks PID 293687/tmp/r56/profile-and-check.log, guarded promotion PID 293688/tmp/r56/promote.log. - End 04:01 UTC./tmp/r55/time-subgroup: 1030.1 ms / 2.6668x./tmp/r55/time-tail: 1031.7 ms / 2.6628x. Moving fetches to the optimizer boundary made the step slower. The subgroup version passed/tmp/r55/quick/check.json(NOT_BITWISE) at 04:06 UTC; isolated commit56203a6, not promoted.
8.5 hRound 56 — combine decoder gate/up projections on physically adjacent weights
-/tmp/r56/profile04:01–04:02 UTC on R52: six leading GEMMs comprise 77.0% of kernel time, reduce-scatter 5.1%, BF16 adds 1.8%, SwiGLU forward and backward about 1.9% combined. R9 concatenated weights every time; R50 now gives many large projections adjacent, shareable storage. Verify addresses, strides, shapes and underlying storage; when they agree, use a view without copying. Try one joined forward product versus two products into a strided joined output. A vectorized CUDA SwiGLU writes its two derivative halves directly into adjacent columns; input and both distinct weight gradients use that joined derivative. R52's final layer is untouched. No change of parameter names/shapes or layout when the check fails./tmp/r56/gate-smoke.logPID 294346: activation and the three BF16 gradients passed tests with strided inputs, unaligned derivatives and both forward modes, including 11,008 columns. Build log/tmp/r56/build.log. Timing PID 298209/tmp/r56/test.log. Profile/conditional full/quick PID 299787/tmp/r57/profile-and-check.log, guarded promotion PID 299788. Additional actual-model trace if not selected: PID 300930/tmp/r56/path-profile.log. CPU-only common-prefix survey PID 300617: generic exact token prefix lengths 35 on image-text and 34 on text-only samples in the sampled groups, not used as runtime constants. - End standalone timing 04:09 UTC: joined/tmp/r56/time-joined1023.3 ms / 2.6845x; split/tmp/r56/time-split1026.4 ms / 2.6765x. The real-model trace/tmp/r56/path-profile/profile_tree.mdconfirms that the adjacent path ran (paired forward 7.7 ms; paired backward 11.3 ms), but combining these GEMMs did not pay./tmp/r56/quick/check.jsonPASS exact items, NOT_BITWISE at 04:14 UTC. Isolated794114f.
8.6 hRound 57 — compute an identical causal text prefix once per batch
-/tmp/r57/profileon checked R52, 04:09–04:10 UTC: six dense GEMM families 76.9% of device time, BF16 gradient reduce-scatter 5.2%. CPU attribution/tmp/r57/prefix-survey.log: 16 independent groups from both modalities share 34–35 ordinary text tokens per row before any image-specific embeddings. Those rows have identical positions and causal histories. In the regular single-image/right-padding merge, derive the true prefix from token IDs, valid masks and truncated lengths on every call. Keep the first copy; each later sequence projects only its suffix. Restore K/V rows for ordinary bottom-right-aligned causal attention with separate query/key lengths, and restore every hidden row before public logits. Prefix K/V adjoints sum over all original occurrences. Never alias equal values from independent image or input embeddings; no prefix cache across forwards or steps, no model or sampler change. Compare ordinary tensor gather (tensor) with joined vectorized BF16 K/V transport and FP32 adjoint reduction (vector). Native build PID 302272/tmp/r57/build.log; correctness PID 304096/tmp/r57/smoke-{cpu,cuda}.log: all outputs, token embedding gradients, independent image gradients and 8 projection gradients matched for two causal layers, including actual BF16 FA3. Native row transport also matched gathered values and summed adjoints bitwise, including strided and missing derivatives. Timing PID 303457/tmp/r57/test.log; vector/tmp/r57/time-vector1010.3 ms / 2.7191x at 04:15 UTC. Profile and conditional full/quick PID 304224/tmp/r58/profile-and-check.log; guarded promotion PID 304225/tmp/r57/promote.log. - End 04:16 UTC. Tensor-only K/V restore/tmp/r57/time-tensor1053.7 ms / 2.6072x; its indexed-copy backward gives back more than the eliminated GEMM work. Vectorized transport/tmp/r57/time-vector1010.3 ms / 2.7191x, peak 116.07 GiB.28b4dafisolated pending full/tmp/r57/check/check.json; verdict recorder PID 308984/tmp/r57/record.log. - Full /tmp/r57/check/check.json: PASS, 2.7124x, 1012.8 ms, N1; canonical +0.023σ, held-out -0.139σ. Promoted the fully checked prefix-sharing implementation. Recorded 04:28 UTC.
8.6 hRound 58 — initialize with ZeRO-3, train with resident BF16 weights
-/tmp/r58/profileon R57 vector, 04:17–04:18 UTC: four dominant dense GEMM kernels 73.8% of aggregate GPU time, BF16 gradient scatter 6.4%, Adam writer 2.6%. Despite the arena, per-layer gather/gradient communication still constrains the two-rank schedule. R2 used stage 2 for both checkpoint loading and training, and failed the first-loss sanity gate. This experiment keeps the original checkpoint/CLIP/projector initialization intact. At the boundary before constructing the HF Trainer, materialize the actual current ZeRO-3 parameter shards, preserving parameter identity, tying, dtype and every loaded bit; run the normal DeepSpeed ZeRO-2 engine on the resulting resident weights. No change to AdamW hyperparameters or precision. Bound individual extents passed to the installed FusedAdam CUDA kernel, whosesizes[]fields are signed 32-bit, without altering optimizer parameter groups. Test 57 and 48 GiB activation budgets for this memory layout. PID 307834/tmp/r58/test.log. Separate two-rank correctness smoke PID 308658/tmp/r58/engine-smoke.log. Profile/conditional full/quick PID 308982/tmp/r59/profile-and-check.log; guarded promotion PID 308983/tmp/r58/promote.log. - End 04:26 UTC./tmp/r58/time-571077.6 ms / 2.5493x;/tmp/r58/time-481121.6 ms / 2.4494x. Because peak allocation with 57 GiB was only 112.82 GiB, tested 72 GiB as well (PID 310242/tmp/r58/test-72.log):/tmp/r58/time-721071.8 ms / 2.5632x, 114.77 GiB peak. Unlike R2, the resident path matched the submitted parent's first logged loss (1.3854), and passed timing sanity. All three were nevertheless slower. The initial model-only smoke needed DeepSpeed's communicator initialization, and the second version called a nonexistent ZeRO-2 teardown method after all assertions passed. Corrected/tmp/r58/engine-final.log(PID 311950): both ranks passed weight-bit/identity/tied-parameter/RNG assertions and two steps of full gradient, master, and moment checks. Isolatedf70323e, not promoted. Revised next-profile/quick PID 310243/tmp/r59/profile-and-check.log. Prefix shape attribution from the R57 trace PID 311951,/tmp/r59/shape-attribution.log.
8.8 hRound 59 — align compact token counts with real prefix rows
-/tmp/r59/profileon fully checked R57, 04:28–04:29 UTC: the four dominant GEMM families consume 72.7% of GPU-kernel time, BF16 reduce-scatter 7.9%./tmp/r59/shape-attribution.logshows 256 square projections of 13,075 rows and 128 weight gradients whose reduction size is also 13,075. Instead of appending fake tokens or rounding every layer independently (R17/R28), retain a few actual duplicate prompt rows: vary the number of physically shared prefix tokens per sequence so the remaining row count is a multiple of 16, 64 or 128. Arbitrary asymmetric causal offsets and fully restored outputs are preserved; the K/V adjoint sums only rows actually shared. Native build PID 312642/tmp/r59/build.log; PID 313005/tmp/r59/smoke.logpassed all dense FP64 and Hopper BF16 two-layer outputs/gradients, partial-prefix row transport and strided/absent gradient cases for 16 and 64. Timing PID 313675/tmp/r59/test.log,/tmp/r59/time-a{16,64,128}. Next profile/conditional full/quick PID 315294/tmp/r60/profile-and-check.log; guarded promotion PID 315295/tmp/r59/promote.log. - End standalone 04:39 UTC. /tmp/r59/time-a16: 1003.1 ms / 2.7386x; /tmp/r59/time-a64: 1011.7 ms / 2.7154x; /tmp/r59/time-a128: 1001.6 ms / 2.7429x. Best standalone 2.7429x, isolated commit 3005e78. See /tmp/r60/parent for selected next profile. /tmp/r57/quick/check.json also passed all exact items after R57's full PASS. - R59 full /tmp/r59/check/check.json: PASS, 2.7143x, 1012.1 ms/step, N1; canonical -0.002σ, held-out -0.145σ. Promoted 3005e78 with /tmp/r59/a128.json. Recorded 04:48 UTC.
9.0 hRound 60 — combine prefix K/V transport with rotations
-/tmp/r60/profileon R59 alignment 128, 04:39–04:40 UTC: six dense GEMMs 73.1% of device-kernel time, gradient scatter 6.2%. Indexed Q/K rotations and prefix K/V transport form adjacent memory passes, the same pattern as on R57 (combined ~16.5 ms there). Fuse them: rotate compact Q once, write every expanded K already rotated and copy V directly. Backward maps every original K/V derivative into its actual representative, accumulates shared rows in FP32, rounds to BF16 before inverse RoPE exactly as in the unfused chain, and handles independently unused Q/K/V. Only use with contiguous supported Hopper BF16 shapes and immutable rotation tables. Test 2 and 8 BF16 lanes. CUDA build PID 316641/tmp/r60/build.log; locked smoke PID 316795/tmp/r60/rope-smoke.logmatched the parent bitwise for all forward values and partial, strided derivatives, at both widths with fixed and uneven prefix sharing. Timing PID 318069/tmp/r60/test.log,/tmp/r60/time-w{2,8}; future six-step profile/conditional full/quick PID 320792/tmp/r61/profile-and-check.log, guarded promotion PID 320793/tmp/r60/promote.log. R58's best resident version passed/tmp/r58/quick/check.jsonat 04:36 UTC, NOT_BITWISE, but was slower. - End standalone 04:46 UTC./tmp/r60/time-w21002.2 ms / 2.7412x;/tmp/r60/time-w8996.4 ms / 2.7572x. The wider fused transport was faster than R59's 2.7429x standalone. Isolated commit72cb976, conditional full/tmp/r60/check/check.jsonafter its six-step profile. R59/R60 full verdict recorders PIDs 324871/324872,/tmp/r59/record.log,/tmp/r60/record.log. R61 CPU feasibility PID 319888/tmp/r61/image-survey.logand exact-count/multiset smoke PID 325303/tmp/r61/stats-smoke.log; its official timings wait for the current-best profile (PID 325304/tmp/r61/test.log). - R60 full /tmp/r60/check/check.json: PASS, 2.7111x, 1013.3 ms/step, N1; canonical +0.005σ, held-out -0.101σ. Kept the already-checked submission 3005e78. Recorded 05:00 UTC.
9.2 hRound 61 — balance image steps against rounded decoder work
-/tmp/r61/profileof R60 width 8, 04:48–04:50 UTC, six production steps. Rank 0 gradient scatter reached 11.5% of aggregate device time, versus 6.2% on the preceding single-step profile./tmp/r61/phases-r{0,1}.log: on the first image step 13,184 vs 12,800 compact projection rows and 69.2 vs 229.5 ms of gradient scatter; on another 12,160 vs 12,672 and 218.8 vs 72.0 ms. R23 tried unconditional pair swaps. Here evaluate a single within-step pair swap against the slower rank's rounded packed GEMM rows, causal quadratic work and padded-head extent, and only take a material predicted gain. Leave all mixed, partial and text steps alone (the existing text policy stays). Use true batched token counts with actual image-marker expansion (batch), a cheap byte/word prefilter before exact counts (screen), or cheap estimates alone (bytes). The actual data pipeline and global step order do not change. CPU survey PID 319888/tmp/r61/image-survey.log: exact scheduling reduced maximum true row count by 116–139 rows on average over three other data orders; selective mode saved fewer rows but only tokenized 13–18 of 72 image batches. PID 325303/tmp/r61/stats-smoke.log: exact batched counts match independent tokenizer-image-token counts including truncations through images; source conversations, step multisets, partial steps and text steps preserved. Timing PID 325304/tmp/r61/test.log, started after profiling. Profile/conditional full/quick PID 328031/tmp/r62/profile-and-check.log; promotion guard PID 328032/tmp/r61/promote.log. - End 05:01 UTC./tmp/r61/time-batch1023.1 ms / 2.6852x,/tmp/r61/time-screen1001.3 ms / 2.7436x,/tmp/r61/time-bytes1014.4 ms / 2.7082x. R61's profiled R60 parent was 2.7572x standalone. Neither true lengths nor cheaper proxies gave an end-to-end win; mathematical load balance did not repay sampling work and shape changes. No promotion or full run, quick check onscreenqueued, isolated commit960dfbd./tmp/r59/quick/check.jsonalso passed all exact items. The next round profiles the fastest fully checked submission, R59, since R60's separate full timing did not beat it.
9.4 hRound 62 — consume branch input derivatives inside RMSNorm backward
-/tmp/r62/profileof checked R59, 05:03–05:04 UTC: six leading GEMMs ~74.9% of kernel time, BF16 gradient scatter 5.9%, regular BF16 vector adds 18.1 ms / 1.8%, main RMS backward 12.4 ms / 1.2%. Decoder Q/K/V and gate/up all consume the same normalized activation; backward ordinarily materializes additions of full hidden gradients before normalization. Return distinct differentiable views of the same normalized storage to these branches, keep the installed RMS forward, and add their BF16 derivatives in the normalization backward kernel as it formsdxanddw. Keep separate BF16 roundings; do not fuse the original trainable parameters. Test attention only, gate/up only, or both; keep R52's final MLP untouched and the regular path for uncached/unsupported shapes. Locked validation PID 327947/tmp/r62/fork-smoke.log: shared storage and distinct autograd edges on all tests; forward bitwise equal to the library; partial and full input/weight gradients agreed for 9, 174 and 1031 rows and two or three branches. Timing PID 329709/tmp/r62/test.log, profile/full-if-faster/quick PID 339170/tmp/r63/profile-and-check.log, guard PID 339171/tmp/r62/promote.log. R63 transport feasibility only (not timing): grouped NCCL versus separate collectives PID 336999/tmp/r63/scatter-smoke.log; real ZeRO check PID 339169/tmp/r63/engine-smoke.log. - End standalone 05:11 UTC./tmp/r62/time-both1001.7 ms / 2.7426x;/tmp/r62/time-attention1000.7 ms / 2.7452x;/tmp/r62/time-mlp1000.6 ms / 2.7455x. This is a small gain against R59's standalone timing; only a full check can justify promotion. Isolatedbe61b34, conditional full/tmp/r62/check/check.json. R60 and R61 separate quick checks both PASS, NOT_BITWISE. The packed-only branch falls back on cached generation and unsupported shapes. - R62 full /tmp/r62/check/check.json: PASS, 2.708x, 1014.5 ms/step, N1; canonical +0.004σ, held-out -0.105σ. Kept the already-checked submission 3005e78. Recorded 05:20 UTC.
9.5 hRound 63 — group direct NCCL gradient scatters without packing tensors
-/tmp/r63/profileof R62 MLP forks, 05:11–05:12 UTC: six main GEMM families 74.8%, BF16 gradient scatter 69.0 ms / 6.9% (11.5% across six steps in R61). R18 writes singleton reduced gradients straight into their ZeRO partitions, but there are hundreds of individual calls per rank. Use larger ZeRO buckets; keep orthogonal tensor storage and launch lists of native NCCL averaged reduce-scatters via the installed process group'sreduce_scatter_tensor_coalesced. Do not concatenate the sources or unpack the destination. Preserve normal partition bookkeeping, gradient stream waits/lifetime and fallback for padding, noncontiguous/mixed/unsupported tensors, accumulation and offload. Compare 32, 64 and 128 Mi-elements. PID 336999/tmp/r63/scatter-smoke.log: grouped outputs matched separate NCCL averages bitwise for varied and misaligned extents and extreme BF16 values. PID 339169/tmp/r63/engine-smoke.log: both ranks actually reduced six-parameter buckets in each of two ZeRO-3 steps, every full gradient matched an independent replica's NCCL average bitwise, and full FP32 master/moments were available. Timing PID 340035/tmp/r63/test.log, next profile/conditional full/quick PID 341673/tmp/r64/profile-and-check.log, guarded promotion PID 341674/tmp/r63/promote.log. - End standalone 05:25 UTC./tmp/r63/time-g32: 1006.8 ms / 2.7288x;/tmp/r63/time-g64: 1006.4 ms / 2.7298x;/tmp/r63/time-g128: 992.6 ms / 2.7678x, peak tracked 116.43 GiB, relative two-window spread 1.39%. The 128 Mi-element case is isolated atb3b5e7cpending full/tmp/r63/checkand exact/tmp/r63/quick; verdict recorder PID 344956,/tmp/r63/record.log. - R63 full /tmp/r63/check/check.json: PASS, 2.6786x, 1025.6 ms/step, N1; canonical +0.050σ, held-out -0.179σ. Kept the already-checked submission 3005e78. Recorded 05:34 UTC.
9.8 hRound 64 — fuse frozen CLIP epilogues without changing BF16 roundings
-/tmp/r64/profileon R63 g128, 05:25–05:26 UTC: six leading GEMMs 76.7% of device kernel time, grouped gradient scatter 44.1 ms / 4.6%, BF16 adds 1.6%./tmp/r64/clip-attribution.logon an image step attributes 5.0 ms to CLIP linear bias additions, 3.5 ms to the two multiplies and sigmoid in QuickGELU, 0.6 ms to residual adds. Fuse the adjacent bias and QuickGELU passes, or also fuse CLIP's attention scaling and residual bias operations; preserve the intervening BF16 casts. Locked/tmp/r64/epilogue-smoke.logmatched the original frozen block bitwise and checked every BF16 QuickGELU input. Timing PID 344955/tmp/r64/test.log, next profile and conditional checks PID 348648/tmp/r65/profile-and-check.log, guarded promotion PID 348649/tmp/r64/promote.log. Experiments run in detached worktrees, not the live submission. - End 05:30 UTC./tmp/r64/time-gelu995.2 ms / 2.7604x;/tmp/r64/time-all991.4 ms / 2.7709x. The latter's improvement over R63's 992.6 ms is too small to distinguish from a normal repeat. Isolated78caf12, quick/tmp/r64/quickqueued. Conditional full verdict recorder PID 353221/tmp/r64/record.log; no full requested if the small standalone difference is the best result.
10.0 hRound 65 — bound extra prefix rows across several GEMM tile sizes
-/tmp/r65/profileon checked R59, 05:34–05:36 UTC. The three-step profile records 1017 ms/step summed device kernels, with seven cuBLAS families at 73.6%, BF16 gradient scatter 5.9% and FA3 backward 2.5%. Instead of always rounding the number of retained *real* prefix rows to a multiple of 128, choose the largest supported tile with at most 32, 96 or 160 extra rows relative to maximal safe sharing. No fictitious tokens, K/V changes or changes to sample order./tmp/r65/layout-smoke.log: 425 randomized checks preserve exact row owners, valid positions and the permitted work bound. PID 348650/tmp/r65/test.logruns a four-way official sweep with unchanged R59 as a same-run control, followed by standalonetime; PID 359869/tmp/r65/finish.logrecords and commits the result. Next profile/conditional full/quick PID 358777/tmp/r66/profile-and-check.log, promotion guard PID 358778/tmp/r65/promote.logand recorder 358779/tmp/r65/record.log. - R66 feasibility only, not a timing: CPU trace attribution PID 355398/tmp/r66/group-overlap.logshows asymmetric durations on grouped NCCL scatters, 44.6 vs 225.5 ms total, despite similar summed GEMM times on the two ranks. CPU CUDA build PID 358207/tmp/r66/build.log; two-rank locked smoke PID 358776/tmp/r66/async-smoke.logvalidated an independent send stream with credit-limited receive slots. Each rank matched NCCL on 19 queued batches per configuration (misalignment, tails, varying streams and extreme BF16 values), then matched every full gradient and exposed the FP32 masters and moments over three real ZeRO steps. More testing waits for the next profile.