GLM-5.3 1.52×

Harness: Claude Code · switches: none

Judged
1.52×
Tier
N1
Self-best
1.53×
Rounds
30
Hours
10.0
API cost
$80.8
Input tokens · 99.6% cached
285 M
Output tokens
1.3 M
Requests
1,992
Sequence packing
✗ not done
Recompute / activations
✓ checkpointed layers kept per memory budget (→ 1.447×)
Optimizer
✓ Triton AdamW (−38 ms)
Communication & libraries
ZeRO-2 dropped as "forward is not the reference forward"; NCCL protocol and bucketing tried, no gain
Other main levers
fused RoPE (−190 ms), RMSNorm (−228 ms), SiLU, unpad / pad; skip grad norm; resident CLIP and cross-step prefetch (no gain)
Raw work log43 sections
0.0 hRound 1 — profile the unchanged reference (start 19:06, end 19:20)
Profile: /tmp/r1/profile/profile_tree.md (rank 0; rank 1 similar). 1 profiled step, GPU kernel time 2820 ms vs wall 2751 ms → GPU essentially saturated (NCCL on separate stream overlaps compute).

Top GPU kernels per step:
| item | ms | share |
|---|---|---|
| cuBLAS bf16 GEMMs (5 variants) | 1333 | 47.3% |
| NCCL AllGather (ZeRO-3 param gather) | 247 | 8.8% |
| NCCL ReduceScatter (grads) | 241 | 8.6% |
| bf16 elementwise/copy/cat/scatter soup (residual adds, RoPE mul/neg/cat, ZeRO copies, prepare_inputs cat/stack/pad, unpad/pad per layer) | ~660 | ~23% |
| flash-attn fwd+bwd | ~70 | 2.5% |
| fp32 elementwise + RMSNorm (pow/mul/mean) + optimizer multi_tensor | ~160 | 5.7% |

CPU side: aten::nonzero 742 ms inclusive per step — transformers 4.37.2's FA2 path calls _get_unpad_data (flatten→nonzero→cumsum) per layer per pass (~96×/step); each nonzero syncs the CPU with the GPU. GPU stays saturated but this serializes host work and will graph-break under torch.compile.

### Analysis (FLOP split, tokens/GPU/step ≈ 16×~1100)
GEMM 1333 ms splits ≈ fwd 333 / recompute 333 / bwd 667 (recompute = 1 extra fwd from gradient checkpointing). Recompute-related total incl. its elementwise + attn + ZeRO-3 fwd all-gather ≈ 660 ms ≈ 24% of the step → biggest lever, memory-bound (peak 73.5 GiB allocated vs 141 GiB Hopper → ~65 GiB headroom).

Second lever: ZeRO-3 per-layer all-gathers (247 ms) — 2-GPU NV18 NVLink; ZeRO-2 or better bucketing should cut most of it.
Third: ~660 ms elementwise soup (RoPE chain, RMSNorm fp32 casts, residual adds, cat/pad in prepare_inputs, per-layer unpad/pad) — torch.compile / fused-RoPE targets.
Fourth: aten::item 32.8 ms/step (per-step logging sync, small).

### Hotspot classification
- recompute: upstream switch enabled for memory, not needed at this batch — L0 "turn off memory-saving switch" (N0, bitwise: recompute runs identical kernels).
- NCCL: configuration (ZeRO stage) — L0/L2.
- elementwise soup: memory-bound → compile/fusion, L3/L4 later.
- per-layer nonzero syncs: known 4.37.2 inefficiency — L3 rewrite (precompute cu_seqlens once per step; N0).

No changes made. Next round: selective gradient checkpointing (checkpoint only every 2nd layer), switch-gated.
0.2 hRound 2 — selective gradient checkpointing (start 19:08, end ~19:45)
Change: llava/model/language_model/llava_llama.py _set_gradient_checkpointing override — leave the first N decoder layers uncheckpointed, rest keep upstream torch.utils.checkpoint (reentrant). N0: the backward recompute runs the same kernels on the same inputs as the original forward, so every value is bitwise identical; only memory/time trade changes.

Attempts:
1. LLAVA_CKPT_STRIDE=2 (fixed, half the layers uncheckpointed): OOM at step 2 (126.8 GiB allocated; measured activation cost ≈ 333 KB per padded token per uncheckpointed layer — much higher than the ~120 KB naive estimate because transformers 4.37.2's RMSNorm saves fp32 casts, RoPE saves mul products, and FA2 saves unpad gathers). Run /tmp/r2/time (ERROR).
2. LLAVA_CKPT_BUDGET=50 (length-adaptive: N = budget / (padded tokens × 0.36 MB), falling back to full checkpointing on long batches; pure function of batch shape, same code path in every mode): first time run (/tmp/r2/time2) thrashed — zero completed steps in 9 min with GPU 100%: each step's differently-shaped ~50 GB of activation blocks fragment the caching allocator; the freed-but-cached blocks then block the optimizer state's large allocations at the end of step 1, forcing free-everything-and-retry cycles. Killed.
3. Added PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (torch 2.1.2 supports it; verified bogus options raise while this one is accepted) and lowered the budget to 45 GiB. A first relaunch died because rope_fused.py had been written to the wrong directory (fixed; committed).

Current switches: {"LLAVA_CKPT_BUDGET": "45", "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"}.

4. The budget runs (2 and 3 above) were not allocator thrash after all: they deadlocked in NCCL (zero steps, GPU 100% = spin kernels). Root cause: with a per-rank token count the two ranks chose *different* checkpointed-layer sets; under ZeRO-3 a checkpointed layer issues extra parameter all-gathers when its backward recomputes the forward, so the ranks' collective sequences diverged. Fix: aggregate the step's padded token count with one MAX all-reduce per forward pass (_publish_step_tokens, outside the layer loop and any checkpointed region) so both ranks always pick the same set.

Result (/tmp/r2/time5, time.json): status ok, 2665.2 ms/step, speedup 1.0323 (bar 1.028, beyond_reference_noise true), window_spread 0.0012, peak_memory_allocated 89.63 GiB. Logged losses identical to the reference (1.3839, 1.288, ...) as expected for an N0 change.

Lesson: any per-layer decision that can differ across ranks deadlocks ZeRO-3. A hung run looks like GPU 100% with no step progress — kill it early, don't wait.

Calibration from the run: peak 89.63 GiB at budget 45 implies the real saved-activation cost is ~0.30 MB/token/layer (my constant says 0.377), i.e. the policy is conservative and the budget can rise. Next: enable fused RoPE (cuts ~33 KB/token/layer of saves + kernel time), recalibrate constants, raise budget.
0.5 hRound 3 — fused RoPE + budget recalibration (start ≈ 19:00 Nov 12, end 19:40 Nov 12)
Changes (all N0 targets):
1. llava/model/rope_fused.py (new): Triton fused RoPE replacing transformers 4.37.2's apply_rotary_pos_emb chain (per layer per pass: mul, rotate_half = neg+cat, mul, add for q and k — 5 kernels + cat + neg per tensor). The kernel reproduces eager bf16 rounding op for op: y = bf16(bf16(x1*c1) + bf16(-x2*s1)), y2 = bf16(bf16(x2*c2) + bf16(x1*s2)); the backward is the transpose rotation dx1 = bf16(bf16(g1*c1)+bf16(g2*s2)), dx2 = bf16(bf16(g2*c2)-bf16(g1*s1)) — verified bitwise (torch.equal fwd+bwd) against the pristine reference on 7 shapes incl. B=16 S=1100 H=32 D=128 and padded position_ids (/tmp/test_rope.py, ALL PASS). Custom autograd Function saves only the gathered cos/sin (~33 KB/token/layer less saved activation, and the gathered tables themselves are no longer duplicated per mul). Switch LLAVA_FUSED_ROPE (off by default), installed by install_fused_rope() at llava_llama.py import. Initial backward was wrong (reused the forward rotation — max abs diff ~9); fixed to the transpose rotation.
2. llava/train/train_mem.py: inert LLAVA_ZERO_STAGE switch (rewrites scripts/zero3.json stage 3↔2 in the run's tree copy before train() imports; unset = file untouched). Prepared for round 4, not enabled.
3. Checkpoint policy constants recalibrated: ACT_BYTES_PER_TOKEN_LAYER 0.40→0.34 MiB, FIXED_BYTES_PER_TOKEN 1.0 MiB; budget raised 45→55 via switch.

Result (/tmp/r3/time, time.json): status ok, 2621.2 ms/step, speedup 1.0496 (bar 1.028), window_spread 0.0026, peak_memory_allocated 99.45 GiB, no problems. Breakdown vs round 2: 2665.2 → 2621.2 ms (−44 ms ≈ rope fusion + a few more uncheckpointed layers from the budget raise).

Logged-loss check vs frozen reference: steps 1-2 exact; steps 3+ wobble in the 4th decimal (e.g. step 3: ref 1.3162 vs cand 1.3164). The reference's own 3 startup runs wobble identically (step 3: 1.3162/1.3163/1.3162; step 6: 1.2164/1.2172/1.2178) — this is production-mode kernel non-determinism (flash-attn backward atomics), not a numerics difference; candidate values sit inside the reference's own spread. Gate 1 bands (0.5%/25%) passed with huge margin. check --quick running to confirm gate 2 bitwise in deterministic mode.

Commit: fused RoPE (ce71c9d, fix d0eb41c), selective checkpointing (638b5cb), rank-aggregated policy tokens (c512821), train_mem.py switch scaffold (this round).

Next: round 4 = ZeRO-2 via LLAVA_ZERO_STAGE=2 (removes the 247 ms/step of ZeRO-3 per-layer parameter all-gathers; static cost ≈ +7 GiB from full bf16 params — ZeRO-2 still partitions grads and optimizer state, and at world size 2 every reduced element is a single fp32 add, so gradients stay bitwise). Then RMSNorm N0 memory trick (custom Function re-deriving the fp32 casts in backward) to push the uncheckpointed-layer count further.
0.5 hRound 3 addendum — gate 2 validation (`check --quick`, /tmp/r3/quick)
PASS, fully bitwise: 4724/4724 tensors bitwise identical, measured_tier N0, zero exact failures, zero violations (compare.json). Both cases (image-text ~16k padded tokens → ~11 uncheckpointed layers; text-only ~30k → 5 layers) reproduce the reference's loss, fp32 gradients, master-weight updates and both Adam states exactly in deterministic mode. The stacked submission (selective checkpointing + fused RoPE, switches LLAVA_CKPT_BUDGET=55, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, LLAVA_FUSED_ROPE=1) is N0 with evidence.
0.7 hRound 4 — ZeRO-2 screening (start 19:38, end 19:55 Nov 12)
Hypothesis: DeepSpeed ZeRO-3 spends ~250 ms/step on per-layer parameter all-gathers (round-1 profile: NCCL 488 ms, mostly latency-bound gathers at 96+ per step); ZeRO-2 keeps parameters resident, still partitions gradients and fp32 optimizer state, and at world size 2 every reduced gradient element is a single fp32 add — bitwise-neutral in theory. Prepared in round 3: inert LLAVA_ZERO_STAGE switch in train_mem.py rewriting scripts/zero3.json (verified byte-identical to upstream scripts/zero2.json for stage 2 and to the committed file for stage 3).

Screened with sweep (/tmp/r4/sweep): reference → z2b55 → z2b62 → z3b62 → reference.

| config | ms/step | speedup vs sweep refs | peak GiB | status |
|---|---|---|---|---|
| ref a / ref b | 2746.3 / 2759.4 (mean 2752.9, spread 0.0048) | 1.0 | 73.5 | ok |
| z3b55 (= round-3 submission) | 2621.2 (round-3 time) | 1.050 | 99.45 | ok |
| z3b62 (budget 55→62) | 2626.8 | 1.048 | 100.7 | ok — budget 62 buys nothing; keep 55 |
| z2b55 | 2511.4 | (1.0955) | 99.74 | ERROR: gate-1 first-loss band |
| z2b62 | 2502.4 | (1.0994) | 101.96 | ERROR: gate-1 first-loss band |

The ZeRO-2 speed is real (−110 to −119 ms/step) but both runs fail gate 1: the first logged loss is 1.3758 vs the reference's 1.3839 (−0.585%, band 0.5%), reproducibly (identical to 4 decimals in both budget configs). The full loss curve still tracks the reference's step-level pattern exactly (dips at steps 4/11/14/19/21/24) with random-sign deviations of ±0.2-1.6% — the signature of a changed within-step micro-batch split (same 32 samples per step, different 16/16 assignment), not of changed forward math.

Root-cause candidate (found by reading the code): LLaVATrainer._get_train_sampler builds LengthGroupedSampler without a generator, so get_modality_length_grouped_indices draws three torch.randperm(..., generator=None) from the global CPU RNG at first dataloader iteration — after DeepSpeed engine init. If stage 2 and stage 3 consume different RNG there, the within-megabatch order (and the 16/16 split) changes; the probe cannot see it (it compares per-step multisets and sizes only, and sweep does not probe at all). The logged loss is the mean over the step's micro-batches of each one's token mean, so it moves with the split — matching the observed deviations.

check --quick with the z2 config running (/tmp/r4/quick-z2): its exact items include the per-step RNG fingerprint, which should confirm or refute the RNG theory directly.

Submission unchanged (validated N0 config: budget 55, stage 3, fused rope; 1.0496×). ZeRO-2 is ON HOLD pending the mechanism.
0.7 hRound 4 conclusion — ZeRO-2 rejected: its forward is not the reference's forward
The check --quick with the z2 config (/tmp/r4/quick-z2) showed all exact items pass (RNG fingerprint, per-step sample multisets, optimizer/precision settings, dtypes) but only 32/4724 tensors bitwise — the loss, gradients, master updates and Adam states all differ in deterministic mode.

Instrumented bisect (throwaway tree copy, judge time runs, /tmp/rngdump-z3 vs /tmp/rngdump-z2, /tmp/r4/diag-z3c + diag-z2b):
- Initial weights: 12/12 key parameter hashes identical (lm_head, embed, q/k/o_proj, norms, gate_proj, mm_projector, CLIP tower) at train start.
- Global CPU RNG state at train start: identical (5feb54a4...), and at dataloader iteration: identical (60c7dd32...); the sampler's first 64 indices are identical → same batches, same within-step split.
- Yet the losses differ (z2 step-9 loss 1.0892 matches the z2 sweep pattern, not the reference's 1.0875).

So with identical weights, data, order and RNG, DeepSpeed stage 2 computes different forward values than stage 3. The only remaining difference is the parameter *storage*: stage 2 keeps every weight as a slice of one flat bf16 buffer, stage 3 gathers each weight into a fresh tensor — same values, different memory layout, different GEMM kernel selection/reduction order. That is intrinsic to the engine and not honestly patchable (the flat-buffer views are load-bearing for ZeRO-2's gradient/optimizer plumbing).

Verdict: ZeRO-2 is rejected — gate 1's first-logged-loss band (0.5%) fails at −0.585% systematically. The ~110-119 ms/step it saves (1.096-1.099× vs 1.050×) is real but unreachable under the gates. The budget control (z3b62) also bought nothing (2626.8 vs 2621.2 ms). Submission stays at the validated N0 config: LLAVA_CKPT_BUDGET=55, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, LLAVA_FUSED_ROPE=1 (1.0496×, gate 2 bitwise 4724/4724).

Lesson: a ZeRO-stage change is not numerics-neutral on this stack even at world size 2 — validate with check --quick before believing any "same math" argument about the engine's memory management.
0.9 hRound 5 — flash-attention unpad-metadata cache (start ≈ 19:56, end 20:01 Nov 12)
Change (committed b693914, claimed tier N0): llava/model/unpad_cache.py — switch LLAVA_UNPAD_CACHE (off by default) replaces transformers 4.37.2's module-level _get_unpad_data with an identity-keyed cache. The reference recomputes it once per decoder layer per pass (~96×/step; nonzero and .item() are host-device syncs); within one forward every layer sees the same attention_mask object, so the cache returns exactly the tensors the reference computes (CPU unit test: zero recomputes on identity hits, bitwise-equal values, new mask evicts). Wired via install_unpad_cache() next to install_fused_rope() in llava_llama.py.

time with LLAVA_UNPAD_CACHE=1 on top of the validated config (/tmp/r5/time, config /tmp/r5-unpad.json): 2630.0 ms/step, speedup 1.0461, window_spread 0.0043, peak 99.45 GiB, no problems. vs the validated config's own 2621.2 ms/step (1.0496×) that is a −0.34% change — inside the run's own window spread (0.43%) and the reference noise band (0.28%). No measurable gain.

Interpretation: the ~96 syncs per step were already hidden. The CPU launch pipeline runs far enough ahead of the GPU that blocking on .item() costs nothing on the critical path — the sync removes CPU stall time, not GPU work, and the GPU is the bottleneck. A metadata cache is the wrong lever here.

Decision: LLAVA_UNPAD_CACHE stays out of /submission/switches.json; the code stays committed and inert. No check needed (nothing enabled in the submission).

---
1.1 hRound 6 — NCCL protocol screen (start 20:02, end 20:13 Nov 12)
Profile first (/tmp/r6/profile, deleted; profile_tree.md numbers below): GPU kernel time 2692 ms/step on rank 0.

| slice | ms/step | share |
|---|---|---|
| cuBLAS GEMMs (5 sm90 variants) | 1239.7 | 46% |
| NCCL AllGather_RING_LL (758 calls) | 260.3 | 9.7% |
| NCCL ReduceScatter_RING_LL (293 calls) | 252.8 | 9.4% |
| elementwise soup (muls/casts/cat/neg/scatter, ~40 kernels) | ~500 | ~19% |
| flash-attn fwd+bwd | ~75 | 2.8% |

Both NCCL ops run the LL (low-latency) protocol — NCCL's small-message path (~25-50% of NVLink bandwidth). Hypothesis: forcing SIMPLE/LL128 or more channels would cut the 513 ms NCCL slice. Swept (/tmp/r6/sweep):

| config | ms/step | vs sweep refs (mean 2750.6) | peak GiB |
|---|---|---|---|
| c-simple (NCCL_PROTO=SIMPLE) | 2635.7 | 1.0436 | 99.45 |
| c-ll128 (NCCL_PROTO=LL128) | 2651.9 | 1.0372 | 99.45 |
| c-simple-ch (SIMPLE + MIN/MAX_NCHANNELS 16/32) | 2631.1 | 1.0453 | 99.45 |
| (current config, r3/r5 measurements) | 2621.2 / 2630.0 | 1.0493 | 99.45 |

No gain — all three variants are at or below the current config inside noise. The default LL protocol is already the best choice here; the NCCL time is bound by per-op latency and peer wait (758+293 small ops per step), not by protocol bandwidth. Switches unchanged. Cheap negative; the remaining comm lever is *volume and call count* (round 8: ZeRO-3 param residency), not protocol.
1.1 hRound 6 by-product — the fused RoPE never ran (round-3 attribution corrected)
Mining the profile's python frames showed apply_rotary_pos_emb_fused called 55×/step and _rope_supported 55×/step with zero Triton kernels in the trace: the round-3 fused rope silently fell back to the eager chain on every call. Root cause: transformers 4.37.2 builds position_ids as [1, S] (arange().unsqueeze(0), never expanded), so cos[position_ids] is the broadcast form [1, S, D], and _rope_supported required x.shape[0] == cos_g.shape[0] (16 ≠ 1). The eager rope chain is visible in the profile: 524 bf16 muls (119.2 ms), 372 aten::cat (43 ms), 174 neg (16.7 ms) ≈ ~190 ms/step of rope kernels that the fusion was supposed to replace.

Corollary: round 3's −44 ms was the LLAVA_CKPT_BUDGET raise (45→55) alone, not the rope fusion; the "fused RoPE" entry in the gain breakdown below was wrong until now.

Fix (committed 40fa515, tier N0): accept cos_g.shape[0] in (1, x.shape[0]) and read table row 0 via a zero batch stride (value-identical to broadcasting); plus do_not_specialize=["S"] so Triton compiles exactly one variant per direction regardless of the padded length's divisibility-by-16 class (verified: 2 cache entries across S ∈ {512, 777, 1627, 2048}) — otherwise a mid-run class change would compile inside a timing window and invalidate it. Bitwise GPU test vs the eager reference (/tmp/test_rope2.py): fwd outputs, q/k grads, broadcast + per-row forms, deterministic mode, H 8/32, D 64/128 — ALL PASS. Timing in round 7.

- Next queue (after round 7's sweep): pick the best of {rope-b55, rope-b62, rope-res-b44, rope-res-b50} → check --quick for gate 2 (rope fix is unit-tested bitwise; residency reuses gathered payloads, both claim N0) → full check once the stack is settled. Then: RMSNorm N0 fusion (fused forward + eager-identical backward that recomputes the fp32 casts; the profile shows ~60-90 ms/step of RMSNorm chain kernels plus 0.5 GiB/layer of saved fp32 casts), then the 118 ms direct-copy bucket (RMSNorm to(fp32) casts + logits.float() ~2 GB). ZeRO-2 and NCCL protocol levers are closed.
1.3 hRound 7 — the fused RoPE actually runs (start 20:13, end 20:24 Nov 12)
The round-6 fix (40fa515) makes the round-3 fusion engage for the first time. No switch changes: LLAVA_FUSED_ROPE=1 was already in the submission config; the fix is in the code path it gates. Swept (/tmp/r7/sweep, ref_a 2739.6 / ref_b 2745.8, mean 2742.7):

| config | ms/step | speedup vs sweep refs | peak GiB |
|---|---|---|---|
| rope-b55 (= submission switches) | 2431.4 | 1.1280 | 99.45 |
| rope-b62 (budget 55→62) | 2431.5 | 1.1280 | 100.7 |
| rope-res-b44 (+ LLAVA_PARAM_RESIDENT=1) | 2457.0 | 1.1164 | 105.0 |
| rope-res-b50 (+ residency, budget 50) | 2435.5 | 1.1263 | 108.8 |

The fused RoPE is worth −190 ms/step (2621.2 → 2431.4; 1.0496× → 1.1280× vs the sweep's own references; 1.1314× vs the frozen reference 2751.2) — matching the profile's eager-rope accounting (~190 ms of mul/cat/neg/gather kernels per step). The custom autograd Function also stops saving the eager chain's intermediates (~0.5 GiB per uncheckpointed layer), but budget 62 shows no further gain from that headroom at 16k-token steps.

ZeRO-3 param residency (LLAVA_PARAM_RESIDENT) is a no-gain: res-b50 ties rope-b55 inside noise (2435.5 vs 2431.4) and res-b44 is worse (the ~3 fewer uncheckpointed layers cost more than the saved gathers). The backward/recompute all-gathers it eliminates (~22 GB/step) were already hidden on DeepSpeed's comm stream — consistent with round 6's finding that NCCL here is latency-bound and overlapped, not volume-bound. The switch stays out of switches.json (committed inert, 0ab37bf).

Gate 2 validation (check --quick, /tmp/r7/quick) running: the rope fix is unit-tested bitwise (fwd, q/k grads, broadcast + per-row, deterministic mode), so the deterministic records should match the reference exactly (tier N0).
1.3 hRound 7 addendum — gate 2 validation and the full check
check --quick (/tmp/r7/quick): PASS — gate 0 clean (partition is the reference's), gate 2 fully bitwise: 4724/4724 tensors identical, measured tier N0, zero exact failures, zero violations (compare.json summary). The fused-rope path (Triton kernels, custom autograd Function saving only the gathered tables) reproduces the reference's loss, fp32 gradients, master-weight updates and both Adam states exactly in deterministic mode, in both the image-text and text-only cases.

The submission config is unchanged (LLAVA_CKPT_BUDGET=55, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, LLAVA_FUSED_ROPE=1); the speedup comes from the code fix making the existing switch work. Best measured: 2431.4 ms/step = 1.1280× vs the sweep's own references (2739.6/2745.8), 1.1314× vs the frozen reference. The full check (gates 0-3, the verdict) launched at 20:29 → /tmp/r7/check.
1.3 hRound 7 verdict — the full check PASSES (20:37)
check on the submission (commit 40fa515, switches LLAVA_CKPT_BUDGET=55, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, LLAVA_FUSED_ROPE=1), out /tmp/r7/check, check.json: status PASS:

- gate 0: clean (13/13 inputs match; partition is the reference's).
- gate 1: candidate 2432.3 ms/step vs frozen reference 2751.2 → speedup 1.1311 (windows 24.51/24.14 s, no compile in window, no interference flags). The bar was max(1.02, 1+2×0.0014) = 1.0028 — cleared by a wide margin.
- gate 2: 4724/4724 tensors bitwise identical, measured tier N0, zero exact failures, zero violations (compare.json).
- gate 3: verdict equivalent — train_loss_last20_canonical d = −0.000127 (−0.041σ, bound 3σ = 0.0094), probe_loss_last20 d = −0.000229 (−0.082σ, bound 0.0084).

This is the stack the submission now holds. Remaining levers are the NCCL call-count (round 8) and the elementwise soup (silu-mul, RMSNorm, unpad gather).
1.7 hRound 8 — ZeRO-3 reduce-bucket coalescing (start 20:29, end 20:50 Nov 12)
Hotspot from the round-6 profile: 293 one-parameter ReduceScatter_RING_LL calls (252.8 ms) + 758 AllGather_RING_LL (260.3 ms) per step. Root cause chase (CPU, while the round-7 full check ran): transformers/integrations/deepspeed.py:225 fills zero_optimization.reduce_bucket_size: "auto" with hidden_size² = 16,777,216 (stage3_prefetch 0.9×hidden²) — DeepSpeed's own default is 5e8. LLaVA-7B's smallest consecutive backward grad pair — a 4096-element RMSNorm weight followed by a 16777216-element o_proj = 16,781,312 — exceeds it by 0.02%, so __reduce_and_partition_ipg_grads flushes on every parameter and reduce_scatter_coalesced's coalescing never sees a list longer than one. The 293-call count in the profile matches the trainable-parameter count exactly (65 norms + 129×16.78M q/k/v/o/proj-l2 + 96×45.1M gate/up/down + embed/lm_head + proj-l1).

Change (committed c869793, inert switches): LLAVA_REDUCE_BUCKET / LLAVA_PREFETCH_BUCKET rewrite scripts/zero3.json before the Trainer reads it; unit-tested 6 paths on a tree copy. Bitwise analysis: grouping is per-element neutral (exact bf16 halving predivide → NCCL fp32 accumulate of the same two values → the same element ranges narrowed back out), tier N0.

Sweep /tmp/r8/sweep (refs 2745.0 / 2747.3, mean 2746.15):

| config | ms/step | vs baseline | peak GiB |
|---|---|---|---|
| sw-a baseline (= submission) | 2431.6 | — | 99.45 |
| sw-b reduce 5e8 | 2439.5 | +7.9 | 100.35 |
| sw-c reduce 1e8 | 2437.9 | +6.3 | 99.6 |
| sw-d reduce 5e8 + prefetch 5e7 | 2422.1 | −9.5 | 100.35 |

Verdict: reduce bucketing is a no-gain — the per-param reduce-scatters were already hidden behind backward compute (consistent with rounds 6/7: NCCL here is overlap/wait-bound, not call-count-bound), and bigger buckets add flat-buffer + interleave copies. Switches stay out of switches.json. The prefetch component of sw-d (−9.5 ms vs baseline, but −17 vs sw-b) is ambiguous at noise level → confirmed separately in round 9's sweep (arm sw-e, prefetch alone).

By-product while the GPUs were busy: the round-9 fused SiLU-mul was written and unit-tested (llava/model/silu_fused.py, switch LLAVA_FUSED_SILU, committed after its sweep): one Triton kernel per direction reproducing ATen's silu/mul/silu_backward op-for-op in fp32 opmath (libdevice __nv_expf, div.rn.f32), full blocks unmasked for vectorized 128-bit access (BLOCK=1024/warps=4 → 4.0 TB/s fwd, 3.5 TB/s bwd). GPU bitwise test /tmp/test_silu.py: fwd, grad_gate, grad_up bitwise-identical on every realistic case (randn full-size 360M elements, odd sizes, deterministic mode, both FMA variants); the only mismatch is the subnormal corner gate ∈ [−88.7, −87.5] where sig ≈ 6e-39 goes subnormal and Triton's toolchain flushes it where ATen preserves it (eager 1.5e-35 vs fused 0.0 — a measure-zero window for real activations; gate 2 decides the tier). Per-layer fwd+bwd on [16,2048,11008]: eager 3.459 ms → fused 2.576 ms (−0.88 ms × 32 layers × passes ≈ −60 ms/step expected).
1.9 hRound 9 — fused SiLU-mul (start 20:48, end 21:10 Nov 12)
Hotspot: the MLP activation chain (silu(gate) * up) — two eager kernels forward, three backward (~5 GB HBM traffic per layer-pass), running up to 3× per step in checkpointed layers. Change: llava/model/silu_fused.py (switch LLAVA_FUSED_SILU), one Triton kernel per direction reproducing ATen's silu/mul/silu_backward op-for-op in fp32 opmath (libdevice __nv_expf, div.rn.f32, fma'd 1 + x*(1-sig)); full blocks unmasked for vectorized 128-bit access (BLOCK=1024/warps=4 → 4.0 TB/s fwd, 3.5 TB/s bwd on [16,2048,11008]); saves only gate+up where the eager graph keeps three tensors.

GPU bitwise test /tmp/test_silu.py: fwd, grad_gate, grad_up bitwise-identical on every realistic case — randn at full training shape (360M elements), odd sizes, tiny, deterministic mode, both FMA variants. Only divergence: the subnormal corner gate ∈ [−88.7, −87.5] where sig = 1/(1+exp(88)) ≈ 6e-39 is subnormal and Triton's toolchain flushes it to 0 where ATen preserves 1.5e-35 (0.13% of a ×40-stress tensor; measure-zero for real activations; both |x|=30 and |x|=200 match). Gate 2 decides N0 vs N1.

Sweep /tmp/r9/sweep (refs 2743.1/2747.9, mean 2745.5):

| config | ms/step | vs baseline | peak GiB |
|---|---|---|---|
| sw-a baseline (= submission) | 2436.4 | — | 99.45 |
| sw-f + LLAVA_FUSED_SILU | 2415.5 | −20.9 | 96.28 |
| sw-e + prefetch 5e7 | 2440.1 | +3.7 (noise) | 99.45 |
| sw-g silu + prefetch | 2419.5 | ties sw-f | 96.28 |

Fused silu-mul: −21 ms/step (1.1359× vs sweep refs), peak memory −3.2 GiB. The gain is smaller than the kernel-level −0.88 ms/layer-pass × ~70 passes ≈ −60 ms estimate — the eager elementwise kernels were partially overlapped with the NCCL stream, so shortening them only pays out the critical-path share. Prefetch bucket (round-8 leftover) is confirmed no-gain — sw-e is noise-level positive-negative and sw-g adds nothing over sw-f; the switch stays out of switches.json.

check --quick on sw-f running → /tmp/r9/quick (gate 2: does the real fixture data ever land in the subnormal corner?).
1.9 hRound 9 closure — gate 2 PASS, silu promoted to the submission (21:06)
check --quick on the silu config (/tmp/r9/quick): PASS — gate 0 clean, gate 2 fully bitwise 4724/4724, tier N0 (compare.json: zero errors, zero violations). The fixture never lands in the subnormal corner, so the fused SiLU-mul is bitwise-identical in the judged cases. Committed bb80c0b; /submission/switches.json now carries LLAVA_FUSED_SILU=1 alongside budget 55, expandable segments and the fused rope. Best measured: 2415.5 ms/step = 1.1359× vs the sweep's own refs (2743.1/2747.9), 1.1392× vs the frozen reference; the full check for the round-10 stack will cover this config's gates 1+3.
2.0 hRound 10 — profile the silu stack, then RMSNorm fusion (start 21:07 Nov 12)
Protocol profile of the new stack first (/tmp/r10/profile, --detail), then the RMSNorm candidate: llava/model/rmsnorm_fused.py (already committed inert in bb80c0b, switch LLAVA_FUSED_RMSNORM) replaces the eager norm chain — 8 kernels fwd (~4.8 GB fp32 round-trips per norm on [16,2048,4096]) + ~8 bwd (~2.5 GB) — with one kernel per direction computing the variance, normalization, weight mul and the analytic gradient in registers. Tier N1 (torch's mean reduces in its own tree order; eager backward rounds at every kernel boundary). Expected −60..−140 ms/step depending on how much of the chain was already overlapped.
2.0 hRound 10 sweep — fused RMSNorm: −228 ms (21:19)
/tmp/r10/sweep (refs 2744.6 / 2751.8, mean 2748.2):

| config | ms/step | vs sweep refs | vs frozen ref | peak GiB |
|---|---|---|---|---|
| sw-h silu-base | 2410.5 | 1.1401 | 1.1413 | 96.28 |
| sw-i + LLAVA_FUSED_RMSNORM | 2182.1 | 1.2594 | 1.2608 | 85.87 |
| sw-j + CUBLASLT_WORKSPACE_SIZE=32MiB | 2415.9 | 1.1375 | 1.1388 | 96.28 |
| sw-k rmsnorm + ltws | 2182.1 | 1.2594 | 1.2608 | 85.87 |

The fused RMSNorm is worth −228 ms/step (2410.5 → 2182.1) and another −10.4 GiB of peak memory (the Function saves only bf16 x, not the eager graph's fp32 cast). Even more than the naive −360 ms/step kernel arithmetic suggested after overlap, i.e. the eager chain's fp32 round-trips were mostly on the critical path. CUBLASLT_WORKSPACE_SIZE=32MiB is a no-gain (+5.4 ms, noise; sw-k identical to sw-i) — the Lt heuristic already picks its best algos at the default 1 MiB workspace for these shapes. Closed.

The submission config now carries LLAVA_FUSED_RMSNORM=1 (tier N1: Triton reduction order vs torch's mean tree, and fused-register intermediate rounding vs eager's per-kernel rounding). The full check (gates 0–3) launched at 21:20 → /tmp/r10/check; gate 2 will read NOT_BITWISE by design (exact items must still pass), gate 3 is the verdict on the N1.
2.0 hRound 10 verdict — the full check PASSES at 1.2612x (21:28)
check on the rmsnorm stack (/tmp/r10/check, commit bb80c0b + LLAVA_FUSED_RMSNORM=1 in switches.json): status PASS.

- gate 0: clean (13/13 inputs; partition is the reference's).
- gate 1: candidate 2181.3 ms/step vs frozen reference 2751.2 → speedup 1.2612 (bar 1.0028; no compile in window, no interference flags).
- gate 2: NOT_BITWISE as designed, tier N1 — zero exact failures, zero violations: the RNG fingerprint, sample multisets, LR/Adam/DeepSpeed settings, trainable set, fp32 master/Adam dtypes and cross-rank pattern all match; only the recorded tensors differ (reduction order in the norm variance, fused intermediate rounding).
- gate 3: verdict equivalent — train_loss_last20_canonical d = −1.8e-05 (−0.006σ, bound 3σ = 0.0094), probe_loss_last20 d = +7.6e-05 (+0.027σ, bound 0.0084). The 60-step trajectory does not amplify the perturbation; the N1 is indistinguishable from the reference at the judge's resolution.

The submission now holds a verified 1.2612× PASS stack: budget 55 + expandable segments + fused rope (N0) + fused silu (N0) + fused rmsnorm (N1, gate-3 equivalent). Total −570 ms/step vs the frozen reference (−86 ckpt budget, −44 budget raise, −190 rope, −21 silu, −228 rmsnorm; the last two measured against their round's baselines).
4.8 hRound 18 first sweep — the checkpoint lever is large: 1982.2 ms/step at s3 (1.388x)
Sweep 1 (/tmp/r18/sweep, reference 2751.8 ± 0.0018, bracketed): s1 (ACT 0.26, n≈12) 2047.3 ms, peak 91.8 GiB; s2 (+ FIXED_TOK 0.6 / CEILING 130 / budget 70, n≈15) 1998.3 ms, peak 96.1 GiB; s3 (FIXED_TOK 0.5 / CEILING 133 / budget 85, n≈16) 1982.2 ms, peak 99.1 GiB. Every extra uncheckpointed layer is worth ~13–16 ms of wall — the recompute was on the critical path after all (the earlier "budgets 62/70 no gain" was the ceiling capping Δn at ≤1 layer, inside noise). Peak memory grows only ~1.5 GiB per typical layer (the peak is set by the longest batch, which self-limits harder), leaving ~30 GiB headroom at s3.

Second sweep in flight (/tmp/r18/sweep2): s4 (ACT 0.24, FIXED_TOK 0.4, CEILING 136, budget 110 → n≈18 typical / 9 worst) and s5 (FIXED_TOK 0.3, CEILING 138, budget 125 → n≈19/10), watching the peak before promoting anything.
4.8 hRound 18 sweeps 2–3 — 1924.0 ms/step at s7 (1.4297x), peak 108.4 GiB
Sweep 2 (/tmp/r18/sweep2, ref 2753.3/2746.0): s4 (ACT 0.24, n≈18/9) 1958.2 ms, peak 104.0; s5 (FIXED_TOK 0.3, CEILING 138, budget 125, n≈19/10) 1944.0 ms, peak 107.0. Sweep 3 (/tmp/r18/sweep3, ref 2742.9/2750.0): s6 (FIXED_TOK 0.25, CEILING 140, budget 140, n≈20/10) 1945.6 ms (window_spread 0.0186, noisy), s7 (FIXED_TOK 0.2, CEILING 142, budget 160, n≈21/11) 1924.0 ms, peak 108.4, window_spread 0.0005 — 1.4297x vs the frozen reference.

The peak grows far slower than the model predicts (worst-batch n 10→11 predicted +7.9 GiB, measured +0.5): the real per-layer activation bytes back-solve to ~0.20 MB/token/layer and real fixed+boundary to ~40 GiB, so the typical step's physical layer limit is ~26; s7's 21 leaves ~5 layers (~−20 ms each) before the wall. Sweep 4 in flight: s8 (ACT 0.22, FIXED_TOK 0.15, CEILING 145, budget 200 → n≈24/12) — the last push before promoting; promotion requires a clean window and peak with ≥15 GiB margin (the final evaluation runs the same fixture, so the measured peak is what the judge will see).
4.8 hRound 18 completed — s8 promoted: 1900.6 ms/step (1.4474x), peak 113.1 GiB
Sweep 4 (/tmp/r18/sweep4, ref pair 2744.7/2754.9, spread 0.0037): s8 (ACT 0.22, FIXED_TOK 0.15, CEILING 145, budget 200 → n=24 typical / 12 worst of 32 layers) 1900.6 ms/step = 1.4474x vs the frozen reference 2751.2, peak_memory_allocated_gib 113.07, window_spread 0.0006, problems none — 23.4 ms faster than s7 and the cleanest window of the sweep. Marginal gain per typical layer is now ~−8 ms (down from ~−16 at low n): the recompute that remains overlaps comm better as fewer layers checkpoint, so this lever is near its knee. Memory check before promoting: 113.07 GiB allocated of ~131 GiB physical (~18 GiB margin, inside the ≥15 GiB criterion); the policy is self-limiting (n ∝ 1/toks, peak ≈ fixed + room×(ACT_real/ACT_model) independent of step length), so a worse-drawn megabatch in gate 3's 60-step trajectory cannot blow past ~116 GiB.

Promoted (commit 5cff951: the four policy constants are now env-tunable with the old values as defaults — inert without the switch; /submission/switches.json now carries LLAVA_CKPT_BUDGET=200, LLAVA_CKPT_ACT_MB=0.22, LLAVA_CKPT_FIXED_TOK_MB=0.15, LLAVA_CKPT_CEILING=145 alongside the 1.3284x stack). Tier N0: uncheckpointing a layer runs the same kernels on the same inputs as its recompute would, so loss/gradients/optimizer states stay bitwise identical — the change moves only the memory/speed tradeoff, and the pre-existing N1 (fused RMSNorm) is what keeps gate 2 NOT_BITWISE.

Full check on the promoted config launched 00:23 → /tmp/r18/check (expect gate 2 NOT_BITWISE-by-design with 0 exact failures and gate 3 to decide the verdict; ~13 min).
4.8 hRound 18 verdict — the full check PASSES at 1.4391x
check on the promoted config (/tmp/r18/check, commit 5cff951): status PASS, speedup 1.4391 (candidate 1911.8 ms/step vs frozen reference 2751.2; the sweep's 1900.6 and this 1911.8 differ by ~0.6%, run noise).

- gate 0: clean (13/13 inputs before and after; the partition is the reference's).
- gate 1: speedup 1.4391, bar 1.02; window_spread 0.0006; peak 113.07 GiB allocated / 117.42 GiB reserved of ~131 GiB physical (~14 GiB headroom on reserved); logged-loss max rel diff 0.0024, first step 7e-05 — both far inside their bands.
- gate 2: NOT_BITWISE as designed, tier N1 (the fused RMSNorm; the checkpoint policy itself is N0) — 0 exact failures, 0 errors; 5 unresolved per-tensor tolerances, diagnostic only.
- gate 3: verdict equivalent on seed 0 (60-step trajectory, wall 181.5 s).
- measured_tier N1. GPU cost 0.28 gpu-hours.

The submission now holds a verified 1.4391× PASS stack — the checkpoint recalibration is worth −154 ms/step over the 1.3284× stack (2066.1 → 1911.8 in like-for-like judge runs; −163 vs the sweep's own timing). Round 18 closed.
5.6 hRound 19 (00:33–00:43) — re-test the inert backward fusions on the n=24 balance: 1890.1 ms/step (1.4556x)
Hypothesis. Rounds 16–17 showed the decoder residual-add fusion and the ZeRO-3 pre-division fold (NCCL AVG) are no-gain *at n=9*: their kernels ran inside the backward's NCCL overlap slack. Round 18 removed ~300 ms of backward recompute, so that slack shrank — the same kernels may now be exposed.

Sweep (/tmp/r19/sweep, ref pair 2744.7/2749.4, spread 0.0017; baseline = the promoted s8 config, 1900.6 in its own sweep / 1911.8 in the round-18 check):

| config | ms/step | vs frozen 2751.2 | peak GiB | window_spread |
|---|---|---|---|---|
| t1 = +LLAVA_FUSED_ADDNORM | 1896.7 | 1.4494 | 113.07 | 0.0016 |
| t2 = +LLAVA_NCCL_AVG | 1890.3 | 1.4543 | 113.07 | 0.0006 |
| t3 = both | 1890.1 | 1.4556 | 113.07 | 0.0004 |

t2 and t3 agree at 1890 — a ~10 ms gain over the baseline, and the reference bracket reproduced sweep 4's 2744.7 exactly, so the gain reads real: the NCCL AVG fold (removes the 17.8 ms pre-division pass from the backward) carries most of it, the addnorm fusion a few ms. Peak memory unchanged. Tier: addnorm bitwise-identical (verified at kernel and decoder-layer level in round 16); nccl_avg empirically bitwise on 1M+ bf16 values (round 17) — formally N1 at most; the stack's tier stays N1 (fused RMSNorm).

Promoted t3: /submission/switches.json now also carries LLAVA_FUSED_ADDNORM=1 and LLAVA_NCCL_AVG=1. No new commit — both modules are already in the tree from rounds 16–17 (5f55f4e, 14f0df7); this round flips configuration only. Full check launched 00:43 → /tmp/r19/check (gate 2 expected NOT_BITWISE-by-design N1 with 0 exact failures; gate 3 decides; fallback on any failure = revert to the round-18 switches, verified PASS 1.4391×).
5.6 hRound 19 verdict — the full check PASSES at 1.4573x
check on the promoted t3 config (/tmp/r19/check): status PASS, speedup 1.4573 (candidate 1887.9 ms/step vs frozen reference 2751.2; the sweep's 1890.1 and this 1887.9 agree — the −10 ms is real).

- gate 0: clean (13/13 inputs before and after).
- gate 1: speedup 1.4573, bar 1.02; window_spread 0.0011; peak 113.07 GiB allocated / 117.66 GiB reserved; logged-loss max rel diff 0.0012 — far inside its band.
- gate 2: NOT_BITWISE as designed, tier N1 — 0 exact failures, 0 errors (the addnorm fusion and the NCCL AVG fold contribute no new exact-item or tolerance deltas beyond the fused-RMSNorm N1 the stack already carried).
- gate 3: verdict equivalent on seed 0.
- measured_tier N1.

The submission now holds a verified 1.4573× PASS stack. Round 19's gain over round 18: −24 ms/step in like-for-like check timings (1911.8 → 1887.9). Lesson banked: overlap slack is config-dependent — re-test "no-gain" fusions whenever the compute/comm balance shifts materially.
6.1 hRound 20 (00:53–01:12) — profile the promoted config; kill the 27 ms ZeRO-3 release bubble: 1876.1 ms/step (1.4665x)
Profile first (/tmp/r20/profile, detail, the 1.4573× config): wall 1888 ms, rank-0 kernel sum 1828 ms — GEMMs ~1176 ms (64%, closed), NCCL 320.5 ms (AllGather 180.2 + ReduceScatter 140.3), elementwise/norm/optimizer ~330 ms. Trace arithmetic corrected an earlier misreading: the NCCL collectives are 94% overlapped with compute (only 19.8 ms exposed) — but the step has 184 ms of GPU idle, dominated by a 31.5 ms gap right after the last reduce-scatter. Inside that gap: PartitionedParameterCoordinator.release_and_reset_all, 27.1 ms of pure Python with zero CUDA runtime calls — DeepSpeed ZeRO-3 releasing every gathered parameter at the step boundary: two full module-tree walks, and per parameter 3–4 print_rank_0(f"...") calls whose f-strings (incl. debug_param2name_id_shape_device name formatting) are built eagerly though force=False never prints, plus dist.get_rank() per call. The optimizer step (4 AdamW launches, +37.8..+85.5 rel) already overlaps the next forward — the release bubble is the single largest exposed host cost.

Change (llava/train/release_fast.py, commit cddc437, switch LLAVA_FAST_RELEASE): no-op the module-local print_rank_0/debug formatter in partition_parameters, and make release_and_reset_all iterate a cached flat parameter list (same per-param operations in the same order, same verification pass). Tier N0 — bookkeeping only.

Two no-gains screened in the same round (per the round-19 lesson, re-tested on the new balance): NCCL_PROTO=Simple 1885.9 ms and LL128 1903.1 ms vs baseline 1887.9/1890.1 (sweep /tmp/r20/sweep, ref 2742.3/2747.6) — the protocol is irrelevant because the comm is 94% overlapped; NCCL's auto-tuned LL choice stays. Closed.

Result: time on switches+LLAVA_FAST_RELEASE=1 (/tmp/r20/time): 1876.1 ms/step = 1.4665× (−12 ms vs the round-19 baseline; the bubble compresses 27 → ~14 ms), window_spread 0.0007, peak 113.07 GiB unchanged, no problems. check --quick (/tmp/r20/quick): PASS — gate 2 NOT_BITWISE N1, 0 exact failures, 0 errors, the same 5 diagnostic tolerances as the established stack (no new deltas). Promoted into /submission/switches.json; full check launched 01:12 → /tmp/r20/check.
6.1 hRound 20 verdict — the full check PASSES at 1.4605x
check on the promoted config (/tmp/r20/check, commit cddc437): status PASS, speedup 1.4605 (candidate 1883.7 ms/step; the standalone time measured 1876.1 — both below round 19's 1887.9, the gain is real though the check-to-check delta is only ~4 ms given ~5 ms run noise).

- gate 0 clean; gate 1 window_spread 0.0024, peak 113.07 allocated / 117.70 reserved; logged-loss max rel diff 0.0023.
- gate 2: NOT_BITWISE as designed, tier N1 — 0 exact failures, 0 errors (no new deltas from the release patch; the stack's tier is still set by the fused RMSNorm).
- gate 3: verdict equivalent on seed 0.

The submission now holds a verified 1.4605× PASS stack (upstream 2751.2 → 1883.7 ms/step, −46.4%).
6.4 hRound 21 (01:22–01:33) — elementwise-kernel bandwidth audit; norm-backward retune is a microbench win but no in-situ gain (reverted)
Profile first (trace arithmetic on /tmp/r20/profile, no new GPU profile needed): per-call effective bandwidth of every fused Triton kernel, from kernel durations × per-call traffic. Closed as at/near bandwidth: rope 3.06 TB/s, rmsnorm_fwd 3.72, _pad_put_kernel 3.24, unpad/pad row-copies (~56 GB/step, 15.8 ms measured vs 16.7 theoretical), and — correcting an earlier mixed-average error — silu_mul fwd ~2.95 / bwd ~3.27 TB/s (I had averaged fwd+bwd bytes incorrectly; both are at bandwidth). The only kernels below half-bandwidth: _rmsnorm_bwd_kernel 461 µs/call (~1.6 TB/s) and _addnorm_bwd_kernel 564 µs/call (~1.0 TB/s) — both chain 32 rows serially per program (tl.static_range(RPP) with two block-wide tl.sum reductions per row), 557 programs on 132 SMs, long dependent chains.

Microbenchmark (/tmp/r21/kbench*.py, flock'd, real shape 17808×4096 bf16, kernel + gw-fold together, bitwise checks): rmsnorm_bwd best = RPP=16, num_warps=16 → 348 µs (1.41× over 492 baseline); more warps shorten each row's block reduction and RPP=16 keeps enough programs in flight. addnorm_bwd best = RPP=8, num_warps=8 → 497 µs (1.14× over 564). Expected in-situ ≈ −6.4 ms/step.

Applied both (inside the existing LLAVA_FUSED_RMSNORM / LLAVA_FUSED_ADDNORM switches) and timed: time /tmp/r21/time = 1876.5 ms = 1.4661× vs 1876.1 = 1.4665× before — no in-situ gain (the isolated −6 ms did not reach the wall). These kernels sit on backward segments where the overlapped NCCL leg binds (94%-overlapped comm): shortening the compute kernel just widens the exposed comm instead of the wall. Reverted to cddc437 (a numerics-perturbing config — gw/gx reduction order changes, tier N1 — with zero measured gain is pure gate-3 risk); no commit.

Round verdict — no-gain, elementwise tuning closed in situ. The ~150 ms of small GPU-idle gaps (418 gaps of 0.23 ms avg) is the remaining structure; that is host-side, so the next levers are host-op reduction or CUDA graphs, not kernel schedules.
6.6 hRound 22 (01:34–01:47) — LLAVA_CLIP_RESIDENT: keep the frozen vision tower's ZeRO-3 parameters resident — 1.486x
Profile first (gap attribution on the round-20 trace /tmp/r20/profile, still valid — fast-release only removed the one 31.5 ms boundary bubble): the remaining GPU idle is host dispatch, not kernels. The 0.1–0.5 ms band (438 gaps, 100.5 ms in the profiled span): 194 gaps between back-to-back elementwise kernels (36.9 ms), 109 before GEMMs (32.2 ms), 30 before NCCL all-gathers (8.5 ms); the gap midpoints land in CheckpointFunction (20.7 ms), allocator ops, AccumulateGrad — Python dispatch between kernels. The 0.5–2 ms band is the ZeRO-3 backward boundary (ReduceScatter → next op with AccumulateGrad / comm-record / copy_ on the host). Elementwise kernels are all at bandwidth (round 21), GEMMs near peak, comm 94% overlapped — so the lever is *removing* per-linear ZeRO-3 work, not making kernels faster.

Finding: the frozen CLIP tower (ViT-L/14-336, 303M params, requires_grad=False everywhere) is still ZeRO-3-sharded and re-gathered through NCCL in every forward *and* backward — 96 wrapped linears per direction per step, each with module pre-hook gather, inflight bookkeeping and release, plus the step-boundary release walk over them all. CLIP attention is eager ([256,577,577] softmax) but that segment is small; the gather traffic and host bookkeeping are the real cost.

Change (llava/train/clip_resident.py + trainer hook + a skip in release_fast.py, commit f9295b1, switch LLAVA_CLIP_RESIDENT):
- mark every frozen ZeRO-3 parameter of modules whose class name contains "VisionTower" as ds_persist=True, applied on the first training_step (the engine exists by then; the first forward gathers them normally, after which every fetch is a no-op because the parameter is already AVAILABLE);
- DeepSpeed re-partitions *everything* at each optimizer step, so both step-boundary passes — release_and_reset_all (release + verification) and DeepSpeedZeRoOffload.partition_all_parameters (release + verification, parameter_offload.py:283) — learn to skip ds_persist and not requires_grad params. Trainable persistent params (the small norm weights) keep the reference behavior so the optimizer's partition-update path is untouched.
- Cost: one resident bf16 copy of the tower, ~0.6 GiB — peak 113.07 → 113.09 GiB. Tier N0 in intent: the resident weights hold exactly the bits the all-gather produced.

Two crash fixes during bring-up (the first time run errored): partition_all_parameters also verifies all-partitioned at every optimizer step (the initial patch only covered release_and_reset_all), and the offload class is DeepSpeedZeRoOffload, not ParameterOffload.

Result: time on switches+LLAVA_CLIP_RESIDENT=1 (/tmp/r21/timec2): 1851.5 ms/step = 1.486x (−24.6 ms vs the committed 1876.1; the noise band is ±5 ms), window_spread 0.0026, peak 113.09 GiB, no problems. check --quick (/tmp/r21/quick): PASS — 0 exact failures, 0 errors, and the *same* violation profile as the established stack (5 violations / 32 bitwise-identical / 4724 checked), i.e. the residency added no numeric deltas at all. Promoted into /submission/switches.json (now 18 vars), committed f9295b1. Full check (/tmp/r21/check): PASS, speedup 1.4812x (1857.4 ms/step; gate 3 equivalent; 0 exact failures). Round 22 banked.
7.0 hRound 23 (01:56–02:08) — profile of the 1.48x stack; three re-screens all land inside noise (no promotion)
Profile first (/tmp/r23/profile, detail 1 step, the f9295b1 + 18-var config): GPU kernel sum 1853 ms/step — GEMMs 1171 ms (63%, near peak, closed), NCCL 339 ms (AllGather 203.9 + ReduceScatter 135.0, 18%; still mostly overlapped), fused elementwise/norm kernels all at their round-21 shares. CPU-op inclusive: AccumulateGrad 626, CheckpointFunction(Backward) 394/236, aten::nonzero 231.5 ms across 90 calls (mean 2.6 ms — the flash-attn unpad metadata: nonzero + cumsum + .max().item() host syncs, once per attention in forward *and* in every checkpointed layer's recompute), Pre/PostBackwardFunction ~150.

Three screens, all no-gain (round-5/round-9 verdicts re-tested at the new balance per the round-19 lesson):
1. LLAVA_UNPAD_CACHE=1 (metadata computed once per mask instead of ~42×/step; module committed and inert since round 5): time /tmp/r23/time = 1857.2 ms = 1.4813× — inside the committed config's own 1851–1876 range. The sync waits are already hidden under queued GPU work (direct GPU idle around the nonzeros is only ~3 ms/step). Stays out of switches.json.
2. zero3.json reduce_bucket_size 5e8→1e9 (tree copy /tmp/r23/treeA): 1851.6 ms, window_spread 0.0092 (noisy run), peak +1.9 GiB — fast edge of the range, not distinguishable.
3. zero3.json stage3_max_live_parameters 1e9→3e9 + stage3_max_reuse_distance 1e9→1e10 (keep checkpointed layers' params live across the recompute; tree copy /tmp/r23/treeB): 1850.4 ms, tight windows, but peak 118.99 GiB (+5.9) for at most ~7 ms — bad risk-reward.

Decision: nothing promoted; the committed config (f9295b1, 18 vars) stands. The remaining measurable structure: AllGather 203.9 ms is structural to ZeRO-3 on 2 ranks (params gathered per direction); the diffuse host-dispatch gaps are the only other pool and need CUDA-graph-class work. Round verdict: no-gain, three avenues closed with data at this balance.

Housekeeping: deleted /tmp/r1–r17 judge outputs and scratch files (kept r18–r23); /tmp at 6% of 14T.
7.2 hRound 24 (02:09–02:2x) — ZeRO-3 prefetch bucket 5e7→5e8 + live-parameter cap 1e9→3e9: 1.4908x
Profile first (fresh gap analysis on /tmp/r23/profile, the f9295b1 stack): the profiled step has 184 ms of GPU idle — 41.6 ms in 3007 sub-100 µs launch gaps, 77.2 ms in 316 gaps of 0.1–0.5 ms, 40.4 ms in 47 gaps of 0.5–2 ms, 25.0 ms in 7 gaps >2 ms. The ≥0.1 ms transitions concentrate on the collectives: nccl→nccl 21.7 ms over 27 gaps (~0.8 ms each — the host preparing the next bucket between collective launches), nccl→GEMM 17.6, elementwise→GEMM 13.3, nccl→Cat 10.9. Counting launches: 520 AllGather + 293 ReduceScatter kernels per step — the gathers fragment far below the reduce-bucket batching because stage3_prefetch_bucket_size defaults to 5e7 elements (~100 MB bf16 per gather batch, ~140 batches per direction at 7B params). The ReduceScatter fragmentation (293 ≈ per-parameter, unchanged from the round-20 trace) is DeepSpeed's own per-partition behavior, not bucket-size-limited (round 23's reduce-bucket screen showed no gain).

Screens (time on tree copies, config = the committed 18-var switches):
- C: prefetch 5e8 + max_live 3e9 → 1845.4 ms = 1.4908×, window_spread 0.0016, peak 116.81 GiB.
- F: prefetch 2.5e8 + max_live 3e9 → 1849.4 ms (windows 0.0001), same peak.
- D: prefetch 5e8, live 1e9 → 1852.0 ms; E: prefetch 2.5e8, live 1e9 → 1852.3 ms — the bigger bucket alone gains nothing: the prefetched batch itself hits the 1e9 live-parameter cap, forcing early releases and re-gathers. The two knobs only pay together.
- (round 23's B: live 3e9 + reuse 1e10 with prefetch 5e7 → 1850.4, +5.9 GiB — the live bump alone is worth less than the combination and costs more memory.)

Change (commit af35fdf, in-tree scripts/zero3.json): stage3_prefetch_bucket_size 5e8, stage3_max_live_parameters 3e9. Tier N0 vs the previous stack — bucketing changes no tensor value (each element's gather and reduction is independent of batching); the judge's exact DeepSpeed items (batch, accumulation, clipping, precision) are untouched. Cost: peak allocated 113.1 → 116.8 GiB (~10 GiB headroom remains; the peak is reproducible across runs). The committed config's own recent runs: 1851.5 / 1857.2 / 1857.4 (full check) — C is ~10 ms below the central value, twice the ±5 ms noise band.

Result: time /tmp/r23/timeC = 1845.4 ms/step = 1.4908×. check --quick on the promoted tree (/tmp/r23/quick2): PASS — 0 exact failures, 0 errors, the same violation profile as the established stack (5 / 4724 checked, 32 bitwise-identical): the config change is numerics-neutral. Full check (/tmp/r23/check2): PASS, speedup 1.4904x (1845.9 ms/step, gate 3 equivalent). Round 24 banked.
7.6 hRound 25 (02:34–02:4x) — plain F.linear for the resident frozen CLIP linears: weak ~3 ms signal, kept for the full check to arbitrate
Profile first (gap attribution on /tmp/r23/profile continued): the >0.5 ms gaps all sit on the per-parameter ZeRO-3 backward boundary (ReduceScatter → next op, host in AccumulateGrad/Pre/PostBackward/aten::copy_, 0.8–1.8 ms each) — the same per-param fragmentation as the 293 ReduceScatter launches. Batching that reduce path is DeepSpeed stage3 surgery with bitwise-ordering risk; deferred. Cheaper target found instead: DeepSpeed patches torch.nn.functional.linear globally (partition_parameters.py:501), so even with the tower resident every CLIP linear still pays LinearFunctionForZeroStage3.apply (192 calls/step) whose forward is exactly F.linear's math and whose backward skips the frozen weight/bias grads the same way the stock node does.

Change (commit 6503f27, rides LLAVA_CLIP_RESIDENT): save the stock F.linear at import time (before zero.Init patches the global) and rebind each vision-tower nn.Linear.forward to it — only modules whose weight is frozen and resident (or not ZeRO-managed); trainable linears keep the wrapper. Bring-up fixes: the rebound must walk each tower's descendants (the outer class-name guard skipped the linears), and the forward needs types.MethodType binding. Unit test: bias/no-bias forwards bitwise-equal to F.linear, trainable untouched, idempotent.

Result: time 1839.0 ms = 1.4961× (windows 0.0038) and 1846.0 = 1.4904× (windows 0.0017), mean 1842.5 vs the promoted config's 1845.4/1845.9 — a weak, consistent ~3 ms inside the ±5 ms band. The change turned out N1, not N0: for the tower 3D inputs the wrapper does matmul + in-place bias add (two bf16 roundings) while stock F.linear fuses the add — the quick check diagnostic violations went 5 -> 23 (new mm-case projector tensors, rel_err 0.039-0.065 vs gates 0.032-0.048; 0 exact failures both). Commit amended to state N1. The full check arbitrates on the deciding seed; revert if its trajectory gate fails. check --quick (/tmp/r23/quick3): PASS, 0 exact failures, 23 diagnostic violations (up from 5). Full check (/tmp/r23/check3): PASS, speedup 1.495x (1840.3 ms/step, window_spread 0.0028, peak 116.81 GiB) — trajectory verdict equivalent with huge margin: train endpoint 0.073 sigma, probe endpoint 0.035 sigma of the 3-sigma bound. Kept.
7.9 hRound 26 (02:56–03:03) — sub_group_size 5e9 breaks training; stability of the promoted config confirmed
Screen: sub_group_size 1e9 → 5e9 (fewer optimizer sub-groups, hoping for fewer Adam launches and coarser comm sub-group boundaries). The run ERRORED: "logged loss is zero or not finite at steps 2–6 where the reference is not: the run broke" — 5e9 elements exceeds int32 indexing somewhere in the flat-buffer/Adam path (5e9 > 2^31), so training breaks. Rejected outright; no timing value is meaningful. (2e9 would sit 7% under the int32 limit — not worth the proximity risk for a marginal expected gain, since the optimizer step already overlaps the next forward.)

Stability characterization of the promoted config (commit 3f5080f + 18-var switches, two more time runs /tmp/r26/stab1, /tmp/r26/stab2): 1844.2 ms (windows 0.0004) and 1842.8 ms (windows 0.0010). All five runs of this config: 1839.0 / 1840.3 (full check) / 1842.8 / 1844.2 / 1846.0 — mean ≈ 1842.5, spread ±0.2%, peak 116.81 GiB every time. The verified full check (1.495×, 1840.3) sits mid-distribution: the config is stable and the recorded speedup is representative, not a lucky draw.

Round verdict: one rejected knob (with the int32 finding recorded), stability banked. The submission remains at the verified 1.495× stack.
8.0 hRound 27 (03:04–03:11) — ReduceScatter batching closed (layout-inherent); CLIP SDPA no-gain (kept inert)
ReduceScatter reconnaissance (the last structural pool): read DeepSpeed 0.12.6's stage3 reduce path. The 293 per-step RS kernels come from __avg_scatter_grads(params_in_ipg_bucket) → reduce_scatter_coalesced, which launches one reduce_scatter_tensor per parameter — inherent to ZeRO-3's per-param rank-partition semantics (rank r holds slice [r·ceil(N/w), …) of *each* param's grad). A single flat RS over the bucket would distribute elements to ranks differently than the optimizer's grad_position mapping expects; preserving the semantics needs a rank-interleaved relayout of the whole 5e8-element bucket per flush — a full buffer pass that costs roughly what the launch overhead saves. Avenue closed with the mechanism recorded; not attempted.

CLIP SDPA screen (commit fae7ad6, switch LLAVA_CLIP_SDPA, NOT promoted): the tower is grad-free (frozen params, frozen pixels), so its eager attention ([bsz·heads,577,577] materialized twice per layer, 48 bmms + 24 softmaxes + >100 copies per step) is pure forward; SDPA replaces it with one fused kernel, falling back to the original forward whenever a mask / output_attentions / nonzero dropout is requested. Unit-tested (7e-7 vs eager on fp32 CPU; fallback exact; grads flow). Tier N2. time pair: 1839.3 / 1845.3 ms, mean 1842.3 vs the promoted config's own mean 1842.5 — no measurable gain (the eager ops were already hidden behind other work, like round 21's norm kernels). An N2 change with zero gain is pure risk: the switch stays out of switches.json; the module is committed inert for the record (the LLAVA_UNPAD_CACHE precedent).

Round verdict: two avenues closed with data. The submission stays at the verified 1.495× stack (3f5080f + 18 vars; fae7ad6 changes nothing while inert).
8.2 hRound 28 (03:09–03:19) — insurance: full check on the EXACT submitted tree → PASS 1.4954x; submission state verified
Round content: no tree change. A full check on /submission/llava at fae7ad6 (the inert LLAVA_CLIP_SDPA commit is the only diff vs the verified 3f5080f) with the committed 18-var /submission/switches.json — the exact state the final evaluation will judge.

Result (/tmp/r26/checkF): PASS, speedup 1.4954× (1839.7 ms/step, window_spread 0.0014, peak 116.81 GiB) — gate 2 NOT_BITWISE by design with 0 exact failures and the same 23 diagnostic violations as check3 (the stack's established profile); gate 3 equivalent with train endpoint 0.030σ and probe endpoint −0.087σ of the 3σ bound (better margins than check3's 0.073σ/0.035σ). The inert SDPA module changed nothing, as intended.

Hygiene pass: switches.json holds exactly the 18 intended vars; the tree is clean through fae7ad6; the four leftover env hooks in train_mem.py (LLAVA_ZERO_STAGE / LLAVA_PARAM_RESIDENT / LLAVA_REDUCE_BUCKET / LLAVA_PREFETCH_BUCKET) are early-return inert when unset and none are in switches.json; no stray GPU processes; /tmp at 6% of 14T.

Also this round (analysis while the check held the GPUs): re-read reduce_scatter_coalesced (deepspeed/runtime/comm/coalesced_collectives.py:73) and the stage3 flush path, and found round 27's "layout-inherent" closure of ReduceScatter batching was wrong: the function batches a whole flush list into ONE reduce_scatter_tensor after a rank-interleaved torch.cat (lines 96–122) — exactly the relayout round 27 deemed too costly, and DeepSpeed pays it only when a flush holds >1 tensor. The 293 one-param RS calls exist because reduce_bucket_size: "auto" resolves to hidden_size² = 16,777,216 (transformers fills it), which is smaller than the smallest large grad (16.78M), so line 1151 of stage3.py flushes on every parameter and the batched path never engages. Round 23's screen of 1e9 (1851.6, spread 0.0092) was unresolvable against the then-1851–1876 baseline; the config now sits at 1842.5 ± 3.5, so the re-screen is decisive. Avenue reopened; round 29 screens treeR (zero3.json reduce_bucket_size 1e9, one-key diff). Promotion criterion pre-registered: ≤1837.5 ms (≥5 ms beyond the five-run mean and the ±3.5 spread); 1837.6–1846 noise; above → reject.

Round verdict: submission verified at its final state (PASS 1.4954×); a mis-closed avenue reopened with a corrected mechanism.
check --quick on the promoted tree (/tmp/r29/quick): PASS — 0 exact failures, 0 errors, and the same violation profile as the established stack (23 diagnostic violations / 32 bitwise-identical / 4724 checked): the bucket change added no numeric deltas, confirming N0. Committed b618001 (in-tree scripts/zero3.json). Full check on the promoted tree (/tmp/r29/check): PASS, 1.5018× (1831.9 ms/step, window_spread 0.0014, peak 118.65 GiB) — gate 2 NOT_BITWISE by design, 0 exact failures, 23 diagnostic violations; gate 3 equivalent with train −0.013σ / probe −0.059σ of the 3σ bound. New verified best.
8.6 hRound 30 (03:36–03:42) — reduce-bucket size bisection: 1e9 is the optimum; both neighbors +5 ms
Screens (time on one-key tree copies, config = the committed 18-var switches + b618001):
- 2e9: 1837.1 ms = 1.4976× (windows 0.0013, peak 120.51 GiB) — +5.2 ms vs 1e9.
- 5e8: 1836.7 ms = 1.4979× (windows 0.0006, peak 117.71 GiB) — +4.8 ms vs 1e9, −0.94 GiB.

Size curve: auto 16.7M ≈ 1842.5 → 5e8 1836.7 → 1e9 1831.9 → 2e9 1837.1. 1e9 sits at a clean optimum: ~7 flushes balance RS batching efficiency against the tail flush's exposure and the bucket's memory; both neighbors pay ~5 ms. The promoted b618001 stands; the knob is closed.

Also analyzed this round (the no-cat flush idea, deprioritized): hook-time rank-interleaved ipg writes would skip the flush cat (~27 GB/step of reduce-stream traffic), but the RS layout constraint (rank-major, equal halves, reduce_scatter_tensor) pins region 1 at a fixed offset — partial flushes would pay a full-bucket RS, and the cat already runs on the reduce stream mostly overlapped. EV ≈ 2–4 ms for an hour of DeepSpeed surgery plus a re-verify cycle. Not attempted.

Round verdict: no-gain, knob closed with the optimum confirmed. Submission stays at b618001 (verified PASS 1.5018×). Parse note: time_tree.json keeps ms_per_step/peak_memory_allocated_gib at top level, not under performance.
8.8 hRound 31 (03:47–04:0x) — checkpoint room FIXED_GIB 50→42 (n=24→26): −30 ms, 1.5284x; the memory cliff found at n=28
Hotspot: the selective-checkpointing policy (llava_llama.py:171-181) sets n_unckpt = min(BUDGET, CEILING − FIXED_GIB − toks·FIXED_TOK) / (toks·ACT_MB); with the committed knobs the dominant ~16.6k-token image-text steps got n=24. LLAVA_CKPT_FIXED_GIB (default 50, env-tunable 1–100, previously unused) is the clean lever: 50→42 adds ~8 GiB of room ≈ +2 uncheckpointed layers. The round-20 calibration said ~6.4 ms per unrecomputed layer; peak was 118.65 alloc / ~133 reserved of ~141.5, so the increment looked memory-safe. Tier N0: which layers checkpoint never changes values (recompute runs the same kernels on the same inputs; round 20's full check established this for the whole scheme).

Screens (time, tree b618001, switch-file copies; the promoted config is switches.json-only, no tree change):
- n=26 (FIXED_GIB=42): 1800.1 ms = 1.5284× (windows 0.0036, peak 122.38 alloc / 136.02 reserved) — −31.8 ms vs 1831.9.
- n=26 confirm: 1803.4 ms = 1.5256× (windows 0.0012, same peaks) — reproducible; mean 1801.8, −30 ms.
- n=27 (FIXED_GIB=38): 1806.7 ms (windows 0.0101, noisy; peak 124.72 / 137.76) — no better.
- n=28 (FIXED_GIB=34): 1860.7 ms (windows 0.0004 — consistently slow; peak 125.9 / 137.74) — +60 ms cliff.

Mechanism of the cliff: reserved ≥ ~137.7 GiB is the allocator wall — at n=27/28 the caching allocator starts reclaiming/re-reserving (cudaFree forces device syncs), which costs far more than the saved recompute. n=26 (reserved 136.0) stays just under it. The measured per-layer saving at this balance is ~15 ms for 2 layers (2.3× the round-20 estimate — recompute got relatively costlier as everything else shrank).

Change: /submission/switches.json gains LLAVA_CKPT_FIXED_GIB=42 (19 vars; switches-only, nothing to commit in the tree). Full check on the promoted config (/tmp/r31/check): PASS, 1.5242× (1805.0 ms/step, window_spread 0.004, peak 122.38 GiB) — gate 2 NOT_BITWISE by design, 0 exact failures, 23 diagnostic violations; gate 3 equivalent, train 0.008σ / probe −0.183σ of the 3σ bound. New verified best.
9.0 hRound 32 (04:04–04:12) — post-shift re-screens of the two inert modules: both flat at n=26
Rationale: round 31 removed ~15 ms/step of recompute, shifting the balance a third time (after the prefetch and RS-batching shifts). Two committed-but-inert modules were rejected in earlier regimes because their cost was hidden behind queued GPU work; less recompute could expose them at the wall. Pre-registered bar: ≤1797.8 (−5 ms vs the n=26 mean 1802.8 over runs 1800.1/1803.4/1805.0).

Screens (time, tree b618001 + 19-var switches + the one switch; /tmp/r32):
- LLAVA_UNPAD_CACHE=1 (flash-attn unpad metadata computed once per mask instead of ~42×/step; the round-23 profile showed 231.5 ms CPU-inclusive of aten::nonzero): 1805.3 ms = 1.5239× (windows 0.0011) — inside the spread. Third strike (rounds 5, 23, 32): the metadata syncs stay hidden under queued work regardless of the recompute level. Stays out of switches.json.
- LLAVA_CLIP_SDPA=1 (fused SDPA for the frozen tower's eager attention; tier N2): 1803.4 ms = 1.5255× (windows 0.0025) — inside the spread. Second strike (rounds 27, 32): the eager attention remains behind other work. Stays out of switches.json (an N2 change with zero gain is pure risk).

Round verdict: no-gain, both avenues confirmed closed at the new balance; no promotion. The submission stays at the verified 1.5242× (b618001 + 19 vars).

Also this round — prefetch bucket 1e9 (the last untried config knob): time on a tree copy with stage3_prefetch_bucket_size 5e8→1e9 (/tmp/r32/tpf): 1840.4 ms, windows 0.0385 (self-inconsistent), reserved 137.74 — +37 ms and right at the allocator wall. Rejected; the forward's gather batching stays at 5e8. All ZeRO-3 knobs are now closed at measured optima: reduce bucket 1e9, prefetch 5e8, live 3e9, FIXED_GIB 42 (n=26), sub_group 1e9.

Process incident (fixed): the round-32 log command removed the CURRENT SUMMARY anchor line without re-adding one, so this round's truncation head -$((L-1)) got an empty L, failed as head --1, and the following unconditional cp wrote an empty file over the log. Restored from the incremental build /tmp/wl32.md (complete through round 32); the lost addendum is retyped above. New rule: log rebuilds copy to WORK_LOG.bak first and assert the anchor exists (grep -q || exit 1) before any truncation.
9.4 hRound 34 (04:31) — insurance: full check on the exact final state → PASS 1.5258x
Round content: no tree change. Full check on /submission/llava at aff914b (the inert agcat commit) + the committed 19-var switches — the exact state the final evaluation will judge.

Result (/tmp/r34/check): PASS, 1.5258× (1803.1 ms/step, window_spread 0.0028, peak 122.38 GiB) — gate 2 NOT_BITWISE by design, 0 exact failures, 23 diagnostic violations (the stack's established profile); gate 3 equivalent, train 0.052σ / probe −0.135σ of the 3σ bound. The inert module changed nothing, as intended.

Stability record for the submitted config: 1800.1 / 1802.3 / 1803.1 / 1803.2 / 1803.4 / 1803.7 / 1805.0 — mean 1803.0, ±0.14%, peak 122.38 GiB every run.

Next (round 35): re-profile and attribute the 3 gaps >2 ms (18 ms — the same size of bubble that fast-release removed in round 19); the r33 trace was deleted in cleanup, so a fresh profile is needed.
9.4 hRound 35 (04:32) — big-gap attribution: the remaining 18 ms is ZeRO-3 step-boundary structure; no safe lever
Profile + attribution (/tmp/r35/profile, the submitted config; script /tmp/r35/biggaps.py). The 3 gaps >2 ms (17.4 ms total):
- 4.79 ms, step start: the logged-loss .item() (_local_scalar_dense, 5×cudaStreamSynchronize) drains the stream, then the host catches up preparing the first fetch (record_param_comms, cat, sum). Deferring the loss log would reorder log lines against the judge's "first logged loss before any optimizer step" exact check — rejected on risk-reward.
- 4.62 ms, first all-gather: host prep (cat 0.07 + _allgather_base_) plus one cudaDeviceSynchronize (allocator/collective warmup) — once per step.
- 7.99 ms, step end: the final gradient flush — last ReduceScatter (6.41 ms kernel) + its cat (1.87 ms) + 0.68 ms host build + a 250×cudaEventQuery poll. Inherently exposed: the optimizer step cannot begin until every gradient is partitioned. This is the un-overlappable tail that round 24's "comm 94% overlapped" already priced in.

Conclusion: round 19 removed the one removable step-boundary bubble (the 31.5 ms release walk); the rest is ZeRO-3's architecture (final-flush-before-Adam, loss sync at logging_steps=1, first-collective warmup). The step-boundary pool (~26 ms) is closed as structural. With GEMMs at 66%/near-peak, comm at its measured optima, and the micro-pool class closed by four flat attempts, no safe lever above ~2 ms remains anywhere in the profile.

Round verdict: no-gain with the mechanism recorded. The submission stays at the verified 1.5258× (aff914b + 19 vars; four timing runs 1800.1–1805.0, mean 1802.9 ±0.14%). Mode from here: stability maintenance — periodic time confirmations, log currency, and the submission never left in a half-edited state.
9.5 hRound 36 (04:36) — stability maintenance: fifth timing point on the submitted state
time on the exact submitted state (aff914b + 19 vars, /tmp/r36/time): 1802.3 ms = 1.5265× (windows 0.0019, peak 122.38 GiB, no problems). Five-run record: 1800.1 / 1802.3 / 1803.1 / 1803.4 / 1805.0 — mean 1802.8, ±0.14%, peak identical every run. The submitted configuration's speed is stable and the verified full checks (1.5242×, 1.5258×) sit mid-distribution. No change; the search remains exhausted (rounds 33–35 closed the micro-pool and step-boundary classes), so the mode is: keep the submission verified, re-time periodically, keep this log current.
9.6 hRound 37 (04:38–04:45) — library-swap lever: flash-attn 2.6.3 screened and rejected; env restored and re-confirmed
Lever: /opt/candidate-env may carry library upgrades (the judge runs candidate processes with it; version differences are not errors). The one perf-relevant candidate compatible with the pinned torch 2.1.2+cu121 is flash-attn: the LLM attention kernels cost 61.4 ms/step (fwd 18.9 + bwd 42.5) on 2.5.8. GitHub release assets: v2.7+ dropped torch2.1 builds; v2.6.3 has cu118/cu123torch2.1 wheels — the cu123/abiFALSE/cp310 wheel imports and runs fine under cu121 torch (same libcudart SONAME).

Smoke: varlen fwd+bwd (head 128, causal, bf16) finite; deterministic=True supported and bitwise-repeatable (the judge's gate-2 deterministic mode holds).

Screen (/tmp/r37/tfa, tree+switches unchanged, fingerprint confirms flash-attn 2.6.3): 1806.8 ms, windows 0.0047 vs the config's mean 1802.8 — +4 ms, no gain (2.6.3 is not faster on this workload: head 128, causal, sm90). Rejected; uninstalled immediately — leaving it would silently shift the judged runs off the verified configuration.

Restoration proof: pip list shows flash-attn 2.5.8 (llava-env, via path); confirmation time on the restored env (/tmp/r37/tconfirm): 1803.7 ms = 1.5253×, windows 0.0015, peak 122.38, run fingerprint flash-attn 2.5.8 — mid-distribution. Six-run record: 1800.1 / 1802.3 / 1803.1 / 1803.4 / 1803.7 / 1805.0 — mean 1802.9, ±0.14%.

The rest of the library lever is closed by construction: torch/DeepSpeed/transformers upgrades break the pinned stack (my patches target DS 0.12.6 internals; train.py's AST is frozen against transformers 4.37.2; a newer torch breaks both); a newer Triton cannot help — the Triton kernels are bandwidth-bound (round 21 measured 2.9–3.3 TB/s), so better codegen moves nothing.

Round verdict: the last untried lever class is explored and closed with data. The submission stays at the verified 1.5258× (aff914b + 19 vars + the pinned library set).
9.7 hRound 38 (04:47) — maintenance: integrity verified, seventh timing point
Integrity pass (all green): tree clean at aff914b; switches.json = the 19-var set (inert switches absent, FIXED_GIB=42 present); candidate env = the verified library set (flash-attn 2.5.8); /tmp at 6% of 14T; no stray GPU processes.

Periodic timing (/tmp/r38/time): 1803.2 ms = 1.5258× (windows 0.0012, peak 122.38 GiB, no problems). Seven-run record: 1800.1 / 1802.3 / 1803.1 / 1803.2 / 1803.4 / 1803.7 / 1805.0 — mean 1803.0, ±0.14%, identical peak every run. The submitted state continues to time exactly where its verified full checks (1.5242×, 1.5258×) sit.

Round verdict: no change. The search remains exhausted (rounds 33–37 closed micro-pools, step boundaries, and library swaps with data); the mode is protection of the verified 1.5258×.
9.9 hRound 39 (04:47–05:05) — vision prefetch: cross-step CLIP tower on a side stream; both placements measured and REJECTED (inert commit 905d5a2)
Target: the frozen CLIP tower's forward is 34.3 ms standalone ([16,3,336,336] bf16, measured /tmp/r39/towerbench.py) and runs serially at the top of every step; the projector is only 0.8 ms. Design: the dataloader peeks one batch ahead, and at the end of training_step a side CUDA stream (gated side.wait_stream(compute)) runs the tower for the NEXT batch; the next step's patched encode_images consumes the stash and runs only the projector with fresh weights. Tier N0 by construction (frozen resident weights under LLAVA_CLIP_RESIDENT, same kernels/input/mode; misses fall back to stock). ZeRO-3 guard: prefetch only when the tower's params are ds_persist (no coordinator state touched).

Bug found and fixed by the screen: first run ERRORED (mat1 and mat2 ... float != BFloat16 at mm_projector): the tower returns features cast to the CALLER's input dtype (clip_encoder.py:55 .to(images.dtype)), and the serial path gets bf16 images only because transformers' _prepare_input casts float inputs to the DeepSpeed config dtype. Fixed by replicating that cast in the launcher (deepspeed plugin dtype, fallback next(tower.parameters()).dtype); the unit test's FakeTower now models the dtype-mirroring return and catches the regression (/tmp/r39/test_vp.py ALL PASS).

Screen (a) — launch at the END of training_step, overlapping the drain window (final grad flush + Adam, ~40 ms): /tmp/r39/tvp2 = 1806.7 ms (windows 0.0013, peak 122.43 alloc / 136.92 reserved — under the 137.7 wall), no problems. +3.7 ms vs the 1803.0 mean: no gain. The window is bandwidth-saturated (NCCL reduce-scatter + memory-bound Adam), so the side-stream tower makes little progress there and the next step blocks on the event for the remainder.

Screen (b) — launch at the START of training_step, overlapping the step's own forward: /tmp/r39/tvp3 = 1974.0 ms (+171!), window spread 8.6% (interference). The forward's GEMM waves leave no SM headroom; the concurrent tower stretches the step far more than it saves.

Round verdict: every region of this step is SM-saturated (GEMMs 66% of kernel time) or bandwidth-saturated (flush RS, Adam), so the tower's 34 ms is genuinely serial — the cross-stream overlap class is closed for the tower with both placements measured. The module ships inert behind LLAVA_VISION_PREFETCH (absent from switches.json) at placement (a), with both measurements recorded in its docstring; committed 905d5a2. The submission's timing state is unchanged (1803.0 mean).

# CURRENT SUMMARY (updated 05:05 Nov 13, after round 39: vision prefetch measured at both placements and rejected — inert commit 905d5a2; verified best 1.5258x unchanged)

- Submission state: /submission/llava branch opt, clean through 905d5a2 (inert LLAVA_VISION_PREFETCH module, switch off; before it inert LLAVA_AG_BATCHED_CAT aff914b); /submission/switches.json = 19 vars (LLAVA_CKPT_FIXED_GIB=42 → n=26; NO LLAVA_AG_BATCHED_CAT / _UNPAD_CACHE / _CLIP_SDPA / _VISION_PREFETCH). Candidate env = the verified library set (flash-attn 2.5.8).
- Latest verified full check: /tmp/r34/check = PASS 1.5258× (the exact submitted config; 1803.1 ms/step, gate 2 NOT_BITWISE by design with 0 exact failures, gate 3 equivalent 0.052σ/−0.135σ, peak 122.38 alloc / ~136 reserved). Earlier verified: 1.5242× (r31), 1.5018× (r29), 1.4954× (r28), 1.495×, 1.4904×, 1.4812×, 1.4605×.
- Timing record (this config): 1800.1 / 1802.3 / 1803.1 / 1803.2 / 1803.4 / 1803.7 / 1805.0 — mean 1803.0, ±0.14%, identical peak 122.38 every run.
- Gain breakdown (ms/step vs upstream 2751.2): −86 ckpt budget, −44 expandable_segments, −190 fused RoPE, −21 fused SiLU, −228 fused RMSNorm, −22 fused unpad, −38 Triton AdamW + spawn, −36 skip grad norm, −23 fused pad, −154 ckpt n=24/12, −21 addnorm + AVG, −12 fast release, −25 CLIP resident, −10 prefetch 5e8 + live 3e9, −5 CLIP plain-linear, −10.6 reduce bucket 1e9, −30 ckpt n=26 → ~1803 ms/step (1.5258×).
- Memory posture: peak 122.38 alloc / 136.0 reserved; the allocator wall is reserved ≈ 137.7 (n=27/28, prefetch 1e9 and the tower's side-stream transients at placement (a) 136.92 all sit under it; anything past it gets SLOWER). Do not add live memory past ~1.5 GiB.
- State of the search: every config knob is at a measured optimum (reduce 1e9, prefetch 5e8, live 3e9, FIXED_GIB 42/n=26, sub_group 1e9); GEMMs near peak (66% of step); comm mostly overlapped; the remaining ~138 ms of GPU idle is host-dispatch granularity needing CUDA-graph-class work (infeasible under ZeRO-3 per-param hooks and dynamic shapes). Cross-stream overlap is now closed TWICE over: micro-pools (4 flat) and the one big serial block (CLIP tower: +3.7 at the drain, +171 over the forward). No well-scoped lever above ~2 ms remains.
- Rejected / no-gain (cumulative): ZeRO-2; reduce_bucket {5e8, 2e9, 1e9@old-balance}; prefetch {5e7, 2.5e8, 1e9}; reuse 1e10; sub_group 5e9 (BREAKS, int32); ckpt n=27/n=28 (allocator wall); NCCL protocols/channels; CUBLASLT workspace; torch.compile; logits slicing; GEMM tuning; fused-kernel schedule tuning; unpad metadata cache (×3); CLIP SDPA (×2); agcat batched stitch (inert); no-cat flush (low EV); residual-add pool (autograd frames, not fusable); flash-attn 2.6.3 swap (+4 ms, env restored); step-boundary bubbles (structural); cross-step CLIP tower prefetch (both placements: 1806.7 drain / 1974.0 forward — inert 905d5a2).
- Process lessons: bench commands run bare; import-test every switch ON; prove the fused path ran; deterministic stand-ins for flash-attn; deep-copy switch files; re-test no-gains after balance shifts; measure overlap before calling comm exposed; microbench wins must be confirmed by time; promotion needs a gain beyond the config's own spread; re-read the code before accepting a closure; watch peak_memory_reserved (the wall is reserved-side); log rebuilds: back up first, assert the anchor line exists before truncating; kernel-time pools behind slack do not convert to wall time; model the real op's dtype contract in unit-test stand-ins (the FakeTower missed the tower's .to(images.dtype) return and the real screen caught it); a side-stream overlap must budget for BANDWIDTH saturation, not just SM idle — the drain window had none to give.
↑ LLaVA