DeepSeek V4.1 Flash-1017 2.22×

Harness: DeepSeek's own harness · switches: none

Judged
2.22×
Tier
N1
Self-best
2.23×
Rounds
32
Hours
7.7
API cost
$2.58
Input tokens · 99.6% cached
249 M
Output tokens
0.6 M
Requests
696
Sequence packing
✓ round 4 LLAVA_PACK, 1.28 → 1.77×
Recompute / activations
✓ stepped from "save MLP only" to no recompute at all (adding qkv / attention saves + expandable segments)
Optimizer
✓ fused AdamW on fp32 master partitions
Communication & libraries
all params resident; ZeRO-3
Other main levers
installed Liger fused kernels (RMSNorm / SwiGLU / RoPE) into /opt/candidate-env; fused linear + cross-entropy; tried caching CLIP features, reverted halfway
Raw work log33 sections
0.2 hRound 1 (start 19:08, GPU lock 19:09)
### Profile of the unchanged tree (/tmp/r1/profile, profile_tree.md)
GPU kernel sum 2707 ms/step of 2785 ms wall (union busy 2409 ms, 192 ms idle). Breakdown:
- GEMM 1380 ms (50.5%): sm90 xmma bf16 (cuBLAS) + a few cutlass kernels.
- elementwise/copy 809 ms (29.6%): RoPE rotate_half cats (1276 cat kernels), RMSNorm chains,
  SwiGLU, residual adds, and DeepSpeed ZeRO-3 gather copies.
- NCCL 333 ms (12.2%): 758 AllGather calls 259 ms + 293 ReduceScatter 71 ms.
- flash-attn fwd+bwd 79 ms (2.9%); reductions 30 ms.
- CPU: 112 aten::nonzero calls/step (791 ms of CPU time incl. device waits) from
  transformers 4.37 _get_unpad_data, recomputed in every layer and again under gradient
  checkpointing recompute. CheckpointFunction 56 calls, aten::item 165 calls.
- GEMMs are already near peak; the win is in non-GEMM work + recompute + ZeRO traffic.

### Changes tested
1. ZeRO-2 (scripts/zero3.json stage 2): /tmp/r1/time 2684.7 ms/step, speedup 1.0375,
   BUT gate-1 first-loss check ERROR: first logged loss 1.3758 vs 1.3839 (0.585% > 0.5%).
   Forward numerics change with the ZeRO stage (different GEMM/linear path + memory footprint).
   Rejected: cannot pass gate 1. Logged as a failed round (no commit).
2. ZeRO-3 with all params persistent (stage3_param_persistence_threshold=1e9,
   prefetch bucket 1e9): /tmp/r1/time_persist 2752.9 ms/step, speedup 1.0118, status ok,
   first logged loss 1.3839 (identical). Keeps the stage-3 forward path, drops the per-layer
   all-gather. N0 (no math change).

### In flight
3. Selective activation checkpointing: keep activations of N of the 32 decoder layers instead
   of recomputing them (LLAVA_GC_SKIP_LAYERS, env switch) + cache _get_unpad_data per
   attention-mask tensor (LLAVA_UNPAD_CACHE), together with (2).
   Note: first round started without a clean WORK_LOG; details above reconstructed from /tmp.
0.4 hRound 2 (19:20-19:35): fused kernels (Liger) + unpad cache
Profile after (1) (persistent params, unpad cache): /tmp/r1/prof_gc/profile_tree.md
GPU 2531 ms/step: GEMM 1325, elementwise/copy 797, NCCL 211, flash 79, reduce 30.
Still 112 aten::nonzero/step (64 with the full 17808-entry mask, from
transformers' _get_unpad_data in every layer and again under recompute).

Changes:
- _get_unpad_data result cached per attention-mask object (exact, no math change).
- Liger-Kernel 0.2.1 Triton ops (installed in /opt/candidate-env, no-deps) for
  Llama RMSNorm, SwiGLU (silu(gate)*up), RoPE.  N1 (rounding order only).
  Switch LLAVA_LIGER (default on).

time /tmp/r1/time_liger: 2245.8 ms/step, speedup_vs_frozen_reference 1.2402,
status ok, peak 83.58 GiB, first logged loss 1.3838 (ref 1.3839). Committed 609551e.

Also implemented (switch off by default, LLAVA_MLP_STORE_LAYERS): selective
activation checkpointing that checkpoints only the attention block of N layers and
keeps the MLP activations (2/3 of the recompute FLOPs).  Whole-layer skipping with
LLAVA_GC_SKIP_LAYERS OOMs at 8 layers (each uncheckpointed layer costs ~4.6 GiB),
so the finer-grained variant is the one to use.

Gate check running: check --skip-timing (gates 0, 2, 3) in /tmp/r1/check_skip.
0.7 hRound 3 (19:35-19:55): selective activation checkpointing
Profile with liger (/tmp/r3/profile_liger): GPU kernels 1980 ms/step, GEMM (incl. flash)
1490, elementwise/copy 237, other 143, NCCL 95. Recompute is ~1/3 of the GEMM work.
Idea: keep the MLP activations of N decoder layers (checkpoint only their attention block),
which removes 2/3 of the recompute of those layers.

- Whole-layer skip (LLAVA_GC_SKIP_LAYERS) OOMs at 8 layers: ~4.6 GiB/layer.
- MLP-only store costs ~1.5 GiB/layer at seq 1113, ~2.7 GiB/layer at seq 2048.
- Fixed 12/32 layers OOM'd because of an implementation bug (non-stored layers were
  left uncheckpointed); fixed by keeping the whole-layer checkpoint for them.  Tiny-model
  test: gradients are bitwise identical between full checkpointing and MLP-store.
- Adaptive count (from the batch's padded sequence length) makes DeepSpeed's ZeRO trace
  cache invalidate ("Invalidate trace cache ... expected module 44, got 32") and the run
  stalls; not usable. Fixed count keeps the graph constant.
- Fixed 32 layers OOMs (133.5 GiB allocated); at seq 2048 the safe fixed count is ~14.

Current experiment: fixed 14 layers (/tmp/r3/time_mlp14), liger on.
0.7 hRound 3 result
- N=14: /tmp/r3/time_mlp14 2113.1 ms/step, speedup 1.3181, peak 120.8 GiB allocated.
  ~9.5 ms saved per stored layer (recompute total ~456 ms).
- N=8: /tmp/r3/time_mlp8 2169.8 ms/step, speedup 1.2837, peak 104.8 GiB.
  Chosen for the submission: the trajectory (60 steps, other data order) contains
  batches up to the 2048-token truncation; at seq 2048 a stored layer costs ~4.9 GiB,
  so N=8 peaks at ~128 GiB, while N=14 would OOM there.
- Gate 3 validation running: check --n1 -> /tmp/r3/check_n1.
- Adaptive per-step N (LLAVA_MLP_STORE_LAYERS=auto) stalls: ZeRO-3 trace cache is
  invalidated whenever the checkpoint graph changes. Default is now 0 (off).
1.0 hRound 4 (19:56-20:10): padding-free packing
Profile of the n=8 config (/tmp/r4/profile): 1905 ms GPU kernels, GEMM 1328,
elementwise 233, NCCL 95, other 89, liger triton 63, flash 82.
25% of the padded rows of a micro-batch are padding (16 x 1113 rows = 17808, real
tokens 13269) and every GEMM processed them.

- Tried dropping the flash padding mask instead (LLAVA_SKIP_PAD_ATTN): 2227 ms,
  slower (the full-causal attention on padded rows costs more than the copies saved).
- Implemented packing: concatenate the real tokens of the micro-batch, run the
  decoder on the packed sequence, flash-attn varlen with cu_seqlens, per-sample
  position ids, first label of each sample masked so the causal shift cannot cross
  a sample boundary. The loss stays the micro-batch token mean (upstream's
  normalization). Caught a real bug with it: Liger's RoPE ignores position_ids and
  uses cos rows 0..seq-1, which is only right for arange positions; the wrapper now
  indexes cos/sin by position_ids (HF semantics), bitwise identical for the
  non-packed path.
- time /tmp/r5/time_pack2: 1572.2 ms/step, speedup 1.7716, status ok,
  first logged loss 1.383 (ref 1.3839), peak 83.6 GiB. Committed cbbc84f.
- Gate 3 validation running: check --n1 -> /tmp/r5/check_n1.

### Full check PASS (20:16-20:24, /tmp/r5/check_full)
Gates 0,1,2,3 all pass. speedup 1.804 (candidate 1544.0 ms/step, frozen reference
2785.3), windows 15.66/15.22 s, window ratio to frozen 0.556/0.552, peak 88.6 GiB,
first logged loss 1.383. Gate 2 NOT_BITWISE with 0 exact failures (29 diagnostic
per-tensor overruns, mostly Adam exp_avg_sq of near-zero-gradient norm weights).
Gate 3: canonical -0.098 sigma, probe -0.210 sigma.
Switch file at this point: {"LLAVA_UNPAD_CACHE":"1","LLAVA_LIGER":"1",
"LLAVA_MLP_STORE_LAYERS":"12","LLAVA_PACK":"1"}.
1.3 hRound 5: push the selective checkpointing further
- N=16 (/tmp/r6/time_pack16): 1522.5 ms/step, speedup 1.8295, peak 93.7 GiB.
  check --n1 PASS (0.046 / 0.202 sigma), trajectory peak sampled ~106 GiB.
- N=24 (/tmp/r6/check_n1_24): gate 3 PASS (0.114 / 0.237 sigma), trajectory peak
  sampled 121.6 GiB (18 GiB headroom). Chosen.
- N=28: gate 3 PASS but peak 142.5 GiB of 140.4 GiB usable -> rejected as too close
  to the OOM limit (only ~1 GiB headroom).

### Full check PASS at N=24 (20:38-20:45, /tmp/r6/check_full24)
status PASS, speedup 1.8967 (candidate 1468.5 ms/step vs frozen 2785.3), timing
windows 14.88/14.49 s, peak 103.2 GiB (timing run), gate 2 NOT_BITWISE with 0 exact
failures, gate 3 0.130 / 0.286 sigma.  Current switch file = the one above with
LLAVA_MLP_STORE_LAYERS=24.

Remaining profile (/tmp/r6/profile24): 1450 ms GPU kernels, GEMM ~1126 (~75% of peak),
NCCL 95 (reduce-scatter 48.7 + all-gather 46.5, the price of persistent params),
optimizer ~50, flash 79, elementwise ~100.  Remaining recompute is the attention block
of all layers plus the MLP of the 8 un-stored layers.
1.8 hRound 6: whole-layer activation keeping on top of MLP-store
LLAVA_GC_SKIP_LAYERS=6 (skip takes precedence over the MLP-store set) + MLP-store 24.
time /tmp/r7/time_mix: 1446.5 ms/step, speedup 1.9255, peak 106.5 GiB.
check --quick PASS (gate-2 case peak sampled 125.7 GiB), check --n1 PASS
(0.052 / 0.235 sigma, trajectory peak sampled 127.8 GiB of 140.4 GiB).
Memory budget note: N=28 peaked at 139.2 GiB (rejected); this mixed config keeps
~12 GiB of headroom at its worst observed point.

### Final full check PASS with the mixed config (20:57-21:03, /tmp/r7/check_full)
status PASS, speedup 1.9245 (candidate 1446.9 ms/step vs frozen 2785.3).
Gate 2 NOT_BITWISE, 0 exact failures. Gate 3 0.052 / 0.235 sigma.
Committed 8cb2f86.
2.3 hRound 7 (21:14-21:30): where the remaining gain comes from
- LLAVA_GC_SKIP_LAYERS=8 (all skip layers inside the MLP-store set): 1387.1 ms/step,
  speedup 2.008, but timing peak 117.5 GiB -> trajectory would be ~142 GiB. Rejected.
- Choosing the skip layers *outside* the MLP-store set (LLAVA_GC_SKIP_MODE=outside)
  keeps a whole checkpointed layer instead of converting an MLP-stored one:
  skip=6 outside -> 1402.3 ms/step, 1.986, timing peak 114.1 GiB.
- preserve_rng_state=False: no measurable effect (1446.9 vs 1446.5), reverted.
- skip=6 outside, check --n1: PASS, trajectory peak 133.0 GiB (7.4 GiB headroom) -
  a bit tight.
- Trying store=23 + skip=6 outside (one less MLP-stored layer, ~4 GiB less peak).
2.7 hRound 8 (21:30-21:55): profiled the validated best, tested the remaining cheap levers
Profile of the submitted config (/tmp/r9/profile, 1436 ms GPU kernels):
GEMM 991 (69%), flash 83, NCCL 102 (reduce-scatter 50.8 + all-gather 49.4),
elementwise/copy 117 (adds 24, DeepSpeed grad copies 37, mul 15, fills 11, rest),
other 87 (optimizer multi_tensor ~65 + fill), liger triton 43, reductions 13.
GPU idle 151 ms in the profiled step, but ~52 ms of the big pre-reduce-scatter gap
has no CPU work in it (profiler-induced), and the real run's kernels sum to within
~4% of the wall time.

Tested (each with the judge's time, all reverted: inside the +-0.3% noise):
- pack text-only micro-batches after embedding them: 1447.1 ms (no change).
- preallocated padded-embedding buffer instead of 16 zeros+cat+stack: 1449.5.
- reduce_bucket_size=1e9: 1443.1.  contiguous_gradients=false: 1447.6.
- 8 dataloader workers: 1444.5.
- LLAVA_GC_SKIP_MODE=outside (keep 4 more layers): 1402.3 ms, 1.986, but dense
  (200 ms) sampling of the 60-step trajectory shows a true peak of **142.2 GiB of
  143.8** -> rejected. The submitted config's true peak, measured the same way, is
  130.2 GiB (13.3 GiB headroom).
Microbenchmarks: the model's GEMM shapes run at 640-760 TF/s, varlen flash at
155 TF/s, NCCL at ~280 GB/s - all near the practical limits of this 2-GPU box.
The remaining recompute (12.7 layer-equivalents ~ 112 ms) would need ~30 GB more
activation memory (~2.1 GiB per stored layer), which the card does not have.
3.3 hRound 9 (21:55-22:30): copy-free loss unlocks two more stored layers
- Dense-sampled (200 ms) trajectory peaks: submitted 24+skip6 = 130.2 GiB;
  store=28+skip6 = 142.0 GiB; outside-selection = 142.2 GiB. Only the submitted one
  has real headroom, and store=26 has the same composition as the 142.2 variant.
- New change: the packed loss is computed directly on the bf16 shifted-logits view
  (cross_entropy(logits[0,:-1], labels[0,1:])) instead of LlamaForCausalLM's
  logits.float() + a second fp32 contiguous copy of the shifted logits. bf16->fp32
  is exact, so the number is the same; it removes ~2.7 GiB of peak memory.
  N1 (loss-path rounding), gate 3 -0.066 / 0.353 sigma, first loss 1.3828 vs 1.3830.
  time 1440.4 ms (1.9336), full check PASS 1.931 -> committed f546e48.
- With that memory back, store=26 + skip6 + fast_loss: time **1400.8 ms/step,
  speedup 1.9883**, timing peak 111.4 GiB, dense-sampled trajectory peak 129.9 GiB
  (10.7 GiB headroom), gate 3 PASS. Full check running (/tmp/r9/check_26fl).

### Full check PASS at store=26 + fast_loss (22:25-22:31, /tmp/r9/check_26fl)
status PASS, speedup 1.9846 (1401.4 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 within 3 sigma, timing peak 111.4 GiB.
- Discrete layer-set analysis: the MLP-store set positions interact with the skip set,
  so the stored-layer counts are not monotone in the switch. store=26 is the
  optimum before the all-layers point: 24 MLP-store + 6 full-store + 2 checkpointed
  (recompute 10 layer-forwards). store=28 is worse (22 + 6 + 4, recompute 11.3);
  store=30 equals 26; store=32 is all-stored (26 + 6, recompute 8.7) but its
  dense-sampled trajectory peak is 139.6 GiB of 143.8 -> rejected.
- Kept: store=26 + skip6 + fast_loss, full check PASS 1.9846 at 10.7 GiB headroom.
- Second full check on the same state (22:37-22:43, /tmp/r9/check_final2): PASS,
  speedup 1.9945 (1400 ms/step). Reproducible (1.9846 and 1.9945).
3.8 hRound 10 (22:45-23:00): trying to shrink the per-layer activation footprint
Goal: drop the SwiGLU product h (down_proj's saved input) from the MLP-store
layers, since it is a cheap elementwise function of gate/up. Tried two ways:
1. checkpoint only the MLP tail (silu*mul + down_proj, keeping gate/up):
   microbenchmark on the real layer shape (T=13269, H=4096, I=11008):
   plain 1834 MiB vs checkpointed tail 1938 MiB -> *more* memory, not less.
2. a purpose-built autograd Function saving only (gate, up, W) and rebuilding h,
   dgate and dup in the backward: 3123 MiB -> much more (all the recompute
   temporaries are live at once).
Both reverted (tree f546e48). A full tail checkpoint would avoid h but costs more
recompute than it saves; the stored-activation set is already minimal.
State restored: switch file = store=26 + fast_loss, the validated 1.985-1.995 tree.
- The next discrete point, store=31 (25 MLP + 6 full + 1 checkpointed), was
  measured too: dense-sampled trajectory peak 139.6 GiB (same as store=32), i.e. the
  memory curve has a cliff between store=26 (129.9 GiB) and store=31. Rejected;
  store=26 remains the validated maximum.
4.2 hRound 11 (23:00-23:25): projections outside the attention checkpoint (2.089x)
Hotspot: with store=26/32 the remaining recompute was the attention block of the MLP-
stored layers. New split in _layer_forward_keep_mlp: for layers flagged
_llava_qkv_store, the q/k/v projections + RoPE run OUTSIDE the checkpoint (their
graph is kept, so the backward never recomputes those three GEMMs) and only the
flash-attention + o_proj tail is checkpointed, which stores q/k/v (~0.18 GiB/layer
at the timing fixture). Same ops in the same order: first logged loss identical,
gate 3 -0.003 / 0.249 sigma.
- store=26 + qkv(all 24 MLP layers): time 1340.2 ms, 2.0782, timing peak
  allocated 119.8 GiB.  store=32 + qkv: time 1333.4 ms, 2.0889, peak 123.0 GiB.
- Important measurement correction: the nvidia-smi "memory cliff" seen in rounds 8-10
  (several configs all reading ~139.6 GiB) is allocator *reservation*, not live
  memory. Temporary env-gated atexit instrumentation of torch.cuda.max_memory_allocated
  on the real 60-step trajectory gives the true peaks: store=26 129.9 (nvidia-smi)
  is ~117 GiB allocated; store=26+qkv 126.8 GiB; store=32+qkv 130.4 GiB; reserved
  stays ~137.6 GiB. The instrumentation was removed before committing.
- Committed a0030b0 (qkv split, no debug code). Full check running (/tmp/r11/check).

### Full check PASS at store=32 + qkv-store (23:22-23:28, /tmp/r11/check)
status PASS, speedup 2.0987 (1327 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 within 3 sigma (0.0 / 0.25 sigma on the n1 run), timing peak 123.0 GiB.
Committed a0030b0.
- Tried removing the last attention checkpoint too (LLAVA_QKV_FULL, store the
  attention output): the 60-step trajectory OOMs at 131.09 GiB allocated with
  5.78 GiB of fragmented reserved memory, i.e. the practical allocated ceiling on
  this card is ~131-133 GiB, not ~140. Consequence: the store=32+qkv config
  (measured 130.4 GiB) has only ~2-3 GiB of margin while store=26+qkv (126.8 GiB)
  has ~6 GiB, for only 0.5% speed difference. Submitting store=26 + qkv
  (time 1340.2 ms, 2.0782; full check re-running).

### Full check PASS at store=26 + qkv-store (23:36-23:42, /tmp/r12/check26q)
status PASS, speedup 2.0653 (1348.6 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 0.060 / 0.266 sigma, timing peak 119.8 GiB. This is the submitted state.
4.8 hRound 12 (23:45-00:05): removing the last recompute via the allocator
Profile of the submitted 2.065 config: 1329 ms kernels (GEMM 905, NCCL 90, flash 76,
elementwise 90, optimizer 65). The only recompute left was the flash+o_proj of the
MLP-stored layers; the attempt to store the attention output had OOM'd not because
the live set was too big but because the caching allocator could not split a 1 GiB
allocation out of 5.8 GiB of reserved-but-fragmented memory.
Fix: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (an allowed switch, not a
judge-managed variable). With it the same aggressive config passes:
- store=32 + qkv-store + attention-store (LLAVA_QKV_FULL): time **1297.9 ms/step,
  speedup 2.1459**; true 60-step peak 133.8 GiB allocated / 137.4 reserved
  (fragmentation-free, ~3.5 GiB under the usable limit); gate 3 0.076 / 0.266 sigma.
- Committed 9a1322e (attention output stored for MLP-store layers, no debug code).
  Full check running (/tmp/r13/check).
At this point no layer recomputes anything: 6 layers are fully stored and 26 keep
q/k/v + attention output + MLP activations, so the step is a pure forward+backward.

### Full check PASS at store=32 + qkv + attention-store + expandable segments
(00:06-00:13, /tmp/r13/check2): status PASS, speedup 2.1394 (1301.9 ms/step),
gate 2 NOT_BITWISE 0 exact failures, gate 3 within 3 sigma, timing peak 126.0 GiB.
Note: the first attempt failed because the rewritten switches.json was missing its
outer closing brace (JSON parse error) - fixed; always validate the file.
Committed 9a1322e.
5.3 hRound 13 (00:15-00:30): framework overhead attempts
Profile of the 2.139 config: 1274 ms kernels (GEMM 866, NCCL 88, elementwise 90,
optimizer 65, flash 62), 28 ms idle -> 98% GPU busy.
- DeepSpeed multiplies every reduced gradient by gradient_predivide_factor, which
  reduce-scatter forces to 1.0: a full 14 GB multiply by 1.0 (294 kernels, 15.4 ms).
  Patched the guard (LLAVA_SKIP_NOOP_GRAD_MUL), exact and correct (first loss
  identical), but wall time only moved 1297.9 -> 1293.9 ms (the kernels overlap
  NCCL), i.e. inside noise; reverted to keep the tree at the validated state.
- Fused AdamW (optim="adamw_torch_fused") is rejected by torch: DeepSpeed hands
  the optimizer the bf16/partitioned params before its BF16_Optimizer re-points the
  groups at the fp32 partitions. A hand-written fused Adam on the fp32 partitions is
  possible (BF16_Optimizer.step calls the client optimizer after the remap) but would
  have to reproduce DeepSpeed's optimizer state API exactly for gate 2; ~30 ms gain
  for a real risk, not taken.
- Microbenchmarks of the exact GEMM shapes confirm the GEMM path is at the box's
  practical limit (fwd/dgrad 720-780 TF/s; the wgrads run in the TN layout at
  ~858 TF/s, whereas an NN-layout matmul of the same shapes only reaches ~110 TF/s).
5.3 hRound 14 (00:30-00:45): fused AdamW on the fp32 master partitions
DeepSpeed's BF16_Optimizer re-points the client optimizer at one flat fp32 partition
per parameter group, then calls its step(), so a custom optimizer can replace
torch's ~5-pass foreach AdamW with one pass. Implemented
llava/train/fused_adam.py: a Triton kernel that reads p/g/m/v and writes p/m/v in
one pass, with torch's AdamW formula (decoupled weight decay first, beta1/beta2 bias
correction, eps inside the denominator); the optimizer keeps the standard
exp_avg/exp_avg_sq/step state (fp32) and the standard betas/eps/
weight_decay param-group keys, so DeepSpeed's safe_get_full_optimizer_state
and the judge's hyperparameter/state checks see the usual interface.
Hooked in via LLaVATrainer.get_optimizer_cls_and_kwargs (LLAVA_FUSED_ADAM=1).
- time /tmp/r15/time: 1251.9 ms/step, speedup 2.2248 (+4% over 2.139); first
  five logged losses bit-identical to the foreach baseline, later ones differ by
  ~0.3% (update rounding, N1). Timing peak unchanged at 126.0 GiB.
- Full check running (/tmp/r15/check).

### Full check PASS with fused AdamW (00:28-00:34, /tmp/r15/check)
status PASS, speedup 2.2235 (1253 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 within 3 sigma, timing peak 126.0 GiB. Committed.
- With the fused Adam the profile changes: ADAM 22 ms (was 57), but the gradient
  reduce-scatter now appears as 207 ms (68 GB/s) instead of 53 ms because it overlaps
  with saturating compute; the step still got faster (kernels 1378 ms > step 1252 ms,
  i.e. NCCL is hidden). overlap_comm: false re-tested with the new balance: 1266.6
  vs 1251.9 ms, still slower; reverted.
5.7 hRound 15 (00:45-01:00): remaining overheads measured, none adopted
Profile with the fused Adam: kernels 1378 ms in a 1252 ms step (NCCL reduce-scatter
207 ms is fully overlapped), busy union 1219 ms of a 1370 ms profiled span. The two
remaining candidate levers were measured and are hidden behind other work:
- smaller reduce bucket (reduce_bucket_size=1e8, shortens the last bucket's
  exposed tail): 1248.7 vs 1251.9 ms -> noise, reverted.
- Skipping DeepSpeed's exact no-op gradient multiply (16 ms of kernels): 1252.1 vs
  1251.9 ms -> fully overlapped now, reverted.
- overlap_comm: false re-tested with the new balance: 1266.6 ms, worse.
The remaining non-GEMM time is DeepSpeed's grad-bucket copies / fp32<->bf16 copies
/ fp64 grad-norm, all inside BF16_Optimizer.step and the ZeRO-3 reduction; they are
hidden behind the compute stream apart from the reduce-scatter tail.
6.0 hRound 16 (01:00-01:10): token-balanced sampling (measured, reverted)
With packing the step time is set by the rank holding the most tokens, so the
sampler's word-count key was replaced by the real token length (allowed: the key may
be replaced by token length, the megabatch membership still comes from randperm, so
the probe's per-step multiset is unchanged).  Verified from the loader reference:
max/mean token imbalance per step 3.3% -> 0.3% (e.g. step 2: 4591/5219 -> 4904/4906,
step 7: 3198/3789 -> 3490/3497).  But image steps carry 16 x 576 constant visual
tokens, so the balance only affects ~30% of the sequence: expected gain ~0.3%, and
the measured step time was 1262.7 vs 1251.9 ms (noise, wrong direction).  Reverted
(also avoids paying a repartition for a sub-noise gain).
6.1 hRound 17 (01:05-01:20): no-fill packed embeddings
Under packing the padded rows of the multimodal embedding batch are never read (the
pack gathers the cu_seqlens slices), so their zero-fill and the per-sample
cat(embed, zeros)+stack are waste. LLAVA_PACK_NO_FILL allocates the padded batch
once with torch.empty and copies only the real rows (right padding, guarded on
LLAVA_PACK). Numerically exact (the pad values are never consumed; first loss
identical). 11 ms of FillFunctor kernels disappear from the profile; wall time
1249.5 vs 1251.9 ms (2.2292 vs 2.2248) - small because those kernels partly
overlapped. Full check running (/tmp/r18/check).

### Full check PASS with no-fill packing (01:13-01:20, /tmp/r18/check)
status PASS, speedup 2.2245 (1252.1 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 0.107 / 0.249 sigma, peak 126.0 GiB. Committed.
6.3 hRound 18 (01:20-01:35): remaining overheads localised, none adopted
Localised the last idle: the 59 ms gap is the gradient reduce-scatter catching up
after the compute stream ends (it runs at 68 GB/s under SM contention during the
backward). Tested NCCL_PROTO=Simple: 1254.2 vs 1249.5 ms, neutral/worse.
The remaining critical-path overhead is DeepSpeed's per-step fp32<->bf16 bookkeeping:
_update_hp_grads (add_ of the bf16 grad into the fp32 buffer, 42 GB of traffic;
copy_ + skipping clear_hp_grads would be exact and save ~9 ms) and
update_lp_params (fp32 partition -> bf16 partition, then the 35 ms param
all-gather; fusing the copy into the fused-Adam kernel would save ~15 ms). Both are
sub-1% DeepSpeed-internal patches that risk the master-weight / gradient invariants
behind a validated 2.22, so they were not taken. Switch set restored to the
validated one (32335fb).
6.4 hRound 19 (01:35-01:45): fp32 master-gradient bookkeeping (measured, reverted)
BF16_Optimizer.update_hp_grads accumulates the bf16 gradient into the fp32 master
gradient with add_ although clear_hp_grads zeroes that buffer every step (so the
add is a copy that also re-reads the zeros), then zeroes the same buffer again.
Patched to copy_ (bit-identical: 0 + x == x) and dropped the re-zeroing, keeping
the has_gradients bookkeeping. Requires accumulation == 1 (fixed by the task).
Measured: 1251.6 ms (2.2254) vs 1249.5 (2.2292) - neutral, so reverted; the removed
traffic was already overlapped. Same for the earlier no-op-multiply and reduce-bucket
experiments: the step is compute-bound on the GEMM stream, not on this bookkeeping.
6.6 hRound 20 (01:45-02:00): final end-to-end confirmation
Fourth full check of the submitted tree/switch set (/tmp/r21/check): status PASS,
speedup 2.2253 (1251.6 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3
0.060 / 0.249 sigma, timing peak 126.0 GiB.
All four full checks of this code path: 2.2235, 2.2257, 2.2245, 2.2253 -> the
submission is stable at ~2.225x with no variance concerns.
6.6 hRound 21 (02:00-02:10): why the GEMM is 15% below the isolated rate
Microbenchmark of the exact fwd shape (T=13269, 4096->11008):
- one hot weight reused:            753 TF/s
- 32 distinct weights cycled (2.9 GB, like the model's 32 layers): 688 TF/s
- interleaved with an RMSNorm-style elementwise: 426 TF/s (does extra work too)
So the model's ~650-690 TF/s is cold-weight streaming from HBM, not a scheduling or
workspace deficiency (memory pressure was already ruled out: 752 -> 749 TF/s with
120 GiB allocated). The weights must be read once per GEMM; there is no layout or
prefetch lever available from PyTorch/cuBLAS here. This closes the GEMM question:
864 ms/step of GEMM is the floor for this op sequence on this box.
6.6 hRound 22 (02:10-02:20): compliance audit of the submitted tree
Checked the whole delta against the rules rather than the clock:
- No reads of LLAVA_BENCH_*, of the bench/researchboost paths, or of any judge
  artefact anywhere in the tree (grep over all .py).
- switches.json uses only LLAVA_* switches (11) plus PYTORCH_CUDA_ALLOC_CONF;
  none of the judge-managed variables (TMPDIR/TORCHINDUCTOR_CACHE_DIR/TRITON_CACHE_DIR/
  CUDA_CACHE_PATH/CUDA_VISIBLE_DEVICES/PYTHONPATH/NVIDIA_TF32_OVERRIDE/
  CUBLAS_WORKSPACE_CONFIG/HF_HOME/MASTER_PORT/MASTER_ADDR/...) is touched.
- No math-mode flag is changed anywhere (no TF32/reduced-precision overrides), so the
  measured tier is N1 ("different but equivalent kernels"), which is within
  bench.json: agent_max_tier = "N2"; the check reports PASS / verdict equivalent.
- Data path untouched (modality_lengths sampler unmodified) so the probe reports the
  candidate's partition into micro-batches is the reference's.
- Delta vs c121f04: 10 commits, 5 files (llava_llama.py, llava_arch.py, fused_adam.py,
  llava_trainer.py, scripts/zero3.json); zero3.json differs only in the ZeRO-3
  persistence parameters (persistent params, max live/reuse distance, no prefetch).
- Gate 2 exact failures: [] (none); gate 3 within 0.06-0.35 sigma on four runs.
6.6 hRound 23 (02:20-02:30): timing stability
Fifth independent time run of the submitted tree: 1252.5 ms/step, speedup 2.2238,
window spread 0.86%, peak 126.0 GiB.
Collected measurements of the submitted code path: time 2.2208 / 2.2238 / 2.2248 /
2.2254 / 2.2292; full checks 2.2235 / 2.2245 / 2.2253 / 2.2257. Mean ~2.225x with a
total spread of 0.4% - the score is stable and the four full checks all PASS.
6.8 hRound 24 (02:30-02:45): fused master-weight sync (attempted, reverted)
Last item on the compute stream: BF16_Optimizer.update_lp_params copies each rank's
fp32 partition into its bf16 slice (a standalone ~21 GB pass) before the bf16
all-gather. Implemented a version where the fused Adam kernel writes that bf16 slice
in the same pass (extra store, HAS_OUT constexpr) and update_lp_params keeps only
the all-gather, with the fp32->bf16 mapping installed in a patched BF16_Optimizer.step.
Two time runs both died with a DataLoader worker SIGSEGV at the first fetch (the
module-level deepspeed.comm import runs before the workers fork); the validated
config re-ran immediately at 2.2251 (1251.8 ms), so the patch was the cause.
Reverted: the expected gain was ~15 ms (1.2%) and a fork-time crash is not worth
debugging against a validated 2.225x.

### Follow-up on the round-24 crash (why it is not environmental)
Host memory is not the cause: 2014 GiB total, 1938 GiB available, no swap, no OOM in
dmesg/kern.log. With the patch absent the same config has now run 10+ times without a
worker crash, so the early deepspeed.comm import (or the patched optimizer step)
disturbs the forked dataloader workers. Not retried: the gain was ~1.2% and the
failure mode is a crash, which is unacceptable against a validated 2.225x.
6.8 hRound 25 (02:45-03:00): NCCL channel count (measured, reverted)
Hypothesis: the ~60 ms exposed reduce-scatter backlog exists because NCCL sustains
only 68 GB/s while the GEMMs saturate the SMs. Tested NCCL_MAX_NCHANNELS=32 +
NCCL_MIN_NCHANNELS=16 (allowed env, not judge-managed): 1249.7 ms (2.2288) vs
1251.8 (2.2251), window spread 0.6% -> inside noise, so the extra channels do not
recover the backlog either. Reverted to the validated switch set.
6.9 hRound 26 (03:00-03:20): fused linear cross-entropy (measured)
Replaced the materialised lm_head logits + cross_entropy with Liger's
LigerFusedLinearCrossEntropyFunction (chunked: loss and both gradients in one
kernel, no 0.85 GiB logits tensor). Speed: 1252.7 ms (2.2234) vs 1251.8 (2.2251),
neutral. Peak memory: 126.04 -> 124.43 GiB (-1.6 GiB), which is worth having because
the 60-step trajectory peak (133.8 GiB) is the tightest constraint in the submission.
Numerics: N1 - first loss 1.3830 vs 1.3828, later steps ~0.1-0.2% apart (chunked
vocabulary summation order). Full check running (/tmp/r25/check) to confirm gate 3.
### Full check PASS with FLCE (02:03-02:09, /tmp/r25/check)
status PASS, speedup 2.2284 (1249.9 ms/step), gate 2 NOT_BITWISE 0 exact failures,
gate 3 0.067 / 0.279 sigma, peak memory 124.43 GiB (-1.6 GiB). Committed.
- Measured FLCE's effect on the real 60-step trajectory (temporary atexit
  instrumentation of torch.cuda.max_memory_allocated, removed again afterwards):
  peak 133.81 -> 132.37 GiB allocated (reserved 137.2). The submission's tightest
  constraint now has ~5 GiB of margin instead of ~3.5 GiB.
- Instrumentation removed; tree clean at 8a54d41 with the 12-switch set above.
7.2 hRound 27 (03:20-03:35): profile of the FLCE version
GPU kernel time 1235 ms/step (was 1378 before FLCE): the fused loss path costs 0.5 ms
in 7 kernels, so the materialised logits and the separate cross-entropy are gone.
Breakdown: GEMM 866, ELEM 113, NCCL 95 (overlapped; the reduce-scatter tail is much
smaller now that the optimizer/loss no longer serialise behind it), FLASH 70,
TRITON 37, ADAM 22, grad-norm 13. The GPU is now ~99% busy across the step
(1235 ms of kernels in a 1249.9 ms step), i.e. there is no idle left to reclaim;
the remaining time is the 866 ms of GEMM plus the irreducible kernel mix.
7.2 hRound 28 (03:35-03:45): BLAS dispatch lever unavailable
Checked whether the dominant 866 ms of GEMM could use cuBLASLt's heuristics instead of
cuBLAS: torch.backends.cuda.preferred_blas_library does not exist in the installed
torch 2.1.2 (added in 2.2), and calling cuBLASLt directly from a custom autograd
Function would change both the numerics and the whole linear path for an unproven
gain. Lever closed. The weight layout (NT for the forward) is already cuBLAS's
preferred one; the wgrads already run in TN at ~858 TF/s.
7.4 hRound 29 (03:45-04:00): final validation of the FLCE version
Second full check of 8a54d41 with the 12-switch set (/tmp/r27/check): status PASS,
speedup 2.2237 (1252.6 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3
0.010 / 0.276 sigma, tier N1, peak 124.43 GiB. Together with /tmp/r25/check (2.2284)
the submitted version is validated at ~2.226x.
7.4 hRound 30 (04:00-04:10): elementwise bucket audited, nothing removable
The 113 ms ELEM bucket breaks down as: 37.2 ms of copy_/add_ (352 kernels, the
ZeRO-3 grad-bucket copies and the fp32<->bf16 master copies), 28.5 ms of bf16
elementwise (472 kernels), 24.3 ms vectorised elementwise (383), 8.3 ms generic
elementwise (246), 6.8 ms CatArrayBatchedCopy (79, the pack gathers). The bf16 ops
move ~100 MB in 0.06 ms each = ~3.3 TB/s, i.e. they are already at achievable
bandwidth; the rest are framework copies that earlier rounds proved are overlapped
(removing them left the wall time unchanged). Nothing in this bucket is removable
without touching DeepSpeed's gradient path for less than the measurement noise.
Combined with ~99% GPU occupancy, 866 ms of GEMM at the cold-weight limit, 70 ms of
flash attention and zero recompute, the implementation is at the floor for this op
sequence on this hardware.
7.5 hRound 31 (04:10-04:35): CLIP vision attention -> SDPA
The profile timeline showed the frozen vision tower spans ~80 ms with only 17.9 ms of
kernels (126 kernels), because transformers 4.37's CLIP has no SDPA path: each of the
24 layers materialises a (12, 577, 577) score matrix per image, softmaxes it and
multiplies again. Patched CLIPAttention.forward (LLAVA_CLIP_SDPA) to use
F.scaled_dot_product_attention, which is valid because the tower's attention is
bidirectional and unmasked; falls back to the original when a mask or
output_attentions is requested. N1 (frozen features differ ~1e-4 relative; first
loss 1.3834 vs 1.3830).
time: 1248.5 ms, 2.2308 (vs 1251.8 / 2.2251), peak unchanged at 124.43 GiB.
Full check running (/tmp/r28/check).
- Full check of the CLIP-SDPA variant: PASS but 2.2227 (1253.1 ms) while time gave
  2.2308 (1248.5) - the two bracket the baseline, i.e. neutral. The vision tower's
  kernels are not on the critical path (its 80 ms span is CPU-bound and largely a
  profiler artifact), so removing the eager-attention score matrices saves kernel
  time but not wall time, and it perturbs the frozen tower's outputs. Reverted;
  switch set restored to the validated 12.
7.7 hRound 32 (04:35-05:00): cached vision features (abandoned mid-implementation)
The rules explicitly allow precomputing anything independent of trainable parameters
("cached image features"), and the frozen vision tower costs ~18 ms of kernels plus
some CPU-bound idle per step. Wired a cache through encode_images /
prepare_inputs_labels_for_multimodal / the collator and started a precompute
callback that would run the tower once over the dataset in the same micro-batch
shape (so the cached values would be bit-identical). The patch set did not apply
cleanly (heredoc error plus two anchors that had been reverted earlier), and with the
remaining working budget I could not both finish it and validate it end to end, so
the whole change was reverted to keep the tree in its validated state. This is the
one identified optimisation with a >1% expected gain (the prompt's allowed feature
cache); it needs a clean implementation pass plus a full check.
↑ LLaVA