DeepSeek V4.1 Flash-1017 2.22×
Harness: DeepSeek's own harness · switches: none
- Judged
- 2.22×
- Tier
- N1
- Self-best
- 2.23×
- Rounds
- 32
- Hours
- 7.7
- API cost
- $2.58
- Input tokens · 99.6% cached
- 249 M
- Output tokens
- 0.6 M
- Requests
- 696
§ 02Key levers
- Sequence packing
- ✓ round 4
LLAVA_PACK, 1.28 → 1.77× - Recompute / activations
- ✓ stepped from "save MLP only" to no recompute at all (adding qkv / attention saves + expandable segments)
- Optimizer
- ✓ fused AdamW on fp32 master partitions
- Communication & libraries
- all params resident; ZeRO-3
- Other main levers
- installed Liger fused kernels (RMSNorm / SwiGLU / RoPE) into
/opt/candidate-env; fused linear + cross-entropy; tried caching CLIP features, reverted halfway
§ ARaw trace
Raw work log33 sections
0.2 hRound 1 (start 19:08, GPU lock 19:09)
### Profile of the unchanged tree (/tmp/r1/profile,profile_tree.md) GPU kernel sum 2707 ms/step of 2785 ms wall (union busy 2409 ms, 192 ms idle). Breakdown: - GEMM 1380 ms (50.5%): sm90 xmma bf16 (cuBLAS) + a few cutlass kernels. - elementwise/copy 809 ms (29.6%): RoPErotate_halfcats (1276 cat kernels), RMSNorm chains, SwiGLU, residual adds, and DeepSpeed ZeRO-3 gather copies. - NCCL 333 ms (12.2%): 758 AllGather calls 259 ms + 293 ReduceScatter 71 ms. - flash-attn fwd+bwd 79 ms (2.9%); reductions 30 ms. - CPU: 112aten::nonzerocalls/step (791 ms of CPU time incl. device waits) from transformers 4.37_get_unpad_data, recomputed in every layer and again under gradient checkpointing recompute.CheckpointFunction56 calls,aten::item165 calls. - GEMMs are already near peak; the win is in non-GEMM work + recompute + ZeRO traffic. ### Changes tested 1. ZeRO-2 (scripts/zero3.jsonstage 2):/tmp/r1/time2684.7 ms/step, speedup 1.0375, BUT gate-1 first-loss check ERROR: first logged loss 1.3758 vs 1.3839 (0.585% > 0.5%). Forward numerics change with the ZeRO stage (different GEMM/linear path + memory footprint). Rejected: cannot pass gate 1. Logged as a failed round (no commit). 2. ZeRO-3 with all params persistent (stage3_param_persistence_threshold=1e9, prefetch bucket 1e9):/tmp/r1/time_persist2752.9 ms/step, speedup 1.0118, status ok, first logged loss 1.3839 (identical). Keeps the stage-3 forward path, drops the per-layer all-gather. N0 (no math change). ### In flight 3. Selective activation checkpointing: keep activations of N of the 32 decoder layers instead of recomputing them (LLAVA_GC_SKIP_LAYERS, env switch) + cache_get_unpad_dataper attention-mask tensor (LLAVA_UNPAD_CACHE), together with (2). Note: first round started without a clean WORK_LOG; details above reconstructed from /tmp.
0.4 hRound 2 (19:20-19:35): fused kernels (Liger) + unpad cache
Profile after (1) (persistent params, unpad cache):/tmp/r1/prof_gc/profile_tree.mdGPU 2531 ms/step: GEMM 1325, elementwise/copy 797, NCCL 211, flash 79, reduce 30. Still 112aten::nonzero/step (64 with the full 17808-entry mask, from transformers'_get_unpad_datain every layer and again under recompute). Changes: -_get_unpad_dataresult cached per attention-mask object (exact, no math change). - Liger-Kernel 0.2.1 Triton ops (installed in /opt/candidate-env, no-deps) for Llama RMSNorm, SwiGLU (silu(gate)*up), RoPE. N1 (rounding order only). SwitchLLAVA_LIGER(default on).time /tmp/r1/time_liger: 2245.8 ms/step, speedup_vs_frozen_reference 1.2402, status ok, peak 83.58 GiB, first logged loss 1.3838 (ref 1.3839). Committed 609551e. Also implemented (switch off by default,LLAVA_MLP_STORE_LAYERS): selective activation checkpointing that checkpoints only the attention block of N layers and keeps the MLP activations (2/3 of the recompute FLOPs). Whole-layer skipping withLLAVA_GC_SKIP_LAYERSOOMs at 8 layers (each uncheckpointed layer costs ~4.6 GiB), so the finer-grained variant is the one to use. Gate check running:check --skip-timing(gates 0, 2, 3) in /tmp/r1/check_skip.
0.7 hRound 3 (19:35-19:55): selective activation checkpointing
Profile with liger (/tmp/r3/profile_liger): GPU kernels 1980 ms/step, GEMM (incl. flash) 1490, elementwise/copy 237, other 143, NCCL 95. Recompute is ~1/3 of the GEMM work. Idea: keep the MLP activations of N decoder layers (checkpoint only their attention block), which removes 2/3 of the recompute of those layers. - Whole-layer skip (LLAVA_GC_SKIP_LAYERS) OOMs at 8 layers: ~4.6 GiB/layer. - MLP-only store costs ~1.5 GiB/layer at seq 1113, ~2.7 GiB/layer at seq 2048. - Fixed 12/32 layers OOM'd because of an implementation bug (non-stored layers were left uncheckpointed); fixed by keeping the whole-layer checkpoint for them. Tiny-model test: gradients are bitwise identical between full checkpointing and MLP-store. - Adaptive count (from the batch's padded sequence length) makes DeepSpeed's ZeRO trace cache invalidate ("Invalidate trace cache ... expected module 44, got 32") and the run stalls; not usable. Fixed count keeps the graph constant. - Fixed 32 layers OOMs (133.5 GiB allocated); at seq 2048 the safe fixed count is ~14. Current experiment: fixed 14 layers (/tmp/r3/time_mlp14), liger on.
0.7 hRound 3 result
- N=14:/tmp/r3/time_mlp142113.1 ms/step, speedup 1.3181, peak 120.8 GiB allocated. ~9.5 ms saved per stored layer (recompute total ~456 ms). - N=8:/tmp/r3/time_mlp82169.8 ms/step, speedup 1.2837, peak 104.8 GiB. Chosen for the submission: the trajectory (60 steps, other data order) contains batches up to the 2048-token truncation; at seq 2048 a stored layer costs ~4.9 GiB, so N=8 peaks at ~128 GiB, while N=14 would OOM there. - Gate 3 validation running:check --n1-> /tmp/r3/check_n1. - Adaptive per-step N (LLAVA_MLP_STORE_LAYERS=auto) stalls: ZeRO-3 trace cache is invalidated whenever the checkpoint graph changes. Default is now 0 (off).
1.0 hRound 4 (19:56-20:10): padding-free packing
Profile of the n=8 config (/tmp/r4/profile): 1905 ms GPU kernels, GEMM 1328, elementwise 233, NCCL 95, other 89, liger triton 63, flash 82. 25% of the padded rows of a micro-batch are padding (16 x 1113 rows = 17808, real tokens 13269) and every GEMM processed them. - Tried dropping the flash padding mask instead (LLAVA_SKIP_PAD_ATTN): 2227 ms, slower (the full-causal attention on padded rows costs more than the copies saved). - Implemented packing: concatenate the real tokens of the micro-batch, run the decoder on the packed sequence, flash-attn varlen with cu_seqlens, per-sample position ids, first label of each sample masked so the causal shift cannot cross a sample boundary. The loss stays the micro-batch token mean (upstream's normalization). Caught a real bug with it: Liger's RoPE ignores position_ids and uses cos rows 0..seq-1, which is only right for arange positions; the wrapper now indexes cos/sin by position_ids (HF semantics), bitwise identical for the non-packed path. -time /tmp/r5/time_pack2: 1572.2 ms/step, speedup 1.7716, status ok, first logged loss 1.383 (ref 1.3839), peak 83.6 GiB. Committed cbbc84f. - Gate 3 validation running:check --n1-> /tmp/r5/check_n1. ### Full check PASS (20:16-20:24, /tmp/r5/check_full) Gates 0,1,2,3 all pass. speedup 1.804 (candidate 1544.0 ms/step, frozen reference 2785.3), windows 15.66/15.22 s, window ratio to frozen 0.556/0.552, peak 88.6 GiB, first logged loss 1.383. Gate 2 NOT_BITWISE with 0 exact failures (29 diagnostic per-tensor overruns, mostly Adam exp_avg_sq of near-zero-gradient norm weights). Gate 3: canonical -0.098 sigma, probe -0.210 sigma. Switch file at this point: {"LLAVA_UNPAD_CACHE":"1","LLAVA_LIGER":"1", "LLAVA_MLP_STORE_LAYERS":"12","LLAVA_PACK":"1"}.
1.3 hRound 5: push the selective checkpointing further
- N=16 (/tmp/r6/time_pack16): 1522.5 ms/step, speedup 1.8295, peak 93.7 GiB.check --n1PASS (0.046 / 0.202 sigma), trajectory peak sampled ~106 GiB. - N=24 (/tmp/r6/check_n1_24): gate 3 PASS (0.114 / 0.237 sigma), trajectory peak sampled 121.6 GiB (18 GiB headroom). Chosen. - N=28: gate 3 PASS but peak 142.5 GiB of 140.4 GiB usable -> rejected as too close to the OOM limit (only ~1 GiB headroom). ### Full check PASS at N=24 (20:38-20:45, /tmp/r6/check_full24) status PASS, speedup 1.8967 (candidate 1468.5 ms/step vs frozen 2785.3), timing windows 14.88/14.49 s, peak 103.2 GiB (timing run), gate 2 NOT_BITWISE with 0 exact failures, gate 3 0.130 / 0.286 sigma. Current switch file = the one above with LLAVA_MLP_STORE_LAYERS=24. Remaining profile (/tmp/r6/profile24): 1450 ms GPU kernels, GEMM ~1126 (~75% of peak), NCCL 95 (reduce-scatter 48.7 + all-gather 46.5, the price of persistent params), optimizer ~50, flash 79, elementwise ~100. Remaining recompute is the attention block of all layers plus the MLP of the 8 un-stored layers.
1.8 hRound 6: whole-layer activation keeping on top of MLP-store
LLAVA_GC_SKIP_LAYERS=6(skip takes precedence over the MLP-store set) + MLP-store 24.time /tmp/r7/time_mix: 1446.5 ms/step, speedup 1.9255, peak 106.5 GiB.check --quickPASS (gate-2 case peak sampled 125.7 GiB),check --n1PASS (0.052 / 0.235 sigma, trajectory peak sampled 127.8 GiB of 140.4 GiB). Memory budget note: N=28 peaked at 139.2 GiB (rejected); this mixed config keeps ~12 GiB of headroom at its worst observed point. ### Final full check PASS with the mixed config (20:57-21:03, /tmp/r7/check_full) status PASS, speedup 1.9245 (candidate 1446.9 ms/step vs frozen 2785.3). Gate 2 NOT_BITWISE, 0 exact failures. Gate 3 0.052 / 0.235 sigma. Committed 8cb2f86.
2.3 hRound 7 (21:14-21:30): where the remaining gain comes from
-LLAVA_GC_SKIP_LAYERS=8(all skip layers inside the MLP-store set): 1387.1 ms/step, speedup 2.008, but timing peak 117.5 GiB -> trajectory would be ~142 GiB. Rejected. - Choosing the skip layers *outside* the MLP-store set (LLAVA_GC_SKIP_MODE=outside) keeps a whole checkpointed layer instead of converting an MLP-stored one: skip=6 outside -> 1402.3 ms/step, 1.986, timing peak 114.1 GiB. -preserve_rng_state=False: no measurable effect (1446.9 vs 1446.5), reverted. - skip=6 outside, check --n1: PASS, trajectory peak 133.0 GiB (7.4 GiB headroom) - a bit tight. - Trying store=23 + skip=6 outside (one less MLP-stored layer, ~4 GiB less peak).
2.7 hRound 8 (21:30-21:55): profiled the validated best, tested the remaining cheap levers
Profile of the submitted config (/tmp/r9/profile, 1436 ms GPU kernels): GEMM 991 (69%), flash 83, NCCL 102 (reduce-scatter 50.8 + all-gather 49.4), elementwise/copy 117 (adds 24, DeepSpeed grad copies 37, mul 15, fills 11, rest), other 87 (optimizer multi_tensor ~65 + fill), liger triton 43, reductions 13. GPU idle 151 ms in the profiled step, but ~52 ms of the big pre-reduce-scatter gap has no CPU work in it (profiler-induced), and the real run's kernels sum to within ~4% of the wall time. Tested (each with the judge'stime, all reverted: inside the +-0.3% noise): - pack text-only micro-batches after embedding them: 1447.1 ms (no change). - preallocated padded-embedding buffer instead of 16 zeros+cat+stack: 1449.5. - reduce_bucket_size=1e9: 1443.1. contiguous_gradients=false: 1447.6. - 8 dataloader workers: 1444.5. -LLAVA_GC_SKIP_MODE=outside(keep 4 more layers): 1402.3 ms, 1.986, but dense (200 ms) sampling of the 60-step trajectory shows a true peak of **142.2 GiB of 143.8** -> rejected. The submitted config's true peak, measured the same way, is 130.2 GiB (13.3 GiB headroom). Microbenchmarks: the model's GEMM shapes run at 640-760 TF/s, varlen flash at 155 TF/s, NCCL at ~280 GB/s - all near the practical limits of this 2-GPU box. The remaining recompute (12.7 layer-equivalents ~ 112 ms) would need ~30 GB more activation memory (~2.1 GiB per stored layer), which the card does not have.
3.3 hRound 9 (21:55-22:30): copy-free loss unlocks two more stored layers
- Dense-sampled (200 ms) trajectory peaks: submitted 24+skip6 = 130.2 GiB; store=28+skip6 = 142.0 GiB; outside-selection = 142.2 GiB. Only the submitted one has real headroom, and store=26 has the same composition as the 142.2 variant. - New change: the packed loss is computed directly on the bf16 shifted-logits view (cross_entropy(logits[0,:-1], labels[0,1:])) instead of LlamaForCausalLM'slogits.float()+ a second fp32 contiguous copy of the shifted logits. bf16->fp32 is exact, so the number is the same; it removes ~2.7 GiB of peak memory. N1 (loss-path rounding), gate 3 -0.066 / 0.353 sigma, first loss 1.3828 vs 1.3830.time1440.4 ms (1.9336), full check PASS 1.931 -> committed f546e48. - With that memory back,store=26 + skip6 + fast_loss:time**1400.8 ms/step, speedup 1.9883**, timing peak 111.4 GiB, dense-sampled trajectory peak 129.9 GiB (10.7 GiB headroom), gate 3 PASS. Full check running (/tmp/r9/check_26fl). ### Full check PASS at store=26 + fast_loss (22:25-22:31, /tmp/r9/check_26fl) status PASS, speedup 1.9846 (1401.4 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 within 3 sigma, timing peak 111.4 GiB. - Discrete layer-set analysis: the MLP-store set positions interact with the skip set, so the stored-layer counts are not monotone in the switch.store=26is the optimum before the all-layers point: 24 MLP-store + 6 full-store + 2 checkpointed (recompute 10 layer-forwards).store=28is worse (22 + 6 + 4, recompute 11.3);store=30equals 26;store=32is all-stored (26 + 6, recompute 8.7) but its dense-sampled trajectory peak is 139.6 GiB of 143.8 -> rejected. - Kept:store=26 + skip6 + fast_loss, full check PASS 1.9846 at 10.7 GiB headroom. - Second full check on the same state (22:37-22:43, /tmp/r9/check_final2): PASS, speedup 1.9945 (1400 ms/step). Reproducible (1.9846 and 1.9945).
3.8 hRound 10 (22:45-23:00): trying to shrink the per-layer activation footprint
Goal: drop the SwiGLU producth(down_proj's saved input) from the MLP-store layers, since it is a cheap elementwise function of gate/up. Tried two ways: 1. checkpoint only the MLP tail (silu*mul + down_proj, keeping gate/up): microbenchmark on the real layer shape (T=13269, H=4096, I=11008): plain 1834 MiB vs checkpointed tail 1938 MiB -> *more* memory, not less. 2. a purpose-built autograd Function saving only (gate, up, W) and rebuilding h, dgate and dup in the backward: 3123 MiB -> much more (all the recompute temporaries are live at once). Both reverted (tree f546e48). A full tail checkpoint would avoid h but costs more recompute than it saves; the stored-activation set is already minimal. State restored: switch file = store=26 + fast_loss, the validated 1.985-1.995 tree. - The next discrete point,store=31(25 MLP + 6 full + 1 checkpointed), was measured too: dense-sampled trajectory peak 139.6 GiB (same as store=32), i.e. the memory curve has a cliff between store=26 (129.9 GiB) and store=31. Rejected; store=26 remains the validated maximum.
4.2 hRound 11 (23:00-23:25): projections outside the attention checkpoint (2.089x)
Hotspot: with store=26/32 the remaining recompute was the attention block of the MLP- stored layers. New split in_layer_forward_keep_mlp: for layers flagged_llava_qkv_store, the q/k/v projections + RoPE run OUTSIDE the checkpoint (their graph is kept, so the backward never recomputes those three GEMMs) and only the flash-attention + o_proj tail is checkpointed, which stores q/k/v (~0.18 GiB/layer at the timing fixture). Same ops in the same order: first logged loss identical, gate 3 -0.003 / 0.249 sigma. - store=26 + qkv(all 24 MLP layers): time 1340.2 ms, 2.0782, timing peak allocated 119.8 GiB. store=32 + qkv: time 1333.4 ms, 2.0889, peak 123.0 GiB. - Important measurement correction: the nvidia-smi "memory cliff" seen in rounds 8-10 (several configs all reading ~139.6 GiB) is allocator *reservation*, not live memory. Temporary env-gated atexit instrumentation of torch.cuda.max_memory_allocated on the real 60-step trajectory gives the true peaks: store=26 129.9 (nvidia-smi) is ~117 GiB allocated; store=26+qkv 126.8 GiB; store=32+qkv 130.4 GiB; reserved stays ~137.6 GiB. The instrumentation was removed before committing. - Committed a0030b0 (qkv split, no debug code). Full check running (/tmp/r11/check). ### Full check PASS at store=32 + qkv-store (23:22-23:28, /tmp/r11/check) status PASS, speedup 2.0987 (1327 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 within 3 sigma (0.0 / 0.25 sigma on the n1 run), timing peak 123.0 GiB. Committed a0030b0. - Tried removing the last attention checkpoint too (LLAVA_QKV_FULL, store the attention output): the 60-step trajectory OOMs at 131.09 GiB allocated with 5.78 GiB of fragmented reserved memory, i.e. the practical allocated ceiling on this card is ~131-133 GiB, not ~140. Consequence: the store=32+qkv config (measured 130.4 GiB) has only ~2-3 GiB of margin while store=26+qkv (126.8 GiB) has ~6 GiB, for only 0.5% speed difference. Submitting store=26 + qkv (time1340.2 ms, 2.0782; full check re-running). ### Full check PASS at store=26 + qkv-store (23:36-23:42, /tmp/r12/check26q) status PASS, speedup 2.0653 (1348.6 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 0.060 / 0.266 sigma, timing peak 119.8 GiB. This is the submitted state.
4.8 hRound 12 (23:45-00:05): removing the last recompute via the allocator
Profile of the submitted 2.065 config: 1329 ms kernels (GEMM 905, NCCL 90, flash 76, elementwise 90, optimizer 65). The only recompute left was the flash+o_proj of the MLP-stored layers; the attempt to store the attention output had OOM'd not because the live set was too big but because the caching allocator could not split a 1 GiB allocation out of 5.8 GiB of reserved-but-fragmented memory. Fix:PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True(an allowed switch, not a judge-managed variable). With it the same aggressive config passes: - store=32 + qkv-store + attention-store (LLAVA_QKV_FULL): time **1297.9 ms/step, speedup 2.1459**; true 60-step peak 133.8 GiB allocated / 137.4 reserved (fragmentation-free, ~3.5 GiB under the usable limit); gate 3 0.076 / 0.266 sigma. - Committed 9a1322e (attention output stored for MLP-store layers, no debug code). Full check running (/tmp/r13/check). At this point no layer recomputes anything: 6 layers are fully stored and 26 keep q/k/v + attention output + MLP activations, so the step is a pure forward+backward. ### Full check PASS at store=32 + qkv + attention-store + expandable segments (00:06-00:13, /tmp/r13/check2): status PASS, speedup 2.1394 (1301.9 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 within 3 sigma, timing peak 126.0 GiB. Note: the first attempt failed because the rewritten switches.json was missing its outer closing brace (JSON parse error) - fixed; always validate the file. Committed 9a1322e.
5.3 hRound 13 (00:15-00:30): framework overhead attempts
Profile of the 2.139 config: 1274 ms kernels (GEMM 866, NCCL 88, elementwise 90, optimizer 65, flash 62), 28 ms idle -> 98% GPU busy. - DeepSpeed multiplies every reduced gradient bygradient_predivide_factor, which reduce-scatter forces to 1.0: a full 14 GB multiply by 1.0 (294 kernels, 15.4 ms). Patched the guard (LLAVA_SKIP_NOOP_GRAD_MUL), exact and correct (first loss identical), but wall time only moved 1297.9 -> 1293.9 ms (the kernels overlap NCCL), i.e. inside noise; reverted to keep the tree at the validated state. - Fused AdamW (optim="adamw_torch_fused") is rejected by torch: DeepSpeed hands the optimizer the bf16/partitioned params before its BF16_Optimizer re-points the groups at the fp32 partitions. A hand-written fused Adam on the fp32 partitions is possible (BF16_Optimizer.step calls the client optimizer after the remap) but would have to reproduce DeepSpeed's optimizer state API exactly for gate 2; ~30 ms gain for a real risk, not taken. - Microbenchmarks of the exact GEMM shapes confirm the GEMM path is at the box's practical limit (fwd/dgrad 720-780 TF/s; the wgrads run in the TN layout at ~858 TF/s, whereas an NN-layout matmul of the same shapes only reaches ~110 TF/s).
5.3 hRound 14 (00:30-00:45): fused AdamW on the fp32 master partitions
DeepSpeed's BF16_Optimizer re-points the client optimizer at one flat fp32 partition per parameter group, then calls its step(), so a custom optimizer can replace torch's ~5-pass foreach AdamW with one pass. Implementedllava/train/fused_adam.py: a Triton kernel that reads p/g/m/v and writes p/m/v in one pass, with torch's AdamW formula (decoupled weight decay first, beta1/beta2 bias correction, eps inside the denominator); the optimizer keeps the standardexp_avg/exp_avg_sq/stepstate (fp32) and the standardbetas/eps/weight_decayparam-group keys, so DeepSpeed'ssafe_get_full_optimizer_stateand the judge's hyperparameter/state checks see the usual interface. Hooked in viaLLaVATrainer.get_optimizer_cls_and_kwargs(LLAVA_FUSED_ADAM=1). -time /tmp/r15/time: 1251.9 ms/step, speedup 2.2248 (+4% over 2.139); first five logged losses bit-identical to the foreach baseline, later ones differ by ~0.3% (update rounding, N1). Timing peak unchanged at 126.0 GiB. - Full check running (/tmp/r15/check). ### Full check PASS with fused AdamW (00:28-00:34, /tmp/r15/check) status PASS, speedup 2.2235 (1253 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 within 3 sigma, timing peak 126.0 GiB. Committed. - With the fused Adam the profile changes: ADAM 22 ms (was 57), but the gradient reduce-scatter now appears as 207 ms (68 GB/s) instead of 53 ms because it overlaps with saturating compute; the step still got faster (kernels 1378 ms > step 1252 ms, i.e. NCCL is hidden).overlap_comm: falsere-tested with the new balance: 1266.6 vs 1251.9 ms, still slower; reverted.
5.7 hRound 15 (00:45-01:00): remaining overheads measured, none adopted
Profile with the fused Adam: kernels 1378 ms in a 1252 ms step (NCCL reduce-scatter 207 ms is fully overlapped), busy union 1219 ms of a 1370 ms profiled span. The two remaining candidate levers were measured and are hidden behind other work: - smaller reduce bucket (reduce_bucket_size=1e8, shortens the last bucket's exposed tail): 1248.7 vs 1251.9 ms -> noise, reverted. - Skipping DeepSpeed's exact no-op gradient multiply (16 ms of kernels): 1252.1 vs 1251.9 ms -> fully overlapped now, reverted. -overlap_comm: falsere-tested with the new balance: 1266.6 ms, worse. The remaining non-GEMM time is DeepSpeed's grad-bucket copies / fp32<->bf16 copies / fp64 grad-norm, all inside BF16_Optimizer.step and the ZeRO-3 reduction; they are hidden behind the compute stream apart from the reduce-scatter tail.
6.0 hRound 16 (01:00-01:10): token-balanced sampling (measured, reverted)
With packing the step time is set by the rank holding the most tokens, so the sampler's word-count key was replaced by the real token length (allowed: the key may be replaced by token length, the megabatch membership still comes from randperm, so the probe's per-step multiset is unchanged). Verified from the loader reference: max/mean token imbalance per step 3.3% -> 0.3% (e.g. step 2: 4591/5219 -> 4904/4906, step 7: 3198/3789 -> 3490/3497). But image steps carry 16 x 576 constant visual tokens, so the balance only affects ~30% of the sequence: expected gain ~0.3%, and the measured step time was 1262.7 vs 1251.9 ms (noise, wrong direction). Reverted (also avoids paying a repartition for a sub-noise gain).
6.1 hRound 17 (01:05-01:20): no-fill packed embeddings
Under packing the padded rows of the multimodal embedding batch are never read (the pack gathers the cu_seqlens slices), so their zero-fill and the per-sample cat(embed, zeros)+stack are waste.LLAVA_PACK_NO_FILLallocates the padded batch once withtorch.emptyand copies only the real rows (right padding, guarded on LLAVA_PACK). Numerically exact (the pad values are never consumed; first loss identical). 11 ms of FillFunctor kernels disappear from the profile; wall time 1249.5 vs 1251.9 ms (2.2292 vs 2.2248) - small because those kernels partly overlapped. Full check running (/tmp/r18/check). ### Full check PASS with no-fill packing (01:13-01:20, /tmp/r18/check) status PASS, speedup 2.2245 (1252.1 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 0.107 / 0.249 sigma, peak 126.0 GiB. Committed.
6.3 hRound 18 (01:20-01:35): remaining overheads localised, none adopted
Localised the last idle: the 59 ms gap is the gradient reduce-scatter catching up after the compute stream ends (it runs at 68 GB/s under SM contention during the backward). TestedNCCL_PROTO=Simple: 1254.2 vs 1249.5 ms, neutral/worse. The remaining critical-path overhead is DeepSpeed's per-step fp32<->bf16 bookkeeping:_update_hp_grads(add_ of the bf16 grad into the fp32 buffer, 42 GB of traffic;copy_+ skippingclear_hp_gradswould be exact and save ~9 ms) andupdate_lp_params(fp32 partition -> bf16 partition, then the 35 ms param all-gather; fusing the copy into the fused-Adam kernel would save ~15 ms). Both are sub-1% DeepSpeed-internal patches that risk the master-weight / gradient invariants behind a validated 2.22, so they were not taken. Switch set restored to the validated one (32335fb).
6.4 hRound 19 (01:35-01:45): fp32 master-gradient bookkeeping (measured, reverted)
BF16_Optimizer.update_hp_gradsaccumulates the bf16 gradient into the fp32 master gradient withadd_althoughclear_hp_gradszeroes that buffer every step (so the add is a copy that also re-reads the zeros), then zeroes the same buffer again. Patched tocopy_(bit-identical: 0 + x == x) and dropped the re-zeroing, keeping the has_gradients bookkeeping. Requires accumulation == 1 (fixed by the task). Measured: 1251.6 ms (2.2254) vs 1249.5 (2.2292) - neutral, so reverted; the removed traffic was already overlapped. Same for the earlier no-op-multiply and reduce-bucket experiments: the step is compute-bound on the GEMM stream, not on this bookkeeping.
6.6 hRound 20 (01:45-02:00): final end-to-end confirmation
Fourth full check of the submitted tree/switch set (/tmp/r21/check): status PASS, speedup 2.2253 (1251.6 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 0.060 / 0.249 sigma, timing peak 126.0 GiB. All four full checks of this code path: 2.2235, 2.2257, 2.2245, 2.2253 -> the submission is stable at ~2.225x with no variance concerns.
6.6 hRound 21 (02:00-02:10): why the GEMM is 15% below the isolated rate
Microbenchmark of the exact fwd shape (T=13269, 4096->11008): - one hot weight reused: 753 TF/s - 32 distinct weights cycled (2.9 GB, like the model's 32 layers): 688 TF/s - interleaved with an RMSNorm-style elementwise: 426 TF/s (does extra work too) So the model's ~650-690 TF/s is cold-weight streaming from HBM, not a scheduling or workspace deficiency (memory pressure was already ruled out: 752 -> 749 TF/s with 120 GiB allocated). The weights must be read once per GEMM; there is no layout or prefetch lever available from PyTorch/cuBLAS here. This closes the GEMM question: 864 ms/step of GEMM is the floor for this op sequence on this box.
6.6 hRound 22 (02:10-02:20): compliance audit of the submitted tree
Checked the whole delta against the rules rather than the clock: - No reads ofLLAVA_BENCH_*, of the bench/researchboost paths, or of any judge artefact anywhere in the tree (grepover all .py). -switches.jsonuses onlyLLAVA_*switches (11) plusPYTORCH_CUDA_ALLOC_CONF; none of the judge-managed variables (TMPDIR/TORCHINDUCTOR_CACHE_DIR/TRITON_CACHE_DIR/ CUDA_CACHE_PATH/CUDA_VISIBLE_DEVICES/PYTHONPATH/NVIDIA_TF32_OVERRIDE/ CUBLAS_WORKSPACE_CONFIG/HF_HOME/MASTER_PORT/MASTER_ADDR/...) is touched. - No math-mode flag is changed anywhere (no TF32/reduced-precision overrides), so the measured tier is N1 ("different but equivalent kernels"), which is withinbench.json: agent_max_tier = "N2"; the check reports PASS / verdict equivalent. - Data path untouched (modality_lengthssampler unmodified) so the probe reports the candidate's partition into micro-batches is the reference's. - Delta vs c121f04: 10 commits, 5 files (llava_llama.py, llava_arch.py, fused_adam.py, llava_trainer.py, scripts/zero3.json); zero3.json differs only in the ZeRO-3 persistence parameters (persistent params, max live/reuse distance, no prefetch). - Gate 2 exact failures: [] (none); gate 3 within 0.06-0.35 sigma on four runs.
6.6 hRound 23 (02:20-02:30): timing stability
Fifth independenttimerun of the submitted tree: 1252.5 ms/step, speedup 2.2238, window spread 0.86%, peak 126.0 GiB. Collected measurements of the submitted code path:time2.2208 / 2.2238 / 2.2248 / 2.2254 / 2.2292; full checks 2.2235 / 2.2245 / 2.2253 / 2.2257. Mean ~2.225x with a total spread of 0.4% - the score is stable and the four full checks all PASS.
6.8 hRound 24 (02:30-02:45): fused master-weight sync (attempted, reverted)
Last item on the compute stream:BF16_Optimizer.update_lp_paramscopies each rank's fp32 partition into its bf16 slice (a standalone ~21 GB pass) before the bf16 all-gather. Implemented a version where the fused Adam kernel writes that bf16 slice in the same pass (extra store,HAS_OUTconstexpr) and update_lp_params keeps only the all-gather, with the fp32->bf16 mapping installed in a patched BF16_Optimizer.step. Twotimeruns both died with a DataLoader worker SIGSEGV at the first fetch (the module-leveldeepspeed.commimport runs before the workers fork); the validated config re-ran immediately at 2.2251 (1251.8 ms), so the patch was the cause. Reverted: the expected gain was ~15 ms (1.2%) and a fork-time crash is not worth debugging against a validated 2.225x. ### Follow-up on the round-24 crash (why it is not environmental) Host memory is not the cause: 2014 GiB total, 1938 GiB available, no swap, no OOM in dmesg/kern.log. With the patch absent the same config has now run 10+ times without a worker crash, so the earlydeepspeed.commimport (or the patched optimizer step) disturbs the forked dataloader workers. Not retried: the gain was ~1.2% and the failure mode is a crash, which is unacceptable against a validated 2.225x.
6.8 hRound 25 (02:45-03:00): NCCL channel count (measured, reverted)
Hypothesis: the ~60 ms exposed reduce-scatter backlog exists because NCCL sustains only 68 GB/s while the GEMMs saturate the SMs. Tested NCCL_MAX_NCHANNELS=32 + NCCL_MIN_NCHANNELS=16 (allowed env, not judge-managed): 1249.7 ms (2.2288) vs 1251.8 (2.2251), window spread 0.6% -> inside noise, so the extra channels do not recover the backlog either. Reverted to the validated switch set.
6.9 hRound 26 (03:00-03:20): fused linear cross-entropy (measured)
Replaced the materialised lm_head logits +cross_entropywith Liger'sLigerFusedLinearCrossEntropyFunction(chunked: loss and both gradients in one kernel, no 0.85 GiB logits tensor). Speed: 1252.7 ms (2.2234) vs 1251.8 (2.2251), neutral. Peak memory: 126.04 -> 124.43 GiB (-1.6 GiB), which is worth having because the 60-step trajectory peak (133.8 GiB) is the tightest constraint in the submission. Numerics: N1 - first loss 1.3830 vs 1.3828, later steps ~0.1-0.2% apart (chunked vocabulary summation order). Full check running (/tmp/r25/check) to confirm gate 3. ### Full check PASS with FLCE (02:03-02:09, /tmp/r25/check) status PASS, speedup 2.2284 (1249.9 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 0.067 / 0.279 sigma, peak memory 124.43 GiB (-1.6 GiB). Committed. - Measured FLCE's effect on the real 60-step trajectory (temporary atexit instrumentation of torch.cuda.max_memory_allocated, removed again afterwards): peak 133.81 -> 132.37 GiB allocated (reserved 137.2). The submission's tightest constraint now has ~5 GiB of margin instead of ~3.5 GiB. - Instrumentation removed; tree clean at 8a54d41 with the 12-switch set above.
7.2 hRound 27 (03:20-03:35): profile of the FLCE version
GPU kernel time 1235 ms/step (was 1378 before FLCE): the fused loss path costs 0.5 ms in 7 kernels, so the materialised logits and the separate cross-entropy are gone. Breakdown: GEMM 866, ELEM 113, NCCL 95 (overlapped; the reduce-scatter tail is much smaller now that the optimizer/loss no longer serialise behind it), FLASH 70, TRITON 37, ADAM 22, grad-norm 13. The GPU is now ~99% busy across the step (1235 ms of kernels in a 1249.9 ms step), i.e. there is no idle left to reclaim; the remaining time is the 866 ms of GEMM plus the irreducible kernel mix.
7.2 hRound 28 (03:35-03:45): BLAS dispatch lever unavailable
Checked whether the dominant 866 ms of GEMM could use cuBLASLt's heuristics instead of
cuBLAS: torch.backends.cuda.preferred_blas_library does not exist in the installed
torch 2.1.2 (added in 2.2), and calling cuBLASLt directly from a custom autograd
Function would change both the numerics and the whole linear path for an unproven
gain. Lever closed. The weight layout (NT for the forward) is already cuBLAS's
preferred one; the wgrads already run in TN at ~858 TF/s.7.4 hRound 29 (03:45-04:00): final validation of the FLCE version
Second full check of 8a54d41 with the 12-switch set (/tmp/r27/check): status PASS, speedup 2.2237 (1252.6 ms/step), gate 2 NOT_BITWISE 0 exact failures, gate 3 0.010 / 0.276 sigma, tier N1, peak 124.43 GiB. Together with /tmp/r25/check (2.2284) the submitted version is validated at ~2.226x.
7.4 hRound 30 (04:00-04:10): elementwise bucket audited, nothing removable
The 113 ms ELEM bucket breaks down as: 37.2 ms ofcopy_/add_(352 kernels, the ZeRO-3 grad-bucket copies and the fp32<->bf16 master copies), 28.5 ms of bf16 elementwise (472 kernels), 24.3 ms vectorised elementwise (383), 8.3 ms generic elementwise (246), 6.8 ms CatArrayBatchedCopy (79, the pack gathers). The bf16 ops move ~100 MB in 0.06 ms each = ~3.3 TB/s, i.e. they are already at achievable bandwidth; the rest are framework copies that earlier rounds proved are overlapped (removing them left the wall time unchanged). Nothing in this bucket is removable without touching DeepSpeed's gradient path for less than the measurement noise. Combined with ~99% GPU occupancy, 866 ms of GEMM at the cold-weight limit, 70 ms of flash attention and zero recompute, the implementation is at the floor for this op sequence on this hardware.
7.5 hRound 31 (04:10-04:35): CLIP vision attention -> SDPA
The profile timeline showed the frozen vision tower spans ~80 ms with only 17.9 ms of kernels (126 kernels), because transformers 4.37's CLIP has no SDPA path: each of the 24 layers materialises a (12, 577, 577) score matrix per image, softmaxes it and multiplies again. PatchedCLIPAttention.forward(LLAVA_CLIP_SDPA) to use F.scaled_dot_product_attention, which is valid because the tower's attention is bidirectional and unmasked; falls back to the original when a mask or output_attentions is requested. N1 (frozen features differ ~1e-4 relative; first loss 1.3834 vs 1.3830). time: 1248.5 ms, 2.2308 (vs 1251.8 / 2.2251), peak unchanged at 124.43 GiB. Full check running (/tmp/r28/check). - Full check of the CLIP-SDPA variant: PASS but 2.2227 (1253.1 ms) whiletimegave 2.2308 (1248.5) - the two bracket the baseline, i.e. neutral. The vision tower's kernels are not on the critical path (its 80 ms span is CPU-bound and largely a profiler artifact), so removing the eager-attention score matrices saves kernel time but not wall time, and it perturbs the frozen tower's outputs. Reverted; switch set restored to the validated 12.
7.7 hRound 32 (04:35-05:00): cached vision features (abandoned mid-implementation)
The rules explicitly allow precomputing anything independent of trainable parameters
("cached image features"), and the frozen vision tower costs ~18 ms of kernels plus
some CPU-bound idle per step. Wired a cache through encode_images /
prepare_inputs_labels_for_multimodal / the collator and started a precompute
callback that would run the tower once over the dataset in the same micro-batch
shape (so the cached values would be bit-identical). The patch set did not apply
cleanly (heredoc error plus two anchors that had been reverted earlier), and with the
remaining working budget I could not both finish it and validate it end to end, so
the whole change was reverted to keep the tree in its validated state. This is the
one identified optimisation with a >1% expected gain (the prompt's allowed feature
cache); it needs a clean implementation pass plus a full check.