Claude Opus 5.5 2.92×
Harness: Claude Code · switches: none
- Judged
- 2.92×
- Tier
- N1
- Self-best
- 2.77×
- Rounds
- 37
- Hours
- 10.0
- API cost
- $90.4
- Input tokens · 99.5% cached
- 287 M
- Output tokens
- 1.1 M
- Requests
- 621
§ 02Key levers
- Sequence packing
- ✓ round 1
LLAVA_PACK, 1.46× from this step alone - Recompute / activations
- ✓ activations kept by memory budget (budget 52 → 60 GiB)
- Optimizer
- ✓ chunked Triton AdamW, kernel also writes the bf16 param copy
- Communication & libraries
- ✓ uses ZeRO-2; cuBLAS 12.6 (12.9's down-projection is actually slower); FA3 for packed attention (−11.8 ms)
- Other main levers
- LM head only at target positions; last layer computes only the target suffix (−12 ms); gradients switched to bidirectional send/recv rotating partition reduce (−21 ms), grad buffer written in partition order (−8.7 ms), no host sync between optimizer step and next forward (−4.1 ms); ends at ~947 ms/step, both GPUs at the ~700 W power cap
§ ARaw trace
Raw work log37 sections
0.2 hRound 1 (2026-11-14 06:34 -> )
Profile (unchanged reference,/tmp/r1/profile, 2 steps, rank 0): GPU kernel time 2875 ms/step, wall 2903 ms/step. - GEMM (cuBLAS sm90 xmma) 1488 ms/step (52%) - elementwise / copy / cat / gather kernels 875 ms/step (30%): RMSNorm (fp32 upcast, pow, mean, rsqrt, mul), RoPE (mul, cat in rotate_half, add), SiLU*mul, residual adds, their backward, flash-attn unpad/pad (gather/scatter/index_put, ~93 ms), DeepSpeed ZeRO-3 grad norm / partition / optimizer copies - NCCL AllGather 232 ms + ReduceScatter 87 ms (side streams), flash-attn 84 ms, optimizer multi_tensor 65 ms - main-stream idle 240 ms/step - Kind: GEMMs are compute-bound but a third of all token work is padding: the micro-batch is padded to its longest sample, and every linear / norm / elementwise op runs on B x max_len rows. Over the 25 timed steps the valid/padded token ratio is 0.67 (image steps 0.55-0.79, text steps 0.43-0.49), see /tmp/ana/lengths.py. Change: padding-free (packed) forward, switchLLAVA_PACK=1(llava/model/packing.py, dispatched from LlavaLlamaForCausalLM.forward). The collator output is unchanged; inside forward the 16 samples are concatenated into one [1, T] sequence of valid tokens (same per-sample embeddings, image features spliced in, truncation at 2048), attention per sample through flash_attn_varlen_func (cu_seqlens), RoPE positions restart per sample, loss = mean CE over the same (position -> next token) pairs. One embedding call for all text tokens (instead of one per sample). Numerics: N1 (different GEMM M / reduction length). Single-GPU test (/tmp/ana/packtest.py, real model, real batches): loss rel diff 1e-7 (image batch) / 0 (text batch); layer-0 gradient rel-L2 difference pack-vs-ref 5e-3 vs the reference's own run-to-run difference 6e-3 (flash-attn atomic backward), deep layers 3e-5. time:/tmp/r1/time(config{"LLAVA_PACK": "1"}): 1906.3 ms/step, speedup_vs_frozen_reference 1.4567, windows 19.353 / 18.773 s (spread 0.004), logged-loss max rel diff 0.0018 (step 1: 0.0), peak 62.8 GiB (ref 73.5). Full check/tmp/r1/check/check.json: PASS, speedup 1.4553 (1908.3 ms/step), tier N1; gate 2 exact items all pass (1057 of 4724 tensors bitwise); gate 3 d = -6.9e-5 (-0.02 sigma) train_loss, -1.6e-4 (-0.06 sigma) probe. Committed af8007a. /submission/switches.json = {"environment": {"LLAVA_PACK": "1"}}. Round end 07:06. Workflow from here: experiments live in the git worktree /tmp/wt (branchwip); /submission/llava (branchopt) only receives changes that passedtime(+ check).
0.7 hRound 2 (2026-11-14 07:05 -> 07:13, full check running after)
Profile (packed,/tmp/r2/profile, rank 0, 2 steps): GPU kernel time 2117 ms/step (2.12 s/step under the profiler). - GEMM 1144 ms (54%); elementwise/copy 544 ms (26%); NCCL AllGather 193 ms + ReduceScatter 45 ms (side streams); flash-attn 89 ms; optimizer multi_tensor 65 ms; main-stream idle 188 ms (one ~45 ms gap per step at the step boundary + ~970 short launch gaps). - elementwise by source: RMSNorm fwd (x2 with GC recompute) + bwd ~150 ms, RoPE (mul, rotate_half cat/neg, add) fwd + bwd ~120 ms, SiLU*up fwd + bwd ~45 ms, ZeRO-3 grad-norm/partition/optimizer copies ~70 ms. - Kind: memory-bound chains (class "memory"), existing ops unfused (kind 1/3). Change: fused Triton kernels (llava/model/fused_ops.py), switches LLAVA_FUSED_SWIGLU / LLAVA_FUSED_ROPE / LLAVA_FUSED_RMSNORM. Unit test /tmp/ana/fusedtest.py (real shapes): SwiGLU fwd/bwd and RoPE fwd/bwd bitwise equal to the eager ops (same bf16 rounding points; libdevice exp, div_rn, explicit fma as torch's silu_backward); RMSNorm N1 (fwd: 61 of 53M elements differ by the mean's reduction order; bwd mirrors eager's rounding points: dx 457/53M elements differ, dw 4/4096). Kernel times at T=13k: RMSNorm fwd 0.755 -> 0.059 ms, RoPE fwd 0.789 -> 0.073 ms, SwiGLU fwd 0.470 -> 0.268 ms. time:/tmp/r2/time_fused(config /tmp/r2/fused.json = PACK + 3 fused switches, tree /tmp/wt @455c40b): 1612.6 ms/step, speedup_vs_frozen_reference 1.722 (round 1: 1.457), window spread 0.002, peak 62.3 GiB. Full check started:/tmp/r2/check(PID in /tmp/r2/check.log). Full check/tmp/r2/check/check.json: PASS, speedup 1.7231 (1611.6 ms/step), tier N1 (gate-2 status NOT_BITWISE, all exact items pass), gate 3 d = -0.009 sigma (train) / -0.093 sigma (probe). /submission/llava fast-forwarded to 455c40b; /submission/switches.json = PACK + FUSED_SWIGLU/ROPE/RMSNORM. Round end 07:19.
1.1 hRound 3 (2026-11-14 07:19 -> 07:40, full check running after)
Profile (packed + fused,/tmp/r3/profile, rank 0, 2 steps; 1.80 s/step under the profiler): - GEMM 1219 ms/step (68%): forward 313, backward thread 906 = recompute of the checkpointed forward 313 + backward 593; flash-attn 95 (fwd 18 + recompute 18 + bwd 59); elementwise now 111; optimizer multi_tensor 65; NCCL AllGather 148 + ReduceScatter 53 (side streams). - main-stream idle 182 ms/step, nearly all "CPU-late" launches (/tmp/ana/idle2.py): ZeRO-3 linear forward hooks 55 ms, ZeRO-3 get_grad_norm_direct per-parameter loop 40 ms, CLIP vision tower small ops 46 ms. - Hotspot this round: the activation recompute (class: redundant compute), ~330 ms/step incl. flash + elementwise. Change: LLAVA_GC_BUDGET_GB (packing.py): the last k decoder layers run without checkpointing, k = budget / (T_max x LLAVA_GC_BYTES_PER_TOKEN_LAYER), T_max = largest micro-batch over ranks. Recompute produces the same activations, so numerics are unchanged (N0 relative to round 2). - First attempt (per-rank k) deadlocked: ranks with different T took different k, so ZeRO-3's parameter all-gathers ran in different orders (GPUs at 100% util, 120 W, no progress; killed after 10 min). Fix: T_max via a gloo (CPU) all-reduce, no GPU sync; under no_grad every layer runs plainly. - budget 40 GiB, 136 KiB/token-layer:/tmp/r3/time_gc401424.4 ms/step (1.9496); peak allocated 97.3 GiB but reserved 132.9 GiB (fragmentation from varying T). Measured cost ~119 KiB per token and layer. - budget 52 GiB, 120 KiB/token-layer, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True:/tmp/r3/time_gc52x1328.1 ms/step, speedup 2.0909, peak allocated 113.9 GiB, reserved 123.0 GiB; windows 13.429 / 13.133 s. At T=14k all 32 layers run without recompute; the worst case T=32k gives k=13 (bounded memory). Committed 7dec83c on wip; full check/tmp/r3/check(config /tmp/r3/gc52x.json) started 07:41. Full check/tmp/r3/check/check.json: PASS, speedup 2.0882 (1329.9 ms/step), tier N1, gate 3 d = +0.003 sigma (train) / -0.011 sigma (probe). /submission/llava fast-forwarded to 7dec83c; switches.json = /tmp/r3/gc52x.json (PACK, FUSED_SWIGLU/ROPE/RMSNORM, GC_BUDGET_GB=52, GC_BYTES_PER_TOKEN_LAYER=122880, expandable_segments). Round end 07:47.
1.6 hRound 4 (2026-11-14 07:47 -> 08:11, full check running after)
Profile (current best: packed + fused + GC budget 52, ZeRO-3;/tmp/r4/profile, rank 0, 1.50 s/step under profiler): GEMM 943 ms/step (63%), NCCL AllGather 143 + ReduceScatter 55 (side streams), elementwise 106, flash 76, optimizer multi_tensor 65; main-stream idle 183 ms/step, nearly all CPU-late launches: ZeRO-3 linear forward hooks 56 ms, ZeRO-3 get_grad_norm_direct per-parameter loop 40 ms, CLIP vision tower small ops ~47 ms. Hotspot class: host/launch. **Changes and time runs (all/tmp/r4/time_*):** | config | ms/step | speedup | note | |---|---|---|---| | ZeRO-2 (scripts/zero2.json, overlap_comm on), budget 45 | 1258.5 | 2.2066 | step-1 loss -0.48%, later up to -1.8% | | + LLAVA_ZERO3_LINEAR | 1266.6 | ERROR | step-1 loss -0.535% > 0.5% band: not the cause | | + LLAVA_ROPE_ZERO_INIT_CACHE | 1268.2 | 2.1897 | step 1 -0.0001, but steps 3+ up to 1.4% off | | overlap_comm off (zero2_nooverlap.json) | 1278.7 | 2.1718 | all steps within 0.24% (like ZeRO-3) | | + LLAVA_FAST_ZERO_NORM | 1246.6 | 2.2277 | within 0.09% | | + LLAVA_FUSED_ADAM | 1217.9 | 2.2802 | rejected: losses up to 2.9% off | | budget 52 (no fused Adam) | 1210.0 | 2.2950 | peak 113.6 alloc / 114.3 reserved GiB, losses within 0.16% | Findings: - The reference's RoPE cache is built under DeepSpeed zero.Init, whose wrapped torch.arange casts LlamaRotaryEmbedding's float position vector to bf16 (positions > 256 rounded). Any non-ZeRO-3 run must rebuild that cache (LLAVA_ROPE_ZERO_INIT_CACHE; bitwise equal to a zero.Init-built cache, /tmp/ana/ropetest.py). - DeepSpeed 0.12.6 ZeRO-2 with overlap_comm=true gives wrong gradients here (loss drift 1-1.4% within 15 steps); overlap off costs ~10 ms only. - torch 2.1 fused AdamW is wrong for tensors > 2^31 elements (ZeRO-2's flat fp32 partition is 3.5e9): 2.4% of the first 2^31 elements mis-updated (/tmp/ana/adambig.py). Chunked views would avoid it (next round). Committed fff42b9 on wip; full check/tmp/r4/check(config /tmp/r4/z2_52.json) started 08:11. Full check/tmp/r4/check/check.json: PASS, speedup 2.2961 (1209.4 ms/step), tier N1, gate 2 exact items pass (RNG fingerprints included) with ZeRO-2, gate 3 d = -0.079 sigma (train) / -0.097 sigma (probe). /submission/llava fast-forwarded to fff42b9; switches.json = /tmp/r4/z2_52.json. Round end 08:18.
1.9 hRound 5 (2026-11-14 08:18 -> 08:27)
Incident: round 4's scripts/zero2_nooverlap.json was not in the commit (.gitignore has *.json), so /submission's switch file pointed at a missing config (profile ERROR at 08:18). Fixed within minutes: force-added, commit 2d7ef93, /submission fast-forwarded; diff /tmp/wt vs /submission clean. Lesson: checkgit status --ignoredbefore merging. Profile (current best, ZeRO-2;/tmp/r5/profile, 1.31 s/step under profiler): GEMM 876 ms/step (67%), flash 73, optimizer multi_tensor 69, elementwise 66, NCCL AllReduce 53 (on the compute stream: overlap off) + AllGather 25, main-stream idle 96 ms (ZeRO-3 hook / CLIP gaps gone; remaining: ZeRO-2 copy_grads_in_partition waits 28 ms, step boundary). Hotspot class: DeepSpeed optimizer step + gradient reduction. Changes: - LLAVA_FUSED_ADAM=1 -> ChunkedFusedAdamW (llava/train/fused_adamw.py): torch._fused_adamw_ on views of <= 2^30 elements; bitwise equal to torch fused on small tensors, 3e-7 relative to foreach on 2.2e9 elements. time/tmp/r5/time_adam: 1202.6 ms/step (2.3092), losses within 0.15%. - LLAVA_ZERO2_OVERLAP_FIX=1 (ds_patches.py): per-bucket CUDA event on the reduction stream, waited on before the bucket is written again. With LLAVA_DS_CONFIG=zero2.json (overlap on):/tmp/r5/time_ovfix1196.2 ms/step (2.3216), logged losses within 0.17% (the race diagnosis holds: without the fix they drifted 1-1.4%). Committed bb632cf on wip (no full check yet; to be checked together with round 6). GEMM microbenchmark at T=13400 (/tmp/ana/gemmbench.py), torch 2.1.2 bundled cuBLAS 12.1 vs cuBLAS <ip> via LD_LIBRARY_PATH: forward-type GEMMs 735->812, 706->810, 733->817 TFLOPS, lm_head 640->727, weight-gradient GEMMs 678->698, 684->705, 639->684. nvidia-cublas-cu12 <ip> installed in /opt/candidate-env (inactive without LD_LIBRARY_PATH; verified via /proc/self/maps).
2.1 hRound 6 (2026-11-14 08:27 -> 08:41)
Hotspot (round-5 profile): GEMMs 876 ms/step (67%) at 640-735 TFLOPS with torch 2.1.2's bundled cuBLAS 12.1 (class: compute, library). Microbenchmark (/tmp/ana/gemmbench.py, T=13400): cuBLAS <ip> is 10-15% faster on the forward/dX GEMMs and 3-7% on the weight-gradient GEMMs; <ip> about equal to 12.6; <ip> regresses on the down projection (729 vs 817 TFLOPS). Change: nvidia-cublas-cu12==<ip> installed in /opt/candidate-env (pip, Aliyun mirror) and selected with the switch LD_LIBRARY_PATH=/opt/candidate-env/lib/python3.10/site-packages/nvidia/cublas/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 (torch finds its bundled libcublas through RUNPATH $ORIGIN, which LD_LIBRARY_PATH precedes; the two original entries do not exist and are kept for completeness; verified via /proc/self/maps). No code change. time:/tmp/r6/time_cublas1149.6 ms/step, speedup 2.4155 (round 5: 1196.2), logged losses within 0.17%. Full check (rounds 5+6, config /tmp/r6/best.json)/tmp/r6/check/check.json: PASS, speedup 2.4214 (1146.9 ms/step), tier N1, gate 3 d = -0.04 sigma (train) / -0.081 sigma (probe). /submission/llava fast-forwarded to bb632cf (trees verified identical); switches.json = /tmp/r6/best.json. Round end 08:41.
2.2 hRound 7 (2026-11-14 08:41 -> 08:48)
Profile (current best with cuBLAS 12.6;/tmp/r7/profile): GEMM 863 ms/step (now cuBLAS "nvjet" kernels; no runtime correlation in this torch profiler, so stage attribution of GEMMs is unavailable), flash 75, NCCL AllReduce 68 (side stream, overlapped), elementwise 67, optimizer multi_tensor 63, AllGather 23, Triton kernels ~50; main-stream idle 62 ms/step (22 ms CPU-late at the step boundary, 12 ms compute stream waiting on the gradient-bucket event, HF logging ~7 ms). Power: windows at ~671 W/GPU average, peaks at the 700 W cap with SM clocks down to 1.3-1.4 GHz: the run is power-bound, so removing work pays more than faster kernels (cuBLAS 12.6: +10-15% in isolation, +4% end to end). Hotspot this round: the output head computes lm_head + fp32 cross entropy for all T tokens, but only target positions (~11%) enter the loss. Change: LLAVA_LABEL_ONLY_HEAD=1 (packing.py): target positions found on the CPU (no sync), hidden states gathered with index_select, lm_head + CE on those rows only. Single-GPU test (/tmp/ana/packtest.py): losses equal to the padded reference (1.158399, 0.227106), lm_head grad rel-L2 6e-5. time:/tmp/r7/time_head1132.7 ms/step, speedup 2.4517 (round 6: 1149.6), logged losses within 0.14%. Committed on wip (not yet full-checked).
2.4 hRound 8 (2026-11-14 08:48 -> 08:55, full check running after)
Hotspot (round-7 profile): the optimizer step: torch 2.1's fused AdamW kernel (multi_tensor_apply, FusedOptimizerTensorListMetadata<4>) takes 60 ms/step for ~98 GB of traffic (~1.6 TB/s, a third of HBM bandwidth); its moment updates are evaluated in double precision (double lr / betas in adam_math). Class: memory-bound kernel. Change: LLAVA_TRITON_ADAM=1 (llava/train/fused_adamw.py): one fp32 Triton kernel per fp32 partition, in torch foreach AdamW's operation order (lerp, mul + addcmul, IEEE sqrt / bc2 + eps, addcdiv), 1-beta formed in double on the host (in fp32, 1 - 0.999 has a 1.3e-5 relative error: the first version differed from foreach by 6.8e-6, the final one by 2.6e-7, closer than torch's fused kernel at 3.3e-7). 3.4e9 elements: 23 ms (Triton) vs 56 ms (torch fused) vs 67 ms (foreach) (/tmp/ana/adamtri.py). - First full run failed: DataLoader workers segfaulted. DeepSpeed's optimizer-state initialization calls step() before the workers are forked, so the Triton kernel was compiled and launched in the main process before fork(). Fix: the first step() call (zero gradients, no-op update) uses torch's kernel; Triton first runs after the fork, like the fused_ops kernels. time:/tmp/r8/time_tadam1092.6 ms/step, speedup 2.5417 (round 7: 1132.7), logged losses within 0.11%. Committed 7f5e44a on wip; full check/tmp/r8/check(rounds 7+8, config /tmp/r8/best.json) started 08:55. Full check (rounds 7+8, config /tmp/r8/best.json)/tmp/r8/check/check.json: PASS, speedup 2.536 (1095.0 ms/step), tier N1, gate 3 d = -0.039 sigma (train) / +0.028 sigma (probe). /submission/llava fast-forwarded to 7f5e44a (trees verified identical); switches.json = /tmp/r8/best.json (round-6 set + LLAVA_LABEL_ONLY_HEAD + LLAVA_TRITON_ADAM). Round end 09:02.
2.6 hRound 9 (2026-11-14 09:01 -> 09:10): no gain
Profile (current best;/tmp/r9/profile): GEMM 846 ms/step, flash 74, NCCL AllReduce 72 (overlapped), elementwise 65, Triton AdamW 23 ms (was 60), param AllGather 21.5 ms exposed after the optimizer step (the main thread sits in DeepSpeed's ThroughputTimer cuda.synchronize while the NCCL all-gather runs; nothing on the compute stream), main-stream idle 64 ms. Hotspot: optimizer epilogue (fp32->bf16 copy + all-gather, ~27 ms sequential after Adam). Tried: - NCCL settings, isolated 2-GPU microbenchmark (/tmp/ana/nccl_bench.py, 6.7 GB/rank all-gather, 400 x 33 MB all-reduce): default 22.1 ms (304 GB/s) / 51.9 ms; NCCL_MIN_NCHANNELS=32 20.7 / 49.3; NCCL_PROTO=Simple 20.5 / 51.8; NCCL_MAX_NCHANNELS=8 54.8 / 85.1. Gains of ~1.5 ms: not pursued. - LLAVA_ZERO2_PIPELINED_STEP (patched ZeRO-2 step): Triton AdamW on 2^28-element chunks writing the bf16 copy itself, each chunk exchanged while the next is computed. Correct (logged losses within 0.15%) but slower: per-chunk broadcasts/tmp/r9/time_pipe1124.4 ms/step (one NVLink direction at a time), send/recv pairs/tmp/r9/time_pipe21112.7 ms/step, vs 1092.6 without. Reverted (code kept in /tmp/r9/*_pipelined.py). Nothing committed; submission unchanged (round-8 state, check PASS 2.536).
2.7 hRound 10 (2026-11-14 09:10 -> 09:17, full check running after)
Hotspot (round-9 profile): gradient-reduction bookkeeping and the CLIP vision tower. - reduce_bucket_size (zero2 configs with 5e7 / 2e8 instead of HF "auto" = hidden^2):/tmp/r10/time_b5e71090.1,/tmp/r10/time_b2e81092.6 ms/step vs 1092.6: no gain (configs deleted). - CLIP (frozen, no_grad, 16 images x 577 tokens x 24 layers) ~40 ms/step on the compute stream: its attention bmm's run on sm75 "cutlass_75 ... align1" kernels (12.9 ms; 577 x 577 matrices are not 16-byte aligned, so cuBLAS 12.6 skips its Hopper kernels) + bf16 softmax 4.2 ms + scale / copies. Kind: missed fast path. Change: LLAVA_CLIP_SDPA=1 (clip_encoder.py): CLIPAttention.forward -> torch scaled_dot_product_attention (flash), query pre-scaled exactly as CLIPAttention (q_proj(x) * scale in bf16), SDPA scale 1.0; falls back to the original with masks / output_attentions / dropout. Numerics: synthetic test flash vs eager 4.9e-3 rel (bf16 rounding). Real images (/tmp/ana/cliptest2.py): the reference's bf16 CLIP is itself 20.8% rel-L2 from fp32 CLIP at the selected layer (rounding is amplified through the layers); SDPA is 20.6% from fp32 (equally close), 11% from eager. LLaVA loss on 8 real micro-batches (/tmp/ana/cliploss.py): -0.25% .. +0.39%, mean +0.03% (random sign). Tier: N1 by the judge's measurement (no dtype / math-mode change); an attention-backend change in the task's words. time:/tmp/r10/time_clipsdpa1079.0 ms/step, speedup 2.5736 (round 8: 1092.6), logged losses within 0.23% (step 1: +0.03%). Committed 1447db9 on wip; full check/tmp/r10/check(config /tmp/r10/best.json) started 09:17 (this change alone on top of the checked round-8 state, so gate 3 judges it in isolation). Full check/tmp/r10/check/check.json: PASS, speedup 2.5756 (1078.2 ms/step), tier N1, gate 3 d = +0.091 sigma (train) / -0.071 sigma (probe). /submission/llava fast-forwarded to 1447db9; switches.json = /tmp/r10/best.json (round-8 set + LLAVA_CLIP_SDPA). Round end 09:24.
2.9 hRound 11 (2026-11-14 09:24 -> 09:29)
Hotspot: cross-rank imbalance: every step waits for the slower rank (ZeRO-2 gradient buckets synchronize the ranks during the backward). Upstream's split balances word counts; over the timed steps the slower rank carries 1.25% more tokens than the mean (/tmp/ana/lengths.py), step 9 (text) 15%. Change: LLAVA_TOKEN_LENGTHS=1: the grouping key is the tokenized LLM length (preprocess() + 575 per <image>, truncated at 2048, sign = modality), and split_to_even_chunks balances tokens + L^2/25000 (flash-attention term). Allowed by the task (length key and balancing target); megabatch composition (which 32 samples form a step) is unchanged. Offline (/tmp/ana/lengths2.py): timed-step imbalance 1.25% -> 0.10% (tokens), 1.32% -> 0.11% (cost). Startup tokenizes the 2560 fixture samples once (~1 min, outside the timed windows). time:/tmp/r11/time_tok1072.6 ms/step, speedup 2.5889 (round 10: 1079.0); data check ok, repartitioned (logged-loss band 48.4%, max rel diff 5.8%). With GC budget 58:/tmp/r11/time_tok581071.5 (no gain, budget stays 52). Committed on wip (full check pending, together with round 12).
3.0 hRound 12 (2026-11-14 09:29 -> 09:32, full check running after)
Hotspot (round-9 profile, CLIP part): after SDPA, CLIP still spends ~5.4 ms/step in 147 separate bias-add kernels (F.linear is DeepSpeed's zero3_linear_wrap: matmul, thenoutput += bias) and ~4 ms in quick_gelu's three elementwise kernels (mul, sigmoid, mul). Change: LLAVA_CLIP_FAST=1 (with LLAVA_CLIP_SDPA, ZeRO < 3): CLIP attention and MLP call aten's linear with the bias fused into the GEMM, quick_gelu is one Triton kernel (fp32 math). Module-path fallback for ZeRO-3 parameters or when gradients are needed. CLIP features: 20.7% rel-L2 from fp32 CLIP (eager bf16 reference: 20.8%), standalone CLIP forward 22.0 -> 19.3 ms (/tmp/ana/cliptest3.py; bias-add savings only appear with the zero3 linear wrap active). time:/tmp/r12/time_clipfast1062.9 ms/step, speedup 2.6126 (round 11: 1072.6), logged-loss max rel diff 5.9% (repartitioned band 48.4%), first step 0.26%. Committed on wip; full check/tmp/r12/check(rounds 11+12, config /tmp/r12/best.json) started 09:32. Full check (rounds 11+12)/tmp/r12/check/check.json: PASS, speedup 2.6145 (1062.2 ms/step), tier N1, gate 3 d = -0.0043 = -1.32 sigma (train_loss_last20_canonical) / +0.054 sigma (probe). The train-loss endpoint is the mean of per-micro-batch token means, so the repartition (round 11) reweights samples in it by construction; the held-out probe endpoint is unchanged. Earlier checks varied by <= 0.1 sigma run to run, so the margin to 3 sigma is ample. /submission/llava fast-forwarded to 4b4a5da (trees verified identical); switches.json = /tmp/r12/best.json (round-10 set + LLAVA_TOKEN_LENGTHS + LLAVA_CLIP_FAST). Round end 09:40.
3.2 hRound 13 (2026-11-14 09:41 -> 09:45, full check running after)
Hotspot (round-9 profile): host/communication at the step boundary. ZeRO-2's all-gather of the updated bf16 partitions (~21 ms) ran with nothing on the compute stream: DeepSpeed's ThroughputTimer synchronizes the device at step end and again at forward start, and HF's per-step loss / flos gathers (NCCL, same communicator) are queued behind the parameter all-gather, so the host blocked until it finished. Change: LLAVA_ASYNC_ALLGATHER=1: - ds_patches: stage_1_and_2.all_gather_dp_groups issues all_gather_into_tensor(async_op=True) and keeps the handles; wait_param_allgather() makes the compute stream wait (no host block). ThroughputTimer.start/stop without the device synchronize (they only feed DeepSpeed's samples/s log). - packing.build_packed_inputs: vision tower, then wait, then projector (= encode_images); non-packed forward / generate wait at entry; LLaVATrainer._save / _save_checkpoint wait. - LLaVATrainer._nested_gather (scalar loss) and store_flos gather over a CPU gloo group (same fp32 values and arithmetic as HF's NCCL path); packing's rank max shares that group. The all-gather now overlaps the logging, the next batch's transfer and the CLIP forward. time:/tmp/r13/time_async1050.5 ms/step, speedup 2.6434 (round 12: 1062.9); logged losses identical to the round-12 run at steps 1-2 and within +-0.15% after (no stale parameters). Committed 25b231f on wip; full check/tmp/r13/check(config /tmp/r13/best.json) started 09:44. Full check/tmp/r13/check/check.json: PASS, speedup 2.6337 (1054.4 ms/step), tier N1, gate 3 d = -1.367 sigma (train canonical, the round-11 repartition effect) / +0.255 sigma (probe). /submission/llava fast-forwarded to 25b231f (trees verified identical); switches.json = /tmp/r13/best.json (round-12 set + LLAVA_ASYNC_ALLGATHER). Round end 09:52.
3.4 hRound 14 (2026-11-14 09:52 -> 09:58): no gain
Profile (current best, async all-gather;/tmp/r14/profile): GEMM 814 ms/step, NCCL AllReduce 91 (side stream), flash 72, elementwise 53, Adam 23, param AllGather 22 (now overlapped); main-stream idle 57 ms, of which 17 ms are compute-stream waits on gradient buckets (11 ms in the overlap fix's bucket-reuse wait, 6 ms in DeepSpeed's), plus host syncs: loss logging 5.5 ms, batch transfer 4.1 ms, grad-norm .item() 3.5 ms, packing plan 2.7 ms. Tried: LLAVA_ZERO2_IPG_BUFFERS=4 (ring of 4 gradient buckets instead of 2) + LLAVA_LAZY_GRAD_NORM=1 (grad norm kept as a GPU tensor, no .item() in the optimizer step):/tmp/r14/time_ipg41049.6 ms/step vs 1050.5 (noise); logged losses identical at steps 1-2. Reverted (code kept in /tmp/r14/ds_patches_ipg_lazy.py).
3.5 hRound 15 (2026-11-14 09:58 -> 10:04)
Hotspot (round-14 profile): optimizer-step copies: DeepSpeed widens the flattened bf16 gradient partition to a new fp32 tensor (9.4 ms, 13.5 GB) before the optimizer and copies the updated fp32 partition back to bf16 (9.2 ms). Change: LLAVA_ZERO2_FUSED_EPILOGUE=1 (ds_patches: patched ZeRO-2 step for bf16 / no clipping / loss scale 1 / Triton Adam; otherwise DeepSpeed's step): the bf16 gradient partition goes straight into the Triton AdamW kernel, which widens it in registers (same values). - First version also wrote the bf16 parameter copy from the kernel:/tmp/r15/time_epi1068.0 ms/step (slower); microbenchmark (/tmp/ana/adamvar.py, 3.4e9 elements): Adam fp32 grad 23.0 ms, bf16 grad 21.5 ms, + bf16 store 56 ms (also with a bit-level round-to-nearest-even store, which is bitwise equal to copy_), copy_ 9.2 ms. Dropped the in-kernel store. - Final:/tmp/r15/time_epi21046.0 ms/step, speedup 2.6548 (round 13: 1050.5); logged losses identical to round 13 at steps 1-3. Committed on wip (full check pending).
3.5 hRound 16 (2026-11-14 10:04 -> 10:08): no gain (measurements only)
Hotspot candidates (round-14 profile): my own Triton kernels (RMSNorm bwd ~15 ms/step, SwiGLU bwd 17 ms) and GEMM
efficiency per shape.
- Launch configs at T=13400 (/tmp/ana/kbench.py): RMSNorm bwd 0.170 ms/call (ROWS=64, 8 warps) vs best 0.142
(ROWS=8, 16 warps): ~1.8 ms/step; SwiGLU bwd 0.396 (BLOCK 2048, 8 warps) vs best 0.385: negligible. Not pursued.
- Per-shape GEMM TFLOPS: cuBLAS 12.6's nvjet kernels carry no usable External id / runtime correlation in this
profiler, so per-shape attribution failed; totals: 886 ms of GEMM kernels for ~552 TFLOP/step (~620 TFLOPS) under
the 700 W cap (isolated GEMMs: 700-820 TFLOPS).
- Residual add + RMSNorm fusion estimated at ~3 ms/step for the attention residual only (~6 ms with a cross-layer
interface): not pursued. Per-step wall times (921-1160 ms) follow the step's token count; no outliers.
Round 15 full check /tmp/r15/check/check.json: PASS, speedup 2.6509 (1047.6 ms/step), tier N1, gate 3
d = -1.315 sigma (train canonical) / +0.152 sigma (probe). /submission/llava fast-forwarded to a131624 (trees identical);
switches.json = /tmp/r15/best.json (round-13 set + LLAVA_ZERO2_FUSED_EPILOGUE). 10:13.3.8 hRound 17 (2026-11-14 10:13 -> 10:20): no gain
Hotspot: the overlapped gradient all-reduce (91 ms of NCCL kernels/step on a side stream) competes with the backward
GEMMs for SMs and power. Tried NCCL_MAX_NCHANNELS caps (fewer NCCL CTAs), on the round-15 configuration:
8 -> 1071.6 ms/step (slower: the async parameter all-gather becomes exposed), 12 -> 1048.2, 16 -> 1043.4, 24 -> 1046.2,
default 1046.0 (/tmp/r17/time_nch*). Within noise; not adopted.3.8 hRound 18 (2026-11-14 10:20 -> 10:23, full check running after)
Hotspots (small, from rounds 14/16): RMSNorm backward launch configuration (~1.8 ms/step) and the vision tower on text-only micro-batches: the collator gives every text-only sample a zero dummy image, the reference runs CLIP on all 16 (~19 ms) although only empty slices of those features join the sequences. Change: - LLAVA_SKIP_DUMMY_IMAGES=1 (packing.build_packed_inputs): a micro-batch without any <image> token skips the vision tower; the projector runs on zeros of the tower's output shape (zero gradients, as in the reference). Mixed and image micro-batches are unchanged. - LLAVA_RMSNORM_BWD_FAST=1 (fused_ops): RMSNorm backward with 8 rows / 16 warps per program (N1: dw partial-sum order). time:/tmp/r18/time_small1039.0 ms/step, speedup 2.6729 (round 15: 1046.0); per-step deltas -45 and -35 ms on two steps (text-only), ~0 elsewhere; logged losses identical at steps 1-2. Committed on wip; full check/tmp/r18/check(config /tmp/r18/best.json) started 10:23. Round 18 full check/tmp/r18/check/check.json: PASS, speedup 2.6798 (1036.3 ms/step), tier N1, gate 3 d = -1.32 sigma (train canonical) / +0.196 sigma (probe). /submission/llava fast-forwarded to 1e36620; switches.json = /tmp/r18/best.json (round-15 set + LLAVA_SKIP_DUMMY_IMAGES + LLAVA_RMSNORM_BWD_FAST). 10:30.
4.0 hRound 19 (2026-11-14 10:30 -> 10:33, full check running after)
Hotspot: DeepSpeed's fp32 -> bf16 parameter copy after the optimizer (9.2 ms/step, ~2.2 TB/s), left in round 15 because a bf16 store inside the Triton AdamW kernel was 2.4x slower. Microbenchmark by block size (/tmp/ana/adamblk.py, 1e9 elements, bf16 gradient): the slowdown came from the 4096-element block; at 1024 elements x 8 warps Adam costs 6.2 ms without and 6.7 ms with the bf16 store (copy_ alone: 2.7 ms); Triton's native fp32->bf16 store is as fast as a bit-level RNE store and bitwise equal to copy_ (/tmp/ana/k2/adamnat.py). Change: Triton AdamW BLOCK 1024 (from 4096), and with LLAVA_ZERO2_FUSED_EPILOGUE the kernel writes the bf16 partition itself; DeepSpeed's copy_ is gone from that path. time:/tmp/r19/time_p161027.0 ms/step, speedup 2.7039 (round 18: 1039.0); logged losses identical to round 18 at steps 1-3. Committed on wip; full check/tmp/r19/check(config /tmp/r19/best.json = round-18 switch file) started 10:33. Round 19 full check/tmp/r19/check/check.json: PASS, speedup 2.6986 (1029.1 ms/step), tier N1, gate 3 d = -1.266 sigma (train canonical) / +0.031 sigma (probe). /submission/llava fast-forwarded to 1e1e438 (trees identical); switches.json unchanged (round-18 file). 10:39.
4.2 hRound 20 (2026-11-14 10:39 -> 10:46, full check running after)
Profile (current best;/tmp/r20/profile): GEMM 812 ms/step (81% of the busy time), flash 72, NCCL AllReduce 83 + AllGather 22 (overlapped), Triton kernels ~55, Adam 23, elementwise 32.5 (down from 53); main-stream idle 48 ms. Finding: per-step times in round 19 had two outliers (1095, 1118 ms). Triton 2.1 specializes integer kernel arguments on divisibility by 16; my RMSNorm backward kernel takes M = the micro-batch's token count, so the first micro-batch with M % 16 == 0 JIT-compiles a second variant during training (/tmp/ana/spectest.py: 2 variants for M = 13400 / 13408 / 13401). Kind: invalid-steady-state overhead (a compile inside timed steps). Change:@triton.jit(do_not_specialize=['M'])on the RMSNorm backward kernel (one variant). All other Triton kernels take integers that are fixed per run (SwiGLU n = T x 11008 is always divisible by 16; Adam / quick_gelu sizes are fixed). time:/tmp/r20/time_nospec1018.6 ms/step, speedup 2.7262 (round 19: 1027.0); the two outlier steps went 1095 -> 1020 and 1118 -> 1039 ms, the rest unchanged. Committed on wip; full check/tmp/r20/checkstarted 10:46. Round 20 full check/tmp/r20/check/check.json: PASS, speedup 2.7112 (1024.3 ms/step), tier N1, gate 3 d = -1.342 sigma (train canonical) / +0.08 sigma (probe). /submission/llava fast-forwarded to aeef24f (trees identical); switches.json unchanged. 10:53.
4.4 hRound 21 (2026-11-14 10:53 -> 10:56, full check running after)
Hotspot: remaining activation recompute on the long text-only steps (T ~ 15.5k: budget 52 GiB leaves ~3 of 32 layers checkpointed; image steps already run without recompute). Change (switch value only): LLAVA_GC_BUDGET_GB=60 (no recompute up to T ~ 16k). Memory stays bounded by the budget formula for any T (worst case T=32k: 16 layers kept, ~124 GiB estimated). time:/tmp/r21/time_gc601015.3 ms/step, speedup 2.7353 (round 20 config: 1018.6); the two text-only steps 1091 -> 1083 and 1117 -> 1092 ms; peak allocated 116.2 GiB (was 113.9), reserved unchanged 117.2 GiB of 139.8. Full check/tmp/r21/check(tree /submission/llava @aeef24f, config /tmp/r21/best.json) started 10:56. Round 21 full check/tmp/r21/check/check.json: PASS, speedup 2.724 (1019.4 ms/step), tier N1, gate 3 d = -1.309 sigma (train canonical) / +0.084 sigma (probe); peak 116.2 GiB allocated / 117.2 GiB reserved. /submission/switches.json = /tmp/r21/best.json (LLAVA_GC_BUDGET_GB=60); tree unchanged (aeef24f). 11:03.
4.6 hRound 22 (2026-11-14 11:03 -> 11:08): no gain
Context: power telemetry of the current best (/tmp/r21/time_gc60/power_tree.csvvs the reference's): busy-sample mean 662 W (reference 665 W), median SM clock 1500 MHz (reference 1785 MHz), 55% of busy samples at the 700 W cap (reference 28%). The step is power-bound; its energy is mostly the GEMM FLOPs, which are all required. Tried (custom kernels): LLAVA_FUSED_RESIDUAL: each residual add fused into the following RMSNorm (Triton kernels _add_rmsnorm_fwd/bwd; layers pass the pair (h, m); the backward adds the skip gradient inside the norm-backward kernel, the bf16 accumulation autograd performs). Single-GPU test with deterministic flash-attn backward (/tmp/ana/residtest.py): bitwise identical loss and gradients to the current path, with and without checkpointing./tmp/r22/time_resid1013.8 ms/step vs 1015.3 (noise). Reverted; code kept in /tmp/r22/*_resid.py.
4.6 hRound 23 (2026-11-14 11:08 -> 11:10): no gain
Hotspot hypothesis (host stragglers): Python's cyclic GC running full collections over the large object graph
(model, DeepSpeed, HF state) inside timed steps. Tried LLAVA_GC_FREEZE (gc.collect() + gc.freeze() at the first
micro-step): /tmp/r23/time_gcf 1020.2 ms/step vs 1015.3 (per-step times unchanged within noise). Reverted.4.7 hRound 24 (2026-11-14 11:10 -> 11:13): no gain
Hotspot: host time in the Trainer's input preparation (round-20 profile: 3.7 ms/step of GPU idle while the training thread converts the fp32 image batch to bf16 for the bf16 DeepSpeed engine). Tried LLAVA_BF16_IMAGES (collator casts images to bf16 in the DataLoader workers; same values, the judge's digest is defined on bf16 images) together with the bitwise-identical LLAVA_FUSED_RESIDUAL of round 22:/tmp/r24/time_bundle1022.0 ms/step vs 1015.3 (round-21 configuration): no gain. Singletimeruns vary by about +-5 ms here, so changes below ~5 ms/step cannot be resolved; reverted.
4.9 hRound 26 (2026-11-14 11:22 -> 11:32): marginal, not adopted
Tried: bundle of the value-preserving micro-changes (LLAVA_FUSED_RESIDUAL, bitwise equal; LLAVA_BF16_IMAGES, same values; LLAVA_LAZY_GRAD_NORM, no host sync in the optimizer step), measured with interleaved A/Btimeruns to get below the single-run noise: bundle 1014.7 / 1018.2 ms/step (/tmp/r26/time_B1,_B2), current submission 1018.6 / 1019.1 (/tmp/r26/time_A1,_A2): -2.4 ms/step (0.24%), consistent in sign but far inside the final evaluation's run-to-run noise. Not adopted (code kept in /tmp/r26/code/); submission unchanged. Adopted after all (the measured gain is small but consistent and the changes are value-preserving): committed 20af8bc, full check/tmp/r26/check/check.json: PASS, speedup 2.7336 (1015.9 ms/step), tier N1, gate 3 d = -1.301 sigma (train canonical) / +0.116 sigma (probe), peak 116.2 GiB. /submission/llava fast-forwarded to 20af8bc (trees identical); switches.json = /tmp/r26/best.json (round-21 set + LLAVA_FUSED_RESIDUAL, LLAVA_BF16_IMAGES, LLAVA_LAZY_GRAD_NORM). 11:38.
5.2 hRound 27 (2026-11-14 11:39 -> 11:52, full check running after)
Hotspot: CLIP self-attention: torch 2.1's SDPA takes its memory-efficient kernel (pytorch_fmha::fmha_fwd_loop_kernel, ~6.5 ms/step for 24 layers of 16 x 16 heads x 577 tokens, ~20 TFLOPS). Change: LLAVA_CLIP_FA2=1 (clip_encoder, CLIP fast path): flash-attn's flash_attn_func on the [batch, seq, heads, dim] layout (also drops the transposes). CLIP forward 18.1 -> 15.5 ms; distance to fp32 CLIP unchanged (13.2% vs 13.6%). time (interleaved A/B): FA2 1012.1 / 1012.4 ms/step (/tmp/r27/time_B*) vs current 1017.3 / 1017.3 (/tmp/r27/time_A*): -5.0 ms/step. Committed on wip; full check/tmp/r27/check(config /tmp/r27/best.json) started 11:52. Round 27 full check/tmp/r27/check/check.json: PASS, speedup 2.7481 (1010.5 ms/step), tier N1, gate 3 d = -1.385 sigma (train canonical) / +0.121 sigma (probe). /submission/llava fast-forwarded to fdf6e6e; switches.json = /tmp/r27/best.json (round-26 set + LLAVA_CLIP_FA2). 11:54.
5.5 hRound 28 (2026-11-14 11:54 -> 12:04): no gain
Hotspot: device copies of DeepSpeed's ZeRO-2 gradient reduction (each bucket flattened into a new tensor, all-reduced,
copied back): ~20 GB/step on the reduction stream.
Tried LLAVA_ZERO2_INPLACE_ALLREDUCE (all-reduce the contiguous bucket region of the gradient buffer in place; same
used values): interleaved A/B 1013.9 / 1012.6 ms/step vs current 1013.8 / 1012.9 (/tmp/r28/time_*): no difference;
the reduction stream is not on the critical path. Reverted (code in /tmp/r28/ds_patches_inplace.py).5.7 hRound 29 (2026-11-14 12:04 -> 12:13): no gain
Hotspot: with contiguous_gradients, DeepSpeed copies every parameter gradient into the bucket buffer on the compute stream (~13.5 GB/step). Tried a ZeRO-2 config with contiguous_gradients false (overlap on; flattening then happens at reduction time on the reduction stream). First attempt crashed in my overlap-fix patch (ipg_index only exists with contiguous gradients); with a guard: correct (logged losses equal to the current configuration within run-to-run noise), peak 110.0 GiB instead of 116.2, but slower: 1024.0 / 1027.2 ms/step vs 1009.9 / 1012.5 (/tmp/r29/time_*). Reverted. ## Rounds 29-30 wrap-up (2026-11-14 12:13 -> 12:21) Second confirmation check of exactly /submission/llava @fdf6e6e + /submission/switches.json (27 entries):/tmp/r30/check_final/check.jsonPASS, speedup 2.7454 (1011.5 ms/step, windows 10.212 / 10.019 s), tier N1, gate 3 d = -1.351 sigma (train canonical) / +0.251 sigma (probe), peak 116.2 GiB. Round 27's check: 2.7481. Considered and not pursued (each ~1 ms or less, or self-defeating): CLIP's unused last encoder layer (~0.7 ms, exact); CLIP residual/LayerNorm fusion; projector bias fusion; prefetching the next batch's CLIP into the optimizer phase (CLIP currently hides the asynchronous parameter all-gather, which would be exposed again).
6.0 hRound 31 (2026-11-14 12:21 -> 12:32, full check running after)
Hotspot (dead computation): with the LM head on target positions only, the last decoder layer's outputs at all other positions (system prompt, the 576 image tokens, the question) never reach the loss, but the layer computed queries, o_proj, residual, norm and MLP for all T positions. Change: LLAVA_LAST_LAYER_SUFFIX=1 (packing): a per-sample "suffix" from the first target position (CPU plan: query indices, cu_seqlens_q, targets in suffix coordinates); the last layer computes keys/values for all positions and everything else only on the suffix; flash_attn_varlen_func with separate q/k cu_seqlens, causal (bottom-right aligned: the query at absolute position p sees keys 0..p, as before); fused_ops.rope1 (single-tensor RoPE). Requires the fused-residual / fused-RoPE / label-only-head path and an uncheckpointed last layer (else the full path). Single-GPU test, deterministic flash-attn backward (/tmp/ana/suffixtest.py): losses bitwise equal (1.158843, 0.227017), lm_head grads bitwise equal, last-layer weight grads rel-L2 2-5e-5. time (interleaved A/B): suffix 1001.6 / 1000.1 ms/step vs current 1014.5 / 1011.3 (/tmp/r31/time_*): -12 ms/step (1.2%), peak 114.8 GiB (was 116.2). Committed on wip; full check/tmp/r31/checkstarted 12:32. Round 31 full check/tmp/r31/check/check.json: PASS, speedup 2.7696 (1002.7 ms/step), tier N1, gate 3 d = -1.333 sigma (train canonical) / +0.139 sigma (probe), peak 114.8 GiB. /submission/llava fast-forwarded to 599ec3e; switches.json = /tmp/r31/best.json (round-27 set + LLAVA_LAST_LAYER_SUFFIX). 12:39.
6.2 hRound 32 (2026-11-14 12:39 -> 12:47): no gain
Tried LLAVA_LAST_LAYER_TARGETS (extension of round 31: after the last layer's attention, o_proj / residual / norm /
MLP / final norm only on the target rows instead of the whole suffix). Single-GPU test: losses bitwise equal, grads at
rounding level. Interleaved A/B 1002.4 / 998.9 ms/step vs current 1000.3 / 1000.8 (/tmp/r32/time_*): no difference
(the non-target rows inside the suffixes are few). Reverted (code in /tmp/r32/packing_targets.py).6.8 hRound 34 (2026-11-14 12:57 -> 13:29)
Profile (current best, round 31;/tmp/r34/profile): main-stream busy 968 ms/step, idle 51 ms: **~25 ms of compute- stream waits for gradient buckets** (the overlap fix's bucket-reuse wait 16.9 ms + 7.7 ms more in DeepSpeed's hook), host syncs ~10 ms (packing plan 4.7, loss logging 4.2). In situ each 33 MB bucket all-reduce takes ~0.33 ms (~100 GB/s) vs 0.13 ms isolated: the NCCL kernels start late behind the GEMMs' thread blocks, and the compute stream then waits for them. Kind: scheduling (communication priority). Change: LLAVA_NCCL_HIGH_PRIORITY=1 (train_mem.py): init_process_group / new_group get ProcessGroupNCCL.Options(is_high_priority_stream=True) for NCCL groups (DeepSpeed's data-parallel group is a clone of the world group made with new_group). Scheduling only. **time (interleaved A/B,/tmp/r34/time_*):** high priority 994.6 / 997.9 vs current 999.9 / 998.8 ms/step; with 8 gradient buckets 996.4 / 995.7 vs 1005.5 / 998.2; with DeepSpeed's reduction stream also at high priority 996.4 / 996.0 vs 998.7 / 1015.5. All six high-priority runs 994.6-997.9 (mean 996.2) vs the current configuration's 998.2-1015.5 (median 999.3): ~-3 ms/step and fewer slow runs. Only the NCCL priority is kept (the other two added nothing; code in /tmp/r34/ds_patches_ipg_redprio.py). Committed 039ea2e on wip; full check/tmp/r34/check(13:22 -> 13:29): PASS 2.789 (995.7 ms/step), tier N1, gate 3 at -1.353 sigma (train canonical) / +0.138 sigma (probe). /submission/llava fast-forwarded to 039ea2e, switches = /tmp/r34/best.json.
7.4 hRound 35 (2026-11-14 13:34 -> 13:59)
Profile (current best, round 34;/tmp/r35/profile, 1 step): main stream idle ~59 ms/step, of which ~26 ms are compute-stream waits for gradient buckets and ~8 ms more around fused_ops / ZeRO linear backward; the gradient all-reduce kernels take 98-126 ms/step on the NCCL stream (13.5 GB/step at ~110-137 GB/s while the GEMMs run), and the reduction stream spends another 16.6 ms/step on DeepSpeed's bucket div_(2) (10.1 ms) and flatten / copy-back memcpys (6.4 ms). Cause: DeepSpeed 0.12.6 all-reduces every bucket in full (allreduce_bucket sets rank=None) although each rank keeps only its own partition's half. Kind: communication volume. Change: LLAVA_ZERO2_P2P_REDUCE=1 + LLAVA_DS_CONFIG=zero2_rr.json (round_robin_gradients). Per bucket (1e8 elements) each rank sends the slices the other rank owns straight from the gradient tensors (one NCCL batch_isend_irecv) and adds what it receives into its own slices with one Triton kernel, own = bf16(bf16(own*0.5) + bf16(recv*0.5)), i.e. DeepSpeed's div_(2) + NCCL bf16 sum bit for bit. With round robin, consecutive parameters belong to alternating ranks, so a bucket's transfer runs in both NVLink directions (isolated micro-benchmark/tmp/ana/p2p_bench.py: 0.165 ms per 90 MB vs 0.318 ms all-reduce; one direction 0.256 ms). DeepSpeed's hp-param linkage (safe_get_full_*) walked the group in its original order, wrong under round robin: fixed to the flat order. Harness/tmp/r35/t(2 ranks, ZeRO-2, LLaMA-like parameter list, 3 steps): gradients via safe_get_full_grad, fp32 master weights, Adam states, bf16 params and losses bitwise identical to the stock path, with and without round robin. **time (interleaved A/B,/tmp/r35/time_*):** P2P + round robin 976.9 / 980.8 ms/step (2.8428 / 2.8313) vs current 997.9 / 1001.0 (2.7828 / 2.7741): -21 ms/step. Peak memory max over ranks 115.0 GB (was 114.75). Full check/tmp/r35/check(13:52 -> 13:59): PASS 2.8347 (979.7 ms/step), tier N1, gate 3 at -1.375 sigma (train canonical) / +0.216 sigma (probe). Committed 5ce801f; /submission/llava fast-forwarded, switches = /tmp/r35/best.json.
7.6 hRound 36 (2026-11-14 14:00 -> 14:25)
Profile (round-35 profile/tmp/r35/profile, step boundary; round-34 two-step profile for the next step's start): after the backward, ~4 ms of gradient-norm kernels (~200 foreach_norm chunks; the norm is only reported, clipping is off), a 4 ms flatten (memcpy) of the 6.8 GB gradient partition into partition order, then AdamW 23.6 ms, then the bf16 all-gather 21.4 ms on the NCCL stream while the next step's compute stream had only ~7 ms of work (CLIP) before the projector waits for it: ~14 ms exposed. During the backward the reduction stream also copies every own gradient into DeepSpeed's grads_in_partition (backward order). Kind: optimizer epilogue (memory copies + serialized communication). Change A — LLAVA_ZERO2_PART_GRAD=1 (commit 1096945): the P2P half-sum kernel writes each own slice's reduced gradient straight into a persistent bf16 buffer per group laid out like the partition (zero padding); averaged_gradients are views with get_flat_partition's extents (safe_get_full_grad unchanged); no grads_in_partition copy, no flatten, the backward's end is a stream wait instead of a host sync; the gradient norm comes from per-block sums in the AdamW kernel (matches an exact fp64 norm to 1e-8; DeepSpeed's foreach path was off by 5e-4). Harness/tmp/r35/t: all gradients, master weights, Adam states, bf16 params and losses bitwise identical to the stock path. time (interleaved A/B,/tmp/r36/time_{A,B}*): 975.0 / 971.0 ms/step (2.8483 / 2.8599) vs current 982.4 / 980.9 (2.8267 / 2.8312): -8.7 ms/step. Change B — LLAVA_ZERO2_PIPE_ALLGATHER=1 (commit 5d8079e): AdamW on 2^28-element chunks, each chunk's bf16 copy exchanged with the other rank (send + recv, both NVLink directions) while the next chunk is computed, instead of the all-gather after the optimizer. Harness: bitwise identical (incl. bf16 params after the exchange). time (pipe on vs off, same tree, interleaved,/tmp/r36/time_{P,Q}*): pipelined 977.3 / 974.2 ms/step vs 972.4 / 970.8 without: +4 ms, slower (the AdamW chunks slow down under the concurrent NVLink traffic, and what the next step waits for at its start is the host: HF's loss logging (.item() after the optimizer step) and the synchronous H2D copies of the next batch block the CPU until AdamW ends, so the all-gather was not the exposed part). Commit dropped from the branch (code in /tmp/r36/stage). Full check/tmp/r36/checkof change A (1096945, switches /tmp/r36/best.json) started 14:17. Result PASS 2.8606 (970.8 ms/step), tier N1, gate 3 at -1.335 sigma (train canonical) / +0.079 sigma (probe). /submission/llava fast-forwarded to 1096945, switches = /tmp/r36/best.json (31 entries). Round end 14:25.
8.1 hRound 37 (2026-11-14 14:25 -> 14:54)
Profile (current best, 2 steps;/tmp/r37/profile): at the step boundary the host blocks until AdamW has finished: HF's NaN filtertorch.isnan(tr_loss_step)reads a GPU value queued behind the backward and DeepSpeed's optimizer step (accelerate runs engine.step() inside training_step), the logging .cpu() of tr_loss, accelerate's blocking H2D copy of the next batch, the packing plan's .cpu() of ids / mask / labels; DeepSpeed's engine.backward also syncs (self.losses += loss.mean().item()) between forward and backward (~2.8 ms GPU gap). After AdamW the bf16 parameter all-gather took 25 ms while the next step's compute stream had only ~8 ms (CLIP) to do before the projector waits for it. Isolated (/tmp/ana/ag_bench.py): NCCL all-gather 21.4 ms vs P2P exchange 18.4 ms for the 6.7 GB partition, so the exchange has to overlap AdamW to disappear, which only pays once the host no longer waits for AdamW. Kind: host synchronization (latency). Change A — LLAVA_HOST_ASYNC=1 (commit 402b536): training_step returns HF's value as a host tensor copied right after the forward (side stream + event); the logged loss comes from a host mirror of tr_loss (HF's rules, fp32); batches stay pinned on the host (DataLoaderShard.device = None after accelerate's prepare) and _prepare_input copies them without blocking, registering the host tensors for the packing plan; DeepSpeed's backward accumulates engine.losses on the device instead of .item(). Logged losses of steps 1-3 identical (1.3802, 1.2878, 1.3478). time (interleaved A/B,/tmp/r37/time_{A,B}*): 967.2 / 967.3 ms/step (2.8712 / 2.8708) vs current 971.4 / 971.3: -4.1 ms/step. Change B — LLAVA_ZERO2_PIPE_ALLGATHER=1 again (round-36 code, cherry-picked as cee554a) on top of A: time (interleaved, same tree,/tmp/r37/time_{P,H}*): 966.9 / 965.0 ms/step (2.8721 / 2.8778) vs host-async alone 968.7 / 967.1: -2 ms (it was +4 ms while the host still waited for AdamW). Full check/tmp/r37/check(A + B, tree cee554a, switches /tmp/r37/best.json; 14:46 -> 14:53): PASS 2.8723 (966.8 ms/step), tier N1, gate 3 at -1.288 sigma (train canonical) / +0.175 sigma (probe). /submission/llava fast-forwarded to cee554a, switches = /tmp/r37/best.json (33 entries). Round end 14:54.
8.5 hRound 38 (2026-11-14 14:55 -> 15:04): no gain
Hotspot (round-37 analysis): the next forward waited for the whole bf16 parameter exchange before the projector.
Change: LLAVA_ZERO2_LAYER_WAIT=1 (commit 152580c, on the pipelined exchange): chunks ordered by first use in the
next forward (projector, embedding, layers, norm, head; small no-decay group first) and the forward waits per module
only for the chunks it needs (all at its end). Chunk plan verified against a brute-force element-level computation
(200 random layouts, /tmp/r37/plantest.py); harness bitwise identical with per-module waits and no wait after the
step.
**time (interleaved A/B, /tmp/r38/time_*):** 966.9 / 966.7 ms/step vs current 967.6 / 966.9: -0.5 ms, within
noise: with AdamW and the exchange pipelined (round 37) the exchange is no longer on the critical path. Dropped
(branch reset to cee554a; code in 152580c). Submission unchanged.8.9 hRound 39 (2026-11-14 15:04 -> 15:40)
Profile (current best,/tmp/r38/profile, 2 steps): main-stream idle down to 21 ms/step (51-59 before rounds 35-37); the step is now GPU compute: nvjet GEMMs ~670 ms, flash-attention 75.8 ms/step (backward 55.5: dq/dk/dv loop 47.9 + preprocess 4.4 + dq convert 3.2; forward 20.4 incl. CLIP's), Triton elementwise ~55, AdamW 23. Hotspot: attention kernels (FlashAttention-2 2.5.8 on Hopper; FlashAttention-3 uses the Hopper TMA/WGMMA pipelines). Change: LLAVA_FA3=1 (commit 38c93d3): packed decoder attention (incl. the last layer's suffix attention) on FlashAttention-3 kernels via llava/model/fa3.py (autograd Function over flashattn_hopper_cuda.varlen_fwd / varlen_bwd). The extension: flash-attention v2.7.0.post2 hopper/, trimmed to bf16 head-dim-128 fwd + bwd (flash_api.cpp dispatch and setup.py sources), built with a local CUDA 12.4.1 toolkit (/tmp/cuda124, extracted from NVIDIA's runfile; the system toolkit is 12.1, FA3 needs >= 12.3) against torch 2.1.2, sm_90a; installed as /opt/candidate-env/lib/python3.10/site-packages/flashattn_hopper_cuda.cpython-310-x86_64-linux-gnu.so (it resolves libcudart.so.12 like the installed FA2 extension does). CLIP stays on FA2 (head dim 64). Numerics (/tmp/r39/fa3test.py): out / dq / dk / dv errors vs an fp32 reference equal FA2's (0.20-0.25% relative), also with q shorter than k (suffix); tier N1 (not bitwise FA2), decided by gate 3. Isolated fwd+bwd per layer call (design micro-benchmark): 2.52 ms vs 3.09 ms. **time (interleaved A/B,/tmp/r39/time_*):** FA3 953.1 / 953.6 ms/step (2.9138 / 2.912) vs current 967.0 / 963.2 (2.8717 / 2.8832): -11.8 ms/step. Logged loss step 1: 1.3804 vs 1.3802 (+0.015%). Full check/tmp/r39/check(15:32 -> 15:39): PASS 2.9137 (953.1 ms/step), tier N1, gate 3 at -1.413 sigma (train canonical) / +0.127 sigma (probe). /submission/llava fast-forwarded to 38c93d3, switches = /tmp/r39/best.json (34 entries). The submission now also requires flashattn_hopper_cuda in /opt/candidate-env (installed).
9.2 hRound 40 (2026-11-14 15:40 -> 16:00)
Profile (current best with FA3,/tmp/r40/profile, 2 steps): GEMM 794 ms/step, flash 57.4 (FA3 bwd 36.8, FA3 fwd 17.6, CLIP FA2 3.0), AdamW 29.8 (chunks overlapped with the exchange), SwiGLU bwd 17.6 / fwd 8.4, add+RMSNorm bwd 12.4 / fwd 6.8, RoPE 8.0, generic elementwise 11.8, of which 7.8 ms are 96 autograd add_ kernels (3 per layer: the input gradients of q/k/v, which share the normed input, and of gate/up). Both ranks equally busy (1903.8 vs 1904.2 ms per 2 steps); still power-capped (median 693 W, 1.47 GHz). The Triton SwiGLU kernels reach 3.7-4.1 TB/s in isolation (copy: 4.08 TB/s), so they are not the problem. Kind: avoidable memory passes. Change: LLAVA_FUSED_DX=1 (commit df452e1): q/k/v (and the last layer's k/v) and gate/up through one autograd function: forward and weight gradients bitwise DeepSpeed's ZeRO-3 linear, input gradient accumulated inside the GEMMs (dx = dy1 W1; dx.addmm_(dy_i, W_i)). dx error vs fp32 2.35e-3 vs 2.85e-3 for the separate path (/tmp/r40/mltest.py). time (interleaved A/B,/tmp/r40/time_*): 947.7 / 947.4 ms/step (2.9303 / 2.9312) vs current 953.4 / 952.7 (2.9126 / 2.9148): -5.5 ms/step. Full check/tmp/r40/check(15:53 -> 16:00): PASS 2.9315 (947.3 ms/step), tier N1, gate 3 at -1.386 sigma (train canonical) / +0.206 sigma (probe). /submission/llava fast-forwarded to df452e1, switches = /tmp/r40/best.json (35 entries).