MiniMax M3.1 Flash 2.34×
Harness: Codex · switches: none
- Judged
- 2.34×
- Tier
- N1
- Self-best
- 2.32×
- Rounds
- 29
- Hours
- 10.0
- API cost
- $5.89
- Input tokens · 97.2% cached
- 238 M
- Output tokens
- 2.4 M
- Requests
- 1,743
§ 02Key levers
- Sequence packing
- ✓ round 3, 1942 ms, 1.42×
- Recompute / activations
- —
- Optimizer
- ✓ fused AdamW
- Communication & libraries
- resident params (1191 ms); ZeRO-3
- Other main levers
- fused norm / rotary / SwiGLU; logits only at supervised positions; grad layout and single-precision grad norm
§ ARaw trace
Raw work log29 sections
0.2 hRound 1 - profile of the unchanged tree (2026-11-12 19:07-19:12 UTC)
Command:bench.py profile --tree /submission/llava --config /submission/switches.json --out /tmp/r1/profile --detail --steps 2(/tmp/r1/profile/profile_tree.md, traces in /tmp/r1/profile/detail_tree.r*.json) Findings (rank 0, per optimizer step = 1 micro-batch of 16 rows/rank): * GPU kernel time 2881 ms/step; compute stream (stream 7) busy 2526 ms/step, NCCL all-gather 202 ms (ZeRO-3 param gather, stream 14) + reduce-scatter 133 ms (stream 26) - both fully overlapped with compute (union 2554 ms/step vs 2881 ms of kernel durations). * GPU idle ~220 ms/step (90% busy), concentrated in the optimizer/transition phase at the end of the step (t=+2.5..2.85 s of the 2.83 s step). * GPU time by op (ms/step): aten::mm (GEMM) 1471 (51%), aten::mul 286 (rotary embedding + norms), allgather 202, copy_ 174, reduce-scatter 135, add 122, cat 54, silu 24, flash-attn fwd+bwd 75, optimizer multi_tensor ~50. * CPU: aten::nonzero 874 ms/step over 112 calls/step (D2H syncs in llava_arch.prepare_inputs_labels_for_multimodal and DeepSpeed); busy_cpus 2.2 of 48. * Padding: each rank's 16 rows are padded to the longest row (1253 tokens for step 0), real tokens 13269 -> 20048 padded positions => ~34% of the LLM GEMM work is padding. * GEMM roofline: microbenchmark on this GPU gives 640-750 TFLOP/s bf16 for the LLM shapes; the step's mm time (1471 ms for ~1050 TFLOP) is at the roofline => the only way to cut GEMM time is to cut FLOPs (recompute, padding).
0.9 hRound 2 (continued) - model-side length grouping (split into groups) - NOT the way forward
The earlier attempt split the micro-batch into 2 or 4 length-homogeneous padded groups (LLAVA_SPLIT_GROUPS). Measured (sweep /tmp/r2/sweep): split2 2416 ms (1.1435x), split4 2615 ms (1.0565x). Two problems: (a) the per-group loss was computed withF.cross_entropyon bf16 logits while the reference casts the logits to fp32 first, which shifted the logged losses by up to 17% (the 25% sanity band did not catch it, gate 3 would have); (b) more groups than 2 made the step *slower*, most likely because every group is a separateLlamaModel.forward, and with DeepSpeed ZeRO-3 that means one more parameter gather/release cycle per group (the all-gather kernel is 203 ms/step in the reference profile). Fixed the fp32 cast, but the direction is dominated by the next round.
0.9 hRound 3 - packed micro-batch (varlen attention), LLAVA_PACK=1 (2026-11-12 19:56-20:05 UTC)
Profile finding (round 1) that drove it: 34% of the LLM GEMM work is padding (16 rows padded to the longest row of the micro-batch: 20048 padded positions for 13269 real tokens). The lengths are strongly bimodal (576 image tokens + 60..950 text tokens), so no reordering of the rows can remove the padding; only removing the padding itself can. Change:llava/model/llava_arch.pypacks the rows of the micro-batch into ONE sequence (_pack_multimodal_rows): the rows keep their own positions and their own attention block, so the attention runs with flash-attention's variable length API (llava/model/language_model/llava_llama.pypatchesLlamaFlashAttention2.forwardfor the packed layout, cu_seqlens travel down as the attention mask). The label of the first token of every row but the first is set to IGNORE_INDEX, so the shifted prediction never crosses a row boundary - the same set of supervised tokens as the padded batch. The loss is the mean cross entropy over exactly those tokens (fp32 logits, as the reference). One forward pass, so the ZeRO-3 gather pattern is unchanged. Numerics: tier N1 (not bitwise: different GEMM shapes, so a different rounding of the same maths). Measured on a real 16-row batch (1 GPU, /tmp/bench_pack.py): loss 0.20693207 (ref) vs 0.20696662 (packed), rel 1.7e-4; the gradient differences (median 0.9-3.4%) are of the same order as the reference's own run-to-run nondeterminism (1-3.7% on the same parameters).time(/tmp/r3/time_pack): 1942.2 ms/step, speedup_vs_frozen_reference 1.4224, peak memory 62.95 GiB (reference 73.5), no problems, logged losses within 0.35% of the reference's (first step 1.3827 vs 1.3839), startup 14.4 s. Full check PASS (/tmp/r3/check/check.json, 2026-11-12 20:10 UTC): status PASS, speedup 1.4207 (candidate 1944.5 ms/step vs the frozen reference 2762.6 ms/step), tier N1, gate 3 "equivalent": train_loss_last20_canonical diff -6.7e-5 (-0.02 sigma, bound 0.0094), probe_loss_last20 diff -1.6e-4 (-0.05 sigma, bound 0.0086). Committed as bf83e56.
1.4 hRound 4 - profile of the packed tree + DeepSpeed ZeRO-3 tuning (20:12-20:25 UTC)
Profile (/tmp/r4/profile/profile_tree.md, packed tree): GPU kernel time 2087 ms/step, GEMM 1132 ms (54%, at the roofline), NCCL all-gather 140 ms + reduce-scatter 46 ms, elementwise ~420 ms, flash attention 80 ms, optimizer 65 ms; the GPU is now 100% busy (no idle left), so only less work helps. Microbenchmarks of the elementwise part (real packed shapes, T=13269): rotary 0.83 ms fwd + 0.97 ms bwd per layer, RMSNorm 0.77 ms fwd + 1.47 ms bwd per norm (2 per layer), SwiGLU 0.38 + 0.63 ms. Per step that is ~115 ms (rotary), ~287 ms (RMSNorm), ~65 ms (SwiGLU). DeepSpeed ZeRO-3 tuning (sweep /tmp/r5/sweep, reference 2770.9 / 2755.3 ms): * ds_a (prefetch + reduce bucket 5e8): 1946.2 ms, 63.7 GiB - no gain (the all-gather is bandwidth bound, not latency bound). * ds_b (ds_a + all the parameters kept gathered: persistence threshold 1e9, max_live/max_reuse 1e10): 1921.4 ms, 1.4378x, 74.7 GiB (+11.8 GiB, the full bf16 model stays gathered). The logged losses are unchanged (max rel diff 0.16% vs the frozen reference), so the numerics are untouched. Baked into scripts/zero3.json (the judge uses the tree's file); the same values are also reachable through the LLAVA_DS_* switches.
1.8 hRound 5 - fused RMSNorm (Triton), LLAVA_FUSED_NORM=1 (20:33-20:57 UTC)
Profile finding that drove it (round 4): the elementwise part of the step is ~420 ms, of which the RMSNorm is the biggest piece (0.77 ms fwd + 1.47 ms bwd per norm at the real packed shape, 2 norms x 32 layers + the recomputation of the checkpointed layers). Change:llava/fused_rmsnorm.pycomputes the RMSNorm of the LLaMA layers in one Triton kernel per direction (forward; rowsum + backward; gradient of the weight), with the same arithmetic as transformers' LlamaRMSNorm (fp32 variance, the normalised value rounded to bf16 before the weight is applied). It is installed byllava/model/language_model/llava_llama.pywhen LLAVA_FUSED_NORM is set; the flag is off by default. Bug found while validating (important): the kernel that reduces the gradient of the weight looped over a fixed chunk of 64 rows without bounding the last chunk to the number of rows, so with 12987 rows it read 37 rows past the end of the tensor. The garbage turned into NaN in the master weights after the first optimizer step (the loss became NaN at step 2 and the run died with an illegal memory access at step 20). Fixed by masking the rows of the last chunk. Numerics: tier N1. On a small model the fused norm reproduces the reference's loss and *all* the gradients bitwise (T=65, 100, 12987 rows). The judge's quick check (gate 0 + the exact items of gate 2) passes: 0 exact failures, no dtype change, no precision-mode change, tier N1.time(/tmp/r6/time_fused): **1757.3 ms/step, speedup_vs_frozen_reference 1.5721** (the packed tree without the fused norm: 1924.6 ms/step, 1.4354x), peak memory 74.25 GiB. The logged losses match the packed tree (1.3829 vs 1.3839 at the first step, <0.02% afterwards).check --quick: PASS.
2.5 hRound 9 - fused rotary embedding (Triton), LLAVA_FUSED_ROTARY=1 (21:28-21:36 UTC)
Profile finding that drove it (round 7, /tmp/r7/profile/profile_tree.md): of the 1721 ms of GPU kernel time per step, the GEMMs are 1087 ms (63%, at the roofline, nothing to gain) and the elementwise part is ~250 ms, of which the rotary embedding is the biggest piece: the reference rotary costs 1.48 ms forward and 2.14 ms backward per layer at the packed shape of this recipe (T=13269, H=32, D=128), and with the gradient checkpointing the forward runs twice, so the rotary costs ~163 ms per step. Change:llava/fused_rotary.pycomputes the rotary embedding of the query and of the key in one Triton kernel per tensor: one pass over the data instead of a negation, a concatenation, two multiplications and an addition that each read and write the whole tensor. The rounding of every bf16 intermediate is kept (the products are rounded to bf16 before they are added), so the kernel reproduces transformers' apply_rotary_pos_emb bitwise, in the forward and in the backward (checked on the real shapes: T=13269, 1992, 4096, 100 -> max difference 0). The kernel is installed by the packed attention ofllava/model/language_model/llava_llama.pywhen LLAVA_FUSED_ROTARY is set; the flag is off by default. Numerics: tier N0 for this change (the rotary is bitwise identical to the reference's).time(/tmp/r9/time_rotary): 1635.7 ms/step, speedup_vs_frozen_reference 1.6889 (the previous tree: 1757.3 ms/step, 1.5721), peak memory 74.26 GiB (unchanged), no problems.check --quick(/tmp/r9/quick_rotary): PASS (gate 0 + the exact items of gate 2). Committed as 65495a4. /submission/switches.json = pack + fused norm + fused rotary. ### Failed direction: selective gradient checkpointing (the last layers keep their activations) Idea: the layers that are not checkpointed are not recomputed during the backward, which is a whole extra forward of the model. With the last 8-12 layers un-checkpointed the timing run went from 1757 ms to 1622 ms/step (1.7027x) and the peak memory rose from 74.25 to 98.62 GiB. The implementation hooks transformers'_set_gradient_checkpointingand lets the last layers of the LLaMA run without the checkpoint (llava/selective_ckpt.py, switch LLAVA_CKPT_SKIP_LAST). It deadlocks: the run hangs in the ZeRO-3 backward, both ranks spinning in a NCCL collective (py-spy: the autograd worker thread of both ranks is busy inside the backward; the GPU is at 100% but no step finishes). Two runs hung the same way (the first one with a version whose counter was also counting the vision tower's layers, the second one with the counter fixed to count only the LLaMA decoder layers). The same tree without the switch runs fine, and an earlier variant of the idea (which un-checkpointed the *first* layers, because of the same counter bug) did finish 24 steps, so the deadlock is not deterministic - it smells like the ZeRO-3 gradient reduction being issued in a different order on the two ranks when the order in which the layers' gradients become ready changes. Not worth the risk: the change is left out of the tree (llava/selective_ckpt.py is not imported by anything). ### Full check of the fused-rotary tree (21:57-22:04 UTC)check(/tmp/r10/check_rotary/check.json): status PASS, speedup 1.6923 (candidate 1632.4 ms/step vs the frozen reference 2762.6 ms/step), tier N1, gate 2: 0 exact failures, 0 dtype changes, 0 precision-mode changes (4724 tensors checked, 32 bitwise identical, 7 beyond the diagnostic tolerances - gate 3 decides), gate 3: passed, peak memory 74.25 GiB (reference 73.5), window spread 1.16%, the logged losses are within 0.15% of the reference's.
3.5 hRound 11 - fused AdamW (torch's fused implementation), LLAVA_FUSED_ADAM=1 (22:12-22:36 UTC)
Profile finding that drove it (round 10, /tmp/r11/profile/profile_tree.md): the AdamW update is 65.2 ms of the 1585 ms of GPU kernel time per step (4.1%), spent inmulti_tensor_applywith one kernel per tensor (the LLaMA-7B model has 686 parameter tensors and the ZeRO-3 partitions are flat per sub-group, so the update is done on ~10 flat tensors - but each of them goes through the multi-tensor path). Change:llava/train/llava_trainer.py::enable_fused_adamwwrapstorch.optim.AdamW.stepso that the parameter group is marked asfusedbefore the update runs. DeepSpeed's ZeRO-3 keeps the client optimizer (torch.optim.AdamW, created by the HF Trainer) and calls it once per sub-group with the flat fp32 partition (DeepSpeedZeroOptimizer_Stage3._optimizer_step), so the fused implementation of AdamW is used with exactly the same hyperparameters (lr, betas, eps, weight decay) and the same fp32 master weights and fp32 Adam state - only the kernels differ (one fused kernel instead of the multi-tensor ones). The flag has to be set at step time and not at creation time because DeepSpeed replaces the parameters with the fp32 GPU partitions after the optimizer is built. The optimizer class is unchanged (torch.optim.AdamW), so gate 2's "Adam hyperparameters" item is untouched. Numerics: tier N1 (the tree was already N1 because of the packed attention).time(/tmp/r11/time_adam4): 1621.4 ms/step, speedup_vs_frozen_reference 1.7039 (the previous tree: 1635.7 ms/step, 1.6889), +0.9%. The logged losses match the reference (1.3829 / 1.2879 / 1.3170 / 0.2573 / 1.2641 vs the reference's 1.3839 / 1.2880 / 1.3171 / 0.2572 / 1.2638).check --quick(/tmp/r11/quick_adam): PASS - gate 0 matches, 0 exact failures, 0 dtype changes, 0 precision-mode changes, and the tensor comparison is the same as the previous tree's (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances), so the fused update did not make the numerics any worse. /submission/switches.json = pack + fused norm + fused rotary + fused adam.
3.7 hRound 12 - partial recompute of the gradient checkpoint (cache the MLP's gate/up), LLAVA_PARTIAL_MLP=1 (22:38-22:47 UTC)
Profile finding that drove it (/tmp/r11/profile/profile_tree.md): of the 1585 ms of GPU kernel time per step, the GEMMs are 1119.6 ms (70.6%) and about a quarter of them are the *recompute* of the gradient checkpoint (the forward of the decoder layer is computed a second time in the backward pass: 404.9 MFLOP per token and layer, of which the MLP's two up-projections are 180.4 MFLOP, i.e. 44.6%). The recompute of the two projections is pure waste: it computes exactly the numbers that the forward pass already computed. Change:llava/partial_mlp_cache.py. The forward pass of the checkpoint (which runs undertorch.no_grad()) keeps the result of the gate and the up projection; the recompute (which runs undertorch.enable_grad()) reuses it and skips the matrix multiplication. The projections are still called through their module, so every hook of DeepSpeed's ZeRO-3 fires at exactly the same place and the trace of the modules is unchanged - only the matrix multiplication is skipped. The gradient of the weight is computed by the backward of the projection itself, with the same two matrix multiplications as the backward of the linear layer (grad_input = mm(grad, W), grad_weight = mm(x^T, grad)^T). Numerics: tier N0 for this change. Checked on the real shapes (T=13269, K=4096, N=11008, bf16, on the GPU): the output of the layer, the gradient of the input and the gradients of the three weights are bitwise identical to the ones of the reference. The judge'scheck --quick: PASS, and the tensor comparison is the same as the previous tree's (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r12/time_partial): 1518.5 ms/step, speedup_vs_frozen_reference 1.8192 (the previous tree: 1621.4 ms/step, 1.7039), +6.8%. Peak memory 93.82 GiB (the reference 73.5, the previous tree 74.25): the two projections of the 32 layers are kept alive, 2 x 13269 x 11008 x 2 bytes x 32 = 18.7 GB more. /submission/switches.json = pack + fused norm + fused rotary + fused adam + partial mlp.
3.8 hRound 13 - cache the down projection too (skip its recompute), LLAVA_PARTIAL_MLP=1 (22:47-22:53 UTC)
Change: the same mechanism as round 12, extended to the third projection of the MLP. The recompute of a layer now only does the elementwise product of the MLP (silu(gate) * up); the matrix multiplications of the gate, the up and the down projection are all skipped. Numerics: tier N0 (checked on the real shapes on the GPU: the output of the layer and the gradients of the input and of the three weights are bitwise identical to the reference's).check --quick(/tmp/r13/quick_down): PASS, the tensor comparison is unchanged (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r13/time_down): 1457.7 ms/step, speedup_vs_frozen_reference 1.8952 (the previous tree: 1518.5 ms/step, 1.8192), +4.2%. Peak memory 97.47 GiB (+3.7 GB: the output of the down projection of the 32 layers).
4.1 hRound 14 (investigation) - the copies of the gradients into the ipg bucket
The profile of the tree of round 13 (/tmp/r14/profile) showed 30 ms more copies of the gradients of the MLP into the ipg bucket of DeepSpeed than the tree of round 11 (96 extra copies of 45.1M elements per step). The instrumentation ofDeepSpeedZeroOptimizer_Stage3.__add_grad_to_ipg_bucket(a copy of the tree in /tmp/dbg/llava, LLAVA_DEBUG_IPG) shows that *every* parameter is added to the ipg bucket twice, with and without the cache of the projections: it is a behaviour of DeepSpeed (the hook on the accumulation of the gradient fires again when the gradient of the parameter is accumulated a second time) and not something the cache introduces. Making the gradient of the weight contiguous (.contiguous()) changes nothing (1460 ms instead of 1457 ms), so the change was reverted.
4.1 hRound 15 - cache the query, key and value projections too, LLAVA_PARTIAL_MLP=1 (23:05-23:11 UTC)
Profile finding that drove it (/tmp/r14/profile/profile_tree.md): the GEMMs are 930.3 ms of the 1429 ms of GPU kernel time per step (65%). Of the matrix multiplications that are still done twice (the forward pass and the recompute of the gradient checkpoint), only the attention is left: the query, the key and the value projections are 100.7 of the 134.2 MFLOP per token and layer, so they are 10% of the matrix multiplications of a step (about 70 ms). Change: the same mechanism as rounds 12 and 13, applied to the query, the key and the value projection of every layer. The recompute of a layer now only does the rotary embedding, the attention itself and the output projection. Numerics: tier N0 (the output of the projection and the gradients of its input and of its weight are bitwise identical to the ones of the reference; the same three matrix multiplications as the backward of the linear layer).check --quick(/tmp/r15/quick_attn): PASS, the tensor comparison is unchanged (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r15/time_attn): 1401.0 ms/step, speedup_vs_frozen_reference 1.9719 (the previous tree: 1457.7 ms/step, 1.8952), +4.0%. Peak memory 108.39 GiB (the reference 73.5): the query, the key and the value of the 32 layers are kept alive, 3 x 13269 x 4096 x 2 bytes x 32 = 10.4 GB more.
4.2 hRound 16 - do not copy the gradients into the flat ipg bucket, LLAVA_PARTIAL_MLP=1 (23:11-23:17 UTC)
Profile finding that drove it (/tmp/r14/profile/profile_tree.md): 80.5 ms of the step is spent in copies of the gradients. DeepSpeed's ZeRO-3 copies every gradient into a flat buffer (__add_grad_to_ipg_bucket) and only reduces that buffer whenreduce_scatteris off; withreduce_scatteron (the default) the reduction reads the gradients of the parameters one by one (__avg_scatter_grads), so the copies into the flat buffer are never used. Change:scripts/zero3.jsonsets"contiguous_gradients": false. The reduction of the gradients is the same code path as before (the flat buffer is not used either way), the gradients are read where they are, and the flat buffer (1 GB) is not allocated at all. Numerics: tier N0 (the same gradients are reduced, in the same order).time(/tmp/r16/time_nocontig): 1395.4 ms/step, speedup_vs_frozen_reference 1.9797 (the previous tree: 1401.0 ms/step, 1.9719). Peak memory 107.46 GiB (-0.93 GB).check --quick(/tmp/r16/quick_nocontig): PASS, the tensor comparison is unchanged.
4.4 hRound 17 - cache the output projection of the attention, LLAVA_CACHE_OPROJ=1 (23:19-23:33 UTC)
Profile of the tree of round 16 (/tmp/r17/profile/profile_tree.md): 1356 ms of GPU kernel time per step, of which 862 ms are the matrix multiplications (63%), 113 ms are plain copies (8%), 85 ms the flash attention, 77 ms the NCCL collectives, 56 ms the fused AdamW and ~100 ms the elementwise operations. The GPU is idle 153 ms of the 1477 ms of the profiled step (one gap of 59 ms at the end of the backward, where the CPU prepares the optimizer step); in the timed run the bubble is ~70 ms of 1395 ms. Profile finding that drove it: after the caches of rounds 12 to 15, the only matrix multiplication of a layer that was still computed twice was the output projection of the attention (the recompute of the checkpoint computed it again). The GPU time of the copies of the gradients and of the parameters is not reducible from our side (it is DeepSpeed's ZeRO-3), and the collectives (46.7 ms of AllGather of the parameters, 30 ms of ReduceScatter of the gradients) are the price of the ZeRO-3 of the recipe. Change: the output projection of the attention is cached like the other projections of the layer (llava/partial_mlp_cache.py, switchLLAVA_CACHE_OPROJ), so the recompute of the checkpoint takes its result instead of computing the matrix multiplication again. Numerics: tier N0 (the same mechanism as the other projections: the result of the forward pass is reused and the gradients of the input and of the weight are computed with the same two matrix multiplications as the backward of the linear layer).check --quick(/tmp/r17/quick_oproj): PASS, the tensor comparison is unchanged (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r17/time_oproj): 1378.1 ms/step, speedup_vs_frozen_reference 2.0047 (the previous tree: 1395.4 ms/step, 1.9797), +1.3%. Peak memory 111.1 GiB (+3.6 GB: the output of the output projection of the 32 layers).
4.5 hRound 18 - the output layer only on the supervised positions, LLAVA_SUP_LOGITS=1 (23:33-23:40 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): the matrix multiplications are 862 ms of the 1356 ms of GPU kernel time per step. Of those, the output layer (the vocabulary projection) is 10.7 TFLOP (2% of the matrix multiplications of the step, about 18 ms), and it is applied to *every* position of the sequence, while only the tokens of the answer are supervised: the image tokens (576 per image) and the prompt are masked. The logits of the whole sequence are also the biggest tensors of the step (13600 x 32000, 1.7 GB in fp32). Change: in the packed path the transformer returns the hidden states and the output layer is applied to the hidden states of the supervised positions only (`llava/model/language_model/ llava_llama.py, switchLLAVA_SUP_LOGITS`). The loss is still the mean cross entropy over the same supervised tokens, andmodel.forward(labels=...)still returns it. Numerics: tier N1 (the output layer is a matrix multiplication with fewer rows, so the rounding of the logits can differ in the last bit; the loss and the gradients are the same to about 1e-7).check --quick(/tmp/r18/quick_sup): PASS, the tensor comparison is unchanged (7 tensors beyond the diagnostic tolerances, the same ones as before, 0 exact failures, 0 dtype changes).time(/tmp/r18/time_sup): 1354.4 ms/step, speedup_vs_frozen_reference 2.0397 (the previous tree: 1378.1 ms/step, 2.0047), +1.7%. Peak memory 110.49 GiB (-0.6 GB).
4.7 hRound 19 - fused SwiGLU, LLAVA_FUSED_SWIGLU=1 (23:40-23:50 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): the elementwise operations of the MLP are the biggest remaining item that is not a matrix multiplication: the SiLU (13.7 ms), the multiplication of the SiLU and the up projection (28.4 ms of the multiplications of the step) and the SiLU of the backward (9.0 ms). The chain reads and writes the two big tensors of the layer (the gate and the up projection, 13600 x 11008) four times. Change:llava/fused_swiglu.pycomputessilu(gate) * upin one Triton kernel and its backward in one more, and the MLP of the LLaMA layers uses it (switchLLAVA_FUSED_SWIGLU). The arithmetic is the one of PyTorch, with the rounding of every intermediate result kept: the SiLU isx / (1 + exp(-x))in fp32, rounded to bf16, the product is rounded to bf16, and the backward isd_up = d_out * silu(gate),d_silu = d_out * up(rounded to bf16) andd_gate = d_silu * s * (1 + gate * (1 - s))withs = 1 / (1 + exp(-gate)). Two details of Triton matter for the exactness:tl.expis the fastex2.approxand differs from theexpfof PyTorch (48% of the values), so the kernel usestl.math.exp(the libdevice__nv_expf); and the division of the sigmoid usestl.math.div_rn. Numerics: tier N0. Checked on the real shapes (13600 x 11008) on the GPU: the result and the two gradients are bitwise identical to the ones of PyTorch (0 differing elements out of 149.7M).check --quick(/tmp/r19/quick_swiglu): PASS, the tensor comparison is unchanged (7 tensors beyond the diagnostic tolerances, the same ones as before, 0 exact failures, 0 dtype changes).time(/tmp/r19/time_swiglu): 1337.6 ms/step, speedup_vs_frozen_reference 2.0653 (the previous tree: 1354.4 ms/step, 2.0397), +1.3%. Peak memory 110.5 GiB (unchanged; the kernel does not need the result of the SiLU to be kept for the backward).
4.8 hRound 20 - no copy of the gradients of the rotary, LLAVA_FUSED_ROTARY=1 (23:51-23:58 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): 113 ms of the step are plain copies. 64 of them are the copies of the gradients of the rotary embedding: the backward of the fused rotary made the gradients contiguous before reading them, but the gradient of the query arrives with the strides of the transposed view of the attention (the transpose of the backward of the transpose of the forward), so the copy was real: two copies of 111 MB per layer and per backward (14 GB of traffic per step, about 5 ms). Change:llava/fused_rotary.pyreads the gradients with their own strides (the kernel already takes the strides of its arguments), so the copies are not needed any more. Numerics: tier N0 (the same values are read, only the copy is gone). Checked on the GPU with the real shapes: the gradient of the query is bitwise identical to the one of the reference implementation of the rotary (the strides of the gradient are the same too).check --quick(/tmp/r20/quick_rotary): PASS, the tensor comparison is unchanged (7 tensors beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r20/time_rotary): 1331.2 ms/step, speedup_vs_frozen_reference 2.0753 (the previous tree: 1337.6 ms/step, 2.0653), +0.5%. Peak memory 110.5 GiB (unchanged).
5.0 hRound 21 - keep the result of the attention for the recompute, LLAVA_CACHE_FLASH=1 (23:57-00:05 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): after the caches of the projections (rounds 12 to 17), the recompute of the checkpoint still ran the attention of every layer a second time: 33.2 ms of the step are the forward of flash-attention (64 calls, one per layer and per pass), half of which is the recompute. Change:llava/flash_cache.pykeeps the result of the attention of the forward pass (the output, the log-sum-exp of the softmax and the state of the random number generator) and the recompute reuses it: the forward of the recompute returns the result of the forward pass and the backward calls the backward of flash-attention with the tensors it saves (the query, the key, the value, the output and the log-sum-exp), with the same arguments as the reference. Numerics: tier N0. Checked on the GPU with the real shapes (13600 tokens, 32 heads): the output, the gradient of the key and the gradient of the value are bitwise identical to the ones of flash-attention's own function. The gradient of the query is the one of flash-attention's backward, which accumulates with atomic operations when it is not deterministic (the reference gives a different result on two runs as well); in the deterministic mode (the mode of the gate 2 of the judge) the two are bitwise identical.check --quick(/tmp/r21/quick_flash): PASS, the tensor comparison is unchanged (7 tensors beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).time(/tmp/r21/time_flash): 1314.5 ms/step, speedup_vs_frozen_reference 2.1016 (the previous tree: 1331.2 ms/step, 2.0753), +1.3%. Peak memory 114.25 GiB (+3.75 GB: the output of the attention of the 32 layers is kept until the recompute).
5.2 hRound 22 - full check of the tree of round 21: ERROR (00:05-00:12 UTC)
Command:bench.py check --tree /submission/llava --config /submission/switches.json --out /tmp/r22/check_full* gate 0: PASS. gate 1: PASS - 1319.1 ms/step, speedup 2.0943 (the frozen reference is 2762.6 ms/step), peak memory 114.25 GiB. * gate 2: NOT_BITWISE with 0 exact failures (7 tensors beyond the diagnostic tolerances, the same ones as in every round since the packed attention), 0 dtype changes. * gate 3: ERROR - "trajectory seed 0 exited 1". The trajectory died at step 41 withRuntimeError: shape '[1, 14122, 32, 128]' is invalid for input of size 60534784inllava_llama.py:67(the view of the query of the packed attention). Root cause (found by reading the traceback): the values that the projections keep for the recompute of the checkpoint were kept per projection, without saying which forward pass had computed them. The bench makes a forward pass of its own before every training step (the forward that measures the loss of the batch) and, at the end of every step of the window of the probe, another one with the probe batch (32 held out rows). Those passes end without a recompute, so their values stayed in the cache; the *next* training pass found them and took them for its own recompute - the values of another batch, with another number of tokens. The first step of the probe window (step 41) is the first one whose micro-batch has another number of tokens than the probe batch, and the shapes did not match: the crash.
5.2 hRound 23 - the values of the projections belong to the pass that computed them (00:12-00:20 UTC)
Change:llava/partial_mlp_cache.pygives every forward pass of the model a new generation of the values (a counter, and the values of the previous generation are dropped), and a recompute only takes the values of the generation that is running. The values of a pass that ends without a recompute are dropped with it, so they cannot be used by another pass. Numerics: N0 for the values (the same values are computed and used; only the values of another pass are not used any more).check --n1(/tmp/r23/n1, gates 0 and 3): PASS. The trajectory of the gate 3 ends with a bias of -0.12 sigma (the canonical loss of the last 20 steps) and -0.03 sigma (the probe loss), the bound is 3 sigma.time(/tmp/r23/time): 1318.5 ms/step, speedup_vs_frozen_reference 2.0953 (the tree of round 21: 1314.5 ms/step, 2.1016 - the same, the fix does not cost anything in the timing run).
5.4 hRound 24 - cheaper release of the gathered parameters of ZeRO-3, LLAVA_FAST_ZERO3_RELEASE=1 (00:23-00:26 UTC)
Profile finding that drove it (/tmp/r17/profile, the trace of the step): the GPU is idle for 58.8 ms of the step, right after the backward, and the CPU is walking the module tree of the model 13 686 times: it isDeepSpeedZeroOptimizer_Stage3.step->_partition_all_parameters->PartitionedParameterCoordinator.release_and_reset_all, which releases the 686 parameters that are still gathered. The release walks the module tree three times per step and asks the process for its rank 2 744 times to decide whether a message is printed, although the message is only printed with force=True and is never printed in the training loop. Change:llava/zero3_release.pykeeps the parameters of a module (they do not change while the model is trained) instead of walking the module tree again, and does not ask for the rank when the message cannot be printed. The same parameters are released, in the same order, with the same calls. Numerics: tier N0 (nothing is computed differently).time(/tmp/r24/time): 1306.0 ms/step, speedup_vs_frozen_reference 2.1153 (the previous tree: 1318.5 ms/step, 2.0953), +1.0%. Peak memory 114.25 GiB (unchanged).
5.4 hRound 25 - the gradient of the weight of a projection in its own layout, LLAVA_CONTIG_GRAD=1 (00:26-00:30 UTC)
Profile finding that drove it (/tmp/r17/profile): 41 ms of the step are the copies of the gradients of the weights of the projections (the shapes [11008, 4096], [4096, 4096] and [4096, 11008] of the trace). The gradient of the weight is computed by the backward of the matrix multiplication in the transposed layout and is copied into the gradient of the parameter; the copy of a transposed tensor reads the memory in the wrong order (96 GB/s against 3.9 TB/s for a contiguous copy on this GPU). Change: the backward of the cached projection computes the gradient of the weight with the transposed multiplication (mm(grad.t(), x)instead ofmm(x.t(), grad).t()), which gives the same numbers in the layout of the weight. Checked on the GPU with the real shapes (13600 tokens): not one element of the 45M differs, for the three shapes of the model. Numerics: tier N0 (the values are bitwise identical, only the layout of the gradient changes).time(/tmp/r25/time): 1266.6 ms/step, speedup_vs_frozen_reference 2.1811 (the previous tree: 1306.0 ms/step, 2.1153), +3.0%. Peak memory 114.25 GiB (unchanged).
5.5 hRound 26 - profile of the tree of round 25 (00:30-00:31 UTC)
Command: bench.py profile --tree /submission/llava --config /submission/switches.json --out /tmp/r26/profile --detail
GPU kernel time 1232 ms/step (the tree of round 17: 1356 ms/step). The GEMMs are 838 ms (68%)
and are at the roofline of the shapes (774 TFLOP/s measured on the GPU for the shape of the
gradient of a weight); the rest is: the fused AdamW 56 ms, the backward of flash-attention 53 ms,
the AllGather 35.5 ms, the ReduceScatter 31.2 ms, the residual adds 27.6 ms (384 calls at
4.2 TB/s, at the roofline), the cats 21.7 ms, the copies 35.6 ms, the fused SwiGLU 29 ms, the
fused RMSNorm 20.8 ms, the fused rotary 13.6 ms. The GPU is idle for 35 ms of the step.5.5 hRound 27 - release the gathered parameters at the end of the optimizer step, LLAVA_DEFER_ZERO3_RELEASE=1 (00:31-00:36 UTC)
Profile finding that drove it (/tmp/r26/profile): the release of the gathered parameters is CPU work that the GPU waits for (the GPU is idle for 35 ms of the step, the release is most of it). The parameters that are gathered are not needed by the optimizer step: the optimizer works on the fp32 master weights and on the fp32 gradients. Releasing them at the end of the step lets the GPU update the parameters while the CPU releases the ones of the next forward pass. Change:llava/zero3_release.pydefers_partition_all_parametersto the end of the optimizer step (the release is done before the gather of the persistent parameters, in the same order as before); a step that ends without a release (the loss scale overflowed) releases the parameters at the beginning of the next step. Numerics: tier N0 (the same parameters are released, in the same order, with the same calls; only the moment of the release changes, and the values of the parameters are refreshed by the gather that follows it, as before).time(/tmp/r27/time): 1247.3 ms/step, speedup_vs_frozen_reference 2.2148 (the previous tree: 1266.6 ms/step, 2.1811), +1.5%. Peak memory 114.25 GiB (unchanged: the peak is in the forward pass, not in the optimizer step).
5.7 hRound 28 - full check of the tree of round 27 (00:37-00:47 UTC)
Command: bench.py check --tree /submission/llava --config /submission/switches.json --out /tmp/r28/check_full
* gate 0: PASS. gate 1: PASS - 1249.8 ms/step, speedup 2.2103 (the frozen reference is
2762.6 ms/step), peak memory 114.25 GiB.
* gate 2: NOT_BITWISE with 0 exact failures (7 tensors beyond the diagnostic tolerances, the
same ones as in every round since the packed attention), 0 dtype changes, 0 precision mode
changes -> tier N1.
* gate 3: PASS - the canonical loss of the last 20 steps is -0.10 sigma, the probe loss is
-0.03 sigma (the bound is 3 sigma).
The tree of the round is the one of the round 27 (commit fa7cfb8): it is the state that is
judged at the end, unless a later round is better.6.4 hRound 30 - a Triton kernel for the fused AdamW, LLAVA_TRITON_ADAM=1 (01:12-01:25 UTC)
Profile finding that drove it (/tmp/r29/profile): the fused AdamW of PyTorch moves 28 bytes per parameter (the gradient, the two moments, the parameter) and reaches 1.7 TB/s on this GPU; the kernel takes 56 ms of the step. Change:llava/fused_adamw.pyis a Triton kernel that does the same update as PyTorch's fused AdamW (the same arithmetic, the same bias corrections, the same rounding) and is installed in place oftorch._fused_adamw_; the state, the counter of the steps and the bias corrections are still PyTorch's. The kernel takes 22 ms of the step (4.2 TB/s, the speed of a copy). Numerics: tier N1 (the update is PyTorch's fused AdamW up to the last bit of the rounding of the arithmetic of the kernel; the parameters differ by at most 4e-11).time(/tmp/r30/time_triton): 1236.5 ms/step, speedup_vs_frozen_reference 2.2341 (the tree without the kernel, measured just before: 1255.5 ms/step, 2.2004).check --quick(/tmp/r30/quick): PASS, tier N1, 0 exact failures, 0 dtype changes. Committed as 09e5bad.
6.4 hRound 31 - skip the work of the optimizer step that nothing uses (01:26-01:32 UTC)
Profile finding that drove it (/tmp/r29/profile): the optimizer step of ZeRO-3 spends 26 ms in the computation of the norm of the gradients (get_grad_norm_direct: the gradients are copied into a buffer of double precision and reduced there, 16.3 ms of copies and 8.7 ms of reduction) and 6.3 ms in the unscale of the gradients (grad.mul_(1/loss_scale)). Both are dead work: the norm is only used to clip the gradients and the recipe does not clip them (scripts/zero3.jsondoes not setgradient_clipping, soclip_gradis zero andunscale_and_clip_gradsnever scales the gradients with the norm; the value is kept inoptimizer._global_grad_norm, which nothing reads), and with bf16 the loss scale is one, so the unscale multiplies the gradients by one. Change:llava/fast_grad_norm.pyandllava/fast_grad_unscale.pypatch the two methods ofDeepSpeedZeroOptimizer_Stage3(LLAVA_FAST_GRAD_NORM,LLAVA_FAST_GRAD_UNSCALE). Numerics: tier N0 (the gradients, the update of the parameters, the state of the optimizer and the loss scale are unchanged; only a value that nothing reads is not computed).time(/tmp/r31/time_norm): 1218.7 ms/step, speedup_vs_frozen_reference 2.2669 (the tree without the change: 1236.5 ms/step, 2.2341). Committed as 3f3e769.
6.5 hRound 32 - make the release of the gathered parameters cheaper, LLAVA_FAST_PARAM_RELEASE=1 (01:32-01:41 UTC)
Profile finding that drove it (/tmp/r29/profile): the GPU is idle for 43.5 ms in one gap, between the release of the parameters that the backward pass gathered and the all-gather of the forward pass. The CPU work of the release is 33 ms: for every one of the 686 parameters, it builds three messages that are never printed, describes the parameter for one of them, queries the current stream of the device (11 us) and allocates the empty tensor that replaces the gathered one. Change:llava/fast_param_release.pybuilds the messages and the description not at all, queries the stream of the device once, and reuses the empty tensor of every dtype and device. Numerics: tier N0 (the same parameters are released, in the same order, with the same calls; only the messages that are not printed and the repeated queries are avoided).time(/tmp/r32/time_release): 1207.6 ms/step, speedup_vs_frozen_reference 2.2877 (the tree without the change: 1218.7 ms/step, 2.2669).check --quick(/tmp/r32/quick): PASS, tier N1, 0 exact failures, 0 dtype changes. Committed as 1c5c927.
6.8 hRound 33 - the norm of the gradients in single precision (2026-11-13 01:40-01:55 UTC)
Profile finding that drove it: DeepSpeed's ZeRO-3 computes the norm of the gradients to clip them (the defaultgradient_clippingof DeepSpeed is 1.0, so the gradients *are* clipped - the earlier round that tried to skip the norm was a no-op). The norm was computed with a buffer of double precision: 16.3 ms of bf16->fp64 copies + 8.7 ms of fp64 reduction + a.item()that costs a synchronization (a 4.8 ms gap of the GPU idle). Change:llava/fast_grad_norm.pycomputes the norm with `torch.linalg.vector_norm(g, 2, dtype=torch.float32)` (the squares of the bf16 gradients are exact in single precision) and sums the per-parameter squares in double precision, as DeepSpeed does. The norm of the reference is computed instead when the result is not finite. Tier N1 (the norm differs by ~1e-7 relative, the gradients are scaled by it).time(/tmp/r33/time_norm32): 1200.9 ms/step, speedup_vs_frozen_reference 2.3005 (was 1207.6 / 2.2877).check --quick(/tmp/r33/quick): PASS (0 exact failures, 7 unresolved). Committed b07099e.
6.8 hRound 34 - the parameters of ZeRO-3 are not persistent any more (2026-11-13 01:55-02:05 UTC)
Profile finding that drove it: the end of the optimizer step is serial. Withstage3_param_persistence_threshold: 1e9every parameter of the model is persistent (7 062 902 784 elements in 686 parameters; the reference has 599 040 in 312), so at the end of every step DeepSpeed releases all the gathered parameters (17.9 ms of CPU, a 28.6 ms gap of the GPU idle) and then re-gathers them with 686 NCCL all-gathers (38 ms of GPU time, nothing else to do while they run). Change:LLAVA_DS_PERSIST_THRESHOLD=100000(the value the reference's "auto" resolves to) in /submission/switches.json. The parameters are then gathered by the prefetcher of ZeRO-3 during the forward and the backward, where the gathering overlaps with the computation, and the end of the step has almost nothing left to do. The numerics do not change (the same parameters are gathered, the same values).time(/tmp/r34/time_persist): 1191.0 ms/step, speedup_vs_frozen_reference 2.3197 (was 1200.9 / 2.3005). The gain is much smaller than the 60 ms the tail seemed to cost: the gathering during the forward and the backward is only partly hidden. ## Rounds 35-41 - the tail of the ZeRO-3 step (2026-11-13 02:05-03:00 UTC) Profile finding that drove them: the critical path of a step is the compute stream (1084 ms of kernels of the 1200 ms of the step); the communication (all-gather 45 ms, reduce-scatter 31 ms), the split of the flat buffer of ZeRO-3 (86 ms) and the gradients (25 ms) run on other streams and are completely hidden. The GEMMs are 844 ms of the 1084 ms and run at the roofline (~818 TFLOP/s of ~990), the recompute of the gradient checkpointing is ~207 ms of them, so what is left to win is the ~175 ms of GPU idle (32 ms in the tail of the step, 17.5 ms in the forward of the vision tower, which is CPU bound) and the recompute. Four changes were tried, each behind its own switch, and each was neutral (the spread of the reference is 0.13% and the runs differ by less than that): *LLAVA_FAST_PARAM_GATHER(llava/fast_param_gather.py): gather every parameter of ZeRO-3 with an all-gather of its own instead of a cat of all of them.time(/tmp/r38/time_gather): 1194.7 ms/step (2.3118). *LLAVA_FAST_PARAM_SPLIT(llava/fast_param_split.py): copy the gathered parameters out of the flat buffer of the gathering with one cat for the whole module instead of one copy per parameter.time(/tmp/r38/time_split): 1202.2 ms/step (2.2961). *LLAVA_NO_PERSIST_GATHER: do not gather the persistent parameters at the end of the step (the prefetcher gathers them anyway, while the GPU computes).time(/tmp/r39/time_nopersist): 1205.6 ms/step (2.2924), but that run was noisy (the two windows of the run differ by 3.7%). *LLAVA_DS_PERSIST_THRESHOLD=100000: the parameters of ZeRO-3 are not persistent any more (the value the reference's "auto" resolves to).time(/tmp/r34/time_persist): 1191.0 ms/step (2.3197).sweep(/tmp/r40/sweep) with the four combinations of the last two switches: 1199.6, 1197.7, 1196.4 and 1200.8 ms/step - all inside 0.4%, that is, inside the noise of the measurement. The tree keeps the code of the first two (off by default) and the last two are on. Fullcheck(/tmp/r41/check): PASS, speedup 2.308, tier N1, 0 exact failures, 7 unresolved (out of 4724 tensors), 32 tensors bitwise identical. This is the best validated state so far.