MiniMax M3.1 Flash 2.34×

Harness: Codex · switches: none

Judged
2.34×
Tier
N1
Self-best
2.32×
Rounds
29
Hours
10.0
API cost
$5.89
Input tokens · 97.2% cached
238 M
Output tokens
2.4 M
Requests
1,743
Sequence packing
✓ round 3, 1942 ms, 1.42×
Recompute / activations
—
Optimizer
✓ fused AdamW
Communication & libraries
resident params (1191 ms); ZeRO-3
Other main levers
fused norm / rotary / SwiGLU; logits only at supervised positions; grad layout and single-precision grad norm
Raw work log29 sections
0.2 hRound 1 - profile of the unchanged tree (2026-11-12 19:07-19:12 UTC)
Command: bench.py profile --tree /submission/llava --config /submission/switches.json --out /tmp/r1/profile --detail --steps 2
(/tmp/r1/profile/profile_tree.md, traces in /tmp/r1/profile/detail_tree.r*.json)

Findings (rank 0, per optimizer step = 1 micro-batch of 16 rows/rank):
* GPU kernel time 2881 ms/step; compute stream (stream 7) busy 2526 ms/step,
  NCCL all-gather 202 ms (ZeRO-3 param gather, stream 14) + reduce-scatter 133 ms (stream 26)
  - both fully overlapped with compute (union 2554 ms/step vs 2881 ms of kernel durations).
* GPU idle ~220 ms/step (90% busy), concentrated in the optimizer/transition phase
  at the end of the step (t=+2.5..2.85 s of the 2.83 s step).
* GPU time by op (ms/step): aten::mm (GEMM) 1471 (51%), aten::mul 286 (rotary embedding
  + norms), allgather 202, copy_ 174, reduce-scatter 135, add 122, cat 54, silu 24,
  flash-attn fwd+bwd 75, optimizer multi_tensor ~50.
* CPU: aten::nonzero 874 ms/step over 112 calls/step (D2H syncs in
  llava_arch.prepare_inputs_labels_for_multimodal and DeepSpeed); busy_cpus 2.2 of 48.
* Padding: each rank's 16 rows are padded to the longest row (1253 tokens for step 0),
  real tokens 13269 -> 20048 padded positions => ~34% of the LLM GEMM work is padding.
* GEMM roofline: microbenchmark on this GPU gives 640-750 TFLOP/s bf16 for the LLM shapes;
  the step's mm time (1471 ms for ~1050 TFLOP) is at the roofline => the only way to cut
  GEMM time is to cut FLOPs (recompute, padding).
0.9 hRound 2 (continued) - model-side length grouping (split into groups) - NOT the way forward
The earlier attempt split the micro-batch into 2 or 4 length-homogeneous padded groups
(LLAVA_SPLIT_GROUPS).  Measured (sweep /tmp/r2/sweep): split2 2416 ms (1.1435x), split4 2615 ms
(1.0565x).  Two problems: (a) the per-group loss was computed with F.cross_entropy on bf16
logits while the reference casts the logits to fp32 first, which shifted the logged losses by up to
17% (the 25% sanity band did not catch it, gate 3 would have); (b) more groups than 2 made the step
*slower*, most likely because every group is a separate LlamaModel.forward, and with DeepSpeed
ZeRO-3 that means one more parameter gather/release cycle per group (the all-gather kernel is
203 ms/step in the reference profile).  Fixed the fp32 cast, but the direction is dominated by the
next round.
0.9 hRound 3 - packed micro-batch (varlen attention), LLAVA_PACK=1 (2026-11-12 19:56-20:05 UTC)
Profile finding (round 1) that drove it: 34% of the LLM GEMM work is padding (16 rows padded to the
longest row of the micro-batch: 20048 padded positions for 13269 real tokens).  The lengths are
strongly bimodal (576 image tokens + 60..950 text tokens), so no reordering of the rows can remove
the padding; only removing the padding itself can.

Change: llava/model/llava_arch.py packs the rows of the micro-batch into ONE sequence
(_pack_multimodal_rows): the rows keep their own positions and their own attention block, so the
attention runs with flash-attention's variable length API (llava/model/language_model/llava_llama.py
patches LlamaFlashAttention2.forward for the packed layout, cu_seqlens travel down as the
attention mask).  The label of the first token of every row but the first is set to IGNORE_INDEX, so
the shifted prediction never crosses a row boundary - the same set of supervised tokens as the
padded batch.  The loss is the mean cross entropy over exactly those tokens (fp32 logits, as the
reference).  One forward pass, so the ZeRO-3 gather pattern is unchanged.

Numerics: tier N1 (not bitwise: different GEMM shapes, so a different rounding of the same maths).
Measured on a real 16-row batch (1 GPU, /tmp/bench_pack.py): loss 0.20693207 (ref) vs 0.20696662
(packed), rel 1.7e-4; the gradient differences (median 0.9-3.4%) are of the same order as the
reference's own run-to-run nondeterminism (1-3.7% on the same parameters).

time (/tmp/r3/time_pack): 1942.2 ms/step, speedup_vs_frozen_reference 1.4224, peak memory
62.95 GiB (reference 73.5), no problems, logged losses within 0.35% of the reference's
(first step 1.3827 vs 1.3839), startup 14.4 s.

Full check PASS (/tmp/r3/check/check.json, 2026-11-12 20:10 UTC): status PASS, speedup
1.4207 (candidate 1944.5 ms/step vs the frozen reference 2762.6 ms/step), tier N1, gate 3
"equivalent": train_loss_last20_canonical diff -6.7e-5 (-0.02 sigma, bound 0.0094),
probe_loss_last20 diff -1.6e-4 (-0.05 sigma, bound 0.0086).  Committed as bf83e56.
1.4 hRound 4 - profile of the packed tree + DeepSpeed ZeRO-3 tuning (20:12-20:25 UTC)
Profile (/tmp/r4/profile/profile_tree.md, packed tree): GPU kernel time 2087 ms/step, GEMM 1132 ms
(54%, at the roofline), NCCL all-gather 140 ms + reduce-scatter 46 ms, elementwise ~420 ms, flash
attention 80 ms, optimizer 65 ms; the GPU is now 100% busy (no idle left), so only less work helps.

Microbenchmarks of the elementwise part (real packed shapes, T=13269): rotary 0.83 ms fwd + 0.97 ms
bwd per layer, RMSNorm 0.77 ms fwd + 1.47 ms bwd per norm (2 per layer), SwiGLU 0.38 + 0.63 ms.
Per step that is ~115 ms (rotary), ~287 ms (RMSNorm), ~65 ms (SwiGLU).

DeepSpeed ZeRO-3 tuning (sweep /tmp/r5/sweep, reference 2770.9 / 2755.3 ms):
* ds_a (prefetch + reduce bucket 5e8): 1946.2 ms, 63.7 GiB - no gain (the all-gather is bandwidth
  bound, not latency bound).
* ds_b (ds_a + all the parameters kept gathered: persistence threshold 1e9, max_live/max_reuse
  1e10): 1921.4 ms, 1.4378x, 74.7 GiB (+11.8 GiB, the full bf16 model stays gathered).
  The logged losses are unchanged (max rel diff 0.16% vs the frozen reference), so the numerics are
  untouched.  Baked into scripts/zero3.json (the judge uses the tree's file); the same values are
  also reachable through the LLAVA_DS_* switches.
1.8 hRound 5 - fused RMSNorm (Triton), LLAVA_FUSED_NORM=1 (20:33-20:57 UTC)
Profile finding that drove it (round 4): the elementwise part of the step is
~420 ms, of which the RMSNorm is the biggest piece (0.77 ms fwd + 1.47 ms bwd
per norm at the real packed shape, 2 norms x 32 layers + the recomputation of
the checkpointed layers).

Change: llava/fused_rmsnorm.py computes the RMSNorm of the LLaMA layers in one
Triton kernel per direction (forward; rowsum + backward; gradient of the
weight), with the same arithmetic as transformers' LlamaRMSNorm (fp32 variance,
the normalised value rounded to bf16 before the weight is applied).  It is
installed by llava/model/language_model/llava_llama.py when LLAVA_FUSED_NORM
is set; the flag is off by default.

Bug found while validating (important): the kernel that reduces the gradient of
the weight looped over a fixed chunk of 64 rows without bounding the last chunk
to the number of rows, so with 12987 rows it read 37 rows past the end of the
tensor.  The garbage turned into NaN in the master weights after the first
optimizer step (the loss became NaN at step 2 and the run died with an illegal
memory access at step 20).  Fixed by masking the rows of the last chunk.

Numerics: tier N1.  On a small model the fused norm reproduces the reference's
loss and *all* the gradients bitwise (T=65, 100, 12987 rows).  The judge's
quick check (gate 0 + the exact items of gate 2) passes: 0 exact failures, no
dtype change, no precision-mode change, tier N1.

time (/tmp/r6/time_fused): **1757.3 ms/step, speedup_vs_frozen_reference
1.5721** (the packed tree without the fused norm: 1924.6 ms/step, 1.4354x),
peak memory 74.25 GiB.  The logged losses match the packed tree (1.3829 vs
1.3839 at the first step, <0.02% afterwards).  check --quick: PASS.
2.5 hRound 9 - fused rotary embedding (Triton), LLAVA_FUSED_ROTARY=1 (21:28-21:36 UTC)
Profile finding that drove it (round 7, /tmp/r7/profile/profile_tree.md): of the 1721 ms of GPU
kernel time per step, the GEMMs are 1087 ms (63%, at the roofline, nothing to gain) and the
elementwise part is ~250 ms, of which the rotary embedding is the biggest piece: the reference
rotary costs 1.48 ms forward and 2.14 ms backward per layer at the packed shape of this recipe
(T=13269, H=32, D=128), and with the gradient checkpointing the forward runs twice, so the rotary
costs ~163 ms per step.

Change: llava/fused_rotary.py computes the rotary embedding of the query and of the key in one
Triton kernel per tensor: one pass over the data instead of a negation, a concatenation, two
multiplications and an addition that each read and write the whole tensor.  The rounding of every
bf16 intermediate is kept (the products are rounded to bf16 before they are added), so the kernel
reproduces transformers' apply_rotary_pos_emb bitwise, in the forward and in the backward
(checked on the real shapes: T=13269, 1992, 4096, 100 -> max difference 0).  The kernel is
installed by the packed attention of llava/model/language_model/llava_llama.py when
LLAVA_FUSED_ROTARY is set; the flag is off by default.

Numerics: tier N0 for this change (the rotary is bitwise identical to the reference's).

time (/tmp/r9/time_rotary): 1635.7 ms/step, speedup_vs_frozen_reference 1.6889 (the previous
tree: 1757.3 ms/step, 1.5721), peak memory 74.26 GiB (unchanged), no problems.
check --quick (/tmp/r9/quick_rotary): PASS (gate 0 + the exact items of gate 2).
Committed as 65495a4.  /submission/switches.json = pack + fused norm + fused rotary.

### Failed direction: selective gradient checkpointing (the last layers keep their activations)
Idea: the layers that are not checkpointed are not recomputed during the backward, which is a
whole extra forward of the model.  With the last 8-12 layers un-checkpointed the timing run went
from 1757 ms to 1622 ms/step (1.7027x) and the peak memory rose from 74.25 to 98.62 GiB.
The implementation hooks transformers' _set_gradient_checkpointing and lets the last layers of
the LLaMA run without the checkpoint (llava/selective_ckpt.py, switch LLAVA_CKPT_SKIP_LAST).

It deadlocks: the run hangs in the ZeRO-3 backward, both ranks spinning in a NCCL collective
(py-spy: the autograd worker thread of both ranks is busy inside the backward; the GPU is at 100%
but no step finishes).  Two runs hung the same way (the first one with a version whose counter was
also counting the vision tower's layers, the second one with the counter fixed to count only the
LLaMA decoder layers).  The same tree without the switch runs fine, and an earlier variant of the
idea (which un-checkpointed the *first* layers, because of the same counter bug) did finish 24
steps, so the deadlock is not deterministic - it smells like the ZeRO-3 gradient reduction being
issued in a different order on the two ranks when the order in which the layers' gradients become
ready changes.  Not worth the risk: the change is left out of the tree (llava/selective_ckpt.py
is not imported by anything).

### Full check of the fused-rotary tree (21:57-22:04 UTC)
check (/tmp/r10/check_rotary/check.json): status PASS, speedup 1.6923 (candidate 1632.4
ms/step vs the frozen reference 2762.6 ms/step), tier N1, gate 2: 0 exact failures, 0 dtype
changes, 0 precision-mode changes (4724 tensors checked, 32 bitwise identical, 7 beyond the
diagnostic tolerances - gate 3 decides), gate 3: passed, peak memory 74.25 GiB (reference
73.5), window spread 1.16%, the logged losses are within 0.15% of the reference's.
3.5 hRound 11 - fused AdamW (torch's fused implementation), LLAVA_FUSED_ADAM=1 (22:12-22:36 UTC)
Profile finding that drove it (round 10, /tmp/r11/profile/profile_tree.md): the AdamW update is
65.2 ms of the 1585 ms of GPU kernel time per step (4.1%), spent in multi_tensor_apply with
one kernel per tensor (the LLaMA-7B model has 686 parameter tensors and the ZeRO-3 partitions
are flat per sub-group, so the update is done on ~10 flat tensors - but each of them goes
through the multi-tensor path).

Change: llava/train/llava_trainer.py::enable_fused_adamw wraps torch.optim.AdamW.step so
that the parameter group is marked as fused before the update runs.  DeepSpeed's ZeRO-3 keeps
the client optimizer (torch.optim.AdamW, created by the HF Trainer) and calls it once per
sub-group with the flat fp32 partition (DeepSpeedZeroOptimizer_Stage3._optimizer_step), so the
fused implementation of AdamW is used with exactly the same hyperparameters (lr, betas, eps,
weight decay) and the same fp32 master weights and fp32 Adam state - only the kernels differ
(one fused kernel instead of the multi-tensor ones).  The flag has to be set at step time and
not at creation time because DeepSpeed replaces the parameters with the fp32 GPU partitions
after the optimizer is built.  The optimizer class is unchanged (torch.optim.AdamW), so gate 2's
"Adam hyperparameters" item is untouched.

Numerics: tier N1 (the tree was already N1 because of the packed attention).

time (/tmp/r11/time_adam4): 1621.4 ms/step, speedup_vs_frozen_reference 1.7039 (the previous
tree: 1635.7 ms/step, 1.6889), +0.9%.  The logged losses match the reference
(1.3829 / 1.2879 / 1.3170 / 0.2573 / 1.2641 vs the reference's 1.3839 / 1.2880 / 1.3171 / 0.2572 /
1.2638).  check --quick (/tmp/r11/quick_adam): PASS - gate 0 matches, 0 exact failures,
0 dtype changes, 0 precision-mode changes, and the tensor comparison is the same as the
previous tree's (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances), so
the fused update did not make the numerics any worse.

/submission/switches.json = pack + fused norm + fused rotary + fused adam.
3.7 hRound 12 - partial recompute of the gradient checkpoint (cache the MLP's gate/up), LLAVA_PARTIAL_MLP=1 (22:38-22:47 UTC)
Profile finding that drove it (/tmp/r11/profile/profile_tree.md): of the 1585 ms of GPU kernel time
per step, the GEMMs are 1119.6 ms (70.6%) and about a quarter of them are the *recompute* of the
gradient checkpoint (the forward of the decoder layer is computed a second time in the backward
pass: 404.9 MFLOP per token and layer, of which the MLP's two up-projections are 180.4 MFLOP,
i.e. 44.6%).  The recompute of the two projections is pure waste: it computes exactly the numbers
that the forward pass already computed.

Change: llava/partial_mlp_cache.py.  The forward pass of the checkpoint (which runs under
torch.no_grad()) keeps the result of the gate and the up projection; the recompute (which runs
under torch.enable_grad()) reuses it and skips the matrix multiplication.  The projections are
still called through their module, so every hook of DeepSpeed's ZeRO-3 fires at exactly the same
place and the trace of the modules is unchanged - only the matrix multiplication is skipped.  The
gradient of the weight is computed by the backward of the projection itself, with the same two
matrix multiplications as the backward of the linear layer (grad_input = mm(grad, W),
grad_weight = mm(x^T, grad)^T).

Numerics: tier N0 for this change.  Checked on the real shapes (T=13269, K=4096, N=11008, bf16,
on the GPU): the output of the layer, the gradient of the input and the gradients of the three
weights are bitwise identical to the ones of the reference.  The judge's check --quick:
PASS, and the tensor comparison is the same as the previous tree's (4724 tensors, 32 bitwise
identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).

time (/tmp/r12/time_partial): 1518.5 ms/step, speedup_vs_frozen_reference 1.8192 (the
previous tree: 1621.4 ms/step, 1.7039), +6.8%.  Peak memory 93.82 GiB (the reference 73.5, the
previous tree 74.25): the two projections of the 32 layers are kept alive, 2 x 13269 x 11008 x 2
bytes x 32 = 18.7 GB more.

/submission/switches.json = pack + fused norm + fused rotary + fused adam + partial mlp.
3.8 hRound 13 - cache the down projection too (skip its recompute), LLAVA_PARTIAL_MLP=1 (22:47-22:53 UTC)
Change: the same mechanism as round 12, extended to the third projection of the MLP.  The
recompute of a layer now only does the elementwise product of the MLP (silu(gate) * up); the
matrix multiplications of the gate, the up and the down projection are all skipped.

Numerics: tier N0 (checked on the real shapes on the GPU: the output of the layer and the
gradients of the input and of the three weights are bitwise identical to the reference's).
check --quick (/tmp/r13/quick_down): PASS, the tensor comparison is unchanged (4724 tensors,
32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).

time (/tmp/r13/time_down): 1457.7 ms/step, speedup_vs_frozen_reference 1.8952 (the previous
tree: 1518.5 ms/step, 1.8192), +4.2%.  Peak memory 97.47 GiB (+3.7 GB: the output of the down
projection of the 32 layers).
4.1 hRound 14 (investigation) - the copies of the gradients into the ipg bucket
The profile of the tree of round 13 (/tmp/r14/profile) showed 30 ms more copies of the gradients
of the MLP into the ipg bucket of DeepSpeed than the tree of round 11 (96 extra copies of 45.1M
elements per step).  The instrumentation of DeepSpeedZeroOptimizer_Stage3.__add_grad_to_ipg_bucket
(a copy of the tree in /tmp/dbg/llava, LLAVA_DEBUG_IPG) shows that *every* parameter is added to
the ipg bucket twice, with and without the cache of the projections: it is a behaviour of
DeepSpeed (the hook on the accumulation of the gradient fires again when the gradient of the
parameter is accumulated a second time) and not something the cache introduces.  Making the
gradient of the weight contiguous (.contiguous()) changes nothing (1460 ms instead of 1457 ms),
so the change was reverted.
4.1 hRound 15 - cache the query, key and value projections too, LLAVA_PARTIAL_MLP=1 (23:05-23:11 UTC)
Profile finding that drove it (/tmp/r14/profile/profile_tree.md): the GEMMs are 930.3 ms of the
1429 ms of GPU kernel time per step (65%).  Of the matrix multiplications that are still done
twice (the forward pass and the recompute of the gradient checkpoint), only the attention is left:
the query, the key and the value projections are 100.7 of the 134.2 MFLOP per token and layer, so
they are 10% of the matrix multiplications of a step (about 70 ms).

Change: the same mechanism as rounds 12 and 13, applied to the query, the key and the value
projection of every layer.  The recompute of a layer now only does the rotary embedding, the
attention itself and the output projection.

Numerics: tier N0 (the output of the projection and the gradients of its input and of its weight
are bitwise identical to the ones of the reference; the same three matrix multiplications as the
backward of the linear layer).  check --quick (/tmp/r15/quick_attn): PASS, the tensor comparison
is unchanged (4724 tensors, 32 bitwise identical, 7 beyond the diagnostic tolerances, 0 exact
failures, 0 dtype changes).

time (/tmp/r15/time_attn): 1401.0 ms/step, speedup_vs_frozen_reference 1.9719 (the previous
tree: 1457.7 ms/step, 1.8952), +4.0%.  Peak memory 108.39 GiB (the reference 73.5): the query,
the key and the value of the 32 layers are kept alive, 3 x 13269 x 4096 x 2 bytes x 32 = 10.4 GB
more.
4.2 hRound 16 - do not copy the gradients into the flat ipg bucket, LLAVA_PARTIAL_MLP=1 (23:11-23:17 UTC)
Profile finding that drove it (/tmp/r14/profile/profile_tree.md): 80.5 ms of the step is spent in
copies of the gradients.  DeepSpeed's ZeRO-3 copies every gradient into a flat buffer
(__add_grad_to_ipg_bucket) and only reduces that buffer when reduce_scatter is off; with
reduce_scatter on (the default) the reduction reads the gradients of the parameters one by one
(__avg_scatter_grads), so the copies into the flat buffer are never used.

Change: scripts/zero3.json sets "contiguous_gradients": false.  The reduction of the
gradients is the same code path as before (the flat buffer is not used either way), the gradients
are read where they are, and the flat buffer (1 GB) is not allocated at all.

Numerics: tier N0 (the same gradients are reduced, in the same order).

time (/tmp/r16/time_nocontig): 1395.4 ms/step, speedup_vs_frozen_reference 1.9797 (the
previous tree: 1401.0 ms/step, 1.9719).  Peak memory 107.46 GiB (-0.93 GB).
check --quick (/tmp/r16/quick_nocontig): PASS, the tensor comparison is unchanged.
4.4 hRound 17 - cache the output projection of the attention, LLAVA_CACHE_OPROJ=1 (23:19-23:33 UTC)
Profile of the tree of round 16 (/tmp/r17/profile/profile_tree.md): 1356 ms of GPU kernel time per
step, of which 862 ms are the matrix multiplications (63%), 113 ms are plain copies (8%), 85 ms the
flash attention, 77 ms the NCCL collectives, 56 ms the fused AdamW and ~100 ms the elementwise
operations.  The GPU is idle 153 ms of the 1477 ms of the profiled step (one gap of 59 ms at the
end of the backward, where the CPU prepares the optimizer step); in the timed run the bubble is
~70 ms of 1395 ms.

Profile finding that drove it: after the caches of rounds 12 to 15, the only matrix multiplication
of a layer that was still computed twice was the output projection of the attention (the recompute
of the checkpoint computed it again).  The GPU time of the copies of the gradients and of the
parameters is not reducible from our side (it is DeepSpeed's ZeRO-3), and the collectives
(46.7 ms of AllGather of the parameters, 30 ms of ReduceScatter of the gradients) are the price of
the ZeRO-3 of the recipe.

Change: the output projection of the attention is cached like the other projections of the layer
(llava/partial_mlp_cache.py, switch LLAVA_CACHE_OPROJ), so the recompute of the checkpoint
takes its result instead of computing the matrix multiplication again.

Numerics: tier N0 (the same mechanism as the other projections: the result of the forward pass is
reused and the gradients of the input and of the weight are computed with the same two matrix
multiplications as the backward of the linear layer).  check --quick (/tmp/r17/quick_oproj):
PASS, the tensor comparison is unchanged (4724 tensors, 32 bitwise identical, 7 beyond the
diagnostic tolerances, 0 exact failures, 0 dtype changes).

time (/tmp/r17/time_oproj): 1378.1 ms/step, speedup_vs_frozen_reference 2.0047 (the previous
tree: 1395.4 ms/step, 1.9797), +1.3%.  Peak memory 111.1 GiB (+3.6 GB: the output of the output
projection of the 32 layers).
4.5 hRound 18 - the output layer only on the supervised positions, LLAVA_SUP_LOGITS=1 (23:33-23:40 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): the matrix multiplications are
862 ms of the 1356 ms of GPU kernel time per step.  Of those, the output layer (the vocabulary
projection) is 10.7 TFLOP (2% of the matrix multiplications of the step, about 18 ms), and it is
applied to *every* position of the sequence, while only the tokens of the answer are supervised:
the image tokens (576 per image) and the prompt are masked.  The logits of the whole sequence are
also the biggest tensors of the step (13600 x 32000, 1.7 GB in fp32).

Change: in the packed path the transformer returns the hidden states and the output layer is
applied to the hidden states of the supervised positions only (`llava/model/language_model/
llava_llama.py, switch LLAVA_SUP_LOGITS`).  The loss is still the mean cross entropy over the
same supervised tokens, and model.forward(labels=...) still returns it.

Numerics: tier N1 (the output layer is a matrix multiplication with fewer rows, so the rounding
of the logits can differ in the last bit; the loss and the gradients are the same to about 1e-7).
check --quick (/tmp/r18/quick_sup): PASS, the tensor comparison is unchanged (7 tensors beyond
the diagnostic tolerances, the same ones as before, 0 exact failures, 0 dtype changes).

time (/tmp/r18/time_sup): 1354.4 ms/step, speedup_vs_frozen_reference 2.0397 (the previous
tree: 1378.1 ms/step, 2.0047), +1.7%.  Peak memory 110.49 GiB (-0.6 GB).
4.7 hRound 19 - fused SwiGLU, LLAVA_FUSED_SWIGLU=1 (23:40-23:50 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): the elementwise operations of the
MLP are the biggest remaining item that is not a matrix multiplication: the SiLU (13.7 ms), the
multiplication of the SiLU and the up projection (28.4 ms of the multiplications of the step) and
the SiLU of the backward (9.0 ms).  The chain reads and writes the two big tensors of the layer
(the gate and the up projection, 13600 x 11008) four times.

Change: llava/fused_swiglu.py computes silu(gate) * up in one Triton kernel and its backward
in one more, and the MLP of the LLaMA layers uses it (switch LLAVA_FUSED_SWIGLU).  The
arithmetic is the one of PyTorch, with the rounding of every intermediate result kept: the SiLU
is x / (1 + exp(-x)) in fp32, rounded to bf16, the product is rounded to bf16, and the backward
is d_up = d_out * silu(gate), d_silu = d_out * up (rounded to bf16) and
d_gate = d_silu * s * (1 + gate * (1 - s)) with s = 1 / (1 + exp(-gate)).

Two details of Triton matter for the exactness: tl.exp is the fast ex2.approx and differs
from the expf of PyTorch (48% of the values), so the kernel uses tl.math.exp (the libdevice
__nv_expf); and the division of the sigmoid uses tl.math.div_rn.

Numerics: tier N0.  Checked on the real shapes (13600 x 11008) on the GPU: the result and the
two gradients are bitwise identical to the ones of PyTorch (0 differing elements out of 149.7M).
check --quick (/tmp/r19/quick_swiglu): PASS, the tensor comparison is unchanged (7 tensors
beyond the diagnostic tolerances, the same ones as before, 0 exact failures, 0 dtype changes).

time (/tmp/r19/time_swiglu): 1337.6 ms/step, speedup_vs_frozen_reference 2.0653 (the
previous tree: 1354.4 ms/step, 2.0397), +1.3%.  Peak memory 110.5 GiB (unchanged; the kernel
does not need the result of the SiLU to be kept for the backward).
4.8 hRound 20 - no copy of the gradients of the rotary, LLAVA_FUSED_ROTARY=1 (23:51-23:58 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): 113 ms of the step are plain
copies.  64 of them are the copies of the gradients of the rotary embedding: the backward of the
fused rotary made the gradients contiguous before reading them, but the gradient of the query
arrives with the strides of the transposed view of the attention (the transpose of the backward of
the transpose of the forward), so the copy was real: two copies of 111 MB per layer and per
backward (14 GB of traffic per step, about 5 ms).

Change: llava/fused_rotary.py reads the gradients with their own strides (the kernel already
takes the strides of its arguments), so the copies are not needed any more.

Numerics: tier N0 (the same values are read, only the copy is gone).  Checked on the GPU with
the real shapes: the gradient of the query is bitwise identical to the one of the reference
implementation of the rotary (the strides of the gradient are the same too).
check --quick (/tmp/r20/quick_rotary): PASS, the tensor comparison is unchanged (7 tensors
beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).

time (/tmp/r20/time_rotary): 1331.2 ms/step, speedup_vs_frozen_reference 2.0753 (the
previous tree: 1337.6 ms/step, 2.0653), +0.5%.  Peak memory 110.5 GiB (unchanged).
5.0 hRound 21 - keep the result of the attention for the recompute, LLAVA_CACHE_FLASH=1 (23:57-00:05 UTC)
Profile finding that drove it (/tmp/r17/profile/profile_tree.md): after the caches of the
projections (rounds 12 to 17), the recompute of the checkpoint still ran the attention of every
layer a second time: 33.2 ms of the step are the forward of flash-attention (64 calls, one per
layer and per pass), half of which is the recompute.

Change: llava/flash_cache.py keeps the result of the attention of the forward pass (the output,
the log-sum-exp of the softmax and the state of the random number generator) and the recompute
reuses it: the forward of the recompute returns the result of the forward pass and the backward
calls the backward of flash-attention with the tensors it saves (the query, the key, the value,
the output and the log-sum-exp), with the same arguments as the reference.

Numerics: tier N0.  Checked on the GPU with the real shapes (13600 tokens, 32 heads): the output,
the gradient of the key and the gradient of the value are bitwise identical to the ones of
flash-attention's own function.  The gradient of the query is the one of flash-attention's
backward, which accumulates with atomic operations when it is not deterministic (the reference
gives a different result on two runs as well); in the deterministic mode (the mode of the gate 2
of the judge) the two are bitwise identical.
check --quick (/tmp/r21/quick_flash): PASS, the tensor comparison is unchanged (7 tensors
beyond the diagnostic tolerances, 0 exact failures, 0 dtype changes).

time (/tmp/r21/time_flash): 1314.5 ms/step, speedup_vs_frozen_reference 2.1016 (the
previous tree: 1331.2 ms/step, 2.0753), +1.3%.  Peak memory 114.25 GiB (+3.75 GB: the output of
the attention of the 32 layers is kept until the recompute).
5.2 hRound 22 - full check of the tree of round 21: ERROR (00:05-00:12 UTC)
Command: bench.py check --tree /submission/llava --config /submission/switches.json --out /tmp/r22/check_full

* gate 0: PASS.  gate 1: PASS - 1319.1 ms/step, speedup 2.0943 (the frozen reference is
  2762.6 ms/step), peak memory 114.25 GiB.
* gate 2: NOT_BITWISE with 0 exact failures (7 tensors beyond the diagnostic tolerances, the
  same ones as in every round since the packed attention), 0 dtype changes.
* gate 3: ERROR - "trajectory seed 0 exited 1".  The trajectory died at step 41 with
  RuntimeError: shape '[1, 14122, 32, 128]' is invalid for input of size 60534784 in
  llava_llama.py:67 (the view of the query of the packed attention).

Root cause (found by reading the traceback): the values that the projections keep for the
recompute of the checkpoint were kept per projection, without saying which forward pass had
computed them.  The bench makes a forward pass of its own before every training step (the
forward that measures the loss of the batch) and, at the end of every step of the window of
the probe, another one with the probe batch (32 held out rows).  Those passes end without a
recompute, so their values stayed in the cache; the *next* training pass found them and took
them for its own recompute - the values of another batch, with another number of tokens.  The
first step of the probe window (step 41) is the first one whose micro-batch has another number
of tokens than the probe batch, and the shapes did not match: the crash.
5.2 hRound 23 - the values of the projections belong to the pass that computed them (00:12-00:20 UTC)
Change: llava/partial_mlp_cache.py gives every forward pass of the model a new generation
of the values (a counter, and the values of the previous generation are dropped), and a
recompute only takes the values of the generation that is running.  The values of a pass that
ends without a recompute are dropped with it, so they cannot be used by another pass.

Numerics: N0 for the values (the same values are computed and used; only the values of another
pass are not used any more).

check --n1 (/tmp/r23/n1, gates 0 and 3): PASS.  The trajectory of the gate 3 ends with
a bias of -0.12 sigma (the canonical loss of the last 20 steps) and -0.03 sigma (the probe
loss), the bound is 3 sigma.

time (/tmp/r23/time): 1318.5 ms/step, speedup_vs_frozen_reference 2.0953 (the tree of round
21: 1314.5 ms/step, 2.1016 - the same, the fix does not cost anything in the timing run).
5.4 hRound 24 - cheaper release of the gathered parameters of ZeRO-3, LLAVA_FAST_ZERO3_RELEASE=1 (00:23-00:26 UTC)
Profile finding that drove it (/tmp/r17/profile, the trace of the step): the GPU is idle for
58.8 ms of the step, right after the backward, and the CPU is walking the module tree of the
model 13 686 times: it is DeepSpeedZeroOptimizer_Stage3.step -> _partition_all_parameters ->
PartitionedParameterCoordinator.release_and_reset_all, which releases the 686 parameters that
are still gathered.  The release walks the module tree three times per step and asks the process
for its rank 2 744 times to decide whether a message is printed, although the message is only
printed with force=True and is never printed in the training loop.

Change: llava/zero3_release.py keeps the parameters of a module (they do not change while the
model is trained) instead of walking the module tree again, and does not ask for the rank when
the message cannot be printed.  The same parameters are released, in the same order, with the
same calls.

Numerics: tier N0 (nothing is computed differently).
time (/tmp/r24/time): 1306.0 ms/step, speedup_vs_frozen_reference 2.1153 (the previous
tree: 1318.5 ms/step, 2.0953), +1.0%.  Peak memory 114.25 GiB (unchanged).
5.4 hRound 25 - the gradient of the weight of a projection in its own layout, LLAVA_CONTIG_GRAD=1 (00:26-00:30 UTC)
Profile finding that drove it (/tmp/r17/profile): 41 ms of the step are the copies of the
gradients of the weights of the projections (the shapes [11008, 4096], [4096, 4096] and
[4096, 11008] of the trace).  The gradient of the weight is computed by the backward of the
matrix multiplication in the transposed layout and is copied into the gradient of the parameter;
the copy of a transposed tensor reads the memory in the wrong order (96 GB/s against 3.9 TB/s
for a contiguous copy on this GPU).

Change: the backward of the cached projection computes the gradient of the weight with the
transposed multiplication (mm(grad.t(), x) instead of mm(x.t(), grad).t()), which gives the
same numbers in the layout of the weight.  Checked on the GPU with the real shapes (13600
tokens): not one element of the 45M differs, for the three shapes of the model.

Numerics: tier N0 (the values are bitwise identical, only the layout of the gradient changes).
time (/tmp/r25/time): 1266.6 ms/step, speedup_vs_frozen_reference 2.1811 (the previous
tree: 1306.0 ms/step, 2.1153), +3.0%.  Peak memory 114.25 GiB (unchanged).
5.5 hRound 26 - profile of the tree of round 25 (00:30-00:31 UTC)
Command: bench.py profile --tree /submission/llava --config /submission/switches.json --out /tmp/r26/profile --detail
GPU kernel time 1232 ms/step (the tree of round 17: 1356 ms/step).  The GEMMs are 838 ms (68%)
and are at the roofline of the shapes (774 TFLOP/s measured on the GPU for the shape of the
gradient of a weight); the rest is: the fused AdamW 56 ms, the backward of flash-attention 53 ms,
the AllGather 35.5 ms, the ReduceScatter 31.2 ms, the residual adds 27.6 ms (384 calls at
4.2 TB/s, at the roofline), the cats 21.7 ms, the copies 35.6 ms, the fused SwiGLU 29 ms, the
fused RMSNorm 20.8 ms, the fused rotary 13.6 ms.  The GPU is idle for 35 ms of the step.
5.5 hRound 27 - release the gathered parameters at the end of the optimizer step, LLAVA_DEFER_ZERO3_RELEASE=1 (00:31-00:36 UTC)
Profile finding that drove it (/tmp/r26/profile): the release of the gathered parameters is CPU
work that the GPU waits for (the GPU is idle for 35 ms of the step, the release is most of it).
The parameters that are gathered are not needed by the optimizer step: the optimizer works on
the fp32 master weights and on the fp32 gradients.  Releasing them at the end of the step lets
the GPU update the parameters while the CPU releases the ones of the next forward pass.

Change: llava/zero3_release.py defers _partition_all_parameters to the end of the optimizer
step (the release is done before the gather of the persistent parameters, in the same order as
before); a step that ends without a release (the loss scale overflowed) releases the parameters
at the beginning of the next step.

Numerics: tier N0 (the same parameters are released, in the same order, with the same calls;
only the moment of the release changes, and the values of the parameters are refreshed by the
gather that follows it, as before).
time (/tmp/r27/time): 1247.3 ms/step, speedup_vs_frozen_reference 2.2148 (the previous
tree: 1266.6 ms/step, 2.1811), +1.5%.  Peak memory 114.25 GiB (unchanged: the peak is in the
forward pass, not in the optimizer step).
5.7 hRound 28 - full check of the tree of round 27 (00:37-00:47 UTC)
Command: bench.py check --tree /submission/llava --config /submission/switches.json --out /tmp/r28/check_full

* gate 0: PASS.  gate 1: PASS - 1249.8 ms/step, speedup 2.2103 (the frozen reference is
  2762.6 ms/step), peak memory 114.25 GiB.
* gate 2: NOT_BITWISE with 0 exact failures (7 tensors beyond the diagnostic tolerances, the
  same ones as in every round since the packed attention), 0 dtype changes, 0 precision mode
  changes -> tier N1.
* gate 3: PASS - the canonical loss of the last 20 steps is -0.10 sigma, the probe loss is
  -0.03 sigma (the bound is 3 sigma).

The tree of the round is the one of the round 27 (commit fa7cfb8): it is the state that is
judged at the end, unless a later round is better.
6.4 hRound 30 - a Triton kernel for the fused AdamW, LLAVA_TRITON_ADAM=1 (01:12-01:25 UTC)
Profile finding that drove it (/tmp/r29/profile): the fused AdamW of PyTorch moves 28 bytes
per parameter (the gradient, the two moments, the parameter) and reaches 1.7 TB/s on this
GPU; the kernel takes 56 ms of the step.

Change: llava/fused_adamw.py is a Triton kernel that does the same update as PyTorch's
fused AdamW (the same arithmetic, the same bias corrections, the same rounding) and is
installed in place of torch._fused_adamw_; the state, the counter of the steps and the
bias corrections are still PyTorch's.  The kernel takes 22 ms of the step (4.2 TB/s, the
speed of a copy).

Numerics: tier N1 (the update is PyTorch's fused AdamW up to the last bit of the rounding of
the arithmetic of the kernel; the parameters differ by at most 4e-11).
time (/tmp/r30/time_triton): 1236.5 ms/step, speedup_vs_frozen_reference 2.2341 (the
tree without the kernel, measured just before: 1255.5 ms/step, 2.2004).
check --quick (/tmp/r30/quick): PASS, tier N1, 0 exact failures, 0 dtype changes.
Committed as 09e5bad.
6.4 hRound 31 - skip the work of the optimizer step that nothing uses (01:26-01:32 UTC)
Profile finding that drove it (/tmp/r29/profile): the optimizer step of ZeRO-3 spends 26 ms
in the computation of the norm of the gradients (get_grad_norm_direct: the gradients are
copied into a buffer of double precision and reduced there, 16.3 ms of copies and 8.7 ms of
reduction) and 6.3 ms in the unscale of the gradients (grad.mul_(1/loss_scale)).

Both are dead work: the norm is only used to clip the gradients and the recipe does not
clip them (scripts/zero3.json does not set gradient_clipping, so clip_grad is zero and
unscale_and_clip_grads never scales the gradients with the norm; the value is kept in
optimizer._global_grad_norm, which nothing reads), and with bf16 the loss scale is one, so
the unscale multiplies the gradients by one.

Change: llava/fast_grad_norm.py and llava/fast_grad_unscale.py patch the two methods of
DeepSpeedZeroOptimizer_Stage3 (LLAVA_FAST_GRAD_NORM, LLAVA_FAST_GRAD_UNSCALE).

Numerics: tier N0 (the gradients, the update of the parameters, the state of the optimizer
and the loss scale are unchanged; only a value that nothing reads is not computed).
time (/tmp/r31/time_norm): 1218.7 ms/step, speedup_vs_frozen_reference 2.2669 (the
tree without the change: 1236.5 ms/step, 2.2341).
Committed as 3f3e769.
6.5 hRound 32 - make the release of the gathered parameters cheaper, LLAVA_FAST_PARAM_RELEASE=1 (01:32-01:41 UTC)
Profile finding that drove it (/tmp/r29/profile): the GPU is idle for 43.5 ms in one gap,
between the release of the parameters that the backward pass gathered and the all-gather of
the forward pass.  The CPU work of the release is 33 ms: for every one of the 686
parameters, it builds three messages that are never printed, describes the parameter for one
of them, queries the current stream of the device (11 us) and allocates the empty tensor
that replaces the gathered one.

Change: llava/fast_param_release.py builds the messages and the description not at all,
queries the stream of the device once, and reuses the empty tensor of every dtype and
device.

Numerics: tier N0 (the same parameters are released, in the same order, with the same
calls; only the messages that are not printed and the repeated queries are avoided).
time (/tmp/r32/time_release): 1207.6 ms/step, speedup_vs_frozen_reference 2.2877 (the
tree without the change: 1218.7 ms/step, 2.2669).
check --quick (/tmp/r32/quick): PASS, tier N1, 0 exact failures, 0 dtype changes.
Committed as 1c5c927.
6.8 hRound 33 - the norm of the gradients in single precision (2026-11-13 01:40-01:55 UTC)
Profile finding that drove it: DeepSpeed's ZeRO-3 computes the norm of the gradients to clip them
(the default gradient_clipping of DeepSpeed is 1.0, so the gradients *are* clipped - the earlier
round that tried to skip the norm was a no-op).  The norm was computed with a buffer of double
precision: 16.3 ms of bf16->fp64 copies + 8.7 ms of fp64 reduction + a .item() that costs a
synchronization (a 4.8 ms gap of the GPU idle).

Change: llava/fast_grad_norm.py computes the norm with `torch.linalg.vector_norm(g, 2,
dtype=torch.float32)` (the squares of the bf16 gradients are exact in single precision) and sums
the per-parameter squares in double precision, as DeepSpeed does.  The norm of the reference is
computed instead when the result is not finite.  Tier N1 (the norm differs by ~1e-7 relative, the
gradients are scaled by it).

time (/tmp/r33/time_norm32): 1200.9 ms/step, speedup_vs_frozen_reference 2.3005 (was 1207.6 /
2.2877).  check --quick (/tmp/r33/quick): PASS (0 exact failures, 7 unresolved).  Committed b07099e.
6.8 hRound 34 - the parameters of ZeRO-3 are not persistent any more (2026-11-13 01:55-02:05 UTC)
Profile finding that drove it: the end of the optimizer step is serial.  With
stage3_param_persistence_threshold: 1e9 every parameter of the model is persistent (7 062 902 784
elements in 686 parameters; the reference has 599 040 in 312), so at the end of every step DeepSpeed
releases all the gathered parameters (17.9 ms of CPU, a 28.6 ms gap of the GPU idle) and then
re-gathers them with 686 NCCL all-gathers (38 ms of GPU time, nothing else to do while they run).

Change: LLAVA_DS_PERSIST_THRESHOLD=100000 (the value the reference's "auto" resolves to) in
/submission/switches.json.  The parameters are then gathered by the prefetcher of ZeRO-3 during the
forward and the backward, where the gathering overlaps with the computation, and the end of the step
has almost nothing left to do.  The numerics do not change (the same parameters are gathered, the
same values).

time (/tmp/r34/time_persist): 1191.0 ms/step, speedup_vs_frozen_reference 2.3197 (was 1200.9 /
2.3005).  The gain is much smaller than the 60 ms the tail seemed to cost: the gathering during the
forward and the backward is only partly hidden.

## Rounds 35-41 - the tail of the ZeRO-3 step (2026-11-13 02:05-03:00 UTC)
Profile finding that drove them: the critical path of a step is the compute stream (1084 ms of
kernels of the 1200 ms of the step); the communication (all-gather 45 ms, reduce-scatter 31 ms),
the split of the flat buffer of ZeRO-3 (86 ms) and the gradients (25 ms) run on other streams and
are completely hidden.  The GEMMs are 844 ms of the 1084 ms and run at the roofline (~818 TFLOP/s
of ~990), the recompute of the gradient checkpointing is ~207 ms of them, so what is left to win
is the ~175 ms of GPU idle (32 ms in the tail of the step, 17.5 ms in the forward of the vision
tower, which is CPU bound) and the recompute.

Four changes were tried, each behind its own switch, and each was neutral (the spread of the
reference is 0.13% and the runs differ by less than that):
* LLAVA_FAST_PARAM_GATHER (llava/fast_param_gather.py): gather every parameter of ZeRO-3 with an
  all-gather of its own instead of a cat of all of them.  time (/tmp/r38/time_gather): 1194.7
  ms/step (2.3118).
* LLAVA_FAST_PARAM_SPLIT (llava/fast_param_split.py): copy the gathered parameters out of the
  flat buffer of the gathering with one cat for the whole module instead of one copy per
  parameter.  time (/tmp/r38/time_split): 1202.2 ms/step (2.2961).
* LLAVA_NO_PERSIST_GATHER: do not gather the persistent parameters at the end of the step (the
  prefetcher gathers them anyway, while the GPU computes).  time (/tmp/r39/time_nopersist):
  1205.6 ms/step (2.2924), but that run was noisy (the two windows of the run differ by 3.7%).
* LLAVA_DS_PERSIST_THRESHOLD=100000: the parameters of ZeRO-3 are not persistent any more (the
  value the reference's "auto" resolves to).  time (/tmp/r34/time_persist): 1191.0 ms/step
  (2.3197).

sweep (/tmp/r40/sweep) with the four combinations of the last two switches: 1199.6, 1197.7,
1196.4 and 1200.8 ms/step - all inside 0.4%, that is, inside the noise of the measurement.  The
tree keeps the code of the first two (off by default) and the last two are on.

Full check (/tmp/r41/check): PASS, speedup 2.308, tier N1, 0 exact failures, 7 unresolved
(out of 4724 tensors), 32 tensors bitwise identical.  This is the best validated state so far.
↑ LLaVA