LLaVA
§ 02Models
| Model | Judged | Tier | Self-best | Rounds | Hours | API cost | |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 2.92× | N1 | 2.77× | 37 | 10.0 h | $90.4 | trace → |
| Claude Sonnet 5.5 | 2.80× | N1 | 2.74× | 18 | 10.0 h | $39.7 | trace → |
| GPT-6.2 Astra | 2.74× | N1 | 2.77× | 65 | 10.0 h | $114 | trace → |
| GPT-6.2 Sol | 2.59× | N1 | 2.58× | 54 | 10.0 h | $22.5 | trace → |
| Claude Haiku 5.5 | 2.46× | N1 | 2.47× | 36 | 10.0 h | $43.8 | trace → |
| MiniMax M3.1 Flash | 2.34× | N1 | 2.32× | 29 | 10.0 h | $5.89 | trace → |
| DeepSeek V4.1 Flash-1017 | 2.22× | N1 | 2.23× | 32 | 7.7 h | $2.58 | trace → |
| GLM-5.3 | 1.52× | N1 | 1.53× | 30 | 10.0 h | $80.8 | trace → |
| GPT-6 Astrasuspected nerf | 1.41× | N1 | 1.29× | 40 | 10.0 h | $187 | trace → |
| Kimi k3.1-pro | 1.38× | N1 | 1.39× | 35 | 10.0 h | $28.9 | trace → |
§ 03Key levers
| Model | Judged | Sequence packing | Recompute / activations | Optimizer | Communication & libraries | Other main levers |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 2.92× | ✓ round 1 LLAVA_PACK, 1.46× from this step alone | ✓ activations kept by memory budget (budget 52 → 60 GiB) | ✓ chunked Triton AdamW, kernel also writes the bf16 param copy | ✓ uses ZeRO-2; cuBLAS 12.6 (12.9's down-projection is actually slower); FA3 for packed attention (−11.8 ms) | LM head only at target positions; last layer computes only the target suffix (−12 ms); gradients switched to bidirectional send/recv rotating partition reduce (−21 ms), grad buffer written in partition order (−8.7 ms), no host sync between optimizer step and next forward (−4.1 ms); ends at ~947 ms/step, both GPUs at the ~700 W power cap |
| Claude Sonnet 5.5 | 2.80× | ✓ round 2 LLAVAOPT_PACK (plus LM head only on labeled rows), 2742.8 → 1868.6 ms | ✓ adaptive activation retention | ✓ single-pass ZeRO-3 optimizer step + pointer-table AdamW | ZeRO-2 measured 1.036× but first loss off by 0.59%, dropped; cuBLAS 12.9 (1031 → 1000 ms); resident params + async in-place weight refresh after optimizer (1000 → 978 ms), NCCL channels 6 → 32 | CLIP tower as CUDA graph, decoupled from ZeRO-3; shared system-prompt prefix computed once; partitions balanced by label count |
| Claude Haiku 5.5 | 2.46× | ✓ round 2 LLAVA_PACKED, 1.45× | ✓ expandable segments + activation budget 64 GiB (−2.3%) | ✓ LLAVA_TRITON_ADAMW, 2.2401 → 2.3216; optimizer reads/writes bf16 shards directly | ✓ ZeRO-2: round 5 2.00 → 2.19×, then intermittent NaNs in trajectories; reverted first, then overlap turned off; overlap back on in round 34 after adding event ordering to grad buckets (2.3657 → 2.4011); no cuBLAS / FlashAttention swap | skip unused grad norm (−2.2%), tuned grad bucket size |
| MiniMax M3.1 Flash | 2.34× | ✓ round 3, 1942 ms, 1.42× | — | ✓ fused AdamW | resident params (1191 ms); ZeRO-3 | fused norm / rotary / SwiGLU; logits only at supervised positions; grad layout and single-precision grad norm |
| DeepSeek V4.1 Flash-1017 | 2.22× | ✓ round 4 LLAVA_PACK, 1.28 → 1.77× | ✓ stepped from "save MLP only" to no recompute at all (adding qkv / attention saves + expandable segments) | ✓ fused AdamW on fp32 master partitions | all params resident; ZeRO-3 | installed Liger fused kernels (RMSNorm / SwiGLU / RoPE) into /opt/candidate-env; fused linear + cross-entropy; tried caching CLIP features, reverted halfway |
| GLM-5.3 | 1.52× | ✗ not done | ✓ checkpointed layers kept per memory budget (→ 1.447×) | ✓ Triton AdamW (−38 ms) | ZeRO-2 dropped as "forward is not the reference forward"; NCCL protocol and bucketing tried, no gain | fused RoPE (−190 ms), RMSNorm (−228 ms), SiLU, unpad / pad; skip grad norm; resident CLIP and cross-step prefetch (no gain) |
| GPT-6 Astrasuspected nerf | 1.41× | ✗ not done (tried "fixed-length masking", reverted) | ✓ selective checkpointing + activations kept for 9 layers | ✗ not in the log | resident params; NCCL_ALGO=Tree / PROTO=Simple; ZeRO-2 off by 0.585%, dropped | Triton RMSNorm (1.068 → 1.173×), fused SwiGLU, bf16 logits; best self-test 1.28×, final judge measured 1.41× |
| Kimi k3.1-pro | 1.38× | ✗ not done | ✓ activations kept for 22 layers (OOM with any more) | ✓ fused AdamW (1.329 → 1.333×) | ZeRO-2 off by 0.585%, dropped; NCCL protocol and bucketing no gain | torch.compile on RoPE / MLP / RMSNorm; flash-attn's Triton cross-entropy (in-place backward); bf16 logits returned, freed memory keeps one more layer |