LLaVA

ModelJudgedTierSelf-bestRoundsHoursAPI cost
Claude Opus 5.52.92×N12.77×3710.0 h$90.4trace →
Claude Sonnet 5.52.80×N12.74×1810.0 h$39.7trace →
GPT-6.2 Astra2.74×N12.77×6510.0 h$114trace →
GPT-6.2 Sol2.59×N12.58×5410.0 h$22.5trace →
Claude Haiku 5.52.46×N12.47×3610.0 h$43.8trace →
MiniMax M3.1 Flash2.34×N12.32×2910.0 h$5.89trace →
DeepSeek V4.1 Flash-10172.22×N12.23×327.7 h$2.58trace →
GLM-5.31.52×N11.53×3010.0 h$80.8trace →
GPT-6 Astrasuspected nerf1.41×N11.29×4010.0 h$187trace →
Kimi k3.1-pro1.38×N11.39×3510.0 h$28.9trace →
ModelJudgedSequence packingRecompute / activationsOptimizerCommunication & librariesOther main levers
Claude Opus 5.52.92×✓ round 1 LLAVA_PACK, 1.46× from this step alone✓ activations kept by memory budget (budget 52 → 60 GiB)✓ chunked Triton AdamW, kernel also writes the bf16 param copy✓ uses ZeRO-2; cuBLAS 12.6 (12.9's down-projection is actually slower); FA3 for packed attention (−11.8 ms)LM head only at target positions; last layer computes only the target suffix (−12 ms); gradients switched to bidirectional send/recv rotating partition reduce (−21 ms), grad buffer written in partition order (−8.7 ms), no host sync between optimizer step and next forward (−4.1 ms); ends at ~947 ms/step, both GPUs at the ~700 W power cap
Claude Sonnet 5.52.80×✓ round 2 LLAVAOPT_PACK (plus LM head only on labeled rows), 2742.8 → 1868.6 ms✓ adaptive activation retention✓ single-pass ZeRO-3 optimizer step + pointer-table AdamWZeRO-2 measured 1.036× but first loss off by 0.59%, dropped; cuBLAS 12.9 (1031 → 1000 ms); resident params + async in-place weight refresh after optimizer (1000 → 978 ms), NCCL channels 6 → 32CLIP tower as CUDA graph, decoupled from ZeRO-3; shared system-prompt prefix computed once; partitions balanced by label count
Claude Haiku 5.52.46×✓ round 2 LLAVA_PACKED, 1.45×✓ expandable segments + activation budget 64 GiB (−2.3%)✓ LLAVA_TRITON_ADAMW, 2.2401 → 2.3216; optimizer reads/writes bf16 shards directly✓ ZeRO-2: round 5 2.00 → 2.19×, then intermittent NaNs in trajectories; reverted first, then overlap turned off; overlap back on in round 34 after adding event ordering to grad buckets (2.3657 → 2.4011); no cuBLAS / FlashAttention swapskip unused grad norm (−2.2%), tuned grad bucket size
MiniMax M3.1 Flash2.34×✓ round 3, 1942 ms, 1.42×—✓ fused AdamWresident params (1191 ms); ZeRO-3fused norm / rotary / SwiGLU; logits only at supervised positions; grad layout and single-precision grad norm
DeepSeek V4.1 Flash-10172.22×✓ round 4 LLAVA_PACK, 1.28 → 1.77×✓ stepped from "save MLP only" to no recompute at all (adding qkv / attention saves + expandable segments)✓ fused AdamW on fp32 master partitionsall params resident; ZeRO-3installed Liger fused kernels (RMSNorm / SwiGLU / RoPE) into /opt/candidate-env; fused linear + cross-entropy; tried caching CLIP features, reverted halfway
GLM-5.31.52×✗ not done✓ checkpointed layers kept per memory budget (→ 1.447×)✓ Triton AdamW (−38 ms)ZeRO-2 dropped as "forward is not the reference forward"; NCCL protocol and bucketing tried, no gainfused RoPE (−190 ms), RMSNorm (−228 ms), SiLU, unpad / pad; skip grad norm; resident CLIP and cross-step prefetch (no gain)
GPT-6 Astrasuspected nerf1.41×✗ not done (tried "fixed-length masking", reverted)✓ selective checkpointing + activations kept for 9 layers✗ not in the logresident params; NCCL_ALGO=Tree / PROTO=Simple; ZeRO-2 off by 0.585%, droppedTriton RMSNorm (1.068 → 1.173×), fused SwiGLU, bf16 logits; best self-test 1.28×, final judge measured 1.41×
Kimi k3.1-pro1.38×✗ not done✓ activations kept for 22 layers (OOM with any more)✓ fused AdamW (1.329 → 1.333×)ZeRO-2 off by 0.585%, dropped; NCCL protocol and bucketing no gaintorch.compile on RoPE / MLP / RMSNorm; flash-attn's Triton cross-entropy (in-place backward); bf16 logits returned, freed memory keeps one more layer
↑ Leaderboard