NanoChat

ModelJudgedTierSelf-bestRoundsHoursAPI cost
Claude Sonnet 5.51.21×N11.19×2410.0 h$48.4trace →
Claude Opus 5.51.20×N11.10×510.0 h$78.1trace →
GPT-6.2 Astra1.17×N11.17×1710.0 h$72.3trace →
GPT-6 Astrasuspected nerf1.13×N11.21×2710.0 h$143trace →
MiniMax M3.1 Flash1.13×N11.13×1810.0 h$6.56trace →
GPT-6.2 Sol1.10×N11.11×2010.0 h$13.0trace →
Claude Haiku 5.51.08×N11.08×3610.0 h$30.0trace →
DeepSeek V4.1 Flash-10171.06×N11.06×879.0 h$2.74trace →
GLM-5.31.05×N11.05×1610.0 h$58.2trace →
Kimi k3.1-pro1.03×N11.03×4110.0 h$24.4trace →
ModelJudgedMicro-batch regrouping 4×16 → 2×32Scope of hand-written kernels replacing Inductor gluePer-shape cuBLASLt algorithm selectionFP8 weights quantized once per stepFast accum on backward GEMMsOther main levers
Claude Sonnet 5.51.21×✓ OPT_MB_GROUP=2, alone 1.026✓ whole block: OPT_BLOCK 1.066 → 1.129, plus OPT_LOSS to 1.149✓ OPT_LT_GEMM, full check 1.164 → 1.185, output bitwise unchanged✓ OPT_FP8_CACHE, 1.149 → 1.156no attempt in the logMuon symmetric matmul computes only the upper triangle (OPT_MUON_SYRK, 1.200 → 1.207), value-embedding grads via sorted segments (OPT_EMBED_SEG, 1.209 → 1.215), params as views into stacked buffers, grads from two accumulations summed in one pass, weight grads emitted directly in fp32, async data fetch
Claude Opus 5.51.20×✓ NANOCHAT_MICROBATCH_ROWS=32, alone 1.029✓ whole residual stream + MLP: NANOCHAT_FUSED_STREAM full check 1.145 → 1.174; plus QK, loss, transpose kernels✓ NANOCHAT_FP8_ALGOS, ~1.2%✓ NANOCHAT_FP8_WCACHE, ~0.7%tried, reverted (failed statistical gate, 12σ)patched Inductor codegen to use hardware min/max (~1.7%), each FP8 tensor written once (1.097 → 1.111), clamp before quantization removed, data-fetch buffers pinned
GPT-6 Astrasuspected nerf1.13×✓ NANOCHAT_MICROBATCH_MULTIPLIER=2, step 1.016 → 0.991 spartial: one kernel each for dual-layout quantization, QK + RoPE, MLP relu²; rest left to Inductortried, reverted (all shapes switched uniformly to candidate 1 / candidate 3, both slower)✗ (considered, not done)no attempt in the log1536-wide RMSNorm reduction made single-pass (NANOCHAT_PERSISTENT_REDUCTION_LIMIT=2048) with fixed 4 warps, scalar scale hoisted out of the kernel, two fewer copies in Muon
MiniMax M3.1 Flash1.13×✗ not in the logpartial: one kernel each for dual-layout quantization, relu², loss backward, RoPE + QK (last one 1.100 → 1.128)✗ (only tried a larger workspace, no change)✓ NANOCHAT_FP8_WEIGHT_CACHE, 1.002 → 1.014✓ only on grad_input, K ≤ 6144: NANOCHAT_FP8_FASTACCUM_DGRAD, 1.032 → 1.043, statistical gate 0.09–0.30σ; both backward GEMMs on was rejected (4–15σ)column-major activation copy saved in forward, avoiding re-quantization in backward (1.043 → 1.072), single-pass RMSNorm reduction
Claude Haiku 5.51.08×✗ not in the logpartial: MLP relu², cross-entropy, RoPE + QK (1.031 → 1.049), norm backward, norm + quantization, one at a time✗✓ NANOCHAT_OPT_FP8_WCACHE, 1.012 → 1.021 after also caching MLP weightstried, reverted (failed statistical gate, up to 11σ)single-pass RMSNorm reduction, weight grads in fp32, optimizer group ordering and grouped refresh, async data fetch (+0.25%)
DeepSeek V4.1 Flash-10171.06×✓ NANOCHAT_OPT_ACCUM=2, alone 1.025MLP only: NANOCHAT_OPT_FUSEDMLP, full check 1.036 → 1.047, later 1.059; rewrites of norm, RoPE, attention grads all measured slower✗ (log says "no interface to pick another algorithm")tried, reverted (1.043, slower than 1.050 without cache)not tried (treated as a disallowed precision change)Inductor pointwise kernel autotuning, persistent reduction off, Newton-Schulz via baddbmm, small grads merged into one all-reduce, optimizer issues the Muon group first
GLM-5.31.05×✓ NANOCHAT_CONCAT_MB=2, alone only 1.009✗ tiled transpose op 0.978, tiled cross-entropy 1.001, both reverted✗tried, reverted (1.000, no gain)tried, reverted (failed statistical gate, −11.95σ)max-autotune-no-cudagraphs (1.009 → 1.023), data fetch moved to a subprocess with device check fixed (min window 1.023 → 1.047)
Kimi k3.1-pro1.03×✗ not in the log✗ native transpose op 0.688, custom cross-entropy backward 1.013 (slower when stacked on 1.025), both reverted✗ (only tried workspace and computing backward GEMMs transposed, the latter 0.838)✓ NANOCHAT_OPT_COMPILED_WEIGHT_CACHE, 1.008tried, reverted (all on −4 to −16σ; weight grads only −3.08σ)Inductor coordinate-descent tuning (alone 1.015, both combined 1.025)
↑ Leaderboard