NanoChat
§ 02Models
| Model | Judged | Tier | Self-best | Rounds | Hours | API cost | |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 1.21× | N1 | 1.19× | 24 | 10.0 h | $48.4 | trace → |
| Claude Opus 5.5 | 1.20× | N1 | 1.10× | 5 | 10.0 h | $78.1 | trace → |
| GPT-6.2 Astra | 1.17× | N1 | 1.17× | 17 | 10.0 h | $72.3 | trace → |
| GPT-6 Astrasuspected nerf | 1.13× | N1 | 1.21× | 27 | 10.0 h | $143 | trace → |
| MiniMax M3.1 Flash | 1.13× | N1 | 1.13× | 18 | 10.0 h | $6.56 | trace → |
| GPT-6.2 Sol | 1.10× | N1 | 1.11× | 20 | 10.0 h | $13.0 | trace → |
| Claude Haiku 5.5 | 1.08× | N1 | 1.08× | 36 | 10.0 h | $30.0 | trace → |
| DeepSeek V4.1 Flash-1017 | 1.06× | N1 | 1.06× | 87 | 9.0 h | $2.74 | trace → |
| GLM-5.3 | 1.05× | N1 | 1.05× | 16 | 10.0 h | $58.2 | trace → |
| Kimi k3.1-pro | 1.03× | N1 | 1.03× | 41 | 10.0 h | $24.4 | trace → |
§ 03Key levers
| Model | Judged | Micro-batch regrouping 4×16 → 2×32 | Scope of hand-written kernels replacing Inductor glue | Per-shape cuBLASLt algorithm selection | FP8 weights quantized once per step | Fast accum on backward GEMMs | Other main levers |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 1.21× | ✓ OPT_MB_GROUP=2, alone 1.026 | ✓ whole block: OPT_BLOCK 1.066 → 1.129, plus OPT_LOSS to 1.149 | ✓ OPT_LT_GEMM, full check 1.164 → 1.185, output bitwise unchanged | ✓ OPT_FP8_CACHE, 1.149 → 1.156 | no attempt in the log | Muon symmetric matmul computes only the upper triangle (OPT_MUON_SYRK, 1.200 → 1.207), value-embedding grads via sorted segments (OPT_EMBED_SEG, 1.209 → 1.215), params as views into stacked buffers, grads from two accumulations summed in one pass, weight grads emitted directly in fp32, async data fetch |
| Claude Opus 5.5 | 1.20× | ✓ NANOCHAT_MICROBATCH_ROWS=32, alone 1.029 | ✓ whole residual stream + MLP: NANOCHAT_FUSED_STREAM full check 1.145 → 1.174; plus QK, loss, transpose kernels | ✓ NANOCHAT_FP8_ALGOS, ~1.2% | ✓ NANOCHAT_FP8_WCACHE, ~0.7% | tried, reverted (failed statistical gate, 12σ) | patched Inductor codegen to use hardware min/max (~1.7%), each FP8 tensor written once (1.097 → 1.111), clamp before quantization removed, data-fetch buffers pinned |
| GPT-6 Astrasuspected nerf | 1.13× | ✓ NANOCHAT_MICROBATCH_MULTIPLIER=2, step 1.016 → 0.991 s | partial: one kernel each for dual-layout quantization, QK + RoPE, MLP relu²; rest left to Inductor | tried, reverted (all shapes switched uniformly to candidate 1 / candidate 3, both slower) | ✗ (considered, not done) | no attempt in the log | 1536-wide RMSNorm reduction made single-pass (NANOCHAT_PERSISTENT_REDUCTION_LIMIT=2048) with fixed 4 warps, scalar scale hoisted out of the kernel, two fewer copies in Muon |
| MiniMax M3.1 Flash | 1.13× | ✗ not in the log | partial: one kernel each for dual-layout quantization, relu², loss backward, RoPE + QK (last one 1.100 → 1.128) | ✗ (only tried a larger workspace, no change) | ✓ NANOCHAT_FP8_WEIGHT_CACHE, 1.002 → 1.014 | ✓ only on grad_input, K ≤ 6144: NANOCHAT_FP8_FASTACCUM_DGRAD, 1.032 → 1.043, statistical gate 0.09–0.30σ; both backward GEMMs on was rejected (4–15σ) | column-major activation copy saved in forward, avoiding re-quantization in backward (1.043 → 1.072), single-pass RMSNorm reduction |
| Claude Haiku 5.5 | 1.08× | ✗ not in the log | partial: MLP relu², cross-entropy, RoPE + QK (1.031 → 1.049), norm backward, norm + quantization, one at a time | ✗ | ✓ NANOCHAT_OPT_FP8_WCACHE, 1.012 → 1.021 after also caching MLP weights | tried, reverted (failed statistical gate, up to 11σ) | single-pass RMSNorm reduction, weight grads in fp32, optimizer group ordering and grouped refresh, async data fetch (+0.25%) |
| DeepSeek V4.1 Flash-1017 | 1.06× | ✓ NANOCHAT_OPT_ACCUM=2, alone 1.025 | MLP only: NANOCHAT_OPT_FUSEDMLP, full check 1.036 → 1.047, later 1.059; rewrites of norm, RoPE, attention grads all measured slower | ✗ (log says "no interface to pick another algorithm") | tried, reverted (1.043, slower than 1.050 without cache) | not tried (treated as a disallowed precision change) | Inductor pointwise kernel autotuning, persistent reduction off, Newton-Schulz via baddbmm, small grads merged into one all-reduce, optimizer issues the Muon group first |
| GLM-5.3 | 1.05× | ✓ NANOCHAT_CONCAT_MB=2, alone only 1.009 | ✗ tiled transpose op 0.978, tiled cross-entropy 1.001, both reverted | ✗ | tried, reverted (1.000, no gain) | tried, reverted (failed statistical gate, −11.95σ) | max-autotune-no-cudagraphs (1.009 → 1.023), data fetch moved to a subprocess with device check fixed (min window 1.023 → 1.047) |
| Kimi k3.1-pro | 1.03× | ✗ not in the log | ✗ native transpose op 0.688, custom cross-entropy backward 1.013 (slower when stacked on 1.025), both reverted | ✗ (only tried workspace and computing backward GEMMs transposed, the latter 0.838) | ✓ NANOCHAT_OPT_COMPILED_WEIGHT_CACHE, 1.008 | tried, reverted (all on −4 to −16σ; weight grads only −3.08σ) | Inductor coordinate-descent tuning (alone 1.015, both combined 1.025) |