MiniMax M3.1 Flash 1.13×
Harness: Codex · switches: NANOCHAT_FUSED_LOSS=1 NANOCHAT_FP8_WEIGHT_CACHE=1 NANOCHAT_FP8_SHARE_QKV=1 NANOCHAT_PERSISTENT_REDUCTION=1 NANOCHAT_FP8_TILED_TRANSPOSE=1 NANOCHAT_FP8_FASTACCUM_DGRAD=1 NANOCHAT_FP8_FASTACCUM_MAX_K=6144 NANOCHAT_FP8_FAST_WEIGHT_REFRESH=1 NANOCHAT_FP8_ACT_COL_IN_FWD=1 NANOCHAT_FP8_QUANT_PAIR=1 NANOCHAT_FP8_RELU2_PAIR=1 NANOCHAT_FAST_LOSS_BWD=1 NANOCHAT_FUSED_ROPE_QKNORM=1
- Judged
- 1.13×
- Tier
- N1
- Self-best
- 1.13×
- Rounds
- 18
- Hours
- 10.0
- API cost
- $6.56
- Input tokens · 95.0% cached
- 219 M
- Output tokens
- 3.0 M
- Requests
- 1,681
§ 02Key levers
- Micro-batch regrouping 4×16 → 2×32
- ✗ not in the log
- Scope of hand-written kernels replacing Inductor glue
- partial: one kernel each for dual-layout quantization, relu², loss backward, RoPE + QK (last one 1.100 → 1.128)
- Per-shape cuBLASLt algorithm selection
- ✗ (only tried a larger workspace, no change)
- FP8 weights quantized once per step
- ✓
NANOCHAT_FP8_WEIGHT_CACHE, 1.002 → 1.014 - Fast accum on backward GEMMs
- ✓ only on grad_input, K ≤ 6144:
NANOCHAT_FP8_FASTACCUM_DGRAD, 1.032 → 1.043, statistical gate 0.09–0.30σ; both backward GEMMs on was rejected (4–15σ) - Other main levers
- column-major activation copy saved in forward, avoiding re-quantization in backward (1.043 → 1.072), single-pass RMSNorm reduction
§ ARaw trace
Raw work log23 sections
0.0 hRound 1 — profile the unchanged reference
(start 2026-11-07 ~02:50 UTC)
0.0 hRound 1 findings (profile of the unchanged reference)
bench.py profile --tree targets/reference --out /tmp/prof-ref-1(torch.profiler, rank 0, updates 12-13). Total GPU kernel time 2016.7 ms for 2 updates -> 1008 ms/update (the measured step time is the same, so the GPU is ~97.5% busy; only 2.5% idle gaps). Breakdown of the 2016 ms: - FP8 GEMMs (cuBLASnvjet_*): 896 ms 44.5% - triton elementwise/reduction kernels: 821 ms 40.7% (the FP8 quantization glue, rms_norm, relu^2, loss) - FlashAttention-3 fwd+bwd: 218 ms 10.8% - NCCL (optimizer reduce-scatter/all-gather): 45 ms 2.2% - aten elementwise: 28 ms 1.4% The FP8 GEMMs run at ~1.28 TFLOP/s of FP8 (H100 class peak 1979 dense) -> little headroom in cuBLAS. The memory-bound part is already at the bandwidth roofline (a pure abs().max() over [32768,1536] bf16 takes 35.8 us = 2.8 TB/s), so speedups must come from *removing* passes, not from tuning kernels.
0.5 hRound 2 — screening (all with `bench.py time --tree /submission/nanochat --config <file>`)
| config | change | s/update | speedup vs frozen reference | |---|---|---|---| | /tmp/sw/base.json | none (tree with all switches off) | 1.0169 | 0.9989 | | /tmp/sw/fastacc.json |NANOCHAT_FP8_FASTACCUM_BWD=1(cuBLAS fast accum in the 2 backward FP8 GEMMs) | 0.9883 | 1.0278 | | /tmp/sw/e2.json | fastacc +NANOCHAT_FP8_SHARE_QKV=1| 0.9887 | 1.0274 | | /tmp/sw/e3.json |NANOCHAT_FP8_SHARE_QKV=1only | 1.0160 | 0.9998 | Notes: * Sharing one FP8 quantization across the q/k/v projections gains nothing: inductor already CSEs the three identical_to_fp8calls on the same tensor. The switch is therefore useless -> will be removed. * Moving the activation quantization out of the autograd.Function (needed to share it) is performance neutral (base row) but adds risk -> keep only if it buys something else. * fast accumulation in the backward GEMMs is the only win so far: +2.8% (tier N1, changes FP8 accumulation order).
1.5 hRound 3 — isolate the two changes (gate 3 = statistical gate)
(start 2026-11-07 04:00 UTC, end 04:16 UTC) Isolation runs withbench.py check --n1(gate 3 only, one 150-update trajectory, seed 1): | config | change | gate 3 verdict | |---|---|---| | /tmp/sw/lossonly.json |NANOCHAT_FUSED_LOSS=1only | PASS (N1), verdictequivalent| | /tmp/sw/fastonly.json |NANOCHAT_FP8_FASTACCUM_BWD=1only | INCORRECT (verdictdifferent) | * The fused loss perturbs the trajectory by 0.03–0.19 sigma (bound is 3 sigma) -> very safe, kept. * The cuBLAS fast accumulation in the two backward FP8 GEMMs perturbs the probe loss by 4–15 sigma (the K=32768 grad_weight contraction accumulates in fp16) -> rejected, the switch is now off. Consequence: the speedup must come from changes that do not change the arithmetic (tier N0) plus the fused loss. The measured speedup of the fused loss alone is ~0.2 %, so >=1.8 % has to come from N0 work. ### Profile analysis of the reference (torch.profiler, rank 0, updates 12-13) Total 1012 ms/update. Per kernel family: - cuBLAS FP8 GEMMs (nvjet_*): 443 ms/u (44 %) ~1270 TFLOP/s effective (peak 1979) - FlashAttention-3 fwd+bwd: 106 ms/u (10 %) - Triton elementwise/reduction (FP8 glue, rms_norm, loss): ~380 ms/u (38 %) <- the target - NCCL (reduce-scatter + all-gather): 27 ms/u - AdamW/Muon optimizer compute: ~40 ms/u Biggest single non-GEMM kernel: 64.6 ms/u, 96 calls/update, grid [96,512] = a 64x64-tiled transposed fp8 store of the [32768,6144] quantized MLP activation (0.6-1.5 TB/s vs ~3 TB/s roofline). Second: the two FP8 amax reductions of the MLP tensors (19.6 + 13.0 ms/u) and the fp8 cast of the MLP activation (18.6 ms/u). FP8 *weight* quantization + column-major copy (recomputed for every micro-batch although the weights only change at the optimizer step): ~13.6 ms/u -> can be cached (N0, bitwise identical).
2.0 hRound 4 — compile-mode screening + where the time really goes
(start 2026-11-07 04:29 UTC, end ~04:45 UTC) Screening withbench.py time(all with fused loss + FP8 weight cache): | config | change | result | |---|---|---| | /tmp/sw/mat.json |NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs| aborted after 10 min — inductor autotunes ~400 mm instances (the Muon polar-express matmuls), compile alone > 45 min. Too slow for the final evaluation (timing timeout 2700 s). | | /tmp/sw/ro.json |NANOCHAT_COMPILE_MODE=reduce-overhead(CUDA graphs) | ERROR: SIGSEGV on rank 0 during the first capture. CUDA graphs are not usable with this model. | Conclusions: inductor's compile mode is a dead end; the wins have to come from the kernels themselves. ### Where the time goes (re-derived from the profile, per *call* not per update) The earlier "ms/u" figures of the triton kernels are totals over the 4 micro-batches, so per-call they are 3-5x smaller than they look. Re-reading the profile with call counts: *nvjet(cuBLAS FP8 GEMMs) 451 ms/u,tritonpointwise/reduction 411 ms/u, FA3 109, NCCL 23, aten 15. * the loss forward is not a hotspot: a standalone micro-benchmark oflog(sum(exp(15*tanh(x/15))))over the [32768, 98304] bf16 logits takes 3.0 ms (2.1 TB/s, i.e. at the roofline); an exp2/tanh rewrite only buys 5 %, a hand written Triton kernel 12 %. -> dropped (the 12.4 "ms/u" figure was 4 calls of 3.1 ms). * the rms_norm backward is the real find: inductor decomposes it into a row-reduction kernel + a separate epilogue kernel, i.e. it reads the activation and the gradient *twice*. A single fused Triton kernel (read x, read grad, write grad_in) runs at 75 us per [16,2048,1536] vs ~180 us for the inductor decomposition.
2.0 hRound 4/5 results (all with `bench.py time`, tree = /submission/nanochat)
| config | change | s/update | speedup vs frozen ref | min per-window ratio |
|---|---|---|---|---|
| /tmp/time-dump-1 (loss only) | fused loss only | 1.0138 | 1.0020 | - |
| /tmp/time-wcache-1 | fused loss + FP8 weight cache | 1.0021 | 1.0137 | 1.0122 |
| /tmp/time-rms-1 | + fused rms_norm backward (custom Triton) | 1.0515 | 0.9660 | - |
| /tmp/time-ws32-1 | + CUBLASLT_WORKSPACE_SIZE=32 MiB | 1.0018 | 1.0140 | - |
| /tmp/time-fadgrad-1 | + fast accum only in the grad_input GEMM (K<=32768) | 0.9896 | 1.0265 | 1.0239 |
Dead ends (documented so they are not retried):
* fused rms_norm backward (custom Triton kernel): the kernel itself is faster (75 us vs 108 us
per [16,2048,1536] norm) but wrapping the norm in a torch.autograd.Function makes dynamo
graph-break, so the *forward* of the norm runs eagerly (fp32 intermediates) -> +50 ms/update.
The inductor rms_norm backward is already at the memory roofline (300 MB in 108 us = 2.8 TB/s).
* max-autotune: compiles for > 45 min (it autotunes the Muon's matmuls) -> unusable.
* reduce-overhead / CUDA graphs: SIGSEGV on this model.
* CUBLASLT_WORKSPACE_SIZE: no effect (1.0018 vs 1.0021, noise).
* fast accumulation in both backward GEMMs (round 3): +2.8 % but 4-15 sigma off on the probe
loss -> gate 3 rejects it. Restricting it to the grad_input GEMM (whose contraction length is
the number of output features, not the 32768 tokens of the micro-batch) keeps almost all of the
speed and is much closer to the reference numerically.2.7 hRound 6 — Inductor coordinate-descent tuning (DEAD END: hangs)
(start 2026-11-07 05:15 UTC, killed 05:20 UTC) Hypothesis. The Triton glue kernels that dominate the memory-bound part of the update run at ~2.1 TB/s while the machine's copy roofline is 4.25 TB/s. Inductor's block sizes for those kernels are heuristics;torch._inductor.config.coordinate_descent_tuning = Truelets Inductor benchmark launch configurations and pick the best one. Change.scripts/base_train.pyreadsNANOCHAT_INDUCTOR_CD_TUNE=1and setstorch._inductor.config.coordinate_descent_tuning = Truebefore the model is compiled (switch is off by default; committed in 118327f). Result. The run *hung* in the first forward: Inductor compiled ~442 kernel variants between 05:15:36 and 05:18:23 and then both ranks slept forever at 0 % GPU utilisation (a deadlock between the two ranks in the autotune cache / file locks, or in the tuning benchmark itself). The processes were killed; no judge JSON was produced (/tmp/time-cdtune-1.log,/tmp/time-cdtune-1/timing_tree.logcontain only the setup lines). Conclusion. Dead end, do not retry without understanding the deadlock. The same lever was already refuted twice (max-autotune: > 45 min compile; reduce-overhead/CUDA graphs: SIGSEGV).
2.7 hRound 6 profile of the *current best* tree (weight cache + fused loss)
bench.py profile --tree /submission/nanochat --config /submission/switches.json --out /tmp/prof-cur-1(rank 0, updates 12-13). Total kernel time 1999.4 ms for 2 updates -> 999.7 ms/update (the reference is 1008.4 ms/update in the same profiler). * FP8 GEMMs (nvjet_*): 452 ms/u (45 %) * FlashAttention-3: 109 ms/u (11 %) * Triton glue: ~390 ms/u (39 %) * NCCL (reduce-scatter + all-gather of the optimizer): 45 ms/u (4.5 %) Top glue kernels (ms/update, calls/update): 65.1 triton_poi_fused__scaled_mm__to_copy_clamp_clone_mul_pow_reciprocal_re (MLP backward: reads h + grad in bf16 [32768,6144], writes e4m3 col-major + e5m2 row-major + e5m2 col-major; 1407 MB/call at 2.07 TB/s -- the transposed stores are the suspect, the neighbouring amax kernel reaches 3.94 TB/s) 20.9 / 18.1 / 14.0 / 13.5 / 10.0 the five*_fused_rms_norm_backward*kernels (76.5 ms/u together: Inductor splits the rms-norm backward into a reduction + a pointwise kernel, so x and grad are read twice) 19.6 triton_red_fused__to_copy_abs_max_mul_pow_relu_threshold_backward (amax of the MLP gradient) 18.6 triton_poi_fused__scaled_mm__to_copy_clamp_mul_pow_relu_t_view_15 (e5m2 cast of that gradient) 18.6 triton_per_fused__to_copy_add_cat_mean_mul_neg_pow_rsqrt_slice_view_7 (norm forward, fused with the residual add and the value-embedding cat) 16.2 triton_per_fused__scaled_mm__to_copy__unsafe_view_clamp_embedding_dens (value-embedding gather) 16.2 triton_per_fused__to_copy_abs_add_max_slice_backward_view_28 (embedding backward, scatter-add) 14.7 triton_poi_fused__scaled_mm__to_copy_add_clamp_clone_mul_reciprocal_sl (forward activation quant) 12.6 the fused soft-capped cross entropy (4 calls/u, 3.1 ms each: it is exp/tanh bound, not memory bound)
3.6 hRound 7 — share one activation quantization between q/k/v (tier N0)
(start 2026-11-07 05:20 UTC, end 05:52 UTC) Hypothesis. The query, key and value projections of every attention block read the *same* activation, so the model quantized it three times (three amax passes + three cast passes, and three copies of the column-major fp8 activation that the weight GEMM needs). Change.NANOCHAT_FP8_SHARE_QKV=1(nanochat/gpt.py): quantize the attention input once and hand the fp8 tensor plus its inverse scale toc_q,c_kandc_v(Float8Linear.forwardalready accepted a pre-quantized activation). Result.bench.py time --tree /submission/nanochat --config /tmp/sw/qkv.json --out /tmp/time-qkv-1->speedup_vs_frozen_reference1.0158 (mean), windows 0.9971 / 1.0024 / 1.0004 s/update. (The switch is bitwise identical to the reference: the same fp8 values, just computed once.)
3.6 hRound 8 — persistent reductions for the 1536-wide rms-norm (tier N1) *** biggest win so far ***
(start 2026-11-07 05:56 UTC, end 06:19 UTC) Profile finding (round 6,bench.py profileof the current best tree). The rms-norm reduces over 1536 elements. Inductor only emits a *persistent* reduction (the whole reduced row stays in registers) when the reduction is <= 64 elements (1024 for an inner reduction), so it generated a *split* reduction: the kernel walks the row twice (once to accumulate the sum of squares, once to apply the scale) and the activation is read from memory twice. Change.nanochat/inductor_tuning.py(new, off by default,NANOCHAT_PERSISTENT_REDUCTION=1) patchesInductorChoices.should_use_persistent_reductionso that reductions of up to 2048 elements stay in registers. Kernels that read or write an fp8 buffer are excluded: a persistent reduction works on a 1-D tile and the transposed fp8 stores this model needs would be uncoalesced there (the unfiltered version was a *catastrophic* regression: 1.1049 s/update, speedup 0.9194).scripts/base_train.pycallsapply_persistent_reduction()before the model is compiled. *Bug found and fixed while implementing this:*SIMDKernelFeatures.buf_accessesis a method, not a property, andDephas no.buffer; the fp8 filter has to go throughV.graph.get_buffer(dep.name).get_dtype(). With the bug the filter always raised and silently disabled the whole switch. Result. `bench.py time --tree /submission/nanochat --config /tmp/sw/persist.json --out /tmp/time-persist-2->speedup_vs_frozen_reference` 1.0224 (mean), windows 0.9889 / 0.9961 / 0.9955 s/update, per-window ratios 1.0215 / 1.0224 / 1.0235 (min 1.0215). The benefit bar is max(1.02, 1 + 2*spread) = 1.02, so the run is now above the bar (was 1.0131 min window). Tier. N1: the sum of squares of the rms-norm is accumulated in a different order, so the parameters are no longer bitwise identical (probe loss at update 0: 10.397739 with the switch vs 10.397737 without it - a ~1e-7 relative difference). Gate 3 (statistical) has to accept it. Current switch file (/submission/switches.json): `{"environment": {"NANOCHAT_FUSED_LOSS": "1", "NANOCHAT_FP8_WEIGHT_CACHE": "1", "NANOCHAT_FP8_SHARE_QKV": "1", "NANOCHAT_PERSISTENT_REDUCTION": "1"}}`
3.6 hDead end (investigated in round 8, not adopted): fast transposed fp8 stores
The biggest single Triton kernel is the MLP backward glue kernel (65 ms/update, 1407 MB per call at 2.07 TB/s while its neighbours reach 3.9 TB/s). It writes two *transposed* fp8 buffers (the column-major copies the weight GEMMs need). A microbenchmark of the exact generated kernel showed: * unmodified: 0.485 ms (2.90 TB/s) * withtl.transon the value only: 0.357 ms (3.94 TB/s) - **but the data ends up transposed twice, i.e. wrong** * withtl.transon the value *and* the matching index (correct, bitwise identical): 0.493 ms - no gain * withtl.transon the index variables: 5.98 ms - catastrophic (Triton materialises the transpose of the index tile in shared memory) Triton cannot express "same logical shape, transposed layout" for a value that comes from an x-contiguous load while the destination pointer is token-contiguous: the layout conversion that the store needs is exactly the expensive part, andtl.transchanges the logical shape (which breaks the data pairing). Conclusion: with the fp8 layout constraints oftorch._scaled_mm(mat1 must be row-major, mat2 column-major - verified: every other combination raises "Only multiplication of row-major and column-major matrices is supported by cuBLASLt") the transposed copies are irreducible, and a fast transposing store is not reachable from Triton's layout system.
3.6 hRound 8, follow-up: the statistical gate (gate 3) accepts the persistent reduction
bench.py check --n1 --tree /submission/nanochat --config /submission/switches.json --out /tmp/check-n1-1-> status PASS, tier N1 (/tmp/check-n1-1/check.json). The six probe-loss endpoints of the 150-update trajectory (seed 1) deviate from the frozen reference by 0.22 - 0.30 sigma (bound = 3 sigma), e.g. endpoint 5-15: mean 0.00506 vs sigma_seed 0.02269 (bound 0.06808). The persistent reduction only changes the order in which the sum of squares of the rms-norm is accumulated, which is far below the seed-to-seed spread. State after round 8: a full check would be PASS (N1) with speedup ~1.022.
4.3 hRound 9: tiled Triton FP8 transpose (N0 candidate) — first attempt was a loss, refined version wins
Profile findings (round 9 start). The judge profile of the current best tree (/tmp/prof-persist-1, rank 0, 989 ms/update of GPU kernel time) breaks down as GEMM (cuBLAS nvjet) 452 ms/u, FlashAttention-3 110 ms/u, Triton glue 389 ms/u, NCCL 23 ms/u. Inside the glue the biggest single kernel is the FP8 gradient kernel that writes the *transposed* FP8 copy the weight GEMM needs: it reads the bf16 gradient with a transposed (uncoalesced) access pattern and takes 0.675 ms per call, i.e. ~220 GB/s while its neighbours reach 3.9 TB/s. The cuBLASLt FP8 GEMM requires the second operand in column-major layout, so those transposed copies are unavoidable; only their *cost* is negotiable. Attempt 1 (regression). A Triton kernel that transposes an already quantized FP8 tensor with a tiled (tl.trans) transpose reaches 3.0-3.9 TB/s in isolation *and* 3.75 TB/s in situ. Wiring it in as atorch.library.custom_op(opaque, so Inductor cannot fuse the quantization into the transposed store) removed every uncoalesced transposed store from the generated code, but the judgetimerun gave speedup 1.0079 (windows 1.0026 / 1.0078 / 1.0129) — a 1.4 % *regression* versus 1.0224. Reason (from the profiles): making the copy opaque split the fused kernel, and Inductor then re-generated the remaining read with a 1-D grid whose index arithmetic (x % 1536,x // 1536,x % 128,x // 128) costs an integer division per element; the same read in the 2-D grid version was ~1.9x faster (0.153 -> 0.285 ms/call). Attempt 2 (adopted). Write *one* custom kernel that reads the bf16 gradient once (coalesced) and writes both FP8 layouts (row-major for the grad-input GEMM, transposed for the grad-weight GEMM), the transposed store going through a shared-memory tile transpose. The scale is computed with the same expression as_to_fp8(the amax is order independent, so it is bit-identical), so the FP8 bytes are unchanged. The activation's column-major copy uses the plain tiled FP8 transpose. Microbenchmark: 3.5-3.85 TB/s (0.057 ms for [32768,1536], 0.217 ms for [32768,6144], 1.14 ms for [32768,32768]). Result (local, 2 GPUs). Loss after updates 0/1: 10.397739 / 10.386679 — bit-identical to the reference. Update time 982.6 ms -> 952.2 ms. Switch:NANOCHAT_FP8_TILED_TRANSPOSE=1(default off).
4.6 hRound 10: fast accumulation in the grad_input FP8 GEMM (N1) — ACCEPTED, 1.0426
Profile findings (round 10 start). The judge profile of the round-9 tree (/tmp/prof-qp-1, rank 0, ~1050 ms of GPU kernel time per update window) breaks down as GEMM (cuBLAS nvjet) 481 ms/u, FlashAttention-3 112 ms/u, Triton glue 321 ms/u (of which the FP8 quantisation path is ~115 ms and the rms-norm family ~62 ms), the custom FP8 kernels 70 ms/u, NCCL 26 ms/u, Muon/AdamW ~30 ms/u and ~43 ms/u of GPU idle (launch latency spread over ~9.8k kernel launches per update). The optimizer step alone is 98 ms of the ~980 ms update (10 %), of which NCCL is 26 ms and the Muon Newton-Schulz GEMMs ~20 ms. What changed.NANOCHAT_FP8_FASTACCUM_DGRAD=1withNANOCHAT_FP8_FASTACCUM_MAX_K=6144turns on cuBLAS' fast-accumulation mode for the *grad_input* FP8 GEMM of every transformer block. The vocabulary head is excluded because its contraction (32768) is much longer than the blocks' (1536-6144), and the earlier experiment that enabled fast accumulation in *both* backward GEMMs was rejected by gate 3 at 4-15 sigma. The switch already existed (commit 118327f, default off); this round validates and adopts it. Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/dgrad.json --out /tmp/time-dgrad-1->speedup_vs_frozen_reference1.0426,s_per_update` 0.9743 (was 1.0321 / 0.9879 for the round-9 tree). `bench.py check --n1 --tree /submission/nanochat --config /tmp/sw/dgrad.json --out /tmp/check-n1-dgrad2` -> status PASS, tier N1. The six probe-loss endpoints of the 150-update trajectory deviate by 0.09 - 0.30 sigma (bound 3 sigma): 0.0038 (0.17s), 0.0029 (0.22s), 0.0024 (0.30s), 0.0024 (0.17s), 0.0031 (0.13s), 0.0031 (0.09s). Current switch file (/submission/switches.json): `{"environment": {"NANOCHAT_FUSED_LOSS": "1", "NANOCHAT_FP8_WEIGHT_CACHE": "1", "NANOCHAT_FP8_SHARE_QKV": "1", "NANOCHAT_PERSISTENT_REDUCTION": "1", "NANOCHAT_FP8_TILED_TRANSPOSE": "1", "NANOCHAT_FP8_FASTACCUM_DGRAD": "1", "NANOCHAT_FP8_FASTACCUM_MAX_K": "6144"}}`
5.1 hRound 11: fast Triton refresh of the cached FP8 weights (N0) — no measurable gain, kept
Profile findings (round 11 start, judge profile/tmp/prof-r11, rank 0, two updates). Kernel time per update (ms), from the profiler trace (2 updates, 1937.8 ms of kernel time => 968.9 ms/u against a 975 ms update, i.e. the GPU is ~99 % busy): | bucket | ms/u | share | |---|---|---| | cuBLAS FP8 GEMMs (nvjet_*) | 448.5 | 46 % | | Inductor Triton glue | 302.1 | 31 % | | FlashAttention-3 (fwd+bwd) | 109.6 | 11 % | | custom FP8 kernels (_fp8_quant_pair_kernel,_fp8_transpose_kernel) | 69.4 | 7 % | | NCCL (reduce-scatter / all-gather) | 23.7 | 2.4 % | | other (eager elementwise, cat, ...) | 15.6 | 1.6 % | A standalone cuBLAS microbenchmark (/tmp/bench_gemm.py) reproduces the 448 ms/u GEMM time exactly, so the GEMMs are at cuBLAS speed: 1346-1375 TFLOPS for the grad-weight GEMMs (contraction 32768), 1311-1328 for the grad-input GEMMs, and 1019-1094 for the forward GEMMs whose contraction is only 1536 (the measured FP8 peak of this GPU is 1518 TFLOPS at 8192^3). FA3 runs at 584 TFLOPS on the full-context layers and 388 on the 512-window layers, i.e. it is at the bf16 GEMM roofline. There is therefore no GEMM or attention headroom left; the only slack is in the Inductor glue (302 ms/u) and the custom FP8 kernels (69 ms/u). What changed.NANOCHAT_FP8_FAST_WEIGHT_REFRESH=1(default off) replaces the compiled weight-quantization helper (_quantize_weight_into_compiled: five Inductor kernels plus the Dynamo guard machinery per weight, 97 weights for D24) with two Triton launches per weight (amax partials + row quantize) plus the existing tiled FP8 transpose. The FP8 bytes are bit-identical: the amax is order independent, the scale is448 / max(double(amax), 1e-12)computed in fp64 with an IEEE round-to-nearest division, and the reciprocal usestl.math.div_rn(triton'stl.fdivis *not* reliably round-to-nearest and produced a 1-ULP different inverse scale, which was caught by the bitwise check). Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/cur.json --out /tmp/time-r11b->speedup_vs_frozen_reference1.0425,s_per_update` 0.9744 (windows 0.9684 / 0.9748 / 0.9765). The previous tree measured 1.0426, so the change is within noise: the refresh costs 0.044 ms per weight instead of 0.095 ms, i.e. ~5 ms of the 975 ms update. It is bit-identical (N0), so it is kept enabled, but it is not a source of gain.
5.4 hRound 12: column-major FP8 activation copy in the forward pass (N0) — ACCEPTED, 1.072
Profile findings. The judge profile of the round-11 tree (/tmp/prof-r11, rank 0, two updates, 1937.8 ms of kernel time = 968.9 ms/u against a 975 ms update) breaks down as cuBLAS FP8 GEMMs 448.5 ms/u, Inductor Triton glue 302.1 ms/u, FlashAttention-3 109.6 ms/u, the custom FP8 kernels 69.4 ms/u, NCCL 23.7 ms/u and other 15.6 ms/u. A standalone cuBLAS microbenchmark reproduces the 448 ms/u of GEMM time exactly (1346-1375 TFLOPS for the grad-weight GEMMs, 1311-1328 for the grad-input GEMMs, 1019-1094 for the forward GEMMs whose contraction is only 1536; the measured FP8 peak of this GPU is 1518 TFLOPS at 8192^3), and FA3 runs at 584/388 TFLOPS on the full-context / 512-window layers, i.e. at the bf16 GEMM roofline. There is no GEMM or attention headroom left. The interesting part was in the *generated* code (TORCH_LOGS=output_codeon a depth-4 model with the same shapes): the backward graph rematerialises the FP8 activation quantisation. The backward only needs the activation in column-major layout (the second operand of the grad-weight GEMM), but instead of saving the FP8 buffer Inductor's min-cut partitioner recomputes the whole quantisation - read the bf16 activation, compute the amax, cast to FP8 - and only then transposes it. For D24 that duplicated ~600 MB of traffic per FP8 linear per micro-batch (~30 ms/u). What changed.NANOCHAT_FP8_ACT_COL_IN_FWD=1now produces the column-major copy in the forward pass, with the tiled Triton transpose (_fp8_transpose_op) instead of the eagerx.t().contiguous().t()copy - the eager copy reads and writes with a transposed access pattern and is several times slower, which is why the earlier attempt with this switch was neutral. Because the backward then consumes the forward's copy, the partitioner no longer has anything to rematerialise. Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/actcol.json --out /tmp/time-actcol->speedup_vs_frozen_reference1.072,s_per_update` 0.9476 (was 1.0425 / 0.9744): +2.9 %. Numerics. The FP8 bytes and the operand of the grad-weight GEMM are unchanged, so the change is bitwise identical (N0). Verified by comparing the 10-update deterministic record of the tree with and without the switch (check --quick,/tmp/check-quick-actcoland/tmp/check-quick-cur): the parameter digests are identical, and the exact items pass (exact_items_ok: true, no problems).
6.3 hRound 13: fused FP8 quantize + column-major copy (N1) — ACCEPTED, 1.0816
Profile findings (round-12 tree,/tmp/prof-r12). 942.6 ms/u of kernel time: cuBLAS FP8 GEMMs 449 ms/u (82-95 % of the 1462 TFLOPS FP8 peak), FlashAttention-3 109 ms/u (at the bf16 roofline), Inductor Triton glue ~225 ms/u, the custom FP8 kernels ~70 ms/u (all at ~3.7 TB/s), the optimizer ~50 ms/u and NCCL ~26 ms/u. There is no headroom left in the GEMMs or in the attention, so the remaining work is in the memory-bound glue. What changed. The FP8 activation has two consumers: the forward GEMM operand (row major) and the column-major copy that the grad-weight GEMM needs. Inductor materialises the row-major buffer with its own cast kernel and the column-major copy is then a second pass over the FP8 bytes (NANOCHAT_FP8_QUANT_PAIR, off by default). A Triton kernel (_fp8_quant_pair_e4m3_kernel) now reads the bf16 source once and writes both layouts. The FP8 bytes are identical to_to_fp8()(checked byte for byte on four shapes), but the custom op is a materialisation boundary: Inductor can no longer fuse the rms-norm into the quantisation, so the norm output is rounded to bf16 before it is quantised. The tree is therefore N1 (it already was, through the fast-accumulation dgrad GEMM). Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/pair.json --out /tmp/time-pair->speedup_vs_frozen_reference1.0816,s_per_update` 0.9392 (was 1.0720 / 0.9476): +0.9 %. The profile of the new tree (/tmp/prof-r13) shows 930.2 ms/u of kernel time (-12.4 ms/u): the separate transpose kernel for the plain activations is gone (_fp8_quant_pair_e4m3_kernel, 16.1 ms/u at 3.6 TB/s) and the forward amax of those activations is now its own reduction (3.5 ms/u at 2.75 TB/s). Numerics.bench.py check --quick-> the exact items pass (exact_items_ok: true) and the parameter digests differ from the previous tree (the quantised bytes now come from the materialised bf16 norm output).bench.py check --n1-> PASS (N1), the six endpoints of the 150-update trajectory are at 0.05 - 0.24 sigma (bound 3 sigma):probe_s005_0150.092,probe_s015_0300.148,probe_s030_0500.237,probe_s050_0800.096,probe_s080_1150.051,probe_s115_1500.104.
7.8 hRound 14: one FP8 kernel for the MLP activation (both layouts) (N0) — ACCEPTED, 1.0907
Profile findings (round-13 tree,/tmp/prof-r13). 930 ms/u of kernel time. The FP8 activation of the MLP's second projection was the last user of the "plain" (Inductor) cast path:triton_poi_fused__scaled_mm__to_copy_clamp_fp8_transpose_mul_pow_reciprocal_relu_t_view_12(20.6 ms/u) fuses relu^2 + the FP8 cast, but its output has two consumers (the operand of the forward GEMM and the column-major copy that the grad-weight GEMM needs), so Inductor materialised the FP8 buffer twice (buf59andbuf62, both [32768, 6144] e4m3) and the column-major copy was then a *third* pass over the data (_fp8_transpose_kernel, 10.7 ms/u). The amax of the activation (triton_red_fused__to_copy_abs_max_pow_relu_view_10, 13.2 ms/u) reads the pre-activation a second time. What changed.NANOCHAT_FP8_RELU2_PAIR=1(new, off by default) quantizes the pre-activation with a dedicated Triton kernel (_fp8_quant_pair_relu_e4m3_kernel) that applies relu^2 in fp32, casts to e4m3 and writes both layouts in one pass. The relu^2 derivative is applied by a new node (_Relu2Grad: identity forward, derivative backward), so Inductor keeps fusing the derivative - and the amax of the gradient that is fused into it - into the backward graph exactly as before. Traffic per MLP per micro-batch: 1608 MB -> 1206 MB. Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/relu2.json --out /tmp/time-relu2c->speedup_vs_frozen_reference1.0907,s_per_update` 0.9313 (was 1.0816 / 0.9392): +0.8 %. Windows 0.9259 / 0.9328 / 0.9351 s/u. Numerics (N0). Unit test: the scale, the row-major FP8 bytes, the column-major FP8 bytes and the relu^2 gradient are bit-identical to the unfused path.bench.py check --quick-> exact items pass (exact_items_ok: true); the 10 record updates x 175 parameter digests are identical to the previous tree (/tmp/check-quick-relu2/record_cand.r0.jsonvs/tmp/check-full-pair/record_cand.r0.json), so the tree stays exactly as it was (N1 vs the frozen reference, through the fast-accumulation dgrad GEMM).
7.9 hRound 15: wider FP8 quantize tiles + a dedicated amax reduction for the relu^2 activation (N1, bitwise-neutral) — 1.0951
Profile findings (round-14 tree,/tmp/prof-r14). The three "quantize + write both layouts" kernels are memory-bound and run at ~3.4 TB/s (the roofline of this GPU), so they can only get faster with fewer bytes or a wider store. The forward amax of the MLP activation was still an Inductor reduction over the *pre-activation* (13.2 ms/u, 2.93 TB/s). What changed (one commit,60122f5, both behind the existing switches): 1. The three pair kernels now use a 64 x 256 tile instead of 64 x 128 (the row-major store is then 256 B wide). Microbenchmark: 32768 x 32768 quantize 1.143 -> 1.112 ms, 32768 x 6144 0.217 -> 0.212 ms (2-3 % per kernel). The tile falls back to 64 x 128 when N % 256 != 0. 2. New Triton kernel + custom opnanochat::fp8_amax_partials_relu(1024 programs, block 4096): the partial maxima of relu(x)^2 (4.18 TB/s vs Inductor's 2.93 TB/s).forward_relu2now uses it and keeps the *same* scale expression, so the FP8 bytes are unchanged (a maximum does not depend on the order of the reduction, and the squares are computed in fp32 exactly like Inductor's fused kernel does). Result (judge). `bench.py time --tree /submission/nanochat --config /submission/switches.json --out /tmp/time-r15a->speedup_vs_frozen_reference1.0951,s_per_update` 0.9276 (was 1.0907 / 0.9313): +0.4 %. Windows 0.9224 / 0.9283 / 0.9321 s/u. Numerics.bench.py check --quick --out /tmp/check-quick-r15-> the exact items pass (exact_items_ok: true,status: NOT_BITWISEbecause the tree is tier N1 through the fast-accumulation dgrad GEMM). The 10 record updates x 175 parameter digests are identical to the round-14 tree (/tmp/check-quick-r15/record_cand.r0.jsonvs/tmp/check-full-pair/record_cand.r0.json) -> the change is numerically neutral (same N1 tier, the statistical gate of the last full check still applies).
8.8 hRound 16: a dedicated Triton kernel for the soft-capped cross entropy backward (N1) — 1.0983
Profile findings (round-15 tree,/tmp/prof-r15). 918 ms/u of kernel time per update. The complete inventory (from the profiler trace) shows that almost everything is at the memory roofline of this GPU (4.2-4.3 TB/s): the FP8 quantization kernels (99 ms/u), the relu^2 gradient + amax (29.8), rope + QK-norm (18.8), the rmsnorm backward (23.5), the gradient accumulation (10.2), the amax of the gradients (16.6). The GEMMs (449 ms/u, 49%) run at ~64 % MFU, FA3 (109 ms/u) is at its roofline, NCCL (23.4 ms/u) is NVLink bound. The one clearly *compute* bound piece left is the loss: the backward kernel (triton_red_fused__to_copy_abs_arange_clamp_div_eq_exp_max_mul_rsub_sub_tanh...) took 3.157 ms/call (12.63 ms/u) for 1.07 G elements, the forward 1.16 ms/call (4.64 ms/u). The backward is slow because the FP8 amax of the head's gradient (the head is an FP8 linear) is fused into it: the kernel then becomes a reduction over the whole vocabulary. What changed.NANOCHAT_FAST_LOSS_BWD=1(new, off by default) computes the gradient of the soft-capped cross entropy with a dedicated Triton kernel (nanochat::loss_bwd), one program per (row, block of 4096 columns). The arithmetic is the one of the compiled graph, in the same order: the division by the soft cap is the multiplication by its fp32 reciprocal that Inductor emits,tanh/expare the same libdevice functions, the scale isgrad_out / countwith Triton's own division (the *same*div.full.f32that Inductor emits - an IEEE division here is worth 1 ulp on ~1.3e-5 of the elements) and the result is rounded to bf16 at the very end. The unit test on a real model: the 21 parameter gradients are bitwise identical to the ones of the Inductor kernel. Result (judge).bench.py time --config /tmp/sw/lossbwd.json->speedup_vs_frozen_reference1.0983,s_per_update0.9249 (was 1.0951 / 0.9276). Numerics (N1).bench.py check --quick-> the exact items pass, the status is NOT_BITWISE (the tree already was tier N1 through the fast-accumulation dgrad GEMM) and the 10 record updates x 175 parameter digests differ from the round-15 candidate: the amax of the head's gradient is now taken over the bf16 gradient (a separate reduction) instead of over the fp32 values that the fused kernel reduced, so the FP8 scale of the head's gradient changes in the last bits. Full check (judge). `bench.py check --tree /submission/nanochat --config /submission/switches.json --out /tmp/check-full-r17-> PASS,performance.speedup` 1.0981, tier N1 (the tree is not bitwise identical to the reference because of the fast-accumulation dgrad GEMM of round 10; the statistical gate passed: every endpoint of the 150-update trajectory is within 3 sigma of the reference)./tmp/check-full-r17/check.json.
8.8 hRound 17: the amax of the head's gradient in the loss backward kernel (N0) — 1.0998
What changed. The Triton kernel of round 16 now also reduces the maximum of the gradient (over the fp32 values, exactly like the fused Inductor kernel did) and hands it over to the FP8 path of the head through a small hand-off slot innanochat/fp8.py(nanochat::grad_amax): the head is the only FP8 layer whose gradient comes out of the loss, and the FP8 path asks for the amax right after the loss backward, in the same order. The maximum does not depend on the order of the reduction, so the value is the one the fused kernel produced. With the hand-off, the separate amax pass over the 2 GB gradient disappears and the whole loss backward becomes one kernel. Result (judge).bench.py time --config /tmp/sw/lossbwd.json->speedup_vs_frozen_reference1.0998,s_per_update0.9236 (windows 0.9178 / 0.9248 / 0.9281), i.e. +0.4 % over the round-15 tree. Numerics (N0). A one-layer model with the FP8 head gives the same loss and the same 15 parameter gradients as the round-15 tree (the only difference is the token embedding's gradient, which is atomic-nondeterministic in *both*).bench.py check --quick-> the 10 record updates x 175 parameter digests are identical to the round-15 candidate (/tmp/check-quick-r17/record_cand.r0.jsonvs/tmp/check-full-pair/record_cand.r0.json) -> the change is numerically neutral, the tree keeps the tier N1 of the fast-accumulation dgrad GEMM and the statistical evidence of the last full check still applies. Dead ends of this round (measured, not guessed). * The FP8 amax of the head's gradient is *not* the amax of the bf16 gradient: Inductor reduces the fp32 values (it removes the bf16 round trip ofgrad.float().abs().max()), so a separate reduction over the bf16 gradient gives a different scale (this was the N1 of round 16). * The scale of the loss backward isgrad_out / countcomputed *inside* the kernel with Triton's non-IEEE division (div.full.f32), not with the IEEE division of eager: with a count that is not a power of two the two differ by 1 ulp on ~1.3e-5 of the elements. * The Inductor kernel of the loss backward and the Triton kernel of round 16 do not have the same arithmetic: the Inductor one is a 2D reduction (the amax is fused into it), the Triton one is pointwise, and the fp32 values of the gradient are the same only if the amax is reduced over the same (fp32) values. * Addressing the gradient to hand the amax over does not work: the tensor that reaches the FP8 path does not have the address of the tensor the loss kernel returned.
9.6 hRound 18: the rotary embedding and the QK-norm in dedicated Triton kernels (N1) — 1.1279, full check PASS 1.1263
Profile findings (round-17 tree,/tmp/prof-r18, 3 updates, 916.6 ms/u). cuBLAS/nvjet FP8 GEMMs 451 ms/u (49 %, 814-1378 TFLOP/s), FlashAttention-3 110 ms/u, the FP8 quantize kernels ~113 ms/u (all at the roofline), NCCL 23 ms/u, the optimizer 49.5 ms/u, **the fused rope + QK-norm kernel 19.0 ms/u (96 calls per update, 191 us each) and its backward 23.7 ms/u** (the tworms_norm_backwardkernels), the value-embedding backward ~21 ms/u, the loss 11.8 ms/u, the gradient accumulation 10.1 ms/u; the GPU is idle only 3.5 ms/u, so the update is GPU-bound. Why the rope kernel is slow. Inductor fusesrms_norm(rope(x)) * 1.2into one kernel that reduces over the head dimension (128 elements) with one row per program: the reduction is a cross-warp reduction for a single 256-byte row, so the kernel moves 600 MB (it also does the rotation of q *and* k) in 191 us, i.e. 3.2 TB/s where the machine does 4.2 TB/s on a copy. What changed.NANOCHAT_FUSED_ROPE_QKNORM=1(new, off by default) computes the rotation, the QK-norm and the 1.2 factor with two dedicated Triton kernels (nanochat/rope_qknorm.py, forward and backward). They do exactly the arithmetic of the compiled graph - the same fp32 operations, the samelibdevice.rsqrt(sum(y^2)/128 + 1.1920928955078125e-07), the same 1.2 factor, the samesum((g*rstd)*(y*1.2))of the backward, the same fp32 inverse rotation - but 16 rows share one program, so the reduction is a shuffle over a few lanes: 49 us for the forward (201 MB) and 74 us for the backward (302 MB), i.e. 4.1 TB/s, the roofline. The backward recomputes the rotation instead of reading back the rotated values the forward used to save, so the forward writes half as many bytes. Numerics (N1). The sum of squares and the dot product are accumulated in a different order (the kernel sums the two halves of the head dimension separately), which changes the fp32 rstd by an ulp: 115 of 50.3 M elements of the forward and 1580 of 50.3 M of the backward differ from the compiled reference (the bf16 rounding hides the difference on the others). On a 2-layer D24 model the loss is bitwise identical and onlyattn.c_proj.weightsees a difference (fp32 noise, 3e-8 on a gradient whose peak is 6e-6). Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/rope.json --out /tmp/time-r18a->speedup_vs_frozen_reference1.1279,s_per_update` 0.9006 (was 1.0998 / 0.9236): +2.6 %. Full check (judge). `bench.py check --tree /submission/nanochat --config /tmp/sw/rope.json --out /tmp/check-full-r18-> PASS,performance.speedup` 1.1263 (windows 1.1263 / 1.1294 / 1.1264), tier N1: the exact items pass (exact_items_ok: true), and the six endpoints of the 150-update trajectory are at 0.11 - 0.20 sigma (bound 3 sigma). Dead end of this round (measured). Rounding the gradient to bf16 *before* the inverse rotation (as the eager_fused_rms_norm_backwardwould do) is *not* what the compiled graph does: the dumped Inductor kernel keeps the whole backward in fp32 and rounds only once, at the end. With the eager rounding 44 % of the elements differ from the compiled reference, without it 0.003 %.
9.9 hRound 19: a dedicated Triton amax for the plain FP8 activations (N0 candidate) — REVERTED (1.1227, a 0.5 % regression)
Profile findings (round-18 tree,/tmp/prof-r19, 2 updates, 886.8 ms/u of kernel time). nvjet FP8 GEMMs 445 ms/u (50 %), FlashAttention-3 101 ms/u, the FP8 quantize kernels 89.6 ms/u (all at the roofline: 3.3-3.9 TB/s), the Inductor reduction that fuses the relu^2 backward with the amax of its gradient 29.8 ms/u (3.9 TB/s), the value-embedding scatter 11.2 ms/u (1.3 TB/s - the only clearly sub-roofline kernel of the update), the value-embedding gate multiply 3.95 ms/u, the cast of the fp32 scatter buffer 3.58 ms/u, the rope kernels 25 ms/u, NCCL 25 ms/u, the optimizer 46 ms/u, the loss 11.8 ms/u, the gradient accumulation 10 ms/u. The GPU is idle only ~13 ms/u: the update is GPU-bound and almost every kernel already runs at the copy roofline. What was tried.NANOCHAT_FP8_AMAX_PARTIALS=1(new, off by default): the amax of a plain activation is computed by a dedicated Triton kernel (4096 partial maxima, 8192 elements each) instead of Inductor's reduction, and the relu^2 amax kernel is retuned (1024->4096 parts, 4096->16384 block). The amax value and therefore the FP8 bytes are bit-identical (a maximum does not depend on the order of the reduction), so this is a tier N0 change. Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/amax.json --out /tmp/time-r19a->speedup_vs_frozen_reference` 1.1227 (windows 0.9004 / 0.9055 / 0.9086), against 1.1279 for the round-18 tree: a 0.5 % regression, reverted (git checkout -- nanochat/fp8.py, nothing to commit). Why (measured). Inductor fuses the amax of an activation into the kernel that *produces* it (the rms-norm backward, the gate multiply, ...). Forcing the amax into a separate custom op breaks that fusion, so the tensor has to be read once more: the standalone partial-max kernel (26 us for 100 MB) plus the extra read of the producer's output costs more than the 11 us that the faster reduction saves. The same argument applies to the retuned relu^2 amax kernel.