MiniMax M3.1 Flash 1.13×

Harness: Codex · switches: NANOCHAT_FUSED_LOSS=1 NANOCHAT_FP8_WEIGHT_CACHE=1 NANOCHAT_FP8_SHARE_QKV=1 NANOCHAT_PERSISTENT_REDUCTION=1 NANOCHAT_FP8_TILED_TRANSPOSE=1 NANOCHAT_FP8_FASTACCUM_DGRAD=1 NANOCHAT_FP8_FASTACCUM_MAX_K=6144 NANOCHAT_FP8_FAST_WEIGHT_REFRESH=1 NANOCHAT_FP8_ACT_COL_IN_FWD=1 NANOCHAT_FP8_QUANT_PAIR=1 NANOCHAT_FP8_RELU2_PAIR=1 NANOCHAT_FAST_LOSS_BWD=1 NANOCHAT_FUSED_ROPE_QKNORM=1

Judged
1.13×
Tier
N1
Self-best
1.13×
Rounds
18
Hours
10.0
API cost
$6.56
Input tokens · 95.0% cached
219 M
Output tokens
3.0 M
Requests
1,681
Micro-batch regrouping 4×16 → 2×32
✗ not in the log
Scope of hand-written kernels replacing Inductor glue
partial: one kernel each for dual-layout quantization, relu², loss backward, RoPE + QK (last one 1.100 → 1.128)
Per-shape cuBLASLt algorithm selection
✗ (only tried a larger workspace, no change)
FP8 weights quantized once per step
✓ NANOCHAT_FP8_WEIGHT_CACHE, 1.002 → 1.014
Fast accum on backward GEMMs
✓ only on grad_input, K ≤ 6144: NANOCHAT_FP8_FASTACCUM_DGRAD, 1.032 → 1.043, statistical gate 0.09–0.30σ; both backward GEMMs on was rejected (4–15σ)
Other main levers
column-major activation copy saved in forward, avoiding re-quantization in backward (1.043 → 1.072), single-pass RMSNorm reduction
Raw work log23 sections
0.0 hRound 1 — profile the unchanged reference
(start 2026-11-07 ~02:50 UTC)
0.0 hRound 1 findings (profile of the unchanged reference)
bench.py profile --tree targets/reference --out /tmp/prof-ref-1 (torch.profiler, rank 0, updates 12-13).
Total GPU kernel time 2016.7 ms for 2 updates -> 1008 ms/update (the measured step time is the same, so the
GPU is ~97.5% busy; only 2.5% idle gaps).
Breakdown of the 2016 ms:
- FP8 GEMMs (cuBLAS nvjet_*):        896 ms  44.5%
- triton elementwise/reduction kernels: 821 ms  40.7%   (the FP8 quantization glue, rms_norm, relu^2, loss)
- FlashAttention-3 fwd+bwd:             218 ms  10.8%
- NCCL (optimizer reduce-scatter/all-gather): 45 ms  2.2%
- aten elementwise:                     28 ms   1.4%
The FP8 GEMMs run at ~1.28 TFLOP/s of FP8 (H100 class peak 1979 dense) -> little headroom in cuBLAS.
The memory-bound part is already at the bandwidth roofline (a pure abs().max() over [32768,1536] bf16 takes
35.8 us = 2.8 TB/s), so speedups must come from *removing* passes, not from tuning kernels.
0.5 hRound 2 — screening (all with `bench.py time --tree /submission/nanochat --config <file>`)
| config | change | s/update | speedup vs frozen reference |
|---|---|---|---|
| /tmp/sw/base.json    | none (tree with all switches off)                | 1.0169 | 0.9989 |
| /tmp/sw/fastacc.json | NANOCHAT_FP8_FASTACCUM_BWD=1 (cuBLAS fast accum in the 2 backward FP8 GEMMs) | 0.9883 | 1.0278 |
| /tmp/sw/e2.json      | fastacc + NANOCHAT_FP8_SHARE_QKV=1               | 0.9887 | 1.0274 |
| /tmp/sw/e3.json      | NANOCHAT_FP8_SHARE_QKV=1 only                   | 1.0160 | 0.9998 |
Notes:
* Sharing one FP8 quantization across the q/k/v projections gains nothing: inductor already CSEs the three
  identical _to_fp8 calls on the same tensor. The switch is therefore useless -> will be removed.
* Moving the activation quantization out of the autograd.Function (needed to share it) is performance neutral
  (base row) but adds risk -> keep only if it buys something else.
* fast accumulation in the backward GEMMs is the only win so far: +2.8% (tier N1, changes FP8 accumulation order).
1.5 hRound 3 — isolate the two changes (gate 3 = statistical gate)
(start 2026-11-07 04:00 UTC, end 04:16 UTC)

Isolation runs with bench.py check --n1 (gate 3 only, one 150-update trajectory, seed 1):

| config | change | gate 3 verdict |
|---|---|---|
| /tmp/sw/lossonly.json | NANOCHAT_FUSED_LOSS=1 only | PASS (N1), verdict equivalent |
| /tmp/sw/fastonly.json | NANOCHAT_FP8_FASTACCUM_BWD=1 only | INCORRECT (verdict different) |

* The fused loss perturbs the trajectory by 0.03–0.19 sigma (bound is 3 sigma) -> very safe, kept.
* The cuBLAS fast accumulation in the two backward FP8 GEMMs perturbs the probe loss by 4–15 sigma
  (the K=32768 grad_weight contraction accumulates in fp16) -> rejected, the switch is now off.

Consequence: the speedup must come from changes that do not change the arithmetic (tier N0) plus the
fused loss.  The measured speedup of the fused loss alone is ~0.2 %, so >=1.8 % has to come from N0 work.

### Profile analysis of the reference (torch.profiler, rank 0, updates 12-13)
Total 1012 ms/update.  Per kernel family:
- cuBLAS FP8 GEMMs (nvjet_*):            443 ms/u (44 %)   ~1270 TFLOP/s effective (peak 1979)
- FlashAttention-3 fwd+bwd:                106 ms/u (10 %)
- Triton elementwise/reduction (FP8 glue, rms_norm, loss): ~380 ms/u (38 %)  <- the target
- NCCL (reduce-scatter + all-gather):       27 ms/u
- AdamW/Muon optimizer compute:            ~40 ms/u
Biggest single non-GEMM kernel: 64.6 ms/u, 96 calls/update, grid [96,512] = a 64x64-tiled transposed
fp8 store of the [32768,6144] quantized MLP activation (0.6-1.5 TB/s vs ~3 TB/s roofline).
Second: the two FP8 amax reductions of the MLP tensors (19.6 + 13.0 ms/u) and the fp8 cast of the MLP
activation (18.6 ms/u).
FP8 *weight* quantization + column-major copy (recomputed for every micro-batch although the weights only
change at the optimizer step): ~13.6 ms/u -> can be cached (N0, bitwise identical).
2.0 hRound 4 — compile-mode screening + where the time really goes
(start 2026-11-07 04:29 UTC, end ~04:45 UTC)

Screening with bench.py time (all with fused loss + FP8 weight cache):

| config | change | result |
|---|---|---|
| /tmp/sw/mat.json | NANOCHAT_COMPILE_MODE=max-autotune-no-cudagraphs | aborted after 10 min — inductor autotunes ~400 mm instances (the Muon polar-express matmuls), compile alone > 45 min.  Too slow for the final evaluation (timing timeout 2700 s). |
| /tmp/sw/ro.json  | NANOCHAT_COMPILE_MODE=reduce-overhead (CUDA graphs) | ERROR: SIGSEGV on rank 0 during the first capture.  CUDA graphs are not usable with this model. |

Conclusions: inductor's compile mode is a dead end; the wins have to come from the kernels themselves.

### Where the time goes (re-derived from the profile, per *call* not per update)
The earlier "ms/u" figures of the triton kernels are totals over the 4 micro-batches, so
per-call they are 3-5x smaller than they look.  Re-reading the profile with call counts:
* nvjet (cuBLAS FP8 GEMMs) 451 ms/u, triton pointwise/reduction 411 ms/u, FA3 109, NCCL 23, aten 15.
* the loss forward is not a hotspot: a standalone micro-benchmark of
  log(sum(exp(15*tanh(x/15)))) over the [32768, 98304] bf16 logits takes 3.0 ms (2.1 TB/s,
  i.e. at the roofline); an exp2/tanh rewrite only buys 5 %, a hand written Triton kernel 12 %.
  -> dropped (the 12.4 "ms/u" figure was 4 calls of 3.1 ms).
* the rms_norm backward is the real find: inductor decomposes it into a row-reduction
  kernel + a separate epilogue kernel, i.e. it reads the activation and the gradient *twice*.
  A single fused Triton kernel (read x, read grad, write grad_in) runs at 75 us per
  [16,2048,1536] vs ~180 us for the inductor decomposition.
2.0 hRound 4/5 results (all with `bench.py time`, tree = /submission/nanochat)
| config | change | s/update | speedup vs frozen ref | min per-window ratio |
|---|---|---|---|---|
| /tmp/time-dump-1 (loss only) | fused loss only | 1.0138 | 1.0020 | - |
| /tmp/time-wcache-1 | fused loss + FP8 weight cache | 1.0021 | 1.0137 | 1.0122 |
| /tmp/time-rms-1 | + fused rms_norm backward (custom Triton) | 1.0515 | 0.9660 | - |
| /tmp/time-ws32-1 | + CUBLASLT_WORKSPACE_SIZE=32 MiB | 1.0018 | 1.0140 | - |
| /tmp/time-fadgrad-1 | + fast accum only in the grad_input GEMM (K<=32768) | 0.9896 | 1.0265 | 1.0239 |

Dead ends (documented so they are not retried):
* fused rms_norm backward (custom Triton kernel): the kernel itself is faster (75 us vs 108 us
  per [16,2048,1536] norm) but wrapping the norm in a torch.autograd.Function makes dynamo
  graph-break, so the *forward* of the norm runs eagerly (fp32 intermediates) -> +50 ms/update.
  The inductor rms_norm backward is already at the memory roofline (300 MB in 108 us = 2.8 TB/s).
* max-autotune: compiles for > 45 min (it autotunes the Muon's matmuls) -> unusable.
* reduce-overhead / CUDA graphs: SIGSEGV on this model.
* CUBLASLT_WORKSPACE_SIZE: no effect (1.0018 vs 1.0021, noise).
* fast accumulation in both backward GEMMs (round 3): +2.8 % but 4-15 sigma off on the probe
  loss -> gate 3 rejects it.  Restricting it to the grad_input GEMM (whose contraction length is
  the number of output features, not the 32768 tokens of the micro-batch) keeps almost all of the
  speed and is much closer to the reference numerically.
2.7 hRound 6 — Inductor coordinate-descent tuning (DEAD END: hangs)
(start 2026-11-07 05:15 UTC, killed 05:20 UTC)

Hypothesis. The Triton glue kernels that dominate the memory-bound part of the update run at
~2.1 TB/s while the machine's copy roofline is 4.25 TB/s. Inductor's block sizes for those kernels are
heuristics; torch._inductor.config.coordinate_descent_tuning = True lets Inductor benchmark launch
configurations and pick the best one.

Change. scripts/base_train.py reads NANOCHAT_INDUCTOR_CD_TUNE=1 and sets
torch._inductor.config.coordinate_descent_tuning = True before the model is compiled (switch is off by
default; committed in 118327f).

Result. The run *hung* in the first forward: Inductor compiled ~442 kernel variants between 05:15:36 and
05:18:23 and then both ranks slept forever at 0 % GPU utilisation (a deadlock between the two ranks in the
autotune cache / file locks, or in the tuning benchmark itself). The processes were killed; no judge JSON was
produced (/tmp/time-cdtune-1.log, /tmp/time-cdtune-1/timing_tree.log contain only the setup lines).

Conclusion. Dead end, do not retry without understanding the deadlock. The same lever was already
refuted twice (max-autotune: > 45 min compile; reduce-overhead/CUDA graphs: SIGSEGV).
2.7 hRound 6 profile of the *current best* tree (weight cache + fused loss)
bench.py profile --tree /submission/nanochat --config /submission/switches.json --out /tmp/prof-cur-1
(rank 0, updates 12-13). Total kernel time 1999.4 ms for 2 updates -> 999.7 ms/update (the reference is
1008.4 ms/update in the same profiler).
* FP8 GEMMs (nvjet_*): 452 ms/u (45 %)
* FlashAttention-3: 109 ms/u (11 %)
* Triton glue: ~390 ms/u (39 %)
* NCCL (reduce-scatter + all-gather of the optimizer): 45 ms/u (4.5 %)
Top glue kernels (ms/update, calls/update):
  65.1  triton_poi_fused__scaled_mm__to_copy_clamp_clone_mul_pow_reciprocal_re  (MLP backward: reads
        h + grad in bf16 [32768,6144], writes e4m3 col-major + e5m2 row-major + e5m2 col-major;
        1407 MB/call at 2.07 TB/s -- the transposed stores are the suspect, the neighbouring amax kernel
        reaches 3.94 TB/s)
  20.9 / 18.1 / 14.0 / 13.5 / 10.0  the five *_fused_rms_norm_backward* kernels (76.5 ms/u together:
        Inductor splits the rms-norm backward into a reduction + a pointwise kernel, so x and grad are
        read twice)
  19.6  triton_red_fused__to_copy_abs_max_mul_pow_relu_threshold_backward  (amax of the MLP gradient)
  18.6  triton_poi_fused__scaled_mm__to_copy_clamp_mul_pow_relu_t_view_15  (e5m2 cast of that gradient)
  18.6  triton_per_fused__to_copy_add_cat_mean_mul_neg_pow_rsqrt_slice_view_7 (norm forward, fused with
        the residual add and the value-embedding cat)
  16.2  triton_per_fused__scaled_mm__to_copy__unsafe_view_clamp_embedding_dens  (value-embedding gather)
  16.2  triton_per_fused__to_copy_abs_add_max_slice_backward_view_28  (embedding backward, scatter-add)
  14.7  triton_poi_fused__scaled_mm__to_copy_add_clamp_clone_mul_reciprocal_sl (forward activation quant)
  12.6  the fused soft-capped cross entropy (4 calls/u, 3.1 ms each: it is exp/tanh bound, not memory bound)
3.6 hRound 7 — share one activation quantization between q/k/v (tier N0)
(start 2026-11-07 05:20 UTC, end 05:52 UTC)

Hypothesis. The query, key and value projections of every attention block read the *same*
activation, so the model quantized it three times (three amax passes + three cast passes, and
three copies of the column-major fp8 activation that the weight GEMM needs).

Change. NANOCHAT_FP8_SHARE_QKV=1 (nanochat/gpt.py): quantize the attention input once and
hand the fp8 tensor plus its inverse scale to c_q, c_k and c_v (Float8Linear.forward already
accepted a pre-quantized activation).

Result. bench.py time --tree /submission/nanochat --config /tmp/sw/qkv.json --out /tmp/time-qkv-1
-> speedup_vs_frozen_reference 1.0158 (mean), windows 0.9971 / 1.0024 / 1.0004 s/update.
(The switch is bitwise identical to the reference: the same fp8 values, just computed once.)
3.6 hRound 8 — persistent reductions for the 1536-wide rms-norm (tier N1) *** biggest win so far ***
(start 2026-11-07 05:56 UTC, end 06:19 UTC)

Profile finding (round 6, bench.py profile of the current best tree). The rms-norm reduces over
1536 elements.  Inductor only emits a *persistent* reduction (the whole reduced row stays in
registers) when the reduction is <= 64 elements (1024 for an inner reduction), so it generated a
*split* reduction: the kernel walks the row twice (once to accumulate the sum of squares, once to
apply the scale) and the activation is read from memory twice.

Change. nanochat/inductor_tuning.py (new, off by default, NANOCHAT_PERSISTENT_REDUCTION=1)
patches InductorChoices.should_use_persistent_reduction so that reductions of up to 2048 elements
stay in registers.  Kernels that read or write an fp8 buffer are excluded: a persistent
reduction works on a 1-D tile and the transposed fp8 stores this model needs would be uncoalesced
there (the unfiltered version was a *catastrophic* regression: 1.1049 s/update, speedup 0.9194).
scripts/base_train.py calls apply_persistent_reduction() before the model is compiled.

*Bug found and fixed while implementing this:* SIMDKernelFeatures.buf_accesses is a method,
not a property, and Dep has no .buffer; the fp8 filter has to go through
V.graph.get_buffer(dep.name).get_dtype().  With the bug the filter always raised and silently
disabled the whole switch.

Result. `bench.py time --tree /submission/nanochat --config /tmp/sw/persist.json --out
/tmp/time-persist-2 -> speedup_vs_frozen_reference` 1.0224 (mean), windows 0.9889 / 0.9961 /
0.9955 s/update, per-window ratios 1.0215 / 1.0224 / 1.0235 (min 1.0215).  The benefit bar is
max(1.02, 1 + 2*spread) = 1.02, so the run is now above the bar (was 1.0131 min window).

Tier. N1: the sum of squares of the rms-norm is accumulated in a different order, so the
parameters are no longer bitwise identical (probe loss at update 0: 10.397739 with the switch vs
10.397737 without it - a ~1e-7 relative difference).  Gate 3 (statistical) has to accept it.

Current switch file (/submission/switches.json):
`{"environment": {"NANOCHAT_FUSED_LOSS": "1", "NANOCHAT_FP8_WEIGHT_CACHE": "1",
"NANOCHAT_FP8_SHARE_QKV": "1", "NANOCHAT_PERSISTENT_REDUCTION": "1"}}`
3.6 hDead end (investigated in round 8, not adopted): fast transposed fp8 stores
The biggest single Triton kernel is the MLP backward glue kernel (65 ms/update, 1407 MB per call at
2.07 TB/s while its neighbours reach 3.9 TB/s).  It writes two *transposed* fp8 buffers (the
column-major copies the weight GEMMs need).  A microbenchmark of the exact generated kernel showed:
* unmodified: 0.485 ms (2.90 TB/s)
* with tl.trans on the value only: 0.357 ms (3.94 TB/s) - **but the data ends up transposed
  twice, i.e. wrong**
* with tl.trans on the value *and* the matching index (correct, bitwise identical): 0.493 ms - no
  gain
* with tl.trans on the index variables: 5.98 ms - catastrophic (Triton materialises the
  transpose of the index tile in shared memory)

Triton cannot express "same logical shape, transposed layout" for a value that comes from an
x-contiguous load while the destination pointer is token-contiguous: the layout conversion that the
store needs is exactly the expensive part, and tl.trans changes the logical shape (which breaks
the data pairing).  Conclusion: with the fp8 layout constraints of torch._scaled_mm (mat1 must be
row-major, mat2 column-major - verified: every other combination raises "Only multiplication of
row-major and column-major matrices is supported by cuBLASLt") the transposed copies are
irreducible, and a fast transposing store is not reachable from Triton's layout system.
3.6 hRound 8, follow-up: the statistical gate (gate 3) accepts the persistent reduction
bench.py check --n1 --tree /submission/nanochat --config /submission/switches.json --out /tmp/check-n1-1
-> status PASS, tier N1 (/tmp/check-n1-1/check.json).

The six probe-loss endpoints of the 150-update trajectory (seed 1) deviate from the frozen reference
by 0.22 - 0.30 sigma (bound = 3 sigma), e.g. endpoint 5-15: mean 0.00506 vs sigma_seed 0.02269
(bound 0.06808).  The persistent reduction only changes the order in which the sum of squares of
the rms-norm is accumulated, which is far below the seed-to-seed spread.

State after round 8: a full check would be PASS (N1) with speedup ~1.022.
4.3 hRound 9: tiled Triton FP8 transpose (N0 candidate) — first attempt was a loss, refined version wins
Profile findings (round 9 start). The judge profile of the current best tree
(/tmp/prof-persist-1, rank 0, 989 ms/update of GPU kernel time) breaks down as
GEMM (cuBLAS nvjet) 452 ms/u, FlashAttention-3 110 ms/u, Triton glue 389 ms/u, NCCL 23 ms/u.
Inside the glue the biggest single kernel is the FP8 gradient kernel that writes the *transposed*
FP8 copy the weight GEMM needs: it reads the bf16 gradient with a transposed (uncoalesced) access
pattern and takes 0.675 ms per call, i.e. ~220 GB/s while its neighbours reach 3.9 TB/s.  The
cuBLASLt FP8 GEMM requires the second operand in column-major layout, so those transposed copies
are unavoidable; only their *cost* is negotiable.

Attempt 1 (regression). A Triton kernel that transposes an already quantized FP8 tensor with a
tiled (tl.trans) transpose reaches 3.0-3.9 TB/s in isolation *and* 3.75 TB/s in situ.  Wiring it
in as a torch.library.custom_op (opaque, so Inductor cannot fuse the quantization into the
transposed store) removed every uncoalesced transposed store from the generated code, but the
judge time run gave speedup 1.0079 (windows 1.0026 / 1.0078 / 1.0129) — a 1.4 % *regression*
versus 1.0224.  Reason (from the profiles): making the copy opaque split the fused kernel, and
Inductor then re-generated the remaining read with a 1-D grid whose index arithmetic
(x % 1536, x // 1536, x % 128, x // 128) costs an integer division per element; the same
read in the 2-D grid version was ~1.9x faster (0.153 -> 0.285 ms/call).

Attempt 2 (adopted). Write *one* custom kernel that reads the bf16 gradient once (coalesced)
and writes both FP8 layouts (row-major for the grad-input GEMM, transposed for the grad-weight
GEMM), the transposed store going through a shared-memory tile transpose.  The scale is computed
with the same expression as _to_fp8 (the amax is order independent, so it is bit-identical), so
the FP8 bytes are unchanged.  The activation's column-major copy uses the plain tiled FP8
transpose.  Microbenchmark: 3.5-3.85 TB/s (0.057 ms for [32768,1536], 0.217 ms for [32768,6144],
1.14 ms for [32768,32768]).

Result (local, 2 GPUs). Loss after updates 0/1: 10.397739 / 10.386679 — bit-identical to the
reference.  Update time 982.6 ms -> 952.2 ms.

Switch: NANOCHAT_FP8_TILED_TRANSPOSE=1 (default off).
4.6 hRound 10: fast accumulation in the grad_input FP8 GEMM (N1) — ACCEPTED, 1.0426
Profile findings (round 10 start). The judge profile of the round-9 tree
(/tmp/prof-qp-1, rank 0, ~1050 ms of GPU kernel time per update window) breaks down as
GEMM (cuBLAS nvjet) 481 ms/u, FlashAttention-3 112 ms/u, Triton glue 321 ms/u (of which the
FP8 quantisation path is ~115 ms and the rms-norm family ~62 ms), the custom FP8 kernels
70 ms/u, NCCL 26 ms/u, Muon/AdamW ~30 ms/u and ~43 ms/u of GPU idle (launch latency spread
over ~9.8k kernel launches per update).  The optimizer step alone is 98 ms of the ~980 ms
update (10 %), of which NCCL is 26 ms and the Muon Newton-Schulz GEMMs ~20 ms.

What changed. NANOCHAT_FP8_FASTACCUM_DGRAD=1 with NANOCHAT_FP8_FASTACCUM_MAX_K=6144
turns on cuBLAS' fast-accumulation mode for the *grad_input* FP8 GEMM of every transformer
block.  The vocabulary head is excluded because its contraction (32768) is much longer than
the blocks' (1536-6144), and the earlier experiment that enabled fast accumulation in *both*
backward GEMMs was rejected by gate 3 at 4-15 sigma.  The switch already existed
(commit 118327f, default off); this round validates and adopts it.

Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/dgrad.json
--out /tmp/time-dgrad-1 -> speedup_vs_frozen_reference 1.0426, s_per_update` 0.9743
(was 1.0321 / 0.9879 for the round-9 tree).
`bench.py check --n1 --tree /submission/nanochat --config /tmp/sw/dgrad.json
--out /tmp/check-n1-dgrad2` -> status PASS, tier N1.  The six probe-loss endpoints of the
150-update trajectory deviate by 0.09 - 0.30 sigma (bound 3 sigma):
0.0038 (0.17s), 0.0029 (0.22s), 0.0024 (0.30s), 0.0024 (0.17s), 0.0031 (0.13s), 0.0031 (0.09s).

Current switch file (/submission/switches.json):
`{"environment": {"NANOCHAT_FUSED_LOSS": "1", "NANOCHAT_FP8_WEIGHT_CACHE": "1",
"NANOCHAT_FP8_SHARE_QKV": "1", "NANOCHAT_PERSISTENT_REDUCTION": "1",
"NANOCHAT_FP8_TILED_TRANSPOSE": "1", "NANOCHAT_FP8_FASTACCUM_DGRAD": "1",
"NANOCHAT_FP8_FASTACCUM_MAX_K": "6144"}}`
5.1 hRound 11: fast Triton refresh of the cached FP8 weights (N0) — no measurable gain, kept
Profile findings (round 11 start, judge profile /tmp/prof-r11, rank 0, two updates).
Kernel time per update (ms), from the profiler trace (2 updates, 1937.8 ms of kernel time
=> 968.9 ms/u against a 975 ms update, i.e. the GPU is ~99 % busy):

| bucket | ms/u | share |
|---|---|---|
| cuBLAS FP8 GEMMs (nvjet_*) | 448.5 | 46 % |
| Inductor Triton glue | 302.1 | 31 % |
| FlashAttention-3 (fwd+bwd) | 109.6 | 11 % |
| custom FP8 kernels (_fp8_quant_pair_kernel, _fp8_transpose_kernel) | 69.4 | 7 % |
| NCCL (reduce-scatter / all-gather) | 23.7 | 2.4 % |
| other (eager elementwise, cat, ...) | 15.6 | 1.6 % |

A standalone cuBLAS microbenchmark (/tmp/bench_gemm.py) reproduces the 448 ms/u GEMM time
exactly, so the GEMMs are at cuBLAS speed: 1346-1375 TFLOPS for the grad-weight GEMMs
(contraction 32768), 1311-1328 for the grad-input GEMMs, and 1019-1094 for the forward GEMMs
whose contraction is only 1536 (the measured FP8 peak of this GPU is 1518 TFLOPS at 8192^3).
FA3 runs at 584 TFLOPS on the full-context layers and 388 on the 512-window layers, i.e. it is
at the bf16 GEMM roofline.  There is therefore no GEMM or attention headroom left; the only
slack is in the Inductor glue (302 ms/u) and the custom FP8 kernels (69 ms/u).

What changed. NANOCHAT_FP8_FAST_WEIGHT_REFRESH=1 (default off) replaces the compiled
weight-quantization helper (_quantize_weight_into_compiled: five Inductor kernels plus the
Dynamo guard machinery per weight, 97 weights for D24) with two Triton launches per weight
(amax partials + row quantize) plus the existing tiled FP8 transpose.  The FP8 bytes are
bit-identical: the amax is order independent, the scale is 448 / max(double(amax), 1e-12)
computed in fp64 with an IEEE round-to-nearest division, and the reciprocal uses
tl.math.div_rn (triton's tl.fdiv is *not* reliably round-to-nearest and produced a 1-ULP
different inverse scale, which was caught by the bitwise check).

Result (judge). `bench.py time --tree /submission/nanochat --config /tmp/sw/cur.json
--out /tmp/time-r11b -> speedup_vs_frozen_reference 1.0425, s_per_update` 0.9744
(windows 0.9684 / 0.9748 / 0.9765).  The previous tree measured 1.0426, so the change is
within noise: the refresh costs 0.044 ms per weight instead of 0.095 ms, i.e. ~5 ms of the
975 ms update.  It is bit-identical (N0), so it is kept enabled, but it is not a source of gain.
5.4 hRound 12: column-major FP8 activation copy in the forward pass (N0) — ACCEPTED, 1.072
Profile findings.  The judge profile of the round-11 tree (/tmp/prof-r11, rank 0, two
updates, 1937.8 ms of kernel time = 968.9 ms/u against a 975 ms update) breaks down as
cuBLAS FP8 GEMMs 448.5 ms/u, Inductor Triton glue 302.1 ms/u, FlashAttention-3 109.6 ms/u,
the custom FP8 kernels 69.4 ms/u, NCCL 23.7 ms/u and other 15.6 ms/u.  A standalone cuBLAS
microbenchmark reproduces the 448 ms/u of GEMM time exactly (1346-1375 TFLOPS for the
grad-weight GEMMs, 1311-1328 for the grad-input GEMMs, 1019-1094 for the forward GEMMs whose
contraction is only 1536; the measured FP8 peak of this GPU is 1518 TFLOPS at 8192^3), and
FA3 runs at 584/388 TFLOPS on the full-context / 512-window layers, i.e. at the bf16 GEMM
roofline.  There is no GEMM or attention headroom left.

The interesting part was in the *generated* code (TORCH_LOGS=output_code on a depth-4 model
with the same shapes): the backward graph rematerialises the FP8 activation quantisation.
The backward only needs the activation in column-major layout (the second operand of the
grad-weight GEMM), but instead of saving the FP8 buffer Inductor's min-cut partitioner
recomputes the whole quantisation - read the bf16 activation, compute the amax, cast to FP8 -
and only then transposes it.  For D24 that duplicated ~600 MB of traffic per FP8 linear per
micro-batch (~30 ms/u).

What changed.  NANOCHAT_FP8_ACT_COL_IN_FWD=1 now produces the column-major copy in the
forward pass, with the tiled Triton transpose (_fp8_transpose_op) instead of the eager
x.t().contiguous().t() copy - the eager copy reads and writes with a transposed access
pattern and is several times slower, which is why the earlier attempt with this switch was
neutral.  Because the backward then consumes the forward's copy, the partitioner no longer
has anything to rematerialise.

Result (judge).  `bench.py time --tree /submission/nanochat --config /tmp/sw/actcol.json
--out /tmp/time-actcol -> speedup_vs_frozen_reference 1.072, s_per_update` 0.9476
(was 1.0425 / 0.9744): +2.9 %.

Numerics.  The FP8 bytes and the operand of the grad-weight GEMM are unchanged, so the
change is bitwise identical (N0).  Verified by comparing the 10-update deterministic record
of the tree with and without the switch (check --quick, /tmp/check-quick-actcol and
/tmp/check-quick-cur): the parameter digests are identical, and the exact items pass
(exact_items_ok: true, no problems).
6.3 hRound 13: fused FP8 quantize + column-major copy (N1) — ACCEPTED, 1.0816
Profile findings (round-12 tree, /tmp/prof-r12).  942.6 ms/u of kernel time: cuBLAS FP8
GEMMs 449 ms/u (82-95 % of the 1462 TFLOPS FP8 peak), FlashAttention-3 109 ms/u (at the bf16
roofline), Inductor Triton glue ~225 ms/u, the custom FP8 kernels ~70 ms/u (all at ~3.7 TB/s),
the optimizer ~50 ms/u and NCCL ~26 ms/u.  There is no headroom left in the GEMMs or in the
attention, so the remaining work is in the memory-bound glue.

What changed.  The FP8 activation has two consumers: the forward GEMM operand (row major)
and the column-major copy that the grad-weight GEMM needs.  Inductor materialises the row-major
buffer with its own cast kernel and the column-major copy is then a second pass over the FP8
bytes (NANOCHAT_FP8_QUANT_PAIR, off by default).  A Triton kernel (_fp8_quant_pair_e4m3_kernel)
now reads the bf16 source once and writes both layouts.  The FP8 bytes are identical to
_to_fp8() (checked byte for byte on four shapes), but the custom op is a materialisation
boundary: Inductor can no longer fuse the rms-norm into the quantisation, so the norm output is
rounded to bf16 before it is quantised.  The tree is therefore N1 (it already was, through the
fast-accumulation dgrad GEMM).

Result (judge).  `bench.py time --tree /submission/nanochat --config /tmp/sw/pair.json
--out /tmp/time-pair -> speedup_vs_frozen_reference 1.0816, s_per_update` 0.9392 (was
1.0720 / 0.9476): +0.9 %.  The profile of the new tree (/tmp/prof-r13) shows 930.2 ms/u of
kernel time (-12.4 ms/u): the separate transpose kernel for the plain activations is gone
(_fp8_quant_pair_e4m3_kernel, 16.1 ms/u at 3.6 TB/s) and the forward amax of those activations
is now its own reduction (3.5 ms/u at 2.75 TB/s).

Numerics.  bench.py check --quick -> the exact items pass (exact_items_ok: true) and the
parameter digests differ from the previous tree (the quantised bytes now come from the
materialised bf16 norm output).  bench.py check --n1 -> PASS (N1), the six endpoints of the
150-update trajectory are at 0.05 - 0.24 sigma (bound 3 sigma): probe_s005_015 0.092,
probe_s015_030 0.148, probe_s030_050 0.237, probe_s050_080 0.096, probe_s080_115 0.051,
probe_s115_150 0.104.
7.8 hRound 14: one FP8 kernel for the MLP activation (both layouts) (N0) — ACCEPTED, 1.0907
Profile findings (round-13 tree, /tmp/prof-r13).  930 ms/u of kernel time.  The FP8
activation of the MLP's second projection was the last user of the "plain" (Inductor) cast
path: triton_poi_fused__scaled_mm__to_copy_clamp_fp8_transpose_mul_pow_reciprocal_relu_t_view_12
(20.6 ms/u) fuses relu^2 + the FP8 cast, but its output has two consumers (the operand of
the forward GEMM and the column-major copy that the grad-weight GEMM needs), so Inductor
materialised the FP8 buffer twice (buf59 and buf62, both [32768, 6144] e4m3) and the
column-major copy was then a *third* pass over the data (_fp8_transpose_kernel, 10.7 ms/u).
The amax of the activation (triton_red_fused__to_copy_abs_max_pow_relu_view_10, 13.2 ms/u)
reads the pre-activation a second time.

What changed.  NANOCHAT_FP8_RELU2_PAIR=1 (new, off by default) quantizes the
pre-activation with a dedicated Triton kernel (_fp8_quant_pair_relu_e4m3_kernel) that applies
relu^2 in fp32, casts to e4m3 and writes both layouts in one pass.  The relu^2 derivative
is applied by a new node (_Relu2Grad: identity forward, derivative backward), so Inductor
keeps fusing the derivative - and the amax of the gradient that is fused into it - into the
backward graph exactly as before.  Traffic per MLP per micro-batch: 1608 MB -> 1206 MB.

Result (judge).  `bench.py time --tree /submission/nanochat --config /tmp/sw/relu2.json
--out /tmp/time-relu2c -> speedup_vs_frozen_reference 1.0907, s_per_update` 0.9313
(was 1.0816 / 0.9392): +0.8 %.  Windows 0.9259 / 0.9328 / 0.9351 s/u.

Numerics (N0).  Unit test: the scale, the row-major FP8 bytes, the column-major FP8 bytes
and the relu^2 gradient are bit-identical to the unfused path.  bench.py check --quick ->
exact items pass (exact_items_ok: true); the 10 record updates x 175 parameter digests are
identical to the previous tree (/tmp/check-quick-relu2/record_cand.r0.json vs
/tmp/check-full-pair/record_cand.r0.json), so the tree stays exactly as it was (N1 vs the
frozen reference, through the fast-accumulation dgrad GEMM).
7.9 hRound 15: wider FP8 quantize tiles + a dedicated amax reduction for the relu^2 activation (N1, bitwise-neutral) — 1.0951
Profile findings (round-14 tree, /tmp/prof-r14).  The three "quantize + write both layouts"
kernels are memory-bound and run at ~3.4 TB/s (the roofline of this GPU), so they can only get
faster with fewer bytes or a wider store.  The forward amax of the MLP activation was still an
Inductor reduction over the *pre-activation* (13.2 ms/u, 2.93 TB/s).

What changed (one commit, 60122f5, both behind the existing switches):
1. The three pair kernels now use a 64 x 256 tile instead of 64 x 128 (the row-major store is
   then 256 B wide).  Microbenchmark: 32768 x 32768 quantize 1.143 -> 1.112 ms, 32768 x 6144
   0.217 -> 0.212 ms (2-3 % per kernel).  The tile falls back to 64 x 128 when N % 256 != 0.
2. New Triton kernel + custom op nanochat::fp8_amax_partials_relu (1024 programs, block 4096):
   the partial maxima of relu(x)^2 (4.18 TB/s vs Inductor's 2.93 TB/s).  forward_relu2 now uses
   it and keeps the *same* scale expression, so the FP8 bytes are unchanged (a maximum does not
   depend on the order of the reduction, and the squares are computed in fp32 exactly like
   Inductor's fused kernel does).

Result (judge).  `bench.py time --tree /submission/nanochat --config /submission/switches.json
--out /tmp/time-r15a -> speedup_vs_frozen_reference 1.0951, s_per_update` 0.9276 (was
1.0907 / 0.9313): +0.4 %.  Windows 0.9224 / 0.9283 / 0.9321 s/u.

Numerics.  bench.py check --quick --out /tmp/check-quick-r15 -> the exact items pass
(exact_items_ok: true, status: NOT_BITWISE because the tree is tier N1 through the
fast-accumulation dgrad GEMM).  The 10 record updates x 175 parameter digests are identical
to the round-14 tree (/tmp/check-quick-r15/record_cand.r0.json vs
/tmp/check-full-pair/record_cand.r0.json) -> the change is numerically neutral (same N1
tier, the statistical gate of the last full check still applies).
8.8 hRound 16: a dedicated Triton kernel for the soft-capped cross entropy backward (N1) — 1.0983
Profile findings (round-15 tree, /tmp/prof-r15).  918 ms/u of kernel time per update.
The complete inventory (from the profiler trace) shows that almost everything is at the
memory roofline of this GPU (4.2-4.3 TB/s): the FP8 quantization kernels (99 ms/u), the
relu^2 gradient + amax (29.8), rope + QK-norm (18.8), the rmsnorm backward (23.5), the
gradient accumulation (10.2), the amax of the gradients (16.6).  The GEMMs (449 ms/u, 49%)
run at ~64 % MFU, FA3 (109 ms/u) is at its roofline, NCCL (23.4 ms/u) is NVLink bound.
The one clearly *compute* bound piece left is the loss: the backward kernel
(triton_red_fused__to_copy_abs_arange_clamp_div_eq_exp_max_mul_rsub_sub_tanh...) took
3.157 ms/call (12.63 ms/u) for 1.07 G elements, the forward 1.16 ms/call (4.64 ms/u).
The backward is slow because the FP8 amax of the head's gradient (the head is an FP8
linear) is fused into it: the kernel then becomes a reduction over the whole vocabulary.

What changed.  NANOCHAT_FAST_LOSS_BWD=1 (new, off by default) computes the gradient of
the soft-capped cross entropy with a dedicated Triton kernel (nanochat::loss_bwd), one
program per (row, block of 4096 columns).  The arithmetic is the one of the compiled
graph, in the same order: the division by the soft cap is the multiplication by its fp32
reciprocal that Inductor emits, tanh/exp are the same libdevice functions, the scale is
grad_out / count with Triton's own division (the *same* div.full.f32 that Inductor
emits - an IEEE division here is worth 1 ulp on ~1.3e-5 of the elements) and the result is
rounded to bf16 at the very end.  The unit test on a real model: the 21 parameter
gradients are bitwise identical to the ones of the Inductor kernel.

Result (judge).  bench.py time --config /tmp/sw/lossbwd.json ->
speedup_vs_frozen_reference 1.0983, s_per_update 0.9249 (was 1.0951 / 0.9276).

Numerics (N1).  bench.py check --quick -> the exact items pass, the status is
NOT_BITWISE (the tree already was tier N1 through the fast-accumulation dgrad GEMM) and
the 10 record updates x 175 parameter digests differ from the round-15 candidate: the
amax of the head's gradient is now taken over the bf16 gradient (a separate reduction)
instead of over the fp32 values that the fused kernel reduced, so the FP8 scale of the
head's gradient changes in the last bits.


Full check (judge).  `bench.py check --tree /submission/nanochat --config /submission/switches.json
--out /tmp/check-full-r17 -> PASS, performance.speedup` 1.0981, tier N1 (the tree is not
bitwise identical to the reference because of the fast-accumulation dgrad GEMM of round 10; the
statistical gate passed: every endpoint of the 150-update trajectory is within 3 sigma of the
reference).  /tmp/check-full-r17/check.json.
8.8 hRound 17: the amax of the head's gradient in the loss backward kernel (N0) — 1.0998
What changed.  The Triton kernel of round 16 now also reduces the maximum of the
gradient (over the fp32 values, exactly like the fused Inductor kernel did) and hands
it over to the FP8 path of the head through a small hand-off slot in nanochat/fp8.py
(nanochat::grad_amax): the head is the only FP8 layer whose gradient comes out of the
loss, and the FP8 path asks for the amax right after the loss backward, in the same order.
The maximum does not depend on the order of the reduction, so the value is the one the
fused kernel produced.  With the hand-off, the separate amax pass over the 2 GB gradient
disappears and the whole loss backward becomes one kernel.

Result (judge).  bench.py time --config /tmp/sw/lossbwd.json ->
speedup_vs_frozen_reference 1.0998, s_per_update 0.9236 (windows 0.9178 / 0.9248 /
0.9281), i.e. +0.4 % over the round-15 tree.

Numerics (N0).  A one-layer model with the FP8 head gives the same loss and the same
15 parameter gradients as the round-15 tree (the only difference is the token embedding's
gradient, which is atomic-nondeterministic in *both*).  bench.py check --quick -> the
10 record updates x 175 parameter digests are identical to the round-15 candidate
(/tmp/check-quick-r17/record_cand.r0.json vs /tmp/check-full-pair/record_cand.r0.json)
-> the change is numerically neutral, the tree keeps the tier N1 of the fast-accumulation
dgrad GEMM and the statistical evidence of the last full check still applies.

Dead ends of this round (measured, not guessed).
* The FP8 amax of the head's gradient is *not* the amax of the bf16 gradient: Inductor
  reduces the fp32 values (it removes the bf16 round trip of grad.float().abs().max()),
  so a separate reduction over the bf16 gradient gives a different scale (this was the
  N1 of round 16).
* The scale of the loss backward is grad_out / count computed *inside* the kernel with
  Triton's non-IEEE division (div.full.f32), not with the IEEE division of eager: with
  a count that is not a power of two the two differ by 1 ulp on ~1.3e-5 of the elements.
* The Inductor kernel of the loss backward and the Triton kernel of round 16 do not have
  the same arithmetic: the Inductor one is a 2D reduction (the amax is fused into it), the
  Triton one is pointwise, and the fp32 values of the gradient are the same only if the
  amax is reduced over the same (fp32) values.
* Addressing the gradient to hand the amax over does not work: the tensor that reaches
  the FP8 path does not have the address of the tensor the loss kernel returned.
9.6 hRound 18: the rotary embedding and the QK-norm in dedicated Triton kernels (N1) — 1.1279, full check PASS 1.1263
Profile findings (round-17 tree, /tmp/prof-r18, 3 updates, 916.6 ms/u).  cuBLAS/nvjet FP8
GEMMs 451 ms/u (49 %, 814-1378 TFLOP/s), FlashAttention-3 110 ms/u, the FP8 quantize kernels
~113 ms/u (all at the roofline), NCCL 23 ms/u, the optimizer 49.5 ms/u, **the fused
rope + QK-norm kernel 19.0 ms/u (96 calls per update, 191 us each) and its backward 23.7 ms/u**
(the two rms_norm_backward kernels), the value-embedding backward ~21 ms/u, the loss 11.8 ms/u,
the gradient accumulation 10.1 ms/u; the GPU is idle only 3.5 ms/u, so the update is GPU-bound.

Why the rope kernel is slow.  Inductor fuses rms_norm(rope(x)) * 1.2 into one kernel that
reduces over the head dimension (128 elements) with one row per program: the reduction is a
cross-warp reduction for a single 256-byte row, so the kernel moves 600 MB (it also does the
rotation of q *and* k) in 191 us, i.e. 3.2 TB/s where the machine does 4.2 TB/s on a copy.

What changed.  NANOCHAT_FUSED_ROPE_QKNORM=1 (new, off by default) computes the rotation,
the QK-norm and the 1.2 factor with two dedicated Triton kernels (nanochat/rope_qknorm.py,
forward and backward).  They do exactly the arithmetic of the compiled graph - the same fp32
operations, the same libdevice.rsqrt(sum(y^2)/128 + 1.1920928955078125e-07), the same 1.2
factor, the same sum((g*rstd)*(y*1.2)) of the backward, the same fp32 inverse rotation - but
16 rows share one program, so the reduction is a shuffle over a few lanes: 49 us for the
forward (201 MB) and 74 us for the backward (302 MB), i.e. 4.1 TB/s, the roofline.  The
backward recomputes the rotation instead of reading back the rotated values the forward used to
save, so the forward writes half as many bytes.

Numerics (N1).  The sum of squares and the dot product are accumulated in a different order
(the kernel sums the two halves of the head dimension separately), which changes the fp32 rstd
by an ulp: 115 of 50.3 M elements of the forward and 1580 of 50.3 M of the backward differ from
the compiled reference (the bf16 rounding hides the difference on the others).  On a 2-layer D24
model the loss is bitwise identical and only attn.c_proj.weight sees a difference (fp32
noise, 3e-8 on a gradient whose peak is 6e-6).

Result (judge).  `bench.py time --tree /submission/nanochat --config /tmp/sw/rope.json
--out /tmp/time-r18a -> speedup_vs_frozen_reference 1.1279, s_per_update` 0.9006 (was
1.0998 / 0.9236): +2.6 %.

Full check (judge).  `bench.py check --tree /submission/nanochat --config /tmp/sw/rope.json
--out /tmp/check-full-r18 -> PASS, performance.speedup` 1.1263 (windows 1.1263 /
1.1294 / 1.1264), tier N1: the exact items pass (exact_items_ok: true), and the six
endpoints of the 150-update trajectory are at 0.11 - 0.20 sigma (bound 3 sigma).

Dead end of this round (measured).  Rounding the gradient to bf16 *before* the inverse
rotation (as the eager _fused_rms_norm_backward would do) is *not* what the compiled graph
does: the dumped Inductor kernel keeps the whole backward in fp32 and rounds only once, at the
end.  With the eager rounding 44 % of the elements differ from the compiled reference, without
it 0.003 %.
9.9 hRound 19: a dedicated Triton amax for the plain FP8 activations (N0 candidate) — REVERTED (1.1227, a 0.5 % regression)
Profile findings (round-18 tree, /tmp/prof-r19, 2 updates, 886.8 ms/u of kernel time).
nvjet FP8 GEMMs 445 ms/u (50 %), FlashAttention-3 101 ms/u, the FP8 quantize kernels
89.6 ms/u (all at the roofline: 3.3-3.9 TB/s), the Inductor reduction that fuses the
relu^2 backward with the amax of its gradient 29.8 ms/u (3.9 TB/s), the value-embedding
scatter 11.2 ms/u (1.3 TB/s - the only clearly sub-roofline kernel of the update), the
value-embedding gate multiply 3.95 ms/u, the cast of the fp32 scatter buffer 3.58 ms/u,
the rope kernels 25 ms/u, NCCL 25 ms/u, the optimizer 46 ms/u, the loss 11.8 ms/u, the
gradient accumulation 10 ms/u.  The GPU is idle only ~13 ms/u: the update is GPU-bound and
almost every kernel already runs at the copy roofline.

What was tried.  NANOCHAT_FP8_AMAX_PARTIALS=1 (new, off by default): the amax of a
plain activation is computed by a dedicated Triton kernel (4096 partial maxima, 8192
elements each) instead of Inductor's reduction, and the relu^2 amax kernel is retuned
(1024->4096 parts, 4096->16384 block).  The amax value and therefore the FP8 bytes are
bit-identical (a maximum does not depend on the order of the reduction), so this is a
tier N0 change.

Result (judge).  `bench.py time --tree /submission/nanochat --config /tmp/sw/amax.json
--out /tmp/time-r19a -> speedup_vs_frozen_reference` 1.1227 (windows 0.9004 / 0.9055 /
0.9086), against 1.1279 for the round-18 tree: a 0.5 % regression, reverted
(git checkout -- nanochat/fp8.py, nothing to commit).

Why (measured).  Inductor fuses the amax of an activation into the kernel that
*produces* it (the rms-norm backward, the gate multiply, ...).  Forcing the amax into a
separate custom op breaks that fusion, so the tensor has to be read once more: the
standalone partial-max kernel (26 us for 100 MB) plus the extra read of the producer's
output costs more than the 11 us that the faster reduction saves.  The same argument
applies to the retuned relu^2 amax kernel.
↑ NanoChat