GPT-6.2 Sol 1.10×
Harness: Codex · switches: NANOCHAT_TRAIN_MICROBATCH_MULTIPLIER=2 NANOCHAT_FUSED_ROTARY_NORM=1 NANOCHAT_TUNE_POINTWISE=1 NANOCHAT_TMA_FP8_RELU_CAST=1 NANOCHAT_FUSED_FP8_MLP_ACTIVATION=1 NANOCHAT_TMA_FP8_LINEAR_GRAD=1
- Judged
- 1.10×
- Tier
- N1
- Self-best
- 1.11×
- Rounds
- 20
- Hours
- 10.0
- API cost
- $13.0
- Input tokens · 98.4% cached
- 70 M
- Output tokens
- 0.4 M
- Requests
- 490
§ ARaw trace
Raw work log20 sections
0.0 hRound 1 — baseline profile and first hotspot
- Start: 2026-11-18 06:44:10 UTC, GPU lock acquired for the unchanged upstream reference. - Detached judge profile: PID 227746,/tmp/nano-r1-profile.log, output/tmp/nano-r1-profile. - Rank-0 profiler, updates 12–13: forward compiled regions 29.8% and backward 65.1% of attributed CUDA time. Kernel sums: nvjet GEMMs 44.7%, named FP8 glue 20.9%, FA3 10.8%, cross-entropy 2.8%, NCCL 2.3%, other 18.5%. The largest non-GEMM kernel was fused MLP backward FP8 cast/transpose: 133.35 ms across 192 calls (6.44%); its amax was another 39.86 ms (1.92%). Percentages are profiler attribution, not throughput. - First experiment/tmp/nano-r1-tree: split the large FP8 gradient transpose from the elementwise producer and use a tiled byte-copy kernel. Switch/tmp/nano-r1-switches.json. First launch failed before GPU acquisition because the switch file was not yet created (PID 243287,/tmp/nano-r1-time.log). Corrected launch PID 243295,/tmp/nano-r1-time-a.log, output/tmp/nano-r1-time-a, GPU lock 06:51:03. - First result 06:53:58:timeok, 1.0026×, 1.0397 s/update. Too small to qualify; not promoted, no numerics tier measured. The extra read/write from splitting likely consumes the transpose gain. - Second experiment/tmp/nano-r1b-tree: explicitly coalesced dual-layout cast for the large MLP gradient, fusing the ReLU-square derivative with E5M2 conversion and transpose. The amax is still computed on the original bf16 derivative; the GEMMs and their precision flags are unchanged. Enabled by/tmp/nano-r1b-switches.json. Detached timing PID 258751,/tmp/nano-r1-time-b.log, output/tmp/nano-r1-time-b, GPU lock 06:56:54. - Second experiment's initial Gluon kernel failed at compile time at 06:58:55 (gl.clampnot available); an isolated GPU correctness test then found a bad layout conversion. Replaced it with a Triton 2D tiled kernel. Isolated checks under the GPU lock matched both reference FP8 layouts byte-for-byte for 512×1536, 129×197, and the workload's 32768×6144. Retiming detached PID 273612,/tmp/nano-r1-time-c.log, output/tmp/nano-r1-time-c, GPU lock at 07:04:26. - End 07:07:27:timeok, 1.0057× (1.0303/1.0363/1.0428 s/update). Not promoted: splitting or retiling this producer alone did not deliver enough end-to-end gain. No judge numerics tier measured because neither timing qualified.
0.4 hRound 2 — GEMM algorithm selection
- Start 07:08:05 UTC: candidate remains upstream. Detached profile PID 289035,/tmp/nano-r2-profile.log, output/tmp/nano-r2-profile, updates 11–12. - Profile finished 07:10:45. FP8 scaled matmuls were 42.77% of CUDA attribution; largest kernel variants individually 15.90%, 8.67%, and 8.21%. FA3 backward 8.26%, largest FP8 cast 6.41%, all collectives 2.27%, optimizer bf16 BMM 2.04%. - Screened native GEMM autotuning and persistent TMA templates with/tmp/nano-r2-switches.json. Detached PID 304925,/tmp/nano-r2-time-a.log, output/tmp/nano-r2-time-a, GPU lock 07:11:28. - Result 07:18:36: 1.0047×, 1.0375 s/update. Autotuning selected the same ATen FP8 GEMMs for all relevant shapes, with several minutes extra compilation; not promoted. - ScreenedCUBLASLT_WORKSPACE_SIZE=262144in/tmp/nano-r2-workspace.json. Detached PID 333437,/tmp/nano-r2-time-b.log, output/tmp/nano-r2-time-b, GPU lock 07:18:38 to 07:21:33. Result 1.0049×, 1.0373 s/update; not promoted. - After switch-only attempts, implemented a direct TMA weight-gradient GEMM for large hidden projections in/tmp/nano-r2c-tree. It reads the original E5M2 and E4M3 token-major tensors and uses FP32 accumulation without materializing either global transpose; it preserves BF16 GEMM output. Deterministic tile-local reduction, no atomics. Isolated numerical tests returned relative norm differences of 0.00057–0.00059 compared with the reference GEMM. Time PID 350213,/tmp/nano-r2-time-c.log, output/tmp/nano-r2-time-c, GPU lock 07:23:36. Switch/tmp/nano-r2c-switches.json. - Direct weight gradient result 07:26:39: 0.5951×, 1.7516 s/update. In-kernel layout/scheduling penalties outweigh the saved transposes by a large margin; not promoted. - Screened GEMM batch geometry in/tmp/nano-r2d-tree: regroup the same ordered rows into two batches of 32 per rank, preserving total tokens, model, loss, and schedules. Existing BOS loader packs the same row sequence independently of its yield boundary. Switch/tmp/nano-r2d-switches.json. Detached time PID 359216,/tmp/nano-r2-time-d.log, output/tmp/nano-r2-time-d, GPU lock at 07:28:51. - Result 07:31:50: 1.0304×; windows 1.0104/1.0099/1.0145 s/update; peak allocated 94.04 GiB, reserved 96.37 GiB. First candidate above threshold. Detached exact check--quickPID 373543,/tmp/nano-r2-quick-d.log, output/tmp/nano-r2-quick-d, GPU lock 07:35:13. - End 07:37:22:--quickreturned NOT_BITWISE, with every exact item passing (10 updates, 175 parameters, 640 token rows/rank). N1 trajectory detached PID 380255,/tmp/nano-r2-n1-d.log, output/tmp/nano-r2-n1-d, GPU lock 07:37:27. Promoted as the only candidate clearing the frozen benefit screen, with the trajectory pending; commita4e92ea. - The trajectory finished 07:42:36:PASS (N1), equivalent, largest endpoint difference 0.424σ against the 3σ bound. Finalized the commit message and updated the submission to commit988d502(same source).
1.0 hRound 3 — profile the N1 candidate
- Start 07:42:40 UTC. Profile PID 397243,/tmp/nano-r3-profile.log, output/tmp/nano-r3-profile, updates 11–12, current submission and switches. - Profile finished 07:45:21. GPU kernel sums: nvjet 46.0%, FP8 glue 20.6%, FA3 11.1%, cross-entropy 2.8%, NCCL 2.4%, other 17.2%. Largest FP8 cast 6.43%. QK RoPE/RMSNorm forward 1.80%, backward normalization 0.95% plus 1.34% rotary adjoint. Large residual RMSNorm backward is also replicated in a series of high-register FP8 amax and scalar-gradient reductions. Chose normalization/backward fusion boundaries. - Ran full check of988d502, PID 410067,/tmp/nano-r2-full-d.log, output/tmp/nano-r2-full-d. Gate locks 07:45:47 (timing), 07:48:47 (record), 07:50:58 (trajectory); finished 07:56:07. PASS, N1, 1.0291×, windows 1.0091/1.0148/1.0149 s/update. - In/tmp/nano-r3-tree, isolated non-QK RMSNorm backward in a row-tiled FP32 reduction kernel to avoid recomputing its derivative in different consumers. Forward is still the PyTorch fused RMSNorm. Switch/tmp/nano-r3-switches.json; eager and compiled unit checks gave exact forward, no more than 2.0e-5 relative gradient error. Detached test/time PID 430348,/tmp/nano-r3-time-a.log, output/tmp/nano-r3-time-a, timing lock 07:56:17. - Prepared independent QK-only rotary/RMSNorm/temperature fusion in/tmp/nano-r3b-treeunder/tmp/nano-r3b-switches.json: preserves the original BF16 rounding boundaries, FP32 normalization reductions, and original FA3. Isolated correctness test prepared before timing. - Isolated RMSNorm backward finished 07:58:48: 1.0276×, 1.0144 s/update, 98.54 GiB allocated. Not an improvement on 32-row batching; materialization raises memory and did not pay off. - QK test/time PID 452723,/tmp/nano-r3-time-b.log, output/tmp/nano-r3-time-b, timing lock 07:59:00–08:01:44. Unit forward/backward relative errors ≤1.5e-5 versus eager and ≤0.00374 versus compiled (Inductor elides some eager BF16 rounding in fused regions). Measured 1.0454× (0.9928/0.9994/0.9990 s/update), about 1.5% additional throughput over the 32-row candidate. Peak allocated unchanged at 94.04 GiB. - QK validation: detached quick→full PID 470025,/tmp/nano-r3-check-b.log, outputs/tmp/nano-r3-quick-band/tmp/nano-r3-full-b.--quicklock 08:03:16. This includes cosmetic kernel renames and a fallback guard for rotary tables requiring gradients. The passing microbatch-only version remains submitted during validation. - End of screening at 08:05:13: quick returned NOT_BITWISE with all exact items passing. Full check locks 08:05:14 (timing), 08:07:58 (record), 08:09:48 (trajectory); finished 08:14:37: PASS N1, 1.0437×, windows 0.9934/1.0007/1.0020 s/update. Largest absolute endpoint drift 0.352σ. Promoted the fusion as commit88c9ead; no dependency changes.
1.6 hRound 4 — profile and activation headroom
- Start 08:15:48 UTC. Detached profile PID 514978,/tmp/nano-r4-profile.log, output/tmp/nano-r4-profile, updates 11–12 of commit88c9eadwith its submitted switches. - Profile finished 08:18:14. FP8 GEMMs 44.75% of attributed CUDA, backward compiled region 64.61%, forward 29.43%, FA3 backward 8.65%, largest FP8 derivative cast 6.55%, optimizer 4.46%, collectives 2.42%. Peak allocation/reservation 94.04/96.37 GiB. Scaling observed 16→32-row activation memory to 64 rows suggests the default plan will not fit. - Screened 64-row microbatches (once per rank per optimizer update, unchanged total tokens) with AOTAutograd activation budget 0.70 in/tmp/nano-r4-prep, switch/tmp/nano-r4-switches.json. This trades saved activations for recomputation, no dtype or model changes. Detached timing PID 530105,/tmp/nano-r4-time-a.log, output/tmp/nano-r4-time-a, GPU lock 08:20:13. - First timing failed 08:22:55 with CUDA OOM on the 4 GiB E5M2 head weight-gradient transpose (131.89 GiB allocated, 5.37 GiB inactive reserved). Retested the same budget with expandable allocator segments via/tmp/nano-r4b-switches.json: PID 547739,/tmp/nano-r4-time-b.log, output/tmp/nano-r4-time-b, GPU lock 08:25:21. - Prepared a more targeted head weight-gradient paneling option in/tmp/nano-r4c-tree. It would reuse the original tensorwise FP8 scales, transpose up to 16,384 token rows at a time, and reduce the weight products in FP32 before the reference BF16 output cast. This replaces the single 4 GiB transpose allocation with smaller panels; deterministic fixed reduction order. - End 08:28:44: expandable segments made 64 rows fit (135.89 GiB allocated, 138.23 GiB reserved), buttimewas 0.8004×, 1.3023 s/update (1.3838/1.3030/1.2200). The overall rematerialization/large-shape cost dominates the head workspace issue. Closed the direction; did not time the head-panel draft because it does not address that regression. No changes promoted.
1.8 hRound 5 — memory-kernel compiler settings
- Start 08:30:26 UTC. Detached profile PID 565865,/tmp/nano-r5-profile.log, output/tmp/nano-r5-profile, updates 11–12 of current passing88c9ead. - Profile finished 08:32:51. FP8 GEMMs 44.71% of CUDA attribution, FA3 backward 8.63%, the dual-layout MLP FP8 cast 6.54%, its amax 1.93%, and optimizer step 4.48%. Focused on pointwise layout and streaming costs. - Config-only screening of max pointwise autotune plus coordinate descent with/tmp/nano-r5-switches.json. Detached timing PID 582935,/tmp/nano-r5-time-a.log, output/tmp/nano-r5-time-a, GPU lock 08:33:33. - Prepared a separate packed CUDA alternative in/tmp/nano-r5b-tree: one 128×128 tile reads BF16 operands, casts to the same E5M2 values, writes 32-bit packs in both layouts using a shared-memory XOR swizzle. The original scale and FP8 GEMMs are unchanged. CPU-only offline build succeeded (32 registers, 16 KiB shared memory, no spills). Queued locked unit check and judgetimeafter the tuner, PID 623277,/tmp/nano-r5-time-b.log, output/tmp/nano-r5-time-b, switch/tmp/nano-r5b-switches.json. - Config timing finished 08:41:53: 1.0633×, 0.9803 s/update, windows 0.9770/0.9819/0.9821. Cold startup is longer owing to autotuning. Exact→full validation queued after the packed-cast screen, PID 655819,/tmp/nano-r5-check-a.log, outputs/tmp/nano-r5-quick-aand/tmp/nano-r5-full-a. - Packed cast matched both reference byte layouts for 384×512 and 65536×6144. Timing acquired GPUs 08:42:01. - Packed cast timing finished 08:44:42: 1.0508×, 0.9920 s/update. A small improvement over the submitted version, but below the compiler-only result and insufficient to justify a CUDA extension. Not promoted. - Stopped the first validation at 08:51:27: in torch 2.9.1 coordinate descent can benchmark after initial configuration selection even with deterministic algorithms enabled, so that record would not provide deterministic evidence. Replaced the environment flags with a single opt-in in/tmp/nano-r5c-treethat uses the same compiler settings in production and skips tuning whentorch.are_deterministic_algorithms_enabled(). Nothing detects bench mode. Switch/tmp/nano-r5c-switches.json, new quick→full PID 694248, log/tmp/nano-r5-check-c.log, outputs/tmp/nano-r5-quick-cand/tmp/nano-r5-full-c; lock 08:52:38. - End 08:59:53: the quick check finished at 08:54:28 with every exact item passing, verdictNOT_BITWISE. No promotion yet; the full check holds the GPUs separately from the round. - Full check finished at 09:14:53:/tmp/nano-r5-full-c/check.jsonPASS (N1), 1.0619×; all exact items matched and the largest drift was 0.447σ. Promoted the deterministic-aware opt-in ase9563a3and enabled it in the submitted switch file.
2.5 hRound 6 — profile the tuned build
- Start 09:15:48 UTC. Detached judge profile PID 813875,/tmp/nano-r6-profile.log, output/tmp/nano-r6-profile, updates 11–12 of submittede9563a3with pointwise tuning enabled. - Profile finished 09:23:52. FP8 GEMMs 45.81%, FA3 backward 8.75%, largest dual-layout FP8 cast 6.61% plus 1.92% for amax, and optimizer step 4.79% of attributed CUDA time. Collectives were 2.52%, with reduce-scatter and all-gather each taking about 25 ms over two updates. This round targets optimizer staging and communication, not compute kernels. - First screen:NCCL_PROTO=Simplein/tmp/nano-r6a-switches.json, PID 868698, log/tmp/nano-r6-time-a.log, output/tmp/nano-r6-time-a, GPU lock 09:24:17. - Protocol result at 09:32:30:timeok, 1.0644× and 0.9793 s/update. Only 0.1% ahead of the tuned production screen (1.0633×); not enough to justify changing the default protocol. No numerics claim made. - Second screen:/tmp/nano-r6b-treepacks Muon weights once into the rank-ordered all-gather layout and stacks the gradients once per update; the same Muon update operates on a view of the owned slice. Exact arithmetic and FP32 storage are unchanged. Switch/tmp/nano-r6b-switches.json, PID 886286, log/tmp/nano-r6-time-b.log, output/tmp/nano-r6-time-b, queued after the protocol screen at 09:26:53. No experiments enabled in the submission. - Packed-buffer timing acquired GPUs at 09:32:34 and finished at 09:40:54: 1.0667×, 0.9772 s/update (0.9727/0.9782/0.9806), just 0.3% ahead of the tuned screen, with all windows improving. Queued quick check PID 978962, log/tmp/nano-r6-quick-b.log, output/tmp/nano-r6-quick-b. - End 09:43:34: quick check/tmp/nano-r6-quick-b/check.jsonhad no exact-item failures, statusNOT_BITWISE. The packed-buffer point estimate is only 0.3% faster than the tuned screen, not enough to spend a full check or adopt the added storage aliasing. Both variants stay off.
3.0 hRound 7 — profile additional kernel traffic
- Start 09:44:10 UTC. Detached judge profile PID 984962,/tmp/nano-r7-profile.log, output/tmp/nano-r7-profile, updates 11–12 of the fully validatede9563a3. - Profile finished 09:52:10. FP8 GEMMs 45.74%, kernels named FP8 cast plus clone 12.89% (largest MLP gradient fusion 6.67%), and loss kernels 2.70% of kernel CUDA time. No standalone large transpose appeared: the compiler already fuses various copies into casts. The remaining question is how it places that work relative to the forward and backward GEMMs. - Screen/tmp/nano-r7a-treesaves the same E4M3 operands for FP8 backward in their column-major layouts during forward, giving Inductor the opportunity to emit both layouts from one cast. Scales, output rounding, and GEMM options are unchanged. Switch/tmp/nano-r7a-switches.json; detached timing PID 1041474, log/tmp/nano-r7-time-a.log, output/tmp/nano-r7-time-a, started 09:52:55 UTC. - Prepacking result at 10:01:18: 1.0618×, 0.9817 s/update (0.9768/0.9835/0.9848), no improvement over the tuned build. Moving the transposes did not reduce end-to-end traffic enough to justify the rewrite. - Independent variant/tmp/nano-r7b-treeasks Inductor to consider multidimensional tiling for the strided casts and their supporting reductions, without changing the optimizer compiler options. Switch/tmp/nano-r7b-switches.json. Detached PID 1055824, log/tmp/nano-r7-time-b.log, output/tmp/nano-r7-time-b, queued at 09:54:47 after the operand-layout screen. - End 10:09:56: tiling variant held GPUs from 10:01:23 through 10:09:52 and measured 1.0606×, 0.9828 s/update (0.9798/0.9852/0.9835). Neither variant improved on the tuned screen. No promotion or numerics claim.
3.4 hRound 8 — reprofile the head loss
- Start 10:10:40 UTC. Detached judge profile PID 1153678,/tmp/nano-r8-profile.log, output/tmp/nano-r8-profile, updates 11–12 of submittede9563a3. - Profile finished 10:18:47. Soft-cap loss normalization took 0.84% of CUDA, its gradient/amax 0.91%, and its gradient cast/transpose 0.89% (51.9 ms total across two updates); gradient casts already fuse with the soft-cap derivative. Earlier main hotspots remain dominant. -/tmp/nano-r8a-treecomputes the same soft-capped mean cross-entropy in FP32 with one row sum rather than max shifting. The ±15 bound makes the unshifted exponential safe. It saves only raw logits and row log-normalizers, and its explicit backward remains visible to the FP8 compiler fusions. CPU comparisons against upstream passed for FP32 and BF16 logits, from near-zero to saturated, including ignore-index rows. Switch/tmp/nano-r8a-switches.json; detached timing PID 1207651, log/tmp/nano-r8-time-a.log, output/tmp/nano-r8-time-a, started 10:20:05 UTC. - Loss rewrite result 10:28:33:timemeasured 1.0635×, 0.9802 s/update (0.9776/0.9819/0.9809), effectively unchanged from 1.0633×. Not promoted. - Second variant/tmp/nano-r8b-treeuses a deterministic Triton row reduction for loss and log-normalizers in forward and the same compiler-visible FP32 backward. Switch/tmp/nano-r8b-switches.json, timing PID 1262155, log/tmp/nano-r8-time-b.log, output/tmp/nano-r8-time-b, started 10:29:43 UTC. - End 10:38:16: Triton loss held GPUs 10:29:44–10:38:06 and measured 1.0623×, 0.9813 s/update (0.9760/0.9826/0.9852). Neither loss variant improves on the tuned screen beyond measurement noise. No judge numerics claim; both switches remain off.
3.9 hRound 9 — profile GPU dispatch gaps
- Start 10:38:47 UTC. Detached judge profile PID 1318129,/tmp/nano-r9-profile.log, output/tmp/nano-r9-profile, updates 11–12 of the fully validatede9563a3. - Profile finished 10:46:53. GPU kernel/copy/memset interval union leaves 23.42 ms of gaps (1.20% of the profiled GPU span), including 3.71 ms in gaps of at least 20 μs (0.19%). Dispatch is a small remaining budget; test graph replay before assuming any gain. -/tmp/nano-r9a-treeenables compiled-model CUDA graphs, with one graph step marker per optimizer update to keep both accumulation microbatches live. The optimizer, loader, and math are unchanged. Switch/tmp/nano-r9a-switches.json; detached timing PID 1373297, log/tmp/nano-r9-time-a.log, output/tmp/nano-r9-time-a, started 10:47:27 UTC. - End 10:50:21: graph timing held GPUs 10:47:27–10:48:46;/tmp/nano-r9-time-a/time.jsonreturnedERRORbecause rank 0 received SIGSEGV before completing the timing steps. No valid speed or numerics result. Graphs remain disabled; no workaround pursued.
4.1 hRound 10 — profile FlashAttention execution
- Start 10:50:42 UTC. Detached judge profile PID 1383209,/tmp/nano-r10-profile.log, output/tmp/nano-r10-profile, updates 11–12 of the fully validatede9563a3. - Profile finished 10:58:42. FP8 GEMMs 45.64%, FA3 backward 8.74% and forward 2.68%, optimizer step 4.86% of CUDA attribution. Tested a different execution path in the same pinned attention backend before considering rewriting kernels. -/tmp/nano-r10a-treecalls FA3 varlen for equal-length self-attention, with fixed cumulative sequence boundaries that match the original batch rows. Causality, local windows, softmax scaling, and BF16 are unchanged. Switch/tmp/nano-r10a-switches.json; detached timing PID 1438639, log/tmp/nano-r10-time-a.log, output/tmp/nano-r10-time-a, started 10:59:34 UTC. - End 11:07:52: FA3 varlen timing held GPUs 10:59:35–11:07:50 and measured 1.0599×, 0.9835 s/update (0.9805/0.9856/0.9844). The equal-length path is slightly slower than the original dense FA3; no promotion or numerics claim.
4.4 hRound 11 — profile the MLP gradient cast for async transfers
- Start 11:08:17 UTC. Detached judge profile PID 1492924,/tmp/nano-r11-profile.log, output/tmp/nano-r11-profile, updates 11–12 of the fully validatede9563a3. - Profile finished 11:16:20. Largest MLP derivative cast/transpose kernel took 129.53 ms (6.62%) and its amax 37.72 ms (1.93%) across two profiled updates. Focused on the dual-layout cast, preserving the same tensorwise scale and BF16 derivative rounding. -/tmp/nano-r11a-treetries Hopper TMA loads and a row-major TMA store alongside a coalesced transposed store. Initial FP8 layout conversion corrupted transpose bytes; changing layout while the derivative is still BF16 and then converting to E5M2 fixed it. A GPU-locked test matched both upstream byte layouts at 384×512 and production 65536×6144. Switch/tmp/nano-r11a-switches.json; detached timing PID 1547986, log/tmp/nano-r11-time-a.log, output/tmp/nano-r11-time-a, started 11:21:09 UTC. - Prepared a second store layout in/tmp/nano-r11b-tree(/tmp/nano-r11b-switches.json) that uses TMA for both writes, with the corrected BF16 layout conversion. Not run or submitted yet. - Queued PID 1560782, log/tmp/nano-r11-time-b.log, output/tmp/nano-r11-time-b. It waits for the first timing to finish, and, if the first is valid but below 1.07×, runs a GPU-locked byte comparison before timing the full-TMA variant. - First timing held GPUs 11:21:10–11:29:26./tmp/nano-r11-time-a/time.json: 1.0701×, 0.9741 s/update (0.9722/0.9773/0.9728), a consistent gain over the tuned screen (1.0633×). The second layout was not run. Queuedcheck --quickfollowed bycheck --n1if exact items pass: PID 1599924, log/tmp/nano-r11-check-a.log, outputs/tmp/nano-r11-quick-aand/tmp/nano-r11-n1-a, started 11:30:05 UTC. - End 11:31:55:/tmp/nano-r11-quick-a/check.jsonwasNOT_BITWISEwith every exact item passing. Source digest matched the timed candidate. The seed-1 trajectory took the GPU lock at 11:31:56, outside the round budget. No promotion before it passes. - Trajectory finished 11:42:13:/tmp/nano-r11-n1-a/check.jsonPASS (N1), worst of the six endpoints 0.478σ vs. the 3σ bound. Its source digest matches both the timing and the exact check. Promoted as9cb4b0e. - Full check of submitted9cb4b0estarted 11:43:21 UTC, PID 1664618, log/tmp/nano-r11-full-a.log, output/tmp/nano-r11-full-a. - Full check finished at 12:03:45: PASS (N1), 1.0724× vs. frozen reference, 0.9720 s/update (0.9706/0.9741/0.9714). Exact items passed and the maximum trajectory deviation was 0.456σ. This supersedes the split-run validation.
5.3 hRound 12 — profile the TMA cast writes
- Start 12:04:21 UTC. Detached judge profile PID 1780382,/tmp/nano-r12-profile.log, output/tmp/nano-r12-profile, updates 11–12 of verified9cb4b0e. - Profile finished 12:12:20. The new derivative cast took 56.57 ms over two updates (2.90%, down from 129.53 ms / 6.62%); its amax took 38.56 ms (1.97%). The new trace also shows a separate log-named ReLU-square activation layout kernel, 46.46 ms (2.37%). GEMMs 46.57%, FA3 backward 9.02%, optimizer step 4.56%. -/tmp/nano-r12a-treesaves the MLP down-projection's E4M3 input in column-major layout while the forward path casts it. Unlike the broad Round 7 attempt, this targets only the MLP activation affected by the new derivative cast. No scale or dtype changes. Switch/tmp/nano-r12a-switches.json, PID 1833017, log/tmp/nano-r12-time-a.log, output/tmp/nano-r12-time-a, started 12:15:22 UTC. - Second alternative is the previously prepared both-stores TMA version/tmp/nano-r11b-tree, which differs from the submitted cast only in the optional transposed-store path. Queued PID 1833368, log/tmp/nano-r12-time-b.log, output/tmp/nano-r12-time-b; it byte-checks both layouts first and times only if they match and the activation prepack did not reach 1.078×. - Activation prepack finished 12:23:32: 1.0700×, 0.9742 s/update (0.9721/0.9751/0.9752). There is no gain over the async-cast build; not promoted. - End 12:31:58: both-store layout matched reference bytes at 384×512 and 65536×6144. Timing held GPUs 12:23:40–12:31:51 and measured 1.0718×, 0.9726 s/update (0.9709/0.9730/0.9738), within run-to-run variation of 9cb4b0e. Neither Round 12 variant was promoted.
5.8 hRound 13 — profile MLP forward cast fusion
- Start 12:32:19 UTC. Detached judge profile PID 1943443,/tmp/nano-r13-profile.log, output/tmp/nano-r13-profile, updates 11–12 of verified9cb4b0e. - Profile finished 12:40:11. GEMMs 46.36%, optimizer step 5.01%, async derivative cast 2.89%, and ReLU-square forward amax 1.26%. Existing activation pack variants did not combine the two layout writes into one read. -/tmp/nano-r13a-treeforms both E4M3 layouts of the squared activation in one Hopper tile and saves only the column-major one for the original weight-gradient GEMM. Computes the same BF16 amax by squaring the clamped maximum. The GEMM precision and existing E5M2 gradient path are unchanged. GPU-locked byte comparisons pass for both layouts and both dtypes at 384×512 and 65536×6144. Switch/tmp/nano-r13a-switches.json; detached timing PID 1997628, log/tmp/nano-r13-time-a.log, output/tmp/nano-r13-time-a, started 12:42:53 UTC. Submission unchanged. - Timing held GPUs 12:42:54–12:51:09./tmp/nano-r13-time-a/time.json: 1.0906×, 0.9558 s/update (0.9528/0.9563/0.9584), faster in all windows than the submitted build. Queued quick exact check then N1 trajectory if exact items pass: PID 2052891, log/tmp/nano-r13-check-a.log, outputs/tmp/nano-r13-quick-aand/tmp/nano-r13-n1-a, started 12:51:45 UTC. No promotion before validation. - End 12:53:35:/tmp/nano-r13-quick-a/check.jsonreturnedNOT_BITWISEwith all exact items passing and the same source digest as timing. N1 trajectory started at 12:53:36, outside the round. No promotion before it passes. - Trajectory finished 13:04:10:/tmp/nano-r13-n1-a/check.jsonPASS (N1); largest endpoint drift 0.394σ. Same source digest in timing, record, and trajectory. Promoted ascea61f8. Full check started 13:05:05 UTC, PID 2116379, log/tmp/nano-r13-full-a.log, output/tmp/nano-r13-full-a. - Full check finished 13:25:28 UTC:/tmp/nano-r13-full-a/check.jsonPASS (N1), 1.0906× vs. the frozen reference, 0.9558 s/update (0.9523/0.9575/0.9576). All exact items passed; maximum endpoint deviation 0.330σ.
6.7 hRound 14 — profile submitted activation fusion
- Start 13:25:40 UTC (GPU lock). Judge profile ofcea61f8, updates 11–12, detached PID 2131719, log/tmp/nano-r14-profile.log, output/tmp/nano-r14-profile; queued at 13:07:09 after the full check. - Profile finished 13:33:48. FP8 GEMMs 47.66%, FA3 backward 9.12% and forward 2.75%, dual-layout MLP cast 5.26%, optimizer 5.09% (BF16 batched products alone 2.73%), collectives 2.63%. This round targets the optimizer's symmetric matrix products. -/tmp/nano-r14a-treeevaluates one triangle of each BF16 Gram product, writes its transpose, and folds the quadratic expression into the second product's output store. Keeps all five iterations, round points, and the originalXupdate and state. Compiled GPU comparisons at 1536×1536, 1536×6144, and 6144×1536 were bitwise on the tested matrices. Switch/tmp/nano-r14a-switches.json; PID 2288267, log/tmp/nano-r14-time-a.log, output/tmp/nano-r14-time-a, timing lock 13:35:28. - Alternate/tmp/nano-r14b-treeusesbaddbmmepilogues for the same quadratic expression andXupdate, leaving matrix dimensions and optimizer state untouched. Switch/tmp/nano-r14b-switches.json. Queued PID 2297850 at 13:36:36, log/tmp/nano-r14-next.log: runs quick check/tmp/nano-r14-quick-aif the triangular kernel clears the screen, otherwise times the alternate into/tmp/nano-r14-time-b. - Triangular products finished 13:43:50:/tmp/nano-r14-time-a/time.json, 1.0952× and 0.9518 s/update (0.9506/0.9537/0.9511). All windows were faster, but 0.4% marginal gain in one screen did not resolve the gain; no quick check or promotion. The alternate took the GPUs at 13:43:57. - End 13:52:04 UTC: the batched-epilogue variant measured 1.0924×, 0.9542 s/update (/tmp/nano-r14-time-b/time.json), a 0.2% point estimate over the submitted build. Neither variant clears the screen for a resolved gain, so neither is promoted or assigned a numerics tier.
7.1 hRound 15 — profile remaining FP8 operand traffic
- Queued 13:52 UTC, detached judge profile PID 2396084, log/tmp/nano-r15-profile.log, output/tmp/nano-r15-profile, updates 11–12 of verifiedcea61f8. Round clock starts at GPU lock. - GPU lock 13:52:36 UTC; profile finished 14:00:37. FP8 GEMMs 47.57%, FA3 backward 9.11%, MLP dual casts 5.29%, optimizer 5.07%. Remaining FP8 cast/transpose family 4.30%, of which the 65536×1536 tiled transpose group alone is 2.19%. Targeted gradient materialization on FP8 attention projections. -/tmp/nano-r15a-treeuses one TMA pass to cast the same tensorwise-scaled BF16 gradients to both E5M2 operand layouts. Vocab and unsupported shapes stay on upstream casts. Both layouts match byte-for-byte at 384×512 and 65536×1536. Switch/tmp/nano-r15a-switches.json; test and timing PID 2450339, log/tmp/nano-r15-time-a.log, output/tmp/nano-r15-time-a, GPU timing lock 14:02:05. -/tmp/nano-r15b-treealso emits partial maxima directly from the rotary adjoint for Q and K, preserving the same BF16 rounding and one tensorwise scale per gradient. Switch/tmp/nano-r15b-switches.json. Queued PID 2466956 at 14:04:23, log/tmp/nano-r15-next.log. It runs quick check/tmp/nano-r15-quick-aif A clears the screen; otherwise checks the fused projection against the original and times B into/tmp/nano-r15-time-b. - Timing finished 14:10:36:/tmp/nano-r15-time-a/time.json, 1.0985× and 0.9489 s/update (0.9458/0.9500/0.9508). All three windows improved, a 0.7% marginal gain; peak allocation unchanged. Quick check took the GPUs 14:10:44. Queued full check outside the round after the exact gate passes, PID 2512139, log/tmp/nano-r15-full-a.log, output/tmp/nano-r15-full-a. The submitted tree stays unchanged pending validation; variant B was not run. - End 14:12:29 UTC:/tmp/nano-r15-quick-a/check.jsonreturnedNOT_BITWISEwith all exact items passing. The pending full check is outside the round. - Queued the next profile after the full check, PID 2524752, log/tmp/nano-r16-profile.log, output/tmp/nano-r16-profile. It profiles the candidate on a confirmed gain and full pass, otherwise the unchanged submission. - Full check/tmp/nano-r15-full-a/check.jsonfinished 14:33:24 UTC: PASS (N1), 1.0988×, 0.9486 s/update (0.9449/0.9497/0.9512); all exact items passed and maximum trajectory deviation was 0.472σ. Promoted the tested files and switch without changes as07e9940at 14:34:21.
7.9 hRound 16 — profile the new projection casts
- Start 14:33:27 UTC (GPU lock). Judge profile of the verified Round 15 tree, now submitted as07e9940, updates 11–12; PID 2524752,/tmp/nano-r16-profile.log, output/tmp/nano-r16-profile. - Profile finished 14:41:43. FP8 GEMMs 47.77%, FA3 backward 9.11%, optimizer 5.43%. The new dual-layout projection gradient casts take 2.29%; the standalone BF16 gradient amax group takes 14.87 ms (0.79%) over two steps and the rotary adjoint 28.00 ms (1.48%). This round targets the maxima produced for Q and K. - Screening the previously drafted/tmp/nano-r15b-treeagainst the now-submitted Round 15 build. The rotary adjoint emits tilewise FP32 maxima of the same BF16 gradients, so the original tensorwise E5M2 scale needs no separate full-matrix scan. Preserves NaN propagation. Switch/tmp/nano-r15b-switches.json; detached verification and timing PID 2690605, log/tmp/nano-r16-time-a.log, output/tmp/nano-r16-time-a. No changes promoted. - GPU verification matched both projections and all three gradients bitwise on the compiled 2×2048×1536 test. Judge timing held GPUs 14:43:02–14:51:00;/tmp/nano-r16-time-a/time.jsonmeasured 1.1014×, 0.9464 s/update, only 0.2% ahead of the validated build. No promotion. Detached quick check/tmp/nano-r16-quick-athen Round 17 profile/tmp/nano-r17-profileat 14:51 UTC, PID 2746140, log/tmp/nano-r16-quick-r17-profile.log. - End 14:53:34 UTC: quick check wasNOT_BITWISEwith all exact items passing. The 0.2% screen did not justify this extra custom path or a full check.
8.2 hRound 17 — profile post-GEMM activation work
- Start 14:53:35 UTC (GPU lock). Judge profile of07e9940, updates 11–12, PID 2746140,/tmp/nano-r16-quick-r17-profile.log, output/tmp/nano-r17-profile. - Profile finished 15:01:44. FP8 GEMMs 47.84%, FA3 backward 9.15%, optimizer 5.18%. The MLP derivative's amax and cast take 2.03% + 2.96%, respectively; its BF16 preactivation forward amax takes 0.98%. Targeted reductions that can be emitted by the preceding FP8 GEMM. -/tmp/nano-r17a-treeuses a Hopper FP8 tensorwise input-gradient product with FP32 accumulation, a BF16 ReLU-square adjoint, and per-tile maxima in the epilogue; the weight-gradient GEMM and E5M2 scaling recipe remain unchanged. GPU tests at 512 and 65536 rows gave 0.000636 and 0.000632 relative L2 against the same cuBLAS product and derivative, with exact returned maxima. Switch/tmp/nano-r17a-switches.json; detached test and timing PID 2806037, log/tmp/nano-r17-time-a.log, output/tmp/nano-r17-time-a, timing lock 15:03:19. - Independent forward variant/tmp/nano-r17b-treeemits each BF16 preactivation tile's maximum from the unchanged tensorwise E4M3 product with fast accumulation, before the original ReLU-square/cast. Switch/tmp/nano-r17b-switches.json. Queued at 15:04:39, PID 2817918, log/tmp/nano-r17-next.log: runs the A quick check (/tmp/nano-r17-quick-a) on a gain, otherwise tests and times B (/tmp/nano-r17-time-b). - Backward epilogue timing finished 15:11:50: 1.0153×, 1.0267 s/update (/tmp/nano-r17-time-a/time.json). The GEMM replacement costs far more than the saved scans; no quick check or promotion. The forward variant is next. - The forward variant matched cuBLAS bitwise in its 512-row and 65536-row unit tests. Timing held GPUs 15:12:04–15:20:30 and measured 1.1031×, 0.9450 s/update (/tmp/nano-r17-time-b/time.json), a 0.4% screen over the validated submission. No promotion on this evidence. Queued quick check/tmp/nano-r17-quick-bthen profile/tmp/nano-r18-profile, PID 2918369, log/tmp/nano-r17-quick-r18-profile.log. - End 15:22:57 UTC: forward epilogue quick checkNOT_BITWISEwith all exact items passing. Its one timing screen is too small to justify promotion.
8.7 hRound 18 — profile GEMM algorithm selection
- Start 15:22:58 UTC (GPU lock). Judge profile of07e9940, updates 11–12, PID 2918369,/tmp/nano-r17-quick-r18-profile.log, output/tmp/nano-r18-profile. - Profile finished 15:31:24. FP8 GEMMs still take 48.12%; the three largest cuBLAS kernel variants account for 17.70%, 15.51%, and 9.02%. FA3 backward 9.22%, optimizer 4.80%. Targeted alternate plans in cuBLASLt, rather than replacing the tensor-core mainloop. -/tmp/nano-r18-treeapplies the same tensorwise FP8 scales, FP32 compute and fast-accumulation flags, and BF16 output while screening alternate supported heuristic plans with no BF16 partial reductions. Keeps the built-in fallback when no allowed plan is found. Verified the first alternate was used and bitwise equal on the three full-size forward, input-gradient and weight-gradient probes. Switch/tmp/nano-r18a-switches.json; detached test and timing PID 2980252, log/tmp/nano-r18-time-a.log, output/tmp/nano-r18-time-a, GPU timing lock 15:33:43. - Queued PID 2983874 at 15:34 UTC, log/tmp/nano-r18-next.log: quick check of the first alternate on a resolved gain, or verify and time the next alternate with/tmp/nano-r18b-switches.json, output/tmp/nano-r18-time-b. - The first alternate finished timing at 15:42:46: 1.0998× and 0.9478 s/update (/tmp/nano-r18-time-a/time.json). The marginal 0.1% did not clear the screen; no promotion. - End 15:51:50 UTC: the second alternate also selected a real plan and was bitwise on all three unit tests. It held the GPUs 15:43:00–15:51:50 and was slower at 1.0823×, 0.9631 s/update (/tmp/nano-r18-time-b/time.json). Neither variant was promoted or taken to judge numerics.
9.1 hRound 19 — profile tensorwise scale reads
- Queued at 15:52 UTC: detached judge profile of07e9940, updates 11–12, PID 3096563, log/tmp/nano-r19-profile.log, output/tmp/nano-r19-profile. - GPU lock 15:52:20; profile finished 16:00:42. FP8 GEMMs 47.68%, FA3 backward 9.15%, optimizer 5.40%. The MLP forward maximum and the family of projection gradient maxima are still 0.98% and 0.79% of CUDA. Test whether the separate small gains from removing these reads add up when combined. -/tmp/nano-r19-treecombines the byte-identical source of the Round 16 rotary-adjoint maxima and Round 17B MLP forward maxima, with both tensorwise scale switches enabled. Both constituents passed their own exact items. Switch/tmp/nano-r19-switches.json; detached timing and quick check PID 3152602, log/tmp/nano-r19-time-a.log, outputs/tmp/nano-r19-time-aand/tmp/nano-r19-quick-a. No change to submission. - End 16:11:31 UTC: time held GPUs 16:01:46–16:09:43 and measured 1.1030×, 0.9451 s/update; quick checkNOT_BITWISEwith all exact items passing. The two changes did not improve over the forward epilogue's standalone screen, and the marginal point estimate over the submission is 0.4%; no promotion.
9.5 hRound 20 — profile the FP8 head and loss gradient
- Queued at 16:12 UTC: detached judge profile of07e9940, updates 11–12, PID 3212874, log/tmp/nano-r20-profile.log, output/tmp/nano-r20-profile. - GPU lock 16:12:15; profile finished 16:20:34. FP8 GEMMs 48.02%, FA3 backward 9.25%, optimizer 5.11%. The loss normalization, gradient amax and dual-layout cast account for 0.88%, 0.95%, and 0.98% respectively. Targeted the loss-gradient scan and cast. -/tmp/nano-r20a-treecomputes the same soft-capped cross-entropy and FP32 maximum candidate for each row during normalization, then uses a single dual-layout E5M2 cast from the BF16-rounded derivative. One exact tensorwise scale, same FP8 GEMMs and loss cap. Compiled unit tests with padding, ignored labels and saturated logits gave ≤1.91e-6 absolute loss difference and ≤0.00240 relative input/weight gradient error. Switch/tmp/nano-r20a-switches.json; detached trial PID 3268410, log/tmp/nano-r20-time-a.log, output/tmp/nano-r20-time-a, GPU timing lock 16:23:03. -/tmp/nano-r20b-treetests chunked vocabulary reductions for the same fused loss statistics. Queued at 16:24:27, PID 3280309, log/tmp/nano-r20-next.log: quick check/tmp/nano-r20-quick-aon a gain, otherwise verify and time B in/tmp/nano-r20-time-bwith the same switch file. - A timing finished 16:31:30:/tmp/nano-r20-time-a/time.json, 1.1100× and 0.9391 s/update, about 1.0% ahead of the submitted build. Quick check running; chunked variant not run. Detached seed-1 trajectory queued after the exact check, PID 3323817, log/tmp/nano-r20-n1-a.log, output/tmp/nano-r20-n1-a. Submission unchanged pending validation. - End 16:33:27 UTC: exact items all passed (NOT_BITWISEin/tmp/nano-r20-quick-a/check.json). Timing windows 0.9361/0.9417/0.9395 and 95.04 GiB peak allocated vs. 103.04 GiB submitted. The trajectory and full check are outside the round. Queued full check on a matching-digest N1 pass: PID 3331095, log/tmp/nano-r20-full-a.log, output/tmp/nano-r20-full-a.