GPT-6.2 Astra 1.17×
Harness: Codex · switches: NANOCHAT_TRAIN_MICROBATCH=32 NANOCHAT_PACK_MLP=1 NANOCHAT_MLP_REGISTER_TRANSPOSE=1 NANOCHAT_FUSE_QK=1 NANOCHAT_RMS_ATTN=1 NANOCHAT_COALESCE_ADAMW=1 NANOCHAT_FLAT_MUON=1 NANOCHAT_PACK_HEAD_LOSS=1 NANOCHAT_HEAD_ROW_CHUNK=4096 NANOCHAT_ADD_ATTN=1 NANOCHAT_ADD_MLP=1 NANOCHAT_ADD_QK=1 NANOCHAT_RESIDUAL_SEARCH=1 NANOCHAT_FP8_LIBRARY=12.9.3 NANOCHAT_VENDOR_SEARCH=1 NANOCHAT_RMS_BRANCHES=1 NANOCHAT_RMS_DIRECT=1 NANOCHAT_RMS_GRAD_ROWS=1 NANOCHAT_RMS_GRAD_WARPS=4 TORCHINDUCTOR_COORDINATE_DESCENT_TUNING=1
- Judged
- 1.17×
- Tier
- N1
- Self-best
- 1.17×
- Rounds
- 17
- Hours
- 10.0
- API cost
- $72.3
- Input tokens · 96.9% cached
- 84 M
- Output tokens
- 0.5 M
- Requests
- 616
§ ARaw trace
Raw work log17 sections
0.0 hRound 1 — unchanged upstream profile
- 19:31:13: judgeprofileof upstream92d63d4acquired the lock. PID 222302, log/tmp/nc-r01-prof.log, output/tmp/nc-r01-prof. - Frozen reference: 1.0286, 1.0349, 1.0342 s/update; spread 0.0039. Benefit floor in screening: 1.02×. - Profile complete 19:33:56. Percentages of summed rank-0 GPU kernel/copy time, two updates: compiled Triton 40.5%, FA3 10.7%, NCCL 2.2%, FP8 GEMMs about 43%. Largest individual fusions: MLP backward FP8 clamp/transpose with ReLU derivative 6.39%, its max reduction 1.93%, MLP forward cast 1.82%; rotary/norm 1.83%. The weight-gradient GEMM family accounts for 15.88%. - First test: enable Inductor coordinate-descent tuning for its generated kernels, without changing model code, in a separate clone (/tmp/nc-dev).timePID 238348, started 19:35:09; log/tmp/nc-r01-time.log, output/tmp/nc-r01-time. -check --quickfor this switch queued as PID 238690 (/tmp/nc-r01-quick.log, output/tmp/nc-r01-quick). Second test of the same FP8 layout bottleneck: use an explicitly tiled byte transpose in the FP8 backward. This gives Inductor a boundary between the quantized operands and the strided writes (the baseline generated MLP transpose recomputes both ReLU² forward quantization and ReLU² gradient quantization). Separate clone/tmp/nc-layout, switchNANOCHAT_FP8_TILED_TRANSPOSE=1. - Layout candidate timing followed by quick check queued at 19:39:02 as PID 268071, log/tmp/nc-r01-layout.log; outputs/tmp/nc-r01-layout-time,/tmp/nc-r01-layout-quick. - Recap (end 19:50:21): tunertimewindows [1.0115, 1.0159, 1.0196] s/update, 1.0166×. Quick: PASS bitwise N0. Disable wall-clock coordinate search when deterministic algorithms are requested, leaving the standard compiler configuration in that mode. The tuned production run is below the 1.02 benefit floor on its own. Layouttime: [1.2903, 1.2987, 1.2981], 0.7969×; quickNOT_BITWISEwith exact items OK. Additional materializations and separate memory passes did not pay for themselves. Layout not promoted.
0.3 hRound 2 — tuned tree
- 19:51:55: started profiling the fastest screened configuration, PID 309070,/tmp/nc-r02-prof.log, output/tmp/nc-r02-prof. - Profile completed 19:59:14: FP8 GEMM families about 44% (largest weight-gradient family 16.28%), Triton 39.0% (MLP fused FP8 packing 6.30%), FA3 11.0%, NCCL 2.4%. Microbatch shape and cuBLASLt algorithm selection for the same tensorwise FP8 GEMMs are the focus of this round. - B=32 with two accumulation steps, tuner unchanged: PID 356235,/tmp/nc-r02-batch.log,timestarting 20:01:33 in/tmp/nc-r02-time, followed by/tmp/nc-r02-quick. Only the training microbatch is regrouped; token count, schedule and validation batch size do not change. - Alternative at original B=16: a C++ cuBLASLt operator considers native matmul algorithms in two equivalent output orientations; it uses the original FP8 tensors, scalar scales, BF16 output, FP32 compute and original fast-accum flags. Split reductions restricted to compute precision. Deterministic policy uses a fixed non-split algorithm without benchmarking./tmp/nc-lt,NANOCHAT_TUNE_FP8_GEMM=1, no coordinate search.timeand quick queued as PID 391937,/tmp/nc-r02-lt.log, outputs/tmp/nc-r02-lt-timeand/tmp/nc-r02-lt-quick. - B=32timefinished 20:10:09: 1.0435×, [0.9862, 0.9922, 0.9905] s/update, peak allocated 94.04 GiB. Proceed to full check if exact items pass. - B=32 quick passed all exact items, candidate needs N1. Promoted this measured candidate and queued its full check with PID 412241, log/tmp/nc-full-b32.log, result/tmp/nc-full-b32/check.json. - Round 2 end 20:17:21: B=16 cuBLASLt selectiontime[1.0368, 1.0416, 1.0426], 0.9926× (20:15:09). Its quick check passed exact items but wasNOT_BITWISE. Extra operator boundaries and alternative GEMM choices did not beat the upstream library integration end-to-end. Not promoted. B=32 incremental gain over tuned B=16: 1.0265×. - Outside rounds: B=32 full check started 20:17:23. Next profile queued as PID 428981 (/tmp/nc-r03-prof.log,/tmp/nc-r03-prof); it will profile B=32 if accepted, otherwise the previous B=16 tuner. While the GPUs are occupied, preparing an isolated Gluon dual-layout MLP packing experiment; both FP8 kernels compiled offline for SM90, no speed or accuracy result claimed. - The full check advanced to seed-1 trajectory at 20:28:14. MLP op equality tests (no timing) queued underflockfollowing the next profile: PID 485868,/tmp/nc-r03-mlp-unit.log. Full check (outside rounds), 20:17:23–20:38:55: PASS, N1, 1.0440×,/tmp/nc-full-b32/check.json. All exact items and all six trajectory endpoints passed. This is a screening full check using the frozen reference timing, not the final R–C–R measurement.
1.1 hRound 3 — profile accepted B=32, investigate MLP hidden packing
- Profile PID 428981,/tmp/nc-r03-prof.log→/tmp/nc-r03-prof, followed by locked kernel unit tests. Separate experiment/tmp/nc-mlp; the accepted submission does not contain experimental kernels. - 20:39:00–20:47:09 profile: FP8 GEMM 44.5%, generated fusions 38.0%, FA3 11.2%, NCCL 2.5%. MLP-wide packing is the most costly non-GEMM kernel at 6.57%; the associated gradient max is 1.95%, forward packing 1.67%, forward max 1.09%. Target these together, not the GEMMs: compute the peak of ReLU² from the peak of the BF16 preactivation, fuse squaring or its derivative with casting to both layouts, and keep the transposed FP8 activation for the weight gradient. All up/down products retain the reference formats, scales and accumulation flags. - Unit test caught incorrect byte transposes on aligned FP8 tiles with layout conversion *after* the cast. First timing dispatch (PID 586051) did not start because tests failed. Converting registers while still in FP32 and casting in each layout fixed it. At 20:49:46/tmp/nc-r03-mlp-unit3.log: both pack layouts, both FP8 formats, masked tiles and production width byte-exact; output, both weight gradients and input gradient equal to an independent tensor-expression implementation. Now screening register transpose vs the accepted tree. - Registered-tile candidatetime+ quick PID 586429, start 20:50:20, log/tmp/nc-r03-reg-fixed.log, outputs/tmp/nc-r03-reg-timeand/tmp/nc-r03-reg-quick. - Register-packed MLPtimefinished 20:58:58, 1.0715× (1.0268× over B=32 timed alone), [0.9616, 0.9665, 0.9632] s/update; 103.04 GiB peak allocated. Pending record/statistical gates; accepted B=32 remains in place. - Round 3 end 21:01:06: quickNOT_BITWISE(exact items passed). Running the full check for the measured register layout, keeping the previous full PASS submitted until its result. - Full check (outside rounds) PID 644183, started 21:01:46,/tmp/nc-full-mlp.log→/tmp/nc-full-mlp/check.json, candidate commit8f40e79in/tmp/nc-mlp. - Next profile PID 644505 queued (/tmp/nc-r04-prof.log). While the full check runs, a separate prototype in/tmp/nc-qkcouples the existing tensorwise FP8 Q/K projections to fused rotary/RMS preparation and its adjoint. No change to FlashAttention, its scale or mask; tile maxima from the adjoint supply the original two independent gradient scales. Forward and backward successfully compiled offline for SM90; not yet timed or accepted. Full check, 21:01:47–21:22:57: PASS N1, 1.0722×,/tmp/nc-full-mlp/check.json, all six endpoint errors between −0.023σ and +0.442σ. Promoted the checked commit and the exact switch mapping.
1.9 hRound 4 — accepted MLP tree
- Profile PID 644505, GPU lock 21:23:01, output/tmp/nc-r04-prof, log/tmp/nc-r04-prof.log. Unit tests for the experimental Q/K preparation queued after the profile as PID 694866 (/tmp/nc-r04-qk-unit.log, under the GPU lock). - Profile ended 21:31:09: FP8 GEMMs 45.9%, generated Triton fusions 30.4%, FA3 11.6%, accepted MLP pack 5.14%, NCCL 2.8%. Q/K preparation: rotary/RMS forward 1.80%; backward head reduction 1.01%, rotary adjoint/slices 1.44%, gradient maxima ≥0.84%, FP8 cast/transpose 1.50%. Screen integrated Q/K preparation, projection and FP8 gradient packing with separate tensorwise gradient scales; no changes inside FA3. Both targeted kernel tests passed (full head shapes and tails, gradients, tile maxima and byte-exact gradient layouts). Timing/quick PID 799375, start 21:31:29, log/tmp/nc-r04-qk.log, outputs/tmp/nc-r04-time,/tmp/nc-r04-quick. -timeended 21:40:28 at 1.0980×, windows [0.9371, 0.9409, 0.9434], 104.16 GiB peak. Q/K adds 1.0247× over the MLP-only screen. Check the same kernel with coordinate search off as a fast-screen calibration now that the hot fusions have been replaced. This changes no deterministic setting (coordinate search is already disabled for deterministic algorithms). - Quick finished 21:42:17NOT_BITWISE, all exact items passed. No-searchtimePID 855975,/tmp/nc-r04-nosrch.log, output/tmp/nc-r04-nosrch, 21:48:06–21:50:45: 1.0817×, [0.9523, 0.9551, 0.9566]. The tuner still helps; no-search is just a lower-cost screen. Round ended 21:50:45. Full check outside rounds for tuned97db88bPID 856314, started 21:50:47,/tmp/nc-full-qk.log→/tmp/nc-full-qk/check.json. - Next profile queued PID 878910 (/tmp/nc-r05-prof.log, output/tmp/nc-r05-prof). Prepared a model-only FP8 GEMM template-search option in a separate copy/tmp/nc-templatesand a non-timing correctness/capability probe for native cuBLASLt epilogues; probe PID 895953 will run under the lock after that profile, log/tmp/nc-epilogue-support.log. Full check, 21:50:47–22:12:50: PASS N1, 1.0991×,/tmp/nc-full-qk/check.json. Six endpoint biases in σ: [−0.005, −0.013, +0.460, +0.106, +0.079, −0.013]. Promoted checked commit97db88band its checked switch mapping.
2.7 hRound 5 — profile accepted Q/K tree
- Profile PID 878910, GPU lock 22:12:51,/tmp/nc-r05-prof.log→/tmp/nc-r05-prof, followed by the non-timing vendor capability probe at production MLP dimensions. - Profile finished ~22:21:30. FP8 GEMM 47.3% (weight gradient family 17.58%); generated fusions 24.5%, FA3 12.0%, the new pack and Q/K kernels 6.40% and 2.63%, NCCL 2.9%.AMAX_D_POINTERon a BF16 FP8-GEMM output and device alpha vectors gave no vendor algorithm; plain ReLU epilogue worked. No unsupported path entered training. Test model-only Inductor GEMM search including persistent TMA, with no pointwise coordinate search (compare 1.0817× round-4 no-search control). Fixed first-choice baseline in deterministic mode instead of wall-clock search. - TemplatetimePID 1038062, 22:21:48,/tmp/nc-r05-templates.log→/tmp/nc-r05-templates-time. Also prepared a separate orientation alternative: compute each weight gradient as the transpose of the reversed product with the two FP8 operands and their inverse scales swapped (same BF16 output rounding and non-fast accumulator). Its natural column-major gradient layout and cuBLAS kernel shape differ from round 2's fixed row-major output experiment. - Wgrad orientationtime+ quick queued as PID 1066570,/tmp/nc-r05-wflip.log, outputs/tmp/nc-r05-wflip-timeand/tmp/nc-r05-wflip-quick. - Template search finished 22:29:16: [0.9519, 0.9522, 0.9547], 1.0836× vs 1.0817× without that search, not resolved. The existing scaled GEMM won the major shapes over the Triton candidates. Quick PID 1069149,/tmp/nc-r05-templates-quick.log→/tmp/nc-r05-templates-quick. Wgrad orientationtimeended 22:31:40: [0.9616, 0.9654, 0.9678], 1.0702×, a regression. Neither change promoted. - Round 5 ended 22:35:17: both quick checks passed exact items,NOT_BITWISE. No full check warranted. Submitted tree and switches unchanged.
3.1 hRound 6 — RMSNorm/FP8 preparation
- Reprofile the unchanged Q/K champion, PID 1083898 (/tmp/nc-r06-prof.log, output/tmp/nc-r06-prof). The immediately preceding profile of the same commit/config attributes 9.76% in aggregate to RMS/rsqrt-related families (including fused neighboring operations). Test coupling the two block-input RMSNorms to their FP8 consumers, independently of each other, if confirmed. - Round clock started 22:35:18. Prepared/tmp/nc-rmswith per-row RMS stats/peaks, dual-layout BF16→FP8 normalized inputs shared by the following projections, and an analytic RMS backward. The global tensorwise scale is unchanged. The attention and MLP sites have separate switches. Unit test, no-coordinate-search combinedtime, andcheck --quickqueued as PID 1122581, log/tmp/nc-r06-both.log, outputs/tmp/nc-r06-both-timeand/tmp/nc-r06-both-quick. - Profile completed 22:43:55: 47.5% FP8 GEMM, 24.5% generated fusions, 12.0% FA3, 8.59% in kernel names containing RMS/rsqrt (overlapping categories). Both focused unit tests passed, including a production-width RMS, low-magnitude and zero inputs, byte-exact layouts and projection input/weight gradients. Combined-site timing started 22:44:01. - No-search combined sites finished 22:46:02: 1.0941×, [0.9413, 0.9439, 0.9462], peak 110.92 GiB; +1.15% vs matched 1.0817× Q/K control. Site-isolation timings queued as PID 1152469 (/tmp/nc-r06-sites.log, outputs/tmp/nc-r06-attn-time,/tmp/nc-r06-mlp-time). - Both-site quickNOT_BITWISE, exact items passed at 22:47:35. Attention only [0.9422, 0.9448, 0.9436], 1.0944×; MLP only [0.9501, 0.9537, 0.9552], 1.0835×. Only attention benefits clearly: it shares the packed activation across three projections. Restored coordinate tuning for attention only: PID 1184518, log/tmp/nc-r06-tuned.log, outputs/tmp/nc-r06-tuned-time,/tmp/nc-r06-tuned-quick; start 22:52:54. - Tuned attention RMS: 1.1113×, [0.9277, 0.9303, 0.9294], peak 108.66 GiB. Gain over tuned Q/K: 1.0121×. Round ended 22:59:54: quickNOT_BITWISE, all exact items passed. Committed7b494d5onexperiment/rms-fp8. Full check outside round: PID 1225310 at 23:00:21, log/tmp/nc-full-rms.log, result/tmp/nc-full-rms/check.json. Next best-tree profile queued as PID 1225624, log/tmp/nc-r07-prof.log, output/tmp/nc-r07-prof. - During the full check, prepared a separate descriptor-based FP8 packing prototype/tmp/nc-tma. No training change in the submission. Non-timing tests queued after the next profile as PID 1247279,/tmp/nc-r07-unit.log(all four packing formulas, aligned/masked shapes, and both input-transfer policies). Full check, 23:00:21–23:14:37: PASS N1, 1.1131×,/tmp/nc-full-rms/check.json. Maximum absolute trajectory bias 0.339σ. Promoted7b494d5and the exactrms-attn.jsonconfiguration.
3.7 hRound 7 — dual-layout FP8 writes
- 23:14:39: profile of the just-accepted RMS candidate acquired the lock (PID 1225624,/tmp/nc-r07-prof.log, output/tmp/nc-r07-prof). The prior Q/K profile had 6.38% dual-layout packing. Confirm its share now that the attention projection inputs are also prepacked, then screen Hopper tensor-map transfers against the existing register-layout kernel. - Profile finished 23:19:38: 48.0% FP8 GEMM, 22.0% compiled fusions, 12.1% FA3; the single dual-layout pack family is 7.00% (240 calls). Initial unit run: TMA loads/stores matched every expected FP8 byte; with ordinary loads, the transposed ReLU derivative failed on all three tile shapes. Screening only the passing all-TMA policy (PID 1341099, 23:20:20,/tmp/nc-r07-tma.log, outputs/tmp/nc-r07-tma-time,/tmp/nc-r07-tma-quick). - In a separate copy/tmp/nc-tma-hybrid, the mixed-load policy uses the verified TMA input path for the ReLU derivative, with a second option to pipeline four tiles per CTA. Unit tests precede both timings. Queue PID 1342630, log/tmp/nc-r07-variants.log, outputs/tmp/nc-r07-pipeline-time,/tmp/nc-r07-stores-time. - Plain all-TMA first timing: 1.0841×, below the matched 1.0944×. Also screen alternative lane/register ownership for the original bulk transfers without descriptors (64×64 compact and 128×128 tall tiles), each only if its byte and gradient tests pass: PID 1353720,/tmp/nc-r07-regtiles.log, outputs/tmp/nc-r07-compact-time,/tmp/nc-r07-tall-time. - Plain all-TMA quick:NOT_BITWISE, exact items passed (23:24:16). Both revised policy suites passed (8 pipelined tests and 2 single-tile transfer tests). Four-tile pipeline: 1.0607×, [0.9703, 0.9756, 0.9745]; mixed loads: 1.0860×, [0.9476, 0.9523, 0.9525], finished 23:29:16. All three tensor-map variants regress against the existing register pack; neither additional shared-memory staging nor pipelining paid off. Archive only,experiment/tma-pack. - Compact register tile: 1.0867×, [0.9478, 0.9522, 0.9506]; tall: 1.0942×, [0.9417, 0.9458, 0.9435]. Six kernel and gradient tests passed per configuration, but neither beats the 1.0944× control. Archive295cfb3onexperiment/pack-tiles. Final quick checks for tall & corrected mixed TMA, then next best-tree profile, PID 1409665 (/tmp/nc-r07-record-r08-prof.log, outputs/tmp/nc-r07-tall-quick,/tmp/nc-r07-hybrid-quick,/tmp/nc-r08-prof). No full check warranted. - Round ended 23:40:03. Both final quick records:NOT_BITWISE, exact items passed. Champion unchanged. - Diagnostic comparison against the already N1 RMS parent (not the upstream verdict): the all-TMA, corrected hybrid and tall-tile records preserve every deterministic parameter digest on both ranks at every recorded update.
4.2 hRound 8 — optimizer collective layout
- 23:40:04: judge profile of the unchanged RMS champion acquired the lock (PID 1409665,/tmp/nc-r07-record-r08-prof.log, output/tmp/nc-r08-prof). The previous profile of this exact tree shows ~50 ms for its GPU optimizer annotation; NCCL 2.8% of total GPU work (AllGather RING_LL 1.47%, plus reductions), with several full-group stack/copies. Screen native NCCL Simple first, and measure a group layout that avoids copying parameters out of each gathered Muon stack. Protocol test queued after profile: PID 1420245 (/tmp/nc-r08-simple.log,/tmp/nc-r08-simple-time). - Profile finished ~23:45: 48.2% FP8 GEMM, 21.9% generated fusions, 12.2% FA3, 3.1% other GEMM, 2.8% NCCL; the GPU optimizer annotation is 49.65 ms. In/tmp/nc-flat, stage each shape group of Muon parameters in contiguous same-dtype canonical storage, scatter from a directly stacked gradient tensor, update the rank's view and all-gather into the parameters in place. Separate switch for launching Muon groups before AdamW. Distributed unit compares every parameter and state to the standard optimizer over three identical updates with partial/empty rank shards. Unit and three judgetimes queued as PID 1457338,/tmp/nc-r08-flat.log; outputs/tmp/nc-r08-flat-muon-time,/tmp/nc-r08-muon-first-time,/tmp/nc-r08-flat-first-time. - NCCL Simple finished 23:47:20: 1.0936×, [0.9405, 0.9458, 0.9461], no improvement vs 1.0944×. Archived26bc37e. Distributed flat/reorder unit passed bitwise on both ranks. Flat storage alone screened at 1.0975× (23:49:42), a small ~0.28% increment. - Muon-first alone: 1.0944×, no gain. Additional data-layout screen: coalesce the multi-table AdamW value-embedding reduce-scatters/all-gathers through the native process group while preserving each table's row shard and optimizer state. Two-rank unit test, isolated and flat-Muon combination timings queued PID 1498291,/tmp/nc-r08-coalesce.log, outputs/tmp/nc-r08-coalesce-adam-time,/tmp/nc-r08-flat-coalesce-time. - Flat plus Muon-first: 1.0972× [0.9379, 0.9420, 0.9434], comparable to flat-only [0.9379, 0.9421, 0.9428]. Archived separate commits4517944(order) andcf4cf5d(flat storage). Combined distributed coalescing/flat test passed bitwise on both ranks, including state tensors. - Coalescing in isolation 1.0936× [0.9417, 0.9451, 0.9459]; together with flat Muon 1.0996× [0.9379, 0.9397, 0.9399], ~0.48% faster than matched no-search. Combo quick PID 1518527,/tmp/nc-r08-combo-quick.log→/tmp/nc-r08-combo-quick. Committed73935ebonexperiment/coalesced-optim. Restored coordinate tuning for combo timing: PID 1530880,/tmp/nc-r08-tuned.log→/tmp/nc-r08-tuned-time. - Tuned combo at 00:06:00 (Nov 18): 1.1170×, [0.9240, 0.9252, 0.9241], 0.51% above the tuned RMS screen with all windows improved. Combo quickNOT_BITWISEvs reference; all ten deterministic parameter digests per parameter on both ranks match the full-PASS RMS parent. Final protocol-only quick/tmp/nc-r08-simple-quick, followed by full check of73935eboutside rounds (/tmp/nc-full-optim), PID 1570602,/tmp/nc-r08-record-full.log. Next best-tree profile queued as PID 1570603,/tmp/nc-r09-prof.log, output/tmp/nc-r09-prof. - Round ended 00:08:04, simple protocol quickNOT_BITWISEwith exact items passing. Full check started 00:08:05. While it runs, prototype/tmp/nc-splitkimplements explicit token-axis partitions of FP8 weight gradients: separate FP32 partials, fixed sum then BF16 result, original two tensorwise scales. Two candidate kernels: pipelined Triton/SM90, or strided-batch cuBLASLt with fast accumulation disabled. CPU-only native build succeeded (PID 1614218,/tmp/nc-split-build.log); GPU correctness and production-shape capability tests queued after the next profile, PID 1617093,/tmp/nc-r09-split-unit.log. No weights, gradients, or optimizer policy from this prototype are in the submitted tree. Full check, Nov 18 00:08:05–00:22:18: PASS N1, 1.1168×,/tmp/nc-full-optim/check.json. All six endpoint biases 0.084–0.407σ. Promoted73935eband the exact checked configuration.
4.9 hRound 9 — split weight-gradient GEMMs
- Nov 18 00:22:23: acquired the lock to profile the just-accepted optimizer tree (PID 1570603,/tmp/nc-r09-prof.log, output/tmp/nc-r09-prof). The preceding RMS profile charged 17.96% to the main weight-gradient kernel; test explicit parallel FP32 partial sums along the large reduction dimension if confirmed. No changes to dgrad, forward GEMMs or quantization. - Profile: FP8 GEMM 48.5%, generated fusions 22.1%, FA3 12.2%, NCCL 2.5%; the main weight-gradient kernel alone is 18.12% (292 calls). Triton split partials passed all unit checks at K blocks 128, 64 and 32: FP32 partial accuracy, repeatability, and BF16 final gradient. The native strided-batch library accepts all three production shapes, but its partials exceeded the 1e-4 unit bound (~1.05e-4), so no vendor timing until investigated. Follow-up tests PID 1685438 (/tmp/nc-r09-accum-tests.log); independent comparison with stock non-fast FP32 GEMM and FP64 arithmetic PID 1686018 (/tmp/nc-r09-vendor-diag.log). Triton screens+quick PID 1685679,/tmp/nc-r09-triton.log, outputs/tmp/nc-r09-split4-time,/tmp/nc-r09-square8-time,/tmp/nc-r09-split4-quick. - Against FP64, the native batched FP32 partial error is identical to stock non-fast FP32_scaled_mmon each diagnostic partition (exactly zero batch-vs-native difference); fast accumulation is distinctly worse. BF16 final gradient relative L2 2.0e-5 / 2.8e-5. Vendor unit now requires bitwise equality to stock non-fast FP32 GEMM for every partial. Unit → timing → quick queued PID 1695503,/tmp/nc-r09-lt.log, outputs/tmp/nc-r09-lt-time,/tmp/nc-r09-lt-quick. General Triton split-fourtimefinished 00:30:40 at 0.8628×, a major regression vs matched 1.0996×. - Distinct MLP layout variant/tmp/nc-rowwgrad: directly consume token-major FP8 inputs/gradients in the weight-gradient kernel; avoid writing their whole-tensor transposes. Projection output and input gradients exactly match the existing MLP and both weight gradients meet relative-L2 unit checks. Timing and quick PID 1694919,/tmp/nc-r09-row.log, outputs/tmp/nc-r09-row-time,/tmp/nc-r09-row-quick. - MLP direct-rowtime: 0.5631×; eliminating transposes does not offset the GEMM penalty of non-K-major operands. Native four-way batched FP32 products: all partials bit-exact to stock non-fast library GEMM in unit, buttimeonly 1.0909×. Screened a second native formulation with cuBLASLt's internal split-K and *compute-type* reduction, not an in-place BF16 reduction (PID 1702642,/tmp/nc-r09-native.log,/tmp/nc-r09-native-time,/tmp/nc-r09-native-quick; correctness/capability/tmp/nc-r09-native-unit.log, all three production shapes supported). Also test just the square block matrices with two explicit vendor batches, PID 1712660,/tmp/nc-r09-ltsq.log, output/tmp/nc-r09-ltsq-time. - Internal native split-K: 1.0823× [0.9502, 0.9549, 0.9571]; explicit native batches square-only: 1.0957× [0.9387, 0.9429, 0.9455]. Both below the 1.0996× matched control. The native-all, internal-native and direct-row quick checks all pass exact items (NOT_BITWISE). Archived23ac421,08f5513,91714ac,e5c5cf1separately. No full check justified. Next best-tree profile queued as PID 1746088,/tmp/nc-r10-prof.log,/tmp/nc-r10-prof. - Square-only custom eight-way splittime: 0.9774×; the general Triton quick record passed exact items (NOT_BITWISE). Round ended 00:49:29. No split variant promoted.
5.3 hRound 10 — soft-capped output loss
- 00:49:31: profile of the unchanged full-PASS optimizer tree acquired the lock (PID 1746088,/tmp/nc-r10-prof.log,/tmp/nc-r10-prof). The preceding profile measured 2.96% summed GPU time in log-softmax/tanh-gradient families alone, across two microbatches of 65,536×32,768 logits. Assess combining the row loss/normalizer with the tensorwise head-gradient peak, then packing both gradient layouts in one pass. - Profile finished ~00:54:30: FP8 GEMM 48.6%, Inductor fusions 22.1%, FA3 12.3%, and the loss-related families 2.995%./tmp/nc-headlosskeeps the head's FP8 GEMMs, BF16 logits, cap=15, ignored-label mean reduction and scalar gradient scale. A row kernel computes sum-exp and the largest off-target derivative numerator alongside the loss, accounting separately for the target. One backward kernel computes the soft-cap adjoint, rounds to BF16 and packs both FP8 layouts. Unit → no-coordinatetime→ quick queued PID 1797713,/tmp/nc-r10-head.log; outputs/tmp/nc-r10-head-time,/tmp/nc-r10-head-quick. - Initial projected-head test passed, but approximate PTX tanh failed the per-row loss tolerance on high-magnitude logits (maximum difference 1.14e-4). Replaced it with standard FP32 CUDA libdevice tanh in both forward and backward; both strict units passed, including exact rounded peaks, byte-transposed gradients, ignored labels, three upstream factors and projected input/weight gradients. New timing/quick PID 1797920 (/tmp/nc-r10-head-libdevice.log), timing acquired the lock 00:57:45. Added an alternate deterministic split-row reduction (4096 vocabulary elements per CTA) and an 8-warp whole-row screen in a clone; chunk unit and both times queued PID 1808213,/tmp/nc-r10-variants.log, outputs/tmp/nc-r10-chunk-time,/tmp/nc-r10-w8-time. - Whole rows, 16 warps: 1.1127× [0.9266, 0.9291, 0.9283], +1.19% over no-search optimizer control; peak 97.71 GiB. Archived asf298ae1. Optional FP32 exp-based evaluation of the same tanh identity, with an extra saturation test, queued after row-geometry screens; PID 1813039,/tmp/nc-r10-exp.log, output/tmp/nc-r10-exp-timeonly if tests pass. - Default quickNOT_BITWISE, all exact items passed. Chunked reduction passed the strict tests and screened at 1.1136× [0.9253, 0.9284, 0.9281]; 8-warp whole rows 1.1066×. Exp-form tanh passed saturation/gradient tests but 1.1103×, not promoted (2f4fbe0). The slight chunk-vs-whole difference alone is not resolved; selected the chunked standard-libdevice path for a tuned measurement/quick (PID 1852903,/tmp/nc-r10-tuned.log, outputs/tmp/nc-r10-tuned-time,/tmp/nc-r10-tuned-quick). Exp-only quick queued afterward as PID 1853255,/tmp/nc-r10-exp-quick.log,/tmp/nc-r10-exp-quick. - Tuned chunked headtime: 1.1282×, [0.9124, 0.9178, 0.9157], 97.71 GiB; +1.00% vs the 1.1170× matched tuned optimizer parent. Tuned quick at 01:16:06:NOT_BITWISE, exact items passed. Archived4c8a58f. Full check queued as PID 1886989 (/tmp/nc-full-head.log, output/tmp/nc-full-head), following the last optional quick. Next-best profile PID 1886990,/tmp/nc-r11-prof.log,/tmp/nc-r11-prof. - Round 10 ended 01:17:45. Exp-form quick passed all exact items (NOT_BITWISE); no reason to select it over the standard libdevice version. The tuned standard-libdevice head's full check acquired the GPU lock at 01:17:48. - While that full check runs, separate clone/tmp/nc-resgemmhas an offline-built cuBLASLt epilogue prototype (CPU build PID 1902975,/tmp/nc-resgemm-build.log, successful). It uses the original FP8 operands/scales, the original fast/non-fast mode and BF16 residual as matrix C with beta=1. Attention output, MLP down projection and Q/K input-gradient sum can be toggled separately. One BF16 rounding boundary moves; not a bitwise claim. Production-shape correctness and branch-gradient units are queued after the next profile, followed by a no-coordinatetimeand quick if tests pass (PID 1935622,/tmp/nc-r11-first.log;/tmp/nc-r11-all-time,/tmp/nc-r11-all-quick). No change to the checked submission. Full check, 01:17:48–01:31:52: PASS N1, 1.1273×,/tmp/nc-full-head/check.json. All exact items passed; six trajectory biases [+0.207, +0.260, +0.383, +0.196, +0.163, +0.166]σ. Promoted4c8a58fand the exacthead-loss-tuned.jsonswitch mapping.
6.0 hRound 11 — FP8 GEMM residual epilogues
- Queued PID 1886990 selected the newly checked head configuration for this round's profile. The preceding trace of its parent showed 48.6% FP8 GEMM, with generated add/RMS-adjoint and residual-mix kernels adjacent to projections. Investigate replacing the separate BF16 residual traffic with matrix-C loads in the GEMM epilogue. - Round clock 01:31:55. Profile ended 01:36:49,/tmp/nc-r11-prof/profile.json: FP8 GEMM 49.3%, generated fusions 19.3%, FA3 12.4%, dual-layout pack 7.10%, NCCL 2.4%. Kernels containingaddaccount for 9.91% inclusive, not the removable share: e.g. 0.99% in add-before-attention-RMS-adjoint, 0.91% in add/MLP RMS/peak. Test all three residual sites, then each site alone (queued PID 1979214,/tmp/nc-r11-sites.log,/tmp/nc-r11-{attn,mlp,qk}-time). An optional official cuBLAS wheel stages in/opt/candidate-env/optional/cublas-12.9.3, outside Python/library search paths, for future FP8 GEMM library comparison (PID 2009618,/tmp/nc-cublas129-install.log). It cannot change the current run or checked tree. - Five locked GPU units passed, including native production shapes, FP32-reference accuracy, deterministic repeated output and branch gradients. All sitestime01:37:00–01:39:21: 1.1075×, [0.9297, 0.9358, 0.9318], peak 97.84 GiB, below 1.1136× no-search control. Quick ended 01:41:06NOT_BITWISE, exact items passed. Archivede40e40c. Alongside site isolation, investigate heuristic choice with a one-time shape-specific search among native, non-split cuBLASLt algorithms; deterministic mode uses fixed first choice. CPU extension build passed with explicit SM90 (PID 2037827); search units/time/quick queued PID 2037830,/tmp/nc-r11-search.log,/tmp/nc-r11-search-{time,quick}. Replaced slow optional library download with range requests for the same official PyPI wheel, checking its published SHA-256 before installation (PID 2039423,/tmp/nc-cublas129-stage.log). - Isolated site timings ended 01:48:18: attention projection 1.1109×, MLP down projection 1.1139× (neutral vs 1.1136× control), Q/K backward 1.1025×. The prior fused add/RMS kernels were already doing useful adjacent work. The alternate algorithm search passed all five units and began timing at 01:48:26. - Warmup algorithm selection improved all-sitestimeto 1.1175×, windows [0.9207, 0.9262, 0.9250] vs no-search parent 1.1136×. Small but consistent enough across windows to test with coordinate tuning restored. Tunedtime/quick queued PID 2074287 (/tmp/nc-r11-tuned.log,/tmp/nc-r11-tuned-time,/tmp/nc-r11-tuned-quick). - No-coordinate search quick passed all exact items (NOT_BITWISE). Tuned all-sitestimeended 01:57:55: 1.1355×, windows [0.9071, 0.9111, 0.9099], peak 97.84 GiB; +0.65% over tuned headtime. Archived as939c507. Full check after the tuned quick, PID 2113732,/tmp/nc-full-resalg.log→/tmp/nc-full-resalg. Next profile of whichever full-check winner, PID 2113733,/tmp/nc-r12-prof.log→/tmp/nc-r12-prof. - Round 11 ended 01:59:41 with tuned quickNOT_BITWISE, all exact items passed. Full check acquired the lock at 01:59:44; the live tree remains the prior full PASS. - Outside the round, while the full check runs:/tmp/nc-vendormmimplements an isolated optional tensorwise FP8 backend for the 49.3%-of-GPU-time GEMM family in the latest profile. It opens only the selected official cuBLASLt library, creates a matching handle, uses scalar tensor scales, original FP8 inputs, FP32 compute, BF16 outputs and the original per-direction fast-accum bit. Fixed-choice and non-split autotuned variants; no library preload or changes to Torch's normal dependencies. Built successfully without GPUs, PID 2136767,/tmp/nc-vendor-build.log. Matched image-library control and optional 12.9.3 tests queued after the next profile as PID 2151315 (/tmp/nc-r12-vendor.log;/tmp/nc-r12-image-time,/tmp/nc-r12-129-time,/tmp/nc-r12-129-search-time,/tmp/nc-r12-129-search-check). Full check, 01:59:44–02:14:03: PASS N1, 1.1346×,/tmp/nc-full-resalg/check.json. All exact items passed. Six endpoint biases [+2.163, −1.122, −0.624, −0.953, −1.124, −1.158]σ; all within the 3σ criterion. Promoted939c507and its checkedresidual-search-tuned.json.
6.7 hRound 12 — cuBLASLt FP8 kernel library
- The accepted residual tree is the input to PID 2113733's profile (/tmp/nc-r12-prof,/tmp/nc-r12-prof.log). Compare the image's 12.8.4 library through the same native wrapper with the separately staged NVIDIA 12.9.3 library. This is an app-level kernel selection; the installed Torch and its dependencies remain untouched. Only the non-residual products are replaced; each residual epilogue keeps its just-validated native path. - Round clock 02:14:04; profile ended 02:18:58. FP8 GEMMs 50.3%, generated fusions 18.5%, FA3 12.6%, NCCL 2.3%; the largest long-K product kernel accounts for 17.02% alone (244 calls). The official <ip> wheel installed after matching the PyPI SHA-256 (download scratch removed); old/new Lt API versions are 120804/120903. Image-library production-shape and adjoint units passed; controltimebegan 02:19:10. Matching 12.8.4 search and selected 12.9.3 long-K / forwardtimes queued PID 2232971,/tmp/nc-r12-sites.log,/tmp/nc-r12-{image-search,129-long-k,129-forward}-time. - Image fixed-choice control ended 02:21:22: 1.1132×, [0.9230, 0.9291, 0.9307], 97.96 GiB. Initial 12.9.3 numerics passed, but the unit process exited 139 after shutdown (reproduced under the lock, PID 2240892,/tmp/nc-r12-teardown.log), so the original queues stopped before any 12.9.3 training. The CUDA DSO now stays mapped for its process lifetime (RTLD_NODELETE); the same two units passed and exited 0. Retry PID 2240989,/tmp/nc-r12-retry.log; fixed-librarytimestarted 02:23:55. Requeue matched image-search and long-K screens; optional forward screen deferred to keep within the round clock. - New 12.9.3 fixed choicetime: 1.1191× [0.9212, 0.9226, 0.9242]. New 12.9.3 per-shape search units passed and exited 0;timeended 02:28:22 at 1.1315×, [0.9107, 0.9129, 0.9143], +1.25% vs no-coordinate 1.1175× parent. Quick started 02:28:23. Matched controls PID 2242463,/tmp/nc-r12-match.log; tunedtimeand quick for the winning library/config queued as PID 2255918 (/tmp/nc-r12-tuned.log,/tmp/nc-r12-tuned-{time,quick}), keeping the previous full-check winner live. - Search quick:NOT_BITWISE, all exact items passed. Matched image-library search 1.1240× [0.9150, 0.9210, 0.9202]; 12.9.3 long-K-only search 1.1168× [0.9210, 0.9275, 0.9254]. Benefit requires more than just the long-K shapes. Archived the library and per-shape selection asd4ae21c. Selected all-product 12.9.3 search for tunedtimeat 02:34:43. Image-library exact-item control quick queued after the winning tuned quick, PID 2278282,/tmp/nc-r12-image-quick.log,/tmp/nc-r12-image-quick. - Tuned all-product 12.9.3time: 1.1561×, windows [0.8914, 0.8941, 0.8942], peak 97.96 GiB, ended 02:39:41; +1.81% vs 1.1355× matched tuned residual parent. Archived4e09cff. Full check queued after the quick controls: PID 2304252,/tmp/nc-full-vendor.log→/tmp/nc-full-vendor; next profile PID 2304253,/tmp/nc-r13-prof.log→/tmp/nc-r13-prof. - Tuned exact-item quick ended 02:41:30NOT_BITWISE; image-search control quick ended 02:43:20NOT_BITWISE, both with all exact items passed. Round 12 ended 02:43:20. Full check acquired the GPU lock at 02:43:22. - While that full check runs, stage activation-memory schedules for the two remaining B=32 forwards/backwards (97.96 GiB peak) in/tmp/nc-remat: a B=64 no-rematerialization control, AOT memory budgeting with custom tensorwise GEMM/FA compute costs, and selective checkpointing that keeps large products and recomputes pointwise intermediates. CPU eager/compiled checkpoint output and adjoints matched bitwise (unit PID 2330660,/tmp/nc-remat-unit.log). After the winner's profile, PID 2335274 runs these B=64time/feasible quicks, log/tmp/nc-r13-batches.log,/tmp/nc-r13-b{64,64-65,64-sac}-{time,quick}. Complementary prototype/tmp/nc-remat-packavoids retaining a transposed FP8 MLP hidden tensor per block; its backward packs the hidden transpose from the same BF16 activation and scale in the same tile as the two FP8 adjoint layouts. Locked byte/gradient unit then B=32/64 screens queued after the batch pilots, PID 2344892,/tmp/nc-r13-hidden.log,/tmp/nc-r13-hidden{32,64}-{time,quick}. Neither experiment affects the submitted tree. Full check, 02:43:22–02:56:49: PASS N1, 1.1557×,/tmp/nc-full-vendor/check.json. All exact items passed; six endpoint biases [+2.165, −1.113, −0.675, −0.960, −1.112, −1.205]σ. Promoted4e09cffand its exactvendor-tuned.json. Its optional official library is installed in/opt/candidate-env/optional/cublas-12.9.3, accessible offline by the candidate user.
7.4 hRound 13 — microbatch size and activation lifetime
- Profile PID 2304253 selected the now-checked vendor configuration,/tmp/nc-r13-prof. Investigate whether the 50%-GEMM training loop can use a single 64-row local microbatch and eliminate one complete set of launches and a gradient accumulation. B=32 is already near 98 GiB, so explicitly separate the unmodified memory schedule, compiler rematerialization, and exact rebuilding of the MLP hidden transpose in backward. Only the group size changes, not token order or tokens per optimizer update. - Round clock 02:56:54; profile ended 03:01:25. FP8 GEMMs 49.7%, generated fusions 18.3%, FA3 12.9%, dual-layout pack 7.26%, NCCL 2.4%; peak 97.96 GiB. The largest weight-gradient kernel alone is 17.56%. B=64 native scheduletimestarted 03:01:33. - Plain B=64timefailed OOM before an update, 137.87 GiB allocated of 139.81 GiB card capacity (/tmp/nc-r13-b64-time). Budget 0.65 failed in Torch's AOT partitioner on missing size metadata for an_assert_asyncnode (/tmp/nc-r13-b64-65-time); no validity assertion was removed. B=64 with selective block checkpoints fit: 1.0881×, [0.9453, 0.9494, 0.9524], 131.58 GiB peak, exact items passed (NOT_BITWISE, 03:08:05). Recomputed activations cost more than the larger batch saves. Archived the generic schedules onexperiment/remat-b64. - Manual MLP transpose rematerialization passed all three locked byte/gradient units. B=32time1.1190×, [0.9183, 0.9237, 0.9263], saved exactly 9 GiB (88.96 GiB peak), but below 1.1315× no-coordinate control. Archived06268ec. B=64 tests continue. Two selective alternatives queued as PID 2422675,/tmp/nc-r13-schedules.log: keep cheap reduction outputs and quantized weights to reduce repeated work (/tmp/nc-r13-sac-stats-time), or allow reduction recomputation in the AOT budget's cost model without changing runtime assertions (/tmp/nc-r13-budget-red-time). Attention-only checkpoint + manual hidden layout (/tmp/nc-r13-attn-hidden-{time,quick}) queued as PID 2427044,/tmp/nc-r13-attncombo.log, to avoid recomputing the rest of the MLP. - Manual B=32 quick passed exact items (NOT_BITWISE). With the corrected cost-model switch, budget 0.65 compiled but went OOM during execution. Saving reductions/quantized weights improved B=64 checkpointtimeto 1.1046×, [0.9313, 0.9363, 0.9366], peak 132.23 GiB; still slower than B=32. Archived243d9b5. Attention-only checkpoint plus hidden rebuilding went OOM allocating the second 4-GiB head-gradient layout,/tmp/nc-r13-attn-hidden-time. B=64 manual-only was rescheduled because theNOT_BITWISEquick returned nonzero to the original queue; manual-only and budget 0.50 both went OOM (PID 2444639,/tmp/nc-r13-last.log,/tmp/nc-r13-hidden64-time,/tmp/nc-r13-budget-half-time). No invalid run used for a speed comparison. - Final memory pilot uses a single scratch buffer, packing the head gradient in the row layout for its input gradient, then repacking the same BF16 logits/scales into its transposed layout for the weight gradient. An initial launch captured the previous tree when a patch did not apply; stopped that own torchrun,/tmp/nc-r13-head-serial-time, not a result. Actual implementation passed both focused head tests. PID 2454032,/tmp/nc-r13-serial2.log,/tmp/nc-r13-serial2-time, timing from 03:24:00. The checked B=32 submission has not changed. - Serial head scratch plus attention-only checkpoint and MLP rebuilding fit at B=64, 1.1032×, [0.9331, 0.9385, 0.9364], 128.30 GiB peak. Round 13 ended 03:26:27. Strongest tested B=64 schedule 1.1046× vs B=32 no-coordinate 1.1315×: no promotion or full check. Next accepted-tree profile PID 2456001,/tmp/nc-r14-prof.log→/tmp/nc-r14-prof.
7.9 hRound 14 — remaining FP8 GEMM dispatch
- Profile the unchanged 1.1557× full-PASS tree. Its preceding profile showed 49.7% FP8 products, including ~8% in identifiable residual epilogue kernel families. In the checked stack only the beta=0 GEMMs use 12.9.3 and its algorithm search; explore the remaining matrix-C sites and additional supported native kernel choices if confirmed. - Round clock 03:26:29. Fresh profile ended 03:31:00: 49.5% FP8 GEMMs, 18.5% generated Triton, 13.0% FA3, 2.4% NCCL. Matrix-C forward and non-fast backward families are 6.41% + 1.83%./tmp/nc-vendor-epioptionally sends these products to the selected cuBLASLt DSO with an independently keyed beta=1 plan. It keeps BF16 residuals, the FP32 compute type, original FP8 scales, and original fast flags. Production-shape/gradient unit then no-coordinatetime/quick PID 2488786 (/tmp/nc-r14-epi.log,/tmp/nc-r14-epi-time,/tmp/nc-r14-epi-quick). - Five residual/adjoint units passed. 12.9.3 beta=1time1.1298×, [0.9096, 0.9149, 0.9175], compared with 1.1315× keeping those sites on 12.8.4; quick exact items passed (NOT_BITWISE, 03:36:17). Forward-only matrix-C 1.1306× (/tmp/nc-r14-epi-forward-time, queue PID 2496486,/tmp/nc-r14-forward.log); no gain. Archivedc9e60f2. - For the larger beta=0 family, screen 48 native heuristics, or both TN operand orientations writing the same row-major result, independently and together. Fixed deterministic mode still selects the first non-split algorithm. CPU build PID 2500216 passed; seven GPU unit tests passed including non-square native output, adjoints and beta=1 layout. No-coordinate screens+quick PID 2501815 (/tmp/nc-r14-layout.log,/tmp/nc-r14-{48-layout,48,layout}-time,/tmp/nc-r14-layout-quick). Both knobs together 1.1324×, [0.9095, 0.9144, 0.9118], close to the 1.1315× parent. Also stage a distinct geometry refinement on the winning algorithm (library-advertised tiles/stages, supported CTA swizzle and checked thread-block cluster shapes, no split reduction). CPU build PID 2512210 successful, locked units/time queued PID 2512211,/tmp/nc-r14-refine.log,/tmp/nc-r14-refine-time. - 48 choices alone 1.1299×, [0.9111, 0.9166, 0.9141]; alternate orientation alone 1.1300×, [0.9087, 0.9148, 0.9180]. Combined quickNOT_BITWISE, exact items passed. Archived3dce631. Geometry refinement passed native accuracy and gradient units,time1.1320×, [0.9095, 0.9122, 0.9150] at 03:49:59. Archivedfb943ee. None of these screening differences resolves a gain over 1.1315×. No tuned/full check warranted; geometry quick PID 2531185,/tmp/nc-r14-follow.log,/tmp/nc-r14-refine-quick. Next current-best profile queued PID 2539317,/tmp/nc-r15-prof.log,/tmp/nc-r15-prof. - Geometry quick ended 03:51:52,NOT_BITWISE, exact items passed. Round 14 ended 03:51:52; no dispatch changes promoted.
8.4 hRound 15 — attention RMS backward
- Current-best profile PID 2539317 acquired the lock at 03:51:53. Prior trace showed about 1% in separate additions preparing the normalized attention input gradient, followed by the RMS adjoint. Inspect the new trace before fusing the Q/K branch, V branch and small value-gate gradient loads into that adjoint's per-row reduction. - The profile finished 03:56:24 (/tmp/nc-r15-prof): 49.24% FP8 GEMM, 12.94% FA3, 1.14% RMS kernels, 0.77% in the two downstream gradient-add kernels. Prototype/tmp/nc-rms-branchessupplies independent autograd outputs and merges Q/K, value and optional gate upstreams in the RMS adjoint; separately switched identity-gradient addition. Shape, packing and eager/compiled projection-gradient units → branch-only and direct-path times → direct quick, PID 2571170,/tmp/nc-r15-first.log, outputs/tmp/nc-r15-{branches,direct}-time,/tmp/nc-r15-direct-quick. - All four GPU units passed, including compiled branched projection gradients. First timing command stopped before launch: missingjqgenerated an empty switch file. Corrected JSON, now screening four variants in/tmp/nc-r15-<variant>-screen, followed by fastestcheck --quickin/tmp/nc-r15-<variant>-record; detached log/tmp/nc-r15-time.log. - Screening job PID 2571877. The first no-coordinate branch-only screen finished 04:03:36 at 1.1351×. In a separate clone,/tmp/nc-value-adjoint, compare an algebraically equivalent place for the Q/K/V sum: calculate all three pullbacks together and add the value input gradient in its tensor-core GEMM epilogue. Unit/time/quick queued PID 2579762,/tmp/nc-r15-epi.log; outputs/tmp/nc-r15-value-{time,quick}. - Fusing the direct gradient as welltime1.1386×. Best branch configuration goes to a coordinate-tunedtimeafter its quick; PID 2588377,/tmp/nc-r15-tuned.log,/tmp/nc-r15-tuned-time. - Four no-coordinate screens (seconds/update windows): branches 1.1351× [0.9050, 0.9114, 0.9128]; plus direct 1.1386× [0.9028, 0.9074, 0.9106]; direct 8 rows/8 warps 1.1380× [0.9036, 0.9080, 0.9105]; direct 1 row/4 warps 1.1458× [0.8984, 0.9035, 0.9018], vs matched 1.1315×. Archived93d1bf9. Best quick finished 04:12:05:NOT_BITWISE, all exact items passed. All three value-adjoint GPU units passed. If the tuned RMS screen clears 1.159×, run the full judge outside the round, PID 2610830 (/tmp/nc-r15-full.log,/tmp/nc-full-rms-branches). - Tunedtimefinished 04:17:18, 1.1674×, [0.8815, 0.8866, 0.8854], unchanged 97.96 GiB peak. Archived configuration02eca15; full check queued after the value-adjoint quick. Next current-best profile PID 2643347,/tmp/nc-r16-prof.logand/tmp/nc-r16-prof, selects the full-checked tree after its verdict. - Value-adjoint epiloguetime1.1325× [0.9098, 0.9133, 0.9123], not resolved over control and slower than fusing into RMS. Archived9c0e66f. Quick finished 04:21:17, exact items passed (NOT_BITWISE). Round ended 04:21:17. Full check PID 2610830 is outside the round. The submission stays on its preceding full PASS pending the verdict. - Full check acquired the lock 04:21:21. While it runs, staged an optional BF16 cuDNN graph in/tmp/nc-attention-graphsfor the previously measured 12.94% attention family: FP32 accumulation/statistics, the same causal diagonal and inclusive left boundary, and a deterministic backward plan. Stored NVIDIA's unmodified frontend 1.14.1 binding in/opt/candidate-env/optional/cudnn-fe-1.14.1(CPU install PID 2650054,/tmp/nc-r16-fetch.log), outside normal imports. No training-environment changes. No GPU results for this new path yet. Full check 04:21:21–04:35:04: PASS N1, 1.1673×,/tmp/nc-full-rms-branches/check.json. All exact items passed, endpoint biases [+2.283, −1.105, −0.664, −0.968, −1.072, −1.181]σ. Promoted02eca15and its checked switch file.
9.1 hRound 16 — attention kernel dispatch
- Profile acquired the lock at 04:35:09 (PID 2643347,/tmp/nc-r16-prof.log→/tmp/nc-r16-prof), using the newly checked tree. Verify the attention family share and its forward/backward split before timing any alternate dispatch. - Profile finished ~04:40: 50.2% FP8 GEMMs, 17.1% generated Triton, 13.1% FA3, 2.4% NCCL. Local/full FA3 backward main kernels 5.16%/2.99%, forward 2.02%/1.05%, plus the pre/postprocess kernels. Target attention only: screen FA3 splitting/SM count and an optional cuDNN graph for local/full windows or one pass direction. Focused units cover inclusive left and causal right boundaries, production shapes, forward and three gradients against pinned FA3, deterministic reproducibility, and compiled dispatch. Detached queue/tmp/nc-r16-first.log, units/tmp/nc-r16-units.log, judge outputs/tmp/nc-r16-<variant>-timeand representative*-quick. - Queue PID 2764539. All three cuDNN attention units passed (including compiled forward/backward and both mixed backend directions) before timing. The new graph explicitly enables the native deterministic algorithm under torch determinism and its repeated VJP test was bitwise exact. - Eight-SM reservationtime1.1420× and two-way full-window splitting 1.1321×, both below 1.1458× unchanged. cuDNN both directions/all layers 1.1168× [0.9203, 0.9257, 0.9278]; isolating sites/directions. Independent short-window Triton FlexAttention with a static inclusive causal block mask: unit/time/quick queued PID 2801460,/tmp/nc-r16-flex.log→/tmp/nc-r16-flex-{time,quick}. - FlexAttention's first diagnostic returned a NaN relative error. No timing was launched: retry with padded rows handled by the safe masked-softmax path, and report finite outputs/each pullback separately. PID 2808113,/tmp/nc-r16-flex-retry.log. - The explicit finite checks localized FlexAttention's issue: the 192-token case passed after restoring the safe path, but 2048 withROWS_GUARANTEED_SAFE=Truestill returned nonfinite output and gradients. Disabled that hint entirely: having a self-key in each row doesn't guarantee an eligible key in the first partial block. Retesting before any timing, log/tmp/nc-r16-flex-safe.log. - Final safe Flex unit PID 2811305 passed at both sequence lengths (forward, Q/K/V gradients and exact boundary). Flextime1.0811×, [0.9513, 0.9566, 0.9575]; quick at 04:58:51NOT_BITWISE, exact items passed. Archivedba6c2dc. Native FA3 dispatch archived6cc5bb3. cuDNN short-only 1.1254×, long-only 1.1361×; cuDNN forward + FA3 backward 1.1376× [0.9032, 0.9090, 0.9110]; reverse mix 1.1201× [0.9177, 0.9230, 0.9251]. Archived9e9666b. Nothing exceeded the 1.1458× no-coordinate control; no full check warranted. Next unchanged-best profile queued PID 2810734,/tmp/nc-r17-prof.log,/tmp/nc-r17-prof. - Native quick finished 05:02:55; graph quick finished 05:04:44. BothNOT_BITWISE, every exact item passed. Round 16 ended 05:04:44; no promotion.
9.6 hRound 17 — shared-input projection products
- Current-best profile acquired the lock 05:04:45 (PID 2810734,/tmp/nc-r17-prof.log,/tmp/nc-r17-prof). The previous trace's largest long-K kernel was 17.55%. Attribute the square weight gradients by shape; if still significant, compare combining Q/K gradients that have the same right-hand operand while preserving each tensor's actual FP8 bytes and separate original scalar scale. - Profile ended 05:09: ~50.1% FP8 GEMM, 17.2% generated Triton, 13.2% FA3, 2.4% NCCL. The top individual product family is 17.75% across many shapes, not wholly removable; 68 back-to-back Q/K weight products alone account for 1.74% before the RMS adjoint (others scheduled later)./tmp/nc-paired-qkpacks two separately scaled gradient transposes directly into adjoining storage, then forms one taller GEMM against their common input. Compare an FP32 accumulator followed by the original two scalar products/BF16 casts, or a BF16 output epilogue broadcasting each original scalar over its own projection. This is not per-row quantization: regression tests check the exact original FP8 bytes and scales at zero and disparate magnitudes. Unit +timefor image vs 12.9.3 variants and best quick queued PID 2886759,/tmp/nc-r17-first.log, outputs/tmp/nc-r17-<backend>-time,/tmp/nc-r17-paired-quick. - Three native FP32-output units passed, and first paired-weighttime1.1478× (small vs 1.1458×). Archived327b13e. A separate clone/tmp/nc-paired-forwardtests the same common-operand GEMM idea on forward Q/K outputs, keeping the two independently quantized weights and reading the adjacent output views directly in RoPE/RMS without separate contiguity copies. Unequal scales, contiguous/pitched forward and backward, real projection gradients and native capabilities tested before timing. PID 2897965,/tmp/nc-r17-fwd.log,/tmp/nc-r17-fwd-<backend>-time,/tmp/nc-r17-fwd-quick. - Repeated-scalar epilogue weights: image library 1.1304×, native 12.9.3 1.1370×. Both native forward products are genuinely supported in the Hopper unit, not fallback: 12.9.3 broadcast epilogue 1.1421×; image broadcast epilogue 1.1445×. Both below the matched parent. If no remaining speedup above 1.15×, queue next best-tree profile PID 2936939,/tmp/nc-r18-prof.log,/tmp/nc-r18-prof. - Best paired-weight screen is close to run noise; a fresh matched no-coordinate parenttime(PID 2944598,/tmp/nc-r17-parent-control.log,/tmp/nc-r17-parent-control) follows the product screens. Requeued next profile so that it waits for the parent as well as both quicks. Grouped raw FP32 forward 1.1280×, slower than the vector epilogue; archived forward code10e71d6.