Claude Sonnet 5.5 1.96×
Harness: Claude Code · switches: OPT_DECODE_GRAPH=1 OPT_DENOISE_FAST=1 OPT_REF_RESHARD=1 OPT_FUSED=1 OPT_OVERLAP_REF=1 OPT_EARLY_FWD=1 OPT_DECODE_SAMPLE_GRAPH=1 OPT_CLONE_CTX=1 OPT_FUSED_TRAIN=1 OPT_FUSED_GEN=1 OPT_PACK_FAST=1 OPT_GPU_YIELD=3000 OPT_DECODE_SKIP_WARM=1 OPT_FSDP_PREFETCH=1 PYTORCH_ALLOC_CONF=expandable_segments:True OPT_NO_EMPTY_CACHE=1
- Judged
- 1.96×
- Tier
- N0
- Self-best
- 1.92×
- Rounds
- 9
- Hours
- 10.0
- API cost
- $37.9
- Input tokens · 99.2% cached
- 209 M
- Output tokens
- 1.1 M
- Requests
- 461
§ 02Key levers
- Text-decode CUDA graph
- ✓
OPT_DECODE_GRAPH, first half of sampling also in the graph (self-estimated ~−9 s per window) - Reward wait hidden behind forward
- ✓
OPT_OVERLAP_REF+OPT_EARLY_FWD;OPT_GPU_YIELD=3000yields the GPU every layer; together with skipping decode warmup, full check 1.74× → 1.85× - Rollout-side bitwise fused kernels
- ✓ Triton: RMSNorm, q/k norm + RoPE, whole denoise layer fused (
OPT_FUSED,OPT_FUSED_GEN) - Train-side bitwise kernels / freeing memory
- ✓
OPT_FUSED_TRAIN(1.7035× → 1.7087×, within noise);OPT_REF_RESHARDpeak down to ~104 GiB; FSDP prefetch + expandable segments + no cache clearing 1.85× → 1.92× - Other main levers
- packing desync, text CFG context copied directly; pipelined decode, fused MLP tried, no gain
§ ARaw trace
Raw work log9 sections
0.4 hRound 0 — profile of the unchanged reference (2026-11-06 05:46 - 06:06)
Command:bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-ref --from 11 --iters 2(steps 11 = refine/edit step, 12 = generate step; rank 0; profiler overhead inflates step times ~10%). Step times seen in the run log (reference, steps 5..11): generate steps 14.6-17.2 s, refine steps 30-35 s. Per window of 2 steps (one generate + one refine) ≈ 47 s -> the score is dominated by the refine step (~2/3). Findings from the 54 s Chrome trace (rank 0, 1.44M kernel launches in two steps; GPU kernel time only 26.5 s of 54 s = 49% busy): | phase (refine step / generate step) | wall (s) | GPU busy | note | |---|---|---|---| | text decode (generate_text_batch) | ~11 / ~8 | ~18% | 130 decode steps per 6 s => ~46 ms/token-step under profiler; ~1770 kernel launches, ~100.item()syncs per token-step | | image denoise (generate_image_mix) | ~13.5 / ~8 | ~65% | ~80k cudaStreamSynchronize per 10 s (3.5 s blocked), 70k.item(), boolean-mask indexing (nonzero) in_merge_static_kv_prefix_and_queryper layer per pass | | reward wait + glue | ~1.5 / ~1.5 | 0% | idle GPU | | training (ref fwd + policy fwd/bwd + opt) | ~8 / ~3 | ~100% | cudaMalloc/cudaFree thrash in backward (1.8 s per window total), memory is ~131 GiB at peak | Hotspot classes: decode = launch/host-sync bound (kind 2: missing graph/sync-free path); denoise = host sync bound; training = GPU bound (GEMM nvjet kernels 15%+7.7%, flex-attn triton, NCCL RS/AG). Top GPU kernels overall: nvjet GEMMs (4.0 s + 2.0 s + ...), index_put (1.38 s, 41k launches), elementwise (~3 s over >300k launches). Plan: (1) sync-free + CUDA-graph decode (same kernels => bitwise), (2) sync-free denoise merge, (3) allocator (expandable segments / free ref-model unsharded params), (4) misc host overlap.
1.0 hRound 1 — sync-free CUDA-graph decode + sync-free denoise (2026-11-06 06:14 - 06:42)
Phase breakdown of the unchanged reference without profiler (scratch tree with CUDA-event phase timer, judgetime: 23.48 s/step vs frozen 23.36, i.e. the timer is free): generate step ≈ 14.3-16 s (decode 4.7-5.4, denoise 5.2-5.7, reward wait 1.1-2.2, training ≈ 2), refine step ≈ 34.5-35 s (decode 9.4-10.9 for 340-385 token-steps = 28 ms/step, denoise 10.1-10.3, reward wait 0.65-2.5 + collective idle 1.4-3.6, ref fwd 1.1-1.5, policy fwd 1.4-3.1, bwd 4.2-4.7). Changes (all N0, each behind a switch, defaults off): -OPT_DECODE_GRAPH=1:Bagel.generate_text_batch->fast_paths.generate_text_batch_fastfor StaticKVCache: host-side bookkeeping (one D2H copy of (next token, prob) per step instead of ~100.item()), per-active-count CUDA graph of the 28-layer decode forward (+lm_head) (captured after one eager step per M,thread_localcapture, no gc/empty_cache), sampling/RNG calls unchanged (eager, same shapes and order),advance_seq_lensunchanged._static_kv_cache_attentiongot a sync-free branch for decode (cache.fast_decode). -OPT_DENOISE_FAST=1:torch.tensor([t]*4096)(4096 device syncs per step!) ->t.expand(N).contiguous(), host-side cfg-interval test, nounique()/sum()syncs in_forward_flow, and_static_kv_cache_readonly_attentionbuilds the merged K/V withtorch.catof [cached prefix | new tokens] per sequence from a host-known plan (cu_seqlens, max_k) instead of ~15 syncs + boolean-mask indexing per layer per pass. Verification: single-GPU harness (real weights, deterministic mode, rollout() gen + refine, rollout_for_eval()): 808/808 outputs bitwise identical (tokens, probs, latents, log-probs, images); harness wall: gen 14.0->7.2 s, refine 29.7->14.6 s, eval 137->75 s. Full check #1 (/tmp/check-1): ERROR - CUDA OOM in the backward of step 1 (129.9 GiB allocated, 248 MiB free; the reference sits at ~131 GiB peak, the ref model's unsharded params stay resident). => memory headroom is mandatory: addedOPT_REF_RESHARD=1(reshard the reference model after its forward) andOPT_OVERLAP_REF=1(prepack + pack + ref forward right after the rollout, overlapping the reward wait; advantages are filled in later via slots, not enabled yet). ### Full check #2 (06:42 - 07:03) — PASS, N0, speedup 1.378/tmp/check-2/check.json: switches{"OPT_DECODE_GRAPH":"1","OPT_DENOISE_FAST":"1","OPT_REF_RESHARD":"1"}(OPT_OVERLAP_REF off), commit bebb2ae. Gate 1: windows 15.97 / 16.61 / 16.83 s per step vs frozen 23.11 / 23.78 / 23.20 -> ratios 1.447 / 1.432 / 1.378, speedup 1.378 (bar 1.0206). Gate 2: bitwise PASS (14 steps, 3200 tensors, 235 reward requests identical). Peak allocated 104.3 GiB (reference ~131) thanks to resharding the reference model. Steps in the timing run: generate steps 9.7-10.5 s, refine 21.5-22.9 s.
2.1 hRound 2/3 — fused exact kernels, overlap of reward wait, in-graph sampling (07:00 - 07:50)
-OPT_FUSED=1: Triton kernels that reproduce the reference's op chains bit for bit, including ATen'smean(-1)summation order (block width W fromsetReduceConfig: thread-strided vec4 accumulators, halving folds, adjacent-pair warp tree) andrsqrt.approx:rms_norm_exact(C=3584, bf16/fp32 in) andqk_norm_rope_exact(head_dim 128, bf16 path for decode/prefill, fp32 path + row scatter for the denoise 'gen' mode).enable_fp_fusion=False,do_not_specializeon row counts, all kernel variants compiled inwarmup()(a first-use JIT compile had cost 3.3 s inside a timed window). Tested bitwise against the module chain: 228 + 264 cases (n = 1..60000, bf16/fp32, 3 scales, idx scatter). Kernel time vs reference chain: RMSNorm n=4096 213 -> 30 us; q-norm+rope n=4096,H=28 586/756 -> 73 us. -OPT_OVERLAP_REF=1: prepack + pack_batch (advantages deferred via slots) + reference forward right after the rollout;OPT_EARLY_FWD=1: also the policy forward (it does not need the rewards); zero_grad right after the optimizer step. -OPT_DECODE_SAMPLE_GRAPH=1: the deterministic front half of_sample_next_token(topk/logsumexp/cumsum/mask) and the raw softmax run inside the decode graph; onlymultinomial(RNG) + gathers stay eager. -OPT_CLONE_CTX=1: the text-CFG context ofedit_imagecopies the image-prefix K/V of the conditional context instead of recomputing VAE+ViT+28 layers. - Phase timer (fused + overlap,/tmp/time-pt3): generate step 8.3-9.0 s, refine 18.2-20 s (was 14.3-16 / 34.5-35). - Full check #3 (all switches except CLONE_CTX): speedup 1.659 (windows 1.717 / 1.659 / 1.707) but INCORRECT: from the first training rollout on, every reward request differed (the 11 step-0 eval requests matched). Cause: my ownwarmup()calledtorch.randnon the default CUDA generator, which the SDE sampler draws from. The harness had masked it by re-seeding before each rollout. Fixed (RNG-free warm-up), harness now seeds once and compares the three RNG states after every rollout. Harness (det mode, all switches): 817/817 entries bitwise equal incl. RNG states; gen 14.07 -> 5.42 s, refine 29.8 -> 10.5 s, eval 136 -> 60 s. ### Full check #4 (07:53 - 08:12) — PASS, N0, speedup 1.7035 (current best;/tmp/check-4/check.json) Tree commit a1169c9 (later edits are gated off by default), switches (= /submission/switches.json):OPT_DECODE_GRAPH, OPT_DENOISE_FAST, OPT_REF_RESHARD, OPT_FUSED, OPT_OVERLAP_REF, OPT_EARLY_FWD, OPT_DECODE_SAMPLE_GRAPH, OPT_CLONE_CTXall "1". Gate 1: windows 13.11 / 13.77 / 13.62 s per step vs 23.11 / 23.78 / 23.20 -> ratios 1.763 / 1.7275 / 1.7035 (bar 1.0206). Gate 2: bitwise PASS (14 steps, 3200 tensors, 235 reward requests identical). Peak allocated 103.9 GiB. Steps in the timing run: generate 8.4-9.1 s, refine 17.8-18.4 s (reference 14.3-16 / 34.5-35).
3.0 hRound 4 — what did not help, reward-service probe, exact fused TRAINING kernels (08:15 - 09:25)
- B1 (OPT_NO_EMPTY_CACHE+OPT_FSDP_PREFETCH, time run on the phase-timer tree, /tmp/time-b1): worse than check #4 (windows 13.81 / 14.81 / 14.40 vs 13.11 / 13.77 / 13.62; peak reserved 133.7 GiB). ctx_enter shrank 0.25 -> 0.14 s but the cached blocks raised fragmentation/peak. Both switches stay OFF. Also the unsharded parameter addresses change every step (pointer signature differs each rollout), so decode graphs cannot be reused across rollouts. - Reward service probe (judge's service, idle GPUs): single request 0.39 s, 4 concurrent 0.97 s each, 16 concurrent makespan 3.75 s (4 waves) => ~4.3 requests/s whatever the concurrency; each step's 16-image burst needs ~3.7 s to be scored. It is hidden by ref+policy forward in refine steps (3.3 s of work) but ~2 s stay exposed in generate steps (only ~0.9 s of reward-independent GPU work exists there). Batching images per request is impossible (the judge compares the request set). - Denoise (non-det harness profile, all switches): 97% GPU busy, GEMM 72%, flash 6.6%, scatter/gather/silu/add ~20%; decode graph steady state 6.1 ms/step (GEMM weight streaming floor ~4.0), capture ~35 ms per M + eager first steps ~0.3 s per rollout. - GEMMs run at 680-800 TFLOPS (q/o 728, gate/up 805, down 678): nothing to gain there. -OPT_FUSED_TRAIN=1: exact autograd Functions for the training forward:RMSNormExact(fwd stores r; bwd reproduces ToCopy/Mul/Rsqrt(-0.5*g*r^3)/Mean(g * fl(1/C))/Pow(2*h)/Add/ToCopy of the reference chain, row reduction in ATen's order, weight grad = ATen sum of the bf16 product) andQKNormRopeExact(rotate_half adjoint, bf16 adds, norm bwd for C=128). Op-level tests: 60 + 42 cases bitwise equal (outputs, grad_x, grad_w). Whole-model test on one GPU (real weights, AC, det mode): loss/logits/mse preds equal and 796/800 gradients equal for generate and refine packings; fwd+bwd 5.27 -> 4.35 s (refine pack). ### Full check #5 (08:44 - 09:04) — PASS, N0, speedup 1.7087 (current best;/tmp/check-5/check.json) Tree commit fa13351, switches (= /submission/switches.json): the 8 of check #4 +OPT_FUSED_TRAIN=1. Gate 1: windows 12.56 / 13.46 / 13.58 s per step -> ratios 1.8407 / 1.7671 / 1.7087 (mean s/step 13.20 vs 13.49 in check #4; the min-window ratio is dominated by window noise ±5%). Gate 2: bitwise PASS (3200 tensors, 235 reward requests). Peak 102.2 GiB.
3.4 hRound 5 — fused gen-mode (denoise) layer, `OPT_FUSED_GEN=1` (09:05 - )
rms_rowsTriton kernel (gather rows by index / add a GEMM output row with one bf16 rounding / RMSNorm in ATen order / write contiguous or scattered / store the residual sum scattered) +Qwen2Model._forward_inference_gen_fused: per layer [attention GEMMs + fused q/k norm-rope + flash + o_proj] then add+post-norm in one kernel, MLP, and the next layer's input norm fused with the MLP residual add (final norm scatters the output rows). Removes ~12 full-tensor gather/scatter/zeros/add passes per layer-pass. Row-kernel test: 36 cases bitwise. Harness (det, everything on): 817/817 bitwise incl. RNG states; gen 5.42 -> 5.10 s, refine 10.54 -> 9.86 s, eval 60 -> 48 s (single GPU). Full check #6 launched (/tmp/check-6). ### Full check #6 (09:10 - 09:32) — PASS, N0, speedup 1.7457 (current best;/tmp/check-6/check.json) Tree commit 8514fbf, switches (= /submission/switches.json): the 9 of check #5 +OPT_FUSED_GEN=1. Gate 1: windows 12.20 / 13.25 / 13.29 s per step -> ratios 1.8946 / 1.795 / 1.7457 (mean 12.91 s/step). Gate 2: bitwise PASS. Windows 2 and 3 are consistently ~1 s/step slower than window 1 (reference too: 23.78 vs 23.11); phase timers of a similar run show a larger exposed reward wait in generate steps 12 and 14 (1.4-1.6 s vs 0.0-0.9 s).
4.2 hRound 6 — small items (09:35 - ), check #7 launched with `OPT_PACK_FAST`
-OPT_PACK_FAST=1:add_latentstores Python floats for the 0-d device scalarstimestep/dt(they were repeated 20K times per latent into lists thattorch.tensorconverted with one device sync per element):pack_batch0.51 -> 0.2 s on the refine packing; all 30 packed tensors/lists identical (gen and refine structure). Matters on the refine critical path (GPU idle during packing). -OPT_DECODE_HOIST(per-step integer index math once instead of per layer): bitwise, no measurable gain -> off. -OPT_FUSED_MLP(exact silu*mul fwd+bwd, whole-model grads identical): fwd+bwd 4.32 -> 4.30 s on the refine packing, rollouts unchanged -> no gain, off. (Triton flushes fp32 denormals; PyTorch does not: differs only for |gate| in [87.5, 88.5] / bf16 subnormals.) - Blocks tried and dropped: flex block-mask cache between ref and policy forward (the ref forward runs in inference_mode, its mask tensors are inference tensors and cannot be saved for backward). - Critical-path analysis of a pair: gen step = rollout 4.4 s + reward service 2.5 s (the 16-image burst, ~4.3 req/s, 2x slower while the GPUs are busy) + backward/optimizer 1.05 s; refine step = rollout 9.0 + service/forwards 3-4.5 + backward 3.6. The service latency, not my GPU work, is what the post-rollout phase waits for; the remaining controllable parts are rollout and backward.
5.8 hRound 7 — the reward service is slowed by my GPU work: yield the GPU (2026-11-06 10:26 - 11:33)
Profile (per-rank phase timer, judgetimeruns /tmp/time-pt5 and /tmp/time-pt6 on a scratch copy with CUDA-event marks; all 4 ranks logged): - The 4 ranks are in lock-step in every phase (rollout end spread < 0.1 s; the "rank imbalance" I suspected does not exist). - Window 3 (steps 14-15) is the weakest because of the post-rollout wait for the reward service: time from the end of the early block (pack + reference fwd + policy fwd, 0.85 s GPU in gen steps, 3.1-3.5 s in refine steps) to the arrival of the 16 rewards = gen steps 8/10/12/14: 1.3 / 1.1 / 2.0 / 1.9 s, refine steps 9/11/13/15: -0.6 / -0.6 / +0.7 / +1.45 s. Service latency (submission -> last result): gen 2.0-2.8 s, refine 2.4-4.9 s (4.9 s in step 15: the early block ran at the same time). In addition the refine text decode grows with the step (et_text_gen 1.4 / 1.8 / 2.1 / 2.6 s for steps 9/11/13/15, data dependent). - Service probe (standalone judge service, /tmp/work/reward_probe3-5.py; 16 requests like one step): idle makespan 3.8-4.2 s with realistic requests (0.9 s with the first 16 prompts: the cost is dominated by colour classification: DataLoader forks + CLIP crops); 8 busy CPU cores: no effect, 48 busy cores: +50%; continuous GEMM load on the 4 GPUs: 6.2 s (x1.65); the same load with idle gaps: 30ms/2ms 3.9-4.4 s, 10ms/1ms 3.9-4.5 s, 100ms/10ms 4.0-4.3 s, 300ms/25ms 3.9-4.5 s, 45ms/3ms 4.1-4.3 s (all ~7-9% idle) but 100ms/2ms 5.1 s, 300ms/5ms 5.2 s, 1000ms/80ms 4.2-4.6 s, 3000ms/250ms 5.0-5.8 s. Cause: GPU time slicing between my process and the service workers (one per GPU); while my queue is never empty, each small dependent kernel of the service waits for a slice. - ChangeOPT_GPU_YIELD=<us>(newmodeling/bagel/gpu_yield.py, hook inQwen2Model.forward_train, begin/end inonline_rl.main): between the submission of the rewards and the arrival of the results, the post-rollout forwards (reference fwd, early policy fwd) synchronise the stream after every decoder layer and sleep<us>(stops by itself when the rank's own futures are done). No tensor is touched: N0.OPT_GPU_YIELD_CHUNK_US(adaptive layers per gap) exists too. - Time runs on the phase-timer copy (all other switches of check #7): no yield 13.14/13.48/13.95 and 12.76/13.60/14.02 (mean 13.53, 13.46);OPT_GPU_YIELD=3000per layer: windows 12.56 / 13.29 / 12.96 (mean 12.94, speedup_vs_frozen 1.806), exposed wait after the early block gen 0.02/0.03/0.55/0.53 s, refine -0.61/-0.61/0.22/-0.07 s; gen-step service latency 1.35-2.35 s (was 2.0-2.8), refine 2.5-3.8 s (was 2.4-4.9). Chunked variant (8 ms gap per ~100 ms of work): 12.75/13.37/13.26 (13.12): worse in gen steps (service latency 1.9-2.75 s) -> per-layer gaps kept. Early block with yields: refine +0.1 s, gen +0.5 s (the gen-step early block is no longer fully hidden in steps 8/10). - Commit 32f7bb8 (off by default). Full check #8 (11:33 - 11:55) with check-7 set +OPT_GPU_YIELD=3000+OPT_DECODE_SKIP_WARM=1: PASS, N0, speedup 1.8454 (/tmp/check-8/check.json): windows 11.94 / 12.67 / 12.57 s per step -> ratios 1.936 / 1.878 / 1.846; gate 2 bitwise PASS (14 steps, 3200 tensors, 235 reward requests)./submission/switches.jsonupdated to this set. - CPU sampler during the check (/tmp/work/cpu_samples_check8.txt): the 4 training ranks use ~4 cores in total all the time; the service bursts use 12-24 cores in the service processes plus short-lived DataLoader workers (machine total up to 60-80 of 96 logical cores): the service is CPU heavy by itself, my ranks hardly add to it.
7.3 hRound 8 — profile of the check-#8 configuration, early-block anatomy, allocator/prefetch (2026-11-06 11:56 - 13:04)
Profile (judgeprofile --from 14 --iters 2of /submission/r3 + switches of check #8, steps 14 (gen) and 15 (refine), rank 0; profiler inflates ~10%): - GPU busy per phase: gen: text decode 70-83%, denoise 98%, post-rollout 43-55% (yield gaps + wait for the reward service), backward 95%; refine: decode 70-80%, contexts 60%, denoise 98%, early block (pack + ref fwd + policy fwd) ~75%, backward 99%. - Denoise kernel mix (gen rollout trace): GEMMs 80% of kernel time at ~750 TFLOPS (nothing to gain without changing GEMM kernels), flash 7.3%, silu+mul 5.6%, rms/qk-norm-rope 5% (already fused), gather/index/cat 1.6%. - Decode graph: 777 kernels per step; kernel time 5.9 ms/step, intra-graph gaps 0.36 ms, per-step host gap ~0.85 ms (D2H sync + python + graph launch). - Blocking allocator calls: cudaMalloc 0.53 s + cudaFree 0.29 s per pair of steps (empty_cache at the end of every rollout context + re-allocation). - Training fwd+bwd kernels (earlier trace): GEMM 66%, flex attention 12.8%, dispatch index_put/gather/cat/copy ~9.5%; a microbenchmark showedindex_copyonly 10-30% faster thanindex_putper op (whole dispatch fusion would gain <1%): dropped. Phase-timer fix: the PT copies run the early policy forward twice (the patch anchor matched the tail of a 20-space line, see apply_pt.py), so absolute PT step times were ~0.7 s/step too high until pt10; comparisons between PT runs and the early-block numbers stay valid. Early block anatomy with yields (PT, rank 0, relative to the reward submission): refine: pack 0.3-0.4 s, reference fwd 1.4-1.7 s, policy fwd 1.2-1.4 s (3.0-3.5 s total, the service answers at 2.4-3.6 s); gen: pack ~0.05 s, ref fwd 0.6-0.7 s, policy fwd 0.6-0.9 s (1.2-1.6 s). Uncontended service latency measured in situ (diagnostic run waiting for the answers before starting the early block): gen steps 8/10/12/14: 1.31 / 1.34 / 2.00 / 2.15 s, refine steps 9/11/13/15: 1.23 / 1.30 / 3.09 / 2.19 s => in refine steps 9, 11, 15 the early block (not the service) is the critical path, in gen steps and refine step 13 the service is. Runs (phase-timer tree, judgetime; windows s/step): check-8 config 12.45/12.91/13.04 (with the double-forward bug); +OPT_FSDP_PREFETCH=1(corrected tree): 11.86/12.36/12.37 (ratios 1.949/1.925/1.876); +PYTORCH_ALLOC_CONF=expandable_segments:True+OPT_NO_EMPTY_CACHE=1: 11.73/12.28/12.15 (1.971/1.937/1.910), peak reserved 122 vs 129 GiB, peak allocated 101 GiB, no graph-capture failures. Full check #9 launched with check-8 set + these three switches (/tmp/work/sw_final2.json). ### Full check #9 (13:04 - 13:26) — PASS, N0, speedup 1.9153 (/tmp/check-9/check.json) Tree commit 32f7bb8, switches = /submission/switches.json (check-8 set +OPT_FSDP_PREFETCH,PYTORCH_ALLOC_CONF=expandable_segments:True,OPT_NO_EMPTY_CACHE). Gate 1: windows 11.73 / 12.38 / 12.11 s per step -> ratios 1.971 / 1.921 / 1.915. Gate 2: bitwise PASS (14 steps, 3200 tensors, 235 reward requests). Peak allocated 101.3 GiB, reserved 128 GiB (was 136). Repeat of the same configuration (accidentally launched twice, 13:50): windows 11.50 / 12.15 / 12.12, speedup 1.9143, bitwise PASS (/tmp/check-10a.check.json) -> the score is reproducible to ~0.1%; window 3 (steps 14-15) is always the minimum.
8.5 hRound 9 — dead ends of this session (13:26 - 14:15), nothing changes in the submission
- Pipelined decode loop (OPT_DECODE_PIPE, step t+1 launched before the host reads step t; device-side token hand-over, lagged bookkeeping, prototype in /tmp/work/tree-pipe, patch /tmp/work/pipe.patch): bitwise (harness 817/817, RNG states equal) but no speed-up: 385 decode steps 2.46-2.68 s plain vs 2.43-2.57 s pipelined (single GPU harness). The decode step is GPU bound (graph ~6.4 ms); the 0.85 ms "host gap" seen in the profiler trace is mostly profiler overhead. Not committed. -OPT_FUSED_MLP(exact fused silu*mul) re-measured in the rollout harness: bitwise, but gen 5.0 -> 5.0 s, refine 9.79 -> 9.83 s, eval 47.6 -> 48.9 s: no gain although silu+mul are 5.6% of the denoise kernel time (the Triton kernel is not faster than the two ATen kernels). - Remaining controllable time per pair of steps (24.3 s): rollout 13.4 s (denoise 8.8 s at 98% GPU busy with GEMMs at ~750 TFLOPS, decode 4.3 s GPU bound, contexts 1.9 s), post-rollout 4.4 s (gen: reward service 1.3-2.2 s uncontended, refine: early block 3.1-3.5 s), backward+opt 6 s. Items still left, each <= 1% and not done: exact fused AdamW (opt step 81-97 ms -> ~25 ms), partial activation-checkpoint bypass for the last K layers (K/28 of the recompute, memory risk), block-mask sharing between the reference and the policy forward (~0.06 s per refine step), image-prefill dedup for the 4 identical refine inputs (short text prefills are NOT batch invariant: flash split heuristic depends on B*H*m_blocks), pack image-transform memoisation (~0.06 s), yield tuning (service-bound gen steps cannot gain: measured S' ~= S_idle). - Background pre-capture of the smaller-M decode graphs (OPT_DECODE_PRECAPTURE, helper thread with its own stream,no_grad, device context, shorter GIL switch interval; prototype in /tmp/work/tree-pre): works and is bitwise (817/817 in the harness) but gives no speed-up (refine 9.79 -> 9.79-9.83 s, gen 5.00 -> 5.03-5.08 s): the helper's tracing competes with the decode loop for the interpreter. Not committed.