GLM-5.3 1.45×

Harness: Claude Code · switches: OPT_SYNC_FREE_TIMESTEP=1 OPT_SYNC_FREE_ATTN=1 OPT_OVERLAP_PREPACK=1 OPT_SYNC_FREE_TOKENLOOP=1 OPT_MERGE_IDX_CACHE=1 OPT_TEXTGEN_CUDAGRAPH=1 OPT_EARLY_FORWARD=1

Judged
1.45×
Tier
N0
Self-best
1.46×
Rounds
17
Hours
10.0
API cost
$31.1
Input tokens · 99.5% cached
106 M
Output tokens
0.6 M
Requests
789
Text-decode CUDA graph
✓ OPT_TEXTGEN_CUDAGRAPH, min window 1.08× → 1.38× (backward OOM three times; switched to lower-level capture, enabled only from step 1)
Reward wait hidden behind forward
✓ OPT_OVERLAP_PREPACK + OPT_EARLY_FORWARD, 1.38× → 1.44×
Rollout-side bitwise fused kernels
✗ not in the log
Train-side bitwise kernels / freeing memory
✗ (peak 131.1 GiB)
Other main levers
desync of timestep, attention length, token loop, merge index; denoise CUDA graph ~0.9 s slower per window, reverted
Raw work log17 sections
0.8 hRound 1 — profile of the unchanged reference (2026-11-06 18:25–19:00)
Profile: bench/bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-r1
(steps 11=refine + 12=gen on rank 0). Trace 3.6 GB, analyzed with streaming scripts in /tmp/analysis
(trace_summary.py, trace_phases.py, trace_window.py). Trace span 57.5 s (≈46 s without profiler overhead).

Phase structure of one gen+refine window (from the trace timeline, 500 ms buckets):

| phase | wall | GPU busy | notes |
|---|---|---|---|
| refine-step think (text gen, ~275 tok) | 12.5 s | 19% | host/sync bound, 22 tok/s |
| refine-step denoise (20 steps × 3 CFG fwd) | 13 s | 66% | ~4500 .item syncs per 500 ms |
| gen-step think (~176 tok) | 8 s | 19% | same as above |
| gen-step denoise (15 steps × 2 CFG fwd) | 7 s | 66% | |
| pure-idle gaps (reward wait etc.) | ~7.6 s | 0% | 4.1 s + 1.1 s + 1.15 s + 1.18 s |
| training fwd/bwd (GPU saturated, host idle) | ~9 s | ~95% | genuinely GPU-bound |

Top hotspot (kind: missed fast path / host syncs): **244,157 aten::item calls = 9.5 s CPU (22% of
CPU total), plus 4.8 s "Command Buffer Full"**. Breakdown found in the trace:

1. bagel.py generate_image_mix: timestep = torch.tensor([t] * x_t.shape[0], device=x_t.device) —
   t is a 0-dim CUDA tensor, so building the list converts all 4096 latent-token elements with one
   blocking D2H each: **runs of exactly 8192 item events (4096 .item + 4096 _local_scalar_dense)
   once per denoising timestep** → ~143k syncs per window (~35 per run of 8192 at ~25 µs). Plus
   t > cfg_interval[0] on a CUDA tensor → 1 more sync per timestep.
2. Per-layer .item() in the KV-cache attention helpers (qwen2_navit.py
   _static_kv_cache_attention 3/layer, _merge_static_kv_prefix_and_query 5/layer): ~47k during
   text gen (582 forwards × 27 layers × 3), ~12k during denoise.
3. Text-gen token loop (bagel.generate_text_batch): ~8 .item() per token.

Second hotspot: reward-service wait — a 4.1 s pure-idle gap per window (all 4 ranks share ONE reward
URL with 4 sync workers; 16 images per round), fully exposed because nothing overlaps it.
Third: training fwd/bwd is GPU-saturated (~9 s), little to gain there without kernel changes.

### Change (this round): OPT_SYNC_FREE_TIMESTEP (N0)

modeling/bagel/bagel.py generate_image_mix: iterate timesteps.cpu() (one D2H per rollout)
instead of the CUDA tensor; build the per-timestep conditioning vector with
torch.full((n,), float(t), dtype=timesteps.dtype, device=...) instead of
torch.tensor([t] * n, device=...); pass the original CUDA element timesteps[i] to
sde_sampler.step and timesteps_for_sde. float(t) is the exact float64 of the float32 t, and
torch.full casts back to the same float32 → every device-side input is bit-identical; the CPU-side
t > cfg_interval comparison yields the same boolean without a sync. Removes ~143k blocking syncs
per window (~4097 per denoising timestep).

- switches.json: {"environment": {"OPT_SYNC_FREE_TIMESTEP": "1"}}
- time result: /tmp/time-r1 — windows [22.2733, 22.7816, 22.3855] s/step,
  speedup_vs_frozen_reference = 1.0268 (mean over windows). Per-window ratios vs the
  frozen reference [22.7266, 23.5265, 22.9922]: 1.0204 / 1.0327 / 1.0271 — the minimum
  (1.0204) is below the gate-1 threshold max(1.02, 1+2×0.0152) = 1.0304, so round 1 alone
  would be judged no_benefit. More sync elimination needed (round 2: attention helpers).
  Peak memory 130.7 GiB (reference ~131) — unchanged.
1.0 hRound 2 — attention-helper host mirrors (2026-11-06 19:00–19:30)
Hotspot (from the round-1 profile, #2): per-layer .item() reads in the KV-cache attention
helpers of modeling/bagel/qwen2_navit.py — _static_kv_cache_attention (3/layer),
_merge_static_kv_prefix_and_query (5/layer), _static_kv_cache_readonly_attention (2 via
merge): ~47k blocking syncs during text gen (582 forwards × 27 layers × 3) + ~12k during
denoise per window. Every read blocks the host on a D2H copy while the GPU idles.

Change: OPT_SYNC_FREE_ATTN (N0) — host mirrors of the exact integers the device reads:

- StaticKVCache (qwen2_navit.py): new seq_lens_host mirror (list of ints, None = stale
  → fall back), new set_seq_lens(values) (identical device op seq_lens[:] = torch.tensor(...)
  plus mirror), advance_seq_lens maintains the mirror (stale-marks it if indices are untagged).
- Producers tag the length tensors with the host values they were built from (attribute
  _r3_host): prepare_prompts text_token_lens, prepare_vit_images / prepare_vae_images /
  prepare_vae_latent packed_seqlens (bagel.py); generate_text_batch query_lens_for_model
  ([1]*n) and active_sequences_original_indices (host list kept in lockstep with the
  boolean-mask filter, also replaces the per-element .item() of original_batch_idx);
  update_context_text/image arange cache indices (rollout_controller.py); the two
  seq_lens[:] = torch.tensor(...) writes become set_seq_lens(kv_lens).
- The three helpers read max/sum/first scalars from the mirrors when present (and skip the
  (query_lens == query_lens[0]).all() GPU compare), else take the original .item() path.
  All device tensors and kernel inputs are unchanged; every mirrored value is the exact
  integer the device read would return, and any untagged/stale mirror falls back.

Unit smoke test of the mirror bookkeeping (incl. deepcopy of cfg contexts, partial indices,
stale marking): /tmp/analysis/test_host_mirrors.py — ALL PASS.

- switches.json: {"environment": {"OPT_SYNC_FREE_TIMESTEP": "1", "OPT_SYNC_FREE_ATTN": "1"}}
- time result: /tmp/time-r2 — windows [21.529, 21.2606, 21.2715] s/step,
  speedup_vs_frozen_reference = 1.0809; per-window ratios 1.0556 / 1.1066 / 1.0809,
  min 1.0556 — above the gate-1 threshold 1.0304. Per-step: refine 14–15.5 s → 12.7–13.3 s,
  gen 31–33 s → 27.3–29.7 s. Peak memory 130.72 GiB (unchanged).
1.5 hRound 3 — overlap prepack with the reward wait (2026-11-06 19:30–19:55)
Hotspot: the post-rollout host phase — the reward futures are resolved (blocking HTTP wait,
~1.1–4.1 s of pure idle per step) before any packing work starts. prepack (tokenization,
timestep subselection via random.sample, data-point assembly) reads only rollout outputs,
never reward values.

Change: OPT_OVERLAP_PREPACK (N0) — train/online_rl.py: when set, run prepack right
after rollout() returns (reward futures still in flight) instead of after the futures loop.
RNG order is provably unchanged: prepack's random.sample is the only Python-global-random
consumer in the loop, and nothing between the two positions consumes Python or torch RNG
(rollout_buffer.sample uses its own seeded torch.Generator). prepack mutates only
per_sample_results, disjoint from every field the futures loop reads.

- switches.json: {"OPT_SYNC_FREE_TIMESTEP": "1", "OPT_SYNC_FREE_ATTN": "1", "OPT_OVERLAP_PREPACK": "1"}
- time result: /tmp/time-r3 — windows [20.9422, 21.7319, 21.4014],
  speedup_vs_frozen_reference = 1.0807; per-window ratios 1.0852 / 1.0826 / 1.0743,
  min 1.0743 (up from 1.0556 — the weakest window improved; gate 1 needs > 1.0304).
  Peak memory 130.72 GiB (unchanged).

### Trace findings for the next rounds (round-1 trace, /tmp/analysis/)

- Allocator stalls: 324 ms cudaStreamSynchronize + 471 ms cudaFree + 270 ms sync
  inside Optimizer.step#AdamW.step, and 113+215+216+212 ms cudaFree/sync inside
  FSDP::root_post_backward_callback — the caching allocator evicting blocks under memory
  pressure (130.7 GiB peak, reward workers on the same GPUs). PYTORCH_CUDA_ALLOC_CONF is
  not in the judge's reserved_env → try expandable_segments:True as a pure switch.
- aten::nonzero × 19,312 (all under aten::index = boolean-mask advanced indexing):
  dominated by _merge_static_kv_prefix_and_query — its past_mask/new_mask flat-index
  computations run once per layer (27×) per forward although they depend only on the sequence
  lengths. ~7 nonzero syncs × 27 layers × ~95 denoise forwards + text forwards ≈ 19k per
  window. Plan: compute the flat index tensors once per forward (cache keyed on the host
  length mirrors) and reuse across layers — same elements in the same row-major order, so
  all gathers/scatters are bit-identical.
- Text-gen token loop still has ~3 blocking reads per token per active sequence (round-5
  change already coded: one .tolist() per tensor per token, OPT_SYNC_FREE_TOKENLOOP).
- PIL pickling for rollout_buffer.all_gather measured trivial (6 ms / 8 images, raw-byte
  transport) — NOT a hotspot, idea dropped.

check --quick on rounds 1–3 (/tmp/check-q1, 2026-11-06 20:07): PASS — gates 0 and 2
bitwise identical to the frozen reference record (14 steps, deterministic mode; all optimizer
parameter/state SHA-256s and every reward-request hash match). Switch set verified:
TIMESTEP + ATTN + PREPACK.
1.7 hRound 4–6 — token-loop batched reads, merge-index cache, allocator conf (2026-11-06 20:00–20:45)
- OPT_SYNC_FREE_TOKENLOOP (N0): generate_text_batch reads the per-token device values
  with one .tolist() per tensor per token (3 blocking reads) instead of ~3 .item() per
  active sequence (~12 at n=4); reads happen before the loop's device writes and iteration i
  only writes the index it reads → identical integers. Unit-verified against the original path.
- OPT_MERGE_IDX_CACHE (N0): _merge_static_kv_prefix_and_query (and the varlen write-back
  in _static_kv_cache_attention) build gather/scatter indices via boolean-mask indexing
  (aten::nonzero blocking syncs) once per layer although they depend only on the sequence
  lengths. Now computed once per length signature (LRU-cached module dict) and reused across
  the 27 layers. Verified bitwise-identical to the boolean-mask path on CPU:
  /tmp/analysis/test_merge_cache.py + test_merge_equiv.py (unequal/zero/equal lens, selected
  subsets, dtypes). ~19k fewer nonzero syncs per window.
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (pure env switch, not in reserved_env):
  targets the 471+215+216+113 ms cudaFree stalls and 324+270+212 ms syncs found inside
  Optimizer.step#AdamW.step and FSDP::root_post_backward_callback — allocator evictions
  under memory pressure (130.7 GiB peak). Bitwise-neutral by construction (allocator only).
- switches.json: all six entries (TIMESTEP, ATTN, PREPACK, TOKENLOOP, MERGE_CACHE,
  PYTORCH_CUDA_ALLOC_CONF).
- time result: /tmp/time-r4 — REGRESSION: windows [22.7692, 25.4165, 23.1565],
  speedup_vs_frozen_reference = 0.9706, per-window ratios 0.9981/0.9256/0.9929, min 0.9256.
  Even steps 31–34.7 s (vs 27.3–29.9 in time-r3), odd 13.9–16.2 s — every phase slower.
  Attribution: the two code changes strictly remove device syncs and ops (unit-verified), so
  PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is the suspect (VMM mapping overhead
  on this workload's many small allocations). Dropped from switches.json; re-timing the five
  OPT_* switches as /tmp/time-r5. Lesson: allocator settings are not free wins here — the
  471 ms cudaFree stalls stay unfixed for now.

- time result (five switches, alloc conf dropped): **/tmp/time-r5 — windows
  [20.282, 21.7367, 20.8982], speedup_vs_frozen_reference = 1.1006; per-window ratios
  1.1205 / 1.0823 / 1.1002, min 1.0823** (up from 1.0743). Confirms the attribution:
  tokenloop + merge-cache add ~0.8 pp on the min ratio and expandable_segments cost ~10 pp.
  Peak memory 130.73 GiB (unchanged). Full check launched on this set: /tmp/check-full-1.

Full check on the five-switch set (/tmp/check-full-1, 20:40–21:06): PASS. Gate 1:
windows [21.429, 21.6817, 20.8499] s/step → ratios 1.0606 / 1.0851 / 1.1027, **speedup
1.0606** (min window ratio; above the 1.0304 gate). Gate 2 (correctness.numerics):
PASS — bitwise identical to the frozen reference record (optimizer params/state SHA-256s
and every reward request). Peak memory 130.73 GiB (unchanged). The five-switch submission
state is fully verified end-to-end. (time-r5 had measured 1.0823 on the same set — run
noise; both well above the gate.)
2.6 hRound 7 — CUDA-graph text-gen step (2026-11-06 20:45–22:44: promoted — full check PASS 1.3739)
Hotspot (round-1 profile): the think phase is host-dispatch bound — ~5,700 host ops and
~1,687 kernel launches per token, GPU busy only 19% (22 tok/s, 12.5 s per refine-think).

Change: OPT_TEXTGEN_CUDAGRAPH (N0) — capture the per-token text forward
(embed_tokens → forward_inference (mode "und") → lm_head) as a CUDA graph, one per
active-batch size n (4,3,2,1; n only decreases within a call, so shared-pool replay order
equals capture order). Inputs live in fixed buffers refilled before each replay; the KV
cache, its seq_lens and selected_cache_indices are read **on device inside the captured
region** (the fast path calls flash_attn_with_kvcache, which has no max_seqlen_k
parameter and takes cache_seqlens as a device tensor — per-batch bounds are read at replay
time), so nothing that changes per token is baked into the graph except the constant
seqlen_q=1. Sampling (explicit torch Generator) stays outside the graph → RNG stream
unchanged. Graphs are rebuilt per generate_text_batch call (they must not outlive the
rollout context or the StaticKVCache), any capture error falls back to eager for the rest of
the run, and the switch requires OPT_SYNC_FREE_ATTN (capture region must be free of
blocking device reads). Committed c721985; switch OFF in /submission/switches.json until
verified — testing via the separate config /tmp/switches-graph.json (six switches).

Pre-flight verification (synthetic, /tmp/analysis/test_cudagraph_kvcache.py, GPU):
flash_attn_with_kvcache is capture-safe, and replay with refilled input buffers and
ADVANCED cache_seqlens (device writes between replays, the exact token-loop pattern) is
bitwise identical to eager across an 8-token loop.

First time (/tmp/time-r6): ERROR — OOM. The run died in loss.backward() at step 0
(rank 1): the graphs engaged (private pools live), but the side-stream warmup blocks +
private pools shifted the allocator's segment layout; the backward's 740 MiB contiguous
request found no free block (3.58 GiB reserved-but-unallocated inside torch, only 214 MiB
free at the driver — the reward worker holds 3.92 GiB of the same GPU). The reference runs
at ~0.5 GiB headroom, so any extra reservation is fatal. A synthetic allocator test
(/tmp/analysis/test_graph_alloc.py) showed the captures themselves are cheap with room to
spare — the failure is fragmentation at the razor edge.

Fix (commit 8971c45): release free segments with torch.cuda.empty_cache() right after
rollout() returns, gated on OPT_TEXTGEN_CUDAGRAPH — the cost lands inside the
reward-service wait (pure idle), and the training phase's large allocations then find
contiguous memory. Memory management only, no tensor value touched. Also cut the graph
warmup from 3 to 2 iterations. Re-timing as /tmp/time-r7.

Evidence the graph path itself works: time-r6's log shows the step-0 think phase at
~108–118 it/s (385 tokens in 3.0 s) vs the reference's ~26 it/s (~14.7 s) — a ~4.2×
speedup of the think phase, before the run died at the backward.

**Re-time with the empty_cache fix (/tmp/time-r7): same ERROR, byte-identical memory
state** (129.94 GiB allocated / 3.58 GiB reserved-unallocated / 214.38 MiB driver-free,
740 MiB request in loss.backward(), rank 1, step 0). So the empty_cache is not the
lever: the train phase's own deterministic re-carve recreates the same layout, and the
graphs' ~10 MB direct footprint plus the rollout-phase carve shift is enough to remove
the one contiguous ≥740 MiB hole the reference's backward relied on. The heap runs at
~214 MiB driver headroom — any perturbation is fatal.

Next attempt (/tmp/time-r8): the failure is textbook fragmentation (free space exists, no
contiguous piece), so add the allocator switch PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256
— large free blocks are then never split for small requests, keeping big holes intact for
the backward's 740 MiB allocations. Zero code, not in reserved_env; expandable_segments
(the setting that regressed 10 pp in round 4) is a different mechanism. Config:
/tmp/switches-graph2.json (six switches + the conf).

/tmp/time-r8: ERROR again, OOM moved — now a 140 MiB request fails with 8.22 GiB
reserved-but-unallocated and 16 MiB driver-free (rank 0): max_split makes the allocator
reserve MORE (small requests carve new segments instead of splitting), so the conf is a
dead end, dropped.

Root cause found (torch source, torch/cuda/graphs.py:240-252):
torch.cuda.graph.__enter__ unconditionally runs torch.cuda.synchronize() +
torch.cuda.empty_cache() before EVERY capture (gc.collect() only when forced — default
off). With 4 captures per think call × 32 calls, that released all free segments
repeatedly and reshaped the allocator; the train phase's deterministic carve then had no
contiguous ≥740 MiB hole. It also explains why the r7 empty_cache changed nothing (the
captures had already done it, and the train phase re-carves identically).

Fix (commit 226faee): capture through the low-level CUDAGraph.capture_begin/capture_end
on a dedicated side stream — same kernels, same pool, but the allocator's segment layout
stays identical to the eager rollout. Dropped the post-rollout empty_cache (it would now be
the only reshaper) and the allocator conf. Synthetic low-level-path capture/replay test
re-verified bitwise-equal (/tmp/analysis/test_cudagraph_kvcache_lowlevel.py).

/tmp/time-r9: ERROR — the SAME OOM, byte-identical totals again (740 MiB request,
129.94 GiB allocated / 3.58 GiB reserved-unallocated / 214.38 MiB driver-free, rank 1).
The totals are live-set-determined and identical across all three capture variants; what
differs from the working five-switch runs is the free-block STRUCTURE (where the free bytes
sit). Re-reading the situation: all three failures are at step 0's backward — the first
optimizer step materializes the entire AdamW state (~44 GiB/rank of fp32 params + moments),
the largest allocation wave of the run, on a heap that also still carries the step-0
evaluation's leftovers. The five-switch runs survive that squeeze by layout; any extra
allocation source re-rolls the dice.

Fix 2 (commit 092b124): engage the graphs only from optimizer step 1 (curr_step >= 1,
set in online_rl.py before the eval/rollout; bagel's _TEXTGEN_CUDAGRAPH_ON defaults off).
Step 0 runs the eager path — exactly what the reference computes, no value changes — and
the measured windows (10–15) plus the gate-2 record (steps 1–13) run the graph path.
Deterministic rule, identical in every run.

**/tmp/time-r10: OK — the big one. Windows [16.3512, 17.01, 16.4138] s/step → per-window
ratios 1.3900 / 1.3831 / 1.4006, min 1.3831, speedup_vs_frozen_reference 1.3912** (gate 1
needs > 1.0304; the five-switch state was 1.0606–1.0823). Per-step: refine 28.5 → ~22.9 s,
gen 12.7 → ~10.3 s. Peak memory 131.2 GiB (reference 130.7 — the pools cost ~0.5 GiB).

check --quick on the six-switch config (/tmp/check-q2, 22:08–22:20): PASS —
correctness.numerics PASS, bitwise identical to the frozen reference record with the
graph path engaged (steps 1–13 graphed, step 0 eager). **Promoted:
/submission/switches.json now has all six switches.**

**Full check on the promoted six-switch state (/tmp/check-full-2, 22:21–22:44): PASS —
speedup 1.3739** (windows [16.1519, 17.0889, 16.7345] s/step, ratios 1.4071 / 1.3767 /
1.3739; gate 2 correctness.numerics PASS — bitwise identical). Round 7 complete: the
submission's verified speedup went from 1.0606 to 1.3739.
4.0 hRound 8 — denoise-loop sync hoist (2026-11-06 22:20–23:00: no clear gain, switch stays off)
Post-graph split (time-r10): gen ~10.5 s, refine ~22.7 s per step; denoise (~15 s/window,
66 % GPU busy) is now the dominant phase. A denoise _forward_flow graph was scoped and
DROPPED: cfg_interval is (0.4, 1.0], so the CFG branch flips mid-loop (two graphs needed),
capture+warmup would cost ~2.4 s/window against ~1–2 s of dispatch savings — net marginal;
the phase is GPU-bound, not dispatch-bound like think was (19 %).

Change: OPT_SYNC_FREE_DENOISE (N0, commit 40c3814, off by default) — hoist the denoise
loop's blocking reads: the SDE index set/first index read once per rollout instead of two
blocking reads per timestep; sum(packed_seqlens) in _forward_flow (per timestep) and in
forward_cache_update_vit/vae (per rollout) from the _r3_host mirrors; the tautological
timestep.unique() assert skipped under the flag (the timestep vector is a uniform fill by
construction); the three end-of-rollout (packed_seqlens - 2).tolist() reads reduced to
mirror arithmetic. Same integers/booleans everywhere; ~150 fewer blocking syncs per window.
To be timed after the full check (config /tmp/switches-r8.json, seven switches).

/tmp/time-r11: ERROR — my bug: `AttributeError: 'list' object has no attribute
'tolist' at the first generate_image_mix — online_rl passes sde_timestep_idx` as
timestep_idx.tolist(), i.e. a plain LIST, so the original membership tests never synced
and my hoisted .tolist() crashed. Fixed (commit 198a945) to handle both shapes; for a
list the hoist is a no-op, for a tensor it saves the two per-timestep blocking reads.
Re-timed as /tmp/time-r12.

/tmp/time-r12: no clear gain — round 8 conclusion: NOT shipped. Windows
[15.9812, 16.9275, 17.0734] → ratios 1.4221 / 1.3898 / 1.3467, mean speedup 1.3854, vs
the six-switch state's 1.3912 mean / 1.3831 min (time-r10): W1 −0.37 s, W2 −0.08 s,
W3 +0.66 s (beyond the ±0.3–0.5 s run noise seen between r10 and check-full-2). The ~80
blocking syncs/window it removes are worth less than the noise. OPT_SYNC_FREE_DENOISE
stays OFF (code committed, off by default — 40c3814 + 198a945); /submission/switches.json
remains the verified six-switch set (full check PASS, 1.3739).
5.1 hRound 9 — pack-time per-element device reads (2026-11-06 23:00–23:40: not promoted; re-measured round 11, stays off)
Profile first (profile --out /tmp/prof-r9, steps 11–12, rank 0; trace deleted after analysis):
- denoise phase now ~90% GPU-busy (rounds 1–2 sync fixes already recovered it) → a denoise
  _forward_flow graph is not worth it (also two variants needed for cfg_interval).
- train phase saturated (kernel time ≥ wall — nothing to win on host there).
- think phase 60–69% GPU-busy (per-token eager sampling + 3 blocking tolists between replays).
- reward-wait idle ~2.5 s/window.
- Top host hotspot: a burst right after Optimizer.zero_grad of 15,691 ×
  (aten::item + aten::_local_scalar_dense + cudaMemcpyAsync + cudaStreamSynchronize)
  in ~0.5 s — pure blocking per-element device reads at pack time.
  Op census attributes it to pack_batch/pack_iterator → add_latent.

Root cause: add_latent repeats the 0-dim CUDA dt/timestep tensors once per latent
token into packed_input['packed_timesteps'], loss_info['dts'],
loss_info['packed_timesteps_for_loss']; _convert_to_tensors then rebuilds those lists
with torch.tensor(list, dtype=float32, device=...), which reads every element back with a
blocking D2H. pack_batch's CFG filter also compares each per-sample timestep tensor against
the bounds (2 syncs per element).

Change (commit 1d87f41, switch OPT_PACK_HOST_FLOATS, off by default, N0):
add_latent converts dt/timestep to host floats once per SDE step; pack_batch hoists
one .tolist() of the shared per-sample timestep tensor for the CFG filter.
Bitwise argument: float32→float64→float32 round-trip is lossless (final torch.tensor
identical; verified with an int32-view equality unit test). The CFG comparison keeps the
same booleans: bounds are 0.4/1.0, and no float32 grid point lies between float64(0.4) and
float32(0.4) (window < 1 ulp), 1.0 is exact — so host float64 comparison ≡ GPU float32
comparison for every possible timestep.

Time (/tmp/time-r13, seven switches incl. OPT_PACK_HOST_FLOATS, 23:41): windows
[16.5587, 16.9045, 17.1882], ratios 1.3724/1.3918/1.3376, min 1.3376, mean
speedup 1.3671 — vs six-switch time-r10 min 1.3831 (windows [16.35, 17.01,
16.41]). The expected gain (~0.25 s/step) is at the run-noise floor (±0.3–0.5
s/window) and the delta is negative on this sample (W3 +0.78 s). **Not
promoted** — /submission/switches.json stays at the verified six-switch set;
the code is committed, off by default. To be re-measured stacked with the next
round's change before any promotion.
5.4 hRound 10 — overlap the reward wait with pack + both forwards (2026-11-06 23:45–00:15)
Hotspot from the round-9 profile: reward-wait idle ~2.5 s/window (the largest remaining
non-GPU phase after denoise/train/think), plus the pack (host walk + VAE encode) and the
ref/policy forwards all running serially after the join.

Change (commit eaaf593, switch OPT_EARLY_FORWARD, off by default, N0; requires
OPT_OVERLAP_PREPACK): the forwards need only packed_input/label_ids, never the
reward-derived advantages. Under the switch, pack_iterator runs once with
defer_advantages=True (pack_batch records every advantage-extend as
(kind, count, (sample_idx, round_idx)) in loss_info['_adv_recording']), then
optimizer.zero_grad() + fsdp_model.train() in the reference's order, then the
forward block — factored into _run_forwards, same code for both paths — runs while the
reward futures are in flight. After the join and advantage computation,
finalize_deferred_advantages replays the recording with the real values.
- Bitwise: same walk, same extend counts and order, same
  text_advantages[sample_idx][round_idx] values (including the else 0 fallback);
  unit test (/tmp/analysis/test_deferred_advantages.py): deferred pack vs normal pack →
  identical packed_input, identical non-advantage loss_info, torch.equal on both
  advantage tensors.
- Collectives: the join, buffer update/all_gather, advantage all_reduces and the stats
  barrier all move together after the forwards; per-rank order stays identical, so every
  collective still matches (no deadlock; collectives complete only when all ranks enter).
- RNG: the moved block consumes none.
- Memory: allocation sequence kept (zero_grad → pack → ref fwd → policy fwd); the only
  trajectory change is rollout_buffer.all_gather()'s torch.cuda.empty_cache() now
  running between the policy forward and the backward on gen steps (was before the pack).
  Step 0 (the knife-edge step) is exercised by the time run.

Time (/tmp/time-r14, six-switch base + OPT_EARLY_FORWARD, 00:01): windows
[15.5648, 16.0194, 15.9711], ratios 1.4601/1.4686/1.4396, min 1.4396, mean
speedup 1.4561 (six-switch time-r10: min 1.3831, mean 1.3912). All three windows
improved by ~0.7–1.0 s — a real gain (~0.85 s/window), not noise. Step 0 (the memory
knife-edge) completed normally (64.55 s incl. eval); peak allocated 131.09 GiB (was
131.45). Promoted into /submission/switches.json (seven switches);
check --quick (bitwise) launched → /tmp/check-q4.
6.2 hRound 11 — denoise-step CUDA graphs (2026-11-07 00:12–01:00: negative, switches stay off)
Profile first (profile --out /tmp/prof-r11, steps 11–12, seven-switch state; 2.0 GB trace,
deleted after analysis; bucketed with /tmp/analysis/trace_phases.py):
- refine step: denoise 3.2 s at ~69% GPU busy (~1.0 s idle; ~45 ms host gap per timestep,
  ~3.5k kernel launches each), think 384 tok 9.2 s at ~94.5% busy (graphs already recovered it),
  pack ~0.5 s with the 15–18k-read storm still present, forwards+train ~7 s saturated.
- gen step: denoise 2.4 s at ~67% busy (~0.8 s idle), think 4.2 s at ~94% busy,
  pack ~0.5 s, forwards+join ~1.6 s (the round-10 overlap works: the reward wait is hidden
  behind pack+forwards), train ~1.4 s saturated.
- Denoise step wall ≈150 ms with ≈103 ms of kernels → the idle is host dispatch, exactly what
  a CUDA graph removes. The scale branches in _forward_flow (if cfg_text_scale > 1.0 gates
  the cfg pair) mean two kernel variants per call; the trace shows the (1.0,1.0) timesteps run
  at half intensity (2–3 of 37 per window) — capturing that variant costs more than it saves,
  so only the full-CFG variant (32/37 steps) is captured.

Change (switch OPT_DENOISE_CUDAGRAPH, off by default, N0; **requires
OPT_SYNC_FREE_DENOISE=1**: without the round-8 hoist, _forward_flow's sum(packed_seqlens)
is a blocking read per timestep, illegal inside capture; the _dn_ok gate enforces it):
_DenoiseStepGraph (same low-level capture pattern as the text graphs: 2 read-only warmups on a
side stream, capture_begin(pool, capture_error_mode="global") on the shared capture stream)
captures one _forward_flow per generate_image_mix call; x_t/timestep live in static buffers
refilled before each replay; the SDE timestep (noise draw) and the 1.0/1.0 timesteps stay eager.
The eager call was rewritten to self._forward_flow(x_t=…, timestep=…, cfg scales…, **_dn_kwargs)
— identical kwargs, one source of truth for both paths. Engagement gate from optimizer step 1
(same rule as the text graphs; step 0 is the memory knife-edge). Capture-failure fallback prints
once and stays eager for the run. Synthetic capture/replay equivalence test:
/tmp/analysis/test_denoise_graph.py.

Time (/tmp/time-r15, eight switches = seven + OPT_PACK_HOST_FLOATS re-measure, 00:41):
windows [15.6157, 16.1178, 16.0789], min-ratio 1.4300, mean 1.4483 — at-or-below the
seven-switch baseline (time-r14 min 1.4396) for the second independent time. The early pack of
round 10 already hides the pack read-storm inside the reward wait, so the fix has nothing left
to win. OPT_PACK_HOST_FLOATS stays OFF permanently (code committed, off by default).
Denoise-graph timing (/tmp/time-r16, nine switches = seven + OPT_SYNC_FREE_DENOISE +
OPT_DENOISE_CUDAGRAPH, 01:00): windows [16.3453, 17.115, 17.0271], min-ratio 1.3503,
mean 1.3715 — a consistent ~0.9 s/window REGRESSION vs the seven-switch baseline (min
1.4396). Zero capture-failure prints: the graphs engaged and simply don't pay. A ~3.5k-kernel
graph instantiate costs ~1.2–1.5 s per call, and one call replays it only 14–18 times
(~0.6–0.8 s of host-gap savings) — structurally negative, unlike the small text-step graphs
(~300 kernels, replayed 196–384× per call). **OPT_DENOISE_CUDAGRAPH and OPT_SYNC_FREE_DENOISE
stay OFF** (code committed 16d1a63, off by default; mechanics unit-tested bitwise — the
failure is economic, not correctness). The denoise phase's ~1.8 s/window idle is locked
behind per-call capture cost: the SDE index is re-sampled per call, so no multi-step or
persistent graph aligns with the loop.

Round 11 close-out. Full check of the seven-switch submission state
(/tmp/check-full-3, launched 00:54, done 01:15): PASS, speedup 1.3854
(gate 1 windows [15.5885, 16.1145, 16.5955], ratios 1.4579/1.4600/1.3854; gate 2
bitwise identical: 14 optimizer steps, param/state hashes and all reward requests
match bench/records/ref.json). This is the submission's headline evidence — the
full judge procedure on the exact tree + switches.json now in /submission.
Window 3 varies ±3% between runs (16.60 here vs 15.97 in time-r14), which is the
whole difference between 1.3854 and 1.4396; the gate needs only > 1.0304.

Post-round-11 screening (no GPU cost worth a round): torch.optim.AdamW(fused=True)
as a possible optimizer-phase win — pre-test /tmp/analysis/test_fused_adamw.py shows
params differ by ~1 ULP (1.19e-07) and exp_avg by 5.6e-09 after 20 steps on float32:
the fused kernel's scalar math (bias correction) does not match the foreach path.
Ruled out for N0. (Also noted: fused keeps step on CUDA, the default on CPU.)
7.1 hRound 12 — batched token-loop bookkeeping (2026-11-07 01:05–01:32: no measurable gain, switch stays off)
Hotspot (from prof-r11): the think phases are 94–95% GPU busy; the residual ~5%
(≈0.74 s/window) is per-token host work: three .tolist() blocking reads, the
boolean-mask index that builds continuing_original_indices (a hidden nonzero
DtoH sync), and one index_put per active row.

Change (switch OPT_TOKENLOOP_BATCHED, off by default, N0; requires
OPT_SYNC_FREE_TOKENLOOP + OPT_SYNC_FREE_ATTN's host mirrors): in
generate_text_batch's token loop, (1) one fused blocking read — active tokens,
current tokens of the active rows and sampled probabilities packed as int32 views
(int64 → two little-endian halves, float32 → raw bits) into one torch.cat and one
.tolist(), reassembled on the host (exact round trip; (lo & 0xffffffff) + (hi << 32),
struct for the floats); (2) the EOS comparison stays host-side on the same values, so
keep_host knows the continuing rows and the boolean-mask indexing (blocking nonzero)
plus the per-row curr_tokens[...] = next_tokens[i] writes collapse into ONE
index_put over host-built unique indices; continuing_original_indices becomes a
host-built int64 tensor with the same values, order and _r3_host mirror; (3) the
last-step end_token_id override becomes the written values themselves
(torch.full), exactly what the reference writes for continuing rows. Per token:
4 blocking reads + ~8 launches → 1 blocking read + ~4 launches.
Unit test /tmp/analysis/test_tokenloop_batched.py: trajectories (incl. values ≥ 2^31
and negatives), EOS corners (all-rows-end, mid-loop, last step), continuing-index
values/mirror, int64+float32 decode round-trips — ALL OK (the first test run
caught a real bug: signed-int32 lo must be masked unsigned before reassembly).

Time (/tmp/time-r17, eight switches, 01:16): windows [15.5388, 15.9394, 16.12],
ratios 1.4628/1.4760/1.4263, min 1.4263, mean 1.4550 vs seven-switch time-r14
min 1.4396, mean 1.4561 (and the same config's gate-1 run in check-full-3: mean
1.434). w1/w2 improve by 0.03/0.08 s, w3 degrades by 0.15 s — window 3 swings
±0.6 s between identical-config runs, so the ~0.1 s/window expected gain is
unresolvable. Switch stays OFF; code committed off-by-default. The submission
remains the seven-switch full-check-PASS state (1.3854).
7.4 hRound 13 — profile-verify round 12; denoise idle re-attributed (2026-11-07 01:33–02:05: both no-gain, submission unchanged)
Round 12 verification by mechanism (profile --out /tmp/prof-r13, steps 11–12, eight
switches = seven + OPT_TOKENLOOP_BATCHED; trace bucketed with trace_phases.py):
- think spans: refine 8.60 s @ 94.2% busy (r11) vs 9.20 s @ 93.6% (r13); gen 4.00 s @
  94.2% vs 4.20 s @ 94.2%. GPU kernel time itself moved +6% between the two profile runs
  (clock variance), and wall/kern is unchanged (1.060-1.061 vs 1.063-1.069). The batched
  token loop does NOT reduce the think host gap measurably — the removed syncs were
  data-ready (cheap) and the residual gap is the sample-chain serial dependency.
  OPT_TOKENLOOP_BATCHED definitively stays OFF (round 12's wall-clock wash confirmed
  by mechanism; unit-tested bitwise, committed ac76a28 off-by-default).

Denoise idle re-attribution (new tools /tmp/analysis/trace_gaps.py + gap_hosts.py on
the r13 trace): the denoise phase's profiled 67-70% busy decomposes into
(a) ~293 ms per 2 s window of sub-µs inter-kernel gaps (320k kernels in 2 s — ~25k per
timestep, avg 4.2 µs; ~0.9 µs launch floor each), (b) ~368 ms in 138 gaps of 1-10 ms —
**and those are filled by PyTorch Profiler [Trace]/Buffer-Flush events, i.e. CUPTI
overhead, not workload stalls**, (c) one 39 ms gap (also a Trace flush). Profiling also
inflates wall (steps 11-12 span 33.0 s profiled vs ~26.4 s in time-r17 — ~25-30%), so
profile-based idle shares are systematically overstated. Cross-checking with wall clocks:
the TRUE denoise host overhead is ~0.5–0.8 s/window — which is exactly the replay saving
the round-11 graph experiment measured against its ~1.2–1.5 s/call instantiate cost
(time-r16 net −0.9 s/window). The denoise lever is closed from both directions, and
round 8's OPT_SYNC_FREE_DENOISE no-gain is explained: the queue it would drain never
builds depth because the host and GPU run at nearly the same pace.

State after round 13: submission = seven-switch set, full check PASS 1.3854
(/tmp/check-full-3). No further lever with positive expected value has survived
profiling; remaining rounds keep the state verified and documented.

Round 13 close-out. check --quick on the CURRENT tree (ac76a28) with the
submission config (/tmp/check-q5, 01:49–02:00): PASS, bitwise identical — the
default switch-off path is verified after round-12's edit (check-full-3 had run on the
pre-round-12 tree). Final full check of the exact stop-time state launched at 02:01
→ /tmp/check-full-4 (gate 1 timing + gate 2, ~40 min).

FINAL VERIFICATION. Full check of the exact stop-time state — tree at ac76a28,
/submission/switches.json (seven switches) — PASS, speedup 1.4357
(/tmp/check-full-4, 02:01–02:22): gate 1 windows [15.5894, 16.1182, 16.0141] →
ratios 1.4578/1.4596/1.4357 (min); gate 2 bitwise identical (14 optimizer steps,
param/state hashes and all reward requests match bench/records/ref.json).
Combined with check-full-3 (PASS 1.3854) and check-q5 (quick PASS on the same tree),
the submission is verified end-to-end at both ends of the window-3 variance band
(±3%); the gate threshold is 1.0304.
8.2 hRound 14 — persistent text-graph pool vs window-3 drift (2026-11-07 02:24–02:36: ERROR, switch stays off)
Target: w3 is the deciding (slowest) window in every timing of the candidate
([15.56, 16.02, 15.97] r14; [15.59, 16.11, 16.60] full-3; [15.59, 16.12, 16.01] full-4;
per-step times rise ~2–4% over steps 10–15 for BOTH step types) while the reference shows
no such rise ([22.73, 23.53, 22.99] — w2 highest). Hypothesis: the per-call text-graph
CUDA-graph pool (created/destroyed ~8× per optimizer step) churns the caching allocator,
increasingly so in later steps.

Change (switch OPT_TEXTGEN_GRAPH_POOL, off by default, N0 by argument — a pool only
decides where graph-private allocations live): point generate_text_batch's pool holder
at a module-level list so all calls share one pool (the documented torch pattern; graphs
still captured/destroyed per call; ~200–500 MB permanently reserved, no step-0 risk since
graphs engage from step 1).

Time (/tmp/time-r18): ERROR on every rank — `RuntimeError: Offset increment
outside graph capture encountered unexpectedly` (torch graph-RNG bookkeeping). Sharing a
pool across calls is incompatible with this rollout's sampling pattern: the generator
advances eagerly (multinomial between graph replays), and re-entering a pool with a new
capture after eager generator advances trips torch's graph-safe-RNG assertion. The run
never reached step 16; the bench released the lock normally. Switch stays OFF
(committed off-by-default); the w3 drift remains unexplained — most plausibly thermal
(the candidate's per-step rise is smooth and the reference's own bootstrap per-step data
would settle it, but no lever exists either way). Also closed by code audit this round:
the denoise host overhead IS the per-layer aten dispatch (~3000 layer-forwards/window ×
~25–35 dispatches each) — only CUDA graphs remove it, and their instantiate economics
are settled negative; the per-timestep invariant embedding computes in _forward_flow
(embed_tokens/latent_pos_embed/text placement) are worth only ~40–70 ms/window — below
measurement resolution, not attempted.

Round 14 close-out — the window-3 "drift" is data-point structure, not our allocator.
The reference's own bootstrap per-step times (runs/bootstrap/full/timing_ref_a/b.r*.json)
show the SAME late rise on the same steps (ref_a gen steps 10→14: 30.95 → 31.96 → 32.29 s;
candidate: 21.2 → 21.5 → 21.7) and large prompt-dependent per-step variation (ref_b refine
step 5: 17.58 s vs step 7: 14.09 s). Each optimizer step processes a different data point
whose context length and reward latency differ; the frozen recipe gives both trees the same
prompts in the same order, but reward-latency effects do not cancel in the ratio (the
reference's waits are exposed, ours are hidden). The candidate's w3 ratio is lower than w1
mostly because the reference's w3 is its fast window (22.99 vs 23.53 in w2). Within a run,
all four ranks' per-step times agree to ~10 ms (collective lockstep), so the slowest-rank
measurement is tight. No lever exists on our side; the window-3 question is closed.
8.6 hRound 15 — third timing sample of the submission state; log coherence pass (2026-11-07 02:47–03:02: no change, state confirmed)
No lever remains with positive expected value (rounds 11–14 closed each candidate with
evidence), so this round used idle GPU time for an additional independent timing sample
of the EXACT submission state (tree f1f8bc4 + the seven switches): /tmp/time-r19 —
windows [15.5068, 16.0102, 16.3602] s/step → ratios 1.4656/1.4695/1.4053, **min 1.4053,
mean 1.4463**, peak 131.09 GiB, run clean. The submission's four timing samples now band
as: w1 15.51–15.59 (very stable), w2 15.94–16.12, w3 15.97–16.60 (the prompt/reward-
structure window); min-ratio 1.385–1.440, mean 1.434–1.456 — every sample ~30× above the
gate-1 threshold (1.0304), and the two full checks of this config PASS at 1.4357 and
1.3854. Also fixed four stale "in progress" round titles (4–6, 7, 8, 9) so the log's
headers state each round's verified conclusion. No code change; nothing to commit.
8.6 hRound 16 — paired (live-reference) timing; final direction sweep (2026-11-07 03:05–, in progress)
Swept the remaining idea space before spending GPU time; each closed by argument:
- AdamW(capturable=True): moves step to CUDA and computes bias correction in the
  kernel — different scalar rounding than the host-float64 path (same class as fused,
  which measured 1-ULP diffs) — not N0.
- NCCL algorithm/transport tuning for the stats collectives: the ~0.4 s/window is
  wait-for-slowest-rank (rollout-duration variance from SDE randomness), not transport;
  and the skew is semantics-locked.
- Reward-worker GPU contention during the wait overlap: inherent — the service processes
  requests on arrival; the overlap is still a large net win vs serial waiting.
- Overlapping the optimizer step with the next rollout: already async (the rollout's
  Python proceeds while optimizer kernels drain; the post-hook sync is the gate's own).
- The _forward_flow per-timestep invariant computes (embed_tokens/latent_pos_embed/text
  placement): ~4–6 ms/window total — below resolution, not attempted.

Measurement: time --paired — the one timing mode never run — times the LIVE
reference immediately before the candidate, the closest proxy to the final evaluation's
gate-1 (fresh reference runs). /tmp/time-r20, launched 03:05, ~25 min.

Paired timing result (/tmp/time-r20, reference at 03:05 → candidate, done 03:33):
live reference windows [23.0216, 23.8625, 23.3631] (1.0–1.5% above the frozen
[22.7266, 23.5265, 22.9922] — machine state stable), candidate windows
[15.5836, 15.9047, 16.0041] (best w2/w3 of any run), **per-window paired ratios
1.4773 / 1.5003 / 1.4598 — min 1.4598**, speedup_vs_paired_reference 1.4598,
peak 131.09 GiB, both runs clean. This is the closest proxy to the final
evaluation's gate 1 (fresh reference → candidate → fresh reference, min(refA,refB)
per window — with two reference runs the expected ratio is at or slightly above
this single-reference figure). Combined with the two full-check PASSes (1.4357,
1.3854) and four frozen-reference samples (min-ratio band 1.385–1.440), the
submission's expected final speedup is ~1.39–1.47 — roughly 30–45× the gate-1
threshold (1.0304). Round 16 conclusion: no change; measurement-only round, and
the last untried timing mode is now used. Tree f1f8bc4, switches.json unchanged.
9.0 hRound 17 — reward-path audit (2026-11-07 03:36–03:45: closed by measurement, no change)
Last unexplored corner: the JPEG-encode + reward-POST path (exposed pre-wait work:
4 serial PIL encodes per step before the batched POST; request order and JPEG bytes are
gate-checked, but PARALLEL encodes would be bitwise-safe — identical bytes, list order
preserved, one request per step). Measured directly on this host (192 cores, fast SIMD
PIL): 1.5 ms/encode at 512², 6.1 ms at 1024² → **6–24 ms/step = 12–48 ms/window — three
orders below the window**; the tensor→uint8→numpy conversion and pickle add ~10 ms.
Also re-checked and rejected by argument: moving the pack before the POST (would expose
it — the wait clock is fixed by the service), OMP/thread pinning (CPU-op reduction order
is not N0-safe), capturable AdamW (kernel-side bias-correction rounding). The reward
path is closed. With every direction now closed by measurement or argument, the round
spends otherwise-idle GPU on one more full check of the exact final state
(/tmp/check-full-5) — it PASSed at 1.4350 (window ratios 1.4632/1.4504/1.4350):
three full-check PASSes on this config now, 1.3854 / 1.4357 / 1.4350.

Third full check (/tmp/check-full-5, gate 1 03:27–03:37, gate 2 03:37–03:48):
PASS, speedup 1.4350 — window ratios 1.4632 / 1.4504 / 1.4350, gate 2 bitwise
identical, peak normal, run clean (the round-17 entry's "launched 03:45" was a
transcription slip; the bench log shows lock at 03:27:08). The exact final state
now has three full-check PASSes: 1.3854, 1.4357, 1.4350 — a tight band —
plus the paired live-reference measurement 1.4598 and four frozen-reference
timing samples (min-ratio 1.385–1.440). No processes remain; GPUs free.
9.8 hRound 18 — second paired sample; last candidates re-closed (2026-11-07 03:52–04:18: no change, band tightened)
Re-closed the remaining "maybe" candidates before spending GPU time: the round-8
OPT_SYNC_FREE_DENOISE hoist (with the shallow dispatch queue its ~80–150 blocking reads
are worth ~2–8 ms/window — consistent with its measured no-gain), switch combinations
(independent zero-gain mechanisms sum to zero), and cross-step prepack overlap (the host
is a shared serial resource: moving host work into the GPU-saturated train phase merely
relocates it; only the reward wait is GPU-idle, already exploited by round 10).

Second paired run (/tmp/time-r21, reference 03:52 → candidate, done 04:17): live
reference [22.7864, 23.5677, 23.0798] (≈0.2% from frozen — machine stable), candidate
[15.5435, 16.0539, 15.9285], paired ratios 1.466 / 1.468 / 1.449 — min 1.449.
The paired band over two samples is 1.449–1.460, the tightest and most final-like
estimate: the final evaluation's min(refA,refB)/cand with two reference runs should land
at or above ~1.45. Total evidence: 3 full-check PASSes (1.4350/1.4357/1.3854), 2 paired
samples (1.4598/1.449), 4 frozen samples (min 1.385–1.440), quick-check PASS on the
current tree. No change; nothing to commit; no processes remain.
9.9 hRound 19 — stack the individually-unresolvable small levers (2026-11-07 04:20–, in progress)
The two committed-but-off switches with positive-but-unresolvable individual signals —
OPT_TOKENLOOP_BATCHED (round 12: W1 −0.03/W2 −0.08 s; mechanism ~60–120 ms/window) and
OPT_SYNC_FREE_DENOISE (round 8: W1 −0.37/W2 −0.08 s before a noisy W3; mechanism ~10–20
ms/window of shallow-queue reads + ~37 assert kernels) — touch disjoint code paths (think
loop vs denoise reads). Their STACK (~0.2–0.3 s/window expected) may clear the seven-
switch baseline band (mean 1.434–1.456, min 1.385–1.440 over 4 samples) where each alone
cannot. Config /tmp/switches-r15.json (nine switches), time run → /tmp/time-r22,
launched 04:20. Decision rule: mean ≥ 1.47 or min ≥ 1.46 → outside the band → quick-check
and promote; inside the band → both stay off (no evidence of gain).
↑ R3