MiniMax M3.1 Flash 1.62×

Harness: Codex · switches: BAGEL_FAST_ATTN=1 BAGEL_FAST_DECODE=1 BAGEL_FAST_MERGE=1 BAGEL_FAST_TIMESTEP=1 BAGEL_CUDA_GRAPH_DECODE=1 BAGEL_FAST_CFG=1 BAGEL_FAST_OVERLAP=1 BAGEL_FAST_OVERLAP_POLICY=1 BAGEL_FAST_ROPE=1 BAGEL_FAST_PACK=1 BAGEL_FAST_MOE=1 BAGEL_FAST_NORM=1 BAGEL_FAST_MOE_NORM=1

Judged
1.62×
Tier
N0
Self-best
1.61×
Rounds
15
Hours
10.0
API cost
$4.84
Input tokens · 97.2% cached
203 M
Output tokens
1.7 M
Requests
1,585
Text-decode CUDA graph
✓ BAGEL_CUDA_GRAPH_DECODE, single-run timing 1.10× → 1.37× (first backward OOM; switched to manual capture, released at end of rollout)
Reward wait hidden behind forward
✓ BAGEL_FAST_OVERLAP + _POLICY, full check 1.44×
Rollout-side bitwise fused kernels
✓ Triton RoPE, RMSNorm (1.50× → 1.57×), generation branch computed on all rows then text rows overwritten (→ 1.61×, all single-run timing)
Train-side bitwise kernels / freeing memory
✗ (peak 130.9 GiB)
Other main levers
timestep, KV merge, packing desync; desync of per-layer length reads in denoising measured slower, turned off
Raw work log22 sections
0.6 hRound 1 — profile the unchanged reference (2026-11-07 03:44 → 04:05)
Command: bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-ref --from 11 --iters 2
(out: /tmp/prof-ref/profile.json, trace /tmp/prof-ref/timing_profile.trace.json, 3.6 GB).

### Stage split (from the run's own log, /tmp/prof-ref/timing_profile.log)

| step kind | wall | rollout | training |
|---|---|---|---|
| gen (even) | ~14 s | ~11.5 s | ~2.5 s |
| edit (odd) | ~30 s | ~21 s | ~9 s |

A window is one gen + one edit step -> (14+30)/2 = 22 s, matching the frozen reference (22.5-23.1 s).
So the rollout is 80 % of the step; the training part is 10-30 %.

### Hotspots

1. The text CoT decode loop dominates the rollout. tqdm "Thinking..." runs at **22.3 tok/s
   = 45 ms per decoded token** (196 tokens in the gen round, 384 in the edit round). A 7 B MoT
   decoding 4 tokens needs ~4-5 ms of GPU time (weights only), so the loop is ~10x off the
   memory-bound floor.
2. Cause: host synchronisations. aten::item = 244 157 calls and
   Memcpy DtoH = 262 991 in the two profiled steps. Per decoded token:
   * _static_kv_cache_attention (modeling/bagel/qwen2_navit.py:380,390): 3 device syncs **per
     layer** -> 28 layers x 3 = 84 syncs/token
     ((effective_seqlens + query_lens).max().item(), (query_lens == query_lens[0]).all(),
     query_lens[0].item());
   * Bagel.generate_text_batch (modeling/bagel/bagel.py:1375-1385): 12 .item() per token.
   Every sync drains the GPU queue, so the CPU can never run ahead: the GPU idles while the CPU
   relaunches, and the CPU idles while the GPU runs.
3. Profile table (/tmp/prof-ref/timing_profile.profile.txt): Self CPU 42.2 s vs Self CUDA 27.5 s
   for 2 steps -> CPU/launch bound, with 4.75 s of Command Buffer Full.
   Kernel time: GEMMs 10.3 s (41 %), flash-attn 1.2 s, elementwise/index/copy ~5 s.
4. Diffusion (_forward_flow, 16 gen / 21 edit steps) calls the *readonly* static-KV attention,
   which has 7 more .item() per layer per forward (588 syncs per denoising step).
5. The training part (aten::mm 72 479 calls, record_param_comms 1.55 s NCCL) is ~20 % of the step.

### Plan

Round 2: make the decode path sync-free (host query metadata computed once per forward +
one host transfer per decoded token). Round 3: the diffusion (readonly attention) syncs.
Both are N0: no arithmetic, no RNG, no shape change.
0.9 hRound 2 — sync-free text decode (2026-11-07 04:05 → 04:35)
Hotspot class: host synchronisations in the rollout (profile: 244 157 aten::item in 2 steps;
84 device syncs per decoded token inside the attention + 12 .item() per token in the sampling loop).

### Changes (both behind their own switch, off by default)

* modeling/bagel/qwen2_navit.py — BAGEL_FAST_ATTN=1
  * _static_kv_cache_attention: the static-KV-cache overflow check and the packed-query
    metadata (seqlen_q, "are all query lengths equal") no longer read the device **per decoder
    layer; they are read once per forward** in Qwen2Model.forward_inference
    (_check_static_kv_capacity, _host_query_meta) and passed down as host_query_meta.
    84 syncs/token -> 1.
* modeling/bagel/bagel.py — BAGEL_FAST_DECODE=1
  * Bagel.generate_text_batch: the sampling bookkeeping is done on the host from **one
    device->host transfer per decoded token** (active_curr_tokens.tolist() and
    next_tokens_probs.tolist()) instead of 12 .item() calls; the "is this sequence finished"
    test uses active_curr_tokens != end_token_id (identical to curr_tokens[obi] == end_token_id
    because the active indices are unique) and the next token is written back with one index_put.
    The sampling itself (_sample_next_token), the RNG and every value are untouched.

Equivalence of the new bookkeeping was checked against the original .item() version with
2000 randomised cases (/tmp/test_decode_equiv.py): identical generated tokens, identical
probabilities, identical curr_tokens, identical active mask.

### Results

* check --quick (gate 0 + gate 2, bitwise): PASS — /tmp/check-quick-1/check.json
  (14 optimizer steps, 3200 tensors, 235 reward requests, all bitwise identical to
  bench/records/ref.json).
* time: running, output /tmp/time-1.
0.9 hRound 2 result — `time` (2026-11-07 04:38 → 04:51)
time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-1

| window | reference (frozen) | candidate | ratio |
|---|---|---|---|
| 10-11 | 22.4628 | 21.7238 | 1.034 |
| 12-13 | 23.0278 | 22.0794 | 1.043 |
| 14-15 | 23.1123 | 22.3903 | 1.032 |

speedup_vs_frozen_reference = 1.0364 (mean-based), s_per_step 22.06 vs 22.87.
Peak memory 130.72 GiB (unchanged).

Why so little?  The Round-1 profile was *profiled*: the profiler inflates the host
cost of every op, and the host synchronisations it exposed were therefore much more
expensive under the profiler than in the real run.  In the real (unprofiled) reference
the decode already ran at 37 tok/s, not 22 tok/s: the sync removal bought only
~1 s per rollout.
1.5 hRound 3 — instrumented phase breakdown (2026-11-07 05:01 → 05:14)
Profiling the unchanged reference was misleading, so the tree was copied to
/tmp/r3-diag and instrumented with torch.cuda.synchronize()-bracketed phase timers
(train/_diag.py -> diag.py in the scratch copy only; the submitted tree was not
touched).  The instrumented run has the same speed as the plain one
(speedup_vs_frozen_reference 1.0394 vs 1.0364), so the breakdown is trustworthy.

Per step (rank 0), seconds:

| phase | gen step (even) | edit step (odd) |
|---|---|---|
| rollout total | 10.26 | 20.63 |
| - diffusion (dit_forward) | 4.86 (15 timesteps) | 9.45 (20 timesteps) |
| - text decode | 4.59 | 9.17 |
| - image/text context (ViT+VAE) | 0.1 | 0.96 |
| reward wait | 1.66 | 2.00 |
| training forward+backward | 2.39 | 7.67 |
| optimizer | 0.14 | 0.15 |

Per window (one gen + one edit step) the 43 s are therefore:
**diffusion 14.35 s (33 %) | text decode 13.76 s (32 %) | training 10.06 s (23 %) |
reward 3.66 s (8 %) | context 0.96 s | optimizer 0.29 s.**
1.5 hHotspots (Round 3)
1. The diffusion is 33 % of the step.  One _forward_flow = 33 (gen) / 44 (edit)
   forward passes of the 14.6 B model; 147 ms per forward (gen) and 215 ms (edit),
   of which only ~40 ms is GPU work.  The attention of the diffusion path merged the
   whole KV prefix per decoder layer and per forward, and every boolean-mask index
   in that merge runs nonzero, which synchronises the host with the GPU:
   8 synchronisations per layer per forward (plus 6 .item() for the sizes) =
   ~400 synchronisations per forward.  The profile counted 19 314 aten::nonzero
   and 244 157 aten::item in two steps.
2. The text decode is 32 % and is much closer to the memory-bandwidth floor
   (14 GB of weights per token at ~23.7 ms/token); the decode's own host cost is
   ~3 ms/token (measured: removing 84 syncs/token moved it from 27 to 24 ms/token,
   i.e. ~36 us per synchronisation).
3. The whole run is host-bound: the four training processes sit at 100 % of one
   core each while the GPU is at 44 % SM utilisation.  So every removed host
   operation and every removed synchronisation is worth roughly its host time.
4. aten::mul 226 514 calls, aten::copy_ 192 258, aten::_index_put_impl_ 52 005,
   aten::nonzero 19 314 per two steps -> ~375 000 dispatcher calls per step at
   ~59 us each.
1.5 hChange (Round 3)
* modeling/bagel/qwen2_navit.py — BAGEL_FAST_ATTN=1
  * StaticKVCache keeps a host mirror of seq_lens (set_seq_lens,
    advance_seq_lens(..., indices_host=...)).
  * Qwen2Model.forward_inference reads the query metadata and the integer sizes the
    attention needs once per forward and hands them to every layer
    (HostQueryMeta, _host_static_kv_meta): 6 .item() per layer -> 0.
* modeling/bagel/qwen2_navit.py — BAGEL_FAST_MERGE=1
  * _merge_static_kv_prefix_and_query and the non-uniform write-back address the
    cached rows and the new rows by index arithmetic (repeat_interleave with an
    explicit output_size) instead of boolean masks: 8 nonzero synchronisations per
    layer per forward -> 0.
* modeling/bagel/bagel.py, train/rollout_controller.py: the cache lengths are set
  through set_seq_lens, and the decode advances the mirror with the host-side
  active indices.

Both rewrites were checked against the original code on randomised cases
(/tmp/test_kv_meta.py, /tmp/test_merge_equiv.py): identical values.
1.5 hRound 3 result — `time` + `check --quick` (2026-11-07 05:15 → 05:45)
* time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-2
  -> speedup_vs_frozen_reference = 1.0739, s_per_step 21.29 (frozen reference 22.87).
  Per window: 10-11 = 20.79 s, 12-13 = 21.57 s, 14-15 = 21.63 s (frozen 22.46 / 23.03 / 23.11).
* check --quick (gate 0 + gate 2, bitwise): PASS — /tmp/check-quick-2/check.json
  (14 optimizer steps, 3200 tensors, 235 reward requests, all bitwise identical to
  bench/records/ref.json).
2.4 hRound 4 — profile the *current* tree (2026-11-07 05:43 → 06:02)
Command: bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-cur --from 11 --iters 2
(out: /tmp/prof-cur/timing_profile.profile.txt, trace /tmp/prof-cur/timing_profile.trace.json, 3.2 GB).
The trace covers exactly one step (step 12 = a *gen* step: decode -> diffusion -> reward -> training).

### Profile table (2 steps of the reference run before, 1 step here; numbers are the profiler's)

Self CPU time total: 37.0 s   Self CUDA time total: 27.1 s  -> the run is host bound
("Command Buffer Full" = 5.59 s of the CPU waiting for the GPU queue).

| op | self CUDA | calls | note |
|---|---|---|---|
| aten::mm | 11.50 s (42 %) | 72 479 | the GEMMs (decode 113 k + diffusion 43 k + training) |
| aten::mul | 2.78 s (10 %) | 228 894 | elementwise |
| aten::_index_put_impl_ | 1.71 s | 52 445 | 34.8 us each: the MoT "gen"-mode index_puts |
| aten::copy_ | 1.35 s | 196 946 | dtype conversions |
| FlashAttnVarlenFunc | 0.94 s | 2 482 | |
| aten::cat | 0.39 s | 31 323 | |
| aten::_local_scalar_dense | 0.41 s | 184 201 | 184 k device->host synchronisations! |

cudaLaunchKernel: 1 199 048 launches (8.6 s of host time); cudaStreamSynchronize: 186 337.

### The remaining synchronisations

The trace shows 184 201 aten::item calls, each with its own Memcpy DtoH, i.e. a
Python loop reading single scalars from the device.  The sweep over the trace shows:

* they are *top level* events (no enclosing torch op) and there is no aten::select
  before them, so they are x.item() on an already 0-dim tensor (not x[i].item());
* they come in bursts of ~8 200/s during the diffusion phase (11.5-24 s of the trace),
  at ~25/s during the decode and not at all during the training phase.

A micro-benchmark (profiled tolist() vs t[i].item()) rules out tolist() (it does one
D2H and no aten::item) and shows that a Python loop of x.item() produces exactly this
signature (aten::item + aten::_local_scalar_dense + one Memcpy DtoH each).

=> Round 5: find the loop with an instrumented probe (/tmp/r3-probe2, a copy of the tree
with torch.Tensor.item/__float__/__int__/__bool__ counted by caller) and remove it.
3.0 hRound 5 — the per-token timestep vector (2026-11-07 06:17 → 06:41)
### Profile finding (where the 184 k device syncs came from)

The profile of the current tree (/tmp/prof-cur) showed aten::_local_scalar_dense
184 201 calls in two steps, each with its own Memcpy DtoH — i.e. ~92 k host
synchronisations per RL step (3.9 s of CPU time, 10 % of the 37 s of CPU).

An instrumented probe (a scratch copy of the tree, /tmp/r3-probe2, with
torch.Tensor.item/__float__/__int__/__bool__ counted *by caller*) found the source:

```
942 086  float()   <- modeling/bagel/bagel.py:729/822/926
   8 719  item()    <- torch/optim/optimizer.py:97  (AdamW's step, unavoidable)
      ~1  everything else
```

modeling/bagel/bagel.py builds the per-token timestep of every denoising step with

```python
timestep = torch.tensor([t] * x_t.shape[0], device=x_t.device)
```

where t is a 0-dim CUDA tensor.  torch.tensor on a list of tensors converts
every element to a Python float, so this is one device synchronisation per token
(1026 tokens per image per denoising step, 15–20 denoising steps per round, two
rounds per step => ~150 k synchronisations per RL step, plus ~0.5 M in the step-0
evaluation, which uses the non-SDE generate_image).

### Change (behind BAGEL_FAST_TIMESTEP=1)

modeling/bagel/bagel.py: new helper _timestep_vector(t, n); the three denoising
loops (generate_image, generate_image_sde, generate_image_mix) call it.  With the
switch on it returns t.expand(n).contiguous() — the same float32 values (a
float32 -> double -> float32 round trip is exact), the same dtype and the same
contiguous layout, with no host transfer.  Without the switch the original
expression is used, so the switch is a no-op for the reference.

Checked on the GPU: values, dtype, shape and the resulting timestep embedding
(TimestepEmbedder) are bitwise identical for n = 1, 2, 3, 1024, 4098, and
unique() still sees a single value (the assert in _forward_flow).

### Result

* time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-3
  -> speedup_vs_frozen_reference = 1.1048, s_per_step 20.70 (frozen reference 22.87).
  Per window: 10-11 = 20.17 s/step, 12-13 = 20.97, 14-15 = 20.95
  (frozen 22.46 / 23.03 / 23.11).
* Committed as ca849c7 (tier N0).

### What the step times look like (from the run's own log)

step time per optimiser step (gen = even, edit = odd):

| step | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| s | 66.2* | 54.1 | 14.1 | 30.4 | 14.1 | 29.7 | 12.3 | 26.6 | 12.2 | 28.1 | 12.5 | 27.8 | 13.2 | 28.7 | 13.0 |

(*includes the step-0 evaluation.)

The edit round is 2.2x the gen round and dominates the average: the edit denoises
with 21 timesteps and three CFG forwards per step (base + cfg_text + cfg_img,
because cfg_img_scale=1.5), the gen round with 16 timesteps and two forwards
(cfg_img_scale=1).

---
3.9 hRound 6 — CUDA graph for the rollout text decode
### Profile finding

The profile of the round-5 tree (/tmp/prof-cur, torch.profiler over the optimiser
steps 11-12) shows the run is host bound:

```
Self CPU time total: 37.007s
Self CUDA time total: 27.074s
```

i.e. the host spends 37 s of CPU on work whose GPU part only takes 27 s.  The
phase timers of the scratch tree /tmp/r3-diag2 put the text decode at 34 % of
the step (25-29 ms per decoded token) while the GPU work of one decoding step is
~3 ms (14 GB of weights read at 4.8 TB/s) — the rest is per-token host work:
~700 kernel launches plus the matching dispatcher/Python overhead.

### Change (behind BAGEL_CUDA_GRAPH_DECODE=1)

modeling/bagel/bagel.py: the rollout text decode of generate_text_batch is
replayed from a CUDA graph:

* one graph per shape (num_active, the set of still-active cache rows); only the
  token ids and the position ids change from step to step, and both are written
  into static input buffers before the replay;
* the KV-cache lengths are read from the device *inside* the graph
  (flash_attn_with_kvcache takes cache_seqlens as a device tensor), so the
  replay attends over exactly the same keys as the eager step;
* the graph is captured by hand (CUDAGraph.capture_begin/capture_end) instead of
  with the torch.cuda.graph context manager, because that manager empties the
  caching allocator before every capture, which changed the block layout the
  (memory hungry) training phase sees — with it the run died with an OOM in the
  training backward;
* every graph gets its own memory pool and the graphs are deleted at the end of
  the rollout, so the pool is back in the allocator before the training phase;
* modeling/bagel/qwen2_navit.py: Qwen2ForCausalLM.forward_inference and
  Qwen2Model.forward_inference take the host mirror of selected_cache_indices
  (selected_cache_indices_host) so that the capture does not have to read the
  indices back from the device.

Anything unexpected during the capture (an unsupported op, a synchronisation, ...)
falls back to the eager path, which computes exactly the same values.

### Validation

On a scratch tree with a small randomly initialised BAGEL (/tmp/test_graph_rng.py,
4 layers, 4 sequences):

* the generated token ids and the returned probabilities are bitwise identical
  to the eager path (same sha256);
* the CUDA generator's offset advances by exactly the same amount (68) with and
  without the graph — the decode forward itself consumes no RNG, only
  torch.multinomial does, and that stays outside the graph;
* the captured step is 3.7x faster than the eager step on that small model.

### Result

(first time run with the graph: /tmp/time-5 — ERROR, out of memory in the
training backward; the decode itself was already 3x faster, 42 -> 115 tokens/s.
After switching to the manual capture and releasing the graphs at the end of the
rollout: see below.)

After the manual capture and the explicit release of the graphs:

* time --tree /submission/r3 --config /tmp/switches-debug.json --out /tmp/time-6
  -> speedup_vs_frozen_reference = 1.372, s_per_step 16.67
  (frozen reference 22.87; previous best 1.108 / 20.64).
  Step times (gen = even, edit = odd): 13.0 / 24.4 / 11.4 / ... / 10.2 / 22.0 /
  10.4 / 23.1 / 10.5 s — the edit round went from 28-30 s to 22-25 s.
  Peak memory 131.6 GiB allocated (the reference peaks at 130.7), so the graph
  does not cost memory.
* `check --quick --tree /submission/r3 --config /submission/switches.json
  --out /tmp/check-quick-3` -> PASS, tier N0: 14 optimizer steps, 3200
  parameter/state tensors and 235 reward requests identical to the frozen
  reference record.
* Committed as 146d0bf.
5.5 hRound 7 — the denoising attention's per-layer device reads (2026-11-07 08:29 → 08:57)
### Profile finding

bench.py profile of the current tree (/tmp/prof-2, steps 11-12) plus an analysis of the
2.2 GB chrome trace (a streaming parser over the events):

* Self CPU 30.98 s vs Self CUDA 26.99 s for the two profiled steps -> the GPU is busy ~72 % of
  the wall time; the rest is the reward wait, the host-side gaps and the collectives.
* aten::item 40 213 calls (1.63 s CPU, 432 ms of GPU idle inside them) and
  aten::is_nonzero 153 calls (899 ms) -> the diffusion attention reads
  max(query_lens) / max(key_values_lens) from the device **once per decoder layer and per
  forward**: 28 layers x 2 x (16 or 21) denoising steps x 3 forwards ~ 2700 synchronisations
  per rollout.
* The largest pure-Python gaps in the trace (no GPU activity): 1.64 s + 0.39 s before the
  all_gather of the rollout buffer -> the reward service, and 1.44 s before the first
  aten::index of the training phase.

### Change (behind BAGEL_FAST_DIFF=1, committed as 0cfac39)

modeling/bagel/qwen2_navit.py + modeling/bagel/bagel.py: the packed lengths of a denoising
forward are read once per rollout (_host_packed_meta) and handed down through
_forward_flow -> Qwen2ForCausalLM -> Qwen2Model -> Qwen2MoTDecoderLayer ->
PackedAttentionMoT; the merged-buffer size and the two flash-attention maxima are then exact
host integers.  The assert timestep.unique().shape[0] == 1 (a device read) is skipped, because
the denoising loops build the vector from a single scalar.
The attention was verified bitwise identical with and without the metadata for 1, 2 and 5
sequences, with and without a cached prefix, for both cache classes (/tmp/test_diff_meta.py).

### Result — no gain

time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-7
-> speedup_vs_frozen_reference = 1.3414, s_per_step 17.05
(the previous best, /tmp/time-6, was 1.372 / 16.67).
All three windows got slightly *worse* (16.37 / 17.49 / 17.25 vs 16.12 / 16.74 / 17.13), which is
inside the reference spread (4.7 %) but not an improvement.  The extra .tolist() of the packed
lengths at the start of every rollout drains the GPU queue once per rollout, which apparently
costs about as much as the ~2700 removed synchronisations save.

Decision: the switch stays in the tree but is off in /submission/switches.json, so the
submitted tree keeps the behaviour of /tmp/time-6 (1.372).
5.5 hRound 8 — where the step time actually goes (2026-11-07 08:57 → 09:12)
### Findings (a diagnostic tree with fine-grained phase timers, /tmp/r3-diag4)

| phase | gen step | edit step |
|---|---|---|
| rollout | 5.9-6.3 s | 12.5-13.2 s |
| reward wait (.result()) | 1.2-2.8 s | 1.3-3.1 s |
| advantage computation | 0.00 s | 0.00-2.58 s |
| rollout-buffer all_gather | 0.02-1.94 s | 0.00 s |
| pack (the packing) | 0.16 s | 0.43-0.45 s |
| policy forward | 0.68 s | 2.93 s |
| policy backward | 1.18 s | 4.20 s |
| optimizer | 0.16-0.18 s | 0.13-0.14 s |

The advantage and all_gather phases are where the ranks wait for each other: they vary
between 0.00 s and 2.58 s from rank to rank, and the spread follows the spread of the reward wait.
So the step time is rollout + reward service + training, and the *reward service* is on the
critical path of the slowest rank -- the training of a step cannot start before its rewards are
there.

### Change — overlap the reward service with the batch preparation

The reward requests are submitted at the end of the rollout, so the service is busy for ~2.6 s
while the CPU waits for the responses.  The work that does not need the rewards is moved in
front of that wait:

* train/rollout_controller.py: pack_batch/pack_iterator can now *defer* the advantages --
  the batch is packed with placeholders and the slots are remembered
  (fill_deferred_advantages fills them with the very same values once the rewards are there).
* train/online_rl.py: the packing and the reference-model forward (which only depends on the
  packed batch, not on the advantages) now run *before* the reward wait; the training loop uses
  the batch and the reference log-probs that were prepared there.

The random draws keep their order (the reference model's randn_like is still the first draw
after the rollout, the policy model's the second), the collectives keep their order on both
process groups, and the advantages that end up in the batch are the same numbers as before.
6.1 hRound 9 — overlap the reward service with the batch preparation (2026-11-07 09:20 → 10:05)
5.5 hWhere the time goes (from the profile of round 8, `/tmp/prof-2`)
The GPU is idle ~23 % of the wall time (8.0 s of gaps in the 34.4 s trace):

| gap | count | total |
|---|---|---|
| < 1 ms (kernel launch bubbles) | 561 688 | 2.25 s |
| 2-5 ms | 381 | 1.00 s |
| 50-200 ms | 11 | 1.25 s (the reward waits) |
| > 200 ms | 3 | 2.31 s (the reward waits + the packing) |

The reward service is on the critical path: its latency (2.5-3 s for the images of an edit
step) is waited for *after* the rollout, and the GPU has nothing to do meanwhile.  The work that
does not need the rewards (packing the rollout, the reference model's forward pass) was moved in
front of that wait.

### Change

* train/rollout_controller.py: pack_batch/pack_iterator can *defer* the advantages -- the
  batch is packed with placeholders and the slots are remembered (fill_deferred_advantages puts
  the very same values into the packed tensors once the rewards have arrived).
* train/online_rl.py: the packing and the reference model's forward pass (which only depends
  on the packed batch, not on the advantages) now run *before* the reward wait.
* The random draws keep their order (the reference model's randn_like is still the first draw
  after the rollout, the policy model's the second), the collectives keep their order on both
  process groups, and the advantages that end up in the batch are the same numbers as before.

### Memory

The peak memory of the run is 131.7 GiB of a 139.8 GiB card, so the previous attempt (which only
reordered the code) died with a CUDA OOM in the backward pass (it needed 740 MiB and had
730 MiB).  Two changes make room:

* the logits are released as soon as they are used (ref_loss_dict["logits"],
  output_dict["logits"]): they are a large tensor (1400 tokens x 152 064 vocab) and the
  backward pass does not need them (the softmax keeps its own output).  Peak: 131.72 -> 130.78 GiB.
* the advantages are filled into the packed tensors in place.

### Result

* check --quick --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/check-quick-ovl2
  -> PASS, tier N0 (14 optimizer steps, 3200 tensors, 235 reward requests identical to the
  frozen reference; peak memory 130.78 GiB).
6.1 hResult (round 9)
* time --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/time-9
  -> speedup_vs_frozen_reference 1.4182 (16.12 s/step; per window
  [15.573, 16.264, 16.537] vs the frozen reference [22.463, 23.028, 23.112]).
  The previous best (the committed CUDA-graph decode tree) was 1.372 (16.67 s/step).
* check --quick --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/check-quick-ovl2
  -> PASS, tier N0 (14 optimizer steps, 3200 tensors, 235 reward requests
  identical to the frozen reference; peak memory 130.78 GiB).
  This snapshot did not contain the CFG-interval change of modeling/bagel/bagel.py
  (the tree digest of the check is edbff586..., the one of the timed tree is
  7b2a4c77...), so the combination was re-validated afterwards.

### Committed

* 69a65bb N0 -- the CFG interval is tested once per diffusion call
  (BAGEL_FAST_CFG=1).
* ef5615e N0 -- the reward service overlaps the batch preparation
  (BAGEL_FAST_OVERLAP=1).
6.5 hRound 10 -- move the *policy's* forward pass in front of the reward wait (2026-11-07 10:05 -> 10:35)
### Where the time goes

The phase boundaries are visible in the tree's own log of the timed run
(/tmp/time-9/timing_tree.log): the line Ref per token logps shape is printed
after the reference model's forward pass (which now runs before the reward wait)
and the line Text advantages after the wait for the reward service:

```
step | prepack->ref fwd | reward wait | advantages->image loss
  10 |       0.0        |     2.0     |        0.0
  11 |       1.0        |     2.0     |        1.0
  12 |       0.0        |     2.0     |        1.0
  13 |       2.0        |     1.0     |        2.0
  14 |       1.0        |     2.0     |        0.0
  15 |       2.0        |     1.0     |        2.0
```

The reward wait is 1-4 s per step (1.7 s on average) and the GPU is idle
during it.  Only the reference model's forward pass (~1 s) is hidden behind it so
far.  The *policy's* forward pass does not depend on the rewards either -- only
the losses do -- so it can be hidden as well.

### Change

train/online_rl.py (BAGEL_FAST_OVERLAP_POLICY=1, requires
BAGEL_FAST_OVERLAP=1): the policy's forward pass runs right after the
reference model's forward pass, i.e. before the wait for the reward service.
The advantages are filled into the packed batch when the rewards arrive and the
losses are computed from the forward's output afterwards, so the values, the
random draws (the forward has none) and the order of the collectives are the
same as before.

### Status

* check --tree /submission/r3 --config /submission/switches.json --out /tmp/check-4
  (running).
6.5 hResult (round 10)
* check --tree /submission/r3 --config /submission/switches.json --out /tmp/check-4
  -> PASS, tier N0, speedup 1.4373.
  (14 optimizer steps, 3200 tensors and 235 reward requests identical to the
  frozen reference; per window [15.544, 15.984, 16.077] s/step against the
  reference [22.463, 23.028, 23.112] -> 1.4451 / 1.4407 / 1.4376.)
* committed as 4f97b7e.

The reward wait did not disappear, it grew: the reward service shares the GPUs
with the training process, so the GPU work that is moved in front of the wait
makes the service slower (the wait went from 1.7 s to 2.4 s per step while
~1 s of forward pass was hidden behind it).  The net effect is still positive
(16.26 -> 15.98 s/step on the second window).
6.8 hRound 11 -- a fused rotary position embedding for the denoising forward (2026-11-07 10:30 -> 11:00)
### Where the time goes

The operator table of the profile (/tmp/prof-2/timing_profile.profile.txt, two
steps) shows the diffusion's memory-bound work next to its GEMMs:

| operator | CUDA time (2 steps) | share |
|---|---|---|
| aten::mm (the GEMMs, nvjet_*) | 10.99 s | 36.5 % |
| aten::mul | 2.52 s | 8.8 % |
| record_param_comms (FSDP) | 1.81 s | 6.7 % |
| aten::_index_put_impl_ | 1.83 s | 6.4 % |
| aten::copy_ | 1.15 s | 4.1 % |
| flash_fwd_kernel | 0.91 s | 3.4 % |

The muls are mostly the rotary position embedding: in the generation branch
of forward_inference the query and key states are float32 and the cos/sin
tables are bfloat16, so the reference runs eight elementwise kernels over
the [tokens, heads, head_dim] tensors (rotate_half is a negation plus a
concatenation, i.e. a full copy, the two products are two more full-size
temporaries and the sum is a fifth one).  That is ~2.7 GB of memory traffic per
decoder layer and per denoising step, and the denoising loop runs 15 (or 20)
steps with up to three forwards each.

### Change

modeling/bagel/qwen2_navit.py (BAGEL_FAST_ROPE=1): a Triton kernel that
computes the rotary embedding of the generation branch in one pass -- it reads
the float32 query/key states and the bfloat16 cos/sin tables and writes the
bfloat16 result.  The arithmetic is float32 and the result is rounded to
bfloat16 once, exactly like the eager path; the sum is emitted as an explicit
add.rn.f32 so that the compiler cannot contract the two products into an fma
(which would round only once and give different values).  Every other case
(the und branch of the text decode, the training forward) keeps the eager
path, so the backward pass is untouched.

### Validation of the kernel (standalone, on the GPU)

* against the eager path on 67 M random elements (float32 queries, bfloat16
  cos/sin): bitwise identical, and also for tiny (1e-30), huge (1e30),
  denormal, zero, ±1 and inf/NaN inputs.
7.2 hRound 12 -- the packing of the timesteps (2026-11-07 11:00 -> 11:20)
### Where the time goes

The trace of the two profiled steps (/tmp/prof-2/timing_profile.trace.json)
shows 40841 aten::item calls (1.5 s of CPU time) and 153
aten::is_nonzero calls (0.9 s).  36708 of the items are in four bursts of a
few thousand items each -- exactly 4 latents x 4096 tokens: the packing of
the rollout puts the 0-dim *device* tensors of the timestep and of the dt into
a python list, once per latent token, and building the tensor out of that list
reads every element back from the device.  Every one of those reads is a
device->host synchronisation in the middle of the step.

### Change

train/rollout_controller.py (BAGEL_FAST_PACK=1): read the value of the
timestep and of the dt once per latent and repeat it.  The numbers are the
same ones (a float32 -> double -> float32 round trip is exact).

### Result

* time --tree /tmp/r3-combo --config /tmp/sw-combo.json --out /tmp/time-5
  -> speedup_vs_frozen_reference 1.4717 (windows [15.278, 15.687, 15.651]
  s/step against the frozen reference [22.463, 23.028, 23.112]).
* check --quick --tree /submission/r3 --config /submission/switches.json --out /tmp/check-quick-5
  -> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward
  requests identical to the frozen reference).
* committed as 5eef3c0 (the rotary kernel) and ecbfe77 (the packing).
7.8 hRound 13 -- the gen branch of the diffusion attention (2026-11-07 11:00 -> 11:40)
### Where the time goes

The profile of the two profiled steps (/tmp/prof-3) shows
aten::_index_put_impl_ at 1.89 s of self CUDA time (6.8 %) over the two
steps, of which index_elementwise_kernel alone is 1.44 s (41039 launches),
plus aten::index (0.53 s, vectorized_gather_kernel 0.45 s).  Those are the
gathers and the scatters of the gen branch of PackedAttentionMoT: for
every decoder layer and every denoising step the vae rows are gathered out of
the packed sequence, projected, normalised, scattered back and gathered again.

### Change

modeling/bagel/qwen2_navit.py (BAGEL_FAST_MOE=1): in the gen branch the
projection (q/k/v_proj_moe_gen) and the normalisation (q/k_norm_moe_gen)
are applied to every row of the packed sequence; the text rows are then
overwritten with the values of the und branch (q/k/v_proj, q/k_norm).
The reference instead gathers the vae rows, projects them and scatters them
back.  A projection only depends on its own row and the normalisation is per
head, so the values are the very same ones; the rows that are overwritten are
the very same text rows.  The eager path is kept for the layouts the shortcut
does not cover (it is guarded by a check on the row counts that reads no
device memory).

### Validation

* Standalone (/tmp/moetest/test_moe.py): the projections, the q/k
  normalisations and o_proj are bitwise equal for the layouts
  N = 4104 (8 text + 4096 vae), N = 2052, N = 1026 and for 12 random
  interleavings of the text and vae rows (maxdiff = 0.0).
* check --quick --tree /tmp/r3-moe --config /tmp/sw-moe.json --out /tmp/check-quick-moe
  -> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward
  requests identical to the frozen reference).

### Result

* time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-moe
  -> speedup_vs_frozen_reference 1.4987 (15.258 s/step against the frozen
  reference's 22.868 s/step), up from 1.4717.
8.3 hRound 14 -- the fused RMSNorm of the inference path (2026-11-07 11:30 -> 12:05)
### Where the time goes

The profile of the two profiled steps (/tmp/prof-3) shows that the
normalisation is the largest remaining non-GEMM cost: aten::mul 2.03 s of
self CUDA time (7.9 %), aten::pow 0.35 s, aten::mean 0.37 s, the
reciprocal square root 0.10 s and a large part of the 1.07 s of aten::copy_
come from the RMSNorm.  The eager Qwen2RMSNorm walks over the activation
tensor six times (to(float32), pow(2), mean(-1), rsqrt, the scaling
multiply and the weight multiply).  There are about 12000 normalisations per
step, almost all of them on the 4096 vae tokens of the diffusion rollout.

### Change

modeling/qwen2/fast_rmsnorm.py (new) and modeling/qwen2/modeling_qwen2.py
(BAGEL_FAST_NORM=1): the normalisation of the inference path (the rollout
and the frozen reference model, i.e. everything that runs without grad) is
done by two Triton kernels: one that computes the sum of the squares of every
row and one that applies the weight and the scaling.

The kernel reproduces the reduction order of aten::mean exactly, which is
the only way to keep the result bitwise identical:

* aten::mean reduces the last dimension of a contiguous float32 tensor with
  one warp per row (blockDim.x == 32) and combines the 32 lanes with the
  shfl_down tree over the offsets 1, 2, 4, 8, 16,
* for a row of 128 elements lane t owns the elements t, t+32, t+64, t+96,
  for a row of 3584 elements the loads are vectorised and lane t owns the
  elements 4*(t+32*m)+j,
* the squares are rounded to float32 *before* they are added and the adds are
  emitted as add.rn.f32 / mul.rn.f32 inline assembly, so that the
  compiler cannot contract them into an fma (which would round only once),
* the 32 lane partials are combined with the same butterfly
  (reshape/split), the mean's factor (rows / (rows * width) in float32)
  and the reciprocal square root are still computed by aten.

The eager path is kept for everything that needs a backward pass and for the
layouts the kernel does not cover.

### Validation

* standalone (/tmp/normtest/test_fused.py): bitwise identical to the
  eager path for [4096, 3584], [4104, 3584], [200, 3584], [16, 3584],
  [4096, 28, 128], [4104, 28, 128], [200, 28, 128] in bfloat16 and in
  float32 (0 of 14.7 M elements differ).
* check --quick --tree /tmp/r3-norm --config /tmp/sw-norm.json --out /tmp/check-quick-norm
  -> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward
  requests identical to the frozen reference).
* the norm itself is 3.1x faster than the eager path.

### Result

* time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-norm
  -> speedup_vs_frozen_reference 1.5740 (14.528 s/step against the frozen
  reference's 22.868 s/step), up from 1.4987.
9.3 hRound 15 -- the norms and the mlp of the gen branch over every row (2026-11-07 12:33 -> 13:05)
### Where the time goes

The profile of the two profiled steps (/tmp/prof-4) shows the scatters of
the decoder layer among the largest non-gemm costs: aten::_index_put_impl_
1.148 s of self CUDA time (38333 calls), aten::index 0.328 s,
aten::index_elementwise_kernel 1.048 s (31631 calls) and
aten::fill_ 0.169 s (the zeros_like of the reference).

### Change

modeling/bagel/qwen2_navit.py (BAGEL_FAST_MOE_NORM=1): in the gen
branch of Qwen2MoTDecoderLayer.forward_inference the reference gathers the
vae rows out of the packed sequence, normalises them and scatters them back
-- once for input_layernorm, once for post_attention_layernorm and the
mlp.  A normalisation and an mlp only depend on their own row, so the gen
branch now applies them to *every* row of the packed sequence and overwrites
the text rows with the values of the und branch.

The gather, the zeros_like and the two index_put per layer and per
denoising step are gone.

### Validation

* the gemms were verified to be bitwise identical for the two row counts
  (M=16384 vs M=16388 and M=4096 vs M=4104 for
  K,N in {(3584,3584),(3584,512),(512,3584),(3584,18944),(18944,3584)}),
* the fused norm was verified to be bitwise identical to the eager path for
  the row counts of both variants ([16384, 3584] vs [16388, 3584],
  [458752, 128] vs [458864, 128], ...),
* check --quick --tree /tmp/r3-moenorm --config /tmp/sw-moenorm.json --out /tmp/check-quick-moenorm
  -> PASS, tier N0.

### Result

* time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-moenorm
  -> speedup_vs_frozen_reference 1.606 (14.239 s/step against the frozen
  reference's 22.868 s/step), up from 1.5740.
↑ R3