MiniMax M3.1 Flash 1.62×
Harness: Codex · switches: BAGEL_FAST_ATTN=1 BAGEL_FAST_DECODE=1 BAGEL_FAST_MERGE=1 BAGEL_FAST_TIMESTEP=1 BAGEL_CUDA_GRAPH_DECODE=1 BAGEL_FAST_CFG=1 BAGEL_FAST_OVERLAP=1 BAGEL_FAST_OVERLAP_POLICY=1 BAGEL_FAST_ROPE=1 BAGEL_FAST_PACK=1 BAGEL_FAST_MOE=1 BAGEL_FAST_NORM=1 BAGEL_FAST_MOE_NORM=1
- Judged
- 1.62×
- Tier
- N0
- Self-best
- 1.61×
- Rounds
- 15
- Hours
- 10.0
- API cost
- $4.84
- Input tokens · 97.2% cached
- 203 M
- Output tokens
- 1.7 M
- Requests
- 1,585
§ 02Key levers
- Text-decode CUDA graph
- ✓
BAGEL_CUDA_GRAPH_DECODE, single-run timing 1.10× → 1.37× (first backward OOM; switched to manual capture, released at end of rollout) - Reward wait hidden behind forward
- ✓
BAGEL_FAST_OVERLAP+_POLICY, full check 1.44× - Rollout-side bitwise fused kernels
- ✓ Triton RoPE, RMSNorm (1.50× → 1.57×), generation branch computed on all rows then text rows overwritten (→ 1.61×, all single-run timing)
- Train-side bitwise kernels / freeing memory
- ✗ (peak 130.9 GiB)
- Other main levers
- timestep, KV merge, packing desync; desync of per-layer length reads in denoising measured slower, turned off
§ ARaw trace
Raw work log22 sections
0.6 hRound 1 — profile the unchanged reference (2026-11-07 03:44 → 04:05)
Command:bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-ref --from 11 --iters 2(out:/tmp/prof-ref/profile.json, trace/tmp/prof-ref/timing_profile.trace.json, 3.6 GB). ### Stage split (from the run's own log,/tmp/prof-ref/timing_profile.log) | step kind | wall | rollout | training | |---|---|---|---| | gen (even) | ~14 s | ~11.5 s | ~2.5 s | | edit (odd) | ~30 s | ~21 s | ~9 s | A window is one gen + one edit step -> (14+30)/2 = 22 s, matching the frozen reference (22.5-23.1 s). So the rollout is 80 % of the step; the training part is 10-30 %. ### Hotspots 1. The text CoT decode loop dominates the rollout.tqdm"Thinking..." runs at **22.3 tok/s = 45 ms per decoded token** (196 tokens in the gen round, 384 in the edit round). A 7 B MoT decoding 4 tokens needs ~4-5 ms of GPU time (weights only), so the loop is ~10x off the memory-bound floor. 2. Cause: host synchronisations.aten::item= 244 157 calls andMemcpy DtoH= 262 991 in the two profiled steps. Per decoded token: *_static_kv_cache_attention(modeling/bagel/qwen2_navit.py:380,390): 3 device syncs **per layer** -> 28 layers x 3 = 84 syncs/token ((effective_seqlens + query_lens).max().item(),(query_lens == query_lens[0]).all(),query_lens[0].item()); *Bagel.generate_text_batch(modeling/bagel/bagel.py:1375-1385): 12.item()per token. Every sync drains the GPU queue, so the CPU can never run ahead: the GPU idles while the CPU relaunches, and the CPU idles while the GPU runs. 3. Profile table (/tmp/prof-ref/timing_profile.profile.txt): Self CPU 42.2 s vs Self CUDA 27.5 s for 2 steps -> CPU/launch bound, with 4.75 s ofCommand Buffer Full. Kernel time: GEMMs 10.3 s (41 %), flash-attn 1.2 s, elementwise/index/copy ~5 s. 4. Diffusion (_forward_flow, 16 gen / 21 edit steps) calls the *readonly* static-KV attention, which has 7 more.item()per layer per forward (588 syncs per denoising step). 5. The training part (aten::mm72 479 calls,record_param_comms1.55 s NCCL) is ~20 % of the step. ### Plan Round 2: make the decode path sync-free (host query metadata computed once per forward + one host transfer per decoded token). Round 3: the diffusion (readonly attention) syncs. Both are N0: no arithmetic, no RNG, no shape change.
0.9 hRound 2 — sync-free text decode (2026-11-07 04:05 → 04:35)
Hotspot class: host synchronisations in the rollout (profile: 244 157aten::itemin 2 steps; 84 device syncs per decoded token inside the attention + 12.item()per token in the sampling loop). ### Changes (both behind their own switch, off by default) *modeling/bagel/qwen2_navit.py—BAGEL_FAST_ATTN=1*_static_kv_cache_attention: the static-KV-cache overflow check and the packed-query metadata (seqlen_q, "are all query lengths equal") no longer read the device **per decoder layer; they are read once per forward** inQwen2Model.forward_inference(_check_static_kv_capacity,_host_query_meta) and passed down ashost_query_meta. 84 syncs/token -> 1. *modeling/bagel/bagel.py—BAGEL_FAST_DECODE=1*Bagel.generate_text_batch: the sampling bookkeeping is done on the host from **one device->host transfer per decoded token** (active_curr_tokens.tolist()andnext_tokens_probs.tolist()) instead of 12.item()calls; the "is this sequence finished" test usesactive_curr_tokens != end_token_id(identical tocurr_tokens[obi] == end_token_idbecause the active indices are unique) and the next token is written back with one index_put. The sampling itself (_sample_next_token), the RNG and every value are untouched. Equivalence of the new bookkeeping was checked against the original.item()version with 2000 randomised cases (/tmp/test_decode_equiv.py): identical generated tokens, identical probabilities, identicalcurr_tokens, identical active mask. ### Results *check --quick(gate 0 + gate 2, bitwise): PASS —/tmp/check-quick-1/check.json(14 optimizer steps, 3200 tensors, 235 reward requests, all bitwise identical tobench/records/ref.json). *time: running, output/tmp/time-1.
0.9 hRound 2 result — `time` (2026-11-07 04:38 → 04:51)
time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-1| window | reference (frozen) | candidate | ratio | |---|---|---|---| | 10-11 | 22.4628 | 21.7238 | 1.034 | | 12-13 | 23.0278 | 22.0794 | 1.043 | | 14-15 | 23.1123 | 22.3903 | 1.032 |speedup_vs_frozen_reference = 1.0364(mean-based),s_per_step22.06 vs 22.87. Peak memory 130.72 GiB (unchanged). Why so little? The Round-1 profile was *profiled*: the profiler inflates the host cost of every op, and the host synchronisations it exposed were therefore much more expensive under the profiler than in the real run. In the real (unprofiled) reference the decode already ran at 37 tok/s, not 22 tok/s: the sync removal bought only ~1 s per rollout.
1.5 hRound 3 — instrumented phase breakdown (2026-11-07 05:01 → 05:14)
Profiling the unchanged reference was misleading, so the tree was copied to/tmp/r3-diagand instrumented withtorch.cuda.synchronize()-bracketed phase timers (train/_diag.py->diag.pyin the scratch copy only; the submitted tree was not touched). The instrumented run has the same speed as the plain one (speedup_vs_frozen_reference1.0394 vs 1.0364), so the breakdown is trustworthy. Per step (rank 0), seconds: | phase | gen step (even) | edit step (odd) | |---|---|---| | rollout total | 10.26 | 20.63 | | - diffusion (dit_forward) | 4.86 (15 timesteps) | 9.45 (20 timesteps) | | - text decode | 4.59 | 9.17 | | - image/text context (ViT+VAE) | 0.1 | 0.96 | | reward wait | 1.66 | 2.00 | | training forward+backward | 2.39 | 7.67 | | optimizer | 0.14 | 0.15 | Per window (one gen + one edit step) the 43 s are therefore: **diffusion 14.35 s (33 %) | text decode 13.76 s (32 %) | training 10.06 s (23 %) | reward 3.66 s (8 %) | context 0.96 s | optimizer 0.29 s.**
1.5 hHotspots (Round 3)
1. The diffusion is 33 % of the step. One_forward_flow= 33 (gen) / 44 (edit) forward passes of the 14.6 B model; 147 ms per forward (gen) and 215 ms (edit), of which only ~40 ms is GPU work. The attention of the diffusion path merged the whole KV prefix per decoder layer and per forward, and every boolean-mask index in that merge runsnonzero, which synchronises the host with the GPU: 8 synchronisations per layer per forward (plus 6.item()for the sizes) = ~400 synchronisations per forward. The profile counted 19 314aten::nonzeroand 244 157aten::itemin two steps. 2. The text decode is 32 % and is much closer to the memory-bandwidth floor (14 GB of weights per token at ~23.7 ms/token); the decode's own host cost is ~3 ms/token (measured: removing 84 syncs/token moved it from 27 to 24 ms/token, i.e. ~36 us per synchronisation). 3. The whole run is host-bound: the four training processes sit at 100 % of one core each while the GPU is at 44 % SM utilisation. So every removed host operation and every removed synchronisation is worth roughly its host time. 4.aten::mul226 514 calls,aten::copy_192 258,aten::_index_put_impl_52 005,aten::nonzero19 314 per two steps -> ~375 000 dispatcher calls per step at ~59 us each.
1.5 hChange (Round 3)
*modeling/bagel/qwen2_navit.py—BAGEL_FAST_ATTN=1*StaticKVCachekeeps a host mirror ofseq_lens(set_seq_lens,advance_seq_lens(..., indices_host=...)). *Qwen2Model.forward_inferencereads the query metadata and the integer sizes the attention needs once per forward and hands them to every layer (HostQueryMeta,_host_static_kv_meta): 6.item()per layer -> 0. *modeling/bagel/qwen2_navit.py—BAGEL_FAST_MERGE=1*_merge_static_kv_prefix_and_queryand the non-uniform write-back address the cached rows and the new rows by index arithmetic (repeat_interleavewith an explicitoutput_size) instead of boolean masks: 8nonzerosynchronisations per layer per forward -> 0. *modeling/bagel/bagel.py,train/rollout_controller.py: the cache lengths are set throughset_seq_lens, and the decode advances the mirror with the host-side active indices. Both rewrites were checked against the original code on randomised cases (/tmp/test_kv_meta.py,/tmp/test_merge_equiv.py): identical values.
1.5 hRound 3 result — `time` + `check --quick` (2026-11-07 05:15 → 05:45)
*time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-2->speedup_vs_frozen_reference = 1.0739,s_per_step21.29 (frozen reference 22.87). Per window: 10-11 = 20.79 s, 12-13 = 21.57 s, 14-15 = 21.63 s (frozen 22.46 / 23.03 / 23.11). *check --quick(gate 0 + gate 2, bitwise): PASS —/tmp/check-quick-2/check.json(14 optimizer steps, 3200 tensors, 235 reward requests, all bitwise identical tobench/records/ref.json).
2.4 hRound 4 — profile the *current* tree (2026-11-07 05:43 → 06:02)
Command:bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-cur --from 11 --iters 2(out:/tmp/prof-cur/timing_profile.profile.txt, trace/tmp/prof-cur/timing_profile.trace.json, 3.2 GB). The trace covers exactly one step (step 12 = a *gen* step: decode -> diffusion -> reward -> training). ### Profile table (2 steps of the reference run before, 1 step here; numbers are the profiler's)Self CPU time total: 37.0 s Self CUDA time total: 27.1 s-> the run is host bound ("Command Buffer Full" = 5.59 s of the CPU waiting for the GPU queue). | op | self CUDA | calls | note | |---|---|---|---| |aten::mm| 11.50 s (42 %) | 72 479 | the GEMMs (decode 113 k + diffusion 43 k + training) | |aten::mul| 2.78 s (10 %) | 228 894 | elementwise | |aten::_index_put_impl_| 1.71 s | 52 445 | 34.8 us each: the MoT "gen"-mode index_puts | |aten::copy_| 1.35 s | 196 946 | dtype conversions | | FlashAttnVarlenFunc | 0.94 s | 2 482 | | |aten::cat| 0.39 s | 31 323 | | |aten::_local_scalar_dense| 0.41 s | 184 201 | 184 k device->host synchronisations! |cudaLaunchKernel: 1 199 048 launches (8.6 s of host time);cudaStreamSynchronize: 186 337. ### The remaining synchronisations The trace shows 184 201aten::itemcalls, each with its ownMemcpy DtoH, i.e. a Python loop reading single scalars from the device. The sweep over the trace shows: * they are *top level* events (no enclosing torch op) and there is noaten::selectbefore them, so they arex.item()on an already 0-dim tensor (notx[i].item()); * they come in bursts of ~8 200/s during the diffusion phase (11.5-24 s of the trace), at ~25/s during the decode and not at all during the training phase. A micro-benchmark (profiledtolist()vst[i].item()) rules outtolist()(it does one D2H and noaten::item) and shows that a Python loop ofx.item()produces exactly this signature (aten::item+aten::_local_scalar_dense+ oneMemcpy DtoHeach). => Round 5: find the loop with an instrumented probe (/tmp/r3-probe2, a copy of the tree withtorch.Tensor.item/__float__/__int__/__bool__counted by caller) and remove it.
3.0 hRound 5 — the per-token timestep vector (2026-11-07 06:17 → 06:41)
### Profile finding (where the 184 k device syncs came from) The profile of the current tree (/tmp/prof-cur) showedaten::_local_scalar_dense184 201 calls in two steps, each with its ownMemcpy DtoH— i.e. ~92 k host synchronisations per RL step (3.9 s of CPU time, 10 % of the 37 s of CPU). An instrumented probe (a scratch copy of the tree,/tmp/r3-probe2, withtorch.Tensor.item/__float__/__int__/__bool__counted *by caller*) found the source: ``` 942 086 float() <- modeling/bagel/bagel.py:729/822/926 8 719 item() <- torch/optim/optimizer.py:97 (AdamW'sstep, unavoidable) ~1 everything else ```modeling/bagel/bagel.pybuilds the per-token timestep of every denoising step with ```python timestep = torch.tensor([t] * x_t.shape[0], device=x_t.device) ``` wheretis a 0-dim CUDA tensor.torch.tensoron a list of tensors converts every element to a Python float, so this is one device synchronisation per token (1026 tokens per image per denoising step, 15–20 denoising steps per round, two rounds per step => ~150 k synchronisations per RL step, plus ~0.5 M in the step-0 evaluation, which uses the non-SDEgenerate_image). ### Change (behindBAGEL_FAST_TIMESTEP=1)modeling/bagel/bagel.py: new helper_timestep_vector(t, n); the three denoising loops (generate_image,generate_image_sde,generate_image_mix) call it. With the switch on it returnst.expand(n).contiguous()— the same float32 values (a float32 -> double -> float32 round trip is exact), the same dtype and the same contiguous layout, with no host transfer. Without the switch the original expression is used, so the switch is a no-op for the reference. Checked on the GPU: values, dtype, shape and the resulting timestep embedding (TimestepEmbedder) are bitwise identical for n = 1, 2, 3, 1024, 4098, andunique()still sees a single value (theassertin_forward_flow). ### Result *time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-3->speedup_vs_frozen_reference = 1.1048,s_per_step20.70 (frozen reference 22.87). Per window: 10-11 = 20.17 s/step, 12-13 = 20.97, 14-15 = 20.95 (frozen 22.46 / 23.03 / 23.11). * Committed asca849c7(tier N0). ### What the step times look like (from the run's own log)step timeper optimiser step (gen = even, edit = odd): | step | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | s | 66.2* | 54.1 | 14.1 | 30.4 | 14.1 | 29.7 | 12.3 | 26.6 | 12.2 | 28.1 | 12.5 | 27.8 | 13.2 | 28.7 | 13.0 | (*includes the step-0 evaluation.) The edit round is 2.2x the gen round and dominates the average: the edit denoises with 21 timesteps and three CFG forwards per step (base + cfg_text + cfg_img, becausecfg_img_scale=1.5), the gen round with 16 timesteps and two forwards (cfg_img_scale=1). ---
3.9 hRound 6 — CUDA graph for the rollout text decode
### Profile finding The profile of the round-5 tree (/tmp/prof-cur, torch.profiler over the optimiser steps 11-12) shows the run is host bound: ``` Self CPU time total: 37.007s Self CUDA time total: 27.074s ``` i.e. the host spends 37 s of CPU on work whose GPU part only takes 27 s. The phase timers of the scratch tree/tmp/r3-diag2put the text decode at 34 % of the step (25-29 ms per decoded token) while the GPU work of one decoding step is ~3 ms (14 GB of weights read at 4.8 TB/s) — the rest is per-token host work: ~700 kernel launches plus the matching dispatcher/Python overhead. ### Change (behindBAGEL_CUDA_GRAPH_DECODE=1)modeling/bagel/bagel.py: the rollout text decode ofgenerate_text_batchis replayed from a CUDA graph: * one graph per shape (num_active, the set of still-active cache rows); only the token ids and the position ids change from step to step, and both are written into static input buffers before the replay; * the KV-cache lengths are read from the device *inside* the graph (flash_attn_with_kvcachetakescache_seqlensas a device tensor), so the replay attends over exactly the same keys as the eager step; * the graph is captured by hand (CUDAGraph.capture_begin/capture_end) instead of with thetorch.cuda.graphcontext manager, because that manager empties the caching allocator before every capture, which changed the block layout the (memory hungry) training phase sees — with it the run died with an OOM in the training backward; * every graph gets its own memory pool and the graphs are deleted at the end of the rollout, so the pool is back in the allocator before the training phase; *modeling/bagel/qwen2_navit.py:Qwen2ForCausalLM.forward_inferenceandQwen2Model.forward_inferencetake the host mirror ofselected_cache_indices(selected_cache_indices_host) so that the capture does not have to read the indices back from the device. Anything unexpected during the capture (an unsupported op, a synchronisation, ...) falls back to the eager path, which computes exactly the same values. ### Validation On a scratch tree with a small randomly initialised BAGEL (/tmp/test_graph_rng.py, 4 layers, 4 sequences): * the generated token ids and the returned probabilities are bitwise identical to the eager path (same sha256); * the CUDA generator's offset advances by exactly the same amount (68) with and without the graph — the decode forward itself consumes no RNG, onlytorch.multinomialdoes, and that stays outside the graph; * the captured step is 3.7x faster than the eager step on that small model. ### Result (firsttimerun with the graph:/tmp/time-5— ERROR, out of memory in the training backward; the decode itself was already 3x faster, 42 -> 115 tokens/s. After switching to the manual capture and releasing the graphs at the end of the rollout: see below.) After the manual capture and the explicit release of the graphs: *time --tree /submission/r3 --config /tmp/switches-debug.json --out /tmp/time-6->speedup_vs_frozen_reference = 1.372,s_per_step16.67 (frozen reference 22.87; previous best 1.108 / 20.64). Step times (gen = even, edit = odd): 13.0 / 24.4 / 11.4 / ... / 10.2 / 22.0 / 10.4 / 23.1 / 10.5 s — the edit round went from 28-30 s to 22-25 s. Peak memory 131.6 GiB allocated (the reference peaks at 130.7), so the graph does not cost memory. * `check --quick --tree /submission/r3 --config /submission/switches.json --out /tmp/check-quick-3` -> PASS, tier N0: 14 optimizer steps, 3200 parameter/state tensors and 235 reward requests identical to the frozen reference record. * Committed as146d0bf.
5.5 hRound 7 — the denoising attention's per-layer device reads (2026-11-07 08:29 → 08:57)
### Profile findingbench.py profileof the current tree (/tmp/prof-2, steps 11-12) plus an analysis of the 2.2 GB chrome trace (a streaming parser over the events): * Self CPU 30.98 s vs Self CUDA 26.99 s for the two profiled steps -> the GPU is busy ~72 % of the wall time; the rest is the reward wait, the host-side gaps and the collectives. *aten::item40 213 calls (1.63 s CPU, 432 ms of GPU idle inside them) andaten::is_nonzero153 calls (899 ms) -> the diffusion attention readsmax(query_lens)/max(key_values_lens)from the device **once per decoder layer and per forward**: 28 layers x 2 x (16 or 21) denoising steps x 3 forwards ~ 2700 synchronisations per rollout. * The largest pure-Python gaps in the trace (no GPU activity): 1.64 s + 0.39 s before theall_gatherof the rollout buffer -> the reward service, and 1.44 s before the firstaten::indexof the training phase. ### Change (behindBAGEL_FAST_DIFF=1, committed as0cfac39)modeling/bagel/qwen2_navit.py+modeling/bagel/bagel.py: the packed lengths of a denoising forward are read once per rollout (_host_packed_meta) and handed down through_forward_flow->Qwen2ForCausalLM->Qwen2Model->Qwen2MoTDecoderLayer->PackedAttentionMoT; the merged-buffer size and the two flash-attention maxima are then exact host integers. Theassert timestep.unique().shape[0] == 1(a device read) is skipped, because the denoising loops build the vector from a single scalar. The attention was verified bitwise identical with and without the metadata for 1, 2 and 5 sequences, with and without a cached prefix, for both cache classes (/tmp/test_diff_meta.py). ### Result — no gaintime --tree /submission/r3 --config /submission/switches.json --out /tmp/time-7->speedup_vs_frozen_reference = 1.3414,s_per_step17.05 (the previous best,/tmp/time-6, was 1.372 / 16.67). All three windows got slightly *worse* (16.37 / 17.49 / 17.25 vs 16.12 / 16.74 / 17.13), which is inside the reference spread (4.7 %) but not an improvement. The extra.tolist()of the packed lengths at the start of every rollout drains the GPU queue once per rollout, which apparently costs about as much as the ~2700 removed synchronisations save. Decision: the switch stays in the tree but is off in/submission/switches.json, so the submitted tree keeps the behaviour of/tmp/time-6(1.372).
5.5 hRound 8 — where the step time actually goes (2026-11-07 08:57 → 09:12)
### Findings (a diagnostic tree with fine-grained phase timers,/tmp/r3-diag4) | phase | gen step | edit step | |---|---|---| | rollout | 5.9-6.3 s | 12.5-13.2 s | | reward wait (.result()) | 1.2-2.8 s | 1.3-3.1 s | | advantage computation | 0.00 s | 0.00-2.58 s | | rollout-bufferall_gather| 0.02-1.94 s | 0.00 s | | pack (the packing) | 0.16 s | 0.43-0.45 s | | policy forward | 0.68 s | 2.93 s | | policy backward | 1.18 s | 4.20 s | | optimizer | 0.16-0.18 s | 0.13-0.14 s | Theadvantageandall_gatherphases are where the ranks wait for each other: they vary between 0.00 s and 2.58 s from rank to rank, and the spread follows the spread of the reward wait. So the step time isrollout + reward service + training, and the *reward service* is on the critical path of the slowest rank -- the training of a step cannot start before its rewards are there. ### Change — overlap the reward service with the batch preparation The reward requests are submitted at the end of the rollout, so the service is busy for ~2.6 s while the CPU waits for the responses. The work that does not need the rewards is moved in front of that wait: *train/rollout_controller.py:pack_batch/pack_iteratorcan now *defer* the advantages -- the batch is packed with placeholders and the slots are remembered (fill_deferred_advantagesfills them with the very same values once the rewards are there). *train/online_rl.py: the packing and the reference-model forward (which only depends on the packed batch, not on the advantages) now run *before* the reward wait; the training loop uses the batch and the reference log-probs that were prepared there. The random draws keep their order (the reference model'srandn_likeis still the first draw after the rollout, the policy model's the second), the collectives keep their order on both process groups, and the advantages that end up in the batch are the same numbers as before.
6.1 hRound 9 — overlap the reward service with the batch preparation (2026-11-07 09:20 → 10:05)
5.5 hWhere the time goes (from the profile of round 8, `/tmp/prof-2`)
The GPU is idle ~23 % of the wall time (8.0 s of gaps in the 34.4 s trace): | gap | count | total | |---|---|---| | < 1 ms (kernel launch bubbles) | 561 688 | 2.25 s | | 2-5 ms | 381 | 1.00 s | | 50-200 ms | 11 | 1.25 s (the reward waits) | | > 200 ms | 3 | 2.31 s (the reward waits + the packing) | The reward service is on the critical path: its latency (2.5-3 s for the images of an edit step) is waited for *after* the rollout, and the GPU has nothing to do meanwhile. The work that does not need the rewards (packing the rollout, the reference model's forward pass) was moved in front of that wait. ### Change *train/rollout_controller.py:pack_batch/pack_iteratorcan *defer* the advantages -- the batch is packed with placeholders and the slots are remembered (fill_deferred_advantagesputs the very same values into the packed tensors once the rewards have arrived). *train/online_rl.py: the packing and the reference model's forward pass (which only depends on the packed batch, not on the advantages) now run *before* the reward wait. * The random draws keep their order (the reference model'srandn_likeis still the first draw after the rollout, the policy model's the second), the collectives keep their order on both process groups, and the advantages that end up in the batch are the same numbers as before. ### Memory The peak memory of the run is 131.7 GiB of a 139.8 GiB card, so the previous attempt (which only reordered the code) died with a CUDA OOM in the backward pass (it needed 740 MiB and had 730 MiB). Two changes make room: * the logits are released as soon as they are used (ref_loss_dict["logits"],output_dict["logits"]): they are a large tensor (1400 tokens x 152 064 vocab) and the backward pass does not need them (the softmax keeps its own output). Peak: 131.72 -> 130.78 GiB. * the advantages are filled into the packed tensors in place. ### Result *check --quick --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/check-quick-ovl2-> PASS, tier N0 (14 optimizer steps, 3200 tensors, 235 reward requests identical to the frozen reference; peak memory 130.78 GiB).
6.1 hResult (round 9)
*time --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/time-9-> speedup_vs_frozen_reference 1.4182 (16.12 s/step; per window [15.573, 16.264, 16.537] vs the frozen reference [22.463, 23.028, 23.112]). The previous best (the committed CUDA-graph decode tree) was 1.372 (16.67 s/step). *check --quick --tree /tmp/r3-ovl --config /tmp/sw-ovl.json --out /tmp/check-quick-ovl2-> PASS, tier N0 (14 optimizer steps, 3200 tensors, 235 reward requests identical to the frozen reference; peak memory 130.78 GiB). This snapshot did not contain the CFG-interval change ofmodeling/bagel/bagel.py(the tree digest of the check isedbff586..., the one of the timed tree is7b2a4c77...), so the combination was re-validated afterwards. ### Committed *69a65bbN0 -- the CFG interval is tested once per diffusion call (BAGEL_FAST_CFG=1). *ef5615eN0 -- the reward service overlaps the batch preparation (BAGEL_FAST_OVERLAP=1).
6.5 hRound 10 -- move the *policy's* forward pass in front of the reward wait (2026-11-07 10:05 -> 10:35)
### Where the time goes The phase boundaries are visible in the tree's own log of the timed run (/tmp/time-9/timing_tree.log): the lineRef per token logps shapeis printed after the reference model's forward pass (which now runs before the reward wait) and the lineText advantagesafter the wait for the reward service: ``` step | prepack->ref fwd | reward wait | advantages->image loss 10 | 0.0 | 2.0 | 0.0 11 | 1.0 | 2.0 | 1.0 12 | 0.0 | 2.0 | 1.0 13 | 2.0 | 1.0 | 2.0 14 | 1.0 | 2.0 | 0.0 15 | 2.0 | 1.0 | 2.0 ``` The reward wait is 1-4 s per step (1.7 s on average) and the GPU is idle during it. Only the reference model's forward pass (~1 s) is hidden behind it so far. The *policy's* forward pass does not depend on the rewards either -- only the losses do -- so it can be hidden as well. ### Changetrain/online_rl.py(BAGEL_FAST_OVERLAP_POLICY=1, requiresBAGEL_FAST_OVERLAP=1): the policy's forward pass runs right after the reference model's forward pass, i.e. before the wait for the reward service. The advantages are filled into the packed batch when the rewards arrive and the losses are computed from the forward's output afterwards, so the values, the random draws (the forward has none) and the order of the collectives are the same as before. ### Status *check --tree /submission/r3 --config /submission/switches.json --out /tmp/check-4(running).
6.5 hResult (round 10)
*check --tree /submission/r3 --config /submission/switches.json --out /tmp/check-4-> PASS, tier N0, speedup 1.4373. (14 optimizer steps, 3200 tensors and 235 reward requests identical to the frozen reference; per window [15.544, 15.984, 16.077] s/step against the reference [22.463, 23.028, 23.112] -> 1.4451 / 1.4407 / 1.4376.) * committed as4f97b7e. The reward wait did not disappear, it grew: the reward service shares the GPUs with the training process, so the GPU work that is moved in front of the wait makes the service slower (the wait went from 1.7 s to 2.4 s per step while ~1 s of forward pass was hidden behind it). The net effect is still positive (16.26 -> 15.98 s/step on the second window).
6.8 hRound 11 -- a fused rotary position embedding for the denoising forward (2026-11-07 10:30 -> 11:00)
### Where the time goes The operator table of the profile (/tmp/prof-2/timing_profile.profile.txt, two steps) shows the diffusion's memory-bound work next to its GEMMs: | operator | CUDA time (2 steps) | share | |---|---|---| |aten::mm(the GEMMs,nvjet_*) | 10.99 s | 36.5 % | |aten::mul| 2.52 s | 8.8 % | |record_param_comms(FSDP) | 1.81 s | 6.7 % | |aten::_index_put_impl_| 1.83 s | 6.4 % | |aten::copy_| 1.15 s | 4.1 % | |flash_fwd_kernel| 0.91 s | 3.4 % | Themuls are mostly the rotary position embedding: in the generation branch offorward_inferencethe query and key states are float32 and the cos/sin tables are bfloat16, so the reference runs eight elementwise kernels over the [tokens, heads, head_dim] tensors (rotate_halfis a negation plus a concatenation, i.e. a full copy, the two products are two more full-size temporaries and the sum is a fifth one). That is ~2.7 GB of memory traffic per decoder layer and per denoising step, and the denoising loop runs 15 (or 20) steps with up to three forwards each. ### Changemodeling/bagel/qwen2_navit.py(BAGEL_FAST_ROPE=1): a Triton kernel that computes the rotary embedding of the generation branch in one pass -- it reads the float32 query/key states and the bfloat16 cos/sin tables and writes the bfloat16 result. The arithmetic is float32 and the result is rounded to bfloat16 once, exactly like the eager path; the sum is emitted as an explicitadd.rn.f32so that the compiler cannot contract the two products into an fma (which would round only once and give different values). Every other case (theundbranch of the text decode, the training forward) keeps the eager path, so the backward pass is untouched. ### Validation of the kernel (standalone, on the GPU) * against the eager path on 67 M random elements (float32 queries, bfloat16 cos/sin): bitwise identical, and also for tiny (1e-30), huge (1e30), denormal, zero, ±1 and inf/NaN inputs.
7.2 hRound 12 -- the packing of the timesteps (2026-11-07 11:00 -> 11:20)
### Where the time goes The trace of the two profiled steps (/tmp/prof-2/timing_profile.trace.json) shows 40841aten::itemcalls (1.5 s of CPU time) and 153aten::is_nonzerocalls (0.9 s). 36708 of the items are in four bursts of a few thousand items each -- exactly4 latents x 4096 tokens: the packing of the rollout puts the 0-dim *device* tensors of the timestep and of the dt into a python list, once per latent token, and building the tensor out of that list reads every element back from the device. Every one of those reads is a device->host synchronisation in the middle of the step. ### Changetrain/rollout_controller.py(BAGEL_FAST_PACK=1): read the value of the timestep and of the dt once per latent and repeat it. The numbers are the same ones (a float32 -> double -> float32 round trip is exact). ### Result *time --tree /tmp/r3-combo --config /tmp/sw-combo.json --out /tmp/time-5-> speedup_vs_frozen_reference 1.4717 (windows [15.278, 15.687, 15.651] s/step against the frozen reference [22.463, 23.028, 23.112]). *check --quick --tree /submission/r3 --config /submission/switches.json --out /tmp/check-quick-5-> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward requests identical to the frozen reference). * committed as5eef3c0(the rotary kernel) andecbfe77(the packing).
7.8 hRound 13 -- the gen branch of the diffusion attention (2026-11-07 11:00 -> 11:40)
### Where the time goes The profile of the two profiled steps (/tmp/prof-3) showsaten::_index_put_impl_at 1.89 s of self CUDA time (6.8 %) over the two steps, of whichindex_elementwise_kernelalone is 1.44 s (41039 launches), plusaten::index(0.53 s,vectorized_gather_kernel0.45 s). Those are the gathers and the scatters of thegenbranch ofPackedAttentionMoT: for every decoder layer and every denoising step the vae rows are gathered out of the packed sequence, projected, normalised, scattered back and gathered again. ### Changemodeling/bagel/qwen2_navit.py(BAGEL_FAST_MOE=1): in thegenbranch the projection (q/k/v_proj_moe_gen) and the normalisation (q/k_norm_moe_gen) are applied to every row of the packed sequence; the text rows are then overwritten with the values of theundbranch (q/k/v_proj,q/k_norm). The reference instead gathers the vae rows, projects them and scatters them back. A projection only depends on its own row and the normalisation is per head, so the values are the very same ones; the rows that are overwritten are the very same text rows. The eager path is kept for the layouts the shortcut does not cover (it is guarded by a check on the row counts that reads no device memory). ### Validation * Standalone (/tmp/moetest/test_moe.py): the projections, the q/k normalisations ando_projare bitwise equal for the layouts N = 4104 (8 text + 4096 vae), N = 2052, N = 1026 and for 12 random interleavings of the text and vae rows (maxdiff = 0.0). *check --quick --tree /tmp/r3-moe --config /tmp/sw-moe.json --out /tmp/check-quick-moe-> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward requests identical to the frozen reference). ### Result *time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-moe-> speedup_vs_frozen_reference 1.4987 (15.258 s/step against the frozen reference's 22.868 s/step), up from 1.4717.
8.3 hRound 14 -- the fused RMSNorm of the inference path (2026-11-07 11:30 -> 12:05)
### Where the time goes The profile of the two profiled steps (/tmp/prof-3) shows that the normalisation is the largest remaining non-GEMM cost:aten::mul2.03 s of self CUDA time (7.9 %),aten::pow0.35 s,aten::mean0.37 s, the reciprocal square root 0.10 s and a large part of the 1.07 s ofaten::copy_come from the RMSNorm. The eagerQwen2RMSNormwalks over the activation tensor six times (to(float32),pow(2),mean(-1),rsqrt, the scaling multiply and the weight multiply). There are about 12000 normalisations per step, almost all of them on the 4096 vae tokens of the diffusion rollout. ### Changemodeling/qwen2/fast_rmsnorm.py(new) andmodeling/qwen2/modeling_qwen2.py(BAGEL_FAST_NORM=1): the normalisation of the inference path (the rollout and the frozen reference model, i.e. everything that runs without grad) is done by two Triton kernels: one that computes the sum of the squares of every row and one that applies the weight and the scaling. The kernel reproduces the reduction order ofaten::meanexactly, which is the only way to keep the result bitwise identical: *aten::meanreduces the last dimension of a contiguous float32 tensor with one warp per row (blockDim.x == 32) and combines the 32 lanes with theshfl_downtree over the offsets 1, 2, 4, 8, 16, * for a row of 128 elements lanetowns the elementst, t+32, t+64, t+96, for a row of 3584 elements the loads are vectorised and lanetowns the elements4*(t+32*m)+j, * the squares are rounded to float32 *before* they are added and the adds are emitted asadd.rn.f32/mul.rn.f32inline assembly, so that the compiler cannot contract them into an fma (which would round only once), * the 32 lane partials are combined with the same butterfly (reshape/split), the mean's factor (rows / (rows * width)in float32) and the reciprocal square root are still computed byaten. The eager path is kept for everything that needs a backward pass and for the layouts the kernel does not cover. ### Validation * standalone (/tmp/normtest/test_fused.py): bitwise identical to the eager path for[4096, 3584],[4104, 3584],[200, 3584],[16, 3584],[4096, 28, 128],[4104, 28, 128],[200, 28, 128]in bfloat16 and in float32 (0 of 14.7 M elements differ). *check --quick --tree /tmp/r3-norm --config /tmp/sw-norm.json --out /tmp/check-quick-norm-> PASS, tier N0 (14 optimizer steps, 3200 tensors and 235 reward requests identical to the frozen reference). * the norm itself is 3.1x faster than the eager path. ### Result *time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-norm-> speedup_vs_frozen_reference 1.5740 (14.528 s/step against the frozen reference's 22.868 s/step), up from 1.4987.
9.3 hRound 15 -- the norms and the mlp of the gen branch over every row (2026-11-07 12:33 -> 13:05)
### Where the time goes The profile of the two profiled steps (/tmp/prof-4) shows the scatters of the decoder layer among the largest non-gemm costs:aten::_index_put_impl_1.148 s of self CUDA time (38333 calls),aten::index0.328 s,aten::index_elementwise_kernel1.048 s (31631 calls) andaten::fill_0.169 s (thezeros_likeof the reference). ### Changemodeling/bagel/qwen2_navit.py(BAGEL_FAST_MOE_NORM=1): in thegenbranch ofQwen2MoTDecoderLayer.forward_inferencethe reference gathers the vae rows out of the packed sequence, normalises them and scatters them back -- once forinput_layernorm, once forpost_attention_layernormand the mlp. A normalisation and an mlp only depend on their own row, so the gen branch now applies them to *every* row of the packed sequence and overwrites the text rows with the values of the und branch. The gather, thezeros_likeand the twoindex_putper layer and per denoising step are gone. ### Validation * the gemms were verified to be bitwise identical for the two row counts (M=16384vsM=16388andM=4096vsM=4104forK,N in {(3584,3584),(3584,512),(512,3584),(3584,18944),(18944,3584)}), * the fused norm was verified to be bitwise identical to the eager path for the row counts of both variants ([16384, 3584]vs[16388, 3584],[458752, 128]vs[458864, 128], ...), *check --quick --tree /tmp/r3-moenorm --config /tmp/sw-moenorm.json --out /tmp/check-quick-moenorm-> PASS, tier N0. ### Result *time --tree /submission/r3 --config /submission/switches.json --out /tmp/time-moenorm-> speedup_vs_frozen_reference 1.606 (14.239 s/step against the frozen reference's 22.868 s/step), up from 1.5740.