MiniMax M3.1 Flash 1.49×

Harness: Codex · switches: BAGEL_COMPILE_RMSNORM=1 BAGEL_COMPILE_ROPE=1 BAGEL_FSDP_LIMIT_ALL_GATHERS=0 BAGEL_FLASH_ATTN=1 BAGEL_NO_AC_FROM_LAYER=18 BAGEL_FUSED_ADAMW=1 BAGEL_MOT_FUSED=1 BAGEL_EMA_OVERLAP=1 BAGEL_VARLEN_ATTN=1

Judged
1.49×
Tier
N1
Self-best
1.48×
Rounds
22
Hours
10.0
API cost
$4.50
Input tokens · 97.2% cached
186 M
Output tokens
1.7 M
Requests
1,298
Attention kernel swap (split by mask structure)
✓ BAGEL_FLASH_ATTN: one FlashAttention call per segment, 1.09 → 1.29×; BAGEL_VARLEN_ATTN merged into one varlen call, 1.435 → 1.477×
Drop activation recompute (28 decoder layers)
partial: final BAGEL_NO_AC_FROM_LAYER=18 (last 10 layers). Log has tests of last 7 layers (stable), last 14 layers (window spread 6.7%), all removed (1.23×, per-step time jumping between 1.6–2.8 s)
Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
torch.compile on RMSNorm, RoPE (1.08×); BAGEL_MOT_FUSED on, no separate record in the log
Optimizer, EMA, step start / end
fused AdamW, EMA side stream both on, no distinguishable individual gain in the log; VAE side stream +15.6 ms/step, turned off after gate 2b blocked it
Other main levers
all-gather rate limiter off (1.080 → 1.094×)
Raw work log26 sections
0.2 hRound 1 — profile the unchanged reference
Start: 2026-11-10 03:02Z. End: 03:08Z.

Command: bench.py profile --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r1/profile
(unchanged tree == reference; profile of rank 0 over optimizer steps 8..9).

### Findings (per step, from the chrome trace; the profiled step is 2.82 s, the real one 2.60 s)

GPU work per step, by kernel (profiled):

| bucket | ms/step | share |
|---|---|---|
| NCCL reduce-scatter (bf16, 58 calls) | 785 | 22.6% |
| NCCL all-gather (bf16, 117 calls) | 525 | 15.1% |
| GEMM (sm90_xmma_gemm*, ~3000 calls) | 760 | 21.9% |
| attention fwd (fmha_cutlassF, SDPA mem-efficient + dense bias) | 238 | 6.9% |
| attention bwd (fmha_cutlassB) | 322 | 9.3% |
| elementwise / index / copy / cat (~9000 small kernels) | ~846 | 24.3% |

Stream accounting per step: compute stream busy 2040 ms, NCCL stream busy 1310 ms,
overlap 700 ms -> GPU union busy 2650 ms of the 2820 ms step (94% duty).
Exposed (not hidden behind compute): all-gather 260 ms/step, reduce-scatter 284 ms/step.

Breakdown of the collectives (from the trace's In/Out msg nelems):

* 28 decoder layers: AG in 116.5 M elems (233 MB bf16) -> out 932 MB, 8.3 ms each;
  RS in 466 M elems (932 MB) -> out 233 MB, 27.7 ms each (the RS kernel duration also
  contains the wait for the slowest rank and the queueing behind the AG on the same
  NCCL communicator).
* 1 root unit (embed_tokens + lm_head + norms, 1.09 B params): AG 50.9 ms, RS 4.9 ms.
* All-gathers happen twice per unit per step: once for the forward and once for the
  activation-checkpoint recompute in the backward.

Op counts per step (CPU): _index_put_impl_ 1440, mul 4444, to/_to_copy 4671/2356,
copy_ 3604, item 129 (host syncs), linear 2212.

### Conclusion / plan

1. The forward all-gather is fully exposed (260 ms/step): FSDP1 has no forward prefetch by
   default, but torch 2.5.1's FSDP(..., forward_prefetch=True) exists -> round 2.
2. ~846 ms/step of tiny elementwise kernels (RMSNorm, RoPE, gather/scatter of the MoT
   und/gen split) -> torch.compile in a later round.
3. Attention (560 ms/step) runs SDPA with a dense float bias; flex_attention with a block
   mask could skip masked blocks -> later round.
4. Optimizer+EMA foreach kernels ~150 ms/step -> later round.


---
1.1 hRound 4 — torch.compile the RMSNorm and the RoPE (N1)
Start: 2026-11-10 03:57Z. End: 04:05Z.

Change: modeling/qwen2/modeling_qwen2.py — added _rms_norm_impl / _rope_impl, compiled with
torch.compile(..., dynamic=True), used by Qwen2RMSNorm.forward and apply_rotary_pos_emb.
Off by default; enabled with BAGEL_COMPILE_RMSNORM=1 / BAGEL_COMPILE_ROPE=1
(/submission/switches.json has both on).  Same expressions, only the kernel scheduling changes.

Micro-benchmark (4096x3584 bf16, weight bf16): eager 0.232 ms -> compiled 0.072 ms (3.2x);
max relative difference 0.0078 (1 bf16 ULP).

bench.py time --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r4/time
* speedup_vs_frozen_reference: 1.0797  (2.4087 s/step vs the frozen 2.6006)
* window_s_per_step [2.4083, 2.4090], window_range 0.0003, beyond_reference_noise true
* peak alloc 82.14 GiB (unchanged), peak reserved 117.6 GiB
* no recompile/cache_size_limit/cudagraph message in the log; compile cost ~7 s, paid in the
  warm-up (startup_to_first_step_s 171 s vs the reference's 164 s), so the timing windows are clean.
* FSDP re-points its parameters every step; dynamo does not guard on the data pointer, so there
  were no recompiles over the 65 steps.

---
1.3 hRound 5 — FSDP `limit_all_gathers=False` (N0: communication layout only)
Start: 2026-11-10 04:12Z. End: 04:18Z.

Change: train/fsdp_utils.py — added limit_all_gathers=_env_flag('BAGEL_FSDP_LIMIT_ALL_GATHERS', True)
(keeps FSDP's default).  BAGEL_FSDP_LIMIT_ALL_GATHERS=0 in /submission/switches.json.

Why: in the reference profile the compute stream starves ~22 ms per decoder layer in the backward
(28 x 22 = 616 ms/step of idle) because the prefetched all-gather of the next unit is queued
*behind* the reduce-scatter on the shared NCCL stream.  The all-gather limiter is what delays
that prefetch.

bench.py time ... --out /tmp/r5/time
* speedup_vs_frozen_reference: 1.0941 (2.3770 s/step)  -- was 1.0797
* window_s_per_step [2.3802, 2.3738], window_range 0.0027, beyond_reference_noise true
* peak alloc 82.14 GiB (unchanged), peak reserved 126.8 GiB (was 117.6) -- more params unsharded
  at the same time; the GPU has 143.7 GiB, so ~17 GiB of headroom left.

---
2.0 hRound 6 — full check of the compiled RMSNorm/RoPE + FSDP tree (PASS, N1)
Start: 2026-11-10 04:25Z. End: 04:55Z.

bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r7/check
* status PASS, tier N1, performance.speedup = 1.0999 (candidate 2.3644 s/step vs the frozen
  reference 2.6006).
* gate 2 (record, 3 steps, deterministic): status NOT_BITWISE, exact_items_ok: true, no problems
  (the compiled RMSNorm/RoPE changes the arithmetic, so the hashes differ; everything else matches).
* gate 2b (conformance, interleaved data): PASS / conforming —
  R_N = 0.00807, R_J = 0.00658, R_B = 0.00644, ratios 1.25 / 1.02 (bound 3.0).
* gate 3 (statistical, 150 steps, seed 1): PASS (all 14 endpoints inside 3 sigma).
* peak alloc 82.14 GiB, peak reserved 127.0 GiB, startup_to_first_step_s 172 s.
* The judge's deterministic run patches flash_attn_func, flash_attn_varlen_func, ... in
  flash_attn.flash_attn_interface (it lists them in the record) — FlashAttention is usable.

---
2.0 hRound 7 — packed attention with FlashAttention, one call per split (N1, off by default)
Start: 2026-11-10 04:56Z.

Idea: upstream computes the packed attention with one dense-mask
scaled_dot_product_attention per sample, which runs the Ampere cutlass
memory-efficient kernel (fmha_cutlassF_bf16_aligned_64x128_rf_sm80) at ~30-50 TFLOP/s.
Attention is 560 ms of the 1861 ms of compute per step.  The mask, however, has a very
specific structure: inside a sample, a query of a split sees the *non-noise* splits that
come before it and, inside its own split, the lower triangle for a causal split and
everything else.  That is exactly a decomposition into per-split FlashAttention calls
(flash_attn_func, causal aligned to the bottom-right corner for the causal splits),
which also skips the masked-out blocks.

The batch may not change (the judge hashes every field of SimpleCustomBatch), and the
dense mask is the only per-sample attention information that reaches the model, so the
split structure is recovered from the mask itself (modeling/bagel/attn_split.py):
the growth of the number of visible columns per row and the columns that disappear give
the split boundaries and their modes.  Two adjacent causal splits are merged (they
produce the very same mask).  The recovered structure is checked against the mask
(BAGEL_FLASH_ATTN_VERIFY, default on) and the dense-mask path is used when it does not
reproduce the mask exactly.

Offline validation: 3000 random split structures (and 1000 with unit-length splits) —
the recovered structure rebuilds the mask bitwise in every case.

Numerical check on the GPU (one sample, 2330 tokens, 28 heads, head_dim 128):
* relative L2 difference against the dense-mask SDPA path: 8e-4 .. 2.9e-3 (bf16 level,
  the same order as the compiled RMSNorm change that passed gate 2b).
* speed of the attention itself: dense-mask SDPA 0.758 ms -> FlashAttention 0.212 ms
  (3.6x).

Switch: BAGEL_FLASH_ATTN=1 (default off) in /submission/switches.json.

bench.py time --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r8/time
* speedup_vs_frozen_reference: 1.2914 (2.0137 s/step) -- was 1.0941
* window_s_per_step [2.0124, 2.0150], window_range 0.0013, beyond_reference_noise true
* peak alloc 82.14 GiB (unchanged), peak reserved 125.5 GiB; the CPU is not the bottleneck
  (cpu.busy_cpus 5.8 of 96), so the extra kernel launches of the per-split attention do not
  hurt; startup_to_first_step_s 171 s (the compile cost is paid in the warm-up).

bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r9/check
* status PASS, tier N1, performance.speedup = 1.2799 (candidate 2.0319 s/step).
* gate 2 (record): NOT_BITWISE, exact_items_ok: true, no problems.
* gate 2b (conformance, interleaved data): PASS / conforming — R_N = 0.00766,
  R_J = 0.00660, R_B = 0.00613; ratios 1.25 / 1.08 (bound 3.0).  The FlashAttention
  decomposition does not move the pools apart.
* gate 3 (statistical): PASS.
* peak alloc 82.14 GiB, reserved 110.2 GiB; all four ranks take the same 2.0319 s/step.

---
3.1 hRound 11 — activation checkpointing switch (N0, off by default)
Start: 2026-11-10 05:38Z.

train/fsdp_utils.py: new switch BAGEL_NO_AC_FROM_LAYER=k — the decoder layers with
layer_idx >= k are not activation-checkpointed any more (their activations stay in memory, the
recomputation of their forward pass is saved).  Default off (upstream's behaviour).

time with BAGEL_NO_AC_FROM_LAYER=20 (the last 8 of the 28 layers):
speedup_vs_frozen_reference 1.3540 (1.9206 s/step), peak alloc 82.14 GiB,
peak reserved 134.13 GiB — only ~9.6 GiB of headroom on a 143.7 GiB card, too tight.
3.1 hRound 12 — activation checkpointing + a non-fragmenting allocator
Profile of the current tree (/tmp/r10/profile, 2 steps, rank 0), per step:
NCCL reduce-scatter 619 ms, all-gather 305 ms, GEMMs 777 ms, FlashAttention 164 ms,
index_elementwise (the MoT gathers/scatters) 124 ms, multi_tensor_apply (AdamW + EMA) 90 ms,
FillFunc (the new_zeros of the MoT buffers) 45 ms, direct_copy 56 ms.
The compute stream is busy 1486 ms of the 2058 ms step; the NCCL stream 924 ms, 607 ms of it
overlapped with the compute; the GPU is fully idle for 65 ms/step (one bubble at the step
boundary, where a pageable 8-byte H2D copy makes the host wait for the device).

time with BAGEL_NO_AC_FROM_LAYER=14 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
(/tmp/r12/time_a): speedup 1.3985 (1.8596 s/step), window range 0.0057,
peak alloc 94.67 GiB, peak reserved 121.65 GiB (was 134.13 GiB at k=20 without the
allocator change).  The allocator change buys ~12 GiB of headroom, which buys more layers
without activation checkpointing.

---
4.3 hRound 15 — no activation checkpointing at all (`BAGEL_NO_AC_FROM_LAYER=0`)
Start: 2026-11-10 06:41Z. End: 06:47Z.

time --config /tmp/r15_sw.json (r13's config with NO_AC=0), /tmp/r15_time:
s_per_step 2.1085, speedup_vs_frozen_reference 1.2334 -- *worse* than NO_AC=21
(1.8505).  peak alloc 126.16 GiB, peak reserved 134.97 GiB (the card has 143.7 GiB), and the
per-step times were erratic (1.60 .. 2.82 s).  Removing the recompute of all 28 decoder layers
costs ~45 GiB of live activations; at that point the caching allocator cannot keep the segments
and the step time becomes dominated by allocation stalls.  **Conclusion: the recompute is not the
bottleneck, the memory is.**  NO_AC=21 (7 layers without checkpointing) stays the best setting.

---
4.3 hRound 16 — profile of the current best tree, then three neutral changes
Profile of the current best tree (/tmp/r14_profile, rank 0, 2 steps, ~1.89 s/step):

| bucket | ms/step | share of the step |
|---|---|---|
| NCCL reduce-scatter (bf16, 28 calls of 932 MB in) | 532 | 28% |
| NCCL all-gather (56 calls of 233 MB in + 1 root unit of 545 MB) | 267 | 14% |
| GEMM (sm90_xmma_gemm*, ~4000 calls) | 748 | 40% |
| FlashAttention fwd+bwd | 160 | 8% |
| elementwise / index / copy / cat (~16000 small kernels) | ~380 | 20% |
| GPU idle (13963 gaps of ~9 us, no gap above 8 ms) | 126 | 7% |

* 74% of the reduce-scatter and 55% of the all-gather overlap with compute; **266 ms/step of
  communication is exposed**, plus 126 ms/step of small bubbles between kernels.
* The host is not the bottleneck: cudaLaunchKernel (26917 calls) takes 907 ms of *host* time
  over 2 steps, but that is the launch queue blocking (the host runs ahead and is throttled by
  the GPU); there is no cudaMalloc/cudaFree in the trace (the allocator does not thrash
  at NO_AC=21), and the CPU keeps only 5 of 96 cores busy.
* The root FSDP unit (embed_tokens + lm_head, 1.09 B parameters) is all-gathered in 52 ms at the
  very start of the forward, fully exposed.
* The int64 count all-reduces (total_ce_tokens, total_mse_tokens, total_samples) are
  100% exposed: 15 ms/step of pure rank-skew barrier between the forward and the backward.

Three changes were tried, each behind a switch, each measured with time:

* r16a BAGEL_FSDP_FORWARD_PREFETCH=1 (FSDP1 prefetches the all-gather of the next unit
  during the current one): /tmp/r16a_time -> 1.8604 s/step (1.3979).  *Neutral to slightly
  worse*: with limit_all_gathers=0 the all-gathers were already in flight.  Reverted.
* r16b BAGEL_FLASH_OUTBUF=1 (the per-split FlashAttention outputs are written into one
  preallocated buffer instead of being concatenated at the end; the CatArrayBatchedCopy of the
  packed attention is 35 ms/step): /tmp/r16b_time -> 1.8666 s/step (1.3932).  *Slightly worse*
  (the per-split copies cost more kernels than the one batched cat).  Reverted.
* r16c NCCL_PROTO=Simple (the collectives run the "LL" protocol): /tmp/r17_time ->
  1.8506 s/step (1.4053), i.e. exactly the same as without it (peak reserved identical to the
  byte), so the NCCL protocol is not the limiter.  Reverted (it also would have changed the
  gradient reduction order for nothing).

Also reverted: a GpuBatchStream prefetch of the next batch (it moves one batch too early, and
gate 2b compares the *whole* list of batches the rank moved to the GPU, not only the first N) and
the non_blocking=True copies of SimpleCustomBatch.cuda (with pageable memory the source
tensors of nested_attention_masks are freed right after the copy is issued -- a data race the
reference does not have).

---
4.3 hRound 17 — the depth of the activation checkpointing, again (`NO_AC=14`)
time --config /tmp/r18_sw.json, /tmp/r18_time: s_per_step 1.8518 (1.4043), peak alloc
94.73 GiB, peak reserved 134.84 GiB, but window_range 0.067 (6.7%): the two 30-step
windows differ by more than the 6% the final evaluation tolerates, so this configuration would be
re-measured as INVALID.  NO_AC=21 (window range 1.0%) is both the fastest and the stable
setting.  Kept.

---
4.3 hRound 18 — full check of the best tree (PASS, N1)
Start: 2026-11-10 07:21Z. End: 07:48Z.

bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r19_check
(the tree is exactly commit 82c039f; the switches are
{"BAGEL_COMPILE_RMSNORM":"1","BAGEL_COMPILE_ROPE":"1","BAGEL_FSDP_LIMIT_ALL_GATHERS":"0","BAGEL_FLASH_ATTN":"1","BAGEL_NO_AC_FROM_LAYER":"21","BAGEL_FUSED_ADAMW":"1","BAGEL_MOT_FUSED":"1"}).

* status PASS, tier N1, performance.speedup = 1.4051 (candidate 1.8508 s/step vs the frozen
  reference 2.6006 s/step; all four ranks agree).
* gate 2 (record, 3 steps, deterministic): NOT_BITWISE, exact_items_ok: true, no problems
  (1223 parameters, 1408 shards per step compared).  The compiled RMSNorm/RoPE, the FlashAttention
  decomposition and the fused AdamW change the arithmetic, so the hashes differ.
* gate 2b (conformance, interleaved data): PASS / conforming — the pools did not move apart.
* gate 3 (statistical, 150 steps, seed 1): PASS (all 14 endpoints inside 3 sigma).
* peak alloc 78.45 GiB, peak reserved ~131 GiB; the two timing windows agree to 0.9%.

---
5.5 hRound 22 — profile of the current best tree (`/tmp/r22_profile`)
Profile of the tree at 82c039f with the round-19 switches (rank 0, steps 10-12, trace
/tmp/r22_profile/timing_profile.trace.json, 253 MB).  The step in the profiled run is ~1.77 s
(profiler overhead); the machine's real numbers are 1.8508 s/step.

Machine facts measured directly (not what the task statement says): the GPUs report as
Hopper, sm90, 132 SMs, 143 GiB, 4258 GB/s HBM and 806 TFLOP/s bf16 GEMM
(measured with a 1 GiB copy and an 8192^3 bf16 matmul), 18 NVLinks of 26.5 GB/s each
(nvidia-smi topo -m: NV18).  The 4 ranks run in lockstep: the per-rank step-end timestamps of
the last timing run differ by <5 us, so there is no persistent rank skew (the 28 ms
"rank skew" of the round-14 note was an artifact of CUPTI inflating the NCCL kernel durations).

Per-step accounting (3 profiled steps, rank 0, GPU kernel time):

| bucket | ms/step | share |
|---|---|---|
| GEMM (sm90_xmma_gemm*, 1920 calls) | 788 | 48% |
| FlashAttention fwd+bwd | 166 | 10% |
| index gather/scatter (index_elementwise_kernel, 1483 calls) | 120 | 7% |
| vectorized/unrolled elementwise | ~190 | 12% |
| compiled RMSNorm+RoPE (triton, 1226 calls) | 54 | 3% |
| foreach optimizer/EMA (multi_tensor_apply + RowwiseMoments) | 64 | 4% |
| NCCL stream (all-gather 217 + reduce-scatter 120) | 337 | 20% (74% overlapped) |
| GPU idle (18004 gaps of ~7 us, none > 8.4 ms) | 123 | 7% |

Step structure (from the trace timeline): optimizer+EMA ~34-80 ms -> root all-gather ~44 ms
(exposed, nothing else runs) -> forward ~420 ms -> 3 int64 count all-reduces -> backward ~1150 ms
(= 2x the forward + the activation-checkpoint recompute of 21 of the 28 layers, ~315 ms) -> EMA.

Exposed (not overlapped with the compute) GPU time per step: all-gather 46 ms, reduce-scatter
17 ms, the int64 count all-reduces 15 ms, the recompute's side-stream ops 76 ms.

Conclusions used for this round:
* The reduce-scatter is not bandwidth starved (each 932 MB RS moves 700 MB over the ring in
  3.4 ms = 208 GB/s, the same rate as the all-gather); the earlier "37 GB/s" note was wrong.
* The activation-checkpoint recompute (~315 ms/step, 19% of the step) is the largest single
  overhead, but it is blocked by memory: NO_AC=14 allocated 94.7 GiB while the allocator had
  134.8 GiB reserved (40 GiB of fragmentation) and NO_AC=0 thrashed the allocator.
  => try the CUDA caching allocator's virtual-memory segments to remove the fragmentation.
5.5 hRound 22b — `NO_AC=14` re-measured with a CUDA allocator option (1.4745x)
time --config /tmp/r23_sw.json, /tmp/r23_time: **1.7637 s/step, speedup_vs_frozen_reference
1.4745**, window range 0.0062 (stable), peak alloc 94.67 GiB, peak reserved 124.82 GiB.

The only difference to round 17's NO_AC=14 run (/tmp/r18_time, 1.8518 s/step, window range
0.067 -> would have been INVALID in the final evaluation, reserved 134.84 GiB) is
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.  The allocator prints
"expandable_segments not supported on this platform" (the flag is not applied), yet the *reserved*
memory dropped by 10 GiB and the run became stable: the reserved/allocated gap is the allocator's
fragmentation (30-40 GiB), and at NO_AC=14 the model is close enough to the 143.7 GiB limit that
the fragmentation decides whether the run thrashes or not.  (Note: garbage_collection_threshold
and roundup_power2_divisions *are* supported by this torch build and are the next knob to try.)

The gain is the activation-checkpoint recompute of 7 more decoder layers (7 x ~15 ms/step).
5.5 hRound 22c — `NO_AC=7` (the memory wall)
/tmp/r24_time (killed): with 7 more layers un-checkpointed the peak reaches 135-142 GiB of the
143.7 GiB card, the allocator starts to thrash (0.47 steps/s at step 20 vs 0.68 at step 7) and the
per-step times become erratic.  NO_AC=7 is not viable with the default caching allocator.

=> next: the CUDA stream-ordered allocator (PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync),
which keeps the freed memory in a driver-managed pool instead of caching size-mismatched blocks in
the process, i.e. it removes the 30-50 GiB of fragmentation that makes the memory wall move around
from run to run (and that caused the 5.2 s stall of round 17's NO_AC=14 run).

---
6.2 hRound 29 — the VAE encoding on a side CUDA stream (1.4473x)
Start: 2026-11-10 08:55Z. End: 09:12Z.

Profile findings (from round 22, unchanged tree): the step is
optimizer+EMA (34-80 ms) -> root FSDP all-gather ~44 ms, fully exposed -> forward ~420 ms ->
3 int64 count all-reduces -> backward ~1150 ms (= 2x forward + the activation-checkpoint recompute of
21 of the 28 layers, ~315 ms) -> EMA.  The VAE encoding sat *before* the forward, so the GPU was idle
during it and the root all-gather had nothing to overlap with.

Change: modeling/bagel/bagel.py + train/pretrain_unified_navit.py — the VAE encoding runs on its
own CUDA stream (torch.cuda.Stream), the packed images and the latents are record_streamed, and
Bagel.forward waits on the recorded event right before the latents are used (after the ViT).  The
encode is still exactly one call per step and still draws its noise inside it (the judge keys the
injected noise by the ordinal of the encode call, not by the stream), so the data and the noise are
unchanged.  Off by default, BAGEL_VAE_OVERLAP=1.

* time --config /tmp/r28_sw.json (/tmp/r28_time): **1.7969 s/step, speedup_vs_frozen_reference
  1.4473** (r27, without the overlap: 1.8125 / 1.4348) -> the overlap wins 15.6 ms/step.
  window range 0.0164, peak alloc 85.72 GiB, peak reserved 134.97 GiB.
* check --quick --config /tmp/r28_sw.json (/tmp/r29_quick): exact_items_ok: true,
  NOT_BITWISE (expected, the tree is N1 overall), no problems: 1223 parameters and 1408 shards per
  step compared.
* Commit 9d1d724.

### Ruled out in this round (measured, not guessed)

* The H2D copy of the batch is not a lever.  A CPU-only run of the judge's data pipeline
  (/tmp/mask_size.py) shows that a batch is ~150 MB, of which **134 MB is the dense attention
  masks** (3-5 tensors of [sample_len, sample_len] float32; the samples are 2.3k-3.9k tokens).
  Micro-benchmark (/tmp/h2d_bench.py): 134 MB pageable H2D = 7.2 ms, pinned = 2.6 ms, and pinning
  costs 0.4 ms.  So the whole transfer is ~8 ms of the 1.8 s step; even a perfect overlap saves
  <1%.
* **expandable_segments:True is *not* applied by this torch build** (the warning comes from
  PYTORCH_CUDA_ALLOC_CONF_EXPANDABLE_SEGMENTS not being a compile-time flag in the wheel), so
  round 17's NO_AC=14 instability (6.7% window range) vs round 22b's stable run (0.62%) was
  run-to-run variance, not the allocator option.  NO_AC=14 stays rejected: the final
  evaluation uses another data seed and a 6.7% window range is INVALID (score 0).

---
6.8 hRound 30 — profile of the tree with the VAE overlap (`/tmp/r30_profile`)
bench.py profile --tree /submission/bagel_sft --config /tmp/r28_sw.json --out /tmp/r30_profile --from 10 --iters 3
(rank 0, steps 10-12, trace /tmp/r30_profile/timing_profile.trace.json, 250 MB; the profiled step is ~1.78 s).

Per-step kernel time (3 steps / 3 = per step; the profiler adds ~15%):

| bucket | ms/step | share |
|---|---|---|
| GEMM (sm90_xmma_gemm*) | 763 | 39% |
| NCCL (all-gather 349 + reduce-scatter 120) | 469 | 24% (mostly overlapped) |
| elementwise (vectorized + unrolled + elementwise) | 286 | 15% |
| FlashAttention fwd (58) + bwd (98) | 156 | 8% |
| index gather/scatter (index_elementwise_kernel) | 115 | 6% |
| optimizer + EMA (multi_tensor_apply, RowwiseMoments) | 55 | 3% |
| compiled RMSNorm/RoPE (triton) | 52 | 3% |
| GPU idle (union of all streams) | 136 | 7% |

Where the GPU idle comes from (this is the interesting part):

* The idle is *not* spread over the step: two windows of 35 ms and 71 ms where no kernel
  runs on any stream (offsets 2184-2297 ms of the trace).  During them the CPU is inside
  aten::linear/aten::addmm, and the CUDA runtime log shows 169 cudaFree calls (40.8 ms)
  — the caching allocator's garbage collection: it cannot serve a request (the large buffers of the
  MoT MLP change size every step, because the number of understanding/generation tokens does), it
  releases *all* its cached blocks — every cudaFree synchronizes the device — and retries.
  In the same 3 steps: 98 + 40 cudaMalloc calls (255 ms + 25 ms).
* The remaining ~60 ms of idle are the ~8 us gap between consecutive kernels (16 700 kernels per
  step, the GPU's launch latency).

Per-op CPU cost (from the profiler's own table): aten::_index_put_impl_ 938 calls/step and
aten::index 716 calls/step; aten::fill_ 1891 calls/step (41 ms of GPU time); aten::copy_
2890 calls/step.

### Ruled out in this round (measured)

* PYTORCH_CUDA_ALLOC_CONF=roundup_power2_divisions:4 (r31, /tmp/r31_time): the reserved
  memory did not go down (134.94 GiB vs 134.97) and the allocated memory went up by 5 GiB
  (the rounding is applied to the block size), so the step got *slower*: 1.8436 s/step, 1.4106.
  A micro-benchmark of the allocation pattern (/tmp/alloc_sim.py) had shown a 2.7x smaller
  reserved memory, but the real workload has many more distinct sizes.  Reverted.
* torch.cuda.empty_cache() after the setup (r32, /tmp/r32_time): 1.7984 s/step, 1.4460,
  window range 0.0 — neutral, and the reserved memory was unchanged (134.89 GiB).  So the
  fragmentation is *not* left over from the model/EMA/optimizer setup, it is created by the training
  loop itself.  Reverted.
* index_copy_ instead of x[idx] = v (/tmp/idx_bench.py, /tmp/idx_bench2.py): the
  gradients are bitwise identical, and the forward is faster (0.088 ms vs 0.130 ms for a
  [10240, 3584] scatter), but the backward of index_copy_ (index_select) is slower than the
  backward of index_put_ (index), so forward+backward is *worse* (0.282 vs 0.262 ms).
  index_select is also slower on the GPU than advanced indexing (0.193 vs 0.131 ms).  Rejected.
* scatter_ with an expanded index: the same forward speed as index_copy_ but it needs the
  index expanded to the full [n, d] shape; not worth the memory.

---
7.2 hRound 33 — the EMA update on its own CUDA stream (1.4694x)
Start: 2026-11-10 09:40Z. End: 09:54Z.

Change (c78b3d2, switch BAGEL_EMA_OVERLAP, off by default): train/pretrain_unified_navit.py — the
EMA update runs on a second CUDA stream (ema_stream) instead of the compute stream.  It is enqueued
*after* the logging block (whose torch.cuda.synchronize() would otherwise wait for it), and the compute
stream waits for it at the next step's loss.backward() (FSDP rewrites the flat parameters when it
reshards its units at the end of the backward pass).  The judge reads the EMA's flat parameters in the
optimizer post-hook, i.e. before the current step's EMA update, exactly as in the reference.

* time --config /submission/switches.json (/tmp/r33_time): **1.7698 s/step,
  speedup_vs_frozen_reference 1.4694**, window range 0.0148, peak alloc 85.72 GiB, reserved 134.96 GiB.
7.2 hRound 34 — full check of the tree with the VAE and the EMA on their own streams: **INCORRECT**
Start: 2026-11-10 09:54Z. End: 10:07Z.

bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r34_check
(the tree is exactly commit c78b3d2; the switches are the nine of /submission/switches.json).

* gate 1 (timing): 1.4499x (candidate 1.79 s/step vs the frozen reference 2.6006 s/step) — the benefit
  gate passes.
* gate 2 (record, 3 steps, deterministic): NOT_BITWISE, exact_items_ok: true, no problems.
* gate 2b (conformance, interleaved data): INCORRECT / nonconforming.  The per-token MSE of the
  candidate becomes non-finite from the 5th forward on (rank 0: forwards 0-3 finite, 4-15 all NaN;
  the same on the other ranks), and the CE with it: **the model diverges to NaN after three optimizer
  steps** on the interleaved data.  R_N = R_J = R_B = nan (the reference's own numbers are finite), the
  per-batch values reach 16.7 (N pool) and 9.6 (J pool).  The production-mode run was not even reached.
* gate 3 was not run.

Reading: the divergence is *not* a small numerical difference — the parameters become NaN.  With the
learning rate still in its warm-up (2e-5 x step/2000, i.e. ~3e-8 at step 3) the parameters cannot diverge
numerically in three steps, so something *writes* garbage into them: a race between the two new side
streams and the compute stream.  The two candidates are the VAE encoding on vae_stream (the images are
read by the VAE on a side stream while the compute stream runs) and the EMA update on ema_stream (it
reads the model's sharded flat parameters while the next step's forward all-gathers them).

Bisection started in round 35 (a direct run of the judge's conformance gate on the interleaved data
configuration, bench/conform.py's conform_run, with one of the two switches turned off).
7.9 hRound 35 — the VAE side stream: root cause hunt, and the decision to drop it
Start: 2026-11-10 10:31Z. End: 11:05Z.

The VAE-on-side-stream tree diverges to NaN on the interleaved data (gate 2b).  Three experiments,
all with the judge's own gate-2b runner (/tmp/run_conform.py, 8-16 steps):

* VAE off, EMA on (/tmp/cf_vaeoff): no NaN; the manual statistic against the frozen
  reference is r_n=0.0089, r_j=0.0069, r_b=0.0056, bound 0.0169, ratios 1.58/1.22 -> conforming.
  The EMA overlap is fine.
* VAE on, EMA off (/tmp/cf_emaoff): NaN again -> the VAE stream is the culprit.
* VAE on + a new diagnostic (BAGEL_VAE_DEBUG3, /tmp/cf_vaedbg3): the diagnostic captures the
  VAE's internal mean (through a torch.randn_like spy) during the side-stream encode and
  re-computes the encoder on the compute stream afterwards.  Result:
  mean_diff=0.0 (the VAE's internals are bitwise identical), the input images' checksum is
  unchanged, the reconstructed latent matches the stored one to 1.4e-6 (bf16 rounding only),
  and the run did not diverge.  The added diagnostic work changes the timing, so the
  divergence is timing sensitive.

Everything that could be checked was checked and came out clean: the stream ordering
(wait_stream/wait_event/record_stream), the H2D copy (the images are pinned, but
Tensor.to(device) with non_blocking=False is synchronous), the RNG (the judge keys the noise
by the call's ordinal, so the stream cannot change it), the VAE's weights (frozen, never freed)
and the autocast cache (the frozen weights are not cached).

**Conclusion: an unexplained, timing-sensitive race between the VAE's side stream and the
compute stream.**  Two full checks have rejected it.  The overlap is worth 2.4% of the speedup,
which is not worth an unexplained race, so the VAE side stream is switched off.  The code stays
in the tree behind BAGEL_VAE_OVERLAP (default off) for the record.

* time --config /submission/switches.json (/tmp/r35/time, EMA overlap only):
  1.8128 s/step, speedup_vs_frozen_reference 1.4346, beyond_reference_noise: true,
  peak allocated 85.72 GiB, reserved 134.22 GiB, cpu.busy_cpus_in_windows 5.04.
7.9 hRound 36 — where the remaining time goes (profile analysis of the r30 trace)
The r30 trace (/tmp/r30_profile/timing_profile.trace.json, one step, 1823 ms) was re-analysed
with the GPU kernels correlated to their CPU ops (/tmp/an_corr2.py):

* compute stream 1403 ms busy, 18 765 kernels; NCCL 610 ms (all-gather 464, reduce-scatter 146,
  95 ms exposed); VAE stream 135 ms; GPU union busy 1710 ms of 1823 ms -> 113 ms idle in
  14 701 small gaps (7.7 us each).
* the idle is CPU-bound: cpu.busy_cpus_in_windows is 5.04 with 4 ranks, i.e. every rank's
  python thread is ~100% busy.  The step is limited by the kernel-launch rate, not by the GPU.
* the compute stream by op: mm 747 ms, flash attention 170 ms, **index_put_ 73 ms (1964
  calls), index 63 ms (716 calls)**, fill_ 38 ms (1661), copy_ 22 ms (2265), add_ 38 ms
  (1590), mul 30 ms, silu 13 ms, fused AdamW 34 ms, the compiled norms/RoPE 41 ms.

So the two levers that are left are (a) the number of kernel launches the CPU has to issue (the
MoT packing alone is 2680 launches and 136 ms of GPU time) and (b) the 95 ms of exposed NCCL at
the start of the forward, which is exactly what the (broken) VAE side stream was meant to cover.
8.3 hRound 37 — full check of the EMA-only tree: **PASS, tier N1, 1.437x**
Start: 2026-11-10 10:53Z. End: 11:20Z.

Config: /submission/switches.json (8 switches, BAGEL_VAE_OVERLAP removed — the VAE side
stream stays off).  Full check, /tmp/r36/check.

* gate 1 (timing): candidate 1.8097 s/step, frozen reference 2.6006 -> speedup 1.437,
  beyond_reference_noise: true, window range 1.8 %, peak allocated 85.72 GiB.
* gate 2 (record): exact_items_ok: true, NOT_BITWISE (expected: FlashAttention instead of
  the dense-mask SDPA changes the arithmetic), 3 steps, 1408 shards/step compared.
* gate 2b (conformance, interleaved data): deterministic `r_n=0.00762, r_j=0.00653,
  r_b=0.00646`, bound 0.01938, ratios 1.18/1.01 -> conforming; production
  r_n=0.00755, r_j=0.00693, r_b=0.00621, ratios 1.22/1.12 -> conforming.
* gate 3 (trajectory, seed 1, 150 steps): every endpoint within the 3 sigma bound
  (the largest is log_grad_norm_s050_100 at 1.60 sigma) -> PASS.

status: PASS, correctness.measured_tier: N1.  This is the tree to keep.
8.3 hRound 38 — batched varlen FlashAttention for the packed attention
Start: 2026-11-10 11:20Z.

The profile says the step is CPU-bound (every rank's python thread is ~100 % busy, 113 ms of
GPU idle in 14 701 small gaps per step, 11.5 k allocations/step -> 55 409 cudaEventQuery
calls/step in the allocator).  The packed attention issues **one FlashAttention call per split
per layer** (840 forward + 1008 backward calls per step) plus the torch.cat of the visible
keys of every split.

Idea: the splits of a sample are contiguous and a split sees the *non-noise* splits before it
plus itself, so the visible keys of a split are the prefix of its sample **unless a noise split
precedes it.  All the prefix splits of a layer can therefore be computed by a single
flash_attn_varlen_func call per mask kind** (causal / not): the queries of the splits are
gathered into one buffer, their visible keys into another, and the split boundaries become the
cumulative sequence lengths.  The splits that follow a noise split keep the per-split path
(their visible keys have a hole).

The plan depends only on the split structure, which is the same for every layer of the step, so
it is built once per step and cached.

Implementation: modeling/bagel/qwen2_navit.py, behind BAGEL_VARLEN_ATTN (default off).
8.3 hRound 38, first attempt — the bug in `_concat_ranges`
The first implementation of the varlen attention was wrong: _concat_ranges wrote the offsets
1..length-1 instead of base+1..base+length-1, so the second and later samples of a group
pointed at the wrong rows.  The unit test caught it (max|diff| = the signal scale).  Fixed.
8.3 hRound 38 — the batched varlen FlashAttention: **1.4769x**
Start: 2026-11-10 11:20Z. End: 12:25Z.

Why.  The profile of the step (r30 trace, /tmp/r30_profile/timing_profile.trace.json, one
step of 1823 ms) says the step is GPU-work-bound, not launch-bound: the union of the busy
intervals of all the streams is 1710 ms of the 1823 ms step (the GPU is 94 % busy) and the idle
is 113 ms spread over 14 700 gaps of ~7 us.  The compute stream (1403 ms) is:

| op | ms/step | calls/step |
|---|---|---|
| aten::mm (the GEMMs) | 746.8 | 2608 |
| FlashAttention backward | 105.5 | 1008 |
| aten::_index_put_impl_ | 73.1 | 1964 |
| FlashAttention forward | 65.0 | 690 |
| aten::index | 63.1 | 716 |
| aten::fill_ | 38.3 | 1661 |
| aten::add_ | 38.1 | 1590 |

and the GEMMs run at 652 TFLOPS while a plain linear of the same shape reaches 730 TFLOPS on
this GPU, i.e. the GEMMs have no headroom — the attention and the packing do.

The attention was the biggest reducible item.  Upstream's packed attention needs **one
FlashAttention call per split of every sample** (690 forward + 1008 backward kernels per step),
and every one of them slices the packed query/key/value, so the **backward of every slice
allocates a zero buffer the size of the whole packed tensor** (aten::slice_backward ->
aten::zeros): 1232 of the 1661 fill_ calls and 1232 of the 2664 copy_ calls of the step
come from there, and they cost ~50 GB of memset per step.

The change.  The visible keys of a split are the *non-noise* splits before it plus the split
itself, so for every split that is not preceded by a noise split they are exactly the prefix
of its sample.  All the prefix splits of a layer can therefore be computed by **one
flash_attn_varlen_func call per mask kind** (causal / not): the queries of the splits are
gathered into one buffer, their visible keys into another, and the split boundaries become the
cumulative sequence lengths.  The splits that follow a noise split keep the per-split path.

Two details that had to be right:

* FlashAttention's varlen causal mask is aligned per sequence to the bottom right (verified
  against flash_attn_func with seqlen_q != seqlen_k: bitwise identical), which is exactly
  the mask of a causal split.
* index_select's backward adds the rows with atomics, so the rows that several splits share
  are summed in an arbitrary order — not reproducible even with
  torch.use_deterministic_algorithms(True).  The gather therefore uses a custom autograd
  function whose backward accumulates with index_put_(accumulate=True) (sorted, reproducible),
  and the scatter uses one whose backward is a plain gather.

modeling/bagel/qwen2_navit.py, behind BAGEL_VARLEN_ATTN (default off).

Validation (unit test, 3 layouts, out/dq/dk/dv against the per-split path):
all prefix and single split are bitwise identical, noise in the middle differs by
4e-3 relative in dk/dv only (the bf16 accumulation of the shared key rows).

* time --config /tmp/sw_varlen.json (/tmp/r38/time): **1.7608 s/step,
  speedup_vs_frozen_reference 1.4769**, beyond_reference_noise: true, window range 0.5 %,
  peak allocated 86.2 GiB, reserved 134.3 GiB.  (The EMA-only tree: 1.8128 s/step, 1.4346x.)
* the judge's own conformance runner on the interleaved data (/tmp/cf_varlen):
  conforming — r_n=0.00768, r_j=0.00654, r_b=0.00612, bound 0.01837, ratios 1.25/1.07.
9.1 hRound 39 — the full check of the tree with the varlen attention
Start: 2026-11-10 11:57Z.

check --tree /submission/bagel_sft --config /submission/switches.json (/tmp/r39/check), with
BAGEL_VARLEN_ATTN=1 added to the switch file.

Result: status: PASS, measured_tier: N1, performance.speedup 1.462** (1.7788 s/step
against the frozen reference's 2.6006).**

* gate 1 (timing): 1.7788 s/step, window range 0.94 %, beyond_reference_noise: true,
  peak allocated 86.2 GiB, reserved 134.0 GiB, cpu.busy_cpus_in_windows 5.09.
* gate 2 (record): exact_items_ok: true, NOT_BITWISE (the FlashAttention arithmetic).
* gate 2b: deterministic r_n=0.00768, r_j=0.00654, r_b=0.00612 (bound 0.01837, ratios
  1.25/1.07) -> conforming; production r_n=0.00760, r_j=0.00657, r_b=0.00601 (ratios
  1.26/1.09) -> conforming.
* gate 3 (trajectory, seed 1): every endpoint inside the 3 sigma bound (the largest is
  log_grad_norm_s050_100 at 2.05 sigma) -> PASS.

**The submission is now the tree of commit ed48c34 with BAGEL_VARLEN_ATTN=1 in
/submission/switches.json: 1.462x (the standalone time run measured 1.4769x).**
9.8 hRound 40 — the varlen attention also covers the splits whose visible keys have a hole
Start: 2026-11-10 12:20Z.  End: 2026-11-10 12:50Z.

Profile findings (r30 trace, 1 step = 1823 ms, GPU busy 94 %): the step is GPU-work bound.
Per step: GEMMs ~674 ms (at 652 TFLOPS, i.e. no headroom), all-gathers ~307 ms and
reduce-scatters ~106 ms (23 % of the step in communication), FlashAttention ~138 ms, and
~305 ms in the *index* round trips of the MoT expert path (aten::index 54 ms,
aten::index_put_ 61 ms, aten::copy_ 56 ms, aten::cat 31 ms, aten::fill_ 35 ms,
aten::add_ 37 ms).

Change: _build_varlen_plan now makes *every* split one sequence of a varlen call, not
only the splits whose visible keys are the prefix of their sample.  A split that follows a
noise split sees its visible keys with a hole (the noise span is skipped); the plan gathers
the visible ranges in the same order as the per-split path, so the key buffer of the
sequence — and therefore the bottom-right aligned causal mask — is unchanged.
The fix that made it work: cu_seqlens_k must hold the *per-sequence* key lengths (the sum
of the ranges of a split), not the per-range lengths.

Validation (unit test, 3 layouts): out and dq are bitwise identical to the
per-split path in all three layouts; dk/dv differ by ~4e-3 relative (1 bf16 ulp) only in
the layout with a hole.  Micro-benchmark of the hole layout: 10.24 ms -> 6.88 ms (1.49x).

* time --config /submission/switches.json (/tmp/r40/time): **1.7581 s/step,
  speedup_vs_frozen_reference 1.4792** (round 39: 1.4769), beyond_reference_noise: true.
  The gain is small (~0.2 %, inside the 0.25 % reference noise) because the timing data is
  the interleaved one, in which only a few splits per batch have a hole.
↑ BAGEL SFT