MiniMax M3.1 Flash 1.49×
Harness: Codex · switches: BAGEL_COMPILE_RMSNORM=1 BAGEL_COMPILE_ROPE=1 BAGEL_FSDP_LIMIT_ALL_GATHERS=0 BAGEL_FLASH_ATTN=1 BAGEL_NO_AC_FROM_LAYER=18 BAGEL_FUSED_ADAMW=1 BAGEL_MOT_FUSED=1 BAGEL_EMA_OVERLAP=1 BAGEL_VARLEN_ATTN=1
- Judged
- 1.49×
- Tier
- N1
- Self-best
- 1.48×
- Rounds
- 22
- Hours
- 10.0
- API cost
- $4.50
- Input tokens · 97.2% cached
- 186 M
- Output tokens
- 1.7 M
- Requests
- 1,298
§ 02Key levers
- Attention kernel swap (split by mask structure)
- ✓
BAGEL_FLASH_ATTN: one FlashAttention call per segment, 1.09 → 1.29×;BAGEL_VARLEN_ATTNmerged into one varlen call, 1.435 → 1.477× - Drop activation recompute (28 decoder layers)
- partial: final
BAGEL_NO_AC_FROM_LAYER=18(last 10 layers). Log has tests of last 7 layers (stable), last 14 layers (window spread 6.7%), all removed (1.23×, per-step time jumping between 1.6–2.8 s) - Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
torch.compileon RMSNorm, RoPE (1.08×);BAGEL_MOT_FUSEDon, no separate record in the log- Optimizer, EMA, step start / end
- fused AdamW, EMA side stream both on, no distinguishable individual gain in the log; VAE side stream +15.6 ms/step, turned off after gate 2b blocked it
- Other main levers
- all-gather rate limiter off (1.080 → 1.094×)
§ ARaw trace
Raw work log26 sections
0.2 hRound 1 — profile the unchanged reference
Start: 2026-11-10 03:02Z. End: 03:08Z. Command:bench.py profile --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r1/profile(unchanged tree == reference; profile of rank 0 over optimizer steps 8..9). ### Findings (per step, from the chrome trace; the profiled step is 2.82 s, the real one 2.60 s) GPU work per step, by kernel (profiled): | bucket | ms/step | share | |---|---|---| | NCCL reduce-scatter (bf16, 58 calls) | 785 | 22.6% | | NCCL all-gather (bf16, 117 calls) | 525 | 15.1% | | GEMM (sm90_xmma_gemm*, ~3000 calls) | 760 | 21.9% | | attention fwd (fmha_cutlassF, SDPA mem-efficient + dense bias) | 238 | 6.9% | | attention bwd (fmha_cutlassB) | 322 | 9.3% | | elementwise / index / copy / cat (~9000 small kernels) | ~846 | 24.3% | Stream accounting per step: compute stream busy 2040 ms, NCCL stream busy 1310 ms, overlap 700 ms -> GPU union busy 2650 ms of the 2820 ms step (94% duty). Exposed (not hidden behind compute): all-gather 260 ms/step, reduce-scatter 284 ms/step. Breakdown of the collectives (from the trace'sIn/Out msg nelems): * 28 decoder layers: AG in 116.5 M elems (233 MB bf16) -> out 932 MB, 8.3 ms each; RS in 466 M elems (932 MB) -> out 233 MB, 27.7 ms each (the RS kernel duration also contains the wait for the slowest rank and the queueing behind the AG on the same NCCL communicator). * 1 root unit (embed_tokens + lm_head + norms, 1.09 B params): AG 50.9 ms, RS 4.9 ms. * All-gathers happen twice per unit per step: once for the forward and once for the activation-checkpoint recompute in the backward. Op counts per step (CPU):_index_put_impl_1440,mul4444,to/_to_copy4671/2356,copy_3604,item129 (host syncs),linear2212. ### Conclusion / plan 1. The forward all-gather is fully exposed (260 ms/step): FSDP1 has no forward prefetch by default, but torch 2.5.1'sFSDP(..., forward_prefetch=True)exists -> round 2. 2. ~846 ms/step of tiny elementwise kernels (RMSNorm, RoPE, gather/scatter of the MoT und/gen split) -> torch.compile in a later round. 3. Attention (560 ms/step) runs SDPA with a dense float bias; flex_attention with a block mask could skip masked blocks -> later round. 4. Optimizer+EMA foreach kernels ~150 ms/step -> later round. ---
1.1 hRound 4 — torch.compile the RMSNorm and the RoPE (N1)
Start: 2026-11-10 03:57Z. End: 04:05Z. Change:modeling/qwen2/modeling_qwen2.py— added_rms_norm_impl/_rope_impl, compiled withtorch.compile(..., dynamic=True), used byQwen2RMSNorm.forwardandapply_rotary_pos_emb. Off by default; enabled withBAGEL_COMPILE_RMSNORM=1/BAGEL_COMPILE_ROPE=1(/submission/switches.jsonhas both on). Same expressions, only the kernel scheduling changes. Micro-benchmark (4096x3584 bf16, weight bf16): eager 0.232 ms -> compiled 0.072 ms (3.2x); max relative difference 0.0078 (1 bf16 ULP).bench.py time --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r4/time*speedup_vs_frozen_reference: 1.0797 (2.4087 s/step vs the frozen 2.6006) *window_s_per_step[2.4083, 2.4090],window_range0.0003,beyond_reference_noisetrue * peak alloc 82.14 GiB (unchanged), peak reserved 117.6 GiB * norecompile/cache_size_limit/cudagraphmessage in the log; compile cost ~7 s, paid in the warm-up (startup_to_first_step_s 171 s vs the reference's 164 s), so the timing windows are clean. * FSDP re-points its parameters every step; dynamo does not guard on the data pointer, so there were no recompiles over the 65 steps. ---
1.3 hRound 5 — FSDP `limit_all_gathers=False` (N0: communication layout only)
Start: 2026-11-10 04:12Z. End: 04:18Z. Change:train/fsdp_utils.py— addedlimit_all_gathers=_env_flag('BAGEL_FSDP_LIMIT_ALL_GATHERS', True)(keeps FSDP's default).BAGEL_FSDP_LIMIT_ALL_GATHERS=0in/submission/switches.json. Why: in the reference profile the compute stream starves ~22 ms per decoder layer in the backward (28 x 22 = 616 ms/step of idle) because the prefetched all-gather of the next unit is queued *behind* the reduce-scatter on the shared NCCL stream. The all-gather limiter is what delays that prefetch.bench.py time ... --out /tmp/r5/time*speedup_vs_frozen_reference: 1.0941 (2.3770 s/step) -- was 1.0797 *window_s_per_step[2.3802, 2.3738],window_range0.0027,beyond_reference_noisetrue * peak alloc 82.14 GiB (unchanged), peak reserved 126.8 GiB (was 117.6) -- more params unsharded at the same time; the GPU has 143.7 GiB, so ~17 GiB of headroom left. ---
2.0 hRound 6 — full check of the compiled RMSNorm/RoPE + FSDP tree (PASS, N1)
Start: 2026-11-10 04:25Z. End: 04:55Z.bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r7/check* status PASS, tier N1,performance.speedup= 1.0999 (candidate 2.3644 s/step vs the frozen reference 2.6006). * gate 2 (record, 3 steps, deterministic):status NOT_BITWISE,exact_items_ok: true, no problems (the compiled RMSNorm/RoPE changes the arithmetic, so the hashes differ; everything else matches). * gate 2b (conformance, interleaved data): PASS / conforming —R_N = 0.00807,R_J = 0.00658,R_B = 0.00644, ratios 1.25 / 1.02 (bound 3.0). * gate 3 (statistical, 150 steps, seed 1): PASS (all 14 endpoints inside 3 sigma). * peak alloc 82.14 GiB, peak reserved 127.0 GiB,startup_to_first_step_s172 s. * The judge's deterministic run patchesflash_attn_func,flash_attn_varlen_func, ... inflash_attn.flash_attn_interface(it lists them in the record) — FlashAttention is usable. ---
2.0 hRound 7 — packed attention with FlashAttention, one call per split (N1, off by default)
Start: 2026-11-10 04:56Z. Idea: upstream computes the packed attention with one dense-maskscaled_dot_product_attentionper sample, which runs the Ampere cutlass memory-efficient kernel (fmha_cutlassF_bf16_aligned_64x128_rf_sm80) at ~30-50 TFLOP/s. Attention is 560 ms of the 1861 ms of compute per step. The mask, however, has a very specific structure: inside a sample, a query of a split sees the *non-noise* splits that come before it and, inside its own split, the lower triangle for acausalsplit and everything else. That is exactly a decomposition into per-split FlashAttention calls (flash_attn_func, causal aligned to the bottom-right corner for thecausalsplits), which also skips the masked-out blocks. The batch may not change (the judge hashes every field ofSimpleCustomBatch), and the dense mask is the only per-sample attention information that reaches the model, so the split structure is recovered from the mask itself (modeling/bagel/attn_split.py): the growth of the number of visible columns per row and the columns that disappear give the split boundaries and their modes. Two adjacent causal splits are merged (they produce the very same mask). The recovered structure is checked against the mask (BAGEL_FLASH_ATTN_VERIFY, default on) and the dense-mask path is used when it does not reproduce the mask exactly. Offline validation: 3000 random split structures (and 1000 with unit-length splits) — the recovered structure rebuilds the mask bitwise in every case. Numerical check on the GPU (one sample, 2330 tokens, 28 heads, head_dim 128): * relative L2 difference against the dense-mask SDPA path: 8e-4 .. 2.9e-3 (bf16 level, the same order as the compiled RMSNorm change that passed gate 2b). * speed of the attention itself: dense-mask SDPA 0.758 ms -> FlashAttention 0.212 ms (3.6x). Switch:BAGEL_FLASH_ATTN=1(default off) in/submission/switches.json.bench.py time --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r8/time*speedup_vs_frozen_reference: 1.2914 (2.0137 s/step) -- was 1.0941 *window_s_per_step[2.0124, 2.0150],window_range0.0013,beyond_reference_noisetrue * peak alloc 82.14 GiB (unchanged), peak reserved 125.5 GiB; the CPU is not the bottleneck (cpu.busy_cpus5.8 of 96), so the extra kernel launches of the per-split attention do not hurt;startup_to_first_step_s171 s (the compile cost is paid in the warm-up).bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r9/check* status PASS, tier N1,performance.speedup= 1.2799 (candidate 2.0319 s/step). * gate 2 (record):NOT_BITWISE,exact_items_ok: true, no problems. * gate 2b (conformance, interleaved data): PASS / conforming —R_N = 0.00766,R_J = 0.00660,R_B = 0.00613; ratios 1.25 / 1.08 (bound 3.0). The FlashAttention decomposition does not move the pools apart. * gate 3 (statistical): PASS. * peak alloc 82.14 GiB, reserved 110.2 GiB; all four ranks take the same 2.0319 s/step. ---
3.1 hRound 11 — activation checkpointing switch (N0, off by default)
Start: 2026-11-10 05:38Z.train/fsdp_utils.py: new switchBAGEL_NO_AC_FROM_LAYER=k— the decoder layers withlayer_idx >= kare not activation-checkpointed any more (their activations stay in memory, the recomputation of their forward pass is saved). Default off (upstream's behaviour).timewithBAGEL_NO_AC_FROM_LAYER=20(the last 8 of the 28 layers):speedup_vs_frozen_reference1.3540 (1.9206 s/step), peak alloc 82.14 GiB, peak reserved 134.13 GiB — only ~9.6 GiB of headroom on a 143.7 GiB card, too tight.
3.1 hRound 12 — activation checkpointing + a non-fragmenting allocator
Profile of the current tree (/tmp/r10/profile, 2 steps, rank 0), per step: NCCL reduce-scatter 619 ms, all-gather 305 ms, GEMMs 777 ms, FlashAttention 164 ms,index_elementwise(the MoT gathers/scatters) 124 ms,multi_tensor_apply(AdamW + EMA) 90 ms,FillFunc(thenew_zerosof the MoT buffers) 45 ms,direct_copy56 ms. The compute stream is busy 1486 ms of the 2058 ms step; the NCCL stream 924 ms, 607 ms of it overlapped with the compute; the GPU is fully idle for 65 ms/step (one bubble at the step boundary, where a pageable 8-byte H2D copy makes the host wait for the device).timewithBAGEL_NO_AC_FROM_LAYER=14andPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True(/tmp/r12/time_a): speedup 1.3985 (1.8596 s/step), window range 0.0057, peak alloc 94.67 GiB, peak reserved 121.65 GiB (was 134.13 GiB at k=20 without the allocator change). The allocator change buys ~12 GiB of headroom, which buys more layers without activation checkpointing. ---
4.3 hRound 15 — no activation checkpointing at all (`BAGEL_NO_AC_FROM_LAYER=0`)
Start: 2026-11-10 06:41Z. End: 06:47Z.time --config /tmp/r15_sw.json(r13's config withNO_AC=0),/tmp/r15_time:s_per_step2.1085,speedup_vs_frozen_reference1.2334 -- *worse* thanNO_AC=21(1.8505). peak alloc 126.16 GiB, peak reserved 134.97 GiB (the card has 143.7 GiB), and the per-step times were erratic (1.60 .. 2.82 s). Removing the recompute of all 28 decoder layers costs ~45 GiB of live activations; at that point the caching allocator cannot keep the segments and the step time becomes dominated by allocation stalls. **Conclusion: the recompute is not the bottleneck, the memory is.**NO_AC=21(7 layers without checkpointing) stays the best setting. ---
4.3 hRound 16 — profile of the current best tree, then three neutral changes
Profile of the current best tree (/tmp/r14_profile, rank 0, 2 steps, ~1.89 s/step): | bucket | ms/step | share of the step | |---|---|---| | NCCL reduce-scatter (bf16, 28 calls of 932 MB in) | 532 | 28% | | NCCL all-gather (56 calls of 233 MB in + 1 root unit of 545 MB) | 267 | 14% | | GEMM (sm90_xmma_gemm*, ~4000 calls) | 748 | 40% | | FlashAttention fwd+bwd | 160 | 8% | | elementwise / index / copy / cat (~16000 small kernels) | ~380 | 20% | | GPU idle (13963 gaps of ~9 us, no gap above 8 ms) | 126 | 7% | * 74% of the reduce-scatter and 55% of the all-gather overlap with compute; **266 ms/step of communication is exposed**, plus 126 ms/step of small bubbles between kernels. * The host is not the bottleneck:cudaLaunchKernel(26917 calls) takes 907 ms of *host* time over 2 steps, but that is the launch queue blocking (the host runs ahead and is throttled by the GPU); there is nocudaMalloc/cudaFreein the trace (the allocator does not thrash atNO_AC=21), and the CPU keeps only 5 of 96 cores busy. * The root FSDP unit (embed_tokens + lm_head, 1.09 B parameters) is all-gathered in 52 ms at the very start of the forward, fully exposed. * The int64 count all-reduces (total_ce_tokens,total_mse_tokens,total_samples) are 100% exposed: 15 ms/step of pure rank-skew barrier between the forward and the backward. Three changes were tried, each behind a switch, each measured withtime: * r16aBAGEL_FSDP_FORWARD_PREFETCH=1(FSDP1 prefetches the all-gather of the next unit during the current one):/tmp/r16a_time-> 1.8604 s/step (1.3979). *Neutral to slightly worse*: withlimit_all_gathers=0the all-gathers were already in flight. Reverted. * r16bBAGEL_FLASH_OUTBUF=1(the per-split FlashAttention outputs are written into one preallocated buffer instead of being concatenated at the end; theCatArrayBatchedCopyof the packed attention is 35 ms/step):/tmp/r16b_time-> 1.8666 s/step (1.3932). *Slightly worse* (the per-split copies cost more kernels than the one batched cat). Reverted. * r16cNCCL_PROTO=Simple(the collectives run the "LL" protocol):/tmp/r17_time-> 1.8506 s/step (1.4053), i.e. exactly the same as without it (peak reserved identical to the byte), so the NCCL protocol is not the limiter. Reverted (it also would have changed the gradient reduction order for nothing). Also reverted: aGpuBatchStreamprefetch of the next batch (it moves one batch too early, and gate 2b compares the *whole* list of batches the rank moved to the GPU, not only the first N) and thenon_blocking=Truecopies ofSimpleCustomBatch.cuda(with pageable memory the source tensors ofnested_attention_masksare freed right after the copy is issued -- a data race the reference does not have). ---
4.3 hRound 17 — the depth of the activation checkpointing, again (`NO_AC=14`)
time --config /tmp/r18_sw.json,/tmp/r18_time:s_per_step1.8518 (1.4043), peak alloc 94.73 GiB, peak reserved 134.84 GiB, butwindow_range0.067 (6.7%): the two 30-step windows differ by more than the 6% the final evaluation tolerates, so this configuration would be re-measured as INVALID.NO_AC=21(window range 1.0%) is both the fastest and the stable setting. Kept. ---
4.3 hRound 18 — full check of the best tree (PASS, N1)
Start: 2026-11-10 07:21Z. End: 07:48Z.bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r19_check(the tree is exactly commit82c039f; the switches are{"BAGEL_COMPILE_RMSNORM":"1","BAGEL_COMPILE_ROPE":"1","BAGEL_FSDP_LIMIT_ALL_GATHERS":"0","BAGEL_FLASH_ATTN":"1","BAGEL_NO_AC_FROM_LAYER":"21","BAGEL_FUSED_ADAMW":"1","BAGEL_MOT_FUSED":"1"}). * status PASS, tier N1,performance.speedup= 1.4051 (candidate 1.8508 s/step vs the frozen reference 2.6006 s/step; all four ranks agree). * gate 2 (record, 3 steps, deterministic):NOT_BITWISE,exact_items_ok: true, no problems (1223 parameters, 1408 shards per step compared). The compiled RMSNorm/RoPE, the FlashAttention decomposition and the fused AdamW change the arithmetic, so the hashes differ. * gate 2b (conformance, interleaved data): PASS / conforming — the pools did not move apart. * gate 3 (statistical, 150 steps, seed 1): PASS (all 14 endpoints inside 3 sigma). * peak alloc 78.45 GiB, peak reserved ~131 GiB; the two timing windows agree to 0.9%. ---
5.5 hRound 22 — profile of the current best tree (`/tmp/r22_profile`)
Profile of the tree at82c039fwith the round-19 switches (rank 0, steps 10-12, trace/tmp/r22_profile/timing_profile.trace.json, 253 MB). The step in the profiled run is ~1.77 s (profiler overhead); the machine's real numbers are 1.8508 s/step. Machine facts measured directly (not what the task statement says): the GPUs report asHopper, sm90, 132 SMs, 143 GiB, 4258 GB/s HBM and 806 TFLOP/s bf16 GEMM (measured with a 1 GiB copy and an 8192^3 bf16 matmul), 18 NVLinks of 26.5 GB/s each (nvidia-smi topo -m: NV18). The 4 ranks run in lockstep: the per-rank step-end timestamps of the last timing run differ by <5 us, so there is no persistent rank skew (the 28 ms "rank skew" of the round-14 note was an artifact of CUPTI inflating the NCCL kernel durations). Per-step accounting (3 profiled steps, rank 0, GPU kernel time): | bucket | ms/step | share | |---|---|---| | GEMM (sm90_xmma_gemm*, 1920 calls) | 788 | 48% | | FlashAttention fwd+bwd | 166 | 10% | | index gather/scatter (index_elementwise_kernel, 1483 calls) | 120 | 7% | | vectorized/unrolled elementwise | ~190 | 12% | | compiled RMSNorm+RoPE (triton, 1226 calls) | 54 | 3% | | foreach optimizer/EMA (multi_tensor_apply+RowwiseMoments) | 64 | 4% | | NCCL stream (all-gather 217 + reduce-scatter 120) | 337 | 20% (74% overlapped) | | GPU idle (18004 gaps of ~7 us, none > 8.4 ms) | 123 | 7% | Step structure (from the trace timeline): optimizer+EMA ~34-80 ms -> root all-gather ~44 ms (exposed, nothing else runs) -> forward ~420 ms -> 3 int64 count all-reduces -> backward ~1150 ms (= 2x the forward + the activation-checkpoint recompute of 21 of the 28 layers, ~315 ms) -> EMA. Exposed (not overlapped with the compute) GPU time per step: all-gather 46 ms, reduce-scatter 17 ms, the int64 count all-reduces 15 ms, the recompute's side-stream ops 76 ms. Conclusions used for this round: * The reduce-scatter is not bandwidth starved (each 932 MB RS moves 700 MB over the ring in 3.4 ms = 208 GB/s, the same rate as the all-gather); the earlier "37 GB/s" note was wrong. * The activation-checkpoint recompute (~315 ms/step, 19% of the step) is the largest single overhead, but it is blocked by memory:NO_AC=14allocated 94.7 GiB while the allocator had 134.8 GiB reserved (40 GiB of fragmentation) andNO_AC=0thrashed the allocator. => try the CUDA caching allocator's virtual-memory segments to remove the fragmentation.
5.5 hRound 22b — `NO_AC=14` re-measured with a CUDA allocator option (1.4745x)
time --config /tmp/r23_sw.json,/tmp/r23_time: **1.7637 s/step,speedup_vs_frozen_reference1.4745**, window range 0.0062 (stable), peak alloc 94.67 GiB, peak reserved 124.82 GiB. The only difference to round 17'sNO_AC=14run (/tmp/r18_time, 1.8518 s/step, window range 0.067 -> would have been INVALID in the final evaluation, reserved 134.84 GiB) isPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The allocator prints "expandable_segments not supported on this platform" (the flag is not applied), yet the *reserved* memory dropped by 10 GiB and the run became stable: the reserved/allocated gap is the allocator's fragmentation (30-40 GiB), and atNO_AC=14the model is close enough to the 143.7 GiB limit that the fragmentation decides whether the run thrashes or not. (Note:garbage_collection_thresholdandroundup_power2_divisions*are* supported by this torch build and are the next knob to try.) The gain is the activation-checkpoint recompute of 7 more decoder layers (7 x ~15 ms/step).
5.5 hRound 22c — `NO_AC=7` (the memory wall)
/tmp/r24_time(killed): with 7 more layers un-checkpointed the peak reaches 135-142 GiB of the 143.7 GiB card, the allocator starts to thrash (0.47 steps/s at step 20 vs 0.68 at step 7) and the per-step times become erratic.NO_AC=7is not viable with the default caching allocator. => next: the CUDA stream-ordered allocator (PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync), which keeps the freed memory in a driver-managed pool instead of caching size-mismatched blocks in the process, i.e. it removes the 30-50 GiB of fragmentation that makes the memory wall move around from run to run (and that caused the 5.2 s stall of round 17'sNO_AC=14run). ---
6.2 hRound 29 — the VAE encoding on a side CUDA stream (1.4473x)
Start: 2026-11-10 08:55Z. End: 09:12Z. Profile findings (from round 22, unchanged tree): the step is optimizer+EMA (34-80 ms) -> root FSDP all-gather ~44 ms, fully exposed -> forward ~420 ms -> 3 int64 count all-reduces -> backward ~1150 ms (= 2x forward + the activation-checkpoint recompute of 21 of the 28 layers, ~315 ms) -> EMA. The VAE encoding sat *before* the forward, so the GPU was idle during it and the root all-gather had nothing to overlap with. Change:modeling/bagel/bagel.py+train/pretrain_unified_navit.py— the VAE encoding runs on its own CUDA stream (torch.cuda.Stream), the packed images and the latents arerecord_streamed, andBagel.forwardwaits on the recorded event right before the latents are used (after the ViT). The encode is still exactly one call per step and still draws its noise inside it (the judge keys the injected noise by the ordinal of the encode call, not by the stream), so the data and the noise are unchanged. Off by default,BAGEL_VAE_OVERLAP=1. *time --config /tmp/r28_sw.json(/tmp/r28_time): **1.7969 s/step,speedup_vs_frozen_reference1.4473** (r27, without the overlap: 1.8125 / 1.4348) -> the overlap wins 15.6 ms/step. window range 0.0164, peak alloc 85.72 GiB, peak reserved 134.97 GiB. *check --quick --config /tmp/r28_sw.json(/tmp/r29_quick):exact_items_ok: true,NOT_BITWISE(expected, the tree is N1 overall), no problems: 1223 parameters and 1408 shards per step compared. * Commit9d1d724. ### Ruled out in this round (measured, not guessed) * The H2D copy of the batch is not a lever. A CPU-only run of the judge's data pipeline (/tmp/mask_size.py) shows that a batch is ~150 MB, of which **134 MB is the dense attention masks** (3-5 tensors of[sample_len, sample_len]float32; the samples are 2.3k-3.9k tokens). Micro-benchmark (/tmp/h2d_bench.py): 134 MB pageable H2D = 7.2 ms, pinned = 2.6 ms, and pinning costs 0.4 ms. So the whole transfer is ~8 ms of the 1.8 s step; even a perfect overlap saves <1%. * **expandable_segments:Trueis *not* applied by this torch build** (the warning comes fromPYTORCH_CUDA_ALLOC_CONF_EXPANDABLE_SEGMENTSnot being a compile-time flag in the wheel), so round 17'sNO_AC=14instability (6.7% window range) vs round 22b's stable run (0.62%) was run-to-run variance, not the allocator option.NO_AC=14stays rejected: the final evaluation uses another data seed and a 6.7% window range is INVALID (score 0). ---
6.8 hRound 30 — profile of the tree with the VAE overlap (`/tmp/r30_profile`)
bench.py profile --tree /submission/bagel_sft --config /tmp/r28_sw.json --out /tmp/r30_profile --from 10 --iters 3(rank 0, steps 10-12, trace/tmp/r30_profile/timing_profile.trace.json, 250 MB; the profiled step is ~1.78 s). Per-step kernel time (3 steps / 3 = per step; the profiler adds ~15%): | bucket | ms/step | share | |---|---|---| | GEMM (sm90_xmma_gemm*) | 763 | 39% | | NCCL (all-gather 349 + reduce-scatter 120) | 469 | 24% (mostly overlapped) | | elementwise (vectorized + unrolled + elementwise) | 286 | 15% | | FlashAttention fwd (58) + bwd (98) | 156 | 8% | | index gather/scatter (index_elementwise_kernel) | 115 | 6% | | optimizer + EMA (multi_tensor_apply,RowwiseMoments) | 55 | 3% | | compiled RMSNorm/RoPE (triton) | 52 | 3% | | GPU idle (union of all streams) | 136 | 7% | Where the GPU idle comes from (this is the interesting part): * The idle is *not* spread over the step: two windows of 35 ms and 71 ms where no kernel runs on any stream (offsets 2184-2297 ms of the trace). During them the CPU is insideaten::linear/aten::addmm, and the CUDA runtime log shows 169cudaFreecalls (40.8 ms) — the caching allocator's garbage collection: it cannot serve a request (the large buffers of the MoT MLP change size every step, because the number of understanding/generation tokens does), it releases *all* its cached blocks — everycudaFreesynchronizes the device — and retries. In the same 3 steps: 98 + 40cudaMalloccalls (255 ms + 25 ms). * The remaining ~60 ms of idle are the ~8 us gap between consecutive kernels (16 700 kernels per step, the GPU's launch latency). Per-op CPU cost (from the profiler's own table):aten::_index_put_impl_938 calls/step andaten::index716 calls/step;aten::fill_1891 calls/step (41 ms of GPU time);aten::copy_2890 calls/step. ### Ruled out in this round (measured) *PYTORCH_CUDA_ALLOC_CONF=roundup_power2_divisions:4(r31,/tmp/r31_time): the reserved memory did not go down (134.94 GiB vs 134.97) and the allocated memory went up by 5 GiB (the rounding is applied to the block size), so the step got *slower*: 1.8436 s/step, 1.4106. A micro-benchmark of the allocation pattern (/tmp/alloc_sim.py) had shown a 2.7x smaller reserved memory, but the real workload has many more distinct sizes. Reverted. *torch.cuda.empty_cache()after the setup (r32,/tmp/r32_time): 1.7984 s/step, 1.4460, window range 0.0 — neutral, and the reserved memory was unchanged (134.89 GiB). So the fragmentation is *not* left over from the model/EMA/optimizer setup, it is created by the training loop itself. Reverted. *index_copy_instead ofx[idx] = v(/tmp/idx_bench.py,/tmp/idx_bench2.py): the gradients are bitwise identical, and the forward is faster (0.088 ms vs 0.130 ms for a [10240, 3584] scatter), but the backward ofindex_copy_(index_select) is slower than the backward ofindex_put_(index), so forward+backward is *worse* (0.282 vs 0.262 ms).index_selectis also slower on the GPU than advanced indexing (0.193 vs 0.131 ms). Rejected. *scatter_with an expanded index: the same forward speed asindex_copy_but it needs the index expanded to the full [n, d] shape; not worth the memory. ---
7.2 hRound 33 — the EMA update on its own CUDA stream (1.4694x)
Start: 2026-11-10 09:40Z. End: 09:54Z. Change (c78b3d2, switchBAGEL_EMA_OVERLAP, off by default):train/pretrain_unified_navit.py— the EMA update runs on a second CUDA stream (ema_stream) instead of the compute stream. It is enqueued *after* the logging block (whosetorch.cuda.synchronize()would otherwise wait for it), and the compute stream waits for it at the next step'sloss.backward()(FSDP rewrites the flat parameters when it reshards its units at the end of the backward pass). The judge reads the EMA's flat parameters in the optimizer post-hook, i.e. before the current step's EMA update, exactly as in the reference. *time --config /submission/switches.json(/tmp/r33_time): **1.7698 s/step,speedup_vs_frozen_reference1.4694**, window range 0.0148, peak alloc 85.72 GiB, reserved 134.96 GiB.
7.2 hRound 34 — full check of the tree with the VAE and the EMA on their own streams: **INCORRECT**
Start: 2026-11-10 09:54Z. End: 10:07Z.bench.py check --tree /submission/bagel_sft --config /submission/switches.json --out /tmp/r34_check(the tree is exactly commitc78b3d2; the switches are the nine of/submission/switches.json). * gate 1 (timing): 1.4499x (candidate 1.79 s/step vs the frozen reference 2.6006 s/step) — the benefit gate passes. * gate 2 (record, 3 steps, deterministic):NOT_BITWISE,exact_items_ok: true, no problems. * gate 2b (conformance, interleaved data): INCORRECT / nonconforming. The per-token MSE of the candidate becomes non-finite from the 5th forward on (rank 0: forwards 0-3 finite, 4-15 all NaN; the same on the other ranks), and the CE with it: **the model diverges to NaN after three optimizer steps** on the interleaved data.R_N = R_J = R_B = nan(the reference's own numbers are finite), the per-batch values reach 16.7 (N pool) and 9.6 (J pool). The production-mode run was not even reached. * gate 3 was not run. Reading: the divergence is *not* a small numerical difference — the parameters become NaN. With the learning rate still in its warm-up (2e-5 x step/2000, i.e. ~3e-8 at step 3) the parameters cannot diverge numerically in three steps, so something *writes* garbage into them: a race between the two new side streams and the compute stream. The two candidates are the VAE encoding onvae_stream(the images are read by the VAE on a side stream while the compute stream runs) and the EMA update onema_stream(it reads the model's sharded flat parameters while the next step's forward all-gathers them). Bisection started in round 35 (a direct run of the judge's conformance gate on the interleaved data configuration,bench/conform.py'sconform_run, with one of the two switches turned off).
7.9 hRound 35 — the VAE side stream: root cause hunt, and the decision to drop it
Start: 2026-11-10 10:31Z. End: 11:05Z. The VAE-on-side-stream tree diverges to NaN on the interleaved data (gate 2b). Three experiments, all with the judge's own gate-2b runner (/tmp/run_conform.py, 8-16 steps): * VAE off, EMA on (/tmp/cf_vaeoff): no NaN; the manual statistic against the frozen reference isr_n=0.0089, r_j=0.0069, r_b=0.0056, bound 0.0169, ratios 1.58/1.22 -> conforming. The EMA overlap is fine. * VAE on, EMA off (/tmp/cf_emaoff): NaN again -> the VAE stream is the culprit. * VAE on + a new diagnostic (BAGEL_VAE_DEBUG3,/tmp/cf_vaedbg3): the diagnostic captures the VAE's internalmean(through atorch.randn_likespy) during the side-stream encode and re-computes the encoder on the compute stream afterwards. Result:mean_diff=0.0(the VAE's internals are bitwise identical), the input images' checksum is unchanged, the reconstructed latent matches the stored one to 1.4e-6 (bf16 rounding only), and the run did not diverge. The added diagnostic work changes the timing, so the divergence is timing sensitive. Everything that could be checked was checked and came out clean: the stream ordering (wait_stream/wait_event/record_stream), the H2D copy (the images are pinned, butTensor.to(device)withnon_blocking=Falseis synchronous), the RNG (the judge keys the noise by the call's ordinal, so the stream cannot change it), the VAE's weights (frozen, never freed) and the autocast cache (the frozen weights are not cached). **Conclusion: an unexplained, timing-sensitive race between the VAE's side stream and the compute stream.** Two full checks have rejected it. The overlap is worth 2.4% of the speedup, which is not worth an unexplained race, so the VAE side stream is switched off. The code stays in the tree behindBAGEL_VAE_OVERLAP(default off) for the record. *time --config /submission/switches.json(/tmp/r35/time, EMA overlap only): 1.8128 s/step,speedup_vs_frozen_reference1.4346,beyond_reference_noise: true, peak allocated 85.72 GiB, reserved 134.22 GiB,cpu.busy_cpus_in_windows5.04.
7.9 hRound 36 — where the remaining time goes (profile analysis of the r30 trace)
The r30 trace (/tmp/r30_profile/timing_profile.trace.json, one step, 1823 ms) was re-analysed with the GPU kernels correlated to their CPU ops (/tmp/an_corr2.py): * compute stream 1403 ms busy, 18 765 kernels; NCCL 610 ms (all-gather 464, reduce-scatter 146, 95 ms exposed); VAE stream 135 ms; GPU union busy 1710 ms of 1823 ms -> 113 ms idle in 14 701 small gaps (7.7 us each). * the idle is CPU-bound:cpu.busy_cpus_in_windowsis 5.04 with 4 ranks, i.e. every rank's python thread is ~100% busy. The step is limited by the kernel-launch rate, not by the GPU. * the compute stream by op:mm747 ms, flash attention 170 ms, **index_put_73 ms (1964 calls),index63 ms (716 calls)**,fill_38 ms (1661),copy_22 ms (2265),add_38 ms (1590),mul30 ms,silu13 ms, fused AdamW 34 ms, the compiled norms/RoPE 41 ms. So the two levers that are left are (a) the number of kernel launches the CPU has to issue (the MoT packing alone is 2680 launches and 136 ms of GPU time) and (b) the 95 ms of exposed NCCL at the start of the forward, which is exactly what the (broken) VAE side stream was meant to cover.
8.3 hRound 37 — full check of the EMA-only tree: **PASS, tier N1, 1.437x**
Start: 2026-11-10 10:53Z. End: 11:20Z. Config:/submission/switches.json(8 switches,BAGEL_VAE_OVERLAPremoved — the VAE side stream stays off). Full check,/tmp/r36/check. * gate 1 (timing): candidate 1.8097 s/step, frozen reference 2.6006 -> speedup 1.437,beyond_reference_noise: true, window range 1.8 %, peak allocated 85.72 GiB. * gate 2 (record):exact_items_ok: true,NOT_BITWISE(expected: FlashAttention instead of the dense-mask SDPA changes the arithmetic), 3 steps, 1408 shards/step compared. * gate 2b (conformance, interleaved data): deterministic `r_n=0.00762, r_j=0.00653, r_b=0.00646`, bound 0.01938, ratios 1.18/1.01 -> conforming; productionr_n=0.00755, r_j=0.00693, r_b=0.00621, ratios 1.22/1.12 -> conforming. * gate 3 (trajectory, seed 1, 150 steps): every endpointwithinthe 3 sigma bound (the largest islog_grad_norm_s050_100at 1.60 sigma) -> PASS.status: PASS,correctness.measured_tier: N1. This is the tree to keep.
8.3 hRound 38 — batched varlen FlashAttention for the packed attention
Start: 2026-11-10 11:20Z. The profile says the step is CPU-bound (every rank's python thread is ~100 % busy, 113 ms of GPU idle in 14 701 small gaps per step, 11.5 k allocations/step -> 55 409cudaEventQuerycalls/step in the allocator). The packed attention issues **one FlashAttention call per split per layer** (840 forward + 1008 backward calls per step) plus thetorch.catof the visible keys of every split. Idea: the splits of a sample are contiguous and a split sees the *non-noise* splits before it plus itself, so the visible keys of a split are the prefix of its sample **unless a noise split precedes it. All the prefix splits of a layer can therefore be computed by a singleflash_attn_varlen_funccall per mask kind** (causal / not): the queries of the splits are gathered into one buffer, their visible keys into another, and the split boundaries become the cumulative sequence lengths. The splits that follow a noise split keep the per-split path (their visible keys have a hole). The plan depends only on the split structure, which is the same for every layer of the step, so it is built once per step and cached. Implementation:modeling/bagel/qwen2_navit.py, behindBAGEL_VARLEN_ATTN(default off).
8.3 hRound 38, first attempt — the bug in `_concat_ranges`
The first implementation of the varlen attention was wrong:_concat_rangeswrote the offsets1..length-1instead ofbase+1..base+length-1, so the second and later samples of a group pointed at the wrong rows. The unit test caught it (max|diff|= the signal scale). Fixed.
8.3 hRound 38 — the batched varlen FlashAttention: **1.4769x**
Start: 2026-11-10 11:20Z. End: 12:25Z. Why. The profile of the step (r30 trace,/tmp/r30_profile/timing_profile.trace.json, one step of 1823 ms) says the step is GPU-work-bound, not launch-bound: the union of the busy intervals of all the streams is 1710 ms of the 1823 ms step (the GPU is 94 % busy) and the idle is 113 ms spread over 14 700 gaps of ~7 us. The compute stream (1403 ms) is: | op | ms/step | calls/step | |---|---|---| |aten::mm(the GEMMs) | 746.8 | 2608 | | FlashAttention backward | 105.5 | 1008 | |aten::_index_put_impl_| 73.1 | 1964 | | FlashAttention forward | 65.0 | 690 | |aten::index| 63.1 | 716 | |aten::fill_| 38.3 | 1661 | |aten::add_| 38.1 | 1590 | and the GEMMs run at 652 TFLOPS while a plainlinearof the same shape reaches 730 TFLOPS on this GPU, i.e. the GEMMs have no headroom — the attention and the packing do. The attention was the biggest reducible item. Upstream's packed attention needs **one FlashAttention call per split of every sample** (690 forward + 1008 backward kernels per step), and every one of them slices the packed query/key/value, so the **backward of every slice allocates a zero buffer the size of the whole packed tensor** (aten::slice_backward->aten::zeros): 1232 of the 1661fill_calls and 1232 of the 2664copy_calls of the step come from there, and they cost ~50 GB of memset per step. The change. The visible keys of a split are the *non-noise* splits before it plus the split itself, so for every split that is not preceded by a noise split they are exactly the prefix of its sample. All the prefix splits of a layer can therefore be computed by **oneflash_attn_varlen_funccall per mask kind** (causal / not): the queries of the splits are gathered into one buffer, their visible keys into another, and the split boundaries become the cumulative sequence lengths. The splits that follow a noise split keep the per-split path. Two details that had to be right: * FlashAttention's varlen causal mask is aligned per sequence to the bottom right (verified againstflash_attn_funcwithseqlen_q != seqlen_k: bitwise identical), which is exactly the mask of a causal split. *index_select's backward adds the rows with atomics, so the rows that several splits share are summed in an arbitrary order — not reproducible even withtorch.use_deterministic_algorithms(True). The gather therefore uses a custom autograd function whose backward accumulates withindex_put_(accumulate=True)(sorted, reproducible), and the scatter uses one whose backward is a plain gather.modeling/bagel/qwen2_navit.py, behindBAGEL_VARLEN_ATTN(default off). Validation (unit test, 3 layouts,out/dq/dk/dvagainst the per-split path):all prefixandsingle splitare bitwise identical,noise in the middlediffers by 4e-3 relative indk/dvonly (the bf16 accumulation of the shared key rows). *time --config /tmp/sw_varlen.json(/tmp/r38/time): **1.7608 s/step,speedup_vs_frozen_reference1.4769**,beyond_reference_noise: true, window range 0.5 %, peak allocated 86.2 GiB, reserved 134.3 GiB. (The EMA-only tree: 1.8128 s/step, 1.4346x.) * the judge's own conformance runner on the interleaved data (/tmp/cf_varlen): conforming —r_n=0.00768, r_j=0.00654, r_b=0.00612, bound 0.01837, ratios 1.25/1.07.
9.1 hRound 39 — the full check of the tree with the varlen attention
Start: 2026-11-10 11:57Z.check --tree /submission/bagel_sft --config /submission/switches.json(/tmp/r39/check), withBAGEL_VARLEN_ATTN=1added to the switch file. Result:status: PASS,measured_tier: N1,performance.speedup1.462** (1.7788 s/step against the frozen reference's 2.6006).** * gate 1 (timing): 1.7788 s/step, window range 0.94 %,beyond_reference_noise: true, peak allocated 86.2 GiB, reserved 134.0 GiB,cpu.busy_cpus_in_windows5.09. * gate 2 (record):exact_items_ok: true,NOT_BITWISE(the FlashAttention arithmetic). * gate 2b: deterministicr_n=0.00768, r_j=0.00654, r_b=0.00612(bound 0.01837, ratios 1.25/1.07) -> conforming; productionr_n=0.00760, r_j=0.00657, r_b=0.00601(ratios 1.26/1.09) -> conforming. * gate 3 (trajectory, seed 1): every endpoint inside the 3 sigma bound (the largest islog_grad_norm_s050_100at 2.05 sigma) -> PASS. **The submission is now the tree of commited48c34withBAGEL_VARLEN_ATTN=1in/submission/switches.json: 1.462x (the standalonetimerun measured 1.4769x).**
9.8 hRound 40 — the varlen attention also covers the splits whose visible keys have a hole
Start: 2026-11-10 12:20Z. End: 2026-11-10 12:50Z. Profile findings (r30 trace, 1 step = 1823 ms, GPU busy 94 %): the step is GPU-work bound. Per step: GEMMs ~674 ms (at 652 TFLOPS, i.e. no headroom), all-gathers ~307 ms and reduce-scatters ~106 ms (23 % of the step in communication), FlashAttention ~138 ms, and ~305 ms in the *index* round trips of the MoT expert path (aten::index54 ms,aten::index_put_61 ms,aten::copy_56 ms,aten::cat31 ms,aten::fill_35 ms,aten::add_37 ms). Change:_build_varlen_plannow makes *every* split one sequence of a varlen call, not only the splits whose visible keys are the prefix of their sample. A split that follows a noise split sees its visible keys with a hole (the noise span is skipped); the plan gathers the visible ranges in the same order as the per-split path, so the key buffer of the sequence — and therefore the bottom-right aligned causal mask — is unchanged. The fix that made it work:cu_seqlens_kmust hold the *per-sequence* key lengths (the sum of the ranges of a split), not the per-range lengths. Validation (unit test, 3 layouts):outanddqare bitwise identical to the per-split path in all three layouts;dk/dvdiffer by ~4e-3 relative (1 bf16 ulp) only in the layout with a hole. Micro-benchmark of the hole layout: 10.24 ms -> 6.88 ms (1.49x). *time --config /submission/switches.json(/tmp/r40/time): **1.7581 s/step,speedup_vs_frozen_reference1.4792** (round 39: 1.4769),beyond_reference_noise: true. The gain is small (~0.2 %, inside the 0.25 % reference noise) because the timing data is the interleaved one, in which only a few splits per batch have a hole.