Claude Opus 5.5 2.20×
Harness: Claude Code · switches: BAGEL_OPT_FLASH_ATTN=1 BAGEL_OPT_MOT_CONTIG=1 BAGEL_OPT_FUSED_ROPE=1 BAGEL_OPT_FUSED_SWIGLU=1 BAGEL_OPT_FUSED_NORM=1 BAGEL_OPT_NO_CKPT_VIT=1 BAGEL_OPT_NO_CKPT_LAYERS=28 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True BAGEL_OPT_SWIGLU_RECOMPUTE=1 BAGEL_OPT_TRITON_ADAMW=1 BAGEL_OPT_FUSED_EMA=1 BAGEL_OPT_FUSED_VAE_GN=1 BAGEL_OPT_FA3=1 BAGEL_OPT_FUSED_QKV=1 BAGEL_OPT_VIT_ONE_UNIT=1 BAGEL_OPT_PLAN_EARLY=1 BAGEL_OPT_NO_VIT_SYNC=1 BAGEL_OPT_VAE_BIAS=1 BAGEL_OPT_VAE_PREFETCH=1 BAGEL_OPT_EMA_STREAM=1
- Judged
- 2.20×
- Tier
- N1
- Self-best
- 2.22×
- Rounds
- 8
- Hours
- 9.3
- API cost
- $48.9
- Input tokens · 99.4% cached
- 152 M
- Output tokens
- 0.6 M
- Requests
- 295
§ 02Key levers
- Attention kernel swap (split by mask structure)
- ✓
BAGEL_OPT_FLASH_ATTN: 2.606 → 2.213 s; then switched to FA3 (−52 ms) - Drop activation recompute (28 decoder layers)
- ✓ all 28 layers + ViT removed.
BAGEL_OPT_SWIGLU_RECOMPUTEcuts per-layer footprint to 1.48 GiB, 1.727 → 1.417 s - Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
- ✓ the two token types each stored contiguously + Triton RoPE / SwiGLU (bitwise) + RMSNorm, 2.213 → 1.727 s
- Optimizer, EMA, step start / end
- ✓ bitwise Triton AdamW + EMA (1.417 → 1.357 s); VAE GroupNorm fusion (−41 ms); next batch's VAE encoded during backward (−17 ms); EMA on a side stream (−11 ms)
- Other main levers
- QKV fusion (1.264 → 1.233 s), ViT merged into one FSDP unit (−10 ms), host sync before ViT removed, VAE conv bias (−5.6 ms)
§ ARaw trace
Raw work log8 sections
0.4 hRound 1 — profile of the unchanged reference; structure-aware FlashAttention (19:17 – )
Profile (/tmp/r1/profile, steps 8–9 on rank 0, torch.profiler): - Over the two steps: compute stream busy 4.19 s, compute ∪ NCCL 5.30 s; 1.39 s of compute-stream gaps (0.62 s during ReduceScatter, 0.58 s during AllGather, 0.07 s AllReduce, 0.11 s none). - NCCL in isolation (own torchrun test,/tmp/work/nccl_bw.py): a 932 MB reduce-scatter / all-gather takes 2.3 ms (~430 GB/s algbw); in training the same calls average 13.6 ms → the "communication" is ranks waiting for the slowest rank (per-rank token counts 6.3K–11.3K, per-sample attention cost quadratic), not link bandwidth. - Step 8 (rank 0 light): 1.18 s of gaps. Step 9 (rank 0 the heaviest rank) — the critical path: busy 2631 ms = attention (mem-efficient SDPA, cutlass sm80 kernels, dense bf16 masks, every tile visited) 881 ms (33.5%), GEMM 820 ms (31.2%), elementwise 499 ms (19.0%), index gather/scatter 181 ms (6.9%), foreach (AdamW, EMA) 90 ms (3.4%), reductions 43 ms, cat 28 ms. Fwd busy 753 ms, bwd (with recompute) 1807 ms, opt+EMA 71 ms. - ViT forward is launch/CPU-bound: each of the 26 small SigLIP layers waits ~1.3 ms for its own all-gather (no forward prefetch). - Data layouts (interleave config, own CPU probe/tmp/work/data_probe.py): 3–6 samples per rank, e.g.f1026+f1371+c12+n1026(editing; two adjacent bidirectional spans),c17+n2306(t2i),f842+c30+c5+...(VLM), editing chains with a noise span followed by further splits. Hotspot: attention (kind 2/3: backend + a new opportunity — masked tiles skipped). Change (switchBAGEL_OPT_FLASH_ATTN=1,modeling/bagel/packed_flash.py): per batch, the dense masks are analysed on the GPU (hi[q] = last visible key, col_hi[k] = last row seeing key k; accepted only if the mask equals the mask rebuilt from these, and holds only 0/−inf, else upstream's path), cut into causal / bidirectional row groups, and each layer runs one causal and one non-causalflash_attn_varlen_funcover permuted queries and gathered keys (native GQA, no K/V repeat, no mask reads). Duplicated keys' gradients summed in fp32 in a fixed order. Unit test (/tmp/work/test_flash.py, real + interleaved + random layouts): error vs fp64 gold on par with the reference SDPA path (out/dq/dk/dv rel. error 2–4e-3, worst ratio new/ref 1.23); strict deterministic mode runs and repeats bitwise. Numerics tier: N1 (different kernel).time(/tmp/r1/time_flash): speedup_vs_frozen_reference 1.1775 (2.2132 s/step; windows 2.2006 / 2.2258; peak 82.1 GiB; no fallback logged). Full check (/tmp/r1/check_flash/check.json, switch file ={"BAGEL_OPT_FLASH_ATTN": "1"}): **PASS, N1, speedup 1.1766** (28 min). Gate 2b: R_N 0.0073, R_J 0.0050, R_B 0.0056 → ratios 1.31 / 0.90 (bound 3). Gate 3: all 14 endpoints within 3σ; largest |d|/σ: log_grad_norm_s100_150 1.33, disp_vit_s050_100 1.22, disp_vit_s100_140 1.01, disp_other_s050_100 0.99 (CE / MSE endpoints < 0.03σ). Note for later rounds: grad-norm and displacement endpoints are already ~1σ off after one N1 change, so every further N1 stack needs its own full check before it replaces the submission. →/submission/switches.json={"environment": {"BAGEL_OPT_FLASH_ATTN": "1"}}. Round 1 ended 20:23.
1.4 hRound 2 — memory-bound class: contiguous MoT layout + fused RoPE / SiLU·up / RMSNorm (20:23 – 20:41)
Profile of the round-1 best (/tmp/r2/profile, flash on): busy over 2 steps 3419 ms: GEMM 1619 (47%), elementwise 694 (20%; direct_copy 170 = fp32 casts of the RMSNorms, fills 68), index gather/scatter 366 (10.7%; MoT routingx[und_idx],y[und_idx] = ...and their sort-based backward), flash attention 338 (9.9%), foreach 181 (5.3%), reductions 82, cat 29. Heavy step: fwd busy 620 ms, bwd 1352 ms, opt 71 ms. Changes (dev worktree/tmp/dev, branchdev, commit e8f0f9c; each behind its own switch): -BAGEL_OPT_MOT_CONTIG=1: the decoder carries the understanding rows (upstream'spacked_und_token_indexesorder) and the generation rows as two contiguous tensors;Bagel._forward_contigbuilds them bycat(no scatter), the attention plan is re-expressed in row coordinates, the losses read the same rows via the inverse permutation. Unit test (/tmp/work/test_contig.py, 2 real-size MoT layers, deterministic FA): outputs, input grads and all 58 parameter grads bitwise identical to the FA path. -BAGEL_OPT_FUSED_ROPE=1,BAGEL_OPT_FUSED_SWIGLU=1(modeling/bagel/fused_ops.py, Triton): reproduce upstream's bf16 rounding points (libdevice expf, div_rn, the fma nvcc contracts in silu's backward): bitwise identical fwd + bwd (/tmp/work/test_fused.py). -BAGEL_OPT_FUSED_NORM=1: Triton RMSNorm fwd/bwd; ~1e-5 of outputs differ in the last bit (variance reduction order), weight grads accumulated in fp32 without per-product rounding (closer to fp64 than upstream): N1.time: contig+RoPE+SwiGLU (/tmp/r2/time_r2b): 1.3677 (1.9053 s/step); + fused RMSNorm (/tmp/r2/time_r2c): 1.509 (1.727 s/step; windows 1.724 / 1.730; peak alloc 82.1 GiB, reserved 118 GiB; no fallback). Attribution: FA −0.393 s, contig+RoPE+SwiGLU −0.308 s, RMSNorm −0.178 s. Full check of r2c launched 20:41 (/tmp/r2/check_r2c, PID ofsetsidjob in/tmp/r2/check_r2c.log). Full check (/tmp/r2/check_r2c/check.json): PASS, N1, speedup 1.5035 (20:41 → 21:07). Gate 2b ratios: det N 1.43 / J 1.02, prod N 1.40 / J 1.01. Gate 3 max |d|/σ = 1.56 (log_grad_norm_s100_150), all others < 1.4. The log-grad-norm deviation was positive in both checks; per-step series (traj_cand_s1.r0.jsonvsref_runs.json): candidate above reference on 72/150 steps, mean log diff driven by a few spikes — chaotic noise, not a bias. Data loader alone (/tmp/work/data_speed.py, 1 worker): 0.29 s/batch mean, 0.47 s max — not a bottleneck yet. → submission branchoptfast-forwarded to e8f0f9c (judge tree digest accc798d… = the checked digest);/submission/switches.json= FLASH_ATTN + MOT_CONTIG + FUSED_ROPE + FUSED_SWIGLU + FUSED_NORM. Round 2 ended 21:17.
2.4 hRound 3 — recompute class: selective activation checkpointing (21:17 – )
Profile of the round-2 best (/tmp/r3/profile): busy over 2 steps 2590 ms: GEMM 1555 (60%), flash attention 353 (13.6%), foreach (AdamW + EMA + clip) 181 (7%), elementwise 151 (5.8%), fused Triton kernels 120 (4.6%), index (attention permutations) 107 (4.1%). Heavy step: busy 1527 ms of 1828 ms. The recomputation of checkpointed layers is ~1/4 of the GEMM time and ~1/2 of the attention forward. Peak allocated 82 GiB = foreach AdamW temporaries; during forward the static memory is only params + Adam + EMA (grads are freed by zero_grad). Changes (dev commits 56a6114, 516577c; all N0 — the recomputation reproduces the forward bitwise, so not recomputing changes no number): -BAGEL_OPT_NO_CKPT_LAYERS=K(last K decoder layers not checkpointed),BAGEL_OPT_NO_CKPT_VIT=1(ViT layers and connector not checkpointed);PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueagainst fragmentation. -BAGEL_OPT_SWIGLU_RECOMPUTE=1: down_proj(silu(gate)·up) as one autograd function that keeps gate/up only and recomputes the product in backward; GEMMs are autograd's own for F.linear (dy.mm(W),dy.t().mm(s)): bitwise identical output and x / W_gate / W_up / W_down grads (/tmp/work/test_swiglu_down.py). −0.38 GiB/layer.time: K=12 + ViT (/tmp/r3/time_r3a): 1.6651 (1.5651 s/step; alloc 90.5 GiB, reserved 134.9 GiB without expandable segments); K=20 + ViT + expandable (/tmp/r3/time_r3b_es): 1.7499 (1.4892; alloc 105.2, reserved 123.9); K=28 + ViT + expandable + SwiGLU recompute (/tmp/r3/time_r3c): 1.8388 (1.4172 s/step; windows 1.427 / 1.407; alloc 108.8 GiB, reserved 127.5 GiB of 139.8). Per un-checkpointed layer: ~1.84 GiB (1.48 with SwiGLU recompute), ~9.5 ms saved. Full check of r3c launched 21:42 (/tmp/r3/check_r3c). Full check (/tmp/r3/check_r3c/check.json): PASS, N1, speedup 1.843 (21:42 → 22:07). Gate 2b det numbers identical to round 2's (R_N 0.00713, R_J 0.00509, R_B 0.00457: the round-3 changes are N0 on that path); prod ratios 1.43 / 1.03. Gate 3 max |d|/σ 0.96 (disp_vit_s050_100). Gate 3 runs in production mode (bench/traj/frozen.jsonspecdeterministic: False): every run of the same tree is a fresh draw, so a passing check does not guarantee the final one. Over the three checks so far the noisy endpoints (grad norm, displacements) have |d|/σ RMS ≈ 0.75 (max 1.56); CE / MSE < 0.05σ. →optfast-forwarded to 516577c (digest 4df53186… = checked);/submission/switches.json= r3c (round-2 set + NO_CKPT_VIT, NO_CKPT_LAYERS=28, SWIGLU_RECOMPUTE, expandable_segments). Round 3 ended 22:18.
3.1 hRound 4 — optimizer / EMA class (22:18 – 22:24)
Profile basis: round-3 profile — foreach kernels (AdamW foreach + EMA mul/add + clip) 7% of busy time, ~90 ms/step; foreach AdamW also allocates a 14.6 GB sqrt temporary. Changes (dev commits 56a6114, 999cfa9; both N0): -BAGEL_OPT_TRITON_ADAMW=1:fused_ops.TritonAdamW(torch.optim.AdamW)keeps upstream's state (hoststep, fp32 moments;_init_groupreused) and replaces the foreach chain by one Triton pass per tensor with the same rounding: lerpm + w·(g−m)as fma,v·β2thenv + (1−β2)·(g·g)as fma, sqrt_rn, div_rn by √bc2, +eps, addcdiv as fma; bias corrections in double on the host as upstream, scalars rounded to fp32. All 8 fma/no-fma variants tested againsttorch.optim.AdamW(5 steps with the warmup schedule,/tmp/work/test_adamw.py): the all-fma variant is bitwise identical (params, exp_avg, exp_avg_sq, step). Out-of-scope groups use the functionaltorch.optim.adamw.adamw(no second optimizer step / hooks). -BAGEL_OPT_FUSED_EMA=1: EMAfma(1−d, new, ema·d)in one pass: bitwise identical to_foreach_mul_+_foreach_add_(alpha)(/tmp/work/test_ema.py; the non-fma variant is not). - Attention plan upload through a persistent double-buffered pinned staging buffer (was ~4 ms of pin_memory per step).time(/tmp/r4/time_r4a, r3c + TRITON_ADAMW + FUSED_EMA): 1.9208 (1.3567 s/step; windows 1.3596 / 1.3538; alloc 108.8 GiB, reserved 127.5 GiB). Not yet full-checked (N0 on top of the checked r3c).
3.6 hRound 5 — communication / launch class and the step boundary (22:24 – 22:52)
Profile of r4a (/tmp/r5/profile): heavy step busy 1193 ms, gaps 257 ms (with profiler overhead): first root all-gather wait 21 ms, loss all-reduce waiting for a rank that is slower in *forward* 39 + 8 ms, 26 ViT layers ~1.3 ms each (no forward prefetch), ~0.9 ms per decoder layer boundary in backward: the CPU sits ~20 ms in FSDP's all-gather rate limiter (_pre_backward_prefetch → rate_limiter → cudaEventSynchronize) and the GPU idles while the hooks + launches catch up. Step start: the VAE encode takes 22–99 ms per rank-step (padded batch), 44% elementwise casts/SiLU, 13.5% GroupNorm statistics, 11.4% NCHW↔NHWC transforms, only 17% convolution. Changes tried (all N0, dev commits aaf9ca9, 01121b0): -BAGEL_OPT_AG_INFLIGHT=4(rate-limiter depth):/tmp/r5/time_r5a1.9238 (1.3546 s/step, reserved 130 GiB) → no gain beyond noise (the profiler's CPU overhead exaggerated those gaps). - +BAGEL_OPT_FWD_PREFETCH=1:/tmp/r5/time_r5b1.9303 (1.3500, reserved 131.3 GiB) → +0.5%, not worth +3.8 GiB. -BAGEL_OPT_VAE_STREAM=1(VAE encode on a side stream from the batch's H2D on, wait before the latents are read):/tmp/r5/time_r5c1.8904 (slower, 1.3785 s/step; reserved 134.5 GiB: side-stream allocations sit in their own allocator pool; concurrent VAE kernels likely delay the latency-bound NCCL calls at the step start). Dropped. VAE microbenchmark (/tmp/work/vae_bench.py, (4,3,768,1024)): eager 113 ms, channels_last 166 ms (GroupNorm converts back), torch.compile 64 ms but 33 s cold compile per run + recompile risk → next: fused GroupNorm+SiLU. Full check of r4a launched 22:52 (/tmp/r4/check_r4a). Full check of r4a (/tmp/r4/check_r4a/check.json): PASS, N1, speedup 1.9176 (22:52 → 23:17). Gate 2b det numbers identical to rounds 2–3 (N0 confirmed again); prod ratios 1.43 / 1.07; gate 3 max |d|/σ 0.93. →optfast-forwarded to 01121b0 (digest bb9a3508… = checked);/submission/switches.json= r4a (r3c + TRITON_ADAMW + FUSED_EMA). Round 5 ended 23:23.
4.3 hRound 6 — VAE encode (frozen module) and attention kernel (23:23 – )
Basis: round-5 profile — VAE encode 22–99 ms per rank-step on the step's critical path (memory-bound: casts, SiLU, GroupNorm statistics); flash attention 13% of busy time. Changes (dev commit f75d80b): -BAGEL_OPT_FUSED_VAE_GN=1(fused_ops.group_norm_act, used in the VAE encoder's ResnetBlock / AttnBlock / norm_out under bf16 autocast without grad): fp32 GroupNorm statistics in two passes (mean, centred sum of squares; fixed chunks, deterministic), y = x·a + b, optional SiLU, stored once in bf16 (the value the next autocast conv casts to). N1./tmp/work/vae_gn_test.py: encoder 113 → 64 ms on (4,3,768,1024), 14.6 → 6.6 ms on (1,3,512,512); output rel. diff 3.3e-3 vs upstream; error vs an fp32-everywhere encoder 5.7e-3 (upstream 5.6e-3). -BAGEL_OPT_FA3=1: FlashAttention-3 (built from flash-attention v2.7.4hopper/with CUTLASS c506e16 — source from codeload.github.com since git clone fails here; trimmed build: bf16, hdim 128, varlen, no fp16/fp8/split/ paged/softcap/local/packgqa/sm80; 1.5 min) installed asflash_attn_interface.py+flash_attn_3_cuda*.soin/opt/candidate-env/lib/python3.10/site-packages(no pip metadata, soflash_attn2.5.8 is unaffected)./tmp/work/test_flash.pywith FA3: error vs fp64 on par with upstream (worst ratio 1.12). FA3's backward is not deterministic here (deterministic=Truestill leaves dq, and with GQA dk/dv, varying,/tmp/work/fa3_det.py), so undertorch.are_deterministic_algorithms_enabled()the calls take FA2 withdeterministic=True(same precision).time: + FUSED_VAE_GN (/tmp/r6/time_r6a): 1.9802 (1.316 s/step, −41 ms); + FA3 (/tmp/r6/time_r6b): 2.0622 (1.2637 s/step, −52 ms; windows 1.275 / 1.2525; alloc 108.8 GiB, reserved 127.5 GiB). Full check of r6b launched 23:35 (/tmp/r6/check_r6b).
6.4 hRound 7 — GEMM check, ViT FSDP layout, step-start syncs (01:03 – )
Profile of r7a (/tmp/r7/profile, 2 steps on rank 0): busy 1872 ms: GEMM 1323 (70.6%), Triton kernels 186 (AdamW 49, SwiGLU fwd 43 incl. recompute, SwiGLU bwd 27, EMA 20 — all at ~4 TB/s, the HBM limit), FA3 + ViT FA2 168 (9.0%), elementwise 100 (residual adds 28, fills 16), index 25, reductions 24. Heavier step: span 1236, busy 1080, gaps 156 ms (first root all-gather 16, final reduce-scatters 16, post-step hook 7, plan building 6.6, ~80 ms of sub-5 ms per-layer stalls). GEMM microbenchmark (/tmp/work/gemm_bench.py): cuBLAS 758 / 782 TFLOP/s on gate-up / down, 638 on q/o, 640 on M=2048, 655 on the weight-gradient shape; cuBLASLt (preferred_blas_library) slower except the dW shape → GEMMs are near the practical peak; no GEMM-library change. -BAGEL_OPT_VIT_ONE_UNIT=1(dev ae1b9d6): the ViT is one FSDP unit (one all-gather / reduce-scatter instead of 26 latency-bound ones); same parameter names / shapes, another flat layout (N1 by layout)./tmp/r7/time_r7a: 2.1298 (1.2236 s/step, −10 ms). Full check/tmp/r7/check_r7a/check.json: PASS, N1, 2.1277; gate 2b 1.44 / 1.02 (det), 1.46 / 1.04 (prod); gate 3 max |d|/σ 1.68 (log_grad_norm_s050_100). →optfast-forwarded to ae1b9d6 (digest = checked);/submission/switches.json= r7a. - Found in the logs:expandable_segments not supported on this platform— that allocator setting has been a no-op all along (kept in the switch file only because every check ran with it; the reserved-memory differences came from the configurations). -BAGEL_OPT_PLAN_EARLY=1(dev 95b0d04): the training loop launches the mask layout analysis right after the batch is on the GPU (before the VAE encode), Bagel.forward resolves it before the ViT → the plan is built on the CPU while the GPU encodes the images (was a 6.6 ms GPU-idle gap at the LM start). EMA update: FSDP handle traversal cached.BAGEL_OPT_NO_VIT_SYNC=1(dev e0e4bd3): max(vit_token_seqlens) read from the batch's host tensor before the move instead oftorch.max(...).item()(a sync that made the CPU wait for the whole VAE encode, then the GPU idle while the ViT was launched). Both N0 (step-0 losses identical to r7a's run).timer7c = r7a + both: first run 1.696 (an 18.4 s stall at step 27 on all ranks while the machine was idle — environmental: CPU load *lower* during it, no log message, not reproduced), re-run (/tmp/r7/time_r7c2): 2.1574 (1.2079 s/step; windows 1.2224 / 1.1934). Full check launched 01:54 (/tmp/r7/check_r7c); its timing stage: windows 1.2234 / 1.1955, no outliers. - Stall census over all runs: 15–18 s stalls in 2 of ~25 runs (r6b check, r7c first time), never reproduced with the same code and data → host noise; the final evaluation marks such runs INVALID and re-measures. Full check of r7c (/tmp/r7/check_r7c/check.json): PASS, N1, speedup 2.1546 (01:54 → 02:18); gate 2b det numbers identical to r7a's (N0 confirmed), prod 1.48 / 1.04; gate 3 max |d|/σ 1.39. →optfast-forwarded to e0e4bd3 (digest 63a8a0a3… = checked);/submission/switches.json= r7c (r7a + PLAN_EARLY + NO_VIT_SYNC). Round 7 ended 02:21.
7.7 hRound 8 — the pre-LM phase: VAE conv bias, VAE prefetch (02:21 – )
Profile of r7c (/tmp/r8/profile): heavy step span 1221 ms, busy 1085: start→LM 127 ms (EMA 10, H2D + analysis 12, VAE ~60 span / 44 busy, root all-gather wait 16 for a VAE-heavier rank, ViT ~25), LM fwd 313, bwd+opt 782. VAE window: convolutions 17.4 ms, cuDNN NCHW↔NHWC transforms 11.6, GroupNorm 9.5, mid-block attention 5.7, and 11.6 ms of non-vectorised broadcast adds = the conv bias adds (PyTorch runs cuDNN without bias, thenout.add_(bias.view(1,C,1,1)), which takes the strided elementwise kernel at ~1 TB/s). Cross-entropy negligible.timer7d = r7c + AG_INFLIGHT=4: 1.2118 s/step (no gain; +2.4 GiB reserved) → dropped. Changes (dev 0b7312b, 5f873b5, 7d20980; all N0): -BAGEL_OPT_VAE_BIAS=1: every encoder conv runs without bias and adds it in one vectorised Triton pass, bf16(float(out) + float(bias_bf16)) = the same rounding; encoder still bitwise (/tmp/work/vae_gn_test.py), 68.7 → 61.0 ms on (4,3,768,1024).timer8a (/tmp/r8/time_r8a): 2.1675 (1.2023 s/step, −5.6 ms). -BAGEL_OPT_VAE_PREFETCH=1: a look-ahead wrapper peeks the next batch (same order); beforeloss.backward()its pinned images are copied on a side stream, and a gradient hook on decoder layer 13's output launches the encode of batch k+1 on that stream mid-backward (activations of 14 layers already freed), autocast/no-grad set inside the hook (thread-local). Step k+1 waits on its event before reading the latents; the batch still goes throughSimpleCustomBatch.cudain its own step. RNG order unchanged (encode(k), noise(k), encode(k+1), …); gate 2b keys noise by call ordinal and compares only the first len(reference) VAE calls (bench.pyline 1004), so the extra encode at a run's end is tolerated. Dead encoder activations (hs[:-1]) freed early.timer8b (/tmp/r8/time_r8b,/tmp/r8/time_r8b2): 2.1999 / 2.1964 (1.1846 / 1.1865 s/step, −17 ms); peak reserved 131.9 GiB (side-stream pool), allocated 108.8 GiB. Step-0 losses identical to r7c's. Full check of r8b launched 02:53 (/tmp/r8/check_r8b); timing stage windows 1.183 / 1.180. Full check of r8b (/tmp/r8/check_r8b/check.json): PASS, N1, speedup 2.2056 (02:53 → 03:17); gate 2b det numbers identical to r7c's (VAE bias + prefetch confirmed N0), prod 1.42 / 1.01; gate 3 max |d|/σ 1.46. →optfast-forwarded to 7d20980 (digest = checked);/submission/switches.json= r8b (r7c + VAE_BIAS + VAE_PREFETCH). -BAGEL_OPT_EMA_STREAM=1(dev e8a9c82, N0): the EMA update runs on a side stream right after the optimizer step (it only reads the model's flat parameters; the nextoptimizer.step()waits for its event; the logging block's device-wide synchronize becomes a current-stream sync — it only feeds the log's steps/sec; the judge's post-step hook still synchronizes the whole device before it hashes the EMA).timer8c (/tmp/r8/time_r8c): 2.2201 (1.1738 s/step, −11 ms); losses at steps 0 and 1 identical to r8b's run. Full check launched 03:32 (/tmp/r8/check_r8c); timing stage windows 1.1769 / 1.1741. - Backward layer-boundary stalls (30.7 ms of sub-2 ms gaps per profiled heavy step): the CPU waits 10–16 ms per layer in FSDP's all-gather rate limiter and is released only when the GPU runs dry. Raising the limiter depth: r8d = r8c + AG_INFLIGHT=8 (/tmp/r8/time_r8d): 1.3556 s/step, much slower (reserved 135 GiB: more unsharded layers in flight → allocator pressure); depth 4 earlier: no gain. Memory is the binding constraint → dropped. Full check of r8c (/tmp/r8/check_r8c/check.json): PASS, N1, speedup 2.2169 (03:32 → 03:56); gate 2b det identical to r8b's (N0), prod 1.45 / 1.07; gate 3 max |d|/σ 0.81. →optfast-forwarded to e8a9c82 (digest = checked);/submission/switches.json= r8c. Round 8 ended 03:59.