DeepSeek V4.1 Flash-1017 1.15×

Harness: DeepSeek's own harness · switches: BAGEL_LOG_FLUSH=50 BAGEL_AC_SKIP_DECODER=12 BAGEL_NO_AC_VIT=1 BAGEL_H2D_NON_BLOCKING=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True BAGEL_FUSED_NORM_APPLY=1

Judged
1.15×
Tier
N0
Self-best
1.23×
Rounds
22
Hours
10.0
API cost
$2.06
Input tokens · 99.6% cached
198 M
Output tokens
0.5 M
Requests
570
Attention kernel swap (split by mask structure)
✗ bucketed cuDNN attention ~1.46× steady state, but window spread 7.5–10.3%, above the 6% limit, reverted; no split by mask structure
Drop activation recompute (28 decoder layers)
partial: 12 layers + ViT (1.067× at 8 layers). All 28 removed: OOM
Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
tried, reverted: N1 config with compiled RMSNorm / RoPE / SwiGLU + fused AdamW passed a full check once (1.29×); whole set dropped after the repeat check failed gate 3; only a bitwise RMSNorm tail kernel kept in the end (+1.8%)
Optimizer, EMA, step start / end
reverted along with the N1 config; EMA via lerp blocked by gate 3
Other main levers
deferred log flush, non-blocking H2D
Raw work log23 sections
0.3 hBaseline profile (round 1, unchanged tree)
profile --from 8 --iters 2 -> /tmp/r1/profile (log /tmp/r1/profile.log).

- GPU busy 94.5% of the profiled window (5.64 s / 2 steps): the step is GPU-bound, not host-bound.
- Kernel time per 2 steps (6.95 s total, overlapping): NCCL 2.68 s (38.6%), GEMM (sm90 xmma) 1.55 s (22.3%),
  attention (fmha_cutlass, mem-efficient SDPA) 1.12 s (16.2%), elementwise 1.36 s (19.6%),
  optimizer AdamW + EMA foreach ~0.29 s (4%).
- NCCL: 116 all-gathers/step... precisely 58/step (all-gather is repeated for the activation-checkpoint
  recompute), 28 reduce-scatters/step; 0.56 s/step of NCCL is *exposed* (not overlapped with compute).
- Attention: per-sample scaled_dot_product_attention with a dense additive mask, backend
  EFFICIENT_ATTENTION; 0.56 s/step. cuDNN SDPA microbenchmark at the real shapes (L=6912/3456/2368,
  H=28, d=128, bf16 additive mask): fwd+bwd 10.3 / 2.7 / 1.5 ms vs 19.2 / 5.0 / 2.85 ms for mem-efficient.
- Host: the training loop's per-step logging block (sync + .item() + 5-6 scalar all-reduces + wandb) costs
  ~0.23 s/step (printed "Steps/Sec" 0.42 vs hook-measured 2.61 s/step).
0.3 hRound 1 (05:00-05:07): profile the unchanged tree
- Top hotspots as above. Actions chosen in order: attention kernel (compute), per-step logging host syncs
  (host), exposed NCCL (communication), elementwise fusion.
- No code change. Profile artifacts /tmp/r1/profile.
0.3 hRound 2 (05:07-05:15): attention backend -> cuDNN SDPA (N1)
- Change: modeling/bagel/qwen2_navit.py _sdpa_train() helper used by both packed-attention forward_train
  paths; BAGEL_ATTN_BACKEND=cudnn selects sdpa_kernel(backends=[SDPBackend.CUDNN_ATTENTION]), with a
  one-time logged fallback to the upstream memory-efficient kernel if cuDNN rejects a shape.
- First attempt used [CUDNN, EFFICIENT]; measured no gain (0.9992x, /tmp/r2/time) because torch's SDPA
  selection prefers mem-efficient whenever both are enabled. Fixed to cuDNN-only.
- Measured: /tmp/r2/time 2.6133 s/step (0.9992x) and /tmp/r2b/time 2.6112 s/step (1.0000x): no gain.
  Log showed [bagel] cuDNN SDPA rejected a shape, so the cuDNN call fell back on the first sample.  The reason
  (found in round 4's diagnostic) was that the per-sample call passed a 3-D mask [1,L,L]; cuDNN needs 4-D.
0.7 hRound 3 (05:21-05:31): deferred per-step logging (N0)
- Change: BAGEL_LOG_FLUSH=50 accumulates the loss/grad-norm scalars on the GPU and flushes (one sync, wandb) every
  50 steps; the logging-only token/seqlen all-reduces move to the host.  Default 0 = upstream.
- Measured: /tmp/r3/time 2.6081 s/step, 1.0012x.  Host-side logging was already hidden behind the GPU queue:
  the step is 94.5% GPU-busy, so this lever is ~0.  Kept (it cannot hurt and removes a per-step device sync), but no
  round of its own again.
0.7 hRound 4 (05:31-...): make cuDNN attention actually run (N1) + FSDP forward prefetch (N0)
- Diagnostic (/tmp/diag/p, one profiled step with shapes printed on the first rejection) showed the mask reaching
  SDPA as [1, 3443, 3443]: the per-layer code did attention_mask_per_sample.to(bf16).unsqueeze(0) on a 2-D mask.
  A 3-D [1,L,L] mask broadcasts over heads exactly like [1,1,L,L] (verified bitwise for the efficient kernel), so
  normalizing to 4-D is numerics-neutral for the fallback path and lets cuDNN run.
- Mask conversion moved to once per step in Bagel.forward (was once per layer and per recompute; bf16 values are
  identical).
- Also: BAGEL_FSDP_FORWARD_PREFETCH=1 -> FSDP(forward_prefetch=True) to overlap the next unit's all-gather with
  the current unit's forward compute (the baseline leaves ~0.56 s/step of NCCL exposed).
- Measured: /tmp/r4a/time (attn + log) and /tmp/r4b/time (attn + log + forward prefetch) -> see below.
1.1 hRound 5 (05:41-05:51): forward prefetch (rejected) and cuDNN attention (rejected)
- BAGEL_FSDP_FORWARD_PREFETCH=1 (FSDP(forward_prefetch=True)): with nested FSDP units + activation-checkpoint
  recompute the prefetch order is wrong (recompute runs in reverse), memory rose 82 -> 111 GiB and the step got
  slower.  Abandoned; the switch stays off.
- cuDNN SDPA with the 4-D mask: ran, but the cuDNN attention workspace took +28 GiB (110 GiB total) and the step
  collapsed to ~19 s/step with the GPUs idle (host-side workspace churn).  Abandoned; BAGEL_ATTN_BACKEND stays off.
- Lesson: on this workload the step is 94.5% GPU-busy; host-side changes (logging) and kernel choices that need big
  workspaces are not the lever.  Memory-for-recompute is.
1.1 hRound 6 (05:51-06:03): elementwise switches
- Fused RMSNorm (aten::rms_norm): microbenchmark said 1.75x on fwd+bwd, end to end it was 0.985x -- the norm weight
  is fp32 in this model, so the fused op returns fp32 and the explicit cast back to bf16 adds a full-tensor copy per
  call.  Off.
- Activation checkpointing off (BAGEL_NO_AC): same arithmetic (recompute is deterministic), but 28 un-checkpointed
  decoder layers need +50 GiB and OOM (132.7 GiB allocated).  -> partial skip switch BAGEL_AC_SKIP_DECODER=N.
1.3 hRound 7 (06:03-06:18): activation-checkpoint recompute is the lever (N0)
- BAGEL_NO_AC (all 28 decoder layers un-checkpointed) OOMs (needs +50 GiB over the 82 GiB baseline).
- BAGEL_FSDP_STRATEGY=SHARD_GRAD_OP (keeps params unsharded across the recompute, same reductions) also runs out
  of memory (130-138 GiB in nvidia-smi) -- rejected.
- BAGEL_AC_SKIP_DECODER=8 + expandable_segments:True: /tmp/r12/time 2.4462 s/step, speedup 1.0674,
  peak allocated 96.29 GiB (baseline 82.14), reserved 109 GiB, window range 0.0008.  Bitwise identical arithmetic
  (an un-checkpointed region computes the same forward values the recompute would), so this is an N0 change: no gate 3.
- Note: expandable_segments:True cuts the reserved-vs-allocated gap (103 -> 109 reserved while allocated grew by
  14 GiB).
- BAGEL_AC_SKIP_DECODER=16: 96.3 -> ~110 GiB allocated and 137-139 GiB in nvidia-smi (right at the 139.8 GiB
  card limit, step 0 9 s) -- too close to OOM for gate 2's other data configuration, rejected.  Per-layer activation
  cost measured: 1.77 GiB per un-checkpointed decoder layer, 0.17 s/step saved per 8 layers.
- Chosen: AC_SKIP_DECODER=8 -> 1.0674x (/tmp/r12/time).  Full check /tmp/check1 started 06:21.
  - gate 1 (from the check's rank files, same window rule): windows 2.4692 / 2.4557 s/step, mean 2.4624,
    speedup 1.0604, window range 0.55%.
  - gate 2 (record): passed the exact items and proceeded to gate 2b (bitwise outcome in check.json).
  - gate 2b: PASS, deterministic r_n=r_j=r_b=0.0 (bitwise), production r ~ 0.0039-0.0047 (ratio 0.78/0.95,
    bound 0.015).
  - VERDICT: status PASS, measured tier N0 (record bitwise, exact items ok), speedup 1.0604.
    Full result copied to /submission/check-pass-1.0604.json; output dir /tmp/check1.
2.3 hRound 9 (07:05-07:13): profile of the current best (AC_SKIP=12 + ViT + H2D)
profile --from 8 --iters 2 -> /tmp/r16/profile.  Window 4.825 s / 2 steps (2.41 s/step), GPU busy 94.8%.
Per 2 steps (kernel time, overlapping): NCCL 2.18 s (reduce-scatter 1.36, all-gather 0.76), GEMM 1.43 s,
elementwise 1.07 s, attention 1.01 s, optimizer/EMA 0.18 s.  Exposed NCCL 0.80 s / 2 steps = 0.40 s/step.
Gaps: 0.25 s / 2 steps (max single gap 9.5 ms) -> not data-bound any more.
Candidates measured and dropped this round:
- flex_attention with a block mask: 92.9 ms eager / 652 ms compiled for L=6912 fwd+bwd vs 19.2 ms mem-efficient
  (block-mask build itself 43 ms steady).  Not competitive; definitively dropped.
- NCCL algorithm/protocol sweep (NCCL_ALGO Ring/Tree/NVLS x PROTO default/Simple) on a 932 MB per-rank
  reduce-scatter and all-gather: all 2.14-2.18 ms, no difference.  The training-time gap to raw bandwidth is
  contention, not a tunable.
- cuDNN SDPA remains excluded by its 28 GiB workspace.
2.3 hRound 10 (07:13-...): compiled RMSNorm chain + fused AdamW (N1)
- BAGEL_COMPILE_RMSNORM=1: torch.compile(dynamic=True) of the exact upstream chain, with the norm weight as a
  *function argument* so the FSDP unshard (which swaps the parameter tensor object every forward) does not
  invalidate the graph.  Verified: 1 unique graph across changing n and changing tensor objects, works under
  torch.use_deterministic_algorithms(True).  Microbenchmark n=10240 x 3584 fwd+bwd: 1.947 -> 0.469 ms (4.15x).
- BAGEL_FUSED_ADAM=1: torch.optim.AdamW(fused=True); same hyper-parameters and state dtypes, one multi-tensor
  kernel: 69.0 -> 36.9 ms per step at the real per-rank parameter size.
- Both are N1 (rounding order), so the full check will need gate 3.
- time run: /tmp/r17/time -> 2.1294 s/step, speedup 1.2262, window range 0.03%, peak allocated
  105.65 GiB (was 111.91: the compiled chain no longer materializes the fp32 intermediates).  So +9.4% over the
  AC_SKIP=12 configuration.
- Full check /tmp/check3 (07:21-07:50): status PASS, tier N1, speedup 1.227 (gate 1 windows 2.1306/2.1254,
  speedup 1.2270; record exact items OK, not bitwise as expected; gate 2b PASS, ratios 1.31/1.07 det and 1.28/1.02
  prod; gate 3 PASS, all 14 endpoints within +/-1 sigma, worst log_grad_norm_s050_100 at 0.99 sigma).
  Record peak 103.91 GiB allocated, timing peak 105.65 GiB.  Result copy /submission/check-pass-1.227.json.
  Switches installed.
2.9 hRound 11 (07:50-...): compiled RoPE + 2 more un-checkpointed layers
- BAGEL_COMPILE_ROPE=1: same treatment as the norm chain (function of tensors only); microbenchmark at the training
  shape (10240 x 28 x 128, q+k, fwd+bwd) 1.96 -> 0.48 ms, i.e. ~65 ms/step over the 44 layer-passes.
- BAGEL_AC_SKIP_DECODER=14: the compiled norm freed 6 GiB, so two more decoder layers fit (expected peak
  ~107 GiB).  ~40 ms/step.
- time /tmp/r18/time: 2.0636 s/step, speedup 1.2653, peak allocated 109.8 GiB (reserved 124.5).
- Full check /tmp/check4 (07:58-08:29): status PASS, tier N1, speedup 1.2623 (gate 1 windows 2.0706/2.0664;
  record exact items OK, peak 107.96 GiB / 122.2 reserved; gate 2b PASS ratios 1.31/1.07 det and 1.20/1.02 prod;
  gate 3 PASS "equivalent", worst endpoint disp_vit_s050_100 at -2.09 sigma, everything else <= 1.26 sigma).
  Result copy /submission/check-pass-1.2623.json.  Switches installed.
3.5 hRound 12 (08:30-...): compiled MoT gather+norm+scatter (N1)
- BAGEL_COMPILE_MOTNORM=1: the pre-attention norm block (fill + 2 gathers + 2 norm chains + 2 scatters on the packed
  sequence) compiled as a function of tensors.  Same arithmetic.
- time /tmp/r19/time: 2.0583 s/step, speedup 1.2686 -- only +0.26% over 1.2653 and inside the run's own noise
  (window range 0.67%), i.e. Inductor does not remove the fill/gather/scatter traffic.  Reverted rather than spending
  a gate-3 risk (disp_vit_s050_100 is already at -2.09 sigma) on a noise-level gain.  Revert commit 2c00239; the
  tree content is then exactly the one check4 validated.
3.8 hRound 13 (08:38-08:48): profile of the current best, then bucketed cuDNN attention
profile --from 8 --iters 2 -> /tmp/r20/profile.  Window 4.273 s / 2 steps (2.14 s/step), GPU busy 95.0%.
Per step: GEMM 0.70 s, mem-efficient attention 0.50 s, elementwise 0.35 s, exposed NCCL 0.35 s,
optimizer/EMA 0.05 s, compiled-Norm/RoPE triton kernels 0.04 s.

- The cuDNN attention post-mortem: a microbenchmark with the real shapes shows cuDNN SDPA is 1.3-1.9x faster than
  mem-efficient at L = 3443/6866/10240 with the *same or less* peak memory (0.16-0.48 GiB extra, not the 28 GiB seen
  in round 5).  The reason it collapsed in round 5 is the plan cache: the first call for a given sequence length
  costs ~1.5 s (1571 ms for L=4000, then 3.8 ms; another 1502 ms for L=4002), and the packed samples present a new
  length almost every step.  With padding to a small set of bucket lengths the plans warm up once.
- BAGEL_ATTN_BUCKET=1024 + BAGEL_ATTN_BACKEND=cudnn: every sample's mask is padded once per step in Bagel.forward
  (pad rows/columns are -inf) and q/k/v are padded in the attention call, output sliced back.  Padded vs unpadded
  attention agrees to 2e-3 relative (bf16 rounding).  ~10 plan builds (~1.5 s each) happen in the warmup steps.
- time /tmp/r21/time (bucket 1024): 2.0797 s/step, 1.2555, window range 22.6% -- window 1 was polluted by the
  plan builds (2.3148) while window 2, after the plans warmed, ran at 1.8446 s/step (1.415x).  That is the size of
  the prize: the attention kernel really is ~1.9x faster once its plan exists.
- Added an explicit plan warm-up at the first forward call (_warm_cudnn_plans, 11 zero-tensor fwd+bwd calls, ~16 s
  once, inside step 0).  time /tmp/r22/time: 1.9022 s/step, 1.3727, but window range still 7.5%
  (window 1 1.9740 vs window 2 1.8305) -- above the 6% invalidity threshold.
- Bucket 2048 (5 plans) /tmp/r23/time: 2.0059 s/step, 1.3017, window range 4.96% -- fewer plans, less gain, still
  noticeable variance; the per-step trace shows periodic +0.6..0.9 s spikes that neither the bucket count nor the
  warm-up removes (cuDNN frontend plan cache / padded-buffer allocator churn).
- Verdict: the ~9% is real but a timing run whose window range exceeds 6% is INVALID in the final evaluation, so this
  direction is rejected despite the gain; both cuDNN bucket commits are reverted (352d3ad, 1348a8d, tree
  content back to the validated ab2e0bf).  Flash SDPA with an arbitrary bool mask is also rejected by torch
  ("No available kernel"), so the memory-efficient kernel is the only usable attention backend here.
4.3 hRound 14 (09:14-...): compiled SwiGLU + single-pass EMA (both N1)
- Profile /tmp/r20/profile (round 13) left elementwise at 0.35 s/step.  Two pieces of it are pure elementwise and
  are now compiled / fused:
  - BAGEL_COMPILE_SWIGLU=1: silu(gate) * up as one compiled kernel instead of two; microbenchmark at the
    training shape [10240, 18944] fwd+bwd 1.773 -> 1.261 ms.  Gated on act_fn is F.silu so a different activation
    keeps the upstream path.
  - BAGEL_EMA_LERP=1: _foreach_lerp_(ema, new, 1-decay) instead of _foreach_mul_ + _foreach_add_
    (~36 -> ~20 ms/step of EMA traffic).
- First time /tmp/r24/time: 2.0568 s/step, 1.2695 -- the SwiGLU path had not engaged, because
  ACT2FN["silu"] is an nn.SiLU *instance*, not F.silu (guard fixed in d7bca6a).
- After the fix time /tmp/r25/time: 2.0500 s/step, 1.2737, and the peak dropped 109.8 -> 103.7 GiB (the fused
  SwiGLU no longer materializes the [tokens, 18944] intermediate for the 14 un-checkpointed layers).
- That headroom paid for two more un-checkpointed layers: AC_SKIP_DECODER=16 -> /tmp/r26/time
  2.0153 s/step, 1.2956, peak 107.07 GiB (reserved 122.4), window range 0.67%.
- Full check /tmp/check5 with AC_SKIP=16 + SwiGLU + EMA lerp: gate 1 1.2947, gate 2b PASS (ratios 1.60/1.08),
  gate 3 INCORRECT: disp_ema_s050_100 at 53 sigma ("different").
  Root cause: the EMA is bf16 and its per-step update is ~1e-4 of the weight, so
  _foreach_lerp_(ema, new, 1-decay) = ema + 1e-4*(new-ema) rounds the small increment differently from the
  upstream ema*decay then + new*(1-decay).  The lerp is therefore *not* a legitimate variant here; removed
  (3e9c5f3, with a comment so it is not retried).
- Re-check without the lerp (AC_SKIP=16 + SwiGLU, /tmp/sw_r27.json) started 10:08 -> /tmp/check6.
6.0 hRound 16 (10:39-11:05): profile, NCCL knobs (no gain), and dead zero-fill removal (N0)
profile --from 8 --iters 2 -> /tmp/r27/profile: window 4.190 s / 2 steps = 2.095 s/step, 94.9% GPU-busy.
Per step: compute 1.651 s (GEMM 0.696, attention 0.494, elementwise/other 0.451, compiled triton 0.063) and NCCL
0.964 s of kernel time of which 0.337 s is exposed (reduce-scatter 0.216, all-gather 0.121).

- GEMM is at ~66% of the 989 TFLOP/s peak by the FLOP count (0.696 s for ~459 TFLOP/rank/step), so cuBLAS has little
  left.  The elementwise block is diffuse: index_elementwise (gathers) 99 ms/step, direct_copy 62, multi-tensor
  (AdamW+EMA) 52, FillFunctor 45, cat 41, plus the rest.
- NCCL_PROTO=Simple + NCCL_MAX_NCHANNELS=32: /tmp/r28/time 2.0234 s/step, 1.2905 vs the 1.2897 of check6 --
  inside the run-to-run noise, i.e. the training-time collective cost is contention, not a tunable.  Not adopted.
- eea8277: in the MoT layer the understanding/generation index sets partition the packed sequence, so the q/k/v
  buffers, the normed-q/k buffers and the attention-output scatter buffer are fully overwritten by their two
  index_puts; new_zeros -> new_empty (guarded by numel(und)+numel(gen)==L, falling back to zeros otherwise)
  removes ~0.6 GiB/step of dead zero-fill traffic.  Bitwise identical, so N0 -- no trajectory risk.
- time /tmp/r29/time: 2.0164 s/step, 1.2949 -- statistically identical to the 2.0153 of /tmp/r26/time
  (the fill traffic was cheaper than its byte count suggested), so the change is reverted (7cd0344); the tree is
  byte-identical to the check6-validated 3e9c5f3 again.
- Also confirmed with a small FSDP model that MixedPrecision(param_dtype=bf16) really does expose bf16 parameters
  inside the forward, so the GEMMs carry no per-call weight cast and the direct_copy kernels in the profile are
  layout copies (attention-output reshape, mask conversion), not dtype casts.
6.3 hRound 17 (11:05-11:16): GEMM/attention backend probes; cuDNN bucketing re-tested and finally rejected
- Source of the 41 ms/step of aten::cat: the only large cats left are the two rotate_half cats per call *inside*
  the compiled RoPE graph (Inductor keeps them as aten.cat) plus the once-per-step latent/position cats; there is no
  stray cat to remove.
- torch.backends.cuda.preferred_blas_library('cublaslt'): microbenchmark over the five real GEMM shapes gives
  6.37 ms vs 5.46 ms for cuBLAS, i.e. 17% slower -- not adopted.  (The same benchmark shows the GEMMs running at
  ~719 TFLOP/s aggregate, 73% of the 989 TFLOP/s peak, confirming there is no GEMM headroom.)
- cuDNN bucketed attention re-applied on top of the current best (16 un-checkpointed layers + SwiGLU) and re-timed:
  /tmp/r30/time 1.8864 s/step, 1.3842, but window 1 = 1.9839 vs window 2 = 1.7889 -> window range 10.3%,
  reproducing (and worsening) the earlier 7.5%.  Steady state is ~1.78 s/step (~1.46x) but a run with a >6% window
  range is INVALID in the final evaluation.
- One more attempt on the variance hypothesis: the plan warm-up ended with torch.cuda.empty_cache(), which releases
  exactly the blocks the first real steps must re-acquire, so it was dropped (359af9f).  /tmp/r31/time:
  1.8523 s/step, 1.4097, window range 7.9% (1.9252 / 1.7795) -- better than 10.3% but still systematic, with
  irregular +0.6-0.7 s spikes at steps 8-11, 17, 21, 33 that are neither plan builds (those are ~1.5 s) nor data
  (the baseline runs with the same data show none).  The direction is now finally closed; all cuDNN commits are
  reverted (7297c59, diff against the validated 3e9c5f3 is empty) and the validated 1.2897 configuration stands.
6.5 hRound 18 (11:25-...): attention-mask probe and reproducibility check
- bool attn_mask vs the additive bf16 mask on the memory-efficient backend: 19.34 vs 19.16 ms at L=6912 and
  bitwise-identical outputs -- the mask type is not a lever.
- Fused MoT gather+RMSNorm+scatter as a custom Triton kernel: costed before writing it, the block is ~1.7 ms per
  layer-pass of which the fused version could save ~0.3-0.5 ms, i.e. 0.3-0.5% of the step (the tensors are only
  73 MB each and the step is 2 s) -- not worth hand-written autograd code, so not implemented.
- The only remaining block above 1% is the attention (0.494 s/step at ~125 TFLOP/s, where a flash-class kernel would
  reach ~590).  Every way of reaching it with existing tools is closed (cuDNN: 7.9-10.3% window ranges; flex: 5x
  slower; flash: rejects the mask; bool mask: no change), and a block-sparse flash rewrite with log-sum-exp merging
  over the split structure is a multi-hour kernel project whose numerics would then need gate-3 validation.
- Therefore this round spends the budget on reproducibility: a second full check of the exact submitted
  configuration (/tmp/check7), to confirm the N1 gates (statistical trajectory in particular) reproduce.
7.0 hRound 19 (11:56-...): gate-3 is chaotic -> switch to the best bitwise-identical (N0) configuration
The repeat check of the *identical* N1 configuration (/tmp/check7) failed gate 3: log_grad_norm_s100_150 went
from -0.47 sigma (check6) to +4.69 sigma, and the per-endpoint comparison between the two runs shows swings of
1-5 sigma on log_grad_norm and disp_vit while ce/mse stay at 0.00-0.04 sigma:

| endpoint | check6 | check7 | delta |
|---|---|---|---|
| log_grad_norm_s100_150 | -0.47 | +4.69 | +5.16 |
| log_grad_norm_s050_100 | +1.37 | +0.26 | -1.11 |
| disp_vit_s050_100 | -1.56 | +0.35 | +1.90 |
| disp_lm_s100_140 | -0.22 | -1.11 | -0.89 |
| ce/mse (all four) | ~0.02 | ~0.02 | <=0.03 |

Same tree, same switches, same seed and data: the only difference is the run's own kernel nondeterminism (the frozen
reference itself records repeat: {seed: 1, identical: false}), so the 150-step trajectory is chaotic on this
horizon and gate 3 is effectively a coin flip for *any* non-bitwise tree, however small the rounding change.  A
measured pass rate of about one in two makes an N1 submission worth ~0.65 expected speedup, which is worse than the
bitwise tier that skips gate 3 entirely (status PASS, tier N0, no trajectory gate).

Decision: submit the best N0 configuration -- every arithmetic-neutral change, no compiled kernels and no fused
optimizer:

    BAGEL_LOG_FLUSH=50, BAGEL_AC_SKIP_DECODER=12, BAGEL_NO_AC_VIT=1, BAGEL_H2D_NON_BLOCKING=1,
    PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

That is the configuration of /tmp/r15/time (2.3240 s/step, speedup 1.1235) plus the bitwise mask handling; 12
un-checkpointed layers is the memory limit without the compile-freed headroom (111.9 GiB allocated / 126.1 reserved
there, ~134 GB of the card).  Full check started 11:58 -> /tmp/check8.
8.2 hRound 20 (12:43-13:11): the foreach-bug, and the final N0 submission
Chasing why the "arithmetically neutral" configuration was not bitwise (/tmp/check8 differed from step 1) led to a
real bug of mine: passing fused=False explicitly to torch.optim.AdamW skips torch's
if fused is None and foreach is None: _default_to_fused_or_foreach(...) block, so foreach stayed None and then
became False -- the upstream foreach AdamW had been silently replaced by the single-tensor loop since commit
9ef80c1, i.e. different rounding (record differs from step 1) *and* a slower optimizer.  Fixed in 1c08f72: the
fused keyword is only passed when the switch is on.

Verification chain after the fix:
- pure reference configuration on the current tree -> check --quick PASS, N0, bitwise true (/tmp/qREF2), so
  the tree now reproduces upstream exactly with all switches off.
- best N0 configuration -> full check /tmp/check9: status PASS, tier N0, bitwise true, speedup 1.1256,
  gate 2b PASS with r_n = r_j = 0.0, no trajectory gate, record peak 110.19 GiB.  Evidence copy
  /submission/check-pass-1.1256-n0.json.

Why N0 rather than the 1.29 N1 configuration: two runs of the *identical* N1 tree (/tmp/check6 and
/tmp/check7) gave gate-3 worst endpoints of 1.56 sigma and 4.69 sigma, with endpoint deltas up to 5.16 sigma
(log_grad_norm_s100_150) while ce/mse stayed at 0.02 sigma -- the 150-step trajectory is chaotic, the frozen reference
itself records repeat: {seed: 1, identical: false}, and gate 3 is therefore close to a coin flip for any non-bitwise
tree.  A bitwise-identical tree skips gate 3 entirely (status PASS, tier N0), so 1.1256 with certainty beats
1.29 x P(pass) for any plausible P.
8.6 hRound 21 (13:12-13:32): can the N1 route be made reproducible? (no -> N0 stands)
- torch._inductor.config.deterministic does not exist in torch 2.5.1 (only the reserved
  TORCHINDUCTOR_DETERMINISTIC env var, which the judge manages), so the compiled-region nondeterminism cannot be
  removed from inside the tree; the switch was inert and has been reverted (b74528d).
- The run it produced is still a third sample of the N1 configuration's gate 3: check --n1 on the 1.29 config ->
  PASS, "equivalent", worst endpoint disp_vit_s050_100 at 1.78 sigma, conformance PASS (/tmp/n1det).
- N1 gate-3 record is now: check6 PASS (1.56 sigma), check7 FAIL (4.69 sigma), n1det PASS (1.78 sigma) -> the pass
  rate is about two in three.  Break-even against the certain N0 result needs P > 0.87, so the N0 submission is the
  better expected score (1.29 x 0.63 ~ 0.81 versus a certain 1.1256), and the 4.7-sigma failure mode is a whole-score
  loss rather than a small one.
8.7 hRound 22 (13:32-13:41): costed out the remaining bitwise levers; final timing confirmation
With the N1 route closed (see round 21), this round worked through every remaining way to get faster *while staying
bitwise identical* -- the only tier that skips the chaotic trajectory gate:

| idea | why it is not usable |
|---|---|
| checkpoint only the RMSNorm's fp32 intermediates to free memory for more AC layers (~10 GB of fp32 h are saved for the backward across 12 layers) | impossible under FSDP: with use_orig_params=False the module's parameters are bare views into the unsharded flat param, which FSDP frees on the post-forward reshard; a recompute inside an un-checkpointed layer would read dangling memory.  That is exactly why FSDP's own layer-level AC re-gathers on recompute. |
| BAGEL_AC_SKIP_DECODER=13 | +1.25% for ~4 GiB; at 12 layers the record peak is already 110.2 GiB allocated / 111.9 timing (~130 GB of the 140.4 GB card).  Any non-zero OOM probability multiplies the *whole* score, so the expected value is negative (1.138 x 0.97 < 1.126). |
| a bitwise Triton RMSNorm (fused fp32 chain) | would need to reproduce PyTorch's reduce_kernel fp32 tree bit for bit in both forward and backward across every real (n, D); the payoff is ~4-6%, the success probability is low and a single 1-ulp miss drops the tree to N1 and the coin flip. |
| fused torch.nn.functional.rms_norm | measured 1 bf16 ulp off the chain (the chain casts to bf16 before the weight multiply, the fused op after) -> N1. |
| CUDA graphs | graphs are shape-specialised and the packed sequence length changes every step; padding to fixed shapes changes the arithmetic. |
| norm-level fusions / new_empty / MoT gather+norm+scatter | each measured at <=0.3% in earlier rounds. |

Final confirmation of the submitted configuration: time /tmp/r32/time 2.3165 s/step, speedup 1.1272,
window range 0.61 %, peak allocated 111.91 GiB -- consistent with the full check's gate 1 (1.1256).
8.9 hRound 23 (13:41-14:00): profile of the N0 configuration; last bitwise-fusion probe
profile --from 8 --iters 2 of the submitted N0 tree -> /tmp/r33/profile.  Window 4.863 s / 2 steps
(2.43 s/step under the profiler), GPU busy 94.0%.  Per step: compute 1.894 s and NCCL 1.091 s, of which **0.391 s is
exposed**; categories: elementwise/other 0.725 s, GEMM 0.717 s, attention 0.508 s, NCCL 1.091 s.  The elementwise
block is much larger than in the N1 tree (0.451 s) because the eager fp32 RMSNorm chain and the extra recompute are
back: the top family is vectorized_elementwise_kernel with 13,364 calls per step.

- Last bitwise-fusion probe: fusing only the pointwise tail of the norm, w * (h*r).to(bf16), as a compiled
  function of tensors.  Not bitwise (forward or gradients): Inductor fuses the intermediate bf16 cast away, so
  the weight multiply happens in fp32 and rounds once instead of twice (eager 0.611 ms -> compiled 0.218 ms at
  n=10240, i.e. 2.3% of the step is there, but it cannot be had in the bitwise tier).  A hand-written Triton kernel
  could force the cast, but the same function's backward contains the h*r broadcast reduction (d_r = sum_D(d_t*h))
  whose order would have to match PyTorch bit for bit -- the same blocker as the full-chain version.
- Confirmed from the gate-3 rank files (/tmp/check7/traj_cand_s1.r0.json) that the trajectory gate runs with
  deterministic_algorithms = False, i.e. production settings, which is why two identical runs differ by up to
  5.16 sigma.  There is no way to make it reproducible from inside the tree (torch 2.5.1 has no
  inductor.config.deterministic; TORCHINDUCTOR_DETERMINISTIC is judge-managed).

Conclusion for this round: the submitted N0 configuration remains the best expected score; the profile shows no
untried block, and the one remaining fusion was measured to be non-bitwise.
9.4 hRound 24 (13:52-14:24): a bitwise custom kernel -- fused RMSNorm tail (N0, +1.8%)
The N0 profile showed 0.725 s/step of elementwise, the eager fp32 RMSNorm chain being the dense part.  Splitting that
chain, the tail w * (h*r).to(bf16) is pure pointwise (the reduction mean(h^2) stays in PyTorch, untouched), and
the tail alone costs three kernels: fp32 multiply, bf16 cast, weight multiply.

- Two Triton kernels reproduce it bit for bit: the forward writes the output *and* the saved bf16 intermediate tb
  (both verified torch.equal to the eager result at n = 3443/5000/10240, D = 3584), and the backward writes d_tb,
  d_t, d_h (verified torch.equal).
- The Function's backward keeps both reductions in PyTorch -- d_r = (d_t*h).sum(dim=1, keepdim=True) and
  d_w = (grad*tb).sum(dim=0) -- and a hand-written version of it reproduces autograd's three gradients bitwise
  (verified), so nothing about the reduction order changes.
- The 3-D q/k norm inputs are handled through the flattened view, which was verified bitwise-equal to the eager 3-D
  chain (including the mean(-1) values).
- time /tmp/r36/time: 2.2789 s/step, speedup 1.1458 (from 2.3165 / 1.1256), window range 0.05%, peak unchanged
  at 111.91 GiB.  check --quick with the switch: PASS, tier N0, bitwise true (/tmp/qNF).
- Full check started 14:23 -> /tmp/check10.
↑ BAGEL SFT