DeepSeek V4.1 Flash-1017 1.15×
Harness: DeepSeek's own harness · switches: BAGEL_LOG_FLUSH=50 BAGEL_AC_SKIP_DECODER=12 BAGEL_NO_AC_VIT=1 BAGEL_H2D_NON_BLOCKING=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True BAGEL_FUSED_NORM_APPLY=1
- Judged
- 1.15×
- Tier
- N0
- Self-best
- 1.23×
- Rounds
- 22
- Hours
- 10.0
- API cost
- $2.06
- Input tokens · 99.6% cached
- 198 M
- Output tokens
- 0.5 M
- Requests
- 570
§ 02Key levers
- Attention kernel swap (split by mask structure)
- ✗ bucketed cuDNN attention ~1.46× steady state, but window spread 7.5–10.3%, above the 6% limit, reverted; no split by mask structure
- Drop activation recompute (28 decoder layers)
- partial: 12 layers + ViT (1.067× at 8 layers). All 28 removed: OOM
- Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
- tried, reverted: N1 config with compiled RMSNorm / RoPE / SwiGLU + fused AdamW passed a full check once (1.29×); whole set dropped after the repeat check failed gate 3; only a bitwise RMSNorm tail kernel kept in the end (+1.8%)
- Optimizer, EMA, step start / end
- reverted along with the N1 config; EMA via lerp blocked by gate 3
- Other main levers
- deferred log flush, non-blocking H2D
§ ARaw trace
Raw work log23 sections
0.3 hBaseline profile (round 1, unchanged tree)
profile --from 8 --iters 2->/tmp/r1/profile(log/tmp/r1/profile.log). - GPU busy 94.5% of the profiled window (5.64 s / 2 steps): the step is GPU-bound, not host-bound. - Kernel time per 2 steps (6.95 s total, overlapping): NCCL 2.68 s (38.6%), GEMM (sm90 xmma) 1.55 s (22.3%), attention (fmha_cutlass, mem-efficient SDPA) 1.12 s (16.2%), elementwise 1.36 s (19.6%), optimizer AdamW + EMA foreach ~0.29 s (4%). - NCCL: 116 all-gathers/step... precisely 58/step (all-gather is repeated for the activation-checkpoint recompute), 28 reduce-scatters/step; 0.56 s/step of NCCL is *exposed* (not overlapped with compute). - Attention: per-samplescaled_dot_product_attentionwith a dense additive mask, backendEFFICIENT_ATTENTION; 0.56 s/step. cuDNN SDPA microbenchmark at the real shapes (L=6912/3456/2368, H=28, d=128, bf16 additive mask): fwd+bwd 10.3 / 2.7 / 1.5 ms vs 19.2 / 5.0 / 2.85 ms for mem-efficient. - Host: the training loop's per-step logging block (sync +.item()+ 5-6 scalar all-reduces + wandb) costs ~0.23 s/step (printed "Steps/Sec" 0.42 vs hook-measured 2.61 s/step).
0.3 hRound 1 (05:00-05:07): profile the unchanged tree
- Top hotspots as above. Actions chosen in order: attention kernel (compute), per-step logging host syncs
(host), exposed NCCL (communication), elementwise fusion.
- No code change. Profile artifacts /tmp/r1/profile.0.3 hRound 2 (05:07-05:15): attention backend -> cuDNN SDPA (N1)
- Change:modeling/bagel/qwen2_navit.py_sdpa_train()helper used by both packed-attentionforward_trainpaths;BAGEL_ATTN_BACKEND=cudnnselectssdpa_kernel(backends=[SDPBackend.CUDNN_ATTENTION]), with a one-time logged fallback to the upstream memory-efficient kernel if cuDNN rejects a shape. - First attempt used[CUDNN, EFFICIENT]; measured no gain (0.9992x,/tmp/r2/time) because torch's SDPA selection prefers mem-efficient whenever both are enabled. Fixed to cuDNN-only. - Measured:/tmp/r2/time2.6133 s/step (0.9992x) and/tmp/r2b/time2.6112 s/step (1.0000x): no gain. Log showed[bagel] cuDNN SDPA rejected a shape, so the cuDNN call fell back on the first sample. The reason (found in round 4's diagnostic) was that the per-sample call passed a 3-D mask[1,L,L]; cuDNN needs 4-D.
0.7 hRound 3 (05:21-05:31): deferred per-step logging (N0)
- Change:BAGEL_LOG_FLUSH=50accumulates the loss/grad-norm scalars on the GPU and flushes (one sync, wandb) every 50 steps; the logging-only token/seqlen all-reduces move to the host. Default 0 = upstream. - Measured:/tmp/r3/time2.6081 s/step, 1.0012x. Host-side logging was already hidden behind the GPU queue: the step is 94.5% GPU-busy, so this lever is ~0. Kept (it cannot hurt and removes a per-step device sync), but no round of its own again.
0.7 hRound 4 (05:31-...): make cuDNN attention actually run (N1) + FSDP forward prefetch (N0)
- Diagnostic (/tmp/diag/p, one profiled step with shapes printed on the first rejection) showed the mask reaching SDPA as[1, 3443, 3443]: the per-layer code didattention_mask_per_sample.to(bf16).unsqueeze(0)on a 2-D mask. A 3-D[1,L,L]mask broadcasts over heads exactly like[1,1,L,L](verified bitwise for the efficient kernel), so normalizing to 4-D is numerics-neutral for the fallback path and lets cuDNN run. - Mask conversion moved to once per step inBagel.forward(was once per layer and per recompute; bf16 values are identical). - Also:BAGEL_FSDP_FORWARD_PREFETCH=1->FSDP(forward_prefetch=True)to overlap the next unit's all-gather with the current unit's forward compute (the baseline leaves ~0.56 s/step of NCCL exposed). - Measured:/tmp/r4a/time(attn + log) and/tmp/r4b/time(attn + log + forward prefetch) -> see below.
1.1 hRound 5 (05:41-05:51): forward prefetch (rejected) and cuDNN attention (rejected)
-BAGEL_FSDP_FORWARD_PREFETCH=1(FSDP(forward_prefetch=True)): with nested FSDP units + activation-checkpoint recompute the prefetch order is wrong (recompute runs in reverse), memory rose 82 -> 111 GiB and the step got slower. Abandoned; the switch stays off. - cuDNN SDPA with the 4-D mask: ran, but the cuDNN attention workspace took +28 GiB (110 GiB total) and the step collapsed to ~19 s/step with the GPUs idle (host-side workspace churn). Abandoned;BAGEL_ATTN_BACKENDstays off. - Lesson: on this workload the step is 94.5% GPU-busy; host-side changes (logging) and kernel choices that need big workspaces are not the lever. Memory-for-recompute is.
1.1 hRound 6 (05:51-06:03): elementwise switches
- Fused RMSNorm (aten::rms_norm): microbenchmark said 1.75x on fwd+bwd, end to end it was 0.985x -- the norm weight is fp32 in this model, so the fused op returns fp32 and the explicit cast back to bf16 adds a full-tensor copy per call. Off. - Activation checkpointing off (BAGEL_NO_AC): same arithmetic (recompute is deterministic), but 28 un-checkpointed decoder layers need +50 GiB and OOM (132.7 GiB allocated). -> partial skip switchBAGEL_AC_SKIP_DECODER=N.
1.3 hRound 7 (06:03-06:18): activation-checkpoint recompute is the lever (N0)
-BAGEL_NO_AC(all 28 decoder layers un-checkpointed) OOMs (needs +50 GiB over the 82 GiB baseline). -BAGEL_FSDP_STRATEGY=SHARD_GRAD_OP(keeps params unsharded across the recompute, same reductions) also runs out of memory (130-138 GiB in nvidia-smi) -- rejected. -BAGEL_AC_SKIP_DECODER=8+expandable_segments:True:/tmp/r12/time2.4462 s/step, speedup 1.0674, peak allocated 96.29 GiB (baseline 82.14), reserved 109 GiB, window range 0.0008. Bitwise identical arithmetic (an un-checkpointed region computes the same forward values the recompute would), so this is an N0 change: no gate 3. - Note:expandable_segments:Truecuts the reserved-vs-allocated gap (103 -> 109 reserved while allocated grew by 14 GiB). -BAGEL_AC_SKIP_DECODER=16: 96.3 -> ~110 GiB allocated and 137-139 GiB in nvidia-smi (right at the 139.8 GiB card limit, step 0 9 s) -- too close to OOM for gate 2's other data configuration, rejected. Per-layer activation cost measured: 1.77 GiB per un-checkpointed decoder layer, 0.17 s/step saved per 8 layers. - Chosen: AC_SKIP_DECODER=8 -> 1.0674x (/tmp/r12/time). Full check/tmp/check1started 06:21. - gate 1 (from the check's rank files, same window rule): windows 2.4692 / 2.4557 s/step, mean 2.4624, speedup 1.0604, window range 0.55%. - gate 2 (record): passed the exact items and proceeded to gate 2b (bitwise outcome in check.json). - gate 2b: PASS, deterministic r_n=r_j=r_b=0.0 (bitwise), production r ~ 0.0039-0.0047 (ratio 0.78/0.95, bound 0.015). - VERDICT: status PASS, measured tier N0 (record bitwise, exact items ok), speedup 1.0604. Full result copied to/submission/check-pass-1.0604.json; output dir/tmp/check1.
2.3 hRound 9 (07:05-07:13): profile of the current best (AC_SKIP=12 + ViT + H2D)
profile --from 8 --iters 2->/tmp/r16/profile. Window 4.825 s / 2 steps (2.41 s/step), GPU busy 94.8%. Per 2 steps (kernel time, overlapping): NCCL 2.18 s (reduce-scatter 1.36, all-gather 0.76), GEMM 1.43 s, elementwise 1.07 s, attention 1.01 s, optimizer/EMA 0.18 s. Exposed NCCL 0.80 s / 2 steps = 0.40 s/step. Gaps: 0.25 s / 2 steps (max single gap 9.5 ms) -> not data-bound any more. Candidates measured and dropped this round: - flex_attention with a block mask: 92.9 ms eager / 652 ms compiled for L=6912 fwd+bwd vs 19.2 ms mem-efficient (block-mask build itself 43 ms steady). Not competitive; definitively dropped. - NCCL algorithm/protocol sweep (NCCL_ALGO Ring/Tree/NVLS x PROTO default/Simple) on a 932 MB per-rank reduce-scatter and all-gather: all 2.14-2.18 ms, no difference. The training-time gap to raw bandwidth is contention, not a tunable. - cuDNN SDPA remains excluded by its 28 GiB workspace.
2.3 hRound 10 (07:13-...): compiled RMSNorm chain + fused AdamW (N1)
-BAGEL_COMPILE_RMSNORM=1:torch.compile(dynamic=True)of the exact upstream chain, with the norm weight as a *function argument* so the FSDP unshard (which swaps the parameter tensor object every forward) does not invalidate the graph. Verified: 1 unique graph across changing n and changing tensor objects, works undertorch.use_deterministic_algorithms(True). Microbenchmark n=10240 x 3584 fwd+bwd: 1.947 -> 0.469 ms (4.15x). -BAGEL_FUSED_ADAM=1:torch.optim.AdamW(fused=True); same hyper-parameters and state dtypes, one multi-tensor kernel: 69.0 -> 36.9 ms per step at the real per-rank parameter size. - Both are N1 (rounding order), so the full check will need gate 3. -timerun:/tmp/r17/time-> 2.1294 s/step, speedup 1.2262, window range 0.03%, peak allocated 105.65 GiB (was 111.91: the compiled chain no longer materializes the fp32 intermediates). So +9.4% over the AC_SKIP=12 configuration. - Full check/tmp/check3(07:21-07:50): status PASS, tier N1, speedup 1.227 (gate 1 windows 2.1306/2.1254, speedup 1.2270; record exact items OK, not bitwise as expected; gate 2b PASS, ratios 1.31/1.07 det and 1.28/1.02 prod; gate 3 PASS, all 14 endpoints within +/-1 sigma, worst log_grad_norm_s050_100 at 0.99 sigma). Record peak 103.91 GiB allocated, timing peak 105.65 GiB. Result copy/submission/check-pass-1.227.json. Switches installed.
2.9 hRound 11 (07:50-...): compiled RoPE + 2 more un-checkpointed layers
-BAGEL_COMPILE_ROPE=1: same treatment as the norm chain (function of tensors only); microbenchmark at the training shape (10240 x 28 x 128, q+k, fwd+bwd) 1.96 -> 0.48 ms, i.e. ~65 ms/step over the 44 layer-passes. -BAGEL_AC_SKIP_DECODER=14: the compiled norm freed 6 GiB, so two more decoder layers fit (expected peak ~107 GiB). ~40 ms/step. -time/tmp/r18/time: 2.0636 s/step, speedup 1.2653, peak allocated 109.8 GiB (reserved 124.5). - Full check/tmp/check4(07:58-08:29): status PASS, tier N1, speedup 1.2623 (gate 1 windows 2.0706/2.0664; record exact items OK, peak 107.96 GiB / 122.2 reserved; gate 2b PASS ratios 1.31/1.07 det and 1.20/1.02 prod; gate 3 PASS "equivalent", worst endpoint disp_vit_s050_100 at -2.09 sigma, everything else <= 1.26 sigma). Result copy/submission/check-pass-1.2623.json. Switches installed.
3.5 hRound 12 (08:30-...): compiled MoT gather+norm+scatter (N1)
-BAGEL_COMPILE_MOTNORM=1: the pre-attention norm block (fill + 2 gathers + 2 norm chains + 2 scatters on the packed sequence) compiled as a function of tensors. Same arithmetic. -time/tmp/r19/time: 2.0583 s/step, speedup 1.2686 -- only +0.26% over 1.2653 and inside the run's own noise (window range 0.67%), i.e. Inductor does not remove the fill/gather/scatter traffic. Reverted rather than spending a gate-3 risk (disp_vit_s050_100 is already at -2.09 sigma) on a noise-level gain. Revert commit2c00239; the tree content is then exactly the one check4 validated.
3.8 hRound 13 (08:38-08:48): profile of the current best, then bucketed cuDNN attention
profile --from 8 --iters 2->/tmp/r20/profile. Window 4.273 s / 2 steps (2.14 s/step), GPU busy 95.0%. Per step: GEMM 0.70 s, mem-efficient attention 0.50 s, elementwise 0.35 s, exposed NCCL 0.35 s, optimizer/EMA 0.05 s, compiled-Norm/RoPE triton kernels 0.04 s. - The cuDNN attention post-mortem: a microbenchmark with the real shapes shows cuDNN SDPA is 1.3-1.9x faster than mem-efficient at L = 3443/6866/10240 with the *same or less* peak memory (0.16-0.48 GiB extra, not the 28 GiB seen in round 5). The reason it collapsed in round 5 is the plan cache: the first call for a given sequence length costs ~1.5 s (1571 ms for L=4000, then 3.8 ms; another 1502 ms for L=4002), and the packed samples present a new length almost every step. With padding to a small set of bucket lengths the plans warm up once. -BAGEL_ATTN_BUCKET=1024+BAGEL_ATTN_BACKEND=cudnn: every sample's mask is padded once per step inBagel.forward(pad rows/columns are -inf) and q/k/v are padded in the attention call, output sliced back. Padded vs unpadded attention agrees to 2e-3 relative (bf16 rounding). ~10 plan builds (~1.5 s each) happen in the warmup steps. -time/tmp/r21/time(bucket 1024): 2.0797 s/step, 1.2555, window range 22.6% -- window 1 was polluted by the plan builds (2.3148) while window 2, after the plans warmed, ran at 1.8446 s/step (1.415x). That is the size of the prize: the attention kernel really is ~1.9x faster once its plan exists. - Added an explicit plan warm-up at the first forward call (_warm_cudnn_plans, 11 zero-tensor fwd+bwd calls, ~16 s once, inside step 0).time/tmp/r22/time: 1.9022 s/step, 1.3727, but window range still 7.5% (window 1 1.9740 vs window 2 1.8305) -- above the 6% invalidity threshold. - Bucket 2048 (5 plans)/tmp/r23/time: 2.0059 s/step, 1.3017, window range 4.96% -- fewer plans, less gain, still noticeable variance; the per-step trace shows periodic +0.6..0.9 s spikes that neither the bucket count nor the warm-up removes (cuDNN frontend plan cache / padded-buffer allocator churn). - Verdict: the ~9% is real but a timing run whose window range exceeds 6% is INVALID in the final evaluation, so this direction is rejected despite the gain; both cuDNN bucket commits are reverted (352d3ad,1348a8d, tree content back to the validatedab2e0bf). Flash SDPA with an arbitrary bool mask is also rejected by torch ("No available kernel"), so the memory-efficient kernel is the only usable attention backend here.
4.3 hRound 14 (09:14-...): compiled SwiGLU + single-pass EMA (both N1)
- Profile/tmp/r20/profile(round 13) left elementwise at 0.35 s/step. Two pieces of it are pure elementwise and are now compiled / fused: -BAGEL_COMPILE_SWIGLU=1:silu(gate) * upas one compiled kernel instead of two; microbenchmark at the training shape [10240, 18944] fwd+bwd 1.773 -> 1.261 ms. Gated onact_fn is F.siluso a different activation keeps the upstream path. -BAGEL_EMA_LERP=1:_foreach_lerp_(ema, new, 1-decay)instead of_foreach_mul_+_foreach_add_(~36 -> ~20 ms/step of EMA traffic). - Firsttime/tmp/r24/time: 2.0568 s/step, 1.2695 -- the SwiGLU path had not engaged, becauseACT2FN["silu"]is annn.SiLU*instance*, notF.silu(guard fixed ind7bca6a). - After the fixtime/tmp/r25/time: 2.0500 s/step, 1.2737, and the peak dropped 109.8 -> 103.7 GiB (the fused SwiGLU no longer materializes the [tokens, 18944] intermediate for the 14 un-checkpointed layers). - That headroom paid for two more un-checkpointed layers:AC_SKIP_DECODER=16->/tmp/r26/time2.0153 s/step, 1.2956, peak 107.07 GiB (reserved 122.4), window range 0.67%. - Full check/tmp/check5withAC_SKIP=16 + SwiGLU + EMA lerp: gate 1 1.2947, gate 2b PASS (ratios 1.60/1.08), gate 3 INCORRECT:disp_ema_s050_100at 53 sigma ("different"). Root cause: the EMA is bf16 and its per-step update is ~1e-4 of the weight, so_foreach_lerp_(ema, new, 1-decay)=ema + 1e-4*(new-ema)rounds the small increment differently from the upstreamema*decaythen+ new*(1-decay). The lerp is therefore *not* a legitimate variant here; removed (3e9c5f3, with a comment so it is not retried). - Re-check without the lerp (AC_SKIP=16 + SwiGLU,/tmp/sw_r27.json) started 10:08 ->/tmp/check6.
6.0 hRound 16 (10:39-11:05): profile, NCCL knobs (no gain), and dead zero-fill removal (N0)
profile --from 8 --iters 2->/tmp/r27/profile: window 4.190 s / 2 steps = 2.095 s/step, 94.9% GPU-busy. Per step: compute 1.651 s (GEMM 0.696, attention 0.494, elementwise/other 0.451, compiled triton 0.063) and NCCL 0.964 s of kernel time of which 0.337 s is exposed (reduce-scatter 0.216, all-gather 0.121). - GEMM is at ~66% of the 989 TFLOP/s peak by the FLOP count (0.696 s for ~459 TFLOP/rank/step), so cuBLAS has little left. The elementwise block is diffuse:index_elementwise(gathers) 99 ms/step,direct_copy62, multi-tensor (AdamW+EMA) 52,FillFunctor45,cat41, plus the rest. -NCCL_PROTO=Simple+NCCL_MAX_NCHANNELS=32:/tmp/r28/time2.0234 s/step, 1.2905 vs the 1.2897 of check6 -- inside the run-to-run noise, i.e. the training-time collective cost is contention, not a tunable. Not adopted. -eea8277: in the MoT layer the understanding/generation index sets partition the packed sequence, so the q/k/v buffers, the normed-q/k buffers and the attention-output scatter buffer are fully overwritten by their two index_puts;new_zeros->new_empty(guarded bynumel(und)+numel(gen)==L, falling back to zeros otherwise) removes ~0.6 GiB/step of dead zero-fill traffic. Bitwise identical, so N0 -- no trajectory risk. -time/tmp/r29/time: 2.0164 s/step, 1.2949 -- statistically identical to the 2.0153 of/tmp/r26/time(the fill traffic was cheaper than its byte count suggested), so the change is reverted (7cd0344); the tree is byte-identical to the check6-validated3e9c5f3again. - Also confirmed with a small FSDP model thatMixedPrecision(param_dtype=bf16)really does expose bf16 parameters inside the forward, so the GEMMs carry no per-call weight cast and thedirect_copykernels in the profile are layout copies (attention-output reshape, mask conversion), not dtype casts.
6.3 hRound 17 (11:05-11:16): GEMM/attention backend probes; cuDNN bucketing re-tested and finally rejected
- Source of the 41 ms/step ofaten::cat: the only large cats left are the tworotate_halfcats per call *inside* the compiled RoPE graph (Inductor keeps them as aten.cat) plus the once-per-step latent/position cats; there is no stray cat to remove. -torch.backends.cuda.preferred_blas_library('cublaslt'): microbenchmark over the five real GEMM shapes gives 6.37 ms vs 5.46 ms for cuBLAS, i.e. 17% slower -- not adopted. (The same benchmark shows the GEMMs running at ~719 TFLOP/s aggregate, 73% of the 989 TFLOP/s peak, confirming there is no GEMM headroom.) - cuDNN bucketed attention re-applied on top of the current best (16 un-checkpointed layers + SwiGLU) and re-timed:/tmp/r30/time1.8864 s/step, 1.3842, but window 1 = 1.9839 vs window 2 = 1.7889 -> window range 10.3%, reproducing (and worsening) the earlier 7.5%. Steady state is ~1.78 s/step (~1.46x) but a run with a >6% window range is INVALID in the final evaluation. - One more attempt on the variance hypothesis: the plan warm-up ended withtorch.cuda.empty_cache(), which releases exactly the blocks the first real steps must re-acquire, so it was dropped (359af9f)./tmp/r31/time: 1.8523 s/step, 1.4097, window range 7.9% (1.9252 / 1.7795) -- better than 10.3% but still systematic, with irregular +0.6-0.7 s spikes at steps 8-11, 17, 21, 33 that are neither plan builds (those are ~1.5 s) nor data (the baseline runs with the same data show none). The direction is now finally closed; all cuDNN commits are reverted (7297c59, diff against the validated3e9c5f3is empty) and the validated 1.2897 configuration stands.
6.5 hRound 18 (11:25-...): attention-mask probe and reproducibility check
-boolattn_mask vs the additive bf16 mask on the memory-efficient backend: 19.34 vs 19.16 ms at L=6912 and bitwise-identical outputs -- the mask type is not a lever. - Fused MoT gather+RMSNorm+scatter as a custom Triton kernel: costed before writing it, the block is ~1.7 ms per layer-pass of which the fused version could save ~0.3-0.5 ms, i.e. 0.3-0.5% of the step (the tensors are only 73 MB each and the step is 2 s) -- not worth hand-written autograd code, so not implemented. - The only remaining block above 1% is the attention (0.494 s/step at ~125 TFLOP/s, where a flash-class kernel would reach ~590). Every way of reaching it with existing tools is closed (cuDNN: 7.9-10.3% window ranges; flex: 5x slower; flash: rejects the mask; bool mask: no change), and a block-sparse flash rewrite with log-sum-exp merging over the split structure is a multi-hour kernel project whose numerics would then need gate-3 validation. - Therefore this round spends the budget on reproducibility: a second full check of the exact submitted configuration (/tmp/check7), to confirm the N1 gates (statistical trajectory in particular) reproduce.
7.0 hRound 19 (11:56-...): gate-3 is chaotic -> switch to the best bitwise-identical (N0) configuration
The repeat check of the *identical* N1 configuration (/tmp/check7) failed gate 3:log_grad_norm_s100_150went from -0.47 sigma (check6) to +4.69 sigma, and the per-endpoint comparison between the two runs shows swings of 1-5 sigma on log_grad_norm and disp_vit while ce/mse stay at 0.00-0.04 sigma: | endpoint | check6 | check7 | delta | |---|---|---|---| | log_grad_norm_s100_150 | -0.47 | +4.69 | +5.16 | | log_grad_norm_s050_100 | +1.37 | +0.26 | -1.11 | | disp_vit_s050_100 | -1.56 | +0.35 | +1.90 | | disp_lm_s100_140 | -0.22 | -1.11 | -0.89 | | ce/mse (all four) | ~0.02 | ~0.02 | <=0.03 | Same tree, same switches, same seed and data: the only difference is the run's own kernel nondeterminism (the frozen reference itself recordsrepeat: {seed: 1, identical: false}), so the 150-step trajectory is chaotic on this horizon and gate 3 is effectively a coin flip for *any* non-bitwise tree, however small the rounding change. A measured pass rate of about one in two makes an N1 submission worth ~0.65 expected speedup, which is worse than the bitwise tier that skips gate 3 entirely (status PASS, tier N0, no trajectory gate). Decision: submit the best N0 configuration -- every arithmetic-neutral change, no compiled kernels and no fused optimizer: BAGEL_LOG_FLUSH=50, BAGEL_AC_SKIP_DECODER=12, BAGEL_NO_AC_VIT=1, BAGEL_H2D_NON_BLOCKING=1, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True That is the configuration of/tmp/r15/time(2.3240 s/step, speedup 1.1235) plus the bitwise mask handling; 12 un-checkpointed layers is the memory limit without the compile-freed headroom (111.9 GiB allocated / 126.1 reserved there, ~134 GB of the card). Full check started 11:58 ->/tmp/check8.
8.2 hRound 20 (12:43-13:11): the foreach-bug, and the final N0 submission
Chasing why the "arithmetically neutral" configuration was not bitwise (/tmp/check8differed from step 1) led to a real bug of mine: passingfused=Falseexplicitly totorch.optim.AdamWskips torch'sif fused is None and foreach is None: _default_to_fused_or_foreach(...)block, soforeachstayed None and then became False -- the upstream foreach AdamW had been silently replaced by the single-tensor loop since commit9ef80c1, i.e. different rounding (record differs from step 1) *and* a slower optimizer. Fixed in1c08f72: thefusedkeyword is only passed when the switch is on. Verification chain after the fix: - pure reference configuration on the current tree ->check --quickPASS, N0, bitwise true (/tmp/qREF2), so the tree now reproduces upstream exactly with all switches off. - best N0 configuration -> full check/tmp/check9: status PASS, tier N0, bitwise true, speedup 1.1256, gate 2b PASS with r_n = r_j = 0.0, no trajectory gate, record peak 110.19 GiB. Evidence copy/submission/check-pass-1.1256-n0.json. Why N0 rather than the 1.29 N1 configuration: two runs of the *identical* N1 tree (/tmp/check6and/tmp/check7) gave gate-3 worst endpoints of 1.56 sigma and 4.69 sigma, with endpoint deltas up to 5.16 sigma (log_grad_norm_s100_150) while ce/mse stayed at 0.02 sigma -- the 150-step trajectory is chaotic, the frozen reference itself recordsrepeat: {seed: 1, identical: false}, and gate 3 is therefore close to a coin flip for any non-bitwise tree. A bitwise-identical tree skips gate 3 entirely (status PASS, tier N0), so 1.1256 with certainty beats 1.29 x P(pass) for any plausible P.
8.6 hRound 21 (13:12-13:32): can the N1 route be made reproducible? (no -> N0 stands)
-torch._inductor.config.deterministicdoes not exist in torch 2.5.1 (only the reservedTORCHINDUCTOR_DETERMINISTICenv var, which the judge manages), so the compiled-region nondeterminism cannot be removed from inside the tree; the switch was inert and has been reverted (b74528d). - The run it produced is still a third sample of the N1 configuration's gate 3:check --n1on the 1.29 config -> PASS, "equivalent", worst endpoint disp_vit_s050_100 at 1.78 sigma, conformance PASS (/tmp/n1det). - N1 gate-3 record is now: check6 PASS (1.56 sigma), check7 FAIL (4.69 sigma), n1det PASS (1.78 sigma) -> the pass rate is about two in three. Break-even against the certain N0 result needs P > 0.87, so the N0 submission is the better expected score (1.29 x 0.63 ~ 0.81 versus a certain 1.1256), and the 4.7-sigma failure mode is a whole-score loss rather than a small one.
8.7 hRound 22 (13:32-13:41): costed out the remaining bitwise levers; final timing confirmation
With the N1 route closed (see round 21), this round worked through every remaining way to get faster *while staying bitwise identical* -- the only tier that skips the chaotic trajectory gate: | idea | why it is not usable | |---|---| | checkpoint only the RMSNorm's fp32 intermediates to free memory for more AC layers (~10 GB of fp32hare saved for the backward across 12 layers) | impossible under FSDP: withuse_orig_params=Falsethe module's parameters are bare views into the unsharded flat param, which FSDP frees on the post-forward reshard; a recompute inside an un-checkpointed layer would read dangling memory. That is exactly why FSDP's own layer-level AC re-gathers on recompute. | |BAGEL_AC_SKIP_DECODER=13| +1.25% for ~4 GiB; at 12 layers the record peak is already 110.2 GiB allocated / 111.9 timing (~130 GB of the 140.4 GB card). Any non-zero OOM probability multiplies the *whole* score, so the expected value is negative (1.138 x 0.97 < 1.126). | | a bitwise Triton RMSNorm (fused fp32 chain) | would need to reproduce PyTorch'sreduce_kernelfp32 tree bit for bit in both forward and backward across every real (n, D); the payoff is ~4-6%, the success probability is low and a single 1-ulp miss drops the tree to N1 and the coin flip. | | fusedtorch.nn.functional.rms_norm| measured 1 bf16 ulp off the chain (the chain casts to bf16 before the weight multiply, the fused op after) -> N1. | | CUDA graphs | graphs are shape-specialised and the packed sequence length changes every step; padding to fixed shapes changes the arithmetic. | | norm-level fusions /new_empty/ MoT gather+norm+scatter | each measured at <=0.3% in earlier rounds. | Final confirmation of the submitted configuration:time/tmp/r32/time2.3165 s/step, speedup 1.1272, window range 0.61 %, peak allocated 111.91 GiB -- consistent with the full check's gate 1 (1.1256).
8.9 hRound 23 (13:41-14:00): profile of the N0 configuration; last bitwise-fusion probe
profile --from 8 --iters 2of the submitted N0 tree ->/tmp/r33/profile. Window 4.863 s / 2 steps (2.43 s/step under the profiler), GPU busy 94.0%. Per step: compute 1.894 s and NCCL 1.091 s, of which **0.391 s is exposed**; categories: elementwise/other 0.725 s, GEMM 0.717 s, attention 0.508 s, NCCL 1.091 s. The elementwise block is much larger than in the N1 tree (0.451 s) because the eager fp32 RMSNorm chain and the extra recompute are back: the top family isvectorized_elementwise_kernelwith 13,364 calls per step. - Last bitwise-fusion probe: fusing only the pointwise tail of the norm,w * (h*r).to(bf16), as a compiled function of tensors. Not bitwise (forward or gradients): Inductor fuses the intermediate bf16 cast away, so the weight multiply happens in fp32 and rounds once instead of twice (eager 0.611 ms -> compiled 0.218 ms at n=10240, i.e. 2.3% of the step is there, but it cannot be had in the bitwise tier). A hand-written Triton kernel could force the cast, but the same function's backward contains theh*rbroadcast reduction (d_r = sum_D(d_t*h)) whose order would have to match PyTorch bit for bit -- the same blocker as the full-chain version. - Confirmed from the gate-3 rank files (/tmp/check7/traj_cand_s1.r0.json) that the trajectory gate runs withdeterministic_algorithms = False, i.e. production settings, which is why two identical runs differ by up to 5.16 sigma. There is no way to make it reproducible from inside the tree (torch 2.5.1 has noinductor.config.deterministic;TORCHINDUCTOR_DETERMINISTICis judge-managed). Conclusion for this round: the submitted N0 configuration remains the best expected score; the profile shows no untried block, and the one remaining fusion was measured to be non-bitwise.
9.4 hRound 24 (13:52-14:24): a bitwise custom kernel -- fused RMSNorm tail (N0, +1.8%)
The N0 profile showed 0.725 s/step of elementwise, the eager fp32 RMSNorm chain being the dense part. Splitting that chain, the tailw * (h*r).to(bf16)is pure pointwise (the reductionmean(h^2)stays in PyTorch, untouched), and the tail alone costs three kernels: fp32 multiply, bf16 cast, weight multiply. - Two Triton kernels reproduce it bit for bit: the forward writes the output *and* the saved bf16 intermediatetb(both verifiedtorch.equalto the eager result at n = 3443/5000/10240, D = 3584), and the backward writesd_tb,d_t,d_h(verifiedtorch.equal). - The Function's backward keeps both reductions in PyTorch --d_r = (d_t*h).sum(dim=1, keepdim=True)andd_w = (grad*tb).sum(dim=0)-- and a hand-written version of it reproduces autograd's three gradients bitwise (verified), so nothing about the reduction order changes. - The 3-D q/k norm inputs are handled through the flattened view, which was verified bitwise-equal to the eager 3-D chain (including themean(-1)values). -time/tmp/r36/time: 2.2789 s/step, speedup 1.1458 (from 2.3165 / 1.1256), window range 0.05%, peak unchanged at 111.91 GiB.check --quickwith the switch: PASS, tier N0, bitwise true (/tmp/qNF). - Full check started 14:23 ->/tmp/check10.