GLM-5.3 1.49×
Harness: Claude Code · switches: BAGEL_AC_MODE=mlp BAGEL_MASK_BF16=1 BAGEL_MOT_QKV=1 BAGEL_LOG_FAST=1 BAGEL_FLEX_ATTN=1 BAGEL_FLEX_LEN=11520 BAGEL_EARLY_ROOT_AG=1 BAGEL_ADAM_ONEPASS=1 BAGEL_EMA_ONEPASS=1 BAGEL_MOT_ROUTING=1 BAGEL_NORM_FUSE=1
- Judged
- 1.49×
- Tier
- N1
- Self-best
- 1.49×
- Rounds
- 10
- Hours
- 10.1
- API cost
- $39.2
- Input tokens · 99.5% cached
- 132 M
- Output tokens
- 0.9 M
- Requests
- 993
§ 02Key levers
- Attention kernel swap (split by mask structure)
- ✓
BAGEL_FLEX_ATTN: block-sparse FlexAttention with fixed padding to 11520, 1.21 → 1.36×; flex kernel still 327 ms/step afterwards - Drop activation recompute (28 decoder layers)
- partial:
BAGEL_AC_MODE=mlp, recompute only MLP sublayers (1.19× together with mask conversion). Removing all: OOM (134.5 GiB); recomputing 10–16 fewer layers was slower - Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
- ✓
BAGEL_MOT_QKV(bitwise),BAGEL_MOT_ROUTING(1.407 → 1.425×),BAGEL_NORM_FUSE(→ 1.492×) - Optimizer, EMA, step start / end
- single-pass AdamW + single-pass EMA (1.365 → 1.407×). EMA via lerp blocked by gate 3
- Other main levers
- merged log reductions, root-unit all-gather moved earlier
§ ARaw trace
Raw work log12 sections
0.2 hRound 1 (2026-11-10 01:35-01:55) — profile the unchanged reference
-profile --tree /submission/bagel_sft --out /tmp/r1/profile(steps 8-9, rank 0). Step span in the profile 2967 ms (profiler overhead ~14% over the 2600 ms real step); GPU idle only 5.9%. - Per-step kernel accounting (one step, rank 0, from the Chrome trace): - attention (SDPA mem-efficient, per-sample loop, dense additive masks): fwd 371 ms (224 calls) + bwd 506 ms (112 calls) = 877 ms, 30% of the step. 224 fwd vs 112 bwd calls => the decoder forward runs TWICE: activation checkpointing recompute (checkpoint_wrapper is applied inside every FSDP unit:layers.N._fsdp_wrapped_module._checkpoint_wrapped_module). - GEMM (sm90 bf16): ~810 ms, 27% (MLP gate/up/down of both MoT experts, q/k/v/o, lm_head, ViT). - elementwise/copy/index/cat/fill: 843 ms, 28%, 16848 kernel launches per step (the MoT gather/scatter/zero plumbing: index_put 180 ms, direct_copy 190 ms, cat 56 ms, fill 43 ms, ...). - NCCL: 445 ms busy, only 130 ms exposed (4.4%) - FSDP comm overlaps compute well. All collectives run RING_LL (NCCL chose the LL protocol even for the 2.2 GB root-unit all-gather, 152 ms). - optimizer (AdamW foreach) + EMA ~70 ms; per-step logging block ~10 tiny all_reduce (33 ms/step incl. waits) +.item()syncs. - 60 FSDP units per rank (root unit = embed_tokens+lm_head = 1.09B params; 28 MoT decoder layers of 466M; ViT + connector units). Rank shards from bench/records/ref.json. - Batches on the interleave timing data: ~4 samples/rank, ~10k tokens (9682/9187/10059/10228 seen in the log). - Chosen hotspot for round 2: **activation checkpointing recompute (~20% of the step: half the attention fwd, a quarter of the GEMM and elementwise time)**. Peak memory is 82.14 GiB of 140 GiB, so there is headroom to trade memory for compute (L0 "turn off memory-saving switches", N0). - No code change; nothing committed. Profile artifacts: /tmp/r1/profile (deleted later).
0.5 hRound 2 (2026-11-10 01:52-01:58) — activation checkpointing: mlp-only (L0 switch, N0)
- Change 1,BAGEL_AC_MODE(train/pretrain_unified_navit.py): upstream wraps every decoder/ViT block in checkpoint_wrapper (recompute of the whole block in backward). New modemlpcheckpoints only Qwen2MLP/SiglipMLP sublayers: recompute is the two MLP GEMMs, the attention side keeps its activations. Defaultfull= upstream, unchanged. -BAGEL_AC_MODE=none(full off) was timed first: OOM at step 0 at 134.35 GiB (would risk the final evaluation on another data seed) -> rejected, keptmlp(peak 103.8 GiB). - Change 2,BAGEL_MASK_BF16(modeling/bagel/bagel.py): cast the packed masks to bf16 once in Bagel.forward so the per-layer.to(bfloat16)(upstream, inside every layer) becomes a no-op and layers share one mask tensor. 0/-inf are exact in bf16, bit for bit identical values. - Time (/tmp/r2/time2, AC=mlp + MASK_BF16): 2.1833 s/step, windows 2.1802/2.1864, peak 103.81 GiB, speedup 1.1924 (frozen ref 2.6034). Bitwise-identity argument: same kernels on same inputs in deterministic mode;check --quickqueued for the full stack (round 5). - Commits 3fe57f4 (AC_MODE) and 0522a52 (MASK_BF16).
0.5 hRound 3 (01:55-02:00) — MoT attention qkv on compacted token subsets (N0)
-BAGEL_MOT_QKV(modeling/bagel/qwen2_navit.py, default off): in PackedAttentionMoT.forward_train, upstream projects q/k/v for the FULL packed sequence separately for the und and gen experts, then scatters each expert's rows into a zero tensor (over half of each of the 4 q/k/v GEMMs is discarded). The new path gathers each expert's token subset (packed_und_token_indexes / packed_gen_token_indexes), projects, q/k-norms and RoPEs it on the subset, and scatters back into the same full-layout zero tensors. Same GEMM rows / norm / RoPE inputs per token, und+gen partition the sequence, zeros stay zeros -> bit for bit the same q/k/v. o_proj, GQA repeat, the per-sample SDPA loop untouched. - Time: as part of the 4-switch stack in/tmp/r3/time(below). - Commit 5363eb6.
0.5 hRound 4 (01:57-02:01) — batched logging collectives (N0)
-BAGEL_LOG_FAST(train/pretrain_unified_navit.py, default off): the loop logs every step; upstream runs ~10 small all_reduces + ~10 blocking.item()reads per step, plus 2 collectives (tokens, sum of squared sample lens) before every forward. New path keeps the two window counters local (Python floats), stacks the per-step log quantities into one SUM all_reduce + one MAX all_reduce, and reads everything back with a single.item(). Same reduced values, only scheduling changes. - Time: as part of the 4-switch stack in/tmp/r3/time(below). - Commit b7c8732.
0.5 hRound 5 (02:00-...) — stack timing + quick check
- Switch file (all four on):
{"environment": {"BAGEL_AC_MODE": "mlp", "BAGEL_MASK_BF16": "1", "BAGEL_MOT_QKV": "1",
"BAGEL_LOG_FAST": "1"}}
- time -> /tmp/r3/time: see summary.0.5 hRound 5 result (02:00-02:14) — stack timing + quick check
-time(/tmp/r3/time): 2.1457 s/step, speedup 1.2133, windows 2.1515/2.1399 (range 0.54%), peak 103.98 GiB. Gains over AC+MASK alone (2.1833): +38 ms from MOT_QKV + LOG_FAST. -check --quick(/tmp/r5/quick/check.json): status PASS, tier N0 — the full 4-switch stack is bitwise identical to the frozen reference record (exact items + all SHA-256s). - Current switch file (the submission state): AC_MODE=mlp, MASK_BF16=1, MOT_QKV=1, LOG_FAST=1.
0.8 hRound 6 (02:14-02:20) — NCCL_PROTO=^LL: no gain, dropped
- Motivation: the profile shows all NCCL collectives on RING_LL (decoder-unit reduce-scatters
~54 ms each for 233 MB, root-unit all-gather 152 ms for 2.2 GB ≈ 15-17 GB/s), and NCCL busy is
~1.3 s/step (mostly overlapped; ~130 ms exposed).
- time with NCCL_PROTO=^LL added (/tmp/r6/time): 2.1947 s/step, **speedup 1.1862 — worse than
without it** (windows 2.1666/2.2227, range 2.6%). Simple kernels presumably contend for SMs with
compute and cost more launch latency on the many medium FSDP collectives; LL was already right.
- No code change; switches.json left without NCCL_PROTO. Not retried with LL128 yet (possible
later cheap follow-up to combine with another round's switch).1.1 hRound 7 (02:25-...) — BAGEL_FLEX_ATTN: block-sparse flex_attention for the decoder (N1)
- Motivation: attention is ~690 ms of the 2146 ms step (32%): a per-sample SDPA loop with dense additive masks (mem-efficient kernel, ~55 TFLOPS effective) over the full L^2 of every sample, plus a 7x GQA materialization per layer. The packed batches visit only ~20% of the block pairs (cross-sample blocks all masked, cross-split upper blocks masked), so a block-sparse kernel should do a fraction of the work. - The upstream tree already has the whole flex path: the decoder's non-List mask branch runs one flex_attention call with enable_gqa over the packed sequence, and Bagel.forward has the create_sparse_mask + create_block_mask machinery. It is dead code in this recipe because the batch carries dense masks (use_flex=False) and the batch fields are frozen (gate 2 digest). - New code (behind BAGEL_FLEX_ATTN, default off): -_derive_packed_splits(bagel.py): reconstructs each sample's (split_lens, attn_modes) from the batch's token layout (vit runs + vit_token_seqlens, vae runs + latent shapes + mse positions; text splits are the gaps; a vae span with loss is 'noise'). Verified element-wise: reproduces the dataset's dense masks exactly on 92 samples of both data configs, and create_sparse_mask over the derived splits equals the dense masks element-wise on 53 more. -_build_packed_block_mask(bagel.py): builds the BlockMask directly from the split structure in O(seq_len) instead of evaluating the mask over the L^2 grid (upstream's create_block_mask compiles a static shape per step with dynamic=False -> one recompile per distinct length; its eager path materializes a (1, 32, L, L) bool mask). Visited blocks over-approximate and full blocks under-approximate the true visibility (exactly what the flex kernel requires); verified against brute-force block truth from the dense masks on 75 samples: 0 missed visible pairs, 0 wrongly-full pairs, and the persistent mask_mod equals create_sparse_mask element-wise. - The mask_mod closes over persistent buffers (allocated once, refilled per step): new tensors every step would recompile the flex kernel on their dynamo guards. - Fixed padding: torch 2.5's flex template autotuner finds no candidates for symbolic shapes (NoValidChoicesError), so the mask and q/k/v are padded to one fixed length, BAGEL_FLEX_LEN (default 11520 = the recipe's max_num_tokens, exactly what upstream's data-side flex path pads to). The padding sample is causal and isolated, as upstream's pad split. - qwen2_navit.py: the MoT flex branch pads to the mask's length (upstream's pad_sequence), and BAGEL_FLEX_AUTOTUNE=1 compiles flex_attention with max-autotune-no-cudagraphs. - GPU smoke test (1 MoT layer, real interleave batch, 28/4 heads): flex output vs SDPA max abs diff 1.2e-4 (rel ~5e-3, bf16 kernel difference); backward fine; rerun-identical including in deterministic mode (no exception); layer fwd+bwd 27.6 -> 20.2 ms on one batch (flex visits 22% of block pairs; full-block fraction 73% after marking noise-span-internal pairs full). - Judge runs in flight: time (/tmp/r7/time) and check --quick (/tmp/r7/quick) on /tmp/switches_flex.json (the 4-switch stack + BAGEL_FLEX_ATTN=1 + BAGEL_FLEX_LEN=11520). /submission/switches.json is unchanged (the validated 1.2133 stack) until flex is evidenced. - Tier: N1 (attention kernel change; not bitwise). Gates 2b + 3 will decide. -timeon the flex config (/tmp/r7/time): 1.9127 s/step, speedup 1.3611 (from 1.2133), windows 1.9203/1.9051 (range 0.79%), peak memory 97.18 GiB (down from 103.98: flex frees the per-layer mask memory). Note this run's tree copy predates the dense-mask H2D skip below. - Dropped: max-autotune for flex. The autotuner's winning forward config (BLOCK_M=128, BLOCK_N=64, warps=8, stages=3) IS torch 2.5's_h100_default_config[(bfloat16,128)]default, so the default compile already picks it (verified from an autotune log: default 3.76 ms vs the other candidates 9.0 ms); the backward autotune precompiles for ~20+ minutes, TORCHINDUCTOR_CACHE_DIR/TRITON_CACHE_DIR are reserved (cold cache every judge run), so autotune is not viable under the judge. - Extra change (part of round 7, after the timing run): with flex on, SimpleCustomBatch no longer pins/uploads the dense masks (the judge hashes the batch on the host before the move; the masks are never read on the GPU). Saves ~100+ MB of pinned H2D per rank per step. -check --quickon the flex config (/tmp/r7/quick/check.json): NOT_BITWISE — every exact item passes (steps, optimizer hyper-parameters, batch digests, parameter names/shapes/dtypes, optimizer-state dtypes); the parameter/gradient/optimizer/EMA hashes differ, as expected for an attention kernel change. Tier N1; gates 2b (conformance on interleaved layouts) and 3 (statistical) decide in the full check. - switches.json updated to the flex stack (AC=mlp + MASK_BF16 + MOT_QKV + LOG_FAST + FLEX_ATTN + FLEX_LEN=11520), the best evidenced state: speedup 1.3611, exact items all pass. Committed (f497500).
1.3 hRound 8 (02:50-...) — less activation checkpointing + expandable segments (N0)
- Motivation: AC=mlp still recomputes both experts' MLP GEMMs in backward (~117 TFLOP/step
≈ 170-190 ms). AC=none OOM'd at 134.35 GiB allocated pre-flex, but flex freed 6.8 GiB (peak
97.18), and reserved runs at ~134 GiB of 140.4 for only 97 allocated — a fragmentation gap
from the per-step-varying shapes. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (allowed:
not reserved) should close most of that gap.
- time with AC=none + expandable_segments on the flex stack -> /tmp/r8/time (in flight).1.3 hRound 8 — activation-checkpointing / memory trade (no gain; direction closed)
Start 02:35, end 03:25. Hotspot: the MLP recompute that BAGEL_AC_MODE=mlp still pays
(~108 TFLOP/step of GEMM recompute ≈ 155 ms estimated). AC=none would remove it but needs
more activation memory than AC=mlp keeps freed (measured delta 37.3 GiB).
Probes (judge time, all on the current stack flex+motqkv+logfast):
- AC=none + expandable_segments → ERROR, real OOM: 134.54 GiB allocated (only 308 MiB
reserved-unallocated, so not fragmentation — genuine capacity). Matches the linear model
97.18 + 37.3 = 134.5. (/tmp/r8/time)
- AC=mlp:10 (new knob: checkpoint the MLPs of only the first N decoder layers) → 2.3256 s/step
(1.1194×), peak 125.25 GiB. LESS recompute but SLOWER: per-step times bimodal (1.6–5.3 s) —
caching-allocator thrash (only ~10 GiB slack under the peak, forced cache releases every step).
(/tmp/r8b/time)
- AC=mlp:10 + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True → 2.7818 s/step (0.9359×): ES is
*worse* here (map/unmap churn, steps 5.4/6.1/6.6 s). (/tmp/r8c/time)
- AC=mlp:16 → 2.0619 s/step (1.2626×), peak 115.55 GiB, steps still spiky (max 4.07 s). (/tmp/r8d/time)
Conclusion: the allocator-pressure knee is below ~110 GiB, where the recompute saving
(≤ 60 ms at mlp:16, ≤ 36 ms at mlp:20) is already eaten by the thrash tax. AC=mlp (all 26
layers) stays. The mlp:<N> knob is kept in the code, default unchanged ("full" = upstream);
submission switches unchanged (AC_MODE=mlp).
Lesson recorded: on this 139.8 GiB GPU with per-step-varying batch shapes, any change that
raises peak allocated above ~110 GiB costs hundreds of ms in allocator churn. Memory headroom
is a performance resource; AC=none-style levers are dead on this machine.2.2 hRound 9 — profile of the flex tree; FSDP root split + forward prefetch (no gain; closed)
Start 03:25, end 03:55. Profile first (/tmp/r9/profile, steps 8-9, rank 0):
- per step: gemm 688 ms busy, elementwise ~495 ms over ~12.7k launches (aten::mul 106 ms,
copy_ 148 ms incl. FSDP's fp32->bf16 shard casts, index/cat ~110 ms), flex triton 327 ms
(84 kernels = 26 layers x fwd+2 bwd), optimizer+EMA ~90 ms.
- NCCL: 905 ms busy but union-exposure ~234 ms/step (reference was 109): the root unit's
all-gather (embed+lm_head+norms, 1.09B params, bf16) runs at forward entry on an otherwise
idle GPU (~91 ms segment seen in the 2-step window), plus a ~50 ms u64 token-count
all-reduce that is mostly rank-skew spin. FSDP1 already all-gathers in bf16 (it casts the
fp32 shard first: _use_low_precision_shard), so there is no dtype lever.
Change (BAGEL_FSDP_SPLIT + BAGEL_FSDP_PREFETCH, default off): custom wrap policy that also
wraps language_model.model.embed_tokens and language_model.lm_head as their own FSDP units
(root unit 1.09B -> ~20M), plus FSDP forward_prefetch=True so each unit's all-gather is
issued one unit early. Judge-verified safe for gate 2's exact items: the judge records the
original parameter fqns inside each flat param (layout-independent); only the flat-shard
SHA grouping changes (N1, like flex). fsdp_ema_update's handle-count assert still holds
(both models get the same policy).
Result: split+prefetch = 1.9383 s/step (1.3431x) vs 1.9127 baseline — 26 ms SLOWER
(/tmp/r9c/time; first attempt crashed on isinstance(set), fixed with tuple()).
Peak memory 95.94 GiB (slightly lower). Conclusion: the ~45 ms/step average exposure saved is
eaten by forward_prefetch's rate-limiter CPU syncs and two extra units' fixed overhead
(shard casts, hooks, extra collectives). Direction closed; code kept, default off.
Next: full check on the 1.9127 config (flex stack has never run gates 2b/3 — the priority),
then fused AdamW / elementwise fusion rounds.8.6 hCurrent summary (updated after round 19, 10:50)
- **Submission switches (/submission/switches.json): AC_MODE=mlp, MASK_BF16=1, MOT_QKV=1, LOG_FAST=1, FLEX_ATTN=1, FLEX_LEN=11520, EARLY_ROOT_AG=1, ADAM_ONEPASS=1, EMA_ONEPASS=1, MOT_ROUTING=1, NORM_FUSE=1 — validated by the full check /tmp/r19/check: PASS, tier N1, speedup 1.4927** (the judge'stime: 1.7445 s/step = 1.4923x, /tmp/r19/time; frozen reference 2.6034 s/step). - Committed in the tree, default off: FLEX_STAGE=1 (rejected), FLEX_STAGE=2 (no gain), EMA_LERP (gate 3 rejects), FUSED_ADAM (sigma margin), FSDP_SPLIT/FSDP_PREFETCH (no gain), AC_MODE=mlp:<N> (allocator thrash), EARLY_TOKEN_AR (r14), EMA_SIDE (r15). - Full-check history: check-full-1 PASS (1.3329); check-full-2 INCORRECT (EMA lerp, -53 sigma); check-full-3 PASS (1.3888); r12 PASS (1.365); r13 PASS (1.4059); r16 INCORRECT (FLEX_STAGE v1, gate 3, -3.40 sigma — root cause: strided flex input = a different compiled kernel = a real value shift, not bitwise-neutral); r17 PASS (1.4302, +MOT_ROUTING); **r19 PASS (1.4927, +NORM_FUSE) — the submitted config's verdict**. - Gains by change (cumulativetimespeedups): LOG_FAST ~1.02x; MASK_BF16 small; MOT_QKV ~1.05x; FLEX_ATTN 1.3611x; mask-skip + EARLY_ROOT_AG 1.3645x (r12); ADAM_ONEPASS + EMA_ONEPASS 1.4068x (r13, PASS); MOT_ROUTING 1.4249x (r17, PASS); NORM_FUSE 1.4923x (r19, PASS). - The working method that produced the last three rounds: (1) change only what moves the same values through the same or replicated rounding sequences — elementwise fusions are bitwise, reductions never move out of torch; (2) prove it standalone (probe, deterministic mode, forward AND backward, real shapes); (3) prove it in situ (quick check record compared against r13's shard for shard — 0 diffs required); (4) full check; (5) adopt on PASS. FMA contraction (enable_fp_fusion=False) and dynamo stride guards (the flex staging lesson) are the two traps the probes caught. - Dead ends (measured, closed): NCCL_PROTO=^LL; AC=none (OOM); AC=mlp:10/16; FSDP split + forward_prefetch; max-autotune flex; EMA lerp (gate 3); fused AdamW (sigma margin); EARLY_TOKEN_AR (r14); EMA_SIDE (r15); head-major staging dim-1 puts (r16); FLEX_STAGE=1 (kernel-changing value shift, gate 3); FLEX_STAGE=2 (puts cost what the cat saves); torch.compile of the model (dynamic shapes, FSDP fragility); cudnn.benchmark (autotune inside timing windows). - Remaining measured pools (r19 trace, per step): GEMMs ~690 ms (untouchable without precision changes); flex triton 328 ms; routing index ops ~50 ms (gathers 28 + scatters 21.5 — phase-3 split residuals could remove ~8 ops/layer, est. ~15-25 ms, big blast radius, needs its own full check); residual adds ~2 ms; ViT LayerNorm ~10 ms; remaining fills ~2 ms (q/k/v targets provably fully overwritten — too small to bother); NCCL busy 877 ms (exposure only, rounds 14-15 closed the scheduling tricks). Next round: phase-3 split residual stream, only if developed carefully with the probe/quick/full-check ladder.