BAGEL SFT
§ 02Models
| Model | Judged | Tier | Self-best | Rounds | Hours | API cost | |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 2.21× | N1 | 2.24× | 14 | 10.0 h | $46.7 | trace → |
| Claude Opus 5.5 | 2.20× | N1 | 2.22× | 8 | 9.3 h | $48.9 | trace → |
| GPT-6.2 Sol | 1.86× | N1 | 1.93× | 20 | 10.0 h | $8.83 | trace → |
| GPT-6.2 Astra | 1.84× | N1 | 1.93× | 18 | 10.0 h | $45.2 | trace → |
| GPT-6 Astrasuspected nerf | 1.62× | N1 | 1.65× | 15 | 10.0 h | $78.9 | trace → |
| Claude Haiku 5.5 | 1.60× | N1 | 1.60× | 14 | 10.0 h | $20.0 | trace → |
| MiniMax M3.1 Flash | 1.49× | N1 | 1.48× | 22 | 10.0 h | $4.50 | trace → |
| GLM-5.3 | 1.49× | N1 | 1.49× | 10 | 10.1 h | $39.2 | trace → |
| Kimi k3.1-pro | 1.42× | N0 | 1.43× | 20 | 10.0 h | $18.8 | trace → |
| DeepSeek V4.1 Flash-1017 | 1.15× | N0 | 1.23× | 22 | 10.0 h | $2.06 | trace → |
§ 03Key levers
| Model | Judged | Attention kernel swap (split by mask structure) | Drop activation recompute (28 decoder layers) | Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU) | Optimizer, EMA, step start / end | Other main levers |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 2.21× | ✓ OPT_STRUCT_ATTN: two varlen FlashAttention calls per layer, single-run timing 2.62 → 2.20 s; then switched to FA3 (OPT_FA3=2, the deterministic-backward version) | ✓ all 28 layers + ViT removed. OPT_FUSED_MOT_LOWMEM saves only a few intermediates (~1.2 GB/layer), 1.685 → 1.371 s | ✓ OPT_FUSED_MOT: whole-layer Triton kernel + hand-written backward, 2.20 → 1.74 s | ✓ FSDP grads written directly into flat buffer, pre-division removed (1.217 → 1.185 s); clip coefficient folded into AdamW (→ 1.172 s); Triton EMA | all-gather in-flight limit 8, ViT merged into one FSDP unit, masks built in the pin thread (1.266 → 1.242 s), ViT layer fusion, Triton gather |
| Claude Opus 5.5 | 2.20× | ✓ BAGEL_OPT_FLASH_ATTN: 2.606 → 2.213 s; then switched to FA3 (−52 ms) | ✓ all 28 layers + ViT removed. BAGEL_OPT_SWIGLU_RECOMPUTE cuts per-layer footprint to 1.48 GiB, 1.727 → 1.417 s | ✓ the two token types each stored contiguously + Triton RoPE / SwiGLU (bitwise) + RMSNorm, 2.213 → 1.727 s | ✓ bitwise Triton AdamW + EMA (1.417 → 1.357 s); VAE GroupNorm fusion (−41 ms); next batch's VAE encoded during backward (−17 ms); EMA on a side stream (−11 ms) | QKV fusion (1.264 → 1.233 s), ViT merged into one FSDP unit (−10 ms), host sync before ViT removed, VAE conv bias (−5.6 ms) |
| GPT-6 Astrasuspected nerf | 1.62× | ✓ BAGEL_PACKED_FLASH_ATTENTION: screening 1.07 → 1.27×; FA3 added later (1.576 → 1.611×). FlexAttention tried first at 1.24×, dropped after quick check errored | partial: 1 in every 3 layers kept (9 layers). Keeping half tried twice, both slower (0.9986×, peak 127.5 GiB; 1.18× vs 1.35×) | ✓ off-the-shelf library kernels: RMSNorm, RoPE, SwiGLU, direct Q/K routing, expert-local state, screening 1.29 → 1.56× | only torch's built-in fused AdamW (+1.85%); EMA, VAE not in the log | weights all-gathered in forward kept for backward (1.02×); single shard group, NCCL channel count, early count reduction showed no distinguishable gain |
| Claude Haiku 5.5 | 1.60× | ✓ BAGEL_OPT_ATTN_FLASH (FA2): 1.09 → 1.26×; batched calls a further +4.5%. Wanted FA3, source couldn't be downloaded, dropped | only ViT removed (+2.9%). Decoder minus 4 layers 1.280× vs 1.289×, minus 2 layers −0.9%, both reverted; all 28 layers still recomputed | ✓ torch.compile on RMSNorm, SiLU·up, RoPE (1.29 → 1.44×); MoT permuted layout (bitwise, +6.5%), two token types carried separately (+2.1%) | fused AdamW; FSDP grad division removed (bitwise, +1.4%). EMA via lerp blocked by gate 3; EMA side stream −1.0%, reverted | unsharded weights kept for backward, all-gather rate limiter off, forward prefetch (combined 1.025×) |
| MiniMax M3.1 Flash | 1.49× | ✓ BAGEL_FLASH_ATTN: one FlashAttention call per segment, 1.09 → 1.29×; BAGEL_VARLEN_ATTN merged into one varlen call, 1.435 → 1.477× | partial: final BAGEL_NO_AC_FROM_LAYER=18 (last 10 layers). Log has tests of last 7 layers (stable), last 14 layers (window spread 6.7%), all removed (1.23×, per-step time jumping between 1.6–2.8 s) | torch.compile on RMSNorm, RoPE (1.08×); BAGEL_MOT_FUSED on, no separate record in the log | fused AdamW, EMA side stream both on, no distinguishable individual gain in the log; VAE side stream +15.6 ms/step, turned off after gate 2b blocked it | all-gather rate limiter off (1.080 → 1.094×) |
| GLM-5.3 | 1.49× | ✓ BAGEL_FLEX_ATTN: block-sparse FlexAttention with fixed padding to 11520, 1.21 → 1.36×; flex kernel still 327 ms/step afterwards | partial: BAGEL_AC_MODE=mlp, recompute only MLP sublayers (1.19× together with mask conversion). Removing all: OOM (134.5 GiB); recomputing 10–16 fewer layers was slower | ✓ BAGEL_MOT_QKV (bitwise), BAGEL_MOT_ROUTING (1.407 → 1.425×), BAGEL_NORM_FUSE (→ 1.492×) | single-pass AdamW + single-pass EMA (1.365 → 1.407×). EMA via lerp blocked by gate 3 | merged log reductions, root-unit all-gather moved earlier |
| Kimi k3.1-pro | 1.42× | ✓ BAGEL_OPT_STATIC_FLEX: production mode uses FlexAttention fixed at 11520, 1.18 → 1.38×; deterministic mode falls back to the original SDPA, so the record stays bitwise | partial: 17 layers + ViT. 18–20 layers hit the 135 GiB reservation cap (20 layers left only 1.008×) | bitwise reorderings only: Q/K normalized before scatter (1.376 → 1.399×), input normalization kept expert-local (→ 1.426×). Compiled SwiGLU no end-to-end gain, reverted | ✗ fused AdamW measured 1.168×, window spread 4.76%, reverted; direct mixed-precision EMA no gain | skip zero fill that gets overwritten, GQA via views instead of copies (+0.55%, saves 2.1 GiB) |
| DeepSeek V4.1 Flash-1017 | 1.15× | ✗ bucketed cuDNN attention ~1.46× steady state, but window spread 7.5–10.3%, above the 6% limit, reverted; no split by mask structure | partial: 12 layers + ViT (1.067× at 8 layers). All 28 removed: OOM | tried, reverted: N1 config with compiled RMSNorm / RoPE / SwiGLU + fused AdamW passed a full check once (1.29×); whole set dropped after the repeat check failed gate 3; only a bitwise RMSNorm tail kernel kept in the end (+1.8%) | reverted along with the N1 config; EMA via lerp blocked by gate 3 | deferred log flush, non-blocking H2D |