BAGEL SFT

ModelJudgedTierSelf-bestRoundsHoursAPI cost
Claude Sonnet 5.52.21×N12.24×1410.0 h$46.7trace →
Claude Opus 5.52.20×N12.22×89.3 h$48.9trace →
GPT-6.2 Sol1.86×N11.93×2010.0 h$8.83trace →
GPT-6.2 Astra1.84×N11.93×1810.0 h$45.2trace →
GPT-6 Astrasuspected nerf1.62×N11.65×1510.0 h$78.9trace →
Claude Haiku 5.51.60×N11.60×1410.0 h$20.0trace →
MiniMax M3.1 Flash1.49×N11.48×2210.0 h$4.50trace →
GLM-5.31.49×N11.49×1010.1 h$39.2trace →
Kimi k3.1-pro1.42×N01.43×2010.0 h$18.8trace →
DeepSeek V4.1 Flash-10171.15×N01.23×2210.0 h$2.06trace →
ModelJudgedAttention kernel swap (split by mask structure)Drop activation recompute (28 decoder layers)Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)Optimizer, EMA, step start / endOther main levers
Claude Sonnet 5.52.21×✓ OPT_STRUCT_ATTN: two varlen FlashAttention calls per layer, single-run timing 2.62 → 2.20 s; then switched to FA3 (OPT_FA3=2, the deterministic-backward version)✓ all 28 layers + ViT removed. OPT_FUSED_MOT_LOWMEM saves only a few intermediates (~1.2 GB/layer), 1.685 → 1.371 s✓ OPT_FUSED_MOT: whole-layer Triton kernel + hand-written backward, 2.20 → 1.74 s✓ FSDP grads written directly into flat buffer, pre-division removed (1.217 → 1.185 s); clip coefficient folded into AdamW (→ 1.172 s); Triton EMAall-gather in-flight limit 8, ViT merged into one FSDP unit, masks built in the pin thread (1.266 → 1.242 s), ViT layer fusion, Triton gather
Claude Opus 5.52.20×✓ BAGEL_OPT_FLASH_ATTN: 2.606 → 2.213 s; then switched to FA3 (−52 ms)✓ all 28 layers + ViT removed. BAGEL_OPT_SWIGLU_RECOMPUTE cuts per-layer footprint to 1.48 GiB, 1.727 → 1.417 s✓ the two token types each stored contiguously + Triton RoPE / SwiGLU (bitwise) + RMSNorm, 2.213 → 1.727 s✓ bitwise Triton AdamW + EMA (1.417 → 1.357 s); VAE GroupNorm fusion (−41 ms); next batch's VAE encoded during backward (−17 ms); EMA on a side stream (−11 ms)QKV fusion (1.264 → 1.233 s), ViT merged into one FSDP unit (−10 ms), host sync before ViT removed, VAE conv bias (−5.6 ms)
GPT-6 Astrasuspected nerf1.62×✓ BAGEL_PACKED_FLASH_ATTENTION: screening 1.07 → 1.27×; FA3 added later (1.576 → 1.611×). FlexAttention tried first at 1.24×, dropped after quick check erroredpartial: 1 in every 3 layers kept (9 layers). Keeping half tried twice, both slower (0.9986×, peak 127.5 GiB; 1.18× vs 1.35×)✓ off-the-shelf library kernels: RMSNorm, RoPE, SwiGLU, direct Q/K routing, expert-local state, screening 1.29 → 1.56×only torch's built-in fused AdamW (+1.85%); EMA, VAE not in the logweights all-gathered in forward kept for backward (1.02×); single shard group, NCCL channel count, early count reduction showed no distinguishable gain
Claude Haiku 5.51.60×✓ BAGEL_OPT_ATTN_FLASH (FA2): 1.09 → 1.26×; batched calls a further +4.5%. Wanted FA3, source couldn't be downloaded, droppedonly ViT removed (+2.9%). Decoder minus 4 layers 1.280× vs 1.289×, minus 2 layers −0.9%, both reverted; all 28 layers still recomputed✓ torch.compile on RMSNorm, SiLU·up, RoPE (1.29 → 1.44×); MoT permuted layout (bitwise, +6.5%), two token types carried separately (+2.1%)fused AdamW; FSDP grad division removed (bitwise, +1.4%). EMA via lerp blocked by gate 3; EMA side stream −1.0%, revertedunsharded weights kept for backward, all-gather rate limiter off, forward prefetch (combined 1.025×)
MiniMax M3.1 Flash1.49×✓ BAGEL_FLASH_ATTN: one FlashAttention call per segment, 1.09 → 1.29×; BAGEL_VARLEN_ATTN merged into one varlen call, 1.435 → 1.477×partial: final BAGEL_NO_AC_FROM_LAYER=18 (last 10 layers). Log has tests of last 7 layers (stable), last 14 layers (window spread 6.7%), all removed (1.23×, per-step time jumping between 1.6–2.8 s)torch.compile on RMSNorm, RoPE (1.08×); BAGEL_MOT_FUSED on, no separate record in the logfused AdamW, EMA side stream both on, no distinguishable individual gain in the log; VAE side stream +15.6 ms/step, turned off after gate 2b blocked itall-gather rate limiter off (1.080 → 1.094×)
GLM-5.31.49×✓ BAGEL_FLEX_ATTN: block-sparse FlexAttention with fixed padding to 11520, 1.21 → 1.36×; flex kernel still 327 ms/step afterwardspartial: BAGEL_AC_MODE=mlp, recompute only MLP sublayers (1.19× together with mask conversion). Removing all: OOM (134.5 GiB); recomputing 10–16 fewer layers was slower✓ BAGEL_MOT_QKV (bitwise), BAGEL_MOT_ROUTING (1.407 → 1.425×), BAGEL_NORM_FUSE (→ 1.492×)single-pass AdamW + single-pass EMA (1.365 → 1.407×). EMA via lerp blocked by gate 3merged log reductions, root-unit all-gather moved earlier
Kimi k3.1-pro1.42×✓ BAGEL_OPT_STATIC_FLEX: production mode uses FlexAttention fixed at 11520, 1.18 → 1.38×; deterministic mode falls back to the original SDPA, so the record stays bitwisepartial: 17 layers + ViT. 18–20 layers hit the 135 GiB reservation cap (20 layers left only 1.008×)bitwise reorderings only: Q/K normalized before scatter (1.376 → 1.399×), input normalization kept expert-local (→ 1.426×). Compiled SwiGLU no end-to-end gain, reverted✗ fused AdamW measured 1.168×, window spread 4.76%, reverted; direct mixed-precision EMA no gainskip zero fill that gets overwritten, GQA via views instead of copies (+0.55%, saves 2.1 GiB)
DeepSeek V4.1 Flash-10171.15×✗ bucketed cuDNN attention ~1.46× steady state, but window spread 7.5–10.3%, above the 6% limit, reverted; no split by mask structurepartial: 12 layers + ViT (1.067× at 8 layers). All 28 removed: OOMtried, reverted: N1 config with compiled RMSNorm / RoPE / SwiGLU + fused AdamW passed a full check once (1.29×); whole set dropped after the repeat check failed gate 3; only a bitwise RMSNorm tail kernel kept in the end (+1.8%)reverted along with the N1 config; EMA via lerp blocked by gate 3deferred log flush, non-blocking H2D
↑ Leaderboard