R3

ModelJudgedTierSelf-bestRoundsHoursAPI cost
Claude Opus 5.52.41×N02.47×810.0 h$129trace →
Claude Sonnet 5.51.96×N01.92×910.0 h$37.9trace →
GPT-6.2 Astra1.77×N01.81×1510.0 h$50.0trace →
Claude Haiku 5.51.64×N01.62×1610.0 h$12.6trace →
MiniMax M3.1 Flash1.62×N01.61×1510.0 h$4.84trace →
GPT-6.2 Sol1.59×N01.61×1410.0 h$12.4trace →
GPT-6 Astrasuspected nerf1.47×N01.54×2210.0 h$89.3trace →
GLM-5.31.45×N01.46×1710.0 h$31.1trace →
Kimi k3.1-pro1.41×N01.38×1510.0 h$15.4trace →
DeepSeek V4.1 Flash-10171.19×N01.28×1610.0 h$1.93trace →
ModelJudgedText-decode CUDA graphReward wait hidden behind forwardRollout-side bitwise fused kernelsTrain-side bitwise kernels / freeing memoryOther main levers
Claude Opus 5.52.41×✓ BAGEL_FAST_DECODE (together with denoise desync and no cache clearing, single-run timing 1.42×)✓ BAGEL_OVERLAP_REWARD; also splits denoising into parts sent for scoring one after another (BAGEL_SPLIT_PARTS), exposed wait in the generation step 1.2–1.5 s → 0.17–0.37 s✓ custom CUDA extension: RMSNorm, q/k norm + RoPE + scatter, silu·mul✓ BAGEL_FUSED_TRAIN (hand-derived backward), BAGEL_SPLIT_TRAIN, fused AdamW; BAGEL_FREE_REF_SHARDS peak down to 113.5 GiB, backward 3.7 → 3.0 sdedup of repeated rows (17664 of 38912 rows in the refine batch shared), CFG branch merging (single-run timing 2.25× → 2.40×), small-batch GEMM, merged multi-branch attention, custom bitwise attention kernel, GC freeze
Claude Sonnet 5.51.96×✓ OPT_DECODE_GRAPH, first half of sampling also in the graph (self-estimated ~−9 s per window)✓ OPT_OVERLAP_REF + OPT_EARLY_FWD; OPT_GPU_YIELD=3000 yields the GPU every layer; together with skipping decode warmup, full check 1.74× → 1.85×✓ Triton: RMSNorm, q/k norm + RoPE, whole denoise layer fused (OPT_FUSED, OPT_FUSED_GEN)✓ OPT_FUSED_TRAIN (1.7035× → 1.7087×, within noise); OPT_REF_RESHARD peak down to ~104 GiB; FSDP prefetch + expandable segments + no cache clearing 1.85× → 1.92×packing desync, text CFG context copied directly; pipelined decode, fused MLP tried, no gain
Claude Haiku 5.51.64×✓ R3OPT_DECODE_GRAPH, 1.10× → 1.31× (single-run timing)✓ R3OPT_LATE_REWARD, 1.31× → 1.42×✓ Triton decode / denoise elementwise fusion, full check 1.45× → 1.50×; rows of the two experts laid out as contiguous blocks + merged kernels → 1.61×✗ listed under "remaining directions", noted as high effort, not done (peak 130.8 GiB)AdamW step counter moved back to CPU; one-step decode lookahead, expandable segments (1.48×, slower) tried, reverted
MiniMax M3.1 Flash1.62×✓ BAGEL_CUDA_GRAPH_DECODE, single-run timing 1.10× → 1.37× (first backward OOM; switched to manual capture, released at end of rollout)✓ BAGEL_FAST_OVERLAP + _POLICY, full check 1.44×✓ Triton RoPE, RMSNorm (1.50× → 1.57×), generation branch computed on all rows then text rows overwritten (→ 1.61×, all single-run timing)✗ (peak 130.9 GiB)timestep, KV merge, packing desync; desync of per-layer length reads in denoising measured slower, turned off
GPT-6 Astrasuspected nerf1.47×✓ BAGEL_DECODE_GRAPHS, 1.12× → 1.33× (single-run timing)✗ not in the log✓ CUDA RMSNorm (full check 1.34× → 1.45×), indexed RMSNorm (→ 1.47×); RoPE fusion single-run timing 1.54×, still waiting on full check when the log ends, not committed✗ local Adam adaptation measured slower, not merged (peak 131.1 GiB)static KV metadata computed once per forward, timestep on GPU
GLM-5.31.45×✓ OPT_TEXTGEN_CUDAGRAPH, min window 1.08× → 1.38× (backward OOM three times; switched to lower-level capture, enabled only from step 1)✓ OPT_OVERLAP_PREPACK + OPT_EARLY_FORWARD, 1.38× → 1.44×✗ not in the log✗ (peak 131.1 GiB)desync of timestep, attention length, token loop, merge index; denoise CUDA graph ~0.9 s slower per window, reverted
Kimi k3.1-pro1.41×✓ BAGEL_OPT_DECODE_GRAPH, 1.08× → 1.32× (gave up after two OOMs, got it running in round 12 by clearing cache before backward)✗ not in the log✗ not in the log✗ (peak 131.6 GiB)timestep on GPU, packed scalar stacking, KV merge index cache (full check 1.31× → 1.38×); denoise graph, shared attention mask etc. measured no gain
DeepSeek V4.1 Flash-10171.19×tried, reverted: bitwise identical, 1.42× on the timing side, but backward OOM in the deterministic record run; only the one-graph-per-full-batch version passed once (1.28×), OOM again on rerun, finally turned off✓ full check 1.14× → 1.17×tried, reverted: RoPE fusion 1.175× (no change), RMSNorm tail no gain and made the record run OOM✗ (peak 130.7 GiB)static KV length read once per forward, timestep t.repeat(n), packed scalars read once
↑ Leaderboard