R3
§ 02Models
| Model | Judged | Tier | Self-best | Rounds | Hours | API cost | |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 2.41× | N0 | 2.47× | 8 | 10.0 h | $129 | trace → |
| Claude Sonnet 5.5 | 1.96× | N0 | 1.92× | 9 | 10.0 h | $37.9 | trace → |
| GPT-6.2 Astra | 1.77× | N0 | 1.81× | 15 | 10.0 h | $50.0 | trace → |
| Claude Haiku 5.5 | 1.64× | N0 | 1.62× | 16 | 10.0 h | $12.6 | trace → |
| MiniMax M3.1 Flash | 1.62× | N0 | 1.61× | 15 | 10.0 h | $4.84 | trace → |
| GPT-6.2 Sol | 1.59× | N0 | 1.61× | 14 | 10.0 h | $12.4 | trace → |
| GPT-6 Astrasuspected nerf | 1.47× | N0 | 1.54× | 22 | 10.0 h | $89.3 | trace → |
| GLM-5.3 | 1.45× | N0 | 1.46× | 17 | 10.0 h | $31.1 | trace → |
| Kimi k3.1-pro | 1.41× | N0 | 1.38× | 15 | 10.0 h | $15.4 | trace → |
| DeepSeek V4.1 Flash-1017 | 1.19× | N0 | 1.28× | 16 | 10.0 h | $1.93 | trace → |
§ 03Key levers
| Model | Judged | Text-decode CUDA graph | Reward wait hidden behind forward | Rollout-side bitwise fused kernels | Train-side bitwise kernels / freeing memory | Other main levers |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 2.41× | ✓ BAGEL_FAST_DECODE (together with denoise desync and no cache clearing, single-run timing 1.42×) | ✓ BAGEL_OVERLAP_REWARD; also splits denoising into parts sent for scoring one after another (BAGEL_SPLIT_PARTS), exposed wait in the generation step 1.2–1.5 s → 0.17–0.37 s | ✓ custom CUDA extension: RMSNorm, q/k norm + RoPE + scatter, silu·mul | ✓ BAGEL_FUSED_TRAIN (hand-derived backward), BAGEL_SPLIT_TRAIN, fused AdamW; BAGEL_FREE_REF_SHARDS peak down to 113.5 GiB, backward 3.7 → 3.0 s | dedup of repeated rows (17664 of 38912 rows in the refine batch shared), CFG branch merging (single-run timing 2.25× → 2.40×), small-batch GEMM, merged multi-branch attention, custom bitwise attention kernel, GC freeze |
| Claude Sonnet 5.5 | 1.96× | ✓ OPT_DECODE_GRAPH, first half of sampling also in the graph (self-estimated ~−9 s per window) | ✓ OPT_OVERLAP_REF + OPT_EARLY_FWD; OPT_GPU_YIELD=3000 yields the GPU every layer; together with skipping decode warmup, full check 1.74× → 1.85× | ✓ Triton: RMSNorm, q/k norm + RoPE, whole denoise layer fused (OPT_FUSED, OPT_FUSED_GEN) | ✓ OPT_FUSED_TRAIN (1.7035× → 1.7087×, within noise); OPT_REF_RESHARD peak down to ~104 GiB; FSDP prefetch + expandable segments + no cache clearing 1.85× → 1.92× | packing desync, text CFG context copied directly; pipelined decode, fused MLP tried, no gain |
| Claude Haiku 5.5 | 1.64× | ✓ R3OPT_DECODE_GRAPH, 1.10× → 1.31× (single-run timing) | ✓ R3OPT_LATE_REWARD, 1.31× → 1.42× | ✓ Triton decode / denoise elementwise fusion, full check 1.45× → 1.50×; rows of the two experts laid out as contiguous blocks + merged kernels → 1.61× | ✗ listed under "remaining directions", noted as high effort, not done (peak 130.8 GiB) | AdamW step counter moved back to CPU; one-step decode lookahead, expandable segments (1.48×, slower) tried, reverted |
| MiniMax M3.1 Flash | 1.62× | ✓ BAGEL_CUDA_GRAPH_DECODE, single-run timing 1.10× → 1.37× (first backward OOM; switched to manual capture, released at end of rollout) | ✓ BAGEL_FAST_OVERLAP + _POLICY, full check 1.44× | ✓ Triton RoPE, RMSNorm (1.50× → 1.57×), generation branch computed on all rows then text rows overwritten (→ 1.61×, all single-run timing) | ✗ (peak 130.9 GiB) | timestep, KV merge, packing desync; desync of per-layer length reads in denoising measured slower, turned off |
| GPT-6 Astrasuspected nerf | 1.47× | ✓ BAGEL_DECODE_GRAPHS, 1.12× → 1.33× (single-run timing) | ✗ not in the log | ✓ CUDA RMSNorm (full check 1.34× → 1.45×), indexed RMSNorm (→ 1.47×); RoPE fusion single-run timing 1.54×, still waiting on full check when the log ends, not committed | ✗ local Adam adaptation measured slower, not merged (peak 131.1 GiB) | static KV metadata computed once per forward, timestep on GPU |
| GLM-5.3 | 1.45× | ✓ OPT_TEXTGEN_CUDAGRAPH, min window 1.08× → 1.38× (backward OOM three times; switched to lower-level capture, enabled only from step 1) | ✓ OPT_OVERLAP_PREPACK + OPT_EARLY_FORWARD, 1.38× → 1.44× | ✗ not in the log | ✗ (peak 131.1 GiB) | desync of timestep, attention length, token loop, merge index; denoise CUDA graph ~0.9 s slower per window, reverted |
| Kimi k3.1-pro | 1.41× | ✓ BAGEL_OPT_DECODE_GRAPH, 1.08× → 1.32× (gave up after two OOMs, got it running in round 12 by clearing cache before backward) | ✗ not in the log | ✗ not in the log | ✗ (peak 131.6 GiB) | timestep on GPU, packed scalar stacking, KV merge index cache (full check 1.31× → 1.38×); denoise graph, shared attention mask etc. measured no gain |
| DeepSeek V4.1 Flash-1017 | 1.19× | tried, reverted: bitwise identical, 1.42× on the timing side, but backward OOM in the deterministic record run; only the one-graph-per-full-batch version passed once (1.28×), OOM again on rerun, finally turned off | ✓ full check 1.14× → 1.17× | tried, reverted: RoPE fusion 1.175× (no change), RMSNorm tail no gain and made the record run OOM | ✗ (peak 130.7 GiB) | static KV length read once per forward, timestep t.repeat(n), packed scalars read once |