Slime GRPO
§ 02Models
| Model | Judged | Tier | Self-best | Rounds | Hours | API cost | |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 2.40× | N1 | 2.45× | 10 | 10.0 h | $58.3 | trace → |
| Claude Opus 5.5 | 2.35× | N1 | 2.48× | 7 | 10.0 h | $87.7 | trace → |
| GPT-6.2 Astra | 2.15× | N1 | 1.98× | 12 | 10.0 h | $59.1 | trace → |
| MiniMax M3.1 Flash | 1.91× | N1 | 1.90× | 17 | 10.0 h | $5.73 | trace → |
| DeepSeek V4.1 Flash-1017 | 1.88× | N1 | 1.94× | 10 | 10.0 h | $1.41 | trace → |
| Claude Haiku 5.5 | 1.85× | N1 | 1.90× | 23 | 10.1 h | $16.8 | trace → |
| GPT-6.2 Sol | 1.85× | N1 | 1.73× | 12 | 10.0 h | $13.9 | trace → |
| GLM-5.3 | 1.79× | N1 | 1.72× | 14 | 10.0 h | $61.8 | trace → |
| Kimi k3.1-pro | 1.35× | N1 | 1.25× | 9 | 10.0 h | $14.7 | trace → |
| GPT-6 Astrasuspected nerf | 1.15× | N1 | 1.13× | 20 | 10.0 h | $217 | trace → |
§ 03Key levers
| Model | Judged | Candidate, three windows (s) | Rollout side | Train side | Rollout host-side fixes | Skip zero-coefficient reference forward | Train-side main levers |
|---|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 2.40× | 76.4 / 77.5 / 76.0 | 2.54× | 1.89× | ✓ sampler desync, batched top-k + compact + shared-memory transfer, fast JSON, round-robin routing | ✓ | ✓ no recompute, NVLS off, 16 CPU threads, GDN split Ulysses-style (forward bitwise identical), comm groups kept, GC freeze, pinned master weights; train window 67.3 → 35.7 s (1.89×), second on train side It also had a "skip backward based on advantage values" item that cut the train window further to ~18 s (train side 3.33×); banned by the task statement, not counted in the score or in this column; see top of page |
| Claude Opus 5.5 | 2.35× | 79.2 / 78.0 / 77.7 | 2.42× | 2.15× | ✓ sampler desync, only compact base64 top-k returned (post-rollout tail 28 → 2 s), top-k copied back once per step | ✓ | ✓ no recompute, 16 CPU threads, free memory queried once before comm, pinned fp32 master weights, training process and rollout engine both resident (no more per-step sleep / wake), GDN backward split by token across 4 GPUs (forward unchanged); train window 66.2 → 30.9 s (2.15×), highest train side overall Round 7 also measured a fused sampling kernel + round-robin routing (rollout 2.51×) and weight sync directly from on-GPU params (train 2.17×); its own full loop ~74 s/step; quick check didn't finish before time ran out, not committed |
| MiniMax M3.1 Flash | 1.91× | 99.0 / 95.8 / 94.3 | 2.07× | 1.60× | ✓ sampler desync, flat base64 top-k, single D2H, request intake polled every 8 steps (1.95 → 2.07×) | ✓ | partial: NCCL groups kept (train 1.22×), no recompute (→ 1.49×), params resident on GPU, micro-batch 8192 tokens (→ 1.60×) |
| DeepSeek V4.1 Flash-1017 | 1.88× | 101.8 / 97.0 / 95.6 | 2.10× | 1.60× | ✓ redundant nested top-k removed + orjson (rollout 111 → 82 s), host-side top-p check (→ 52 s) | ✓ | partial: no recompute, training model resident, logprob chunk 1024 → 4096 |
| Claude Haiku 5.5 | 1.85× | 98.3 / 100.2 / 99.6 | 2.15× | 1.50× | ✓ sampler desync, engine sends binary top-k only, round-robin | ✓ | partial: no recompute, 24 training CPU threads, GC freeze + param buffers resident (1.40 → 1.50×) |
| GLM-5.3 | 1.79× | 105.3 / 100.7 / 102.0 | 2.19× | ~1.29× | ✓ host-side top-p flag, flat top-k | ✗ reference forward still there (~10 s/step); tried merging its micro-batches into larger ones, net slower | partial: training CPU threads, NCCL kept across sleep/wake; recompute only changed to stride-1, not turned off |
| Kimi k3.1-pro | 1.35× | 138.2 / 133.1 / 137.1 | 1.32× | 1.35× | partial: only sampler sync removed and top-k row copies merged; top-k JSON serialization untouched | ✗ | partial: training resident, no recompute; pairing micro-batches measured 1.44×, blocked by direction cosine 0.931 < 0.955 |
| GPT-6 Astrasuspected nerf | 1.15× | 162.1 / 156.1 / 161.5 | ≈0.98× | 1.48× | ✗ rollout untouched (measured 0.98–0.99× all three times) | ✓ | partial: training process group kept, selective recompute committed only in round 19 (1.20 → 1.48×) |