MiniMax M3.1 Flash 1.91×

Harness: Codex · switches: SLIME_OPT_KEEP_NCCL_GROUPS=1 SLIME_OPT_NO_RECOMPUTE=1 SLIME_OPT_SGLANG_RECV_INTERVAL=8 SLIME_OPT_KEEP_PARAMS_ON_GPU=1 SLIME_OPT_SKIP_WAKE_GC=1 SLIME_OPT_MAX_TOKENS_PER_GPU=8192

Judged
1.91×
Tier
N1
Self-best
1.90×
Rounds
17
Hours
10.0
API cost
$5.73
Input tokens · 97.4% cached
259 M
Output tokens
1.5 M
Requests
1,753
Candidate, three windows (s)
99.0 / 95.8 / 94.3
Rollout side
2.07×
Train side
1.60×
Rollout host-side fixes
✓ sampler desync, flat base64 top-k, single D2H, request intake polled every 8 steps (1.95 → 2.07×)
Skip zero-coefficient reference forward
✓
Train-side main levers
partial: NCCL groups kept (train 1.22×), no recompute (→ 1.49×), params resident on GPU, micro-batch 8192 tokens (→ 1.60×)
RoundChangeMeasuredStepSide, cumulativeStatus
Rollout side
1sampler top-p check moved to host, per-step GPU sync removedrollout-side speedup: — → 1.3×1.301.30×kept
3return only flat base64 top-k logprob arrays, no nested JSONrollout-side speedup: 1.3 → 1.88×1.451.88×kept
7decode top-k as one bulk D2H + one tolistrollout-side speedup: 1.88 → 1.95×1.041.95×kept
12sglang request intake polled every 8 steps (recv_interval=8)rollout-side speedup: 1.95 → 2.07×1.062.07×kept
12recv_interval=32 (requests fetched too late, worse than 8)rollout-side speedup: 2.07 → 1.9——reverted
16CUDA graph fill via foreach_copy, bypassing Triton small copy (no gain, reverted)rollout-side speedup: 2.07 → 1.89——reverted
Train side
2skip useless reference forward when kl_coef=0train-side speedup (window 2 is a host-anomaly outlier): — → 0.892—0.89×kept
5keep NCCL process groups across offload/onload instead of destroying and rebuildingtrain-side speedup: — → 1.22—1.22×kept
6activation recompute off (SLIME_OPT_NO_RECOMPUTE)train-side speedup: 1.22 → 1.49×1.221.49×kept
8skip the separate actor log-prob forward (reference wasn't running anyway, reverted)train window s (no change): — → ———reverted
10per-param-group batched CPU Adam + pinned fp32 master (no gain, turned off)train-side speedup: 1.51 → 1.49——reverted
14params stay on GPU during rollout, only grad_buffer releasedtrain-side speedup: 1.51 → 1.54×1.021.54×kept
14skip cyclic GC on the wake path (SKIP_WAKE_GC)sleep/wake timing (no separate side speedup): — → ———kept
16per-GPU token budget raised to 8192, micro-batches halved (64→32)train-side speedup: 1.54 → 1.6×1.041.60×kept
Raw work log18 sections
0.4 hRound 1 — 2026-11-13 04:35 → 05:35 UTC
Profile (reference, unchanged): bench.py profile --side rollout → /tmp/r1/profile/kernel_report.md
- 200 decode steps, window 3078 ms; GPU busy only 32% (990 ms of kernel time, ~4.95 ms/step) but the real step is
  ~10.3 ms (sglang "gen throughput" 3400 tok/s over 35 running reqs).
- CPU side of the trace: cudaStreamSynchronize 811 ms / 260 calls (3.1 ms each) and aten::item 818 ms / 643 calls.
  200 of the syncs (811 ms) come from sglang/srt/layers/sampler.py Sampler.forward → one GPU sync per decode step.
- Cause: sampler.py:321 bool(sampling_info.return_top_p_log_probs.any().item()) — the mask is a device tensor, so
  the check stalls the step. The value is known on the host (per-request custom_params).
- Other per-step CPU cost: copy_result_to_cpu 2.2 ms/step (10 async D2H enqueues), step[DECODE] 1.98 ms,
  get_next_batch_to_run 0.35 ms, recv_requests 0.3 ms.
- Top-128 topk chain (mbtopk) = 107 µs/step; LM-head GEMM 248 µs/step; TP allreduce ~0.42 ms/step (64/step).

Change 1 (rollout side, sglang): host-side need_return_top_p_log_probs flag replaces the device .any().item().
- sglang/python/sglang/srt/sampling/sampling_batch_info.py: new field + any(...) at construction, |= on merge,
  exact re-derivation on filter.
- sglang/python/sglang/srt/layers/sampler.py:321: if sampling_info.need_return_top_p_log_probs:
- Numerics: N0 (host bookkeeping only; GPU op order unchanged). Commit in sglang repo.

Change 1 result (rollout side): bench.py time --side rollout → /tmp/r1b/time/time.json
- candidate windows [86.276, 86.737, 85.117] s vs frozen reference [111.899, 116.057, 111.527] s
- rollout side speedup 1.297 (min window ratio), projected step speedup 1.1662 (train side unchanged)
- sglang "gen throughput" rose from ~3400 tok/s (10.3 ms/decode step) to ~5800 tok/s (6.2 ms/decode step):
  the CPU no longer waits for the GPU inside the sampler, so the scheduler's CPU work overlaps the
  GPU work again (the overlap scheduler can only overlap if nothing blocks the CPU).
- Commit: sglang 64305b5 "sglang: keep the top-p logprob eligibility on the host" (tier N0, rollout 1.297).
0.7 hRound 2 — 2026-11-13 06:10 → (running) UTC
Profile (reference): the reference's own perf metrics (runs/bootstrap/full/full_ref_a/train.log) give the
train step breakdown: train_time 59.1 s = ref_log_probs_time 11.58 s + actor_train_time 47.1 s;
the rest of the RL step is rollout 109.5 s, sleep 4.6 s, wake_up 1.3 s, update_weights 1.9 s.

Change 2 (train side, slime): the recipe runs --use-kl-loss --kl-loss-coef 0.00 with the reward KL
coefficient at 0, so the reference log-probs are consumed by nothing:
- compute_advantages_and_returns (loss.py:764) already uses kl = zeros when args.kl_coef == 0;
- the loss-level KL is multiplied by kl_loss_coef = 0.0.
The reference forward (+ the weight round trip) is dead work worth 11.6 s of the 59 s train step.
- actor.py: _ref_log_probs_needed() (kl_coef != 0 or use_kl_loss and kl_loss_coef != 0) gates the
  reference forward; loss.py: the KL term is only computed when its coefficient is non-zero.
- The loss value and the gradient are unchanged (the KL term contributed exactly 0.0).
- Commit: slime 8a161b13 (tier N0).

Change 2 result (train side): bench.py time --side train → /tmp/r2/time/time.json
- candidate windows [57.15, 74.24, 55.30] s vs frozen reference [65.599, 66.182, 66.715] s
- window ratios [1.148, 0.892, 1.206] → train side speedup 0.892 because window 2 (step 5) was an outlier
  (actor_train 65.4 s instead of 46 s; the neighbouring steps took 47.9/46.3 s and the reference's step 5 took
  46.9/45.6 s, so the outlier is not explained by the change - it is a step where the host was slow).
- The log confirms the ref forward is gone: the perf dict no longer has perf/ref_log_probs_time and
  perf/actor_train_time dropped from ~47.1 s (reference, with the 11.6 s ref forward) to 46.3 s.
- The reference's own warm-up steps are 222/119/67/119 s (steps 0-3: the ref forward is 31 s there because the
  dump's shapes are compiled twice); our steps 0-3 are 209/112/57/109 s - the same pattern without the ref
  forward.
- Commit: slime 8a161b13 (tier N0).
1.1 hRound 3 — 2026-11-13 05:30 → (running) UTC
Profile (reference, rollout client side): the reference rollout window (109.5 s) is much longer than the
engine's own decode phase (~53 s) and its own per-request e2e latency (max 84.9 s): ~25 s of every rollout is
spent *after* the engines finished, in the client. Cause: sglang returns the top-128 logprobs of every token
twice - as the flat base64 arrays (output_topk_token_ids/output_topk_log_probs, which is what the trainer
uses) and as the nested per-position lists (output_top_logprobs, one JSON object per (token, top-k) pair).
For 8192 tokens x 128 that is ~1M JSON objects (~25 MB) per response, which the engine serializes, the client
parses and Ray pickles - all of it discarded.

Change 3 (rollout side, sglang + slime): a request can now ask for the flat top-k arrays only.
- sglang io_struct.py: return_flat_output_top_logprobs on GenerateReqInput/TokenizedGenerateReqInput.
- sglang tokenizer_manager.py: the nested output_top_logprobs list is not built when the request asks for the
  flat form and the response is not streamed (the nested lists are only used by the streaming chunks and by the
  OpenAI-compatible endpoints).
- slime rollout/sglang_rollout.py: the score-centering request sets the flag.
- The values that reach the trainer are unchanged (the trainer already reads the flat arrays).

Change 3 result (rollout side): bench.py time --side rollout -> /tmp/r3/time/time.json
- candidate windows [53.888, 57.467, 59.462] s vs frozen reference [111.899, 116.057, 111.527] s
- rollout side speedup 1.8756, projected step speedup 1.4094 (train side unchanged at ~1.15-1.20)
- Commit: sglang d9db56a + slime b953a6ea (tier N0: the flat top-k arrays the trainer reads are unchanged).
1.6 hRound 4 - 2026-11-13 05:40 -> 06:12 UTC (verification of rounds 1-3)
Check (the verdict): bench.py check --quick -> /tmp/r4/quick/check.json: status PASS, tier N1
- rollout side: PASS. Token forcing: candidate mean|d| 0.0077768 vs the reference's 0.0076856 (ratio 1.0119 < 2),
  the |d|>tau event rate 2.9e-6 vs the reference's 8.2e-6 (binomial p=0.997), the sampler paired t=-1.41 (|t|<4),
  top-k overlap 0.98800 vs 0.98806 (deficit ratio 1.004), the top-k mean|diff| ratio 0.998, bias t=0.51.
  Train-inference mismatch 0.008036 vs the reference's 0.008119 (ratio 0.99).
- train side: PASS at N1 (the reference itself is not bitwise repeatable: 6400 tensors differ).
  Statistical gate: update_norm d=-6.9e-5 (tol 6.6e-4), exp_avg_sq_norm d=2.0e-7 (tol 1.9e-6),
  loss_mean d=2.6e-6 (tol 2.5e-5), grad_norm_mean d=6.0e-4 (tol 3.4e-2), update direction cos 0.9932 >= 0.9554.
- The judge's rollout windows (from the check run): [53.34, 51.85, 56.50] s vs the reference's
  [111.899, 116.057, 111.527] -> the rollout side is ~2.0-2.2x here.
- Note: the judge materializes the tree once per check, so edits made while a check runs are not measured.

Changes since round 3 (diagnostics only, no behaviour change): timing instrumentation for the actor's
sleep/wake (clear_memory / destroy_pg / pause / resume / reload_pg) and for the HybridDeviceOptimizer step
(D2H enqueue / CPU Adam / state sync), to locate the next hotspot in the 47 s actor_train.
2.6 hRound 5 - 2026-11-13 06:12 -> 07:07 UTC (train side: the offload/optimizer overhead)
Profile (train side, bench.py profile --side train -> /tmp/r5/profile/kernel_report.md):
- The train step is 47-50 s. The torch-profiler trace of one step shows the GPU busy 63.5% of the window
  (47.3 s of kernels) and, inside that, the TP/CP collectives are 30% of the kernel time:
  ncclDevKernel_AllGather_RING_LL 9.66 s (19.9%, 18113 calls, median 156 us, SM cov 0.18 - latency bound,
  not bandwidth bound: NVLink is 18x per GPU pair), ReduceScatter_Sum_bf16 3.23 s (6.7%),
  AllReduce_Sum_f32 1.01 s, SendRecv 0.93 s. The GEMMs (nvjet) are ~17 s and the rest ~7 s.
  The collectives run on the main stream, so they are exposed, but they are latency/synchronization bound;
  the recipe fixes TP2 x CP2 and the SP layout, so this is hard to attack quickly.
- The host-side costs, measured with the new instrumentation in the same run:
  sleep = clear_memory 0.70 + destroy_pg 1.79 + pause 1.68 + stack 0.00 = 4.2 s,
  wake_up = resume 0.47 + clear_memory 0.50 + reload_pg 0.13 = 1.1 s,
  optimizer (HybridDeviceOptimizer.step) = d2h_enqueue 0.003 + cpu_adam 4.75 s.
  The CPU Adam is at the machine's memory-bandwidth limit: 4 ranks x 4.5 Gparams x 28 B = 504 GB of
  traffic per step at ~106 GB/s. A 4-process benchmark with 96/48/24/16 threads per rank gives the same
  aggregate (1.8-2.0 Gparam/s), so thread-count tuning is useless -> that direction was dropped.

Change 4 (train side, slime): the actor destroyed and recreated its ~65 registered process groups twice
per train step (in sleep() and again in update_weights()); the destroy alone costs ~1.8 s per
occurrence, i.e. ~3.6 s of the 55 s train window. The communicator memory is not needed by the rollout
engines (their KV cache is allocated at engine start-up), so the groups now stay alive; a gloo barrier keeps
the rank synchronization that the destroy provided.
- Behind the switch SLIME_OPT_KEEP_NCCL_GROUPS (off by default, on in /submission/switches.json).
- Numerics: N0 (no arithmetic change; the collectives and their order are unchanged).

Change 4 result: bench.py time --side train -> /tmp/r6/time/time.json
- candidate windows [52.181, 52.336, 54.592] s vs the frozen reference [65.599, 66.182, 66.715] s
- window ratios [1.2571, 1.2646, 1.2221] -> train side speedup 1.2221 (was 1.148-1.206 in round 2,
  which was noisy because of one host outlier step).
- The log confirms the change: sleep breakdown: destroy_pg 0.001 (was 1.794) and reload_pg 0.000.
- Commit: slime fd5d0625.
3.1 hRound 6 - 2026-11-13 07:15 -> 07:40 UTC (train side: the activation recompute)
Profile (train side, the round-5 profile /tmp/r5/profile/kernel_report.md): the step is 63.5% GPU busy,
47.3 s of kernels; the TP/CP collectives are 30% of the kernel time, the GEMMs ~17 s. The kernel report cannot
separate the forward from its recompute, so the recompute's share was derived from the work structure: with
--recompute-granularity full --recompute-method uniform --recompute-num-layers 1 every transformer layer is
checkpointed, so the layer's forward runs twice (once grad-enabled, once no_grad in the backward) and its SP
all-gathers run twice as well: 2/3 of the 18113 all-gathers belong to the forward, i.e. ~11 s of the 47 s of
kernels is the recompute pass.

Change 6 (train side, slime): SLIME_OPT_NO_RECOMPUTE=1 (off by default, on in switches.json) drops the
activation recompute: slime/backends/megatron_utils/model_provider.py::_apply_recompute_override sets
recompute_granularity/method/num_layers = None on the TransformerConfig.
- The recompute is a memory-for-compute trade, not a change of the mathematics: the recomputed activations are
  the ones the forward pass produced.
- Memory: with the recompute the step's peak allocation is ~32 GB of 139.8 GB (the log's allocated_GB 30.87 /
  reserved_GB 32.15 at the end of the step, free_GB 102 while the engines are offloaded), so keeping the
  activations (~20 GB more) fits. The run confirms it: no OOM, the peak allocation rises as expected.
- Numerics: N1. The forward is bitwise identical (step 0's loss -0.0007013520690397197, its entropy and its
  train/inference mismatch 0.008164666641911946 are exactly the reference's), but the backward now uses the
  grad-enabled forward's saved activations instead of the no_grad recomputed ones: the step-0 gradient norm moves
  from 0.04709615930914879 to 0.04716501384973526 (1.5e-3 relative). Against the judge's statistical gate
  (loss_mean tol 2.5e-5, grad_norm_mean 0.034, exp_avg_sq_norm 9.3e-7, update_norm 6.6e-4, cos_min 0.9554) the
  shift of the 3-step loss mean is 2.0e-7, of the gradient norm 3.4e-5 and of the second moment 6e-10; the
  update direction cosine stays > 0.999. The train/inference mismatch at step 0 is bitwise the reference's.

Change 6 result: bench.py time --side train -> /tmp/r8/time/time.json
- candidate windows [43.810, 44.363, 43.174] s vs the frozen reference [65.599, 66.182, 66.715] s
- window ratios [1.4974, 1.4918, 1.5453] -> train side speedup 1.4918 (was 1.2221 in round 5)
- the log confirms it: actor_train_time 45.5 s -> 37.2 s, actor_train_tflops 144 -> 175, step_time 52.5 -> 43.9.
- Commit: slime 74de0bf0 (tier N1).
3.7 hRound 7 - 2026-11-13 07:40 -> 08:15 UTC (rollout side: the per-step host transfers)
Check of rounds 1-6 (bench.py check --quick -> /tmp/r9/quick/check.json): PASS, tier N1 (both sides
judged).  The rollout's token forcing is statistically equivalent (mean_abs_ratio 0.9994, the event rate is
below the reference's, the sampler's paired t is -0.011), the top-k overlap is 0.98803 vs the reference's
0.98806, and the training side passes the statistical gate.

Profile (the round-7 rollout trace /tmp/r7/profile): the decode step is GPU-bound (the kernels of one
step take 4.8 ms of GPU time) but the step itself takes ~6.6 ms.  The CPU trace of the scheduler shows why:
66 async_d2h calls per decode step (1.94 ms of inclusive CPU time) - 64 of them the top-128 rows of the
log-probabilities (one pinned allocation + one D2H copy per row) - plus 68 tolist and 165 slice/select
calls per step.

Change 7 (rollout side, sglang): every request of the batch asks for the same number of top-k entries
(128), so get_top_logprobs_raw keeps the (bs, k) block instead of splitting it into bs rows, the copy to
the host is one D2H for the whole block, and the rows are converted with one tolist per step.
- Files: sglang/python/sglang/srt/layers/logprob_processor.py, managers/utils.py,
  managers/scheduler_components/batch_result_processor.py.
- Numerics: N0 (the same numbers are returned, only the transfer and the conversion are batched).

Change 7 result: bench.py time --side rollout -> /tmp/r10/time/time.json
- candidate windows [51.116, 51.506, 57.245] s vs the frozen reference [111.899, 116.057, 111.527] s
- window ratios [2.1891, 2.2533, 1.9482] -> rollout side speedup 1.9482 (the previous state measured
  [53.523, 56.806, 54.154] s in the round-9 check, i.e. the same within the run-to-run spread of the
  rollout windows, which is +-6 s).
- The remaining gap to the GPU: the request e2e latency gives 5.57 ms per decode step against 4.8 ms of GPU
  kernel time, so ~0.8 ms/step (6 s per rollout) of scheduler overhead is left.
- Commit: sglang 6cce324 (tier N0).
4.2 hRound 8 - 2026-11-13 08:10 -> 09:05 UTC (train side: the redundant actor log-prob forward) - NO-OP, REVERTED
Check of rounds 1-7 (bench.py check --quick -> /tmp/r11/quick/check.json): PASS, tier N1 (both sides
judged).  The rollout's token forcing is statistically equivalent (mean_abs 0.007713 vs the reference's
0.007686, i.e. 1.003x), the top-k overlap is within the reference's own, and the training side passes the
statistical gate.  This is the fallback state (commit slime 74de0bf0 + sglang 6cce324).

Profile finding (from the round-5 train trace and the source): the actor runs *two* full forward passes
over the batch per train step: compute_log_prob() (a no-grad forward that fills
rollout_data["log_probs"]) and the training forward+backward.  The first one recomputes the log-probs of
the sampled tokens with the same weights the training forward uses (the reference forward was already
removed in round 2, and the actor's weights do not change between the two passes), so it is a third of the
compute of the step (fwd+bwd+logprob-fwd = 4 forward-equivalents; the log-prob forward is 1 of them) and it
also repeats all the SP/CP collectives of the forward.

What the separate forward's values actually feed (checked in slime/backends/megatron_utils/loss.py):
- the TIS importance ratio: train_log_probs_for_tis is overwritten with the *training* forward's log-probs
  when the PG objective is reinforce (line 1183) - our recipe;
- the mismatch metric train_rollout_logprob_abs_diff: uses the training forward's log-probs whenever
  score centering is on (line 1352) - our recipe;
- the loss itself: pg_loss = -advantages * log_probs for reinforce - ppo_kl (the only consumer of
  old_log_probs) does not enter it;
- the advantages: the KL term is exactly zero when kl_coef == 0 (our recipe);
- the KL loss term: exactly zero when kl_loss_coef == 0 (our recipe);
- only diagnostics: ppo_kl, ois, correct_entropy.  In the reference these are already exactly 0 / 1
  (the two forwards compute the same values bit for bit), so they do not change either.

Change 8 (train side, slime): skip the separate actor log-prob forward when it cannot reach the loss
(SLIME_OPT_SKIP_ACTOR_LOGPROB=1, off by default).  The guard covers every consumer of the old-policy
log-probs (kl_coef, kl_loss_coef, a non-reinforce PG objective, OPSM/GSPO, the critic/OPD/routing-replay
paths and use_rollout_logprobs); with any of them active the forward still runs.  The correct_entropy
diagnostic tolerates the missing log-probs.
- Files: slime/slime/backends/megatron_utils/actor.py, slime/slime/observability/train_metric_utils.py.

Outcome: the change was a NO-OP and was REVERTED (no commit).  While reading the code it turned out that
the separate forward never runs in the reference either: can_reuse_log_probs_in_loss is already True because
it tests len(num_microbatches) == 1, and num_microbatches counts the *VPP* stages, not the micro-batches
(our recipe has no pipeline parallelism).  The reference logs have no Timer log_probs end line, ppo_kl is
already exactly 0.0 and ois exactly 1.0, which confirms that the log-prob forward is skipped in the
reference as well.  The measured gain was therefore 0.0 s (within the +-0.5 s noise of the train windows), and
the guard was removed with git checkout of the two files so the tree stays minimal.  Lesson recorded here:
read the Megatron-side guard conditions before assuming a redundant forward.
4.5 hRound 9 - 2026-11-13 09:00 -> (running) UTC (full check of rounds 1-7)
Started the full check of the accumulated state (rounds 1-7, commits slime 74de0bf0 + sglang 6cce324, switches
SLIME_OPT_KEEP_NCCL_GROUPS=1, SLIME_OPT_NO_RECOMPUTE=1):

```
cd /opt/researchboost/ws
/opt/sglang/bin/python3 bench/bench.py check --tree /submission/slime_grpo --config /submission/switches.json --out /tmp/r13/check
```

(log: /tmp/r13/check.log, detached, PID 875057).  The check runs the rollout side and the training side with
cold compile caches and applies every correctness gate; it is the verdict the final evaluation re-runs.
4.5 hRound 9 - 2026-11-13 09:00 -> 09:25 UTC (full check of rounds 1-7) - the judge crashed, not the tree
Started the full check of the accumulated state (rounds 1-7, slime 74de0bf0 + sglang 6cce324):

```
cd /opt/researchboost/ws
/opt/sglang/bin/python3 bench/bench.py check --tree /submission/slime_grpo --config /submission/switches.json --out /tmp/r13/check
```

The rollout candidate (381 s wall) and the train candidate (743 s wall) both ran fine, and the check then died
in the judge's own code:

```
File "/opt/researchboost/ws/bench/bench.py", line 996, in judge_rollout
    st, why = score(dumps, ids, out / 'scores.npz', out / 'score.log')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/r13/check/rollout/score.log'
```

judge_rollout() writes score.log into <out>/rollout, but in the non-quick path that directory is never
created (in the --quick path judge_rollout calls time_side() itself, which creates it as a side effect).
The final evaluation uses --final, where side_runs is empty and judge_rollout therefore also calls
time_side() itself - so the final evaluation is not affected by this bug; only the agent's full check
is.  Workaround for the next full check: create <out>/rollout before starting the judge.
5.6 hRound 10 - 2026-11-13 09:25 -> 10:10 UTC (train side: the CPU Adam and the fp32 master copy)
Profile finding (the round-5 train trace plus the HDO log): the train step is 36.8 s of which the
optimizer is 4.8 s: the hybrid device optimizer builds one CPU optimizer per parameter (one fused Adam
call each, ~290 of them, each with its own fixed overhead) and the fp32 master weights of the offloaded
parameters are pageable, so the copy back to the device after every step runs at the pageable PCIe rate
(measured: 10.1 GB/s pageable vs 54.1 GB/s pinned).

Change 10 (train side, Megatron): SLIME_OPT_BATCH_CPU_ADAM=1 builds one CPU optimizer per parameter
*group* instead of one per parameter; SLIME_OPT_PIN_FP32_MASTER=1 pins the fp32 master of the offloaded
parameters.  Both off by default.  Files: megatron/core/optimizer/cpu_offloading/hybrid_optimizer.py.

Measured (bench.py time --side train):
| variant | windows (s) | train speedup | cpu_adam |
|---|---|---|---|
| baseline (r12) | 43.498 / 43.500 / 42.792 | 1.5081 | 4.75 s |
| batch+pin (r14) | 44.016 / 44.396 / 42.798 | 1.4903 | 3.40 s |
| batch only (r15) | 44.593 / 44.171 / 44.077 | 1.4711 | 4.60 s |

Conclusion: no gain, both switches left off.  The CPU Adam itself gets cheaper (4.75 s -> 3.40 s), but
the step does not: the copy of the master weights back to the device does not get faster either, because with
four ranks copying at once the *host memory* bandwidth (not PCIe) is the limit - the cost just moves from the
CPU (a blocking pageable copy inside the optimizer step) to the GPU (waiting for the pinned copy at the next
forward).  The per-parameter Adam overhead in the real parameter mix is only ~0.5 ms per call, not the ~10 ms
a micro-benchmark with small tensors suggests.  The actor_train timer is 36.7/36.8/36.1/38.0 s (baseline),
36.8/37.6/35.6/37.3 s (batch+pin) and 36.7/37.2/37.1/36.7 s (batch only) - all within the +-0.7 s noise.
Committed as Megatron 6967b3b with the switches off.
6.0 hRound 11 (2026-11-13 10:09-10:33 UTC) - full-loop validation of the current tree
Why: no full check had ever completed in this task (the judge's own non-quick check path crashes in
judge_rollout(): it writes score.log into <out>/rollout, a directory that is only created in the
--quick/--final paths).  The end-to-end RL loop (rollout -> train -> weight sync -> rollout) had therefore
never been exercised on the candidate tree, and the final evaluation times exactly that loop.

What: ran the judge's own time_side('full') through a small driver that imports bench.bench
(/tmp/b/fullrun.py), i.e. the same code path the final evaluation uses: a prime run (train x5 on the fixture
dumps, to fill the compile caches) and then the timed full loop (5 RL steps).  Output /tmp/r16/full/,
log /tmp/r16/full.log.  *This is a functional check, not a reported speedup - only the judge's own commands
count.*

Result: STATUS ok, windows [104.892, 99.612, 99.218] s per RL step against the frozen reference step of
181.863 s; train-inference mismatch per step 0.0073-0.0082 (the reference's own magnitude is ~0.0077).
The loop works end to end on the candidate tree (weight update, engine sleep/wake, rollout -> train hand-off).
6.0 hRound 12 (2026-11-13 10:34 UTC) - rollout: poll the tokenizer less often
Profile (py-spy dump of the sglang scheduler's MainThread during the round-11 full loop, 60 samples):
30 % of the samples are in ingest_requests -> recv_requests -> broadcast_pyobj (the gloo broadcast of the
request list to the other TP rank, ~20 % of the samples are inside dist.broadcast itself), 20 % in the
forward launch (mostly torch.cuda.Stream.synchronize, i.e. waiting for the GPU), 17 % in the result
processing/detokenizer, 10 % in the batch preparation.  The decode step costs ~4.8 ms of GPU work against a
5.8-6.2 ms wall, so ~1 ms per step is CPU overhead that keeps the GPU idle.

Change: sglang's own request-receiver skipper polls the tokenizer every N steps instead of every step
(scheduler_recv_interval).  The skip decision only depends on the forward mode of the last batch, which is
identical on every TP rank, so the broadcast collective stays in sync.  slime passes it to the engine behind
the switch SLIME_OPT_SGLANG_RECV_INTERVAL (off by default = the reference behaviour), in
slime/backends/sglang_utils/sglang_engine.py::_compute_server_args.

Measured (bench.py time --side rollout, one run per variant, against the frozen reference timing):
| variant | windows (s) | rollout speedup |
|---|---|---|
| interval 8 (/tmp/r17/time/time.json) | 53.571 / 53.213 / 54.002 | 2.0652 (was 1.9482) |
| interval 32 (/tmp/r17/time32/time.json) | 52.601 / 53.419 / 58.626 | 1.9023 |

Interval 8 is the sweet spot: 32 delays the pickup of the requests too much (its last window is 5 s slower).
The switch stays off by default; /submission/switches.json now carries
SLIME_OPT_SGLANG_RECV_INTERVAL=8 together with the two switches of the earlier rounds.

Scheduler stage metrics (sglang's own sglang:scheduler_stage_seconds_total, sampled from the engine's
/metrics endpoint during a live rollout): per decode step the scheduler loop spends
process_batch_result 2.3-5.0 ms (mostly the wait for the GPU's result copy), run_batch 1.7-2.4 ms,
get_next_batch_to_run 0.33-0.41 ms and - with the interval on - recv_requests only 0.09-0.20 ms
(it was 18 % of the loop before the change).

Verdict: /tmp/r18/quick/check.json = PASS, tier N1 (both sides judged).
- rollout: token forcing mean |d| 0.007750 vs the reference's 0.007686 (ratio 1.0083), the event rate
  1.4e-6 vs the reference's 8.2e-6 (p = 0.9998), the sampler t = -0.78, the top-k overlap deficit ratio
  1.0067 and the mean-abs ratio 1.0019, the train-inference mismatch ratio 1.0003 -> PASS.
- train: the record is not bitwise identical (6418 tensors differ), the statistical gate passes
  (update norm d = -6.3e-5 vs the tolerance 6.6e-4, second moment d = 1.7e-7 vs 1.9e-6, loss mean
  d = 7.1e-6 vs 2.5e-5, grad norm d = -4.1e-4 vs 3.4e-2, update-direction cosine 0.9928 vs the frozen
  minimum 0.9554) -> N1.
Committed as slime 7122bc5f (N1).

Note: the rollout windows of that check run were 53.0 / 52.9 / 68.8 s - the third one is an outlier
caused by my own profiling (py-spy dumps and /metrics polling) running on the same machine during the
run; the clean measurement in /tmp/r17/time gave 53.6 / 53.2 / 54.0 s.
7.3 hRound 13 (2026-11-13 11:29 -> 12:00 UTC) - profiling the rollout's remaining bubble and the train step
Profile (round 12's tree, rollout side): a live rollout of the current tree was sampled three ways:
- nvidia-smi dmon during the decode: the four GPUs run at 95-100 % SM utilization, so the decode
  step (6.2 ms wall) is GPU-bound; there is no CPU bubble left to remove on the rollout side.
- py-spy on the sglang scheduler (1200 stack samples): 43 % of the samples are in
  result.copy_done.synchronize() (waiting for the GPU), the rest is the launch path
  (run_batch 27 %, get_next_batch_to_run 5 %, ingest_requests 5 %).  The remaining CPU work per decode
  step is ~3.3 ms against ~6.2 ms of wall time, and the GPU is busy the whole time.
- the engine's own stage metrics: run_batch 2.2 ms/step, process_batch_result 3.4 ms/step (mostly the
  GPU wait), get_next_batch_to_run 0.7 ms/step, recv_requests 0.7 ms/step.
=> the rollout's GPU work (the weight reads of the decode GEMMs, ~43 % of the kernel time at 3.0-3.4
TB/s) is at the roofline; the only remaining rollout lever would be fusing the ~490 small kernels per
decode step, which is a large, risky project.

Where the RL step's time goes now (from the full-loop run of round 11, per step, s):
rollout 52-56 (decode 8192 steps x 6.2 ms) | actor_train 36.8-39.1 (of which the CPU-bound Adam 4.7 and
the GPU ~32) | sleep 2.3 + wake_up 1.1 | update_weights 0.9 | reward+conversion ~1-2.
The train GPU time is dominated by the sequence-parallel collectives (the round-5 kernel report:
AllGather_RING_LL 19.9 %, ReduceScatter_bf16 6.7 %, AllReduce_f32 2.1 %, SendRecv 1.9 % of the kernel
time; the all-gather's median is 156 us but its average 533 us - it is latency/skew bound, not
bandwidth bound) and by 64 micro-batches per step (one per sample: a sample of ~8300 tokens does not fit
into the 4096-token-per-GPU bin, so every micro-batch holds exactly one sample).

Change (train side, slime): SLIME_OPT_MICRO_BATCH_SCALE=N (off by default) scales the dynamic
micro-batch size, so the same tokens are processed in fewer, larger micro-batches: fewer
sequence-parallel collectives and fewer per-micro-batch launches per step.  The step still processes the
same tokens in the same order and the loss is the same sum over the same tokens (only the reduction
order inside the step changes).  File: slime/utils/arguments.py.

Measured: see below (the timing run of this round).
8.1 hRound 14 (2026-11-13 12:10 -> 13:00 UTC) - train side: the model offload overhead + a fresh rollout profile
Profile (the fresh rollout profile of the current tree, /tmp/r22/profile/kernel_report.md): the decode step of
the engine costs ~6.2 ms of wall time with ~4.6 ms of GPU kernels (the union of the kernels over 200 decode steps
is 1011 ms; the model forward is at the memory roofline - 18 GB of weights per GPU per step at ~4.2 TB/s - and the
sampler/top-k/logits post-processing adds ~0.4 ms).  The remaining ~1.6 ms per step is the scheduler's CPU work
that is not overlapped with the GPU; sglang already overlaps the result processing of the previous batch with the
current forward, so the bubble is the per-step preparation (batch metadata, sampling metadata) plus the launch
latency of the next batch.  That is the biggest remaining rollout lever (~13 s per rollout) but it needs a
scheduler-level change (double buffering), not a kernel change.

Train side: the model offload (the actor's sleep/wake).  The log of the full-loop run shows per step
sleep = clear_memory 0.6-1.15 + pause 1.37-1.55 and wake_up = resume 0.37-0.53 + clear_memory 0.50-0.56, i.e.
~3.4 s per step.  The pause copies every parameter (9 GB per rank) to the host and frees it on the GPU; the
rollout engines do not need that memory (their KV cache is allocated when the engine starts and released while the
actor trains), so only the gradient buffer (18 GB per rank, dead after the optimizer step) has to go.

Change 14a (train side, slime): SLIME_OPT_KEEP_PARAMS_ON_GPU=1 releases only the tagged grad_buffer region
of the memory saver (torch_memory_saver.pause(tag="grad_buffer")), so the parameters stay on the GPU during the
rollout.  The memory saver's semantics were verified with a standalone probe (a tagged pause frees only the tagged
allocation, the default region stays valid, and the resume re-allocates the region at the *same* virtual address, so
Megatron's param.main_grad views stay valid).
- The measured sleep/wake: pause 1.45 -> 0.017 s, resume 0.40 -> 0.024 s, i.e. ~1.5 s less per train step.
- The GPU holds 9 GB more during the rollout (the engine's KV pool still fits: 97.9 GB of 139.8 GB).

Change 14b (train side, slime): SLIME_OPT_SKIP_WAKE_GC=1 skips the cyclic garbage collection of the wake-up
path (the sleep already collected it).  The new instrumentation (SLIME_OPT_MEMORY_TIMING=1) shows that the
collection is the whole cost of clear_memory: sync 0.000 gc 0.41-0.63 empty_cache 0.00-0.04.

Measured (bench.py time --side train, /tmp/r23/time/time.json, change 14a only):
| variant | windows (s) | train speedup |
|---|---|---|
| baseline (r12) | 43.498 / 43.500 / 42.792 | 1.5081 |
| 14a (params on the GPU) | 42.189 / 42.902 / 41.719 | 1.5426 |

Commits: slime 9b474233 (14a), slime 03cdfd67 (14b + the instrumentation).  Both switches are off by default.
8.7 hRound 15 (2026-11-13 12:55 -> 14:00 UTC) - profiling the rollout's CPU path and the step's dead time
Rollout profile (live, current tree, bench.py time --side rollout + py-spy on the sglang scheduler):
- bench.py time --side rollout -> /tmp/r25/time/time.json: windows [54.139, 54.021, 53.548] s vs the frozen
  reference [111.899, 116.057, 111.527] => rollout side 2.0669x (unchanged by the round-14 changes).
- py-spy (11952 samples at 200 Hz over 60 s of the decode loop, the scheduler's main thread):
  27.0 % of the samples are in process_batch_result_decode -> synchronize (the CPU waiting for the previous
  batch's D2H copy), and the rest is spread thin: _stream_output_generation 3.3 %, send_pyobj 2.9 %,
  _normalize_decode_outputs 2.8 %, fill_from/try_small_copy 4.0 %, record_batch_in_overlap 2.5 %,
  _apply_cuda_graph_metadata 2.2 %, set_mamba_track_indices_from_reqs 2.0 %, prepare_for_decode 1.5 %,
  alloc_for_decode 1.4 %, _compute_mrope_positions_decode 1.2 %, init_new 0.9 %.  There is no single
  CPU hotspot left; the decode step is GPU-bound with a ~1 ms/step scheduling bubble.
- The trace of the profiled rollout (/tmp/r22/profile/trace): per decode step the kernels sum to 5.8 ms
  (5.06 ms of union) over a 5.2 ms span, with 0.3 ms of gaps inside the step; the decode step's period is
  6.2 ms => ~1 ms/step (8 s per rollout) of scheduling bubble: the CPU work between the end of one
  forward and the launch of the next (the batch preparation + the CUDA-graph input fill) is not overlapped
  with the GPU.  Removing it needs a scheduler-level double buffer (prepare the next step's metadata while
  the GPU runs the current one) - too deep for the time left; every individual piece is 1-2 % of the step.
- The GPU work itself is at the roofline: the weight GEMMs run at 3.0 TB/s (HBM ~4.2 TB/s), the TP
  all-reduces cost 0.74 ms/step (6 s per rollout) and the GDN kernels 0.49 ms/step.

The step's dead time.  From the full-loop log of round 11 (/tmp/r16/full/full_cand/train.log): the rollout
metric is logged at 10:28:02 and the trainer only starts wake_up at 10:28:05 and the train at 10:28:07 =>
~3-5 s per step between "the samples are complete" and "the train step starts", i.e. 4-6 % of the step,
all of it on an idle GPU.  The candidates are the reward/conversion/publish path in the rollout manager and the
Ray handover of the converted batch (the top-128 logprobs are 512 MB per rollout).
- Instrumentation added (off by default, SLIME_OPT_PHASE_TIMING=1): slime/ray/rollout.py times
  _get_rollout_data / log_rollout_data / batch_builder.begin / convert / publish / split_by_dp,
  and train.py times the generate call, the handover and the train call.
- A CPU micro-benchmark of the parts that were suspected: the top-k validation (_validate_head_arrays) is
  3.8 ms per sample (0.24 s for the 64 samples of a rollout) and the base64 decode 2.6 ms per sample, so the
  validation is NOT the dead time.

Train side, the CPU Adam.  The instrumentation in the hybrid device optimizer shows cpu_adam 4.75 s of
the 37 s step.  The optimizer is torch.optim.AdamW(..., fused=True) (Megatron's CPUAdam) and it runs at the
host memory bandwidth (~60 GB/s of traffic: the fp32 master + the two moments + the fp32 grads, read and
written once per step, plus the master->bf16 copy).  PyTorch's fused CPU Adam is bitwise identical to the
foreach path on 6 of 7 test tensors and differs by 1.8e-7 relative on one (the largest) => N1 at worst.
A fused CPU Adam of our own could cut the traffic by ~30 % (2.5 s of the step), but it needs a C++ extension
and the gain is small compared with the risk of touching the optimizer.

Full check running (/tmp/r26/check, started 13:05) on the tree with the round-14 changes.
9.2 hRound 16 (2026-11-13 13:05 -> 14:46 UTC) - the full check of the round-14 tree PASSES
Full check (bench.py check --tree /submission/slime_grpo --config /submission/switches.json --out /tmp/r26/check,
log /tmp/r26/check.log, started 13:05, ended 13:45): status PASS, tier N1, performance.speedup 1.7778
(the projected step speedup from the frozen reference timings: rollout 2.0669x, train 1.5426x).
Stages: rollout cand (386 s), train cand (731 s), token forcing, the train/inference mismatch rollout (359 s),
the deterministic train record (576 s), then the statistical gate.
This is the first *complete* verdict of this task on the accumulated tree (slime 03cdfd67 + sglang 6cce324 +
Megatron 6967b3b, switches KEEP_NCCL_GROUPS / NO_RECOMPUTE / SGLANG_RECV_INTERVAL=8 / KEEP_PARAMS_ON_GPU /
SKIP_WAKE_GC).

**Profiling of what is left (CPU-only analysis of the rollout trace /tmp/r22/profile/trace and the train trace
/tmp/r5/profile/trace):**
- Rollout: the decode step's kernels sum to 5.06 ms (union) over a 6.2 ms period, i.e. ~1-1.5 ms of scheduling
  bubble per step (~12 s per rollout).  The inter-step gap contains the CUDA-graph buffer fill
  (fill_from -> _grouped_foreach_copy_ -> the Triton try_small_copy, 200 calls, ~0.4-1.2 ms of host time
  each in the profiled run) and the graph replay.  cudaGraphLaunch itself is only 0.5 us, so the launch is not
  the problem - the *host* work between the result of a step and the next launch is.
  The per-step .tolist() of the (bs, 128) top-k block costs 2 x 53 us (measured on the host).
- Train: the SP/CP collectives are ~11.6 s of the ~31 s of GPU kernels (the AllGather alone 9.66 s over 18113
  calls with a median of 156 us and a long tail up to 61 ms - the tail is the ranks waiting for each other, so the
  collectives are latency/imbalance-bound, not bandwidth-bound).  The CPU Adam is 4.70-4.84 s on the four ranks
  (balanced) and is host-bandwidth-bound (504 GB of host traffic per step).

Change 16a (train side, slime): SLIME_OPT_MAX_TOKENS_PER_GPU (off by default) enlarges the dynamic batch
packer's per-micro-batch token budget.  With the recipe's 4096 the packer puts one ~8.2k-token sample per
micro-batch (64 micro-batches per optimizer step); with 8192 it packs two (32 micro-batches), which halves the
number of sequence/context parallel collectives.  The samples, their order, the batch and the loss are unchanged -
only the grouping of the gradient accumulation (a reduction order) changes.  Numerics: N1.
- Files: slime/slime/utils/arguments.py.

Change 16b (rollout side, sglang): SLIME_OPT_SGLANG_NO_SMALL_COPY (off by default) skips the grouped Triton
"small copy" kernel in the CUDA-graph buffer registry and uses the torch._foreach_copy_ fallback, which copies
the same values with one multi-tensor kernel per dtype group instead of a Triton launch whose Python/JIT dispatch
costs a few hundred microseconds of host time per decode step (the decode loop cannot launch the next forward
before the metadata is in the graph's static buffers, so that host time is on the critical path).
Numerics: N0 (the same values are copied; verified that _foreach_copy_ handles the mixed shapes/dtypes of the
graph's metadata slots and the int64->int32 conversion like the Triton kernel).
- Files: sglang/python/sglang/srt/model_executor/cuda_graph_buffer_registry.py.

Change 16a result (train side): bench.py time --side train with SLIME_OPT_MAX_TOKENS_PER_GPU=8192
(/tmp/r27/time/time.json): candidate windows [40.732, 41.326, 40.898] s vs the frozen reference
[65.599, 66.182, 66.715] s -> train side speedup 1.6014 (was 1.5426).  The actor train time drops from
37.3/37.9/36.6/37.6 s to 36.2/36.6/36.5/38.1 s, the step time from 42.6 to 41.0 s; the per-step
train/rollout log-prob mismatch moves by <= 2e-4 (0.00816/0.00793/0.00752/0.00853/0.00843/0.00855/0.01010/
0.01051 vs the reference run's 0.00816/0.00798/0.00751/0.00855/0.00840/0.00849/0.01008/0.01055).  No OOM: the
GPU still holds 102 GB free after the step (the activations double with the micro-batch, the persistent
buffers are unchanged at 30.9 GB).  Commit: slime f7a06d3c.  The projected step speedup goes 1.7778 -> 1.844.

Change 16b result (rollout side): REVERTED.  bench.py time --side rollout with
SLIME_OPT_SGLANG_NO_SMALL_COPY=1 (/tmp/r28/time/time.json): windows [53.460, 56.495, 59.038] s vs the
frozen reference -> speedup 1.8891, i.e. *not* better than the 2.0669 of the unchanged tree (the rollout
windows of this machine have a +-6 s spread, so a ~1 % effect is not measurable with one run).  Since the
change brought no measurable gain, the file was restored (git checkout): the tree stays minimal and the
copy path (the grouped Triton kernel) stays as sglang ships it.

Incident (structure gate): the phase-timing diagnostics of the previous round had also been added to
slime/train.py.  The judge's structure gate compares the *AST* of slime/train.py with the reference's and
rejects any difference, so check --quick returned ERROR ("slime/train.py differs from the reference").
train.py was restored from the pre-diagnostics commit (slime 04d3b416) and its AST now compares equal to
/opt/researchboost/ws/targets/reference/train.py.  Lesson: only slime/train.py is AST-frozen; every other
file may change, but the entry point must stay byte-for-byte the reference's.
9.6 hRound 17 (2026-11-13 14:07 UTC) - validating the larger micro-batch
bench.py check --quick --tree /submission/slime_grpo --config /submission/switches.json --out /tmp/r30/quick
(the correctness gates only: the rollout side's token forcing / top-k / mismatch and the training side's
statistical gate, on the tree with SLIME_OPT_MAX_TOKENS_PER_GPU=8192).
↑ Slime GRPO