MiniMax M3.1 Flash 1.40×

Harness: Codex · switches: OPD_SKIP_REF_LOGPROBS=1 OPD_REUSE_LOGPROBS_IN_LOSS=1 OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 32768 --max-prefill-tokens 32768 OPD_SAMPLER_NO_SYNC=1 OPD_CPU_THREADS=8 SLIME_DESTROY_WORLD_PROCESS_GROUP=0 OPD_KEEP_PROCESS_GROUPS=1 NCCL_PROTO=Simple OPD_MAX_TOKENS_PER_GPU=12288

Judged
1.40×
Tier
N1
Self-best
1.50×
Rounds
26
Hours
10.0
API cost
$4.92
Input tokens · 97.9% cached
236 M
Output tokens
1.2 M
Requests
1,544
Candidate, three windows (s)
108.6 / 109.3 / 111.8
Rollout side
1.27×
Teacher side
—
Train side
1.55×
Per-step sampler sync
✓ (round 8)
Train-side main levers
partial: skip reference forward, reuse logprob, two samples per micro-batch, process groups kept, NCCL_PROTO=Simple; full recompute kept (selective recompute peaks at 118 GiB, OOM)
Teacher-side changes
chunking only: teacher chunk 32768
Raw work log33 sections
0.2 hRound 1 (start 2026-11-13 07:35 UTC)
Profile first: profile --side train (unchanged reference) -> /tmp/r1/profile (see below).
Hotspot found: the reference-model forward pass (ref_log_probs, 11.8-13.2 s/step = 8-9 % of the whole
step) runs even though --kl-loss-coef 0.00 and --kl-coef (default 0) weight both KL-to-reference terms
by zero, so its output cannot change the loss, the gradients or the updated weights.
Change: slime/slime/backends/megatron_utils/actor.py + loss.py
  - skip the ref forward when kl_coef == 0 and (not use_kl_loss or kl_loss_coef == 0),
    behind the switch OPD_SKIP_REF_LOGPROBS=1 (off by default in the code).
  - the loss then uses kl_loss = 0 (the term is multiplied by kl_loss_coef = 0 => bitwise same loss/grads).
Numbers/tier: N0-equivalent (bitwise identical loss/grads/weights), only the logged kl_loss/ref_log_probs
metrics disappear. Expected: train side 73.2 s -> ~60 s, projected step speedup ~1.08.
0.2 hRound 1 results (08:00-08:25 UTC)
- time --side train (switch OPD_SKIP_REF_LOGPROBS=1) -> /tmp/r1/time/time.json
  train side speedup 1.1086 (candidate windows 64.0/65.9/64.5 s vs the frozen reference 74.1/73.1/72.5 s),
  projected step speedup 1.0487 (the projection uses the train-mode window, which also carries sleep/wake).
  Per-phase (train mode, apples to apples with the reference run in runs/bootstrap/timing_train_2):
  reference ref_log_probs 10.6 s + log_probs 9.9 s + actor_train 46.0 s = 66.8 s  ->  candidate
  log_probs 10.8 s + actor_train 46.1 s = 57.6 s.  The ~10 s of the reference forward is gone; the
  actor log-prob forward got ~0.8 s slower (it now pays the cold-cache cost the ref forward used to pay).
- The train record run must still pass the N1 gate (loss/grad_norm/params unchanged; only the logged
  kl_loss / ref_log_probs metrics disappear).
- Lesson: profile --side train is unusable on this box: the torch profiler export (with_stack +
  profile_memory + record_shapes) writes >800 MB *per rank* of chrome JSON and takes >10 min to dump;
  the kernel events are written last, so a partial trace has no kernel data. Use the per-phase timers
  in slime's log (perf N: {...}) instead, and the judge's time for the numbers.
0.9 hRound 2 (08:25 UTC) — the teacher's prefill chunking
The teacher's scoring tail is 16.3 s of the 154 s step (10.6 %). The judge's own teacher noise set is
{"conc1", "chunk8192", "nooverlap", "noradix"} (bench.json teacher_rules.rebatchings), i.e. re-batching
the teacher with a different prefill chunking is *inside* the reference's own re-batching noise, and the
preparation log records the control "the teacher with another prefill chunking" as accepted at N1.
So the teacher's prefill chunk size is a sanctioned lever (OPD_TEACHER_SERVER_ARGS).
- --chunked-prefill-size 8192: time --side teacher -> /tmp/r2/teacher8192/time.json:
  windows 17.51/17.84/17.30 vs the reference 18.27/18.25/17.88 => teacher side 1.0229 (17.55 s vs 18.13 s).
1.6 hRound 3 (08:35-08:45 UTC) — profile of the student's decode (rollout side)
profile --side rollout -> /tmp/r3/profile (kernel_report.md). 200 decode steps of rollout 1 on one
student engine: window 2270 ms, 92 327 kernels, GPU busy only 40.7 % of the window (sum of kernel
durations 1016 ms -> 5.1 ms/step of kernel time vs a real step of 7.5 ms: the engine's "gen throughput"
4267 tok/s with 32 running reqs). Kernel classes: nvjet GEMMs 613 ms (63 % of the kernel time; the
weights are re-read every step -> 9 GB/step/GPU at TP2, i.e. ~2.9 TB/s = 87 % of the HBM peak, so the
GEMMs are at the roofline), GDN decode 97 ms, TP allreduce 98 ms, the sampling path (softmax over the
248 320-wide vocab, argmax, log, rand) ~65 ms. Conclusion: the student's decode is close to the memory
roofline; the gap is CPU/launch overhead (inflated by the profiler), so there is little to win there.
The teacher's tail, in contrast, is *latency* bound: the teacher scores the last samples one request at
a time at 30-55 k token/s (TP4 allreduce bound), which is why its chunked-prefill size matters.
1.6 hRound 4 (08:46-09:00 UTC) — the student's log-prob forward pass (train side)
Hotspot: the separate log-prob forward pass (9.9 s/step = 6.4 % of the step). It exists because the
OPD advantage needs the student's log-probs *before* the training forward.
The OPD advantage is base_advantage - opd_kl_coef * (student_logp - teacher_logp), and with GRPO
(get_grpo_returns = per-sample reward broadcast) and no --normalize-advantages, the base advantage
does not depend on the student's log-probs at all. So the penalty can be applied inside the training
loss with the log-probs the training forward just computed -- the same values (same logits, same
kernel) that the separate pass produces.
Change (slime, commit 672a012): loss.py gets apply_opd_kl_to_advantages_in_loss; actor.py skips
the separate log-prob pass when the conditions hold (behind OPD_REUSE_LOGPROBS_IN_LOSS=1).
Numerics: step 0's train/loss = 0.05341373383998871 and train_rollout_logprob_abs_diff =
0.008052160032093525 are bitwise identical to the reference's; step 1 differs only by the
reference's own run-to-run noise (the reference is not bitwise repeatable).
time --side train -> /tmp/r4/time/time.json: windows 53.08/52.48/52.97 s vs the frozen reference
74.08/73.09/72.46 s => train side 1.3678x, projected step speedup 1.1462.
(Reference step 1: ref_log_probs 11.1 s + log_probs 9.85 s + actor_train 46.1 s = 74.2 s.
 Candidate step 1: actor_train 47.6 s = 56.3 s. Both extra forward passes are gone.)
1.6 hRound 5 (09:06 UTC) — correctness check of rounds 1-4
check --quick --side rollout --side teacher --side train -> /tmp/r5/quick (all three sides, because
the teacher chunking shows up in the rollout's teacher answers too). Status: see below.
1.6 hRound 5 results (09:06-09:21 UTC) — correctness of rounds 1-4
check --quick --side rollout --side teacher --side train -> /tmp/r5/quick/check.json
status PASS, tier N1 (judged sides: teacher + train, because the teacher's prefill chunking
shows up in the rollout's teacher answers too).
- teacher: mean_abs ratio 1.0128 (bound 2.0), event rate 5.34e-6 vs the reference's 4.25e-6
  (binomial p = 0.265), per-sample bias t = 0.025 -> PASS.
  Teacher windows 17.157 / 17.219 / 16.875 s vs the frozen reference 18.13 s mean (chunk 16384).
- train: update_norm 0.14831 vs 0.14845, exp_avg_sq within, loss_mean within, grad_norm_first
  within, update cosine 0.9941 (floor 0.9566), train_rollout_logprob_abs_diff ratio 1.0
  (bitwise equal to the reference) -> PASS at N1.
(The reference is not bitwise repeatable, so N0 is unreachable; everything is judged at N1.)
2.1 hRound 6 (09:21-09:34 UTC) — the activation recompute (train side)
Hotspot: the recipe asks for `--recompute-granularity full --recompute-method uniform
--recompute-num-layers 1`: every transformer layer's forward is run a second time in the backward
pass. Recompute only trades memory for compute -- the recomputed forward produces the same
activations, so the loss, the gradients and the updated weights are unchanged (step-0 loss and
train_rollout_logprob_abs_diff are bitwise identical to the reference).
Change (slime, commit 29ab85b4): OPD_NO_RECOMPUTE=1 sets args.recompute_granularity = None
(off by default).
time --side train -> /tmp/r6/time/time.json: windows 46.0 / 44.8 / 44.9 s vs the frozen
reference 74.1 / 73.1 / 72.5 s => train side 1.6104x, projected step speedup 1.2192.
actor_train 47.6 s -> 38.5 s (the extra forward is gone).
Not enabled: memory. The train-mode peak GPU memory (nvidia-smi poll) grows from ~50 GB to
73.7 GB; in the colocated full loop the teacher (35 GB: 13.5 weights + 10.1 mamba state + 11.2 KV)
and the engines (~15 GB while their memory occupation is released) are resident on the same GPUs,
so the full-loop peak would be ~123 GB of 139.8 GB. Round 7 makes it memory-safe.
2.3 hRound 7 (09:43 UTC) — memory-safe no-recompute + the rollout side as a gate
Two questions at once:
1. The final evaluation judges all three sides (bench.py: if a.final: sides = set(ALL_SIDES)),
   so the rollout side has to pass with the teacher's chunk-16384 setting even though the diff does not
   touch the rollout files. An accidental edit to slime/slime/utils/arguments.py (a file outside the
   three sides) made this check judge all three sides -> the rollout side is being validated here.
2. The no-recompute variant needs a memory-safe micro-batch cap. Added OPD_MAX_TOKENS_PER_GPU
   (override of --max-tokens-per-gpu, off by default).
   *Measured afterwards*: the fixture's samples are 2.0k-8.7k tokens and the recipe's cap is
   4096*cp_size = 8192 tokens, so every sample already lands alone in its own micro-batch
   (the reference logs 63-64 micro-batches for 64 samples). Lowering the cap changes nothing:
   the packing, the per-micro-batch shapes and the peak memory are identical. The knob is
   therefore useless for the memory problem and will be dropped.
   The real numbers: the train-only peak (nvidia-smi poll) is 73.7 GB with the recompute dropped
   vs ~51 GB with it; in the colocated full loop the teacher (36.4 GB) and the engines (13.4 GB)
   are resident on the same GPUs, so the full-loop peak would be ~123.5 GB of 139.8 GB (16 GB
   free), while the reference's own rollout phase peaks at 98.5 GB. Enabling it is an OOM risk in
   the final evaluation (score 0), so it stays off unless the memory can be cut elsewhere.
2.3 hRound 7 results (09:43-10:18 UTC) — the rollout side passes with all three sides judged
check --quick (config = the submission's switches + OPD_NO_RECOMPUTE=1 + OPD_MAX_TOKENS_PER_GPU=2048)
-> /tmp/r7/quick/check.json: status PASS, tier N1, judged sides: rollout + teacher + train.
- rollout: exact items ok; token forcing mean_abs ratio 0.9862 (limit 2.0), events 2 (rate 9.7e-7 vs the
  reference's 1.46e-6, p = 0.80), sampler paired t = 1.66 (limit 4); the rollout's teacher answers mean_abs
  ratio 0.9899, bias t = 1.44; train-inference mismatch ratio 0.9962 -> PASS.
- rollout windows 75.3 / 76.7 / 75.9 s (mean 75.96) vs the frozen reference 76.21 / 76.55 / 76.53
  -> the rollout side is ~1.01x; the teacher tail is 15.3-16.1 s of it.
- The teacher chunk 16384 and the two skipped forward passes therefore pass every gate of the final
  evaluation (which judges all three sides).
2.8 hRound 8 (10:20 UTC) — the student's decode: a device-to-host sync in the sampler
Profile of the student's sglang worker (bench.py profile --side rollout, /tmp/r3/profile): the decode
window (198 steps, bs=32) has the GPU busy only 40.7% of the time; the host spends 8.7 ms between
steps. The cause: one aten::item per decode step inside Sampler.forward
(sglang/srt/layers/sampler.py: bool(sampling_info.return_top_p_log_probs.any().item()), 199 calls,
781 ms of the 2.27 s window, each one a device-to-host sync). It blocks the host until the whole forward
has finished on the device, so the GPU idles while the host prepares the next step: the decode is
host-bound, not device-bound.
Change (sglang, switch OPD_SAMPLER_NO_SYNC, off by default): the per-request "return top-p log-probs"
flags are already known on the host when the batch is built, so the batch carries a host-side flag
(need_return_top_p_log_probs) and the sampler asks the question without touching the device. The answer
is identical (the device tensor is built from the same host list), so the sampling, the tokens and the
log-probabilities do not change; only the synchronization disappears.
2.8 hRound 8 results (10:20-11:05 UTC) — the sampler's per-step sync removed
time --side rollout (config /tmp/r8/switches.json = the submission's switches + OPD_SAMPLER_NO_SYNC=1)
-> /tmp/r8/time/time.json: windows 58.70 / 60.10 / 58.52 s vs the frozen reference 76.21 / 76.55 / 76.53
-> rollout side 1.2736x (was 1.0126x). The student's decode: e2e latency 59.4 s -> 40.8-42.4 s
(decode throughput per request 138 -> 200 tok/s); the teacher tail is unchanged (14.7-16.0 s).
check --quick (all three sides judged, the diff touches sglang/ and slime/) -> /tmp/r8/quick/check.json:
status PASS, tier N1.
- rollout: token forcing mean_abs ratio 0.9867 (limit 2.0), events 3 (rate 1.46e-6 = the reference's own,
  p = 0.58), sampler paired t = 1.85 (limit 4); the rollout's teacher answers mean_abs ratio 0.9875,
  bias t = 1.51; train-inference mismatch ratio 0.9954 -> PASS.
- teacher: mean_abs ratio 1.0128, events p = 0.26, bias t = 0.025 -> PASS.
- train: mismatch ratio 1.0 -> PASS.
The switch is now in /submission/switches.json; sglang commit 5db2c09.
4.5 hRound 9 (11:00-11:50 UTC) — the no-recompute variant: faster in training mode, *slower* in the full loop
Hypothesis: the full activation recompute (`--recompute-granularity full --recompute-method uniform
--recompute-num-layers 1`) costs a whole extra forward pass (~9.8 s of the 46 s train step) and could be
dropped. Change (slime, switch OPD_NO_RECOMPUTE, off by default): skip the checkpointing so the
activations are kept and the backward does not recompute them. Tier: N0-equivalent (the same forward, the
same loss, the same gradients).
- time --side train (with OPD_NO_RECOMPUTE=1) -> /tmp/r9/time_train_nr: 45.65 / 45.37 / 45.24 s
  vs the frozen reference 74.08 / 73.09 / 72.46 -> the train side 1.6016x (the best train number seen).
- But a candidate-only full-loop run (/tmp/r9/full_nr, perf N: in its train.log) shows the opposite
  in the loop that is actually scored:
    step 1: actor_train 38.5 s, step_time 106.2 s   (reference step 1: 48.7 / 158.6)
    step 2: actor_train 93.4 s, step_time 163.8 s (reference step 2: 50.1 / 156.6)
  and the GPU peak during that run was 129.1 GB of 139.8 GB (the reference's full-loop peak is ~105 GB):
  with the teacher (36.4 GB) and the student engines (13.4 GB) resident on the same GPUs, dropping the
  recompute leaves the actor ~24 GB more activations and the step becomes memory-bound (the alternating
  92-102 s steps are the same phenomenon in train mode, where the peak is only 73.7 GB: the allocator
  thrashes on the 25 GB larger activation footprint).
  => Rejected: OPD_NO_RECOMPUTE stays OFF. The train-mode number (1.60x) is misleading; the scored
  loop is the full loop. (The same alternating slow step exists in the *reference* at the same step index,
  so it is not caused by the change, but the candidate's slow step is much worse.)
- Teacher chunking sweep (time --side teacher, the teacher alone, /tmp/r9/time_t_*):
  chunk 16384 -> 1.058x | chunk 32768 -> 1.061x | chunk 65536 -> 1.057x | **chunk 32768 +
  --max-prefill-tokens 32768 -> 1.0907x** (16.63 / 16.68 / 16.39 s vs the reference 18.13) | chunk 16384
  + max-prefill 32768 -> 1.0599x. The teacher's share of a step is ~10.6 % (the tail after the students
  finish), so this is worth ~0.4 % of the loop.
- The teacher's own log (r9 full run): it prefills 32768-token batches at ~32k token/s, i.e. ~1.7 PFLOP/s
  over 4 GPUs (~30-45 % MFU); the hybrid model's linear-attention scan is the likely limit. No sglang knob
  changes that without changing the teacher's numerics.
4.5 hRound 10 (11:50-12:20 UTC) — where the remaining time goes (diagnostics, no code change yet)
Accounting of one full-loop step of the current candidate (from the r9 full run's perf N: and the
reference's bootstrap log):
  student decode 42 s | teacher tail 15 s | actor_train 46.5 s (fwd 9.8 + recompute 9.8 + bwd 19.6 +
  optimizer ~7) | sleep 4.4 + wake 1.4 + update_weights 2.0 + engines' offload/onload ~2.2 = 114 s
  vs the reference's 158.6 s -> a projected full-loop speedup of ~1.39x.
The decode is at the memory roofline (~94 % of the HBM peak), the optimizer is bound by the host memory
bandwidth (the fp32 master + the two Adam moments + the fp32 grads are ~126 GB of host traffic per GPU
per step), and the recompute cannot be dropped (round 9). The remaining candidates are the sleep/wake
overhead (5.8 s) and the teacher's chunking (+0.45 s).
4.5 hRound 10 results (12:00-12:35 UTC) — the GPU-idle overhead around the training step
Diagnostics (1 Hz nvidia-smi utilization poll next to the judge's own time --side train, /tmp/r10):
the training step itself is GPU-bound (95-100 % utilization on all four GPUs during the forward /
recompute / backward), but every step has a ~12 s GPU-idle window: the sleep (4.2 s) + the wake_up
(1.4 s) + the CPU-offloaded optimizer (~7 s). The per-phase timers of the reference show the same shape.
Two candidate cuts were measured:
1. Skip the host-memory release in the sleep (clear_memory(clear_host_memory=True) ends in
   malloc_trim(0) over a multi-hundred-GB heap): change (slime, switch OPD_SKIP_HOST_CLEAR, off by
   default) -> time --side train /tmp/r10/time_train_hc: windows 53.55 / 55.22 / 55.21 s, train side
   1.3123x vs the baseline's 55.25 / 53.84 / 52.60 s (1.3407x) and the sleep times unchanged
   (4.2 / 4.1 / 6.3 / 4.0 / 4.4 / 4.5 / 4.1 s). => No effect; reverted (the trim is not the cost).
2. The CPU optimizer is at the host-memory roofline: measured on this box, a 1e9-parameter Adam-like
   chain (lerp_ + mul_/addcmul_ + sqrt + addcdiv_) costs 380 ms, i.e. 179 GB/s of host traffic - the same
   190 GB/s that a plain add_ reaches. The optimizer moves ~68 bytes per parameter per step
   (master + 2 moments + grads) over 4.5 B parameters per GPU, i.e. ~1.2 TB per step at 175 GB/s = the
   measured ~7 s. A fused kernel would remove the denominator temporary (-24 % of the traffic, ~1.7 s) and
   keeping the moments on the GPU is impossible in the colocated memory budget (the peak would go from
   105 GB to ~129 GB, which round 9 showed is catastrophic). => No safe win; the optimizer stays.
Other measured facts of this round:
- The engines' offload/onload (the KV cache + the CUDA graphs + the weights, released to the host and
  re-allocated every step) costs ~2.4 s of the step. Keeping the engines' weights on the GPU would save
  ~0.9 s but raises the training-phase peak by 9 GB (105 -> 114 GB of 140 GB), which is too close to the
  regime where round 9's variant collapsed. => Rejected.
- The teacher's prefill chunking is the only sanctioned teacher lever that does not change its kernels
  (a different GDN prefill backend would move the linear-attention state at every chunk, i.e. far beyond
  the reference's own re-batching spread). Best measured: `--chunked-prefill-size 32768
  --max-prefill-tokens 32768` (the teacher side 1.0907x vs 1.058x for chunk 16384) -> now in
  /submission/switches.json; worth ~0.45 s of the ~114 s step.
5.3 hRound 11 (12:35-13:0x UTC) — teacher prefill chunking validated (pending) + the CPU optimizer found
- The r11 check --quick (started 12:35:39) runs the candidate with the new teacher arguments
  (--chunked-prefill-size 32768 --max-prefill-tokens 32768); result in check.json when it finishes.
- New hotspot (train side): the CPU-offloaded Adam runs on ONE core.
  Measured facts:
  - Megatron prints number of parameters on (tensor, gtp_weight_remat, pipeline) model parallel rank: 5,285,884,416
    -> 5.29 B parameters *per rank* (TP2 x CP2; CP does not shard weights), i.e. 21.2 B parameters in total.
  - --optimizer-offload-fraction defaults to 1.0, so the whole optimizer state (fp32 master + both Adam
    moments + the fp32 grad copy) lives on the host: 5.29 B x (4+4+4+4) B = 212 GB of host traffic per rank
    per step, 846 GB per step over the 4 ranks.
  - Megatron builds one torch.optim.AdamW per parameter (build_cpu_optimizer_list) with fused=True,
    and Ray pins OMP_NUM_THREADS=1 in every actor (one CPU per actor) -> the whole update streams 846 GB
    through a single core per rank.
  - Micro-benchmark on this box (300 M params, fused=True AdamW, fp32 master/moments/grad):
    1 thread 331 ms (36 GB/s) | 4 threads 92 ms | 8 threads 60 ms (201 GB/s) | 16 threads 46 ms (256 GB/s).
    With 4 concurrent processes (the 4 ranks): 1 thread 350 ms/rank (aggregate 137 GB/s) vs 8 threads
    88-128 ms/rank (aggregate 390-540 GB/s).
  - Extrapolated: the optimizer is ~6.2 s of the ~53 s train step (846 GB / 137 GB/s) and ~2 s with 8 threads.
    The update is element-wise, so the arithmetic (and therefore the training record) does not change: N0-safe.
- Change (slime, switch OPD_CPU_THREADS, off by default): slime/backends/megatron_utils/actor.py calls
  torch.set_num_threads(n) at the start of MegatronTrainRayActor.init (Ray sets OMP_NUM_THREADS=1 before
  torch starts, but torch.set_num_threads still changes the OpenMP pool afterwards - verified in the
  micro-benchmark: 331 ms -> 60 ms).
- r11 result: PASS (tier N1) (/tmp/r11/quick/check.json), all three sides judged with the new teacher
  arguments (--chunked-prefill-size 32768 --max-prefill-tokens 32768):
  - rollout token forcing mean|d| ratio 0.9864 (limit 2.0), sampler t ok, events p=0.353;
  - rollout's own teacher answers: mean|d| ratio 0.9839, bias 1.7e-05;
  - teacher side: mean|d| ratio 1.0164 (limit 2.0), events p=0.975, bias -7.4e-06 -> the larger prefill
    chunk is inside the reference teacher's own re-batching noise;
  - train-inference mismatch ratio 0.9989; train N1: loss_mean d=1.4e-06 (tol 3.8e-05), update cos 0.9942
    (min 0.9566).
  - teacher replay mean 16.59 s vs the reference's 18.13 s.
6.0 hRound 12 (13:13-13:27 UTC) — the CPU-offloaded optimizer, threaded (train side 1.3407x -> 1.3777x)
Change (slime commit 7684b59b, switch OPD_CPU_THREADS, default off, set to 8 in switches.json):
MegatronTrainRayActor.init calls torch.set_num_threads(n). Ray pins OMP_NUM_THREADS=1 in every actor, and
torch.set_num_threads still changes the OpenMP pool after torch has started (verified in the micro-benchmark:
331 ms -> 60 ms for a 300 M-parameter fused CPU AdamW step).
Measurement (time --side train on the instrumented copy /tmp/exp_tree, /tmp/r12/time_train_cpu8/time.json):
- windows 53.77 / 52.78 / 50.74 s vs the r10 baseline 55.25 / 53.84 / 52.60 s -> train side 1.3777x
  (was 1.3407x), i.e. ~1.5 s less per train step.
- The instrumented optimizer step (OPTSTEP-PROFILE): 10.75 / 10.95 s for the first two steps (they also
  allocate the pinned grad copies and the Adam moments), then 3.10-3.50 s in the steady state.
- The update is element-wise, so the training record is unchanged: the same fused CPU AdamW kernel, only the
  thread count differs -> bitwise identical master weights (verified in the micro-benchmark for 1/4/8/16
  threads) => tier N0 for this change.
Also measured in the same run (the instrumented sleep/wake):
- sleep: clear 0.72 s + destroy_pg 1.79 s + tms_pause 1.88 s + reset_stack 0.00 = 4.40 s
- wake: tms_resume 0.61 s + clear 0.49 s + reload_pg 0.12 = 1.22 s
- the TMS pause copies 11.3 GB to the host at every sleep (host_used 858.5 -> 869.8 GB) and restores it at
  the wake (-11.1 GB): a host round trip of the paused GPU memory (~2.5 s per step). At the init sleep (before
  any training) the same pause takes 0.05 s, so the backup is of memory that only exists after the first train
  step (the vocabulary-sized logit/activation buffers, not the param/grad buffer: that one is already in a
  torch_memory_saver.region(..., enable_cpu_backup=False) because disable_grad_buffers_cpu_backup=True).
6.2 hRound 13 (13:31-13:44 UTC) — do not destroy the NCCL WORLD group at the sleep (train side 1.3777x -> 1.3884x)
Change: the upstream switch SLIME_DESTROY_WORLD_PROCESS_GROUP=0 (no code change; the env var is in slime's
MegatronTrainRayActor.init). With it, the accelerator WORLD group is not destroyed/recreated at every
sleep/wake, so its communicator memory stays on the GPU.
Measured (time --side train, /tmp/r13/time_train_nopg/time.json): windows 53.36 / 51.84 / 50.08 s ->
train side 1.3884x (r12: 53.77/52.78/50.74 = 1.3777x), i.e. ~0.5-0.9 s less per step.
Cost: the GPU keeps the WORLD communicator's memory: used_GB after the offload 1.93 instead of 0.82
(+1.1 GB per GPU), i.e. the full-loop peak would rise from ~105 to ~106 GB of 140 GB (the round-9 collapse
was at 129 GB).
Note: the *sub*groups (65 of them) are still destroyed at every sleep: destroy_pg 1.25-1.82 s after the first
train step (0.17 s at init, before the groups exist). The WORLD destroy itself was 1.79 s.
6.4 hRound 14 (13:44-13:53 UTC) — the teacher's prefill chunking, final sweep (rejected 65536)
time --side teacher with --chunked-prefill-size 65536 --max-prefill-tokens 65536
-> /tmp/r14/time_teacher_64k/time.json: windows 16.941 / 16.965 / 16.929 s -> teacher side 1.0561x,
i.e. *worse* than the --chunked-prefill-size 32768 --max-prefill-tokens 32768 setting
(1.0907x, 16.6 s). The teacher's prefill is already compute bound at ~400 TFLOP/s per GPU
(27 B params, 2 FLOP/param/token, 31-32 k token/s over 4 GPUs -> ~40 % MFU); a bigger chunk does not
move the GEMMs, it only changes where the linear-attention state crosses a chunk boundary (which the
teacher gate tolerates, but the gain is zero). => the 32768/32768 setting stays in switches.json.
6.4 hRound 15 (13:55 UTC) — CUDA MPS: can the teacher overlap the student's decode?
Where the rollout's 57.6 s goes (from the judge's own records of the r8 run, /tmp/r8/time/rollout_tree/judge):
- the 64 samples are generated in 41.6 s (the first one at 32.7 s into the rollout, the last at 41.6 s);
- the teacher's answers all arrive at 57.6 s -> a tail of 16.0 s after the last generation;
- the teacher alone (replay mode) needs 16.6 s of GPU time for the same 525 k tokens.
So the teacher's scoring is a *serial tail*: the requests only exist once a sample is finished (the first
one at 32.7 s), so at best ~9 s of the teacher's work could hide under the student's decode.
Two ways to overlap: (a) make the teacher cheaper (the prefill is compute bound at 40 % MFU; the chunk
size sweep above shows nothing there), (b) let the teacher's kernels and the student's kernels run on the
GPU *at the same time* - the student's decode is memory bound (the GEMMs run at 2.9 of the 3.2 TB/s
read bandwidth) and the teacher's prefill is compute bound, so they use different resources.
Without MPS the two processes take turns on the GPU (context time slicing); with MPS their kernels run
concurrently in one server context per GPU.
Pre-checks on this box (cheap, no full run):
- nvidia-cuda-mps-control -d works here; the daemon starts and the clients connect (the control log
  shows NEW CLIENT / NEW SERVER).
- NCCL works under MPS (a 2-rank allreduce gives the right values, twice, with a sleep in between) -
  this matters because sglang's TP2 and Megatron's collectives are NCCL.
- torch_memory_saver (the Megatron offload) works under MPS (region/pause/resume OK) - it needs the
  LD_PRELOAD hook, exactly as slime sets it.
- a synthetic memory-bound + compute-bound pair on one GPU: the wall time is already ~= the compute-bound
  time without MPS (0.557 s vs 0.53 s alone) and 0.542 s with MPS -> the gain from MPS on this pair is
  small, so the real measurement decides.
Change (sglang, commit pending): sglang/srt/mps.py + a call in launch_server.py that starts the MPS
daemon when CUDA_MPS_PIPE_DIRECTORY is set (off by default; nothing else changes, the same kernels run).
The teacher server is the first process of a run (bench/launch.sh starts it before slime), so the daemon is
up before any CUDA context exists and every later process (the rollout engines, Megatron) joins it.
Measurement running: time --side rollout with the MPS switches -> /tmp/r15/time_rollout_mps.
6.4 hRound 15 results (13:56-14:06 UTC) — CUDA MPS: no gain, rejected
time --side rollout with the MPS switches (/tmp/sw_mps.json, the MPS daemon started by the teacher's
launch_server) -> /tmp/r15/time_rollout_mps/time.json: windows 57.98 / 57.83 / 57.69 s -> the rollout
side 1.3145x, i.e. the same as the same tree without MPS (the teacher-args gain alone explains it).
=> MPS rejected, the sglang change was reverted (the tree is back to the sampler commit only).

Why MPS cannot help here (the analysis that led to the experiment):
- the student's decode is memory bound: the profile shows the decode GEMMs moving 9 GB of weights per
  step per GPU at ~2.9 of the 3.2 TB/s read bandwidth this box measures, i.e. ~90 % of the roofline;
- the teacher's prefill is compute bound (27 B params, 2 FLOP/param/token, ~31 k token/s over 4 GPUs =
  ~420 TFLOP/s per GPU = ~42 % MFU);
- but the two do not overlap in the first place: the response lengths are 1947..8192 with a median of
  8192, so almost all 64 samples finish within the last ~2 s of the 42 s decode. The teacher's requests
  therefore only exist in the last few seconds and its 16.6 s of prefill is a serial tail after the
  students are done (measured: the last sample's generation ends 12.6-16.0 s before the rollout ends).
- so the teacher's tail cannot be hidden under the student's decode; it can only be made cheaper
  (the prefill chunking, swept in round 14) or the samples would have to exist earlier (they cannot).
6.8 hRound 16 (14:07-14:20 UTC) — keep the process groups across the sleep/wake (train side 1.3884x -> 1.4893x)
Hotspot: the actor's sleep destroys the accelerator WORLD group and the ~65 cached subgroups and the
wake rebuilds them; the instrumented sleep was clear 0.72 + destroy_pg 1.79 + tms_pause 1.88 = 4.40 s
per training step, of which the process-group teardown is 1.79 s and the matching reload 0.12 s.
Change (slime, commit 82c4799d, switch OPD_KEEP_PROCESS_GROUPS, off by default):
slime/utils/reloadable_process_group.py: destroy_process_groups() returns early when the switch is on,
so the communicators stay alive across the sleep; the matching reload_process_groups() is then a no-op
(it already skips groups that are still valid).
Measurement (time --side train on the instrumented copy /tmp/exp_tree, /tmp/r16/time_train_keeppg):
windows 49.74 / 47.73 / 47.53 s vs the r13 numbers 53.36 / 51.84 / 50.08 s (the same tree with
SLIME_DESTROY_WORLD_PROCESS_GROUP=0 only) -> train side 1.4893x (was 1.3884x), i.e. ~3.5 s less
per training step.
The instrumented sleep now reads clear 0.73 + destroy_pg 0.00 + tms_pause 1.77 = 2.50 s.
Cost: the GPU keeps the communicators' memory: used_GB after the offload 2.36 instead of 0.82
(+1.5 GB per GPU), so the colocated full-loop peak goes from ~105 to ~107 GB of 140 GB.
Numerics: unchanged (N0) - the same communicators run the same collectives, only their lifetime changes.
Adopted: OPD_KEEP_PROCESS_GROUPS=1 (with SLIME_DESTROY_WORLD_PROCESS_GROUP=0) are now in
/submission/switches.json.
6.8 hRound 17 (14:20 UTC) — the full judge check on the new switch set
bench.py check --tree /submission/slime_opd --config /submission/switches.json --out /tmp/r17/check
(running; the verdict of the full check, all three sides judged because the diff touches sglang/,
slime/slime/rollout/, slime/slime/backends/megatron_utils/ and slime/slime/utils/).
6.8 hRound 17 results (14:20-14:46 UTC) — the timings of the full check are in; the check itself hit a judge-side bug
bench.py check (without --quick) ran all three timing runs and then crashed in the *correctness* phase:
FileNotFoundError: /tmp/r17/check/rollout/score.log. The judge's evaluate() runs the timing runs into
<out>/<side>_cand but the correctness phase writes its scores into <out>/rollout/, a directory that is
only created when the correctness phase runs the rollout itself (the --quick path, run=None). The
reference hits the same path (the bootstrap's own check was a --quick check), and the final evaluation
(check --final) does not hit it (there side_runs is empty and the correctness phase runs its own
rollout). Nothing to fix on our side: the timings it produced are valid, and the correctness gates are run
with check --quick instead (round 18).

Timings of the r17 runs (the same runs the check would have used):
- rollout windows 59.17 / 59.29 / 57.88 s vs the frozen reference 76.21 / 76.55 / 76.53 -> the rollout
  side 1.2879x (the teacher tail is inside these windows);
- teacher windows 16.63 / 16.69 / 16.42 s vs 18.27 / 18.25 / 17.88 -> the teacher side 1.0889x;
- train windows 48.91 / 48.06 / 49.39 s vs 74.08 / 73.09 / 72.46 -> the train side 1.4686x;
- the judge's projection 1/(1 - f_rollout*(1-1/1.2879) - f_train*(1-1/1.4686)) = 1.355x
  (the teacher's term is dropped because the rollout windows already contain its share).
The full loop should follow: the candidate's step is the rollout (59 s) + the train (49 s) + ~5 s of
sleep/wake, weight update and engine offload = ~113 s against the reference's 154 s.
7.8 hRound 18 (14:47-15:22 UTC) — `check --quick` on the current switch set: **PASS, tier N1**
bench.py check --quick --tree /submission/slime_opd --config /submission/switches.json --out /tmp/r18/quick
-> /tmp/r18/quick/check.json: status PASS, tier N1, all three sides judged (the diff touches sglang/,
slime/slime/rollout/ (via the teacher's arguments), slime/slime/backends/megatron_utils/ and
slime/slime/utils/ (a file outside every side -> all three sides are judged)).
- rollout: token forcing mean|d| ratio 0.9961 (limit 2.0), events 4 (rate 1.94e-6 vs the reference's
  1.46e-6, binomial p = 0.35), sampler paired t = 0.886 (limit 4); the rollout's own teacher answers
  mean|d| ratio 0.9941, bias t = 0.74; the train-inference mismatch 0.008066 vs the reference's
  0.008052 (ratio 1.0) -> PASS.
- teacher: mean|d| ratio 1.0164 (limit 2.0), events p = 0.975, bias t = -0.49 -> PASS.
  The teacher's windows: 16.643 / 16.667 / 16.400 s (the reference's frozen mean 18.13 s).
- train: not bitwise (the reference is not bitwise repeatable: 6344 tensors differ), statistical gate:
  update_norm 0.148303 vs 0.148452 (tol 0.001195), exp_avg_sq within, loss_mean within,
  the mismatch ratio 1.0 -> PASS at N1.
The rollout's windows of this run: 59.37 / 58.20 / 61.12 s; the teacher's tail inside them: 15.4 / 14.8 /
14.1 s (the rest is the student's decode, ~42 s, at the HBM roofline).
8.0 hRound 19 (15:22-15:27 UTC) — the teacher's profile: the judge's profiler never fires (no data)
bench.py profile --side teacher -> /tmp/r19/profile/profile.json: **status ERROR, "the profiler wrote no
trace"**. The judge's teacher profile calls sglang's /start_profile with num_steps=64 before the replay of
rollout 1, but one rollout of the teacher's replay is only ~16 prefill batches (32768 tokens each), so the
profiler never reaches 64 forward steps and the teacher is SIGTERM'd 2 s later, before it dumps. The teacher's
kernel breakdown therefore cannot be measured with the judge's instrument (the same happens for the
reference). Read from the teacher's own log instead: the prefill runs at a steady 32 k token/s over
4 GPUs (= 376 TFLOP/s per GPU, ~38 % MFU) in batches of exactly 32768 tokens, and the log confirms
- the GDN prefill already runs on the FlashInfer backend ("Defaulting SM90 GDN prefill backend to
  FlashInfer", which sglang enables automatically only while chunked_prefill_size <= 32768 - that is why
  the 65536 chunk was slower: it falls back to Triton),
- the teacher's requests arrive as a burst at the end of the student's decode (the response lengths are
  1947..8192 with a median of 8192, so nearly all 64 samples finish within the last ~2 s of the 42 s
  decode). The teacher's tail is therefore serial: it cannot overlap with the student's decode, and its
  floor is the teacher's own throughput.
8.0 hRound 20 (15:27-15:32 UTC) — the teacher's prefill batch budget: no effect
time --side teacher with --chunked-prefill-size 32768 --max-prefill-tokens 65536 (the chunked-prefill
size must stay <= 32768 to keep the FlashInfer GDN prefill) -> /tmp/r20/time_teacher_mp64k/time.json:
windows 16.655 / 16.697 / 16.390 s -> the teacher side 1.0908x, i.e. identical to the 32768/32768
setting (1.0907x, 16.58 s). The teacher's replay is not limited by the batch size but by the model's
forward cost. => the teacher's arguments stay as they are.
8.2 hRound 21 (15:33-15:46 UTC) — the CPU-offloaded optimizer with 16 threads: **rejected** (the worst window blows up)
time --side train with OPD_CPU_THREADS=16 -> /tmp/r21/time_train_cpu16/time.json: windows
48.20 / 68.54 / 46.87 s -> the train side 1.0665x (8 threads: 1.4686x). The per-step timers show why:
the *big* fixture step (dump 2, ~2x the tokens) took 63.6 s of actor_train instead of 42.7 s with 8
threads, while every other step is unchanged (42.1-43.6 s). The host has 96 cores, so this is not core
oversubscription: with 4 ranks x 16 threads the host memory bus (the optimizer streams ~1.4 TB per step)
saturates and the step that is most sensitive to it pays. => OPD_CPU_THREADS=8 stays.
8.7 hRound 22 (15:47-16:15 UTC) — the train profile (the first one that produced data)
bench.py profile --side train with the light-profiler switch -> /tmp/r22/profile/kernel_report.md
(the judge's own torch.profiler over one train step on every rank; trace 154 MB/rank).
Window 51.5 s, GPU busy 79.8 %, sum of kernel durations 42.2 s.
- NCCL = 9.19 s (22 % of the window): AllGather 5.86 s (18113 calls, median 157 us, 24 blocks,
  SM cov 0.18 -> badly under-filled), ReduceScatter 2.59 s (9344 calls), AllReduce(f32) 0.35 s,
  SendRecv 0.35 s. The SP all-gather carries [2176,1,4096] bf16 = 17.8 MB, the reduce-scatter 35.6 MB,
  i.e. an effective 130-220 GB/s per rank (the box has no NVLink for these GPUs).
- GEMMs ~17.4 s (nvjet/cutlass classes, SM cov 1.00, well filled).
- The main compute stream has 11.8 s of gaps (5.87 s in gaps > 1 ms): CPU/optimizer stalls, inflated by
  the profiler.
=> the two candidate hotspots for the train side are the collectives (9.2 s, under-filled: 18113
tiny all-gathers) and the CPU stalls (the CPU-offloaded optimizer + Python overhead).
8.9 hRound 23 (16:13-16:27 UTC) — NCCL "Simple" protocol on the train side: small gain, adopted
Hotspot (from round 22): the collectives are on the main compute stream, 18113 all-gathers of 17.8 MB
(average 157 us, 24 blocks, SM cov 0.18) -> the LL protocol's per-call overhead is a candidate.
Change (no code change): the switch NCCL_PROTO=Simple (NCCL_ALGO is reserved by the judge, which
pins Ring; the protocol is free). Tier N0: the transport changes, not the reduction order - proven by
the judge's own train-inference mismatch, which is bitwise the reference's value
(mismatch_steps[0] = 0.008052160032093525, the same number as the reference's).
Measurement (time --side train, /tmp/r23/time_train_proto/time.json): windows **48.239 / 47.932 /
48.217 s -> the train side 1.5027x** (the r17 baseline with the same switches: 48.91 / 48.06 /
49.39 -> 1.4677x), i.e. ~0.7 s less per train step.
=> NCCL_PROTO=Simple is adopted (added to /submission/switches.json).
9.0 hRound 24 (16:27-16:30 UTC) — Transformer Engine's TP/comm overlap (user buffers): **crashes, rejected**
Hotspot: the sequence-parallel collectives (9.19 s of the 51.5 s train window) sit on the main compute
stream, so they are serialized with the GEMMs; TE's tp_comm_overlap fuses them into the GEMM
epilogue (the standard fix).
Change (slime, switch OPD_TP_COMM_OVERLAP, off by default): args.tp_comm_overlap = True in
_set_default_megatron_args (Megatron's own initialize.py then calls
_initialize_tp_communicators() -> TE initialize_ub).
Result: the run dies after 74 s in TE's bootstrap:
`CommOverlapHelper: Assertion failed: intra_domain_group.value()->getBackendType() == backend.
Internal TE error: Intra-node group must be on the same backend (%s) as the world group!undefined`
(TE creates its own new_group(backend="nccl") world group and a new_subgroups_by_enumeration TP
domain group; on this torch build the two report different backend types, and TE aborts).
Even if it started, the user buffers are sized from --seq-length x --micro-batch-size while this
recipe packs variable-length sequences (up to 8.5k tokens per micro-batch), so the declared buffer
shape would not match the real tensors. => rejected, the code change was reverted (the switch is
gone; the tree carries only the changes listed in the summary).
9.2 hRound 25 (16:31-16:47 UTC) — two samples per micro-batch (train side 1.5027x -> **1.5481x**)
Hotspot (round 22): 18113 all-gathers and 9344 reduce-scatters per train step, i.e. ~9 collectives per
transformer layer per micro-batch, and one Python layer call per layer per micro-batch. The recipe's
--max-tokens-per-gpu 4096 (x cp 2 = 8192 tokens per micro-batch) is smaller than one sample
(prompt + 8192 response = ~8.4k tokens), so every sample lands alone in its own micro-batch: the
step runs 64 micro-batches for 64 samples.
Change (slime, switch OPD_MAX_TOKENS_PER_GPU, off by default): raise the cap to 12288 (x cp 2 =
24576 tokens), which packs two samples per micro-batch -> 29-32 micro-batches per step.
The loss is normalized per sample (loss * num_microbatches / step_global_batch_size in
slime/backends/megatron_utils/loss.py, cancelled by Megatron's own 1/num_microbatches), so the
computed loss does not depend on the grouping; only the order of the fp32 accumulation and the GEMM
shapes change (tier N1).
Measurement (time --side train, /tmp/r25/time_train_mtpg/time.json): windows 47.36 / 47.21 / 46.39 s
-> the train side 1.5481x (1.5027x with only NCCL_PROTO=Simple), i.e. ~1.4 s less per step.
Per-step timers: actor_train 43.3 -> 38.7 s (the micro-batch count halves), but the sleep/wake
grows 3.6 -> 6.6 s (the live memory at the sleep grows 37.5 -> 41.2 GB, so the torch-memory-saver
backup copies more), which eats a third of the gain.
The train-inference mismatch moves from 0.0080522 to 0.0080190 (0.4 %, the different GEMM shapes),
i.e. the ratio to the reference is 0.996 - far inside the gate's 2x.
=> OPD_MAX_TOKENS_PER_GPU=12288 is adopted (added to /submission/switches.json).
9.5 hRound 26 (16:48-17:01 UTC) — selective activation recompute: **OOM, rejected**
Hotspot (round 22): the activation recompute is ~9.8 s of the 43 s actor_train (a whole extra
forward pass over the batch). Megatron's selective granularity recomputes only the modules named in
--recompute-modules and keeps the other activations, which would remove most of that extra pass.
Change (slime, switch OPD_SELECTIVE_RECOMPUTE, off by default): recompute_granularity=selective,
recompute_modules=["core_attn"], recompute_method=None, recompute_num_layers=None
(the first attempt died in Megatron's validation: "When using recompute_granularity: selective
recompute_num_layers must be None").
Result: the run dies with a CUDA OOM in the backward (grad_input = torch.empty_like(logits),
5.68 GiB, 118.2 GiB already allocated by PyTorch): without the layer-level checkpointing *every*
layer's activations stay alive until the backward pass, so the peak goes from ~42 GB to ~118 GB.
(The memory print at the sleep still showed 41.8 GB - the blow-up happens inside the backward.)
=> rejected; the experiment was reverted (commit f373cbb2). The full activation recompute is what
keeps this recipe inside the memory budget of the colocated loop.
↑ Slime OPD