DeepSeek V4.1 Flash-1017 1.43×

Harness: DeepSeek's own harness · switches: SLIME_OPD_SKIP_REF_FORWARD=1 SLIME_OPD_NO_RECOMPUTE=1 SLIME_OPD_KEEP_TRAIN_RESIDENT=1 OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 16384 --max-prefill-tokens 32768 SLIME_STUDENT_SGLANG_ARGS=--disable-radix-cache

Judged
1.43×
Tier
N1
Self-best
1.43×
Rounds
13
Hours
10.0
API cost
$2.06
Input tokens · 99.6% cached
213 M
Output tokens
0.4 M
Requests
519
Candidate, three windows (s)
107.1 / 108.1 / 106.6
Rollout side
1.29×
Teacher side
1.06×
Train side
1.58×
Per-step sampler sync
✓
Train-side main levers
partial: skip reference forward, no recompute, training resident
Teacher-side changes
chunking only: teacher chunk 16384 + single-batch prefill 32768; student radix cache off
Raw work log15 sections
0.0 hRound 1 — baseline profiles
Thu Nov 12 11:25:40 PM UTC 2026
0.0 hRound 1 (start 23:25 UTC) — baseline profiling
- Started all three bench.py profile runs on the unchanged reference (job 501481, log /tmp/r1/profile.log).
- Reference component timings from runs/bootstrap/timing_train_1/.../train.log (steady step):
  step window 74.6 s = ref_log_probs 10.8 + actor log_probs 10.1 + actor_train 46.7 + sleep/wakeup 6.6.
  Full loop 153.8 s = rollout 76.3 (generation ~59.9 + teacher tail 16.4) + train 67.8 + sleep/wake 5.2
  + update_weights 2.0 + misc.
- Key facts: student decode is bandwidth-bound (~4950 tok/s/engine at short context, ~3700 tok/s at 8k);
  teacher replay (64 x ~8.5k tokens prefill) is ~45% MFU; reference train record is NOT bitwise repeatable
  (repeatable=false), so the statistical N1 gate is the realistic target.
- Candidate changes prepared:
  T1 SLIME_OPD_SKIP_REF_FORWARD: the reference-model log-prob forward pass is dead code when every KL
     coefficient is 0 (recipe: kl_coef 0, kl_loss_coef 0). Saves ~10.8 s/step (train side, loss/grad unchanged).
  T2 SLIME_OPD_NO_RECOMPUTE: drop full activation recompute (memory strategy; arithmetic unchanged).
  Teacher OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 8192 (one of the reference's own frozen rebatchings,
     so it is inside the teacher tolerance by construction).
Thu Nov 12 11:30:02 PM UTC 2026
1.1 hRound 2 (23:45–00:30 UTC) — train: dead ref forward + no recompute; teacher chunk
Measured with time (frozen reference windows in brackets):
- train, switches SKIP_REF_FORWARD+NO_RECOMPUTE (/tmp/sw_train.json): windows 56.239/75.342/55.065 s
  [76.649/88.036/73.417] -> side speedup 1.1685, mean step 62.2 s vs 79.4. Components (steady):
  log_probs 10.8 s (unchanged), actor_train 38.0 s [46.7], sleep+wake+wait 7.4 s [6.6].
  Window 2 hit the same host-side stall the reference shows (actor_train 57 s instead of 38 s).
- OMP_NUM_THREADS=24 + max-tokens-per-gpu=8192 (v3): step 57.4/58.8 s — neutral, dropped.
  max-tokens-per-gpu=16384 OOMs (a 32k-token microbatch needs >100 GB of activations), dropped.
- teacher, OPD_TEACHER_SERVER_ARGS="--chunked-prefill-size 8192": 17.536/17.821/17.324 s
  [18.375/18.303/17.912] -> 1.027. Adding --max-prefill-tokens 32768: identical (1.028); the
  scheduler caps the prefill batch at chunked_prefill_size, and throughput is flat in batch size
  (~30k tok/s, ~41% MFU) -> the teacher is compute-bound, not batching-bound.
- rollout profile (200 decode steps, 1 engine): kernel time 4.99 ms/step of a 6.4 ms step;
  GEMM 68.5% (3.42 ms, all ~0.73-wave 96-block cuBLASLt nvjet kernels at ~1.5-1.9 TB/s of 4.8),
  GDN/mamba 11.7%, comm (allreduce fusion + allgather) 11.0%, elementwise 6.1%, attention 1.5%.
  The allreduce fusion is already active (backend auto -> trtllm); sampling backend already flashinfer.
  -> decode is bandwidth/occupancy-bound at ~30% of HBM peak; no config fast path identified.

Full check started 00:30 UTC on /tmp/r3/check with switches
{SKIP_REF_FORWARD, NO_RECOMPUTE, teacher chunked-prefill 8192}.
Fri Nov 13 12:31:05 AM UTC 2026
2.5 hRound 3 (00:30–01:47 UTC) — FULL CHECK
- First attempt (/tmp/r3) crashed in the judge itself: judge_rollout reuses gate-1's run and never
  creates <out>/rollout, so score() died on open(<out>/rollout/score.log). Pre-creating
  <out>/rollout (+ teacher, train) fixed it. Gate-1 data from that run was still valid:
  rollout windows 75.94/76.46/76.41, train windows 55.4/56.1/56.7.
- Second attempt: /tmp/r4/check → PASS, speedup 1.1424, tier N1 (check.json).
  side speedups: rollout 1.010, teacher 1.0283, train 1.3032.
  - rollout: exact ok; token forcing mean|d| ratio 0.9905 (<2), 4 events (p=0.35), sampler t=1.52 (<4);
    train-inference mismatch ratio 1.0034.
  - teacher: mean|d| ratio 1.0149 of the reference's pooled rebatching bound (limit 2), 10 events (p=0.38),
    bias t=-1.03 (limit 4) — exactly the frozen rebatch_chunk8192 variant, i.e. bitwise that rebatching.
  - train: exact ok; not bitwise (reference itself is not repeatable); statistical gate all within:
    update_norm 0.148319 vs 0.148452 (tol 1.20e-3), exp_avg_sq 0.0016464 vs 0.0016418 (tol 4.4e-4),
    loss_mean 0.0516844 vs 0.0516909 (tol 3.8e-5), grad_norm_first 0.99652 vs 0.99674 (tol 1.65e-3),
    update-direction cosine 0.99358 (min 0.95660). mismatch ratio 1.0.
2.9 hRound 4 (01:53–02:16 UTC) — marginal levers, separate trees (no submission change yet)
- teacher --disable-radix-cache (on top of chunk 8192): windows 17.063/17.156/16.746 s -> teacher side
  1.0669 (vs 1.0283 for chunk only). Numerics validated by check --quick on a reference-shaped tree
  (/tmp/reftree + teacher switch): PASS, i.e. bitwise the frozen rebatch_noradix variant.
  Full-loop value is small (~0.4 s of the teacher tail, ~0.3 %), so NOT put into switches.json; the
  submission keeps the config that passed the full check.
- teacher --linear-attn-prefill-backend flashinfer: 1.0289 (no gain over chunk 8192).
- student --linear-attn-decode-backend flashinfer: rollout windows 86.6/87.2/87.4 s -> 0.883x
  (12 % SLOWER than the reference). Rejected.
Fri Nov 13 02:16:17 AM UTC 2026
3.5 hRound 5 (02:16–02:52 UTC) — student radix cache off (accepted)
- time --side rollout with student SLIME_STUDENT_SGLANG_ARGS="--disable-radix-cache" (teacher chunk 8192):
  windows 74.177/74.216/74.173 s vs frozen 77.402/77.246/77.164 -> rollout side 1.0403
  (was 1.010 with the radix cache on). The radix cache never hits (prefix_cache_hit_rate 0) and its
  mamba extra-buffer state tracking costs ~2.1 s per rollout.
- Gate validation: check --quick --tree /tmp/rolltree (reference slime/Megatron + sglang copy whose
  disable_radix_cache default is True) -> rollout gate PASS: exact ok, token forcing mean|d| ratio
  0.9957 (limit 2), 7 events (rate 2.3x reference but binomial p=0.034 >> 0.001), sampler t=0.739,
  train-inference mismatch ratio ~1.0; teacher gate PASS.
- Final switches.json:
  {SLIME_OPD_SKIP_REF_FORWARD=1, SLIME_OPD_NO_RECOMPUTE=1,
   OPD_TEACHER_SERVER_ARGS="--chunked-prefill-size 8192",
   SLIME_STUDENT_SGLANG_ARGS="--disable-radix-cache"}
  (teacher kept at chunk 8192 only: that exact teacher was validated end-to-end by the full check;
   adding --disable-radix-cache to the teacher is validated on the teacher gate but gains only ~0.4 s
   and would leave the online teacher+rollout combination untested, so it is not used.)
- Expected projection with the final switches: 1/(1 - 0.501*(1-1/1.0403) - 0.5146*(1-1/1.3032)) = 1.161.
- A final full check (/tmp/r7/check) with the final switches was queued at 02:52 UTC.
Fri Nov 13 02:52:27 AM UTC 2026
4.2 hRound 6 (02:52–03:30 UTC) — final full check on the final switches
- /tmp/r7/check: gate 1 rollout 1.0302 (windows 74.104/74.222/74.899), teacher 1.0255,
  train 0.9869 -> status UNAVAILABLE (speedup 1.0079). The train window 3 hit the host-side stall
  (+17 s inside the *microbatch loop*, actor_train 56 s instead of 38 s), so the min ratio fell below 1.
- Diagnosis of the stall: it appears only in --debug-train-only runs (fixture dumps 1–3):
  v1 timing (step 5), r7 (step 6), the reference's own train timing runs (step 5, both runs), while
  the bootstrap full-loop reference A/B and control runs (5 RL steps each, 10 timed steps) show no
  stall (spreads 0.8 %). Not Triton first-use (first_use.seconds = 0 in every timed window), not
  cgroup CPU throttling (0.16 s total throttled), not memory pressure (1 TiB container limit, ~200 GB
  used). Treated as a train-only artifact; the final evaluation times the full loop.
- Because a stalled full-loop window would zero the score, the last large lever (keep Megatron resident
  across the rollout, ~5 s/step) was re-rejected: it cannot be validated in the full loop with the
  judge's tools (--final is unavailable), the colocate path with offload_train=False also changes
  the grad/param CPU-backup flags used by the weight sync, and an OOM/regression would zero the score.
- Final state: see the summary below. No processes left running; GPUs idle.
4.6 hRound 7 (03:35–04:03 UTC) — expensive levers tried
- Log-prob pass microbatch repacking (SLIME_OPD_LOGPROB_TOKENS_PER_MB in actor.py: repack the
  samples into 4-sample bins for the inference-only log-prob forward, which stores no activations;
  forward_only already restores the sample order from the iterator). Implemented, timed, rejected:
  the log-prob pass did get faster (10.8 -> 9.3 s) but the transient 8.6 GB logits buffers grew the
  caching allocator's reserved memory 33 -> 40 GB, which made the colocated sleep grow 4.1 -> 9.1 s and
  wake 1.4 -> 2.8 s: step 60.5-66.1 s vs 56-58 s before. Code reverted (git checkout).
- Student engine --enable-torch-compile (inductor max-autotune compiles the decode forward and
  captures it in the CUDA graph; the autotune log shows BLOCK_M=32 Triton GEMMs, promising against the
  0.73-wave 128x32 cuBLASLt kernels in the trace). First attempt with the recipe's mem-fraction 0.45
  OOMs during capture (needs >12 GB on top of the 63 GB static pool + 35 GB teacher). Retesting with
  --mem-fraction-static 0.35 (KV pool still ~2.5x the used capacity). Compile time is outside every
  timed window (graph capture is at engine init), but it makes each run ~10-20 min longer.
Fri Nov 13 04:02:48 AM UTC 2026
- Student --enable-torch-compile — rejected (memory). Even with --mem-fraction-static 0.35
  (max-autotune) or 0.40 with SGLANG_TORCH_COMPILE_MODE=default, the decode-graph capture +
  inductor workspaces OOM during engine init (needs 8.3-12.2 GB on top of the 49-56 GB static pool and
  the 35 GB teacher; only 1.3-2.7 GB free). The colocated 4-GPU layout has no room for the compile
  workspace. Note: with compile on, the engine takes a different GDN decode path ("fused GDN decode
  QKVZ/BA unpack + indexed Conv1D") and inductor autotunes BLOCK_M=32 Triton GEMMs, so it is the
  promising direction if the memory were available; it is not on this layout.
- Submission left at the validated Round-5 configuration (no further switch changes).
5.4 hRound 8 (04:20–04:46 UTC) — keep the actor resident across the rollout (ACCEPTED)
- Change: SLIME_OPD_KEEP_TRAIN_RESIDENT=1 (committed 0ae905df, slime/utils/arguments.py):
  with the colocated recipe the actor otherwise offloads to host inside every train step
  (sleep 4.1 s + wake 1.4 s + process-group destroy/reload + cold-cache effects). The resident
  actor fits: the student engines still release their KV/state pool during training
  (offload_rollout untouched).
- time --side train (judge tool): windows 47.072 / 46.565 / 46.323 s vs
  [76.649 / 88.036 / 73.417] -> side speedup 1.5849. Steady components: log_probs 9.5 s
  (was 10.8-12), actor_train 35.2 s (was 38-39, no cold allocator/process-group reload),
  train_wait 1.7 s (was 7.4).
- Full-mode validation (the judge's own bench/launch.sh, MODE=full, NUM_ROLLOUT=2, forced lengths,
  run under the judge's GPU flock, no timing read): exit 0, 2 RL steps, no OOM; peak GPU memory
  125 GB of 140 GB during generation, 96 GB after the weight sync; the step-1 train-inference
  mismatch 0.00810 (reference 0.00805) so the weight sync is correct; update_weights 0.73-1.36 s.
- switches.json now: SKIP_REF_FORWARD, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT, teacher chunk 8192,
  student radix cache off. Projected step speedup with rollout 1.030 and train 1.585 = 1.257;
  expected full-loop speedup ~154.2/125.7 = ~1.23.
- check --quick on the final switches started 04:46 UTC (/tmp/r12/quick) to re-validate the
  three correctness gates with the resident actor.
Fri Nov 13 04:46:36 AM UTC 2026
5.4 hRound 8 verdict — quick check PASS with the final switches
/tmp/r12/quick/check.json (04:46–05:21 UTC): status PASS, all three sides PASS with the resident
actor: rollout PASS (exact + token forcing + teacher-on-rollout + mismatch), teacher PASS, train PASS
(exact ok; update_norm 0.148195 vs 0.148452, loss_mean 0.051672 vs 0.051691, grad_norm_first 0.996354
vs 0.996736, update-direction cosine 0.99253 vs min 0.95660).
6.8 hRound 9 (05:23–06:11 UTC) — FULL CHECK on the final configuration
/tmp/r13/check/check.json: status PASS, projected step speedup 1.2554, tier N1.
- gate 1: rollout side 1.0294 (74.568/74.591/74.959 s vs 77.402/77.246/77.164), teacher side 1.0294
  (17.49/17.78/17.34 vs 18.375/18.303/17.912), train side 1.581 (46.55/46.92/47.00 vs
  76.649/88.036/73.417). No per-window stall this time.
- correctness: rollout PASS (exact, token forcing, teacher-on-rollout, mismatch), teacher PASS,
  train PASS (update_norm 0.148440 vs 0.148452, loss_mean 0.051687 vs 0.051691,
  grad_norm_first 0.996474 vs 0.996736, update-direction cosine 0.99564 vs min 0.95660).
- Expected final full-loop speedup: 154.2 / (74.6 rollout + 47.0 train + ~4 sync/misc) = ~1.23.
Fri Nov 13 06:14:12 AM UTC 2026
7.2 hRound 10 (06:15–06:35 UTC) — profile + torch.compile retry with one captured bs; no gain
- Profiled the current best rollout (200 decode steps, radix cache off, /tmp/r14/profile_rollout):
  kernel time 5.00 ms/step of a 6.4 ms step; GEMM 63% (same 0.73-wave nvjet classes), GDN decode
  9.9% (0.50 ms, 66% of HBM peak), allreduce fusion 9.4%, attention 2.6%, elementwise ~4%,
  sampling 1.5%. The mamba state-tracking kernel is gone (that is the radix-off gain). No new hotspot.
- Retried student --enable-torch-compile with a single captured decode bs (`--cuda-graph-bs-decode
  [32], --mem-fraction-static 0.30`, keep-resident active) to fit the compile workspace. The engine
  still dies during graph capture: an inductor-generated kernel allocates a 1.88e9-element fp32 buffer
  (empty_strided_cuda((1882718208,)), 7.5 GB) inside the compiled linear-attention decode path and
  OOMs. Inductor's codegen for this hybrid GDN decode is not usable at this model/batch; the direction
  is closed. Parser change reverted; tree back at 0ae905df.
Fri Nov 13 06:34:23 AM UTC 2026
7.6 hRound 11 (06:35–07:25 UTC) — second full check (reproducibility)
- /tmp/r16/check (same tree/switches as the PASSING /tmp/r13/check): gate-1 rollout windows
  74.473/74.147/74.380 s (stable, side ~1.037), teacher 1.029, train windows
  46.37 / 47.11 / 46.99 / 62.87 s -> the last timed window hit the train-only host stall again
  (+16 s on actors), so this check projects ~1.10 instead of r13's 1.2554. The stall has only ever
  been seen in --debug-train-only runs (fixture dumps), not in full-loop runs.
- Robustness note: with the resident actor the full step is ~125 s vs the reference's 154 s, so even a
  +16 s stall in a full-loop window would still give a ratio of ~141 -> 154/141 = 1.09, above the
  max(1.02, 1+2*spread) bar; the change also widened that margin.
- Correctness gates of r16 are re-running; the r13 full check already PASSed all three with the same
  configuration (and /tmp/r12/quick PASSed with the identical switches).
Fri Nov 13 07:01:35 AM UTC 2026
9.2 hRound 12 (07:33–08:32 UTC) — sampler per-step device sync removed (ACCEPTED, big)
- CPU-profiled a student engine's scheduler with py-spy during a rollout: **70.9 % of the samples sat in
  one line of Sampler.forward** — sampling_info.return_top_p_log_probs.any().item(). The field is
  always a CUDA bool tensor (never None), so a reduction + device-to-host sync ran on *every decode
  step* even though the recipe never asks for top-p logprobs, stalling the scheduler and breaking the
  overlap pipeline.
- Fix (sglang commit 453bb3d, sglang/): carry the same information as a CPU flag
  (need_return_top_p_log_probs, built from the per-request custom params and OR-ed on batch merge),
  exactly equivalent to the tensor check. Sampling results unchanged; only the sync is gone.
- time --side rollout: windows 60.10 / 59.44 / 58.85 s (was 74.57/74.15/74.38) -> rollout side
  1.2879 (was 1.030). ~14 s per rollout recovered.
- FULL CHECK /tmp/r19/check: PASS, projected step speedup 1.4342 (side: rollout 1.2943,
  teacher 1.0283, train 1.5796), all three correctness gates PASS (rollout exact/token forcing/
  teacher-on-rollout/mismatch; teacher; train statistical).
- Expected final full-loop speedup: 154.2 / (59.4 rollout + 47.0 train + ~4 sync/misc) = ~1.40.
Fri Nov 13 08:34:04 AM UTC 2026
9.8 hRound 13 (08:48–09:13 UTC) — CPU profiles of the train and teacher; teacher prefill chunk 16384
- py-spy of a Megatron rank during a train step: 26 % of the CPU samples in TE's CP-attention backward
  (context_parallel.py:2869, the dq[cu_seqlens_q_padded[-1]:].fill_(0) GPU-scalar slice that syncs
  once per full-attention layer per backward), ~19 % in Triton JIT compilation (warm-up steps only;
  first_use inside the timed windows is 0), 8.9 % torch.cuda.synchronize from the Triton autotuner
  (also warm-up), and the rest diffuse. The TE sync only makes the CPU wait while the GPU runs, and the
  patched alternative would need monkey-patching an installed package, so it was not pursued.
- py-spy of the teacher scheduler during a replay: the process is mostly waiting on the GPU (60 samples
  in 12 s; 55 % memory-pool alloc, 23 % CUDA-graph replay), i.e. the 15 s teacher tail is GPU-bound
  prefill (42 % MFU), no CPU serialization to remove.
- Teacher --chunked-prefill-size 16384 --max-prefill-tokens 32768 (one chunk per 8.5k-token request
  instead of two): time --side teacher windows 17.165/17.160/16.941 s -> teacher side 1.0573
  (was 1.0283); teacher gate PASS (mean|d| ratio 1.0128, 11 events p=0.26, bias t=0.02,
  /tmp/r23/quick_t). ~0.4 s off the rollout's teacher tail.
- switches.json updated to the 16k teacher chunks; full check /tmp/r24/check started 09:13 UTC.
Fri Nov 13 09:13:42 AM UTC 2026
↑ Slime OPD