DeepSeek V4.1 Flash-1017 1.88×

Harness: DeepSeek's own harness · switches: SLIME_OPT_SKIP_ZERO_KL_REF=1 SLIME_OPT_NO_RECOMPUTE=1 SLIME_OPT_KEEP_TRAIN_RESIDENT=1 SLIME_OPT_LOGPROBS_CHUNK=4096

Judged
1.88×
Tier
N1
Self-best
1.94×
Rounds
10
Hours
10.0
API cost
$1.41
Input tokens · 99.8% cached
153 M
Output tokens
0.3 M
Requests
407
Candidate, three windows (s)
101.8 / 97.0 / 95.6
Rollout side
2.10×
Train side
1.60×
Rollout host-side fixes
✓ redundant nested top-k removed + orjson (rollout 111 → 82 s), host-side top-p check (→ 52 s)
Skip zero-coefficient reference forward
✓
Train-side main levers
partial: no recompute, training model resident, logprob chunk 1024 → 4096
RoundChangeMeasuredStepSide, cumulativeStatus
Rollout side
1redundant nested top-logprobs removed + orjsonrollout-side speedup: — → 1.34×1.341.34×kept
2FlashInfer GDN decodedecode throughput tok/s: 107 → 94.8×0.89—reverted
3CuTe DSL GDN decoderollout-side speedup: 1.34 → 1.25×0.93—reverted
4sampler top-p flag on host, per-step sync removedrollout-side speedup: 1.34 → 2.1×1.562.10×kept
4Megatron-native TP/CP-sharded GDN implementationnone (weight checkpoint load failed): — → ———reverted
8batched per-request top-k D2H copiesnone (corrupted responses, rolled back): — → ———reverted
Train side
1skip reference model forward when KL coefficient is 0ref_log_probs time s: 12.2 → 0——kept
1activation recompute off (enough training memory)actor_train s (train side 1.3725 after A+B): 47.3 → 38.9—1.37×kept
2actor resident in GPU memory, skip per-step NCCL group teardown/rebuildtrain window s: 47.7 → 41×1.161.59×kept
3pack 2 samples per microbatchtrain-side speedup: 1.59 → 1.74×1.09—reverted
5TE TP comm overlap (userbuffer)none (init / UB buffer shape assertion failed): — → ———reverted
8log-softmax chunked by 4096 tokenstrain-side speedup: 1.56 → 1.61×1.031.61×kept
9gradient all-reduce in bf16train-side speedup: 1.61 → 1.55×0.96—reverted
9fla fused GDN gatetrain-side speedup: 1.61 → 1.6×0.99—reverted
10NCCL_PROTO=Simple protocol tuningtrain-side speedup: 1.61 → 1.62×1.00—reverted
Raw work log18 sections
0.1 hRound 1 — profile the unchanged reference, then remove redundant train work + rollout payload
Start ~22:22. Profiles (reference tree, unchanged) launched detached:
/tmp/r1/run_profiles.sh -> /tmp/r1/profiles.log (train profile then rollout profile).

Change A (train side, switch SLIME_OPT_SKIP_ZERO_KL_REF, default off):
the recipe passes --use-kl-loss --kl-loss-coef 0.00, which makes with_ref=True and runs a full
extra reference-model forward (~12.2 s/step) whose only consumers (KL advantage penalty, KL loss
term) both have coefficient 0. Skip it, and skip the KL loss term in loss.py; advantages already
fall back to zero KL for kl_coef==0. Parameters are unaffected (the ref pass is forward-only).
Change B (train side, switch SLIME_OPT_NO_RECOMPUTE, default off):
--recompute-granularity full --recompute-method uniform --recompute-num-layers 1 recomputes all
32 layers during backward; free GPU memory during training is ~105 GB so drop it. Recomputation
re-runs the same forward, so the arithmetic is unchanged (peak activation memory rises).
Change C (rollout side): sglang tokenizer_manager puts the nested output_top_logprobs
(8192 x 128 tuples per sample, ~2x the compact payload) into every non-streaming response in
addition to the base64 output_topk_* arrays that slime actually consumes. Drop the nested field
for non-streaming compact responses (streaming/text-logprob clients unchanged). No data change.
Change D (rollout side): parse the response with orjson instead of stdlib json.
0.1 hRound 1 results (time, 2026-11-12 22:52-23:14, out /tmp/r1/time)
- rollout side speedup 1.344 (windows candidate 82.27, 82.24, 82.08 s vs frozen 110.57, 112.877, 110.458)
- train side speedup 1.3725 (windows candidate 47.96, 47.51, 47.65 s vs frozen 66.812, 66.467, 65.4)
- projected step speedup 1.3423 (bar: max(1.02, 1+2*spread) = 1.079)
- train perf after A+B: actor_train 38.9 s (was 47.3), ref_log_probs gone, step_time 48.3 s (was 67.5)
- rollout after C+D: 4 rollouts 87.7 / 80.2 / 80.7 / 78.9 s (was 118.3 / 108.5 / 111.9 / 108.3)
- quick check started 23:16 -> /tmp/r1/quick (log /tmp/r1/quick.log)

Reference profile: the judge's train profile (torch.profiler on all ranks) was killed after its
export hung for ~10 min; the 359 MB/rank gz traces were truncated and unusable. The rollout
profile was killed with it to start timing. Lesson: train traces are enormous; do not block a round
on them.
0.1 hRound 1 verdict
check --quick (both sides, switches A+B): PASS, tier N1 (out /tmp/r1/quick, 23:14-23:45).
- rollout exact ok; token forcing mean|d| 0.007744 (ref 0.007686), max_abs 0.749 (ref 1.785),
  mismatch 0.008024 (ref 0.008119) -> PASS
- train record exact ok; bitwise differs in 6386 tensors (the reference itself differs in 6388);
  statistical gate PASS: update_norm 0.121607 (ref 0.121669, tol 6.6e-4), loss -0.0023683 (tol 2.5e-5),
  grad_norm 0.048408 (tol 0.034), cos 0.99255 (min 0.9554), mismatch exactly 0.0081647 -> PASS
1.9 hRound 2 — colocated memory/offload (train+rollout interface)
Smoke test of the judge's own launcher in MODE=full (NUM_ROLLOUT=2, /tmp/smoke_full.sh,
log /tmp/smoke.log): tests the colocated full loop, which check/time never run.
- Found and fixed two bugs in the resident path: the FlashInfer override was applied before
  the server-args dict existed; the resident wake-up must rebuild the NCCL groups that the
  weight update tears down.
- FlashInfer GDN decode (SLIME_OPT_FLASHINFER_GDN_DECODE=1) measured in the colocated loop:
  decode 94.8 tok/s vs 107.0 tok/s for Triton -> 11% SLOWER. Rejected, switch left at 0.
- keep-resident (SLIME_OPT_KEEP_TRAIN_RESIDENT=1): sleep 4.4-4.8 s -> 0.42-0.83 s,
  wake 1.2-1.4 s -> 0.49-0.54 s; but update_weights grew 2.0 -> 2.9-3.9 s (group churn while
  the actor is resident). Added a follow-up that skips the per-step NCCL group teardown and
  rebuild in resident mode; re-running the full-loop smoke test with 3 steps.
1.9 hRound 2 results
Full-loop smoke test (judge's launch.sh, MODE=full, 3 RL steps, keep-resident on, FlashInfer off):
exit 0, so the colocated loop fits in memory. Per-step perf (s):
  step 1 (cold): step 146.5 | step 2: step 122.7 (train 37.3, sleep 0.69, wake 0.50, update_weights 0.94) |
  step 3: step 129.2
vs the reference full loop ~182 s/step. Sleep 4.4-4.8 -> 0.4-0.8 s, wake 1.2 -> 0.5 s,
update_weights 2.0 -> 0.9-1.5 s after skipping the group teardown/rebuild.

check (non-quick) on these switches is broken by a judge-side bug: judge_rollout reuses the
gate-1 timing run and then writes <out>/rollout/score.log without creating <out>/rollout
(FileNotFoundError at bench.py:858). It does not affect check --final (which uses the full
loop). Worked around by using check --quick (gates 0,2,3,4) plus time (gate 1).
The timing runs of that attempt did complete; their judge clocks give
rollout windows 82.04 / 90.27 / 86.90 s (ref 110.57 / 112.877 / 110.458) -> 1.250, and
train windows 40.97 / 41.03 / 41.07 s (ref 66.812 / 66.467 / 65.4) -> 1.592;
projected step speedup 1.347. In the no-dump rollout time run the windows were
82.27 / 82.24 / 82.08 s (spread 0.2%), so the 90 s window here is the judge's 748 MB dump
interfering; the final evaluation does not dump.
3.6 hRound 3 — train-side dynamic-batching cap and log-probs chunk
- time --side train with SLIME_OPT_MAX_TOKENS_PER_GPU=8448 (2 samples per microbatch,
  32 microbatches instead of 64) and SLIME_OPT_LOGPROBS_CHUNK=4096:
  train windows 37.48 / 38.07 / 36.84 s -> side speedup 1.7423 (was 1.592 with 41 s windows)
  (out /tmp/r3/batch, log /tmp/r3/batch.log). Adopted.
- Also tested CuTe DSL GDN decode: rollout windows 88.40 / 90.25 / 87.68 s -> side speedup
  1.2507, i.e. ~7% slower than the Triton decode (82.2-82.3 s). Rejected, switch stays off
  (out /tmp/r3/cutedsl).
- Re-validating all gates with check --quick on the five switches: /tmp/r4/quick.
3.6 hRound 3 verdict — FAILED, reverted
check --quick with MAX_TOKENS_PER_GPU=8448 + LOGPROBS_CHUNK=4096 (/tmp/r4/quick, 01:59-02:29):
rollout PASS, but the train gate is INCORRECT:
  update_norm d = -1.04e-3 (tolerance 6.6e-4), update-direction cos 0.9308 (min 0.9554).
Packing two samples into one microbatch changes the per-sample weighting of the loss reducer
(sum_of_sample_mean normalizes per microbatch), i.e. it is not just a reduction order.
Reverted to the three-switch configuration validated at /tmp/r3/quick (PASS N1, both sides).
The log-probs chunk switch alone was not separated/retested (would need another 13 min + 34 min
for a gain of at most a fraction of a second).
5.8 hRound 4 — rollout profile: per-step host sync in the sampler (biggest win so far)
profile --side rollout on the current tree (out /tmp/r6/profile, 12 min) gave a 42 MB trace per
TP rank for 200 decode steps. Analysis (not a timing tool: kernel/CPU accounting of that trace):
- GPU busy (main stream, sum of kernel durations) 4.59 ms per decode step; the wall span between
  graph replays was 8.22 ms -> ~44% of every decode step was NOT kernel time.
- aten::item/_local_scalar_dense: 645 calls, 820 ms CPU, of which 200 calls at ~4.1 ms
  (one per decode step), each followed by cudaStreamSynchronize. The enclosing Python frame was
  sglang/srt/layers/sampler.py:forward, right after aten::log + aten::any.
- Source: `if sampling_info.return_top_p_log_probs is not None and
  bool(sampling_info.return_top_p_log_probs.any().item()). That .item()` syncs the whole decode
  step on every token even when no request asked for top-p logprobs (score centering uses top-k).
- Fix: SamplingBatchInfo already builds the per-request flag list on the host; mirror its any()
  into a bool field (need_return_top_p_log_probs, OR-ed in merge) and read that in the sampler.
  The device tensor stays for its other consumers. Branch taken is unchanged (N0).

Effect (time --side rollout, out /tmp/r6/t1): decode throughput 107 -> ~150 tok/s; rollout windows
52.8 / 53.2 / 58.6 s (the 58.6 s was an outlier; the next run measured 52.7 / 51.8 / 52.4 s),
side speedup 1.8857 on that run and 2.099 by the clocks of the next one.
check --quick on the fix (out /tmp/r6/quick): PASS N1, both sides; token forcing mean|d|
0.0077233 (unchanged to 7 digits vs before the fix), mismatch ratio 1.0026, train statistical gate
PASS (update_norm d -8.9e-5, cos 0.9947), so the fast path computes exactly the same rollout.

Also tried this round: SLIME_OPT_GDN_IMPLEMENTATION=megatron (Megatron's TP/CP-sharded GDN
implementation). It fails at checkpoint load — the frozen torch_dist checkpoint was converted with
the replicated HF layout (separate in_proj_qkv/z/b/a, conv1d) and dist_checkpointing errors on the
fused native names. Rejected (the checkpoint is read-only).
5.8 hRound 4 official timing (judge `time`, both sides, out /tmp/r6/t2, 04:04-04:17)
- rollout windows 52.692 / 51.819 / 52.398 s vs frozen 110.57 / 112.877 / 110.458 -> side speedup 2.0984
- train windows 41.670 / 42.555 / 41.338 s vs frozen 66.812 / 66.467 / 65.4 -> side speedup 1.5619
- projected step speedup 1.82; expected full loop (rollout 52 + train cycle ~37 + sync ~1)
  ~90 s/step vs the reference's 182.2 -> ~2.0x.
5.8 hRound 4 full-loop verification and state
Judge's launcher in MODE=full, 2 RL steps (log /tmp/smoke3.log): exit 0.
  step 1 (cold) 116.8 s; step 2 93.9 s = rollout 49.3 + train 38.7 (actor_train 38.5,
  sleep 0.69, wake 0.49) + update_weights 1.19. Engine decode 170-175 tok/s (was 107).
  vs the reference full loop 182.2 s -> ~1.94x.
Measured bf16 GEMM peak on this GPU (8192^3 cuBLAS): 789 TFLOP/s; the training step runs at
~213 TFLOP/s of useful 6N FLOPs (27% of peak), so the train side may still have headroom;
profiling it now (profile --side train, /tmp/r7/profile).
6.9 hRound 5 — train profile, communication and GDN attempts
profile --side train on the current tree (out /tmp/r7/profile, 04:30-04:57, 694 MB trace/rank,
kernel report written by bench/kernel_util.py). GPU busy 38.4 s of the profiled 54.3 s step.
Kernel time by class (of 39.6 s):
  - NCCL collectives 33%: AllGather 7.52 s (12481 calls, median 157 us, p90 1.68 ms),
    ReduceScatter 3.11 s, AllReduce f32 1.56 s, SendRecv 0.91 s; all flagged under-filled
    (24 blocks, SM coverage 0.18) and *skewed* (mean >> median: waiting for a peer).
  - cuBLAS/nvjet GEMMs ~33% (13.0 s), the largest at 132 blocks / full SM coverage.
  - elementwise/copy/fill 14% (~5.5 s, mostly small under-filled kernels from the GDN plugin's
    eager gating math and the fp32 casts), fla GDN kernels 10% (~4.0 s), FlashAttention bwd 1.1%.
Actions:
  - SLIME_OPT_TP_COMM_OVERLAP=1 (TransformerEngine userbuffer overlap of the SP all-gather /
    reduce-scatter): fails at init -- TE's CommOverlapHelper asserts the intra-domain group's
    backend type, which slime's reloadable process-group wrapper does not satisfy
    (comm_gemm_overlap.cpp:42 Assertion failed: intra_domain_group...getBackendType() == backend).
    Rejected (would need the reloadable-group wrapper disabled, which interacts with the CPU
    offload/optimizer path).
  - Tried a dense CUDA-graph batch-size bucket list: not pursued (the decode is mostly
    weight-bandwidth bound, so padding 36 -> 40 wastes little).
Measured bf16 GEMM peak: 789 TFLOP/s; the train step does ~213 TFLOP/s of 6N work (27% of peak),
and a third of its busy time is un-overlapped collectives, which is the next real opportunity
(blocked by the TE/group-wrapper interaction above).

State after round 5 (submission): switches = SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT;
rollout side 2.0984, train side 1.5619 (time 04:04-04:17, /tmp/r6/t2), projected 1.82,
full-loop smoke 93.9 s/step vs 182.2 s reference (~1.94x).
The two switches added after the last full gate run (GDN implementation, TP comm overlap) are
default-off and unset, so the validated behaviour is unchanged.
7.0 hRound 6 — rollout re-profile after the sync fix
profile --side rollout again (out /tmp/r8/profile): GPU busy (union of kernels) 4.77 ms per
decode step (was 5.2 ms), 441 aten::item calls with 0 above 2 ms (was 200 at ~4.1 ms), and
cudaStreamSynchronize 0.4 ms in the whole 200-step trace (was 814 ms). The decode step is now
6.35 ms wall (52 s / 8192 tokens): ~4.8 ms GPU kernel time (55% of it memory-bound cuBLAS GEMMs
reading the weights, 10% the fused GDN decode kernel, 10% TP allreduce) plus ~1.6 ms of per-step
scheduler/sampler host work. Remaining rollout upside would need the GDN weights to be TP-sharded
(the replicated HF wrapper reads the full 1.6 B-parameter GDN on both ranks per step) -- the
in-tree Megatron GDN does that but cannot load the frozen checkpoint; or a rewrite of sglang's
per-step Python. Both are beyond this round's budget.
7.8 hRound 7 — TP communication overlap investigation (rejected: incompatible with dynamic batching)
The train profile showed 33% of kernel time in sequence-parallel NCCL collectives, so the next
target was Megatron/TransformerEngine userbuffer (UB) overlap of the all-gather / reduce-scatter.
- Attempt 1 (SLIME_OPT_TP_COMM_OVERLAP=1 alone): TE's CommOverlapHelper asserts the
  intra-domain group's backend type == the comm backend. slime wraps every torch.distributed group
  in ReloadableProcessGroup (a plain ProcessGroup with the real one inside), so the backend
  reads as undefined. Failed at actor init.
- Added SLIME_OPT_NO_RELOADABLE_GROUPS=1: skips only monkey_patch_torch_dist() (keeping
  offload_train=True so the torch_memory_saver setup the actor needs stays intact; the
  destroy/rebuild calls are already skipped by the resident path). Now TE initialises its
  userbuffers and the run gets into training, but fails at the first overlapped GEMM:
  `comm_gemm_overlap.cpp:455 copy_into_buffer: Assertion failed: _ubufs[_tp_id].numel() ==
  input_size`. The UB buffers are sized once from
  (seq_length * micro_batch_size) // cp_size, while this recipe uses dynamic batching with a
  per-microbatch token count (M = 4096-4224 after CP2 slicing/padding). TE requires the copied
  input to match the buffer *exactly*, so UB overlap cannot be used with variable-length
  microbatches without reshaping every microbatch to one fixed M (which would change the
  batching/data order). Rejected; both switches stay off.

Also examined and rejected for this round: fla's fused gate (use_gate_in_kernel) in the GDN
plugin would remove ~3-4 eager elementwise kernels per layer (gradients for A_log/dt_bias are
supported in the fla backward), worth well under 1% of the step; and fla's native GVA
(HV > H) only removes the q/k repeat copy (~0.03 s/step) because the qk products scale with
HV either way.
8.0 hRound 8 — small, safe leftovers
- SLIME_OPT_LOGPROBS_CHUNK=4096 (log-softmax token chunking; the reduction is per row over the
  vocab, so this is bitwise identical arithmetic): time --side train windows
  40.867 / 40.952 / 40.509 s -> side speedup 1.6145 (base 1.5619). Adopted in switches.json.
- sglang copy_to_cpu: the top-k logprob rows were copied device->host one small tensor per
  request (~64 non-blocking copies plus Python launches per decode step; the rollout trace showed
  2.1 ms CPU per step in that annotation). Stack the equally shaped rows, copy once, return
  per-request CPU views. Committed (7b5cce0); timing below.
8.0 hRound 8 results and revert
- SLIME_OPT_LOGPROBS_CHUNK=4096: train windows 40.867 / 40.952 / 40.509 s -> side speedup 1.6145
  (adopted; bitwise-identical arithmetic, it only changes how many tokens the log-softmax
  processes per chunk).
- Batched per-request top-k D2H copy (commit 7b5cce0): reverted (575ed34). A rollout failed
  with a malformed top-k response ("string indices must be integers" after the client's top-k
  validation rejected the payload). The batched torch.stack(...).to("cpu") did not reproduce
  async_d2h's pinned host buffer and record_stream lifetime handling, so the copied rows were
  not safe under the overlap scheduler. The per-request copies stay.
- Final switch file now: SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT, LOGPROBS_CHUNK=4096.
- Final gate run on that exact tree + switches: check --quick -> /tmp/r8/quick2.
9.0 hRound 9 — two more train-side kernel/communication levers
- SLIME_OPT_BF16_GRAD_REDUCE=1 (drop --accumulate-allreduce-grads-in-fp32 so the row-parallel
  weight gradients all-reduce in bf16; the train profile had 1.56 s/step in the fp32 all-reduce):
  time --side train windows 41.141 / 41.362 / 42.173 s -> side speedup 1.5508, i.e. ~3%
  SLOWER than the fp32 path (bf16 also makes the gradient accumulation buffer bf16). Rejected,
  switch stays off.
- SLIME_OPT_GDN_FUSED_GATE=1: pass the raw a/b projections plus A_log/dt_bias into fla's
  chunk_gated_delta_rule(use_gate_in_kernel=True, use_beta_sigmoid_in_kernel=True), so
  -exp(A_log)*softplus(a+dt_bias), sigmoid(b) and the chunk cumsum run inside one kernel
  instead of ~6 eager elementwise kernels per GDN layer. fla's backward returns dA_log/ddt_bias
  (verified in fla/ops/gated_delta_rule/gate.py), and the fused gate formula matches the eager one
  exactly. Timing in flight.
9.0 hRound 9 results
- bf16 gradient reduction: side speedup 1.5508 (41.141 / 41.362 / 42.173 s) -> ~3% SLOWER than the
  fp32 path. Rejected, switch off.
- Fused GDN gate: side speedup 1.6038 (40.886 / 41.443 / 40.456 s) vs 1.6145 for the chunk-only
  baseline -- within noise, no gain; the eager gate kernels overlap with the GPU work. Rejected,
  switch off. (Both switches remain in the tree, default off.)
- Final configuration unchanged and still the validated one:
  SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT, LOGPROBS_CHUNK=4096.

### Consolidated final timing (time, both sides, final switches; out /tmp/r8/final, 07:36-07:55)
- rollout windows 52.675 / 50.977 / 59.806 s -> side 1.8469 (third window hit the recurring
  last-rollout host jitter; the same code measured 52.692 / 51.819 / 52.398 = 2.0984 at 04:04)
- train windows 41.763 / 40.783 / 40.755 s -> side 1.5998
- projected 1.7135 (with the jittered rollout window); best measured sides: rollout 2.0984,
  train 1.6145 -> projected ~1.84.
- Expected full loop: rollout ~51 + train cycle ~37 + weight sync ~1 = ~89 s vs the reference
  182.2 s -> ~2.0x.
9.8 hRound 10 — NCCL protocol / channel tuning for the un-overlapped collectives
The train profile put 33% of kernel time in sequence-parallel NCCL. Since TE's userbuffer overlap
cannot be used with dynamic microbatch shapes, try pure NCCL configuration (env overrides in the
switch file; NCCL_ALGO itself is reserved and untouched):
- NCCL_PROTO=Simple (the profile's collectives ran the RING_LL low-latency protocol even for
  34 MB messages): train windows 40.679 / 41.105 / 40.305 s -> side 1.617, i.e. the same as the
  1.6145 baseline. No gain (the profiled skew is partly a profiler artifact).
- NCCL_MIN_NCHANNELS=32 + NCCL_PROTO=Simple: running.
↑ Slime GRPO