DeepSeek V4.1 Flash-1017 1.88×
Harness: DeepSeek's own harness · switches: SLIME_OPT_SKIP_ZERO_KL_REF=1 SLIME_OPT_NO_RECOMPUTE=1 SLIME_OPT_KEEP_TRAIN_RESIDENT=1 SLIME_OPT_LOGPROBS_CHUNK=4096
- Judged
- 1.88×
- Tier
- N1
- Self-best
- 1.94×
- Rounds
- 10
- Hours
- 10.0
- API cost
- $1.41
- Input tokens · 99.8% cached
- 153 M
- Output tokens
- 0.3 M
- Requests
- 407
§ 02Key levers
- Candidate, three windows (s)
- 101.8 / 97.0 / 95.6
- Rollout side
- 2.10×
- Train side
- 1.60×
- Rollout host-side fixes
- ✓ redundant nested top-k removed + orjson (rollout 111 → 82 s), host-side top-p check (→ 52 s)
- Skip zero-coefficient reference forward
- ✓
- Train-side main levers
- partial: no recompute, training model resident, logprob chunk 1024 → 4096
§ 03Step by step
| Round | Change | Measured | Step | Side, cumulative | Status |
|---|---|---|---|---|---|
| Rollout side | |||||
| 1 | redundant nested top-logprobs removed + orjson | rollout-side speedup: — → 1.34 | ×1.34 | 1.34× | kept |
| 2 | FlashInfer GDN decode | decode throughput tok/s: 107 → 94.8 | ×0.89 | — | reverted |
| 3 | CuTe DSL GDN decode | rollout-side speedup: 1.34 → 1.25 | ×0.93 | — | reverted |
| 4 | sampler top-p flag on host, per-step sync removed | rollout-side speedup: 1.34 → 2.1 | ×1.56 | 2.10× | kept |
| 4 | Megatron-native TP/CP-sharded GDN implementation | none (weight checkpoint load failed): — → — | — | — | reverted |
| 8 | batched per-request top-k D2H copies | none (corrupted responses, rolled back): — → — | — | — | reverted |
| Train side | |||||
| 1 | skip reference model forward when KL coefficient is 0 | ref_log_probs time s: 12.2 → 0 | — | — | kept |
| 1 | activation recompute off (enough training memory) | actor_train s (train side 1.3725 after A+B): 47.3 → 38.9 | — | 1.37× | kept |
| 2 | actor resident in GPU memory, skip per-step NCCL group teardown/rebuild | train window s: 47.7 → 41 | ×1.16 | 1.59× | kept |
| 3 | pack 2 samples per microbatch | train-side speedup: 1.59 → 1.74 | ×1.09 | — | reverted |
| 5 | TE TP comm overlap (userbuffer) | none (init / UB buffer shape assertion failed): — → — | — | — | reverted |
| 8 | log-softmax chunked by 4096 tokens | train-side speedup: 1.56 → 1.61 | ×1.03 | 1.61× | kept |
| 9 | gradient all-reduce in bf16 | train-side speedup: 1.61 → 1.55 | ×0.96 | — | reverted |
| 9 | fla fused GDN gate | train-side speedup: 1.61 → 1.6 | ×0.99 | — | reverted |
| 10 | NCCL_PROTO=Simple protocol tuning | train-side speedup: 1.61 → 1.62 | ×1.00 | — | reverted |
§ ARaw trace
Raw work log18 sections
0.1 hRound 1 — profile the unchanged reference, then remove redundant train work + rollout payload
Start ~22:22. Profiles (reference tree, unchanged) launched detached:/tmp/r1/run_profiles.sh-> /tmp/r1/profiles.log (train profile then rollout profile). Change A (train side, switch SLIME_OPT_SKIP_ZERO_KL_REF, default off): the recipe passes--use-kl-loss --kl-loss-coef 0.00, which makeswith_ref=Trueand runs a full extra reference-model forward (~12.2 s/step) whose only consumers (KL advantage penalty, KL loss term) both have coefficient 0. Skip it, and skip the KL loss term in loss.py; advantages already fall back to zero KL for kl_coef==0. Parameters are unaffected (the ref pass is forward-only). Change B (train side, switch SLIME_OPT_NO_RECOMPUTE, default off):--recompute-granularity full --recompute-method uniform --recompute-num-layers 1recomputes all 32 layers during backward; free GPU memory during training is ~105 GB so drop it. Recomputation re-runs the same forward, so the arithmetic is unchanged (peak activation memory rises). Change C (rollout side): sglang tokenizer_manager puts the nestedoutput_top_logprobs(8192 x 128 tuples per sample, ~2x the compact payload) into every non-streaming response in addition to the base64output_topk_*arrays that slime actually consumes. Drop the nested field for non-streaming compact responses (streaming/text-logprob clients unchanged). No data change. Change D (rollout side): parse the response with orjson instead of stdlib json.
0.1 hRound 1 results (time, 2026-11-12 22:52-23:14, out /tmp/r1/time)
- rollout side speedup 1.344 (windows candidate 82.27, 82.24, 82.08 s vs frozen 110.57, 112.877, 110.458) - train side speedup 1.3725 (windows candidate 47.96, 47.51, 47.65 s vs frozen 66.812, 66.467, 65.4) - projected step speedup 1.3423 (bar: max(1.02, 1+2*spread) = 1.079) - train perf after A+B: actor_train 38.9 s (was 47.3), ref_log_probs gone, step_time 48.3 s (was 67.5) - rollout after C+D: 4 rollouts 87.7 / 80.2 / 80.7 / 78.9 s (was 118.3 / 108.5 / 111.9 / 108.3) - quick check started 23:16 -> /tmp/r1/quick (log /tmp/r1/quick.log) Reference profile: the judge's train profile (torch.profiler on all ranks) was killed after its export hung for ~10 min; the 359 MB/rank gz traces were truncated and unusable. The rollout profile was killed with it to start timing. Lesson: train traces are enormous; do not block a round on them.
0.1 hRound 1 verdict
check --quick (both sides, switches A+B): PASS, tier N1 (out /tmp/r1/quick, 23:14-23:45).
- rollout exact ok; token forcing mean|d| 0.007744 (ref 0.007686), max_abs 0.749 (ref 1.785),
mismatch 0.008024 (ref 0.008119) -> PASS
- train record exact ok; bitwise differs in 6386 tensors (the reference itself differs in 6388);
statistical gate PASS: update_norm 0.121607 (ref 0.121669, tol 6.6e-4), loss -0.0023683 (tol 2.5e-5),
grad_norm 0.048408 (tol 0.034), cos 0.99255 (min 0.9554), mismatch exactly 0.0081647 -> PASS1.9 hRound 2 — colocated memory/offload (train+rollout interface)
Smoke test of the judge's own launcher in MODE=full (NUM_ROLLOUT=2, /tmp/smoke_full.sh, log /tmp/smoke.log): tests the colocated full loop, whichcheck/timenever run. - Found and fixed two bugs in the resident path: the FlashInfer override was applied before the server-args dict existed; the resident wake-up must rebuild the NCCL groups that the weight update tears down. - FlashInfer GDN decode (SLIME_OPT_FLASHINFER_GDN_DECODE=1) measured in the colocated loop: decode 94.8 tok/s vs 107.0 tok/s for Triton -> 11% SLOWER. Rejected, switch left at 0. - keep-resident (SLIME_OPT_KEEP_TRAIN_RESIDENT=1): sleep 4.4-4.8 s -> 0.42-0.83 s, wake 1.2-1.4 s -> 0.49-0.54 s; but update_weights grew 2.0 -> 2.9-3.9 s (group churn while the actor is resident). Added a follow-up that skips the per-step NCCL group teardown and rebuild in resident mode; re-running the full-loop smoke test with 3 steps.
1.9 hRound 2 results
Full-loop smoke test (judge's launch.sh, MODE=full, 3 RL steps, keep-resident on, FlashInfer off): exit 0, so the colocated loop fits in memory. Per-step perf (s): step 1 (cold): step 146.5 | step 2: step 122.7 (train 37.3, sleep 0.69, wake 0.50, update_weights 0.94) | step 3: step 129.2 vs the reference full loop ~182 s/step. Sleep 4.4-4.8 -> 0.4-0.8 s, wake 1.2 -> 0.5 s, update_weights 2.0 -> 0.9-1.5 s after skipping the group teardown/rebuild.check(non-quick) on these switches is broken by a judge-side bug:judge_rolloutreuses the gate-1 timing run and then writes<out>/rollout/score.logwithout creating<out>/rollout(FileNotFoundError at bench.py:858). It does not affectcheck --final(which uses the full loop). Worked around by usingcheck --quick(gates 0,2,3,4) plustime(gate 1). The timing runs of that attempt did complete; their judge clocks give rollout windows 82.04 / 90.27 / 86.90 s (ref 110.57 / 112.877 / 110.458) -> 1.250, and train windows 40.97 / 41.03 / 41.07 s (ref 66.812 / 66.467 / 65.4) -> 1.592; projected step speedup 1.347. In the no-dump rollouttimerun the windows were 82.27 / 82.24 / 82.08 s (spread 0.2%), so the 90 s window here is the judge's 748 MB dump interfering; the final evaluation does not dump.
3.6 hRound 3 — train-side dynamic-batching cap and log-probs chunk
-time --side trainwith SLIME_OPT_MAX_TOKENS_PER_GPU=8448 (2 samples per microbatch, 32 microbatches instead of 64) and SLIME_OPT_LOGPROBS_CHUNK=4096: train windows 37.48 / 38.07 / 36.84 s -> side speedup 1.7423 (was 1.592 with 41 s windows) (out /tmp/r3/batch, log /tmp/r3/batch.log). Adopted. - Also tested CuTe DSL GDN decode: rollout windows 88.40 / 90.25 / 87.68 s -> side speedup 1.2507, i.e. ~7% slower than the Triton decode (82.2-82.3 s). Rejected, switch stays off (out /tmp/r3/cutedsl). - Re-validating all gates withcheck --quickon the five switches: /tmp/r4/quick.
3.6 hRound 3 verdict — FAILED, reverted
check --quick with MAX_TOKENS_PER_GPU=8448 + LOGPROBS_CHUNK=4096 (/tmp/r4/quick, 01:59-02:29):
rollout PASS, but the train gate is INCORRECT:
update_norm d = -1.04e-3 (tolerance 6.6e-4), update-direction cos 0.9308 (min 0.9554).
Packing two samples into one microbatch changes the per-sample weighting of the loss reducer
(sum_of_sample_mean normalizes per microbatch), i.e. it is not just a reduction order.
Reverted to the three-switch configuration validated at /tmp/r3/quick (PASS N1, both sides).
The log-probs chunk switch alone was not separated/retested (would need another 13 min + 34 min
for a gain of at most a fraction of a second).5.8 hRound 4 — rollout profile: per-step host sync in the sampler (biggest win so far)
profile --side rollouton the current tree (out /tmp/r6/profile, 12 min) gave a 42 MB trace per TP rank for 200 decode steps. Analysis (not a timing tool: kernel/CPU accounting of that trace): - GPU busy (main stream, sum of kernel durations) 4.59 ms per decode step; the wall span between graph replays was 8.22 ms -> ~44% of every decode step was NOT kernel time. -aten::item/_local_scalar_dense: 645 calls, 820 ms CPU, of which 200 calls at ~4.1 ms (one per decode step), each followed bycudaStreamSynchronize. The enclosing Python frame wassglang/srt/layers/sampler.py:forward, right afteraten::log+aten::any. - Source: `if sampling_info.return_top_p_log_probs is not None and bool(sampling_info.return_top_p_log_probs.any().item()). That.item()` syncs the whole decode step on every token even when no request asked for top-p logprobs (score centering uses top-k). - Fix:SamplingBatchInfoalready builds the per-request flag list on the host; mirror itsany()into a bool field (need_return_top_p_log_probs, OR-ed in merge) and read that in the sampler. The device tensor stays for its other consumers. Branch taken is unchanged (N0). Effect (time --side rollout, out /tmp/r6/t1): decode throughput 107 -> ~150 tok/s; rollout windows 52.8 / 53.2 / 58.6 s (the 58.6 s was an outlier; the next run measured 52.7 / 51.8 / 52.4 s), side speedup 1.8857 on that run and 2.099 by the clocks of the next one.check --quickon the fix (out /tmp/r6/quick): PASS N1, both sides; token forcing mean|d| 0.0077233 (unchanged to 7 digits vs before the fix), mismatch ratio 1.0026, train statistical gate PASS (update_norm d -8.9e-5, cos 0.9947), so the fast path computes exactly the same rollout. Also tried this round:SLIME_OPT_GDN_IMPLEMENTATION=megatron(Megatron's TP/CP-sharded GDN implementation). It fails at checkpoint load — the frozentorch_distcheckpoint was converted with the replicated HF layout (separate in_proj_qkv/z/b/a, conv1d) and dist_checkpointing errors on the fused native names. Rejected (the checkpoint is read-only).
5.8 hRound 4 official timing (judge `time`, both sides, out /tmp/r6/t2, 04:04-04:17)
- rollout windows 52.692 / 51.819 / 52.398 s vs frozen 110.57 / 112.877 / 110.458 -> side speedup 2.0984 - train windows 41.670 / 42.555 / 41.338 s vs frozen 66.812 / 66.467 / 65.4 -> side speedup 1.5619 - projected step speedup 1.82; expected full loop (rollout 52 + train cycle ~37 + sync ~1) ~90 s/step vs the reference's 182.2 -> ~2.0x.
5.8 hRound 4 full-loop verification and state
Judge's launcher in MODE=full, 2 RL steps (log /tmp/smoke3.log): exit 0.
step 1 (cold) 116.8 s; step 2 93.9 s = rollout 49.3 + train 38.7 (actor_train 38.5,
sleep 0.69, wake 0.49) + update_weights 1.19. Engine decode 170-175 tok/s (was 107).
vs the reference full loop 182.2 s -> ~1.94x.
Measured bf16 GEMM peak on this GPU (8192^3 cuBLAS): 789 TFLOP/s; the training step runs at
~213 TFLOP/s of useful 6N FLOPs (27% of peak), so the train side may still have headroom;
profiling it now (profile --side train, /tmp/r7/profile).6.9 hRound 5 — train profile, communication and GDN attempts
profile --side trainon the current tree (out /tmp/r7/profile, 04:30-04:57, 694 MB trace/rank, kernel report written by bench/kernel_util.py). GPU busy 38.4 s of the profiled 54.3 s step. Kernel time by class (of 39.6 s): - NCCL collectives 33%: AllGather 7.52 s (12481 calls, median 157 us, p90 1.68 ms), ReduceScatter 3.11 s, AllReduce f32 1.56 s, SendRecv 0.91 s; all flagged under-filled (24 blocks, SM coverage 0.18) and *skewed* (mean >> median: waiting for a peer). - cuBLAS/nvjet GEMMs ~33% (13.0 s), the largest at 132 blocks / full SM coverage. - elementwise/copy/fill 14% (~5.5 s, mostly small under-filled kernels from the GDN plugin's eager gating math and the fp32 casts), fla GDN kernels 10% (~4.0 s), FlashAttention bwd 1.1%. Actions: - SLIME_OPT_TP_COMM_OVERLAP=1 (TransformerEngine userbuffer overlap of the SP all-gather / reduce-scatter): fails at init -- TE's CommOverlapHelper asserts the intra-domain group's backend type, which slime's reloadable process-group wrapper does not satisfy (comm_gemm_overlap.cpp:42 Assertion failed: intra_domain_group...getBackendType() == backend). Rejected (would need the reloadable-group wrapper disabled, which interacts with the CPU offload/optimizer path). - Tried a dense CUDA-graph batch-size bucket list: not pursued (the decode is mostly weight-bandwidth bound, so padding 36 -> 40 wastes little). Measured bf16 GEMM peak: 789 TFLOP/s; the train step does ~213 TFLOP/s of 6N work (27% of peak), and a third of its busy time is un-overlapped collectives, which is the next real opportunity (blocked by the TE/group-wrapper interaction above). State after round 5 (submission): switches = SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT; rollout side 2.0984, train side 1.5619 (time04:04-04:17, /tmp/r6/t2), projected 1.82, full-loop smoke 93.9 s/step vs 182.2 s reference (~1.94x). The two switches added after the last full gate run (GDN implementation, TP comm overlap) are default-off and unset, so the validated behaviour is unchanged.
7.0 hRound 6 — rollout re-profile after the sync fix
profile --side rolloutagain (out /tmp/r8/profile): GPU busy (union of kernels) 4.77 ms per decode step (was 5.2 ms), 441aten::itemcalls with 0 above 2 ms (was 200 at ~4.1 ms), andcudaStreamSynchronize0.4 ms in the whole 200-step trace (was 814 ms). The decode step is now 6.35 ms wall (52 s / 8192 tokens): ~4.8 ms GPU kernel time (55% of it memory-bound cuBLAS GEMMs reading the weights, 10% the fused GDN decode kernel, 10% TP allreduce) plus ~1.6 ms of per-step scheduler/sampler host work. Remaining rollout upside would need the GDN weights to be TP-sharded (the replicated HF wrapper reads the full 1.6 B-parameter GDN on both ranks per step) -- the in-tree Megatron GDN does that but cannot load the frozen checkpoint; or a rewrite of sglang's per-step Python. Both are beyond this round's budget.
7.8 hRound 7 — TP communication overlap investigation (rejected: incompatible with dynamic batching)
The train profile showed 33% of kernel time in sequence-parallel NCCL collectives, so the next target was Megatron/TransformerEngine userbuffer (UB) overlap of the all-gather / reduce-scatter. - Attempt 1 (SLIME_OPT_TP_COMM_OVERLAP=1alone): TE'sCommOverlapHelperasserts the intra-domain group's backend type == the comm backend. slime wraps every torch.distributed group inReloadableProcessGroup(a plain ProcessGroup with the real one inside), so the backend reads as undefined. Failed at actor init. - AddedSLIME_OPT_NO_RELOADABLE_GROUPS=1: skips onlymonkey_patch_torch_dist()(keepingoffload_train=Trueso the torch_memory_saver setup the actor needs stays intact; the destroy/rebuild calls are already skipped by the resident path). Now TE initialises its userbuffers and the run gets into training, but fails at the first overlapped GEMM: `comm_gemm_overlap.cpp:455 copy_into_buffer: Assertion failed: _ubufs[_tp_id].numel() == input_size`. The UB buffers are sized once from(seq_length * micro_batch_size) // cp_size, while this recipe uses dynamic batching with a per-microbatch token count (M = 4096-4224 after CP2 slicing/padding). TE requires the copied input to match the buffer *exactly*, so UB overlap cannot be used with variable-length microbatches without reshaping every microbatch to one fixed M (which would change the batching/data order). Rejected; both switches stay off. Also examined and rejected for this round: fla's fused gate (use_gate_in_kernel) in the GDN plugin would remove ~3-4 eager elementwise kernels per layer (gradients for A_log/dt_bias are supported in the fla backward), worth well under 1% of the step; and fla's native GVA (HV > H) only removes the q/k repeat copy (~0.03 s/step) because the qk products scale with HV either way.
8.0 hRound 8 — small, safe leftovers
-SLIME_OPT_LOGPROBS_CHUNK=4096(log-softmax token chunking; the reduction is per row over the vocab, so this is bitwise identical arithmetic):time --side trainwindows 40.867 / 40.952 / 40.509 s -> side speedup 1.6145 (base 1.5619). Adopted in switches.json. - sglangcopy_to_cpu: the top-k logprob rows were copied device->host one small tensor per request (~64 non-blocking copies plus Python launches per decode step; the rollout trace showed 2.1 ms CPU per step in that annotation). Stack the equally shaped rows, copy once, return per-request CPU views. Committed (7b5cce0); timing below.
8.0 hRound 8 results and revert
-SLIME_OPT_LOGPROBS_CHUNK=4096: train windows 40.867 / 40.952 / 40.509 s -> side speedup 1.6145 (adopted; bitwise-identical arithmetic, it only changes how many tokens the log-softmax processes per chunk). - Batched per-request top-k D2H copy (commit 7b5cce0): reverted (575ed34). A rollout failed with a malformed top-k response ("string indices must be integers" after the client's top-k validation rejected the payload). The batchedtorch.stack(...).to("cpu")did not reproduceasync_d2h's pinned host buffer andrecord_streamlifetime handling, so the copied rows were not safe under the overlap scheduler. The per-request copies stay. - Final switch file now: SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT, LOGPROBS_CHUNK=4096. - Final gate run on that exact tree + switches:check --quick-> /tmp/r8/quick2.
9.0 hRound 9 — two more train-side kernel/communication levers
-SLIME_OPT_BF16_GRAD_REDUCE=1(drop--accumulate-allreduce-grads-in-fp32so the row-parallel weight gradients all-reduce in bf16; the train profile had 1.56 s/step in the fp32 all-reduce):time --side trainwindows 41.141 / 41.362 / 42.173 s -> side speedup 1.5508, i.e. ~3% SLOWER than the fp32 path (bf16 also makes the gradient accumulation buffer bf16). Rejected, switch stays off. -SLIME_OPT_GDN_FUSED_GATE=1: pass the raw a/b projections plus A_log/dt_bias into fla'schunk_gated_delta_rule(use_gate_in_kernel=True, use_beta_sigmoid_in_kernel=True), so-exp(A_log)*softplus(a+dt_bias),sigmoid(b)and the chunk cumsum run inside one kernel instead of ~6 eager elementwise kernels per GDN layer. fla's backward returns dA_log/ddt_bias (verified in fla/ops/gated_delta_rule/gate.py), and the fused gate formula matches the eager one exactly. Timing in flight.
9.0 hRound 9 results
- bf16 gradient reduction: side speedup 1.5508 (41.141 / 41.362 / 42.173 s) -> ~3% SLOWER than the
fp32 path. Rejected, switch off.
- Fused GDN gate: side speedup 1.6038 (40.886 / 41.443 / 40.456 s) vs 1.6145 for the chunk-only
baseline -- within noise, no gain; the eager gate kernels overlap with the GPU work. Rejected,
switch off. (Both switches remain in the tree, default off.)
- Final configuration unchanged and still the validated one:
SKIP_ZERO_KL_REF, NO_RECOMPUTE, KEEP_TRAIN_RESIDENT, LOGPROBS_CHUNK=4096.
### Consolidated final timing (time, both sides, final switches; out /tmp/r8/final, 07:36-07:55)
- rollout windows 52.675 / 50.977 / 59.806 s -> side 1.8469 (third window hit the recurring
last-rollout host jitter; the same code measured 52.692 / 51.819 / 52.398 = 2.0984 at 04:04)
- train windows 41.763 / 40.783 / 40.755 s -> side 1.5998
- projected 1.7135 (with the jittered rollout window); best measured sides: rollout 2.0984,
train 1.6145 -> projected ~1.84.
- Expected full loop: rollout ~51 + train cycle ~37 + weight sync ~1 = ~89 s vs the reference
182.2 s -> ~2.0x.9.8 hRound 10 — NCCL protocol / channel tuning for the un-overlapped collectives
The train profile put 33% of kernel time in sequence-parallel NCCL. Since TE's userbuffer overlap cannot be used with dynamic microbatch shapes, try pure NCCL configuration (env overrides in the switch file; NCCL_ALGO itself is reserved and untouched): -NCCL_PROTO=Simple(the profile's collectives ran the RING_LL low-latency protocol even for 34 MB messages): train windows 40.679 / 41.105 / 40.305 s -> side 1.617, i.e. the same as the 1.6145 baseline. No gain (the profiled skew is partly a profiler artifact). -NCCL_MIN_NCHANNELS=32+NCCL_PROTO=Simple: running.