Kimi k3.1-pro 1.35×
Harness: Claude Code · switches: SLIME_KEEP_TRAINING_RESIDENT=1 SLIME_DISABLE_ACTIVATION_RECOMPUTE=1
- Judged
- 1.35×
- Tier
- N1
- Self-best
- 1.25×
- Rounds
- 9
- Hours
- 10.0
- API cost
- $14.7
- Input tokens · 98.4% cached
- 58 M
- Output tokens
- 0.1 M
- Requests
- 387
§ 02Key levers
- Candidate, three windows (s)
- 138.2 / 133.1 / 137.1
- Rollout side
- 1.32×
- Train side
- 1.35×
- Rollout host-side fixes
- partial: only sampler sync removed and top-k row copies merged; top-k JSON serialization untouched
- Skip zero-coefficient reference forward
- ✗
- Train-side main levers
- partial: training resident, no recompute; pairing micro-batches measured 1.44×, blocked by direction cosine 0.931 < 0.955
§ 03Step by step
| Round | Change | Measured | Step | Side, cumulative | Status |
|---|---|---|---|---|---|
| Rollout side | |||||
| 1 | sampler top-p check via host-side flag, per-step any().item() sync removed | rollout-side speedup: — → 1.27 | ×1.27 | 1.27× | kept |
| 2 | decode top-k per-request row D2H merged into one bulk copy of the underlying tensor | rollout-side speedup: 1.27 → 1.32 | ×1.04 | 1.32× | kept |
| 3 | CUDA graph input copy via _foreach_copy_ instead of Triton small_copy (no gain, reverted) | rollout-side speedup: 1.32 → 1.3 | — | — | reverted |
| 4 | uniform top-k packed before D2H + one tolist on the whole tensor (no gain, reverted) | rollout-side speedup: 1.32 → 1.31 | — | — | reverted |
| Train side | |||||
| 6 | training side resident, no offload (rollout offload kept) | train-side speedup: — → 1.15 | ×1.15 | 1.15× | kept |
| 7 | full-layer activation recompute off (recompute_granularity=None) | train-side speedup: 1.15 → 1.35 | ×1.17 | 1.35× | kept |
| 8 | actor micro-batch pair packing at 8704 tokens/GPU (faster but judge INCORRECT, reverted) | train-side speedup: 1.35 → 1.44 | ×1.07 | — | reverted |
| 9 | token cap raised to 8704 for the no-grad log-prob forward only | train window (this round only started timing, no result in the log): — → — | — | — | reverted |
§ ARaw trace
Raw work log9 sections
0.0 hRound 1 — reference profile (started 2026-11-12 23:01 UTC)
- GPU allocation: 4 × Hopper, all idle before launch;/tmphas ample free space. - Trees and switches are unchanged from the reference; all three repositories are clean on branchopt. - Frozen reference: rollout windows 110.466 / 114.317 / 110.612 s (spread 1.19%); train windows 78.308 / 65.628 / 66.051 s (spread 4.28%); full-loop windows 181.049 / 179.582 / 182.965 s. Frozen full-loop fractions: rollout 0.617, train 0.3863. - Profiling the unchanged submission, rollout then train. Detached driver PID 461090; log/tmp/r1/driver.log; outputs/tmp/r1/profile-rolloutand/tmp/r1/profile-train. - Rollout profile/tmp/r1/profile-rollout/kernel_report.md: 3,022.962 ms trace window, GPU busy only 1,009.521 ms (33.4%); 97,940 kernels. GEMMs are the largest kernel classes (19.1%, 9.6%, 9.2%), fused recurrent GDN decode is 9.0%, and TP all-reduces are 11.2%, but the dominant opportunity is host/launch scheduling rather than a new kernel. - Rollout trace attribution: 200 decode steps spent 806.682 ms inaten::item/ 805.335 ms inaten::_local_scalar_dense; the 200 long items (805.039 ms) haveSampler.forwardas their Python parent.Sampler.forwardtotals 1,086.591 ms. The item is thereturn_top_p_log_probs.any().item()guard, evaluated even though every request has the flag false for this top-p=1 recipe. - Training profile ran successfully after 2,378 s at 23:50 UTC and wrote very large traces (about 1.3 GB/rank) under/tmp/r1/profile-train/trace. Its post-run report generation was stopped to avoid CPU contention with the timed run. Reference log stage timings in steady state: ref forward about 10.7-11.2 s, actor train about 46 s, train wait about 9-11 s per roughly 66 s step. A background wait for the oversized profiler hit its 10-minute tool limit and was stopped; the profiler itself continued and completed. - Change (sglang commitb42fcf2): added a CPU-derivedneed_return_top_p_log_probsflag, merged it like the existing token-id flag, and used it instead of a deviceany().item()inSampler.forward. No sampling, log-probability, or model arithmetic changed; tier N0. - Validation before commit:py_compileandgit diff --checkpassed. - Timing command was queued at 23:33 UTC (PID 501191, log/tmp/r1/time-driver.log) and held the GPU lock from 23:50 to 23:58 UTC. Result/tmp/r1/time/time.json: candidate windows 86.756 / 86.109 / 84.313 s; ratios 1.2733 / 1.3276 / 1.3119;side_speedup.rollout = 1.2733;speedup_vs_frozen_reference= 1.2733; projected step speedup = 1.1526. All four GPUs returned to 0 MiB. - Full check attempt at 00:01 UTC (detached PID 526957, log/tmp/r1/check-driver.log, output/tmp/r1/check) did not produce a verdict. The candidate rollout completed successfully in 502 s, then the judge raisedFileNotFoundErrorwhile opening/tmp/r1/check/rollout/score.logfor token forcing. The timed-check path did not create therolloutparent beforescore()opened its log; nocheck.jsonexists. This is a judge-side directory failure, not a gate rejection. Retrying the unchanged command in a fresh output directory with only the expectedrolloutoutput subdirectory pre-created; the judge and gates remain unchanged. - Full check retry/tmp/r1/check-retry/check.jsoncompleted at 00:37 UTC: PASS, N1, projected step speedup 1.1654x. Candidate rollout windows were 84.800 / 85.202 / 85.164 s; side speedup 1.2988x. Exact items, layout, token forcing, top-k, and train-inference mismatch all passed (mismatch ratio 0.9883). All four GPUs returned to 0 MiB. The retry was detached PID 541011 with log/tmp/r1/check-retry-driver.log.
1.6 hRound 2 — optimized rollout profile (started 2026-11-13 00:38 UTC)
- Starting point: sglangb42fcf2, full check/tmp/r1/check-retry/check.jsonPASS at N1 with projected step speedup 1.1654x; switches remain empty and all three repositories are clean. - Re-profiling the current best rollout before selecting another hotspot. Detached PID 570165; log/tmp/r2-profile-driver.log; output/tmp/r2/profile-rollout. - Training-stage evidence retained from round 1: steady-state actor training is about 46-48 s, reference forward about 11-12 s, and train wait about 9 s (including 4.3-4.7 s sleep and 1.3 s wake-up). The profiler's traced step is heavily instrumented and is not a timing result. - Profile/tmp/r2/profile-rollout/kernel_report.md: 2,501.550 ms trace window, GPU busy 952.662 ms (38.1%); 96,156 kernels. The disabled-top-paten::itemoperations are gone. Host attribution shows CUDA-graph buffer filling at 441 ms/200 decode steps, result D2H at 405 ms, and decode top-k formatting at 157 ms. The result path made 64 separate per-request value/index D2H copies per step from two contiguous top-k tensors. - Change (uncommitted): coalesce each decode top-k value/index row list into one pinned D2H copy of its common contiguous base on the existing copy stream, then recreate equivalent host row views. A size guard preserves the old path for truncated/heterogeneous rows. Arithmetic, sampled tokens, and copy-stream ordering are unchanged; code tier N0. - Validation:py_compile,git diff --check, and a GPU functional test passed; both coalesced uniform rows and fallback truncated rows matched their source tensors exactly. - Timing/tmp/r2/time/time.json: candidate windows 81.785 / 86.470 / 82.150 s; ratios 1.3507 / 1.3220 / 1.3465; rollout speedup 1.3220x, projected step speedup 1.1769x. Mean rollout time fell from 85.726 s in round 1 to 83.468 s. Status ok, layout unchanged, all GPUs idle afterward. - Full check/tmp/r2/check/check.jsoncompleted at 01:23 UTC: PASS, N1, projected step speedup 1.1711x. Rollout windows 84.303 / 86.038 / 83.767 s, side speedup 1.3103x; exact items, token forcing, top-k and mismatch passed (mismatch ratio 0.9883). GPUs returned to idle. - Committed as sglang5572a62(Coalesce decode top-k host copies (N0)), with the/tmp/r2/timeresult in the commit message. Round ended 01:23 UTC; the full check is outside the round's 30-minute budget.
2.4 hRound 3 — graph-input and remaining rollout host overhead (started 2026-11-13 01:24 UTC)
- Starting point: sglang5572a62, full check/tmp/r2/check/check.jsonPASS at N1 with projected step speedup 1.1711x; switches empty and repositories clean. - Re-profiling the current best rollout before selecting the next hotspot. Detached PID 626178; log/tmp/r3-profile-driver.log; output/tmp/r3/profile-rollout. - Profile/tmp/r3/profile-rollout/kernel_report.md: trace window 2,159.247 ms (down 342 ms from round 2), GPU busy 951.128 ms (44.0%); 96,180 kernels. Host attribution:CudaGraphBufferRegistry.fill_from473 ms/200 steps and Tritontry_small_copy441 ms; coalesced result D2H helpers are now 102 ms. The next hotspot is graph-input copy launch processing, not model arithmetic. - Change (uncommitted): when PyTorch_foreach_copy_is available, use the registry's existing grouped foreach path instead of the per-step Tritonsmall_copyPython launcher. The Triton path remains the fallback for older PyTorch. Copies and dtypes are unchanged; code tier N0. - Validation:py_compile,git diff --check, and a GPU test of mixed shapes/dtypes (including int32-to-int64) matched ordinarycopy_exactly. - Timing/tmp/r3/time/time.json: candidate windows 85.018 / 85.953 / 82.939 s; ratios 1.2993 / 1.3300 / 1.3337; rollout speedup 1.2993x, projected step 1.1657x. This is not a gain over/tmp/r2/time(1.3220x / 1.1769x), so the one-line foreach experiment was reverted. No correctness run was needed for the reverted experiment; the submission is back to clean, fully checked5572a62. Round ended 01:43 UTC.
2.8 hRound 4 — packed decode top-k formatting (started 2026-11-13 01:45 UTC)
- Starting point: restored clean sglang5572a62; latest full check/tmp/r2/check/check.jsonPASS N1, projected step 1.1711x. The round-3 foreach experiment was reverted. - Profiling the restored best rollout before changing the remaining top-k formatting path. Detached PID 652771; log/tmp/r4-profile-driver.log; output/tmp/r4/profile-rollout. - Profile/tmp/r4/profile-rollout/kernel_report.md: trace window 1,885.505 ms, GPU busy 997.626 ms (52.9%). Current-best host attribution:get_top_logprobs_raw178 ms/200 steps,_async_d2h_rows106 ms,_normalize_decode_outputs67 ms. Uniform top-k was still split into 64 GPU row views before transfer and converted with 64 separatetolist()calls afterward. - Change (uncommitted):OutputLogprobProcessornow keeps uniform decode top-k values/indices packed through the asynchronous D2H transfer; result normalization performs onetolist()per packed tensor. Heterogeneous k retains the previous row-wise/fallback representation. Both overlap and non-overlap result conversion handle packed tensors. Top-k arithmetic and output values are unchanged; code tier N0. - Validation:py_compile,git diff --check, and GPU tests passed. Packed uniform and heterogeneous-fallback outputs matched the originalget_top_logprobs(..., no_copy_to_cpu=False)Python lists exactly. - Timing/tmp/r4/time/time.json: candidate windows 84.392 / 86.786 / 84.594 s; ratios 1.3090 / 1.3172 / 1.3076; rollout speedup 1.3076x, projected step 1.1698x. No gain over the current-best/tmp/r2/time(1.3220x / 1.1769x), and the mean was slower (85.257 vs 83.468 s). The four-file packed-formatting experiment was reverted. The submission is clean at fully checked5572a62; round ended 02:05 UTC.
3.1 hRound 5 — training-side profile (started 2026-11-13 02:06 UTC)
- Starting point: clean sglang5572a62; Megatron-LM and slime unchanged; switches empty. Latest full check/tmp/r2/check/check.jsonPASS N1, projected step 1.1711x. - After two no-gain rollout host-formatting experiments, moving to the untouched training side. Profiling the current best training tree before selecting a hotspot. Detached PID 679264; log/tmp/r5-profile-driver.log; output/tmp/r5/profile-train. Round-1 training profiling took 2,378 s and produced about 1.3 GB/rank; allow this profile to finish rather than extrapolating from its instrumented step time. - Profile completed successfully at 02:52 UTC (wall 2,425 s;/tmp/r5/profile-train/profile.jsonstatus ok). Reports are/tmp/r5/profile-train/kernel_report.mdand/tmp/r5/rank{0,1,3}-report.md. On rank 2 the 97,260.662 ms instrumented trace window had 66,495.084 ms GPU busy (68.4%) and 68,271.326 ms summed kernel time. All-gather was 17,338.216 ms (25.4%, 23,809 calls) and reduce-scatter 5,886.406 ms (8.6%, 11,968 calls), followed by the large GEMM classes. These are not timing results. - Collective durations are strongly skewed by rank waiting: all-gather totals were 18,060.684 / 14,536.043 / 17,338.216 / 10,820.273 ms on ranks 0/1/2/3, while total GPU-busy time was 68,600.631 / 64,828.815 / 66,495.084 / 57,601.797 ms. Rank 3 has both the least all-gather time and least GPU work. Do not interpret collective totals as pure wire cost or change reduction algorithms on this evidence. - Rank-2 host attribution includedCheckpointFunctionBackward43,685.328 ms/2,048,CheckpointFunction18,544.793 ms/2,048,_AllGather8,049.910 ms/4,608,_AttnFuncWithCPAndKVP2P6,317.442 ms/1,536,_VocabParallelLogProbEntropy5,204.984 ms/954, and_maybe_all_reduce5,840.681 ms/4,134. The latter calls are the required TP vocab-parallel softmax reductions, not removable metric collectives.HybridDeviceOptimizer.stepwas 4,627.374 ms for the traced update, so the CPU optimizer is not the leading target. - Uninstrumented stage logs provide a larger scheduling/memory target: each steady step spends about 4-5 s insleepand 1.3-1.4 s inwake_up, in addition to about 11 s reference forward and 40-47 s actor training. Full-loop logs show about 40.5 GB training GPU use before offload and 39.06 GB free after engine graph capture with training offloaded. SGLang sizes its KV pool from currently free memory, so keeping training resident reduces KV capacity rather than adding its full static allocation on top of training. The fixture needs only about 262K tokens per engine, far below the reference capacity of 3,023,676. Round 5 ended at 03:15 UTC with resident-training memory strategy selected for round 6.
4.3 hRound 6 — keep training resident between colocated steps (started 2026-11-13 03:15 UTC)
- Starting point remains clean sglang5572a62; no training-side edits yet. One hotspot: repeated colocated training sleep/wake copies and process-group reconstruction. - Planned switchable change:SLIME_KEEP_TRAINING_RESIDENT=1disables only training offload while preserving rollout offload, synchronous ordering, model layout, and arithmetic. The switch is off by default. This file is outside either individual side, so a full check will judge both sides; initial screening uses judgetime --side train, followed by a real full-loop memory smoke run before considering acceptance. - Implementation and validation: added the environment override inslime/utils/arguments.py, enabled it in/submission/switches.json, and passedpy_compileandgit diff --check. Committed separately as slime23473f1fafter timing. - Judge timing/tmp/r6/time/time.json: candidate train windows 56.712 / 57.049 / 56.419 s; ratios 1.3808 / 1.1504 / 1.1707 against frozen 78.308 / 65.628 / 66.051 s; train speedup 1.1504x, isolated-side projected step gain 1.0532x. Status ok, no layout problems, first-use 0/0/0 s. The timer started at 03:18 and finished at 03:32 UTC. - Full-loop functional smoke/tmp/r6/full-smokeused the realbench/launch.shin MODE=full for two forced-length RL steps with live engines, rollout offload, optimizer updates, and weight synchronization. It exited 0 at about 03:46 UTC with 64 samples in each rollout and no CUDA OOM. Engines retained 3,023,676-token capacity and both post-update rollouts succeeded; all GPUs and processes returned to idle. This was a functional/memory check, not a timing claim. - Full check launched at 03:47 UTC, detached PID 757495, log/tmp/r6/check-driver.log, output/tmp/r6/check(with the expectedrolloutoutput parent pre-created). Both sides are judged becauseslime/utils/arguments.pylies outside the side-specific paths. - Full check/tmp/r6/check/check.jsoncompleted at 04:31 UTC: PASS, N1, projected step speedup 1.2432x. Rollout windows 84.220 / 86.156 / 84.994 s, side speedup 1.3014x; train windows 57.269 / 56.668 / 56.270 s, side speedup 1.1581x. Rollout exact items, token forcing, top-k and mismatch passed (mismatch ratio 0.9890). Training was not bitwise repeatable (bitwise: false), but every N1 statistic and direction passed; update cosine 0.99219 versus minimum 0.95541, and train-inference mismatch ratio was 1.0. All GPUs returned to idle and all three repositories are clean. Round 6 ended at 04:32 UTC.
5.5 hRound 7 — activation recomputation in the resident-training step (started 2026-11-13 04:33 UTC)
- Starting point: full check/tmp/r6/check/check.jsonPASS N1, projected step speedup 1.2432x; current switchesSLIME_KEEP_TRAINING_RESIDENT=1; all repositories clean. - Re-profiling the exact current best training tree before selecting the next hotspot. Detached PID 823207; log/tmp/r7-profile-driver.log; output/tmp/r7/profile-train. The prior trace exposedCheckpointFunctionBackward43,685.328 ms andCheckpointFunction18,544.793 ms in its instrumented step; this run must confirm that recompute remains a leading target after the lifecycle change. The planned bottleneck class is redundant full-layer activation recompute, with any change kept behind a separate switch and tested for memory and judge equivalence. - Profile/tmp/r7/profile-train/profile.jsoncompleted successfully at 05:06 UTC (run wall 1,730 s). Rank-3 trace window 80,670.928 ms, GPU busy 57,002.434 ms (70.7%), 547,229 kernels. All-gather was 12,217.647 ms (20.8%) but remains skewed by rank waiting; the leading GEMM classes followed. Host attribution in the new trace:CheckpointFunctionBackward40,495.526 ms / 2,048 calls andCheckpointFunction15,714.649 ms / 2,048 calls. Forward GDN kernels such ascausal_conv1d_fwd_kernelhad 6,144 calls versus 1,536 backward calls, confirming a full extra forward pass. This is a configuration/memory hotspot, not a request to alter model arithmetic. Verified uninstrumented steady steps in/tmp/r6/check/train_cand/train.loghad about 89 GB free and 45 GB reserved after backward, leaving headroom to retain activations. - Change (uncommitted): default-offSLIME_DISABLE_ACTIVATION_RECOMPUTE=1sets Megatron'srecompute_granularitytoNoneduring argument validation. Data, microbatching, model weights, loss, optimizer and update ordering are unchanged; the initial forward activations are retained instead of recomputed in backward. Expected code tier N0, subject to the judge's training record. The verified/submission/switches.jsonis unchanged during screening;/tmp/r7/switches-exp.jsonenables both this experiment and resident training.py_compileandgit diff --checkpassed. - Judge timing launched at 05:13 UTC, detached PID 852370, log/tmp/r7/time-driver.log, output/tmp/r7/time, usingtime --side trainand the experimental switch file. - Timing/tmp/r7/time/time.jsoncompleted at 05:26 UTC: candidate train windows 48.897 / 48.739 / 48.563 s; ratios 1.6015 / 1.3465 / 1.3601 against the frozen reference; cumulative train speedup 1.3465x, isolated-side projected step gain 1.1104x. Status ok, layout unchanged, first-use 0/0/0 s. For direct attribution,/tmp/r6/timewith only resident training had 56.712 / 57.049 / 56.419 s, so this change removed about 8 s per steady train step; steady actor training fell from about 42.8 s to 34.6-34.9 s. Peak reserved memory rose from about 45 GB to 73 GB, with about 61 GB still free; no OOM or invalid-run signature occurred. - Correctnesscheck --quicklaunched with the experimental switches, detached PID 871217, log/tmp/r7/quick-driver.log, output/tmp/r7/quick(expectedrolloutparent pre-created)./submission/switches.jsonstill contains only the fully checked resident-training switch while the verdict runs. - Quick check/tmp/r7/quick/check.jsoncompleted at 05:57 UTC: PASS, N1. Rollout exact items, forcing, top-k and mismatch passed (mismatch ratio 1.0078). Training record was not bitwise identical, as expected from the reference's own non-repeatability; all N1 gates passed with margin. Update direction cosine 0.99470 versus minimum 0.95541, update norm within tolerance, loss mean difference -8.87e-7, and train-inference mismatch ratio 1.0. The experimental switches are now promoted to/submission/switches.json; all GPUs are idle. - Committed separately as slimec1a1aeb6(Retain activations instead of full-layer recompute (N0)), with/tmp/r7/timeand/tmp/r7/quickevidence in the commit message. Full check launched at 05:59 UTC, detached PID 916848, log/tmp/r7/check-driver.log, output/tmp/r7/check. - Full check/tmp/r7/check/check.jsoncompleted at 06:43 UTC: PASS, N1, projected step speedup 1.2510x. Train windows 49.424 / 48.659 / 48.460 s, side speedup 1.3487x. Rollout windows 92.428 / 85.690 / 85.121 s, side speedup 1.1952x; the slow first window, rather than a training regression, limited the combined projection. All rollout and training gates passed. Training update cosine was 0.99255 versus minimum 0.95541; train-inference mismatch ratio 1.0 and rollout mismatch ratio 0.9993. All GPUs returned to idle. Round 7 ended at 06:44 UTC.
7.7 hRound 8 — actor dynamic-microbatch packing (started 2026-11-13 06:44 UTC)
- Starting point: full check/tmp/r7/check/check.jsonPASS N1, projected step 1.2510x; slimec1a1aeb6; switches enable resident training and no activation recompute; repositories clean. - Re-profiling the exact current best training tree before selecting the next hotspot. Detached PID 981284; log/tmp/r8-profile-driver.log; output/tmp/r8/profile-train. Candidate bottleneck class: small per-microbatch shapes, kernel launches and collectives. With CP2 the current actor packing cap is 8,192 full-sequence tokens, so each roughly 8.2K-token fixture sample occupies its own microbatch. A separate actor-only cap near 8,704 tokens/GPU would pack pairs while leavinglog_probs_max_tokens_per_gpu=4096and the reference forward unchanged. Global batch, sample order and one optimizer step remain fixed; any experiment uses a separate default-off switch and must pass memory and N1 gates. - Profile/tmp/r8/profile-train/profile.jsoncompleted at 07:11 UTC (run wall 1,425 s). Rank-2 trace window 61,689.698 ms, GPU busy 42,413.832 ms (68.8%, hence 31.2% idle), 451,628 kernels. Under-filled classes accounted for 10,237.490 ms (23.7%, 126,866 launches) and short classes 2,975.892 ms (157,645 launches). All-gather was 5,676.773 ms / 18,177 calls (13.2%) and reduce-scatter 1,936.315 ms / 9,408 calls; these still include waiting but their per-microbatch launch count is a target. Removing recompute reduced forward GDNcausal_conv1d_fwd_kernelcalls from 6,144 to 4,608 and total GPU busy from 57.0 s in the prior trace to 42.4 s; these instrumented values are not timing results. Pairing actor microbatches should halve actor-side per-layer launches and collectives while increasing GEMM token dimensions. - Change (uncommitted): default-offSLIME_ACTOR_MAX_TOKENS_PER_GPU=8704overrides onlyargs.max_tokens_per_gpuafterlog_probs_max_tokens_per_gpuhas already defaulted to the recipe's 4096. Actor packing uses 17,408 full-sequence tokens per bin under CP2 and pairs ordinary fixture samples; reference-log-probability batching remains unchanged. Expected tier N1 because microbatch reduction order changes, while data membership/order, global batch, loss formula and one optimizer update remain fixed.py_compileandgit diff --checkpassed. A direct schedule test on 64 lengths of 8,308 tokens verified 64 -> 32 microbatches, two samples per new bin, and unchanged flattened sample order/membership./submission/switches.jsonremains at the verified two-switch configuration during screening;/tmp/r8/switches-exp.jsonenables all three. - Judge timing launched at 07:14 UTC, detached PID 1007145, log/tmp/r8/time-driver.log, output/tmp/r8/time, usingtime --side trainand the experimental switch file. - Timing/tmp/r8/time/time.jsoncompleted at 07:27 UTC: candidate train windows 45.541 / 45.423 / 45.229 s; ratios 1.7195 / 1.4448 / 1.4604; cumulative train speedup 1.4448x, isolated-side projected step gain 1.1350x. Status ok, layout unchanged, first-use 0/0/0 s. Compared with/tmp/r7/time, pairing removed another 3.3 s per steady step; actor train fell to 31.85-32.25 s and ref log-probs remained 8.7-8.9 s. Steady reserved memory was 106.8-109.5 GB with about 25-27 GB free. The first cold step reached 134.06 GB reserved and nearly exhausted GPU memory but completed successfully; subsequent full-loop/cold-record validation is required. No OOM or invalid-run signature occurred. - Correctnesscheck --quicklaunched with the experimental switches, detached PID 1025928, log/tmp/r8/quick-driver.log, output/tmp/r8/quick(expectedrolloutparent pre-created)./submission/switches.jsonremains the fully checked two-switch configuration while the verdict runs. - Quick check/tmp/r8/quick/check.jsoncompleted at 08:00 UTC: INCORRECT at the training statistical gate. Rollout passed (mismatch ratio 0.9974), but paired actor microbatches produced update cosine 0.93118 < 0.95541 and update norm difference -0.00105466, exceeding the 0.00065901 paired-noise tolerance. Loss mean, gradient norm and second-moment norm were within their gates, but that does not override the failed direction/update-norm checks. No bitwise claim was made. Per the gate protocol, the experiment is rejected rather than tuned around the gate. The uncommitted actor-cap override was fully reverted;/submission/switches.jsonnever enabled it, all repositories are clean at the fully checked round-7 candidate, and all GPUs are idle. Round 8 ended at 08:02 UTC with the 1.4448x training timing recorded only as a rejected experiment, not an accepted speedup.
9.0 hRound 9 — no-gradient log-probability microbatch packing (started 2026-11-13 08:02 UTC)
- Starting point: round-8 actor pairing was rejected and fully reverted. Restored candidate is exactly full-check/tmp/r7/check/check.jsonPASS N1, projected step 1.2510x, with resident training and no activation recompute; all repositories clean. - Re-profiling the restored current best before another change. Detached PID 1071493; log/tmp/r9-profile-driver.log; output/tmp/r9/profile-train. Proposed independent hotspot: the no-gradient log-probability pass still useslog_probs_max_tokens_per_gpu=4096and about 8.7-9.5 s per steady step. Only this separate cap may be changed; actor forward/backward packing stays at the accepted 4096 setting after the round-8 statistical failure. The same global samples, order and log-probability computation must remain, with judge N1 validation. - Profile/tmp/r9/profile-train/profile.jsoncompleted at 08:29 UTC (run wall 1,442 s). Rank-2 trace window 62,142.301 ms, GPU busy 44,876.208 ms (72.2%), 451,628 kernels. Under-filled classes were 12,733.922 ms (27.9%, 126,866 launches); all-gather was 7,720.848 ms / 18,177 calls and reduce-scatter 3,000.115 ms / 9,408 calls, both including rank waiting. The no-gradient forward still accounts for 2,048 of the 4,608causal_conv1d_fwd_kernelcalls (actor initial forward and reference forward); reducing its launch count is the selected hotspot. - Fast-path inspection found thatlog_probs_max_tokens_per_gpuis defaulted inarguments.pybut otherwise unused; merely overriding it would have no effect.forward_onlyrestores original sample positions from the iterator's flattenedmicro_batch_indices, so a separate iterator can merge adjacent bins and preserve output order. Implementation merges only within original optimizer-step boundaries, uses total sequence lengths times the CP-aware cap, keeps the actor iterator andnum_microbatchesunchanged, and activates only when the separate log-probability cap differs from the actor cap. - Change (uncommitted): default-offSLIME_LOG_PROBS_MAX_TOKENS_PER_GPU=8704sets only the separate log-probability cap.get_log_prob_data_iteratorinbackends/megatron_utils/data.pyrepacks adjacent no-gradient bins within each original optimizer-step boundary, andcompute_log_probuses it only when dynamic batching is active and the separate cap differs from the actor cap. Actor forward/backward continues with the original iterator and counts. Expected tier N1 because no-gradient forward kernel/reduction order can change; samples, output positions and actor optimization remain unchanged.py_compileandgit diff --checkpassed. Direct tests verified pair packing, varied lengths, cap handling, unchanged input schedule, flattened output order, and no merging across step boundaries./submission/switches.jsonremains at the verified two-switch configuration;/tmp/r9/switches-exp.jsonenables the experiment. - Judge timing launched at about 08:33 UTC, detached PID 1097715, log/tmp/r9/time-driver.log, output/tmp/r9/time, usingtime --side trainand the experimental switch file.