Kimi k3.1-pro 1.35×

Harness: Claude Code · switches: SLIME_KEEP_TRAINING_RESIDENT=1 SLIME_DISABLE_ACTIVATION_RECOMPUTE=1

Judged
1.35×
Tier
N1
Self-best
1.25×
Rounds
9
Hours
10.0
API cost
$14.7
Input tokens · 98.4% cached
58 M
Output tokens
0.1 M
Requests
387
Candidate, three windows (s)
138.2 / 133.1 / 137.1
Rollout side
1.32×
Train side
1.35×
Rollout host-side fixes
partial: only sampler sync removed and top-k row copies merged; top-k JSON serialization untouched
Skip zero-coefficient reference forward
✗
Train-side main levers
partial: training resident, no recompute; pairing micro-batches measured 1.44×, blocked by direction cosine 0.931 < 0.955
RoundChangeMeasuredStepSide, cumulativeStatus
Rollout side
1sampler top-p check via host-side flag, per-step any().item() sync removedrollout-side speedup: — → 1.27×1.271.27×kept
2decode top-k per-request row D2H merged into one bulk copy of the underlying tensorrollout-side speedup: 1.27 → 1.32×1.041.32×kept
3CUDA graph input copy via _foreach_copy_ instead of Triton small_copy (no gain, reverted)rollout-side speedup: 1.32 → 1.3——reverted
4uniform top-k packed before D2H + one tolist on the whole tensor (no gain, reverted)rollout-side speedup: 1.32 → 1.31——reverted
Train side
6training side resident, no offload (rollout offload kept)train-side speedup: — → 1.15×1.151.15×kept
7full-layer activation recompute off (recompute_granularity=None)train-side speedup: 1.15 → 1.35×1.171.35×kept
8actor micro-batch pair packing at 8704 tokens/GPU (faster but judge INCORRECT, reverted)train-side speedup: 1.35 → 1.44×1.07—reverted
9token cap raised to 8704 for the no-grad log-prob forward onlytrain window (this round only started timing, no result in the log): — → ———reverted
Raw work log9 sections
0.0 hRound 1 — reference profile (started 2026-11-12 23:01 UTC)
- GPU allocation: 4 × Hopper, all idle before launch; /tmp has ample free space.
- Trees and switches are unchanged from the reference; all three repositories are clean on branch opt.
- Frozen reference: rollout windows 110.466 / 114.317 / 110.612 s (spread 1.19%); train windows 78.308 / 65.628 / 66.051 s (spread 4.28%); full-loop windows 181.049 / 179.582 / 182.965 s. Frozen full-loop fractions: rollout 0.617, train 0.3863.
- Profiling the unchanged submission, rollout then train. Detached driver PID 461090; log /tmp/r1/driver.log; outputs /tmp/r1/profile-rollout and /tmp/r1/profile-train.
- Rollout profile /tmp/r1/profile-rollout/kernel_report.md: 3,022.962 ms trace window, GPU busy only 1,009.521 ms (33.4%); 97,940 kernels. GEMMs are the largest kernel classes (19.1%, 9.6%, 9.2%), fused recurrent GDN decode is 9.0%, and TP all-reduces are 11.2%, but the dominant opportunity is host/launch scheduling rather than a new kernel.
- Rollout trace attribution: 200 decode steps spent 806.682 ms in aten::item / 805.335 ms in aten::_local_scalar_dense; the 200 long items (805.039 ms) have Sampler.forward as their Python parent. Sampler.forward totals 1,086.591 ms. The item is the return_top_p_log_probs.any().item() guard, evaluated even though every request has the flag false for this top-p=1 recipe.
- Training profile ran successfully after 2,378 s at 23:50 UTC and wrote very large traces (about 1.3 GB/rank) under /tmp/r1/profile-train/trace. Its post-run report generation was stopped to avoid CPU contention with the timed run. Reference log stage timings in steady state: ref forward about 10.7-11.2 s, actor train about 46 s, train wait about 9-11 s per roughly 66 s step. A background wait for the oversized profiler hit its 10-minute tool limit and was stopped; the profiler itself continued and completed.
- Change (sglang commit b42fcf2): added a CPU-derived need_return_top_p_log_probs flag, merged it like the existing token-id flag, and used it instead of a device any().item() in Sampler.forward. No sampling, log-probability, or model arithmetic changed; tier N0.
- Validation before commit: py_compile and git diff --check passed.
- Timing command was queued at 23:33 UTC (PID 501191, log /tmp/r1/time-driver.log) and held the GPU lock from 23:50 to 23:58 UTC. Result /tmp/r1/time/time.json: candidate windows 86.756 / 86.109 / 84.313 s; ratios 1.2733 / 1.3276 / 1.3119; side_speedup.rollout = 1.2733; speedup_vs_frozen_reference = 1.2733; projected step speedup = 1.1526. All four GPUs returned to 0 MiB.
- Full check attempt at 00:01 UTC (detached PID 526957, log /tmp/r1/check-driver.log, output /tmp/r1/check) did not produce a verdict. The candidate rollout completed successfully in 502 s, then the judge raised FileNotFoundError while opening /tmp/r1/check/rollout/score.log for token forcing. The timed-check path did not create the rollout parent before score() opened its log; no check.json exists. This is a judge-side directory failure, not a gate rejection. Retrying the unchanged command in a fresh output directory with only the expected rollout output subdirectory pre-created; the judge and gates remain unchanged.
- Full check retry /tmp/r1/check-retry/check.json completed at 00:37 UTC: PASS, N1, projected step speedup 1.1654x. Candidate rollout windows were 84.800 / 85.202 / 85.164 s; side speedup 1.2988x. Exact items, layout, token forcing, top-k, and train-inference mismatch all passed (mismatch ratio 0.9883). All four GPUs returned to 0 MiB. The retry was detached PID 541011 with log /tmp/r1/check-retry-driver.log.
1.6 hRound 2 — optimized rollout profile (started 2026-11-13 00:38 UTC)
- Starting point: sglang b42fcf2, full check /tmp/r1/check-retry/check.json PASS at N1 with projected step speedup 1.1654x; switches remain empty and all three repositories are clean.
- Re-profiling the current best rollout before selecting another hotspot. Detached PID 570165; log /tmp/r2-profile-driver.log; output /tmp/r2/profile-rollout.
- Training-stage evidence retained from round 1: steady-state actor training is about 46-48 s, reference forward about 11-12 s, and train wait about 9 s (including 4.3-4.7 s sleep and 1.3 s wake-up). The profiler's traced step is heavily instrumented and is not a timing result.
- Profile /tmp/r2/profile-rollout/kernel_report.md: 2,501.550 ms trace window, GPU busy 952.662 ms (38.1%); 96,156 kernels. The disabled-top-p aten::item operations are gone. Host attribution shows CUDA-graph buffer filling at 441 ms/200 decode steps, result D2H at 405 ms, and decode top-k formatting at 157 ms. The result path made 64 separate per-request value/index D2H copies per step from two contiguous top-k tensors.
- Change (uncommitted): coalesce each decode top-k value/index row list into one pinned D2H copy of its common contiguous base on the existing copy stream, then recreate equivalent host row views. A size guard preserves the old path for truncated/heterogeneous rows. Arithmetic, sampled tokens, and copy-stream ordering are unchanged; code tier N0.
- Validation: py_compile, git diff --check, and a GPU functional test passed; both coalesced uniform rows and fallback truncated rows matched their source tensors exactly.
- Timing /tmp/r2/time/time.json: candidate windows 81.785 / 86.470 / 82.150 s; ratios 1.3507 / 1.3220 / 1.3465; rollout speedup 1.3220x, projected step speedup 1.1769x. Mean rollout time fell from 85.726 s in round 1 to 83.468 s. Status ok, layout unchanged, all GPUs idle afterward.
- Full check /tmp/r2/check/check.json completed at 01:23 UTC: PASS, N1, projected step speedup 1.1711x. Rollout windows 84.303 / 86.038 / 83.767 s, side speedup 1.3103x; exact items, token forcing, top-k and mismatch passed (mismatch ratio 0.9883). GPUs returned to idle.
- Committed as sglang 5572a62 (Coalesce decode top-k host copies (N0)), with the /tmp/r2/time result in the commit message. Round ended 01:23 UTC; the full check is outside the round's 30-minute budget.
2.4 hRound 3 — graph-input and remaining rollout host overhead (started 2026-11-13 01:24 UTC)
- Starting point: sglang 5572a62, full check /tmp/r2/check/check.json PASS at N1 with projected step speedup 1.1711x; switches empty and repositories clean.
- Re-profiling the current best rollout before selecting the next hotspot. Detached PID 626178; log /tmp/r3-profile-driver.log; output /tmp/r3/profile-rollout.
- Profile /tmp/r3/profile-rollout/kernel_report.md: trace window 2,159.247 ms (down 342 ms from round 2), GPU busy 951.128 ms (44.0%); 96,180 kernels. Host attribution: CudaGraphBufferRegistry.fill_from 473 ms/200 steps and Triton try_small_copy 441 ms; coalesced result D2H helpers are now 102 ms. The next hotspot is graph-input copy launch processing, not model arithmetic.
- Change (uncommitted): when PyTorch _foreach_copy_ is available, use the registry's existing grouped foreach path instead of the per-step Triton small_copy Python launcher. The Triton path remains the fallback for older PyTorch. Copies and dtypes are unchanged; code tier N0.
- Validation: py_compile, git diff --check, and a GPU test of mixed shapes/dtypes (including int32-to-int64) matched ordinary copy_ exactly.
- Timing /tmp/r3/time/time.json: candidate windows 85.018 / 85.953 / 82.939 s; ratios 1.2993 / 1.3300 / 1.3337; rollout speedup 1.2993x, projected step 1.1657x. This is not a gain over /tmp/r2/time (1.3220x / 1.1769x), so the one-line foreach experiment was reverted. No correctness run was needed for the reverted experiment; the submission is back to clean, fully checked 5572a62. Round ended 01:43 UTC.
2.8 hRound 4 — packed decode top-k formatting (started 2026-11-13 01:45 UTC)
- Starting point: restored clean sglang 5572a62; latest full check /tmp/r2/check/check.json PASS N1, projected step 1.1711x. The round-3 foreach experiment was reverted.
- Profiling the restored best rollout before changing the remaining top-k formatting path. Detached PID 652771; log /tmp/r4-profile-driver.log; output /tmp/r4/profile-rollout.
- Profile /tmp/r4/profile-rollout/kernel_report.md: trace window 1,885.505 ms, GPU busy 997.626 ms (52.9%). Current-best host attribution: get_top_logprobs_raw 178 ms/200 steps, _async_d2h_rows 106 ms, _normalize_decode_outputs 67 ms. Uniform top-k was still split into 64 GPU row views before transfer and converted with 64 separate tolist() calls afterward.
- Change (uncommitted): OutputLogprobProcessor now keeps uniform decode top-k values/indices packed through the asynchronous D2H transfer; result normalization performs one tolist() per packed tensor. Heterogeneous k retains the previous row-wise/fallback representation. Both overlap and non-overlap result conversion handle packed tensors. Top-k arithmetic and output values are unchanged; code tier N0.
- Validation: py_compile, git diff --check, and GPU tests passed. Packed uniform and heterogeneous-fallback outputs matched the original get_top_logprobs(..., no_copy_to_cpu=False) Python lists exactly.
- Timing /tmp/r4/time/time.json: candidate windows 84.392 / 86.786 / 84.594 s; ratios 1.3090 / 1.3172 / 1.3076; rollout speedup 1.3076x, projected step 1.1698x. No gain over the current-best /tmp/r2/time (1.3220x / 1.1769x), and the mean was slower (85.257 vs 83.468 s). The four-file packed-formatting experiment was reverted. The submission is clean at fully checked 5572a62; round ended 02:05 UTC.
3.1 hRound 5 — training-side profile (started 2026-11-13 02:06 UTC)
- Starting point: clean sglang 5572a62; Megatron-LM and slime unchanged; switches empty. Latest full check /tmp/r2/check/check.json PASS N1, projected step 1.1711x.
- After two no-gain rollout host-formatting experiments, moving to the untouched training side. Profiling the current best training tree before selecting a hotspot. Detached PID 679264; log /tmp/r5-profile-driver.log; output /tmp/r5/profile-train. Round-1 training profiling took 2,378 s and produced about 1.3 GB/rank; allow this profile to finish rather than extrapolating from its instrumented step time.
- Profile completed successfully at 02:52 UTC (wall 2,425 s; /tmp/r5/profile-train/profile.json status ok). Reports are /tmp/r5/profile-train/kernel_report.md and /tmp/r5/rank{0,1,3}-report.md. On rank 2 the 97,260.662 ms instrumented trace window had 66,495.084 ms GPU busy (68.4%) and 68,271.326 ms summed kernel time. All-gather was 17,338.216 ms (25.4%, 23,809 calls) and reduce-scatter 5,886.406 ms (8.6%, 11,968 calls), followed by the large GEMM classes. These are not timing results.
- Collective durations are strongly skewed by rank waiting: all-gather totals were 18,060.684 / 14,536.043 / 17,338.216 / 10,820.273 ms on ranks 0/1/2/3, while total GPU-busy time was 68,600.631 / 64,828.815 / 66,495.084 / 57,601.797 ms. Rank 3 has both the least all-gather time and least GPU work. Do not interpret collective totals as pure wire cost or change reduction algorithms on this evidence.
- Rank-2 host attribution included CheckpointFunctionBackward 43,685.328 ms/2,048, CheckpointFunction 18,544.793 ms/2,048, _AllGather 8,049.910 ms/4,608, _AttnFuncWithCPAndKVP2P 6,317.442 ms/1,536, _VocabParallelLogProbEntropy 5,204.984 ms/954, and _maybe_all_reduce 5,840.681 ms/4,134. The latter calls are the required TP vocab-parallel softmax reductions, not removable metric collectives. HybridDeviceOptimizer.step was 4,627.374 ms for the traced update, so the CPU optimizer is not the leading target.
- Uninstrumented stage logs provide a larger scheduling/memory target: each steady step spends about 4-5 s in sleep and 1.3-1.4 s in wake_up, in addition to about 11 s reference forward and 40-47 s actor training. Full-loop logs show about 40.5 GB training GPU use before offload and 39.06 GB free after engine graph capture with training offloaded. SGLang sizes its KV pool from currently free memory, so keeping training resident reduces KV capacity rather than adding its full static allocation on top of training. The fixture needs only about 262K tokens per engine, far below the reference capacity of 3,023,676. Round 5 ended at 03:15 UTC with resident-training memory strategy selected for round 6.
4.3 hRound 6 — keep training resident between colocated steps (started 2026-11-13 03:15 UTC)
- Starting point remains clean sglang 5572a62; no training-side edits yet. One hotspot: repeated colocated training sleep/wake copies and process-group reconstruction.
- Planned switchable change: SLIME_KEEP_TRAINING_RESIDENT=1 disables only training offload while preserving rollout offload, synchronous ordering, model layout, and arithmetic. The switch is off by default. This file is outside either individual side, so a full check will judge both sides; initial screening uses judge time --side train, followed by a real full-loop memory smoke run before considering acceptance.
- Implementation and validation: added the environment override in slime/utils/arguments.py, enabled it in /submission/switches.json, and passed py_compile and git diff --check. Committed separately as slime 23473f1f after timing.
- Judge timing /tmp/r6/time/time.json: candidate train windows 56.712 / 57.049 / 56.419 s; ratios 1.3808 / 1.1504 / 1.1707 against frozen 78.308 / 65.628 / 66.051 s; train speedup 1.1504x, isolated-side projected step gain 1.0532x. Status ok, no layout problems, first-use 0/0/0 s. The timer started at 03:18 and finished at 03:32 UTC.
- Full-loop functional smoke /tmp/r6/full-smoke used the real bench/launch.sh in MODE=full for two forced-length RL steps with live engines, rollout offload, optimizer updates, and weight synchronization. It exited 0 at about 03:46 UTC with 64 samples in each rollout and no CUDA OOM. Engines retained 3,023,676-token capacity and both post-update rollouts succeeded; all GPUs and processes returned to idle. This was a functional/memory check, not a timing claim.
- Full check launched at 03:47 UTC, detached PID 757495, log /tmp/r6/check-driver.log, output /tmp/r6/check (with the expected rollout output parent pre-created). Both sides are judged because slime/utils/arguments.py lies outside the side-specific paths.
- Full check /tmp/r6/check/check.json completed at 04:31 UTC: PASS, N1, projected step speedup 1.2432x. Rollout windows 84.220 / 86.156 / 84.994 s, side speedup 1.3014x; train windows 57.269 / 56.668 / 56.270 s, side speedup 1.1581x. Rollout exact items, token forcing, top-k and mismatch passed (mismatch ratio 0.9890). Training was not bitwise repeatable (bitwise: false), but every N1 statistic and direction passed; update cosine 0.99219 versus minimum 0.95541, and train-inference mismatch ratio was 1.0. All GPUs returned to idle and all three repositories are clean. Round 6 ended at 04:32 UTC.
5.5 hRound 7 — activation recomputation in the resident-training step (started 2026-11-13 04:33 UTC)
- Starting point: full check /tmp/r6/check/check.json PASS N1, projected step speedup 1.2432x; current switches SLIME_KEEP_TRAINING_RESIDENT=1; all repositories clean.
- Re-profiling the exact current best training tree before selecting the next hotspot. Detached PID 823207; log /tmp/r7-profile-driver.log; output /tmp/r7/profile-train. The prior trace exposed CheckpointFunctionBackward 43,685.328 ms and CheckpointFunction 18,544.793 ms in its instrumented step; this run must confirm that recompute remains a leading target after the lifecycle change. The planned bottleneck class is redundant full-layer activation recompute, with any change kept behind a separate switch and tested for memory and judge equivalence.
- Profile /tmp/r7/profile-train/profile.json completed successfully at 05:06 UTC (run wall 1,730 s). Rank-3 trace window 80,670.928 ms, GPU busy 57,002.434 ms (70.7%), 547,229 kernels. All-gather was 12,217.647 ms (20.8%) but remains skewed by rank waiting; the leading GEMM classes followed. Host attribution in the new trace: CheckpointFunctionBackward 40,495.526 ms / 2,048 calls and CheckpointFunction 15,714.649 ms / 2,048 calls. Forward GDN kernels such as causal_conv1d_fwd_kernel had 6,144 calls versus 1,536 backward calls, confirming a full extra forward pass. This is a configuration/memory hotspot, not a request to alter model arithmetic. Verified uninstrumented steady steps in /tmp/r6/check/train_cand/train.log had about 89 GB free and 45 GB reserved after backward, leaving headroom to retain activations.
- Change (uncommitted): default-off SLIME_DISABLE_ACTIVATION_RECOMPUTE=1 sets Megatron's recompute_granularity to None during argument validation. Data, microbatching, model weights, loss, optimizer and update ordering are unchanged; the initial forward activations are retained instead of recomputed in backward. Expected code tier N0, subject to the judge's training record. The verified /submission/switches.json is unchanged during screening; /tmp/r7/switches-exp.json enables both this experiment and resident training. py_compile and git diff --check passed.
- Judge timing launched at 05:13 UTC, detached PID 852370, log /tmp/r7/time-driver.log, output /tmp/r7/time, using time --side train and the experimental switch file.
- Timing /tmp/r7/time/time.json completed at 05:26 UTC: candidate train windows 48.897 / 48.739 / 48.563 s; ratios 1.6015 / 1.3465 / 1.3601 against the frozen reference; cumulative train speedup 1.3465x, isolated-side projected step gain 1.1104x. Status ok, layout unchanged, first-use 0/0/0 s. For direct attribution, /tmp/r6/time with only resident training had 56.712 / 57.049 / 56.419 s, so this change removed about 8 s per steady train step; steady actor training fell from about 42.8 s to 34.6-34.9 s. Peak reserved memory rose from about 45 GB to 73 GB, with about 61 GB still free; no OOM or invalid-run signature occurred.
- Correctness check --quick launched with the experimental switches, detached PID 871217, log /tmp/r7/quick-driver.log, output /tmp/r7/quick (expected rollout parent pre-created). /submission/switches.json still contains only the fully checked resident-training switch while the verdict runs.
- Quick check /tmp/r7/quick/check.json completed at 05:57 UTC: PASS, N1. Rollout exact items, forcing, top-k and mismatch passed (mismatch ratio 1.0078). Training record was not bitwise identical, as expected from the reference's own non-repeatability; all N1 gates passed with margin. Update direction cosine 0.99470 versus minimum 0.95541, update norm within tolerance, loss mean difference -8.87e-7, and train-inference mismatch ratio 1.0. The experimental switches are now promoted to /submission/switches.json; all GPUs are idle.
- Committed separately as slime c1a1aeb6 (Retain activations instead of full-layer recompute (N0)), with /tmp/r7/time and /tmp/r7/quick evidence in the commit message. Full check launched at 05:59 UTC, detached PID 916848, log /tmp/r7/check-driver.log, output /tmp/r7/check.
- Full check /tmp/r7/check/check.json completed at 06:43 UTC: PASS, N1, projected step speedup 1.2510x. Train windows 49.424 / 48.659 / 48.460 s, side speedup 1.3487x. Rollout windows 92.428 / 85.690 / 85.121 s, side speedup 1.1952x; the slow first window, rather than a training regression, limited the combined projection. All rollout and training gates passed. Training update cosine was 0.99255 versus minimum 0.95541; train-inference mismatch ratio 1.0 and rollout mismatch ratio 0.9993. All GPUs returned to idle. Round 7 ended at 06:44 UTC.
7.7 hRound 8 — actor dynamic-microbatch packing (started 2026-11-13 06:44 UTC)
- Starting point: full check /tmp/r7/check/check.json PASS N1, projected step 1.2510x; slime c1a1aeb6; switches enable resident training and no activation recompute; repositories clean.
- Re-profiling the exact current best training tree before selecting the next hotspot. Detached PID 981284; log /tmp/r8-profile-driver.log; output /tmp/r8/profile-train. Candidate bottleneck class: small per-microbatch shapes, kernel launches and collectives. With CP2 the current actor packing cap is 8,192 full-sequence tokens, so each roughly 8.2K-token fixture sample occupies its own microbatch. A separate actor-only cap near 8,704 tokens/GPU would pack pairs while leaving log_probs_max_tokens_per_gpu=4096 and the reference forward unchanged. Global batch, sample order and one optimizer step remain fixed; any experiment uses a separate default-off switch and must pass memory and N1 gates.
- Profile /tmp/r8/profile-train/profile.json completed at 07:11 UTC (run wall 1,425 s). Rank-2 trace window 61,689.698 ms, GPU busy 42,413.832 ms (68.8%, hence 31.2% idle), 451,628 kernels. Under-filled classes accounted for 10,237.490 ms (23.7%, 126,866 launches) and short classes 2,975.892 ms (157,645 launches). All-gather was 5,676.773 ms / 18,177 calls (13.2%) and reduce-scatter 1,936.315 ms / 9,408 calls; these still include waiting but their per-microbatch launch count is a target. Removing recompute reduced forward GDN causal_conv1d_fwd_kernel calls from 6,144 to 4,608 and total GPU busy from 57.0 s in the prior trace to 42.4 s; these instrumented values are not timing results. Pairing actor microbatches should halve actor-side per-layer launches and collectives while increasing GEMM token dimensions.
- Change (uncommitted): default-off SLIME_ACTOR_MAX_TOKENS_PER_GPU=8704 overrides only args.max_tokens_per_gpu after log_probs_max_tokens_per_gpu has already defaulted to the recipe's 4096. Actor packing uses 17,408 full-sequence tokens per bin under CP2 and pairs ordinary fixture samples; reference-log-probability batching remains unchanged. Expected tier N1 because microbatch reduction order changes, while data membership/order, global batch, loss formula and one optimizer update remain fixed. py_compile and git diff --check passed. A direct schedule test on 64 lengths of 8,308 tokens verified 64 -> 32 microbatches, two samples per new bin, and unchanged flattened sample order/membership. /submission/switches.json remains at the verified two-switch configuration during screening; /tmp/r8/switches-exp.json enables all three.
- Judge timing launched at 07:14 UTC, detached PID 1007145, log /tmp/r8/time-driver.log, output /tmp/r8/time, using time --side train and the experimental switch file.
- Timing /tmp/r8/time/time.json completed at 07:27 UTC: candidate train windows 45.541 / 45.423 / 45.229 s; ratios 1.7195 / 1.4448 / 1.4604; cumulative train speedup 1.4448x, isolated-side projected step gain 1.1350x. Status ok, layout unchanged, first-use 0/0/0 s. Compared with /tmp/r7/time, pairing removed another 3.3 s per steady step; actor train fell to 31.85-32.25 s and ref log-probs remained 8.7-8.9 s. Steady reserved memory was 106.8-109.5 GB with about 25-27 GB free. The first cold step reached 134.06 GB reserved and nearly exhausted GPU memory but completed successfully; subsequent full-loop/cold-record validation is required. No OOM or invalid-run signature occurred.
- Correctness check --quick launched with the experimental switches, detached PID 1025928, log /tmp/r8/quick-driver.log, output /tmp/r8/quick (expected rollout parent pre-created). /submission/switches.json remains the fully checked two-switch configuration while the verdict runs.
- Quick check /tmp/r8/quick/check.json completed at 08:00 UTC: INCORRECT at the training statistical gate. Rollout passed (mismatch ratio 0.9974), but paired actor microbatches produced update cosine 0.93118 < 0.95541 and update norm difference -0.00105466, exceeding the 0.00065901 paired-noise tolerance. Loss mean, gradient norm and second-moment norm were within their gates, but that does not override the failed direction/update-norm checks. No bitwise claim was made. Per the gate protocol, the experiment is rejected rather than tuned around the gate. The uncommitted actor-cap override was fully reverted; /submission/switches.json never enabled it, all repositories are clean at the fully checked round-7 candidate, and all GPUs are idle. Round 8 ended at 08:02 UTC with the 1.4448x training timing recorded only as a rejected experiment, not an accepted speedup.
9.0 hRound 9 — no-gradient log-probability microbatch packing (started 2026-11-13 08:02 UTC)
- Starting point: round-8 actor pairing was rejected and fully reverted. Restored candidate is exactly full-check /tmp/r7/check/check.json PASS N1, projected step 1.2510x, with resident training and no activation recompute; all repositories clean.
- Re-profiling the restored current best before another change. Detached PID 1071493; log /tmp/r9-profile-driver.log; output /tmp/r9/profile-train. Proposed independent hotspot: the no-gradient log-probability pass still uses log_probs_max_tokens_per_gpu=4096 and about 8.7-9.5 s per steady step. Only this separate cap may be changed; actor forward/backward packing stays at the accepted 4096 setting after the round-8 statistical failure. The same global samples, order and log-probability computation must remain, with judge N1 validation.
- Profile /tmp/r9/profile-train/profile.json completed at 08:29 UTC (run wall 1,442 s). Rank-2 trace window 62,142.301 ms, GPU busy 44,876.208 ms (72.2%), 451,628 kernels. Under-filled classes were 12,733.922 ms (27.9%, 126,866 launches); all-gather was 7,720.848 ms / 18,177 calls and reduce-scatter 3,000.115 ms / 9,408 calls, both including rank waiting. The no-gradient forward still accounts for 2,048 of the 4,608 causal_conv1d_fwd_kernel calls (actor initial forward and reference forward); reducing its launch count is the selected hotspot.
- Fast-path inspection found that log_probs_max_tokens_per_gpu is defaulted in arguments.py but otherwise unused; merely overriding it would have no effect. forward_only restores original sample positions from the iterator's flattened micro_batch_indices, so a separate iterator can merge adjacent bins and preserve output order. Implementation merges only within original optimizer-step boundaries, uses total sequence lengths times the CP-aware cap, keeps the actor iterator and num_microbatches unchanged, and activates only when the separate log-probability cap differs from the actor cap.
- Change (uncommitted): default-off SLIME_LOG_PROBS_MAX_TOKENS_PER_GPU=8704 sets only the separate log-probability cap. get_log_prob_data_iterator in backends/megatron_utils/data.py repacks adjacent no-gradient bins within each original optimizer-step boundary, and compute_log_prob uses it only when dynamic batching is active and the separate cap differs from the actor cap. Actor forward/backward continues with the original iterator and counts. Expected tier N1 because no-gradient forward kernel/reduction order can change; samples, output positions and actor optimization remain unchanged. py_compile and git diff --check passed. Direct tests verified pair packing, varied lengths, cap handling, unchanged input schedule, flattened output order, and no merging across step boundaries. /submission/switches.json remains at the verified two-switch configuration; /tmp/r9/switches-exp.json enables the experiment.
- Judge timing launched at about 08:33 UTC, detached PID 1097715, log /tmp/r9/time-driver.log, output /tmp/r9/time, using time --side train and the experimental switch file.
↑ Slime GRPO