Kimi k3.1-pro 1.59×

Harness: Claude Code · switches: OPD_SKIP_ZERO_KL_REF=1 OPD_DISABLE_FULL_RECOMPUTE=1 OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 16384 OPD_RESIDENT_ACTOR=1 OPD_FUSE_OPD_ACTOR_LOGPROBS=1 OPD_CPU_OPTIMIZER_THREADS=24

Judged
1.59×
Tier
N1
Self-best
1.57×
Rounds
23
Hours
10.0
API cost
$22.4
Input tokens · 98.6% cached
89 M
Output tokens
0.2 M
Requests
627
Candidate, three windows (s)
96.9 / 96.8 / 96.9
Rollout side
1.28×
Teacher side
1.06×
Train side
2.11×
Per-step sampler sync
✓ only working by round 23 (rollout 1.28×, its own estimate of overall up to 1.56×), changed only in a separate code tree, not merged into the commit; score recorded from the version merged on its behalf
Train-side main levers
✓ skip reference forward, no full recompute, training resident, fused actor logprob, CPU optimizer 24 threads (train side 2.11×)
Teacher-side changes
chunking only: teacher chunk 16384 (teacher side 1.06×)
Raw work log27 sections
0.0 hRound 1 — reference profile (start 2026-11-12 21:44 UTC)
- Tree: unchanged reference on branch opt; switches: {"environment": {}}.
- Frozen reference: full loop 156.479 s/step; rollout 77.473 s (49.51% of step), train 74.928 s (47.88%), teacher critical-path tail 16.501 s (10.55%). Full-loop reference spread 3.47%; final acceptance requires >1.0694 speedup.
- GPUs checked idle (4 × Hopper, 0 MiB used); /tmp has ample space.
- Plan: profile all three unchanged sides before editing, classify hotspots, then choose one dominant hotspot. Judge profile commands write to /tmp/r1/profile-{rollout,teacher,train}.

- Rollout profile completed (/tmp/r1/profile-rollout, status ok). In the 200-decode-step trace, GPU kernels cover 41.1% of the instrumented window. Decode GPU time: GEMMs about 60%, gated-delta recurrence 10.0%, FlashInfer all-reduce 10.7%; full decode CUDA graphs and overlap scheduling are active. This is instrumented localization, not a timing result.
- Teacher profile completed (/tmp/r1/profile-teacher, status ok). GPU coverage is 92.2%; TP4 NCCL all-reduce is 22.3% (1923.9/8639.5 ms), the largest non-GEMM class. FlashInfer all-reduce is initialized, but apply_flashinfer_allreduce_fusion only accepts <=2048 tokens and the stock 4096-token chunks therefore take NCCL.
- Train profile reached and traced step 5, but four ~500 MB gzip traces were still being exported after 50 minutes. I stopped the job and its remaining workers; no train kernel report was produced and the partial traces are not used as evidence. Future rounds will use stage logs/one targeted profile rather than repeat this expensive observation.
- End: 2026-11-12 22:33 UTC. No source or switch change was made during profiling.
0.9 hRound 2 — teacher prefill all-reduce fallback (start 2026-11-12 22:34 UTC)
- Hotspot: teacher TP4 communication, 22.3% of teacher GPU time (configuration/backend problem: existing fused path missed at 4096-token chunks).
- Change: OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 2048, enabling the already initialized FlashInfer all-reduce path. Same model, dtype, scoring requests and token order; expected numerics tier N1 because hybrid prefill chunk boundaries and collective arithmetic change.
- Validation plan: judge time --side teacher in /tmp/r2/time, then check --quick --side teacher in /tmp/r2/quick.

- Result: /tmp/r2/time/time.json; teacher windows [20.156, 20.168, 19.782] s versus reference [18.532, 18.478, 18.129] s; side speedup 0.9162x, projected step 0.9904x. Rejected and reverted; no correctness check run because the change has no benefit.
- End: 2026-11-12 22:38 UTC. No repository diff to commit for this configuration-only rejected round.
0.9 hRound 3 — fused all-reduce at stock 4096-token chunks (start 2026-11-12 22:39 UTC)
- Hotspot: same teacher TP4 communication class, isolating collective selection from the extra-chunk regression in round 2.
- Change: FUSE_ALLREDUCE_MAX_BATCH_SIZE is configurable with OPD_FUSE_ALLREDUCE_MAX_BATCH_SIZE (default remains 2048); candidate sets 4096. The workspace initialization imports the same constant, so eligibility and allocation stay consistent.
- Expected tier N1 (different collective kernel/reduction order); teacher timing in /tmp/r3/time.

- Result: /tmp/r3/time/time.json; teacher windows [19.036, 18.978, 18.568] s; side speedup 0.9735x, projected step 0.9971x. Rejected. Committed experiment f1abfab and restoration 24f181b; no correctness run because there is no benefit.
- End: 2026-11-12 22:45 UTC.
1.0 hRound 4 — NCCL protocol for large teacher collectives (start 2026-11-12 22:46 UTC)
- Hotspot: the stock teacher's 42 MB-per-rank TP4 all-reduces run as RING_LL; profile shows 22.3% of GPU time. Test the existing NCCL Simple protocol, which can have higher large-message bandwidth than LL.
- Change: configuration-only NCCL_PROTO=Simple (no source diff), expected N1 wherever collective reduction order changes. Screen teacher first in /tmp/r4/time-simple; retain only with a clear gain.

- Result: /tmp/r4/time-simple/time.json; teacher windows [18.559, 18.450, 18.070] s; side speedup 0.9985x, projected step 0.9998x. No resolved gain; reverted without a correctness run. End 2026-11-12 22:50 UTC.
1.2 hRound 5 — remove zero-coefficient reference-KL forward (start 2026-11-12 22:54 UTC)
- Train profile/stage evidence: normal steps 1, 3 and 4 took 68.5, 69.2 and 72.1 s in train; ref_log_probs took 11.5, 10.9 and 11.8 s, actor log_probs 9.9–11.0 s, and actor_train 46.7–48.9 s. Inter-step train_wait was 6.5–7.6 s. Step 2 was straggled (122.0 s); profiler step 5 is not used for timing.
- Hotspot: an approximately 11 s reference-model forward is computed solely for the reference-KL diagnostic even though kl_coef=0 and kl_loss_coef=0. OPD advantages instead use the separately supplied teacher log-probs.
- Change: off-by-default OPD_SKIP_ZERO_KL_REF=1 skips the ref forward and reference-KL evaluation only when both coefficients are zero; retains a finite zero kl_loss metric and the existing loss + 0 * kl_loss operation. Actor/teacher forwards, data order, advantages and optimizer are unchanged. Intended tier N0 for training state.
- Validation: judge train timing /tmp/r5/time, then train quick correctness /tmp/r5/quick.
- Timing result: /tmp/r5/time/time.json; candidate train windows [65.580, 64.514, 64.825] s; side speedup 1.1395x, projected step 1.0623x, first-use 0 s in every window. Ref forward absent from the timed log.
- Quick job: PID 438215, log /tmp/r5-quick-driver.log; output /tmp/r5/quick.

- Quick result: /tmp/r5/quick/check.json PASS N1. Update norm, second-moment norm, mean loss, first gradient norm and update cosine all pass; mismatch ratio 1.0. Bitwise is false (6338 tensors), consistent with the reference's own non-repeatability. End 2026-11-12 23:23 UTC.
- Commit: a2b3bfab (Skip zero-coefficient reference KL forward (N1); train time 1.1395x).
1.7 hRound 6 — remove full activation recompute (start 2026-11-12 23:25 UTC)
- Hotspot: actor_train remains 45–48 s after round 5 and includes a full extra forward for activation recomputation. Rank logs show 102.98 GiB free before model offload and 106.27 GiB free after wake-up; train-only peak used memory is about 37 GiB of 139.81 GiB. This is a memory-strategy change, not a model/objective change.
- Change: OPD_DISABLE_FULL_RECOMPUTE=1 sets recompute granularity/method/layers to None only when the recipe requested full recompute. Round 5 remains enabled. Intended N0/N1 depending on the record; watch OOM and full-loop colocation headroom separately.
- Timing result: /tmp/r6/time/time.json; candidate train windows [55.696, 56.026, 55.090] s; window ratios [1.3602, 1.3415, 1.3408]; side speedup 1.3408x, projected full-step 1.1386x, first-use 0 s in every window. No OOM in train-only mode. Full-loop colocated memory remains to be validated.
- Correctness result: /tmp/r6/quick/check.json PASS N1. All statistical gates pass (update cosine 0.993053 >= 0.956603; grad-norm delta -0.000215 within 0.001651) and mismatch ratio is 1.0. Bitwise is false (6334 tensors). End 2026-11-12 23:57 UTC.
- Commit: ea0a0c37 (Disable full activation recompute (N1); train time 1.3408x).
- Full-check note: train-only timed check would apply its frozen train spread (21.36%) and stop at benefit unless train side exceeds 1.4272x; final full-loop threshold is instead 1.0694x. The available CLI has no --final; all-side correctness can be requested separately. Keep the quick-passing state and increase margin before a long timed check.
2.3 hRound 7 — reuse the training forward for OPD actor log-probs (start 2026-11-13 00:05 UTC)
- Current-best profile/stage evidence (/tmp/r6/time/train_tree/train.log): actor log_probs 11.6 s, actor_train 37.6 s, train_wait 6.6 s. The separate actor forward is the next redundant-forward hotspot after removing the ref forward and activation recompute.
- Fixed-recipe facts: GRPO, no advantage whitening, one optimizer step after all microbatches, zero dropout, teacher log-probs already CP-sliced in every training batch. Thus the training forward can provide old/current actor log-probs and the per-token OPD reverse-KL advantage in the same pass.
- Experiment: off-by-default OPD_FUSE_OPD_ACTOR_LOGPROBS=1, restricted to one-step GRPO/OPD without whitening, routing replay, custom advantages or critic. Skip the separate actor forward; compute detached reverse-KL advantages from the training-forward log-probs, capture them by rollout position for metrics/debug data, and preserve sample order and the single optimizer update. Expected N1 because grad-enabled and no-grad forward kernels/reductions can differ.
- First timing result: /tmp/r7/time/time.json; windows [59.054, 47.162, 45.807] s, ratios [1.2828, 1.5937, 1.6125], side 1.2828x, projected 1.1180x. Steps 5–7 have actor_train 37.8–39.7 s and no separate actor forward, but step 4 hit a ~12 s straggler (actor_train 51.7 s). This is below round 6 by the minimum-window rule, so not accepted yet.
- Repeat: same tree/switches, judge timing /tmp/r7/time-repeat, PID 515394, driver /tmp/r7-time-repeat-driver.log, to distinguish random train straggling from a regression. Round 6 itself had warmup stragglers and frozen train spread is 21.36%.

- Repeat result: /tmp/r7/time-repeat/time.json; windows [48.326, 59.816, 45.708] s, ratios [1.5676, 1.2565, 1.6160], side 1.2565x, projected 1.1083x. A similar ~12 s stall moved to another window. Both runs lose to round 6's 1.3408x under the required minimum-window rule; no quick gate run and no commit.
- Conclusion: rejected for now and fully restored to ea0a0c37 plus the two verified switches. The separate 11 s forward may hide asynchronous optimizer/offload work; steady fused steps are faster, but worst-window stalls are unresolved. End 2026-11-13 00:33 UTC.
2.8 hRound 8 — train sleep/wake gap investigation (start 2026-11-13 00:34 UTC)
- Current-best stage evidence: /tmp/r6/time/train_tree/train.log timed steps have train_wait 6.63–7.03 s, including model sleep/offload and wake/reload. This is the next non-GEMM stage hotspot after the rejected forward-reuse experiment.
- Change: OPD_KEEP_HOST_CACHE=1 avoids torch._C._host_emptyCache() in every actor sleep while retaining GPU cache clearing, process-group destruction/reload, and torch_memory_saver.pause()/resume(). The CPU-offloaded optimizer can reuse pinned host buffers instead of recreating them. Memory/cache strategy only; intended N0. Host has about 2 TB RAM and over 1 TB available in the current runs.
- Validation: syntax/diff check passed. Judge train timing /tmp/r8/time running as PID 535410, driver /tmp/r8-time-driver.log; quick gate if the memory-only change unexpectedly affects numerics.

- Result: /tmp/r8/time/time.json; windows [56.216, 58.563, 55.230] s, ratios [1.3476, 1.2834, 1.3374], side 1.2834x, projected 1.1182x. Mean 56.670 s is also worse than round 6's 55.604 s. Rejected and restored; no quick run and no commit. End 2026-11-13 00:50 UTC.
- Full-loop memory evidence: reference bootstrap full logs show about 50 GiB/GPU of teacher+rollout residency. Round 6 train-only observed peak was about 69 GiB, implying about 119 GiB colocated and roughly 20 GiB headroom on 139.81 GiB GPUs. This supports keeping no-recompute, but is an estimate rather than a final-mode memory test.
3.2 hRound 9 — student GDN/mamba kernel backend (start 2026-11-13 00:58 UTC)
- Profile: unchanged rollout profile /tmp/r1/profile-rollout/kernel_report.md remains applicable because no accepted rollout change exists. Gated-delta recurrence is about 10.0% of decode GPU time; full decode graphs, overlap scheduling, FA3 and extra_buffer are already active. Cookbook docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx confirms extra_buffer as the throughput strategy and FlashInfer as an alternative NVIDIA GDN backend; flashinfer.mamba imports successfully.
- Change: off-by-default OPD_STUDENT_MAMBA_BACKEND=flashinfer, applied only to student engine construction in ray/serving.py. No teacher argument, layout, dtype, sampling or length change. Expected rollout N1 due to kernel/reduction differences.
- First attempt: /tmp/r9/time/time.json ERROR before rollout; engine startup succeeded, but attach_training rejected the changed retained configuration field. It also showed that mamba_backend=flashinfer alone selects FlashInfer prefill but leaves decode on Triton.
- Correction: apply the override at both construction and attachment, and explicitly set linear_attn_decode_backend=flashinfer; the source's gdn_flashinfer.py supports SM90 decode with FP32 state. Retry /tmp/r9/time-decode.
- Result: /tmp/r9/time-decode/time.json; candidate rollout windows [87.692, 88.002, 88.192] s versus frozen reference [77.381, 77.185, 77.853] s; window ratios [0.8824, 0.8771, 0.8828], side speedup 0.8771x, projected step 0.9351x. Logs confirmed decode=FlashInferGDNKernel on both student engines, but the path reported packed_decode=False and was substantially slower than the stock packed Triton decode. Rejected; ray/serving.py restored and the extra switch removed. No correctness run because there is no benefit, and no experiment commit. End 2026-11-13 01:16 UTC.
3.6 hRound 10 — student decode torch.compile (start 2026-11-13 01:18 UTC)
- Restored round 9 before starting. Current-best rollout profile remains /tmp/r1/profile-rollout: GPU coverage 41.1%, GEMMs about 60%, FlashInfer all-reduce 10.7%, and full decode CUDA graphs already active. Single-batch overlap was inspected but rejected without timing because its implementation is specific to DeepSeek/GLM MoE models, not this dense Qwen3.5 student.
- Change: off-by-default OPD_STUDENT_TORCH_COMPILE=1 enables sglang's existing compiled decode-graph path, with compile maximum batch 32 matching each engine's round-robin share of the 64-request rollout. It is applied before both serving construction and retained-trainer attachment. No layout, dtype, sampling, teacher or length change. Expected rollout N1 from compiler kernel/reduction selection.
- Validation: syntax and diff checks passed. Judge rollout timing /tmp/r10/time; quick rollout/teacher correctness only if timing improves. Cold compilation occurs during startup/graph capture, before timed windows.
- First attempt /tmp/r10/time (PID 584476, driver /tmp/r10-time-driver.log) never reached rollout: max-autotune compilation during decode graph capture grew GPU memory until repeated 8–15 GB allocation failures and OOM retries. Stopped and cleaned up after about 11 minutes; no timing JSON or correctness run.
- Retry /tmp/r10/time-default (PID 605072, driver /tmp/r10-time-default-driver.log) with SGLANG_TORCH_COMPILE_MODE=default hit the same capture-time memory growth and repeated OOM retries. This rules out max-autotune GEMM selection alone as the cause. Rejected, all workers terminated, GPUs verified idle, source and switches restored. No result JSON, quick gate or commit. End 2026-11-13 01:43 UTC.
4.1 hRound 11 — larger dynamic training microbatches (start 2026-11-13 01:47 UTC)
- Current-best stage profile /tmp/r6/time/train_tree/train.log: actor log_probs about 11.6 s and actor_train about 37.6 s at max_tokens_per_gpu=4096; no full activation recompute is enabled. Dynamic packing uses max_tokens_per_gpu * CP per bin, so many short samples can share forward/backward launches and larger GEMMs.
- Memory constraint: train mode does not start the teacher (bench/launch.sh:115-117); full mode's teacher is fixed at mem-fraction 0.25. Previous train-only no-recompute peak was about 69 GiB. A moderate 6144 cap (1.5x, not 8192) is the first safe screening point; actual colocated memory still needs separate validation before relying on a gain.
- Change: off-by-default OPD_TRAIN_MAX_TOKENS_PER_GPU=6144, read only by the existing dynamic schedule builder. Same samples, global batch, optimizer step, sample order within each packed bin and model precision; batching/reduction order changes, expected N1. This file is outside all exclusive side lists, so a correctness check will judge all sides.
- Validation: syntax and diff checks passed. Judge train timing /tmp/r11/time, inspect microbatch counts/memory and run all-side quick correctness only if there is a benefit.
- Result: /tmp/r11/time/time.json; because the file was outside all exclusive lists, all sides ran. Rollout 0.9907x ([76.985, 77.908, 77.542] s) and teacher 0.9986x are unchanged-side noise. Train windows [55.946, 56.024, 56.959] s, side 1.2968x, all-side projected 1.1172x. Timed steps still had about 62–63 microbatches and actor_train about 37.3 s; 6144 cannot pack two normal 8K responses. It is below round 6's 1.3408x, so rejected without quick correctness or a commit; source restored. End 2026-11-13 02:13 UTC.
4.5 hRound 12 — two-response training microbatches (start 2026-11-13 02:14 UTC)
- Profile: round 11 shows that 6144 leaves about 62 one-sample microbatches, actor forward about 11 s and actor_train about 37 s. Fixture responses are mostly 8192 tokens; a cap of 9216 per GPU (18432 across CP2) is required to pack two complete samples.
- Change: same off-by-default OPD_TRAIN_MAX_TOKENS_PER_GPU, now 9216 and applied in the train-side Megatron argument hook before the driver builds its dynamic schedule. This confines judged timing to train, unlike round 11. Expected N1 from batching and reduction order.
- Memory warning: train mode has no teacher, while full-loop reference logs retain about 50 GiB/GPU after actor offload. Two-response microbatches may not be promotable even if train-only timing wins; require memory evidence and serving-reservation headroom before retention.
- Validation: syntax/diff checks passed; judge train timing /tmp/r12/time. Run quick correctness only if timing improves, and do not treat train-only success as full-loop memory validation.
- Result: /tmp/r12/time/time.json; train windows [53.038, 53.316, 54.648] s, side 1.3517x, projected 1.1423x, 32 microbatches. This is only slightly above round 6, while observed train-only GPU usage reached about 107.5 GiB; adding the reference full-loop residency of about 50 GiB implies 157.5 GiB and OOM. Steady actor_train remained about 37.1 s. Rejected on full-loop memory despite the small train-only gain; no quick run or commit, source restored. End 2026-11-13 02:31 UTC.
4.8 hRound 13 — larger vocabulary log-prob chunks (start 2026-11-13 02:32 UTC)
- Profile: the current 4096-token sample microbatches run actor log-probs and training loss through _ChunkedLogProbEntropy with chunk size 1024. Each typical CP-local sample therefore executes four sequential forward reduction chunks and four sequential backward recompute chunks, each with vocab-parallel collectives. Actor log_probs is about 11 s and actor_train about 37 s.
- Change: off-by-default OPD_LOG_PROBS_CHUNK_SIZE=4096, set on the Megatron actor before model initialization. This preserves the full logits, token targets and per-token vocabulary reduction but uses one chunk for typical samples. Unlike round 12, model microbatching and retained layer activations are unchanged; only the temporary reduction/autograd workspace grows. Expected N1 from kernel shape/reduction scheduling.
- Validation: syntax/diff checks passed. Judge train timing /tmp/r13/time, watch memory and run train quick correctness if timing improves.
- Result: /tmp/r13/time/time.json; windows [56.619, 55.729, 54.459] s, side 1.3380x, projected 1.1376x, mean 55.602 s (effectively identical to round 6's 55.604 s). Actor log_probs remained about 11 s and steady actor_train about 37.5 s. Rejected under minimum-window scoring; no quick run or commit, source restored. End 2026-11-13 02:46 UTC.
5.1 hRound 14 — remove zero-coefficient entropy diagnostic (start 2026-11-13 02:47 UTC)
- Profile/code evidence: _VocabParallelLogProbEntropy always computes log-prob softmax, but with_entropy=True additionally performs a full [T,V] einsum and TP all-reduce for every forward chunk. The recipe has entropy_coef=0, and with_entropy_grad is already false, so this work is diagnostic-only and cannot affect gradients.
- Change: off-by-default OPD_SKIP_ZERO_ENTROPY=1, restricted to entropy_coef == 0. Skip only the extra full-vocabulary entropy reduction; return finite zero per-token entropy tensors so metric shapes and the existing loss = pg_loss - 0 * entropy_loss operation remain. Log-probs, advantages, policy loss and all model/optimizer computation are unchanged. Intended training N0; quick gate will verify.
- Validation: syntax/diff checks passed. Judge train timing /tmp/r14/time, then train quick correctness if it wins.
- Result: /tmp/r14/time/time.json; windows [56.796, 55.551, 54.961] s, ratios [1.3338, 1.3530, 1.3440], side 1.3338x, projected 1.1361x, mean 55.770 s; first-use 0 s. Some steady actor-train timers improved, but the minimum window is below round 6's 1.3408x. Rejected and source restored; no quick gate or commit. The inert switch was removed from /submission/switches.json. End 2026-11-13 03:03 UTC.
5.5 hRound 15 — manual cyclic garbage collection (start 2026-11-13 03:12 UTC)
- Profile: round 7's movable ~12 s actor-train stalls coincided with host-used memory dropping by about 300 GiB within the stalled step, implicating automatic cyclic GC rather than a model kernel. The current actor already calls clear_memory(), which explicitly runs gc.collect(), at initialization, sleep and wake (actor.py:167,190,213; memory_utils.py:43-48). Automatic collection inside forward/backward is therefore redundant and can straggle one rank.
- Change: off-by-default OPD_MANUAL_GC=1 disables automatic cyclic GC in each Megatron actor process, retaining the existing explicit full collections at sleep/wake. No tensor computation, data, optimizer or synchronization order changes; intended N0. This round tests GC scheduling alone on the verified round 6 path, not the rejected actor-forward reuse.
- Result: /tmp/r15/time/time.json; windows [57.931, 58.312, 58.732] s, ratios [1.3077, 1.2889, 1.2577], side 1.2577x, projected 1.1088x, mean 58.325 s; first-use 0 s. Host memory stayed flat near 647 GiB instead of sawtoothing, but explicit collections increased sleep/train_wait by roughly 1–2 s and steady actor compute did not improve. Rejected as a standalone change; no quick gate or commit. It remains only as a possible enabling mitigation for round 7's larger forward-reuse gain, to be tested explicitly rather than presented as an accepted result. End 2026-11-13 03:27 UTC.
5.8 hRound 16 — actor-forward reuse with coordinated GC (start 2026-11-13 03:30 UTC)
- Profile: round 7 removed the separate ~11 s actor forward and reduced steady actor-train time to about 38–40 s, but automatic-GC-like movable ~12 s stalls made its worst windows lose. Round 15 confirms manual GC eliminates the memory-release sawtooth, though its standalone cost is not acceptable. Test the large forward-reuse hotspot with that stall mitigation, not GC as a new independent optimization.
- Change: reapply the tightly guarded OPD_FUSE_OPD_ACTOR_LOGPROBS=1 experiment on top of OPD_MANUAL_GC=1. Compute detached OPD reverse-KL advantages from each training-forward log-prob, preserve sample order and one optimizer update, and capture log-probs for metrics/debug output. The guards exclude whitening, rollout-logprob use, TIS, routing replay, critic, OPSM, custom advantages, pipeline parallelism and dropout. Expected N1 because old log-probs now come from the grad-enabled forward and PPO's old/current values are exactly shared.
- First attempt /tmp/r16/time/time.json ERROR after step 0: capture covered zero samples because the restoration omitted round 7's model.py change that adds partition to the training keys when fusion is active. Restored that enabling condition; this was an incomplete experiment restoration, not a numerics/performance failure.
- Retry: judge train timing /tmp/r16/time-fixed completed as PID 762009, driver /tmp/r16-time-fixed-driver.log, same tree plus the partition-key fix.
- Result: /tmp/r16/time-fixed/time.json; windows [46.950, 47.837, 62.291] s, ratios [1.6136, 1.5712, 1.1858], side 1.1858x, projected 1.0811x, first-use 0 s. The separate actor forward was absent and steady actor_train was 38.6–41.4 s, but the final window's wake_up took 16.84 s (normally 1.4–1.8 s), producing train_wait 23.44 s. Manual GC did not prevent host-memory releases or this sleep/wake straggler. Rejected; no quick gate or commit. Restored actor.py, loss.py, model.py and the two verified switches. End 2026-11-13 03:55 UTC.
6.2 hRound 17 — larger teacher prefill chunks (start 2026-11-13 03:56 UTC)
- Profile: unchanged teacher profile /tmp/r1/profile-teacher has 92.2% GPU coverage and TP4 NCCL all-reduce at 22.3%; rounds 2–4 show smaller chunks, fused all-reduce at 4096 and NCCL Simple do not help. Every fixture response is at most 8192 tokens, while the stock 4096 chunked prefill splits a request's hybrid linear-attention state across at least two sequential chunks. Larger chunks are the remaining prefill configuration lever and are explicitly within the teacher's accepted re-batching noise controls.
- Experiment plan: screen OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 8192 using the unchanged reference tree so only the teacher side runs; if useful, consider 16384 to cover prompt plus 8192 response in one chunk. Same model, requests, BF16 and TP4; expected N1 from changed chunk boundaries and kernel/reduction shapes. This does not alter the verified training tree or its switches.
- 8192 result: /tmp/r17/time-8192/time.json; teacher windows [17.669, 18.043, 17.487] s versus [18.532, 18.478, 18.129] s, ratios [1.0488, 1.0241, 1.0367], side 1.0241x, projected step 1.0025x. No OOM; server confirmed chunk 8192, TP4, BF16 and 56-request Mamba-state cap.
- 16384 result: /tmp/r17/time-16384/time.json; windows [17.296, 17.335, 17.055] s, ratios [1.0715, 1.0659, 1.0630], side 1.0630x, projected step 1.0063x. No OOM; available GPU memory 98.38 GiB in teacher-only mode. This is a useful teacher gain, pending correctness and colocated-memory caution.
- Cache-320 result: /tmp/r17/time-16384-cache320/time.json; windows [17.312, 17.404, 17.056] s, side 1.0617x, no improvement over 16384 with the stock cache (1.0630x). The server confirmed cap 64 and lower token capacity 641126, so reject the extra cache argument and retain only chunk 16384 for correctness.
- Correctness: /tmp/r17/quick-16384/check.json PASS N1 (teacher only). Mean |d| ratio 1.0128; tail events 11 at rate 5.338e-6 versus reference 4.246e-6, binomial p=0.2648; bias t=0.0249, no failures. Promoted OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 16384 in /submission/switches.json. Configuration-only change, no repository commit. End 2026-11-13 04:16 UTC.
6.2 hCurrent summary (updated after round 17)
- Current switch file: {"environment": {"OPD_SKIP_ZERO_KL_REF": "1", "OPD_DISABLE_FULL_RECOMPUTE": "1", "OPD_TEACHER_SERVER_ARGS": "--chunked-prefill-size 16384"}}; all three repositories are clean at the verified source state.
- Latest candidate correctness: /tmp/r6/quick/check.json PASS N1 (train), /tmp/r17/quick-16384/check.json PASS N1 (teacher). No candidate full-loop/final run is available through the CLI.
- Verified gains: zero-KL ref-forward removal (a2b3bfab) plus no full activation recompute (ea0a0c37) = train 1.3408x and projected step 1.1386x (/tmp/r6/time/time.json). Teacher chunk 16384 = teacher 1.0630x, projected incremental step 1.0063x (/tmp/r17/time-16384/time.json); combined projection is about 1.145x before final full-loop measurement. Estimated colocated training peak is about 119/139.81 GiB per GPU, leaving about 20 GiB headroom; larger teacher chunks passed teacher-only memory but final colocation is not directly measurable with the available CLI.
- Rejected after timing: actor-forward reuse rounds 7/16 had large steady gains but movable actor-train/wake stalls; worst side results 1.2828x, 1.2565x and 1.1858x with manual GC. Manual GC alone round 15 = 1.2577x. Host-cache retention round 8 = 1.2834x. FlashInfer GDN decode round 9 = 0.8771x. Student torch.compile round 10 stalled in capture-time OOM retries. Microbatches: 6144 = 1.2968x; 9216 = 1.3517x but unsafe estimated full-loop memory. Vocabulary chunks 4096 = 1.3380x; zero-entropy removal = 1.3338x. Teacher: chunk 2048 = 0.9162x, chunk 8192 = 1.0241x, chunk 16384 + cache320 = 1.0617x (no gain over chunk alone), FlashInfer allreduce at 4096 = 0.9735x, NCCL Simple = 0.9985x.
6.6 hRound 18 — student scheduler receive interval (start 2026-11-13 04:20 UTC)
- Profile: /tmp/r1/profile-rollout/kernel_report.md shows full decode CUDA graphs but 41.1% GPU coverage, with 1351.5 ms gaps on the main stream in a 2299.9 ms instrumented window. Backend replacement (round 9) and torch.compile (round 10) already failed. num_continuous_decode_steps is unused in this sglang source; scheduler_recv_interval is live through SchedulerRecvSkipper and can skip host request-receive work between decode steps.
- Change in isolated /tmp/r18/tree: off-by-default OPD_STUDENT_SCHEDULER_RECV_INTERVAL=4, applied to student sglang_scheduler_recv_interval both when constructing retained serving and when validating attachment. No teacher argument, fixed layout, dtype, sampling or forced lengths change. Scheduling can change online batching/kernel numerics, so rollout/teacher statistical gates are required; training state is untouched.
- Validation: syntax/diff checks passed. Judge rollout timing /tmp/r18/time-recv4 ran detached as PID 793689, driver /tmp/r18-time-recv4-driver.log, against the frozen reference. Both student engine logs confirmed scheduler_recv_interval=4; fixed TP2 × two-engine layout remained intact.
- Result: /tmp/r18/time-recv4/time.json; candidate rollout windows [76.738, 82.862, 83.496] s versus frozen [77.381, 77.185, 77.853] s, ratios [1.0084, 0.9315, 0.9324], side 0.9315x, projected step 0.9649x. Skipping receives delayed useful request handling and made windows 2–3 substantially slower. Rejected without quick correctness or a commit; isolated experiment only, promoted source and switches untouched. End 2026-11-13 04:30 UTC.
6.6 hCurrent summary (updated after round 18)
- Current switch file: {"environment": {"OPD_SKIP_ZERO_KL_REF": "1", "OPD_DISABLE_FULL_RECOMPUTE": "1", "OPD_TEACHER_SERVER_ARGS": "--chunked-prefill-size 16384"}}; all three repositories are clean at the verified source state.
- Latest candidate correctness: /tmp/r6/quick/check.json PASS N1 (train), /tmp/r17/quick-16384/check.json PASS N1 (teacher). No candidate full-loop/final run is available through the CLI.
- Verified gains: zero-KL ref-forward removal (a2b3bfab) plus no full activation recompute (ea0a0c37) = train 1.3408x, projected full-step 1.1386x (/tmp/r6/time/time.json). Teacher chunk 16384 = teacher 1.0630x, projected incremental step 1.0063x (/tmp/r17/time-16384/time.json); combined projection about 1.145x.
- Rejected after timing: student scheduler receive interval 4 = rollout 0.9315x (round 18). Actor-forward reuse rounds 7/16 had large steady gains but movable stalls and worst results 1.2828x, 1.2565x and 1.1858x. Manual GC = 1.2577x; host-cache retention = 1.2834x; FlashInfer GDN decode = 0.8771x; student torch.compile OOM-stalled; 6144/9216 microbatches = 1.2968x/1.3517x (larger unsafe on estimated colocated memory); vocab chunks = 1.3380x; zero entropy = 1.3338x. Teacher chunk 2048 = 0.9162x; cache320 added no gain; FlashInfer allreduce = 0.9735x; NCCL Simple = 0.9985x.
7.0 hRound 19 — resident actor between training steps (start 2026-11-13 04:31 UTC)
- Profile: current-best train profile /tmp/r19/profile-train (PID 809812, driver /tmp/r19-profile-train-driver.log). Its completed steps report sleep 4.8–6.7 s, wake 1.6–1.7 s and train_wait 7.2–9.7 s per step, while the actor holds 30.87 GiB of live parameter/gradient storage before offload. This is the largest non-kernel stage cost after the accepted forward/recompute changes. Student startup still reports about 51 GiB available after teacher+engine allocation, so residency is plausible but requires colocated-memory scrutiny. The profiler captured the requested step but spent over 12 minutes exporting with no trace artifact; it was stopped at 04:57 UTC and the judge cleaned up its workers, so no kernel report is claimed.
- Change in isolated /tmp/r19/tree: off-by-default OPD_RESIDENT_ACTOR=1, applied in the train-side Megatron validation hook after slime's colocate defaulting. It disables only actor sleep/wake (offload_train=False) for the no-critic colocated recipe; rollout offload, teacher residency, model/optimizer computation, layout and synchronous weight updates remain unchanged. Intended N0. The train-only judge run does not prove full-loop memory safety.
- Validation: syntax and diff checks passed. Judge train timing /tmp/r19/time-resident ran as PID 831457, driver /tmp/r19-time-resident-driver.log, with /tmp/r19/switches-resident.json; promoted /submission remained unchanged.
- First attempt: teacher replay completed, but train initialization failed before step 0 (/tmp/r19/time-resident/time.json, ERROR). Slime had already set the two disable_*_cpu_backup flags while defaulting colocated offload; with residency they still entered a TMS allocation region despite no LD_PRELOAD hook. Explicitly clear both flags in the resident branch. This was an incomplete configuration change, not a timing or numerics rejection.
- Retry: /tmp/r19/time-resident-fixed, PID 849796, driver /tmp/r19-time-resident-fixed-driver.log, using train-only /tmp/r19/switches-resident-train.json to avoid rerunning the unchanged teacher. It completed successfully.
- Timing result: /tmp/r19/time-resident-fixed/time.json; train windows [47.247, 46.798, 46.890] s versus frozen [75.757, 75.161, 73.865] s, ratios [1.6034, 1.6061, 1.5753], side 1.5753x, projected step 1.2119x, first-use 0 s. This improves the accepted train 1.3408x by removing sleep/wake and their host-memory stalls; offload_train=False was confirmed in the actual run. No sleep/wake timers remain and steady train_wait is about 1.7 s.
- Correctness: /tmp/r19/quick-resident/check.json PASS N1 (PID 867458, driver /tmp/r19-quick-resident-driver.log). Update direction cosine 0.995279 (minimum 0.956603), update norm difference -3.68e-5 within tolerance, loss mean difference -8.59e-6, and train/inference mismatch ratio 1.0. It is not N0. Full-loop colocated memory is not covered by this train-only run; require a separate un-timed full-loop completion before promotion.
7.7 hRound 20 — actor-forward reuse without sleep/wake stalls (start 2026-11-13 05:27 UTC)
- Profile: current-best resident timing /tmp/r19/time-resident-fixed/train_tree/train.log has steady actor log-probs about 9.7 s, actor train about 35.5–37.5 s and train_wait about 1.7 s. The separate actor forward is again the largest removable stage. Round 16's fused steady windows were 46.95/47.84 s even with sleep/wake, but a 16.84 s wake stall wrecked its third window; round 19 removes that mechanism and passes N1.
- Change in isolated /tmp/r20/tree: reapply only the guarded OPD_FUSE_OPD_ACTOR_LOGPROBS=1 forward reuse on top of round 19, without manual GC. The training forward supplies detached old log-probs and per-token OPD reverse-KL advantages, with complete per-sample capture for metrics/debug data and one unchanged optimizer update. Same data/order, teacher scores, GRPO base advantages and loss; expected N1 from grad-enabled forward numerics.
- Validation: syntax/diff checks passed. Judge train timing /tmp/r20/time-fused-resident ran detached as PID 884230, driver /tmp/r20-time-fused-resident-driver.log, with /tmp/r20/switches-fused-resident-train.json.
- Timing result: /tmp/r20/time-fused-resident/time.json; windows [37.458, 37.201, 36.920] s, ratios [2.0225, 2.0204, 2.0007], side 2.0007x, projected step 1.3149x, first-use 0 s. Separate actor log-probs are absent, steady total actor train is about 35–36 s, train_wait about 1.6 s, and there is no movable wake stall. This confirms the large round 7/16 gain once residency removes the sleep/wake mechanism.
- Correctness: /tmp/r20/quick-fused-resident/check.json PASS N1 (PID 901013, driver /tmp/r20-quick-fused-resident-driver.log). Update direction cosine 0.994338 (minimum 0.956603), update norm difference -6.57e-5, second-moment norm difference -8.05e-5, loss mean difference -9.22e-6, and mismatch ratio 1.0; every statistical tolerance passed.
- Full-loop memory/completion: un-timed two-step full recipe /tmp/r20/full-memory ran under the GPU lock as PID 917386, driver /tmp/r20-full-memory-driver.log, with forced fixture lengths and the accepted teacher chunk 16384. It completed both rollouts, training steps and weight updates with exit_code=0; step 0 and step 1 metrics are present. Observed colocated use was about 136.7/143.8 GB per GPU during rollout 1, tight but without OOM. This was only a completion/memory check; no speed is claimed from it. All validator processes exited and the GPUs returned to 0 MiB.
- Promotion: applied the exact tested sources to /submission. Round 19 commit 451e151f (Keep actor resident between train steps (N1); train time 1.5753x); round 20 commit e01c3bf2 (Reuse training forward for OPD actor log-probs (N1); train time 2.0007x). Enabled OPD_RESIDENT_ACTOR=1 and OPD_FUSE_OPD_ACTOR_LOGPROBS=1 in /submission/switches.json. End 2026-11-13 06:04 UTC.
- All-side quick verdict: /tmp/r20/quick-all/check.json PASS N1 on the promoted submission, finished 06:58:48 UTC. Rollout, teacher and training gates all passed; key rollout/token-forcing, online-teacher, training-direction and mismatch statistics are recorded in the current summary below.
7.7 hCurrent summary (updated after round 20)
- Current switch file: {"environment": {"OPD_SKIP_ZERO_KL_REF": "1", "OPD_DISABLE_FULL_RECOMPUTE": "1", "OPD_TEACHER_SERVER_ARGS": "--chunked-prefill-size 16384", "OPD_RESIDENT_ACTOR": "1", "OPD_FUSE_OPD_ACTOR_LOGPROBS": "1"}}.
- Latest candidate correctness: /tmp/r20/quick-all/check.json PASS N1 for rollout, teacher and train on the exact promoted /submission tree and switches (finished 2026-11-13 06:58:48 UTC). Rollout token forcing mean-|d| ratio 0.9884, tail-event p=0.8009, sampler t=1.3059, online-teacher mean-|d| ratio 0.9908 and mismatch ratio 0.9906; teacher replay passed as before; training update-direction cosine 0.992736, all statistical tolerances passed and mismatch ratio 1.0. Separate train-only evidence: /tmp/r20/quick-fused-resident/check.json PASS N1. /tmp/r20/full-memory/exit_code is 0 after two complete colocated steps, with about 136.7/143.8 GB observed per GPU. No final-mode run is available through the CLI.
- Verified gains: zero-KL ref-forward removal (a2b3bfab) + no full recompute (ea0a0c37) gave train 1.3408x. Resident actor (451e151f) alone gives train 1.5753x, projected step 1.2119x (/tmp/r19/time-resident-fixed/time.json). Actor-forward reuse on resident (e01c3bf2) gives train 2.0007x, projected step 1.3149x (/tmp/r20/time-fused-resident/time.json). Teacher chunk 16384 gives teacher 1.0630x (/tmp/r17/time-16384/time.json); combined projection is approximately 1.326x, not a final full-loop measurement.
- Rejected directions: scheduler receive interval 4 = rollout 0.9315x; actor-forward reuse with offloaded actor/GC had worst windows 1.2828x, 1.2565x and 1.1858x due sleep/wake stalls; manual GC = 1.2577x; host-cache retention = 1.2834x; FlashInfer GDN decode = 0.8771x; student torch.compile OOM-stalled; 6144/9216 microbatches = 1.2968x/1.3517x (larger memory-unsafe); vocab chunks = 1.3380x; zero entropy = 1.3338x. Teacher chunk 2048 = 0.9162x; cache320 added no gain; FlashInfer allreduce = 0.9735x; NCCL Simple = 0.9985x.
8.6 hRound 21 — CPU optimizer intra-op thread count (start 2026-11-13 06:18 UTC)
- Profile/code evidence: /tmp/r20/time-fused-resident/train_tree/train.log has steady fused actor train 35–36 s. The recipe offloads all FP32 master weights/moments and fused AdamW to the CPU, with one optimizer per parameter and four colocated actor processes. PyTorch reports 96 intra-op threads by default on the 96-CPU host, with no OMP_NUM_THREADS or Megatron override, so the actors can create 384 worker threads and contend on their NUMA-affined CPUs.
- Change in isolated /tmp/r21/tree: off-by-default OPD_CPU_OPTIMIZER_THREADS=24, setting actor intra-op threads once at initialization. Model/optimizer arithmetic, data/order, FP32 state and GPU layout are unchanged; intended N0 on top of the existing N1 candidate. This tests one bottleneck class (CPU Adam thread oversubscription), not a new optimizer.
- Validation: syntax/diff checks passed. Judge train timing /tmp/r21/time-threads24 ran as PID 957190, driver /tmp/r21-time-threads24-driver.log, using /tmp/r21/switches-threads24.json.
- 24-thread result: /tmp/r21/time-threads24/time.json; windows [35.525, 35.278, 34.944] s, ratios [2.1325, 2.1305, 2.1138], side 2.1138x, projected step 1.3374x, first-use 0 s. This improves round 20's 2.0007x by about 1.9 s per train step; the actual actor logs confirm the 24-thread setting.
- 16-thread comparison: same source, /tmp/r21/switches-threads16.json, judge timing /tmp/r21/time-threads16 ran as PID 981308. Windows [35.446, 35.208, 34.947] s, ratios [2.1373, 2.1348, 2.1136], side 2.1136x, projected step 1.3374x, first-use 0 s. It is effectively tied with 24 threads; retain 24 because its judged side minimum is marginally higher (2.1138x versus 2.1136x).
- Correctness for the selected 24-thread setting: /tmp/r21/quick-threads24/check.json PASS N1 (PID 1014732, driver /tmp/r21-quick-threads24-driver.log). Update-direction cosine 0.990590 (minimum 0.956603), update norm difference -1.90e-4, second-moment norm difference +1.85e-4, loss mean difference -3.79e-6 and mismatch ratio 1.0; all tolerances passed.
- Promotion: applied the exact tested actor change and enabled OPD_CPU_OPTIMIZER_THREADS=24. Commit 38b748ef (Limit CPU optimizer intra-op threads (N1); train time 2.1138x). End 2026-11-13 07:09 UTC.
8.6 hCurrent summary (updated after round 21)
- Current switch file: {"environment": {"OPD_SKIP_ZERO_KL_REF": "1", "OPD_DISABLE_FULL_RECOMPUTE": "1", "OPD_TEACHER_SERVER_ARGS": "--chunked-prefill-size 16384", "OPD_RESIDENT_ACTOR": "1", "OPD_FUSE_OPD_ACTOR_LOGPROBS": "1", "OPD_CPU_OPTIMIZER_THREADS": "24"}}.
- Latest correctness: /tmp/r20/quick-all/check.json PASS N1 for all three sides on round 20; /tmp/r21/quick-threads24/check.json PASS N1 for the added thread setting. /tmp/r20/full-memory/exit_code is 0 after two complete colocated steps. No final-mode run is available through the CLI.
- Verified gains: train 2.1138x, projected train-only step 1.3374x (/tmp/r21/time-threads24/time.json, commit 38b748ef). Teacher chunk 16384 = 1.0630x (/tmp/r17/time-16384/time.json); combined projection is approximately 1.349x, not a final full-loop measurement. The earlier train chain is a2b3bfab + ea0a0c37 + 451e151f + e01c3bf2 + 38b748ef.
- Rejected directions are unchanged from round 20. The 16-thread alternative tied at 2.1136x and was not promoted.
9.1 hRound 22 — teacher scheduler prefill batch size (start 2026-11-13 06:52 UTC)
- Profile: unchanged teacher profile /tmp/r1/profile-teacher/kernel_report.md has 92.2% GPU coverage, 43.0% in the dominant GEMM class and 22.3% in TP4 all-reduce. Accepted chunk 16384 removes per-request hybrid-state chunk boundaries but leaves the scheduler's default prefill batch cap at 16384 tokens. Larger batches may reduce launch/collective count and improve GEMM utilization; this is the same prefill-batching hotspot as round 17, not a new model or kernel.
- Change: configuration-only isolated teacher experiment /tmp/r22-switches-32768.json, setting both --chunked-prefill-size 32768 and --max-prefill-tokens 32768. Same TP4 BF16 teacher, requests and reward path; expected N1 from re-batching/chunk boundaries. A 24576 alternative is prepared at /tmp/r22-switches-24576.json if 32768 is memory-limited.
- Timing result: /tmp/r22/time-32768/time.json; teacher windows [16.787, 16.831, 16.607] s versus frozen [18.532, 18.478, 18.129] s, ratios [1.1039, 1.0979, 1.0916], side 1.0916x, projected step 1.0089x. This improves accepted 16384's 1.0630x by about 2.7% on the teacher side. The actual server confirmed TP4, BF16, chunk/batch 32768 and 56-request Mamba-state cap; teacher-only available memory 95.82 GiB.
- Correctness: /tmp/r22/quick-32768/check.json PASS N1 (PID 1045040, driver /tmp/r22-quick-32768-driver.log). Mean |d| ratio 1.0164; tail events 4 at rate 1.941e-6 versus reference 4.246e-6, binomial p=0.9747; bias t=-0.4910. Every teacher tolerance passed.
- Full colocated validation: combined round-21 + round-23 + teacher-32768 tree /tmp/r23/combined-tree, switches /tmp/r23/switches-combined.json. The un-timed two-step full recipe /tmp/r23/full-memory-32768 ran under the GPU lock as PID 1060412 and finished with exit_code=0. Both forced-length rollouts, train steps 0/1 and weight updates completed; no OOM. This was only a completion/memory check, not a speed measurement. GPUs returned to 0 MiB.
- Combined correctness: /tmp/r23/quick-combined running detached as PID 1079637, driver /tmp/r23-quick-combined-driver.log, judges rollout, teacher and train on the exact combined candidate. Promotion requires PASS; /submission remains at round 21.
9.3 hRound 23 — remove per-decode optional-top-p GPU synchronization (start 2026-11-13 06:59 UTC)
- Profile: existing current-source rollout trace /tmp/r1/profile-rollout/trace/1794520227.9295971-TP-0.trace.json.gz has 41.1% GPU coverage and 1351.5 ms main-stream gaps. CPU-op/stack attribution shows 200 calls of aten::item in Sampler.forward totaling 795.761 ms of its 2299.949 ms profile window (maximum 17.343 ms); the other per-step item calls are CPU mirrors and together take under 2 ms. The synchronizing call is return_top_p_log_probs.any().item(), although this OPD recipe never enables return_top_p_log_probs. This is the dominant measured rollout host bottleneck.
- Change in isolated /tmp/r23/tree: retain the request-derived top-p log-prob flags as a host list in SamplingBatchInfo, filter and merge that list with the existing device mask, and use any(host_list) in the sampler. Manually constructed metadata without the host mirror retains the old device reduction as a fallback, including mixed merges. Same sampled distribution, log-prob computation and output; intended N0 because only the location of the boolean reduction changes. No environment switch is needed for this generic semantics-preserving fix.
- Validation: syntax compilation and git diff --check passed. Existing CPU unit suite test/registered/unit/sampling/test_sampling_batch_info.py: 47 passed plus 13 subtests. Judge rollout timing /tmp/r23/time ran as PID 1030716, driver /tmp/r23-time-driver.log, using /tmp/r23/switches.json (the verified round-20 switches).
- Timing result: /tmp/r23/time/time.json; rollout windows [58.535, 60.304, 60.317] s versus frozen [77.381, 77.185, 77.853] s, ratios [1.3220, 1.2799, 1.2907], side 1.2799x, projected rollout-only step 1.1214x, teacher tails [16.129, 14.456, 14.529] s. Combined with round-21 train 2.1138x, the projected full-step speedup is approximately 1.564x (the rollout window already includes the teacher critical-path share). Actual layout checks passed.
- Next: run all-side quick correctness on a combined tree containing round-21 train threading, this sampler change and any teacher configuration that passes round 22. Full colocated memory/completion is required before promotion. /submission remains at round 21.
↑ Slime OPD