GLM-5.3 1.46×
Harness: Claude Code · switches: OPD_SKIP_KL0_REF_PASS=1 OPD_FUSED_ADVANTAGES=1 OMP_NUM_THREADS=8 OPD_RECOMPUTE_OFF=1 OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 16384 OPD_CPU_ADAM_THREADS=8 OPD_SAMPLER_CPU_TOPP_FLAG=1 OPD_SHARDED_INPUT_LOGPROBS=1
- Judged
- 1.46×
- Tier
- N1
- Self-best
- 1.58×
- Rounds
- 13
- Hours
- 10.0
- API cost
- $80.6
- Input tokens · 99.7% cached
- 292 M
- Output tokens
- 0.8 M
- Requests
- 2,083
§ 02Key levers
- Candidate, three windows (s)
- 105.0 / 105.3 / 106.8
- Rollout side
- 1.32×
- Teacher side
- 1.16×
- Train side
- 1.71×
- Per-step sampler sync
- ✓ (round 13)
- Train-side main levers
- partial: skip reference forward, fused advantage computation, no recompute, CPU Adam 8 threads
- Teacher-side changes
- ✓ teacher computes input logprobs per shard (−1.4 s), teacher chunk 16384
§ ARaw trace
Raw work log18 sections
0.2 hRound 1 (2026-11-13 01:03–01:39): profile the unchanged reference
Profile of all three sides launched detached (train → rollout → teacher), logs/tmp/r1/profile-all.log, outputs/tmp/r1/profile-{train,rollout,teacher}. Findings from slime's own timers in the reference train-mode log (runs/bootstrap/timing_train_1/train_ref_0/train.log) — per steady-state train step (~73 s window): | segment | seconds | note | |---|---|---| |ref_log_probsforward-only pass | ~10.7 | ref-model log-probs; only used askl_loss_coef(0.00) × kl→ contributes exactly 0 | |log_probsactor forward-only pass | ~9.9 | current-policy log-probs; used only for the OPD advantage and asold_log_probsin the loss — with 1 train step/rollout the train pass recomputes the same values | |actor_train(fwd+recompute+bwd+optimizer) | ~45 | the real train pass | | sleep + wake_up (colocate offload) | ~6 | torch_memory_saver pause/resume + NCCL group destroy/reload | | train_wait (fixture dump loading) | ~6.7 | judge's instrument path — not to be optimized | Microbatch structure: 64 samples ≈ 8280 tokens each (responses ~8040 forced), `max_tokens_per_gpu 4096 × cp 2 = 8192` per bin → every microbatch holds a single sample → 64 microbatches/step on TP2×CP2 (4140 local tokens per rank). Data parallel (with CP) = 1. Verified from the reference run's own log (runs/bootstrap/timing_train_1/train_ref_0/train.log): - steady state (rollout 1): sleep 4.26 + wake 1.62 + train_wait 6.70 + ref_log_probs 10.86 + log_probs 9.88 + actor_train 45.89 = step 73.77 s. -rollout/advantages= −rollout/opd_reverse_klexactly (GRPO base advantages are 0 with zero task rewards;rollout/kl= 0.0), so the advantage is purely −1.0 × reverse KL. -train/kl_loss= 0.0 at step 0 (ref == actor before any update) and ~3e-4 later, butloss == pg_lossexactly every step → the0.00 × kl_lossterm contributes nothing.train/ppo_kl = 0.0andtrain/pg_clipfrac = 0.0every step → the separate log-prob pass and the train pass agree bitwise (low_var_klis non-negative, so a 0.0 mean forces every element to 0). - The judge reads onlytrain/loss,train/grad_norm,train/train_rollout_logprob_abs_difffrom thestep N: {...}lines (bench.py:1261) — droppingtrain/kl_lossis safe. - Recipe args confirmed: use_kl_loss=True with kl_loss_coef=0.0, kl_coef=0.0, opd_kl_coef=1.0, use_opd=True, opd_type=sglang, grpo, policy_loss, normalize_advantages=False, use_rollout_logprobs=False, get_mismatch_metrics=False, use_critic/use_tis/use_opsm/use_score_centering/use_routing_replay/ use_rollout_entropy all False, entropy_coef=0.0, num_steps_per_rollout=None, save_debug_train_data=None. ### Change implemented this round (train side, behind switches, default off) 1.OPD_SKIP_KL0_REF_PASS=1— skip the separate ref-model log-prob forward whenkl_coef == 0andkl_loss_coef == 0(its only consumer is0 × kl). Also skips the pointless_switch_model("actor")re-restore when no model was switched. Loss skips the KL term whenref_log_probsis absent and the coef is 0;train/kl_lossmetric then not reported (judge reads only train/loss, grad_norm, train_rollout_logprob_abs_diff — verified in bench.py:1262-1267). 2.OPD_FUSED_ADVANTAGES=1— skip the separate actor log-prob forward and apply the OPD reverse-KL advantage term inside the train loss from the train pass's own (detached) log-probs; base GRPO advantages still come fromcompute_advantages_and_returns. Gated on: 1 train step/rollout, policy_loss, kl_coef 0, kl_loss_coef 0 (or no use_kl_loss), no rollout-logprobs/mismatch/critic/ normalization/score-centering/tis/opsm/routing-replay/rollout-entropy, grpo estimator, use_opd with sglang teacher, teacher_log_probs present. The loss's existingold_log_probsfallback ([lp.detach() for lp in log_probs]) then reproduces the reference'sold_log_probsexactly. 3.OPD_FUSED_ADVANTAGES_VERIFY=1(evidence mode, not for submission) — runs the reference path AND captures the train pass's log-probs, loggingmax|train_lp − separate_lp|and the count of bitwise-equal samples per rollout. Profile status at round end: the train-side torch-profiler run finished its 8 steps and wrote 4 complete raw traces (23.2–23.4 GB each), but the in-process gzip export was only 37% done after 15 min (~14 MB/s per rank, single-core) — killed to free the GPUs for Round 2. The traces were then lost: bench'sscratch_dircleanup deleted/tmp/sg-run-*(raw.jsons) when the profile process exited, and the partial.gzfiles were unusable. Lesson: copy traces out of the scratch dir before the bench process exits (also saved to agent memory). Train-side kernel attribution will instead come from the per-segment Timer lines (fwd 10 s / recompute ~10 s / bwd ~20 s of actor_train's 45.9 s) and, if needed, a re-run of the train profile. The rollout/teacher profile runs were postponed; they run in Round 3 (their traces are small and export fast). Expected train-side saving ≈ 20.6 s of 73.3 s → train speedup ≈ 1.39×, projected step ≈ 1/(1 − 0.4621 × (1 − 1/1.39)) ≈ 1.15×. Files:slime/backends/megatron_utils/{actor,loss,model}.py(train side only).
0.8 hRound 2 (2026-11-13 01:39–): train-side skips (OPD_SKIP_KL0_REF_PASS + OPD_FUSED_ADVANTAGES)
check --quick(train side, record run, switches ON, config/tmp/r2/switches-on.json) → PASS, tier N1 (/tmp/r2/quick/check.json, finished 01:50:34, record wall 654 s vs reference 702 s): - exact items ok; bitwise attempt: 6316 differing tensors — the unchanged reference's own repeat differs in 6342 (same tensor classes: conv1d/in_proj/fc1/fc2) → statistically indistinguishable from a reference rerun. - statistical endpoints (candidate vs reference, tolerance): update_norm Δ4.1e-5 (tol 1.2e-3), exp_avg_sq_norm Δ4.1e-5 (tol 4.4e-4), loss_mean Δ1.25e-5 (tol 3.75e-5), grad_norm_first Δ5.9e-5 (tol 1.65e-3) — allwithin: true; train–inference mismatch ratio 1.0 (max 2.0). - per-step canaries vs the reference record run: step 0 loss BITWISE identical (0.05341373383998871); steps 1–2 within the reference's own repeat noise (loss Δ3.2e-5 / Δ4.6e-5); mismatch metric Δ<1e-7.time --side train(01:50:51–02:04:38,/tmp/r2/time, wall 809 s) → **train speedup 1.3214×, projected step 1.1266×** — PASSes the benefit gate (max(1.02, 1+2×0.0226) = 1.045): - windows [55.22, 53.15, 55.05] s vs frozen reference [72.97, 73.49, 73.54] (ratios 1.321/1.383/1.336; side speedup = min = 1.3214); first_use 0.0 in every window. - step_time series: cold 213, dump-1 first visit 71.2, dump-2 first visit 105.7 (first-use artifacts, same family as the reference's own 128.5 s outlier on its dump-2 first visit), then 53.4/55.1/52.9/55.3/53.7. - steadyactor_train46.3–48.6 s ≈ the reference's 45.9 — the skips do not change the train pass itself; the whole gain is the removed 20.7 s of forward-only passes. - mismatch trajectory 0.00805 → 0.0289 over 8 steps, same as the reference's own (0.008–0.029). Committed as358e676f(actor.py, loss.py, model.py); switches promoted to/submission/switches.json(OPD_SKIP_KL0_REF_PASS=1, OPD_FUSED_ADVANTAGES=1). Mid-round discovery (informs Round 3+): Megatron'sHybridDeviceOptimizer.step()(Megatron-LM/megatron/core/optimizer/cpu_offloading/hybrid_optimizer.py:170-174) runs the CPU Adam synchronously insideactor_train: per-paramtorch.optim.AdamW(fused=True)instances stepped sequentially in the calling thread,optimizer_offload_fraction=1.0→ 4.5 B params/rank on CPU. Ray forcesOMP_NUM_THREADS=1when unset → each fused AdamW is single-threaded. This is the main remainingactor_traintail;OMP_NUM_THREADS=8(Round 3) attacks it directly (microbenchmark: 1 thread 1.25 s per 400 M params, 8 threads 0.08 s; bitwise identical). Host: 96 cores, 2 TB RAM — 4 ranks × 8 threads is safe.
1.0 hRound 3 (2026-11-13 02:04–02:19): `OMP_NUM_THREADS=8` (CPU Adam multi-thread, config-only)
time --side trainwith/tmp/r3/switches-omp.json(Round 2 switches + OMP_NUM_THREADS=8), run 02:05–02:18 (/tmp/r3/time) → train speedup 1.4241×, projected step 1.1596× (vs Round 2's 1.3214×): - windows [51.23, 50.19, 51.64] s vs frozen reference [72.97, 73.49, 73.54] (ratios 1.424/1.464/1.424); first_use 0.0. - the gain (~4 s/window) sits in the steady window steps; the noisy first-visit actor_train timers (46.7/45.3 s) had masked it — the judged windows are the evidence. - mismatch trajectory bitwise-identical to R2 and the reference (step 0: 0.008052160032093525). - Ray only defaults OMP_NUM_THREADS when it is unset (ray/_private/utils.py:244), so the switches entry reaches the Megatron actors; host has 96 cores, 2 TB RAM (4 ranks × 8 threads safe). Config-only: no code change to commit. Switches promoted to/submission/switches.json(OPD_SKIP_KL0_REF_PASS=1, OPD_FUSED_ADVANTAGES=1, OMP_NUM_THREADS=8).
1.0 hRound 4 (2026-11-13 02:18–02:31): `OPD_RECOMPUTE_OFF=1` (store activations, no full-layer recompute)
time --side trainwith/tmp/r4/switches-recompute.json(Round 3 switches + recompute off), run 02:18:20–02:30:52 (/tmp/r4/time) → train speedup 1.5812×, projected step 1.2046×: - windows [43.57, 46.48, 42.98] s vs frozen reference [72.97, 73.49, 73.54] (ratios 1.675/1.581/1.711; side speedup = min = 1.5812); first_use 0.0. −7 s/window vs Round 3. - mismatch trajectory: step 0 bitwise identical (0.008052160032093525), later steps within 2e-5 of R2/R3 — the train pass's log-probs are unchanged (recompute is bitwise-neutral, as predicted). - Train-mode memory: no OOM (109 GB free there). Full-loop risk (teacher resident during the train phase, engines offloaded) probed separately below. Committed as2dd9cdb2(arguments.py); switches promoted to/submission/switches.json(4 entries: skips + OMP_NUM_THREADS=8 + OPD_RECOMPUTE_OFF=1). ### Full-loop de-risk (not a timing run)/tmp/r4/fullrisk.sh(02:31:20–02:46:33, rc=0, log/tmp/r4/fullrisk.log): the recipe's ownlaunch.shin MODE=full with the candidate switches, NUM_ROLLOUT=3, forced lengths. Passed: 3 rollouts (rollout_time 80.5/74.5/75.5 s — steady rollouts match the reference's 74.6–76.1, as expected for train-side-only changes) and 3 train steps, zero OOM/CUDA errors. Train-phase peak memory with the no-recompute activations beside the resident teacher: 120 GB / 144 GB per GPU (23 GB headroom), measured mid-step. The full loop works end-to-end with the fused-advantage path and live engines + teacher. ### Full check (the verdict) — PASScheck --tree /submission/slime_opd --config /submission/switches.json --out /tmp/r4/check(02:47–03:08, train side) → status PASS, tier N1, performance.speedup 1.2269: - benefit: candidate timing windows [43.76, 42.95, 42.41] s vs frozen reference [72.97, 73.49, 73.54] → train speedup 1.6674×; projected step speedup 1.2269× (bar 1.0452, passed). - record run (wall 530 s): exact items ok; bitwise attempt 6324 differing tensors (the unchanged reference's own repeat: 6342 — statistically indistinguishable); statistical endpoints allwithin: update_norm Δ1.0e-4 (tol 1.2e-3), exp_avg_sq_norm Δ5.1e-5 (tol 4.4e-4), loss_mean Δ1.27e-5 (tol 3.75e-5), grad_norm_first Δ2.4e-5 (tol 1.65e-3); update_direction_cos 0.9940 ≥ 0.9566 (reference's own noise cos 0.9913); train–inference mismatch ratio exactly 1.0 (0.008052160032093525 on both sides). - verdictequivalent, measured tier N1.
2.1 hRound 5 (2026-11-13 03:08–): rollout side — profile first
The rollout (74.6–76.1 s of the ~159 s reference step; ≈85 s of train_wait in the full loop) is the largest remaining component. Evidence so far: student engines already capture CUDA graphs (decode backend=full bs≤138, prefill breakable); per-engine decode ~4700–5000 tok/s ≈ 6.4 ms/step at batch 32 vs ~3 ms theoretical floor (9 GB weights/GPU + GDN state); dips to 1400–2900 tok/s during teacher prefill bursts; teacher tail 16.4 s ≈ drain of the ~30+ samples that finish in the last second (mamba cache caps the teacher at 56 concurrent).num_continuous_decode_stepsis a dead arg in this sglang (defined, never read). Staged candidate: teacher--chunked-prefill-size 8192viaOPD_TEACHER_SERVER_ARGS(/tmp/r5/switches-teacher-chunk.json) — numerically pre-validated (re-batching deviation 0.006912 at the reference's pooled noise floor 0.0068, gate bound 0.0137), makes the tail drain and the interference bursts more efficient; not in the layout-gated fields.profile --side rollout(03:08–03:16,/tmp/r5/profile, wall 441 s) — 200 decode steps on every student worker, kernel report of one TP rank: - window 2337 ms for 200 steps (11.7 ms/step with profiler overhead; unprofiled 6.4–6.8 ms); GPU busy (this process's kernels) 39.2 % — the rest is teacher contention on the same GPU + scheduler gaps. - student kernel time ≈ 5.0 ms/step: GEMMs ≈2.9 ms (nvjet bf16, memory-bound at ~65 % of peak BW), GDNfused_recurrent_gated_delta_rule0.48 ms (24/step), fused allreduce+RMSNorm (oneshot lamport) 0.39 ms (63/step), LM head 0.24 ms (at its 2 GB/step vocab-matrix memory floor), flash-attn 0.11 ms, sampler softmax 0.08 ms. One under-filled GDN GEMM (64x32_64x16_1x1_h_bz_splitK, 24/step, 3 blocks, SM cov 0.02) costs 0.31 ms/step. - stream gaps: main stream 1417 ms of gaps over the window (median 0.4 µs — graph replay is tight); the ~1.3 ms/step wall-vs-kernel gap is teacher SM contention, not launch overhead. - Conclusion: decode-side gains need deep kernel work (bounded by ~2.5 ms/step memory floor); the cheap lever is the teacher's interference and tail drain → teacher chunk 8192 (Round 5's change).
2.1 hRound 5 timings (03:17–03:30, `/tmp/r5/time-chain.log`)
- teacher side: speedup 1.0253× — windows [17.49, 17.80, 17.29] s vs frozen reference [18.31, 18.25, 17.89] (ratios 1.046/1.025/1.035; teacher benefit bar 1.0222 — passed). - rollout side: speedup 1.0045× — windows [74.67, 75.81, 74.87] s vs [75.98, 76.15, 75.77] (real ~0.85 s/window but at the rollout noise floor, spread 0.0125). - Combined contribution to the projected step ≈ +0.5 % (teacher fraction 0.1033 × 2.5 % + rollout 0.4787 × 0.45 %) → expected final speedup ≈ 1.234× vs 1.2269× without. - Decision: promote the 5-switch config (OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 8192added; numerics pre-validated — chunk 8192 is one of the reference's own accepted re-batchings, deviation 0.006912 vs pooled noise 0.0068, bound 0.0137) and run the 3-sided fullcheckas the verdict (launched 03:31, PID 468712, log/tmp/r5/check.log→/tmp/r5/check), covering the rollout and teacher gates with the chunk-8192 teacher online — the same gates the final evaluation runs. If any gate fails, revert to the 4-switch config (full check PASS at 1.2269×).
2.1 hRound 5 full check (03:31–03:55, `/tmp/r5/check`) — PASS
The 5-switch config's full check judged the sides the diff+config touch — teacher + train (the rollout side is untouched by both): - status PASS, tier N1, performance.speedup 1.2269 (bar 1.0452): teacher speedup 1.0257× (gates PASS — the chunk-8192 replay stayed within the re-batching tolerance, as pre-validated), train speedup 1.6518× (record 539 s, gates PASS N1). - The teacher chunk's rollout-side gain (1.0045×) is not in this projection (side not judged) but lands in the final evaluation's full-loop measurement. - Remaining uncovered gate stack: the rollout side with the chunk-8192 teacher online (token forcing + the rollout's teacher log-probs vs the reference teacher's replay) — exactly what the final evaluation judges.check --quick --side rolloutlaunched 03:58 (PID 506643,/tmp/r5/quick-rollout); if it fails, revert switches to the 4-switch config.
2.1 hRound 5 rollout-gate check (03:57–04:31, `/tmp/r5/quick-rollout`) — PASS
check --quick --side rollout with the 5-switch config (the chunk-8192 teacher online during rollout):
- rollout PASS: token forcing mean |d| ratio 0.9837 (bar 2.0 — candidate 0.00766 vs reference's
own 0.00779), event rate 4.85e-7 vs reference 1.46e-6 (p = 0.95), sampler paired t 1.77 (bar 4.0).
- teacher PASS (rollout's teacher log-probs vs the reference teacher's replay), train PASS N1
(train–inference mismatch).
- All three sides' gates now PASS with the exact submission config. The 5-switch config stands.3.5 hRound 6 (2026-11-13 04:32–04:58): train-side profile — findings
profile --side train(/tmp/r6/profile, wall 1370 s, traces exported to<out>/traceintact). One timed train step, rank 2: window 52.0 s, GPU busy 62.1 % (the idle part = the judge's dump loading + CPU Adam, as expected). Kernel time 33.1 s: | class | time | share | note | |---|---:|---:|---| | NCCL AG (RING_LL) | 4.05 s | 12.2 % | 12481 calls (≈195/microbatch), median 93 µs, SM cov 0.18, skewed — SP/CP gathers, LL protocol on ~34 MB messages | | GEMMs (nvjet) | 13.6 s | 41 % | full SM coverage, 1.00 waves — already efficient | | NCCL RS (RING_LL) | 2.05 s | 6.2 % | 6784 calls, same flags | | GDN kernels | ≈4.5 s | 13.6 % | causal_conv1d_bwd 0.82 s (waves 265!), chunk_gated_delta_rule fwd/bwd ≈1.5 s, prepare_wy_repr 0.43 s | | elementwise/add/copy | ≈2.1 s | 6.4 % | 22528 add-bf16 + 19072 add-f32 + fill/copy — fusion candidates | | NCCL SendRecv + f32 AllReduce | 1.06 s | 3.2 % | f32 AllReduce very skewed (wait spin) | Per microbatch (625 ms): NCCL ≈95 ms, GEMMs ≈212 ms, GDN ≈70 ms, elementwise ≈33 ms, rest ≈90 ms, gaps ≈125 ms. NCCL total 7.16 s = 21.6 % of kernel time — the LL protocol on NVLink is the config-level lever;NCCL_ALGOis reserved for the judge's record runs butNCCL_PROTOis not, and timing runs pick their own algo (the profile's RING_LL was NCCL's own choice).
4.0 hRound 7 (2026-11-13 05:00–): `NCCL_PROTO=LL128` (config-only)
time --side trainwith/tmp/r7/switches-ll128.json(5-switch config + NCCL_PROTO=LL128), launched 05:00 (PID 584290 →/tmp/r7/time). LL128 keeps LL-class latency with ~2× LL bandwidth on NVLink; expected −1 to −2.5 s on the AG/RS pair (part of their time is peer-wait spin that will not improve). Numerics: Ring+LL128 is still a fixed ring order (record runs force NCCL_ALGO=Ring anyway); judged by the N1 gates if promoted. - Switch file:/submission/switches.json— Round 4's config (`OPD_SKIP_KL0_REF_PASS=1, OPD_FUSED_ADVANTAGES=1, OMP_NUM_THREADS=8, OPD_RECOMPUTE_OFF=1`), train speedup 1.5812×, projected step 1.2046× (/tmp/r4/time). - Latest checks: 4-switch full check PASS N1 1.2269 (/tmp/r4/check); 5-switch (adds teacher chunk 8192) full check PASS N1 1.2269 with teacher gates PASS (/tmp/r5/check, 03:55) — teacher 1.0257×, train 1.6518×. Rollout-side quick check with the chunk teacher in flight (/tmp/r5/quick-rollout). - Commits:358e676f(Round 2 skips),2dd9cdb2(Round 4 recompute off). Round 5 is config-only. - Next candidates: train-side profile (attribute actor_train's ~30 % MFU; rescue traces properly), then microbatch packing or kernel work. Full-loop-only components (sleep+wake 5.4 s, update_weights 2.0 s) are unmeasurable bytime(debug modes skip them) — deferred as unjustifyable risk. - Full-loop de-risk PASSED (see above): no OOM, 23 GB headroom, steady rollouts unchanged. - Queue after R3/R4: fullcheckwith the winning config → train profile (rescue traces before the scratch cleanup!) → rollout/teacher profiles. Microbatch packing (2 samples/mb) investigated and deprioritized: the plan is built in the rollout manager (needs a parse-path hook outside all sides) and the estimated gain is only ~1.3 s. - Round 3 evidence (CPU-offloaded Adam):actor_train46.5 s steady contains the synchronous serial per-param CPU Adam (4.5 B params/rank,optimizer_offload_fraction=1.0). Ray forcesOMP_NUM_THREADS=1in its workers only when the variable is unset (ray/_private/utils.py:244), andOMP_NUM_THREADSis not in the judge's reserved_env → a switches.json entry reaches the Megatron actors. Micro-benchmark (torch.optim.AdamW fused=True, the exact class Megatron uses asCPUAdam, 400M fp32 params): 1 thread 1.25 s, 8 threads 0.08 s; the update is bitwise IDENTICAL across 1/8/16 threads (verified twice). - Round 4 candidate:OPD_RECOMPUTE_OFF=1(implemented, default off, inslime/backends/megatron_utils/arguments.py_set_default_megatron_args): sets recompute_granularity/method/ num_layers to None post-parse (memory strategy — allowed). Recomputed activations equal stored ones bitwise; expected −8–10 s (the recompute is one extra forward). Memory risk: ~11 GB/rank of activations vs ~29 GB headroom (141 − engines 63.5 − teacher 35 − model/grads/logits/NCCL ~13). - Later candidates: rollout window 76 s ≈ student decode ~58 s + teacher tail 16.4 s (teacher prefill is compute-bound at ~45% MFU: 2×27e9×530e3 FLOPs / 4 GPUs ≈ 16 s — chunk-size tweaks gain ~1-2 s at most;--chunked-prefill-size 8192viaOPD_TEACHER_SERVER_ARGSis numerically pre-validated: its re-batching deviation 0.006912 sits at the reference's pooled noise floor 0.0068, gate bound 2× pooled ≈ 0.0137); sleep + wake 5.9 s/step (torch_memory_saver + NCCL reload). - Rollout-side evidence (from the reference bootstrap rollout log): the student engines DO capture CUDA graphs (prefill backend=breakable up to 8192 tokens, decode backend=full bs=[1..138]; the legacycuda_graph: Falseserver-args field is dead) — decode at batch 32 is graph-replayed. Per-engine decode throughput ~4700–5000 tok/s (≈6.4 ms/step) with dips to 1400–2900 tok/s during the teacher's prefill bursts and one 56 tok/s stall; theoretical floor ≈3 ms/step (9 GB weights/GPU + GDN state R/W). The gap is per-step scheduler/sampler overhead + teacher SM contention.num_continuous_decode_steps>1and the teacher chunk size are the config-level levers;profile --side rolloutdecides. - train_wait 6.7 s exists only in the standalone train timing (fixture loading), not in the full loop; judge's instrument path — not to be touched. - Canaries for the fused path: reference logs showtrain/ppo_kl = 0.0andtrain/pg_clipfrac = 0.0every step (the two passes' log-probs agree exactly), andtrain/train_rollout_logprob_abs_diff≈ 0.008–0.029 per step; my run must reproduce these. Record-run step-0 canary:train/lossmust equal the reference's 0.05341373383998871 exactly (the forward loss on dump 0 — comm algo changes do not affect it). Result (record run, 01:45:46): EXACT MATCH — loss 0.05341373383998871, pg_loss 0.05341373383998871, entropy_loss 0.3821689486503601, pg_clipfrac 0.0, ppo_kl 0.0, mismatch 0.008052160032093525 — every one identical to the reference; the fusedtrain/opd_reverse_kl0.05341373383998871 equals the reference'srollout/opd_reverse_kl. grad_norm 0.9966776967048645 vs reference-record 0.9967363476753235 (Δ5.9e-5, within the reference's own 3.3e-4 repeat noise — comm/atomics, not the loss math). The train pass's log-probs are therefore bitwise equal to the skipped separate pass's, and the fused advantage term is bitwise equal to the reference's.
4.0 hRound 7 result (05:00–05:13, `/tmp/r7/time`): no gain — not promoted
NCCL_PROTO=LL128: train speedup 1.6622× (windows [43.22, 44.21, 42.25], mean 43.23) vs the current config's own train timing 1.6518× (mean ≈43.04) — within the 0.0226 reference spread. The AG/RS time is dominated by peer-wait spin (the skew flags), not LL bandwidth; NCCL's protocol choice was not the bottleneck.NCCL_PROTOstays out of the switches.
4.2 hRound 8 (2026-11-13 05:14–): thread-pool the per-param CPU Adam steps
The post-loop optimizer tail ≈6.5 s of actor_train:HybridDeviceOptimizer.step()(Megatron-LM/.../cpu_offloading/hybrid_optimizer.py:170-174) steps ~700 per-paramtorch.optim.AdamWinstances SEQUENTIALLY in the calling thread, each with its ownd2h_event.synchronize()and a step-post-hook that enqueues that param's H2D copy. With OMP=8 the Adam math is already fast; what's left is serial per-param launch/sync overhead.OPD_CPU_ADAM_THREADS=Nruns the per-optimizer steps in an N-thread pool: each param's update is independent and deterministic (same inputs, same op), and the H2D copies touch disjoint buffers whose order doesn't matter (each hook makes the compute stream wait on its own copy) → bitwise-identical parameters. N0 in principle; the record run decides.
4.2 hRound 8 result (05:14–05:27, `/tmp/r8/time`): train speedup 1.6981× — promoted
OPD_CPU_ADAM_THREADS=8: train speedup 1.6981× (windows [42.97, 42.56, 42.26], mean 42.60) vs the current config's own train timings 1.6518× (R5 check) and 1.6622× (R7) — every window faster than every previous window; the ~0.5 s gain exceeds the 0.0226 reference spread. Projected step speedup 1.2345× (from 1.2269×). Step-0 train-inference mismatch bitwise identical to the reference (0.008052160032093525). The pool parallelizes the serial per-param sync/launch overhead; the largest params' updates still bound the tail (Amdahl) — the hoped −3–4 s was optimistic, real gain ≈0.5–1 s. Promoted:/submission/switches.jsonnow carries 6 switches (addedOPD_CPU_ADAM_THREADS=8); committed in Megatron-LM as4f73434(tier N0 in principle — per-param updates deterministic and disjoint; the record run decides, else statistical N1).
4.7 hRound 9 (2026-11-13 05:50–, in flight): native Megatron GDN for the linear-attention layers
Offline profile (kernel→op→python-function attribution of the R6 train trace,/tmp/r8/attr.log,/tmp/r8/attr2.log): the train loop's elementwise pool is dominated by the **replicated HuggingFace GDN implementation** that the recipe uses for the 24 linear-attention layers (--qwen-gdn-implementationdefaults toreplicated): per window ≈0.6 s ofQwen3NextRMSNorm(modeling_qwen3_next.py:158, 3072 calls:floatupcast +pow+mean+rsqrt+type_as), 0.27 srepeat_interleave, 0.27 sfloatcasts, 0.14 scontiguous(flashqla branch), plus softplus/sigmoid per layer-mb — and the shapes[1,8704,4096]show each rank processes full microbatch sequences (CP2 does not shard this path) with unshardednn.Linearprojections (TP2 does not shard them either): the GDN compute (~4.5 s of kernels: fla chunk/conv/wy kernels + the norm soup) is duplicated 4× per rank. The loss-side[1024,124160]chunk soup (log-softmax+entropy, ≈0.9 s) is already response-chunked and buffer-reused — not this round. Change:OPD_GDN_NATIVE=1(in/tmp/r9/tree, a copy of the submission) setsargs.qwen_gdn_implementation = "megatron"→ the spec buildsQwen3_5NativeAttention(Megatron'sGatedDeltaNet: TP-sharded in_proj/out_proj, CP-aware cu_seqlens, fused TE norms) for the linear-attention layers; and redirectsargs.ref_load = args.hf_checkpoint, because the frozen torch_dist asset follows the replicated layout while the HF loader (hf_to_megatron/qwen3_5.py) converts the same weights into the native layout (itsnative_mapping+ fusedin_projmerge exist for exactly this path;megatron_to_hf/qwen3_5.pysplits the fused in_proj back for weight publication, so the engine sync keeps working). Same math, different sharding/reduction order → tier N1 (statistical gates); train-inference mismatch bound 2× reference. Also staged but NOT in this round's config:OPD_TP_COMM_OVERLAP(TE user-buffer GEMM+comm fusion for the 6.1 s AG/RS; risk: UB's fixed-shape buffers vs dynamic microbatch lengths) — next round if GDN-native lands or fails fast. ### Full check on the 6-switch config (05:29–05:53,/tmp/r8/check): PASS, N1, speedup 1.2408 The promoted submission config (6 switches incl.OPD_CPU_ADAM_THREADS=8) passed the full check: teacher 1.0262×, train 1.7071×, projected step speedup 1.2408 (from 1.2269). All gates PASS (record judged statistically, tier N1, as before). This is the submission's certified state.
7.0 hRound 14 (08:00–): teacher chunking 16384 — the 16.4 s rollout tail
Profile findings (from the R13 runs' teacher logs and clocks, no separate profile run): the teacher replay round is 17.5 s for ~530k tokens (64 samples × ~8.3k). The GEMM floor for 557k tokens through 27B TP4 is ≈ 10.7 s (2·27e9·557e3 FLOP / 4 GPUs / ~700 TFLOPS), so ~40% is non-GEMM. The chunk schedule at the promoted--chunked-prefill-size 8192: each ~8.3k-token request prefills as an 8192-token chunk plus a ~130-token remainder → **~128 chunk-forwards per round**, each with fixed costs (scheduler iteration, GDN state handling across the boundary, per-chunk logprob extraction, allreduce). The rollout window's teacher tail (16.4 s) equals this replay: all 64 samples finish generating together (lengths cluster at the 8192 cap), so the entire scoring pass lands after generation — halving the teacher's per-round cost takes ~4 s off every rollout window. Change (switch-only,OPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 16384): each request fits one chunk (~8.3k < 16384; the scheduler may pack a second request's head into the same forward) → ~64–70 forwards per round instead of ~128, halving the per-forward fixed costs. Numerics: every prefill chunking lands in the same re-batching noise family (the frozen spread pooled one-at-a-time / chunk-8192 / no-overlap / no-radix, all ≈ 0.0068 mean|d|, gate bound 2× = 0.0136; the startup controls accepted "another prefill chunking" at N1, and the promoted 8192 passed the full check). Screen:time --side teacherwith/tmp/r14/switches.json(7 switches, teacher arg → 16384) on the promoted submission tree →/tmp/r14/time. R14 screen result (/tmp/r14/time): teacher 1.0623× — windows 17.099 / 17.181 / 16.789 s vs the frozen reference 18.305/18.252/17.893, and ~0.4–0.6 s faster per round than the promoted 8192 config (17.489/17.794/17.285). Gain is far beyond the teacher spread (0.0011). Halving the chunk count (128 → ~64–70 forwards/round) removed ~0.5 s of per-forward fixed cost per round; the rest of the replay is GEMM + per-token kernels.OPD_TEACHER_SERVER_ARGSin/submission/switches.jsonupdated to--chunked-prefill-size 16384(8 switches now);check --quicklaunched 08:04 →/tmp/r14/quickto validate the teacher gate (and the rollout gate's teacher sub-check) before this counts — every chunking sits in the same ~0.007 noise family, and the startup controls accepted another prefill chunking at N1, but the gate verdict is what certifies it. **R14 correctness verdict:check --quick(/tmp/r14/quick, 08:37) = PASS, tier N1, all three sides** — the promoted 8-switch config (teacher--chunked-prefill-size 16384) passes the teacher replay gate, the rollout gate with its teacher sub-check (the rollout now carries 16384-chunked teacher logprobs), and the train record gate. Teacher side timed 1.0623× (/tmp/r14/time), rollout windows in the two quick-check runs stayed at 57.3–58.7 s (≈1.30–1.32×), train unchanged. Projected step speedup for the promoted config ≈ 1.44× (1/(1 − 0.4787·(1−1/1.317) − 0.4621·(1−1/1.712)), teacher term left out).
7.6 hRound 15 (08:37–): profile the teacher's scoring pass (the remaining rollout tail)
The rollout window is now ~57.7 s ≈ 41 s GPU-bound decode + ~16 s teacher tail, and the teacher replay (all-at-once, 64 × ~8.3k tokens) is ~17.0 s at the promoted 16384 chunking. The GEMM floor for 557k tokens through 27B TP4 is ≈ 10.7 s, so up to ~6 s is non-GEMM: per-forward scheduler work, the 24 GDN layers' chunked delta-rule Triton kernels, per-chunk logprob extraction (input_token_logprobsneeds a [tokens, 151k] log_softmax + gather per chunk), and TP4 collectives. Round protocol: profile first —profile --side teacheron the promoted submission (launched 08:37 →/tmp/r15/profile), then pick the lever from the kernel report (kernel classes, SM coverage, gaps between the ~64 chunk-forwards). R15 profile attempt 1 (/tmp/r15/profile): ERROR, no trace — judge/sglang profiler race. The judge's teacher profile posts/start_profilewithnum_steps: 64(bench/teacher_replay.py:58); sglang's profiler auto-stops and flushes after 64 counted forward steps. At the promoted 16384 chunking the profiled round has only ~64–70 chunk-forwards in total, so the auto-stop raced the end of the round: the last/generatereturned 08:39:43, the judge found no trace and sent SIGTERM at 08:39:45, killing the profiler before it flushed (trace/empty; "the profiler wrote no trace"). At 8192 chunking the round has ~128 forwards and the stop lands safely. The profile is diagnostic only (no gate, no timing), so retrying at 8192 (/tmp/r15/switches8192.json,/tmp/r15/profile2) — the kernel-class mix (GEMM MFU, GDN Triton share, logprob extraction) transfers to 16384. The promotion itself is unaffected: the timing and all gates at 16384 are already measured and passed. R15 profile attempt 2 (/tmp/r15/profile2, 8192 chunking, diagnostic): OK. Traces/tmp/r15/profile2/trace/1794559361.235472-TP-{0..3}.trace.json.gz, report/tmp/r15/profile2/kernel_report.md. Window 17496 ms, GPU busy 95.6% — the teacher is kernel-bound, not gap-bound (stream 96 gap total 772 ms). TP-0 kernel classes: - GEMMs (nvjet, SM cov 1.00): 22.7% + 18.4% + 10.3% + 0.5% ≈ 52% — well-tuned, at the expected MFU for 27B bf16 TP4 on ~8.2k-token chunks. -ncclDevKernel_AllReduce_Sum_bf16_RING_LL20.8% (8256 calls, median 420 µs, 24 blocks, SM cov 0.18) +AllGather_RING_LL3.8% (256 calls, median 2465 µs, 24 blocks) ≈ 24.6% in TP4 collectives, both in the LL protocol — the low-latency protocol whose 8-byte data + 8-byte flag packing halves wire bandwidth on the ~64 MB prefill allreduces (~8.2k tokens × 4096 hidden × bf16) and the ~620 MB logits all-gathers ([8.2k, 151936/4]). Nothing in the trees forces LL (grep: no NCCL_PROTO anywhere); it is NCCL 2.28's own tuner model on this Hopper interconnect. Simple protocol should roughly double the wire efficiency of both, and the ring reduction order is protocol-independent (each element's partial sums still accumulate in ring order from its chunk owner) → expected bitwise-neutral. - GDNFullyFusedDeltaRule6.3% (2928 calls, 24 blocks, SM cov 0.18 — under-filled by design, one CTA per head-block), causal-conv/norm helpers ≈ 2%. - fused_add_rmsnorm 4.1%, act_and_mul 1.2%, FA3 2.0%. - Logprob extraction:direct_copy/elementwise at 993280 blocks (7524 waves!) 2.3% + 2.0%,_row_logsumexp0.7% — the [tokens, 151k] logits copy + gather chain, ≈ 5%. - Repeated chains: nothing above 1% of kernel time (chain 1: GDN pre-fused QKV/gating/norm epilogue, 1.0%); the copies above are large single kernels, not chains. R15 lever:NCCL_PROTO=Simple(pure env switch, not inreserved_env) for the teacher's TP4 collectives. Ring AllReduce/AllGather/RS in LL vs Simple move the same elements in the same ring order (LL only adds flow-control flags at finer granularity), so the reduction result should be bitwise unchanged; only the wire efficiency changes. It applies to all my processes (teacher, engines, Megatron) — the engines' decode ARs use the flashinfer fused ones (not NCCL), and Megatron's AG/RS bucket (6.1 s/step, also RING_LL) may gain as well; if the train record drifts, the N1 statistical gate decides. Screen withtime --side teacherfirst. R15 result: two no-gain switches on the teacher's collectives. -NCCL_PROTO=Simple(env switch,/tmp/r15/switches_simple.json, screen/tmp/r15/time_simple): teacher windows [17.089, 17.161, 16.786] → speedup 1.0636 vs frozen — **identical to the promoted config's own 1.0623** (R14: [17.099, 17.181, 16.789]). No gain, no regression. ---enable-nccl-nvlson the teacher (server arg viaOPD_TEACHER_SERVER_ARGS,/tmp/r15/switches_nvls.json, screen/tmp/r15/time_nvls): [17.082, 17.154, 16.832] → 1.063 — identical again. Either NVLS did not engage (sglang setsNCCL_CUMEM_ENABLE=0when symm-mem is off) or NVLS does not beat ring here. - Cross-rank check (kernel_util on TP-1/2/3 traces): busy 95.7/95.7/95.8%, AR medians 399/402/392 µs vs TP-0's 420 — no straggler; the ~400 µs is real collective time everywhere. - Bandwidth accounting: T = 8192 (FA3 timing confirms), hidden 5120 → each per-layer AllReduce is ~84 MB; 420 µs → ~300 GB/s algorithmic. With LL's 8B-data/8B-flag packing that is ~600 GB/s physical, only possible on a Gen5-speed NV18 fabric; there Simple should roughly halve AR time and NVLS cut its traffic — the total lack of any timing effect suggests **neither switch actually engaged** (protocol/algo chosen elsewhere), or the collectives are capped by channel count (24) rather than protocol. - Not promoted./submission/switches.jsonunchanged (7 env vars, quick-check PASS N1). R16 plan: decisive check —profile --side teacherwithNCCL_PROTO=Simpleat 8192 chunking (avoids the profiler race): if the AR kernels are stillRING_LL, the env is being ignored and the follow-up is to find where the protocol is pinned; if they areRING_Simpleat ~400 µs, the collective is channel-capped and the lever is NCCL channel/thread tuning (NCCL_MAX_NCHANNELS,NCCL_NTHREADS) — which changes chunk→owner mapping (hence bf16 add order) and must pass the teacher gate statistically.
8.1 hRound 16 (09:00–09:40): why the teacher's collective switches do nothing — the tuner is already optimal; real collective sizes established
R16 question (from R15): why didNCCL_PROTO=Simpleand--enable-nccl-nvlschange nothing? -profile --side teacherwithNCCL_PROTO=Simpleat 8192 chunking (/tmp/r16/profile): kernels are stillRING_LL, identical counts/durations — but this was the wrong read: at 8192 chunking the per-layer AR is 84 MB and the tuner prefers LL there; the **promoted 16384 config's ARs are a different size** (below). - Diagnostic screen withNCCL_DEBUG=INFO, NCCL_DEBUG_SUBSYS=INIT,ENV,TUNING+--enable-nccl-nvls+NCCL_CUMEM_ENABLE=1(/tmp/r16/time_nvls2, timing 1.0631 = no change, debug-only config, not promoted). The teacher log answers everything: - The switches env does reach NCCL ("NCCL_CUMEM_ENABLE set by environment to 1", "NCCL_NVLS_ENABLE set by environment to 1"). - NVLS multicast support is available on all 4 devs (nChannels 16, chunk 131072). - The tuner's per-collective lines: the promoted config's per-layer AllReduce is 167772160 Bytes = [16384 tok, 5120 hid] bf16 = 160 MB, and the tuner already picksAlgo RING proto SIMPLEchannels 0..23 for it (and for the 254 MB logits AllGathers[2048, 62080], 255/round). NVLS was available and NOT chosen; myNCCL_PROTO=Simplescreen was a no-op because SIMPLE was already the choice at 160 MB. - So R15's two no-gain switches are explained: the collectives are at the tuner's floor (Ring-Simple, 24 channels, NVLS rejected by its model). No protocol/algo switch has headroom. Structural overlap (async-TP RS+AG pipelined into the GEMMs) would be a large rewrite — parked. - New lever found while reading the tuner lines: the teacher's logprob pipeline gathers the full[2048, 248320]logits (254 MB/rank shard, 1.02 GB gathered) per 2048-token chunk — 255 chunks/round — then does two ~1 GB HBM copies + a full-vocab row logsumexp + gather: ≈ 1.7 s of the 17 s round (AllGather 0.63 s + copies 0.71 s + gather 0.27 s + logsumexp 0.12 s from the R15 profile). Mathematically replaceable by per-shard logsumexp + tiny fp32logaddexpcombine + shard-local target gather (distributed softmax): the full-vocab gather and full-row copies vanish; expected save ≈ 1.4 s/round (teacher 1.063 → ~1.17, step ≈ +1.4%). Medium rewrite inlogits_processor/logprob_processor; numerics change is fp32-combine-order only (~1e-6 in log-space), far inside the teacher gate's noise family. Ranked as the next teacher-side lever. - Bigger lever for the next round (R17): the rollout's decode still leaks ~0.7 ms CPU per 5.0 ms step (GPU ~4.3 ms): ≈ 5.7 s of the 41 s generation per window, pure scheduling cost, zero numerics risk. R17 = re-profile the rollout side on the promoted config (the R13 CPU breakdown predates the sampler fix) and micro-optimize the top CPU item (R13 pointers:fill_from1.34 ms,try_small_copy1.17 ms,process_batch_result_decode1.1 ms). No code changes in R15/R16 (screens and profiles only); nothing to commit. Submission unchanged: 7-env switches, quick-check PASS N1, certified rollout 1.2457 / train 1.7119 / teacher 1.0636.
9.0 hRound 17 (09:40–10:10): sharded input-logprob processing — teacher tail 17.0 -> 15.6 s (PROMOTED)
Profile first (profile --side rollouton the promoted config,/tmp/r17/profile): the post-sampler-fix decode CPU path is flat — the apparent top item (event_loop_overlapself 2.52 ms/step) is mostly CUPTI's per-launch cost absorbed into the caller (458 kernels/step x ~6 us); no real CPU item exceeds ~0.5 ms/step. Decode micro-opt has poor ROI. The teacher's logits pipeline (quantified in R15/R16) was the round's hotspot instead: ~1.7 s per round in the input-logprob path — a 254 MB/rank full-vocab AllGather ([2048, 248320] x 255 chunks/round, 2.46 ms each) + two ~1 GB HBM copies + a full-vocab row logsumexp + target gather, all to read one target logit per row. Change (behindOPD_SHARDED_INPUT_LOGPROBS, default off; sglang commit381495a): shard-local input-logprob processing. Each TP rank reduces its own lm-head vocab shard to a per-row(max, logsumexp-max)pair; two tiny fp32 all-gathers combine the per-shard normalizers (m = max_r m_r; log_sum = log(sum_r exp(m_r + ls_r - m))) and the shard-owned target logits (owner =token // shard_vocab, masked sum picks the owned value); sampled rows (one per sequence — sglang always creates one even atmax_new_tokens=0, which made the first screen a no-op) are gathered per chunk into the same side-by-side layout the full gather produces. 199 insertions, 0 deletions; gathered path untouched when the switch is off; falls back for top-k/token-id logprobs, DP attention, attn-TP, softcapping, logit scaling, replicated lm_heads, tp 1. Pure-torch check: target logit bitwise identical, normalizer differs <= 2e-6 (fp32 combine order only). - Screen 1 (/tmp/r17/time_shard): identical timing — the path never engaged (sample rows always present). Fixed by gathering sample rows per chunk. - Screen 2 (/tmp/r17/time_shard2): teacher windows [15.635, 15.687, 15.385], mean 15.57 -> speedup 1.163 vs frozen (was 1.0636): the predicted ~1.44 s/round off the tail. -check --quick(/tmp/r17/quick_shard): PASS, tier N1, verdict equivalent — rollout (samples, token forcing, teacher replay on my tokens, mismatch), teacher replay gates, train record all pass. - Promoted:OPD_SHARDED_INPUT_LOGPROBS=1added to/submission/switches.json(8 env vars). - Rollout-window certification with the promoted config launched (/tmp/r17/time_rollout): expected ~57.7 -> ~56.3 s (ratio ~1.35, projected step ~1.46 from rollout 1.35 / train 1.71). Remaining ranked levers: (1) train side — the old (pre-R12) profile showed GPU busy only 62%: the CPU-offloaded Adam serialization is the biggest unexplored pool; re-profile train on the current config. (2) teacher GDNFullyFusedDeltaRule6.3% at 24 blocks (under-filled by design). (3) decode CPU micro-opts (flat, low ROI). Dead ends this round: decode CPU (flat),--enable-tp-lm-head-all-to-all(requires token-scattered hidden states, not our TP layout).