GPT-6.2 Sol 1.37×
Harness: Codex · switches: SLIME_SKIP_ZERO_WEIGHT_KL_LOSS=1 SLIME_DISABLE_ACTIVATION_RECOMPUTE=1 SLIME_OPD_REUSE_TRAIN_LOG_PROBS=1 SLIME_KEEP_TRAIN_MODEL_RESIDENT=1 SLIME_PUBLISH_RESIDENT_ACTOR_WEIGHTS=1 SLIME_RETAIN_LOGPROB_SOFTMAX=1 SLIME_STREAM_LOGPROB_SOFTMAX=1 SGLANG_ENABLE_SHARDED_INPUT_LOGPROBS=1
- Judged
- 1.37×
- Tier
- N1
- Self-best
- 1.31×
- Rounds
- 18
- Hours
- 10.0
- API cost
- $16.9
- Input tokens · 96.5% cached
- 78 M
- Output tokens
- 0.4 M
- Requests
- 551
§ ARaw trace
Raw work log19 sections
0.1 hRound 1 — zero-weight reference KL
- Started 2026-11-18 02:07:13 UTC (4-GPU lock). Unchanged rollout profile: PID 669806,/tmp/r1-profile.log,/tmp/r1/profile. - Unchanged timings: rollout 75.772 s (including a 16.391 s teacher tail), training 72.872 s, full loop 153.815 s. The startup training trace at/opt/researchboost/ws/runs/bootstrap/control_profile_train/kernel_report.mdshows BF16 ring all-gather (23.3%) and reduce-scatter (8.2%) as the leading communication kernels, with dense GEMMs much of the remainder. Its uninstrumented warm-up steps show 10.77–11.72 s in the frozen reference actor's log-probability pass, even though the reference KL coefficient is zero. This is separate from the teacher and OPD reverse KL. - The startup teacher profile shows about 53% in its top two GEMM classes and 24% in BF16 all-reduce and all-gather combined. - Fresh rollout profile completedokat 02:15:21 UTC. Its largest trace was 41.9% GPU-busy under profiling; leading GPU-time classes were the top three BF16 GEMMs (39.1% combined), fused all-reduce (10.1%), and recurrent GDN decode (9.4%). - Experimental tree/tmp/opd-r1disables only the zero-weight KL regularizer and its unweighted diagnostic, avoiding the frozen reference actor load/forward. Nonzero reference KL and the online teacher's OPD KL path are unchanged. Timing PID 683047,/tmp/r1-time.log, output/tmp/r1/time; lock acquired at 02:15:44 UTC. - Ended 02:30:50 UTC.timewasok: train windows 64.774, 63.578, 69.127 s;speedup_vs_frozen_reference1.1085× train and 1.0486× projected step. Window 3 includes a 10.53 s Triton first-use warning forcausal_conv1d_fwd_kernel; the report retains this window. No layout differences, and first-step train–rollout mismatch was 0.0080767. Intended N0; correctness replay deferred to the stacked round because the change only removes arithmetic with zero weight and its optional diagnostic.
0.4 hRound 2 — activation recomputation
- Profile first: Round 1's uninstrumented stage counters on the current best show the forward/backward/optimizer path at about 46 s of a 64–66 s train step (~71%); the remaining student log-probability forward is 10.7–12.0 s. The unchanged operator trace attributes about 31.5% of GPU kernel time to BF16 all-gather/reduce-scatter, operations repeated by full layer checkpointing along with every layer's projections. This round uses the live stage profile and unchanged operator trace rather than repeating the 50-minute train trace. - Retain activations instead of checkpointing every layer when sufficient memory is available. Precision, CP/TP, microbatches, weights and CPU-offloaded optimizer remain the same. Teacher and student rollout code are unchanged. - Started 2026-11-18 02:32:38 UTC (4-GPU lock). Experimental tree/tmp/opd-r2hasSLIME_DISABLE_ACTIVATION_RECOMPUTE=1atop Round 1. Timing PID 707192,/tmp/r2-time.log, output/tmp/r2/time. Train startup confirms both switches took effect. - Timing completed 02:46:12 UTC:ok, layout unchanged, train windows 58.582, 57.310, 60.480 s.speedup_vs_frozen_reference1.2401× train (1.1187× relative to the Round 1 measured ratios), projected step 1.1010×. Window 3 again has ~10.5 s first-use warning; result reported without subtracting it. First-step mismatch and loss were unchanged from Round 1. Combinedcheck --quick: PID 723507,/tmp/r2-quick.log,/tmp/r2/quick, locked at 02:46:34 UTC. - Ended 02:57:51 UTC./tmp/r2/quick/check.jsonPASS, N1 for both changes. Exact metadata passed; all four numerical statistics passed; update cosine 0.99441 (minimum 0.95330), train–inference mismatch exactly the reference value (0.0080767). Promoted to the submission. The stacked 1.1010× projection is not enough to clear the train-side frozen-reference spread threshold (1.1376×), so a timing-enabled check is deferred until gains accumulate.
0.7 hRound 3 — reuse the student training forward
- Preparing in a copy while Round 2 replays. Stage profiling of the Round 2 timed steps shows a separate student forward costing 10.9–12.9 s (~19–22% of training), andtrain/ppo_klis exactly zero in all observed windows. The GRPO objective takes one update from unchanged weights and uses no dropout or advantage whitening here; the same pre-update student log-probabilities can be detached from the training forward, while computing the reverse KL against the unchanged teacher for every token. Reuse is guarded to the applicable single-update GRPO configuration. No optimizer, batch, or sampling changes. - Experimental tree/tmp/opd-r3switches onSLIME_OPD_REUSE_TRAIN_LOG_PROBS. CPU verification: 89 policy-loss/argument-validation tests passed, plus a focused loss test with nonzero GRPO rewards and two sequences found exactly equal losses, gradients, and metrics with explicit vs deferred OPD KL. Timing queued after a passing Round 2 replay, PID 741878,/tmp/r3-time.log,/tmp/r3/time. - Started 2026-11-18 02:58:14 UTC (4-GPU lock); timing entered the training forward with no separate student pass. Quick replay queued after timing, PID 757217,/tmp/r3-quick.log,/tmp/r3/quick. - Timing completed 03:10:32 UTC:ok; windows 44.986, 45.176, 45.089 s, no first-use time and no layout differences.speedup_vs_frozen_reference1.6164× train (1.3034× vs Round 2 timing ratios); projected step 1.2205×. First-step train–inference mismatch 0.0080767. This is above the frozen-spread threshold, so a timing-enabled check is warranted if quick replay passes. - Quick replay locked at 03:10:56 UTC. Timing-enabledcheckqueued on PASS: PID 773393,/tmp/r3-check.log,/tmp/r3/check. - Ended 03:21:54 UTC./tmp/r3/quick/check.jsonPASS, N1; update cosine 0.99554, all statistical gates passed, mismatch exactly the reference value (0.0080767). Promoted. Timing-enabledcheckrunning separately from the round. - Timing-enabledchecklocked at 03:22:13 UTC; N1-only confirmation queued if that check passes at N1 (PID 793111,/tmp/r3-n1.log,/tmp/r3/n1). - Timing-enabled/tmp/r3/check/check.jsonPASS, N1, completed 03:44:21 UTC. Judge projected speedup 1.2161×; all training statistics passed (update cosine 0.99559) and mismatch ratio was 1.0. This is a train-side check with frozen-reference timing, not the final paired full-loop score. - N1-only confirmation/tmp/r3/n1/check.jsonPASS, N1, completed 03:55:04 UTC; first-step train–rollout mismatch again matched the reference exactly.
1.3 hRound 4 — training model residency
- Prepared in a copy during the Round 3 timing-enabled check. The current best's stage profile shows sleep around 4–5 s and wake around 1.5 s, together ~12–15% of a ~45 s train step. The unchanged rollout profile reports ~51 GiB free per GPU after teacher and student-engine graph capture; training's baseline between-step residency is ~33–37 GiB per GPU. -/tmp/opd-r4addsSLIME_KEEP_TRAIN_MODEL_RESIDENT=1: opt-in to keep trainer buffers resident and retain the existing rollout sleep, synchronous weight publication and CPU-offloaded FP32 Adam. Not promoted. Validate time and training precision before promotion; co-located headroom must also be considered. - Timing queued after Round 3's N1 repeat (PID 812522,/tmp/r4-time.log,/tmp/r4/time); quick check queued after valid timing (PID 812523,/tmp/r4-quick.log,/tmp/r4/quick). - Locked at 03:55:27 UTC. Firsttimefinished ERROR at 03:57:08 UTC: disabling train offload left its no-CPU-backup allocation flags set, so the buffer allocator tried to enter a memory-saver region without a preload hook. No speedup or quick result. The experimental switch now clears those offload-only flags too. Retry timing: PID 846606,/tmp/r4-time-retry.log,/tmp/r4/time-retry; quick queued on valid retry: PID 846607,/tmp/r4-quick-retry.log,/tmp/r4/quick-retry. - Retry locked at 03:58:25 UTC and finished 04:09:24 UTC:timeok, train windows 37.191, 37.206, 37.223 s, no first-use time or layout differences.speedup_vs_frozen_reference1.9586× training (1.2117× relative to Round 3's measured ratio), 1.3019× projected. First-step train–rollout mismatch remains 0.0080767. Quick retry locked 04:09:45 UTC. - Ended 04:19:09 UTC./tmp/r4/quick-retry/check.jsonPASS, N1; first-step train–rollout mismatch exactly matches the reference. Timing and training precision are verified; residency is not submitted until the colocation check passes. - A two-iteration colocation correctness smoke using the supplied launcher and GPU lock is queued after the quick retry (PID 873106, output/tmp/r4/colocation-smoke, wrapper log/tmp/r4-smoke.log). It validates teacher/student/actor coexistence and synchronous weight publication; its clocks are not used as a timing result. - Colocation check finished successfully at 04:31 UTC: both rollouts scored, both optimizer updates completed, and both engines accepted weight versions through 3. Train–rollout mismatches were 0.0081662 and 0.0078980; no CUDA OOM. The engine warnings were startup/freeze_gcconnection retries, not failed generation. Promoted residency as3079decbafter this check.
2.2 hRound 5 — partition duplicate GDN compute
- Prep and profile: Round 4's current candidate train stage counters showactor_trainat 35.3–35.4 of 37.1–37.4 s (~95%). The unchanged train trace's leading classes include 23.3% BF16 all-gather, 8.2% BF16 reduce-scatter and 4.3% send/receive; GDN's reference wrapper all-gathers activations across TP and CP and runs the full sequence and all heads on four ranks. Causal convolution and GDN chunk kernels are also visible in the trace. Use the current stage profile and archived operator trace because an instrumented train run took 44 minutes before report generation. - The/tmp/opd-r5experiment enablesSLIME_QWEN35_SHARD_GDN_ACTIVATIONS=1: project local tokens, exchange token shards for head shards through all-to-all, compute the same packed-sequence GDN per head and return to the original token layout. Weight and optimizer-state names, layouts, and dtypes are unchanged. Megatron reduces replicated GDN parameter gradients across TP in addition to its existing CP reduction. - CPU stand-in passed four-rank Gloo forward, input-gradient, and replicated parameter-gradient comparisons with unequal packed sequence lengths; 30 existing linear-attention tests passed. Judgetimequeued after the Round 4 colocation smoke: PID 879820,/tmp/r5-time.log,/tmp/r5/time; quick queued on valid time: PID 879821,/tmp/r5-quick.log,/tmp/r5/quick. Experimental only. - Started 2026-11-18 04:31:35 UTC (GPU lock fortime). -timecompleted 04:41:25 UTC, ok: 31.824, 29.061, 29.133 s, no timed first use and no layout differences.speedup_vs_frozen_reference2.4320× training (1.2417× relative to Round 4), 1.3869× projected. First-step train–rollout mismatch 0.0080407. A timing-enabled check is queued on a faster, N1-passing quick replay (PID 916232,/tmp/r5-check.log,/tmp/r5/check). - Ended 04:50:26 UTC. Quick replay locked at 04:41:48;/tmp/r5/quick/check.jsonINCORRECT at the training statistical gate. First-step gradient norm 0.962348 vs 0.948656 (allowed paired delta 0.000898); update cosine 0.934000 vs minimum 0.953303. Update norm, second moment and loss passed. A toy mathematical equivalence check does not establish N1 on the actual BF16 kernels. The experiment was rejected, not submitted; its conditional timing-enabled check did not run.
2.6 hRound 6 — teacher prefill collectives
- Started 2026-11-18 04:51:09 UTC (GPU lock for profile). Profile PID 916233,/tmp/r6-profile.log,/tmp/r6/profile, using the current residency snapshot/tmp/opd-r4. Finished 04:55:28 UTC, ok. GPU-busy 92.3% in this trace; top two GEMM classes 52.3% combined; BF16 ring all-reduce 20.4% (206.6 us median, 24 CTAs), ring all-gather 3.6%. The hardware reports NV18 links between all pairs. - Try the existing SGLang symmetric-memory all-reduce for the teacher only, preserving BF16, TP4 and chunking. Standalone teacher experiment/tmp/opd-r6-teacher/switches.jsonsetsOPD_TEACHER_SERVER_ARGS=--enable-torch-symm-mem. Timing PID 934179,/tmp/r6-time.log,/tmp/r6/time; quick replay queued on meaningful gain, PID 934180,/tmp/r6-quick.log,/tmp/r6/quick. Not submitted. - Ended 2026-11-18 04:59:55 UTC./tmp/r6/time/time.jsonwasok: 18.286, 18.568, 18.382 s, 0.9848× teacher (speedup_vs_frozen_reference), 0.9984× projected. No gain; the quick replay was not run and the switch was not promoted.
3.0 hRound 7 — weight publication without a CPU snapshot
- Started 2026-11-18 05:00:42 UTC (GPU lock for timing). The current Round 4 stage profile shows ~35.4 s inactor_trainout of ~37.2 s per train-only window; the remaining post-train path takes a pinned CPU snapshot of all actor weights. In the colocated path the weight updater also restores that snapshot for its GPU collectives. These copies are redundant when the actor buffers are resident and the actor never switches to a reference/teacher checkpoint. - Experiment/tmp/opd-r7enablesSLIME_PUBLISH_RESIDENT_ACTOR_WEIGHTS=1for this case only, sourcing publication from detached live parameters. The actor updates and the synchronous rollout barrier are unchanged. CPU source-adapter smoke passed; timing PID 937076,/tmp/r7-time.log,/tmp/r7/time; conditional quick PID 937077,/tmp/r7-quick.log,/tmp/r7/quick. Not submitted. A colocated publication smoke is required if the train timing and quick check justify promotion. - Colocated smoke queued after a passing quick check: PID 951995,/tmp/r7-smoke.log,/tmp/r7/colocation-smoke(two iterations, verification only). Timing-enabled check queued after the smoke: PID 951996,/tmp/best-check.log,/tmp/best-check. It checks the Round 7 tree if the smoke passes and the verified Round 4 tree otherwise. -/tmp/r7/time/time.jsoncompleted 2026-11-18 05:11:22 UTC,ok: 36.555, 36.591, 36.537 s; 1.9932× train (speedup_vs_frozen_reference, 1.0177× relative to Round 4) and 1.3091× projected. Layout unchanged and zero timed first use. Quick check took the lock at 05:11:41 UTC. -/tmp/r7/quick/check.jsonfinished 05:21:02 UTC: PASS, N1. All exact metadata and four statistics passed, cosine 0.99519 (minimum 0.95330), mismatch exactly the reference (0.0080767). Colocation smoke began at about 05:21 UTC; this experiment is still not submitted. - Ended 2026-11-18 05:30:10 UTC. The colocation smoke exited 0 after scoring both rollouts, taking both optimizer steps, and publishing both updates without OOM. Its train–rollout mismatches were 0.0080222 and 0.0079789. Promoted the tested patch and switches asf6a28d18; the timing-enabled check is running on this stack in/tmp/best-check.
3.0 hTiming-enabled check of Round 7
- Completed 2026-11-18 05:49:27 UTC: /tmp/best-check/check.json PASS, N1. Train windows 36.577, 36.644, 36.599 s, 1.9907× train against the frozen reference and 1.3085× projected step (benefit bar 1.1376×). No parallel-layout differences or timed first use; first-step train–inference mismatch is exactly the reference. The train-only check is not the final paired full-loop measurement.3.8 hRound 8 — teacher prefill batching
- Started 2026-11-18 05:50:04 UTC (GPU lock for teacher profile). Profiling the current verified stack in/tmp/opd-r7, PID 971885,/tmp/r8-profile.log,/tmp/r8/profile. No experiment or submission switch has changed yet. - Profile completed 05:54:42 UTC,ok. The GPU was busy 91.6% of the trace; the top two BF16 GEMM classes account for 52.2% of kernel time, BF16 ring all-reduce 20.5%, and ring all-gather 3.6%. Larger prefill batches may amortize projection weight reads and per-forward scheduling; only the chunk size changes. - Teacher-only experiment/tmp/opd-r8-teacher/switches.jsonsetsOPD_TEACHER_SERVER_ARGS=--chunked-prefill-size 8192. Timing PID 1021581,/tmp/r8-time.log,/tmp/r8/time. Quick check of both the teacher replay and online rollout queued if teacher speedup exceeds 1.02×: PID 1021582,/tmp/r8-quick.log,/tmp/r8/quick. The bootstrap teacher-only chunking control passed, but this change needs the online gate before promotion. -/tmp/r8/time/time.jsoncompleted 05:58:53 UTC,ok: 17.611, 17.962, 17.666 s, 1.0218× teacher (speedup_vs_frozen_reference) and 1.0023× projected step. The projected gain was too small to justify the roughly 30-minute online check. Ended 06:01:13 UTC after canceling that check during priming; itsERRORwas the deliberate SIGTERM, not a numerics verdict. Teacher settings remain unchanged in the submission.
3.9 hRound 9 — retain the policy log-probability softmax
- Started 2026-11-18 06:01:38 UTC (GPU lock for timing). First-level profile of the verified Round 7 stack during its timing-enabled check shows ~35.3 of 36.6 s inactor_train(~96%). In the archived train trace,_ChunkedLogProbEntropyBackwardruns 64 times and occupies 1.70 s of CPU profiler ranges (not a measured GPU gain). Its chunked path recomputes the full TP vocabulary softmax in backward to bound memory. Reusing the existing unchunked path retains the softmax and removes these repeated reductions. - Test copy/tmp/opd-r9addsSLIME_RETAIN_LOGPROB_SOFTMAX=1for gradient-carrying, unmasked logits with metric-only entropy. The loss, entropy metric, precision, response extraction and TP reductions are preserved. All 89 existing loss and argument CPU tests passed; a focused test of the switched path with two packed samples matched response log-probs, entropy and input gradients. No unverified change is submitted. - Timing PID 1027087,/tmp/r9-time.log,/tmp/r9/time; quick PID 1027088,/tmp/r9-quick.log,/tmp/r9/quick, queued on a further train gain of at least 1%. A colocated smoke will be needed to verify activation-memory headroom before promotion. - Colocation smoke queued on a passing quick check: PID 1035682,/tmp/r9-smoke.log,/tmp/r9/colocation-smoke(verification only). Follow-up PID 1037840,/tmp/r9-followup.log, runs the timing-enabled check if the smoke succeeds, or a statistical-only repeat of the verified Round 7 stack otherwise. - Timing completed 06:12:14 UTC,ok: 36.103, 36.044, 36.047 s, zero first use and no layout changes./tmp/r9/time/time.jsonreports 2.0206× train (speedup_vs_frozen_reference, 1.0137× relative to Round 7) and 1.3146× projected. Quick check took the lock at 06:12:40 UTC. The submission is unchanged until correctness and colocation pass. - Quick check completed at 06:21:46 UTC:/tmp/r9/quick/check.jsonPASS N1. Exact metadata and all numerical statistics passed, including update cosine 0.99413 (minimum 0.95330) and mismatch exactly the reference (0.0080767). The colocation smoke is running with the same patch. - Ended 06:30:11 UTC. The colocation smoke exited 0 after two scored rollouts, two optimizer steps, and publication through weight version 3 on both engines. No OOM; the step mismatches were 0.0080570 and 0.0080124. Follow-up timing-enabled check took the lock at 06:30:27 UTC (/tmp/r9/check); Round 7 remains submitted until that check finishes. - Follow-up/tmp/r9/check/check.jsoncompleted 06:49:44 UTC: PASS N1. Train windows 36.144, 35.998, 35.969 s, zero timed first use and no layout differences; 2.0221× train and 1.3149× projected step. Update cosine 0.99489, first-step mismatch exactly the reference. Promoted the exact tested patch as721d452band enabled its switch.
4.4 hRound 10 — packed GDN decode occupancy
- Queued a fresh decode profile of whichever of Round 9 and Round 7 has a passing timing-enabled check, after the current check releases the GPUs. PID 1085998, log/tmp/r10-profile.log, output/tmp/r10/profile. No candidate change yet; choose the hotspot after the report. - Started at 2026-11-18 06:50:01 UTC (GPU lock). Profiling the passing Round 9 stack. - Profile finished 06:57:24 UTC,ok. Largest trace GPU busy 37.6% under profiling; top three BF16 GEMMs 40.6% of kernel time, packed recurrent GDN 9.4%, fused FlashInfer all-reduce 8.8%. The recurrent kernel uses 205 registers/thread at a one-warp CTA, with 2048 blocks and 20.2 µs median. - Test copy/tmp/opd-r10usesSGLANG_GDN_PACKED_DECODE_WIDE_CTA=1to spread the same 128-by-32 packed state tile over four warps for 128-dimensional heads. No changes to cache/state dtype, recurrence, sampling, prefill or parallel layout. A direct 12-step kernel test with BF16 and FP32 state, padded slots and batches 1/8/32 passed tolerance (not bitwise). Timing PID 1122449, log/tmp/r10-time.log, output/tmp/r10/time; rollout side only. The experiment is not submitted. - Ended 07:14:58 UTC./tmp/r10/time/time.jsonwasok: 76.143, 76.166, 76.488 s, 0.9935× rollout (speedup_vs_frozen_reference), 0.9968× projected for this change alone. No gain; no quick check or promotion.
5.2 hRound 11 — training collective protocol
- Started a fresh train profile on the verified Round 9 submission. PID 1152132, log/tmp/r11-profile.log, output/tmp/r11/profile. No further change submitted. - GPU lock acquired at 2026-11-18 07:16:42 UTC; 30-minute round ends at 07:46:42 UTC. - Profile completed 07:35:45 UTC,ok: GPU busy 71.4% under profiling. Ring all-gather 13.7% of GPU kernel time, ring BF16 reduce-scatter 6.1%, top four BF16 GEMMs 26.8% combined; CP send/receive is only 0.9%. Gather and reduce-scatter run LL with only 24 CTAs. - Test copy/tmp/opd-r11usesSLIME_TRAIN_NCCL_LL128=1to select the existing LL128 transfer protocol in the trainer's Ray worker environments, leaving the topology, algorithms, optimization and serving workers untouched. Timing PID 1172832, log/tmp/r11-time.log, output/tmp/r11/time. No production change. - Round budget ended 07:46:42 UTC with the timing run in flight. Not promoted; record the pending timing result when it writestime.json. - Timing finished 07:48:21 UTC:/tmp/r11/time/time.jsonwasok, train windows 36.215, 36.267, 36.183 s, 2.0119× train vs frozen reference (speedup_vs_frozen_reference), 1.3129× projected. Slight regression from Round 9; no quick check or promotion.
5.7 hRound 12 — joint TP/CP activation gather
- Profile first: the still-current best version's fresh Round 11 trace shows all-gather (13.7%) and BF16 reduce-scatter (6.1%) dominate communication. The replicated GDN wrapper performs both a TP and a CP activation gather in each of 24 layers per microbatch; in backward it reduce-scatters the whole CP shard and then discards the other TP token shard. This round targets these same input activations, not the GDN arithmetic. - Test copy/tmp/opd-r12usesSLIME_GDN_JOINT_ACTIVATION_GATHER=1. A single joint TP/CP all-gather feeds the unchanged CP token reordering, GDN and output scatter. Backward selects the same TP shard before CP reduce-scatter, preserving the CP sum for every retained element. Slime initializes both supported rank layouts with TP fastest and CP next. The LL128 trial is not included. - Four-rank GPU correctness test took the lock at about 07:51 UTC: forward activations and input gradients matched bitwise for BF16 and FP32, including noncontiguous output gradients. The earlier Gloo stand-in failed inside the old subgroup scatter implementation, before making the comparison; the real NCCL comparison passed. All 30 existing CP/GDN CPU tests passed after putting the opt-in check in forward rather than relying on a cached attribute absent from constructor-bypassing tests. - Timing PID 1190586, log/tmp/r12-time.log, output/tmp/r12/time. Quick PID 1190587, log/tmp/r12-quick.log, output/tmp/r12/quick, queued only if train timing improves by at least 1% relative to Round 9. Not submitted. - Ended 08:04:07 UTC./tmp/r12/time/time.jsonwasok: 37.192, 36.624, 36.558 s and zero timed first use; 1.9807× train (speedup_vs_frozen_reference), 1.3065× projected. The joint gather was a slight regression, despite bitwise equivalence of the communication primitive. No quick check or promotion.
6.0 hRound 13 — larger fused prefill collectives
- Fresh teacher profile on the verified Round 9 stack. PID 1207131, log/tmp/r13-profile.log, output/tmp/r13/profile. No experiment or production switch changed yet. - Started 2026-11-18 08:05:41 UTC (GPU lock). Profile finished 08:09:59 UTC,ok: GPU busy 92.3%, top two BF16 GEMMs 52.2%; BF16 ring all-reduce 20.5%, unfused-with-collective residual RMSNorm 4.0%, ring all-gather 3.6%. The fusion cutoff of 2048 disables the existing FlashInfer all-reduce + residual + RMSNorm for full 4096-token teacher prefill chunks. - Test copy/tmp/opd-r13setsSGLANG_FLASHINFER_ALLREDUCE_MAX_TOKENS=4096. It raises only the fusion eligibility cutoff and its preallocated workspace size; request scheduling, model precision and topology stay fixed. Timing PID 1210034, log/tmp/r13-time.log, output/tmp/r13/time; teacher side only. The submission is unchanged. - Ended 08:14:46 UTC./tmp/r13/time/time.jsonwasok: 18.588, 18.864, 18.681 s, all TP workers used the 4096-token fusion workspace. 0.9691× teacher (speedup_vs_frozen_reference) and 0.9966× projected; no gain, quick check or promotion.
6.3 hRound 14 — sharded vocabulary normalization for scoring
- Profile first: the unchanged best version's Round 13 teacher trace shows 128 ring all-gathers (3.6% of kernel time, 2.38 ms median), paired full-vocab copy and index-gather kernels (4.1% combined), and row log-sum-exp (0.7%). Two logit chunks per prefill consume these large gather/copy operations; only one chosen input log-probability per token is requested. - Test copy/tmp/opd-r14enablesSGLANG_ENABLE_SHARDED_INPUT_LOGPROBS=1for unpadded, homogeneous TP vocabulary shards in prefill-only scoring, with no top-k or alternative-token result requested. The same BF16 LM-head projections and FP32 normalizer run per shard. Gather per-row max, log-sum-exp and target logit, merge in FP32, and gather the few sampled rows to keep output bookkeeping unchanged. Actual generation, LoRA, padded vocabularies, DP attention, and normal score modes fall back. Not submitted. - Started 08:28:45 UTC (GPU lock for correctness test). Four 62,080-element vocabulary shards with BF16 logits matched a full FP32 log-softmax within 1.91e-6 on synthetic targets from every rank (including shard boundaries). Timing PID 1213242, log/tmp/r14-time.log, output/tmp/r14/time. Quick PID 1213243, log/tmp/r14-quick.log, output/tmp/r14/quick, conditional on at least 4% teacher gain. - Timing finished 08:33:04 UTC,ok: 16.597, 16.900, 16.685 s without layout problems./tmp/r14/time/time.jsonreports 1.0840× teacher (speedup_vs_frozen_reference), 1.0083× projected for the teacher change alone. The all-side quick check took the GPU lock at 08:33:31 UTC; the submitted stack remains unchanged. - Online rollout timing queued on a passing all-side quick check: PID 1249318, log/tmp/r14-online.log, output/tmp/r14/time-online. It tests whether the teacher replay gain is also visible with the student engines online, before any promotion. - Round budget ended at 08:58:45 UTC, with the all-side check replaying the candidate rollout through reference training for mismatch. No promotion while validation is unfinished; queued follow-up runs only on a PASS. - All-side/tmp/r14/quick/check.jsonfinished 09:13:01 UTC: PASS N1, including forced student scoring, the rollout's online teacher, teacher replay, and the training record. On 2,069,553 fixture response tokens the standalone teacher mean absolute delta was 7.63e-8, maximum 1.91e-6, with zero tail events. Rollout mismatch ratio was 1.003; frozen training mismatch was exactly the reference. Online timing took the lock at 09:13:31 UTC. - Online timing finished 09:28:49 UTC,ok: 75.704, 75.144, 75.323 s./tmp/r14/time-online/time.jsonwas 1.0051× rollout vs frozen reference, a near-noise gain online (reference spread 0.65%) alongside the clear 1.0840× replay gain. All three numerical sides passed; promoted the byte-for-byte tested serving patch as1725944with its switch enabled.
6.9 hRound 15 — adjacent training microbatches
- Profile first: the verified training path's Round 11 trace is unchanged by the pending teacher experiment. It has 64 microbatches, 319,330 kernel launches (109,526 in short classes), 71.4% GPU busy under profiling, and 19.8% of kernel time in BF16 gather/reduce-scatter. Combining neighboring bins can amortize per-microbatch work without moving any sample across an update or changing the flattened sample order. - Separate copy/tmp/opd-r15enablesSLIME_COALESCE_ACTOR_MICROBATCHES=1in the dynamic, single-update, DP1/PP1 actor without critic or routing replay. It only concatenates consecutive pairs of scheduled microbatches and uses the resulting accumulation count. Activation memory and BF16 reduction order must be validated; no promotion on a timing-only result. - Timing queued after the Round 14 checks finish: PID 1264742, log/tmp/r15-time.log, output/tmp/r15/time. Quick PID 1264743, log/tmp/r15-quick.log, output/tmp/r15/quick, conditional on a 3% further train gain. A passing experiment would also need to demonstrate colocation memory headroom. - Started 2026-11-18 09:29:08 UTC (GPU lock for timing). The teacher is absent in the train-only arm; the training-only change is measured separately from the now-promoted sharded teacher scoring. - Timing finished 09:39:26 UTC,ok: 33.994, 33.960, 33.763 s, 2.1493× train (speedup_vs_frozen_reference, about 1.064× vs Round 9), 1.3393× projected without the serving gain./tmp/r15/time/time.json. Quick validation is running. Two colocated iterations with both optimizations are queued on a PASS: PID 1337458,/tmp/r15-smoke.log,/tmp/r15/colocation-smoke. This smoke is correctness and memory validation only. - Timing-enabled all-side check queued after the smoke result: PID 1345077,/tmp/latest-check.log,/tmp/latest-check. It checks the combined Round 15 tree only if both R15 validations pass, otherwise the already verified Round 14 stack. This is the judge's screening check, not the final paired full-loop evaluation. - Quick check completed 09:48:51 UTC: PASS N1 (/tmp/r15/quick/check.json), mismatch ratio 1.0048. Gradient norm 0.9478105 vs 0.9486560 (paired tolerance 0.0008976), and update cosine 0.95600 vs minimum 0.95330, passed narrowly. Colocation smoke is running. Require a fresh train-only N1 replay before any promotion; PID 1355839,/tmp/r15-repeat.log,/tmp/r15/repeat-n1, queued on successful smoke. Replaced the still-waiting full check wrapper with PID 1355840 to also require that repeat. The current submission remains Round 14. - Ended 2026-11-18 09:54:34 UTC: colocated smoke exited 1. The first rollout was scored correctly, but the first train step could not allocate a 3.91 GiB FP32 vocabulary-softmax workspace with the teacher resident (2.67 GiB device memory free; not allocator fragmentation). The N1 repeat was skipped. No promotion: an isolated training gain and a narrow precision pass don’t establish that the colocated job runs. - The timing-enabled all-side check selected the unchanged Round 14 stack and took the first GPU lock at 09:55:08 UTC. PID 1355840, log/tmp/latest-check.log, output/tmp/latest-check/check.jsonwhen finished.
8.0 hRound 16 — tiled vocabulary softmax for training
- Profile first: attributed the verified training stack’s Round 11 trace through CPU ranges and CUDA external IDs. Vocabulary softmax forward/backward accounted for 2,475 kernels and 638.28 ms (1.97% of kernel durations): entropy GEMV 130.22 ms and dense FP32 elementwise passes were the largest components. Round 15’s colocated failure also located its allocation in this exact path. - Prototype in/tmp/opd-r16, opt-inSLIME_STREAM_LOGPROB_SOFTMAX=1, no coalescing. Tile reductions compute softmax statistics without a dense probability buffer; backward reads the original logits and reconstructs the same log-probability gradient directly. Metric-only entropy, target ownership, and stable FP32 normalization across TP are preserved. Masked or entropy-gradient modes fall back. Not submitted. - Triton interpreter passed single-rank and two-rank Gloo tests on uneven tiles and the production 124,160-entry vocab shard, including targets at shard boundaries. GPU parity, judge timing, and a conditional quick check remain pending. The verified Round 14 stack is untouched. - Detached runner PID 1387213,/tmp/r16-run.log, waits for the current all-side check to pass, then takes the GPU lock for parity before judgetimeat/tmp/r16/time. Quick check at/tmp/r16/quickruns only on valid timing at least 0.5% faster than the verified trainer. Round clock has not started. - Before timing, removed the same path’s dense BF16-to-FP32 transport cast using Megatron’s existingfp32_output=Falsefor eligible policy-loss forwards. The tile loads promote BF16 logits to FP32 exactly as before, and the gradient is stored at the original model-output dtype. CPU interpreter parity passed fp32 and bf16 inputs, using the reference’s explicit upcast for the latter. No production change. - The 50 existing PPO log-probability, CP-loss and policy-loss CPU tests passed. GPU round started about 10:45:15 UTC. The standalone test initially stopped at an entropy-only 4.58e-5 difference against the reference’s FP32 GEMV at production vocab size. Log-prob parity passed there. Retry uses an FP64 entropy oracle to distinguish the two reductions (keeps tight 1e-5 on the candidate and the repo GPU entropy-test tolerance 1e-4 on the reference); this doesn’t change any judge tolerance. Candidate source unchanged since before the first GPU test. Retry PID 1443547,/tmp/r16-run-retry.log, then the same judge timing and conditional quick validation. - Two-rank CUDA parity passed all shapes and both dtypes. With the production shard width the new entropy was within 1.33e-6 of FP64 vs 4.71e-5 for reference FP32 GEMV. Timing locked 10:47:18 UTC and completed 10:57:46:/tmp/r16/time/time.jsonok, 35.524, 35.405, 35.286 s, 2.0582× train (speedup_vs_frozen_reference, 1.0165× vs latest verified stack) and 1.3220× projected without serving changes. No timed first use or layout differences. Quick check took the lock at 10:58:03 UTC. - Two-iteration colocation smoke queued on a quick PASS for the exact candidate plus the verified sharded teacher, PID 1469060,/tmp/r16-smoke.logand/tmp/r16/colocation-smoke. The copy is/tmp/opd-r16-full; this run checks memory and weight publication only. Round 14 stays submitted until the precision and smoke results are available. - The judge’s/tmp/r16/quick/check.jsoncompleted 11:07:06 UTC, PASS N1. First gradient norm 0.9486416 vs 0.9486560, update cosine 0.99487 vs minimum 0.95330, and mismatch ratio 1.0000. The colocated smoke is starting; no promotion yet. - Round budget ended 2026-11-18 11:15:15 UTC. Timing and train correctness passed, but the two colocated steps are still in progress; no submission change. The smoke result will determine promotion. - Colocation smoke exited 0 at 2026-11-18 11:15:46 UTC. Both optimizer updates and all weight publications succeeded, both engines accepted weight version 3, no OOM, and train–rollout mismatch was 0.0081385 and 0.0080859. Promoted the byte-for-byte tested training patch as91d7a250and enabled its switch.
9.2 hRound 17 — larger microbatches with the tiled loss
- Fresh judge train profile queued after colocation, PID 1491575, log/tmp/r17-profile.log, output/tmp/r17/profile. It uses the tiled training stack only if its colocated smoke passes, otherwise the unchanged verified trainer. No experiment has been applied; choose the hotspot from the new trace. Round clock starts at the GPU lock. - Started 2026-11-18 11:16:22 UTC (GPU lock for the newly verified tiled-logit stack). Round budget ends 11:46:22 UTC. - Fresh profile finished 11:35:27 UTC,ok(/tmp/r17/profile/kernel_report.md). GPU busy 63.4% under profiling; 317,373 launches including 108,322 short ones (6.6% of kernel durations). Ring all-gather 9.3%, BF16 reduce-scatter 4.0%, top four GEMM classes 29.8%. Consolidate the per-microbatch overhead without changing sample order or update count. - Re-test the byte-for-byte Round 15 scheduling patch in/tmp/opd-r17(SLIME_COALESCE_ACTOR_MICROBATCHES=1), now on the verified tiled FP32 normalizer with BF16 logit storage. It removes the large forward workspace that failed in Round 15. Timing PID 1512166,/tmp/r17-run.log,/tmp/r17/time; train quick/tmp/r17/quickfollows on valid timing with at least 2% further gain. A passing colocation smoke and sufficient N1 margin remain required. No production change. - On a quick PASS, the exact combined stack in/tmp/opd-r17-fullruns the two-iteration colocation smoke (PID 1519387, log/tmp/r17-smoke.log, output/tmp/r17/colocation-smoke). If that passes but quick precision is still close to the gradient-norm or update-direction limit, PID 1519388 repeats the judge’s N1 gate at/tmp/r17/repeat-n1(log/tmp/r17-repeat.log). No promotion until those conditions are satisfied. - Ended 2026-11-18 11:46:22 UTC at the 30-minute cap. The timing run is still finishing. No promotion; record judge timing, precision, and colocation results when available. - Timing completed 11:46:55 UTC (/tmp/r17/time/time.json): ok, train windows 33.224, 33.186, 33.239 s, 2.1939×speedup_vs_frozen_reference(1.0659× vs R16), 1.3474× train-only projected step, with zero timed compilation and no layout issues. First-step mismatch 0.0081153. Quick check locked at 11:47:13 UTC; candidate still isolated. - Quick check completed 11:56:11 UTC (/tmp/r17/quick/check.json): PASS N1. Gradient norm 0.9478502 vs 0.9486560 (allowed delta 0.0008976); cosine 0.95800 vs minimum 0.95330. Both pass, with enough proximity to require the queued repeat. Colocation smoke starts first. Keep R16 submitted. - Colocation failed at 2026-11-18 12:03:35 UTC. The tiled loss removed the earlier FP32 forward allocation, but backward could not allocate its 2.01 GiB BF16 logit gradient: 766 MiB of GPU memory free with 1.05 GiB unallocated in the allocator. Teacher used 40.22 GiB on that rank. The stat repeat was skipped. Reject Round 17 despite better timing and a passing isolated N1 record; no coalescing in the submission.
9.7 hRound 18 — next rollout hotspot
- Queued a fresh rollout profile after Round 17’s pending validation, PID 1528455,/tmp/r18-profile.log,/tmp/r18/profile. It profiles the R17 variant only if the pending colocation and required N1 validation pass; otherwise it profiles the verified R16 stack. No change chosen before profiling. - Started 2026-11-18 12:04:24 UTC (GPU lock for profile), using the verified R16 stack after the larger-microbatch colocation failure. Cap at 12:34:24 UTC.