GPT-6.2 Sol 1.86×

Harness: Codex · switches: BAGEL_PACKED_FLASH_ATTENTION=1 BAGEL_DECODER_KEEP_ACTIVATIONS=28 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True BAGEL_EXPERT_MAJOR_LAYOUT=1 BAGEL_FUSED_RMS_NORM=1 BAGEL_FUSED_ADAMW=1 BAGEL_FUSED_SWIGLU=1 BAGEL_FUSED_EMA=1 BAGEL_FSDP_AVERAGE_REDUCTION=1 BAGEL_FUSED_PACKED_ROTARY=1 BAGEL_OVERLAP_VAE=1

Judged
1.86×
Tier
N1
Self-best
1.93×
Rounds
20
Hours
10.0
API cost
$8.83
Input tokens · 98.0% cached
46 M
Output tokens
0.3 M
Requests
335
Raw work log20 sections
0.0 hRound 1 — grouped FlashAttention for packed masks
- Start: 2026-11-18 05:15:58 UTC; four GPUs. Detached judge profile PID 9519, log /tmp/r1/profile.log, output /tmp/r1/profile.
- Baseline profile PASS; GPU lock 05:16:22 UTC. The frozen reference is 2.59997 s/step (0.2% spread). Actual judge devices report Hopper with 143771 MiB device memory; profiler CUDA properties report SM90.
- Profile (rank 0, steps 8–9; percentages of summed GPU durations, not wall time): NCCL 38.4% (370 calls), GEMM 22.9% (6,941), attention 16.4% (766, including ViT/VAE), remaining kernels 21.9%, explicit copies/sets 0.5%. 1.06 s of NCCL intervals did not overlap other GPU work, but these include waiting for other ranks. Decoder efficient attention alone is 1.11 s of aggregate GPU duration. PyTorch repeats K/V to all 28 query heads and re-runs dense masked attention during decoder checkpointing.
- Candidate /tmp/r1/tree: switch BAGEL_PACKED_FLASH_ATTENTION=1 groups exactly representable dense-mask rows into causal/noncausal FlashAttention varlen calls, keeping four KV heads and preserving the distinct noise and bidirectional spans. Falls back to the original backend for masks with other biases or unsupported layouts; FlashAttention uses deterministic backward when requested. A CPU reconstruction probe reproduced the full masks on 100 randomized packed layouts and checked biased-mask fallback.
- Detached judge time PID 10918, log /tmp/r1/time.log, output /tmp/r1/time, launched 05:23:14 UTC.
- Timing completed 05:29:23 UTC: 1.1712× frozen reference, 2.2199 s/step, windows 2.2160/2.2238, peak 82.14 GiB allocated / 116.89 GiB reserved. time cost 0.356 GPU hours.
- Detached judge check --quick PID 12463, log /tmp/r1/quick.log, output /tmp/r1/quick, launched 05:29:38 UTC.
- Round end 05:34:33 UTC: quick check NOT_BITWISE, all exact items passed. Committed as 9953d10 and enabled the candidate in the submission. N1 is pending further gates.
- Detached check --n1 PID 14254, log /tmp/r1/n1-valid.log, output /tmp/r1/n1-valid, launched 05:35:10 UTC. The preliminary launch PID 14221 was stopped before GPU acquisition because the commit had not landed yet.
- check --n1 completed 05:55:02 UTC: PASS (N1), both conformance modes and 150-step trajectory passed. Largest N/J conformance ratio was 1.465 (limit 3), largest endpoint bias 1.503 sigma (limit 3). Cost 1.271 GPU hours.
0.7 hRound 2 — FSDP communication
- Start: 2026-11-18 05:55:25 UTC; four GPUs. Detached judge profile PID 18201, log /tmp/r2/profile.log, output /tmp/r2/profile.
- Profile completed 05:59:59 UTC: NCCL 40.1% of GPU kernel durations (1.317 s reduce-scatter, 1.013 s all-gather; 0.988 s of NCCL intervals had no concurrent GPU work, including possible rank waiting), GEMM 26.5%, attention 5.8%, other 27.0%, explicit copies/sets 0.6%. All four GPU pairs report NV18; both large collectives default to Ring/LL. Peak GPU memory 82.13 GiB allocated, 109.05 GiB reserved.
- Screening two independent changes in the communication class: NCCL_PROTO=Simple on accepted tree, and BAGEL_FSDP_RETAIN_COMPUTE_WEIGHTS=1 in /tmp/r2/tree (hybrid ZeRO-2 retains each BF16 gathered training weight between forward and backward; FP32 local master/optimizer/EMA and BF16 reduce-scatter unchanged).
- Detached time PID 19599, log /tmp/r2/time-simple.log, output /tmp/r2/time-simple, launched 06:00:40 UTC. Detached time PID 20394, log /tmp/r2/time-retain.log, output /tmp/r2/time-retain, launched 06:01:46 UTC, queued behind the Simple test via the judge lock.
- Simple timing completed 06:06:48 UTC: 1.1756× frozen, 2.2116 s/step, only 0.37% faster than Round 1; no robust incremental gain, not enabling it.
- Retention timing completed 06:12:30 UTC: 1.1978× frozen, 2.1707 s/step, windows 2.1664/2.1749; 2.22% incremental speedup over Round 1. Peak 89.36 GiB allocated / 134.48 GiB reserved. Cost 0.379 GPU hours.
- Detached check --quick PID 22180, log /tmp/r2/quick.log, output /tmp/r2/quick, launched 06:12:44 UTC. Compare deterministic tensor digests against the Round 1 candidate as well as the exact-item result against upstream.
- Round end 06:17:34 UTC: combined quick check NOT_BITWISE with exact items passing, and all four ranks' parameter, gradient, optimizer and EMA digests bitwise identical to the N1 attention candidate. Retention is an N0 delta, committed as 6fb5982 and enabled. Full combined validation pending.
1.1 hRound 3 — decoder activation recomputation
- Start: 2026-11-18 06:18:12 UTC; four GPUs. Detached judge profile PID 23700, log /tmp/r3/profile.log, output /tmp/r3/profile.
- Profile completed 06:22:44 UTC: NCCL 41.0%, GEMM 25.6%, attention 5.7%, other 27.0%, copies/sets 0.6% (summed GPU durations). All-gather launches dropped from 234 to 116. GEMM replay inside backward: 0.361 s and 1,046 kernels (6.2%); FlashAttention replay: 0.071 s and 164 kernels (1.2%); both exclude associated indexing, norms and pointwise work. Peak memory 89.25 GiB allocated / 133.50 GiB reserved.
- Candidate /tmp/r3/tree, BAGEL_DECODER_KEEP_ACTIVATIONS: replace whole-block checkpointing on the requested number of final decoder layers with an inner MLP checkpoint. Save the two BF16 gate/up projection outputs and replay their gating activation; still checkpoint ViT and connector. CPU probe found bitwise identical MLP output and all gradients in BF16 and FP32. Both screening configurations enable expandable CUDA segments to reduce memory fragmentation.
- Time all 28 layers (without weight retention): detached PID 24927, log /tmp/r3/time-all.log, output /tmp/r3/time-all, launched 06:23:48 UTC. Time 22 layers with weight retention: detached PID 25352, log /tmp/r3/time-retain.log, output /tmp/r3/time-retain, launched 06:24:17 UTC, queued behind all-28.
- All-28 timing completed 06:29:32 UTC: 1.3990× frozen, 1.8584 s/step, windows 1.8702/1.8465; 16.8% incremental speedup over Round 2. Peak 117.19 GiB allocated / 134.88 GiB reserved; cost 0.330 GPU hours.
- 22-layer + retention timing completed 06:36:06 UTC: 0.7688× frozen, 3.3820 s/step, peak 129.12 GiB allocated / 134.96 GiB reserved. Not selected: high memory pressure and regression. The installed allocator reports expandable segments unsupported, so that override will not be submitted.
- Detached check --quick of all-28, PID 27691, log /tmp/r3/quick.log, output /tmp/r3/quick, launched 06:36:20 UTC.
- Round end 06:41:11 UTC: quick check NOT_BITWISE with all exact items passing and all deterministic tensor digests bitwise identical to Round 1 (all ranks/steps). N0 delta, committed e54f24c. Enabled all-28 without weight retention or allocator override.
- Detached full check PID 29184, log /tmp/r3/check.log, output /tmp/r3/check, launched 06:41:42 UTC. This validation is outside the round clock.
- Full check completed 07:09:25 UTC: PASS (N1), 1.3041×, 1.9937 s/step (windows 2.0024/1.9851), peak 117.28 GiB allocated / 134.96 GiB reserved, cost 1.794 GPU hours. Check used no allocator override. Conformance ratios ≤1.461, all trajectory endpoint biases ≤1.833 sigma. Investigating the speed gap to the earlier time run separately rather than assuming that the allocator warning describes every PyTorch translation unit.
1.9 hRound 4 — CUDA allocator fragmentation
- Start: 2026-11-18 07:10:03 UTC; four GPUs. Detached profile PID 35735, log /tmp/r4/profile.log, output /tmp/r4/profile.
- Profile completed 07:14:29 UTC: NCCL 43.7%, GEMM 23.9%, attention 5.2%, other 26.4%, copies/sets 0.7% (summed GPU durations). CUDA-runtime trace reports 82 cudaMalloc and 74 cudaFree calls (57 ms of host time) in the two-step capture; host event waits add to the exposed allocation pressure. Full timing at default allocation had median step 1.949 s versus 1.781 s for the earlier expanded allocation run.
- Locked capability probe: a 32 MiB allocation reserves 32 MiB with expandable_segments:False, 40 MiB with True; snapshot is_expandable changes from false to true. The warning is not evidence of inactivity for the native caching allocator.
- Detached screening time: expandable_segments:True PID 37068, /tmp/r4/time-expandable{,.log}; backend:cudaMallocAsync PID 37069, /tmp/r4/time-async{,.log}. Both launched 07:15:32 UTC; judge lock serializes the runs. Neither switch changes the source or arithmetic.
- cudaMallocAsync errored with CUDA OOM before steady state at 07:19:30 UTC; not selected (0.211 GPU hours). Expandable segments timing completed 07:24:22 UTC: 1.4153× frozen, 1.8371 s/step (windows 1.8386/1.8356), 8.52% incremental speedup over the last full check; peak 117.19 GiB allocated / 134.94 GiB reserved, cost 0.325 GPU hours.
- Detached check --quick for the allocator switch, PID 39611, log /tmp/r4/quick.log, output /tmp/r4/quick, launched 07:25:01 UTC.
- Round end 07:29:47 UTC: NOT_BITWISE against upstream (attention), but all deterministic tensor digests and exact items match the Round 3 full check across all ranks/steps. Enabled expandable segments as an N0 delta; documented switches in commit d474cc6.
2.3 hRound 5 — contiguous MoT expert routing
- Start: 2026-11-18 07:30:43 UTC; four GPUs. Detached profile PID 41070, log /tmp/r5/profile.log, output /tmp/r5/profile.
- Profile completed 07:35:04 UTC: NCCL 37.65%, aten::mm 24.80%, aten::copy_ 5.45%, aten::mul 4.66%, FlashAttention backward 3.93%, AdamW 3.23%, tensor index/write 4.21% combined (3,632 calls). Many index/write operations route decoder tokens to or from their expert despite identical routing across layers.
- Switched variant /tmp/r5/tree, BAGEL_EXPERT_MAJOR_LAYOUT=1: permute the decoder hidden state and RoPE positions once so understanding and generation experts occupy contiguous slices, remap the exact FlashAttention indices, and restore the original order before losses. Keep the upstream layout for incomplete partitioning, unsupported masks, or frozen understanding expert. A two-layer CPU probe with alternate full/noise/causal layouts matched forward outputs and all gradients exactly.
- Detached time PID 42326, log /tmp/r5/time.log, output /tmp/r5/time, launched 07:36:59 UTC.
- Timing completed 07:42:31 UTC: 1.4889× frozen, 1.7462 s/step, windows 1.7302/1.7621; 5.21% incremental speedup. Peak 117.11 GiB allocated / 134.92 GiB reserved, 0.317 GPU hours.
- Detached check --quick PID 43584, log /tmp/r5/quick.log, output /tmp/r5/quick, launched 07:42:49 UTC.
- Round end 07:47:47 UTC: NOT_BITWISE with all exact items passing and parameter, gradient, optimizer and EMA digests matching Round 4 across every step/rank. N0 delta, committed 73c97f5 and enabled.
- Detached full check PID 45055, log /tmp/r5/check.log, output /tmp/r5/check, launched 07:48:48 UTC. Full validation is outside the round clock.
- Full check finished 08:15:11 UTC: PASS (N1), 1.4865×, 1.7491 s/step (windows 1.7221/1.7760), peak 117.11 GiB allocated / 134.92 GiB reserved, cost 1.706 GPU hours. Largest held-out N/J conformance ratio 1.461; largest paired endpoint bias 1.713 sigma.
3.0 hRound 6 — fused BF16 RMS normalization
- Start: 2026-11-18 08:15:36 UTC; four GPUs. Detached profile PID 51531, log /tmp/r6/profile.log, output /tmp/r6/profile.
- Profile finished 08:20:03 UTC: NCCL 38.51%, GEMM aten::mm 25.08%, copies 6.33%, multiplies 4.93%, AdamW 3.39%. RMSNorm power/mean/rsqrt consume 22.9 ms GPU time over two steps; repeated casts and multiplies add bandwidth and saved activation pressure.
- Variant /tmp/r6/tree, BAGEL_FUSED_RMS_NORM=1: Triton per-row RMSNorm forward/backward for contiguous BF16 inputs and weights. Preserves FP32 accumulation and the BF16 normalized-value and product rounding points; the parallel reduction order can change. Weight gradient uses a deterministic sum with no atomics in the custom kernels. Probe on five production/small shapes in strict deterministic mode: largest output/input-or-weight-grad relative L2 0.000021.
- Detached time PID 52925, log /tmp/r6/time.log, output /tmp/r6/time, launched 08:24:40 UTC. The current submission remains fully validated Round 5 pending evidence.
- Timing finished 08:30:01 UTC: 1.6848×, 1.5432 s/step (windows 1.5531/1.5332), 13.34% incremental speedup over the latest full check. Peak allocated fell from 117.11 GiB to 103.63 GiB; peak reserved 122.53 GiB. Cost 0.304 GPU hours.
- Detached check --quick PID 54424, log /tmp/r6/quick.log, output /tmp/r6/quick, launched 08:30:31 UTC.
- Round end 08:35:30 UTC: NOT_BITWISE with all exact items passing. Digests differ from Round 5 on all four ranks, so this is an N1 candidate, not an N0 delta. Committed as f36ed31 on the experimental worktree only.
- Detached full check PID 56075, log /tmp/r6/check.log, output /tmp/r6/check, launched 08:35:59 UTC. Keep the passing Round 5 configuration submitted until this check passes.
- Full check completed 09:01:30 UTC: PASS (N1), 1.6805×, 1.5471 s/step (windows 1.5575/1.5366), max conformance ratio 1.523 and max trajectory bias 2.407 sigma. Cost 1.648 GPU hours. Promoted as e52d6c7 and enabled after PASS.
3.8 hRound 7 — selective FSDP compute-weight retention
- Start: 2026-11-18 09:01:52 UTC; four GPUs. Detached profile PID 63385, log /tmp/r7/profile.log, output /tmp/r7/profile.
- Profile completed 09:06:03 UTC: NCCL 38.34% of aggregate GPU kernel durations, split mainly between all-gather (18.82%, 234 kernels) and reduce-scatter (18.53%, 116); aten::mm 28.18%, FlashAttention backward 4.78%, copies 4.17%, AdamW 3.65%. Fused normalization freed ~13.5 GiB relative to Round 5.
- Variant /tmp/r7/tree: BAGEL_FSDP_RETAIN_DECODER_WEIGHTS=16 uses PyTorch's per-module wrapping policy to retain gathered BF16 weights for just the last 16 decoder blocks, which free early in backward. All sharding groups, flat layout, and optimizer/EMA precision are intended to remain unchanged.
- Detached time PID 64791, log /tmp/r7/time.log, output /tmp/r7/time, launched 09:07:26 UTC.
- Round end 09:12:45 UTC: 1.6974× frozen, 1.5317 s/step (windows 1.5316/1.5317), only 0.75% faster than the Round 6 screening and with over 12 GiB more allocated memory (115.78/134.88 GiB allocated/reserved). Cost 0.302 GPU hours. No quick check: not selecting this small/uncertain gain. Experimental commit 12a330f, not on opt.
4.0 hRound 8 — fused AdamW update
- Start: 2026-11-18 09:14:25 UTC; four GPUs. Detached profile PID 66275, log /tmp/r8/profile.log, output /tmp/r8/profile.
- Profile completed 09:18:35 UTC: NCCL 37.89%, aten::mm 28.30%, copy 4.15%, AdamW 3.70% (146.55 ms over two steps). AdamW's foreach update makes separate passes for first and second moments, denominator, and parameter update.
- Variant /tmp/r8/tree, BAGEL_FUSED_ADAMW=1: opt-in PyTorch stock fused AdamW, with the same optimizer/scheduler, parameter groups and FP32 master/moment tensors. Fused arithmetic may introduce rounding differences. Detached time PID 67718, log /tmp/r8/time.log, output /tmp/r8/time, launched 09:19:48 UTC.
- Timing completed 09:25:19 UTC: 1.7230× frozen, 1.5090 s/step, windows 1.5235/1.4944, 2.53% incremental rate over the latest full check, unchanged peak memory (103.63 GiB allocated/122.53 GiB reserved), 0.316 GPU hours.
- Detached check --quick PID 69224, log /tmp/r8/quick.log, output /tmp/r8/quick, launched 09:25:42 UTC.
- Round end 09:30:28 UTC: quick NOT_BITWISE with all exact items passing. Experimental commit 423743b, not submitted pending validation.
- Detached check --n1 PID 70873, log /tmp/r8/n1.log, output /tmp/r8/n1, launched 09:30:50 UTC. This adds conformance and statistical gates without duplicating timing or the record.
- check --n1 completed 09:47:42 UTC: PASS (N1), largest conformance ratio 1.527 and endpoint bias 1.527 sigma. Cost 1.071 GPU hours. Enabled and promoted as 8132914 after all gates passed in separate commands.
4.6 hRound 9 — fused SwiGLU gating
- Start: 2026-11-18 09:48:24 UTC; four GPUs. Detached profile PID 75216, log /tmp/r9/profile.log, output /tmp/r9/profile.
- Profile completed 09:52:41 UTC: NCCL 38.68%, GEMM aten::mm 28.75%, copies 4.32%, multiply 2.25%, AdamW 1.75% (34.0 ms/step, down from 73.3). SiLU and its backward add up to 47.65 ms over two steps before separate multiply work.
- Variant /tmp/r9/tree, BAGEL_FUSED_SWIGLU=1: fuse SiLU, elementwise multiplication and their gradients with Triton, keeping the two explicit BF16 rounding boundaries. Deterministic probe found exact bit patterns for output and gradients on two random decoder shapes; the all-finite-BF16-pattern probe had low-magnitude differences but passed the numerical bound.
- Detached time PID 76741, log /tmp/r9/time.log, output /tmp/r9/time, launched 09:54:49 UTC.
- Timing completed 10:00:06 UTC: 1.7472× frozen, 1.4881 s/step (windows 1.4997/1.4765), 1.40% incremental rate. Peak 103.41 GiB allocated, 122.27 GiB reserved; cost 0.299 GPU hours.
- Detached check --quick PID 78326, log /tmp/r9/quick.log, output /tmp/r9/quick, launched 10:00:43 UTC.
- Round end 10:05:31 UTC: exact items passed and all recorded parameter, gradient, optimizer state and EMA digests match Round 8 over every rank/step. N0 delta, committed 3b112b3 and enabled.
- Detached combined full check PID 80038, log /tmp/r9/check.log, output /tmp/r9/check, launched 10:06:08 UTC. Full validation is outside the round clock.
- Full check completed 10:31:48 UTC: PASS (N1), 1.7513×, 1.4846 s/step (windows 1.4962/1.4730). Largest deterministic/production conformance ratio 1.517, largest endpoint bias 1.935 sigma. Cost 1.657 GPU hours.
5.3 hRound 10 — fused EMA update
- Start: 2026-11-18 10:32:10 UTC; four GPUs. Detached judge profile PID 87415, log /tmp/r10/profile.log, output /tmp/r10/profile.
- GPU lock 10:32:33 UTC. Profile completed 10:36:20 UTC: NCCL 37.54%, GEMM aten::mm 30.93%, FlashAttention 6.91%, copies 4.27%, AdamW 1.73%, EMA foreach operations 1.00% (39.03 ms across two profiled steps; 698 kernel launches). EMA was selected as a memory-bandwidth fusion opportunity.
- Variant /tmp/r10/tree, BAGEL_FUSED_EMA=1: fuse the FP32 decay and scaled addition in one kernel per shard, keeping FP32 rounding after decay and the existing fused multiply-add for the scaled addition. A strict-deterministic CUDA probe matched all bit patterns for five tensor sizes (up to 121,637,888 elements) at two decay values.
- Detached time PID 88822, log /tmp/r10/time.log, output /tmp/r10/time, launched 10:38:16 UTC.

- Timing completed 10:43:33 UTC: 1.7591× frozen, 1.4780 s/step, 0.68% faster than the Round 9 screening and 0.45% faster than its full check. Detached check --quick PID 90201, log /tmp/r10/quick.log, output /tmp/r10/quick, launched 10:43:59 UTC.
- Round end 10:50:03 UTC: exact items passed. All deterministic records apart from timing fields matched Round 9 on every rank and step. N0 delta, committed c2a9620 and enabled on the N1 stack.
5.6 hRound 11 — native gradient averaging
- Start: 2026-11-18 10:50:03 UTC; four GPUs. Detached judge profile PID 91775, log /tmp/r11/profile.log, output /tmp/r11/profile.

- GPU lock 10:50:26 UTC. Profile completed 10:54:13 UTC: NCCL 38.28% of GPU activity (reduce-scatter 18.47%), GEMM aten::mm 30.78%, FlashAttention 6.90%, copies 4.11%, standalone FSDP gradient divisors 1.42% (55.21 ms across two steps). FSDP host rate limiter time 1.436 s across the trace, including normal backpressure; dataset fetch 0.18 ms.
- Variant /tmp/r11/tree, BAGEL_FSDP_AVERAGE_REDUCTION=1: a single replica group uses equivalent full sharding and NCCL BF16 AVG for reduce-scatter, eliminating the two scaling passes. Other meshes retain the reference reduction. A four-rank deterministic BF16 probe (1 to 67,108,864 elements per rank, five sizes) matched the reduce-scatter and two separate scaling passes bitwise.
- Detached time PID 93304, log /tmp/r11/time.log, output /tmp/r11/time, launched 10:57:38 UTC.

- Timing completed 11:02:54 UTC: 1.7779× frozen, 1.4624 s/step (windows 1.4767/1.4480), a 1.07% incremental gain over Round 10. Peak 103.41 GiB allocated / 121.92 GiB reserved; cost 0.299 GPU hours.
- Detached check --quick PID 94634, log /tmp/r11/quick.log, output /tmp/r11/quick, launched 11:03:39 UTC.

- Round end 11:09:27 UTC: quick check passed exact items; all tensor digests match Round 10 bitwise on every rank and step. N0 delta, committed b16b94c and enabled. The cumulative timing gain over the latest full check warrants a new combined check.

- Detached combined full check PID 96149, log /tmp/r11/check.log, output /tmp/r11/check, launched 11:09:27 UTC. Validation is outside the round clock.

- Full check completed 11:34:39 UTC: PASS (N1), 1.7723×, 1.4670 s/step (windows 1.4682/1.4657), 1.20% faster than the Round 9 full check. Cost 1.626 GPU hours.
6.3 hRound 12 — packed rotary layout fusion
- Start: 2026-11-18 11:35:16 UTC; four GPUs. Detached judge profile PID 102806, log /tmp/r12/profile.log, output /tmp/r12/profile.

- GPU lock 11:35:40 UTC. Profile completed 11:39:28 UTC: NCCL 39.57%, GEMM aten::mm 31.49%, FlashAttention 7.13%, copies 4.19%, concatenation 2.37%, multiply 0.91%, add 0.38%, negation 0.16%. Rotary and expert reassembly contribute to the repeated CAT and pointwise kernels on packed BF16 queries and keys.
- Variant /tmp/r12/tree, BAGEL_FUSED_PACKED_ROTARY=1: fuse the reassembly of normalized query/key expert slices with rotary embedding and split the backward gradient directly. Both BF16 product roundings are kept. Strict deterministic probes matched eager outputs and gradients bitwise on two empty-expert layouts and a full 11,520-row 28-query/4-key-head layout.
- Detached time PID 104434, log /tmp/r12/time.log, output /tmp/r12/time, launched 11:42:27 UTC.

- Timing completed 11:47:52 UTC: 1.8185× frozen, 1.4297 s/step (windows 1.4438/1.4156), a 2.28% incremental rate over Round 11's screening. Peak 103.41 GiB allocated / 121.86 GiB reserved; cost 0.308 GPU hours.
- Detached check --quick PID 105856, log /tmp/r12/quick.log, output /tmp/r12/quick, launched 11:49:07 UTC.

- Round end 11:54:52 UTC: quick check exact items passed; all deterministic tensor digests matched Round 11 on every step and rank. N0 delta, committed 745a8f0 and enabled. The cumulative gain warrants a new full check.

- Detached combined full check PID 107445, log /tmp/r12/check.log, output /tmp/r12/check, launched 11:54:52 UTC. Validation is outside the round clock.

- Full check completed 12:20:00 UTC: PASS (N1), 1.8211×, 1.4277 s/step (windows 1.4288/1.4265). Largest deterministic/production conformance ratio 1.519. Cost 1.622 GPU hours.
7.1 hRound 13 — inverse gathers for attention permutations
- Start: 2026-11-18 12:20:47 UTC; four GPUs. Detached judge profile PID 114533, log /tmp/r13/profile.log, output /tmp/r13/profile.

- GPU lock 12:21:11 UTC. Profile completed 12:24:59 UTC: NCCL 39.25%, GEMM aten::mm 32.61%, FlashAttention 7.33%, copies 4.22%, concatenation 2.02%, index-add 1.08%, index-select 0.94%. All 40.10 ms of profiled index-add came from index-select backpropagation. Fused rotary was active in the trace (5.57 ms across two steps).
- Variant /tmp/r13/tree, BAGEL_PACKED_ATTENTION_PERMUTATION=1: gather queries as one bijection, split into their existing FlashAttention groups, and restore outputs in the original order. Backward gathers by inverse permutation, avoiding scatter-add and zero-fill for Q and output; repeated K/V rows still use the existing operations. A strict deterministic CUDA probe with mixed causal, adjacent full and nonterminal noise spans matched all outputs and gradients bitwise in original and permuted token order.
- Detached time PID 116215, log /tmp/r13/time.log, output /tmp/r13/time, launched 12:27:18 UTC.

- Timing completed 12:32:33 UTC: 1.8182× frozen, 1.4300 s/step (windows 1.4400/1.4200). No gain compared with Round 12's 1.4297 s/step screening or 1.4277 s/step full check. Peak 103.41 GiB allocated / 121.82 GiB reserved; cost 0.297 GPU hours. Do not promote.
- Detached check --quick PID 117627, log /tmp/r13/quick.log, output /tmp/r13/quick, launched 12:33:18 UTC to record exactness of the experimental change.

- Round end 12:38:52 UTC: exact items passed; deterministic digests match Round 12 on every rank and step. Experimental N0 delta committed 1bdda93, not enabled or merged.
7.4 hRound 14 — FSDP all-gather overlap
- Start: 2026-11-18 12:38:52 UTC; four GPUs. Detached judge profile PID 119233, log /tmp/r14/profile.log, output /tmp/r14/profile.

- GPU lock 12:39:16 UTC. Profile completed 12:42:59 UTC: NCCL 39.78% (all-gather 19.52%, reduce-scatter 19.11%), GEMM aten::mm 32.05%, FlashAttention 7.35%, copies 4.26%. Across two steps, 311.1 ms of all-gather occurs without concurrent GPU compute, including waits for other ranks. FSDP host rate limiter accounts for 1,416.1 ms of blocking across 234 calls, also including normal queue backpressure.
- Retest full BF16 weight retention with the current lower-memory activation path and native AVG-enabled sharding. Detached time PID 120633, log /tmp/r14/time-retain.log, output /tmp/r14/time-retain, config /tmp/r14/switches-retain.json, launched 12:44:03 UTC.

- Variant /tmp/r14/tree, BAGEL_FSDP_ALL_GATHER_QUEUE_DEPTH=6: raise the FSDP limiter's queue depth from 2 to 6 without retaining weights, to let the CPU issue gathers further ahead. Detached time PID 121429, log /tmp/r14/time-queue.log, output /tmp/r14/time-queue, launched 12:44:43 UTC, queued behind retention.

- Retention completed 12:50:37 UTC: 0.9666× frozen, 2.6898 s/step (windows 2.7167/2.6629), peak 126.85/137.04 GiB allocated/reserved. High memory pressure and large regression; left disabled. Cost 0.385 GPU hours.
- Queue depth 6 completed 12:55:06 UTC: 1.8123× frozen, 1.4346 s/step (windows 1.4527/1.4165), peak 103.41/126.22 GiB allocated/reserved, no gain with 4.36 GiB more reserved. Cost 0.295 GPU hours.
- Round end 12:56:14 UTC: neither configuration selected. Queue depth variant committed as experimental 01733da (expected N0, scheduling only). No additional quick check for these non-promoted configurations.
7.7 hRound 15 — merged MLP gate and up projections
- Start: 2026-11-18 12:56:14 UTC; four GPUs. Detached judge profile PID 123455, log /tmp/r15/profile.log, output /tmp/r15/profile.

- GPU lock 12:56:38 UTC. Profile completed 13:00:21 UTC: NCCL 39.24%, GEMM aten::mm 32.61% (1.202 s across two steps, BF16 SM90 tensor-core kernels), FlashAttention 7.41%, copies 4.04%, concatenation 2.04%, AdamW 1.83%.
- Variant /tmp/r15/tree, BAGEL_PACKED_MLP_PROJECTIONS=1: use one native BF16 linear projection over a checked view of two adjacent FSDP weights, retaining the original parameters. Apply the existing SwiGLU arithmetic directly on the packed rows and return packed gradients. An isolated strict deterministic probe matched outputs and both weight gradients bitwise on two shapes; the merged input-gradient GEMM changes rounding (maximum 0.00333 relative L2 difference in the probe).
- Detached time PID 125205, log /tmp/r15/time.log, output /tmp/r15/time, launched 13:02:59 UTC.

- Timing completed 13:08:13 UTC: 1.8147× frozen, 1.4327 s/step (windows 1.4473/1.4181), no gain. Peak 103.41 GiB allocated / 122.08 GiB reserved; cost 0.296 GPU hours. Detached check --quick PID 126589, log /tmp/r15/quick.log, output /tmp/r15/quick, launched 13:08:47 UTC.

- Round end 13:14:12 UTC: quick check passed exact items; changed digests confirm the packed GEMM was active. Experimental N1 candidate committed 01c285e, not enabled or merged. No conformance/trajectory run because the timing did not improve.
8.0 hRound 16 — VAE channels-last encoder
- Start: 2026-11-18 13:14:12 UTC; four GPUs. Detached judge profile PID 128171, log /tmp/r16/profile.log, output /tmp/r16/profile.

- GPU lock 13:14:36 UTC. Profile completed 13:18:20 UTC: NCCL 39.42%, GEMM aten::mm 32.30%, FlashAttention 7.33%, copies 4.35%; VAE cuDNN convolution 0.94% (34.83 ms), group norm 0.80% (29.42 ms) across two steps. The frozen encoder's contiguous images and weights can limit convolution/layout efficiency.
- Variant /tmp/r16/tree, BAGEL_VAE_CHANNELS_LAST=1: use channels-last weights and images inside the frozen encoder; restore the encoder moments to the original layout before the posterior sample. A deterministic probe with the released VAE matched the posterior shape, strides, dtype and final CUDA RNG state.
- Detached time PID 129984, log /tmp/r16/time.log, output /tmp/r16/time, launched 13:20:12 UTC.

- Timing completed 13:25:29 UTC: 1.7614× frozen, 1.4761 s/step (windows 1.4882/1.4639), 3.39% slower in step time than Round 12's full check, with unchanged peak memory. Cost 0.300 GPU hours. The isolated latent relative L2 difference was 0.01770 with the same noise draw. Detached check --quick PID 131375, log /tmp/r16/quick.log, output /tmp/r16/quick, launched 13:26:00 UTC.

- Round end 13:31:26 UTC: quick check exact items passed, not bitwise. Experimental N1 candidate committed ce4185a; not merged or enabled because of the timing regression.
8.3 hRound 17 — overlap batch-count reduction
- Start: 2026-11-18 13:31:26 UTC; four GPUs. Detached judge profile PID 132976, log /tmp/r17/profile.log, output /tmp/r17/profile.

- GPU lock 13:31:50 UTC. Profile completed 13:35:33 UTC: NCCL 39.33%, GEMM aten::mm 32.75%, FlashAttention 7.29%, copies 4.11%. Six integer scalar all-reduces occupied 66.81 ms across the two steps (1.80%, includes rank rendezvous). FP32 scalar all-reduces occupied 2.15 ms.
- Variant /tmp/r17/tree, BAGEL_OVERLAP_BATCH_COUNTS=1: pack CE/MSE and sample counts into one int64 sum, launch it before the VAE/forward, and consume the exact same global counts for loss normalization and logging. No changes to floating-point loss arithmetic.
- Detached time PID 134420, log /tmp/r17/time.log, output /tmp/r17/time, launched 13:37:11 UTC.

- Timing completed 13:42:24 UTC: 1.8193× frozen, 1.4291 s/step (windows 1.4384/1.4198), no repeatable gain versus Round 12. Peak 103.41/121.86 GiB allocated/reserved, cost 0.295 GPU hours. Detached check --quick PID 135811, log /tmp/r17/quick.log, output /tmp/r17/quick, launched 13:43:02 UTC to check the alternate normalization path.

- Round end 13:48:26 UTC: exact items and all tensor digests matched Round 12. N0 delta 8b5b346 not enabled or merged because time did not improve.
8.5 hRound 18 — reuse local BF16 FSDP casts
- Start: 2026-11-18 13:48:26 UTC; four GPUs. Detached judge profile PID 137410, log /tmp/r18/profile.log, output /tmp/r18/profile.

- GPU lock 13:48:49 UTC. Profile completed 13:52:33 UTC: NCCL 40.62%, GEMM aten::mm 31.61%, FlashAttention 7.18%, copies 4.10% (155.31 ms across two steps). FSDP pre-forward/pre-backward copies account for 31.12/36.90 ms, respectively; post-backward copies account for 38.70 ms.
- Variant /tmp/r18/tree, BAGEL_FSDP_CACHE_COMPUTE_SHARDS=1: retain each trained non-root handle's local BF16 forward cast until its backward unshard if the parameter version still matches. Gathered weights are released as before; FSDP still frees the local cache after backward consumption. Master, gradient and optimizer dtypes are unchanged.
- Detached time PID 138822, log /tmp/r18/time.log, output /tmp/r18/time, launched 13:54:18 UTC.

- Timing completed 13:59:31 UTC: 1.8216× frozen, 1.4273 s/step (windows 1.4424/1.4123), essentially unchanged compared to Round 12's full run. Peak 109.25/127.06 GiB allocated/reserved, +5.84/+5.20 GiB. Cost 0.295 GPU hours. Detached check --quick PID 140209, log /tmp/r18/quick.log, output /tmp/r18/quick, launched 14:00:01 UTC to validate cast reuse.

- Round end 14:05:33 UTC: quick check passed exact items and matched every tensor digest of Round 12. N0 delta 4f54fbe, not enabled or merged because the timing was unchanged with higher memory.
8.8 hRound 19 — overlap the VAE with current-batch vision work
- Start: 2026-11-18 14:05:33 UTC; four GPUs. Detached judge profile PID 141826, log /tmp/r19/profile.log, output /tmp/r19/profile.

- GPU lock 14:05:56 UTC. Profile completed 14:09:43 UTC: NCCL 41.85%, GEMM aten::mm 31.33%, FlashAttention 6.97%, copies 4.09%. VAE convolution 0.89%, group norm 0.77%, VAE attention 0.16%; the convolution interval envelopes were 31.3 ms and 102.9 ms for the two profiled batches.
- Variant /tmp/r19/tree, BAGEL_OVERLAP_VAE=1: launch the current batch's frozen VAE encode and unchanged FSDP input cast on a side stream, then wait for its event immediately before latent use in Bagel. Independently scheduled earlier embedding, mask, ViT and connector work can overlap; the batch, posterior and latent-noise call sites and call order are unchanged.
- Detached time PID 143215, log /tmp/r19/time.log, output /tmp/r19/time, launched 14:11:25 UTC.

- Timing completed 14:16:36 UTC: 1.8689× frozen, 1.3912 s/step (windows 1.4078/1.3747), 2.63% incremental rate over the Round 12 full check. Peak allocated unchanged (103.41 GiB); reserved increased to 131.57 GiB (+9.71). Cost 0.293 GPU hours.
- Detached check --quick PID 144629, log /tmp/r19/quick.log, output /tmp/r19/quick, launched 14:17:12 UTC.

- Round end 14:23:04 UTC: quick check passed exact items and all tensor digests matched Round 12 bitwise on every rank and step. N0 delta committed 9f8e87d and enabled. Incremental gain warrants a new full check.

- Detached full check PID 146251, log /tmp/r19/check.log, output /tmp/r19/check, launched 14:23:04 UTC. Validation is outside the round clock.

- Full check completed 14:48:12 UTC: PASS (N1), 1.8698×, 1.3905 s/step (windows 1.4077/1.3734). Largest conformance ratio 1.512; largest endpoint bias 1.855 sigma. Cost 1.622 GPU hours.
9.6 hRound 20 — retain ViT and connector activations
- Start: 2026-11-18 14:49:05 UTC; four GPUs. Detached judge profile PID 153239, log /tmp/r20/profile.log, output /tmp/r20/profile.

- GPU lock 14:49:29 UTC. Profile completed 14:53:12 UTC: NCCL 40.11%, GEMM aten::mm 32.30%, FlashAttention 7.08%, copies 4.09%, aten::addmm 1.55%. Vision-layer replay includes 7.22 ms of biased GEMMs, 6.32 ms of FlashAttention forward and 1.55 ms of forward LayerNorm across two steps, not including other replayed work.
- Variant /tmp/r20/tree, BAGEL_KEEP_VISION_ACTIVATIONS=1: keep ViT/connector activations instead of checkpointing those blocks; decoder policy unchanged. Detached time PID 154658, log /tmp/r20/time.log, output /tmp/r20/time, launched 14:54:42 UTC.

- Timing completed 14:59:49 UTC: 1.9260× frozen, 1.3499 s/step (windows 1.3628/1.3371), 3.01% incremental rate over Round 19's full check. Peak 108.79/136.79 GiB allocated/reserved, +5.38/+5.22 GiB. Cost 0.289 GPU hours; full validation required before promotion due to the higher reserved memory.
- Detached check --quick PID 156045, log /tmp/r20/quick.log, output /tmp/r20/quick, launched 15:00:37 UTC.

- Round end 15:06:01 UTC: quick check passed exact items and matched all Round 19 digests bitwise. N0 delta 7f73585 committed in /tmp/r20/tree, not promoted while checking reserved-memory headroom across the full workload.
- Detached full check PID 157629, log /tmp/r20/check.log, output /tmp/r20/check, launched 15:06:01 UTC on the isolated tree. Validation is outside the round clock.
↑ BAGEL SFT