GPT-6 Astrasuspected nerf 1.62×
Harness: Codex · switches: BAGEL_RETAIN_SHARDED_WEIGHTS=1 BAGEL_DECODER_CHECKPOINT_SKIP_INTERVAL=3 BAGEL_PACKED_FLASH_ATTENTION=1 BAGEL_FUSED_ADAMW=1 BAGEL_FUSED_RMSNORM=1 BAGEL_FUSED_HEAD_RMSNORM=1 BAGEL_FUSED_ROTARY=1 BAGEL_DIRECT_QK_ROUTING=1 BAGEL_FUSED_SWIGLU=1 BAGEL_PARTITION_ROUTING=1 BAGEL_PARTITION_SKIP_ZERO=1 BAGEL_LOCAL_EXPERT_STATES=1 BAGEL_FLASH_ATTENTION_3=1
- Judged
- 1.62×
- Tier
- N1
- Self-best
- 1.65×
- Rounds
- 15
- Hours
- 10.0
- API cost
- $78.9
- Input tokens · 92.0% cached
- 32 M
- Output tokens
- 0.4 M
- Requests
- 297
§ 02Key levers
- Attention kernel swap (split by mask structure)
- ✓
BAGEL_PACKED_FLASH_ATTENTION: screening 1.07 → 1.27×; FA3 added later (1.576 → 1.611×). FlexAttention tried first at 1.24×, dropped after quick check errored - Drop activation recompute (28 decoder layers)
- partial: 1 in every 3 layers kept (9 layers). Keeping half tried twice, both slower (0.9986×, peak 127.5 GiB; 1.18× vs 1.35×)
- Intra-layer MoT fusion (routing / RMSNorm / RoPE / SwiGLU)
- ✓ off-the-shelf library kernels: RMSNorm, RoPE, SwiGLU, direct Q/K routing, expert-local state, screening 1.29 → 1.56×
- Optimizer, EMA, step start / end
- only torch's built-in fused AdamW (+1.85%); EMA, VAE not in the log
- Other main levers
- weights all-gathered in forward kept for backward (1.02×); single shard group, NCCL channel count, early count reduction showed no distinguishable gain
§ ARaw trace
Raw work log21 sections
0.0 hRound 1 — baseline profiling
- Start: 2026-11-10T03:32:49Z; four Hopper GPUs. - Submission initially matches upstreama2fa77d;/submission/switches.jsonhas an empty environment mapping. - First action: judge profile of the unchanged submission; no training changes made. - Profile started 2026-11-10T03:33:41Z: PID9387, log/tmp/r1/profile.log, output/tmp/r1/profile. - Frozen reference timing: 2.59833 seconds/step; spread 0.0022. - Profile finished 03:38:16Z, statusok, 221.8 s run wall time. Rank-0 CUDA accounting (overlapping sums): collectives 38.8% (reduce-scatter 22.45%, all-gather 15.38%), GEMMs 20.62%, memory-efficient attention backward 9.14% and forward 6.86%, copies 4.50%, multiplies 3.61%. 234 all-gathers / two steps. Peak reference memory 82.14 GiB. - Round hotspot: communication; configuration fast path. Test retaining gathered bf16 parameters from forward until backward (_HYBRID_SHARD_ZERO2), avoiding backward all-gathers without changing sharded master weights or arithmetic. Expected N0; judge will determine tier. SwitchBAGEL_RETAIN_SHARDED_WEIGHTS=1, default off. Experiment isolated in/tmp/r1/tree; submission remains upstream. - Time job: PID10524, log/tmp/r1/time.log, output/tmp/r1/time. - Retained-weight timing completed 03:46:09Z:/tmp/r1/time/time.json, statusok, 2.5475 s/step, reported speedup 1.02 (unrounded ratio 1.01995, too close to the 1.02 benefit gate). Not promoted; most backward all-gathers apparently overlap useful work. - Next communication-class configuration: standard NCCLSimpleprotocol, retain-weight switch off. This tests whether the large LL collective kernels in the baseline reflect a bandwidth/backend limitation; no model code or arithmetic changed. - NCCL timing job: PID11608, log/tmp/r1/time-simple.log, output/tmp/r1/time-simple. - NCCLSimplecompleted 03:53:34Z:/tmp/r1/time-simple/time.json, statusok, 2.5957 s/step, speedup 1.001, no resolved gain. Rejected this direction. Protocol remains automatic. - Baseline trace overlap accounting: 2.704 s total NCCL kernels over two steps, of which 1.572 s overlapped other GPU kernels and 1.132 s had communication alone. This explains the modest benefit from removing prefetched all-gathers. - Retained-weight exactness job: PID12675, log/tmp/r1/quick.log, output/tmp/r1/quick. - Quick check completed 03:58:57Z:/tmp/r1/quick/check.json,PASS, tierN0. - End: 2026-11-10T03:59:26Z (25 minutes from first GPU lock). Retained-weight option promoted as the best screened version; its 2% gain alone is borderline for gate 1. No full verdict claimed. Peak 89.66 GiB; timing windows 2.5458 / 2.5492 s.
0.5 hRound 2 — reprofile retained-weight version
- Start: 2026-11-10T03:59:26Z. Profile current best before selecting the next hotspot. - Profile job: PID13937, log/tmp/r2/profile.log, output/tmp/r2/profile. - Profile finished 04:04:29Z,ok: all-gathers reduced from 234 to 116 over two steps, confirming the retained-weight fast path. CUDA shares: communication 38.06% (overlap included), GEMM 20.19%, attention backward 9.44% / forward 7.11%, copies 4.49%, multiplies 3.71%. Attention forward has 338 calls versus 168 backward calls, exposing checkpoint recomputation. Peak retained-weight timing memory was only 89.66 GiB. - Hotspot: decoder activation recomputation (configuration/memory tradeoff). Isolated experiment/tmp/r2/tree:BAGEL_DECODER_CHECKPOINT_INTERVAL=2checkpoints alternate decoder layers, leaving ViT and connectors unchanged. Same parameters, arithmetic, optimizer, data and loss; expected N0, to be measured. Default interval 1 preserves upstream behavior. - Timing job: PID15010, log/tmp/r2/time.log, output/tmp/r2/time. - Half-checkpoint experiment completed 04:11:28Z:/tmp/r2/time/time.json, 2.6020 s/step (windows 2.5697 / 2.6343), speedup 0.9986, peak 127.54 GiB. No gain and little memory headroom: rejected, not promoted. Possible allocator pressure; not established as a cause. - Second configuration of this hotspot: retain activations for only one quarter of decoder layers (every fourth), usingBAGEL_DECODER_CHECKPOINT_SKIP_INTERVAL=4, default 0. Retained-weight optimization unchanged. This reduces memory pressure versus the unsuccessful half-checkpoint case. - Quarter-retention time job: PID16108, log/tmp/r2/time-quarter.log, output/tmp/r2/time-quarter. - Quarter-retention completed 04:19:03Z:/tmp/r2/time-quarter/time.json,ok, 2.4136 s/step, speedup 1.0765, windows 2.4063 / 2.4209 s, peak 107.10 GiB. Clear gain above frozen noise, with significantly more headroom than the half-retention case. - Quick job: PID17181, log/tmp/r2/quick.log, output/tmp/r2/quick. - Quick completed 04:24:32Z:/tmp/r2/quick/check.json,PASS N0. Promoted quarter-layer retention with retained gathered weights. - End: 2026-11-10T04:24:45Z (24.4 minutes from round-2 GPU lock). Clear 1.0765x combined screening gain justifies the full check; full check is outside the round budget.
1.2 hRound 3 — profile fully validated candidate
- Start: 2026-11-10T04:43:24Z. Submission remains the full-check-passing rounds 1–2 version; subsequent experiments remain isolated. - Profile job: PID22668, log/tmp/r3/profile.log, output/tmp/r3/profile. - Profile completed 04:48:29Z,ok. CUDA shares (overlapping kernel totals): communication 35.59%, GEMM 21.45%, memory-efficient attention backward 10.14% plus forward 6.67%, copies 4.64%, multiplies 3.81%, AdamW 2.37%. Attention still uses SM80 CUTLASS kernels, with explicit GQA head duplication. - Hotspot/class: attention backend, L1 library substitution. Experiment in/tmp/r3/tree:BAGEL_DENSE_FLEX_ATTENTION=1converts the actual upstream zero/-infinity dense sample masks into a block-diagonal boolean mask, then uses existing compiled FlexAttention with native GQA. No inference of image/noise layout: every allowed edge comes directly from the original mask. Computational padding uses a minimum 16384 power-of-two bin to keep compilation shapes stable; padded tokens are excluded from all original queries and sliced off output. Default switch off. Expected N1. - Isolated deterministic correctness probe (interleaved noise, adjacent full spans, causal text; outputs and all Q/K/V gradients) followed by judgetime: PID23758, logs/tmp/r3/probe.logand/tmp/r3/time.log, timing output/tmp/r3/time. Probe does not time or rank implementations; only the judge does. - Probe passed: relative L2 output 0.0008794, dQ 0.003690, dK 0.004020, dV 0.002602; all finite under strict deterministic algorithms. This is diagnostic evidence only, not a gate verdict. - Judge timing completed 04:56:33Z:/tmp/r3/time/time.json,ok, 2.0911 s/step, speedup1.2426, beyond frozen noise. The existing fully validated N0 submission is unchanged pending gate checks. - Timing windows 2.0752 / 2.1071 s; peak 106.06 GiB. Quick job: PID25546, log/tmp/r3/quick.log, output/tmp/r3/quick. - Quick completed 05:01:16Z:/tmp/r3/quick/check.json,ERROR, not a numerical verdict. At the optimizer post-hook, enumeration of live FSDP objects encounteredReferenceError: weakly-referenced object no longer exists(record_cand.log:677, stack ending in the existing FSDP-root inventory). The standalone computation probe is correct but does not override this gate failure. Discarding the FlexAttention path, without modifying or bypassing the judge or its object inventory. Submission remains the fullPASS N0version. - Remaining attention-library capability probe this round: test whether PyTorch/cuDNN supports the required explicit arbitrary additive mask and deterministic backward. An unsupported backend will be rejected without a training run. - cuDNN probe: production-mode additive-mask outputs/dQ/dK/dV have relative L2 0.0000408 / 0.003517 / 0.003549 / 0.0000831. Strict deterministic mode explicitly reportscuDNN SDPA is not deterministicand refuses to launch. Per the task's deterministic-path requirement, the cuDNN option uses original efficient attention whenever deterministic algorithms are enabled, and cuDNN otherwise, at identical bf16 precision with unchanged masks. No compile or inventory workaround. SwitchBAGEL_CUDNN_ATTENTION=1; isolated/tmp/r3/cudnn-treeis based on the accepted N0 tree, not the rejected FlexAttention code. - cuDNN time job: PID26893, log/tmp/r3/time-cudnn.log, output/tmp/r3/time-cudnn. - cuDNN first timing failed 05:07:04Z (ERROR, no timing): the backend explicitly requires a 2-D or 4-D mask, whereas upstream passes a broadcast-equivalent 3-D mask. Root cause is a documented shape guard, not incorrect computation. The optional cuDNN path now adds one singleton mask dimension; the deterministic/original path retains its exact 3-D mask. Retesting real noncontiguous query layout and non-aligned sequence length before the judge timing. - Corrected cuDNN probe/time job: PID27944, logs/tmp/r3/probe_cudnn_4d.logand/tmp/r3/time-cudnn-4d.log, timing output/tmp/r3/time-cudnn-4d. - Round-3 30-minute boundary: 2026-11-10T05:14:09Z. No more changes in this round; the already-started cuDNN timing is still being awaited. Round conclusion so far: FlexAttention rejected by the quick gate; cuDNN shape correction has a passing local production-mode probe, but no end-to-end result yet. Submission stays at fullPASS N0, 1.0733x. - Late timing result, 05:23:34Z:/tmp/r3/time-cudnn-4d/time.json,ok, 10.0904 s/step, speedup0.2575, windows 10.7771 / 9.4036 s, peak 106.32 GiB. Window spread 13.61% is also flagged as noisy. This backend is substantially slower and is rejected; no precision check is warranted for a no-gain candidate. The already-running measurement overran the round boundary; no changes were made after the boundary. Future rounds will reserve more time for slow backend measurements.
1.9 hRound 4 — reprofile before attention rewrite
- Start: 2026-11-10T05:23:48Z. All round-3 changes were rejected; the submission remains the full-check-passing N0 version. - Profile job: PID29375, log/tmp/r4/profile.log, output/tmp/r4/profile. - Profile finished 05:29:12Z,ok: communication 36.25%, GEMM 21.56%, attention backward 9.98% / forward 6.56%, copies 4.53%, multiplies 3.75%. Same attention hotspot remains after the library-only alternatives failed. - L3 attention rewrite in/tmp/r4/tree, switchBAGEL_PACKED_FLASH_ATTENTION=1(default off): derive full/causal query groups from each actual dense mask's adjacent diagonals, select each group's keys from its actual final row, and batch groups through native-GQA FlashAttention varlen. Prior noise columns are therefore excluded directly from the source mask; adjacent full spans stay separate. A boundary after an isolated one-token noise span is identified from the lower diagonal. No batch changes, no compiler, same bf16 precision; explicit deterministic FlashAttention backward when deterministic algorithms are required. Expected N1, subject to all gates. - CPU edge-equivalence / GPU forward-backward probe, then judge time: PID30454, logs/tmp/r4/probe.logand/tmp/r4/time.log, timing output/tmp/r4/time. - Probe passed: all edges exactly reconstructed for 505 layouts (500 randomized, including one-token noise and adjacent full spans). Strict deterministic outputs and all Q/K/V gradients repeat bitwise. Relative L2 versus efficient attention: output 0.0008301, dQ 0.0036865, dK 0.0048183, dV 0.0034330; production-mode differences similarly small. Diagnostic only, not a judge verdict. - Time completed 05:37:16Z:/tmp/r4/time/time.json,ok, 2.0470 s/step, speedup1.2693, beyond reference noise. No change promoted yet. - Timing windows 2.0479 / 2.0460 s; peak 105.67 GiB. Quick job PID31859, log/tmp/r4/quick.log, output/tmp/r4/quick. - Quick completed 05:42:41Z:/tmp/r4/quick/check.json,NOT_BITWISE: every exact item passes, numerical hashes differ as expected. No evaluator error. N1 remains provisional until the full conformance and trajectory gates pass. - End: 2026-11-10T05:42:48Z (17.9 minutes from round-4 GPU lock). Clear 1.2693x gain justifies the full check. Keeping the fully validated N0 submission while validating this isolated candidate.
1.9 hFull validation of round 4
- Full unabridged check of/tmp/r4/treewith/tmp/r4/flash.json, including gate 1, exact-item record, both conformance modes, and the N1 trajectory when required. - Full-check job: PID33101, log/tmp/r4/check.log, output/tmp/r4/check. - Completed 2026-11-10T06:11:25Z:/tmp/r4/check/check.json,PASS, tierN1, performance speedup1.2674. Exact items, deterministic and production interleaved conformance, and all trajectory endpoints passed. Promoting the exact tested source patch and switch configuration into the submission.
2.7 hRound 5 — profile accepted N1 implementation
- Start: 2026-11-10T06:12:25Z. Reprofile the newly promoted, fully validated version before choosing another hotspot. - Promotion initially encountered a patch-format newline marker; no source was changed by that rejected patch. Retried with the marker removed, then confirmed all four changed files and the switches are byte-identical to the full-check candidate. An early profile launch (38349) was stopped; its incomplete output is not used. The valid profile uses a fresh output directory. - Accepted-tree profile: PID39123, log/tmp/r5/profile-accepted.log, output/tmp/r5/profile-accepted. - Profile completed 06:18:38Z,ok. CUDA shares: communication 37.08%, GEMM 25.20%, copies 5.05%, multiplies 4.57%, indexing/scatter 4.99%, FlashAttention forward/backward 6.01%, AdamW 2.81% (73.1 ms per step). Memory-bound elementwise work is now the selected bottleneck class. Standard fused paths are tried before custom kernels. - First memory-class substitution (L0):BAGEL_FUSED_ADAMW=1, PyTorch's fused implementation, in/tmp/r5/tree-adam; off by default. Hyperparameters and fp32 master/optimizer-state precision unchanged. N1 expected. - Fused AdamW time job: PID40182, log/tmp/r5/time-adam.log, output/tmp/r5/time-adam. - Preparing the second memory-class substitution in a separate copy (/tmp/r5/tree-norm): library Triton RMSNorm for wide hidden-state norms, leaving 128-wide head norms on the original implementation. The library backward uses one partial-weight-gradient block per SM and a deterministic fp32 final reduction; no atomics or Torch compilation. Guarded for nonempty bf16 CUDA inputs with matching bf16 weights and supported widths. SwitchBAGEL_FUSED_RMSNORM=1, off by default. This preparation uses no GPUs while AdamW is timed. - AdamW time completed 06:25:47Z:/tmp/r5/time-adam/time.json,ok, 2.0098 s/step, speedup1.2928; windows 2.0140 / 2.0057 s, peak 105.67 GiB. About 1.85% faster than the accepted FlashAttention screening run; retained for combined validation, not yet promoted. - Wide RMSNorm probe/time job: PID41244, logs/tmp/r5/probe-norm.logand/tmp/r5/time-norm.log, timing output/tmp/r5/time-norm. The wait enforces a 06:38:30Z cutoff if this experiment stalls, reserving the remainder of the round for quick validation of the best configuration. - RMSNorm correctness probe passed: wide-shape outputs/input gradients/weight gradients have relative L2 about 0.0028–0.0032, deterministic repeats are bitwise, and narrow/empty guarded fallbacks are exact. Dtypes are unchanged. - Combined timing finished 06:33:19Z:/tmp/r5/time-norm/time.json,ok, mean 1.9844 s/step, reported speedup1.3094, peak 103.52 GiB. Windows 2.0595 / 1.9094 s have 7.57% spread, above the final check's 6% limit. This run is not sufficient to accept the RMSNorm gain; a fresh stable timing is required. Possible cold kernel specialization is a hypothesis, not an established diagnosis. - Proceeding with the combined quick exact-item check. A subsequent full check will independently repeat timing; no unvalidated or noisy candidate is promoted. - Combined quick job: PID43003, log/tmp/r5/quick.log, output/tmp/r5/quick. - Quick completed 06:40:39Z:/tmp/r5/quick/check.json,NOT_BITWISE; all exact items pass, including optimizer hyperparameters and state dtypes. The library fusion also avoids the earlier FlexAttention inventory error. - End: 2026-11-10T06:40:53Z (26.4 minutes from the accepted-tree profile GPU lock). Two independent memory-class changes are recorded separately. Their combined candidate stays isolated pending a fresh stable timing and complete numerical validation.
2.7 hFull validation of round 5
- Unabridged full check of/tmp/r5/tree-normwith/tmp/r5/norm.json. It must pass all numerical gates and provide stable timing before promotion; the initial RMSNorm timing alone is not acceptance evidence. - Full-check job: PID44567, log/tmp/r5/check.log, output/tmp/r5/check. - Completed 2026-11-10T07:09:36Z:/tmp/r5/check/check.json,PASS N1, speedup1.3532, candidate 1.9201 s/step, no timing notes. Fresh full-check timing is stable; all exact items, both conformance modes, and all trajectory endpoints pass. Both fusion changes promoted in separate commits, with changed source files checked byte-for-byte against the tested tree.
3.6 hRound 6 — profile the validated fusion stack
- Start: 2026-11-10T07:09:46Z. Profile current best before selecting another change; keep the newly accepted stack intact during experiments. - Profile job: PID51386, log/tmp/r6/profile.log, output/tmp/r6/profile. - Profile completed 07:16:01Z,ok; GPU lock began 07:11:40Z (round boundary 07:41:40Z). Overlapping CUDA totals: communication 38.88%, GEMM 27.02%, FlashAttention backward 3.77% / forward 2.62%, copies 2.94%, index-put 2.86%, multiplies 2.70%, indexing 2.50%. Activation recomputation remains a target; later fusions have reduced memory traffic since the unsuccessful round-2 half-retention experiment. - Experiment: retain half the decoder activations with existing switchBAGEL_DECODER_CHECKPOINT_SKIP_INTERVAL=2, isolated config/tmp/r6/half.json. Submission remains at the accepted interval 4. Retest is motivated by the fused stack's approximately 103.3 GiB allocated peak versus round 2's higher memory use; reserved-memory fragmentation remains a hypothesis, not a finding. - Half-retention time job: PID52779, log/tmp/r6/time-half.log, output/tmp/r6/time-half. - Half-retention completed 07:27:55Z:/tmp/r6/time-half/time.json,ok, 2.1989 s/step, speedup 1.1816; windows 2.1674 / 2.2305 (2.87% range), allocated peak 120.45 GiB, reserved peak 134.85 GiB. Slower than the accepted 1.9201 s/step: not promoted. Test expandable allocator segments with the same policy to investigate memory-pressure overhead;/tmp/r6/half-expandable.jsonis isolated. - Verified round-5 full timing directly: windows 1.9170 / 1.9233 s, 0.33% range, diagnostic window ratios 1.3563 / 1.3501. This confirms stable timing rather than relying on a nonexistent top-level notes field. - Half-retention expandable-segments time job: PID54164, log/tmp/r6/time-half-expandable.log, output/tmp/r6/time-half-expandable. - Expandable-segments run completed 07:34:56Z:/tmp/r6/time-half-expandable/time.json,ok, 2.3654 s/step, speedup 1.0985; windows 2.2296 / 2.5013, 11.49% range (noisy, unsuitable for acceptance), allocated peak 120.40 GiB, reserved peak 134.85 GiB. Both windows lose to the accepted configuration. No gain demonstrated; rejected. No quick check needed for configurations that will not be submitted. - End: 2026-11-10T07:35:12Z, within 30 minutes of the profile GPU lock. No source change or new optimization promoted. Retaining quarter-layer activations remains the validated policy.
4.1 hRound 7 — profile before choosing the next hotspot
- Start: 2026-11-10T07:35:12Z. Profile the unchanged accepted stack; round-6 configuration experiments stay disabled. - Profile job: PID55545, log/tmp/r7/profile.log, output/tmp/r7/profile. - Profile completed 07:40:02Z,ok; lock began 07:35:54Z. Overlapping CUDA shares: communication 40.08%, GEMM 26.04%, FlashAttention backward/forward 3.76%/2.64%; copies 2.94%, index-put 2.82%, multiply 2.67%, indexing 2.50%. Hotspot class: non-GEMM normalization/memory traffic; 128-wide query/key norms still use the original sequence of casts, reductions and multiplies. - Experiment/tmp/r7/tree,/tmp/r7/head.json: extend the existing library RMSNorm fusion to head dimensions 128–1023 behind separate default-offBAGEL_FUSED_HEAD_RMSNORM; wide norms retain their original switch. Expected N1; fp32 accumulation and bf16 compute, parameters and optimizer state unchanged. First measure the library before considering a custom narrow-row kernel. - Head-norm time job: PID56909, log/tmp/r7/time-head.log, output/tmp/r7/time-head. - Head-norm timing completed 07:46:45Z:/tmp/r7/time-head/time.json,ok, 1.8646 s/step, speedup 1.3935; windows 1.8618 / 1.8674 (0.30% range), allocated peak 102.30 GiB. About 3% faster than the accepted stack, with stable timing. Run quick and full numerical validation before promotion. - Head-norm quick job: PID58501, log/tmp/r7/quick.log, output/tmp/r7/quick. - Quick completed 07:51:55Z:/tmp/r7/quick/check.json,NOT_BITWISE; all exact items pass, all four ranks see three identical batches, no dtype or optimizer changes. The existing stack is already N1; complete statistical validation remains required. - End: 2026-11-10T07:52:12Z (16.3 minutes from profile lock). Candidate remains isolated pending the full check. No failure or gate workaround.
4.1 hFull validation of round 7
- Run the unabridged check on/tmp/r7/tree, config/tmp/r7/head.json, including fresh timing, both interleaved conformance modes and trajectory endpoints. Main submission stays on round 5 until the result is known. - Full-check job: PID60293, log/tmp/r7/check.log, output/tmp/r7/check. - Completed 2026-11-10T08:20:04Z:/tmp/r7/check/check.json,PASS N1, speedup 1.3708, 1.8955 s/step; windows 1.8705 / 1.9205 (2.64% range, below the 6% threshold). All exact items, both conformance modes, and all statistical endpoints pass. Promote the exact tested source and configuration; the earlier stable screening was 1.3935x.
4.8 hRound 8 — profile the accepted head-norm stack
- Start: 2026-11-10T08:21:00Z. Reprofile the newly validated stack before selecting the next hotspot. All new work remains isolated until validated. - Profile job: PID68240, log/tmp/r8/profile.log, output/tmp/r8/profile. - Profile completed 08:25:45Z,ok; lock began 08:21:41Z (round boundary 08:51:41Z). Overlapping CUDA totals: communication 40.06%, GEMM 26.80%, FlashAttention backward/forward 3.78%/2.53%; index-put 2.88%, indexing 2.52%, copies 2.19%, zero-fill 2.14%, multiply 1.73%, concatenation 1.67%. Head fusion reduced casts/multiplies; the next class is Q/K preparation's remaining pointwise/layout traffic. - First experiment: existing FlashAttention rotary forward/backward for packed bf16 Q/K, default-offBAGEL_FUSED_ROTARY; canonical duplicated Qwen2 cos/sin are supplied directly, with no change to positions or attention masks. CPU, other dtypes/shapes, and trainable cos/sin retain the original implementation. Isolated/tmp/r8/tree, config/tmp/r8/rotary.json; expected N1. A direct GPU probe checks outputs, gradients and deterministic repetition without timing. - Rotary probe then time job: PID69813, probe log/tmp/r8/probe-rotary.log, job log/tmp/r8/time-rotary.log, timing output/tmp/r8/time-rotary. - Rotary probe passed for lengths 1, 17, 1024 and 10240: output/Q/K-gradient relative L2 errors below 0.003, deterministic repetitions bitwise identical. Timing completed 08:32:55Z:/tmp/r8/time-rotary/time.json,ok, 1.8355 s/step, speedup 1.4156, windows 1.8414 / 1.8296 (0.64% range), allocated peak 102.32 GiB. - Second change in the same Q/K-preparation memory class:BAGEL_DIRECT_QK_ROUTING=1normalizes each expert's projected Q/K before scattering into packed order, removing the intermediate global scatter/gather pair and two zero-filled global buffers. Projection sizes, token indices and masks are unchanged. Frozen-understanding mode uses the unmodified original path. Isolated/tmp/r8/treeand/tmp/r8/direct.json; expected N1 because autograd accumulation order can change. Rotary-only source is retained at/tmp/r8/tree-rotary. - Direct-routing time job: PID71790, log/tmp/r8/time-direct.log, output/tmp/r8/time-direct. - Direct-routing timing completed 08:41:05Z:/tmp/r8/time-direct/time.json,ok, 1.8002 s/step, speedup 1.4434, windows 1.8018 / 1.7987 (0.18% range), allocated peak 102.32 GiB. This adds a measured gain over rotary-only 1.8355 s/step. Before numerical checks, constrained the optional direct path to CUDA bf16 packed inputs (the measured training path) so other input precisions retain their original cast boundaries; the full check will supply fresh timing for this guarded version. - Combined quick job: PID73432, log/tmp/r8/quick.log, output/tmp/r8/quick. - Combined quick completed 08:47:00Z:/tmp/r8/quick/check.json,NOT_BITWISE; every exact item passes, with no reported problems. End of round: 2026-11-10T08:48:28Z, within 30 minutes of the profile GPU lock. The 1.4434x screened stack remains isolated until full validation.
4.8 hFull validation of round 8
- Run the complete judge on/tmp/r8/treewith/tmp/r8/direct.json, including fresh timing of the guarded direct path, interleaved conformance and statistical trajectories. Submission remains the full-check-passing round-7 stack. - Full-check job: PID75280, log/tmp/r8/check.log, output/tmp/r8/check. - Completed 2026-11-10T09:15:44Z:/tmp/r8/check/check.json,PASS N1, speedup 1.4414, candidate 1.8026 s/step; windows 1.7989 / 1.8063 (0.41% range). All exact items, both conformance modes and all statistical endpoints pass. Promoted rotary fusion and direct Q/K routing in separate commits; changed source checked byte-for-byte against the tested candidate.
5.7 hRound 9 — profile the accepted Q/K-preparation stack
- Start: 2026-11-10T09:16:35Z. Reprofile the new accepted stack before selecting another hotspot. Submission contains no unvalidated changes. - Profile job: PID83524, log/tmp/r9/profile.log, output/tmp/r9/profile. - Profile completed 09:21:47Z,ok; GPU lock began 09:17:46Z (round boundary 09:47:46Z). Overlapping CUDA totals: communication 37.45%, GEMM 28.78%, FlashAttention backward/forward 4.26%/2.84%; copies 2.53%, index-put 2.43%, indexing 2.18%, zero-fill 2.02%, multiplication 1.19%, SiLU 0.44%. Q/K changes reduced routing traffic. Next hotspot class: MLP activation memory, including the SiLU/multiply chain and its stored intermediate. - Experiment: default-offBAGEL_FUSED_SWIGLUselects FlashAttention's installed Jiterator SwiGLU forward/backward for nonempty bf16 CUDA inputs with the original SiLU activation. Other activations/dtypes/CPU/empty inputs retain the original path. Parameters, GEMMs and their precision stay unchanged; fp32 fused elementwise arithmetic changes rounding (expected N1). This also avoids retaining the separate SiLU output. Isolated/tmp/r9/tree, config/tmp/r9/swiglu.json. - SwiGLU probe then time job: PID85152, probe log/tmp/r9/probe-swiglu.log, job log/tmp/r9/time-swiglu.log, timing output/tmp/r9/time-swiglu. - SwiGLU probe passed three representative shapes through 1024×18944: output/gradient relative L2 errors below 0.003, repeated deterministic results bitwise identical. Timing completed 09:28:56Z:/tmp/r9/time-swiglu/time.json,ok, 1.7629 s/step, speedup 1.4739, windows 1.7651 / 1.7606 (0.26% range), allocated peak 99.48 GiB (down from 102.32 GiB). - Within the same activation-memory class, test exchanging the saved intermediate memory for less recomputation: existingBAGEL_DECODER_CHECKPOINT_SKIP_INTERVAL=3, isolated/tmp/r9/third.json. This retains nine rather than seven decoder layers' activations; no new source change. The earlier failed half-retention policy is not used. Main submission stays on its full-check-passing configuration. - Third-retention time job: PID86870, log/tmp/r9/time-third.log, output/tmp/r9/time-third. - Third-retention completed 09:35:22Z:/tmp/r9/time-third/time.json,ok, 1.7467 s/step, speedup 1.4876, windows 1.7494 / 1.7440 (0.31% range), allocated peak 105.10 GiB, reserved peak 134.61 GiB. This adds a smaller but resolved screening gain over SwiGLU with interval 4 (1.7629 s/step). Validate the combined interval-3 configuration before promotion. - Combined quick job: PID88524, log/tmp/r9/quick.log, output/tmp/r9/quick. - Quick completed 09:41:20Z:/tmp/r9/quick/check.json,NOT_BITWISE, every exact item passed without problems. End of round: 2026-11-10T09:41:34Z, within 30 minutes of the profile GPU lock. Combined candidate remains isolated until full validation.
5.7 hFull validation of round 9
- Complete check of/tmp/r9/tree, config/tmp/r9/third.json: fresh timing, record, deterministic and production interleaved conformance, and statistical endpoints. Submission remains on the full-check-passing round-8 stack. - Full-check job: PID90365, log/tmp/r9/check.log, output/tmp/r9/check. - Completed 2026-11-10T10:09:18Z:/tmp/r9/check/check.json,PASS N1, speedup 1.4932, candidate 1.7401 s/step; windows 1.7436 / 1.7366 (0.40% range). All exact items, both conformance modes and all statistical endpoints pass. Promoted the exact tested runtime code and switch values. Added a matching optional in-repo preset for reproducibility; it is not read automatically by training.
6.6 hRound 10 — profile the accepted activation-memory stack
- Start: 2026-11-10T10:09:27Z. Reprofile the current full-check-passing stack before selecting the next bottleneck. Experiments remain isolated. - Profile job: PID98616, log/tmp/r10/profile.log, output/tmp/r10/profile. - Profile completed 10:14:48Z,ok; GPU lock began 10:10:44Z (round boundary 10:40:44Z). Overlapping CUDA totals: communication 37.24% (reduce-scatter 25.26%, all-gather 7.98%, other reductions 4.00%), GEMM 28.88%, FlashAttention backward/forward 4.42%/2.85%; copies 2.50%, index-put 2.49%, indexing 2.22%, zero-fill 2.11%. Next hotspot class: FSDP communication scheduling and topology. - Experiment: default-offBAGEL_SINGLE_SHARD_GROUPcollapses the degenerate one-replica hybrid layout into a single shard group, usingSHARD_GRAD_OPwhen gathered weights are retained (otherwiseFULL_SHARD). Guarded by one replica and world size equal to shard count; other topologies use the original path. This removes the size-one replica reduction and its dedicated stream. Parameters, masters, state and reduction dtype remain unchanged; the full stack is N1. Isolated/tmp/r10/tree, config/tmp/r10/single.json. - Profile launch accounting: large all-gather and reduce-scatter kernels each use 24 CUDA blocks of 544 threads; scalar reductions use one block of 96 threads. This provides a concrete channel-occupancy configuration to test if the topology simplification does not help. - Single-group time job: PID100250, log/tmp/r10/time-single.log, output/tmp/r10/time-single. - Single-group timing completed 10:21:24Z:/tmp/r10/time-single/time.json,ok, 1.7292 s/step, speedup 1.5026; windows 1.7335 / 1.7249 (0.50% range), allocated peak 105.10 GiB. The gain over the accepted full check (1.7401 s/step) is small and needs further confirmation; not promoted. - Second communication configuration: cap NCCL channels at 12 (profile default was 24 blocks), on top of the single-group candidate, via/tmp/r10/channels12.json. This tests communication/compute occupancy rather than changing tensor precision or collective semantics. Installed NCCL is 2.21.5; the existing channel-cap setting was checked against NVIDIA's official documentation. No library was changed. - Channel-cap time job: PID101839, log/tmp/r10/time-channels12.log, output/tmp/r10/time-channels12. - Channel-cap timing completed 10:28:17Z:/tmp/r10/time-channels12/time.json,ok, 1.7386 s/step, speedup 1.4945; windows 1.7565 / 1.7207 (2.06% range). No resolved gain over the accepted 1.7401 s/step and inconsistent per-window changes: rejected. The single-group result's approximately 0.6% gain is also smaller than twice its own window range, so it remains unconfirmed and is not promoted. No precision verdict is claimed for either experiment. - End: 2026-11-10T10:28:34Z, within 30 minutes of the profile lock. No submitted source/configuration change; optional experimental topology code stays only in/tmp/r10/tree. No full check is justified by these marginal measurements.
7.0 hRound 11 — profile before the next communication rewrite
- Start: 2026-11-10T10:29:00Z. Profile the unchanged full-check-passing round-9 stack before selecting another target. - Profile job: PID103428, log/tmp/r11/profile.log, output/tmp/r11/profile. - Profile completed 10:34:41Z,ok; lock began 10:30:40Z (round boundary 11:00:40Z). Overlapping CUDA totals: communication 37.71%, GEMM 28.49%, FlashAttention backward/forward 4.42%/2.83%; scalar all-reductions 4.16%, almost all from six int64 reductions over two steps. The loop constructs and synchronizes CE/MSE count tensors only after forward, creating an avoidable loss-normalization barrier. - Experiment: default-offBAGEL_PREFETCH_LOSS_COUNTScombines the same two integer lengths in one int64 reduction before forward, overlapping it with VAE encoding. It waits before entering FSDP's model forward to preserve ordering between process groups. Loss formulas, count values/dtypes, batch fields and RNG order are unchanged; reweighted CE keeps its original weight reduction. Isolated/tmp/r11/tree,/tmp/r11/counts.json; expected numerically unchanged, with the existing stack still N1. - Count-prefetch time job: PID105073, log/tmp/r11/time-counts.log, output/tmp/r11/time-counts. - Count-prefetch timing completed 10:41:41Z:/tmp/r11/time-counts/time.json,ok, 1.7460 s/step, speedup 1.4882; windows 1.7447 / 1.7473 (0.15% range), allocated peak 105.10 GiB. No gain over the accepted 1.7401 s/step; not promoted. Run the quick gate to record exact-item correctness of the changed normalization plumbing before closing this direction. - Count-prefetch quick job: PID106726, log/tmp/r11/quick.log, output/tmp/r11/quick. - Quick completed 10:48:41Z:/tmp/r11/quick/check.json,NOT_BITWISE; all exact items pass without problems. No full check is justified because timing showed no benefit. End: 2026-11-10T10:49:08Z, within 30 minutes of the profile lock. Count-prefetch code remains isolated and disabled in the submission.
7.3 hRound 12 — profile before further memory-traffic work
- Start: 2026-11-10T10:49:08Z. Profile the unchanged accepted stack before selecting a new hotspot; inspect remaining native fast paths during setup. - Profile job: PID108569, log/tmp/r12/profile.log, output/tmp/r12/profile. - Profile completed 10:53:59Z,ok; lock began 10:50:00Z (round boundary 11:20:00Z). Overlapping CUDA totals: communication 39.20%, GEMM 27.32%, FlashAttention backward/forward 4.38%/2.80%; copies 2.54%, index-put 2.49%, indexing 2.18%, zero-fill 2.07%. Selected hotspot class: expert-routing forward/backward memory traffic; generic independent indexing creates avoidable dense gradient copies and zero fills. - Experiment: default-offBAGEL_PARTITION_ROUTINGuses joint autograd gather/scatter for the upstream disjoint understanding/generation index sets. Nativeindex_select/index_copy_handle both directions; uncovered positions remain zero, and empty partitions are supported. Changes apply only to bf16 CUDA MoT training with understanding unfrozen; original paths remain otherwise. The installed PyTorch 2.5.1 documents deterministicindex_copy, and a strict deterministic GPU probe verifies copied outputs and gradients. Parameters, masks and projection arithmetic stay unchanged; accumulation scheduling may differ (N1 stack). Isolated/tmp/r12/tree,/tmp/r12/partition.json. - Partition-routing probe then time job: PID110213, probe log/tmp/r12/probe-partition.log, job log/tmp/r12/time-partition.log, output/tmp/r12/time-partition. - Native partition probe passed: outputs and input gradients match the original indexing exactly for 2-D/3-D tensors, 10240×3584, empty understanding partitions, and uncovered positions, under strict determinism. Timing completed 11:01:11Z:/tmp/r12/time-partition/time.json,ok, 1.7292 s/step, speedup 1.5026, windows 1.7226 / 1.7359 (0.77% range), allocated peak 105.05 GiB. Its small gain alone is not resolved above window variability. - Next memory-class variant:BAGEL_PARTITION_SKIP_ZERO=1uses empty allocation only when the upstream disjoint partitions cover every row; all rows are overwritten before any read. Incomplete partitions keep zero initialization. The prior hole-coverage probe is preserved as/tmp/r12/probe_partition_holes.py; an updated strict probe exercises complete partitions, including an empty understanding partition. This changes no arithmetic and removes remaining redundant fills; private config/tmp/r12/partition-empty.json. - Complete-partition probe then time job: PID112136, probe log/tmp/r12/probe-empty.log, job log/tmp/r12/time-empty.log, output/tmp/r12/time-empty. - Complete-partition strict probes passed (including empty understanding): outputs and gradients equal the original indexing. Timing completed 11:10:58Z:/tmp/r12/time-empty/time.json,ok, 1.7235 s/step, speedup 1.5076, windows 1.7290 / 1.7179 (0.65% range), allocated peak 105.05 GiB. Both routing variants screen slightly faster, but the incremental gain remains small relative to window variability. Quick validation precedes any further work; main submission remains unchanged. - Routing quick job: PID114061, log/tmp/r12/quick.log, output/tmp/r12/quick. - Quick completed 11:18:24Z:/tmp/r12/quick/check.json,NOT_BITWISE, all exact items passed without problems. End of round: 2026-11-10T11:18:28Z, within 30 minutes of the profile lock. Full validation is deferred until a larger cumulative gain is established; the best screened routing experiment is retained privately, not promoted.
7.8 hRound 13 — profile the best screened routing experiment
- Start: 2026-11-10T11:18:28Z. Profile/tmp/r12/treewith/tmp/r12/partition-empty.json, the best screened experiment (1.5076x, quick exact items pass), before another rewrite. The submission remains the fully validated round-9 tree at 1.4932x. - Profile job: PID115908, log/tmp/r13/profile.log, output/tmp/r13/profile. - Profile completed 11:23:27Z,ok; lock began 11:19:31Z (round boundary 11:49:31Z). Overlapping CUDA totals: communication 37.59%, GEMM 28.75%, FlashAttention backward/forward 4.58%/2.95%; index-select 4.29%, index-copy 2.83%, copies 2.51%. Native routing still spends substantial time materializing intermediate packed/expert layouts. - New graph rewriteBAGEL_LOCAL_EXPERT_STATES: keep each expert's residual, normalized inputs and MLP activations in expert order within a decoder layer. Attention accepts the already-normalized expert inputs and can return expert-local output projections; packed Q/K/V, positions, attention masks and all GEMM shapes remain unchanged. Only the layer boundary scatters the final states. Guarded by bf16 CUDA, unfrozen understanding, existing routing switches, and complete disjoint partitions; all other paths remain original. Isolated/tmp/r13/tree,/tmp/r13/local.json; expected N1 due accumulation scheduling. A small strict deterministic layer probe checks full outputs and parameter/input gradients on adjacent-full/noise-exclusion masks before timing. - Local-state probe then time job: PID117541, probe log/tmp/r13/probe-local.log, job log/tmp/r13/time-local.log, output/tmp/r13/time-local. - Local-state strict layer probes passed for lengths 37 and 127 with adjacent full spans and later noise exclusion: forward outputs exact, input-gradient relative error 0.0, all parameter gradients within bf16 tolerances. Timing completed 11:31:25Z:/tmp/r13/time-local/time.json,ok, 1.6634 s/step, speedup 1.5621, windows 1.6763 / 1.6504 (1.56% range), allocated peak 105.02 GiB. The cumulative gain over the accepted 1.7401 s/step now justifies full validation after the quick gate. - Local-state quick job: PID119702, log/tmp/r13/quick.log, output/tmp/r13/quick. - Quick completed 11:38:07Z:/tmp/r13/quick/check.json,NOT_BITWISE; all exact items passed without problems. End of round: 2026-11-10T11:38:10Z, within 30 minutes of the profile lock. The cumulative 1.5621x screen justifies full validation of rounds 12–13 together.
8.6 hRound 14 — profile the validated local-expert stack
- Start: 2026-11-10T12:08:13Z. Reprofile the newly accepted stack before selecting the next hotspot. Submission is fully validated; subsequent work remains isolated. - Profile job: PID129833, log/tmp/r14/profile.log, output/tmp/r14/profile. - Profile completed 12:14:05Z,ok; GPU lock began 12:10:10Z (round deadline 12:40:10Z). Overlapping CUDA totals: communication 37.64% (reduce-scatter 25.41%, all-gather 7.61%, other reductions 4.63%), matrix multiply 29.46%, FlashAttention backward 4.78% and forward 3.10%, index-select 3.01%, index-copy 2.21%, norm backward 1.70%, fused AdamW 1.67%. Next class: replace the remaining FA2 attention kernels with the official Hopper-specialized library, without changing mask construction or precision. - Official Dao-AILab/flash-attention v2.8.3 (060c918) isolated source build: PID131670, log/tmp/r14/build.log, wheel destination/tmp/r14/wheels. Head dimension 128 and bf16 only; GQA, varlen and deterministic backward retained. Main submission and its installed FA2 stay untouched. Local toolkit is CUDA 12.4; this official build downloads its own nvcc/ptxas into its source/cache paths, not into the training environment. - First build failed at 12:22:37Z before compilation: NVIDIA toolchain download SSL handshake timeout. Retrying using the already installed CUDA 12.4 (which meets the library's minimum 12.3), with only the optional toolchain-download block disabled in the isolated setup script. Retry PID132088, log/tmp/r14/build-local.log; no training library has been replaced. An immediately stopped retry (PID 131972) preceded the corrected setup guard. - Local build succeeded 12:24:56Z. Installed separateflash-attn-3 3.0.0b1into the candidate environment, without dependencies or changes to FA2/PyTorch. Wheel SHA-256 a97a2d551c622cbf3ee1c3e392f9649b879c62b9ccf7c6c3a296bf43d9dac1ba. Isolated adapter/tmp/r14/tree, switchBAGEL_FLASH_ATTENTION_3in/tmp/r14/fa3.json, default off. Strict production-shape and interleaved-mask correctness probe PID132492, log/tmp/r14/probe.log(GPU lock held). - FA3 correctness investigation: the dense-mask comparison had output relative L2 0.00102 and gradient relative L2 at most 0.00477, but strict repeated backward was not bitwise deterministic. An isolated bare-FA3 call on 8100 tokens, 28/4 heads and width 128 confirmed this in both causal modes withdeterministic=True(/tmp/r14/determinism.log), independent of the packing adapter. Therefore the optional backend uses the existing FA2 implementation undertorch.are_deterministic_algorithms_enabled(), as required by the task, and FA3 only in production. Both remain bf16 compute with unchanged model/state precision. The strict probe remains strict; its assertions are not relaxed. - Fallback correctness probe plus judge time and quick job: PID132971, log/tmp/r14/screen.log; probe/tmp/r14/probe-fallback.log, timing/tmp/r14/time, quick/tmp/r14/quick. Submission stays on the validated round-13 stack. - Round completed 2026-11-10T12:36:58Z, within 30 minutes of the profile GPU lock./tmp/r14/time/time.json:ok, 1.6124 s/step, speedup 1.6115, windows 1.6172 / 1.6076 (0.60% range), peak allocation 105.02 GiB. This is a resolved screening improvement over round 13's 1.6491 s/step./tmp/r14/quick/check.json:NOT_BITWISE, all exact items pass without problems. Correctness probe passes 506 layouts and strict-repeat assertions with FA2 fallback; production FA3 tensor/gradient errors remain small. Candidate remains isolated pending a full check.
8.6 hFull validation of round 14
- Full validation of/tmp/r14/treeand/tmp/r14/fa3.json: new timing, record, deterministic and production interleaved conformance, and statistical endpoints. Submission remains on the full-check-passing round-13 stack. - Full-check job: PID136781, log/tmp/r14/check.log, output/tmp/r14/check. - Full check completed 2026-11-10T13:03:45Z:/tmp/r14/check/check.json,PASS N1; every exact item, both conformance modes and all trajectory endpoints pass. Timing was noisy: speedup 1.5618, 1.6637 s/step, windows 1.7350 / 1.5923 (8.58% range), including a 5.4244-second outlier; the judge reports interference. This does not supersede the stable screening evidence (1.6115x, 0.60% window range). Promoted the exact validated runtime code based on that stable screening improvement and full numerical validation; fresh timing will confirm it. The old round-13 commit remains available if the gain does not repeat.
9.6 hRound 15 — profile the fully validated Hopper stack
- Start: 2026-11-10T13:06:01Z. Profile the current stack, then obtain fresh timing before selecting/promoting another change; the preceding full check established correctness but its timing had a large outlier. - Profile and fresh timing job: PID145035, log/tmp/r15/baseline.log, outputs/tmp/r15/profileand/tmp/r15/time-baseline. - Profile completed 13:10:19Z; GPU lock began 13:06:25Z (round deadline 13:36:25Z). CUDA totals overlap: communication 38.33% (reduce-scatter 24.87%, all-gather 7.88%), GEMM 31.37%, index-select 3.11%, index-copy 2.28%, cat 1.81%; FA3 backward/forward now 2.45%/1.41%. Detailed kernel accounting gives all RMSNorm backward/forward 1.77%/0.57% and rotary only 0.25%. Therefore defer the considered head-norm/rotary fusion: existing kernels leave little ceiling. Select attention-layout traffic instead, reusing the accepted joint-partition autograd rather than writing a new kernel. - Fresh baseline time completed 13:15:52Z:/tmp/r15/time-baseline/time.json,ok, 1.5985 s/step, speedup 1.6255, windows 1.5999 / 1.5972 (0.17% range), peak allocation 105.02 GiB. This confirms the FA3 improvement and supersedes the noisy full-check timing as screening evidence, not as a new full-check verdict. - IsolatedBAGEL_ATTENTION_PARTITION_ROUTING: jointly gather the disjoint causal/bidirectional query partitions and directly scatter their outputs, eliminating concatenation/inverse-gather traffic and independent zero-filled query-gradient buffers. A single query partition is already in original token order and needs neither permutation. Key/value gathering and every mask edge remain unchanged. Uses the existing native, deterministic partition operators; no custom kernel or precision change. - Correctness probe and judge time/quick job: PID148392, log/tmp/r15/screen.log, outputs/tmp/r15/probe.log,/tmp/r15/time-routing,/tmp/r15/quick. - Screening completed 13:29:06Z./tmp/r15/time-routing/time.json:ok, speedup 1.6472, 1.5774 s/step, windows 1.5700 / 1.5848 (0.94% range), peak allocation 105.02 GiB. Both windows improve on the fresh baseline, but the mean gain is modest enough to repeat. Quick:NOT_BITWISE, all exact items pass. The strict probe is bitwise equal to the original adapter's output and all Q/K/V gradients, and matches 506 mask-layout cases. - Additional single-partition strict probe and repeat timing job: PID152220, log/tmp/r15/repeat.log, outputs/tmp/r15/single-probe.logand/tmp/r15/time-repeat.