GPT-6 Astrasuspected nerf 1.41×

Harness: Codex · switches: none

Judged
1.41×
Tier
N1
Self-best
1.29×
Rounds
40
Hours
10.0
API cost
$187
Input tokens · 99.2% cached
165 M
Output tokens
0.2 M
Requests
1,079
Sequence packing
✗ not done (tried "fixed-length masking", reverted)
Recompute / activations
✓ selective checkpointing + activations kept for 9 layers
Optimizer
✗ not in the log
Communication & libraries
resident params; NCCL_ALGO=Tree / PROTO=Simple; ZeRO-2 off by 0.585%, dropped
Other main levers
Triton RMSNorm (1.068 → 1.173×), fused SwiGLU, bf16 logits; best self-test 1.28×, final judge measured 1.41×
Raw work log40 sections
—Round 0 (2026-11-13 UTC)
- Baseline profile: /tmp/r0/profile/profile_tree.md (unchanged upstream tree). One steady step was 2.692 s GPU time; top kernels were bf16 GEMMs (19.7%, 13.5%, 7.1%, 5.3%, 5.2%; ~73% combined), NCCL all-gather 10.0%, and elementwise/layout kernels. CPU busy was 2.34/48 cores; no compile in the profile window.
- Baseline timing: /tmp/r0/time/time_tree.json, 2781.3 ms/step, speedup_vs_frozen_reference=0.9992, valid but within reference noise.
- Hotspot classification: compute plus ZeRO-3 communication; first configuration lever is reducing parameter collectives.
—Round 1 (2026-11-13 UTC)
- Change: added switchable ZeRO-2 config (scripts/zero2.json) selected by LLAVA_USE_ZERO2=1 in llava/train/train_mem.py; default remains ZeRO-3.
- Timing: /tmp/r1/time-zero2/time_tree.json, 2684.4 ms/step (speedup_vs_frozen_reference=1.0352, beyond noise), peak memory 82.58 GiB. The run was rejected as ERROR because the first logged loss differed by 0.585% (limit 0.5%); data probe inputs matched exactly. ZeRO-2 therefore remains an experiment and is not enabled.
—Round 2 (2026-11-13 UTC)
- Change: switchable ZeRO-3 config (scripts/zero3_persist.json, LLAVA_USE_PERSISTENT_ZERO3=1) with a 100M persistence threshold and large live/reuse limits; 6.8B parameters became persistent and peak memory rose to 84.54 GiB.
- Timing: /tmp/r2/time-persist/time_tree.json, 2747.3 ms/step (speedup_vs_frozen_reference=1.0115, beyond measured noise but below the 1.02 benefit bar). Loss and data checks passed. Not enabled.
—Round 3 (2026-11-13 UTC)
- Profile finding used: LLaMA FlashAttention _get_unpad_data appeared 64 times in the one-step trace, with 807 ms CPU nonzero and 839 ms metadata/copy work; this is repeated mask metadata and launch/synchronization overhead.
- Change: switchable cache of unpadding metadata attached to the immutable attention mask (LLAVA_CACHE_UNPAD=1) in llava/model/language_model/llava_llama.py.
- Timing: /tmp/r3/time-cache-unpad/time_tree.json, 2784.9 ms/step (speedup_vs_frozen_reference=0.9979, no benefit). The metadata work overlaps enough with GPU compute that the cache is a small regression. Not enabled.
—Round 4 (2026-11-13 UTC)
- Profile hotspot: baseline recomputation was 1.664 s, so layer checkpointing was the next lever.
- Torch-compile layer experiment (LLAVA_COMPILE_LAYERS=1) failed before step 1 inside DeepSpeed hooks (TypeError: 'int' object is not iterable); disabled.
—Round 5 (2026-11-13 UTC)
- Removing gradient checkpointing (LLAVA_NO_CHECKPOINT=1) OOMed at 139.7 GiB before the first step; rejected.
—Round 6 (2026-11-13 UTC)
- Selective checkpointing retaining half the layers (LLAVA_CHECKPOINT_STRIDE=2) OOMed at 139.7 GiB after step 1; rejected.
—Round 7 (2026-11-13 UTC)
- Selective checkpointing retaining 8 evenly spaced decoder layers (LLAVA_CHECKPOINT_STRIDE=4) reduced timing to 2623.0 ms/step, speedup_vs_frozen_reference=1.0595, peak 120.14 GiB.
- /tmp/r7/quick/check.json: PASS, bitwise N0 (4,724/4,724 exact).
- /tmp/r8/skip/check.json: PASS; trajectory equivalent (train endpoint bias -0.000159, probe +0.000142).
—Round 8/9 (2026-11-13 UTC)
- Full check timing was noisy (candidate 2717.3 ms/step, one 6.8% window spread) and its text record process had a transient native double-free after writing records. The non-timing rerun passed all exact and trajectory gates.
- /tmp/r9/profile/profile_tree.md: selective profile GPU 2495 ms; checkpoint backward 1247 ms, NCCL all-gather 211 ms on the nonpersistent path.
—Round 10/11 (2026-11-13 UTC)
- Retaining one additional layer (early or late) OOMed around 137–138 GiB after 10 steps; current 8-layer memory ceiling confirmed.
- Parameterized checkpoint exclusions; extra-layer timing artifacts: /tmp/r10/time-extra and /tmp/r11b/time-extra31 (OOM).
—Round 12/13/14 (2026-11-13 UTC)
- Non-reentrant checkpointing was valid but slower: /tmp/r12/time-nonreentrant/time_tree.json, 2636.9 ms/step, speedup 1.0539; disabled.
- Combining 8 retained layers with persistent ZeRO-3 OOMed; balancing to 7 retained layers plus scripts/zero3_persist.json (100M persistence threshold) fit at 125.39 GiB and timed 2614.8 ms/step, speedup 1.0628.
- /tmp/r15/quick/check.json: PASS bitwise; /tmp/r16/skip/check.json: PASS trajectory equivalent (train bias -0.000208, probe +0.000216).
—Round 15/16 (2026-11-13 UTC)
- Lower persistence threshold (20M) with 8 retained layers still OOMed at 138.4 GiB (/tmp/r17/time-persist20).
- Context-aware FlashAttention unpadding cache reuses metadata across checkpoint-detached masks; /tmp/r21/time-cache-context/time_tree.json: 2610.3 ms/step, speedup 1.0646. /tmp/r21/quick/check.json: PASS bitwise.
- Dropping the right-padding mask to fixed-length attention increased memory and OOMed (/tmp/r20/time-dropmask); disabled.
—Round 17/18 (2026-11-13 UTC)
- NCCL backend screen with the validated stack: Tree gave 2605.2 ms/step (1.0667×); Tree + Simple protocol gave 2603.7 ms/step (1.0673×), stable windows and unchanged memory. Ring + Simple (2610.8 ms) and Tree + LL128 (2612.1 ms) were slower.
- /tmp/r23/quick/check.json: PASS bitwise with Tree + Simple.
—Round 19 (2026-11-13 UTC)
- Start/end: 07:42:59–07:57:38 UTC. Profiled the current six-switch stack at /tmp/r19/profile/profile_tree.md; GPU kernels remained GEMM-dominated (top GEMMs 19.5%, 15.6%, 8.0%, 6.2%, 5.0%), with NCCL reduce-scatter/all-gather at 2.5%/2.1%. The detailed CPU view showed a 323 ms aten::item wait inside DeepSpeedEngine.backward, caused by unconditional loss-monitor logging while monitoring is disabled. This was classified as a host-sync candidate, with the profiler trace as the evidence pack.
- Change: added switchable runtime patch LLAVA_SKIP_DS_LOSS_ITEM in llava/train/fast_deepspeed.py and train_mem.py; it preserves the monitor-enabled path and skips only the disabled-monitor scalar logging conversion.
- Timing with the switch enabled: /tmp/r19/time-loss-item/time_tree.json, valid 2605.3 ms/step (speedup_vs_frozen_reference=1.0667, window spread 0.05%). This is slightly slower than the validated 2602.1 ms/step stack and provides no gain; the profiler-only scalar wait is overlapped or instrumentation-induced in the real path.
- Exactness with the switch enabled: /tmp/r19/quick-loss/check.json, PASS, N0, 4,724/4,724 bitwise records and matching data probe. The switch is removed from /submission/switches.json; the patch remains available but disabled.
- Conclusion: failed/no-gain host-sync direction; keep the prior six-switch configuration as the current best.
—Round 20 (2026-11-13 UTC)
- Start/end: 07:59:58–08:37:40 UTC. Fresh profile /tmp/r20/profile/profile_tree.md confirmed the prior stack: 2294 ms GPU kernel time, GEMMs at 19.7%, 15.6%, 8.1%, 6.3%, 5.0%, and the largest non-GEMM elementwise class at 122.8 ms plus bf16 adds at 54.5 ms and multiplies at 42.9 ms.
- Change: added LLAVA_USE_TRITON_RMSNORM=1 in llava/model/language_model/llava_llama.py, replacing Transformers' eager LlamaRMSNorm forward with the pinned FlashAttention Triton RMSNorm kernel. Parameter names/shapes and the fp32 reduction contract remain unchanged; the kernel's reduction ordering makes this N1.
- The first timed attempt /tmp/r20/time-triton-rms failed before step 1 because the wrapper call passed an unsupported keyword; corrected in place and reran the same hotspot.
- Valid timing: /tmp/r20/time-triton-rms2/time_tree.json, 2363.3 ms/step (speedup_vs_frozen_reference=1.1759), stable 0.43% window spread, peak allocated 110.74 GiB, no compile/fallback signature. The full check timing was 2369.1 ms/step (speedup=1.173).
- Quick exactness: /tmp/r20/quick-triton-rms/check.json, PASS, N1, data/setup exact and no exact-item failures (32/4724 tensors bitwise due to the expected reduction reorder).
- Full check: /tmp/r20/full-triton-rms/check.json, PASS, N1; trajectory equivalent with train endpoint bias -0.000288 and probe bias +0.000129.
—Round 21 (2026-11-13 UTC)
- Start/end: 08:44:56–08:56:09 UTC. Profiled the accepted Triton RMSNorm stack at /tmp/r21/profile/profile_tree.md; GPU time fell to 2091 ms, with GEMMs at 22.1%, 17.4%, 8.9%, 6.9%, 5.6%, the remaining bf16 binary class at 87.2 ms, bf16 add at 55.0 ms, and NCCL reduce-scatter/all-gather at 52.2/45.0 ms.
- Change: tested LLAVA_FUSE_RESIDUAL_RMSNORM=1, which routes the attention residual add and post-attention RMSNorm through FlashAttention Triton rms_norm_fn(..., prenorm=True) while retaining the pre-norm residual for the MLP skip.
- Timing: /tmp/r21/time-fused-rms/time_tree.json, valid 2364.3 ms/step (speedup_vs_frozen_reference=1.1754) versus the accepted RMSNorm-only 2363.3 ms/step; internal window spread was 1.04%, so there is no measurable gain.
- Exactness: /tmp/r21/quick-fused-rms/check.json, PASS N1 with no exact failures; one diagnostic unresolved tolerance is left for gate 3. The switch is removed from /submission/switches.json; the fusion remains in code but disabled.
- Conclusion: residual-plus-RMSNorm fusion is a neutral/no-gain direction; keep Triton RMSNorm-only as the current best.
—Round 22 (2026-11-13 UTC)
- Start/end: 09:00:15–09:17:22 UTC. The Round 21 profile /tmp/r21/profile/profile_tree.md identified the remaining separate SiLU and multiply kernels after Triton RMSNorm (about 18/36 ms classes), so this round targeted the SwiGLU activation fusion.
- Change: added LLAVA_USE_FUSED_SWIGLU=1 in llava/model/language_model/llava_llama.py; LlamaMLP.forward now calls the pinned FlashAttention swiglu autograd kernel for the recipe's pretraining_tp=1, SiLU model, retaining the original path for other configurations.
- Timing: /tmp/r22/time-swiglu/time_tree.json, valid 2332.4 ms/step (speedup_vs_frozen_reference=1.1915), 0.68% window spread, peak allocated 106.23 GiB, no invalid signature. Full check timing was 2334.2 ms/step (speedup=1.1906).
- Quick exactness: /tmp/r22/quick-swiglu/check.json, PASS N1, no exact failures; six diagnostic tolerance violations are left for the statistical gate.
- Full check: /tmp/r22/full-swiglu/check.json, PASS N1; trajectory equivalent (train endpoint bias -0.000271, probe bias +0.0000058).
—Round 23 (2026-11-13 UTC)
- Start/end: 09:19:23–09:36:37 UTC. Profile /tmp/r23/profile/profile_tree.md showed checkpoint recomputation still dominated at 1117 ms CPU per step, while fused kernels reduced peak memory to 106 GiB.
- Change: tested eight retained decoder layers with the existing stride-4 selective checkpointing switch and no extra checkpoint at layer 28 (/tmp/r23/switches-8.json). This removes one recomputed layer while keeping the same data, parameters, optimizer, and schedule.
- Timing: /tmp/r23/time-8retain/time_tree.json, valid 2319.9 ms/step (speedup_vs_frozen_reference=1.1979), 0.51% window spread, 109.46 GiB peak allocation.
- Quick exactness: /tmp/r23/quick-8retain/check.json, PASS N1, no exact failures.
- Full check: /tmp/r23/full-8retain/check.json, PASS N1 at 2318.0 ms/step (speedup=1.1989); trajectory equivalent (train endpoint bias -0.000163, probe bias -0.000096).
- Promotion: removed LLAVA_CHECKPOINT_EXTRA=28 from /submission/switches.json; the code default retains all stride-4 layers, so the submitted configuration now uses eight retained layers.
—Round 24 (2026-11-13 UTC)
- Start/end: 09:39:17–09:57:23 UTC. Profile /tmp/r24/profile/profile_tree.md showed checkpoint backward at 1073 ms/step, with 30 GiB of memory headroom after the fused kernels.
- Change: added switchable LLAVA_CHECKPOINT_RETAIN support in llava/model/language_model/llava_llama.py; the temporary config retained late decoder layer 31 in addition to the eight stride-4 layers.
- Timing: /tmp/r24/time-retain31/time_tree.json, valid 2303.8 ms/step (speedup_vs_frozen_reference=1.2063), 0.27% window spread, 112.69 GiB peak allocation.
- Quick exactness: /tmp/r24/quick-retain31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r24/full-retain31/check.json, PASS N1 at 2312.6 ms/step (speedup=1.2017); trajectory equivalent (train endpoint bias -0.000211, probe bias -0.000035).
- Promotion: /submission/switches.json now sets LLAVA_CHECKPOINT_RETAIN=31.
—Round 25 (2026-11-13 UTC)
- Start/end: 10:00:14–10:17:32 UTC. Profile /tmp/r25/profile/profile_tree.md showed checkpoint backward at 1012 ms/step after retaining layer 31, leaving memory margin for another late layer.
- Change: promoted the same LLAVA_CHECKPOINT_RETAIN mechanism with layers 30,31, retaining two adjacent late decoder layers in addition to the eight stride-4 layers.
- Timing: /tmp/r25/time-retain30-31/time_tree.json, valid 2286.7 ms/step (speedup_vs_frozen_reference=1.2153), 0.54% window spread, 115.93 GiB peak allocation.
- Quick exactness: /tmp/r25/quick-retain30-31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r25/full-retain30-31/check.json, PASS N1 at 2288.8 ms/step (speedup=1.2142); trajectory equivalent (train endpoint bias +0.000027, probe bias -0.000044).
- Promotion: /submission/switches.json now sets LLAVA_CHECKPOINT_RETAIN=30,31.
—Round 26 (2026-11-13 UTC)
- Start/end: 10:19:56–10:37:18 UTC. Profile /tmp/r26/profile/profile_tree.md showed checkpoint backward at 981 ms/step after retaining layers 30–31.
- Change: tested LLAVA_CHECKPOINT_RETAIN=29,30,31, retaining a third adjacent late decoder layer while preserving the stride-4 checkpoints.
- Timing: /tmp/r26/time-retain29-31/time_tree.json, valid 2269.4 ms/step (speedup_vs_frozen_reference=1.2245), 0.36% window spread, 119.16 GiB peak allocation.
- Quick exactness: /tmp/r26/quick-retain29-31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r26/full-retain29-31/check.json, PASS N1 at 2278.9 ms/step (speedup=1.2194); trajectory equivalent (train endpoint bias -0.000160, probe bias -0.000229).
- Promotion: /submission/switches.json now sets LLAVA_CHECKPOINT_RETAIN=29,30,31.
—Round 27 (2026-11-13 UTC)
- Start/end: 10:39:14–10:56:47 UTC. Profile /tmp/r27/profile/profile_tree.md showed checkpoint backward at 939 ms/step after retaining layers 29–31.
- Change: tested LLAVA_CHECKPOINT_RETAIN=27,29,30,31, retaining a fourth late decoder layer.
- Timing: /tmp/r27/time-retain27-31/time_tree.json, valid 2257.2 ms/step (speedup_vs_frozen_reference=1.2312), 0.69% window spread, 122.4 GiB peak allocation.
- Quick exactness: /tmp/r27/quick-retain27-31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r27/full-retain27-31/check.json, PASS N1 at 2259.6 ms/step (speedup=1.2299); trajectory equivalent (train endpoint bias -0.000082, probe bias +0.000087).
- Promotion: /submission/switches.json now sets LLAVA_CHECKPOINT_RETAIN=27,29,30,31.
—Round 28 (2026-11-13 UTC)
- Start/end: 10:58:54–11:16:52 UTC. Profile /tmp/r28/profile/profile_tree.md showed checkpoint backward at 898 ms/step after retaining layers 27 and 29–31.
- Change: tested LLAVA_CHECKPOINT_RETAIN=26,27,29,30,31, retaining a fifth late decoder layer.
- Timing: /tmp/r28/time-retain26-31/time_tree.json, valid 2237.1 ms/step (speedup_vs_frozen_reference=1.2423), 0.46% window spread, 125.63 GiB peak allocation.
- Quick exactness: /tmp/r28/quick-retain26-31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r28/full-retain26-31/check.json, PASS N1 at 2236.3 ms/step (speedup=1.2427); trajectory equivalent (train endpoint bias -0.000190, probe bias +0.000073).
- Promotion: /submission/switches.json now sets LLAVA_CHECKPOINT_RETAIN=26,27,29,30,31.
—Round 29 (2026-11-13 UTC)
- Start/end: 11:18:26–11:32:47 UTC. Profile /tmp/r29/profile/profile_tree.md showed checkpoint backward at 852 ms/step after retaining layers 26,27,29–31.
- Change: screened a sixth late retained layer with LLAVA_CHECKPOINT_RETAIN=25,26,27,29,30,31 in /tmp/r29/switches-retain25-31.json.
- Timing: /tmp/r29/time-retain25-31/time_tree.json, valid 2217.3 ms/step (speedup_vs_frozen_reference=1.2534), 0.33% window spread, 128.87 GiB peak allocation.
- Quick exactness: /tmp/r29/quick-retain25-31/check.json, PASS N1, no exact failures.
- Full check: /tmp/r29/full-retain25-31/check.json, rejected with ERROR: the 60-step trajectory OOMed at step 10 while timing remained valid (125.99 GiB allocated, 8.83 GiB reserved, 2.77 GiB free, attempted 3.75 GiB allocation). The temporary switch was not promoted; the accepted five-layer configuration remains submitted.
- Conclusion: sixth retained layer is unsafe under the statistical trajectory memory pattern; keep disabled.
—Round 30 (2026-11-13 UTC)
- Start/end: 11:34:59–11:53:27 UTC. Profile /tmp/r30/profile/profile_tree.md for the six-layer candidate showed 1986 ms/step GPU kernel time; the largest kernels were bf16 GEMMs at 21.0%, 18.2%, 9.6%, 7.2%, and 5.3%, with checkpoint backward still 852.5 ms/step and DeepSpeed aten::item CPU time 335.2 ms/step. The allocator had enough timing headroom but the prior six-layer trajectory OOMed.
- Change: retained decoder layers 25,26,27,29,30,31 and enabled PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce allocator fragmentation. This is the same computation and checkpoint policy; only segment allocation behavior changed.
- Timing: /tmp/r30/time-retain25-expand/time_tree.json, valid 2217.5 ms/step (speedup_vs_frozen_reference=1.2532), 0.61% window spread, 128.84 GiB peak allocation.
- Quick exactness: /tmp/r30/quick/check.json, PASS N1, data and exact-item checks clean.
- Full check: /tmp/r30/full/check.json, PASS N1 at 2214.3 ms/step (speedup=1.255); trajectory equivalent (train endpoint bias -0.000094, probe bias -0.000385). The allocator setting and sixth retained layer are promoted.
—Round 31 (2026-11-13 UTC)
- Start/end: 11:55:36–11:57:55 UTC. The Round 30 profile identified DeepSpeed's disabled-monitor loss.mean().item() synchronization as a large CPU hotspot.
- Change: screened LLAVA_SKIP_DS_LOSS_ITEM=1 on the promoted six-layer/expandable allocator stack in /tmp/r31/switches-skip-item.json.
- Timing: /tmp/r31/time-skip-item/time_tree.json, valid 2214.5 ms/step (speedup_vs_frozen_reference=1.2549), 0.29% window spread, essentially neutral versus Round 30. The switch remains disabled.
- No quick/full check was needed for this N0 no-gain screen; the existing Round 30 full check remains the submitted validation.
—Round 32 (2026-11-13 UTC)
- Start/end: 11:59:54–12:16:33 UTC. Profile /tmp/r32/profile/profile_tree.md for the promoted six-layer/expandable stack measured 1986 ms/step GPU time; checkpoint backward was 807.3 ms/step, while the top five bf16 GEMMs accounted for 60.7% of GPU kernel time.
- Change: added decoder layer 23 to LLAVA_CHECKPOINT_RETAIN, keeping PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True so the extra live activation is allocated without the fragmentation seen in Round 29.
- Timing: /tmp/r32/time-retain23-expand/time_tree.json, valid 2200.1 ms/step (speedup_vs_frozen_reference=1.2631), 0.62% window spread, 132.08 GiB peak allocation.
- Quick exactness: /tmp/r32/quick-retain23-expand/check.json, PASS N1, data and exact-item checks clean.
- Full check: /tmp/r32/full-retain23-expand/check.json, PASS N1 at 2201.6 ms/step (speedup=1.2622); trajectory equivalent. The layer-23 retention is promoted.
—Round 33 (2026-11-13 UTC)
- Start/end: 12:26:28–12:44:15 UTC. Profile /tmp/r33/profile/profile_tree.md for the promoted layer-23 stack measured 2091 ms/step GPU time; bf16 GEMMs remained dominant and NCCL reduce-scatter was 184.3 ms in this sample, while checkpoint backward was 767.9 ms/step.
- Change: added decoder layer 22 to LLAVA_CHECKPOINT_RETAIN, again using expandable allocator segments.
- Timing: /tmp/r33/time-retain22-expand/time_tree.json, valid 2188.1 ms/step (speedup_vs_frozen_reference=1.2701), 0.02% window spread, 135.31 GiB peak allocation.
- Quick exactness: /tmp/r33/quick-retain22-expand/check.json, PASS N1, data and exact-item checks clean.
- Full check: /tmp/r33/full-retain22-expand/check.json, PASS N1 at the measured candidate timing (speedup=1.2728); trajectory equivalent. Layer 22 retention is promoted.
—Round 34 (2026-11-13 UTC)
- Start/end: 12:45:25–12:49:34 UTC. Profile /tmp/r34/profile/profile_tree.md for the promoted layer-22 stack measured 1939 ms/step GPU time; checkpoint backward was 719.5 ms/step and the five largest GEMMs were 61.0% of GPU kernel time.
- Change: screened one more retained decoder layer, 21, with the expandable allocator in /tmp/r34/switches-retain21-expand.json.
- Timing: /tmp/r34/time-retain21-expand/time_tree.json failed at step 10 with CUDA OOM (137.41 GiB in use, 2.39 GiB free, attempted 3.75 GiB allocation), so no timing speedup was available.
- Conclusion: retaining layer 21 in addition to the current set is unsafe; the submitted layer-22 stack remains unchanged.
—Round 35 (2026-11-13 UTC)
- Start/end: 12:49:56–12:54:12 UTC. Profile /tmp/r35/profile/profile_tree.md for the layer-22 candidate measured 1939 ms/step GPU time; checkpoint backward remained the leading recomputation cost.
- Change: replaced retained layer 22 with layer 21 (21,23,25,26,27,29,30,31) at the same retention count and memory footprint.
- Timing: /tmp/r35/time-replace22-with21/time_tree.json, valid 2185.5 ms/step (speedup_vs_frozen_reference=1.2715), 0.37% window spread, 135.31 GiB peak allocation. This did not improve the full-check result (1.2728x), so the existing layer-22 set remains submitted.
- Conclusion: retention placement at layer 22 is at least as good as replacing it with layer 21; no promotion.
—Round 36 (2026-11-13 UTC)
- Start/end: 12:55:00–13:07:09 UTC. Profile /tmp/r36/profile/profile_tree.md for the layer-22 stack again showed separate residual/elementwise work among the leading non-GEMM costs.
- Change: re-screened the existing Triton residual-plus-RMSNorm fusion (LLAVA_FUSE_RESIDUAL_RMSNORM=1) and also tested a native FlashAttention dropout_add_rms_norm variant behind LLAVA_USE_APEX_RESIDUAL_RMSNORM.
- Timing: Triton fusion /tmp/r36/time-fused-resid/time_tree.json was 2188.9 ms/step (speedup_vs_frozen_reference=1.2696) with 1.15% window spread, slower than the validated stack. The native variant /tmp/r36/time-apex-resid/time_tree.json failed at import because the pinned environment lacks the dropout_layer_norm extension. Both switches remain disabled; the experimental code was reverted.
—Round 37 (2026-11-13 UTC)
- Start/end: 13:09:28–13:14:02 UTC. The current profile-guided target was repeated rotary-table dtype conversion in each decoder layer.
- Change: added an opt-in per-layer cache for converted RoPE tables (LLAVA_CACHE_ROTARY=1), preserving the original table generation and values.
- Timing: /tmp/r37/time-cache-rotary/time_tree.json, valid 2193.6 ms/step (speedup_vs_frozen_reference=1.2669) with 1.41% window spread and 135.31 GiB peak allocation, slower than the validated 1.2728x stack. The cache was reverted and remains disabled; the measured overhead indicates the model buffers were already in the compute dtype.
—Round 38 (2026-11-13 UTC)
- Start/end: 13:22:57–13:39:41 UTC. Profile /tmp/r38/profile/profile_tree.md measured 1940 ms/step GPU time; the output path still contained large elementwise bandwidth work after the dominant LM-head GEMM.
- Change: added LLAVA_KEEP_BF16_LOGITS, which skips the inherited Llama fp32 logits copy before cross entropy and keeps the logits in their bf16 compute dtype. The model, optimizer, and parameter set remain unchanged.
- Timing: /tmp/r38/time-bf16-logits/time_tree.json, valid 2174.1 ms/step (speedup_vs_frozen_reference=1.2782), 0.74% window spread, 129.68 GiB peak allocation.
- Quick exactness: /tmp/r38/quick-bf16-logits/check.json, PASS; no exact-item failures.
- Full check: /tmp/r38/full-bf16-logits/check.json, PASS N1 at 2179.6 ms/step (speedup=1.2752); trajectory equivalent (train endpoint bias -0.000463, probe bias -0.000030). The switch is promoted.
—Round 39 (2026-11-13 UTC)
- Start/end: 13:41:18–13:55:17 UTC. Profile /tmp/r39/profile/profile_tree.md for the promoted bf16-logits stack measured 1943 ms/step GPU time; checkpoint backward was 717.2 ms/step and the main GEMMs remained dominant.
- Change: retried retaining decoder layer 21 in addition to the eight retained extras, now with the bf16-logits memory reduction (/tmp/r39/switches-retain21-bf16.json).
- Timing: /tmp/r39/time-retain21-bf16/time_tree.json, valid 2158.6 ms/step (speedup_vs_frozen_reference=1.2874), 0.70% window spread, 132.92 GiB peak allocation.
- Quick exactness: /tmp/r39/quick-retain21-bf16/check.json, PASS.
- Full check: /tmp/r39/full-retain21-bf16/check.json rejected at the bitwise record gate because the text case aborted with double free or corruption (!prev) after recording its step. The layer-21 addition is not promoted despite the faster timing; the validated layer-22 stack remains submitted.
—Round 40 (2026-11-13 UTC)
- Start/end: 13:58:05–14:02:01 UTC. Profile /tmp/r40/profile/profile_tree.md for the validated bf16-logits stack measured 1943 ms/step GPU time; checkpoint backward and the large GEMMs remained the leading costs.
- Change: replaced retained layer 22 with layer 21 at the same retention count while keeping bf16 logits (/tmp/r40/switches-replace22-with21-bf16.json), avoiding the extra live activation that destabilized Round 39.
- Timing: /tmp/r40/time-replace22-with21-bf16/time_tree.json, valid 2180.2 ms/step (speedup_vs_frozen_reference=1.2746), 0.35% window spread, 129.68 GiB peak allocation. This is slower than the validated layer-22 stack, so no promotion or quick check was needed.
—Round 41 (2026-11-13 UTC)
- Start/end: 14:02:43–14:05:56 UTC. Profile /tmp/r41/profile/profile_tree.md was taken on the validated bf16-logits stack before testing allocator controls.
- Change: added max_split_size_mb:512 to the expandable-segments allocator for the fast extra-layer-21 candidate in /tmp/r41/switches-retain21-maxsplit512.json.
- Timing: /tmp/r41/time-retain21-maxsplit512/time_tree.json failed with PyTorch's !block->expandable_segment_ internal allocator assertion during ZeRO parameter gather. The setting is rejected and not promoted.
—Round 42 (2026-11-13 UTC)
- Start/end: 14:06:48–14:23:35 UTC. Profile /tmp/r42/profile/profile_tree.md for the validated stack showed 1943 ms/step GPU time with checkpoint backward still dominant.
- Change: enabled garbage_collection_threshold:0.8 alongside expandable segments and retried retaining layer 21 in /tmp/r42/switches-retain21-gc.json.
- Timing: /tmp/r42/time-retain21-gc/time_tree.json, valid 2153.5 ms/step (speedup_vs_frozen_reference=1.2905), 0.51% window spread, 132.92 GiB peak allocation.
- Quick exactness: /tmp/r42/quick-retain21-gc/check.json, PASS.
- Full check: /tmp/r42/full-retain21-gc/check.json, PASS N1 at 2165.8 ms/step (speedup=1.2832); trajectory equivalent (train endpoint bias -0.000463, probe bias +0.000312). The layer-21 retention and allocator threshold are promoted; the threshold avoided Round 39's record double-free.
—Round 43 (2026-11-13 UTC)
- Start/end: 14:25:25–14:40:57 UTC. Profile /tmp/r43/profile/profile_tree.md for the promoted layer-21 stack measured 1943 ms/step GPU time; checkpoint backward was still the largest recomputation component.
- Change: added decoder layer 19 to the retained set with bf16 logits and allocator garbage collection (/tmp/r43/switches-retain19-gc.json).
- Timing: /tmp/r43/time-retain19-gc/time_tree.json, valid 2148.1 ms/step (speedup_vs_frozen_reference=1.2937), 1.15% window spread, 136.15 GiB peak allocation.
- Quick exactness: /tmp/r43/quick-retain19-gc/check.json, PASS.
- Full check: /tmp/r43/full-retain19-gc/check.json failed in the 60-step trajectory at step 10 with CUDA OOM (138.05 GiB in use, 1.75 GiB free, attempted 1.88 GiB). The layer-19 addition remains disabled.
—Round 44 (2026-11-13 UTC)
- Start/end: 14:41:45–14:46:04 UTC. Profile /tmp/r44/profile/profile_tree.md for the passing layer-21 stack was collected before changing only the retained-layer placement.
- Change: replaced layer 21 with layer 19 while keeping nine extra retained layers, bf16 logits, and allocator garbage collection (/tmp/r44/switches-replace21-with19-gc.json).
- Timing: /tmp/r44/time-replace21-with19-gc/time_tree.json, valid 2156.5 ms/step (speedup_vs_frozen_reference=1.2886), 0.88% window spread, 132.92 GiB peak allocation. It is slower than the validated layer-21 stack's full-check timing, so no quick/full check or promotion was needed.
—Round 45 (2026-11-13 UTC)
- Start/end: 14:47:02–14:51:07 UTC. Profile /tmp/r45/profile/profile_tree.md was taken on the passing layer-21/GC stack before testing more aggressive allocator reclamation.
- Change: screened layer 19 with garbage_collection_threshold:0.5 (/tmp/r45/switches-retain19-gc05.json) to recover memory for the extra retained layer.
- Timing: /tmp/r45/time-retain19-gc05/time_tree.json, valid but much slower at 2206.3 ms/step (speedup_vs_frozen_reference=1.2596), with 5.08% window spread and 136.15 GiB peak allocation. The setting is rejected without a quick/full check.
↑ LLaVA