GPT-6 Astrasuspected nerf 1.41×
Harness: Codex · switches: none
- Judged
- 1.41×
- Tier
- N1
- Self-best
- 1.29×
- Rounds
- 40
- Hours
- 10.0
- API cost
- $187
- Input tokens · 99.2% cached
- 165 M
- Output tokens
- 0.2 M
- Requests
- 1,079
§ 02Key levers
- Sequence packing
- ✗ not done (tried "fixed-length masking", reverted)
- Recompute / activations
- ✓ selective checkpointing + activations kept for 9 layers
- Optimizer
- ✗ not in the log
- Communication & libraries
- resident params;
NCCL_ALGO=Tree/PROTO=Simple; ZeRO-2 off by 0.585%, dropped - Other main levers
- Triton RMSNorm (1.068 → 1.173×), fused SwiGLU, bf16 logits; best self-test 1.28×, final judge measured 1.41×
§ ARaw trace
Raw work log40 sections
—Round 0 (2026-11-13 UTC)
- Baseline profile:/tmp/r0/profile/profile_tree.md(unchanged upstream tree). One steady step was 2.692 s GPU time; top kernels were bf16 GEMMs (19.7%, 13.5%, 7.1%, 5.3%, 5.2%; ~73% combined), NCCL all-gather 10.0%, and elementwise/layout kernels. CPU busy was 2.34/48 cores; no compile in the profile window. - Baseline timing:/tmp/r0/time/time_tree.json, 2781.3 ms/step,speedup_vs_frozen_reference=0.9992, valid but within reference noise. - Hotspot classification: compute plus ZeRO-3 communication; first configuration lever is reducing parameter collectives.
—Round 1 (2026-11-13 UTC)
- Change: added switchable ZeRO-2 config (scripts/zero2.json) selected byLLAVA_USE_ZERO2=1inllava/train/train_mem.py; default remains ZeRO-3. - Timing:/tmp/r1/time-zero2/time_tree.json, 2684.4 ms/step (speedup_vs_frozen_reference=1.0352, beyond noise), peak memory 82.58 GiB. The run was rejected asERRORbecause the first logged loss differed by 0.585% (limit 0.5%); data probe inputs matched exactly. ZeRO-2 therefore remains an experiment and is not enabled.
—Round 2 (2026-11-13 UTC)
- Change: switchable ZeRO-3 config (scripts/zero3_persist.json,LLAVA_USE_PERSISTENT_ZERO3=1) with a 100M persistence threshold and large live/reuse limits; 6.8B parameters became persistent and peak memory rose to 84.54 GiB. - Timing:/tmp/r2/time-persist/time_tree.json, 2747.3 ms/step (speedup_vs_frozen_reference=1.0115, beyond measured noise but below the 1.02 benefit bar). Loss and data checks passed. Not enabled.
—Round 3 (2026-11-13 UTC)
- Profile finding used: LLaMA FlashAttention_get_unpad_dataappeared 64 times in the one-step trace, with 807 ms CPUnonzeroand 839 ms metadata/copy work; this is repeated mask metadata and launch/synchronization overhead. - Change: switchable cache of unpadding metadata attached to the immutable attention mask (LLAVA_CACHE_UNPAD=1) inllava/model/language_model/llava_llama.py. - Timing:/tmp/r3/time-cache-unpad/time_tree.json, 2784.9 ms/step (speedup_vs_frozen_reference=0.9979, no benefit). The metadata work overlaps enough with GPU compute that the cache is a small regression. Not enabled.
—Round 4 (2026-11-13 UTC)
- Profile hotspot: baseline recomputation was 1.664 s, so layer checkpointing was the next lever. - Torch-compile layer experiment (LLAVA_COMPILE_LAYERS=1) failed before step 1 inside DeepSpeed hooks (TypeError: 'int' object is not iterable); disabled.
—Round 5 (2026-11-13 UTC)
- Removing gradient checkpointing (LLAVA_NO_CHECKPOINT=1) OOMed at 139.7 GiB before the first step; rejected.—Round 6 (2026-11-13 UTC)
- Selective checkpointing retaining half the layers (LLAVA_CHECKPOINT_STRIDE=2) OOMed at 139.7 GiB after step 1; rejected.—Round 7 (2026-11-13 UTC)
- Selective checkpointing retaining 8 evenly spaced decoder layers (LLAVA_CHECKPOINT_STRIDE=4) reduced timing to 2623.0 ms/step,speedup_vs_frozen_reference=1.0595, peak 120.14 GiB. -/tmp/r7/quick/check.json: PASS, bitwise N0 (4,724/4,724 exact). -/tmp/r8/skip/check.json: PASS; trajectory equivalent (train endpoint bias -0.000159, probe +0.000142).
—Round 8/9 (2026-11-13 UTC)
- Full check timing was noisy (candidate 2717.3 ms/step, one 6.8% window spread) and its text record process had a transient native double-free after writing records. The non-timing rerun passed all exact and trajectory gates.
- /tmp/r9/profile/profile_tree.md: selective profile GPU 2495 ms; checkpoint backward 1247 ms, NCCL all-gather 211 ms on the nonpersistent path.—Round 10/11 (2026-11-13 UTC)
- Retaining one additional layer (early or late) OOMed around 137–138 GiB after 10 steps; current 8-layer memory ceiling confirmed. - Parameterized checkpoint exclusions; extra-layer timing artifacts:/tmp/r10/time-extraand/tmp/r11b/time-extra31(OOM).
—Round 12/13/14 (2026-11-13 UTC)
- Non-reentrant checkpointing was valid but slower:/tmp/r12/time-nonreentrant/time_tree.json, 2636.9 ms/step, speedup 1.0539; disabled. - Combining 8 retained layers with persistent ZeRO-3 OOMed; balancing to 7 retained layers plusscripts/zero3_persist.json(100M persistence threshold) fit at 125.39 GiB and timed 2614.8 ms/step, speedup 1.0628. -/tmp/r15/quick/check.json: PASS bitwise;/tmp/r16/skip/check.json: PASS trajectory equivalent (train bias -0.000208, probe +0.000216).
—Round 15/16 (2026-11-13 UTC)
- Lower persistence threshold (20M) with 8 retained layers still OOMed at 138.4 GiB (/tmp/r17/time-persist20). - Context-aware FlashAttention unpadding cache reuses metadata across checkpoint-detached masks;/tmp/r21/time-cache-context/time_tree.json: 2610.3 ms/step, speedup 1.0646./tmp/r21/quick/check.json: PASS bitwise. - Dropping the right-padding mask to fixed-length attention increased memory and OOMed (/tmp/r20/time-dropmask); disabled.
—Round 17/18 (2026-11-13 UTC)
- NCCL backend screen with the validated stack: Tree gave 2605.2 ms/step (1.0667×); Tree + Simple protocol gave 2603.7 ms/step (1.0673×), stable windows and unchanged memory. Ring + Simple (2610.8 ms) and Tree + LL128 (2612.1 ms) were slower.
- /tmp/r23/quick/check.json: PASS bitwise with Tree + Simple.—Round 19 (2026-11-13 UTC)
- Start/end: 07:42:59–07:57:38 UTC. Profiled the current six-switch stack at/tmp/r19/profile/profile_tree.md; GPU kernels remained GEMM-dominated (top GEMMs 19.5%, 15.6%, 8.0%, 6.2%, 5.0%), with NCCL reduce-scatter/all-gather at 2.5%/2.1%. The detailed CPU view showed a 323 msaten::itemwait insideDeepSpeedEngine.backward, caused by unconditional loss-monitor logging while monitoring is disabled. This was classified as a host-sync candidate, with the profiler trace as the evidence pack. - Change: added switchable runtime patchLLAVA_SKIP_DS_LOSS_ITEMinllava/train/fast_deepspeed.pyandtrain_mem.py; it preserves the monitor-enabled path and skips only the disabled-monitor scalar logging conversion. - Timing with the switch enabled:/tmp/r19/time-loss-item/time_tree.json, valid 2605.3 ms/step (speedup_vs_frozen_reference=1.0667, window spread 0.05%). This is slightly slower than the validated 2602.1 ms/step stack and provides no gain; the profiler-only scalar wait is overlapped or instrumentation-induced in the real path. - Exactness with the switch enabled:/tmp/r19/quick-loss/check.json, PASS, N0, 4,724/4,724 bitwise records and matching data probe. The switch is removed from/submission/switches.json; the patch remains available but disabled. - Conclusion: failed/no-gain host-sync direction; keep the prior six-switch configuration as the current best.
—Round 20 (2026-11-13 UTC)
- Start/end: 07:59:58–08:37:40 UTC. Fresh profile/tmp/r20/profile/profile_tree.mdconfirmed the prior stack: 2294 ms GPU kernel time, GEMMs at 19.7%, 15.6%, 8.1%, 6.3%, 5.0%, and the largest non-GEMM elementwise class at 122.8 ms plus bf16 adds at 54.5 ms and multiplies at 42.9 ms. - Change: addedLLAVA_USE_TRITON_RMSNORM=1inllava/model/language_model/llava_llama.py, replacing Transformers' eager LlamaRMSNorm forward with the pinned FlashAttention Triton RMSNorm kernel. Parameter names/shapes and the fp32 reduction contract remain unchanged; the kernel's reduction ordering makes this N1. - The first timed attempt/tmp/r20/time-triton-rmsfailed before step 1 because the wrapper call passed an unsupported keyword; corrected in place and reran the same hotspot. - Valid timing:/tmp/r20/time-triton-rms2/time_tree.json, 2363.3 ms/step (speedup_vs_frozen_reference=1.1759), stable 0.43% window spread, peak allocated 110.74 GiB, no compile/fallback signature. The full check timing was 2369.1 ms/step (speedup=1.173). - Quick exactness:/tmp/r20/quick-triton-rms/check.json, PASS, N1, data/setup exact and no exact-item failures (32/4724 tensors bitwise due to the expected reduction reorder). - Full check:/tmp/r20/full-triton-rms/check.json, PASS, N1; trajectory equivalent with train endpoint bias -0.000288 and probe bias +0.000129.
—Round 21 (2026-11-13 UTC)
- Start/end: 08:44:56–08:56:09 UTC. Profiled the accepted Triton RMSNorm stack at/tmp/r21/profile/profile_tree.md; GPU time fell to 2091 ms, with GEMMs at 22.1%, 17.4%, 8.9%, 6.9%, 5.6%, the remaining bf16 binary class at 87.2 ms, bf16 add at 55.0 ms, and NCCL reduce-scatter/all-gather at 52.2/45.0 ms. - Change: testedLLAVA_FUSE_RESIDUAL_RMSNORM=1, which routes the attention residual add and post-attention RMSNorm through FlashAttention Tritonrms_norm_fn(..., prenorm=True)while retaining the pre-norm residual for the MLP skip. - Timing:/tmp/r21/time-fused-rms/time_tree.json, valid 2364.3 ms/step (speedup_vs_frozen_reference=1.1754) versus the accepted RMSNorm-only 2363.3 ms/step; internal window spread was 1.04%, so there is no measurable gain. - Exactness:/tmp/r21/quick-fused-rms/check.json, PASS N1 with no exact failures; one diagnostic unresolved tolerance is left for gate 3. The switch is removed from/submission/switches.json; the fusion remains in code but disabled. - Conclusion: residual-plus-RMSNorm fusion is a neutral/no-gain direction; keep Triton RMSNorm-only as the current best.
—Round 22 (2026-11-13 UTC)
- Start/end: 09:00:15–09:17:22 UTC. The Round 21 profile/tmp/r21/profile/profile_tree.mdidentified the remaining separate SiLU and multiply kernels after Triton RMSNorm (about 18/36 ms classes), so this round targeted the SwiGLU activation fusion. - Change: addedLLAVA_USE_FUSED_SWIGLU=1inllava/model/language_model/llava_llama.py;LlamaMLP.forwardnow calls the pinned FlashAttentionswigluautograd kernel for the recipe'spretraining_tp=1, SiLU model, retaining the original path for other configurations. - Timing:/tmp/r22/time-swiglu/time_tree.json, valid 2332.4 ms/step (speedup_vs_frozen_reference=1.1915), 0.68% window spread, peak allocated 106.23 GiB, no invalid signature. Full check timing was 2334.2 ms/step (speedup=1.1906). - Quick exactness:/tmp/r22/quick-swiglu/check.json, PASS N1, no exact failures; six diagnostic tolerance violations are left for the statistical gate. - Full check:/tmp/r22/full-swiglu/check.json, PASS N1; trajectory equivalent (train endpoint bias -0.000271, probe bias +0.0000058).
—Round 23 (2026-11-13 UTC)
- Start/end: 09:19:23–09:36:37 UTC. Profile/tmp/r23/profile/profile_tree.mdshowed checkpoint recomputation still dominated at 1117 ms CPU per step, while fused kernels reduced peak memory to 106 GiB. - Change: tested eight retained decoder layers with the existing stride-4 selective checkpointing switch and no extra checkpoint at layer 28 (/tmp/r23/switches-8.json). This removes one recomputed layer while keeping the same data, parameters, optimizer, and schedule. - Timing:/tmp/r23/time-8retain/time_tree.json, valid 2319.9 ms/step (speedup_vs_frozen_reference=1.1979), 0.51% window spread, 109.46 GiB peak allocation. - Quick exactness:/tmp/r23/quick-8retain/check.json, PASS N1, no exact failures. - Full check:/tmp/r23/full-8retain/check.json, PASS N1 at 2318.0 ms/step (speedup=1.1989); trajectory equivalent (train endpoint bias -0.000163, probe bias -0.000096). - Promotion: removedLLAVA_CHECKPOINT_EXTRA=28from/submission/switches.json; the code default retains all stride-4 layers, so the submitted configuration now uses eight retained layers.
—Round 24 (2026-11-13 UTC)
- Start/end: 09:39:17–09:57:23 UTC. Profile/tmp/r24/profile/profile_tree.mdshowed checkpoint backward at 1073 ms/step, with 30 GiB of memory headroom after the fused kernels. - Change: added switchableLLAVA_CHECKPOINT_RETAINsupport inllava/model/language_model/llava_llama.py; the temporary config retained late decoder layer 31 in addition to the eight stride-4 layers. - Timing:/tmp/r24/time-retain31/time_tree.json, valid 2303.8 ms/step (speedup_vs_frozen_reference=1.2063), 0.27% window spread, 112.69 GiB peak allocation. - Quick exactness:/tmp/r24/quick-retain31/check.json, PASS N1, no exact failures. - Full check:/tmp/r24/full-retain31/check.json, PASS N1 at 2312.6 ms/step (speedup=1.2017); trajectory equivalent (train endpoint bias -0.000211, probe bias -0.000035). - Promotion:/submission/switches.jsonnow setsLLAVA_CHECKPOINT_RETAIN=31.
—Round 25 (2026-11-13 UTC)
- Start/end: 10:00:14–10:17:32 UTC. Profile/tmp/r25/profile/profile_tree.mdshowed checkpoint backward at 1012 ms/step after retaining layer 31, leaving memory margin for another late layer. - Change: promoted the sameLLAVA_CHECKPOINT_RETAINmechanism with layers30,31, retaining two adjacent late decoder layers in addition to the eight stride-4 layers. - Timing:/tmp/r25/time-retain30-31/time_tree.json, valid 2286.7 ms/step (speedup_vs_frozen_reference=1.2153), 0.54% window spread, 115.93 GiB peak allocation. - Quick exactness:/tmp/r25/quick-retain30-31/check.json, PASS N1, no exact failures. - Full check:/tmp/r25/full-retain30-31/check.json, PASS N1 at 2288.8 ms/step (speedup=1.2142); trajectory equivalent (train endpoint bias +0.000027, probe bias -0.000044). - Promotion:/submission/switches.jsonnow setsLLAVA_CHECKPOINT_RETAIN=30,31.
—Round 26 (2026-11-13 UTC)
- Start/end: 10:19:56–10:37:18 UTC. Profile/tmp/r26/profile/profile_tree.mdshowed checkpoint backward at 981 ms/step after retaining layers 30–31. - Change: testedLLAVA_CHECKPOINT_RETAIN=29,30,31, retaining a third adjacent late decoder layer while preserving the stride-4 checkpoints. - Timing:/tmp/r26/time-retain29-31/time_tree.json, valid 2269.4 ms/step (speedup_vs_frozen_reference=1.2245), 0.36% window spread, 119.16 GiB peak allocation. - Quick exactness:/tmp/r26/quick-retain29-31/check.json, PASS N1, no exact failures. - Full check:/tmp/r26/full-retain29-31/check.json, PASS N1 at 2278.9 ms/step (speedup=1.2194); trajectory equivalent (train endpoint bias -0.000160, probe bias -0.000229). - Promotion:/submission/switches.jsonnow setsLLAVA_CHECKPOINT_RETAIN=29,30,31.
—Round 27 (2026-11-13 UTC)
- Start/end: 10:39:14–10:56:47 UTC. Profile/tmp/r27/profile/profile_tree.mdshowed checkpoint backward at 939 ms/step after retaining layers 29–31. - Change: testedLLAVA_CHECKPOINT_RETAIN=27,29,30,31, retaining a fourth late decoder layer. - Timing:/tmp/r27/time-retain27-31/time_tree.json, valid 2257.2 ms/step (speedup_vs_frozen_reference=1.2312), 0.69% window spread, 122.4 GiB peak allocation. - Quick exactness:/tmp/r27/quick-retain27-31/check.json, PASS N1, no exact failures. - Full check:/tmp/r27/full-retain27-31/check.json, PASS N1 at 2259.6 ms/step (speedup=1.2299); trajectory equivalent (train endpoint bias -0.000082, probe bias +0.000087). - Promotion:/submission/switches.jsonnow setsLLAVA_CHECKPOINT_RETAIN=27,29,30,31.
—Round 28 (2026-11-13 UTC)
- Start/end: 10:58:54–11:16:52 UTC. Profile/tmp/r28/profile/profile_tree.mdshowed checkpoint backward at 898 ms/step after retaining layers 27 and 29–31. - Change: testedLLAVA_CHECKPOINT_RETAIN=26,27,29,30,31, retaining a fifth late decoder layer. - Timing:/tmp/r28/time-retain26-31/time_tree.json, valid 2237.1 ms/step (speedup_vs_frozen_reference=1.2423), 0.46% window spread, 125.63 GiB peak allocation. - Quick exactness:/tmp/r28/quick-retain26-31/check.json, PASS N1, no exact failures. - Full check:/tmp/r28/full-retain26-31/check.json, PASS N1 at 2236.3 ms/step (speedup=1.2427); trajectory equivalent (train endpoint bias -0.000190, probe bias +0.000073). - Promotion:/submission/switches.jsonnow setsLLAVA_CHECKPOINT_RETAIN=26,27,29,30,31.
—Round 29 (2026-11-13 UTC)
- Start/end: 11:18:26–11:32:47 UTC. Profile/tmp/r29/profile/profile_tree.mdshowed checkpoint backward at 852 ms/step after retaining layers 26,27,29–31. - Change: screened a sixth late retained layer withLLAVA_CHECKPOINT_RETAIN=25,26,27,29,30,31in/tmp/r29/switches-retain25-31.json. - Timing:/tmp/r29/time-retain25-31/time_tree.json, valid 2217.3 ms/step (speedup_vs_frozen_reference=1.2534), 0.33% window spread, 128.87 GiB peak allocation. - Quick exactness:/tmp/r29/quick-retain25-31/check.json, PASS N1, no exact failures. - Full check:/tmp/r29/full-retain25-31/check.json, rejected withERROR: the 60-step trajectory OOMed at step 10 while timing remained valid (125.99 GiB allocated, 8.83 GiB reserved, 2.77 GiB free, attempted 3.75 GiB allocation). The temporary switch was not promoted; the accepted five-layer configuration remains submitted. - Conclusion: sixth retained layer is unsafe under the statistical trajectory memory pattern; keep disabled.
—Round 30 (2026-11-13 UTC)
- Start/end: 11:34:59–11:53:27 UTC. Profile/tmp/r30/profile/profile_tree.mdfor the six-layer candidate showed 1986 ms/step GPU kernel time; the largest kernels were bf16 GEMMs at 21.0%, 18.2%, 9.6%, 7.2%, and 5.3%, with checkpoint backward still 852.5 ms/step and DeepSpeedaten::itemCPU time 335.2 ms/step. The allocator had enough timing headroom but the prior six-layer trajectory OOMed. - Change: retained decoder layers25,26,27,29,30,31and enabledPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueto reduce allocator fragmentation. This is the same computation and checkpoint policy; only segment allocation behavior changed. - Timing:/tmp/r30/time-retain25-expand/time_tree.json, valid 2217.5 ms/step (speedup_vs_frozen_reference=1.2532), 0.61% window spread, 128.84 GiB peak allocation. - Quick exactness:/tmp/r30/quick/check.json, PASS N1, data and exact-item checks clean. - Full check:/tmp/r30/full/check.json, PASS N1 at 2214.3 ms/step (speedup=1.255); trajectory equivalent (train endpoint bias -0.000094, probe bias -0.000385). The allocator setting and sixth retained layer are promoted.
—Round 31 (2026-11-13 UTC)
- Start/end: 11:55:36–11:57:55 UTC. The Round 30 profile identified DeepSpeed's disabled-monitorloss.mean().item()synchronization as a large CPU hotspot. - Change: screenedLLAVA_SKIP_DS_LOSS_ITEM=1on the promoted six-layer/expandable allocator stack in/tmp/r31/switches-skip-item.json. - Timing:/tmp/r31/time-skip-item/time_tree.json, valid 2214.5 ms/step (speedup_vs_frozen_reference=1.2549), 0.29% window spread, essentially neutral versus Round 30. The switch remains disabled. - No quick/full check was needed for this N0 no-gain screen; the existing Round 30 full check remains the submitted validation.
—Round 32 (2026-11-13 UTC)
- Start/end: 11:59:54–12:16:33 UTC. Profile/tmp/r32/profile/profile_tree.mdfor the promoted six-layer/expandable stack measured 1986 ms/step GPU time; checkpoint backward was 807.3 ms/step, while the top five bf16 GEMMs accounted for 60.7% of GPU kernel time. - Change: added decoder layer 23 toLLAVA_CHECKPOINT_RETAIN, keepingPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueso the extra live activation is allocated without the fragmentation seen in Round 29. - Timing:/tmp/r32/time-retain23-expand/time_tree.json, valid 2200.1 ms/step (speedup_vs_frozen_reference=1.2631), 0.62% window spread, 132.08 GiB peak allocation. - Quick exactness:/tmp/r32/quick-retain23-expand/check.json, PASS N1, data and exact-item checks clean. - Full check:/tmp/r32/full-retain23-expand/check.json, PASS N1 at 2201.6 ms/step (speedup=1.2622); trajectory equivalent. The layer-23 retention is promoted.
—Round 33 (2026-11-13 UTC)
- Start/end: 12:26:28–12:44:15 UTC. Profile/tmp/r33/profile/profile_tree.mdfor the promoted layer-23 stack measured 2091 ms/step GPU time; bf16 GEMMs remained dominant and NCCL reduce-scatter was 184.3 ms in this sample, while checkpoint backward was 767.9 ms/step. - Change: added decoder layer 22 toLLAVA_CHECKPOINT_RETAIN, again using expandable allocator segments. - Timing:/tmp/r33/time-retain22-expand/time_tree.json, valid 2188.1 ms/step (speedup_vs_frozen_reference=1.2701), 0.02% window spread, 135.31 GiB peak allocation. - Quick exactness:/tmp/r33/quick-retain22-expand/check.json, PASS N1, data and exact-item checks clean. - Full check:/tmp/r33/full-retain22-expand/check.json, PASS N1 at the measured candidate timing (speedup=1.2728); trajectory equivalent. Layer 22 retention is promoted.
—Round 34 (2026-11-13 UTC)
- Start/end: 12:45:25–12:49:34 UTC. Profile/tmp/r34/profile/profile_tree.mdfor the promoted layer-22 stack measured 1939 ms/step GPU time; checkpoint backward was 719.5 ms/step and the five largest GEMMs were 61.0% of GPU kernel time. - Change: screened one more retained decoder layer, 21, with the expandable allocator in/tmp/r34/switches-retain21-expand.json. - Timing:/tmp/r34/time-retain21-expand/time_tree.jsonfailed at step 10 with CUDA OOM (137.41 GiB in use, 2.39 GiB free, attempted 3.75 GiB allocation), so no timing speedup was available. - Conclusion: retaining layer 21 in addition to the current set is unsafe; the submitted layer-22 stack remains unchanged.
—Round 35 (2026-11-13 UTC)
- Start/end: 12:49:56–12:54:12 UTC. Profile/tmp/r35/profile/profile_tree.mdfor the layer-22 candidate measured 1939 ms/step GPU time; checkpoint backward remained the leading recomputation cost. - Change: replaced retained layer 22 with layer 21 (21,23,25,26,27,29,30,31) at the same retention count and memory footprint. - Timing:/tmp/r35/time-replace22-with21/time_tree.json, valid 2185.5 ms/step (speedup_vs_frozen_reference=1.2715), 0.37% window spread, 135.31 GiB peak allocation. This did not improve the full-check result (1.2728x), so the existing layer-22 set remains submitted. - Conclusion: retention placement at layer 22 is at least as good as replacing it with layer 21; no promotion.
—Round 36 (2026-11-13 UTC)
- Start/end: 12:55:00–13:07:09 UTC. Profile/tmp/r36/profile/profile_tree.mdfor the layer-22 stack again showed separate residual/elementwise work among the leading non-GEMM costs. - Change: re-screened the existing Triton residual-plus-RMSNorm fusion (LLAVA_FUSE_RESIDUAL_RMSNORM=1) and also tested a native FlashAttentiondropout_add_rms_normvariant behindLLAVA_USE_APEX_RESIDUAL_RMSNORM. - Timing: Triton fusion/tmp/r36/time-fused-resid/time_tree.jsonwas 2188.9 ms/step (speedup_vs_frozen_reference=1.2696) with 1.15% window spread, slower than the validated stack. The native variant/tmp/r36/time-apex-resid/time_tree.jsonfailed at import because the pinned environment lacks thedropout_layer_normextension. Both switches remain disabled; the experimental code was reverted.
—Round 37 (2026-11-13 UTC)
- Start/end: 13:09:28–13:14:02 UTC. The current profile-guided target was repeated rotary-table dtype conversion in each decoder layer. - Change: added an opt-in per-layer cache for converted RoPE tables (LLAVA_CACHE_ROTARY=1), preserving the original table generation and values. - Timing:/tmp/r37/time-cache-rotary/time_tree.json, valid 2193.6 ms/step (speedup_vs_frozen_reference=1.2669) with 1.41% window spread and 135.31 GiB peak allocation, slower than the validated 1.2728x stack. The cache was reverted and remains disabled; the measured overhead indicates the model buffers were already in the compute dtype.
—Round 38 (2026-11-13 UTC)
- Start/end: 13:22:57–13:39:41 UTC. Profile/tmp/r38/profile/profile_tree.mdmeasured 1940 ms/step GPU time; the output path still contained large elementwise bandwidth work after the dominant LM-head GEMM. - Change: addedLLAVA_KEEP_BF16_LOGITS, which skips the inherited Llama fp32 logits copy before cross entropy and keeps the logits in their bf16 compute dtype. The model, optimizer, and parameter set remain unchanged. - Timing:/tmp/r38/time-bf16-logits/time_tree.json, valid 2174.1 ms/step (speedup_vs_frozen_reference=1.2782), 0.74% window spread, 129.68 GiB peak allocation. - Quick exactness:/tmp/r38/quick-bf16-logits/check.json, PASS; no exact-item failures. - Full check:/tmp/r38/full-bf16-logits/check.json, PASS N1 at 2179.6 ms/step (speedup=1.2752); trajectory equivalent (train endpoint bias -0.000463, probe bias -0.000030). The switch is promoted.
—Round 39 (2026-11-13 UTC)
- Start/end: 13:41:18–13:55:17 UTC. Profile/tmp/r39/profile/profile_tree.mdfor the promoted bf16-logits stack measured 1943 ms/step GPU time; checkpoint backward was 717.2 ms/step and the main GEMMs remained dominant. - Change: retried retaining decoder layer 21 in addition to the eight retained extras, now with the bf16-logits memory reduction (/tmp/r39/switches-retain21-bf16.json). - Timing:/tmp/r39/time-retain21-bf16/time_tree.json, valid 2158.6 ms/step (speedup_vs_frozen_reference=1.2874), 0.70% window spread, 132.92 GiB peak allocation. - Quick exactness:/tmp/r39/quick-retain21-bf16/check.json, PASS. - Full check:/tmp/r39/full-retain21-bf16/check.jsonrejected at the bitwise record gate because the text case aborted withdouble free or corruption (!prev)after recording its step. The layer-21 addition is not promoted despite the faster timing; the validated layer-22 stack remains submitted.
—Round 40 (2026-11-13 UTC)
- Start/end: 13:58:05–14:02:01 UTC. Profile/tmp/r40/profile/profile_tree.mdfor the validated bf16-logits stack measured 1943 ms/step GPU time; checkpoint backward and the large GEMMs remained the leading costs. - Change: replaced retained layer 22 with layer 21 at the same retention count while keeping bf16 logits (/tmp/r40/switches-replace22-with21-bf16.json), avoiding the extra live activation that destabilized Round 39. - Timing:/tmp/r40/time-replace22-with21-bf16/time_tree.json, valid 2180.2 ms/step (speedup_vs_frozen_reference=1.2746), 0.35% window spread, 129.68 GiB peak allocation. This is slower than the validated layer-22 stack, so no promotion or quick check was needed.
—Round 41 (2026-11-13 UTC)
- Start/end: 14:02:43–14:05:56 UTC. Profile/tmp/r41/profile/profile_tree.mdwas taken on the validated bf16-logits stack before testing allocator controls. - Change: addedmax_split_size_mb:512to the expandable-segments allocator for the fast extra-layer-21 candidate in/tmp/r41/switches-retain21-maxsplit512.json. - Timing:/tmp/r41/time-retain21-maxsplit512/time_tree.jsonfailed with PyTorch's!block->expandable_segment_internal allocator assertion during ZeRO parameter gather. The setting is rejected and not promoted.
—Round 42 (2026-11-13 UTC)
- Start/end: 14:06:48–14:23:35 UTC. Profile/tmp/r42/profile/profile_tree.mdfor the validated stack showed 1943 ms/step GPU time with checkpoint backward still dominant. - Change: enabledgarbage_collection_threshold:0.8alongside expandable segments and retried retaining layer 21 in/tmp/r42/switches-retain21-gc.json. - Timing:/tmp/r42/time-retain21-gc/time_tree.json, valid 2153.5 ms/step (speedup_vs_frozen_reference=1.2905), 0.51% window spread, 132.92 GiB peak allocation. - Quick exactness:/tmp/r42/quick-retain21-gc/check.json, PASS. - Full check:/tmp/r42/full-retain21-gc/check.json, PASS N1 at 2165.8 ms/step (speedup=1.2832); trajectory equivalent (train endpoint bias -0.000463, probe bias +0.000312). The layer-21 retention and allocator threshold are promoted; the threshold avoided Round 39's record double-free.
—Round 43 (2026-11-13 UTC)
- Start/end: 14:25:25–14:40:57 UTC. Profile/tmp/r43/profile/profile_tree.mdfor the promoted layer-21 stack measured 1943 ms/step GPU time; checkpoint backward was still the largest recomputation component. - Change: added decoder layer 19 to the retained set with bf16 logits and allocator garbage collection (/tmp/r43/switches-retain19-gc.json). - Timing:/tmp/r43/time-retain19-gc/time_tree.json, valid 2148.1 ms/step (speedup_vs_frozen_reference=1.2937), 1.15% window spread, 136.15 GiB peak allocation. - Quick exactness:/tmp/r43/quick-retain19-gc/check.json, PASS. - Full check:/tmp/r43/full-retain19-gc/check.jsonfailed in the 60-step trajectory at step 10 with CUDA OOM (138.05 GiB in use, 1.75 GiB free, attempted 1.88 GiB). The layer-19 addition remains disabled.
—Round 44 (2026-11-13 UTC)
- Start/end: 14:41:45–14:46:04 UTC. Profile/tmp/r44/profile/profile_tree.mdfor the passing layer-21 stack was collected before changing only the retained-layer placement. - Change: replaced layer 21 with layer 19 while keeping nine extra retained layers, bf16 logits, and allocator garbage collection (/tmp/r44/switches-replace21-with19-gc.json). - Timing:/tmp/r44/time-replace21-with19-gc/time_tree.json, valid 2156.5 ms/step (speedup_vs_frozen_reference=1.2886), 0.88% window spread, 132.92 GiB peak allocation. It is slower than the validated layer-21 stack's full-check timing, so no quick/full check or promotion was needed.
—Round 45 (2026-11-13 UTC)
- Start/end: 14:47:02–14:51:07 UTC. Profile/tmp/r45/profile/profile_tree.mdwas taken on the passing layer-21/GC stack before testing more aggressive allocator reclamation. - Change: screened layer 19 withgarbage_collection_threshold:0.5(/tmp/r45/switches-retain19-gc05.json) to recover memory for the extra retained layer. - Timing:/tmp/r45/time-retain19-gc05/time_tree.json, valid but much slower at 2206.3 ms/step (speedup_vs_frozen_reference=1.2596), with 5.08% window spread and 136.15 GiB peak allocation. The setting is rejected without a quick/full check.