GLM-5.3 1.45×
Harness: Claude Code · switches: OPT_SYNC_FREE_TIMESTEP=1 OPT_SYNC_FREE_ATTN=1 OPT_OVERLAP_PREPACK=1 OPT_SYNC_FREE_TOKENLOOP=1 OPT_MERGE_IDX_CACHE=1 OPT_TEXTGEN_CUDAGRAPH=1 OPT_EARLY_FORWARD=1
- Judged
- 1.45×
- Tier
- N0
- Self-best
- 1.46×
- Rounds
- 17
- Hours
- 10.0
- API cost
- $31.1
- Input tokens · 99.5% cached
- 106 M
- Output tokens
- 0.6 M
- Requests
- 789
§ 02Key levers
- Text-decode CUDA graph
- ✓
OPT_TEXTGEN_CUDAGRAPH, min window 1.08× → 1.38× (backward OOM three times; switched to lower-level capture, enabled only from step 1) - Reward wait hidden behind forward
- ✓
OPT_OVERLAP_PREPACK+OPT_EARLY_FORWARD, 1.38× → 1.44× - Rollout-side bitwise fused kernels
- ✗ not in the log
- Train-side bitwise kernels / freeing memory
- ✗ (peak 131.1 GiB)
- Other main levers
- desync of timestep, attention length, token loop, merge index; denoise CUDA graph ~0.9 s slower per window, reverted
§ ARaw trace
Raw work log17 sections
0.8 hRound 1 — profile of the unchanged reference (2026-11-06 18:25–19:00)
Profile:bench/bench.py profile --tree /submission/r3 --config /submission/switches.json --out /tmp/prof-r1(steps 11=refine + 12=gen on rank 0). Trace 3.6 GB, analyzed with streaming scripts in /tmp/analysis (trace_summary.py, trace_phases.py, trace_window.py). Trace span 57.5 s (≈46 s without profiler overhead). Phase structure of one gen+refine window (from the trace timeline, 500 ms buckets): | phase | wall | GPU busy | notes | |---|---|---|---| | refine-step think (text gen, ~275 tok) | 12.5 s | 19% | host/sync bound, 22 tok/s | | refine-step denoise (20 steps × 3 CFG fwd) | 13 s | 66% | ~4500.itemsyncs per 500 ms | | gen-step think (~176 tok) | 8 s | 19% | same as above | | gen-step denoise (15 steps × 2 CFG fwd) | 7 s | 66% | | | pure-idle gaps (reward wait etc.) | ~7.6 s | 0% | 4.1 s + 1.1 s + 1.15 s + 1.18 s | | training fwd/bwd (GPU saturated, host idle) | ~9 s | ~95% | genuinely GPU-bound | Top hotspot (kind: missed fast path / host syncs): **244,157aten::itemcalls = 9.5 s CPU (22% of CPU total), plus 4.8 s "Command Buffer Full"**. Breakdown found in the trace: 1.bagel.py generate_image_mix:timestep = torch.tensor([t] * x_t.shape[0], device=x_t.device)—tis a 0-dim CUDA tensor, so building the list converts all 4096 latent-token elements with one blocking D2H each: **runs of exactly 8192 item events (4096.item+ 4096_local_scalar_dense) once per denoising timestep** → ~143k syncs per window (~35 per run of 8192 at ~25 µs). Plust > cfg_interval[0]on a CUDA tensor → 1 more sync per timestep. 2. Per-layer.item()in the KV-cache attention helpers (qwen2_navit.py_static_kv_cache_attention3/layer,_merge_static_kv_prefix_and_query5/layer): ~47k during text gen (582 forwards × 27 layers × 3), ~12k during denoise. 3. Text-gen token loop (bagel.generate_text_batch): ~8.item()per token. Second hotspot: reward-service wait — a 4.1 s pure-idle gap per window (all 4 ranks share ONE reward URL with 4 sync workers; 16 images per round), fully exposed because nothing overlaps it. Third: training fwd/bwd is GPU-saturated (~9 s), little to gain there without kernel changes. ### Change (this round): OPT_SYNC_FREE_TIMESTEP (N0)modeling/bagel/bagel.pygenerate_image_mix: iteratetimesteps.cpu()(one D2H per rollout) instead of the CUDA tensor; build the per-timestep conditioning vector withtorch.full((n,), float(t), dtype=timesteps.dtype, device=...)instead oftorch.tensor([t] * n, device=...); pass the original CUDA elementtimesteps[i]tosde_sampler.stepandtimesteps_for_sde.float(t)is the exact float64 of the float32t, andtorch.fullcasts back to the same float32 → every device-side input is bit-identical; the CPU-sidet > cfg_intervalcomparison yields the same boolean without a sync. Removes ~143k blocking syncs per window (~4097 per denoising timestep). - switches.json:{"environment": {"OPT_SYNC_FREE_TIMESTEP": "1"}}-timeresult: /tmp/time-r1 — windows [22.2733, 22.7816, 22.3855] s/step,speedup_vs_frozen_reference= 1.0268 (mean over windows). Per-window ratios vs the frozen reference [22.7266, 23.5265, 22.9922]: 1.0204 / 1.0327 / 1.0271 — the minimum (1.0204) is below the gate-1 threshold max(1.02, 1+2×0.0152) = 1.0304, so round 1 alone would be judged no_benefit. More sync elimination needed (round 2: attention helpers). Peak memory 130.7 GiB (reference ~131) — unchanged.
1.0 hRound 2 — attention-helper host mirrors (2026-11-06 19:00–19:30)
Hotspot (from the round-1 profile, #2): per-layer.item()reads in the KV-cache attention helpers ofmodeling/bagel/qwen2_navit.py—_static_kv_cache_attention(3/layer),_merge_static_kv_prefix_and_query(5/layer),_static_kv_cache_readonly_attention(2 via merge): ~47k blocking syncs during text gen (582 forwards × 27 layers × 3) + ~12k during denoise per window. Every read blocks the host on a D2H copy while the GPU idles. Change: OPT_SYNC_FREE_ATTN (N0) — host mirrors of the exact integers the device reads: -StaticKVCache(qwen2_navit.py): newseq_lens_hostmirror (list of ints,None= stale → fall back), newset_seq_lens(values)(identical device opseq_lens[:] = torch.tensor(...)plus mirror),advance_seq_lensmaintains the mirror (stale-marks it if indices are untagged). - Producers tag the length tensors with the host values they were built from (attribute_r3_host):prepare_promptstext_token_lens,prepare_vit_images/prepare_vae_images/prepare_vae_latentpacked_seqlens (bagel.py);generate_text_batchquery_lens_for_model ([1]*n) and active_sequences_original_indices (host list kept in lockstep with the boolean-mask filter, also replaces the per-element.item()of original_batch_idx);update_context_text/imagearange cache indices (rollout_controller.py); the twoseq_lens[:] = torch.tensor(...)writes becomeset_seq_lens(kv_lens). - The three helpers read max/sum/first scalars from the mirrors when present (and skip the(query_lens == query_lens[0]).all()GPU compare), else take the original.item()path. All device tensors and kernel inputs are unchanged; every mirrored value is the exact integer the device read would return, and any untagged/stale mirror falls back. Unit smoke test of the mirror bookkeeping (incl. deepcopy of cfg contexts, partial indices, stale marking): /tmp/analysis/test_host_mirrors.py — ALL PASS. - switches.json:{"environment": {"OPT_SYNC_FREE_TIMESTEP": "1", "OPT_SYNC_FREE_ATTN": "1"}}-timeresult: /tmp/time-r2 — windows [21.529, 21.2606, 21.2715] s/step,speedup_vs_frozen_reference= 1.0809; per-window ratios 1.0556 / 1.1066 / 1.0809, min 1.0556 — above the gate-1 threshold 1.0304. Per-step: refine 14–15.5 s → 12.7–13.3 s, gen 31–33 s → 27.3–29.7 s. Peak memory 130.72 GiB (unchanged).
1.5 hRound 3 — overlap prepack with the reward wait (2026-11-06 19:30–19:55)
Hotspot: the post-rollout host phase — the reward futures are resolved (blocking HTTP wait, ~1.1–4.1 s of pure idle per step) before any packing work starts.prepack(tokenization, timestep subselection viarandom.sample, data-point assembly) reads only rollout outputs, never reward values. Change: OPT_OVERLAP_PREPACK (N0) —train/online_rl.py: when set, runprepackright afterrollout()returns (reward futures still in flight) instead of after the futures loop. RNG order is provably unchanged: prepack'srandom.sampleis the only Python-global-random consumer in the loop, and nothing between the two positions consumes Python or torch RNG (rollout_buffer.sampleuses its own seededtorch.Generator). prepack mutates onlyper_sample_results, disjoint from every field the futures loop reads. - switches.json:{"OPT_SYNC_FREE_TIMESTEP": "1", "OPT_SYNC_FREE_ATTN": "1", "OPT_OVERLAP_PREPACK": "1"}-timeresult: /tmp/time-r3 — windows [20.9422, 21.7319, 21.4014],speedup_vs_frozen_reference= 1.0807; per-window ratios 1.0852 / 1.0826 / 1.0743, min 1.0743 (up from 1.0556 — the weakest window improved; gate 1 needs > 1.0304). Peak memory 130.72 GiB (unchanged). ### Trace findings for the next rounds (round-1 trace, /tmp/analysis/) - Allocator stalls: 324 mscudaStreamSynchronize+ 471 mscudaFree+ 270 ms sync insideOptimizer.step#AdamW.step, and 113+215+216+212 mscudaFree/sync insideFSDP::root_post_backward_callback— the caching allocator evicting blocks under memory pressure (130.7 GiB peak, reward workers on the same GPUs).PYTORCH_CUDA_ALLOC_CONFis not in the judge'sreserved_env→ tryexpandable_segments:Trueas a pure switch. -aten::nonzero× 19,312 (all underaten::index= boolean-mask advanced indexing): dominated by_merge_static_kv_prefix_and_query— itspast_mask/new_maskflat-index computations run once per layer (27×) per forward although they depend only on the sequence lengths. ~7 nonzero syncs × 27 layers × ~95 denoise forwards + text forwards ≈ 19k per window. Plan: compute the flat index tensors once per forward (cache keyed on the host length mirrors) and reuse across layers — same elements in the same row-major order, so all gathers/scatters are bit-identical. - Text-gen token loop still has ~3 blocking reads per token per active sequence (round-5 change already coded: one.tolist()per tensor per token, OPT_SYNC_FREE_TOKENLOOP). - PIL pickling forrollout_buffer.all_gathermeasured trivial (6 ms / 8 images, raw-byte transport) — NOT a hotspot, idea dropped.check --quickon rounds 1–3 (/tmp/check-q1, 2026-11-06 20:07): PASS — gates 0 and 2 bitwise identical to the frozen reference record (14 steps, deterministic mode; all optimizer parameter/state SHA-256s and every reward-request hash match). Switch set verified: TIMESTEP + ATTN + PREPACK.
1.7 hRound 4–6 — token-loop batched reads, merge-index cache, allocator conf (2026-11-06 20:00–20:45)
- OPT_SYNC_FREE_TOKENLOOP (N0):generate_text_batchreads the per-token device values with one.tolist()per tensor per token (3 blocking reads) instead of ~3.item()per active sequence (~12 at n=4); reads happen before the loop's device writes and iteration i only writes the index it reads → identical integers. Unit-verified against the original path. - OPT_MERGE_IDX_CACHE (N0):_merge_static_kv_prefix_and_query(and the varlen write-back in_static_kv_cache_attention) build gather/scatter indices via boolean-mask indexing (aten::nonzeroblocking syncs) once per layer although they depend only on the sequence lengths. Now computed once per length signature (LRU-cached module dict) and reused across the 27 layers. Verified bitwise-identical to the boolean-mask path on CPU: /tmp/analysis/test_merge_cache.py + test_merge_equiv.py (unequal/zero/equal lens, selected subsets, dtypes). ~19k fewer nonzero syncs per window. - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (pure env switch, not in reserved_env): targets the 471+215+216+113 mscudaFreestalls and 324+270+212 ms syncs found insideOptimizer.step#AdamW.stepandFSDP::root_post_backward_callback— allocator evictions under memory pressure (130.7 GiB peak). Bitwise-neutral by construction (allocator only). - switches.json: all six entries (TIMESTEP, ATTN, PREPACK, TOKENLOOP, MERGE_CACHE, PYTORCH_CUDA_ALLOC_CONF). -timeresult: /tmp/time-r4 — REGRESSION: windows [22.7692, 25.4165, 23.1565],speedup_vs_frozen_reference= 0.9706, per-window ratios 0.9981/0.9256/0.9929, min 0.9256. Even steps 31–34.7 s (vs 27.3–29.9 in time-r3), odd 13.9–16.2 s — every phase slower. Attribution: the two code changes strictly remove device syncs and ops (unit-verified), soPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueis the suspect (VMM mapping overhead on this workload's many small allocations). Dropped from switches.json; re-timing the five OPT_* switches as /tmp/time-r5. Lesson: allocator settings are not free wins here — the 471 ms cudaFree stalls stay unfixed for now. -timeresult (five switches, alloc conf dropped): **/tmp/time-r5 — windows [20.282, 21.7367, 20.8982],speedup_vs_frozen_reference= 1.1006; per-window ratios 1.1205 / 1.0823 / 1.1002, min 1.0823** (up from 1.0743). Confirms the attribution: tokenloop + merge-cache add ~0.8 pp on the min ratio and expandable_segments cost ~10 pp. Peak memory 130.73 GiB (unchanged). Fullchecklaunched on this set: /tmp/check-full-1. Fullcheckon the five-switch set (/tmp/check-full-1, 20:40–21:06): PASS. Gate 1: windows [21.429, 21.6817, 20.8499] s/step → ratios 1.0606 / 1.0851 / 1.1027, **speedup 1.0606** (min window ratio; above the 1.0304 gate). Gate 2 (correctness.numerics): PASS — bitwise identical to the frozen reference record (optimizer params/state SHA-256s and every reward request). Peak memory 130.73 GiB (unchanged). The five-switch submission state is fully verified end-to-end. (time-r5 had measured 1.0823 on the same set — run noise; both well above the gate.)
2.6 hRound 7 — CUDA-graph text-gen step (2026-11-06 20:45–22:44: promoted — full check PASS 1.3739)
Hotspot (round-1 profile): the think phase is host-dispatch bound — ~5,700 host ops and ~1,687 kernel launches per token, GPU busy only 19% (22 tok/s, 12.5 s per refine-think). Change: OPT_TEXTGEN_CUDAGRAPH (N0) — capture the per-token text forward (embed_tokens→forward_inference(mode "und") →lm_head) as a CUDA graph, one per active-batch size n (4,3,2,1; n only decreases within a call, so shared-pool replay order equals capture order). Inputs live in fixed buffers refilled before each replay; the KV cache, itsseq_lensandselected_cache_indicesare read **on device inside the captured region** (the fast path callsflash_attn_with_kvcache, which has nomax_seqlen_kparameter and takescache_seqlensas a device tensor — per-batch bounds are read at replay time), so nothing that changes per token is baked into the graph except the constantseqlen_q=1. Sampling (explicit torch Generator) stays outside the graph → RNG stream unchanged. Graphs are rebuilt pergenerate_text_batchcall (they must not outlive the rollout context or the StaticKVCache), any capture error falls back to eager for the rest of the run, and the switch requiresOPT_SYNC_FREE_ATTN(capture region must be free of blocking device reads). Committed c721985; switch OFF in/submission/switches.jsonuntil verified — testing via the separate config/tmp/switches-graph.json(six switches). Pre-flight verification (synthetic, /tmp/analysis/test_cudagraph_kvcache.py, GPU):flash_attn_with_kvcacheis capture-safe, and replay with refilled input buffers and ADVANCEDcache_seqlens(device writes between replays, the exact token-loop pattern) is bitwise identical to eager across an 8-token loop. Firsttime(/tmp/time-r6): ERROR — OOM. The run died inloss.backward()at step 0 (rank 1): the graphs engaged (private pools live), but the side-stream warmup blocks + private pools shifted the allocator's segment layout; the backward's 740 MiB contiguous request found no free block (3.58 GiB reserved-but-unallocated inside torch, only 214 MiB free at the driver — the reward worker holds 3.92 GiB of the same GPU). The reference runs at ~0.5 GiB headroom, so any extra reservation is fatal. A synthetic allocator test (/tmp/analysis/test_graph_alloc.py) showed the captures themselves are cheap with room to spare — the failure is fragmentation at the razor edge. Fix (commit 8971c45): release free segments withtorch.cuda.empty_cache()right afterrollout()returns, gated on OPT_TEXTGEN_CUDAGRAPH — the cost lands inside the reward-service wait (pure idle), and the training phase's large allocations then find contiguous memory. Memory management only, no tensor value touched. Also cut the graph warmup from 3 to 2 iterations. Re-timing as /tmp/time-r7. Evidence the graph path itself works: time-r6's log shows the step-0 think phase at ~108–118 it/s (385 tokens in 3.0 s) vs the reference's ~26 it/s (~14.7 s) — a ~4.2× speedup of the think phase, before the run died at the backward. **Re-time with the empty_cache fix (/tmp/time-r7): same ERROR, byte-identical memory state** (129.94 GiB allocated / 3.58 GiB reserved-unallocated / 214.38 MiB driver-free, 740 MiB request inloss.backward(), rank 1, step 0). So the empty_cache is not the lever: the train phase's own deterministic re-carve recreates the same layout, and the graphs' ~10 MB direct footprint plus the rollout-phase carve shift is enough to remove the one contiguous ≥740 MiB hole the reference's backward relied on. The heap runs at ~214 MiB driver headroom — any perturbation is fatal. Next attempt (/tmp/time-r8): the failure is textbook fragmentation (free space exists, no contiguous piece), so add the allocator switchPYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256— large free blocks are then never split for small requests, keeping big holes intact for the backward's 740 MiB allocations. Zero code, not in reserved_env;expandable_segments(the setting that regressed 10 pp in round 4) is a different mechanism. Config: /tmp/switches-graph2.json (six switches + the conf). /tmp/time-r8: ERROR again, OOM moved — now a 140 MiB request fails with 8.22 GiB reserved-but-unallocated and 16 MiB driver-free (rank 0): max_split makes the allocator reserve MORE (small requests carve new segments instead of splitting), so the conf is a dead end, dropped. Root cause found (torch source, torch/cuda/graphs.py:240-252):torch.cuda.graph.__enter__unconditionally runstorch.cuda.synchronize()+torch.cuda.empty_cache()before EVERY capture (gc.collect() only when forced — default off). With 4 captures per think call × 32 calls, that released all free segments repeatedly and reshaped the allocator; the train phase's deterministic carve then had no contiguous ≥740 MiB hole. It also explains why the r7 empty_cache changed nothing (the captures had already done it, and the train phase re-carves identically). Fix (commit 226faee): capture through the low-levelCUDAGraph.capture_begin/capture_endon a dedicated side stream — same kernels, same pool, but the allocator's segment layout stays identical to the eager rollout. Dropped the post-rollout empty_cache (it would now be the only reshaper) and the allocator conf. Synthetic low-level-path capture/replay test re-verified bitwise-equal (/tmp/analysis/test_cudagraph_kvcache_lowlevel.py). /tmp/time-r9: ERROR — the SAME OOM, byte-identical totals again (740 MiB request, 129.94 GiB allocated / 3.58 GiB reserved-unallocated / 214.38 MiB driver-free, rank 1). The totals are live-set-determined and identical across all three capture variants; what differs from the working five-switch runs is the free-block STRUCTURE (where the free bytes sit). Re-reading the situation: all three failures are at step 0's backward — the first optimizer step materializes the entire AdamW state (~44 GiB/rank of fp32 params + moments), the largest allocation wave of the run, on a heap that also still carries the step-0 evaluation's leftovers. The five-switch runs survive that squeeze by layout; any extra allocation source re-rolls the dice. Fix 2 (commit 092b124): engage the graphs only from optimizer step 1 (curr_step >= 1, set in online_rl.py before the eval/rollout; bagel's_TEXTGEN_CUDAGRAPH_ONdefaults off). Step 0 runs the eager path — exactly what the reference computes, no value changes — and the measured windows (10–15) plus the gate-2 record (steps 1–13) run the graph path. Deterministic rule, identical in every run. **/tmp/time-r10: OK — the big one. Windows [16.3512, 17.01, 16.4138] s/step → per-window ratios 1.3900 / 1.3831 / 1.4006, min 1.3831,speedup_vs_frozen_reference1.3912** (gate 1 needs > 1.0304; the five-switch state was 1.0606–1.0823). Per-step: refine 28.5 → ~22.9 s, gen 12.7 → ~10.3 s. Peak memory 131.2 GiB (reference 130.7 — the pools cost ~0.5 GiB).check --quickon the six-switch config (/tmp/check-q2, 22:08–22:20): PASS —correctness.numericsPASS, bitwise identical to the frozen reference record with the graph path engaged (steps 1–13 graphed, step 0 eager). **Promoted:/submission/switches.jsonnow has all six switches.** **Fullcheckon the promoted six-switch state (/tmp/check-full-2, 22:21–22:44): PASS — speedup 1.3739** (windows [16.1519, 17.0889, 16.7345] s/step, ratios 1.4071 / 1.3767 / 1.3739; gate 2correctness.numericsPASS — bitwise identical). Round 7 complete: the submission's verified speedup went from 1.0606 to 1.3739.
4.0 hRound 8 — denoise-loop sync hoist (2026-11-06 22:20–23:00: no clear gain, switch stays off)
Post-graph split (time-r10): gen ~10.5 s, refine ~22.7 s per step; denoise (~15 s/window, 66 % GPU busy) is now the dominant phase. A denoise_forward_flowgraph was scoped and DROPPED: cfg_interval is (0.4, 1.0], so the CFG branch flips mid-loop (two graphs needed), capture+warmup would cost ~2.4 s/window against ~1–2 s of dispatch savings — net marginal; the phase is GPU-bound, not dispatch-bound like think was (19 %). Change: OPT_SYNC_FREE_DENOISE (N0, commit 40c3814, off by default) — hoist the denoise loop's blocking reads: the SDE index set/first index read once per rollout instead of two blocking reads per timestep;sum(packed_seqlens)in_forward_flow(per timestep) and inforward_cache_update_vit/vae(per rollout) from the_r3_hostmirrors; the tautologicaltimestep.unique()assert skipped under the flag (the timestep vector is a uniform fill by construction); the three end-of-rollout(packed_seqlens - 2).tolist()reads reduced to mirror arithmetic. Same integers/booleans everywhere; ~150 fewer blocking syncs per window. To be timed after the full check (config /tmp/switches-r8.json, seven switches). /tmp/time-r11: ERROR — my bug: `AttributeError: 'list' object has no attribute 'tolist'at the firstgenerate_image_mix— online_rl passessde_timestep_idx` astimestep_idx.tolist(), i.e. a plain LIST, so the original membership tests never synced and my hoisted.tolist()crashed. Fixed (commit 198a945) to handle both shapes; for a list the hoist is a no-op, for a tensor it saves the two per-timestep blocking reads. Re-timed as /tmp/time-r12. /tmp/time-r12: no clear gain — round 8 conclusion: NOT shipped. Windows [15.9812, 16.9275, 17.0734] → ratios 1.4221 / 1.3898 / 1.3467, mean speedup 1.3854, vs the six-switch state's 1.3912 mean / 1.3831 min (time-r10): W1 −0.37 s, W2 −0.08 s, W3 +0.66 s (beyond the ±0.3–0.5 s run noise seen between r10 and check-full-2). The ~80 blocking syncs/window it removes are worth less than the noise. OPT_SYNC_FREE_DENOISE stays OFF (code committed, off by default — 40c3814 + 198a945);/submission/switches.jsonremains the verified six-switch set (full check PASS, 1.3739).
5.1 hRound 9 — pack-time per-element device reads (2026-11-06 23:00–23:40: not promoted; re-measured round 11, stays off)
Profile first (profile --out /tmp/prof-r9, steps 11–12, rank 0; trace deleted after analysis): - denoise phase now ~90% GPU-busy (rounds 1–2 sync fixes already recovered it) → a denoise_forward_flowgraph is not worth it (also two variants needed for cfg_interval). - train phase saturated (kernel time ≥ wall — nothing to win on host there). - think phase 60–69% GPU-busy (per-token eager sampling + 3 blocking tolists between replays). - reward-wait idle ~2.5 s/window. - Top host hotspot: a burst right afterOptimizer.zero_gradof 15,691 × (aten::item+aten::_local_scalar_dense+cudaMemcpyAsync+cudaStreamSynchronize) in ~0.5 s — pure blocking per-element device reads at pack time. Op census attributes it topack_batch/pack_iterator→add_latent. Root cause:add_latentrepeats the 0-dim CUDAdt/timesteptensors once per latent token intopacked_input['packed_timesteps'],loss_info['dts'],loss_info['packed_timesteps_for_loss'];_convert_to_tensorsthen rebuilds those lists withtorch.tensor(list, dtype=float32, device=...), which reads every element back with a blocking D2H.pack_batch's CFG filter also compares each per-sample timestep tensor against the bounds (2 syncs per element). Change (commit 1d87f41, switchOPT_PACK_HOST_FLOATS, off by default, N0):add_latentconvertsdt/timestepto host floats once per SDE step;pack_batchhoists one.tolist()of the shared per-sample timestep tensor for the CFG filter. Bitwise argument: float32→float64→float32 round-trip is lossless (finaltorch.tensoridentical; verified with an int32-view equality unit test). The CFG comparison keeps the same booleans: bounds are 0.4/1.0, and no float32 grid point lies between float64(0.4) and float32(0.4) (window < 1 ulp), 1.0 is exact — so host float64 comparison ≡ GPU float32 comparison for every possible timestep. Time (/tmp/time-r13, seven switches incl. OPT_PACK_HOST_FLOATS, 23:41): windows [16.5587, 16.9045, 17.1882], ratios 1.3724/1.3918/1.3376, min 1.3376, mean speedup 1.3671 — vs six-switch time-r10 min 1.3831 (windows [16.35, 17.01, 16.41]). The expected gain (~0.25 s/step) is at the run-noise floor (±0.3–0.5 s/window) and the delta is negative on this sample (W3 +0.78 s). **Not promoted** —/submission/switches.jsonstays at the verified six-switch set; the code is committed, off by default. To be re-measured stacked with the next round's change before any promotion.
5.4 hRound 10 — overlap the reward wait with pack + both forwards (2026-11-06 23:45–00:15)
Hotspot from the round-9 profile: reward-wait idle ~2.5 s/window (the largest remaining non-GPU phase after denoise/train/think), plus the pack (host walk + VAE encode) and the ref/policy forwards all running serially after the join. Change (commit eaaf593, switchOPT_EARLY_FORWARD, off by default, N0; requires OPT_OVERLAP_PREPACK): the forwards need onlypacked_input/label_ids, never the reward-derived advantages. Under the switch,pack_iteratorruns once withdefer_advantages=True(pack_batch records every advantage-extend as (kind, count, (sample_idx, round_idx)) inloss_info['_adv_recording']), thenoptimizer.zero_grad()+fsdp_model.train()in the reference's order, then the forward block — factored into_run_forwards, same code for both paths — runs while the reward futures are in flight. After the join and advantage computation,finalize_deferred_advantagesreplays the recording with the real values. - Bitwise: same walk, same extend counts and order, sametext_advantages[sample_idx][round_idx]values (including theelse 0fallback); unit test (/tmp/analysis/test_deferred_advantages.py): deferred pack vs normal pack → identical packed_input, identical non-advantage loss_info, torch.equal on both advantage tensors. - Collectives: the join, buffer update/all_gather, advantage all_reduces and the stats barrier all move together after the forwards; per-rank order stays identical, so every collective still matches (no deadlock; collectives complete only when all ranks enter). - RNG: the moved block consumes none. - Memory: allocation sequence kept (zero_grad → pack → ref fwd → policy fwd); the only trajectory change isrollout_buffer.all_gather()'storch.cuda.empty_cache()now running between the policy forward and the backward on gen steps (was before the pack). Step 0 (the knife-edge step) is exercised by the time run. Time (/tmp/time-r14, six-switch base + OPT_EARLY_FORWARD, 00:01): windows [15.5648, 16.0194, 15.9711], ratios 1.4601/1.4686/1.4396, min 1.4396, mean speedup 1.4561 (six-switch time-r10: min 1.3831, mean 1.3912). All three windows improved by ~0.7–1.0 s — a real gain (~0.85 s/window), not noise. Step 0 (the memory knife-edge) completed normally (64.55 s incl. eval); peak allocated 131.09 GiB (was 131.45). Promoted into/submission/switches.json(seven switches);check --quick(bitwise) launched → /tmp/check-q4.
6.2 hRound 11 — denoise-step CUDA graphs (2026-11-07 00:12–01:00: negative, switches stay off)
Profile first (profile --out /tmp/prof-r11, steps 11–12, seven-switch state; 2.0 GB trace, deleted after analysis; bucketed with /tmp/analysis/trace_phases.py): - refine step: denoise 3.2 s at ~69% GPU busy (~1.0 s idle; ~45 ms host gap per timestep, ~3.5k kernel launches each), think 384 tok 9.2 s at ~94.5% busy (graphs already recovered it), pack ~0.5 s with the 15–18k-read storm still present, forwards+train ~7 s saturated. - gen step: denoise 2.4 s at ~67% busy (~0.8 s idle), think 4.2 s at ~94% busy, pack ~0.5 s, forwards+join ~1.6 s (the round-10 overlap works: the reward wait is hidden behind pack+forwards), train ~1.4 s saturated. - Denoise step wall ≈150 ms with ≈103 ms of kernels → the idle is host dispatch, exactly what a CUDA graph removes. The scale branches in_forward_flow(if cfg_text_scale > 1.0gates the cfg pair) mean two kernel variants per call; the trace shows the (1.0,1.0) timesteps run at half intensity (2–3 of 37 per window) — capturing that variant costs more than it saves, so only the full-CFG variant (32/37 steps) is captured. Change (switchOPT_DENOISE_CUDAGRAPH, off by default, N0; **requires OPT_SYNC_FREE_DENOISE=1**: without the round-8 hoist,_forward_flow'ssum(packed_seqlens)is a blocking read per timestep, illegal inside capture; the_dn_okgate enforces it):_DenoiseStepGraph(same low-level capture pattern as the text graphs: 2 read-only warmups on a side stream,capture_begin(pool, capture_error_mode="global")on the shared capture stream) captures one_forward_flowper generate_image_mix call; x_t/timestep live in static buffers refilled before each replay; the SDE timestep (noise draw) and the 1.0/1.0 timesteps stay eager. The eager call was rewritten toself._forward_flow(x_t=…, timestep=…, cfg scales…, **_dn_kwargs)— identical kwargs, one source of truth for both paths. Engagement gate from optimizer step 1 (same rule as the text graphs; step 0 is the memory knife-edge). Capture-failure fallback prints once and stays eager for the run. Synthetic capture/replay equivalence test: /tmp/analysis/test_denoise_graph.py. Time (/tmp/time-r15, eight switches = seven + OPT_PACK_HOST_FLOATS re-measure, 00:41): windows [15.6157, 16.1178, 16.0789], min-ratio 1.4300, mean 1.4483 — at-or-below the seven-switch baseline (time-r14 min 1.4396) for the second independent time. The early pack of round 10 already hides the pack read-storm inside the reward wait, so the fix has nothing left to win. OPT_PACK_HOST_FLOATS stays OFF permanently (code committed, off by default). Denoise-graph timing (/tmp/time-r16, nine switches = seven + OPT_SYNC_FREE_DENOISE + OPT_DENOISE_CUDAGRAPH, 01:00): windows [16.3453, 17.115, 17.0271], min-ratio 1.3503, mean 1.3715 — a consistent ~0.9 s/window REGRESSION vs the seven-switch baseline (min 1.4396). Zero capture-failure prints: the graphs engaged and simply don't pay. A ~3.5k-kernel graph instantiate costs ~1.2–1.5 s per call, and one call replays it only 14–18 times (~0.6–0.8 s of host-gap savings) — structurally negative, unlike the small text-step graphs (~300 kernels, replayed 196–384× per call). **OPT_DENOISE_CUDAGRAPH and OPT_SYNC_FREE_DENOISE stay OFF** (code committed 16d1a63, off by default; mechanics unit-tested bitwise — the failure is economic, not correctness). The denoise phase's ~1.8 s/window idle is locked behind per-call capture cost: the SDE index is re-sampled per call, so no multi-step or persistent graph aligns with the loop. Round 11 close-out. Fullcheckof the seven-switch submission state (/tmp/check-full-3, launched 00:54, done 01:15): PASS, speedup 1.3854 (gate 1 windows [15.5885, 16.1145, 16.5955], ratios 1.4579/1.4600/1.3854; gate 2 bitwise identical: 14 optimizer steps, param/state hashes and all reward requests matchbench/records/ref.json). This is the submission's headline evidence — the full judge procedure on the exact tree + switches.json now in/submission. Window 3 varies ±3% between runs (16.60 here vs 15.97 in time-r14), which is the whole difference between 1.3854 and 1.4396; the gate needs only > 1.0304. Post-round-11 screening (no GPU cost worth a round):torch.optim.AdamW(fused=True)as a possible optimizer-phase win — pre-test /tmp/analysis/test_fused_adamw.py shows params differ by ~1 ULP (1.19e-07) and exp_avg by 5.6e-09 after 20 steps on float32: the fused kernel's scalar math (bias correction) does not match the foreach path. Ruled out for N0. (Also noted: fused keepsstepon CUDA, the default on CPU.)
7.1 hRound 12 — batched token-loop bookkeeping (2026-11-07 01:05–01:32: no measurable gain, switch stays off)
Hotspot (from prof-r11): the think phases are 94–95% GPU busy; the residual ~5% (≈0.74 s/window) is per-token host work: three.tolist()blocking reads, the boolean-mask index that buildscontinuing_original_indices(a hiddennonzeroDtoH sync), and oneindex_putper active row. Change (switchOPT_TOKENLOOP_BATCHED, off by default, N0; requires OPT_SYNC_FREE_TOKENLOOP + OPT_SYNC_FREE_ATTN's host mirrors): ingenerate_text_batch's token loop, (1) one fused blocking read — active tokens, current tokens of the active rows and sampled probabilities packed as int32 views (int64 → two little-endian halves, float32 → raw bits) into onetorch.catand one.tolist(), reassembled on the host (exact round trip;(lo & 0xffffffff) + (hi << 32), struct for the floats); (2) the EOS comparison stays host-side on the same values, sokeep_hostknows the continuing rows and the boolean-mask indexing (blocking nonzero) plus the per-rowcurr_tokens[...] = next_tokens[i]writes collapse into ONEindex_putover host-built unique indices;continuing_original_indicesbecomes a host-built int64 tensor with the same values, order and_r3_hostmirror; (3) the last-stepend_token_idoverride becomes the written values themselves (torch.full), exactly what the reference writes for continuing rows. Per token: 4 blocking reads + ~8 launches → 1 blocking read + ~4 launches. Unit test /tmp/analysis/test_tokenloop_batched.py: trajectories (incl. values ≥ 2^31 and negatives), EOS corners (all-rows-end, mid-loop, last step), continuing-index values/mirror, int64+float32 decode round-trips — ALL OK (the first test run caught a real bug: signed-int32lomust be masked unsigned before reassembly). Time (/tmp/time-r17, eight switches, 01:16): windows [15.5388, 15.9394, 16.12], ratios 1.4628/1.4760/1.4263, min 1.4263, mean 1.4550 vs seven-switch time-r14 min 1.4396, mean 1.4561 (and the same config's gate-1 run in check-full-3: mean 1.434). w1/w2 improve by 0.03/0.08 s, w3 degrades by 0.15 s — window 3 swings ±0.6 s between identical-config runs, so the ~0.1 s/window expected gain is unresolvable. Switch stays OFF; code committed off-by-default. The submission remains the seven-switch full-check-PASS state (1.3854).
7.4 hRound 13 — profile-verify round 12; denoise idle re-attributed (2026-11-07 01:33–02:05: both no-gain, submission unchanged)
Round 12 verification by mechanism (profile --out /tmp/prof-r13, steps 11–12, eight switches = seven + OPT_TOKENLOOP_BATCHED; trace bucketed with trace_phases.py): - think spans: refine 8.60 s @ 94.2% busy (r11) vs 9.20 s @ 93.6% (r13); gen 4.00 s @ 94.2% vs 4.20 s @ 94.2%. GPU kernel time itself moved +6% between the two profile runs (clock variance), and wall/kern is unchanged (1.060-1.061 vs 1.063-1.069). The batched token loop does NOT reduce the think host gap measurably — the removed syncs were data-ready (cheap) and the residual gap is the sample-chain serial dependency. OPT_TOKENLOOP_BATCHED definitively stays OFF (round 12's wall-clock wash confirmed by mechanism; unit-tested bitwise, committed ac76a28 off-by-default). Denoise idle re-attribution (new tools /tmp/analysis/trace_gaps.py + gap_hosts.py on the r13 trace): the denoise phase's profiled 67-70% busy decomposes into (a) ~293 ms per 2 s window of sub-µs inter-kernel gaps (320k kernels in 2 s — ~25k per timestep, avg 4.2 µs; ~0.9 µs launch floor each), (b) ~368 ms in 138 gaps of 1-10 ms — **and those are filled byPyTorch Profiler [Trace]/Buffer-Flush events, i.e. CUPTI overhead, not workload stalls**, (c) one 39 ms gap (also a Trace flush). Profiling also inflates wall (steps 11-12 span 33.0 s profiled vs ~26.4 s in time-r17 — ~25-30%), so profile-based idle shares are systematically overstated. Cross-checking with wall clocks: the TRUE denoise host overhead is ~0.5–0.8 s/window — which is exactly the replay saving the round-11 graph experiment measured against its ~1.2–1.5 s/call instantiate cost (time-r16 net −0.9 s/window). The denoise lever is closed from both directions, and round 8's OPT_SYNC_FREE_DENOISE no-gain is explained: the queue it would drain never builds depth because the host and GPU run at nearly the same pace. State after round 13: submission = seven-switch set, full check PASS 1.3854 (/tmp/check-full-3). No further lever with positive expected value has survived profiling; remaining rounds keep the state verified and documented. Round 13 close-out.check --quickon the CURRENT tree (ac76a28) with the submission config (/tmp/check-q5, 01:49–02:00): PASS, bitwise identical — the default switch-off path is verified after round-12's edit (check-full-3 had run on the pre-round-12 tree). Final fullcheckof the exact stop-time state launched at 02:01 → /tmp/check-full-4 (gate 1 timing + gate 2, ~40 min). FINAL VERIFICATION. Fullcheckof the exact stop-time state — tree at ac76a28,/submission/switches.json(seven switches) — PASS, speedup 1.4357 (/tmp/check-full-4, 02:01–02:22): gate 1 windows [15.5894, 16.1182, 16.0141] → ratios 1.4578/1.4596/1.4357 (min); gate 2 bitwise identical (14 optimizer steps, param/state hashes and all reward requests matchbench/records/ref.json). Combined with check-full-3 (PASS 1.3854) and check-q5 (quick PASS on the same tree), the submission is verified end-to-end at both ends of the window-3 variance band (±3%); the gate threshold is 1.0304.
8.2 hRound 14 — persistent text-graph pool vs window-3 drift (2026-11-07 02:24–02:36: ERROR, switch stays off)
Target: w3 is the deciding (slowest) window in every timing of the candidate ([15.56, 16.02, 15.97] r14; [15.59, 16.11, 16.60] full-3; [15.59, 16.12, 16.01] full-4; per-step times rise ~2–4% over steps 10–15 for BOTH step types) while the reference shows no such rise ([22.73, 23.53, 22.99] — w2 highest). Hypothesis: the per-call text-graph CUDA-graph pool (created/destroyed ~8× per optimizer step) churns the caching allocator, increasingly so in later steps. Change (switchOPT_TEXTGEN_GRAPH_POOL, off by default, N0 by argument — a pool only decides where graph-private allocations live): pointgenerate_text_batch's pool holder at a module-level list so all calls share one pool (the documented torch pattern; graphs still captured/destroyed per call; ~200–500 MB permanently reserved, no step-0 risk since graphs engage from step 1). Time (/tmp/time-r18): ERROR on every rank — `RuntimeError: Offset increment outside graph capture encountered unexpectedly` (torch graph-RNG bookkeeping). Sharing a pool across calls is incompatible with this rollout's sampling pattern: the generator advances eagerly (multinomial between graph replays), and re-entering a pool with a new capture after eager generator advances trips torch's graph-safe-RNG assertion. The run never reached step 16; the bench released the lock normally. Switch stays OFF (committed off-by-default); the w3 drift remains unexplained — most plausibly thermal (the candidate's per-step rise is smooth and the reference's own bootstrap per-step data would settle it, but no lever exists either way). Also closed by code audit this round: the denoise host overhead IS the per-layer aten dispatch (~3000 layer-forwards/window × ~25–35 dispatches each) — only CUDA graphs remove it, and their instantiate economics are settled negative; the per-timestep invariant embedding computes in_forward_flow(embed_tokens/latent_pos_embed/text placement) are worth only ~40–70 ms/window — below measurement resolution, not attempted. Round 14 close-out — the window-3 "drift" is data-point structure, not our allocator. The reference's own bootstrap per-step times (runs/bootstrap/full/timing_ref_a/b.r*.json) show the SAME late rise on the same steps (ref_a gen steps 10→14: 30.95 → 31.96 → 32.29 s; candidate: 21.2 → 21.5 → 21.7) and large prompt-dependent per-step variation (ref_b refine step 5: 17.58 s vs step 7: 14.09 s). Each optimizer step processes a different data point whose context length and reward latency differ; the frozen recipe gives both trees the same prompts in the same order, but reward-latency effects do not cancel in the ratio (the reference's waits are exposed, ours are hidden). The candidate's w3 ratio is lower than w1 mostly because the reference's w3 is its fast window (22.99 vs 23.53 in w2). Within a run, all four ranks' per-step times agree to ~10 ms (collective lockstep), so the slowest-rank measurement is tight. No lever exists on our side; the window-3 question is closed.
8.6 hRound 15 — third timing sample of the submission state; log coherence pass (2026-11-07 02:47–03:02: no change, state confirmed)
No lever remains with positive expected value (rounds 11–14 closed each candidate with evidence), so this round used idle GPU time for an additional independent timing sample of the EXACT submission state (tree f1f8bc4 + the seven switches): /tmp/time-r19 — windows [15.5068, 16.0102, 16.3602] s/step → ratios 1.4656/1.4695/1.4053, **min 1.4053, mean 1.4463**, peak 131.09 GiB, run clean. The submission's four timing samples now band as: w1 15.51–15.59 (very stable), w2 15.94–16.12, w3 15.97–16.60 (the prompt/reward- structure window); min-ratio 1.385–1.440, mean 1.434–1.456 — every sample ~30× above the gate-1 threshold (1.0304), and the two full checks of this config PASS at 1.4357 and 1.3854. Also fixed four stale "in progress" round titles (4–6, 7, 8, 9) so the log's headers state each round's verified conclusion. No code change; nothing to commit.
8.6 hRound 16 — paired (live-reference) timing; final direction sweep (2026-11-07 03:05–, in progress)
Swept the remaining idea space before spending GPU time; each closed by argument: -AdamW(capturable=True): movesstepto CUDA and computes bias correction in the kernel — different scalar rounding than the host-float64 path (same class as fused, which measured 1-ULP diffs) — not N0. - NCCL algorithm/transport tuning for the stats collectives: the ~0.4 s/window is wait-for-slowest-rank (rollout-duration variance from SDE randomness), not transport; and the skew is semantics-locked. - Reward-worker GPU contention during the wait overlap: inherent — the service processes requests on arrival; the overlap is still a large net win vs serial waiting. - Overlapping the optimizer step with the next rollout: already async (the rollout's Python proceeds while optimizer kernels drain; the post-hook sync is the gate's own). - The_forward_flowper-timestep invariant computes (embed_tokens/latent_pos_embed/text placement): ~4–6 ms/window total — below resolution, not attempted. Measurement:time --paired— the one timing mode never run — times the LIVE reference immediately before the candidate, the closest proxy to the final evaluation's gate-1 (fresh reference runs). /tmp/time-r20, launched 03:05, ~25 min. Paired timing result (/tmp/time-r20, reference at 03:05 → candidate, done 03:33): live reference windows [23.0216, 23.8625, 23.3631] (1.0–1.5% above the frozen [22.7266, 23.5265, 22.9922] — machine state stable), candidate windows [15.5836, 15.9047, 16.0041] (best w2/w3 of any run), **per-window paired ratios 1.4773 / 1.5003 / 1.4598 — min 1.4598**,speedup_vs_paired_reference1.4598, peak 131.09 GiB, both runs clean. This is the closest proxy to the final evaluation's gate 1 (fresh reference → candidate → fresh reference, min(refA,refB) per window — with two reference runs the expected ratio is at or slightly above this single-reference figure). Combined with the two full-check PASSes (1.4357, 1.3854) and four frozen-reference samples (min-ratio band 1.385–1.440), the submission's expected final speedup is ~1.39–1.47 — roughly 30–45× the gate-1 threshold (1.0304). Round 16 conclusion: no change; measurement-only round, and the last untried timing mode is now used. Tree f1f8bc4, switches.json unchanged.
9.0 hRound 17 — reward-path audit (2026-11-07 03:36–03:45: closed by measurement, no change)
Last unexplored corner: the JPEG-encode + reward-POST path (exposed pre-wait work:
4 serial PIL encodes per step before the batched POST; request order and JPEG bytes are
gate-checked, but PARALLEL encodes would be bitwise-safe — identical bytes, list order
preserved, one request per step). Measured directly on this host (192 cores, fast SIMD
PIL): 1.5 ms/encode at 512², 6.1 ms at 1024² → **6–24 ms/step = 12–48 ms/window — three
orders below the window**; the tensor→uint8→numpy conversion and pickle add ~10 ms.
Also re-checked and rejected by argument: moving the pack before the POST (would expose
it — the wait clock is fixed by the service), OMP/thread pinning (CPU-op reduction order
is not N0-safe), capturable AdamW (kernel-side bias-correction rounding). The reward
path is closed. With every direction now closed by measurement or argument, the round
spends otherwise-idle GPU on one more full check of the exact final state
(/tmp/check-full-5) — it PASSed at 1.4350 (window ratios 1.4632/1.4504/1.4350):
three full-check PASSes on this config now, 1.3854 / 1.4357 / 1.4350.
Third full check (/tmp/check-full-5, gate 1 03:27–03:37, gate 2 03:37–03:48):
PASS, speedup 1.4350 — window ratios 1.4632 / 1.4504 / 1.4350, gate 2 bitwise
identical, peak normal, run clean (the round-17 entry's "launched 03:45" was a
transcription slip; the bench log shows lock at 03:27:08). The exact final state
now has three full-check PASSes: 1.3854, 1.4357, 1.4350 — a tight band —
plus the paired live-reference measurement 1.4598 and four frozen-reference
timing samples (min-ratio 1.385–1.440). No processes remain; GPUs free.9.8 hRound 18 — second paired sample; last candidates re-closed (2026-11-07 03:52–04:18: no change, band tightened)
Re-closed the remaining "maybe" candidates before spending GPU time: the round-8 OPT_SYNC_FREE_DENOISE hoist (with the shallow dispatch queue its ~80–150 blocking reads are worth ~2–8 ms/window — consistent with its measured no-gain), switch combinations (independent zero-gain mechanisms sum to zero), and cross-step prepack overlap (the host is a shared serial resource: moving host work into the GPU-saturated train phase merely relocates it; only the reward wait is GPU-idle, already exploited by round 10). Second paired run (/tmp/time-r21, reference 03:52 → candidate, done 04:17): live reference [22.7864, 23.5677, 23.0798] (≈0.2% from frozen — machine stable), candidate [15.5435, 16.0539, 15.9285], paired ratios 1.466 / 1.468 / 1.449 — min 1.449. The paired band over two samples is 1.449–1.460, the tightest and most final-like estimate: the final evaluation's min(refA,refB)/cand with two reference runs should land at or above ~1.45. Total evidence: 3 full-check PASSes (1.4350/1.4357/1.3854), 2 paired samples (1.4598/1.449), 4 frozen samples (min 1.385–1.440), quick-check PASS on the current tree. No change; nothing to commit; no processes remain.
9.9 hRound 19 — stack the individually-unresolvable small levers (2026-11-07 04:20–, in progress)
The two committed-but-off switches with positive-but-unresolvable individual signals — OPT_TOKENLOOP_BATCHED (round 12: W1 −0.03/W2 −0.08 s; mechanism ~60–120 ms/window) and OPT_SYNC_FREE_DENOISE (round 8: W1 −0.37/W2 −0.08 s before a noisy W3; mechanism ~10–20 ms/window of shallow-queue reads + ~37 assert kernels) — touch disjoint code paths (think loop vs denoise reads). Their STACK (~0.2–0.3 s/window expected) may clear the seven- switch baseline band (mean 1.434–1.456, min 1.385–1.440 over 4 samples) where each alone cannot. Config /tmp/switches-r15.json (nine switches), time run → /tmp/time-r22, launched 04:20. Decision rule: mean ≥ 1.47 or min ≥ 1.46 → outside the band → quick-check and promote; inside the band → both stay off (no evidence of gain).