GLM-5.3 1.76×
Harness: Claude Code · switches: GS_OPT_FUSED_ADAM=1 GS_OPT_PIPELINE=1 GS_OPT_NO_EMPTY_CACHE=1 GS_OPT_SEPARATE_SHS=1 GS_OPT_LOG_EVERY=100 GS_OPT_NO_DEPTH=1 GS_OPT_SH_ADAM=1 GS_OPT_BWD_PIXCACHE=1 GS_OPT_GROUP_ADAM=1 GS_OPT_FAST_EXP=1 GS_OPT_ACT_FUSE=1 GS_OPT_STATS_FUSE=1 GS_OPT_CLAMP_FUSE=1 GS_OPT_TIGHT_RECT=1
- Judged
- 1.76×
- Tier
- N1
- Self-best
- 1.74×
- Rounds
- 21
- Hours
- 10.0
- API cost
- $55.2
- Input tokens · 99.5% cached
- 187 M
- Output tokens
- 1.2 M
- Requests
- 1,354
§ 02Key levers
- Smaller tiles + rewritten binning
- ✗ tile size unchanged; last round tightened candidate rectangles
GS_OPT_TIGHT_RECT=1(radix sort −39%); binning one warp per Gaussian (1.55 → 1.58) - Bypass autograd, activations folded into raster kernel
- ✗ activations as a pair of standalone kernels outside rasterization
GS_OPT_ACT_FUSE=1(1.60 → 1.70) - Lazy Adam (update only visible Gaussians)
- ✗ dense; SH grad and Adam fused into one kernel
GS_OPT_SH_ADAM=1(first version regressed to 1.28, after three reorderings 1.34 → 1.46) - Fused loss / SSIM
- only clamp backward fused (−11 µs / step); SSIM only changed to not zero its output
- Other main levers
- depth skipped, backward pixel state in shared memory (1.46 → 1.53), four small param groups in one Adam kernel,
__expf, densification stats kernel (+0.003)
§ ARaw trace
Raw work log21 sections
0.2 hRound 1 — profile + fused Adam (GS_OPT_FUSED_ADAM)
Start: 2026-11-07 05:31 (UTC-ish container time). End: (below) Profile of the unchanged reference (/tmp/r1/profile, steps 7000–7019, N≈1.3M): GPU busy ≈ 9.54 ms/step, window idle 16% (36.4/227 ms) — gaps at Adam.step/zero_grad,aten::item,aten::index/nonzero(4 nonzero syncs/step from the visibility.nonzero()+ indexed updates). Top per-step CUDA time: | kernel | ms/step | share | |---|---|---| | PerGaussianRenderCUDA (backward) | 2.62 | 27% | | Gaussian Adam (foreach multi-tensor ×7) | 2.52 | 26% | | renderCUDA (forward) | 1.59 | 17% | | preprocessCUDA fwd | 0.49 | 5% | | duplicateWithKeys | 0.48 | 5% | | cub radix sort | 0.22 | 2% | | index/gather/scatter (densif. stats) | ~0.4 | 4% | | fill/zero/misc | ~0.5 | 5% | Change:scene/gaussian_model.py—torch.optim.Adam(..., fused=True)behindGS_OPT_FUSED_ADAM. Tier: N1 (same Adam rule, different rounding/fusion order; state stays float32, step is 0-dim so the dtype gate item is unaffected). Time (/tmp/r1/time-fused): speedup_vs_frozen_reference = 1.1432 (total_s 267.797 vs ref 306.144; windows 76.733 / 97.764 / 93.300 s; peak mem 5.31 GiB;run_status ok). Note window 3 < window 2 although N is larger: densification (every 100 iters until 15k) costs real time inside windows 1–2. End of round 1: ~05:46. Committed asopt: fused Adam for the Gaussian optimizer (N1).
0.2 hRound 2 — host-sync / pipeline batch (GS_OPT_PIPELINE, GS_OPT_NO_EMPTY_CACHE)
Start: ~05:47. Class: host syncs + launch gaps (16% GPU idle in the profile, gaps ataten::item,aten::nonzero/aten::index,Optimizer.step/zero_gradboundaries). Changes (each off by default): -gaussian_renderer/__init__.py(GS_OPT_PIPELINE): visibility filter as bool mask instead of.nonzero()(removes 1 sync + gather/scatter sizing); means2D without the extra+ 0copy. -gaussian_renderer/__init__.py(GS_OPT_SEPARATE_SHS): pass dc / rest SH separately to the rasterizer (skips the per-iterationcatof a ~250 MB tensor); same values, bitwise. -train.py(GS_OPT_PIPELINE): EMA loss kept on GPU, read back every GS_OPT_LOG_EVERY=100 (no per-iterationloss.item());training_report/elapsed_timeonly when it would do something;max_radii2Dviatorch.maximum(radii==0 for invisible, max_radii2D>=0 → identical). -scene/gaussian_model.py(GS_OPT_PIPELINE): maskedadd_densification_stats(invisible Gaussians have exactly-zero mean2D grad → masked add adds zero; no gather/scatter, no sync). (GS_OPT_NO_EMPTY_CACHE): skiptorch.cuda.empty_cache()indensify_and_prune(device sync + pool drop every 100 iters; changes no computed value). Plan:check --quickwith all switches on (exact items incl. densification steps), thentime. Quick check (/tmp/r2/quick, pipeline switches on, NO_DEPTH off): PASS (exact_items_pass), 3000 steps in 19.4 s, final count 1,095,984 (reference 1,092,971 — counts are not an exact item; the change steps are, and they match).
0.4 hRound 3 — skip the never-consumed inverse-depth path (GS_OPT_NO_DEPTH, CUDA)
Start: ~05:50. The recipe's cameras carry no depth maps (--depths ''→depth_reliable=False), so the depth term of the loss never runs: the rasterizer's invdepth image and its gradient chain are dead work, computed every iteration. Changes (off by default;compute_invdepththreaded fromGaussianRenderSettingsinto both kernels as a template parameterCOMPUTE_DEPTH): -forward.cu renderCUDA: skipexpected_invdepthaccumulation (a divide + fma per contributing pair), thesampled_ardcheckpoint writes, theinvdepthstore, and thedepthsread in the pair loop. -backward.cu PerGaussianRenderCUDA: skip theard/dL_invdepthshuffle cascade, thesampled_ard/pixel_invDepths/dL_invdepthsreads, the per-pair depth gradient terms (incl. a1/(1-alpha)divide), and the finaldL_dinvdepthsatomicAdd. -rasterizer_impl.cu: skip thepixel_invDepthsD2D copy;out_invdepthstays the zero tensor it was allocated as. -gaussian_renderer/__init__.py:compute_invdepth = not GS_OPT_NO_DEPTH or camera.depth_reliable(a camera with a real depth map still gets depth). - Also in this round (pipeline class):scene/gaussian_model.pyexposure optimizer_SkipAllNoneGradAdam—step()returns before the per-parameter bookkeeping while every grad is None (torch's Adam updates nothing in that case anyway; the global step hook wraps the class'sstep, so the step is still counted). Numerics: the RGB image and every gradient that reaches the loss are value-identical (the removed terms were all multiplied bydL_invdepth = 0); possible ±0.0 sign differences in unused outputs. Tier N1. Build: 3 attempts (2 signature plumbings fixed);/tmp/gs-ext/6fc0d54b5e15a16ad8ecbda2builds ok. Quick check (/tmp/r3/quick, all switches incl. NO_DEPTH): PASS (exact_items_pass), 3000 steps in 18.7 s, final count 1,094,792. Time (/tmp/r3/time, all switches on): speedup_vs_frozen_reference = 1.3263 (total_s 230.821 vs ref 306.144; windows 63.078 / 85.104 / 82.639 s, each below the fused-Adam-only windows 76.7/97.8/93.3; peak mem 5.27 GiB). Breakdown so far: fused Adam 1.1432 → +pipeline/no-depth/separate-SHs → 1.3263. Full check (/tmp/r3/check, finished 06:10): status PASS, speedup 1.3275, tier N1 (candidate_s 230.611, windows 62.99/84.95/82.67). Statistical gate (seed 1, vs 3σ bounds): PSNR −0.13σ, SSIM +1.36σ, LPIPS −1.74σ, log_count +0.38σ — all within. This is the banked verdict for the R1–R3 configuration.
0.7 hRound 4 — remove the rasterizer's second per-iteration sync + render Python overhead
Start: 06:02 (validated 06:10–). Class: host syncs / launch gaps, N0 (bitwise-identical outputs). -rasterizer_impl.cu: the sample state is sized by the worst-case bucket count (num_rendered/32 + num_tiles, a valid upper bound because Σ ceil(inst/32) ≤ floor(Σ/32) + T), so thebucket_sumdevice-to-host read that blocked the host between the bucket scan and the render launch is gone; the forward returns the capacity. -backward.cu PerGaussianRenderCUDA: warps beyond the true bucket count stop on a device-side read of the inclusive-scan total (per_tile_bucket_offset[num_tiles-1]), so the oversized grid is safe. -gaussian_renderer/__init__.py(GS_OPT_PIPELINE): callrasterize_gaussians(...)directly instead of constructing aGaussianRasterizernn.Module (and re-validating its args) every iteration;empty_likefor the 2D means buffer (its values are never read — the rasterizer recomputes them and only the gradient of that input is consumed). Quick check (/tmp/r4/quick): PASS (exact_items_pass), 3000 steps in 18.5 s, final count 1,094,097 (counts vary run-to-run within the atomics noise floor; the change steps are the exact item and match). Time (/tmp/r4/time, all switches on): speedup_vs_frozen_reference = 1.3409 (total_s 228.305 vs ref 306.144; windows 63.132 / 83.908 / 81.265; R3 windows were 63.078 / 85.104 / 82.639 → the gain is in windows 2–3, the post-densification steady state; peak mem 5.27 GiB). End of round 4: ~06:19. Committed asopt: capacity-sized sample state, no bucket_sum sync; direct rasterize_gaussians call (N0).
1.1 hRound 5 — fused SH-gradient + Adam for f_dc/f_rest (GS_OPT_SH_ADAM)
Start: ~06:25. Class: memory traffic of the SH-coefficient gradient chain, tier N1. Profile of the R4 tree (/tmp/r5/profile, steps 7000–7019, P≈1.28M): GPU busy ≈ 7.8 ms/step vs 7.6 ms/step wall → the sync removals left essentially no idle; the remaining work is kernel time. Per step: PerGaussianRenderCUDA (bwd) 2.348 ms (30%), renderCUDA (fwd) 1.392 (18%), fused Adam 1.130 (14.5%), preprocess bwd 0.489 (6.3%), duplicateWithKeys 0.484 (6.2%), radix sort 0.225, zeros/fill ~0.30, SSIM 0.134, norm+div ~0.19. Change (GS_OPT_SH_ADAM, off by default): the SH gradient of a point isbasis_j(dir) * (clamp-masked per-point color gradient)— exactly whatBACKWARD::preprocessmaterializes intodL_ddc/dL_dsh(P×1×3 + P×15×3 ≈ 240 MB zeroed + written every step, then read back by Adam). The fused path: -backward.cu:computeColorFromSH/preprocessCUDAskip the dL_ddc/dL_dsh writes when fusing (thedL_dmeansdirection term still runs, soshsis still read). -rasterize_points.cu: in fused mode the two gradient tensors are 1-element stand-ins (their zeroing disappears); the backward returns the byte offset ofclampedin the geom buffer. -adam.cu(newADAM::shAdamUpdate): a 2-thread prep kernel advances the twostepcounters on device (as torch's fused kernel does) and precomputes the bias corrections; the main kernel walks P×(M·3+3) elements, recomputes the SH basis per element (same formulas/values ascomputeColorFromSH, gated by the active degree like the original'sdeg > 0/1/2nesting), and applies torch's fused-Adam update (m = m·β1 + g·(1−β1),v = v·β2 + g²·(1−β2),p −= (lr/bc1)·m/(√v/√bc2 + eps)). Elements beyond the active degree and invisible points get g = 0, exactly the zeros the reference materializes; every element is updated (dense, like the reference). -diff_gaussian_rasterization/__init__.py: in fused mode the backward stashes (dc, sh, means3D, dL_dcolors, geomBuffer, clamped offset, campos, degree) and returnsNoneas the dc/sh input grads (so.gradstays None);consume_sh_grad(f_dc, f_rest)hands the stash to the optimizer when it matches exactly those parameter tensors. -scene/gaussian_model.py(_ShFusedAdam, used when both Adam switches are on):step()runs the fused kernel for the f_dc/f_rest groups (torch-style lazy state init, float32 step/exp_avg/exp_avg_sq) and then callssuper().step()for the other four groups. When densification/pruning/opacity-reset has replaced a parameter tensor since the backward, the reference optimizer sees aNonegradient and skips (no update, no step increment); the identity check reproduces that exactly. Numerics: the gradient values are the same formulas on the same inputs (per-point, no atomics, same nesting) → bitwise-equal g up to compiler FMA choices; the Adam update replicates torch's fused kernel formula (powf-based bias corrections), so ulp-level differences are possible → N1, same class as the R1 foreach→fused switch that passed gate 3 with margin. Traffic saved ≈ 0.7 GB/step (dL_dsh/dL_ddc zero + write + Adam grad read), expected ~0.4 ms/step. Build: first attempt failed (missingstep_shparameter in the binding definition — fixed); second attempt ok, but the first quick check ERROR (illegal memory access at the first optimizer step). Debugging (repro at/tmp/r5/repro.pypassed; the judge's own entry reproduced it underCUDA_LAUNCH_BLOCKING=1; compute-sanitizer only surfaced 660 pre-existing benign shared-memory OOB writes inside stock fused-ssim): an env-gated dump of the exact kernel inputs showedclamped_off = -10203508— negative.GeometryState::fromChunktakeschar*&and *advances* the chunk pointer, so the offset I returned was measured from the chunk *end*. Fixed by capturing the buffer start beforefromChunk(rasterizer_impl.cu), rebuilt. Unit parity (/tmp/r5/parity.py): my kernel vs torch fused Adam given the identically materialized gradient, 3 steps, P=5000, all degrees: max |Δparam| = 2.98e-08 (≈1–3 ulps, rel ~7e-08), step counters identical — the update rule matches torch's up to compiler rounding, and the gradient values are the same formulas on the same inputs (one 5.6% outlier in the first run turned out to be a wrong SH_C3 sign *in my test script*, not in the kernel). Tier N1. Quick check 2 (after the offset fix,/tmp/r5/quick2): PASS (exact_items_pass), 3000 steps ok — the exact items (camera sequence, densification steps, lrs, dtypes) are unchanged by the fusion. Time (/tmp/r5/time): speedup_vs_frozen_reference = 1.2777 — a REGRESSION vs R4's 1.3409 (total_s 239.608 vs 228.305; windows 65.897 / 87.670 / 86.042 vs R4's 63.132 / 83.908 / 81.265 — every window slower, the gap growing with the point count: a per-element kernel cost). Why it lost (working theory, to be confirmed by the R6 profile): the fused kernel parallelizes over the P×(M·3+3) ≈ 61M elements, soj = e/3varies *within every warp* and the 15-wayif-chain that evaluates the SH basis diverges — each warp serially executes most of the 15 branch bodies (~15× the basis ALU). The traffic the fusion saved (~0.4 GB/step ≈ 0.2 ms) is smaller than that divergence penalty. A branch-free (predicated-arithmetic) basis, or a thread-per-point layout like the originalpreprocessCUDA(sequential over j ⇒ no divergence), should bring the kernel to ~0.5 ms vs the ~0.63 ms it replaced — a ceiling of only ~+0.1 ms/step, so the fix is deferred behind bigger fish. Decision:GS_OPT_SH_ADAMremoved from/submission/switches.json(back to the banked 6-switch config, time 1.3409); the code stays in the tree, off by default, committed for a later cheap retry with a branch-free basis.
1.5 hRound 6 — fix the fused SH-Adam kernel layout (GS_OPT_SH_ADAM, retry)
Start: ~06:58. Class: same hotspot as R5 (the fused SH-gradient + Adam kernel), now with the profile evidence. Profile (/tmp/r6/profile, R5 tree with all 7 switches, steps 7000–7019): per step — PerGaussianRenderCUDA (bwd) 2.478 ms (30.1%), renderCUDA (fwd) 1.487 (18.1%), shAdamCUDA 1.548 (17.9%), duplicateWithKeys 0.473, torch fused Adam on the other 4 groups 0.298, radix sort 0.249, preprocessCUDA fwd 0.143, div 0.142, preprocessCUDA bwd (SH-write skipped) 0.114, computeCov2D 0.050, SSIM fwd+bwd 0.134. Root cause of R5's regression confirmed: the kernel is element-parallel (one thread per coefficient), soj = e/3differs across the lanes of every warp and the 15-way SH-basisif-chain diverges — each warp serializes nearly all 15 branch bodies. A traffic-bound version of this kernel should cost ~0.5 ms, not 1.55. Bug found while reviewing (latent in R5, never enabled in a scored run):shAdamPrepCUDAused one learning rate for both groups, but upstream gives f_restfeature_lr/20(0.000125 vs f_dc's 0.0025) — a 20× error on the rest coefficients. The exact-items gate cannot see it (it checks the groups' Python-side lrs), and the R5 parity test used equal lrs for both groups, so it slipped through; gate 3 would have rejected it. Fixed: the binding takeslr_dcandlr_sh, the prep kernel writes each group's own bias-corrected step size, andgaussian_model.pypassesgdc['lr'], grst['lr']. The new parity test (/tmp/r6/parity2.py) uses the two different upstream lrs. Change:adam.cu— one thread per (point, color channel); the basis indexjbecomes a sequential loop variable identical across the warp (no divergence; withM=15known at compile time the loop unrolls and each basis formula constant-folds). Same gradient values (identical formulas on the same inputs), same torch fused-Adam update rule, same step-counter and densification-skip semantics. Grid 3P threads instead of P·(M·3+3); traffic unchanged (~1.5 GB/step) but now the intended ~0.5 ms. Tier N1. Plan: parity →check --quick(7-switch temp config/tmp/r6/sh_switches.json) →time; if it beats 1.3409, putGS_OPT_SH_ADAM=1back into/submission/switches.jsonand commit. Iteration 1 (thread per (point, channel), sequential warp-uniform j loop): build hiccup —rasterize_points.hstill had the old arity (undefined symbol at import; ext.cpp binds against the header), fixed. Parity (/tmp/r6/parity2.py, now with the two different upstream lrs): dc max|Δ| 2.98e-08 (~1-3 ulps), rest ≤3.7e-09, step counters identical, at every degree. Quick (/tmp/r6/quick): PASS. Time (/tmp/r6/time): 1.3266 — even with R4's 1.3409 (230.78 s; windows 62.893/84.956/82.931 vs R4's 63.132/83.908/81.265: the fused path costs slightly more at full size). Profile of iteration 1 (/tmp/r6/profile2):shAdamCUDA= 1.462 ms/step — barely better than the divergent 1.548. The divergence theory was wrong (ptxas had evidently predicated the old chain anyway); ptxas: 48 registers, 0 spills, so occupancy was fine. Bandwidth measurement (synthetic,flocked): a 460 MB device copy runs in 0.111 ms → this GPU sustains ~4.1 TB/s. The kernel moves only ~1.31 GB, i.e. it *should* take ~0.35 ms, not 1.46. Diagnosis: with one thread per (point, channel), every warp instruction touches eleven 12-byte clusters (2.75× sector amplification), and each warp's 15-iteration working set is ~5.9 KB × 3 arrays; at ~64 warps/SM that is ~380 KB — it thrashes the 256 KB L1, so the amplification goes all the way to DRAM (~3.6 GB effective). Iteration 2 (element-parallel, branch-free ternary basis, dc/rest split inside the kernel): build ok, parity 1–3 ulps (dc 2.98e-08, rest 3.7e-09, steps identical), but the in-training profile showedshAdamCUDA= 1.322 ms — still 2× its ~0.6 ms of traffic (1.38 GB at ~4.1 TB/s measured streaming). Bisect (standalone replica/tmp/r6/microbench2.cu= verbatim kernel + harness → 1.175–1.192 ms/call, reproduces the slowness; variants inmb_a/b/c.cu, 30 s compiles): | variant | ms/call | conclusion | |---|---|---| | replica (verbatim) | 1.18 | — | | A: dc branch deleted (if (true), sh-only) | 0.962 | the dc-vs-rest split costs ~0.22 ms (mixed warps execute both paths serially) | | B:intidx instead oflong long| 1.178 | 64-bit indexing is free | | C: constant dir/grad/prep (no input loads, no normalize) | 0.704 | per-element direction recompute + input loads cost ~0.48 ms | The per-element kernel recomputed the normalized view direction (3 amplifiedmeans3Dloads +sqrtf+ three IEEE divides) for every one of the ~61M coefficients instead of once per point — 48× redundant — and the amplified-input theory (2.75× sector amplification thrashing L1, iteration 1) was wrong: it is plain redundant ALU+loads. Iteration 3 (the fix): hoist the per-point invariants into a newshAdamDirCUDA(one thread per (point, channel), P×3 ≈ 3.8M threads) that computes the normalized direction and the clamp-masked color gradient once per point (bitwise the same values — a float32 store/load round-trip is lossless) into a workspace, and folds the DC update into the same kernel (its basis is the constant SH_C0, so it needs no direction; this also removes the dc/rest branch from the main kernel). The mainshAdamCUDAis now sh-only, P×(M·3) threads, thread index = the tensor's flat index (fully coalesced RMW), reading dir/gcol from the workspace. The workspace (P×6floats ≈ 30 MB) is allocated inside the C++ binding per call (caching allocator, stream-ordered free — Python-facing signature unchanged). ptxas: 26/22 registers, 0 spills. - Parity (/tmp/r6/parity2.py, extensioncf278ebc893ad3e3a9111d73): identical to iteration 2 — dc max|Δ| 2.98e-08, rest ≤3.7e-09, steps identical, every degree. Tier N1. - Quick check (/tmp/r6/quick3, 7 switches): PASS (exact_items_pass). - Time (/tmp/r6/time3): speedup_vs_frozen_reference = 1.4605 (total 209.623 s; windows 58.064 / 76.963 / 74.596 vs R4's 63.132 / 83.908 / 81.265 — every window down ~5–7 s ≈ 0.62 ms/step; peak allocated 5.68 GiB). Decision:GS_OPT_SH_ADAM=1added to/submission/switches.json(7 switches). End of round 6: ~07:48 (long round — the bisect needed it; logged in full above).
2.8 hRound 7 — shared-memory staging of the backward's per-pixel accumulators (GS_OPT_BWD_PIXCACHE)
Start: ~07:57. Class: the #1 hotspot,PerGaussianRenderCUDA2.48 ms/step (30%). The kernel launches one 32-thread block per bucket; every cascade iteration lane 0 issues ~10 scalar global loads (sampled_T,sampled_ar×3, pixel colors×3,final_T,n_contrib,dL_dpix×3) that feed the next iteration — a long latency chain where each 4-byte load drags a 32-byte sector (8× amplification), and the pixel-indexed ones are re-read by every bucket of the same tile. Change (GS_OPT_BWD_PIXCACHE, off by default):PerGaussianRenderCachedCUDA— a 256-thread block covers 8 consecutive buckets; each active warp stages its own per-pixel checkpoint (4 KB) coalesced into shared memory, and the block stages the tile's per-pixel data once for all 8 warps. Values identical (lossless float32 staging, same math) → N0 up to atomic-order rounding. Depth-enabled cameras keep the original kernel (also keeps static smem at 42 KB < the 48 KB limit; 62 registers).pixel_cachethreadedGaussianRenderSettings→ binding →Rasterizer::backward→BACKWARD::render;use_cachefalls back to the original global loads when a block's active warps span two tiles. First attempt — ERROR at step 10701 (/tmp/r7/time): illegal memory access after a densification step. A/B with the judge's entry (traj mode, 3100 steps, seed 0) showed the training had diverged: final count 10,432,987 vs the switch-off 945,796. Root cause: the per-warp staging wrotes_T[warp_id][threadIdx.x]— each warp staged only the 32 entries atthreadIdx.x ∈ [warp_id·32, warp_id·32+32)of its own 256-entry array; 224 of 256 checkpoint entries per bucket were uninitialized shared memory. Garbage T/ar → garbage gradients → runaway densification (the quick gate still PASSED — it checks the *steps* at which the count changes, not the counts; a lesson:check --quickcannot see gradient-value bugs). The garbage params then made the next forward's tile ids undefined → the fault. Fix: each warp's 32 lanes walke = lane, lane+32, …, lane+224(all 256 entries, coalesced). Value validation (the quick gate can't see it, and the backward's atomics make bitwise comparison impossible even run-to-run): three 500-step entry runs — off, off, on (seed 0, before densification). Pairwise max|Δparam|: off-off (pure atomics noise) 1.26e-01/7.3e-01/4.1/1.19/4.9e-01 (xyz/dc/opacity/scaling/rotation) vs off-on 1.22e-01/8.3e-01/4.0/1.16/4.6e-01 — same magnitude on every parameter → the cached kernel is value-equivalent within the noise floor (N0 in exact arithmetic; differences are atomic-order rounding, the same class as any rerun). - Quick check (/tmp/r7/quick2): PASS. - Time (/tmp/r7/time2): speedup_vs_frozen_reference = 1.5276 (total 200.404 s; windows 55.811 / 73.431 / 71.162 vs R6's 58.064 / 76.963 / 74.596 — every window down, ~0.31 ms/step; peak allocated 5.68 GiB). Decision:GS_OPT_BWD_PIXCACHE=1added to/submission/switches.json(8 switches). End of round 7: ~08:16.
3.1 hRound 8 — conservative exp pre-cull in both render kernels (GS_OPT_EXP_CULL) — NO GAIN, REVERTED
Start: ~08:28. Class: MUFU/exp throughput in the render kernels (forward renderCUDA 1.38 ms, backward cached 2.10 ms; ~355M walked (pixel, instance) pairs per step, each computingexp(power)before thealpha < 1/255blend test). Change: skip the exp whenpower < log(1/(255·opacity)) − 0.01— a conservative bound that *implies* the originalcontinue(1% margin covers float rounding), so the execution is bitwise-identical (N0); the threshold is onelogfper instance (staged in shared, forward) or per splat (registers, backward), not per pair. Plumbed asexp_cull(runtime bool) through settings → both bindings →Rasterizer::forward/backward→ both render kernels, behindGS_OPT_EXP_CULL. - A/B vs noise floor (500 steps, off/off/on, seed 0): off-on deltas at the run-to-run atomics level on every parameter (counts identical) — value-safe. - Quick check (/tmp/r8/quick): PASS. Time (/tmp/r8/time): 1.5207 vs the banked 1.5276 — a slight regression (201.316 s; windows 55.845 / 73.810 / 71.661 vs 55.385 / 73.532 / 71.486). - Why it lost: pairs withpower ≤ 0are mostly *inside* the Gaussian's effective ellipse, whereopacity·Gpasses the 1/255 threshold — thepower > 0test already did the spatial culling, so the extra compare + shared load per pair cost more than the few saved exps. The "85–95% culled" estimate was wrong. - Decision: reverted to the banked tree (git checkout, back at commit 9996eae; the switch was never in switches.json). Dead code that adds a per-pair compare even when off is a pure cost. End of round 8: ~08:40.
3.3 hRound 9 — occupancy of the cached backward kernel (launch_bounds) — NO GAIN, REVERTED
Start: ~08:42. The cached backward runs 62 registers → 4 blocks/SM = 32 warps (half the SM's warp slots), and its 287-iteration cascade is a serial per-warp latency chain — more resident warps should hide more latency. (The forward renderCUDA is already at full occupancy: 32 regs, 8.2 KB smem → 8 blocks/SM.) Change:__launch_bounds__(256, 5)onPerGaussianRenderCachedCUDA(scheduling only, no value change → N0). ptxas: 48 registers + 16 B stack (spills). Time (/tmp/r9/time, banked 8-switch config): 1.4713 — a big regression (208.076 s vs 200.404; +7.7 s). The spills land in the hot cascade and 5 × 42 KB of smem nearly fills the SM's 227 KB; the natural 62-register allocation at 4 blocks is the better operating point. Reverted (tree back at 9996eae). End of round 9: ~08:49.
3.5 hRound 10 — fused Adam for the four dense groups (GS_OPT_GROUP_ADAM)
Start: ~08:50. Class: the optimizer step (torch's fused Adam on xyz/opacity/scaling/rotation costs 0.29 ms/step over ~330 MB of traffic — 3.6× its bandwidth floor, one multi-tensor launch per parameter). Change (GS_OPT_GROUP_ADAM, off by default):groupAdamCUDA— one launch for all four groups, each *block* owning exactly one group (block-uniform selection; the R6 bisect showed a per-lane group split serializes). A 4-thread prep kernel advances the groups' step counters on device and writes each group's bias-corrected step size (per-group lrs: xyz's is scheduled, read from the group at step time). Same torch fused-Adam formula as the SH kernels.scene/gaussian_model.py:_ShFusedAdam.stepruns it when all four groups carry a gradient (the common case) and then returns withoutsuper().step()when the SH path also ran; on densification steps (replaced tensors → None grads) it falls back to torch's Adam, which skips them exactly like the reference. Mixed configs (group on, SH off) mask the four grads around the fallback so nothing is updated twice. - Parity (/tmp/r10/parity_grp.py, P=5000, 3 steps, per-group upstream lrs): xyz 2.98e-08, opacity 1.49e-07, scaling 5.96e-08, rotation 2.98e-07... rotation 2.98e-08 — all 1–3 ulps, step counters identical. Tier N1. - A/B (500 steps, off×3 vs on): counts identical; off-on deltas 1.1–3× the off-off noise floor (xyz 0.31 vs 0.13–0.17, opacity 9.3 vs 3.8–4.9) — the expected ulp-compounding signature of an N1 optimizer change, not a gross bug; gate 3 is the verdict. - Quick check (/tmp/r10/quick): PASS. - Time (/tmp/r10/time): speedup_vs_frozen_reference = 1.5534 (total 197.076 s vs 200.404; ~0.11 ms/step off the optimizer). Decision:GS_OPT_GROUP_ADAM=1added to/submission/switches.json(9 switches). End of round 10: ~09:00.
3.8 hRound 11 — occupancy of duplicateWithKeys (launch_bounds) — NO GAIN, REVERTED
Start: ~09:05.duplicateWithKeyscosts 0.48 ms/step at 53 registers → 4 blocks/SM = 32 warps (half the warp slots). One thread per point walks its tile rect serially (avg ~16 tiles/point → ~800 dependent instructions per thread), so more resident warps could hide that chain. Change:__launch_bounds__(256, 5)onduplicateWithKeys(scheduling only, values unchanged → N0). ptxas: 48 registers + 8 B stack. Time (/tmp/r11/time, banked 9-switch config): 1.5537 (197.041 s) — a wash vs the banked 1.5534/1.5592. The occupancy bump bought nothing: the kernel is bound by its serial per-thread instruction chain, not by resident-warp latency. Reverted (git checkout, tree back at commit 3b134bf). End of round 11: ~09:12. Conclusion: occupancy levers are exhausted on both render kernels and the binning kernel — the remaining win induplicateWithKeysis to shorten the serial chain itself (lane-parallel rect walk, R12).
4.1 hRound 12 — lane-parallel duplicateWithKeys (one warp per Gaussian)
Start: ~09:14. Class: the binning kernel — 0.48 ms/step, one thread per Gaussian walking its tile rect serially (avg ~16 tiles/point ≈ 800 dependent instructions per thread; R11 showed occupancy is not the bottleneck, the serial chain is). Change (kernel-internal, unconditional — bitwise-identical output): one warp per Gaussian (8 points per 256-thread block, grid(P+7)/8). Lanes walk the rect in row-major rounds of 32 (t = 32r + lane→y = rect_min.y + t/rect_w,x = rect_min.x + t − y·rect_w), each lane runs the same closed-formmax_contrib_power_rect_gaussian_floatper-tile opacity test, and a__ballot_sync/__popccompaction writes the passing tiles atoff + popc(mask & lanes-below-me)— which is exactly the original row-major emission order, so the unsorted key/value arrays are bitwise identical. Sentinel padding of the unused slots (theidentifyTileRangesguard skips tile-id0xFFFFFFFF, so sentinels are never walked) is lane-strided. 39 registers, 0 spills. ptxas note:__lanemask_ltis not declared in this CUDA — used(1u << lane) − 1instead. Validation (stronger than the usual noise-floor A/B, because this change claims bitwise identity): - *Bitwise forward test*: rendered the 6 cameras 0/1/3/9/40/120 from the 500-step model state with the old extension (twice) and the new one — images, radii and depths bitwise identical in all three pairings (the image is a deterministic function of the sorted binning output, so the unsorted arrays are proven identical)./tmp/r12/bw_*.pt. - *500-step A/B* (traj, seed 0, off/off/on): identical counts (136029) and SH degree; off-on max|Δparam| at or below the off-off atomics noise floor on every parameter (dc 0.17–0.34 vs 0.34; opacity 1.8–2.5 vs 2.5; scaling 0.97–1.06 vs 1.05; rotation 0.43–0.46 vs 0.46). Note: in this configuration xyz is bitwise identical even off-off, so it carries no signal. - Quick check (/tmp/r12/quick): PASS. - Time (/tmp/r12/time): speedup_vs_frozen_reference = 1.5834 (total 193.342 s vs 197.076; windows 52.484/71.669/69.189 vs 53.913/72.359/70.069 — every window down, ~0.12 ms/step). Decision: committed as7526a5c(unconditional kernel change, tier N0 — no switch needed: identical arrays, no Python surface). Full check (/tmp/r12/check): PASS, speedup 1.5773, tier N1 (PSNR −1.55σ, SSIM +0.93σ, LPIPS −1.24σ, log_count −2.15σ; windows 52.831/71.836/69.428). End of round 12: ~09:45.
4.7 hRound 13 — fast exp (__expf) in the render hot loops (GS_OPT_FAST_EXP)
Start: ~09:45 (profile) → implementation 09:50–10:05. Class: the #1/#2 hotspots — forward renderCUDA 1.40 ms + cached backward 2.10 ms; both compute the ACCURATE multi-instructionexpfper walked (pixel, instance) pair (~355M pairs/step each). The R13 profile (steps 7000–7019,/tmp/r13/profile) also showed the torch elementwise/launch soup (fills ×24, zeros ×13, div ×9, norm ×2 per regular step; DivBackward+NormBackward+ExpandBackward 232 µs — the sigmoid/exp/normalize activation chain) and that the aten::index/nonzero spike (~60 calls, 683 µs gathers) is the densify step only (amortized ~120 µs/step). Change (GS_OPT_FAST_EXP, off by default):__expf(power)(one MUFU.EX2, ≤2 ulp) instead ofexp(power)in the forward renderCUDA and both backward cascade kernels, selected as a compile-timeFAST_EXPtemplate variant from a new runtimeGaussianRenderSettings.fast_expflag (plumbed likepixel_cache: NamedTuple → binding → Rasterizer → FORWARD/BACKWARD::render explicit-branch dispatch — a runtime bool cannot feed a template argument). Tier N1 (render values change by ~2 ulp). Validation: - *Render sanity*: images from the 500-step state differ by mean 2.3e-8 (≈2 ulp; max up to 5.7e-4 where hundreds of splats composite), radii identical — no systematic shift (/tmp/r13/bw_fastexp.ptvs/tmp/r12/bw_old1.pt). - *500-step A/B* (same build, env off/off/on): identical counts (136029); off-on deltas at the off-off noise floor (dc 0.14–0.16 vs 0.134, opacity 1.89 vs 1.98, scaling 1.22–1.29 vs 1.33, rotation 4.9–5.1 vs 4.3). - Quick check (/tmp/r13/quick, switch ON): PASS — the count-change steps over the first 3,000 steps still match the reference. - Time (/tmp/r13/time): speedup_vs_frozen_reference = 1.6039 (total 190.878 s vs 193.342; ~0.08 ms/step off the render kernels — less than the optimistic estimate; the kernels are not purely exp-issue-bound). Decision: committed as36d01e2;GS_OPT_FAST_EXP=1added to/submission/switches.json(10 switches). Full check (/tmp/r13/check): PASS, speedup 1.6125, tier N1 (PSNR +0.45σ, SSIM +0.41σ, LPIPS −1.19σ, log_count −1.37σ — every endpoint comfortable, PSNR/count moved *toward* the reference vs R12's −1.55σ/−2.15σ; more evidence the single-seed σ readings wobble ±0.5σ regardless of the change). End of round 13: ~10:20.
5.0 hRound 14 — fused activations (sigmoid/exp/normalize) forward+backward (GS_OPT_ACT_FUSE)
Start: ~10:20. Class: the torch elementwise/launch soup around the rasterizer — the R13 profile showed the activation chain (sigmoid on _opacity, exp on _scaling, normalize on _rotation, forward + Sigmoid/Exp/Div/Norm/Expand backward) at ~350 µs/step across ~10 launches (DivBackward 113 µs + NormBackward 71 µs + ExpandBackward 48 µs + forward norm/div/sigmoid/exp). Change (GS_OPT_ACT_FUSE, off by default):cuda_rasterizer/act.cu—actForwardCUDA(one thread per point: sigmoid, exp×3, quaternion normalize with torch'sclamp_min(1e-12)) andactBackwardCUDA(composite of torch's backward chain, clamp branch included), bound as_C.act_forward/_C.act_backward, called from a_FusedActivationsautograd Function inrender()(saves op/sc/raw_ro; backward recomputes the norm). Tier N1 (backward ~1-2 ulp; forward bitwise except 1 ulp on the normalize). - Parity (/tmp/r14_parity.py, P=20000): forward op/sc bitwise, ro max|diff| 1.79e-07 (norm reduction order); backward rel 7.0e-08 (op), 0 (sc), 1.74e-07 (ro); tiny-norm (< eps) path exact. - 500-step A/B (same build, env off/off/on): identical counts (136029); off-on deltas at/below the off-off noise floor on every parameter. - Quick check (/tmp/r14/quick, switch ON): PASS. - Time (/tmp/r14/time): speedup_vs_frozen_reference = 1.6972 (total 180.377 s vs 190.878; ~0.35 ms/step — above the kernel-time estimate, the removed launches were also stalling the pipeline). Decision: committed as7da8173;GS_OPT_ACT_FUSE=1added to/submission/switches.json(11 switches). Full check (/tmp/r14/check): PASS, speedup 1.6942, tier N1 (PSNR −2.20σ, SSIM −0.51σ, LPIPS −2.12σ, log_count −0.79σ). A second seed-1 draw (check --n1,/tmp/r14/n1b) gave PSNR −1.26σ, SSIM −0.42σ, LPIPS −1.74σ, log_count −1.90σ — the identical-code wobble is ±0.5–0.9σ per endpoint, so the act-fuse systematic is roughly PSNR −1.7σ, LPIPS −1.9σ, count −1.3σ, leaving ~1.1–1.3σ of margin. EV keeps the change (+0.09 speedup vs a few % gate-3 risk); going forward prefer N0 changes and watch the margins in every full check. End of round 14: ~10:40.
5.6 hRound 15 — shAdamCUDA restructure (one thread per (point, basis)) — NO GAIN, REVERTED
Start: ~10:40 → end ~10:55. Class: the fused SH-Adam kernel (R6's, ~640 µs/step at ~30k Gaussians) — its per-thread serial loop over the 15 rest coefficients plus theif-chain SH basis derivative looked issue-bound. Change (no switch; kernel-internal): one thread per (point, basis-element) — thread (p, b) handles all 3 channels of basis b for point p, basis polynomial evaluated once per thread. Parity (/tmp/r15/parity_sh.py): not bitwise — sh 1.86e-9, m_sh 7.45e-9, v_sh 3.7e-9 max|diff| (FMA contraction in the basis polynomials; ~1 ulp), everything else 0. Results: 500-step A/B deltas at the off-off noise floor (but that window only exercises the sh path at degree 0); 1600-step A/B: count-change step lists identical 11/11 while counts diverge even off-off (501812 vs 503105 — the atomics noise flips marginal densify decisions). Quick check PASS. Time (/tmp/r15/time): 1.6527 = −0.045 REGRESSION vs 1.6972. Diagnosis: the kernel is memory/L1-bound, not issue-bound — ~1.05 GB of traffic per call (a ~350 µs floor on this GPU) vs 640 µs measured. The (p, b) layout made the gathers stride-3 (thread b reads channel-interleaved floats), quadrupling L1 transactions; the instruction savings were dominated. A precompute-the-basis variant would ADD traffic. Reverted (git checkoutofadam.cu; tree back at 7da8173, whose sources hash to the cached R14 build — no rebuild needed).
5.6 hRound 16 — densify-step host-sync reduction (hoisted mask indices)
Start: ~10:55 → end ~11:10. Class: the densify step's GPU-idle gaps — the R15 trace showed a 16.43 ms span (12.04 ms GPU busy + ~4.4 ms gaps) every 100 steps in the densify window, with ~60aten::nonzerohost-syncing calls per event: every boolean-index gathert[mask]recomputesnonzero()and re-syncs for the output size (16 gathers indensify_and_clone/densify_and_split, 18 in_prune_optimizer, 4 inprune_points, thesum()readback for the prune_filter pad size, and the bool-maskgrads[isnan] = 0assignment). Change (commita4cfc6e, no switch — value-identical code path, tier N0): each event computes its index once (mask.nonzero().squeeze(-1)) and every gather usest[idx]— same values, same ascending order;prune_pointspasses its index into_prune_optimizer; the pad size comes fromselected_idx.shape[0](host int, no readback);grads[grads.isnan()] = 0.0→torch.where(torch.isnan(grads), 0.0, grads)(a bool-maskindex_put_also converts via nonzero → sync). ~60 syncs → ~5 per event. Bug caught by the A/B before any judge run:nonzero()on a 1-D mask returns shape (K, 1) — indexing with it added a trailing dim ((K,1,3)params) and the run died at the first densify ("Tensors must have same number of dimensions: got 2 and 3", step 600)..squeeze(-1)fixes it; the 800-step A/B then completed. Validation: 800-step A/B (on/off, same build): count-change steps identical (600/700/800 both); count values differ by 1 at the first event and drift afterwards — exactly the established off-off atomics pattern. Quick check (/tmp/r16/quick): PASS. Time (/tmp/r16/time): 1.7009 (windows 48.86/66.69/64.43 vs R14's 49.12/66.79/64.47 — the densify-free window 3 moved −0.04 s = the noise floor, so the real gain is the −0.26 s in w1 + −0.10 s in w2 ≈ 2.4 ms per densify event, ~60% of the trace's gap estimate; the remaining stall is the ~5 surviving syncs plus the gather/cat/nn.Parameter-rewiring soup between them). Decision: keep (N0, no gate-3 exposure, small but consistent in the affected windows). No full check yet — +0.004 alone doesn't justify holding the GPU ~15 min; bank more N0 gains and run one full check over the accumulated set.
6.2 hRound 17 — densify gather cost: index_select (no gain) + warp-per-row gather_rows kernel + a REAL BUG FIX (frozen xyz)
Start: ~11:05 → end ~11:45 (overran the 30-min pace; the bug discovery forced the full check). Class: the densify event's gather kernels — the R17 profile (/tmp/r17/profile, steps 7000–7019, one densify event) putaten::indexat 9.37 ms of CUDA time per event (60 calls;vectorized_gather_kernel<16>9 × 679 µs on the 45-wide f_rest rows ≈ 90 GB/s). Attempt 1 —torch.index_select: no gain — profile2 (/tmp/r17/profile2) shows index_select dispatches to the *same* per-element gather kernel (8.41 ms, vectorized_gather 6.11 nested). Time 1.6951 ≈ noise. Attempt 2 — customgather_rows(cuda_rasterizer/gather.cu, commit925bf15): one warp per output row, lane-strided coalesced 4-byte transfers; bitwise parity verified on GPU over all site shapes (N,)/(N,1)/(N,3)/(N,4)/(N,1,3)/(N,15,3), keep-fractions 0–1, permutations, empty. int32tmp_radiiand non-contiguous tensors fall back to torch's path (dispatch order matters: tmp_radii is *contiguous* int32 and reached the C++ TORCH_CHECK before the fix — caught by the quick check's record run). The bug the A/B surfaced (commita9c8db1):_gather_rows(self._xyz, ...)reported_xyzas column-major — shape (N,3), strides (1,N). Root cause:fetchPlybuilds points asnp.vstack([...]).T(F-order) andtorch.tensorpreserves those strides, so_xyzwas transposed from creation until the first densify rebuilt it — *in the reference tree too*. Torch ops are stride-aware, but my fused group-Adam wrapper writes throughparam.contiguous().data_ptr(): for a transposed parameter that is a discarded copy, so xyz (and its exp_avg/exp_avg_sq) was frozen for steps 1..599 while the reference's torch Adam trains it from step 1. Probe: max|xyz(300)−xyz(init)| = 0.0 before the fix, 0.20 after. Present since R10; invisible to gate 2 (it compares no parameter values) — and it explains the systematic gate-3 drift I had been attributing to act-fuse. Fix:.contiguous()at creation increate_from_pcd(+load_ply); values identical everywhere (every consumer already calls.contiguous()internally), layout only. Validation: 300-step probe (xyz trains), quick check PASS (/tmp/r17/quick3), full check PASS, speedup 1.6946, tier N1 (/tmp/r17/check): PSNR +0.96σ, SSIM +1.25σ, LPIPS −2.45σ, log_count +0.03σ — versus R14's −2.20/−0.51/−2.12/−0.79. The frozen-xyz bug was the systematic source; margins are now healthy (LPIPS sits ~2σ *below* reference, i.e. better, and inside the gate). Non-gain noted: windows [49.008, 66.854, 64.796] ≈ R16's — the densify event is *latency*-bound (host-launched kernel chain, ~5 surviving syncs), so making its kernels faster does not shrink the span. The remaining densify lever is fewer launches/syncs/allocations, not faster kernels; treat this path as near-exhausted (~144 events × ~14 ms ≈ 2.0 s ≈ 1.1% of the run, mostly structural). Decision: both commits kept (a9c8db1 fix — mandatory; 925bf15 gather — bitwise N0, no risk, gates pass). End of round 17: ~11:45.
6.4 hRound 18 — the per-step zero-fill/launch soup: backward arena + empty for fully-written outputs
Start: ~11:45 → end ~11:57. Class: per-step allocation/zeroing overhead — the R17 trace attribution showed each regular step pays ~25fill_launches: 13torch::zerosinside the rasterizer backward's 228 µs CPU span (~122 µs of CPU launch triples), 5 image-sizedzeros_likein fused-ssim (4 forward + 1 backward), 3torch::fullin the rasterizer forward, plus the clamp-backward's scalar machinery. Change (commit071c81d, N0, no switch — value-identical): -rasterize_points.cubackward: the eleven P-sized gradient buffers (dL_dmeans3D/means2D/colors/invdepths/conic/opacity/cov3D/dc/sh/scales/rotations) become 16-byte-aligned contiguous views into ONE arena allocation with a single zero fill. First try crashed with *misaligned address*:dL_dconicanddL_drotationsare written throughfloat4casts in backward.cu (lines 405/489) and my offsets were unaligned (P is odd) — fixed by padding every view offset to 4 floats. -radii:torch::full({P},0)→torch::empty(preprocessCUDA writesradii[idx]for every point before any early return). -ssim.cu: the fivezeros_like(img1)→empty_like(both kernels write every element; the only guard is the pixel bounds check). Validation: 300-step loss parity vs the pre-change run at the atomics noise floor (1e-6 in the 6th-7th digit, one mark exactly equal), counts identical; quick check PASS (/tmp/r18/quick); time 1.7125 (/tmp/r18/time; windows 48.15/66.42/64.21 — all three windows improved including the densify-free window 3, confirming a per-step saving; total 178.78 s vs 179.99). Decision: committed. No full check yet — N0-only delta over the R17 full check; bank it and check once more after the next round. End of round 18: ~11:57.
6.8 hRound 19 — input-layout soup: contiguous gt image + view-matrix copy at camera load
Start: ~12:00 → end ~12:18. Class: per-step device clones hiding behind non-contiguous inputs — the R19 profile/trace (/tmp/r19/profile, steps 7000+20) counted 4aten::contiguous→cloneevents per step: the SSIM forward cloning the ground-truth image (position 93), the rasterizer forward cloningworld_view_transform(164), and two in the rasterizer backward (278/284 — one is the viewmatrix again, the backward re-clones its copy). Direct loader replication confirmed the layout facts:PILtoTorchreturns a permuted (H,W,C)-physical view,clamp(0.0,1.0)and.to(cuda)preserve those strides (HWC physical, strides (1, 2937, 3)), andworld_view_transform/projection_matrixare transposed views (strides (1,4)). Change (commit6ab5045, N0, no switch — layout only): -Camera.__init__:original_image = gt_image.clamp(0.0,1.0).contiguous().to(data_device)— the clamp is elementwise (value-exact regardless of layout) and.contiguous()is a bitwise relayout, so the stored image is bit-identical; the SSIM kernels now read it directly instead of cloning it twice per step. - The stored matrices stay EXACTLY as before (a contiguousworld_view_transformwould feed the bmm/inverse a different layout and could shiftfull_proj_transform/camera_centerby ulps through cublas kernel selection — N1 risk for zero benefit); insteadself.raster_viewmatrix = world_view_transform.contiguous()(64 floats, once per camera) and the renderer passes that, with agetattr(..., world_view_transform)fallback for non-Camera callers. Validation: 300-step probe loss marks 0.2505075/0.1984794/0.1514734 vs R18's 0.2505064/0.1984812/0.1514813 (atomics noise floor), count 136029, dtypes unchanged; quick check PASS (/tmp/r19/quick); time 1.724 (/tmp/r19/time; windows 47.75/66.05/63.78 vs 48.15/66.42/64.21 — all three windows improved ≈ −0.4 s each ≈ 40 µs/step, consistent with removing 3–4 clone launches and their strided reads; total 177.58 s). Decision: committed. N0-only delta over the R17 full check (R18 arena + R19 layout). End of round 19: ~12:18.
7.4 hRound 20 — fused per-step densification bookkeeping (stats kernel)
Start: ~12:20 → end ~13:05 (two rebuilds: the first norm formula was wrong at the ulp level). Class: the per-step aten soup that is NOT inside the big kernels — the R19 profile's per-op GPU table showed the GPU is saturated (6.23 ms GPU / 5.96 ms wall), so every GPU-µs removed is wall-µs. Between backward and the optimizer step, every iteration of the densification window paid ~7 kernels:radii > 0(gt), an int32→float32 promotion copy + out-of-placetorch.maximumfor max_radii2D, a strided (P,2)torch.norm, a bool→float mask product, and twoadd_accumulations ≈ 60 µs/step at P≈1.7M. Change (commitc5460da, switchGS_OPT_STATS_FUSE=1, tier N0): newcuda_rasterizer/stats.cu—densify_stats(radii, grad, max_radii2D, accum, denom) -> vis, one thread per point, in-place state updates; wired throughGaussianModel.densify_stats_step(layout-guarded, falls back to the unfused chain), train.py (fused call replaces the maximum + add_densification_stats pair; falls back cleanly if the kernel call declines) and the renderer (skips itsradii > 0under the switch). The ulp hunt (why two rebuilds): the standalone bitwise parity test — run BEFORE any training — caught the norm's sum-of-squares at 1 ulp off on ~3% of points.torch.norm's two-element reduction issqrt(round(gx*gx) + round(gy*gy))with NO FMA contraction (verified against numpy float32 emulation, 0/12345 mismatches); nvcc's default contraction of a plaingx*gx + gy*gy— in either operand order — differs. Fixed with__fmul_rn/__fadd_rnintrinsics that pin the exact form. Also documented: the only inputs where fused ≠ unfused are non-finite screen-space gradients at INVISIBLE points (NaN·0 = NaN in the masked reference) — unreachable, since the rasterizer backward zero-fills them and only atomically writes visible Gaussians. Validation: standalone parity PASS on all reachable states (P=1..1M, visible fractions 0–1, NaN/Inf at visible points, NaN in max_radii2D; vis/max_radii/accum/denom all bitwise). 800-step A/B on/off/off... plus a same-config second ON run: cross-config loss deltas equal the same-config spread (atomics noise floor), count-change steps identical 600/700/800, cameras identical. Quick check PASS (/tmp/r20/quick). Time 1.727 (/tmp/r20/time; windows 47.71/65.77/63.79 vs R19's 47.75/66.05/63.78 — the −0.29 s sits in window 2, which covers the densification window; w1/w3 at the noise floor). Honest accounting: +0.003 total, not the +0.013 estimated from the profile — the profile point (step 7000, P=1.7M) over-represents the stats window, whose average P is ~1M (counts: 176k at step 800, 1.7M at 7000). Kept: bitwise N0, gates pass, fewer launches also relieve the CPU side. Decision: kept. End of round 20: ~13:05. Full check after rounds 18-20 (/tmp/r20/check, tree =c5460da, all-N0 delta over the R17 check): PASS, speedup 1.7354, tier N1 — gates 0-2 clean, timing windows 47.23/65.52/63.65. Gate-3 margins on seed 1: PSNR −0.09σ, SSIM −0.60σ, LPIPS −0.24σ, log_count +0.25σ (max |0.60σ|) — far healthier than R17's reading (PSNR +0.96, SSIM +1.25, LPIPS −2.45, count +0.03): the −2.45σ LPIPS was run-to-run wobble around a near-zero true bias, not a systematic shift. The final evaluation's single-seed roll now has a wide margin.
7.8 hRound 21 — backward arena split (no gain, reverted)
Start: ~13:06 → end ~13:22. Class: continuation of the zero-fill soup — the backward's single zero-filled arena costs a 41.5 µs fill per step (the largest remaining fill); only the atomically-accumulated buffers actually need their zeros. Investigation (all facts verified in source before editing):dL_dscales/dL_drotationsare written only by computeCov3D (via preprocessCUDA, after its radii guard),dL_dcov3Donly by computeCov2DCUDA (full 6-entry write, existing zero branch at line 297), anddL_dmeans3Dby computeCov2DCUDA (direct assign — it runs FIRST, before preprocessCUDA's two+=contributions). So those four (16 of the arena's 30 floats/point) can be allocated uninitialized if the two kernels store explicit zeros on their culled-point early-return paths. Critical catch:dL_dinvdepthsis NOT dead — computeCov2DCUDA *reads* it unconditionally (dL_dinvdepth[idx] * tz2in the means gradient); with COMPUTE_DEPTH=false it is never written, so its zeros are load-bearing and it must stay in the zeroed arena (alsodL_ddc/dL_dshwhen the SH gradients are not fused — preprocessCUDA writes them only for visible points). Change (uncommitted, N0 by construction): split the arena — zeroed 12/pt (means2D, colors, conic, opacity, invdepths, dc/sh stand-ins) + uninitialized 16/pt (means3D, cov3D, scales, rotations, 16B-aligned views); guard-path zero stores in computeCov2DCUDA (9 floats) and preprocessCUDA (7 floats). Results: 300-step parity at the atomics noise floor (0.2505082/0.1984816/0.1514915, count unchanged), quick check PASS (/tmp/r21/quick), time 1.7354 = exactly the R20 check's number (/tmp/r21/time; windows 47.26/65.54/63.61 vs 47.23/65.52/63.65 — every window within ±0.04 s = noise). No gain. Diagnosis: at P≈1.7M most Gaussians are culled in any given view, so the explicit culled-point zero stores (~16/pt × culled fraction) eat most of the 16/pt of removed fill traffic; the net (~5-10/pt ≈ 9-17 µs/step) is below a single timing run's noise floor. The fill itself is also only ~40 µs of a 6.2 ms step. Decision: reverted (git checkoutof backward.cu and rasterize_points.cu; tree back atc5460da, the full-checked state). The zero-fill soup is now exhausted: the remaining fill is load-bearing (atomics + the invdepths read). End of round 21: ~13:22.