Claude Sonnet 5.5 4.44×

Harness: Claude Code · switches: GS_OPT_CUSTOM_ADAM=1 GS_OPT_FASTLOOP=1 GS_OPT_RAWRENDER=1 GS_OPT_FASTLOSS=1 GS_OPT_KEEP_CACHE=1 GS_OPT_FASTDENSIFY=1 GS_OPT_VISADAM=1 GS_OPT_LAZYSH=1

Judged
4.44×
Tier
N1
Self-best
4.06×
Rounds
20
Hours
10.0
API cost
$56.5
Input tokens · 99.3% cached
325 M
Output tokens
1.5 M
Requests
650
Smaller tiles + rewritten binning
✓ 8×8 + exact tile enumeration via two-pass warp cooperation (no padding entries in the sort); 8×8 alone nets 0, with enumeration 1.25 → 1.52
Bypass autograd, activations folded into raster kernel
partial: GS_OPT_RAWRENDER=1 has rasterization take raw params, activations and their backward folded into the kernel (1.52 → 1.86); training loop and raster calls still in PyTorch
Lazy Adam (update only visible Gaussians)
✓ SH only (48 of 59 floats) GS_OPT_LAZYSH=1, 3.31 → 3.51; position / opacity / scale / rotation stay dense
Fused loss / SSIM
✓ GS_OPT_FASTLOSS=1 loss grad without building an autograd graph (2.56 → 2.62); rewritten SSIM kernel (forward 100 → 44 µs, 3.14 → 3.20)
Other main levers
custom single-launch Adam (2.20 → 2.56), SH update on a low-priority stream concurrent with the binning chain (3.81 → 4.06), sort by depth then stable sort by tile (4.06 → 4.24), backward passes only scalar color accumulators (→ 4.35), single-instruction exp / rcp (→ 4.51), fixed-point atomic adds
Raw work log20 sections
0.2 hRound 1 — profile of the unchanged reference (start 13:27, GPU lock 13:27:58)
Judge profile of the unchanged tree, steps 7000..7019 (/tmp/r1/profile) and 25000..25019 (/tmp/r1/profile25k).
GPU time per iteration: 9.65 ms at step 7000 (N ~ 1.9M Gaussians), 10.5 ms at step 25000. Profile span has ~18% GPU idle
(launch gaps + host syncs; aten::nonzero blocks the CPU, then loss.item(), then two cudaMemcpy D2H in the rasterizer forward).

Share of GPU time (step 7000 / step 25000):
- PerGaussianRenderCUDA (rasterizer backward render): 27% / 29%  (2.6 / 3.06 ms)
- Optimizer.step#Adam (torch foreach implementation: ~7 foreach kernels per group, ~18 passes over 59 floats/Gaussian): 26% / 29% (2.6 / 3.06 ms)
- renderCUDA (forward render): 16.6% / 16.8% (1.6 / 1.76 ms)
- forward preprocessCUDA 5% / 6% (0.49 / 0.62 ms), duplicateWithKeys 5% / 5.3% (0.49 / 0.56 ms), radix sort ~2.5%
- aten::index/gather/scatter of the densification statistics (visibility_filter indexing): ~5%
- backward preprocessCUDA + computeCov2D 1.5-2%; ~100 small elementwise kernels (exp/sigmoid/normalize fwd+bwd, L1, SSIM glue) ~5% plus their launch gaps
Hotspot kinds: Adam = "missed existing fast path" (torch fused=True); idle gaps/syncs = host problem; the rasterizer kernels = new opportunity.
0.8 hRound 2 — fused Adam + sync-free loop (13:35–13:48, GPU lock for 2 timing runs)
Profile finding used: Adam foreach = 26-29% of GPU time (kind 1: missed fast path), host syncs = ~18% idle (kind 2).
Changes (each behind a switch, defaults off): GS_OPT_FUSED_ADAM (torch fused Adam, N1), GS_OPT_FASTLOOP (no
.nonzero()/.item() per iteration, mask-form densification statistics, skip all-ones alpha mask, N0/N1).
time results (frozen reference 308.311 s):
- fused Adam only: 263.884 s, speedup_vs_frozen_reference 1.1684 (windows 74.2/96.5/93.2 s) — /tmp/r1/time_fused
- fused Adam + FASTLOOP: 246.029 s, 1.2531 (67.8/90.6/87.7 s) — /tmp/r1/time_fused_fast
- check --quick PASS (/tmp/r1/quick_fused_fast). Profile after: GPU 92% busy (was 82%), Adam 1.19 ms/iter (was 2.4).
0.8 hRound 3 — 8x8 tiles + cooperative exact tile enumeration (13:50–14:17)
Profile: PerGaussianRenderCUDA 2.62 ms + renderCUDA 1.60 ms dominate. Experiment: BLOCK_X=BLOCK_Y=8 only (/tmp/r1/prof_b8):
backward render 1.60 ms, forward render 1.05 ms, but duplicateWithKeys 1.70 ms (thread-per-Gaussian loop over 4x more tiles) and
sort 0.50 ms (padding entries) -> net 0. Fix: (a) candidate set defined at the reference's 16x16 granularity + the
reference's own per-tile test (so the tile set matches the reference), refined by a conservative exact min-of-quadratic
test per 8x8 tile; (b) two warp-cooperative kernels (tileEnumCUDA<false> exact count, tileEnumCUDA<true> write) so
the sort has no padding. Numerics probe (/tmp/dev/tools/numcheck.py on a step-3000 snapshot of the reference): rendered
images bitwise identical to the reference on all 6 test cameras; gradients differ at 1e-3 relL2 — the same size as the
reference's own run-to-run difference (two runs of the reference differ by up to 6e-3 relL2 in vp_grad). radii identical.
time (switches fused_adam+fastloop): 203.425 s, speedup 1.5156 (57.1/74.6/71.7 s) — /tmp/r1/time_t8v2. check --quick PASS.
Profile at 7000 (/tmp/r1/prof_t8v2): GPU busy 7.01 ms/iter (8.35 before): bwd render 1.63, Adam 1.19, fwd render 1.05,
preprocess 0.49, tileEnum 0.71 (count 0.34 + write 0.37), sort 0.21. Only 32% of Gaussians are visible per view (P=1.71M, 542K visible).
No Nsight Compute counters (ERR_NVGPUCTRPERM), so kernel work is guided by the judge's profile + a snapshot profiler.
Committed on opt: 53199f0 (switches), 7b5f0e1 (CUDA tiles). switches.json = fused_adam + fastloop.
1.3 hRound 4 — raw-parameter rasterizer path, tile-enum optimisations, smem-staged backward render (14:27–)
Profile (snapshot profiler /tmp/dev/tools/kprof.py on the step-7000 snapshot, rasterizer+L1 only): the PyTorch elementwise
kernels around the rasterizer (exp/sigmoid/normalize fwd+bwd, zero fills, index ops) were ~0.5 ms/iter + ~100 launches.
Change 1 (GS_OPT_RAWRENDER): rasterize_gaussians_raw / _backward_raw take the raw parameters; preprocess applies
exp/sigmoid/normalize, one merged backward-preprocess kernel does cov2D+cov3D+SH+projection+activation backward with dense
raw-gradient outputs (SH gradients staged through smem); render kernels templated on DEPTH (depth-free instances);
densify_stats kernel. kprof: sum of rasterizer kernels 4.90 -> 3.67 ms/iter. time (fused_adam+fastloop+rawrender, /tmp/r1/time_raw):
165.957 s, speedup 1.8578 (46.7/61.1/58.2 s); quick PASS. Banked as b99a466.
Full check #2 (/tmp/check-2): PASS, speedup 1.8673, N1 (opt @ b99a466, switches fused_adam+fastloop+rawrender).
Change 2: tile enumeration: candidate rectangle tightened to the AABB of the ellipse {f <= thr} (bitwise-identical images),
reciprocals hoisted: tileEnum 0.68 -> 0.46 ms/iter. Change 3: conservative exp early-out in render kernels (no measurable gain:
kernels are latency-bound, not issue-bound). Change 4: backward render kernel stages the tile's per-pixel data in shared memory
(only T and ar travel on the shuffle chain): 1.463 -> 1.069 ms/iter (gradients identical up to atomics noise).
Tile-shape experiment (kprof): 16x8 worse (3.89 vs 3.43 ms), 8x4 equal (3.46) -> keep 8x8.
1.8 hRound 5 — latency fixes in the render kernels + custom Adam (14:40–15:10)
kprof/SASS analysis: the backward render loop is ~58 instructions per step for non-contributing pairs and the kernel is
issue-bound (95 steps x 160K buckets x 58 = ~0.9G warp-instr = 1.0 ms), i.e. the structural cost; the forward kernel was
stalled on a per-contribution global gather of the Gaussian color (latency): staging colors in smem with the batch
(+ register prefetch of the next batch of 128): 0.77 -> 0.52 ms. Backward: per-bucket smem staging of the tile's pixel data:
1.46 -> 1.07 ms. Atomics in the backward are NOT a bottleneck (removing them changed nothing). Tile shapes 16x8 / 8x4 tried
(16x8 worse; 8x4 equal): keep 8x8. Custom Adam (GS_OPT_CUSTOM_ADAM, one launch over all groups, float4): first attempt failed
('tensors must be contiguous': the initial xyz is column-major from the PLY reader) -> made contiguous.
time results (/tmp/r1/time_k2 and time_k2c2): torch fused Adam 139.836 s (2.2048; windows 42.4/49.6/47.8);
custom Adam 120.638 s (2.5557; windows 34.6/44.1/41.9). Quick check PASS. Banked as 0c29c07
(switches: custom_adam + fastloop + rawrender). Full check #3 running (/tmp/check-3).
2.3 hRound 6 — loss fusion, backward loop, densification (15:10–15:50)
Judge profile at step 25000 of the then-best config (/tmp/r1/prof_fl2): GPU busy 96.5% of the span (3.89 of 4.03 ms/iter), 43
kernels/iter (was 220): bwd render 0.98 ms (25%), Adam 0.825 ms (21%; = 4 TB/s, at the memory floor), fwd render 0.58, tileEnum
0.57 (count 0.26 + write 0.31), bwd preprocess 0.28, sort 0.26, fwd preprocess 0.16, fused SSIM 0.13.
Observation: in the backward render kernel ~6 of 32 lanes contribute per step, so the warp executes the contributing path
on ~90% of the steps: instruction count of the contributing path (not the early-out path) is what matters.
Changes: (1) GS_OPT_FASTLOSS: loss gradient without autograd graph (clamp + fused-SSIM fwd/bwd + lossCombineCUDA);
bitwise-equal gradient (first attempt mismatched because the ground-truth image tensor has channels-last strides -> made contiguous
once). time 117.69 s (2.620). (2) backward render loop rewrite + __expf + one reciprocal: time 112.12 s (2.750).
(3) Densification iteration cost: judge profile around iteration 14900 (/tmp/r1/prof_dens): 40.9 ms vs 4 ms normal; 13.5 ms is a
CPU/allocator gap (torch.cuda.empty_cache() -> cudaMalloc churn), the rest 4 concatenate/mask passes over params+states with ~60
syncs. GS_OPT_KEEP_CACHE=1 (no empty_cache): 28.9 ms; GS_OPT_FASTDENSIFY=1 (one gather per tensor, verified bit-identical to the
reference procedure incl. RNG stream with /tmp/dev/tools/densify_test.py): 16.0 ms. time 109.12 s (2.825).
Banked as 2482fed; quick PASS; full check #4 running (/tmp/check-4).
2.7 hRound 7 — tile pass reuse, one fewer host sync, visibility-aware gradients (15:50–16:10)
Judge profile at 25000 (/tmp/r1/prof_best2, tree 2482fed): busy 3.76 of 3.89 ms/iter (96.7%), 41 kernels/iter:
bwd render 0.99, Adam 0.83 (4 TB/s: floor), fwd render 0.58, bwd preprocess 0.28, tileCount 0.27, sort 0.25, tileWrite 0.17,
fwd preprocess 0.16, SSIM 0.13. Full check #4 on 2482fed: PASS 2.819 (dPSNR +0.89 sigma, dSSIM +1.78, dLPIPS -1.02, dlogN +0.67).
Changes: (1) tile pass 1 stores the keep mask + packed rectangle for Gaussians with <=32 candidates, pass 2 expands pairs from the mask
(no tests): tile kernels 0.457 -> 0.362 ms (images bitwise identical); sample state sized with R/32+tiles (no second D2H sync); no
pixel-color D2D copy. time 105.81 s (2.914), banked ec462e8. (2) GS_OPT_VISADAM: the raw backward writes only visible rows, the
custom Adam reads radii and skips invisible gradient rows (verified bitwise vs dense Adam with zero rows and NaN-filled invisible rows);
bwd preprocess 0.233 -> 0.211 ms; time 103.97 s (2.965), banked e443f00, quick PASS.
Extra-seed bias estimate (not a gate): /tmp/dev/tools/seedcheck.py runs the judge's trajectory+evaluation code on reference seeds 2-4.
3.1 hRound 8 — extra-seed bias estimate, CPU overhead, packed shared-memory records (16:10–)
Extra-seed estimate (judge trajectory+evaluation code, seeds 2,3,4, tree e443f00; /tmp/seedcheck1.log): paired d in sigma units
PSNR -0.21/+0.05/+0.01, SSIM -1.03/+0.26/+0.36, LPIPS +0.28/-0.30/+0.66, log N 0.0/-0.14/+0.58; means -0.05/-0.14/+0.21/+0.15 sigma:
no systematic bias. Seed-1 d over four checks: PSNR -1.2..+0.9, SSIM -1.2..+1.8, LPIPS -1.8..-0.2 (the frozen seed-1 reference has
LPIPS +2.07 sigma above the 8-seed mean), logN +0.1..+1.2 -> estimated fail risk of gate 3 ~2-3%.
Early-phase profile (/tmp/r1/prof_early, step 3000): steady iterations are 1.8-2.9 ms with the GPU ~96% busy; the apparent 16% idle came
from one densification iteration in the window. CPU overhead cuts anyway (python Adam step without CPU-tensor ops, no-op exposure step
skipped, contiguous camera matrices cached, clamped image written by the forward kernel): time 103.12 s (2.990); banked 32f7d1d.
Full check #5 (/tmp/check-5): PASS, speedup 2.9814 (N1) for 32f7d1d (dPSNR -1.17, dSSIM -0.51, dLPIPS -1.36, dlogN +0.57 sigma).
Feasibility probe /tmp/dev/tools/overlap_test.py: Adam(SH) on a side stream does NOT overlap with the forward (2.127 vs 2.123 ms) -> dropped.
Packed shared-memory records (forward: {conic,pmin} + {rgb,opacity} float4s, 2 LDS on the non-contributing path; backward: per-pixel
{dL0,dL1,dL2,-T_final*bg.dL}): kprof bwd 0.910 -> 0.838, fwd 0.508 -> 0.438 ms at step 7000, images bitwise identical.
3.3 hRound 9 — packed records, SH-degree probe (16:35–16:50)
Packed smem records: time 99.03 s (3.113) for the forward/backward first step, 98.21 s (3.139) with the backward {init, pixel, dL} float4s;
banked f115bf2. 2-pixel-per-thread forward tried: slower (0.63 vs 0.44 ms), reverted. kprof with SH degree 0 vs 3 at step 7000: fwd
preprocess 0.083 vs 0.135 ms, bwd preprocess 0.184 vs 0.231 ms -> SH handling is only ~0.05 ms in each, not worth restructuring.
tileCount is issue-bound (~200 warp-instructions per visible Gaussian; 0.22 ms): reduce only by dropping the reference's parent-tile test (not done).
Full check #6 (/tmp/check-6): PASS, speedup 3.1488 (N1) for f115bf2 (candidate 97.9 s; dPSNR +0.35, dSSIM +1.59, dLPIPS -0.22, dlogN +0.35 sigma).
3.7 hRound 10 — SSIM kernels, dead ends (16:55–17:10)
Fast fused-SSIM (submodules/fused-ssim/ssim.cu: horizontal pass of all five quantities into smem + vertical pass in registers, two barriers
per channel instead of ~25, channel in grid z): forward 100 -> 44 us, backward 58 -> 36 us (/tmp/dev/tools/ssim_test.py; values within 1-3 ulp
of the original kernels). time 96.44 s (3.197), banked e33206a, quick PASS; full check #7 running (/tmp/check-7).
Dead ends tried this round (all reverted, no gain): active-pixel compaction per bucket in the backward render kernel (0.79 vs 0.76-0.80 ms),
early-out for visible Gaussians without gradient in the backward preprocess (22-40% of visible Gaussians have zero gradient, yet 0.238 vs 0.231 ms),
8x4 tiles again with the new kernels (2.525 vs 2.32 ms), Adam launch-config sweep (VPT 2/4/8 x block 128/256/512: 0.80-0.83 ms, memory bound).
4.5 hRound 11 — profile at step 25000, 64-Gaussian buckets (failed), per-lane tile paths (17:25–17:55)
Profile of e33206a at step 25000 (/tmp/r1/prof_best3, 20 iterations): span 3.549 ms/iter, GPU busy 3.448 ms (97.2%), 38 kernels/iter:
PerGaussianRenderRawCUDA (backward render) 0.865 ms (25.1%), adamFusedCUDA 0.799 (23.2%, at the HBM floor: ~1.5 kB/Gaussian x 2.3 M),
renderRawCUDA 0.499 (14.5%), tileCountCUDA 0.277 (8.0%), preprocessBackwardRawCUDA 0.259 (7.5%), radix sort onesweep 0.233 (+0.020 histogram),
tileWriteCUDA 0.164 (4.8%), preprocessRawCUDA 0.164 (4.7%), fused-SSIM fwd 0.045 + bwd 0.036, small kernels ~0.07.
Dead end: 64-Gaussian buckets (two-pass backward, half the bucket checkpoints) gave wrong gradients; bisecting showed that the original
early-exit bucket_idx_in_tile * 32 >= max_contrib[tile_id] relies on its *unsigned* comparison (a signed-int version gives 7-20% gradient
errors; root cause - negative bucket index or a huge max_contrib - not found); reverted.
Change: tile count/write kernels handle Gaussians with at most SMALL_N=6 candidate tiles in their own lane (no ballot/shuffle enumeration, no
integer divisions; the write pass expands the keep mask of the first 6 candidates in place). Same tests, same order: num_rendered and
num_buckets identical to e33206a on 8 cameras of the step-7000 snapshot (/tmp/dev/tools/run_nrcheck.sh), images bitwise identical.
kprof (step-7000 snapshot): tileCount 0.224 -> 0.157 ms, tileWrite 0.139 -> 0.113 ms (SMALL_N 4: 0.169/0.117, 8: 0.161/0.113).
time (/tmp/r12/time): 93.06 s = 3.313x (was 96.44 s); check --quick /tmp/r12/quick PASS; banked adaaf81 [N0].
5.3 hRound 12 — lazy SH Adam (18:00–18:50)
Hotspot: adamFusedCUDA 0.80 ms/iter (23%), at the HBM floor because it updates all 59 floats of every Gaussian (2.3 GB/iter) although the SH
parameters (48 of 59 floats) are only observed for Gaussians that are rendered (~30%) and the others have a gradient of exactly zero.
Change (GS_OPT_LAZYSH=1, needs CUSTOM_ADAM/RAWRENDER/VISADAM/FASTDENSIFY): Adam for xyz/opacity/scaling/rotation stays dense (needed every
step for the culling); the SH update is applied lazily: per Gaussian applied[row] (steps incorporated) and gs[row] (step whose gradient is in a
persistent buffer written by the backward preprocess). After the tile count the rendered Gaussians (tiles_touched > 0) are brought up to date by
lazyShRowsCUDA (one warp per 32 rows, two Gaussians at a time over half-warps; first the pending gradient step with the dense update formula,
then the k missed zero-gradient steps in closed form: m*b1^r, sqrt(v)*b2^(r/2), p -= ss*m*b1^r/(sqrt(v)*b2^(r/2)/bc2+eps), float32-rounding
equivalent to the sequential update), then shColorCUDA computes their colors. Densification flushes all rows (and skips the SH step like the
dense optimizer does when the parameters are replaced), the last optimizer step (step count == iterations-1) flushes before the judge's hook
reads the parameters, state_dict/save_ply flush too. Step tables (lr/bc1, sqrt(bc2)) are built with the dense update's Python formulas.
Tests: /tmp/dev/tools/lazy_test.py (dense vs lazy over 76 steps with random visibility, densify flushes; passes; the first version with a
sequential catch-up was bitwise identical), lazy_bench.py (kernel time). First (flat, per-element) catch-up kernel: 1.3 ms -> time 112.6 s
(slower!); row-oriented kernel with closed-form runs 0.36-0.43 ms -> time 87.82 s (3.511x); memory-only floor of the access pattern is
0.295 ms (scattered 180-byte rows, ~2.3 TB/s); more rows per warp / prefetching / forced occupancy gave nothing.
Profile at step 25000 (/tmp/r14/prof): span 3.265 ms/iter (was 3.549): backward render 0.864, forward render 0.502, lazy catch-up 0.430,
backward preprocess 0.267, sort 0.253, tileCount 0.199, geometry Adam 0.164, tileWrite 0.144, shColor 0.096.
Quick check PASS (/tmp/r14/quick). Full check #9 (/tmp/check-9): PASS 3.5264 (N1), candidate 87.43 s; d: PSNR +0.67, SSIM +1.84,
LPIPS -1.94, logN +0.75 sigma. Banked 538f172, switches.json + GS_OPT_LAZYSH.
5.7 hRound 13 — pipeline bubbles and render-loop overhead (18:50–19:10)
Profile gaps at step 25000 (/tmp/r15/prof): ~50 us/iter of GPU idle, 25 us of it from the blocking loss_hist.tolist() of the progress bar every
10 iterations (drains the GPU queue) and ~11 us from the num_rendered host sync. Changes: (1) the lazy kernels are enqueued between an async D2H
copy of num_rendered (pinned buffer + event) and the host's event sync (no measurable gain: 87.82 -> 87.73 s); (2) the loss read-back is lagged by 10
iterations through a ring of pinned buffers (87.07 s, -0.75%). SASS of the render loops showed ~14 of ~35 instructions per light-path iteration were
bookkeeping: forward inner loop restructured (checkpoint per bucket of 32 in an outer loop, contributor counter replaced by batch_base+j+1 at
contribution, one combined cull test; images bitwise identical, kprof fwd 0.438 -> 0.384 ms); backward: __fdividef(1, 1-alpha) instead of the IEEE
reciprocal and the constant factors (0.5*W, -0.5) applied after the loop (kprof bwd 0.799 -> 0.729; gradients at noise level, relL2 1e-5..1e-4 vs the
previous version). Splitting the 95-step pipeline into ramp-up/steady/ramp-down phases to drop the range checks: no gain, reverted.
time /tmp/r16/time: 83.43 s (3.696x); quick PASS; full check #10 (/tmp/check-10): PASS 3.7073 (N1), candidate 83.16 s;
d: PSNR -1.82, SSIM +0.23, LPIPS -2.26, logN +0.99 sigma. Banked (opt HEAD).
Gate-3 risk note: over the 10 full checks d(LPIPS) has mean -1.13 sigma (sd 0.8; ref seed 1 sits +2.07 sigma above the 8-seed mean, so a candidate
that was fully independent of it would be centered at -2.07 and fail ~18% of the time), d(logN) mean +0.59 (ref seed 1 at -1.38), PSNR/SSIM within +-1.9.
The last two checks have the largest |d(LPIPS)|; keep watching it.
5.9 hRound 14 — grouped forward loop, gate-3 risk estimate (19:10–19:25)
Forward inner loop evaluates the exponents of 4 Gaussians first (independent work, hides LDS latency), then composites in order: kprof fwd
0.384 -> 0.329 ms, images bitwise identical (groups of 8: same, 16: slower). time /tmp/r17/time 81.42 s (3.787x); quick PASS;
full check #11 (/tmp/check-11): PASS 3.7661, candidate 81.86 s; d: PSNR -1.19, SSIM +1.10, LPIPS -2.09, logN +0.59 sigma.
Within-version spread of the statistical gate (3 more check --n1 runs of the same tree, /tmp/n1-c11-*, tool /tmp/dev/tools/n1_repeat.sh):
d(PSNR) -1.19/-0.79/-0.30/-1.41, d(SSIM) +1.10/+0.97/-0.17/+0.69, d(LPIPS) -2.09/-1.29/-1.56/-0.46, d(logN) +0.59/+1.10/+0.41/+0.53 sigma
-> mean/sd: PSNR -0.92/0.5, SSIM +0.65/0.57, LPIPS -1.35/0.68, logN +0.66/0.3: estimated P(any |d| > 3) ~1% for this version. Different versions
shift the mean (candidate LPIPS z vs the 8-seed mean: c5 +0.71, c6 +1.85, c7 +0.54, c8 +2.09, c9 +0.13, c10 -0.19, c11 -0.02; ref seed 1 is +2.07):
re-screen the final version with 3-4 n1 samples. Banked.
6.4 hRound 15 — visible-list compaction (dead end), interleaved SH state (19:25–19:50)
Profile at step 8000 (/tmp/r18/prof8k): GPU idle 0.389 ms/iter (11.8%) around the densification iteration (host syncs of the python densify + 3.3 ms of
index_select gathers per event): ~1 s of the run; not addressed.
Dead end: compacting the visible Gaussians (radii > 0) into a list built by the preprocess (warp-aggregated atomics, unordered: pair order comes from the
scan in Gaussian order) and running tileCount / tileWrite / shColor / backward preprocess over it with bounded grid-stride loops. Images and pair counts
identical (N0), kprof tileCount 0.157 -> 0.124, tileWrite 0.113 -> 0.105, backward preprocess unchanged (0.224 vs 0.211-0.231), but judge time
82.29 s vs 81.42 s: slower. Lane utilisation (30% visible) was not what limits these kernels. Reverted.
AoS microbenchmark (/tmp/dev/aos/aos_test2.py, 2M rows, 30% visited, p/m/v/g access): separate arrays 0.95 ms, p separate + (m,v,g) interleaved in one
576-byte row 0.28 ms, everything interleaved 0.24 ms -> adopted the (m, v, g) interleaving: st (N, 144) = m | v | g of the 48 SH values, the optimizer
state tensors of f_dc/f_rest are views into it, the backward writes the gradient into columns 96..143, densification gathers st with one index_select.
Catch-up kernel 0.363 -> 0.294 ms (bench), 0.432 -> 0.372 (step 25000), backward preprocess +0.013 (strided G rows). time /tmp/r19/time **80.97 s
(3.808x); quick PASS; full check #12 (/tmp/check-12): PASS 3.8433 (N1)**, candidate 80.22 s; d: PSNR +0.67, SSIM +1.46, LPIPS -1.43, logN +0.09 sigma. Banked.
Profile (step 25000, /tmp/r19/prof): span 2.962 ms/iter: backward render 0.795, forward render 0.377, lazy catch-up 0.372, backward preprocess 0.280,
sort 0.253, tileCount 0.186, geometry Adam 0.164, tileWrite 0.127, shColor 0.095, preprocess 0.075, SSIM 0.081.
Ideas not done: p inside the interleaved row (another ~0.05 ms, needs strided parameters), shColor fused into the catch-up (~0.05), depth-sort-then-tile-sort
restructure (~0.1, gathers), densification without host syncs (~1 s).
6.7 hRound 16 — stream concurrency (19:55–20:12)
The lazy SH update (catch-up + colors, DRAM-bound, 0.47 ms) only depends on tiles_touched, not on the binning chain (tileWrite, sort, ranges, bucket scan,
0.55 ms of latency/atomic-bound kernels). (1) Side stream (non-blocking): lazy + shColor concurrent with the chain: time 80.97 -> 77.88 s (3.959x);
the timeline showed the chain's first kernel queued behind the lazy kernel (blocks dispatched in launch order). (2) Chain on a high-priority stream
(cudaStreamCreateWithPriority), SH update on a low-priority stream, both waiting for an event after the scan, the render waiting for both: **75.94 s
(4.060x)** (/tmp/r21/time; timeline /tmp/r21/prof: lazy 305-730, chain 342-1005, render 1006; kernel durations stretch from contention, GPU 98.9% busy,
iteration span 2.657 ms at step 25000 vs 2.96 before). Quick PASS; full check #13 (/tmp/check-13): PASS 4.0713 (N1), candidate 75.73 s; d: PSNR +0.21,
SSIM +0.23, LPIPS -1.44, logN +0.89 sigma. Banked. Numerics unchanged (N0): same kernels, only scheduling.
Ideas next: lazy/colors on radii > 0 right after the preprocess so they overlap with tileCount (ALU-bound) instead of the chain; zero-fill of the backward
accumulators on the side stream during the forward; split the backward preprocess so the SH-gradient part runs beside the geometry Adam.
7.3 hRound 17 — more overlap and kernel experiments, mostly dead ends (20:12–20:55)
Gate-3 samples of the check-13 tree (3 more check --n1, /tmp/n1-c13-*): d(PSNR) +0.21/+1.47/-0.19/-1.39, d(SSIM) +0.23/+2.23/+0.20/+0.03,
d(LPIPS) -1.44/-1.87/-1.35/-1.81, d(logN) +0.89/+0.91/+1.72/+0.55 sigma -> all within 2.3 sigma, estimated fail risk ~2%.
Tried and reverted (all measured with the judge's time, base 75.94 s):
 - SH update started right after the preprocess on radii > 0 (overlapping tileCount instead of the binning chain; tileCount/scan/chain on the high-priority
   stream): 75.82 s, no gain (the GPU is saturated; shifting work between overlapped kernels does not change the total).
 - Lazy kernel with a bounded grid (grid-stride, 528 / 264 blocks) to leave resources to the chain: 78.74 / 79.10 s, slower (the kernel needs the parallelism for DRAM MLP).
 - Speculative tileWrite launched before the host reads num_rendered (persistent pair buffers with a capacity guard) to hide the 25-40 us host gap: 76.13 s, no gain.
 - Contribution masks (forward records which Gaussians of each bucket contributed per pixel, backward drives its heavy path from the masks and drops the cull tests
   and the logf): backward 0.729 -> 0.698 ms but forward 0.329 -> 0.342 ms (mask stores): net ~0.7%, reverted.
 - Visible-list compaction (round 15).
Measurements for ideas: instrumented forward (temporary): 99.3% of the tile-list pairs reach alpha >= 1/255 at some pixel of their tile, i.e. the (reference-exact) tile
test has <1% false positives; 85-88% of the processed pairs contribute to at least one pixel, ~10% of the pairs are never reached (all pixels of the tile terminated).
CUB microbenchmark (/tmp/dev/cubtest/sort_bench.cu): sort of 2.9M (tile|depth) 64-bit keys 267 us; depth sort of 0.6M 32-bit keys 77 us + tile-only stable sort of
2.9M 64-bit keys (bits 32..46) 105 us = 182 us -> a depth-sort-then-tile-sort restructure could save ~100 us/iter on the critical path (not done: needs index-order
compaction, K read-back, base-offset scatter, ~2 h).
7.8 hRound 18 — accumulators in the forward, depth-first binning (20:40–21:20)
(1) The backward accumulators (88 MB, torch.zeros in the backward = 21 us on the critical path) are allocated by the forward (returned as an extra output, saved
in ctx) and zeroed there on the side stream: time 75.0 s (4.11x). (2) Depth-first binning (rasterizer_impl.cu forwardRaw): visible Gaussians selected in
index order (cub DeviceSelect::If, K read back through pinned memory), depth keys + stable CUB sort of K items (stream A, high priority) beside the tile count
(low-priority stream, then lazy SH update); counts gathered in depth order, exclusive scan, per-Gaussian base offsets scattered; tileWrite (exclusive_base)
writes the pairs in depth order; only bits 32..46 (tile) of the 64-bit keys are sorted afterwards (stable): final order within a tile is (depth, Gaussian index),
identical to the old (tile|depth) sort. Images bitwise identical, num_rendered/num_buckets identical on 8 cameras. First version (both streams high priority):
74.63 s (the depth chain started only after tileCount: same-priority blocks are dispatched in arrival order); with tileCount on the low-priority stream so that the
depth chain preempts it: 72.94 s (4.23x), re-run 73.17 s (run-to-run noise ~0.25 s). Variants that were slower (reverted): SH update on all visible Gaussians
overlapping the tile count 74.69 s; accumulator memset moved before the depth chain 74.23 s. Quick PASS, full check #14 (/tmp/check-14): PASS 4.2409 (N1),
candidate 72.70 s; d: PSNR +0.29, SSIM +1.70, LPIPS -1.41, logN +1.03 sigma. Banked.
Lesson: never edit /tmp/dev/cu while a judge command is running on it (the judge copies the tree at the start of every sub-run); git stash instead (done once, check
validated by the extension cache key: no new digest was built during the check).
Timeline now (step 25000, span 2.56 ms): preprocess 75, tileCount 244 (beside the depth chain), then gather/scan/scatter ~150 (beside the lazy kernel), tileWrite 228
(contended), tile sort ~215, render 306, loss ~105, backward render 738, backward preprocess 263, geometry Adam 164.
8.5 hRound 19 — instruction-level work in the render kernels (21:20–21:58)
Fused offset scan with CUB iterators (gather input / scatter output): exact but 73.60 s (no gain); tileWrite over the depth-sorted list: exact, 74.20 s (slower);
catch-up kernel RPW=4: 73.16 s (same): all reverted. n1 samples of the banked c209abf tree (3 more): PSNR -1.80/-1.44/+0.80, SSIM -0.36/-1.23/+1.56, LPIPS -1.02/-0.35/-1.65,
logN +1.02/+0.95/+0.42 sigma (with check 14: all within 1.8 sigma).
SASS of the backward render loop showed the color accumulator pipeline was the cost: the gradient only needs ar . dL/dpixel, so the pixel pipeline carries (T, A = ar . dL)
instead of (T, ar[3]): 2 shuffles per step instead of 4, dL/dalpha = fma(1/(1-a), A + bg_term, T (c . dL)) (7 instructions instead of 13 in the heavy path):
kprof bwd 0.729 -> 0.633 ms, time 70.93 s (4.35x), full check #15: PASS 4.3463, d: PSNR -1.60, SSIM -1.62, LPIPS -0.24, logN +1.21 sigma.
Then inline PTX ex2.approx.ftz / rcp.approx.ftz instead of __expf / __fdividef (they carry range handling: 3 and 6 extra instructions; results identical in
range, images bitwise identical): bwd 0.633 -> 0.584, fwd 0.329 -> 0.318 ms; power as fma(dx, fma(-a/2, dx, -b dy), (-c/2 dy) dy) with -a/2, -b, -c/2 stored in the
forward smem record / per-lane constants in the backward: fwd 0.297 ms. time 68.64 s (4.49x), quick PASS, full check #16 (/tmp/check-16): PASS 4.514 (N1),
candidate 68.30 s; d: PSNR -1.61, SSIM +0.01, LPIPS -1.15, logN +0.49 sigma. Banked.
(log2(e) folded into the records + w = alpha*T for the color accumulation: kprof -1%, time 68.71 s = noise, reverted.)
Ideas: fast reciprocals/exp in tileCount and the backward preprocess (latency-bound, doubtful), p in the interleaved row, shColor fused with the catch-up.
9.3 hRound 20 — deterministic training (22:00–22:42)
Goal: remove the run-to-run noise of gate 3 (~6% fail risk estimated for a single sample): make the whole training bitwise reproducible, so that the d measured in a
check is exactly the d of the final evaluation.
Sources of nondeterminism found: (1) the float atomics of the render backward (order of the additions) -> 64-bit fixed-point accumulators (fxAdd: round(v * 2^46)
(conic: 2^36) added with an integer atomicAdd; converted back in the backward preprocess; dL/dmean2D is written as float by the preprocess for the densification
statistics); (2) imgState.max_contrib of the forward render kernel: cub::BlockReduce<uint32_t, BLOCK_SIZE> is used with a two-dimensional thread block, so its linear
thread index is threadIdx.x only, 8 threads of every column race on the same slot and the slots of the other warps are never written (uninitialised shared memory): the
per-tile maximum of the last contributors, which bounds the buckets of the backward, depended on stale shared memory. (The REFERENCE rasterizer has the same code,
forward.cu:427, with 16x16 blocks.) Now a warp-shuffle max + one shared slot per warp = the exact maximum.
Verification: two 3,000-step runs and two full 29,999-step runs (seed 1, the judge's entry in traj mode): every parameter tensor bitwise identical (2,031,811 Gaussians);
check --n1 twice: identical json. Cost: time 69.2 s (4.45x, was 68.64 s: the 64-bit atomics and the doubled accumulator memset cost 0.8%).
d (seed 1, in sigma): PSNR -0.35, SSIM -0.75, LPIPS +0.73, logN +0.46 (the old build's samples went up to -1.6 / +1.8 sigma).
Full check #17 (/tmp/check-17): PASS 4.4513 (N1), candidate 69.263 s, same d. Banked: opt @ 9c6e3c0 (= cuda-dev 8063d48).
Consequences: the d of the final evaluation = this d as long as the tree and switches are unchanged and the training stays deterministic. Every later change of the
numerics re-rolls d (a different, equally valid trajectory): run check --n1 on the final tree after the last numerics change (3 min); never add float atomics.
↑ 3DGS