Claude Haiku 5.5 2.35×
Harness: Claude Code · switches: GS_DENSE_ADAM=1 GS_FUSED_NORMALIZE=1 GS_FUSED_REST_ADAM=1
- Judged
- 2.35×
- Tier
- N1
- Self-best
- 2.34×
- Rounds
- 39
- Hours
- 10.0
- API cost
- $58.8
- Input tokens · 99.7% cached
- 1026 M
- Output tokens
- 2.1 M
- Requests
- 1,805
§ 02Key levers
- Smaller tiles + rewritten binning
- ✗ stays 16×16; instead warp-level culling in forward (1.92 → 2.08), binning with one thread per (Gaussian, tile) slot (1.73 → 1.86)
- Bypass autograd, activations folded into raster kernel
- ✗ only normalization made into one kernel
GS_FUSED_NORMALIZE=1(1.55 → 1.63) - Lazy Adam (update only visible Gaussians)
- ✗ dense; rest-SH Adam folded into backward preprocess
GS_FUSED_REST_ADAM=1(1.86 → 1.91) - Fused loss / SSIM
- two SSIM kernel attempts (16×16 blocks, redundant flushes removed), both within noise, not committed
- Other main levers
--use_fast_math(1.35 → 1.43), custom dense Adam, coalesced SH grad writes (1.63 → 1.73), forward color and inverse depth in shared memory (+5.2%), backward visits only possibly contributing pixels, tiles launched in descending length order
§ ARaw trace
Raw work log43 sections
0.2 hRound 1: profile of the unchanged reference (14:00-14:02 UTC)
Profile:bench.py profile(steps 7000..7019, torch.profiler;/tmp/r1/profile). GPU kernel time per step ~9.4 ms, kernels busy 85% of the GPU span. Shares of kernel time: | Hotspot | share | per step | |---|---|---| |PerGaussianRenderCUDA(rasterizer backward, one warp per 32-splat bucket) | 27.9% | 2.63 ms | | Adam: foreachmulti_tensor_apply(about 10 foreach ops x 6 groups) | 23.5% | 2.22 ms | |renderCUDA(rasterizer forward blend) | 17.3% | 1.63 ms | |preprocessCUDA(forward projection and SH) | 5.2% | 0.49 ms | |duplicateWithKeys(tile key generation) | 5.2% | 0.49 ms | | gathers / index (densification, visibility masks) | 3.3% + 2.3% | 0.31 + 0.22 ms | | radix sort | 2.4% | 0.23 ms | | fill / zero | 1.7% | 0.16 ms | Host syncs per step (runtime trace):nonzeroforvisibility_filter,loss.item(),elapsed_timeof the iteration events, and twocudaMemcpydevice-to-host reads in the rasterizer (num_rendered,bucket_sum).
0.2 hRound 1 result: fused Adam (switch `GS_FUSED_ADAM=1`, N1) — commit `3f921e0`
- Change:torch.optim.Adam(..., fused=True)for the Gaussian optimizer, behind the switch. Same update rule, different operation grouping (the fused kernel computes lerp / addcmul / sqrt / addcdiv in one pass per group). -time(/tmp/r2/time): total 263.0 s vs frozen reference 305.4 s, speedup_vs_frozen_reference 1.161; windows 73.8 / 96.1 / 93.1 s. -check --quick(/tmp/r2/quick): PASS, exact_items_pass (3000 steps, final count 1,087,587). - Full check: not run yet (gate 3, N1, pending). -/submission/switches.json={"environment": {"GS_FUSED_ADAM": "1"}}.
0.2 hRound 2: remove per-step host syncs (N0, unconditional; `/tmp/dev`, not yet committed)
-visibility_filteris now the boolean maskradii > 0(wasnonzero(), a host sync per step). The max-radii and densification-statistics updates are masked (torch.where) and produce the same values as the indexed updates. - The loss EMA is kept on the GPU and read back only when the progress bar updates (wasloss.item()per step). -iter_start.elapsed_time(iter_end)is evaluated only when a TensorBoard writer exists (it synchronizes the device; TensorBoard is not installed in the training environment, so this path is unused).
0.2 hRound 2 result: host syncs removed (N0) — commit `d533acd`
-time(/tmp/r2/time_sync): total 245.9 s, speedup_vs_frozen_reference 1.242 (windows 67.6 / 90.6 / 87.6 s); against round 1 (1.161) that is about 7% more. -check --quick(/tmp/r2/quick_sync): PASS, exact_items_pass.
0.4 hRound 3: forward blend kernel, colour/inverse depth staged in shared memory — NO GAIN, not committed
- Change (/tmp/dev3):collected_colorandcollected_invdepthgathered with the batch fetch, so the inner loop reads shared memory instead offeatures[...]anddepths[...]per pixel and Gaussian. -time(/tmp/r3/time): total 246.6 s, speedup_vs_frozen_reference 1.238 (round 2: 1.242). Within noise: no gain. - Probe (/tmp/rtest, reference build vs this build, same inputs): colour, depth and radii are bitwise identical. - Conclusion: the blend kernel is not limited by those global gathers. Dropped (gitnot touched).
0.5 hRound 4: rasterizer backward kernel (N0) — commit `c3a3620`
- Profile before (round 1 profile, unchanged):PerGaussianRenderCUDAwas 27.9% of kernel time, one 32-thread block per bucket. Each of its ~290 steps per warp issued global loads from lane 0 (pixel state, gradient inputs) that the shuffle chain had to wait for: a latency chain of about 290 global round trips per bucket. - Change: the pixel inputs of 32 consecutive pixel indices are staged in shared memory (lane j loads pixel chunk+j, coalesced); lane 0 reads one entry per step. Background colour is read once into registers. Blocks have 8 warps (256 threads) instead of 1, so more warps are resident per SM. Arithmetic per warp is unchanged. -time(/tmp/r4/time): total 228.1 s, speedup_vs_frozen_reference 1.339 (round 2: 1.242; +7.8%); windows 62.8 / 83.9 / 81.4 s. -check --quick(/tmp/r4/quick): PASS, exact_items_pass. - Probe (/tmp/rtest/rast_test.py, reference build vs this build, fixed inputs): colour, depth, radii bitwise identical; gradients differ by up to 3.7e-7 relative, the same size as reference-vs-reference (atomics).
0.9 hRound 5: warp-cooperative duplicateWithKeys (N0) — NO GAIN, not committed (`/tmp/dev5`)
- One warp per Gaussian, tiles tested 32 at a time, accepted keys written to consecutive positions (same output layout and sentinel padding as duplicateWithKeys). -time(/tmp/r5/time): total 238.4 s, speedup 1.281 (round 4: 1.339). Windows 73.0 / 84.0 / 81.5 s: the last two are identical to round 4 within 0.1 s; the first window differs by 10 s, the size of the window-1 run-to-run swing seen before (62.8 to 73.8 s). Conclusion: no measurable gain from this kernel. -check --quick: exact_items_pass.
0.9 hRound 6: launch bounds (256, 4) on the backward kernel (64 registers, no spills), on top of round 5 — NO GAIN
-time(/tmp/r6/time): total 230.0 s, speedup 1.328 (round 4: 1.339); windows 62.5 / 85.1 / 82.4 s. Not better. - Not committed. Submission stays atc3a3620.
1.0 hRound 7: no per-densification `torch.cuda.empty_cache()` (N0) — commit `550356a`
- Profile of the committed state (/tmp/r7/profile, steps 7000-7019; kernel busy about 7.7 ms per step): top kernelsPerGaussianRenderCUDA26% (2.01 ms/step),renderCUDA21% (1.61 ms), fused Adammulti_tensor_apply15% (1.15 ms),preprocessCUDA6.4% (0.49 ms),duplicateWithKeys6.2% (0.48 ms). Runtime:cudaFree26.6 ms in the 20-step window, all from the one densification step that the window contains (empty_cacheat the end ofdensify_and_prune, every 100 steps). Amortized, about 0.3 ms per step. - Change: the call removed; allocator behaviour only. -time(/tmp/r7/time): total 226.3 s, speedup_vs_frozen_reference 1.349 (previous commit 1.339; windows 62.2 / 83.4 / 80.7 s). Small but consistent gain in windows 2 and 3. -check --quick(/tmp/r7/quick): exact_items_pass.
1.4 hRound 8 (cont.): rasterizer built with `--use_fast_math` (N1) — commit `c97a3c9`
- Change:submodules/diff-gaussian-rasterization/setup.pynvcc flags gain--use_fast_math(exp, div, sqrt in the rasterizer kernels). Numerics tier N1 (statistical gate required). -time(/tmp/r8/time_fastmath): windows 58.4 / 78.4 / 76.6 s, total 213.4 s, speedup_vs_frozen_reference 1.431 (550356ain the same kind oftimerun: 1.349). -check --quick(/tmp/r8/quick_fastmath): exact_items_pass (PASS). - Full check ofc97a3c9(/tmp/check-3, started 15:24:43 UTC): pending, result appended below when it finishes. - Profile facts recorded from/tmp/r7/profile(steps 7000-7019, 7.78 ms kernel time per step): rasterizer backward 26%, forward 21%, fused Adam 15% (1.13 ms, about 2 TB/s effective), preprocess fwd 6%, duplicateWithKeys 6%, radix sort 3%. The tile test that drops non-contributing (Gaussian, tile) pairs already exists in this 3dgs_accel rasterizer (max_contrib_power_rect_gaussian_floatinduplicateWithKeys), so it is not a lever.
1.4 hRound 9 (in progress): dense Adam kernel behind `GS_DENSE_ADAM` (`/tmp/dev11`)
- Idea: the Adam step (15%) runs at about 2 TB/s; one bandwidth-bound kernel per parameter tensor (float4 streams, same update rule as torch.optim.Adam, state layout unchanged) should do better. Expected 0.3-0.5 ms per step. - Code:cuda_rasterizer/adam_dense.cu(new, in the rasterizer extension),ext.cpp(adamDense), andscene/gaussian_model.py(DenseAdam(torch.optim.Adam), selected byGS_DENSE_ADAM=1). - Status: written in/tmp/dev11, not yet compiled or timed. - Full check ofc97a3c9(/tmp/check-3, 15:24-15:33 UTC): PASS, tier N1, speedup 1.4264 (spread as frozen). This is the current verified submission (/submission/switches.json=GS_FUSED_ADAM=1, rasterizer with--use_fast_math).
1.4 hRound 9: dense Adam kernel (`GS_DENSE_ADAM`, N1) — correctness checked, timing pending
-/tmp/r9/adam_test.py(correctness only, not timing):DenseAdamagainsttorch.optim.Adam(fused=True)on random tensors with the shapes of the six groups, 200 steps with a decaying lr: worst max|diff| / max|param| = 3.3e-7 after 200 steps (float32 rounding level). The test script's last line failed (a tensor-truthiness bug in the test itself, not in the kernel); the state keys are the torch Adam ones (step,exp_avg,exp_avg_sq). - Kernel build:bench.py buildsucceeded (/tmp/gs-ext/a22b6c9d0a666dac630614b4). - Backward v1 (/tmp/dev12: per-pixel constants of the backward kernel read from shared memory per lane instead of six shuffles per step; bitwise-same arithmetic, N0 intended):time(/tmp/r9/time_bwd) total 218.2 s, windows 59.8 / 80.3 / 78.1 s, speedup 1.3996 — slower than the 213.4 s ofc97a3c9(1.431), about 2%.check --quickPASS. Not adopted in this form. - Dense Adam, firsttime(/tmp/r9/time_dense): ERROR on the first optimizer step. Cause: the point positions fromfetchPlyare a column-major array, so_xyz(torch.tensor(...).float().cuda()) is a non-contiguous parameter (strides (1, N)), which my kernel'sTORCH_CHECKrejected. Reproduced with a 3-iteration run of the judge's recipe (/tmp/r9/repro.py). Fix (in/tmp/dev11and/tmp/dev13):_xyzis made contiguous at creation (values and shape unchanged, a memory-layout change), the Adam state is created contiguous, andDenseAdamupdates a contiguous copy of any non-contiguous parameter (copied back). The repro now completes two optimizer steps. - Dense Adam, fixed (/tmp/r9/time_dense2, the same code as commit5b80905): **total 197.6 s, windows 53.6 / 73.0 / 71.0 s, speedup_vs_frozen_reference 1.5456** (c97a3c9in the same kind of run: 213.4 s, 1.431). Peak memory 5.3 GiB allocated. - Dense Adam plus backward v2 (/tmp/r9/time_dense13, packed per-pixel constants as float2/float4): 196.8 s, speedup 1.5519: 0.4% more, within the window-to-window noise. Not committed (kept the backward kernel unchanged).check --quickof that tree: PASS. - Committed5b80905(dense Adam behindGS_DENSE_ADAM=1;/submission/switches.json= `{"environment": {"GS_DENSE_ADAM": "1"}}).check --quickof the committed tree (/tmp/r9/quick_submit_dense`): PASS. - Full check of5b80905(/tmp/check-4, 15:59-16:07 UTC): PASS, tier N1 (equivalent), speedup 1.5478 (candidate 197.3 s, windows 53.4 / 73.0 / 70.9 s; peak 5.32 GiB allocated). Gate 2: exact items OK (3000 steps, final count 1,090,845). Gate 3 (seed 1, sigma 3): PSNR +0.0006 dB (bound 0.117), SSIM +0.0002 (bound 0.00069), LPIPS -0.0008 (bound 0.0010), log count +0.0057 (bound 0.0163). This is the current verified submission.
2.2 hRound 10 (in progress): profile of `5b80905` and the next two changes
- Profile (/tmp/r10/profile, steps 7000-7019, kernel time 6.8 ms per step): backward rasterizer 1.71 ms (25%), forward rasterizer 1.40 ms (21%), dense Adam 0.66 ms (10%; about 3.5 TB/s, close to the bandwidth limit),preprocessCUDAbackward 0.50 ms (7%),duplicateWithKeys0.50 ms (7%),preprocessCUDAforward 0.14 ms (2%), radix sort 0.23 ms (3%), SSIM 0.13 ms (2%), densification gathers about 0.1 ms per step amortized, and about 0.8 ms per step in small torch kernels (fills, elementwise, the normalisation's autograd, the concatenation of the SH parameters). - Correction to the round-7 notes: the 0.49 ms kernel namedpreprocessCUDAis the backward one (its signature takesfloat3 const* means; the forward takesfloat const* orig_points). Its cost is the per-Gaussian SH gradient (15 vec3 stores at a 180-byte stride, uncoalesced) and the 180-byte SH read; the forward preprocess is 0.14 ms. - Planned changes (each behind a switch): 1.GS_SEPARATE_SH=1: passdcand the rest of the SH coefficients to the rasterizer as two tensors instead of the per-steptorch.cat; colours and gradients identical (N0)./tmp/dev14a, timing/tmp/r10/time_sep. 2.GS_FUSED_NORMALIZE=1: the rotation normalisation (F.normalize, about 0.3 ms per step of autograd kernels) as one fused forward and one fused backward kernel (N1)./tmp/dev14b, not built yet. 3. Next: coalesced writes of the backward SH gradients (stage each warp's 32 x 45 floats in shared memory), N0. - Change 1 result (GS_SEPARATE_SH=1,/tmp/dev14a,/tmp/r10/time_sep): total 197.4 s, windows 53.5 / 73.1 / 70.9 s, speedup 1.5469, against 197.6 s for dense Adam alone (/tmp/r9/time_dense2). No measurable gain: the per-steptorch.catof the SH parameters is cheaper than estimated. Not adopted. Why no gain: the committed path already uses separate SH tensors.SparseGaussianAdamis defined in the rasterizer package, soSPARSE_ADAM_AVAILABLEis True andseparate_sh=Trueintrain.py(the rasterizer then getsdc(N,1,3) and the 15 rest coefficients (N,15,3)). TheGS_SEPARATE_SHswitch was a no-op; the cat I expected never happens. The earlier estimate of its cost was wrong. - Change 2 result (GS_FUSED_NORMALIZE=1,/tmp/dev14b,/tmp/r10/time_norm): **total 187.2 s, windows 51.0 / 69.2 / 66.9 s, speedup 1.6317**, against 197.6 s for dense Adam alone (1.5456): about 5% more. The fused normalisation saved more than the profile suggested (about 10 s per run). Correctness:/tmp/r10/norm_test.py(forward max diff 2.4e-7; backward 4e-6 absolute on unclamped rows, 1e-8 relative on clamped rows).check --quickof the committed tree (/tmp/r10/quick_submit_norm): PASS. - Committed3e2cf5b(GS_FUSED_NORMALIZE=1,/submission/switches.json= `{"environment": {"GS_DENSE_ADAM": "1", "GS_FUSED_NORMALIZE": "1"}}). Full check:/tmp/check-5`, started 16:24:44 UTC (pending). - Full check of3e2cf5b(/tmp/check-5, 16:24-16:32 UTC): PASS, tier N1 (equivalent), speedup 1.6354 (candidate 186.7 s, windows 51.0 / 69.0 / 66.8 s). Gate 2: exact items OK (3000 steps, final count 1,092,293). Gate 3 (seed 1): PSNR -0.055 dB (bound 0.117), SSIM -0.00027 (bound 0.00069), LPIPS +0.00011 (bound 0.0010), log count +0.0023 (bound 0.0163). The PSNR shift is inside the bound but about half of it; noted for later full checks. - Change 3 (/tmp/dev16, committed as5f62082): the backward preprocess (preprocessCUDA, backward, 0.50 ms per step in the round-10 profile) staged each block's SH gradients in shared memory and wrote them coalesced; the 252 MBdL_dshmemset is gone (torch::emptyon this path);computeColorFromSHwrites to ansh_offsetslot.time(/tmp/r10/time_sh): total 176.7 s, windows 48.6 / 65.2 / 63.0 s, speedup 1.7281 (previous 187.2 s, 1.6317).check --quick(/tmp/r10/quick_sh): PASS. - Correctness of change 3 (/tmp/r10/sh_grad_test.py,compare_sh.py): the rendered image is bitwise identical between the committed build and the new one (and between two runs of the committed build). The SH gradients differ between any two runs of the same build by about 2.5e-5 relative (atomicAdd ordering in the rasterizer backward); the new build differs from the committed one by a similar 1.6e-5 relative. No systematic error (a misplaced coefficient would be orders of magnitude larger). Tier N0 apart from that existing nondeterminism. - Full check of5f62082: pending. - Full check of5f62082(/tmp/check-6, started 16:38:39 UTC): PASS, speedup 1.7299 (windows 48.5 / 65.1 / 62.9 s). Gate 2: exact items OK. Gate 3 (seed 1): verdictequivalent; PSNR -0.008 dB (bound 0.117), SSIM +0.00003 (bound 0.00069), LPIPS -0.00044 (bound 0.0010), log count +0.0050 (bound 0.0163). This is the current verified submission.
2.8 hRound 11 (in progress): profile of `5f62082` and the flattened pair kernel
- Profile (/tmp/r11/profile, steps 7000-7019, 6.08 ms kernel time per step): backward rasterizer 1.71 ms (28%), forward rasterizer 1.40 ms (23%), dense Adam 0.66 ms (11%),duplicateWithKeys0.49 ms (8%), densification gathers and index kernels about 0.1 ms per step amortised, radix sort 0.22 ms (4%), backward preprocess 0.21 ms (3%, down from 0.50 ms after the coalesced SH writes), forward preprocess 0.14 ms (2%), SSIM 0.13 ms (2%), other small kernels about 0.3 ms. The normalisation kernels are now 0.04 ms per step together. - Hypothesis forduplicateWithKeys: one thread per Gaussian, looping over its rectangle, so each warp runs as long as its largest Gaussian (about 4.6 M slots per step, but about 9x more warp iterations than slots). Change: one thread per (Gaussian, tile) slot, with the Gaussian of every slot marked first (markPairGaussians), so the work is balanced and the writes are coalesced. Surviving slots keep their relative order, so the stable sort gives the same list; the expected output is bitwise identical (checked with the render test). Tree:/tmp/dev17, sequence/tmp/r11/run_g.sh. - Change 4 result (/tmp/dev17, committed asee9d83d):markPairGaussians+duplicateWithKeysFlat(one thread per (Gaussian, tile) slot).time(/tmp/r11/time_flat): total 164.0 s, windows 45.1 / 60.7 / 58.1 s, speedup 1.8625 (previous 176.7 s, 1.7281; about 7.8% faster). Render test (/tmp/r10/sh_grad_test.py): the image is bitwise identical to the previous build; the SH gradients differ by the same ~3e-5 relative atomicAdd noise.check --quick: PASS (/tmp/r11/quick_flat). The oldduplicateWithKeyskernel is kept but not launched. - Full check ofee9d83d: pending (/tmp/check-7). - Full check ofee9d83d(/tmp/check-7, started 16:55:46 UTC): PASS, speedup 1.8746 (windows 44.3 / 60.5 / 58.1 s). Gate 2: exact items OK. Gate 3 (seed 1): verdictequivalent; PSNR -0.0045 dB (bound 0.117), SSIM +0.000006 (bound 0.00069), LPIPS +0.00002 (bound 0.0010), log count +0.0027 (bound 0.0163). This is the current verified submission.
3.1 hRound 12 (in progress): profile of `ee9d83d` and the inverse-depth change
- Profile (/tmp/r12/profile, steps 7000-7019, 5.72 ms kernel time per step): backward rasterizer 1.71 ms (30%), forward rasterizer 1.44 ms (25%), dense Adam 0.66 ms (12%), radix sort 0.22 ms (4%), backward SH preprocess 0.21 ms (4%), forward preprocess 0.14 ms (2%), SSIM fwd+bwd 0.13 ms (2%), flattened duplicate and mark kernels 0.07 ms (1%), small torch kernels about 0.2 ms, densification gathers about 0.1 ms per step amortised. - Reading of the profile: the rasterizer kernels (about 55%) are limited by their per-pair arithmetic; the remaining items are each about 1% (sort is bandwidth-bound with six passes; Adam is at about 3.5 TB/s). Round-12 candidates in order: the inverse depth per Gaussian for the forward blend (the forward used1.f / depths[id]per contributing pair; now computed once in preprocess, same expression, bitwise expected); coalesced SH reads in the backward preprocess (about 1%, not started). - Change 5 (/tmp/dev18, sequence/tmp/r12/run_h.sh):invdepthsarray written in the forward preprocess (1.f / p_view.z), read in the forward blend instead of1.f / depths[id]. Verification: render test comparing the colour and the inverse-depth image with theee9d83dbuild; thentime,check --quick, and a full check on the tree before any promotion to/submission. - Change 5 result (/tmp/dev18,/tmp/r12/run_h.sh): the colour image and the inverse-depth image are bitwise identical to theee9d83dbuild (render test,/tmp/r12/depth_compare.log); gradients differ only at the run-to-run noise level.time(/tmp/r12/time_inv): total 162.7 s, windows 44.2 / 60.4 / 58.1 s, speedup 1.8773 (theee9d83dtime run gave 1.8625; its full check 1.8746). The gain is within noise (about 0.1-0.8%).check --quick: PASS. Not promoted on its own; it is part of the next candidate (change 6). - Change 6 (in progress,/tmp/dev18on top of change 5,/tmp/r12/run_i.sh): the SH coefficients of each block are loaded into shared memory with coalesced reads in the forward and the backward preprocess (the per-Gaussian reads are strided by 180 bytes); preprocess blocks of 128 threads. Expected 0.5-1%. - Change 6 result (staged SH reads in both preprocess kernels,/tmp/dev18,/tmp/r12/time_sh2): regression, total 180.7 s (windows 48.8 / 67.1 / 64.8 s; speedup 1.6902) against 162.7 s for change 5 alone. Not adopted. Correctness: the colour and inverse-depth images are bitwise identical; the SH gradients agree with the committed build within run-to-run noise in two of three runs (/tmp/r12/staged_run_a.ptand_b.pt), but the first staged run (new2_depth.pt) had a rare outlier (max difference 0.07 on the xyz gradient against about 0.003 noise). Repeat runs (_c,_d,_e) were started to check whether that is a fluke. The change is abandoned either way: the staging costs more than the strided reads it replaces. - Change 5 (inverse depth, bitwise identical):time162.7 s, speedup 1.8773 — within noise of the committed build (1.8746 full check). Not promoted;/submissionstays atee9d83d. - Round 12 conclusion: no promoted change. The remaining per-step items (rasterizer kernels, dense Adam, sort, preprocess) are either arithmetic-bound or within a few per cent of their bandwidth limits; SH-read staging is a regression. - Repeat runs of the staged build (_c,_d,_e): all three agree with the committed build within run-to-run noise (SH gradient maxima 9e-5 to 2e-4 and xyz 1.5e-3 to 2.8e-3; image bitwise identical). The earlier outlier was a single run and is not reproduced, so it is noted as a rare atomics fluctuation; the change is abandoned on timing.
3.6 hRound 13: forward blend with eight alphas computed first (`/tmp/dev19`, `/tmp/r13/run_a.sh`): NO GAIN
- Change: the forward blend loop computes the alphas of eight pairs in a first phase, then blends them in the original order (same expressions, thepower > 0skip written as!(power > 0)so NaN takes the original path). - Correctness: colour and inverse-depth images bitwise identical toee9d83d; gradients at noise level. -time(/tmp/r13/time_blend): 163.1 s, windows 44.3 / 60.5 / 58.2 s, speedup 1.8727, the same as the committed build (windows 44.3 / 60.5 / 58.1 s in its full check).check --quick: PASS. Not promoted: the compiler already overlaps the exponentials; the forward is not latency-bound on the alpha chain.
3.6 hRound 14 (in progress): fused Adam step of the rest-SH coefficients in the backward preprocess (`/tmp/dev20`)
- Idea: the rest-SH gradient (45 floats per Gaussian, 252 MB per step) is written by the backward preprocess and read back by the Adam kernel. Every gradient is final after that kernel, so the update of the rest coefficients is applied from the staged gradients in shared memory; the gradient is never written. Expected 2-3% (about 0.14 ms per step). - Armed from Python (GaussianModel.prepare_fused_rest_adam,GS_FUSED_REST_ADAM=1) on the iterations that take an ordinary optimizer step: not on densification iterations (densify_and_prune replaces the parameters, so the ordinary step skips them there) and not on the last iteration (no step). On those, the ordinary DenseAdam path applies. - Update arithmetic: the same expressions asadamElementinadam_dense.cu; the bias corrections of the iteration are taken from the optimizer's step counter (t = steps so far + 1). - Verification plan: a 4-iteration training run with and without the switch (/tmp/r13/fused_check.py), comparing the rest and DC parameters (expected at the run-to-run noise level); thentime,check --quick, a full check. - Verification of the fused rest update (/tmp/r13/fused_exact.py, test-only build/tmp/dev21, which also writes the gradient the fused update used): the fused parameters agree with the exact first Adam step computed from that same gradient to 1.4e-9 absolute (updates up to 1.25e-4; float rounding), so the indexing and arithmetic are correct. The earlier unfused-vs-fused comparison was dominated by run-to-run atomics noise: two unfused runs differ from each other by as much as fused and unfused do (Adam's first step moves a parameter by about lr for any non-zero gradient, however small). Equivalence therefore rests on this exact test, not on the multi-run comparison. - Fused rest-SH Adam, first version (/tmp/dev20,/tmp/r13/time_restfused): regression, total 182.6 s (windows 58.1 / 63.4 / 61.1 s, speedup 1.6725). The fused loop read and wrote each element in turn, so the global loads were serialised behind the stores.check --quick: PASS. - Fused rest-SH Adam, restructured (four elements per thread gathered before any store,__restrict__on the Adam arrays; the same per-element expressions;/tmp/r13/time_restfused2): **total 159.6 s, windows 43.6 / 59.5 / 56.6 s, speedup 1.9131** (the committedee9d83din the same kind of run: 164.0 s, 1.8625).check --quick: PASS. - Full check of this candidate:/tmp/check-8(started 17:59:06 UTC, tree/tmp/dev20, switchesGS_DENSE_ADAM=1, GS_FUSED_NORMALIZE=1, GS_FUSED_REST_ADAM=1): pending. Promotion to/submissiononly after PASS. - **Full check of the fused rest-SH candidate (/tmp/check-8, 17:59-18:05 UTC): PASS, tier N1 (equivalent), speedup 1.9118** (windows 43.9 / 59.1 / 56.7 s). Gate 2: exact items OK. Gate 3 (seed 1): PSNR +0.019 dB (bound 0.117), SSIM +0.00017 (bound 0.00069), LPIPS -0.00035 (bound 0.0010), log count +0.0023 (bound 0.0163). - Committedbf03379(promotion of the candidate; the submission tree is byte-identical to the checked tree/tmp/dev20)./submission/switches.json= `{"environment": {"GS_DENSE_ADAM": "1", "GS_FUSED_NORMALIZE": "1", "GS_FUSED_REST_ADAM": "1"}}`.
4.1 hRound 15 (in progress): profile of `bf03379` and the vectorised fused rest update (`/tmp/dev22`)
- Profile (/tmp/r15/profile, steps 7000-7019, 5.57 ms kernel time per step): backward rasterizer 1.72 ms (31%), forward rasterizer 1.41 ms (25%), backward SH preprocess with the fused rest Adam 0.58 ms (10%; 0.21 ms before the fusion, so the fused update costs about 0.37 ms), dense Adam for the other groups 0.16 ms (3%), radix sort 0.23 ms (4%), forward preprocess 0.14 ms (2.5%), SSIM 0.15 ms (2.4%), densification about 0.1 ms per step amortised, small kernels about 0.2 ms. - Change: the fused rest update reads and writes its parameters, moments and staged gradients asfloat4(the block range starts on a 16-byte boundary: 256 x 45 floats), two float4 units per thread per iteration, a scalar tail; the per-element expressions are unchanged. Tree:/tmp/dev22, sequence/tmp/r15/run_v.sh. Expected 1-2%. - Vectorised fused rest update (/tmp/dev22,/tmp/r15/time_v4):time158.7 s, windows 43.5 / 58.8 / 56.4 s, speedup 1.924 (committedbf03379: 159.6 s, 1.9131). About 0.5%, consistent in all three windows;check --quick: PASS. - Exposure optimizer on the dense Adam kernel (GS_DENSE_EXPOSURE=1,/tmp/dev23,/tmp/r15/time_v5, bundled with the vectorised update):time159.0 s, speedup 1.9203. No measurable gain over the vectorised update alone (158.7 s), so the switch is not used. Not committed. - Full check of the vectorised update (/tmp/check-9, started 18:20 UTC): pending; promotion only if PASS. - Full check of the vectorised update (/tmp/check-9, 18:20-18:27 UTC): PASS, tier N1 (equivalent), speedup 1.9295 (windows 43.4 / 58.6 / 56.3 s). Gate 2: exact items OK. Gate 3 (seed 1): PSNR +0.031 dB (bound 0.117), SSIM -0.00001 (bound 0.00069), LPIPS -0.00035 (bound 0.0010), log count +0.0030 (bound 0.0163). - Committed84d6fdc(submodules/diff-gaussian-rasterization/cuda_rasterizer/backward.cuonly; the submission tree is identical to the checked tree/tmp/dev22).check --quickof the committed tree: running (/tmp/r15/quick_submit_v4).
4.5 hRound 16 (in progress): profile of `84d6fdc` and a measurement of the forward blend's wasted rounds (`/tmp/dev24`)
- Profile (/tmp/r16/profile, steps 7000-7019, 5.54 ms kernel time per step): backward rasterizer 1.71 ms (31%), forward rasterizer 1.41 ms (25%), backward SH preprocess with the fused rest update 0.56 ms (10%), radix sort 0.22 ms (4%), dense Adam for the other groups 0.16 ms (3%), forward preprocess 0.14 ms (2.5%), SSIM 0.13 ms (2.4%), densification gathers about 0.09 ms per step amortised (1.7%), small kernels about 0.2 ms. - Question: the forward blend's block loops over a tile's list until all of its 256 pixels are done, so a warp whose own pixels are all done still runs the rounds. Measured in a test-only copy (/tmp/dev24, counters inrenderCUDA, read by/tmp/r16/round_test.py): the number of warp-rounds with an active lane against the block-rounds. If the warp-level exit would save much, a warp-level loop is worth a restructure; otherwise the forward has no such waste. - The first instrumented build of/tmp/dev24failed (a duplicated test-only counter declaration; fixed). Rebuild running (/tmp/r16/build24b.log); then/tmp/r16/round_test.pyrenders once and prints the counters. - Measured (/tmp/r16/round_test.pyon the instrumented copy, one render of the synthetic scene): warp-rounds with an active lane 84,854 against 716,800 / 8 = 89,600 block-rounds, so on average about 0.95 of the 8 warps of a block are active in a block-round. A warp with no live pixel already leaves its inner loop at once, so the warp-level exit the hypothesis was about is already in effect. The remaining cost is the active warp itself: it runs a full 256-pair batch for however many of its 32 pixels are still live. A lane-level count (active lanes per active warp-round) is being measured next, which bounds what a compaction of live pixels could save. - Lane-level measurement (same copy, second counter): 2,557,078 active lanes over 84,854 active warp-rounds, i.e. 30.1 of 32 lanes active per active warp-round (94% efficient). The forward blend has no lane-level waste to compact; round 16 ends with no change to the submission.
4.7 hRound 17 (diagnosis, no change): where the backward's time goes
- Measured on the instrumented copy (/tmp/r16/bw_test.py, one forward and one backward of the synthetic scene): contributing (splat, pixel) pairs 144,028,408 against 579,537,952 lane-steps executed by the backward warps, so **24.9% lane efficiency**. Three quarters of the backward's lane-steps evaluate a pair that does not contribute (alpha below 1/255, beyond the pixel's last contributor, or outside the image). - Why it is structural: in the bucket wavefront, lane k handles splat k of a 32-splat bucket at pixel (step - k), so every step pairs 32 different splats with 32 different pixels. The gradient body of a step runs for the whole warp whenever one lane contributes, and with about a quarter of the pairs contributing, that is nearly every step. Changing the pixel-to-thread mapping does not change this (the diagonal pairs splats with pixels, not with a patch). - The only remedy is a different backward: per-pixel chains over each pixel's contributing splats in depth order, which needs a compacted list per pixel (or per tile). That is a rewrite of the reference's bucketed backward and was not attempted; it is the one large lever left (the backward is 31% of the step). - Forward: active lanes 94% per active warp-round (round 16). Its contributing fraction within a batch is not measured; the same SIMT effect applies to its blend body, so a similar restructure would be needed there too. - Round 17 conclusion: no change to the submission (84d6fdc, full check speedup 1.9295, quick check PASS).
5.0 hRound 18: warp-level culling of the forward blend (committed `9a4353b`)
- Times (UTC):timerun 18:55-18:58 (/tmp/r18/time_fwdcull); quick check 18:58 (/tmp/r18/quick_fwdcull); full check 19:00-19:07 (/tmp/r18/check). - Profile: the round 16 profile of84d6fdc(/tmp/r16/profile, steps 7000-7019, 5.54 ms of kernel time per step): forwardrenderCUDA1.41 ms (25%), backwardPerGaussianRenderCUDA1.71 ms (31%), backward preprocess with the fused rest update 0.56 ms (10%), radix sort 0.25 ms (4%), dense Adam 0.17 ms (3%), forward preprocess 0.14 ms (2.5%), SSIM 0.13 ms (2.4%). - Hotspot: the forward blend. Each warp stepped through every entry of every 256-entry batch for its 32 pixels, and each entry costs the power and exp body whether or not any lane of the warp contributes. - Change: per bucket of 32 entries, a ballot of the entries whose opacity factor can fall below ln(255 * opacity) at some pixel of the warp's 16 x 2 strip. The test is the exact minimum of the factor over the strip's closed box (zero if the mean is inside, otherwise the least of the four edge minima, each a clamped one-dimensional minimum) with a margin of 0.01. The blend loop visits only the kept entries. Bucket checkpoints are still written every 32 entries, and last_contributor is the entry index plus one, as before. - Why an exact minimum: the existingmax_contrib_power_rect_gaussian_floatis not a strict lower bound. In 1,088 of 60,000 random strip and tile cases it exceeds the true pixel minimum, by up to 2.5e-3 in factor units (/tmp/r18/rect_test.py). The new test has 0 violations in 120,000 cases (/tmp/r18/exact_rect_test.py). The tile culling induplicateWithKeyskeeps the existing bound, so the pair set of the reference is unchanged. - Verification: forward image bitwise identical to the previous build on a synthetic 200,000-splat render (/tmp/r18/exact_test.py), radii equal. Backward gradients differ between the builds by the same amount as two runs of one build (atomics).check --quick: PASS, exact items OK over 3,000 steps. -time: 146.6 s, windows 40.3 / 54.2 / 52.0 s,speedup_vs_frozen_reference2.0837 (previous 158.7 s, windows 43.4 / 58.6 / 56.3 s, 1.924): 7.6% less time. - Full check (/tmp/r18/check): PASS, speedup 2.0805, tier N1 (equivalent), exact items OK. Gate 3, seed 1: PSNR +0.030 dB (bound 0.117), SSIM -0.00010 (bound 0.00069), LPIPS -0.00002 (bound 0.0010), log count +0.0023 (bound 0.0163). - Committed9a4353b(onlyforward.cu). The submission tree is identical to the checked tree/tmp/dev25.
5.6 hRound 19: backward wavefront over the pixels that can contribute (N1, committed `ca7674a`)
- Times (UTC):timerun 19:21-19:26 (/tmp/r19/time_rb); quick check 19:27-19:28 (/tmp/r19/quick); full check from 19:29 (/tmp/r19/check, speedup and gate values below). - Profile:/tmp/r19/profileat9a4353b(steps 7000-7019, 5.10 ms of kernel time per step): backwardPerGaussianRenderCUDA1.72 ms (34%), forwardrenderCUDA0.95 ms (19%), backward preprocess with the fused rest update 0.56 ms (11%), radix sort 0.23 ms (4%), dense Adam 0.17 ms (3%), forward preprocess 0.14 ms (3%), SSIM 0.13 ms (2.5%). - Hotspot: the backward wavefront. Each 32-splat bucket ran 287 steps over all 256 pixels of its tile, and for a pixel whose last contributor lies before the bucket, every step is wasted. - Change: the forward sorts each tile's 256 pixels by last contributor withcub::BlockRadixSort(descending) and writes the rank -> pixel order (tile_pix_order) and the sorted keys (tile_lc_sorted). A bucket computesn_active, the number of ranks whose last contributor lies beyond its first splat (ballots overtile_lc_sorted), and runsn_active + 31steps. Staging loads pixels by rank, and each lane carries its pixel index along the shuffle chain (p_lane, one extra shuffle per step). The per-pixel inputs and the per-pair arithmetic are unchanged. - Why N1: the per-lane sums over pixels now run in rank order, so float reassociation changes the gradients slightly. The forward image is bitwise identical. - Build fault and fix: the first build faulted (illegal shared writes, found with compute-sanitizer,/tmp/r19/sanitizer.log). cub'sBlockRadixSort<uint32_t, BLOCK_SIZE, ...>derives the thread id from its X dimension only, which is wrong for the 16 x 16 block. Declared withBLOCK_DIM_X = BLOCK_XandBLOCK_DIM_Y = BLOCK_Y. - Open observation (not changed): the existingcub::BlockReduce<uint32_t, BLOCK_SIZE>formax_contribhas the same 1-D thread-id assumption (RowMajorTidignores the Y dimension whenBLOCK_DIM_Y == 1). It is the reference's behavior, so it stays. Whethermax_contribis computed over the whole tile has not been verified; it only sets the early exit of buckets. - Verification: forward image bitwise equal to the previous build (synthetic 200,000-splat render,/tmp/r18/exact_test.py,/tmp/r19/sub_9a.ptagainst/tmp/r19/dev27_b.pt), radii equal. Gradient differences are within the spread of the previous build's own runs (max relative 3e-5 to 1e-4 between runs 2, 3 and the new build; run 1 differs from all later runs by the same order). -check --quick: PASS, exact items OK over 3,000 steps. -time: 143.74 s, windows 40.26 / 53.21 / 50.27 s,speedup_vs_frozen_reference2.1246 (previous 146.6 s, windows 40.3 / 54.2 / 52.0 s, 2.0837): 2.0% less time. The gain is smaller than the estimate in the profile discussion. - Full check (/tmp/r19/check): PASS, speedup 2.1254, tier N1 (equivalent), exact items OK. Gate 3, seed 1: PSNR -0.040 dB (bound 0.117), SSIM +0.00005 (bound 0.00069), LPIPS -0.00024 (bound 0.0010), log count +0.0023 (bound 0.0163). - Committedca7674a(six rasterizer files). The submission tree is identical to the checked tree/tmp/dev27.
5.8 hRound 20: forward culling test with per-entry constants computed once per batch (committed `9de7de0`)
- Times (UTC):timerun 19:39-19:43 (/tmp/r20/time_fc, includes the candidate compile); full check 19:44-19:50 (/tmp/r20/check). - Profile:/tmp/r20/profileatca7674a(steps 7000-7019, 5.01 ms of kernel time per step): backwardPerGaussianRenderCUDA1.59 ms (31.6%), forwardrenderCUDA1.01 ms (20.2%; the per-tile sort of round 19 adds about 0.065 ms to it), backward preprocess with the fused rest update 0.56 ms (11.2%), radix sort 0.23 ms (4.5%), dense Adam 0.17 ms (3.3%), forward preprocess 0.14 ms (2.8%), SSIM 0.13 ms (2.7%), torch fills 0.10 ms (1.9%), densification gathers about 0.10 ms per step amortised (1.9%). - Hotspot: the forward's warp-level culling test. Each of the block's 8 warps recomputed ln(255 * opacity) and two reciprocals for each of the 256 batch entries, so the per-entry constants were computed eight times over. - Change: the constants (co.x, co.z, co.y / co.x, co.y / co.z, co.y, and the threshold ln(255 * opacity) + 0.01) are computed once per batch while the batch is fetched, into shared memory (collected_strip_a,collected_strip_b). The test reads them. The arithmetic of each decision is the same expression as before. - Verification: forward image bitwise equal to theca7674abuild (synthetic 200,000-splat render,/tmp/r20/ca_a.ptagainst/tmp/r20/dev28_a.pt), radii equal. Gradient differences are the backward's atomics noise (max relative 2.4e-5 to 2.8e-5). -time: 142.68 s, windows 39.72 / 52.92 / 50.04 s, speedup 2.1405 (previous 143.74 s, windows 40.26 / 53.21 / 50.27 s, 2.1246): 0.75% less time, smaller in every window. - Full check (/tmp/r20/check): PASS, speedup 2.1294 (windows 40.06 / 53.14 / 50.22 s), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR -0.090 dB (bound 0.117), SSIM -0.00020 (bound 0.00069), LPIPS +0.00001 (bound 0.0010), log count +0.0024 (bound 0.0163). - Committed9de7de0(forward.cuonly). The submission tree is identical to the checked tree/tmp/dev28. - Checked and not changed:dL_dshis not zero-filled in the judge's configuration (the rest coefficients are 15, so3 * M = 45takes theemptybranch inrasterize_points). The torch fills (about 0.10 ms per step) come from the other backward tensors that are accumulated with+=or never written for culled Gaussians. Removing them would save at most about 1%.
6.0 hRound 21 (no gain, not committed): backward per-pixel constants read per lane from a ring (`/tmp/dev29`)
- Times (UTC):timerun 19:56-20:01 (/tmp/r21/time_ring). Build/tmp/gs-ext/899bc0b5b2b1b35a202fab80. - Hotspot and hypothesis: the backward kernel (31.6% of the step atca7674a,/tmp/r20/profile) carries 12 values through the lane pipeline per wavefront step (shfl_up), and lane 0 reloads 11 staged values per step. The hypothesis was that the shuffle traffic dominates the step. - Change: the per-pixel constants of a rank (last contributor, T_final, dL_dpixel x3, dL_invdepth, pixel index) are staged for each 32-rank chunk in a 64-slot ring (rank r in slot r & 63), and each lane reads the constants of its own rank from it. Only the states (T, ar x3, ard) keep the shuffle chain, so the shuffles fall from 12 to 5 per step. The arithmetic is unchanged (patch/tmp/r21/patch_ring.py). - Verification: forward image and radii bitwise equal to9de7de0; gradients of xyz, DC, rest, opacity and scale within 1.3e-7 (max relative) of the9de7de0build. The rotation gradient differs by the same run-to-run amount as between theca7674aand9de7de0builds, whose backward kernels are identical (rms relative about 9 for rotations, pre-existing). -time: 144.12 s, windows 40.22 / 53.45 / 50.45 s, speedup 2.1190, against 142.68 s and 2.1405 at9de7de0(windows 39.72 / 52.92 / 50.04 s). That is about 1% slower in every window. Regression, not promoted. The submission stays at9de7de0. - Conclusion: the shuffle chain is not the limit of the wavefront. The per-lane shared-memory reads sit on the step's critical path, and their address arithmetic and latency cost more than the seven shuffles they replace. The next candidate works on the forward instead.
6.2 hRound 22 (no gain, not committed): forward warp blocks of 8 x 4 pixels (`/tmp/dev30`)
- Times (UTC):timerun 20:06-20:11 (/tmp/r22/time_84); build/tmp/gs-ext/feef8a9a15be6333b9a76906. - Change (forward.cu only): each warp covers an 8 x 4 pixel block of the tile instead of two rows of 16, so the warp's culling box is more compact. The linear pixel index (row-major) stays the index of the bucket checkpoints and of the sort values, so the backward is unchanged. The culling boxes are 8 x 4 with the same exact minimum. - Verification: forward image and radii bitwise equal to9de7de0. The first version gave gradients off by up to 1e-3: the sort is stable, and the thread order had changed the order of ties (pixels with equal last contributor). Feeding the sort through a shared-memory transpose in pixel order restored the tie order; the gradients then differ by about 2.5e-5, the atomics noise. -time: 143.29 s, windows 39.92 / 53.14 / 50.23 s, speedup 2.1313, against 142.68 s and 2.1405 at9de7de0. About 0.4% slower in every window. No gain, not promoted. The compact boxes did not reduce the kept entries enough to pay for the other mapping costs.
6.4 hRound 23: forward sort with the key bits of the tile length (exact, committed `90f9198`)
- Times (UTC):timerun 20:15-20:18 (/tmp/r23/time_bits); full check 20:19-20:27 (/tmp/r23/check). - Profile used:/tmp/r20/profile(headca7674a). The forward's per-tile block radix sort of the 256 pixels by last contributor (about 65 us per step, added in round 19) ran 8 radix passes over full 32-bit keys. - Change: a last contributor is at most the length of its tile's list, so the sort takes only the bits of that length (SortDescending(keys, values, 0, key_bits),key_bitsfrom__clzof the tile length). The sort is stable, so the order is the same, with fewer radix passes. The change is inforward.cuonly. - Verification: forward image and radii bitwise equal to9de7de0(synthetic 200,000-splat render,/tmp/r23/dev31_a.ptagainst/tmp/r20/dev28_a.pt). Gradients differ by about 2e-7 relative (atomics noise). -time: 141.59 s, windows 39.47 / 52.50 / 49.62 s, speedup 2.1569, against 142.68 s and 2.1405 at9de7de0(windows 39.72 / 52.92 / 50.04 s). 0.8% less time, faster in every window. - Full check (/tmp/r23/check): PASS, speedup 2.1446 (candidate 142.401 s; windows [39.693, 52.829, 49.879]), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR -0.086 dB (bound 0.117), SSIM -0.00015 (bound 0.00069), LPIPS -0.00037 (bound 0.0010), log count +0.0007 (bound 0.0163). - Committed90f9198(forward.cuonly). The submission tree is identical to the checked tree/tmp/dev31.
6.6 hRound 24 (no gain, not committed): fused-SSIM kernels with 16 x 16 blocks (`/tmp/dev32`)
- Times (UTC):timerun 20:28-20:32 (/tmp/r24/time_ssim16); same-sessiontimeof HEAD90f919820:33-20:37 (/tmp/r24/time_head_now). - Change (submodules/fused-ssim/ssim.cuonly):BXandBY32 -> 16, so each SSIM block handles a 16 x 16 tile and more blocks fit per SM (the kernels run one block per tile with about 20 barriers per channel). - Verification (/tmp/r24/ssim_unit.py, one fixed random pair at the training resolution): the SSIM value is bitwise equal to the HEAD build; the gradient differs by 4.5e-13 absolute (about 8e-8 relative, rounding only). -time: 142.44 s, windows 39.94 / 52.68 / 49.82 s, speedup 2.1439. HEAD in the same session: 142.41 s, windows 39.78 / 52.75 / 49.88 s, speedup 2.1444. The difference is 0.02%, inside the run-to-run noise (about 0.5%: HEAD measured 141.6 s in round 23 and 142.4 s in its full check). No gain, not promoted. - Conclusion: the SSIM kernels are not the limit at this size; more blocks in flight do not change their time measurably.
6.8 hRound 25: backward colour and depth terms regrouped (N1, committed `798f78c`)
- Times (UTC):timerun 20:38-20:41 (/tmp/r25/time_body); same-sessiontimeof HEAD90f919820:33-20:37 (/tmp/r24/time_head_now); full check 20:43-20:52 (/tmp/r25/check). - Profile used:/tmp/r20/profile(the backward is 31.6% of the step, about 90 instructions per active lane-step in the pair body, of which the per-channel colour and depth terms are a large part). - Change (backward.cu only): the terms of dL/dalpha regroup exactly, with q = 1 / (1 - alpha), into dL/dalpha = T (c . dL + invd dL_inv) + q (ar . dL + ard dL_inv - T_final bg . dL). The channel loop accumulates the dot products c . dL, ar . dL and bg . dL; the depth term and the background term are then one expression. The order of the state updates (ar, Register_dL_dcolors, ard) is unchanged. - Verification: forward image and radii bitwise equal to9de7de0(/tmp/r25/dev33_a.ptagainst/tmp/r20/dev28_a.pt). Gradients differ by about 2.4e-7 (max relative) and rms about 2e-6, reassociation level. -time: 141.90 s, windows 39.60 / 52.59 / 49.71 s, speedup 2.1522. HEAD90f9198in the same session: 142.41 s, windows 39.78 / 52.75 / 49.88 s, speedup 2.1444. About 0.36% less time, faster in every window. - Full check (/tmp/r25/check): PASS, speedup 2.1533 (candidate 141.823 s; windows [39.607, 52.569, 49.647]), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR -0.005 dB (bound 0.117), SSIM -0.00004 (bound 0.00069), LPIPS -0.00007 (bound 0.0010), log count +0.0024 (bound 0.0163). - Committed798f78c(backward.cuonly). The submission tree is identical to the checked tree/tmp/dev33.
7.1 hRound 26: forward blend reads per-entry colour and inverse depth from shared memory (exact, committed `3c06e63`)
- Times (UTC): pairedtimeruns 20:52-21:00 (/tmp/r26/time_stagethen/tmp/r26/time_head); full check 20:59-21:07 (/tmp/r26/check). - Hotspot: the blend body (the forward is 20% of the step). Each contributing lane read the colour features and the depth of the pair's Gaussian from global memory (random, dependent loads inside the loop), and divided by the depth again for every contributing pair. - Change (forward.cu only): the batch fetch stages the three features of every entry (collected_feat) and the reciprocal of its depth (collected_invd) in shared memory, computed once per entry. The blend reads them. The values and the order of the operations per pair are the same expressions as before. - Verification: forward image and radii bitwise equal to798f78c(/tmp/r26/dev34_a.ptagainst/tmp/r25/dev33_a.pt). The gradients differ by the run-to-run atomics spread (max relative 2e-4 to 1e-3, the same magnitude as between two renders of one build). -time: paired in one chain: 134.05 s (speedup 2.2781, windows 37.73 / 49.48 / 46.84 s) against 141.38 s for HEAD798f78cin the same chain (speedup 2.1601, windows 39.39 / 52.46 / 49.52 s): 5.2% less time, faster in every window. - Full check (/tmp/r26/check): PASS, speedup 2.2791 (candidate 133.998 s; windows [37.794, 49.445, 46.759]), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR -0.044 dB (bound 0.117), SSIM +0.00029 (bound 0.00069), LPIPS -0.00034 (bound 0.0010), log count -0.0033 (bound 0.0163). - Note on the order of the paired runs: the staged build ran first in the chain. The full check times the candidate against the frozen reference on its own and gives the same candidate time (133.998 s against 134.054 s in the chain), so the order does not explain the gain. The candidate is 5.5% faster than round 25's full check (141.823 s). - Committed3c06e63(forward.cuonly). The submission tree is identical to the checked tree/tmp/dev34.
7.2 hRound 27 (inconclusive diagnostic, not committed): cost of the scattered staging loads of the backward (`/tmp/dev35`)
- Times (UTC): diagnostictimerun 21:02-21:08 (/tmp/r27/time_diag); build/tmp/gs-ext/433f37fbac72541ebfa22d0d. - Question: since round 19 the backward stages each 32-rank chunk with loads at scattered pixel positions (the rank order of the tile). By a count of cache-line requests per chunk, that is about 200 requests against about 30 in the linear order. The question was whether this costs a measurable part of the backward. - Diagnostic (never to be submitted): every staged pixel load of the backward was redirected to sequential addresses. The outputs are then wrong, the training diverges and does far more densification work (windows 43.5 / 106.3 / 105.1 s, total 254.9 s, speedup 1.198). The timing therefore does not measure the staging cost. - Conclusion: inconclusive. A valid measurement needs a correct rank-ordered gather (for example, a kernel that writes the pixel data of each tile in rank order once per pass, so that the backward's staging reads are sequential). That is a larger change. The submission stays at3c06e63.
7.4 hRound 28 (no gain, not committed): rank-ordered records for the backward's staging (`/tmp/dev36`)
- Times (UTC): pairedtimechain 21:18-21:25: HEAD3c06e63first (/tmp/r28/time_head), then the record build (/tmp/r28/time_rec); build/tmp/gs-ext/c446065e331cc064dd2ae73b. - Change: a kernelgatherTileRecordswrites, once per backward pass, the per-pixel data of each tile in rank order (the final transmittance, the inverse depth, the inverse depth gradient, the three colour channels and their gradients). The staging ofPerGaussianRenderCUDAthen reads consecutive ranks of these records, and the last contributor comes fromtile_lc_sorted, which is already in rank order. The values read are the same, so the staging is the same computation with sequential loads. Checkpoint loads stay indexed by pixel. - Verification: forward image and radii bitwise equal to3c06e63(/tmp/r26/dev34_a.ptagainst/tmp/r28/dev36_a.pt). Gradients differ by 2-5e-5 (max relative), the atomics noise. -time: HEAD3c06e63: 134.32 s, windows 37.75 / 49.65 / 46.92 s. Record build: 147.65 s, windows 51.01 / 49.68 / 46.96 s. The second and third windows are the same for both builds, so the sequential staging gives no steady-state gain. The first window of the record build was 13 s slower; no cause is visible in the logs (the workload is the same, and the forward is bitwise identical). No gain, not promoted. - Conclusion: the scattered staging loads are not what limits the backward. The round 27 diagnostic (inconclusive) and this measurement point the same way. The submission stays at3c06e63.
7.4 hCurrent summary (updated after round 28; HEAD `3c06e63` unchanged since round 26)
- Submission:/submission/gaussian-splattingat3c06e63(branchopt, clean). Last full check:/tmp/r26/check, PASS, speedup 2.2791 (candidate 133.998 s), tier N1, exact items OK./submission/switches.json= `{"environment": {"GS_DENSE_ADAM": "1", "GS_FUSED_NORMALIZE": "1", "GS_FUSED_REST_ADAM": "1"}}`. - Gains by change (timespeedups; full-check speedups in brackets), in order: the earlier changes (see the gains table in the earlier summary), then the forward culling test with per-entry constants (9de7de0, 2.1405 [2.1294]), the forward sort with the tile's key bits (90f9198, 2.1569 [2.1446]), the backward regrouping (798f78c, 2.1522 [2.1533]), and the forward blend's shared-memory staging of colour and inverse depth (3c06e63, 2.2781 [2.2791]). - No gain (not promoted): the backward ring for the per-pixel constants (round 21); the forward 8 x 4 blocks (round 22); the fused-SSIM 16 x 16 blocks (round 24); the inconclusive staging diagnostic (round 27); the rank-ordered records for the backward's staging (round 28: the second and third windows equal HEAD). - Remaining ideas (all below about 1%, or unmeasured): the zero-fills of the rasterizer backward's gradient tensors; the densification gathers (about 0.9% of the run); dense Adam for the remaining groups (about 1%); further exact regroupings of the backward's per-pair arithmetic (conic accumulators). The backward is about 31% of the step and its per-pair structure limits the remaining gains. - Measurement practice: time paired runs in one chain, HEAD first and the candidate first in turn when the difference is below about 1%; the full check is the verdict and its candidate time is a second sample.
7.6 hRound 29 (no gain, not committed): software-pipelined batch fetch in the forward (`/tmp/dev37`)
- Profile first (/tmp/r29/profile, HEAD3c06e63, steps 7000-7019, 4.76 ms of kernel time per step): backwardPerGaussianRenderCUDA1.573 ms (33.1%), forwardrenderCUDA0.774 ms (16.3%; it was 1.014 ms before the blend's shared-memory staging of round 26), backward preprocess 0.560 ms (11.8%), radix sort 0.225 ms (4.7%), dense Adam 0.167 ms (3.5%), forward preprocess 0.139 ms (2.9%), SSIM 0.130 ms (2.7%). - Hypothesis: the forward's per-batch fetch (four or five dependent random global loads per thread, then a barrier) is on the critical path. The change issues the loads of batch i + 1 into registers during the blend of batch i, and stores the values into shared memory at the start of the next batch. The values and the arithmetic are unchanged. - Verification: forward image and radii bitwise equal to3c06e63; gradients at the atomics noise. -time(paired chain, HEAD first): HEAD3c06e63134.25 s (speedup 2.2747, windows 37.78 / 49.58 / 46.90 s); pipelined forward 134.08 s (speedup 2.2777, windows 37.81 / 49.46 / 46.81 s). The difference is 0.13%, inside the noise. **No gain, not promoted.** - Conclusion: the batch fetch's latency is already hidden by the other blocks on each SM; the round 26 gain came from the per-pair loads inside the blend, which the batch staging removed.
7.8 hRound 30 (no gain, not committed): fused-SSIM without the scratch flushes and without zero-filled outputs (`/tmp/dev38`)
- Times (UTC): pairedtimechain 21:40-21:47 (/tmp/r30/time_head, then/tmp/r30/time_ssim); build/tmp/gs-ext/b3b685432610dfeca2a586cc. - Change (submodules/fused-ssim/ssim.cuonly): the eight flushes of the separable convolution's scratch buffer are removed (each pass writes every entry the next pass reads, so the zeroing and its barrier were redundant), and the five SSIM outputs are allocated without a zero fill (the kernels write every element they return). - Verification (/tmp/r24/ssim_unit.py): the SSIM value and the gradient are bitwise equal to the original kernels. -time: HEAD3c06e63134.16 s (speedup 2.2763, windows 37.69 / 49.56 / 46.91 s); SSIM variant 134.15 s (speedup 2.2765, windows 38.29 / 49.28 / 46.58 s). The difference is 0.01%; the windows net out to zero. No gain, not promoted.
7.9 hRound 31 (regression, not committed): launch bounds of four blocks per SM for the backward (`/tmp/dev39`)
- Profile context:PerGaussianRenderCUDAuses 80 registers per thread (cuobjdump --dump-resource-usageof the round 26 build), so at 256 threads per block only 3 blocks (24 warps) fit per SM.renderCUDAuses 40 registers. - Change:__launch_bounds__(BLOCK_SIZE, 4)onPerGaussianRenderCUDA(64 registers, a 32-byte stack frame, so some values spill). - Verification: forward image and radii bitwise equal to HEAD; gradients differ by 4.5e-5 to 8.7e-5 (max relative). -time(paired, HEAD first): HEAD3c06e63133.90 s (speedup 2.2808); launch-bounds build 139.42 s (speedup 2.1904), about 4% slower in every window. Regression, not promoted. The spills cost more than the extra blocks gain.
8.2 hRound 32: per-Gaussian backward kernel with one warp per block (N1, committed `b3eec74`)
- Times (UTC): pairedtimechain 22:00-22:04 (/tmp/r32/time_w1first, then/tmp/r32/time_head); full check 22:07-22:14 (/tmp/r32/check). - Hypothesis: the per-Gaussian backward kernel ran eight buckets per block of 256 threads. A block holds its slot until its slowest bucket finishes, and buckets of one tile differ in length, so warps that finish early hold resources that could hold work. The register footprint (80 per thread, see round 31) limits the kernel to three blocks per SM, so the idle slots lower the warps that hide latency. - Change (backward.cuonly): the per-Gaussian kernel is launched with 32-thread blocks (BWD_WARPS= 1), so that each bucket retires with its own block. The staging arrays are sized by the warps of the block. The arithmetic per bucket is unchanged. - Verification: forward image and radii bitwise equal to HEAD3c06e63; gradients differ by 3e-5 to 4e-5 (max relative), the atomics noise. -time: paired in one chain (the candidate first): 132.80 s, speedup 2.2996, windows 37.35 / 49.09 / 46.37 s, against 134.31 s (speedup 2.2739, windows 37.69 / 49.65 / 46.97 s) for HEAD3c06e63. About 1.1% less time, faster in every window. - Full check (/tmp/r32/check): PASS, speedup 2.2968 (candidate 132.966 s; windows [37.365, 49.142, 46.459]), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR -0.007 dB (bound 0.117), SSIM -0.000006 (bound 0.00069), LPIPS -0.00023 (bound 0.0010), log count -0.0005 (bound 0.0163). - Committedb3eec74(backward.cuonly). The submission tree is identical to the checked tree/tmp/dev40.
8.4 hRound 33 (regression, not committed): 72 registers for the one-warp backward kernel (`/tmp/dev41`)
- Change:__launch_bounds__(32, 28)on the one-warp per-Gaussian kernel (72 registers, no stack frame, so no spills; the occupancy would rise from about 25 to 28 warps per SM). - Verification: forward image and radii bitwise equal to3c06e63; gradients differ by 3.6e-5 to 9.3e-5 (max relative). -time(paired, HEAD first): HEADb3eec74132.69 s (speedup 2.3016); 72-register build 137.75 s (speedup 2.2170), about 3.8% slower in every window. Regression, not promoted. A lower register budget costs more instruction-level parallelism than the extra warps gain, even without spills.
8.5 hRound 34 (no gain, not committed): per-bucket record for the backward's warp setup (`/tmp/dev42`)
- Hypothesis: each backward warp reads its bucket through a chain of about seven dependent loads (bucket to tile, tile range, bucket base,max_contrib, the last contributors, the splat index, the Gaussian data). The forward already knows a bucket's tile, first splat, index in the tile and splat count when it writesbucket_to_tile, so it now writes them as one record (bucket_info, 16 bytes per bucket) and the backward reads one record at its start. - Verification: forward image and radii bitwise equal tob3eec74(/tmp/r32/dev40_a.ptagainst/tmp/r34/dev42_a.pt); gradients differ by 1.5e-4 to 3.0e-4 (max relative), within the atomics spread seen before. -time(paired, HEAD first): HEADb3eec74133.21 s (speedup 2.2925, windows 37.40 / 49.28 / 46.53 s); bucket-record build 134.46 s (speedup 2.2713, windows 37.87 / 49.68 / 46.91 s). About 0.9% slower in every window. No gain, not promoted. - Conclusion: the start-of-warp chain is not what limits the backward at its one-warp occupancy.
8.7 hRound 35 (no gain, not committed): bounding-box fast reject in the forward's culling test (`/tmp/dev43`)
- Profile first (/tmp/r35/profile, HEADb3eec74, steps 7000-7019, 4.73 ms of kernel time per step): backwardPerGaussianRenderCUDA1.540 ms (32.6%), forwardrenderCUDA0.778 ms (16.5%), backward preprocess 0.560 ms (11.8%), radix sort 0.224 ms (4.7%), dense Adam 0.167 ms (3.5%), forward preprocess 0.139 ms (2.9%), SSIM 0.130 ms (2.8%). - Change (forward.cu only): each entry's contour {f <= T}, T = ln(255 o) + 0.01, lies within |dx| <= sqrt(2 T Mi_xx) and |dy| <= sqrt(2 T Mi_yy) (M = [[co.x, co.y], [co.y, co.z]]). The warp test first rejects the entries whose contour box does not reach the strip, then runs the unchanged exact test. The rejection is conservative, so the kept set is the same. - Verification: forward image and radii bitwise equal tob3eec74(/tmp/r32/dev40_a.ptagainst/tmp/r35/dev43_a.pt). -time(paired, HEAD first): HEADb3eec74133.42 s (speedup 2.2889, windows 37.56 / 49.36 / 46.50 s); fast-reject build 147.48 s (speedup 2.0708, windows 51.09 / 49.53 / 46.85 s). Windows 2 and 3 are 0.3% and 0.7% slower; the first window of this build was again slow (as in round 28, with no visible cause). No gain, not promoted.
8.9 hRound 36 (regression, not committed): eight blocks per SM for the forward kernel (`/tmp/dev44`)
- Change:__launch_bounds__(BLOCK_X * BLOCK_Y, 8)onrenderCUDA(32 registers, a 24-byte stack frame, so some spills). - Verification: forward image bitwise equal to HEAD. -time(paired, HEAD first): HEADb3eec74132.94 s (speedup 2.2972); eight-block forward 152.28 s (speedup 2.0055), with the second window at 64.7 s against 49.2 s. Regression, not promoted. As in round 31, the register cap costs more in spills than the extra blocks gain.
9.2 hRound 37 (rejected by gate 3, not committed): tiles in the forward launched longest-first (`/tmp/dev45`)
- Hypothesis (the forward's blocks run in waves, one block per tile): with the tiles launched in row-major order, a long tile can start late and leave the SMs idle at the end of the launch. Launching the tiles in order of decreasing list length (a cub descending sort of the list lengths, then the blocks mapblockIdx.xthroughtile_order) should shorten the tail. The outputs are per tile, so they should not change. - Verification: forward image and radii bitwise equal tob3eec74(/tmp/r32/dev40_a.ptagainst/tmp/r37/dev45_a.pt); gradients differ by 6e-5 to 1e-4 (max relative, the atomics spread). -time(paired, HEAD first): HEADb3eec74133.28 s (speedup 2.2913, windows 37.47 / 49.24 / 46.57 s); longest-first build 131.09 s (speedup 2.3296, windows 36.86 / 48.48 / 45.75 s): 1.6% less time, faster in every window. - Full check (/tmp/r37/check): INCORRECT (verdictdifferent). Gate 2 exact items OK, but gate 3 seed 1: **PSNR -0.159 dB, beyond the bound of 0.117 dB** (SSIM -0.00023 within 0.00069, LPIPS -0.00063 within 0.0010, log count +0.0012 within 0.0163). The speedup of this check is 2.3258 (candidate 131.30 s), but the candidate is rejected by the statistical gate. - Decision: not promoted; the submission stays atb3eec74. The gate result is recorded as it is. The forward itself is bitwise identical, so the difference comes from the order of the backward's atomic accumulations, which changes which trajectory the training follows; the -0.16 dB shift is larger than the other atomics-only candidates (at most about 0.09 dB in absolute value). A reordering of the tiles would have to be shown to keep the trajectory statistically equivalent before it could be used. - Addendum (diagnostic,check --n1on the same tree,/tmp/r37/n1, not a verdict): the statistical gate alone, seed 1 again, gave PASS (equivalent): PSNR -0.012 dB (bound 0.117), SSIM -0.00016 (bound 0.00069), LPIPS -0.00023 (bound 0.0010), log count +0.0087 (bound 0.0163). Two runs of identical code therefore differ in PSNR by about 0.15 dB (-0.159 and -0.012), well above the reference's seed spread (sigma 0.039 dB, bound 0.117 dB). Atomics-only variation of the training trajectory can move a single-seed endpoint across the bound. The official verdict of the full check stands (INCORRECT); the change is not promoted, because a candidate is promoted only on a full-check PASS. Evidence for the judge's owners: the one-seed statistical gate is sensitive to the run-to-run variation of the atomics for changes that reorder the backward's accumulation.
9.5 hRound 38: forward and backward launch the tiles and their buckets longest-first (N1, committed `0e726ad`)
- Times (UTC): pairedtimechain 23:19-23:24 (HEAD first:/tmp/r38/time_head, then/tmp/r38/time_combo); full check 23:26-23:33 (/tmp/r38/check). - Hypothesis (from round 37): the forward's blocks run in waves, and a long tile launched late leaves the SMs idle at the end. The backward's buckets are launched tile by tile in row-major order, so the same tail can occur there. - Change: the forward launches one block per tile in the order of decreasing list length (a cub descending sort of the list lengths; block b renders tile tile_order[b]). The backward launches its buckets in the same tile order: a binary search over the prefix sums of the bucket counts in that order maps each block to its bucket (one warp per block, as before). - Verification: the forward image and radii are bitwise equal to HEAD (/tmp/r38/dev46_a.pt); the gradients differ only by atomics noise (6e-5 max relative at the largest element). -time(paired, HEAD first): HEADb3eec74133.61 s (speedup 2.2857, windows 37.58 / 49.37 / 46.66 s); launch-order build 130.98 s (speedup 2.3316, windows 36.86 / 48.40 / 45.72 s): 2.0% less time, faster in every window. - Full check (/tmp/r38/check): PASS, speedup 2.3419 (candidate 130.404 s; windows [36.655, 48.214, 45.535]), tier N1 (equivalent), exact items OK = True. Gate 3, seed 1: PSNR +0.012 dB (bound 0.117), SSIM -0.00013 (bound 0.00069), LPIPS -0.00021 (bound 0.0010), log count +0.0074 (bound 0.0163). - Relation to round 37: the forward-only longest-first tree of round 37 was rejected by its own full check (gate 3 PSNR -0.159 dB, INCORRECT); itscheck --n1repeat passed. The round 37 tree is not promoted and its verdict is not overturned. This round's tree (forward and backward together) is a separate candidate with its own full check. - Committed0e726ad(six rasterizer files). The submission tree is identical to the checked tree/tmp/dev46.
9.8 hRound 39: profile of `0e726ad`; atomic and drain experiments (no change promoted)
- Times (UTC): profile of HEAD 23:32-23:33 (/tmp/r39/profile, steps 7000-7019, GPU 93.0 ms in the 20 steps; the window holds one densification event, about 9 ms of gathers and index ops); conic-atomic experiment 23:38-23:43 (/tmp/dev47,/tmp/r39/profile_dev47); drain diagnostic 23:48- (/tmp/dev48,/tmp/r39/profile_dev48). - Profile of HEAD:PerGaussianRenderCUDA(backward blend) 30.25 ms (32.5 %, 1.51 ms per step);renderCUDA(forward blend) 14.05 ms (15.1 %, 0.70 ms); backwardpreprocessCUDA(SH and covariance backward with the fused rest-SH Adam) 11.20 ms (12.0 %, 0.56 ms; about 1.5 KB of traffic per Gaussian, close to the memory floor); forward preprocess 2.79 ms (3.0 %); radix sorts 4.9 ms (5.2 %); DenseAdam 3.3 ms (3.6 %); fused SSIM 2.7 ms (2.9 %). - Experiment 1 (/tmp/dev47, rejected, no gain): the three scalaratomicAdds ondL_dconic2Dreplaced by onefloat4atomic (sm_90 vector atomic; the unused z slot stays zero). Built;profile:PerGaussianRenderCUDA30.551 ms against 30.247 ms (+1 %, within noise). The number of scalar atomics per splat is therefore not the limit; packing mean2D, opacity and colour into vector atomics is not pursued. - Reading of the backward (backward.cu:541-805): one warp per bucket; the pixels of the tile enter lane 0 one per step and pass the 32 splats of the bucket (lane = splat); a bucket runsn_active + 31steps, and the 31 fill and 31 drain steps have partly idle lanes. Per step: 12shfl_up(7 of them carry per-pixel constants that do not change along the pipeline), the lane-0 staging path, and about 100 instructions of body work. - Reading of the forward (forward.cu:396-481): already warp-culled (kept entries as a ballot mask); about 40 issue slots per kept entry and warp; no structural waste beyond the tail of each warp. - Experiment 2 (/tmp/dev48, diagnostic only, output wrong): the loop bound changed fromn_active + 31ton_active, dropping the 31 drain steps of every bucket. The time difference bounds what chaining consecutive buckets in one warp (no drain or fill between them) could save. - Result of experiment 2 (/tmp/r39/profile_dev48,profile23:50):PerGaussianRenderCUDA27.331 ms against 30.247 ms (-9.6 %, per step 1.367 ms against 1.512 ms). The forward also changed (-5 %) because the wrong gradients changed the trajectory and the workload of the window, so the number is an estimate. Reading: the 31 drain steps of a bucket cost about 10 % of the backward kernel. Chaining K consecutive buckets in one warp (the pixel wave of the next bucket enters lane 0 as the previous one leaves lane 31; each lane switches its splat when its last pixel of the bucket has passed) would leave one fill and one drain per chain, about (1 - 1/K) of that overhead: roughly 8 % of the backward kernel at K = 8, or about 3 % of a training step. This is the most promising structural change found so far; it is not started (per-lane switches, flushes at staggered steps, and a prefetch of the next bucket's splats into shared memory need a careful implementation and an exactness test). - Outcome of round 39: no change promoted; the submission stays at0e726ad(clean). Next round: the per-iteration glue (fills, copies, the features concatenation), about 8 % of a step in total by the profile (fills 96 us, cat 69 us, clone 35 us, copy 44 us, memcpy 27 us per iteration); exact changes only.