MiniMax M3.1 Flash 2.54×
Harness: Codex · switches: GS_FUSED_ADAM=1 GS_VECTOR_STATS=1 GS_ASYNC_LOSS=1 GS_FUSED_NORMALIZE=1
- Judged
- 2.54×
- Tier
- N1
- Self-best
- 1.60×
- Rounds
- 26
- Hours
- 10.0
- API cost
- $5.47
- Input tokens · 96.8% cached
- 209 M
- Output tokens
- 2.4 M
- Requests
- 1,587
§ 02Key levers
- Smaller tiles + rewritten binning
- ✓ 8×8 alone flat (1.59 vs 1.60); 1.76 with tight bounding boxes; 1.98 with one warp per Gaussian in binning
- Bypass autograd, activations folded into raster kernel
- ✗ only quaternion normalization made into one bitwise-identical kernel
GS_FUSED_NORMALIZE=1(2.21 → 2.40) - Lazy Adam (update only visible Gaussians)
- ✗
- Fused loss / SSIM
- last commit fused L1 and its grad (2.403 → 2.410, per commit message); SSIM untouched
- Other main levers
- bitwise-identical fused Adam, desync, backward per-pixel state packed into float4 (inner loop 176 → 129 instructions, 1.43 → 1.62), unused depth path off (→ 2.06), contiguous SH grad writes (→ 2.21)
§ ARaw trace
Raw work log27 sections
0.1 hRound 1 — baseline profile of the unchanged reference
Start: 2026-11-09 06:55 UTC. End: 2026-11-09 07:12 UTC. Command:bench/bench.py profile --tree /submission/gaussian-splatting --config /submission/switches.json --out /tmp/r1/profile(unchanged tree; 20 Gaussian-optimizer steps from 7,000; output/tmp/r1/profile/timing_profile.profile.txt). Profile findings (per step, GPU time 9.47 ms at step 7,000; GPU idle 0.35 ms/step ≈ 3.5 %): | kernel | ms/step | share | |---|---|---| |PerGaussianRenderCUDA<3u>(rasterizer backward) | 2.62 | 27.7 % | |renderCUDA<3u>(rasterizer forward) | 1.60 | 16.8 % | | Adammulti_tensor_apply(7 foreach passes) | 2.25 | 23.7 % | |preprocessCUDA<3>x2 | 0.63 | 6.6 % | |duplicateWithKeys| 0.49 | 5.2 % | | CUB radix sort (6 passes) | 0.23 | 2.4 % | |index/gather/nonzero (densification stats) | 0.51 | 5.4 % | |fusedssimfwd+bwd | 0.13 | 1.4 % | Kinds of hotspot: 1. Adam is a memory-traffic problem, not a compute problem — 7 separate foreach passes over the 59 floats/Gaussian state = 18 array passes (≈6.4 GB/step) where a fused kernel needs 7 (≈2.5 GB). 2. Rasterizer backward kernel is the single largest kernel. 3. Host: one device sync per iteration (loss.item()) plusnonzero(); ≈3.5 % bubble, plus a ~7.9 ms bubble on every 100th iteration (densification host work). Gate-2 sensitivity note (important): the recorded "steps at which the number of Gaussians changes" are exactly{1} ∪ {600,700,…,14900}— i.e. schedule-determined, not value-determined. The numeric gate is gate 3 (PSNR/SSIM/LPIPS/log(count) within 3σ on seed 1): σ_psnr 0.0389, σ_ssim 2.30e-4, σ_lpips 3.34e-4, σ_log_count 5.44e-3 (3σ_log ⇒ final count within ±1.6 %). ⇒ Numerics must stay *very* close; the optimizer will be made bit-exact, not merely equivalent.
0.8 hRound 2 — fused Adam (one CUDA kernel for the whole optimizer step)
Start: 2026-11-09 07:12 UTC. End: 2026-11-09 07:26 UTC. Change: newsubmodules/diff-gaussian-rasterization/fused_adam.{h,cu}— a single kernel that does the whole Adam step for the six Gaussian parameter tensors (param, grad, exp_avg, exp_avg_sq) plus the six learning-rate tensors (they are tensors in upstream, solris read from device memory). Exposed throughext.cpp(fused_adam_step) and used byscene/gaussian_model.pythrough aFusedAdamclass (atorch.optim.Optimizersubclass with the six parameter groupsxyz, f_dc, f_rest, opacity, scaling, rotation), enabled by the switchGS_FUSED_ADAM=1. Numerics: verified bit-exact against torch's foreach Adam (60 steps over 6 tensors of the real size, 0 mismatches, using the exact float semantics torch uses:lerp=fmaf(w,g-m,m),mul=v*b2,addcmul=fmaf(val, g*g, v)withg*grounded first,div= true float32 division byfl32(scalar)(not a reciprocal multiply),addcdiv=fmaf(ss, m/d, p)). Microbenchmark at N=1.5 M: 2.121 ms -> 0.693 ms per step. Result (time, out dir/tmp/r2/time): wall 255.2 s,speedup_vs_frozen_reference= 1.225.check --quick(out dir/tmp/r2/quick): PASS (exact items identical for the first 3,000 steps). Commite906b69— tier N0 (bit-exact).
0.8 hRound 3 — remove per-iteration host synchronisation, vectorise the densification statistics
Start: 2026-11-09 07:26 UTC. End: 2026-11-09 07:38 UTC. Change (switchGS_VECTOR_STATS=1,GS_ASYNC_LOSS=1): *visibility_filternow returns a bool mask instead of.nonzero();max_radii2Dis updated withtorch.maximum(..., out=...)(equivalent: invisible radii are 0 and the max is >= 0), andadd_densification_statscomputes the norm over all N (bit-identical, verified) — removes the gather/scatter and thenonzerodevice sync. * the training loop keeps the loss EMA on the device and only callsiter_end.synchronize()when atb_writerexists, so the per-iterationloss.item()sync is gone. Result (time, out dir/tmp/r3/time): wall 237.9 s,speedup_vs_frozen_reference= 1.313.check --quick(out dir/tmp/r3/quick): PASS. Commitce10ea3— tier N0 (bit-exact).
0.8 hRound 4 baseline profile (`/tmp/r4/profile`, step 7,000, all switches on)
GPU busy 7.93 ms/iter (was 9.47 ms), idle 6.7 %. | kernel | ms/step | share | |---|---|---| |PerGaussianRenderCUDA<3u>(rasterizer backward) | 2.664 | 33.6 % | |renderCUDA<3u>(rasterizer forward) | 1.592 | 20.1 % | |fused_adam_kernel| 0.755 | 9.4 % | |aten::index(gather/scatter) | 0.456 | 5.8 % | |duplicateWithKeys| 0.484 | 6.1 % | | backwardpreprocessCUDA<3>| 0.471 | 5.9 % | | CUB radix sort (6 passes) | 0.236 | 3.0 % | | forwardpreprocessCUDA<3>| 0.140 | 1.8 % | P ~ 1.66 M Gaussians, B ~ 44 K buckets, L ~ 5.4 M (Gaussian, tile) pairs, ~1.38 G Gaussian-pixel evaluations/iter, 2170 tiles (62x35).
0.8 hRound 4 — rasterizer: hoist the per-Gaussian work out of the per-pixel loops (first attempt)
Start: 2026-11-09 07:55 UTC. End: 2026-11-09 08:10 UTC. Profile findings (round-4 baseline/tmp/r4/profile, all switches on, step 7,000): GPU busy 7.93 ms/iter.PerGaussianRenderCUDA<3u>2.664 ms (33.6 %),renderCUDA<3u>1.592 ms (20.1 %),fused_adam_kernel0.758 ms (9.6 %),duplicateWithKeys0.493 ms (6.2 %), backwardpreprocessCUDA0.472 ms (6.0 %), radix sort 0.237 ms (3.0 %). Grid sizes from the profiler trace: the backward runs 44,530 blocks of 32 threads (64 registers, so only 32 warps/SM = 50 % occupancy), the forward 2,170 blocks of 256 threads. Change: forward — the1.f / depths[...]division (5 SASS instructions per evaluation) moved into the shared-memory prefetch; backward — the per-pixelbg_dot_dpixelmoved to the per-pixel state, plus two guards (dL_invdepth != 0,bg_dot_dpixel != 0) to skip the depth and background terms. Result (time,/tmp/r4/time): 1.317 (231.27 s).check --quick(/tmp/r4/quick): PASS. The profile after the change showedrenderCUDA1.592 -> 1.476 ms (-7.3 %, as predicted) butPerGaussianRenderCUDA2.664 -> 2.737 ms (+2.7 %): the compiler turned the two guards into *predication* (the instructions are still issued) and added BSSY/BSYNC branch overhead. The guards were removed again.
1.7 hRound 5 — rasterizer: fetch the colour once per Gaussian, per-pixel background term
Start: 2026-11-09 08:10 UTC. End: 2026-11-09 08:33 UTC. SASS analysis of the two hot kernels (cuobjdump on the judge's own build): * the forward inner loop is 96 instructions for one (Gaussian, pixel) evaluation, of which ~13 were the address arithmetic + 3 global loads + 3 byte permutes for the colour (features[collected_id[j]*3+ch]); * the backward inner loop is 211 instructions per pixel iteration: 86 in the prologue (18 LDG + 12 SHFL + 18 IMAD + 11 LDC) and 120 in the work part. Both kernels are close to the instruction-issue limit, so removing instructions is what pays. Change (commit2c3f6fe): * forward: the colour of a Gaussian is fetched once per Gaussian into shared memory, packed with the reciprocal depth in a singlefloat4, so the inner loop replaces ~13 instructions with oneLDS.128;collected_idis not needed any more; * forward: the depth reciprocal is fetched once per Gaussian (already done in round 4); * backward:bg_dot_dpixelis computed once per pixel and shuffled with the other per-pixel state. Result (time,/tmp/r5/time): 1.3637 (223.358 s; windows 60.66 / 82.71 / 79.99 vs 63.16 / 85.56 / 82.55 in round 4).check --quick(/tmp/r5/quick): PASS (exact items identical). Numerics tier: N0 (bit-exact).
3.4 hRound 6 — fused Adam: four elements per thread
Start: 2026-11-09 08:33 UTC. End: 2026-11-09 08:50 UTC. The Adam kernel walked the parameters one float per thread; it now walks a float4 per thread (the same operations in the same order, so the result is bit for bit the same). Result (time,/tmp/r6/time): 1.3787 (220.9 s).check --quick: PASS. Commit44f445f— tier N0.
3.4 hRound 7 — rasterizer backward: unsigned pixel index and one reciprocal per pixel
Start: 2026-11-09 08:50 UTC. End: 2026-11-09 09:05 UTC. The backward kernel computed the pixel index with a signed modulo/division and divided by the depth for every pixel; both are done with unsigned arithmetic and one reciprocal per pixel now (the values are the same). Result (time,/tmp/r7/time): 1.3706 (222.2 s).check --quick: PASS. Commitc892524— tier N0.
3.4 hRound 8 — rasterizer: `__expf` instead of `exp` (REJECTED, reverted)
Start: 2026-11-09 09:05 UTC. End: 2026-11-09 09:20 UTC.__expf(the fast hardware exponential) instead ofexpin the forward and the backward. Result (time,/tmp/r8/time): 1.4514 (209.8 s) — the fastest so far — butcheck --n1INCORRECT: on seed 1 the SSIM bias was -1.006e-3 against a bound of 6.89e-4 (-4.38 sigma), PSNR -1.87 sigma, LPIPS +2.94 sigma, log(count) -2.11 sigma. The model is chaotic: a change of one ulp in the rasterizer ends up at 2-4 sigma after 30,000 iterations. From here on every change has to be bit exact. The__expfchange was reverted.
3.4 hRound 9 — rasterizer: empty gradient buffers + kernel-side zero writes (REGRESSION)
Start: 2026-11-09 09:20 UTC. End: 2026-11-09 09:35 UTC. The five gradient buffers that the backward writes in full were allocated withtorch::emptyinstead oftorch::zeros, and the kernels wrote the zeros for the Gaussians that were not rendered. Result (time,/tmp/r9/time): 1.3277 (229.4 s) — a 3 % regression. The profile (/tmp/r10/profile) showed why: the per-Gaussian backward preprocessing went from 0.47 ms to 1.16 ms. The scattered per-Gaussian zero stores are much more expensive than the 300 MB bulk memset they replace.
3.4 hRound 10 — rasterizer: reciprocal hoisted out of the tile loop
Start: 2026-11-09 09:35 UTC. End: 2026-11-09 09:40 UTC.
The reciprocals of the conic's diagonal depend only on the Gaussian, so they are
computed once per Gaussian instead of once per tile it overlaps
(duplicateWithKeys 0.49 ms -> 0.34 ms). Bit exact.3.4 hRound 11 — rasterizer backward: the per-pixel state comes from memory
Start: 2026-11-09 09:40 UTC. End: 2026-11-09 10:04 UTC. Profile of the current best version (round 10,/tmp/r10/profile, step 7000): GPU busy 7.9 ms/iter;PerGaussianRenderCUDA<3u>(the backward) 2.68 ms (33 %),renderCUDA<3u>(the forward) 1.29 ms (16 %), the per-Gaussian backward preprocessing 1.16 ms (14 %, the zero-write experiment), the fused Adam 0.69 ms,duplicateWithKeys0.34 ms, the radix sort 0.24 ms. SASS of the backward's inner loop (cuobjdump on the judge's own build): **176 instructions per pixel iteration, 18 of them global loads** (9 for the per-pixel state, 9 for the bucket state), 5 shuffles, 20 address IMADs. Changes (all bit exact): * the per-pixel state (final transmittance, number of contributors, the colour and depth gradients, the background term) is read by the lane that needs it instead of being passed along the warp with shuffles; * the zero-fill experiment of round 9 was reverted (the buffers are zeroed again); *duplicateWithKeyscomputes the reciprocals once per Gaussian (round 10); *prune_optimizerdoes not empty the caching allocator on every prune; * the progress bar reads the loss back every 100 iterations instead of every 10. Result (time,/tmp/r11/time): 1.4263 (213.5 s; windows 57.7 / 79.1 / 76.7).check --quick(/tmp/r11/quick): PASS (the exact items are identical). Commit6dd3745— tier N0.
3.4 hRound 12 — rasterizer backward: the per-pixel state in two float4s
Start: 2026-11-09 10:04 UTC. End: 2026-11-09 10:21 UTC. The backward read six scalars per pixel (three colour gradients, the depth gradient, the final transmittance, the number of contributors) plus the three components of the background colour, and multiplied the background colour with the gradients for every pixel of every bucket. A small kernel now packs the state into two float4s (packPixelStateCUDA): the colour and depth gradients in one, the final transmittance, the background term and the number of contributors in the other. The background term is accumulated in the same order with the same fused multiply-add, so it is the same number. The inner loop of the backward drops from 176 to 152 instructions and from **18 to 11 memory instructions** (the two float4 loads plus the nine loads of the bucket state). Result (time,/tmp/r12/time): 1.5148 (201.1 s).check --quick(/tmp/r12/quick): PASS (the exact items are identical). Commitf9ddce8— tier N0.
3.7 hRound 13 — rasterizer: pack the bucket state and the pixel colour into float4s
Start: 2026-11-09 10:21 UTC. End: 2026-11-09 10:40 UTC. The forward wrote the state that starts every bucket (the transmittanceTand the three colour sumsC[0..2]) into four separate arrays (sampled_T,sampled_ar[3]), and the backward read them with five separate global loads per pixel. They are now written as onefloat4(sampled_TC) and read with one load. The colour and the inverse depth of the pixel (which the backward needs per pixel) are written directly by the forward as onefloat4(pixel_color_invdepth), which also removes the two device-to-device copies that used to copy the rendered colour and the inverse depth into the chunk. Bit exact: the values stored are the same bits, and the backward still computes-color + C[ch]and-invdepth + ardin the same order. The inner loop of the backward drops from 152 to 129 instructions and from **11 to 5 memory instructions** per pixel iteration. Result (time,/tmp/r13b/time): 1.6169 (188.4 s; windows 51.1 / 69.7 / 67.6).check --quick(/tmp/r13b/quick): PASS (the exact items are identical). Commiteae55f2— tier N0.
4.4 hRound 14 — the full check of round 13 FAILED the statistical gate: the fused Adam was not bit-exact
Start: 2026-11-09 10:45 UTC. End: 2026-11-09 11:20 UTC. The full check of the round-13 tree (/tmp/r14c/check/check.json) came backINCORRECT: gate 2 (exact items) passed, gate 3 failed with SSIM -5.58 sigma, LPIPS +3.55 sigma, PSNR -2.09 sigma. No full check had ever been run before (rounds 1-13 only rancheck --quick, which compares the exact items and not the trained model), so the culprit could be any of the changes, not only the last three. ### Diagnosis (cheap probes instead of 5-minute N1 runs) * The judge's reference record (bench/records/ref.json) has the number of Gaussians after 3,000 steps: 1,092,971 (the bootstrap run of the reference gave 1,095,412, so the run-to-run spread of the *count* is ~0.2 %). The candidate gave 1,056,479 (-3.3 %): a systematic difference, not noise. * A probe that runs the tree's own entry for 3,000 steps takes 20 s, so the switches could be bisected with it: without switches 1,094,119,GS_VECTOR_STATS=11,094,136,GS_ASYNC_LOSS=11,096,426,GS_FUSED_ADAM=11,053,183. The fused Adam was the culprit; the CUDA changes (rounds 5, 11, 12, 13) are fine. *torch.normover a strided view and over a gathered copy is bit-identical (checked), so the vectorised densification statistics are not the problem either. ### Root causecreate_from_pcdbuilds the positions withtorch.tensor(np.asarray(pcd.points)), which keeps the strides of the numpy array:self._xyzis column-major, i.e. not contiguous. The fused Adam kernel walks the parameter, the gradient and the optimizer state as flat buffers, so for the whole training (the positions only become contiguous at the first densification, iteration 600) it wrote the update of the first Gaussian into the first elements of the buffer and left the other positions untouched: the position update was simply wrong. Torch's own Adam uses the foreach route, which respects the layout, which is why the reference is right. ### Fix (commitba310c4)FusedAdam.stepuses the fused kernel only when the parameter and the two state tensors are contiguous, and otherwise falls back to the operations oftorch.optim.Adam(lerp_,mul_+addcmul_,sqrt/div/add,addcdiv_) - the same operations the reference runs, so the values are the same. The fallback is only used for the first 600 iterations. ### Verification * The parameters of the first steps are now identical to the ones of thetorch.optim.Adampath (only the last-bit differences of the rasterizer's atomics remain). * The number of Gaussians after 3,000 steps: 1,093,883 with the fused Adam, against 1,093,831 and 1,095,026 for two runs of the reference path (the broken version: 1,053,183).
4.9 hRound 15 — the fix of round 14 verified, and the first full check
Start: 2026-11-09 11:20 UTC. End: 2026-11-09 11:43 UTC. The fused-Adam fix (commitba310c4) was verified with the cheap probes: * the number of Gaussians after 3,000 steps: 1,093,883 (fused Adam) against 1,093,831 / 1,095,026 for two runs of the reference path (1,053,183 with the broken fused Adam); * the parameters of the first steps are identical to the ones of thetorch.optim.Adampath (only the last-bit differences of the rasterizer's atomics remain). Full check (/tmp/r15/check/check.json): PASS, tier N1, speedup 1.4846. Gate 2 (exact items) passed; gate 3: SSIM -1.29 sigma, LPIPS -0.67 sigma, PSNR -2.08 sigma, log(count) +0.84 sigma (all inside +/- 3 sigma).
4.9 hRound 16 — the zero-fill of the gradient buffers with a device memset
Start: 2026-11-09 11:30 UTC. End: 2026-11-09 11:42 UTC. Profile of the steady state (bench.py profile --from 7010 --iters 20,/tmp/r17/profile): 5.95 ms of GPU time per iteration, of which ~0.29 ms was spent filling zeroed buffers (aten::fill_/aten::zerosinside the rasterizer: the eleven gradient buffers of the backward, the colour image, the inverse depth image and the radii).torch::zeros/torch::fulllaunch the generic elementwise fill kernel, which reaches only a fraction of the memory bandwidth. The eleven gradient buffers of the backward are large (the SH gradient alone is P x 45 floats). They are now allocated withtorch::emptyand zeroed withcudaMemsetAsyncon the current stream: a float 0.0f is all-zero bits, so the bytes written are exactly the same. *time(/tmp/r18/time): 1.5961 (190.8 s; windows 52.1 / 70.3 / 68.4). *check --quick(/tmp/r18/quick): PASS (the exact items are identical). * Commitcc91305— tier N0.
4.9 hRound 17 — the full check of round 16
Start: 2026-11-09 11:43 UTC. End: 2026-11-09 11:51 UTC.
Full check (/tmp/r19/check/check.json): PASS, tier N1, speedup 1.6018.
Gate 3: PSNR -0.94 sigma, SSIM -0.05 sigma, LPIPS -1.20 sigma, log(count) +0.42 sigma.
The submission is now a verified N1 at 1.60x.6.7 hRound 18 — the reciprocal and the dummy screen-space tensor (no-ops)
Start: 2026-11-09 12:00 UTC. End: 2026-11-09 12:25 UTC. Two micro-changes in the tree:__frcp_rn(1.0f-alpha)instead of the division in the rasterizer backward (verified: identical SASS, 360 instructions, the same opcode histogram - a no-op, reverted) andtorch.empty_likeinstead oftorch.zeros_likefor the dummy screen-space mean tensor ofgaussian_renderer.render(the rasterizer never reads the values, only the.gradof the leaf matters). *time(/tmp/r20/time): 1.5681;check --quick(/tmp/r20/quick): PASS. * Commitf1983df- tier N0.
6.7 hRound 19 — 8x8 tiles instead of 16x16 (measured: a wash)
Start: 2026-11-09 12:25 UTC. End: 2026-11-09 12:32 UTC. The rasterizer renders a Gaussian against every pixel of every tile its bounding box overlaps, so a smaller tile rejects most of the (Gaussian, pixel) pairs early: with 8x8 tiles the forward and backward render kernels need 2.2x less pixel work (3.10 ms -> 1.83 ms per iteration), but the binning gets 4x more expensive (the rectangle of a Gaussian covers 4x more tiles): duplicateWithKeys 0.35 -> 1.22 ms, the radix sort 0.22 -> 0.50 ms. Net: a wash. *time(/tmp/r21b/time): 1.5912;check --quick(/tmp/r21b/quick): PASS. * The number of Gaussians after 3,000 steps moves from 1,093,883 to 1,100,796 (+0.6 %), i.e. the finer culling (a tile is kept if one of its pixels has alpha >= 1/255) changes the image slightly - it stays inside the 3 sigma of the statistical gate, but it eats margin.
6.7 hRound 20 — the tight tile box
Start: 2026-11-09 12:35 UTC. End: 2026-11-09 13:01 UTC. Profile of the steady state (bench.py profile --from 7010 --iters 20,/tmp/r23/profile): 5.36 ms of GPU time per iteration. The binning (duplicateWithKeys 1.05 ms + the radix sort 0.26 ms) and the two render kernels (1.20 + 0.58 ms) are the hot spots. A Gaussian is rasterized against every tile of its 3-sigma square, but the square is the bounding box of the *largest* principal axis in both directions. The tiles that can change the image are the ones overlapping the ellipse of the pixels whose alpha is at least 1/255 (q <= log(255*opacity)), whose axis-aligned bounding box issqrt(2k*cov_xx) x sqrt(2k*cov_yy)- much tighter for an elongated Gaussian. Every tile that is dropped this way contains only pixels with alpha < 1/255, which the render kernel skips anyway, so the image is unchanged; the tile rectangles are stored per Gaussian (int2* rects) and used by both the preprocess and the binning kernel. *time(/tmp/r22/time): 1.7606 (173.0 s; windows 49.2 / 62.7 / 61.1), against 1.6018. *check --quick(/tmp/r22/quick): PASS (exact items identical). *check --n1(/tmp/r23/n1): PASS, tier N1 - PSNR -0.53 sigma, SSIM -0.71 sigma, LPIPS -1.46 sigma, log(count) +1.09 sigma (all inside 3 sigma). * Commitscb3d2f0(8x8 tiles) anddf2879f(the tight box) - tier N1.
6.7 hRound 21 — one warp per Gaussian in the binning kernel
Start: 2026-11-09 13:09 UTC. End: 2026-11-09 13:35 UTC. Diagnostic (a build that skips the Gaussians whose rectangle has more than 1024 tiles): duplicateWithKeys falls from 1020 us to 293 us - **a few Gaussians with a huge rectangle (a big Gaussian close to the camera) serialize the whole kernel**, because the kernel gave one *thread* per Gaussian and a thread walks the whole rectangle of its Gaussian alone, so a warp can only advance as fast as its slowest Gaussian. One warp per Gaussian now: the 32 lanes walk the rectangle together, and the pairs that pass the opacity test are written with a ballot (__ballot_sync/__popc), so the entries of a Gaussian are bit-identical to the ones the old kernel wrote (the image does not change at all). *time(/tmp/r26/time): 1.976 (154.1 s), against 1.7606 before. *check --quick(/tmp/r26/quick): PASS (exact items identical). * Commit00fd509- tier N0 (bit-identical output). Profile after the change (4.76 ms of GPU time per iteration): | kernel | ms/step | share | |---|---|---| |PerGaussianRenderCUDA<3u>(rasterizer backward) | 1.236 | 26 % | | fused Adam | 0.696 | 15 % | |preprocessCUDA<3>(rasterizer backward) | 0.452 | 9.5 % | |duplicateWithKeys| 0.441 | 9 % | |renderCUDA<3u>(rasterizer forward) | 0.407 | 8.5 % | | CUB radix sort | 0.261 | 5.5 % | | device memsets | 0.153 | 3 % | |preprocessCUDA<3>(rasterizer forward) | 0.145 | 3 % | | the torch ops of the training loop (rotation normalisation, SSIM, loss, ...) | ~0.8 | 17 % |
7.3 hRound 22 - the depth path of the rasterizer and the exponential of the rejected pairs
Start: 2026-11-09 13:40 UTC. End: 2026-11-09 14:12 UTC. Profile of the tree at the start of the round (/tmp/r26/profile, 4.76 ms of GPU time per iteration): rasterizer backward 1.85 ms (39 %), rasterizer forward 1.47 ms (31 %), fused Adam 0.70 ms (15 %), the torch ops of the loop 0.72 ms (15 %). The CPU runs at about the same rate as the GPU (the wall clock is 5.14 ms per iteration), so every percent of GPU time is a percent of the wall clock. Two changes, both inside the rasterizer: 1. The exponential of the rejected pairs (alpha_power_thresholdinauxiliary.h). The alpha of a (Gaussian, pixel) pair ismin(0.99, opacity * exp(power))and the pair is skipped when it is below 1/255, that is whenpower < -log(255 * opacity). The exponential is the most expensive part of that test and most of the pixels of the tiles a Gaussian touches fail it, so the threshold is now computed once per Gaussian and the power is compared with it *before* the exponential is evaluated. The margin (0.01) makes the test conservative: every pair that passes it is still tested exactly as before, so the set of accepted pairs - and therefore the image - does not change. The forward kernel got 5 % faster (578 -> 550 us in the profile of/tmp/r27/profile); in the backward kernel the test did not pay off (the inner loop is register bound), so it was left out there. 2. The depth path of the rasterizer is now switched off when the depth regularization is inactive, which is the case of the recipe (--depths "", so no camera has a depth map). The rasterizer then does not compute the inverse depth of the image, does not carry the inverse depth of the buckets through the sampled state, and does not evaluate the depth part of the gradients (the gradient of the depth image is exactly zero when the depth is not used, so the gradients are unchanged). The flag travels fromtrain.py(the condition of the depth regularization) throughgaussian_renderer.renderand the rasterizer settings into the four kernels that touch the depth. The per-Gaussian gradient buffer of the inverse depth is not even allocated any more. *time(/tmp/r29/time): 2.0559 (148.2 s; windows 41.3 / 54.2 / 52.7), against 1.976 before. *check --quick(/tmp/r29/quick): PASS (the exact items of the first 3,000 steps are identical). * Numerics: N0 (the image and all the gradients are bit for bit the same). * An earlier attempt at the same round (/tmp/r27) put the exponential test into the backward kernel as well; it was neutral (1.9694) and was reverted.
8.4 hRound 23 - the spherical harmonics gradient, written in a contiguous way
Start: 2026-11-09 15:00 UTC. End: 2026-11-09 15:16 UTC. Profile of the tree at the start of the round (/tmp/r32/profile, 4.61 ms of GPU time per iteration): rasterizer backward 1.06 ms (23 %), rasterizer forward 0.52 ms (11 %), the fused Adam 0.65 ms (14 %), the backwardpreprocessCUDA0.49 ms (10.5 %), the binning kernel 0.45 ms (10 %), the radix sort 0.26 ms (6 %), the device memsets 0.15 ms (3 %), the torch ops of the loop about 0.60 ms (13 %). The hotspot of the round is the backwardpreprocessCUDA. It computes, for every Gaussian, the gradient of the loss with respect to its 15 spherical harmonics coefficients, and writes it to the buffer of shape [P][15][3] that the optimizer reads. The buffer gives every Gaussian 180 bytes, so the 45 floats of a Gaussian are contiguous but the Gaussians of a warp are 180 bytes apart: the 45 stores of a thread touch 45 different cache lines, and the 45 stores of the 32 threads of a warp touch 1440 of them. The stores of the warp use 12 % of the bandwidth they move. The gradients are now staged in the shared memory of the block and written to the global buffer by the block at the end of the kernel, in the order the buffer uses: the 45 floats of the Gaussians of a block are then written by the 32 threads of a warp one after the other, so the stores are contiguous. The staging costs 45 shared stores and 45 shared loads per thread, and it replaces 1440 cache line transactions per warp by 135. The buffer is not zeroed any more, because the kernel fills every element of it (the Gaussians that are not visible get a zero gradient, as before). The values are the ones the old code computed:computeColorFromSHis unchanged except for the place where it writes its result, so the arithmetic, and the way the compiler contracts it, are the same. This was checked with a test program (/tmp/shgx) that runs the old and the newpreprocessCUDAon the same random data and compares the 2,250,000 floats of the gradient buffer: **0 of them differ**. *time(/tmp/r36/time): 2.2058 (138.08 s), against 2.0559 before. *check --quick(/tmp/r35/quick): PASS (the exact items of the first 3,000 steps are identical). * Numerics: N0 (the gradient buffer is bit for bit the same). * A first attempt at this round put the computation of the gradients in a separate kernel (computeSHGradCUDA) that recomputed the direction of the Gaussians. It was 1 to 5 ULP off the reference (the compiler contracted the length of the direction differently in the new context) and the run crashed with SIGSEGV, so it was dropped in favour of the staging above.
8.8 hRound 24 - the normalisation of the quaternions, in one kernel
The profile of the tree at the start of the round (/tmp/r37/profile, 4.86 ms of GPU time per iteration over the window 7,000..7,019) put the composite optorch.nn.functional.normalize(self._rotation, dim=1)at about 0.29 ms of the iteration, that is 6 % of it. It is not one kernel but eight: the norm is reduced twice (once in the forward, once again in the backward of the norm, as torch recomputes it), the sum of the four squares is a reduction over the broadcast dimension of the division, and the rest are elementwise passes over the quaternions and their gradients. In the trace the two reductions of the norm cost 77 us per iteration, the reduction of the division 48 us, and the elementwise passes about 150 us more. The normalisation is now a single CUDA kernel per direction (submodules/diff-gaussian-rasterization/quat_normalize.cu, behind theGS_FUSED_NORMALIZEswitch), which computes the norm, the divisor and the result, and in the backward the whole chain of the gradient. The arithmetic is the one of torch, operation by operation and rounding by rounding, and it was measured before writing the kernel: * the norm issqrt((q0*q0 + q1*q1) + (q2*q2 + q3*q3)); the four squares are summed in pairs, which is what the reduction oflinalg_vector_normdoes on this shape (the sequential sum differs from it on 12 % of the samples); * the divisor ismax(n, 1e-12), and the result isq / d; * the backward isu_k = (q_k/d)/d,t_k = -g_k * u_k,gd = (t0+t1) + (t2+t3),gn = (n >= eps) ? gd : 0,gx_k = (g_k/d) + gn*(q_k/n); the sum of the fourt_kis again in pairs, and the productgn*(q_k/n)is formed as it is written above and not as(gn*q_k)/n(which differs on 17 % of the samples); * a zero norm is replaced by one before the division by the norm, as torch does in the backward of the norm, so that the gradient of a zero quaternion is zero and not a NaN. The kernel uses__fmul_rn,__fadd_rn,__fdiv_rnand__fsqrt_rn, so that the compiler cannot contract the operations into fused multiply-adds, which would round differently. The comparison with torch over about 30 million quaternions (random ones, ones over ten scales from 1e-8 to 1e30, heavy-tailed ones, and the edge cases: the zero quaternion, the ones whose norm is exactly the epsilon, the infinite and the NaN ones) gives 0 differences in the forward and 0 differences in the backward, and the NaNs land on the same elements (/tmp/normtest). *time(/tmp/r38/time): 2.396 (127.12 s), against 2.2058 before. *check --quick(/tmp/r38/quick): PASS (the exact items of the first 3,000 steps are identical). * Numerics: N0 (the forward and the backward are bit for bit the ones of torch).
9.6 hRound 25 - two experiments on the backward render (no gain, reverted)
The backward render (PerGaussianRenderCUDA, 1081 us per iteration, 26 % of the time) is the biggest kernel of the iteration. Two changes were tried on it, both measured withbench.py profile(the time of the kernel is in the profile): 1. The alpha log-threshold pre-filter. The forward render skips the exponential of a pair whose contribution to the tile is below 1/255 (it compares the power withlog(1/(255*opacity))). The same test was added to the backward render, before the exponential. The backward render went from 1059 us to 1068 us: the backward is not bound by the exponential (itslast_contributorearly exit already removes most of the pairs, and the warp divergence prevents the lanes from skipping the test), so the extra comparison only costs. 2. Thepix_grad4load moved after the alpha test. The backward render loads the state of the pixel it works on before it knows whether the pair contributes. Deferring the load made it slower (1081 us). Both changes were reverted (git checkout). The threshold itself was proved safe (0 violations over 16 million samples,/tmp/normtest/thr.py), it simply buys nothing.
9.6 hRound 26 - the occupancy of the backward render (a wash)
The backward render uses 56 registers (nvcc -Xptxas -v) and was launched with one warp per block, so that the number of resident warps per multiprocessor was limited by the hardware limit of 32 blocks per multiprocessor and not by the registers: 32 warps out of the 64 a multiprocessor can hold. The kernel does not care how many warps share a block (it computes the bucket index from the warp index), so the launch was made configurable (GS_BWD_WARPS, off by default,cuda_rasterizer/backward.cu): with four warps per block the limit becomes the number of registers, 36 warps. *time(/tmp/r43/time): 2.4032 (126.74 s), against 2.396 before. The difference is inside the noise of the measurement (the same tree measured 2.3856 in the full check), so the kernel is not bound by its occupancy. * The change is kept and enabled, it is bit for bit the same computation.