MiniMax M3.1 Flash 2.54×

Harness: Codex · switches: GS_FUSED_ADAM=1 GS_VECTOR_STATS=1 GS_ASYNC_LOSS=1 GS_FUSED_NORMALIZE=1

Judged
2.54×
Tier
N1
Self-best
1.60×
Rounds
26
Hours
10.0
API cost
$5.47
Input tokens · 96.8% cached
209 M
Output tokens
2.4 M
Requests
1,587
Smaller tiles + rewritten binning
✓ 8×8 alone flat (1.59 vs 1.60); 1.76 with tight bounding boxes; 1.98 with one warp per Gaussian in binning
Bypass autograd, activations folded into raster kernel
✗ only quaternion normalization made into one bitwise-identical kernel GS_FUSED_NORMALIZE=1 (2.21 → 2.40)
Lazy Adam (update only visible Gaussians)
✗
Fused loss / SSIM
last commit fused L1 and its grad (2.403 → 2.410, per commit message); SSIM untouched
Other main levers
bitwise-identical fused Adam, desync, backward per-pixel state packed into float4 (inner loop 176 → 129 instructions, 1.43 → 1.62), unused depth path off (→ 2.06), contiguous SH grad writes (→ 2.21)
Raw work log27 sections
0.1 hRound 1 — baseline profile of the unchanged reference
Start: 2026-11-09 06:55 UTC. End: 2026-11-09 07:12 UTC.

Command: bench/bench.py profile --tree /submission/gaussian-splatting --config /submission/switches.json --out /tmp/r1/profile
(unchanged tree; 20 Gaussian-optimizer steps from 7,000; output /tmp/r1/profile/timing_profile.profile.txt).

Profile findings (per step, GPU time 9.47 ms at step 7,000; GPU idle 0.35 ms/step ≈ 3.5 %):
| kernel | ms/step | share |
|---|---|---|
| PerGaussianRenderCUDA<3u> (rasterizer backward) | 2.62 | 27.7 % |
| renderCUDA<3u> (rasterizer forward) | 1.60 | 16.8 % |
| Adam multi_tensor_apply (7 foreach passes) | 2.25 | 23.7 % |
| preprocessCUDA<3> x2 | 0.63 | 6.6 % |
| duplicateWithKeys | 0.49 | 5.2 % |
| CUB radix sort (6 passes) | 0.23 | 2.4 % |
| index/gather/nonzero (densification stats) | 0.51 | 5.4 % |
| fusedssim fwd+bwd | 0.13 | 1.4 % |

Kinds of hotspot:
1. Adam is a memory-traffic problem, not a compute problem — 7 separate foreach passes over the
   59 floats/Gaussian state = 18 array passes (≈6.4 GB/step) where a fused kernel needs 7 (≈2.5 GB).
2. Rasterizer backward kernel is the single largest kernel.
3. Host: one device sync per iteration (loss.item()) plus nonzero(); ≈3.5 % bubble, plus a
   ~7.9 ms bubble on every 100th iteration (densification host work).

Gate-2 sensitivity note (important): the recorded "steps at which the number of Gaussians changes"
are exactly {1} ∪ {600,700,…,14900} — i.e. schedule-determined, not value-determined. The
numeric gate is gate 3 (PSNR/SSIM/LPIPS/log(count) within 3σ on seed 1): σ_psnr 0.0389,
σ_ssim 2.30e-4, σ_lpips 3.34e-4, σ_log_count 5.44e-3 (3σ_log ⇒ final count within ±1.6 %).
⇒ Numerics must stay *very* close; the optimizer will be made bit-exact, not merely equivalent.
0.8 hRound 2 — fused Adam (one CUDA kernel for the whole optimizer step)
Start: 2026-11-09 07:12 UTC. End: 2026-11-09 07:26 UTC.

Change: new submodules/diff-gaussian-rasterization/fused_adam.{h,cu} — a single kernel that does the whole
Adam step for the six Gaussian parameter tensors (param, grad, exp_avg, exp_avg_sq) plus the six learning-rate
tensors (they are tensors in upstream, so lr is read from device memory). Exposed through ext.cpp
(fused_adam_step) and used by scene/gaussian_model.py through a FusedAdam class (a torch.optim.Optimizer
subclass with the six parameter groups xyz, f_dc, f_rest, opacity, scaling, rotation), enabled by the switch
GS_FUSED_ADAM=1.

Numerics: verified bit-exact against torch's foreach Adam (60 steps over 6 tensors of the real size, 0
mismatches, using the exact float semantics torch uses: lerp=fmaf(w,g-m,m), mul=v*b2,
addcmul=fmaf(val, g*g, v) with g*g rounded first, div = true float32 division by fl32(scalar) (not a
reciprocal multiply), addcdiv=fmaf(ss, m/d, p)).
Microbenchmark at N=1.5 M: 2.121 ms -> 0.693 ms per step.

Result (time, out dir /tmp/r2/time): wall 255.2 s, speedup_vs_frozen_reference = 1.225.
check --quick (out dir /tmp/r2/quick): PASS (exact items identical for the first 3,000 steps).
Commit e906b69 — tier N0 (bit-exact).
0.8 hRound 3 — remove per-iteration host synchronisation, vectorise the densification statistics
Start: 2026-11-09 07:26 UTC. End: 2026-11-09 07:38 UTC.

Change (switch GS_VECTOR_STATS=1, GS_ASYNC_LOSS=1):
* visibility_filter now returns a bool mask instead of .nonzero(); max_radii2D is updated with
  torch.maximum(..., out=...) (equivalent: invisible radii are 0 and the max is >= 0), and
  add_densification_stats computes the norm over all N (bit-identical, verified) — removes the gather/scatter
  and the nonzero device sync.
* the training loop keeps the loss EMA on the device and only calls iter_end.synchronize() when a
  tb_writer exists, so the per-iteration loss.item() sync is gone.

Result (time, out dir /tmp/r3/time): wall 237.9 s, speedup_vs_frozen_reference = 1.313.
check --quick (out dir /tmp/r3/quick): PASS.
Commit ce10ea3 — tier N0 (bit-exact).
0.8 hRound 4 baseline profile (`/tmp/r4/profile`, step 7,000, all switches on)
GPU busy 7.93 ms/iter (was 9.47 ms), idle 6.7 %.
| kernel | ms/step | share |
|---|---|---|
| PerGaussianRenderCUDA<3u> (rasterizer backward) | 2.664 | 33.6 % |
| renderCUDA<3u> (rasterizer forward) | 1.592 | 20.1 % |
| fused_adam_kernel | 0.755 | 9.4 % |
| aten::index (gather/scatter) | 0.456 | 5.8 % |
| duplicateWithKeys | 0.484 | 6.1 % |
| backward preprocessCUDA<3> | 0.471 | 5.9 % |
| CUB radix sort (6 passes) | 0.236 | 3.0 % |
| forward preprocessCUDA<3> | 0.140 | 1.8 % |
P ~ 1.66 M Gaussians, B ~ 44 K buckets, L ~ 5.4 M (Gaussian, tile) pairs,
~1.38 G Gaussian-pixel evaluations/iter, 2170 tiles (62x35).
0.8 hRound 4 — rasterizer: hoist the per-Gaussian work out of the per-pixel loops (first attempt)
Start: 2026-11-09 07:55 UTC. End: 2026-11-09 08:10 UTC.

Profile findings (round-4 baseline /tmp/r4/profile, all switches on, step 7,000):
GPU busy 7.93 ms/iter. PerGaussianRenderCUDA<3u> 2.664 ms (33.6 %), renderCUDA<3u> 1.592 ms (20.1 %),
fused_adam_kernel 0.758 ms (9.6 %), duplicateWithKeys 0.493 ms (6.2 %), backward preprocessCUDA 0.472 ms
(6.0 %), radix sort 0.237 ms (3.0 %). Grid sizes from the profiler trace: the backward runs 44,530 blocks of
32 threads (64 registers, so only 32 warps/SM = 50 % occupancy), the forward 2,170 blocks of 256 threads.

Change: forward — the 1.f / depths[...] division (5 SASS instructions per evaluation) moved into the
shared-memory prefetch; backward — the per-pixel bg_dot_dpixel moved to the per-pixel state, plus two
guards (dL_invdepth != 0, bg_dot_dpixel != 0) to skip the depth and background terms.

Result (time, /tmp/r4/time): 1.317 (231.27 s). check --quick (/tmp/r4/quick): PASS.
The profile after the change showed renderCUDA 1.592 -> 1.476 ms (-7.3 %, as predicted) but
PerGaussianRenderCUDA 2.664 -> 2.737 ms (+2.7 %): the compiler turned the two guards into *predication*
(the instructions are still issued) and added BSSY/BSYNC branch overhead. The guards were removed again.
1.7 hRound 5 — rasterizer: fetch the colour once per Gaussian, per-pixel background term
Start: 2026-11-09 08:10 UTC. End: 2026-11-09 08:33 UTC.

SASS analysis of the two hot kernels (cuobjdump on the judge's own build):
* the forward inner loop is 96 instructions for one (Gaussian, pixel) evaluation, of which ~13 were the
  address arithmetic + 3 global loads + 3 byte permutes for the colour (features[collected_id[j]*3+ch]);
* the backward inner loop is 211 instructions per pixel iteration: 86 in the prologue (18 LDG + 12 SHFL +
  18 IMAD + 11 LDC) and 120 in the work part.
Both kernels are close to the instruction-issue limit, so removing instructions is what pays.

Change (commit 2c3f6fe):
* forward: the colour of a Gaussian is fetched once per Gaussian into shared memory, packed with the
  reciprocal depth in a single float4, so the inner loop replaces ~13 instructions with one LDS.128;
  collected_id is not needed any more;
* forward: the depth reciprocal is fetched once per Gaussian (already done in round 4);
* backward: bg_dot_dpixel is computed once per pixel and shuffled with the other per-pixel state.

Result (time, /tmp/r5/time): 1.3637 (223.358 s; windows 60.66 / 82.71 / 79.99 vs 63.16 / 85.56 /
82.55 in round 4). check --quick (/tmp/r5/quick): PASS (exact items identical).
Numerics tier: N0 (bit-exact).
3.4 hRound 6 — fused Adam: four elements per thread
Start: 2026-11-09 08:33 UTC. End: 2026-11-09 08:50 UTC.
The Adam kernel walked the parameters one float per thread; it now walks a float4 per
thread (the same operations in the same order, so the result is bit for bit the same).
Result (time, /tmp/r6/time): 1.3787 (220.9 s). check --quick: PASS.
Commit 44f445f — tier N0.
3.4 hRound 7 — rasterizer backward: unsigned pixel index and one reciprocal per pixel
Start: 2026-11-09 08:50 UTC. End: 2026-11-09 09:05 UTC.
The backward kernel computed the pixel index with a signed modulo/division and divided
by the depth for every pixel; both are done with unsigned arithmetic and one reciprocal
per pixel now (the values are the same).
Result (time, /tmp/r7/time): 1.3706 (222.2 s). check --quick: PASS.
Commit c892524 — tier N0.
3.4 hRound 8 — rasterizer: `__expf` instead of `exp` (REJECTED, reverted)
Start: 2026-11-09 09:05 UTC. End: 2026-11-09 09:20 UTC.
__expf (the fast hardware exponential) instead of exp in the forward and the
backward. Result (time, /tmp/r8/time): 1.4514 (209.8 s) — the fastest so far —
but check --n1 INCORRECT: on seed 1 the SSIM bias was -1.006e-3 against a bound of
6.89e-4 (-4.38 sigma), PSNR -1.87 sigma, LPIPS +2.94 sigma, log(count) -2.11 sigma.
The model is chaotic: a change of one ulp in the rasterizer ends up at 2-4 sigma after
30,000 iterations. From here on every change has to be bit exact. The __expf change
was reverted.
3.4 hRound 9 — rasterizer: empty gradient buffers + kernel-side zero writes (REGRESSION)
Start: 2026-11-09 09:20 UTC. End: 2026-11-09 09:35 UTC.
The five gradient buffers that the backward writes in full were allocated with
torch::empty instead of torch::zeros, and the kernels wrote the zeros for the
Gaussians that were not rendered.
Result (time, /tmp/r9/time): 1.3277 (229.4 s) — a 3 % regression. The profile
(/tmp/r10/profile) showed why: the per-Gaussian backward preprocessing went from
0.47 ms to 1.16 ms. The scattered per-Gaussian zero stores are much more expensive
than the 300 MB bulk memset they replace.
3.4 hRound 10 — rasterizer: reciprocal hoisted out of the tile loop
Start: 2026-11-09 09:35 UTC. End: 2026-11-09 09:40 UTC.
The reciprocals of the conic's diagonal depend only on the Gaussian, so they are
computed once per Gaussian instead of once per tile it overlaps
(duplicateWithKeys 0.49 ms -> 0.34 ms). Bit exact.
3.4 hRound 11 — rasterizer backward: the per-pixel state comes from memory
Start: 2026-11-09 09:40 UTC. End: 2026-11-09 10:04 UTC.
Profile of the current best version (round 10, /tmp/r10/profile, step 7000):
GPU busy 7.9 ms/iter; PerGaussianRenderCUDA<3u> (the backward) 2.68 ms (33 %),
renderCUDA<3u> (the forward) 1.29 ms (16 %), the per-Gaussian backward
preprocessing 1.16 ms (14 %, the zero-write experiment), the fused Adam 0.69 ms,
duplicateWithKeys 0.34 ms, the radix sort 0.24 ms.
SASS of the backward's inner loop (cuobjdump on the judge's own build): **176
instructions per pixel iteration, 18 of them global loads** (9 for the per-pixel
state, 9 for the bucket state), 5 shuffles, 20 address IMADs.

Changes (all bit exact):
* the per-pixel state (final transmittance, number of contributors, the colour and
  depth gradients, the background term) is read by the lane that needs it instead of
  being passed along the warp with shuffles;
* the zero-fill experiment of round 9 was reverted (the buffers are zeroed again);
* duplicateWithKeys computes the reciprocals once per Gaussian (round 10);
* prune_optimizer does not empty the caching allocator on every prune;
* the progress bar reads the loss back every 100 iterations instead of every 10.

Result (time, /tmp/r11/time): 1.4263 (213.5 s; windows 57.7 / 79.1 / 76.7).
check --quick (/tmp/r11/quick): PASS (the exact items are identical).
Commit 6dd3745 — tier N0.
3.4 hRound 12 — rasterizer backward: the per-pixel state in two float4s
Start: 2026-11-09 10:04 UTC. End: 2026-11-09 10:21 UTC.
The backward read six scalars per pixel (three colour gradients, the depth gradient,
the final transmittance, the number of contributors) plus the three components of the
background colour, and multiplied the background colour with the gradients for every
pixel of every bucket. A small kernel now packs the state into two float4s
(packPixelStateCUDA): the colour and depth gradients in one, the final
transmittance, the background term and the number of contributors in the other. The
background term is accumulated in the same order with the same fused multiply-add, so
it is the same number.
The inner loop of the backward drops from 176 to 152 instructions and from **18 to
11 memory instructions** (the two float4 loads plus the nine loads of the bucket state).

Result (time, /tmp/r12/time): 1.5148 (201.1 s). check --quick
(/tmp/r12/quick): PASS (the exact items are identical).
Commit f9ddce8 — tier N0.
3.7 hRound 13 — rasterizer: pack the bucket state and the pixel colour into float4s
Start: 2026-11-09 10:21 UTC. End: 2026-11-09 10:40 UTC.
The forward wrote the state that starts every bucket (the transmittance T and the
three colour sums C[0..2]) into four separate arrays (sampled_T, sampled_ar[3]),
and the backward read them with five separate global loads per pixel. They are now
written as one float4 (sampled_TC) and read with one load. The colour and the
inverse depth of the pixel (which the backward needs per pixel) are written directly
by the forward as one float4 (pixel_color_invdepth), which also removes the two
device-to-device copies that used to copy the rendered colour and the inverse depth
into the chunk.

Bit exact: the values stored are the same bits, and the backward still computes
-color + C[ch] and -invdepth + ard in the same order.

The inner loop of the backward drops from 152 to 129 instructions and from **11 to
5 memory instructions** per pixel iteration.

Result (time, /tmp/r13b/time): 1.6169 (188.4 s; windows 51.1 / 69.7 / 67.6).
check --quick (/tmp/r13b/quick): PASS (the exact items are identical).
Commit eae55f2 — tier N0.
4.4 hRound 14 — the full check of round 13 FAILED the statistical gate: the fused Adam was not bit-exact
Start: 2026-11-09 10:45 UTC. End: 2026-11-09 11:20 UTC.

The full check of the round-13 tree (/tmp/r14c/check/check.json) came back INCORRECT: gate 2
(exact items) passed, gate 3 failed with SSIM -5.58 sigma, LPIPS +3.55 sigma, PSNR -2.09 sigma.
No full check had ever been run before (rounds 1-13 only ran check --quick, which compares the
exact items and not the trained model), so the culprit could be any of the changes, not only the last
three.

### Diagnosis (cheap probes instead of 5-minute N1 runs)
* The judge's reference record (bench/records/ref.json) has the number of Gaussians after 3,000
  steps: 1,092,971 (the bootstrap run of the reference gave 1,095,412, so the run-to-run spread of
  the *count* is ~0.2 %). The candidate gave 1,056,479 (-3.3 %): a systematic difference, not noise.
* A probe that runs the tree's own entry for 3,000 steps takes 20 s, so the switches could be bisected
  with it: without switches 1,094,119, GS_VECTOR_STATS=1 1,094,136, GS_ASYNC_LOSS=1 1,096,426,
  GS_FUSED_ADAM=1 1,053,183. The fused Adam was the culprit; the CUDA changes (rounds 5, 11, 12,
  13) are fine.
* torch.norm over a strided view and over a gathered copy is bit-identical (checked), so the
  vectorised densification statistics are not the problem either.

### Root cause
create_from_pcd builds the positions with torch.tensor(np.asarray(pcd.points)), which keeps the
strides of the numpy array: self._xyz is column-major, i.e. not contiguous. The fused Adam
kernel walks the parameter, the gradient and the optimizer state as flat buffers, so for the whole
training (the positions only become contiguous at the first densification, iteration 600) it wrote
the update of the first Gaussian into the first elements of the buffer and left the other positions
untouched: the position update was simply wrong. Torch's own Adam uses the foreach route, which
respects the layout, which is why the reference is right.

### Fix (commit ba310c4)
FusedAdam.step uses the fused kernel only when the parameter and the two state tensors are
contiguous, and otherwise falls back to the operations of torch.optim.Adam (lerp_,
mul_+addcmul_, sqrt/div/add, addcdiv_) - the same operations the reference runs, so the
values are the same. The fallback is only used for the first 600 iterations.

### Verification
* The parameters of the first steps are now identical to the ones of the torch.optim.Adam path
  (only the last-bit differences of the rasterizer's atomics remain).
* The number of Gaussians after 3,000 steps: 1,093,883 with the fused Adam, against 1,093,831 and
  1,095,026 for two runs of the reference path (the broken version: 1,053,183).
4.9 hRound 15 — the fix of round 14 verified, and the first full check
Start: 2026-11-09 11:20 UTC. End: 2026-11-09 11:43 UTC.

The fused-Adam fix (commit ba310c4) was verified with the cheap probes:
* the number of Gaussians after 3,000 steps: 1,093,883 (fused Adam) against
  1,093,831 / 1,095,026 for two runs of the reference path (1,053,183 with the broken
  fused Adam);
* the parameters of the first steps are identical to the ones of the torch.optim.Adam
  path (only the last-bit differences of the rasterizer's atomics remain).

Full check (/tmp/r15/check/check.json): PASS, tier N1, speedup 1.4846.
Gate 2 (exact items) passed; gate 3: SSIM -1.29 sigma, LPIPS -0.67 sigma,
PSNR -2.08 sigma, log(count) +0.84 sigma (all inside +/- 3 sigma).
4.9 hRound 16 — the zero-fill of the gradient buffers with a device memset
Start: 2026-11-09 11:30 UTC. End: 2026-11-09 11:42 UTC.

Profile of the steady state (bench.py profile --from 7010 --iters 20, /tmp/r17/profile):
5.95 ms of GPU time per iteration, of which ~0.29 ms was spent filling zeroed
buffers (aten::fill_ / aten::zeros inside the rasterizer: the eleven gradient
buffers of the backward, the colour image, the inverse depth image and the radii).

torch::zeros / torch::full launch the generic elementwise fill kernel, which
reaches only a fraction of the memory bandwidth. The eleven gradient buffers of the
backward are large (the SH gradient alone is P x 45 floats). They are now allocated
with torch::empty and zeroed with cudaMemsetAsync on the current stream: a float
0.0f is all-zero bits, so the bytes written are exactly the same.

* time (/tmp/r18/time): 1.5961 (190.8 s; windows 52.1 / 70.3 / 68.4).
* check --quick (/tmp/r18/quick): PASS (the exact items are identical).
* Commit cc91305 — tier N0.
4.9 hRound 17 — the full check of round 16
Start: 2026-11-09 11:43 UTC. End: 2026-11-09 11:51 UTC.

Full check (/tmp/r19/check/check.json): PASS, tier N1, speedup 1.6018.
Gate 3: PSNR -0.94 sigma, SSIM -0.05 sigma, LPIPS -1.20 sigma, log(count) +0.42 sigma.
The submission is now a verified N1 at 1.60x.
6.7 hRound 18 — the reciprocal and the dummy screen-space tensor (no-ops)
Start: 2026-11-09 12:00 UTC. End: 2026-11-09 12:25 UTC.

Two micro-changes in the tree: __frcp_rn(1.0f-alpha) instead of the division in the
rasterizer backward (verified: identical SASS, 360 instructions, the same opcode
histogram - a no-op, reverted) and torch.empty_like instead of torch.zeros_like for the
dummy screen-space mean tensor of gaussian_renderer.render (the rasterizer never reads
the values, only the .grad of the leaf matters).
* time (/tmp/r20/time): 1.5681; check --quick (/tmp/r20/quick): PASS.
* Commit f1983df - tier N0.
6.7 hRound 19 — 8x8 tiles instead of 16x16 (measured: a wash)
Start: 2026-11-09 12:25 UTC. End: 2026-11-09 12:32 UTC.

The rasterizer renders a Gaussian against every pixel of every tile its bounding box
overlaps, so a smaller tile rejects most of the (Gaussian, pixel) pairs early: with 8x8
tiles the forward and backward render kernels need 2.2x less pixel work (3.10 ms -> 1.83 ms
per iteration), but the binning gets 4x more expensive (the rectangle of a Gaussian covers
4x more tiles): duplicateWithKeys 0.35 -> 1.22 ms, the radix sort 0.22 -> 0.50 ms. Net: a
wash.
* time (/tmp/r21b/time): 1.5912; check --quick (/tmp/r21b/quick): PASS.
* The number of Gaussians after 3,000 steps moves from 1,093,883 to 1,100,796 (+0.6 %), i.e.
  the finer culling (a tile is kept if one of its pixels has alpha >= 1/255) changes the
  image slightly - it stays inside the 3 sigma of the statistical gate, but it eats margin.
6.7 hRound 20 — the tight tile box
Start: 2026-11-09 12:35 UTC. End: 2026-11-09 13:01 UTC.

Profile of the steady state (bench.py profile --from 7010 --iters 20, /tmp/r23/profile):
5.36 ms of GPU time per iteration. The binning (duplicateWithKeys 1.05 ms + the radix sort
0.26 ms) and the two render kernels (1.20 + 0.58 ms) are the hot spots.

A Gaussian is rasterized against every tile of its 3-sigma square, but the square is
the bounding box of the *largest* principal axis in both directions. The tiles that can
change the image are the ones overlapping the ellipse of the pixels whose alpha is at least
1/255 (q <= log(255*opacity)), whose axis-aligned bounding box is
sqrt(2k*cov_xx) x sqrt(2k*cov_yy) - much tighter for an elongated Gaussian. Every tile
that is dropped this way contains only pixels with alpha < 1/255, which the render kernel
skips anyway, so the image is unchanged; the tile rectangles are stored per Gaussian
(int2* rects) and used by both the preprocess and the binning kernel.
* time (/tmp/r22/time): 1.7606 (173.0 s; windows 49.2 / 62.7 / 61.1), against 1.6018.
* check --quick (/tmp/r22/quick): PASS (exact items identical).
* check --n1 (/tmp/r23/n1): PASS, tier N1 - PSNR -0.53 sigma, SSIM -0.71 sigma,
  LPIPS -1.46 sigma, log(count) +1.09 sigma (all inside 3 sigma).
* Commits cb3d2f0 (8x8 tiles) and df2879f (the tight box) - tier N1.
6.7 hRound 21 — one warp per Gaussian in the binning kernel
Start: 2026-11-09 13:09 UTC. End: 2026-11-09 13:35 UTC.

Diagnostic (a build that skips the Gaussians whose rectangle has more than 1024 tiles):
duplicateWithKeys falls from 1020 us to 293 us - **a few Gaussians with a huge rectangle
(a big Gaussian close to the camera) serialize the whole kernel**, because the kernel gave
one *thread* per Gaussian and a thread walks the whole rectangle of its Gaussian alone, so
a warp can only advance as fast as its slowest Gaussian.

One warp per Gaussian now: the 32 lanes walk the rectangle together, and the pairs
that pass the opacity test are written with a ballot (__ballot_sync/__popc), so the
entries of a Gaussian are bit-identical to the ones the old kernel wrote (the image does
not change at all).
* time (/tmp/r26/time): 1.976 (154.1 s), against 1.7606 before.
* check --quick (/tmp/r26/quick): PASS (exact items identical).
* Commit 00fd509 - tier N0 (bit-identical output).

Profile after the change (4.76 ms of GPU time per iteration):
| kernel | ms/step | share |
|---|---|---|
| PerGaussianRenderCUDA<3u> (rasterizer backward) | 1.236 | 26 % |
| fused Adam | 0.696 | 15 % |
| preprocessCUDA<3> (rasterizer backward) | 0.452 | 9.5 % |
| duplicateWithKeys | 0.441 | 9 % |
| renderCUDA<3u> (rasterizer forward) | 0.407 | 8.5 % |
| CUB radix sort | 0.261 | 5.5 % |
| device memsets | 0.153 | 3 % |
| preprocessCUDA<3> (rasterizer forward) | 0.145 | 3 % |
| the torch ops of the training loop (rotation normalisation, SSIM, loss, ...) | ~0.8 | 17 % |
7.3 hRound 22 - the depth path of the rasterizer and the exponential of the rejected pairs
Start: 2026-11-09 13:40 UTC. End: 2026-11-09 14:12 UTC.

Profile of the tree at the start of the round (/tmp/r26/profile, 4.76 ms of GPU
time per iteration): rasterizer backward 1.85 ms (39 %), rasterizer forward 1.47 ms
(31 %), fused Adam 0.70 ms (15 %), the torch ops of the loop 0.72 ms (15 %).
The CPU runs at about the same rate as the GPU (the wall clock is 5.14 ms per
iteration), so every percent of GPU time is a percent of the wall clock.

Two changes, both inside the rasterizer:

1. The exponential of the rejected pairs (alpha_power_threshold in
   auxiliary.h). The alpha of a (Gaussian, pixel) pair is
   min(0.99, opacity * exp(power)) and the pair is skipped when it is below
   1/255, that is when power < -log(255 * opacity). The exponential is the most
   expensive part of that test and most of the pixels of the tiles a Gaussian
   touches fail it, so the threshold is now computed once per Gaussian and the
   power is compared with it *before* the exponential is evaluated. The margin
   (0.01) makes the test conservative: every pair that passes it is still tested
   exactly as before, so the set of accepted pairs - and therefore the image - does
   not change. The forward kernel got 5 % faster (578 -> 550 us in the profile of
   /tmp/r27/profile); in the backward kernel the test did not pay off (the inner
   loop is register bound), so it was left out there.
2. The depth path of the rasterizer is now switched off when the depth
   regularization is inactive, which is the case of the recipe (--depths "", so
   no camera has a depth map). The rasterizer then does not compute the inverse
   depth of the image, does not carry the inverse depth of the buckets through the
   sampled state, and does not evaluate the depth part of the gradients (the
   gradient of the depth image is exactly zero when the depth is not used, so the
   gradients are unchanged). The flag travels from train.py (the condition of
   the depth regularization) through gaussian_renderer.render and the rasterizer
   settings into the four kernels that touch the depth. The per-Gaussian gradient
   buffer of the inverse depth is not even allocated any more.

* time (/tmp/r29/time): 2.0559 (148.2 s; windows 41.3 / 54.2 / 52.7),
  against 1.976 before.
* check --quick (/tmp/r29/quick): PASS (the exact items of the first 3,000
  steps are identical).
* Numerics: N0 (the image and all the gradients are bit for bit the same).
* An earlier attempt at the same round (/tmp/r27) put the exponential test into
  the backward kernel as well; it was neutral (1.9694) and was reverted.
8.4 hRound 23 - the spherical harmonics gradient, written in a contiguous way
Start: 2026-11-09 15:00 UTC. End: 2026-11-09 15:16 UTC.

Profile of the tree at the start of the round (/tmp/r32/profile, 4.61 ms of GPU
time per iteration): rasterizer backward 1.06 ms (23 %), rasterizer forward 0.52 ms
(11 %), the fused Adam 0.65 ms (14 %), the backward preprocessCUDA 0.49 ms
(10.5 %), the binning kernel 0.45 ms (10 %), the radix sort 0.26 ms (6 %), the
device memsets 0.15 ms (3 %), the torch ops of the loop about 0.60 ms (13 %).

The hotspot of the round is the backward preprocessCUDA. It computes, for every
Gaussian, the gradient of the loss with respect to its 15 spherical harmonics
coefficients, and writes it to the buffer of shape [P][15][3] that the optimizer
reads. The buffer gives every Gaussian 180 bytes, so the 45 floats of a Gaussian
are contiguous but the Gaussians of a warp are 180 bytes apart: the 45 stores of
a thread touch 45 different cache lines, and the 45 stores of the 32 threads of a
warp touch 1440 of them. The stores of the warp use 12 % of the bandwidth they
move.

The gradients are now staged in the shared memory of the block and written to the
global buffer by the block at the end of the kernel, in the order the buffer uses:
the 45 floats of the Gaussians of a block are then written by the 32 threads of a
warp one after the other, so the stores are contiguous. The staging costs 45
shared stores and 45 shared loads per thread, and it replaces 1440 cache line
transactions per warp by 135. The buffer is not zeroed any more, because the
kernel fills every element of it (the Gaussians that are not visible get a zero
gradient, as before).

The values are the ones the old code computed: computeColorFromSH is unchanged
except for the place where it writes its result, so the arithmetic, and the way
the compiler contracts it, are the same. This was checked with a test program
(/tmp/shgx) that runs the old and the new preprocessCUDA on the same random
data and compares the 2,250,000 floats of the gradient buffer: **0 of them
differ**.

* time (/tmp/r36/time): 2.2058 (138.08 s), against 2.0559 before.
* check --quick (/tmp/r35/quick): PASS (the exact items of the first 3,000
  steps are identical).
* Numerics: N0 (the gradient buffer is bit for bit the same).
* A first attempt at this round put the computation of the gradients in a
  separate kernel (computeSHGradCUDA) that recomputed the direction of the
  Gaussians. It was 1 to 5 ULP off the reference (the compiler contracted the
  length of the direction differently in the new context) and the run crashed
  with SIGSEGV, so it was dropped in favour of the staging above.
8.8 hRound 24 - the normalisation of the quaternions, in one kernel
The profile of the tree at the start of the round (/tmp/r37/profile, 4.86 ms of
GPU time per iteration over the window 7,000..7,019) put the composite op
torch.nn.functional.normalize(self._rotation, dim=1) at about 0.29 ms of the
iteration, that is 6 % of it. It is not one kernel but eight: the norm is
reduced twice (once in the forward, once again in the backward of the norm, as
torch recomputes it), the sum of the four squares is a reduction over the
broadcast dimension of the division, and the rest are elementwise passes over
the quaternions and their gradients. In the trace the two reductions of the
norm cost 77 us per iteration, the reduction of the division 48 us, and the
elementwise passes about 150 us more.

The normalisation is now a single CUDA kernel per direction
(submodules/diff-gaussian-rasterization/quat_normalize.cu, behind the
GS_FUSED_NORMALIZE switch), which computes the norm, the divisor and the
result, and in the backward the whole chain of the gradient.

The arithmetic is the one of torch, operation by operation and rounding by
rounding, and it was measured before writing the kernel:

* the norm is sqrt((q0*q0 + q1*q1) + (q2*q2 + q3*q3)); the four squares are
  summed in pairs, which is what the reduction of linalg_vector_norm does on
  this shape (the sequential sum differs from it on 12 % of the samples);
* the divisor is max(n, 1e-12), and the result is q / d;
* the backward is u_k = (q_k/d)/d, t_k = -g_k * u_k,
  gd = (t0+t1) + (t2+t3), gn = (n >= eps) ? gd : 0,
  gx_k = (g_k/d) + gn*(q_k/n); the sum of the four t_k is again in pairs,
  and the product gn*(q_k/n) is formed as it is written above and not as
  (gn*q_k)/n (which differs on 17 % of the samples);
* a zero norm is replaced by one before the division by the norm, as torch does
  in the backward of the norm, so that the gradient of a zero quaternion is
  zero and not a NaN.

The kernel uses __fmul_rn, __fadd_rn, __fdiv_rn and __fsqrt_rn, so that
the compiler cannot contract the operations into fused multiply-adds, which
would round differently.

The comparison with torch over about 30 million quaternions (random ones, ones
over ten scales from 1e-8 to 1e30, heavy-tailed ones, and the edge cases: the
zero quaternion, the ones whose norm is exactly the epsilon, the infinite and
the NaN ones) gives 0 differences in the forward and 0 differences in
the backward, and the NaNs land on the same elements (/tmp/normtest).

* time (/tmp/r38/time): 2.396 (127.12 s), against 2.2058 before.
* check --quick (/tmp/r38/quick): PASS (the exact items of the first
  3,000 steps are identical).
* Numerics: N0 (the forward and the backward are bit for bit the ones of
  torch).
9.6 hRound 25 - two experiments on the backward render (no gain, reverted)
The backward render (PerGaussianRenderCUDA, 1081 us per iteration, 26 % of the
time) is the biggest kernel of the iteration.  Two changes were tried on it, both
measured with bench.py profile (the time of the kernel is in the profile):

1. The alpha log-threshold pre-filter.  The forward render skips the
   exponential of a pair whose contribution to the tile is below 1/255 (it
   compares the power with log(1/(255*opacity))).  The same test was added to
   the backward render, before the exponential.  The backward render went from
   1059 us to 1068 us: the backward is not bound by the exponential (its
   last_contributor early exit already removes most of the pairs, and the warp
   divergence prevents the lanes from skipping the test), so the extra
   comparison only costs.
2. The pix_grad4 load moved after the alpha test.  The backward render
   loads the state of the pixel it works on before it knows whether the pair
   contributes.  Deferring the load made it slower (1081 us).

Both changes were reverted (git checkout).  The threshold itself was proved
safe (0 violations over 16 million samples, /tmp/normtest/thr.py), it simply
buys nothing.
9.6 hRound 26 - the occupancy of the backward render (a wash)
The backward render uses 56 registers (nvcc -Xptxas -v) and was launched with
one warp per block, so that the number of resident warps per multiprocessor was
limited by the hardware limit of 32 blocks per multiprocessor and not by the
registers: 32 warps out of the 64 a multiprocessor can hold.

The kernel does not care how many warps share a block (it computes the bucket
index from the warp index), so the launch was made configurable
(GS_BWD_WARPS, off by default, cuda_rasterizer/backward.cu): with four warps
per block the limit becomes the number of registers, 36 warps.

* time (/tmp/r43/time): 2.4032 (126.74 s), against 2.396 before.  The
  difference is inside the noise of the measurement (the same tree measured
  2.3856 in the full check), so the kernel is not bound by its occupancy.
* The change is kept and enabled, it is bit for bit the same computation.
↑ 3DGS