Skip to content

Fix index out of bounds panic in Cluster::split by computing means from exact integer sums - #4

Open
ManuelLR wants to merge 1 commit into
DarthSim:masterfrom
CartoDB:fix/split-panic
Open

ManuelLR wants to merge 1 commit into
DarthSim:masterfrom
CartoDB:fix/split-panic

Conversation

@ManuelLR

Copy link
Copy Markdown

We hit this in production rendering map tiles: some nearly uniform 512×512 tiles (one colour plus
a few antialiasing shades) made quantize_histogram panic with index out of bounds: the len is 4 but the index is 18446744073709551615 in Cluster::split (attempt to subtract with overflow in
debug).

Repro (quantizr 1.4.3, any Rust ≥ 1.85): a 512×512 image that is one colour plus 17
antialiasing shades, quantized to 16 colours.

fn main() {
    let h: [([u8; 4], u32); 18] = [
        ([250, 246, 240, 255], 261911), ([251, 248, 243, 255], 193), ([249, 244, 237, 255], 6),
        ([249, 245, 238, 255], 5), ([249, 246, 239, 255], 4), ([250, 245, 238, 255], 4),
        ([250, 247, 242, 255], 4), ([250, 247, 241, 255], 3), ([250, 246, 239, 255], 2),
        ([250, 247, 240, 255], 2), ([250, 246, 241, 255], 2), ([249, 246, 240, 255], 2),
        ([250, 245, 239, 255], 1), ([250, 245, 240, 255], 1), ([249, 245, 239, 255], 1),
        ([249, 244, 238, 255], 1), ([250, 246, 242, 255], 1), ([251, 248, 242, 255], 1),
    ];
    let mut raw = vec![];
    for (c, n) in h { for _ in 0..n { raw.extend_from_slice(&c); } }
    let img = quantizr::Image::new(&raw, 512, 512).unwrap();
    let mut hist = quantizr::Histogram::new();
    hist.add_image(&img);
    let mut o = quantizr::Options::default();
    o.set_max_colors(16).unwrap();
    let r = quantizr::QuantizeResult::quantize_histogram(&hist, &o); // panics
    println!("{} colours", r.get_palette().count);
}

Cause. The cluster means are sums of colour × weight accumulated in f32, which round once they
pass 2^24 (one colour on ~66k pixels). The mean can then land outside the values it averages
(254.99998 with every alpha at 255): that error alone makes a constant channel the "widest", every
entry compares > mean, gt -= 1 underflows and the next swap is out of bounds. The same rounding
makes the means depend on the histogram's iteration order. kmeans computes the final palette
colours with the same kind of f32 sums.

Change (src/cluster.rs, src/colormap.rs):

  • Cluster::calc_stats and kmeans accumulate colour × weight and the weight in u64 and divide
    once in f64. Below 2^45 pixels both convert to f64 exactly, so each mean is the correctly rounded
    value: within the values, equal to them when they are all the same, independent of iteration order.
    A constant channel then has zero spread and is never picked for a split.
  • split uses an exclusive upper bound, so it cannot underflow even if every entry lands on one side.
  • The f32 SIMD add_color helpers (and their unsafe intrinsics) go away: the integer loop is plain
    Rust and the compiler vectorises it. The spread sums (add_diff) and total_err stay in f32, as
    they only rank channels and clusters.

Effect. On 480 real map tiles (256² and 512²) plus upscaled and synthetic near-uniform images up
to 4096², with 16–256 colours: no panics; output identical to 1.4.3 except a handful of palette-order
ties with the same error, and the histogram above now gets 16 colours instead of panicking.
Quantization speed, benchmarked with criterion on a dedicated machine, is within ±1.5% of 1.4.3,
which is the same spread we got from rebuilding 1.4.3 itself. We also tried an f64 SIMD
version (measurably slower) and a u32 SIMD one (no measurable gain, more code), so we kept the
simplest.

Tests (in cluster.rs): the histogram above, 1024² to 4096² near-uniform histograms
asserting every cluster mean stays within its values, and a seeded loop of random near-uniform
histograms up to 4096². All three fail on master. cargo fmt --check and
cargo test --all-features --tests pass on 1.85 and 1.90; cargo clippy --all-features -- -D warnings
only reports the two collapsible_if that current stable clippy already flags in vpsearch.rs on
master (untouched here; #3 also fixes them).

🤖 Generated with Claude Code

quantize_histogram could panic with "index out of bounds" in
Cluster::split on nearly uniform images (one colour plus a few
antialiasing shades, e.g. a 512x512 map tile).

The means were sums of colour x weight accumulated in f32, which round
once they pass 2^24 (one colour on ~66k pixels). A cluster's mean could
then fall outside the values it averages (254.99998 with every alpha at
255): the rounding error made a constant channel look like the widest
one, every entry compared above the mean, and `gt -= 1` underflowed.
The rounded sums also made the means depend on the histogram's
iteration order.

- Cluster::calc_stats and kmeans accumulate colour x weight and the
  weight in u64 and divide once in f64. Below 2^45 pixels the sums
  convert to f64 exactly, so each mean is the correctly rounded value:
  within the channel's values, equal to them when they are all the same,
  and independent of the iteration order. A constant channel therefore
  has zero spread and is never picked for a split, and the split pivot
  always has entries on both sides.
- `split` uses an exclusive upper bound, so it cannot underflow even if
  every entry lands on the same side.
- The f32 SIMD add_color helpers are removed; the integer loop is plain
  Rust and the compiler vectorises it. The spread sums (add_diff) and
  kmeans' total_err stay in f32: they only rank channels and clusters.

Output only changes where the rounded sums mattered (palette order or
ties with the same error on a few real map tiles, and a better palette
on the histogram that used to panic). Quantization speed is unchanged
within measurement noise.

Tests: the histogram that panicked, 1024x1024 to 4096x4096 near-uniform
histograms asserting every cluster mean stays within its values, and a
seeded loop of random near-uniform histograms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123cwhkXd7j38L65hK9hzU1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant