Repository navigation
Conversation
quantize_histogram could panic with "index out of bounds" in Cluster::split on nearly uniform images (one colour plus a few antialiasing shades, e.g. a 512x512 map tile). The means were sums of colour x weight accumulated in f32, which round once they pass 2^24 (one colour on ~66k pixels). A cluster's mean could then fall outside the values it averages (254.99998 with every alpha at 255): the rounding error made a constant channel look like the widest one, every entry compared above the mean, and `gt -= 1` underflowed. The rounded sums also made the means depend on the histogram's iteration order. - Cluster::calc_stats and kmeans accumulate colour x weight and the weight in u64 and divide once in f64. Below 2^45 pixels the sums convert to f64 exactly, so each mean is the correctly rounded value: within the channel's values, equal to them when they are all the same, and independent of the iteration order. A constant channel therefore has zero spread and is never picked for a split, and the split pivot always has entries on both sides. - `split` uses an exclusive upper bound, so it cannot underflow even if every entry lands on the same side. - The f32 SIMD add_color helpers are removed; the integer loop is plain Rust and the compiler vectorises it. The spread sums (add_diff) and kmeans' total_err stay in f32: they only rank channels and clusters. Output only changes where the rounded sums mattered (palette order or ties with the same error on a few real map tiles, and a better palette on the histogram that used to panic). Quantization speed is unchanged within measurement noise. Tests: the histogram that panicked, 1024x1024 to 4096x4096 near-uniform histograms asserting every cluster mean stays within its values, and a seeded loop of random near-uniform histograms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0123cwhkXd7j38L65hK9hzU1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We hit this in production rendering map tiles: some nearly uniform 512×512 tiles (one colour plus
a few antialiasing shades) made
quantize_histogrampanic withindex out of bounds: the len is 4 but the index is 18446744073709551615inCluster::split(attempt to subtract with overflowindebug).
Repro (quantizr 1.4.3, any Rust ≥ 1.85): a 512×512 image that is one colour plus 17
antialiasing shades, quantized to 16 colours.
Cause. The cluster means are sums of colour × weight accumulated in f32, which round once they
pass 2^24 (one colour on ~66k pixels). The mean can then land outside the values it averages
(
254.99998with every alpha at 255): that error alone makes a constant channel the "widest", everyentry compares
> mean,gt -= 1underflows and the nextswapis out of bounds. The same roundingmakes the means depend on the histogram's iteration order.
kmeanscomputes the final palettecolours with the same kind of f32 sums.
Change (
src/cluster.rs,src/colormap.rs):Cluster::calc_statsandkmeansaccumulate colour × weight and the weight in u64 and divideonce in f64. Below 2^45 pixels both convert to f64 exactly, so each mean is the correctly rounded
value: within the values, equal to them when they are all the same, independent of iteration order.
A constant channel then has zero spread and is never picked for a split.
splituses an exclusive upper bound, so it cannot underflow even if every entry lands on one side.add_colorhelpers (and theirunsafeintrinsics) go away: the integer loop is plainRust and the compiler vectorises it. The spread sums (
add_diff) andtotal_errstay in f32, asthey only rank channels and clusters.
Effect. On 480 real map tiles (256² and 512²) plus upscaled and synthetic near-uniform images up
to 4096², with 16–256 colours: no panics; output identical to 1.4.3 except a handful of palette-order
ties with the same error, and the histogram above now gets 16 colours instead of panicking.
Quantization speed, benchmarked with criterion on a dedicated machine, is within ±1.5% of 1.4.3,
which is the same spread we got from rebuilding 1.4.3 itself. We also tried an f64 SIMD
version (measurably slower) and a u32 SIMD one (no measurable gain, more code), so we kept the
simplest.
Tests (in
cluster.rs): the histogram above, 1024² to 4096² near-uniform histogramsasserting every cluster mean stays within its values, and a seeded loop of random near-uniform
histograms up to 4096². All three fail on master.
cargo fmt --checkandcargo test --all-features --testspass on 1.85 and 1.90;cargo clippy --all-features -- -D warningsonly reports the two
collapsible_ifthat current stable clippy already flags invpsearch.rsonmaster (untouched here; #3 also fixes them).
🤖 Generated with Claude Code