NEON-vectorized union2by2

[gitRasheed]

NEON kernel for union2by2, the array-container union, on arm64. Fixes #288. Both arrays need at least 32 elements and 128 between them. Smaller inputs keep the scalar assembly from #287.

This replaces #538, which carried a partial block between steps and only paid off from 384 elements per side. This one carries nothing. Each step takes a 16-element window from both sides, lets x be the smaller of the two window maxima, sorts all 32 through a bitonic merge network, drops repeats and lanes above x, stores the rest, and advances each cursor by its count of values at or below x. Every step retires a closed value interval, so there is no seam state and the unread tails are strictly above the last stored value. Windows entirely below the other side are copied straight through. An 8-lane stage handles the last blocks and inputs under 64, then the scalar merge takes the tails.

Array unions had a hand-written scalar path on arm64 and nothing wider.

I made a page that steps through this kernel on worked examples and annotates the assembly line by line, to make the design choices easier to follow: https://claude.ai/code/artifact/5f77af0d-89c2-4965-9d15-95fde361c47f#union (the other tabs cover the other three kernels).

Changes

• setutil_arm64.go: the lane-compaction table (its own commit, shared with NEON-vectorized intersection2by2 #543, NEON-vectorized difference #568 and NEON-vectorized exclusiveUnion2by2 #569 so the four merge in any order) and the dispatch: thresholds, the zero-length buffer lazyorArray passes with capacity, and the in-place self-union geometry (set2 at the output base with set1 an identical copy), which goes to the scalar routine because its reads stay ahead of its writes.
• setutil_neon_part_arm64.s: the kernel.
• setutil_arm64.s: Union2by2 arm64 assembly #287's routine renamed union2by2scalar, unchanged.
• Tests against a sort-and-dedup oracle: the dispatch boundaries, the container-level paths through iorArray and lazyorArray, the partition prefixes the kernel returns at several sizes and shapes, window ties, mixed window ownership, and copy-on-write bitmaps.
• BenchmarkUnion2By2 rotating fixed-seed datasets, and BenchmarkUnionRealData over the real-roaring-datasets corpus behind BENCH_REAL_DATA=1.

Testing

go test passes for all packages on Graviton 4, for this branch alone and with the other three kernel branches merged in. go fmt clean. Smat fuzzer (through the wrapper noted in #538, -tags gofuzz), 300 seconds on Graviton 4: 286,276 executions, no failures.

Performance

Graviton 4 (c8g.xlarge), Go 1.26.2, medians of five runs. BenchmarkUnion2By2 from this PR, dispatch against the #287 scalar routine, ns/op:

dense/64x64                   76.6      95.9   (1.25x)
dense/256x256                239.9     938.8   (3.91x)
dense/4096x4096             3472     24413     (7.03x)
sparse/256x256               248.4    1022     (4.11x)
sparse/4096x4096            3562     25869     (7.26x)
interleaved/256x256          226.5     394.3   (1.74x)
interleaved/4096x4096       3186      6104     (1.92x)
runs/256x256                 240.7     387.5   (1.61x)
runs/4096x4096              3662      5990     (1.64x)
identical/4096x4096         3038      3126     (1.03x)
sharedprefix/4096x4096      2811      3116     (1.11x)

Real-roaring-datasets, master against this branch, BenchmarkUnionRealData over all twelve datasets, raw and optimized, pairwise, accumulated and FastOr: geomean 7 percent less time over 72 cells, with census-income raw accumulated 663 to 428 µs, census1881 raw FastOr 3.18 to 2.13 ms, wikileaks-noquotes raw accumulated 2.11 to 1.63 ms. No allocation changes.