Support Spark-compatible native hash and xxhash64 for decimals with precision >18

[sunchao]

What is the problem the feature request solves?

Comet cannot currently evaluate SQL hash and xxhash64 natively when an input contains a decimal with declared precision above 18, including decimals nested in arrays, structs, and maps. Enabling native support would keep these expressions and their surrounding operators in native execution without JVM codegen dispatch.

The gap remains on main at 58ab5f618e1e715dee06165424672fdd820cafe4:

• The native wide-decimal branch hashes the unscaled i128 using to_le_bytes(), a fixed-width 16-byte little-endian representation.
• Spark's decimal hashing uses toJavaBigDecimal().unscaledValue().toByteArray() above precision 18: minimal-length signed two's-complement big-endian bytes.
• Comet's SQL admission guard rejects these types recursively, protecting SQL hash results from this mismatch.

For example, an unscaled value of 128 in DECIMAL(20,0) is hashed by Spark as 00 80; the existing native branch hashes 80 followed by fifteen zero bytes. The encoding choice depends on declared precision, even when a particular value fits in i64.

Java BigDecimal is not intrinsically required to reproduce this representation. It can be computed directly from Arrow's unscaled integer in Rust.

Describe the potential solution

Implement Spark-compatible wide-decimal encoding in the native hash path, then admit the supported decimal inputs for both SQL hash functions. Preserve the existing unscaled-long hashing path for precision <=18.

Acceptance criteria:

• Both hash (Murmur3) and xxhash64 match Spark for precision 19-38, including small values, values beyond 64 bits, precision-38 extremes, zero, negatives, and sign-padding/byte-length boundaries such as 127/128 and -128/-129.
• Preserve null handling, multi-column hash chaining, and seeds. Test non-default seeds through the native/Catalyst APIs rather than treating an extra SQL hash argument as a seed.
• Cover decimals in arrays, structs, and maps where Spark permits hashing, including null containers, null parent structs, and null decimal children.
• End-to-end Spark parity tests require native execution and no JVM dispatcher activity, so fallback cannot make the tests pass silently. Retain narrow-decimal native controls.
• Review and regression-test the shared native shuffle caller when changing the common encoding helper. Different partition numbers from Spark alone are not evidence of incorrect query results.
• Add a focused benchmark against the existing fallback/dispatcher paths, with matched results and explicit routing checks. Report measured results without assuming a speedup.

Keep dynamic sha2, TimeType, and enabling wide decimals for approx_count_distinct outside this issue's scope.

Additional context

• Result mismatch with vanilla spark in hash function with decimal input #1294 reported the original decimal hash mismatch. fix: partially fix consistency issue of hash functions with decimal input #1295 fixed the narrow-decimal path; fix: Fall back to Spark when hashing decimals with precision > 18 #1325 closed the remaining issue by adding Spark fallback for precision >18.
• Hash functions fall back to Spark for decimal precision > 18, and sha2 for a non-literal numBits #5581 and feat: route wide-decimal hash and non-literal sha2 through the codege… #5835 address these unsupported expressions through JVM codegen dispatch. This issue tracks the separate native implementation.
• Coordinate with feat: route compatible xxhash64 args through SparkXxhash64 #5960 / Consolidate date/timestamp truncation and xxhash64 with upstream DataFusion implementations #5103, which consolidate compatible xxhash64 cases with upstream SparkXxhash64; native admission must be justified by Spark parity on the selected implementation.
• Hash partitioning not falling back to Spark for high precision decimal partition keys #3079 discussed wide-decimal shuffle partitioning and was closed after noting that partition IDs need not match Spark.

This is a native-support enhancement with an encoding-correctness prerequisite. The source analysis above does not establish an exposed wrong-result regression in the currently guarded SQL path, and no new native implementation or benchmark has been run for this issue.