[Variant] Remove the extra Spark byte-reconstruction pass

[peterxcli]

What is the problem the feature request solves?

#5868 materializes an Arrow-unshredded value and then rebuild_spark_variant decodes/rebuilds it into Spark's byte representation. Write the Spark representation during the original unshredding traversal, using Arrow's existing typed decoders and value/object/list builders.

This requires more than changing object-key order. #5474 covers that part; it cannot justify deleting the full byte-reconstruction layer.

Verified encoding contract

Direct reconstruction probes against Spark 4.0.4, 4.1.3, and 4.2.0 confirmed the following. The source is ShreddingUtils.rebuild and VariantBuilder.

Input	Spark output requirement
Typed BIGINT 1	Narrow integer bytes 0c01; Arrow 59.3.0 retains Int64
Residual eight-byte integer 1	Preserve 180100000000000000
Typed DECIMAL(38,2) 1.00	Decimal4 bytes 200264000000
Typed string "x"	Short-string bytes 0578
Residual long-string encoding of "x"	Preserve 400100000078
Typed NaN with a noncanonical payload	Canonical Java NaN bits
Residual NaN with that payload	Preserve the payload bits
No typed_value in the schema	Preserve original value and metadata bytes, including unused keys
typed_value exists but is NULL in this row	Rebuild metadata from visited values; preserve residual scalar encoding

Spark registers keys during traversal: typed fields in schema order, recursively, then residual fields in their stored field order. For typed b.inner, then a, then residual z, the dictionary is [b, inner, a, z]. Sorting object field references happens separately and must not reorder the payload or dictionary.

The tested Spark profiles write UTF-16 object-key order and clear the metadata sorted flag. Arrow 59.3.0 emitted 1101000161 for the metadata of {"a":1}; Spark emitted 0101000161. Spark master has a different comparator, so output ordering must follow the supported Spark profile rather than an assumption based on master.

Concrete implementation

1. Add Spark output encoding to the options-based Arrow unshredder proposed in [Variant] Consolidate Spark-compatible missing-value validation and errors #5977. Reuse UnshredVariantRowBuilder and its recursive decoders; keep ordinary unshred_variant defaults unchanged. Compose the writer with metadata rebuilding from [Variant] Replace missing-key metadata repair with native permissive reconstruction #5979, typed precedence from [Variant] Preserve typed-value precedence without residual pre-rewriting #5980, and missing-state checks from [Variant] Consolidate Spark-compatible missing-value validation and errors #5977. Produce the final value and metadata in this traversal; a final BinaryView-to-Binary layout cast is acceptable, a second Variant decode/re-encode is not.

2. Normalize only typed scalars. Apply the logic currently in Comet's spark_typed_scalar at the typed append sites (AppendToVariantBuilder and DecimalUnshredRowBuilder). Choose integer/decimal widths from the value and scale, use the short-string threshold, and canonicalize typed floating-point NaNs while preserving signed zero. Residual append sites must bypass this conversion. Applying it to every append_value would incorrectly narrow residual integers and erase residual NaN/string encodings.

3. Reuse the writable dictionary and object/list builders. Build a fresh dictionary per shredded row, registering only emitted keys before descending into their values. Finish with Spark's metadata flags. For object sorting, add a default comparison method to MetadataBuilder and have ObjectBuilder::finish call it. The default compares real UTF-8 names; the Spark writer overrides it with the required order. This propagates through existing ParentState nesting and replaces Comet's synthetic spark_sort_key strings. Preserve payload insertion order and calculate field IDs/offset widths using the existing builders.

4. Preserve original residual traversal while remapping IDs. Decode residual field names using the source dictionary and register them in the output dictionary. Recursively remap objects, including those inside residual lists; retain scalar encodings. Do not copy raw containers across different dictionaries.

   Legacy residuals require a fallible compatibility reader that visits the original field sequence. Expose/reuse the existing checked shallow parsing and iteration inside parquet-variant (currently try_new_with_metadata_and_shallow_validation and iter_try_with_shallow_validation) for this reconstruction path. Validate bounds, IDs, types, nesting depth, unique names, and UTF-8-or-legacy-UTF-16 ordering as the reader descends. Keep canonical full-validation and lookup contracts unchanged; a legacy object must not be treated as a canonical object suitable for UTF-8 binary search.

   Feeding only a UTF-8-sorted temporary residual into the writer is insufficient: it changes the order in which Spark would assign dictionary IDs. A probe with U+E000 and an emoji confirmed that rebuilding the original legacy residual visits the emoji first. Retain the existing native implementation for legacy inputs until this reader is available.

5. Preserve the schema-level passthrough decision. Only a schema without typed_value takes the unchanged-bytes path. A NULL typed value in a shredded schema still uses the fresh output dictionary. Preserve parent SQL NULL, explicit Variant null, empty keys, and errors. Keep storage normalization and any separately required metadata repair until their own replacements are available.

6. Remove the second pass after integration. Once the pinned dependency provides the writer and required compatibility reading, switch normalize_variant_array to its final output. Delete rebuild_spark_variant, append_spark_variant, and the now-replaced local SparkValueBuilder, SparkOutputMetadata, and spark_typed_scalar code. Remove legacy residual sorting from this path only when the checked reader replaces it. Error translation into Spark remains in Comet.

Verification and completion

• Keep exact value and metadata byte comparisons. Cover signed integer width boundaries, decimal precision/scale boundaries, 63/64-byte strings, typed/residual NaNs and signed zero, metadata flags, unused keys, and NULL typed values.
• Add dictionary traversal and ID/offset-width boundary fixtures, including nested residual containers and multi-row dictionaries. Test Unicode keys in residuals as well as typed objects; include more than Spark's binary-search threshold of fields.
• Run the existing native regressions, CometVariantProjectionSuite, and unchanged upstream Spark Variant assertions on 4.0.4, 4.1.3, and the experimental 4.2.0 profile. Assert native scans. Canonical, legacy, partially shredded, and empty-key inputs must all retain the current contract before closing this issue.
• Repeat benchmark_variant_buffer_reuse and CometVariantReadBenchmark against the current two-pass implementation, using matched files, full byte consumption, and forward/reverse reader order. Measure allocation traffic and scan time separately. Keep results in the implementing PR description.

Additional context

Parent: #5477; current implementation: #5868. These upstream interfaces are proposals. Research verified the listed Spark helper behavior and Arrow 59.3.0 builder differences; it did not implement or benchmark the proposed writer.

The existing PR measurements establish a cost but do not isolate the second pass's share. A canonical-input-only optimization may land first, but this issue remains open until the extra pass is removed for every currently supported input.