HIVE-30019: ProbeDecode for the vectorized Parquet reader - #6758
Open
abstractdog wants to merge 3 commits into
Open
HIVE-30019: ProbeDecode for the vectorized Parquet reader#6758abstractdog wants to merge 3 commits into
abstractdog wants to merge 3 commits into
Conversation
abstractdog
force-pushed
the
probe-decode-parquet
branch
from
September 5, 2026 09:57
faf24ac to
b468102
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
Port the Hive ProbeDecode runtime path from the ORC-encoded LLAP reader to
the vectorized Parquet reader on Apache master. When a mapjoin is small and
has a much smaller key ratio than its big side, the compiler already tags
the big-side TableScan with a ProbeDecodeContext; this change makes
VectorizedParquetRecordReader honour that context.
Runtime flow, per batch:
1. Look up the small-side VectorMapJoinHashTable from the ObjectCache via
the cache key on ProbeDecodeContext (lazy, once per reader).
2. Decode the probe-key column first (plain 3-arg readBatch).
3. Probe each row against the long-key hash table -> bitmap filter.
4. Decode the remaining columns via the new
readBatch(int, ColumnVector, TypeInfo, ParquetProbeFilter) overload.
For filtered-out rows the reader either skips the value on the page
(dataColumn.skip()) or -- when the page is dict-encoded -- marks the
dict-id slot null so decodeDictionaryIds' downstream materialisation
short-circuits. The filter is honoured uniformly across every primitive
read helper: readDictionaryIDs, readIntegers, readSmallInts,
readTinyInts, readLongs, readFloats, readDoubles, readBooleans,
readDecimal, readDecimal64, readString, readChar, readVarchar,
readBinaries, readDate, readTimestamp. The biggest per-row saving is on
byte-array types (string/char/varchar/binary) where filtered rows no
longer trigger a BytesColumnVector.setVal allocation + copy;
BinaryPlainValuesReader.skip() only reads the length prefix and
advances the buffer.
5. Compact the surviving rows into batch.selected[] so downstream operators
don't re-test them.
Fast-path linkage to parquet-java:
- readDictionaryIDs coalesces contiguous filtered rows into a single
dataColumn.skip(n). That reaches DictionaryValuesReader.skip(int) ->
RunLengthBitPackingHybridDecoder.skipInts, the bulk-skip fast-path
added upstream in parquet-java, which consumes a whole RLE run in O(1).
This is the largest single win because dictionary indices are always
RLE / bit-packed regardless of column type.
- Other readers use per-row dataColumn.skip(); the underlying PLAIN /
DELTA_BINARY_PACKED / BinaryPlain readers have no bulk-skip fast-path
(PLAIN is a byte-offset bump, DELTA is cumulative, BinaryPlain is a
per-value length read), so coalescing would gain nothing.
Interface changes:
- VectorizedColumnReader gains a default filter-aware readBatch overload
(list/map/struct/dummy inherit the default no-op delegation, primitive
reader overrides).
- ParquetDataColumnReader gains skip() and skip(int); skip(int) delegates
to ValuesReader.skip(int) so readDictionaryIDs' per-batch coalescing
reaches the parquet-java bulk skip.
Compiler:
- TezCompiler.removeSemijoinsParallelToMapJoin no longer gates the
ProbeDecodeContext plumbing on LLAP mode -- Parquet reads on Tez
non-LLAP consume it too now. Readers that don't consume it (regular
ORC, text) simply ignore the extra hint on the TableScanOperator.
Scope for this first cut: single long/int key probe. String, multi-key and
Decimal64 key-probe variants and q-test golden updates follow up.
Unit tests: all 43 existing parquet-vector reader tests
(TestVectorizedColumnReader, TestVectorizedDictionaryEncodingColumnReader,
TestVectorizedListColumnReader, TestVectorizedMapColumnReader) still pass.
Adds three test artifacts covering the Parquet ProbeDecode path landed in
HIVE-30019:
1. Q-test (probedecode_mapjoin_simple_parquet.q) mirroring the ORC template
probedecode_mapjoin_simple.q. Sets up item_dim_pq + orders_fact_pq stored
as parquet, runs the join both with and without hive.optimize.scan.probedecode
so the golden captures the plan (with EXPLAIN VECTORIZATION DETAIL) and
asserts the result set matches the baseline. Auto-picked up by
MiniLlapLocalCliConfig's sweep of ql/src/test/queries/clientpositive; the
.q.out golden will be generated on first CI run.
2. TestParquetProbeFilter (10 tests) -- pure unit test of the
ParquetProbeFilter contract: newBitmap null-guard, isSelected bounds and
bitmap fidelity, compact() idempotency, and correct materialization of
selected[] under all-accept / all-reject / mixed / empty / smaller-batch
cases.
3. TestVectorizedParquetProbeDecodeReader (4 tests) -- end-to-end test that
writes a Parquet file, opens VectorizedParquetRecordReader, reflects out
the per-column readers, and drives the filter-aware readBatch signature
directly (bypassing the MapJoin-operator plumbing that nextBatch needs to
resolve ProbeDecodeState). Verifies:
- Surviving rows decode correctly across int / long / double / string.
- Filtered rows come back as null-marked slots.
- noNulls is cleared once any filtered row is emitted.
- allPass filter produces vectors identical to the unfiltered baseline.
Runs on both dictionary-encoded pages (exercises readDictionaryIDs ->
pendingSkip -> RunLengthBitPackingHybridDecoder.skipInts fast-path) and
PLAIN-encoded pages (per-row dataColumn.skip()).
4. VectorizedParquetProbeDecodeBench (JMH) -- crosses two encodings (dict,
plain) with four filter shapes (no-filter baseline, all-pass, all-fail,
half). Establishes the regression floor for the isFilteredOut branch
added to every primitive read helper and quantifies the win on filter=
all-fail (dict path drops entire runs via skipInts).
Test results:
- TestParquetProbeFilter: 10/10 pass
- TestVectorizedParquetProbeDecodeReader: 4/4 pass
- No regression on TestVectorizedColumnReader (19), TestVectorized-
DictionaryEncodingColumnReader (13), TestVectorizedListColumnReader (6),
TestVectorizedMapColumnReader (5)
- itests/hive-jmh compiles clean
Signed-off-by: Laszlo Bodor <bodorlaszlo2011@gmail.com>
- HiveConf: text block for the plain-filter description (S6126, LineLength) - VectorizedColumnReader: continuation indent for the filter-aware default - VectorizedParquetRecordReader: split nextBatch into probe/plain helpers to drop cognitive complexity from 37 to well under 15 (S3776) - VectorizedPrimitiveColumnReader: extract readDateValue helper (S3776); suppress S107 on the 9-arg constructor with a rationale - ParquetProbeDecodeState: pattern-matching instanceof (S6201); replace the TODO comment with a follow-up note (S1135) - ParquetProbeLongHashTable: switch expression over the hash-table kind (S6880); reshape the probe loop to remove the two 'continue's (S135) - VectorizedParquetReadBench (JMH): checked exceptions in place of 'throws Exception' (S112), FileInputFormat static setInputPaths (S3252), Files.deleteIfExists in tearDown (S4042/S899), package-private @PARAM fields (VisibilityModifier), and rationale-tagged suppressions for the reflection (S3011) and temp-dir (S5443) that are inherent to a JMH benchmark - Tests: drop the LinkedHashMap declaration (IllegalTypeCheck), fix '{ x }' whitespace and single-line block layout, remove unused locals in TestVectorizedParquetProbeDecodeReader Ignoring the switch-case indentation alerts on ParquetProbeLongHashTable per the reviewer. Verified: mvn compile (common, ql, itests/hive-jmh) and the three probe-decode unit test classes still pass. Co-Authored-By: Claude Code <noreply@anthropic.com>
abstractdog
force-pushed
the
probe-decode-parquet
branch
from
September 6, 2026 19:22
b468102 to
99a98be
Compare
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| */ | ||
| private ParquetProbeFilter probeFilter; | ||
|
|
||
| @Setup(Level.Trial) |
Contributor
Author
|
benchmark results: different filter selectivity, done for plain and dictionary types: Dict path: the bulk-skip fast path pays. At 10% pass (skip 90% of rows), reject runs coalesce into single RunLengthBitPackingHybridDecoder.skipInts calls and the bench saves ~24% wall-clock. At 50% pass it's still ~18%. At 90% pass, the gap collapses because most rows still materialize. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?