HIVE-29854: Iceberg: Estimate a pruned scan's column statistics from the table's manifests - #6740
Open
deniskuzZ wants to merge 7 commits into
Open
HIVE-29854: Iceberg: Estimate a pruned scan's column statistics from the table's manifests#6740deniskuzZ wants to merge 7 commits into
deniskuzZ wants to merge 7 commits into
Conversation
deniskuzZ
changed the title
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 30, 2026 10:22
d9600eb to
9013747
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 30, 2026 11:56
9013747 to
fee3e19
Compare
asf-ci-hive
added
tests unstable
tests pending
tests passed
and removed
tests pending
tests unstable
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 31, 2026 14:25
fee3e19 to
dfcd4b0
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 31, 2026 18:26
dfcd4b0 to
b6abd65
Compare
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 07:52
b6abd65 to
1c72550
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 09:19
1c72550 to
ae6dee8
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 09:16
a560fd8 to
f24dc11
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 18:14
f24dc11 to
174b238
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
2 times, most recently
from
September 2, 2026 18:41
aafea8b to
0ca87bf
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 23:01
0ca87bf to
bd4d154
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 3, 2026 18:51
bd4d154 to
de3b2ef
Compare
…umn statistics MIN, MAX and COUNT over a column are facts the stored statistics already state, so a query asking only for them is answered from what a storage handler holds rather than by reading the rows. The statistics of every aggregate in a query are fetched at once, and a partitioned table is answered only from statistics that describe the partitions, the columns and the snapshot the scan asks about - a partition whose statistics do not cover every asked column, or which a live delete of no named partition may have changed, is not answered for. A table's size comes from its storage handler rather than from listing what its location holds, and whether a join can be a sort-merge is decided before its big table is elected, so an election made on the handler's numbers is not undone by one made on the listing's.
Entries in a partition blob answer by their field id even when a read decodes the whole blob, so what a dropped column left behind is stepped over rather than served under a namesake added since. And values aggregated from only some of the partitions a scan reads carry that mark, so the filter-reduction rule keeps estimating from them but never folds a predicate to a constant over them.
…gregates A partition-level file holds the table-level aggregates at its tail, so a session reading at table level answers from them rather than finding nothing at its own granularity - but only while the file states the table, which its registered entry marks. A gather over every partition states it, and a merge does while the file it carried from did; a partition-scoped gather with nothing to carry describes its partition alone and marks nothing. The write side stays strict: a table-level increment merges only into a file gathered as one, and a partition-level increment only into a partition-level file, so the leniency is a read-side courtesy over what is physically already there, never a cross-granularity merge.
The reduce rule folds IS NULL and IS NOT NULL when a column's null count equals the row count. The null count comes from the snapshot the scan reads - a branch or as-of, resolved through the handler - so the row count must come from the same snapshot, not the current table's metastore parameters. A branch null count read against the main row count folds IS NOT NULL to false and drops the branch's non-null rows.
The ScanColStats doc claimed an unanswerable aggregate always costs nothing, but the fetch is lazy on the first aggregate that needs it, so a later one that cannot answer has already paid. And a partition-stats test comment read that a span of a written and an untouched partition cannot be answered, where both partitions are in fact fresh and the span is answered - the stale-subset case it describes is the one above.
The metastore holds one unversioned set of parameters describing the current table, so a branch, a tag or a point in time cannot be answered from it. getBasicStatistics guarded only the named-ref case and read the current parameters for an as-of scan; guarding on the whole qualifier, as the row count and the column-stats freshness already do, sends every versioned scan to its own snapshot summary.
…the table's manifests Column statistics stored for a table describe every partition of it, so a scan pruned to part of the table is handed bounds it mostly cannot hold. Where a column's values follow the partition key - a measure that grows with time, an id range loaded per batch - the bounds are wrong by the whole spread of the table, and a predicate outside the partition the scan reads is estimated as satisfiable. The manifests already record what each file holds. Where no per partition statistics answer, the bounds and null counts they state answer for the files the scan reads, folded a file at a time - a file states its metrics once for all of its columns, so what holds them is read where the file is in hand rather than again by each column of it. A column the spec partitions by identity is bounded by the partition value itself, which is exact and is stated even for a file that recorded no metrics of its own. They are not the answer, they are what the filter is then estimated from: a distinct count still comes from the statistics a gather stored, and a column without one states nothing. Planning the scan reads the manifests while a query compiles, so it plans with its own pool rather than the one split generation plans with. Off by default, under hive.iceberg.stats.use.manifests.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
depends on #6707