IN LIST: optimize Utf8View and BinaryView filters - #24088
Merged
alamb merged 4 commits intoSep 13, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24088 +/- ##
==========================================
- Coverage 81.90% 81.90% -0.01%
==========================================
Files 1133 1134 +1
Lines 424993 425156 +163
Branches 424993 425156 +163
==========================================
+ Hits 348083 348216 +133
- Misses 56278 56293 +15
- Partials 20632 20647 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 5, 2026 07:02
ec9cb2e to
65b67ba
Compare
Contributor
Author
|
run benchmark in_list |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_listCPU Details (lscpu)Details
Resource Usagein_list — base (merge-base)
in_list — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (65b67ba) to db0c31b (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 6, 2026 16:01
65b67ba to
a44ee0d
Compare
geoffreyclaude
changed the title
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
August 9, 2026 06:21
a44ee0d to
b0443b6
Compare
12 tasks
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
10 times, most recently
from
August 11, 2026 16:29
61cc7a6 to
156d76b
Compare
github-actions
Bot
added
auto detected api change
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
2 times, most recently
from
August 12, 2026 10:18
6a5284e to
f8ae3ad
Compare
Contributor
Author
|
run benchmark in_list_strategy |
alamb
approved these changes
Sep 10, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thank you @geoffreyclaude -- this looks good to me
| /// `u128` views. | ||
| struct ByteViewFilter<T: ByteViewType> { | ||
| inner: StaticFilterRef, | ||
| _marker: PhantomData<T>, |
Contributor
There was a problem hiding this comment.
Nothing in the filter's behavior actually depends on T — both StringViewArray and BinaryViewArray , we could potentially use a single structure here and dispatch at runtime rather than compile time and save compilation time.
Contributor
Author
There was a problem hiding this comment.
We could, but there are only two possible ByteViewTypes, so we won't win much here!
EDIT: Although on deeper look, it does propagate quite a bit, so I'll rework this part.
|
|
||
| fn contains(&self, v: &dyn Array, negated: bool) -> Result<BooleanArray> { | ||
| let array = downcast_byte_view::<T>(v)?; | ||
| self.inner.contains(&as_decimal128(array), negated) |
Contributor
There was a problem hiding this comment.
this is correct because the haystack is guaranteed to be all inline views, right? And thus we are guaranteed that any larger view in v can not possible match
Contributor
Author
There was a problem hiding this comment.
yes exactly, I can add a clarifying comment over this line
Contributor
|
I apologize for the delay in reviewing |
Contributor
|
I wonder if merging this PR will also close this issue? |
Contributor
|
CI failure I think is due to
|
Treat inline Utf8View and BinaryView values as 128-bit primitive keys and pass them to the shared primitive filter selector. This reuses direct comparisons for short lists and the primitive hash-set path for larger lists without reading backing buffers. Lists containing a non-inline value continue to use the general filter. Exact view types, dictionaries, slices, nulls, IN, and NOT IN keep their existing behavior.
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
September 11, 2026 06:53
e31670c to
d75dad7
Compare
geoffreyclaude
force-pushed
the
codex/in-list-byte-view-hashset
branch
from
September 11, 2026 07:32
d75dad7 to
28809d8
Compare
Contributor
Author
|
run benchmark in_list_strategy |
Contributor
Author
@alamb Probably... I think we still need to work on the OR-rewrite for lists of 4 items or less before the epic can be completely closed. EDIT: I added a new sub-task for the OR-rewrite to the main issue (as well as a fix PR for floating point zeroes, which were not behaving consistently.) |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-byte-view-hashset (28809d8) to 7e5f40a (merge-base) diff Run configurationrun benchmark in_list_strategyResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-byte-view-hashset (28809d8) to 7e5f40a (merge-base) diff Run configurationrun benchmark in_list_strategyCPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
This was referenced Sep 11, 2026
Contributor
|
Merging |
Contributor
|
Thanks again for your patience @geoffreyclaude |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 11, 2026
Contributor
|
merge queue failure seems to be infrastructure: https://github.com/apache/datafusion/actions/runs/34646039081/job/103420287587 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 11, 2026
Contributor
Author
Thank you for your meticulous reviews and always great feedback! |
alamb
enabled auto-merge
September 13, 2026 20:13
Contributor
That is kind of you to say -- I just wish I had about 2x more time to devote to reviewing PRs |
Contributor
|
Woohoo ! We got it in! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Arrow represents every
Utf8ViewandBinaryViewvalue with a 16-byte view. When a value is at most 12 bytes long, the complete value is stored directly in that view:Equal short values therefore have the same 128-bit view. DataFusion can look up that view directly, without reading another buffer or using Arrow's general value comparison.
This PR uses that faster path when every non-null value in the
INlist is at most 12 bytes long. The byte-view adapter treats each inline view as aDecimal128key and passes it to the shared primitive selector from #24283, the same selector used by #24102:Decimal128is only a 16-byte container here; no decimal operations are performed.A longer input value cannot match a short list value because its encoded length is different, so it can be rejected without reading its backing bytes. If the list itself contains a value longer than 12 bytes, DataFusion keeps using the general filter.
Null list entries do not affect this choice, but are still recorded for SQL null behavior. After unwrapping any dictionary, the input and list must use the same view type:
Utf8Viewis not mixed withBinaryView, and regularUtf8andBinarykeep their existing paths.What changes are included in this PR?
Utf8ViewandBinaryView.IN, andNOT IN.Are these changes tested?
Tests cover both view types and both faster paths; the 4/5-value and 12/13-byte boundaries; long inputs and lists that require the general filter; exact-type routing; slices and dictionaries; input and list nulls; and
INandNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Automated benchmark snapshot
Automated run on an aarch64 Neoverse V2 runner, comparing #24102 with this PR at exact commit
00dd083e. The benchmark evaluates 8,192-row batches after filter construction. Lower is better.All 15 reported rows improved, with a 35.9% geometric-mean time reduction.
utf8view/short_8b/list=4/match=0%utf8view/short_8b/list=4/match=50%utf8view/short_8b/list=16/match=0%utf8view/short_8b/list=16/match=50%utf8view/short_8b/list=64/match=0%utf8view/short_8b/list=64/match=50%utf8view/short_8b/list=256/match=0%utf8view/short_8b/list=256/match=50%utf8view/len_12b/list=16/match=0%utf8view/len_12b/list=16/match=50%utf8view/len_12b/list=64/match=0%utf8view/len_12b/list=64/match=50%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_INnulls/utf8view/short_8b/list=16/match=50%/nulls=50%