Skip to content

fix: make Parquet file schema type coercion work on nested schemas - #25193

Open
adragomir wants to merge 1 commit into
apache:mainfrom
hstack:fix-make-parquet-schema-type-coercion-nested
Open

fix: make Parquet file schema type coercion work on nested schemas#25193
adragomir wants to merge 1 commit into
apache:mainfrom
hstack:fix-make-parquet-schema-type-coercion-nested

Conversation

@adragomir

@adragomir adragomir commented Sep 11, 2026

Copy link
Copy Markdown
  • ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields.
  • For some Parquet schemas, where we request a view transform for a nested field, this is translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data on that nested field, the actual predicate execution needs a cast, because the schemas are different - the predicate refers to a string view, but the actual data schema refers to a string

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

  • Modification of the apply_file_schema_type_coercions function to work on nested fields
  • Respective tests for the functionality

What is the testing strategy for this PR?

  • All tests pass
  • Newly added unit tests pass

Are there any user-facing changes?

  • Unsure ? Probably not ? The documentation in the function does NOT explicitly say the current behavior.

- ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields. 
- For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different.
@github-actions github-actions Bot added the datasource Changes to the datasource crate label Sep 11, 2026
@aditanase

Copy link
Copy Markdown
Contributor

cc @adriangb - this is closely related with the nested schema pruning work that is ongoing. We discovered this by investigating a 2x slowdown on delta-rs next vs old TableProviders. Turned out that inconistent Utf8View handling in nested schemas was triggering the more expensive Utf8 decoding unnecessary

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.52381% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.92%. Comparing base (517ac6d) to head (3248cda).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
...tafusion/datasource-parquet/src/schema_coercion.rs 99.52% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25193    +/-   ##
========================================
  Coverage   81.91%   81.92%            
========================================
  Files        1132     1132            
  Lines      421311   421520   +209     
  Branches   421311   421520   +209     
========================================
+ Hits       345137   345337   +200     
- Misses      55768    55774     +6     
- Partials    20406    20409     +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields

3 participants