Nested predicate push down to Parquet Reader #7045
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
3 times, most recently
from
6afa71e to
c073b6b
Compare
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
2 times, most recently
from
ffe3335 to
47dfe1c
Compare
|
@dain @nezihyigitbasi @martint any comments or suggestions? |
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
from
a80cd4f to
6334ef9
Compare
|
@martint @dain @nezihyigitbasi comments or suggestions? |
|
In an offline discussion we decided that @zhenxiao was going to investigate using synthetic virtual-columns in the connector to enable this push-down feature. |
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
from
6334ef9 to
6422c0a
Compare
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
from
6422c0a to
3aef172
Compare
zhenxiao
force-pushed
the
parquet-nested-predicate
branch
from
3aef172 to
0f34d07
Compare
|
I assume #13271 superceds this one, hence, closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Built on top of
#6892
Currently Parquet TupleDomain is constructed based on HiveColumnHandle. This would not work if Nested predicate are pushed down, e.g.
select s.a from t where s.b > 10In this implementation:
Analyze s.b, and put s.b > 10 as an optional Nested predicate in ExtrationResult
add Nested predicate to TableLayout
pass Nested predicate to File Scan, same as flat predicate
Skip reading row groups when Nested predicate does not match Parquet statistics