Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for page-level filter pushdown #105

Closed
wants to merge 4 commits into from
Closed

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Mar 19, 2022

This PR introduces a page iterator that leverages page indexes and column indexes to apply filter push-down at the page level. This is the most efficient way of loading a large row group with a filter as the group can be read without having to read the every page.

Opened for visibility, marked as draft since it is missing tests.

@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2022

Codecov Report

Merging #105 (b5e7751) into main (6511088) will decrease coverage by 3.10%.
The diff coverage is 34.65%.

@@            Coverage Diff             @@
##             main     #105      +/-   ##
==========================================
- Coverage   61.75%   58.65%   -3.11%     
==========================================
  Files          66       69       +3     
  Lines        2782     2999     +217     
==========================================
+ Hits         1718     1759      +41     
- Misses       1064     1240     +176     
Impacted Files Coverage Δ
parquet-tools/src/lib/meta.rs 0.00% <ø> (ø)
src/error.rs 20.00% <0.00%> (-2.23%) ⬇️
src/indexes/index.rs 0.00% <0.00%> (ø)
src/indexes/read.rs 0.00% <0.00%> (ø)
src/metadata/column_descriptor.rs 100.00% <ø> (+26.66%) ⬆️
src/page/page_dict/mod.rs 78.94% <0.00%> (ø)
src/read/page/indexed_reader.rs 0.00% <0.00%> (ø)
src/read/page/stream.rs 0.00% <0.00%> (ø)
src/schema/types/basic_type.rs 100.00% <ø> (ø)
src/schema/types/converted_type.rs 35.71% <ø> (ø)
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6511088...b5e7751. Read the comment docs.

@jorgecarleitao jorgecarleitao force-pushed the use_indexes branch 3 times, most recently from 467e1ef to b5e7751 Compare March 20, 2022 18:02
@jorgecarleitao jorgecarleitao marked this pull request as ready for review March 25, 2022 20:39
@jorgecarleitao
Copy link
Owner Author

Closing in favor of #107

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants