Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Statistics #686

Closed
norberttech opened this issue Oct 31, 2023 · 1 comment · Fixed by #755
Closed

Parquet Statistics #686

norberttech opened this issue Oct 31, 2023 · 1 comment · Fixed by #755
Labels
lib-parquet
Milestone
0.5.0

Comments

@norberttech
Copy link
Member

Parquet Writer is already collecting statistics, but it's not turning them into ColumnChunk statistics that can be later used by the Reader to reduce the number of partitions required to read when looking for specific data.
@norberttech
Copy link
Member Author

DataPageV2 is also allowing to create statistics for a given data page, this might be useful for data page pruning while reading chunks which in case of very large datasets might bring significant performance improvements.

@norberttech norberttech added this to the 0.5.0 milestone Nov 6, 2023
This was referenced Nov 6, 2023
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lib-parquet
Projects
Roadmap
Archived in project
Milestone
0.5.0
Development

Successfully merging a pull request may close this issue.

686-parquet-statistics norberttech/flow
Added DataPageV2 statistics norberttech/flow
1 participant
@norberttech