Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet: Support identifying hierarchical partitioning and schema from Spark/Dask metadata files #868

Closed
rcaudy opened this issue Jul 19, 2021 · 1 comment
Closed

Parquet: Support identifying hierarchical partitioning and schema from Spark/Dask metadata files #868

rcaudy opened this issue Jul 19, 2021 · 1 comment
Assignees
@rcaudy
Labels
core Core development tasks feature request New feature or request parquet Related to the Parquet integration
Milestone
July 2021

Comments

@rcaudy
Copy link
Member

rcaudy commented Jul 19, 2021

We can and should be able to filter partitions according to the values they contain when thrift metadata files are present.
Part of #294, maybe.
@rcaudy rcaudy added feature request New feature or request core Core development tasks parquet Related to the Parquet integration labels Jul 19, 2021
@rcaudy rcaudy added this to the July 2021 milestone Jul 19, 2021
@rcaudy rcaudy self-assigned this Jul 19, 2021
@rcaudy rcaudy mentioned this issue Jul 22, 2021
Closed
@rcaudy rcaudy changed the title Support range indices derived from parquet metadata files Parquet: Support identifying hierarchical partitioning and schema from Spark/Dask metadata files Aug 4, 2021
@rcaudy
Copy link
Member Author

rcaudy commented Aug 4, 2021

I'm moving the requirements for this issue a bit. We've got good support for partitioning columns driven by by metadata files, but we're lacking support for predicate pushdown to the row group level or below. See #968 for future work.

@rcaudy rcaudy mentioned this issue Aug 6, 2021
Merged
@rcaudy rcaudy closed this as completed Aug 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcaudy rcaudy

Labels
core Core development tasks feature request New feature or request parquet Related to the Parquet integration
Projects
None yet
Milestone
July 2021
Development

No branches or pull requests
1 participant
@rcaudy