Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Support datetime in predicate during hive partition pruning #17545

Merged
merged 2 commits into from
Jul 11, 2024

Conversation

nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Jul 10, 2024

We can now apply predicates on date/datetime parts:

lf = pl.scan_parquet(
    "hive_dates/date1=2024-01-01/date2=2023-01-01%2000%3A00%3A00.000000/00000000.parquet",
    hive_partitioning=True,
)

lf.filter(pl.col("date1") == date(1970, 1, 1)).collect()
# parquet file can be skipped, the statistics were sufficient to apply the predicate.

Closes #14712

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Jul 10, 2024
@nameexhaustion nameexhaustion marked this pull request as ready for review July 10, 2024 07:35
@nameexhaustion nameexhaustion changed the title feat: Support datetime in predicate during hive partition pruning perf: Support datetime in predicate during hive partition pruning Jul 10, 2024
@github-actions github-actions bot added the performance Performance issues or improvements label Jul 10, 2024
Copy link

codecov bot commented Jul 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.47%. Comparing base (daf2e49) to head (c97cdd0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #17545      +/-   ##
==========================================
- Coverage   80.48%   80.47%   -0.01%     
==========================================
  Files        1483     1483              
  Lines      195118   195118              
  Branches     2778     2778              
==========================================
- Hits       157039   157022      -17     
- Misses      37568    37585      +17     
  Partials      511      511              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ritchie46 ritchie46 merged commit 662696a into pola-rs:main Jul 11, 2024
27 checks passed
@nameexhaustion nameexhaustion deleted the date-predicate branch July 19, 2024 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

File list pruning with hive partition columns doesn't work with date type
2 participants