-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: If there are many small chunks in write_parquet(), convert to a single chunk (#14484) #14487
perf: If there are many small chunks in write_parquet(), convert to a single chunk (#14484) #14487
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #14487 +/- ##
=======================================
Coverage ? 80.35%
=======================================
Files ? 1352
Lines ? 174669
Branches ? 2436
=======================================
Hits ? 140353
Misses ? 33844
Partials ? 472 ☔ View full report in Codecov by Sentry. |
@@ -133,7 +133,11 @@ pub fn split_series(s: &Series, n: usize) -> PolarsResult<Vec<Series>> { | |||
split_array!(s, n, i64) | |||
} | |||
|
|||
pub fn split_df_as_ref(df: &DataFrame, n: usize) -> PolarsResult<Vec<DataFrame>> { | |||
pub fn split_df_as_ref( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is technically a public API, though nothing uses it at least in Polars, just split_df()
, so if backwards compat is an issue then this could be a new function and split_df_as_ref
could be a wrapper. (Shame Rust has no default keyword args, makes this sort of change easier to be backwards compat.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @itamarst. Looks good to me. 👍
Fixes #14484
Before:
After: