Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: Add upfront partitioning in ColumnChunkMetadata #18584

Merged
merged 1 commit into from
Sep 7, 2024
Merged

perf: Add upfront partitioning in ColumnChunkMetadata #18584

merged 1 commit into from
Sep 7, 2024

Conversation

nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Sep 6, 2024 •
edited
Loading

Also byte-range calculation

Benchmark FORCE_ASYNC=1 with projection -

# 1.6.0 (pypi)
shape (1000, 50000)
python .env/y.py  6.65s user 0.25s system 109% cpu 6.290 total
# This branch (build-opt)
shape (1000, 50000)
python .env/y.py  0.92s user 0.22s system 257% cpu 0.445 total

@github-actions github-actions bot added performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars labels Sep 6, 2024
@nameexhaustion nameexhaustion force-pushed the column-chunk-metadata branch 3 times, most recently from 4c743f4 to 34e30f5 Compare September 6, 2024 12:12
@nameexhaustion nameexhaustion marked this pull request as ready for review September 7, 2024 07:42
@ritchie46
Copy link
Member

Wow, that's an impressive improvement. It shows we had a lot of accidental linear scans allover. :)

@ritchie46 ritchie46 merged commit de344d6 into pola-rs:main Sep 7, 2024
15 of 21 checks passed
@c-peters c-peters added the accepted Ready for implementation label Sep 9, 2024
@nameexhaustion nameexhaustion deleted the column-chunk-metadata branch September 11, 2024 08:04
@cmdlineluser cmdlineluser mentioned this pull request Sep 13, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 ritchie46 approved these changes

@orlp orlp Awaiting requested review from orlp orlp is a code owner

@c-peters c-peters Awaiting requested review from c-peters c-peters is a code owner

@coastalwhite coastalwhite Awaiting requested review from coastalwhite coastalwhite is a code owner

Assignees

@nameexhaustion nameexhaustion

Labels
accepted Ready for implementation performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars
Projects
Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nameexhaustion @ritchie46 @c-peters