perf: Improve cloud scan performance #19728
Merged
+498
−137
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
performance
nameexhaustion
commented
Nov 11, 2024
|
|
||
| // Dropping is delayed for tokio async files so we need to explicitly | ||
| // flush here (https://github.com/tokio-rs/tokio/issues/2307#issuecomment-596336451). | ||
| file.sync_all().await.map_err(PolarsError::from)?; |
There was a problem hiding this comment.
drive-by - moved here from the callsite at file_cache
nameexhaustion
commented
Nov 11, 2024
nameexhaustion
commented
Nov 11, 2024
| // We have a dedicated code-path for a full projection that performs a | ||
| // single range request for the entire row group. During testing this | ||
| // provided much higher throughput from cloud than making multiple range | ||
| // request with `get_ranges()`. |
There was a problem hiding this comment.
We had this previously, which was fast if the file contained many nicely sized row groups, but not for the 12,000 column single row group file.
We now just use get_ranges(), which handles the download optimization for us
|
Great speedups! 🙌 |
nameexhaustion
requested review from
ritchie46,
orlp and
c-peters
as code owners
nameexhaustion
commented
Nov 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref #18443
We were previously slow either due to making a single very large request, or making thousands of tiny requests. This PR splits/combines the range requests to make them evenly distributed with a reasonable chunk size.
Benchmarks - the source file is 12,000 columns x 24,000 rows on S3 (see linked issue for generator). Tested on EC2.