Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: speed up PrefixRecord loading #984

Merged
merged 10 commits into from
Dec 16, 2024
Merged

Conversation

wolfv
Copy link
Contributor

@wolfv wolfv commented Dec 16, 2024

I have tried out rayon to parallelize the loading of PrefixRecord files. The results look pretty good:

load_serially           time:   [10.369 ms 10.403 ms 10.440 ms]
load_as_prefix_record   time:   [697.90 ns 699.63 ns 701.53 ns]
load_as_package_record  time:   [698.94 ns 700.45 ns 702.11 ns]

Unfortunately loading the subset (PackageRecord) doesn't seem to be a big improvement.

@wolfv
Copy link
Contributor Author

wolfv commented Dec 16, 2024

For very long PrefixRecord's it does seem to make a difference (20k files):

load_long_prefix_record time:   [1.8900 ms 1.8961 ms 1.9029 ms]
load_long_package_record time:  [380.69 µs 382.43 µs 384.41 µs]

@wolfv wolfv merged commit 35658e9 into conda:main Dec 16, 2024
14 of 15 checks passed
@baszalmstra baszalmstra mentioned this pull request Dec 16, 2024
@wolfv wolfv deleted the speed-up-record-loading branch December 18, 2024 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants