-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[python] Optimization of ExperimentAxisQuery
to_anndata
#3359
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3359 +/- ##
==========================================
+ Coverage 85.64% 85.85% +0.20%
==========================================
Files 57 56 -1
Lines 6201 6107 -94
==========================================
- Hits 5311 5243 -68
+ Misses 890 864 -26
Flags with carried forward coverage won't be shown. Click here to find out more.
|
ExperimentAxisQuery
to_anndata
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is super-cool @bkmartinjr ! Just some minor comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚢
Thanks @bkmartinjr !! :)
) | ||
|
||
approx_X_shape = tuple(b - a + 1 for a, b in matrix.non_empty_domain()) | ||
# heuristically derived number (benchmarking). Thesis is that this is roughly 80% of a 1 GiB io buffer, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @bkmartinjr !
Apply partitioning optimization and improved concurrency to the ExperimentAxisQuery.to_anndata, similar to the approach used in #3328
Other changes related to this:
As part of testing I validated this on both the sparse-with-dups and sparse-without-dups readers via S3. There was a performance improvement on both.