Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Optimization of ExperimentAxisQuery to_anndata #3359

Merged
merged 4 commits into from
Nov 21, 2024
Merged

Conversation

bkmartinjr
Copy link
Member

@bkmartinjr bkmartinjr commented Nov 21, 2024

Apply partitioning optimization and improved concurrency to the ExperimentAxisQuery.to_anndata, similar to the approach used in #3328

Other changes related to this:

  • removed numba package dependency, as it is no longer needed
  • removal of dead code in both ExperimentAxisQuery and the old CSR converter
  • remove obsolete comments related to resource handling (no more reference cycles now that we use the context thread pool).
  • fixed a constant value typo in outgest
  • style/cleanup of small amount of code in ExperimentAxisQuery._read
  • improve empty matrix handling in CompresedMatrix.from_soma

As part of testing I validated this on both the sparse-with-dups and sparse-without-dups readers via S3. There was a performance improvement on both.

Copy link

codecov bot commented Nov 21, 2024

Codecov Report

Attention: Patch coverage is 94.33962% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.85%. Comparing base (ea3cd1a) to head (155fcad).
Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3359      +/-   ##
==========================================
+ Coverage   85.64%   85.85%   +0.20%     
==========================================
  Files          57       56       -1     
  Lines        6201     6107      -94     
==========================================
- Hits         5311     5243      -68     
+ Misses        890      864      -26     
Flag Coverage Δ
python 85.85% <94.33%> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
python_api 85.85% <94.33%> (+0.20%) ⬆️
libtiledbsoma ∅ <ø> (∅)
---- 🚨 Try these New Features:

@johnkerl johnkerl changed the title optimization of ExperimentAxisQuery to_anndata [python] Optimization of ExperimentAxisQuery to_anndata Nov 21, 2024
@bkmartinjr bkmartinjr marked this pull request as ready for review November 21, 2024 18:31
Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super-cool @bkmartinjr ! Just some minor comments.

apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_query.py Show resolved Hide resolved
apis/python/src/tiledbsoma/io/outgest.py Show resolved Hide resolved
Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢
Thanks @bkmartinjr !! :)

)

approx_X_shape = tuple(b - a + 1 for a, b in matrix.non_empty_domain())
# heuristically derived number (benchmarking). Thesis is that this is roughly 80% of a 1 GiB io buffer,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bkmartinjr !

@bkmartinjr bkmartinjr merged commit 4d7bff2 into main Nov 21, 2024
21 checks passed
@bkmartinjr bkmartinjr deleted the eaq-x-reader branch November 21, 2024 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants