Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Remove double open For SOMAArray reads #3293

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

nguyenv
Copy link
Member

@nguyenv nguyenv commented Nov 4, 2024

Issue and/or context:

This finishes up #3053 [sc-54744]

Changes:

  • Remove clib.SOMAArray.open in all SOMAArray derived class read calls
  • Pass query arguments coords, column names, result order, value filter, and platform config that were formerly passed to clib.SOMAArray.open or clib.SOMAArray.reset instead to TableReadIter and SparseNDArrayRead
  • _set_coords on the ManagedQuery rather than through SOMAArray
  • _arrow_table_reader and SparseTensorReadIterBase initialize a ManagedQuery object and set the query arguments listed above; set up the read; and submit reads and return results until the query is complete

@nguyenv nguyenv linked an issue Nov 4, 2024 that may be closed by this pull request
@johnkerl johnkerl changed the title [python] Remove Double Open For SOMAArray Reads [python] Remove double open For SOMAArray reads Nov 4, 2024
Copy link

codecov bot commented Nov 4, 2024

Codecov Report

Attention: Patch coverage is 94.04762% with 5 lines in your changes missing coverage. Please review.

Project coverage is 85.32%. Comparing base (d0cbd2d) to head (9124aff).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3293      +/-   ##
==========================================
+ Coverage   85.09%   85.32%   +0.22%     
==========================================
  Files          53       53              
  Lines        5577     5567      -10     
==========================================
+ Hits         4746     4750       +4     
+ Misses        831      817      -14     
Flag Coverage Δ
python 85.32% <94.04%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
python_api 85.32% <94.04%> (+0.22%) ⬆️
libtiledbsoma ∅ <ø> (∅)

Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really really nice --a well-thought-out refactor, well executed. Just some minor mods requested. :)

apis/python/src/tiledbsoma/_dense_nd_array.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_dense_nd_array.py Outdated Show resolved Hide resolved
apis/python/src/tiledbsoma/_read_iters.py Outdated Show resolved Hide resolved
@nguyenv nguyenv force-pushed the viviannguyen/sc-54744/python-soma-read-does-an-open-on-an-already branch from 89cd943 to 47c6f9f Compare November 5, 2024 00:31
@johnkerl johnkerl self-requested a review November 5, 2024 00:45
Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

Copy link
Member

@bkmartinjr bkmartinjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Two small items:

  1. platform_config handling - noted inline
  2. ManagedQuery GIL - the read path moves some logic up from C++ to Python -- we need to make sure the pybind entry points are all releasing GIL where appropriate. My quick skim is that it looks OK, but I suggest we benchmark after landing this to confirm.

@nguyenv nguyenv force-pushed the viviannguyen/sc-54744/python-soma-read-does-an-open-on-an-already branch 3 times, most recently from ac764f8 to 9124aff Compare November 13, 2024 20:11
@nguyenv nguyenv force-pushed the viviannguyen/sc-54744/python-soma-read-does-an-open-on-an-already branch from 9124aff to b659f76 Compare November 21, 2024 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[python/c++] The read method does an open on an already open array
3 participants