[python] Python 3.12 support

[johnkerl]

For issue #1849 -- which has a long history.

See also #2999 and [sc-53002].

This needs single-cell-data/SOMA#222

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.65%. Comparing base (928c281) to head (b7db9ed).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3001      +/-   ##
==========================================
+ Coverage   83.54%   83.65%   +0.11%     
==========================================
  Files          51       51              
  Lines        5434     5434              
==========================================
+ Hits         4540     4546       +6     
+ Misses        894      888       -6     

Flag	Coverage Δ
python	83.65% <ø> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	83.65% <ø> (+0.11%)	⬆️
libtiledbsoma	∅ <ø> (∅)

[thetorpedodog]

right now leaving comments as I go through each commit. apologies if this is already resolved in a later one

[thetorpedodog]

From the build log:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/__init__.py", line 103, in <module>
    from ._collection import Collection
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/_collection.py", line 3[7](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10895777089/job/30234556256#step:18:8), in <module>
    from ._dataframe import DataFrame
  File "/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.10/site-packages/tiledbsoma/_dataframe.py", line 17, in <module>
    from tiledbsoma import _new_shape_feature_flag_enabled
ImportError: cannot import name '_new_shape_feature_flag_enabled' from partially initialized module 'tiledbsoma' (most likely due to a circular import) (/__w/TileDB-SOMA/TileDB-SOMA/venv-soma/lib/python3.[10](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10895777089/job/30234556256#step:18:11)/site-packages/tiledbsoma/__init__.py)

This is because at the point this line is executed, the symbol _new_shape_feature_flag_enabled is not yet available. The line

TileDB-SOMA/apis/python/src/tiledbsoma/__init__.py

Line 138 in a96c72d

_new_shape_feature_flag = os.getenv("SOMA_PY_NEW_SHAPE") is not None

needs to be moved above all the imports. (Alternately, it might make sense to create a tiledbsoma._flags module so that there’s no circular dependency between submodules and the tiledbsoma/__init__.py file.

[johnkerl]

needs to be moved above all the imports

@thetorpedodog thank you!!! I got some 'new style lint' here -- I don't know why, I didn't change any linter versions -- and was forced to move imports :(

There's something weird to debug here -- I'll see if I can make everything in the CI toolchain happy simultaneously -- thank you!!!

Also I'll try the _flags module idea -- this sounds promising -- thank you! :)

[johnkerl]

@thetorpedodog

• [python] Remove pyarrow pin on MacOS #3004
• [python] Move temporary feature flag into a module #3005

And re

There's something weird to debug here -- I'll see if I can make everything in the CI toolchain happy simultaneously -- thank you!!!

I have traced that to here:

https://github.com/single-cell-data/TileDB-SOMA/pull/3004/files#r1763424424

[iosonofabio]

Hi guys, thanks for your amazing package.

I understand this is in flux as we write - as a power user I would kindly like to ask if there's anything I can do to help push this through - I really need it and workarounds are a decent pain.

(As I understand, now there is a buggy typechecker stalling CI stalling 3.12 adoption, which itself was released 12 months ago. If inaccurate feel free to correct)

[johnkerl]

@iosonofabio thank you! Your assessment is correct -- current status is here: #1849 (comment) and @ryan-williams will be moving this forward

[ryan-williams]

I can't add you as a reviewer since you opened this PR, @johnkerl, but I would like your review on the latest changes that I've pushed here 🙏

Depending on typeguard@HEAD may trip up SOMA developers who have existing typeguard 4.2.1 or 4.3.0 installations, but I'm not sure what the other option is. As soon as agronholm/typeguard#490 makes it into a release, we can move to that. Could also try to only depend on typeguard@HEAD in Python 3.12, but I thought that was more complex than it was worth.

[johnkerl]

LGTM!! :)

[johnkerl]

Depending on typeguard@HEAD may trip up SOMA developers who have existing typeguard 4.2.1 or 4.3.0 installations, but I'm not sure what the other option is

Agreed. And it only affects folks on 3.12, so any dev who really wants to avoid this can use 3.11.