[python] Run upgrade-shapes on notebook example experiments #3289

johnkerl · 2024-11-03T15:23:40Z

Issue and/or context: As tracked on issue #2407 / [sc-51048].

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

Changes:

Run tiledbsoma.io.upgrade_experiment_shapes on the notebook example experiments.

Notes for Reviewer:

PRs for notebook content are soon to come, to be stacked on top of this.

Stacking as of today:

main
[python] Include tiledbsoma_upgrade_shape for DenseNDArray #3288
[python] Run upgrade-shapes on notebook example experiments #3289
Branch kerl/dataframe-shape-temp which is a copy of of [python/r] Expose shape accessor for DataFrame [RFC] #3276, inserted here as a simplifying prerequisite as noted above
[python] Re-run, and update, example notebooks with new shape feature #3290

The following was run on a system with core dev (soon to be 2.27) on it, wherein we have support for new shape on dense arrays.

Shapes before:

>>> tiledbsoma.io.show_experiment_shapes('apis/python/notebooks/data/sparse/pbmc3k')

[DataFrame] obs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/obs
  count                2638
  domain               ((0, 2637),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DataFrame] ms/RNA/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/var
  count                1838
  domain               ((0, 1837),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[SparseNDArray] ms/RNA/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/X/data
  used_shape           ((0, 2637), (0, 1837))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_draw_graph_fr
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_draw_graph_fr
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_pca
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_pca
  used_shape           ((0, 2637), (0, 49))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_tsne
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_tsne
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_umap
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_umap
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsp/connectivities
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsp/connectivities
  used_shape           ((0, 2637), (0, 2637))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsp/distances
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsp/distances
  used_shape           ((0, 2637), (1, 2637))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/varm/PCs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/varm/PCs
  used_shape           ((0, 1837), (0, 49))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[DataFrame] ms/raw/varm/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/raw/var
  count                13714
  domain               ((0, 2147483646),)
  maxdomain            ((0, 2147483646),)
  upgraded             False

[SparseNDArray] ms/raw/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/raw/X/data
  used_shape           ((0, 2637), (0, 13713))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False
True



>>> tiledbsoma.io.show_experiment_shapes('apis/python/notebooks/data/dense/pbmc3k')

[DataFrame] obs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/obs
  count                2638
  domain               ((0, 2637),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DataFrame] ms/RNA/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/var
  count                1838
  domain               ((0, 1837),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DenseNDArray] ms/RNA/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/X/data
  shape                (2638, 1838)
  maxshape             (2638, 1838)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_draw_graph_fr
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_draw_graph_fr
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_pca
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_pca
  used_shape           ((0, 2637), (0, 49))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_tsne
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_tsne
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_umap
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_umap
  used_shape           ((0, 2637), (0, 1))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsp/connectivities
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsp/connectivities
  used_shape           ((0, 2637), (0, 2637))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/obsp/distances
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsp/distances
  used_shape           ((0, 2637), (1, 2637))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[SparseNDArray] ms/RNA/varm/PCs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/varm/PCs
  used_shape           ((0, 1837), (0, 49))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False

[DataFrame] ms/raw/varm/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/raw/var
  count                13714
  domain               ((0, 2147483646),)
  maxdomain            ((0, 2147483646),)
  upgraded             False

[SparseNDArray] ms/raw/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/raw/X/data
  used_shape           ((0, 2637), (0, 13713))
  shape                (2147483646, 2147483646)
  maxshape             (2147483646, 2147483646)
  upgraded             False
True

The upgrade:

>>> tiledbsoma.io.upgrade_experiment_shapes('apis/python/notebooks/data/sparse/pbmc3k')
>>> tiledbsoma.io.upgrade_experiment_shapes('apis/python/notebooks/data/dense/pbmc3k')

Shapes after:

>>> tiledbsoma.io.show_experiment_shapes('apis/python/notebooks/data/sparse/pbmc3k')

[DataFrame] obs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/obs
  count                2638
  domain               ((0, 2637),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DataFrame] ms/RNA/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/var
  count                1838
  domain               ((0, 1837),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[SparseNDArray] ms/RNA/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/X/data
  used_shape           ((0, 2637), (0, 1837))
  shape                (2638, 1838)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_draw_graph_fr
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_draw_graph_fr
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_pca
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_pca
  used_shape           ((0, 2637), (0, 49))
  shape                (2638, 50)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_tsne
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_tsne
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_umap
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsm/X_umap
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsp/connectivities
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsp/connectivities
  used_shape           ((0, 2637), (0, 2637))
  shape                (2638, 2638)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsp/distances
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/obsp/distances
  used_shape           ((0, 2637), (1, 2637))
  shape                (2638, 2638)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/varm/PCs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/RNA/varm/PCs
  used_shape           ((0, 1837), (0, 49))
  shape                (1838, 50)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[DataFrame] ms/raw/varm/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/raw/var
  count                13714
  domain               ((0, 13713),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[SparseNDArray] ms/raw/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/sparse/pbmc3k/ms/raw/X/data
  used_shape           ((0, 2637), (0, 13713))
  shape                (2638, 13714)
  maxshape             (2147483646, 2147483646)
  upgraded             True
True




>>> tiledbsoma.io.show_experiment_shapes('apis/python/notebooks/data/dense/pbmc3k')

[DataFrame] obs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/obs
  count                2638
  domain               ((0, 2637),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DataFrame] ms/RNA/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/var
  count                1838
  domain               ((0, 1837),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[DenseNDArray] ms/RNA/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/X/data
  shape                (2638, 1838)
  maxshape             (2638, 1838)
  upgraded             False

[SparseNDArray] ms/RNA/obsm/X_draw_graph_fr
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_draw_graph_fr
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_pca
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_pca
  used_shape           ((0, 2637), (0, 49))
  shape                (2638, 50)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_tsne
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_tsne
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsm/X_umap
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsm/X_umap
  used_shape           ((0, 2637), (0, 1))
  shape                (2638, 2)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsp/connectivities
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsp/connectivities
  used_shape           ((0, 2637), (0, 2637))
  shape                (2638, 2638)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/obsp/distances
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/obsp/distances
  used_shape           ((0, 2637), (1, 2637))
  shape                (2638, 2638)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[SparseNDArray] ms/RNA/varm/PCs
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/RNA/varm/PCs
  used_shape           ((0, 1837), (0, 49))
  shape                (1838, 50)
  maxshape             (2147483646, 2147483646)
  upgraded             True

[DataFrame] ms/raw/varm/var
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/raw/var
  count                13714
  domain               ((0, 13713),)
  maxdomain            ((0, 2147483646),)
  upgraded             True

[SparseNDArray] ms/raw/X/data
  URI file:///Users/kerl/git/single-cell-data/TileDB-SOMA/apis/python/notebooks/data/dense/pbmc3k/ms/raw/X/data
  used_shape           ((0, 2637), (0, 13713))
  shape                (2638, 13714)
  maxshape             (2147483646, 2147483646)
  upgraded             True
True

johnkerl added 3 commits November 3, 2024 01:04

[python] Include tiledbsoma_upgrade_shape for DenseNDArray

86752ec

[python] Support upgrade of even older experiments

bf956fd

Run upgrade-shapes on notebook example experiments

e7ecff7

johnkerl requested a review from nguyenv November 3, 2024 15:23

johnkerl mentioned this pull request Nov 3, 2024

[python/r/c++] Revisit shape for component arrays #2407

Open

johnkerl changed the title ~~Kerl/notebook shape upgrade~~ [python] Run upgrade-shapes on notebook example experiments Nov 3, 2024

johnkerl requested a review from aaronwolen November 3, 2024 15:26

This was referenced Nov 4, 2024

[python] Re-run, and update, example notebooks with new shape feature #3290

Merged

[python] Include tiledbsoma_upgrade_shape for DenseNDArray #3288

Merged

nguyenv approved these changes Nov 4, 2024

View reviewed changes

Base automatically changed from kerl/dense-ugrsh to main November 4, 2024 18:43

johnkerl merged commit a515eb7 into main Nov 4, 2024
1 check passed

johnkerl deleted the kerl/notebook-shape-upgrade branch November 4, 2024 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python] Run upgrade-shapes on notebook example experiments #3289

[python] Run upgrade-shapes on notebook example experiments #3289

johnkerl commented Nov 3, 2024 •

edited

Loading

[python] Run upgrade-shapes on notebook example experiments #3289

[python] Run upgrade-shapes on notebook example experiments #3289

Conversation

johnkerl commented Nov 3, 2024 • edited Loading

johnkerl commented Nov 3, 2024 •

edited

Loading