TileDB-SOMA 1.15.0rc4

This is a release candidate for 1.15.0 which we anticipate releasing in earlly December 2024. It contains several important developments.

• Spatial transcriptomics: alpha version of the feature.
• Performance improvements: particularly for tiledbsoma.io.to_anndata / tiledbsoma.io.to_h5ad
• Seurat I/O: incremental work for Seurat v5
• New shape feature: Python tutorial here; Academy docs to be shipped by the time of 1.15.0
• Documentation infrastructure: please see the pre-release docs (note latest in the URL)
• For developers only: We now build our C++ library using C++ 20. This doesn't affect non-developer installs, which will get artifacts from PyPI/r-universe/Conda.
• The TileDB-Py package is no longer a required dependency.

Upcoming deprecation notices

• The .used_shape accessor for SparseNDArray and DenseNDArray is scheduled for removal in 1.16: as of 1.15 the .shape accessor replaces it.

Spatial transcriptomics

• [python] Add export for MultiscaleImage to SpatialData by @jp-dark in #3355
• [python] Add PointCloudDataFrame to SpatialData points model by @jp-dark in #3357
• [python] Add MultiscaleImage level SpatialData exporter by @jp-dark in #3342
• [python] Spatial changes required for somacore 1.0.24 by @jp-dark in #3333
• [python] Fix and test SpatialData transformation exporter by @jp-dark in #3330
• [python] Rewrite Xarray support to use dask arrays by @jp-dark in #3325
• [python] Return identity for MultiscaleImage transform to/from level 0 by @jp-dark in #3323
• [python] Add MultiscaleImage methods to access level URI by @jp-dark in #3322
• [python] Add export for PointCloudDataFrame to SpatialData shapes by @jp-dark in #3298
• [python] Create xarray backend for DataArray types by @jp-dark in #3243
• [python] Update spatial API to match somacore 1.0.23 by @jp-dark in #3274
• [c++] Cast vertex list to WKB by @XanthosXanthopoulos in #3222
• [c++] Geometry Dataframe by @XanthosXanthopoulos in #3212

Performance improvements

• [python] Optimization of ExperimentAxisQuery to_anndata by @bkmartinjr in #3359
• [python] Fix incorrect type handling in ExperimentAxisQuery sparse reader by @bkmartinjr in #3375
• [python/C++] Release GIL in IntIndexer C++ code by @bkmartinjr in #3344
• [python] Partition sparse matrix reads in tiledbsoma.io.to_anndata by @bkmartinjr in #3328
• [python/c++] COO to CSX conversion optimization by @bkmartinjr in #3304
• [python] Performance improvement for experiment-level resize by @johnkerl in #3308
• [python] Add ManagedQuery bindings by @nguyenv in #3281
• [python] Ingest somacore classes by @ryan-williams in #3307
• [c++] Fix heap management in fastercsx by @johnkerl in #3365
• [c++] Add ManagedQuery constructor that takes SOMAArray by @nguyenv in #3279
• [c++] Add ManagedQuery::set_layout by @nguyenv in #3278
• [c++] Use schema from tiledb::Array instead of ManagedQuery by @nguyenv in #3277
• [c++] Move casting from SOMAArray to ManagedQuery by @nguyenv in #3275

Seurat I/O

• [r] Add initial support for ragged array writing for Seurat v5 by @mojaveazure in #2523

New shape

• [c++] Some use_current_domain unit-test/feature-flag teardown, part 4 of 4 by @johnkerl in #3372
• [c++] Some use_current_domain unit-test/feature-flag teardown, part 3 of 4 by @johnkerl in #3371
• [c++] Some use_current_domain unit-test/feature-flag teardown, part 2 of 4 by @johnkerl in #3370
• [python/r] Remove 2.27-related feature flag by @johnkerl in #3368
• [r/python] Revert #3300 by @johnkerl in #3358
• [python/r] Enforce dataframe domain lower bound == 0 by @johnkerl in #3300
• [c++] Use valid ASCII for wide-as-possible string current domain by @johnkerl in #3367
• [python] Use new shape in one more spot by @johnkerl in #3321
• [python] Fix bad merge of soma-shapes notebook by @johnkerl in #3309
• [python] Tutorial notebook for the new-shape feature by @johnkerl in #3294
• [r] Data refresh for new shapes by @johnkerl in #3303
• [python/r] Move beyond the new-shape feature flag by @johnkerl in #3301
• [python] Fix dense example data for core 2.27 by @johnkerl in #3296
• [python/docs] Replace experiment directories with .tgz files in source control by @johnkerl in #3295
• [python] Doc updates for new-shape feature by @johnkerl in #3285
• [python] Run upgrade-shapes on notebook example experiments by @johnkerl in #3289
• [python] Include tiledbsoma_upgrade_shape for DenseNDArray by @johnkerl in #3288
• [python] More auto-detect of function_name_for_messages by @johnkerl in #3286
• [r] More fixes for unit-test cases with dense + core 2.27 by @johnkerl in #3280
• [python] Fix last 2.27+Python+dense failing test case by @johnkerl in #3269
• [r] Fixes for dense+2.27 by @johnkerl in #3270
• [python] Fix 3D/4D cases with core 2.27 by @johnkerl in #3268
• [c++] Apply subarrays for dense reads and writes by @johnkerl in #3263
• [python] Fix some dense+2.27 failing test cases by @johnkerl in #3265
• [python] Centralize sparse/dense pybind11 shape methods by @johnkerl in #3261
• [python] Re-run, and update, example notebooks with new shape feature by @johnkerl in #3290

Documentation infastructure

• [docs] Fix Sphinx issue with linkless summaries by @johnkerl in #3284
• [docs] Add new things to .rst files by @johnkerl in #3283
• [python] Misc. formatting fix-ups by @johnkerl in #3287
• [python] _multiscale_image: add missing docstrings, typo fixes by @ryan-williams in #3312

C++20

• [ci] Change manylinux version for sdist&wheels CI (C++ 20) by @XanthosXanthopoulos in #3351
• [c++] Another C++ 20 doc-spot by @johnkerl in #3352
• [python] Update README.md to remove fmt reference by @nguyenv in #3347
• [c++] Use std::format in spdlog (C++ 20) by @XanthosXanthopoulos in #3343
• [c++/python] Switch to std::span by @XanthosXanthopoulos in #3340
• [c++/python/r] Migrate to C++20 by @XanthosXanthopoulos in #3331
• [python/docs] Use ubuntu-24.04 (C++ 20) for readthedocs by @ryan-williams in #3339

Packaging and CI

• [c++/python/r] Use core 2.27.0-rc3 by @johnkerl in #3366
• [packaging] More failure info in update-tiledb-versions.py by @johnkerl in #3363
• [python] Run pre-commit autoupdate by @johnkerl in #3326
• [r/python] Note on dev installs and spdlog failures by @johnkerl in #3324
• [r/ci] Remove tiledb-r specific install by @mojaveazure in #3319
• [r/ci] Try to unbreak r-valgrind CI by @johnkerl in #3318
• [r/ci] Avoid log truncation with "Last 13 lines of output" by @johnkerl in #3313
• [r] Run tests in closer to internal-dependency order by @johnkerl in #3311
• [python] Unbreak CI with anndata==0.11 by @johnkerl in #3305
• [python] Fix some warnings at anndata==0.11 by @johnkerl in #3306

Remove TileDB-Py as a required dependency

• [python] Update README.md to remove tiledb-py by @nguyenv in #3346

Miscellaneous internals

• [r] Use standard name zzz.R for init code by @johnkerl in #3332
• [r] Address two more compiler warnings by @johnkerl in #3320
• [c++] Fix a compiler warning by @johnkerl in #3310
• [python] Consolidate all _set_coords methods into single _util function by @nguyenv in #3292
• [python] Rename set_reader_coords to set_coords by @johnkerl in #3253

TileDB-SOMA 1.15.0rc3

Summary

• NumPy 2 support
• Python 3.12 support
• Significant work on spatial transcriptomics
• Progress on new shape (#2407) to be released with 1.15.0
• The TileDB-Py package is no longer a dependency

Changes

• [Backport release-1.15] [python] Support NumPy 2 by @github-actions in #3248
• [Backport release-1.15] [python] Use typeguard==4.4.0, somacore==1.0.22 by @github-actions in #3254
• [Backport release-1.15] [python] Rename set_reader_coords to set_coords by @github-actions in #3255
• File add/removes for bulk-sync from main to release-1.15 by @johnkerl in #3258
• Bulk-sync main to release-1.15 by @johnkerl in #3251
• Bulk sync main -> release-1.15 by @johnkerl in #3256
• [r] Prepare for 1.15.0rc3 by @johnkerl in #3257

Full Changelog: 1.15.0rc2...1.15.0rc3

See also: 1.15.0rc2...main

TileDB-SOMA 1.14.5

This is a single-bug-fix release

• [Backport release-1.14] [c++/python] Check arrow_is_string_type in _update_dataframe by @github-actions in #3225
• [r] Prepare for 1.14.5 by @johnkerl in #3226

Full Changelog: 1.14.4...1.14.5

1.14.4

What's Changed

This release does not affect any Python code.

• [r] Backport sparse COO writer to 1.14 by @mojaveazure in #3217
• [r] Prep for 1.14.4 release by @johnkerl in #3219

Full Changelog: 1.14.3...1.14.4

TileDB-SOMA 1.14.3

What's Changed

• [r] Backport #3145 to release-1.14; 1.14.3 by @johnkerl in #3147

Full Changelog: 1.14.2...1.14.3

TileDB-SOMA 1.15.0rc2

The main content of this release candidate is a technical preview of the new spatial-transcriptomics featureset.

New features added

• [python] Update TileDB-SOMA to use the somacore release with spatial datatypes by @jp-dark in #3078
• [python] Add Python implementation of the PointCloudDataFrame class by @jp-dark in #3109
• [python/c++] Update TileDB-SOMA to somacore 1.0.19 by @jp-dark in #3098
• [python/r/c++] Update allowed soma types and the metadata encoding version by @jp-dark in #3086
• [python] Add implementation of the MultiscaleImage to main by @jp-dark in #3112
• [python] Add the Python implementation of the Scene collection by @jp-dark in #3116
• [python] Fix MultiscaleImage.read_spatial_region full region read by @jp-dark in #3128
• [python] Create Group base class for Collection by @jp-dark in #3024
• [python] Drop extra index array schema name by @jp-dark in #3033
• [python] Update coordinate space <-> JSON converters by @nguyenv in #3114
• Add Visium ingestor and sample spatial notebook by @jp-dark in #3123
• [c++] Add SOMAMultiscaleImage class by @jp-dark in #3094
• [c++] Add SOMAScene class by @jp-dark in #3093
• [c++] Add SOMAPointCloudDataFrame class by @jp-dark in #3092

Breaking changes

• [python] Drop support for Python 3.8 by @ryan-williams in #3021
• [python] Remove more Python 3.8 references by @ryan-williams in #3073

Bug fixes

• [Backport release-1.15] [python] Drop anndata version pin; use CSCDataset/CSRDataset by @github-actions in #3144
• [python] Avoid RCs of dependent packages for our RC by @johnkerl in #3142
• [Backport release-1.15] Do not fatal on 1.15-related assertions by @github-actions in #3138
• [python] Fix retrieval of named levels from a MultiscaleImage by @aaronwolen in #3119
• [python] Fix f-strings in multiscaleimage error messages by @aaronwolen in #3118
• [c++] Fix bug in nnz of variant-indexed dataframes by @johnkerl in #2990

Minor changes

• [python] Add --maxfail=50 to GHA pytest invocations by @ryan-williams in #3034
• [python] Replace deprecated wheel.bdist_wheel with setuptools.command.bdist_wheel by @ryan-williams in #3076
• [python] setuptools>=70.1 in sdist GHA by @ryan-williams in #3106
• [c++/python] Check arrow_is_string_type in _update_dataframe by @nguyenv in #3111
• [python] Fix deprecation warnings [main] by @johnkerl in #3014
• [python] One more rename in prep for domainish pushdown by @johnkerl in #3026
• [python] Use somacore 1.0.16 by @johnkerl in #3035
• [python] Use somacore 1.0.17 by @johnkerl in #3070
• [python/r] Use pushdown domainish accessors at the Python/R UX level by @johnkerl in #3027
• [python] Use same default max domain between Python and R by @johnkerl in #3088
• [python/r] DataFrame resizer by @johnkerl in #3091
• [python] Use somacore 1.0.20 by @johnkerl in #3120
• [c++/python] Push update_dataframe_schema down to libtiledbsoma by @johnkerl in #3113
• [c++/python/r] Use libtiledbsoma for R schema evolution by @johnkerl in #3100
• [c++/python/r] Rename a helper function by @johnkerl in #3127
• [r] Address some pkgdown GHA warnings by @ryan-williams in #3077
• [r] Post-1.14 version for main branch by @johnkerl in #3009
• [r] Run clang-format on apis/r/src (except autogen code) by @johnkerl in #3048
• [r] Remove the unused fragment_count accessor by @johnkerl in #3054
• [r] Push ndim down to C++ by @johnkerl in #3066
• [r] Fix up doc blocks and re-run roxygenise by @johnkerl in #3071
• [r] Improve test-case field names for DataFrame by @johnkerl in #3067
• [r] Push dimnames down to C++ by @johnkerl in #3065
• [r] Run devtools::document() by @johnkerl in #3084
• [r] Implement missing domain argument to DataFrame create by @johnkerl in #3032
• [r] Make DataFrame objects shapeable at ingest by @johnkerl in #3089
• [r] Push attrnames down to C++ by @johnkerl in #3121
• [r] DESCRIPTION and NEWS.md for #3121 by @johnkerl in #3122
• [r] Push schema accessor down to libtiledbsoma by @johnkerl in #3079
• [c++] Pre-neatens for polymorphic domainish accessors by @johnkerl in #3011
• [c++] Update dataframe unit-test writes in prep for polytype domainish accessors by @johnkerl in #3017
• [c++] Readback-testing pieces for polytype domainish accessors by @johnkerl in #3018
• [c++] Have a DataFrame test case with soma_joinid not first by @johnkerl in #3019
• [c++] Be a bit more careful testing dim-max vs shape by @johnkerl in #3020
• [c++] Implementation and unit testing for domainish accessors by @johnkerl in #3012
• [c++] Fix bad merge from #3012 and #3020 by @johnkerl in #3025
• [c++] Fix a valgrind issue in unit-test code by @johnkerl in #3029
• [c++] Fix memory-management issues in new domainish helpers by @johnkerl in #3030
• [c++] Improve current-domain signaling for string dims by @johnkerl in #3028
• [c++] Extend some unit-test cases for new shape by @johnkerl in #3068
• [c++] Expose custom DataFrame domain for libtiledbsoma unit-test cases by @johnkerl in #3069
• [c++] Add can-resize helpers in prep for experiment-level resize by @johnkerl in #3095
• Use core 2.26.2 by @johnkerl in #3096
• Minor update to .gitignore by @johnkerl in #3103
• [c++] Fail earlier if n_buffers is not as expected by @johnkerl in #3107
• [c++] Fix a minor typo in logging statements by @johnkerl in #3099
• [c++] Trivial name-shortens in unit-test code by @johnkerl in #3125
• [c++] Propagate Python/R function names to C++ for upgrade/resize methods by @johnkerl in #3130

Full Changelog: 1.14.2...1.15.0rc2

TileDB-SOMA 1.15.0rc1

The main content of this release candidate is a technical preview of the new spatial-transcriptomics featureset.

New features added

• [python] Update TileDB-SOMA to use the somacore release with spatial datatypes by @jp-dark in #3078
• [python] Add Python implementation of the PointCloudDataFrame class by @jp-dark in #3109
• [python/c++] Update TileDB-SOMA to somacore 1.0.19 by @jp-dark in #3098
• [python/r/c++] Update allowed soma types and the metadata encoding version by @jp-dark in #3086
• [python] Add implementation of the MultiscaleImage to main by @jp-dark in #3112
• [python] Add the Python implementation of the Scene collection by @jp-dark in #3116
• [python] Fix MultiscaleImage.read_spatial_region full region read by @jp-dark in #3128
• [python] Create Group base class for Collection by @jp-dark in #3024
• [python] Drop extra index array schema name by @jp-dark in #3033
• [python] Update coordinate space <-> JSON converters by @nguyenv in #3114
• Add Visium ingestor and sample spatial notebook by @jp-dark in #3123
• [c++] Add SOMAMultiscaleImage class by @jp-dark in #3094
• [c++] Add SOMAScene class by @jp-dark in #3093
• [c++] Add SOMAPointCloudDataFrame class by @jp-dark in #3092

Breaking changes

• [python] Drop support for Python 3.8 by @ryan-williams in #3021
• [python] Remove more Python 3.8 references by @ryan-williams in #3073

Bug fixes

• [Backport release-1.15] Do not fatal on 1.15-related assertions by @github-actions in #3138
• [python] Fix retrieval of named levels from a MultiscaleImage by @aaronwolen in #3119
• [python] Fix f-strings in multiscaleimage error messages by @aaronwolen in #3118
• [c++] Fix bug in nnz of variant-indexed dataframes by @johnkerl in #2990

Minor changes

• [python] Add --maxfail=50 to GHA pytest invocations by @ryan-williams in #3034
• [python] Replace deprecated wheel.bdist_wheel with setuptools.command.bdist_wheel by @ryan-williams in #3076
• [python] setuptools>=70.1 in sdist GHA by @ryan-williams in #3106
• [c++/python] Check arrow_is_string_type in _update_dataframe by @nguyenv in #3111
• [python] Fix deprecation warnings [main] by @johnkerl in #3014
• [python] One more rename in prep for domainish pushdown by @johnkerl in #3026
• [python] Use somacore 1.0.16 by @johnkerl in #3035
• [python] Use somacore 1.0.17 by @johnkerl in #3070
• [python/r] Use pushdown domainish accessors at the Python/R UX level by @johnkerl in #3027
• [python] Use same default max domain between Python and R by @johnkerl in #3088
• [python/r] DataFrame resizer by @johnkerl in #3091
• [python] Use somacore 1.0.20 by @johnkerl in #3120
• [c++/python] Push update_dataframe_schema down to libtiledbsoma by @johnkerl in #3113
• [c++/python/r] Use libtiledbsoma for R schema evolution by @johnkerl in #3100
• [c++/python/r] Rename a helper function by @johnkerl in #3127
• [r] Address some pkgdown GHA warnings by @ryan-williams in #3077
• [r] Post-1.14 version for main branch by @johnkerl in #3009
• [r] Run clang-format on apis/r/src (except autogen code) by @johnkerl in #3048
• [r] Remove the unused fragment_count accessor by @johnkerl in #3054
• [r] Push ndim down to C++ by @johnkerl in #3066
• [r] Fix up doc blocks and re-run roxygenise by @johnkerl in #3071
• [r] Improve test-case field names for DataFrame by @johnkerl in #3067
• [r] Push dimnames down to C++ by @johnkerl in #3065
• [r] Run devtools::document() by @johnkerl in #3084
• [r] Implement missing domain argument to DataFrame create by @johnkerl in #3032
• [r] Make DataFrame objects shapeable at ingest by @johnkerl in #3089
• [r] Push attrnames down to C++ by @johnkerl in #3121
• [r] DESCRIPTION and NEWS.md for #3121 by @johnkerl in #3122
• [r] Push schema accessor down to libtiledbsoma by @johnkerl in #3079
• [c++] Pre-neatens for polymorphic domainish accessors by @johnkerl in #3011
• [c++] Update dataframe unit-test writes in prep for polytype domainish accessors by @johnkerl in #3017
• [c++] Readback-testing pieces for polytype domainish accessors by @johnkerl in #3018
• [c++] Have a DataFrame test case with soma_joinid not first by @johnkerl in #3019
• [c++] Be a bit more careful testing dim-max vs shape by @johnkerl in #3020
• [c++] Implementation and unit testing for domainish accessors by @johnkerl in #3012
• [c++] Fix bad merge from #3012 and #3020 by @johnkerl in #3025
• [c++] Fix a valgrind issue in unit-test code by @johnkerl in #3029
• [c++] Fix memory-management issues in new domainish helpers by @johnkerl in #3030
• [c++] Improve current-domain signaling for string dims by @johnkerl in #3028
• [c++] Extend some unit-test cases for new shape by @johnkerl in #3068
• [c++] Expose custom DataFrame domain for libtiledbsoma unit-test cases by @johnkerl in #3069
• [c++] Add can-resize helpers in prep for experiment-level resize by @johnkerl in #3095
• Use core 2.26.2 by @johnkerl in #3096
• Minor update to .gitignore by @johnkerl in #3103
• [c++] Fail earlier if n_buffers is not as expected by @johnkerl in #3107
• [c++] Fix a minor typo in logging statements by @johnkerl in #3099
• [c++] Trivial name-shortens in unit-test code by @johnkerl in #3125
• [c++] Propagate Python/R function names to C++ for upgrade/resize methods by @johnkerl in #3130

Full Changelog: 1.14.2...1.15.0rc0

TileDB-SOMA 1.14.2

What's Changed

• Bulk-backport Python 3.8 and somacore 1.0.17 mods to release-1.14 branch by @johnkerl in #3074

Full Changelog: 1.14.1...1.14.2

TileDB-SOMA 1.14.2rc0

What's Changed

• Bulk-backport Python 3.8 and somacore 1.0.17 mods to release-1.14 branch by @johnkerl in #3074

Full Changelog: 1.14.1...1.14.2

TileDB-SOMA 1.14.1

What's Changed

• [python] Fix deprecation warnings [release-1.14] by @johnkerl in #3013
• [r] TileDB-SOMA 1.14.1 by @johnkerl in #3016

Full Changelog: 1.14.0...1.14.1