Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow force-overwriting existing files (non-backing) #344

Merged
merged 12 commits into from
Jan 6, 2024
Merged

Allow force-overwriting existing files (non-backing) #344

merged 12 commits into from
Jan 6, 2024

Conversation

aeisenbarth
Copy link
Contributor

When I want to create and force-write SpatialData to some location, not caring whether there exist older files, I would intuitively use the overwrite=True argument. A use case is when developing/retrying a script that is supposed to be idempotent, repeatable no matter whether it has been run previously.

However, the code assumes from existing files that the SpatialData is backed (although path is None) and refuses to overwrite the supposed backing file.

@aeisenbarth aeisenbarth force-pushed the fix-overwrite-unbacked branch from a4f69e0 to 8a309ac Compare August 29, 2023 13:35
@aeisenbarth aeisenbarth force-pushed the fix-overwrite-unbacked branch from b546185 to b820821 Compare August 29, 2023 15:06
@codecov
Copy link

codecov bot commented Aug 29, 2023 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0eff295) 91.61% compared to head (c99aa4a) 91.68%.

❗ Current head c99aa4a differs from pull request most recent head 05d690d. Consider uploading reports for the commit 05d690d to get more accurate results

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #344      +/-   ##
==========================================
+ Coverage   91.61%   91.68%   +0.07%     
==========================================
  Files          37       37              
  Lines        4997     5006       +9     
==========================================
+ Hits         4578     4590      +12     
+ Misses        419      416       -3
Files Coverage Δ
src/spatialdata/__init__.py 100.00% <100.00%> (ø)
src/spatialdata/_core/spatialdata.py 93.41% <100.00%> (+0.63%) ⬆️
src/spatialdata/_io/__init__.py 100.00% <100.00%> (ø)
src/spatialdata/_io/_utils.py 91.33% <100.00%> (+0.23%) ⬆️

@giovp
Copy link
Member

giovp commented Sep 29, 2023

thanks @aeisenbarth for this PR! I think it's related to #329 , will take a look over weekend!

@LucaMarconato
Copy link
Member

Thank you for reporting and sorry for the late answer @aeisenbarth, the thesis got in between. The PR linked by Giovanni is now under review and we want to address this soon.

melonora
melonora reviewed Oct 11, 2023
View reviewed changes
Copy link
Collaborator

@melonora melonora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not entirely agree with the test, but for the rest, it looks good to me:)

tests/io/test_readwrite.py Outdated Show resolved Hide resolved
@LucaMarconato LucaMarconato added this to the Data format milestone Nov 8, 2023
@LucaMarconato
Copy link
Member

LucaMarconato commented Jan 6, 2024 •
edited
Loading

@aeisenbarth it took me a while to get back to this, it's a delicate part of the code so I wanted to examine it with care and I couldn't find the time before.

The use case that you described is important and the logic that you implemented using the function self.is_backed() is valid, but unfortunately as it is, the new code could lead to data loss.

This is due to the fact that self.is_backed() only checks if the data was written to disk using write() (this is when self.path is set to a non-None value), but it is possible for the user to construct an in-memory object, thus non-backed, by combining together backed objects (for instance by summing two lazy image tensors). In such a case, if the user tries to overwrite a backed file, the user would incur in data loss.

Anyway, this has a simple solution, which is to modify the is_backed() function to check if the in-memory object depends on some storage files by looking at the computational graph instead than looking at self.path. I will make a patch in this PR.

@LucaMarconato
Copy link
Member

Anyway, this has a simple solution, which is to modify the is_backed() function to check if the in-memory object depends on some storage files by looking at the computational graph instead than looking at self.path. I will make a patch in this PR.

Actually, instead of modifying SpatialData.is_backed() I have improved a function that already existed and made it public; I have now named it get_dask_backing_files(). With this new function I have added the support for the case that you fixed in this PR by avoiding the risk of data loss for the case I introduced above.

@LucaMarconato
Copy link
Member

Ready for review, I kindly ask you to have a look.

Also CC-ing who previously reviewed @giovp @melonora

giovp
giovp approved these changes Jan 6, 2024
View reviewed changes
Copy link
Member

@giovp giovp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me, this will probably need to be reworked again when we implement incremental update.

CHANGELOG.md Outdated
@@ -8,7 +8,11 @@ and this project adheres to [Semantic Versioning][].
[keep a changelog]: https://keepachangelog.com/en/1.0.0/
[semantic versioning]: https://semver.org/spec/v2.0.0.html

## [0.0.15] - tbd
## [0.1.0] - tbd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't release 0.1.0 just yet, there are too many changes around storage, data models and transformations that needs to happen, wdyt? @melonora @LucaMarconato @kevinyamauchi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe for multiple tables let's make 0.1.0, but we could also wait for a all the big PRs being merged. I changed for the moment.

src/spatialdata/_core/spatialdata.py Outdated Show resolved Hide resolved
@LucaMarconato
Copy link
Member

Thanks for the review! Fixing the tests and merging!

@LucaMarconato LucaMarconato merged commit bf17f4d into scverse:main Jan 6, 2024
6 of 7 checks passed
@aeisenbarth aeisenbarth deleted the fix-overwrite-unbacked branch January 6, 2024 16:52
melonora added a commit that referenced this pull request Jan 17, 2024
@melonora @pre-commit-ci
@giovp @LucaMarconato @aeisenbarth
Multi table (#410)
75d66f1 
* [pre-commit.ci] pre-commit autoupdate (#394)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/psf/black: 23.10.1 → 23.11.0](psf/black@23.10.1...23.11.0)
- [github.com/pre-commit/mirrors-prettier: v3.0.3 → v3.1.0](pre-commit/mirrors-prettier@v3.0.3...v3.1.0)
- [github.com/pre-commit/mirrors-mypy: v1.6.1 → v1.7.0](pre-commit/mirrors-mypy@v1.6.1...v1.7.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.3 → v0.1.6](astral-sh/ruff-pre-commit@v0.1.3...v0.1.6)

* ficx pre-precommit

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: giovp <giov.pll@gmail.com>

* initial tests multi_table design

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add mock new table

* create test class and cleanup

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add pseudo methods

* Change table type in init

* make tables plural and add to validation in __init__

* revert to old public accessor

* Validate each table in dictionary

* iterate dict values

* add comment

* adjust table getter

* Add tables getter

* Fix missing parenthesis

* change to warnings.warn DeprecationWarning

* allow for backward compatibility in init

* [pre-commit.ci] pre-commit autoupdate (#408)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix dict subscriptable

* fix string representation of sdata

* add deprecation decorator for future

* Allow for tables not annotating elements

* switch to using tables with deprecation

* fix string representation

* write tables element group

* adjust io to multi_table

* Alter io to give None as default value for spatialdata attrs keys

* add tables setter

* raise keyerror table getter

* remove commented tables setter

* raise keyerror in table deleter

* add deprecation warning

* fix tests

* add DeprecationWarning

* comment test

* change setter into method

* circumvent mappingproxy set issue

* adjust set get test

* add get table keys

* add column getters

* add change set target table

* Give default table name

* fix spatialdata without table

* add int32 because of windows and add docstring

* fix filtering by coordinate system

* Change to Path to not be linux / mac specific

* Change to Path to not be linux / mac specific

* table should annotate existing element

* return table with AnnData having 0 rows

* Adjust for windows

* adjust for accessing table elements

* fix change annotation target

* fix set annotation target

* fix/add tests

* fix init from elements

* fix init from elements tests

* add validation check

* add table validation SpatialData.__init

* fix ruff

* only concatenate if annotating

* change into warning because of filtering

* fix last tests

* adjust to tables

* use tables parameter

* fix some mypy

* some mypy fixes

* some more mypy

* fix another mypy

* circumvent typing error on py3.9

* mypy yet again

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix pre_commit

* down to 12 mypy errors

* down to 1mypy error

* fixed mypy errors

* fix set_table_annotation

* added docstring

* refactor data loader (#299)

Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* add documentation

* add documentation

* minor adjustment docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added / adjusted docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix mypy after merge

* refactor function name

This is to avoid confusion. Many not easily resolved errors are created if we let this function generate table values. This makes it clear that
only spatial element values are generated and not tables. This in opposite to gen_elements which does return tables as well.

* [pre-commit.ci] pre-commit autoupdate (#411)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* small fixes

* added gen_elements docstrings

* tiny comments

* fix ruff pre-commit

* removed types from docstring

* refactor of set_table_annotation_target

* add quotes

* fix (?)

* refactor error messages

* fix incremental update (#329)

Co-authored-by: Wouter-Michiel Vierdag <w-mv@hotmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add concatenate argument

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add util functions to init

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add util functions to init

* add tables class

* add table class

* add deprecation back

* rename function in tests

* rename function in tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix precommit

* update precommit and remove add_table, store_table and general fixes

* adjust tables init to incremental update pr

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix deletion of deprecated table

* revert filter change

* add public generators

* adjust to public generators

* Find element uses public generator

* add validation in sdata for tables

* add deprecation version number

* fix mypy errors

* Fix backing when deleting table

* Fix mypy

* cleanup

* chance target_element_name to region

* refactor test

* adjust concatenate regarding not concatenating tables

* add utility function

* concatenate if in multiple sdata objects

* minor docstring refactor

* fix import

* concatenate with tables

* cleanup

* fix test

* [pre-commit.ci] pre-commit autoupdate (#415)

updates:
- [github.com/psf/black: 23.11.0 → 23.12.1](psf/black@23.11.0...23.12.1)
- [github.com/pre-commit/mirrors-prettier: v4.0.0-alpha.4 → v4.0.0-alpha.8](pre-commit/mirrors-prettier@v4.0.0-alpha.4...v4.0.0-alpha.8)
- [github.com/pre-commit/mirrors-mypy: v1.7.1 → v1.8.0](pre-commit/mirrors-mypy@v1.7.1...v1.8.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.7 → v0.1.9](astral-sh/ruff-pre-commit@v0.1.7...v0.1.9)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fixed typings; made pytest raises explicit

* minor fixes human readable strings

* fixed tests

* fix

* Add changes to changelog

* make private

* Remove commented code

* add cache to ignore

* updated changelog with giovp old pr

* refactor into private function

* Fix docstring

* Fix import

* specify key reuse in docstring

* add orphan_table argument

* Change docstring

* remove todo

* add example

* change concatenate logic

* updated changelog

* Allow force-overwriting existing files (non-backing) (#344)

* Add test for writing unbacked data over existing files

* Protect overwriting existing file only if it is backing file

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Simplify assertion, remove try/except

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed pre-commit

* added get_dask_backing_files(); improved sdata.write with overwrite=True

* fix docs

* changed version in changelog

* fix exception string

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>

* Added error message for removed add_elements functions (#420)

* added error message for removed add_elements functions

* moved _error_message_add_element() to _utils

* added validate and set region key

* fix docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added public functions Spatialdata

* fix tests

* add docstrings

* Added subset API; fix behavior with zero-len table (#426)

* added subset API, returning None instead of empty table for APIs with  filter_table=True

* fix 3.9

* [pre-commit.ci] pre-commit autoupdate (#424)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.1.11](astral-sh/ruff-pre-commit@v0.1.9...v0.1.11)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* remove docstring typehint

* Warn user over overwrite in docstring

* Fix query of 2D/3D data with 2D/3D bounding box (#409)

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix 2d/3d bb for raster data

* support for 2d/3d bb for 2d/3d points

* better tests

* applied suggestions from giovp

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#430)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.11 → v0.1.13](astral-sh/ruff-pre-commit@v0.1.11...v0.1.13)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* updated changelog

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor subset

* remove todo

* Made _locate_spatial_element public, renamed to locate_element() (#427)

* made _locate_spatial_element public, renamed to locate_element()

* returning path instead of tuple in locate_element()

* updated changelog

* locate_elements() now returns a list

* fix test

* Update test_and_deploy.yaml (#434)

Triggering the tests for pull requests to any branch.

* change docstring

* fix query test

* add todo

* refactor filter_by_coordinate_system

* test filter with keep table

* adjust docstring

* adjust docstring

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: giovp <giov.pll@gmail.com>
Co-authored-by: Giovanni Palla <25887487+giovp@users.noreply.github.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>
melonora added a commit that referenced this pull request Mar 14, 2024
@LucaMarconato @melonora
@pre-commit-ci @giovp @aeisenbarth
multi table support (#455)
c2e4ef9 
* initial tests multi_table design (#405)

* initial tests multi_table design

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add mock new table

* create test class and cleanup

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add pseudo methods

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Multi table (#410)

* [pre-commit.ci] pre-commit autoupdate (#394)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/psf/black: 23.10.1 → 23.11.0](psf/black@23.10.1...23.11.0)
- [github.com/pre-commit/mirrors-prettier: v3.0.3 → v3.1.0](pre-commit/mirrors-prettier@v3.0.3...v3.1.0)
- [github.com/pre-commit/mirrors-mypy: v1.6.1 → v1.7.0](pre-commit/mirrors-mypy@v1.6.1...v1.7.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.3 → v0.1.6](astral-sh/ruff-pre-commit@v0.1.3...v0.1.6)

* ficx pre-precommit

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: giovp <giov.pll@gmail.com>

* initial tests multi_table design

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add mock new table

* create test class and cleanup

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* additional cleanup

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add pseudo methods

* Change table type in init

* make tables plural and add to validation in __init__

* revert to old public accessor

* Validate each table in dictionary

* iterate dict values

* add comment

* adjust table getter

* Add tables getter

* Fix missing parenthesis

* change to warnings.warn DeprecationWarning

* allow for backward compatibility in init

* [pre-commit.ci] pre-commit autoupdate (#408)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix dict subscriptable

* fix string representation of sdata

* add deprecation decorator for future

* Allow for tables not annotating elements

* switch to using tables with deprecation

* fix string representation

* write tables element group

* adjust io to multi_table

* Alter io to give None as default value for spatialdata attrs keys

* add tables setter

* raise keyerror table getter

* remove commented tables setter

* raise keyerror in table deleter

* add deprecation warning

* fix tests

* add DeprecationWarning

* comment test

* change setter into method

* circumvent mappingproxy set issue

* adjust set get test

* add get table keys

* add column getters

* add change set target table

* Give default table name

* fix spatialdata without table

* add int32 because of windows and add docstring

* fix filtering by coordinate system

* Change to Path to not be linux / mac specific

* Change to Path to not be linux / mac specific

* table should annotate existing element

* return table with AnnData having 0 rows

* Adjust for windows

* adjust for accessing table elements

* fix change annotation target

* fix set annotation target

* fix/add tests

* fix init from elements

* fix init from elements tests

* add validation check

* add table validation SpatialData.__init

* fix ruff

* only concatenate if annotating

* change into warning because of filtering

* fix last tests

* adjust to tables

* use tables parameter

* fix some mypy

* some mypy fixes

* some more mypy

* fix another mypy

* circumvent typing error on py3.9

* mypy yet again

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix pre_commit

* down to 12 mypy errors

* down to 1mypy error

* fixed mypy errors

* fix set_table_annotation

* added docstring

* refactor data loader (#299)

Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* add documentation

* add documentation

* minor adjustment docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added / adjusted docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix mypy after merge

* refactor function name

This is to avoid confusion. Many not easily resolved errors are created if we let this function generate table values. This makes it clear that
only spatial element values are generated and not tables. This in opposite to gen_elements which does return tables as well.

* [pre-commit.ci] pre-commit autoupdate (#411)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* small fixes

* added gen_elements docstrings

* tiny comments

* fix ruff pre-commit

* removed types from docstring

* refactor of set_table_annotation_target

* add quotes

* fix (?)

* refactor error messages

* fix incremental update (#329)

Co-authored-by: Wouter-Michiel Vierdag <w-mv@hotmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add concatenate argument

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add util functions to init

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add util functions to init

* add tables class

* add table class

* add deprecation back

* rename function in tests

* rename function in tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix precommit

* update precommit and remove add_table, store_table and general fixes

* adjust tables init to incremental update pr

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix deletion of deprecated table

* revert filter change

* add public generators

* adjust to public generators

* Find element uses public generator

* add validation in sdata for tables

* add deprecation version number

* fix mypy errors

* Fix backing when deleting table

* Fix mypy

* cleanup

* chance target_element_name to region

* refactor test

* adjust concatenate regarding not concatenating tables

* add utility function

* concatenate if in multiple sdata objects

* minor docstring refactor

* fix import

* concatenate with tables

* cleanup

* fix test

* [pre-commit.ci] pre-commit autoupdate (#415)

updates:
- [github.com/psf/black: 23.11.0 → 23.12.1](psf/black@23.11.0...23.12.1)
- [github.com/pre-commit/mirrors-prettier: v4.0.0-alpha.4 → v4.0.0-alpha.8](pre-commit/mirrors-prettier@v4.0.0-alpha.4...v4.0.0-alpha.8)
- [github.com/pre-commit/mirrors-mypy: v1.7.1 → v1.8.0](pre-commit/mirrors-mypy@v1.7.1...v1.8.0)
- [github.com/astral-sh/ruff-pre-commit: v0.1.7 → v0.1.9](astral-sh/ruff-pre-commit@v0.1.7...v0.1.9)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fixed typings; made pytest raises explicit

* minor fixes human readable strings

* fixed tests

* fix

* Add changes to changelog

* make private

* Remove commented code

* add cache to ignore

* updated changelog with giovp old pr

* refactor into private function

* Fix docstring

* Fix import

* specify key reuse in docstring

* add orphan_table argument

* Change docstring

* remove todo

* add example

* change concatenate logic

* updated changelog

* Allow force-overwriting existing files (non-backing) (#344)

* Add test for writing unbacked data over existing files

* Protect overwriting existing file only if it is backing file

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Simplify assertion, remove try/except

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed pre-commit

* added get_dask_backing_files(); improved sdata.write with overwrite=True

* fix docs

* changed version in changelog

* fix exception string

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>

* Added error message for removed add_elements functions (#420)

* added error message for removed add_elements functions

* moved _error_message_add_element() to _utils

* added validate and set region key

* fix docstring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added public functions Spatialdata

* fix tests

* add docstrings

* Added subset API; fix behavior with zero-len table (#426)

* added subset API, returning None instead of empty table for APIs with  filter_table=True

* fix 3.9

* [pre-commit.ci] pre-commit autoupdate (#424)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.1.11](astral-sh/ruff-pre-commit@v0.1.9...v0.1.11)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* remove docstring typehint

* Warn user over overwrite in docstring

* Fix query of 2D/3D data with 2D/3D bounding box (#409)

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix 2d/3d bb for raster data

* support for 2d/3d bb for 2d/3d points

* better tests

* applied suggestions from giovp

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* added comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#430)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.11 → v0.1.13](astral-sh/ruff-pre-commit@v0.1.11...v0.1.13)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* updated changelog

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor subset

* remove todo

* Made _locate_spatial_element public, renamed to locate_element() (#427)

* made _locate_spatial_element public, renamed to locate_element()

* returning path instead of tuple in locate_element()

* updated changelog

* locate_elements() now returns a list

* fix test

* Update test_and_deploy.yaml (#434)

Triggering the tests for pull requests to any branch.

* change docstring

* fix query test

* add todo

* refactor filter_by_coordinate_system

* test filter with keep table

* adjust docstring

* adjust docstring

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: giovp <giov.pll@gmail.com>
Co-authored-by: Giovanni Palla <25887487+giovp@users.noreply.github.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>

* Enforce instance key to be dtype int (#444)

* change genes in blobs points

* force instance_id of int dtype

* Change error

* Check unique instance_key values per region

* Join elements table (#445)

* change genes in blobs points

* force instance_id of int dtype

* Change error

* Check unique instance_key values per region

* silence warning

* add left inner join

* change to left join

* add left_exclusive join

* add to Enum

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add inner join

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add right join

* add ugly right exclusive join

* return none instead of empty df

* add left join tests

* test left_exclusive_join

* test inner join

* refactor get_table_keys

* assert valid element in elements_dict

* test warnings

* add fail join tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* silence warning

* add left inner join

* change to left join

* add left_exclusive join

* add to Enum

* add inner join

* fix merge conflict

* Add right join

* add ugly right exclusive join

* return none instead of empty df

* add left join tests

* test left_exclusive_join

* test inner join

* refactor get_table_keys

* assert valid element in elements_dict

* test warnings

* add fail join tests

* fix comments

* lowercase + docstring

* complete docstring

* lowercase enum

* explicit instance_id column in test

* test instance_id and region column

* change error

* add get method

* add contain

* get rid of create_element_dict

* add todo

* adjust enums

* add ability for matching rows

* some cleanup

* make any element none in right exclusive join

* add check of table order conservation

* fix tests

* add match_element_to_table

* change docstring

* revert to old match_table_to_element

* readd match_element_to_table

* added tests for label joins

* add points tests

* fixed type and docstring

* fixed comments and docstrings

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change typehint

* update changelog with joins

* include join functions

* docs for joins

* added test for right match rows

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation of match rows

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix docstring

* fix docs

* fix typo in changelog

* attempt fix docs

* fix docs

* fix docs

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>

* fix tests

* fix docs

* add possibility for custom table name (#459)

* add possibility for custom table name

* change docstring

* updated changelog

* added table_name to SpatialData.aggregate()

---------

Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* Update locate values (#460)

* add possibility for custom table name

* change docstring

* updated changelog

* added table_name parameter

* update changelog

---------

Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com>

* Filter table annotate (#462)

* add get_element_annotator

* add docstring

* add test

* wip get_centroids

* implemented get_centroids()

* made _assert_spatialdata_objects_seem_identical() into a util

* fix docs, attemp

* allow table to be None in get_values and _locate_values (#466)

* allow table to be None

* fixes to aggregate

* check annotation

* adjust docstrings

* remove table check

* add sphinx_pytest

* Added `validate_table_annotation_target()` (#468)

* added validate_table_annotation_target()

* removed test that is no longer relevant

* fix docs

* fixed way to test no warnings emitted

* merged validate_table_annotation_target() into validate_table_in_spatialdata()

* fix test

* silence warning

* fix gettin dtype from multiscale

* add else dtype back

* silence scipy.misc.face deprecation

* Operation `to_circles()` (#473)

* `get_centroids()` (#465)

* wip get_centroids

* implemented get_centroids()

* code suggestions from kevin

* wip vectorize.py

* added vectorize; still no tests

* refactored testing functions; wip tests to_circles()

* considering removing the target_coordinate_system parameter in to_circles()

* adjusted tests, docs, changelog

* added changelog

* fix tests, remove inject_docs

* fix sphinx

* attempt fix sphinx

* deedcopy() utils function (#480)

* deedcopy() utils function

* fixed missings seeds for default_rng()

* wip fix

* wip fix

* fix bug due to data being computed in-place and then failing validation

* add pooch requirement

* Update src/spatialdata/_core/_deepcopy.py

* Update src/spatialdata/_core/_deepcopy.py

* Update src/spatialdata/_core/_deepcopy.py

* Update src/spatialdata/_core/_deepcopy.py

---------

Co-authored-by: Wouter-Michiel Vierdag <w-mv@hotmail.com>

* fix bug deepcopy() of points wrong columns order

* workaround wrong order points columns after deepcopy

* rechunking raster data after spatial query (#479)

* rechunking raster data after spatial query

* using xarray chunk() instead of dask rechunk()

* Test joins with string indices and instance id (#485)

* test join strings

* fix dtype aggregate

---------

Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>

* cleanup tests

* remove comments

---------

Co-authored-by: Wouter-Michiel Vierdag <w-mv@hotmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: giovp <giov.pll@gmail.com>
Co-authored-by: Giovanni Palla <25887487+giovp@users.noreply.github.com>
Co-authored-by: aeisenbarth <54448967+aeisenbarth@users.noreply.github.com>
Co-authored-by: Wouter-Michiel Vierdag <michiel.vierdag@embl.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@melonora melonora melonora left review comments

@LucaMarconato LucaMarconato LucaMarconato left review comments

@giovp giovp giovp approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Data format
Development

Successfully merging this pull request may close these issues.
4 participants
@aeisenbarth @giovp @LucaMarconato @melonora