[BPT-155] Spatial Data Refactor #104

dylanclam12 · 2023-08-07T21:34:42Z

Refactoring Bento with SpatialData

add download badge

notion-workspace · 2023-08-07T21:45:40Z

Removed all files that have not been refactored with SpatialData

Added a modified SpatialData query script that works around their intrinsic_axes issue.

Added a formatting function that reads a Spatial Data object to: sjoin points to shapes, sjoin shapes to shapes, and changes shape object indices to strings. Sjoin functions changed in geometry.

Refactor of _shape_features to read and save to SpatialData.

Refactor point features to read and save to SpatialData.

Refactoring lp function and implementing lp_diff for discrete phenotypes

Implemented lp_diff_continuous, caught phenotype error in lp_diff_discrete, and clipped infinite values to max for -log10p and -log10padj.

Added _neighborhoods.py with no changes

Added sync_points function to sync points dask dataframe to groud truth sdata.table anndata + minor fixes

Refactored flux and fluxmap functions to SpatialData. Minor changes to format data and sindex_points.

Removed dask functionality since there is no speed up in the point computations. Changed geometry functions and edited how they were used in io, shape features, point features, and flux.

Refactored _flux_enrichment.py and removed register_points functionality since we are placing cell_raster in a points element that is dynamic instead of leaving it in unstructured.

Refactored colocation.py which relies on _decomposition.py being refactored. Current _composition.py comp_diff function does not work correctly. Refactored to the point where we run across the same error.

Refactoring plotting script and all of its depending scripts: colors, layers, utils

Refactoring plotting scripts for localization patterns

ckmah

change all file permissions back to 755
cleanup commented out imports and decorators (track)
add summary list of changes
unit tests

bento/datasets/_datasets.py

bento/plotting/_signatures.py

bento/query/__init__.py

bento/query/_query.py

bento/_utils.py

bento/geometry/_geometry.py

bento/plotting/_lp.py

bento/plotting/_plotting.py

bento/tools/_flux_enrichment.py

dylanclam12 · 2024-01-16T21:13:27Z

change all file permissions back to 755

cleanup commented out imports and decorators (track)

add summary list of changes

unit tests

Moved to PR Summary

#104 (comment) - defer to native SpatialData API for query - change bento.geo.get_points astype parameter options to lowercase - change bento.geo.sindex_points unindexed points from "None" to empty string - in bento.tl.lp pull out gene expression array into variable so it is not queried twice - cleanup unused @ track, batch, copy - in bento.tl.fe make getting sparse matrix in csr format more readable

pyproject.toml

used instance_key as an indicator of cell_shapes across all point features.

used {shape_key}_index as an indicator of whether a point is inside or outside a specified shape. Changed this because points element and the pulled in shape elements have columns with the same name. Changing the columns in the points element to have the _index suffix to differentiate.

Since we aren't holding all polygons that have been sjoined to cells in the cell shapes element, I added a join to pull those polygons into points_df. Points and shape elements will have similarly named columns so I am adding the _index suffix to the columns in the points element.

wrote test cases for point_features.py

moved reading and formatting sdata to setup. Condensed feature lists

moved test cases to a loop instead of listing them one by one

Changed PATTERN_FEATURES to match the new naming conventions (cell_boundaries_x and nucleus_boundaries_x)

Incorporated the use of instance_key into lp to be consistent with the rest of the package

ckmah

lgtm

edit: nvm didnt finish reviewing

Updated set_metadata documentation

changed flux to use instance_key instead of hard coding cell_boundaries

made the default value for gene feature_name to be consistent with the Xenium format like the rest of the package

test script for flux and fluxmap

changed overwriting cell_boundaires_raster to adding metadata with set_points_metadata

Changed all hard coded cell_raster to use the instance key

Test cases for flux enrichment functions

removed import random from test_flux.py and test_flux_enrichment.py

added instance_key and feature_key to colocation and _colocation_tensor

added instance_key, feature_key, points key to coloc_quotient, _cell_clq, and _clq_statistic

Tests for _colocation.py

Tests for _geometry.py

ckmah

looks good, need to find new home for datasets that supports zarr streaming

ckmah · 2024-03-24T21:59:43Z

bento/tools/_lp.py

@@ -87,13 +92,15 @@ def lp(sdata: SpatialData, groupby: str = "gene"):
    sdata.table.uns["lp"] = indicator_df.reset_index()
    sdata.table.uns["lpp"] = pattern_prob.reset_index()

-def lp_stats(sdata: SpatialData):
+def lp_stats(sdata: SpatialData, instance_key: str = "cell_boundaries"):


should default value be set for instance_key?

ckmah · 2024-03-26T17:17:17Z

bento/tools/_colocation.py

@@ -106,28 +98,23 @@ def _colocation_tensor(data: AnnData, copy: bool = False):
    tensor = s.todense()
    print(tensor.shape)


remove print

ckmah · 2024-03-26T17:19:15Z

bento/tools/_colocation.py

-    data: AnnData,
-    shapes: List[str] = ["cell_shape"],
+    sdata: SpatialData,
+    shapes: List[str] = ["cell_boundaries"],


remove as default, infer from points instance_key (I believe this is the static key we use for cells?)

nvm we can do this when we update tutorials

ckmah · 2024-04-05T20:50:41Z

bento/tools/_flux_enrichment.py

@@ -83,13 +84,13 @@ def fe(
    Returns
    -------
    sdata : SpatialData
-        .points["cell_raster"]["flux_fe"] : DataFrame
+        .points["cell_boundaries_raster"]["flux_fe"] : DataFrame


should be {instance_key}_raster

LucaMarconato · 2024-04-06T00:09:38Z

🥳

ckmah added 2 commits July 12, 2023 10:56

Update README.md

f0f5329

Merge pull request #103 from ckmah/download-badge

7d3ec21

add download badge

dylanclam12 self-assigned this Aug 7, 2023

dylanclam12 requested a review from ckmah August 7, 2023 21:34

dylanclam12 changed the title ~~Spatialdata Refactor~~ [BPT-155] Spatialdata Refactor Aug 7, 2023

dylanclam12 changed the title ~~[BPT-155] Spatialdata Refactor~~ [BPT-155] Port Spatial Features Aug 7, 2023

dylanclam12 changed the title ~~[BPT-155] Port Spatial Features~~ [BPT-155] Spatial Data Refactor Aug 7, 2023

Delete Non-refactored Files

1d582ac

Removed all files that have not been refactored with SpatialData

dylanclam12 force-pushed the spatialdata_refactor branch from 39153f8 to 1d582ac Compare August 8, 2023 17:43

dylanclam12 added 14 commits August 8, 2023 10:52

SpatialData Modified Query Script

22e1141

Added a modified SpatialData query script that works around their intrinsic_axes issue.

SpatialData Formatting in _io.py and _geometry.py

a7ff251

Added a formatting function that reads a Spatial Data object to: sjoin points to shapes, sjoin shapes to shapes, and changes shape object indices to strings. Sjoin functions changed in geometry.

_shape_features.py Refactor

efda092

Refactor of _shape_features to read and save to SpatialData.

_point_features.py Refactor

f1912d5

Refactor point features to read and save to SpatialData.

_lp.py Refactor Part 1

7ff7f69

Refactoring lp function and implementing lp_diff for discrete phenotypes

_lp.py Refactor Part 2

ed6e1e9

Implemented lp_diff_continuous, caught phenotype error in lp_diff_discrete, and clipped infinite values to max for -log10p and -log10padj.

_neighborhoods.py Addition

e47a265

Added _neighborhoods.py with no changes

_utils.py sync_points function and minor fixes

f83ff83

Added sync_points function to sync points dask dataframe to groud truth sdata.table anndata + minor fixes

_flux.py Refactor

e724986

Refactored flux and fluxmap functions to SpatialData. Minor changes to format data and sindex_points.

Removed Dask Functionality

cb29d2d

Removed dask functionality since there is no speed up in the point computations. Changed geometry functions and edited how they were used in io, shape features, point features, and flux.

_flux_enrichment.py Refactor

55766cf

Refactored _flux_enrichment.py and removed register_points functionality since we are placing cell_raster in a points element that is dynamic instead of leaving it in unstructured.

_colocation.py, _decomposition.py, _composition.py Refactor

7068c43

Refactored colocation.py which relies on _decomposition.py being refactored. Current _composition.py comp_diff function does not work correctly. Refactored to the point where we run across the same error.

_plotting.py, _colors.py, _layers.py, _utils.py Refactor

0a33e1b

Refactoring plotting script and all of its depending scripts: colors, layers, utils

_lp.py, _multidimensional.py Plotting Refactor

f17bf6e

Refactoring plotting scripts for localization patterns

ckmah requested changes Dec 19, 2023

View reviewed changes

dylanclam12 requested a review from ckmah January 16, 2024 21:47

ckmah reviewed Jan 16, 2024

View reviewed changes

pyproject.toml Show resolved Hide resolved

ckmah reviewed Jan 16, 2024

View reviewed changes

pyproject.toml Show resolved Hide resolved

dylanclam12 added 9 commits March 22, 2024 11:07

_points_features.py: instance_key

06b615a

used instance_key as an indicator of cell_shapes across all point features.

test_point_features.py

495fdec

wrote test cases for point_features.py

simplifying test_point_features.py

cc2113e

moved reading and formatting sdata to setup. Condensed feature lists

simplifying test_shape_features.py

4eb7435

moved test cases to a loop instead of listing them one by one

Change PATTERN_FEATURES

f0dfe93

Changed PATTERN_FEATURES to match the new naming conventions (cell_boundaries_x and nucleus_boundaries_x)

_lp.py: instance_key

bf5f276

Incorporated the use of instance_key into lp to be consistent with the rest of the package

_lp.py Tests

ebf3e36

ckmah approved these changes Mar 24, 2024

View reviewed changes

dylanclam12 added 12 commits March 28, 2024 12:38

_geometry.py: set_metadata documentation

9fa35a3

Updated set_metadata documentation

_flux.py: instance_key

79cb21e

changed flux to use instance_key instead of hard coding cell_boundaries

_neighborhoods.py: feature_name

f6e44af

made the default value for gene feature_name to be consistent with the Xenium format like the rest of the package

_flux.py Tests

0fbd7a0

test script for flux and fluxmap

_flux_enrichment.py: set_points_metadata

cc18e88

changed overwriting cell_boundaires_raster to adding metadata with set_points_metadata

_flux_enrichement.py: instance_key

a179937

Changed all hard coded cell_raster to use the instance key

_flux_enrichment.py Tests

9034dd5

Test cases for flux enrichment functions

remove import random test_flux.py and test_flux_enrichment.py

dad41c4

removed import random from test_flux.py and test_flux_enrichment.py

_colocation.py: colocation instance_key and feature_key

285ee56

added instance_key and feature_key to colocation and _colocation_tensor

_colocation.py: coloc_quotient keys

48c8d62

added instance_key, feature_key, points key to coloc_quotient, _cell_clq, and _clq_statistic

_colocation.py Tests

411f62f

Tests for _colocation.py

_geometry.py Tests

30ecb71

Tests for _geometry.py

ckmah marked this pull request as ready for review April 5, 2024 20:44

ckmah approved these changes Apr 5, 2024

View reviewed changes

ckmah added 3 commits April 5, 2024 21:48

cleanup misc constants

ee1f007

Merge branch 'v2.1' into spatialdata_refactor

8fdfdf1

more merge conflicts

fcb72ec

ckmah merged commit e5e193c into v2.1 Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BPT-155] Spatial Data Refactor #104

[BPT-155] Spatial Data Refactor #104

dylanclam12 commented Aug 7, 2023 •

edited by ckmah

Loading

notion-workspace bot commented Aug 7, 2023

ckmah left a comment

dylanclam12 commented Jan 16, 2024 •

edited by ckmah

Loading

ckmah left a comment •

edited

Loading

ckmah left a comment

ckmah Mar 24, 2024

ckmah Mar 26, 2024

ckmah Mar 26, 2024

ckmah Apr 5, 2024

ckmah Apr 5, 2024

LucaMarconato commented Apr 6, 2024

		@@ -106,28 +98,23 @@ def _colocation_tensor(data: AnnData, copy: bool = False):
		tensor = s.todense()
		print(tensor.shape)

[BPT-155] Spatial Data Refactor #104

[BPT-155] Spatial Data Refactor #104

Conversation

dylanclam12 commented Aug 7, 2023 • edited by ckmah Loading

notion-workspace bot commented Aug 7, 2023

ckmah left a comment

Choose a reason for hiding this comment

dylanclam12 commented Jan 16, 2024 • edited by ckmah Loading

ckmah left a comment • edited Loading

Choose a reason for hiding this comment

ckmah left a comment

Choose a reason for hiding this comment

ckmah Mar 24, 2024

Choose a reason for hiding this comment

ckmah Mar 26, 2024

Choose a reason for hiding this comment

ckmah Mar 26, 2024

Choose a reason for hiding this comment

ckmah Apr 5, 2024

Choose a reason for hiding this comment

ckmah Apr 5, 2024

Choose a reason for hiding this comment

LucaMarconato commented Apr 6, 2024

dylanclam12 commented Aug 7, 2023 •

edited by ckmah

Loading

dylanclam12 commented Jan 16, 2024 •

edited by ckmah

Loading

ckmah left a comment •

edited

Loading