Single mantis directory #1061

camisowers · 2023-09-11T19:51:44Z

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #669 and closes #1059. Creates a single top level mantis directory for notebooks 2, 3, and 4, to avoid duplicate files.

How did you implement your changes

Set the mantis folder to be created as one subdirectory within the base_dir, rather than within each specific notebook folder.
Change the output masks from notebook 4 to be named with _post_clustering_cell_mask instead of just _cell_mask to avoid overwriting notebook 3 cell cluster output files.
Change the output masks from the generic_cell_clustering notebook to be named _generic_cell_mask.

Remaining issues

Run mantis generation in all clustering notebooks

review-notebook-app · 2023-09-11T19:51:48Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

srivarra

Looks good!

cliu72

Just a small comment - I think I confused you/I was also confused when I commented on the issue about the naming thing.

I don't think mask_prefix is actually needed. The masks are generated using generate_and_save_pixel_cluster_masks, which doesn't even have an option for a prefix, only a suffix. So the way the code is, there would be no situation where the masks have a prefix. I was originally thinking we could add the prefix when the masks are copied over into the mantis directory, however, after looking at this code, I don't think it's actually needed - I think new_mask_suffix is enough. So like users who want to can use the new_mask_suffix, can make it like population_pixel_mask_whatevertheywant.tiff in the mantis folder. So like I think we can jut remove mask_prefix and leave everything else as is.

alex-l-kong

Nothing to add on top of Candace's comments.

ngreenwald · 2023-09-12T21:00:28Z

I think we should have a prefix so that all the masks are next to one another in alphabetical order when opening files in Mantis. Makes it easier to find the one you're looking for

cliu72 · 2023-09-12T21:06:48Z

I think we should have a prefix so that all the masks are next to one another in alphabetical order when opening files in Mantis. Makes it easier to find the one you're looking for

They all start with "population_" then with the suffix after - which will make them all easy to find I think? Since you're opening one FOV in mantis at a time anyways, you can just look for all the files that start with "population"

camisowers · 2023-09-12T21:09:11Z

I think we should have a prefix so that all the masks are next to one another in alphabetical order when opening files in Mantis. Makes it easier to find the one you're looking for

Currently they all have a built in prefix of "population". So masks by default are population_pixel_mask.tiff, population_cell_mask.tiff, etc. We can change that to be something higher alphabetically if that would be more convenient.

ngreenwald · 2023-09-12T21:38:10Z

Okay perfect

cliu72

Looks good!

* single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix

* added plotting notebook * updated readmed * git merge issues * git merge fixes v2 - docs/landing.md * type * Remove cells that don't have any pixel clusters expressed prior to Pixie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells * Include `img_sub_folder` as parameter in Mantis calls (#1050) * Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for * fixed clipped plots in the notebooks. * Update HuggingFace version (#1053) * v0.1.10 (#1057) * Added continuous variable segmentation plots * Pin to scikit-image to v0.19.3 (#1060) * pin to 0.19.3 * requirement less than 0.19.3 * Make sure overwrite functionality for pixel clustering propagates into helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite * Single mantis directory (#1061) * single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix * strange merge issue * removed branch filters on CI, so if you branch off of a branch and run GHA, the CI workflow should work * changed ci.yml, filter push on main branch only * python 3.9 doesn't support | for Union type overloading * added fovs parameter for continuous variable plots * updated notebook to include plots * updated cell clustering notebooks to add erosion (default to True), cleaned up some notebook outputs * fixed style imports * re-added test_generate_summary_stats. Was accidently deleted in a refactoring step. * newline at eof fiber_segmentation_test.py * seaborn-paper -> seaborn-v0_8-paper * added cell to view all plotting styles i mask gen nb * pycodestyle --------- Co-authored-by: alex-l-kong <31424707+alex-l-kong@users.noreply.github.com> Co-authored-by: camisowers <38049893+camisowers@users.noreply.github.com>

* plot_cluster Next Release v0.6.4 (#1012) * 0.6.4 * updated environment.yml * Update README.md * updated windows setup docs * added repo verison in git clone * file fix Deepcell upload loop cleaning (#1023) * loop check for successful deepcell upload * update test zip paths * helper functions pls * zip input, upload, and extract per batch * cleaning * docstring indent * fov -> fovs file name * remove overwrite warning * new dir for each test call * fix bad logic warning testing * remove arg from docstring * add helper function test * batch_num start at 1 * add previously processed warning * set timeouts and scrap retry * correct timeout errors and update tests * timeout 5 mins * add to unprocessed list after loop closes * continue loop after extraction * break instead of continue * skipped processing print fixes * adjust print statements * 3 seconds before second zip call update * updated readthedocs python version * updated plot_utils docstring * docstrings v2 * use GS [1,1] if cbar_visible is False, otherwise use GS [1, 2] for plot + colorbar * added cbar_visible T/F for tests * pycodestyle * added numpy list to cmap / norm function * Remove cells that don't have any pixel clusters expressed prior to Pixie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells * Include `img_sub_folder` as parameter in Mantis calls (#1050) * Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for * Update HuggingFace version (#1053) * v0.1.10 (#1057) * Pin to scikit-image to v0.19.3 (#1060) * pin to 0.19.3 * requirement less than 0.19.3 * Make sure overwrite functionality for pixel clustering propagates into helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite * Single mantis directory (#1061) * single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix * Mass plotting notebook (#1052) * added plotting notebook * updated readmed * git merge issues * git merge fixes v2 - docs/landing.md * type * Remove cells that don't have any pixel clusters expressed prior to Pixie (#1051) * Drop cells without any SOM cluster expressions * Add testing for dropping non-expressed cells * Include `img_sub_folder` as parameter in Mantis calls (#1050) * Ensure mantis can read the image sub folder specified * Test on example dataset * Default img_sub_folder to None in create_mantis_dir (standard used elsewhere) * Doc fix and updated example dataset * Update segmentation labels again * Clarify img_sub_folder (mostly for refreshing tests) * Force .github/get_example_dataset.py to use updated revision * Update the file names to look for * fixed clipped plots in the notebooks. * Update HuggingFace version (#1053) * v0.1.10 (#1057) * Added continuous variable segmentation plots * Pin to scikit-image to v0.19.3 (#1060) * pin to 0.19.3 * requirement less than 0.19.3 * Make sure overwrite functionality for pixel clustering propagates into helper functions (#1058) * Propagate overwrite functionality for pixel clustering * Ensure re-normalization prevented on overwrite for pixel SOM re-assignment * normalize_data should be the opposite of overwrite * Single mantis directory (#1061) * single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix * strange merge issue * removed branch filters on CI, so if you branch off of a branch and run GHA, the CI workflow should work * changed ci.yml, filter push on main branch only * python 3.9 doesn't support | for Union type overloading * added fovs parameter for continuous variable plots * updated notebook to include plots * updated cell clustering notebooks to add erosion (default to True), cleaned up some notebook outputs * fixed style imports * re-added test_generate_summary_stats. Was accidently deleted in a refactoring step. * newline at eof fiber_segmentation_test.py * seaborn-paper -> seaborn-v0_8-paper * added cell to view all plotting styles i mask gen nb * pycodestyle --------- Co-authored-by: alex-l-kong <31424707+alex-l-kong@users.noreply.github.com> Co-authored-by: camisowers <38049893+camisowers@users.noreply.github.com> * default ero * added erode=True for generate_cluster_mask --------- Co-authored-by: alex-l-kong <31424707+alex-l-kong@users.noreply.github.com> Co-authored-by: camisowers <38049893+camisowers@users.noreply.github.com>

camisowers added 2 commits September 11, 2023 11:42

single mantis dir, no recopying

d4ffa11

post clustering mask suffix

55e91da

camisowers added the enhancement New feature or request label Sep 11, 2023

camisowers self-assigned this Sep 11, 2023

camisowers added 2 commits September 11, 2023 16:54

generic clustering adjustments

5b763f2

mask prefix

6613588

camisowers requested review from srivarra and alex-l-kong September 12, 2023 17:50

srivarra approved these changes Sep 12, 2023

View reviewed changes

camisowers requested a review from cliu72 September 12, 2023 18:21

cliu72 requested changes Sep 12, 2023

View reviewed changes

remove mask prefix

7fd8ff1

alex-l-kong reviewed Sep 12, 2023

View reviewed changes

camisowers requested a review from cliu72 September 12, 2023 20:31

cliu72 approved these changes Sep 12, 2023

View reviewed changes

camisowers requested a review from ngreenwald September 12, 2023 23:31

ngreenwald approved these changes Sep 12, 2023

View reviewed changes

ngreenwald added this pull request to the merge queue Sep 12, 2023

Merged via the queue into main with commit 5ee9a5d Sep 13, 2023
16 checks passed

ngreenwald deleted the single_mantis_dir branch September 13, 2023 00:12

srivarra pushed a commit that referenced this pull request Sep 13, 2023

Single mantis directory (#1061)

ec690c5

* single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix

srivarra pushed a commit that referenced this pull request Sep 13, 2023

Single mantis directory (#1061)

ab241fc

* single mantis dir, no recopying * post clustering mask suffix * generic clustering adjustments * mask prefix * remove mask prefix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Single mantis directory #1061

Single mantis directory #1061

camisowers commented Sep 11, 2023 •

edited

Loading

review-notebook-app bot commented Sep 11, 2023

srivarra left a comment

cliu72 left a comment

alex-l-kong left a comment

ngreenwald commented Sep 12, 2023

cliu72 commented Sep 12, 2023

camisowers commented Sep 12, 2023

ngreenwald commented Sep 12, 2023

cliu72 left a comment

Single mantis directory #1061

Single mantis directory #1061

Conversation

camisowers commented Sep 11, 2023 • edited Loading

review-notebook-app bot commented Sep 11, 2023

srivarra left a comment

Choose a reason for hiding this comment

cliu72 left a comment

Choose a reason for hiding this comment

alex-l-kong left a comment

Choose a reason for hiding this comment

ngreenwald commented Sep 12, 2023

cliu72 commented Sep 12, 2023

camisowers commented Sep 12, 2023

ngreenwald commented Sep 12, 2023

cliu72 left a comment

Choose a reason for hiding this comment

camisowers commented Sep 11, 2023 •

edited

Loading