v0.5.2

The latest version of ark-analysis include many improvements to the Pixie pipeline such as allowing the ability to rerun SOM training, and allowing users to skip many of the regionprops fields which are computationally expensive to generate.

🚀 Features

• Remove deterministic argument from som function in pyFlowSOM @alex-l-kong (#893)
• GitHub Actions @srivarra (#895)
• Ensure cluster_helpers.PixelSOMCluster receives the fovs argument @alex-l-kong (#905)
• Allow user to set an explicit skip flag for custom regionprops and marker extraction functionality in notebook 1 @alex-l-kong (#897)

🐛 Bug Fixes

• Turn off minor ticks for pixel and cell cluster overlay colorbar @alex-l-kong (#912)
• Remove zero column pixel cluster columns from cell clustering pipeline @alex-l-kong (#884)
• Ensure z-score slider adjusts colorbar range when changed @alex-l-kong (#904)
• Ensure pixel SOM training reruns if new markers specified @alex-l-kong (#903)
• Correct 99.9% normalization to 100% normalization for "many-zero" columns @alex-l-kong (#894)

🧰 Maintenance

• Bump ipython from 8.4.0 to 8.10.0 in /docs @dependabot (#908)
  @alex-l-kong, @dependabot, and @srivarra

v0.5.1

This latest version of Ark Analysis includes moving shared utility functions across angelolab/ark-analysis, angelolab/toffy, and angelolab/mibi-bin-tools to their new home in angelolab/tmi. In addition some other updates include the ability to convert to and from OME-TIFFs and adding inverse as a custom method to allow for more recent versions of matplotlib. In addition a majority of the requirements were adjusted to be less strict.

🚀 Features

• Added nuc_seg_suffix for filter_with_nuclear_mask @srivarra (#879)
• Add inverse method to ZScoreNorm to integrate with current matplotlib @alex-l-kong (#871)
• OME-TIFF Read / Write @srivarra (#819)
• Load Utils - Expanding Acceptable File Types @srivarra (#878)

🧰 Maintenance

• Next Release v0.5.1 @srivarra (#889)
• Adding TMI @srivarra (#825)
  @alex-l-kong and @srivarra

v0.5.0

The latest release of Ark Analysis mainly focused on removing R dependencies. In order to do so, we have created a small version of FlowSOM over at angelolab/pyFlowSOM.

Other major features include more example datasets, and performance improvements.
Notable bug fixes include removing duplicate FOV loading and making the mantis file extension independent from the segmentation file extension.
Documentation improvements include improved git cloning / installation instructions, and ejecting an imposter Pixie author (ඞ).

🚀 Features

• Load Utils - Expanding Acceptable File Types @srivarra (#878)
• Example Dataset - Cropped FOV 0 @srivarra (#859)
• Integrate pyFlowSOM with Pixie @alex-l-kong (#863)
• Clean up Pixie pipeline @alex-l-kong (#856)
• Make segmentation_dir definition more clear for pixel and cell clustering processes @alex-l-kong (#855)
• Additional Input Validation @srivarra (#841)
• Add native Python consensus clustering process @alex-l-kong (#839)
• Subset pixel channel averaging to remove bottleneck @alex-l-kong (#823)
• Example Dataset - Spatial Datasets @srivarra (#832)
• Use random_state parameter in KMeans clustering for reproducibility @alex-l-kong (#834)
• Added Parameter for Seg Mask Suffix for Mantis @srivarra (#828)
• Rename _feature_0 to _whole_cell and _feature_1 to _nuclear @alex-l-kong (#786)
• Refactor distance matrix saving @alex-l-kong (#803)

🐛 Bug Fixes

• Make Mantis mask file extension independent from segmentation file extension @alex-l-kong (#870)
• seg_suffix Requires Explicit File Extension @srivarra (#848)
• Fix neighborhood matrix indexing due to subsetting @alex-l-kong (#824)
• Removed duplicate mantis fov loading @srivarra (#838)
• Don't load the pixel, cell, and neighborhood FOVs in batches for mask generation @alex-l-kong (#816)
• Ensure neighborhood inertia and silhouette score generation works with varying min_k @alex-l-kong (#820)
• Travis patchelf pypi_deploy Updates @srivarra (#805)

🧰 Maintenance

• Next Release v0.5.0 @srivarra (#877)

📚️ Documentation

• Tagged Installation Instructions @srivarra (#840)
• Example Dataset - Adding more info to Overwrite Existing @srivarra (#826)
• Fix Pixie authorship @ngreenwald (#813)
• Cell Table Assumptions @srivarra (#808)
  @alex-l-kong, @ngreenwald and @srivarra

v0.4.4

Users will now be able to tile their images with a script, and use the notebooks with example data. In addition this release adds more validation checks for user provided paths, optimization and a handful of bug fixes.

🚀 Features

• Standardize usage of .tiff extension @alex-l-kong (#801)
• Adjust load tiled imgs for toffy use @camisowers (#802)
• Compressed csv Option in Notebook 1 @srivarra (#800)
• Increase validate paths usage @camisowers (#796)
• Testbook Adjustments @srivarra (#791)
• Local Caching @srivarra (#794)
• Notebook Update Tests with string args @srivarra (#782)
• Example Dataset - Notebook 4 @srivarra (#763)
• Make non-parallel loops canon for Pixie @alex-l-kong (#779)
• Example Dataset - Notebook 3 @srivarra (#760)
• Image tiling script @camisowers (#702)
• Example Dataset - Notebook 2 @srivarra (#736)
• Remove mibitiff option in notebooks @camisowers (#772)
• Optimize dataframe merging @camisowers (#762)
• Add inertia and silhouette score progress bars for neighborhood analysis notebook @alex-l-kong (#758)
• Fiber segmentation @ackagel (#569)
• Add Read the Docs status badge to README @alex-l-kong (#755)

🐛 Bug Fixes

• In k-means neighborhood notebook, fix excluded_channels=None @cliu72 (#788)
• Run pytest as a separate job to avoid scripts from getting overwritten @alex-l-kong (#776)
• Patch up PyPI requirements @alex-l-kong (#753)
• Settings name update @camisowers (#741)

🧰 Maintenance

• Next Release - v0.4.4 @srivarra (#804)
  @ackagel, @alex-l-kong, @camisowers, @cliu72 and @srivarra

v0.4.3

Users will now be able to use the Docker image without any installation errors with the arrow library. Additionally, the Docker startup script now ensures only one container gets created per image.

🚀 Features

• Use existing Docker container when possible to prevent redundancy @alex-l-kong (#742)
• Remove batch_size parameter from spatial analysis functions @alex-l-kong (#752)
• Rework mask saving and cell labeling to support just single FOV parameter @alex-l-kong (#737)

🐛 Bug Fixes

• Patch up arrow library in Docker @alex-l-kong (#748)

🧰 Maintenance

• Bump to v0.4.3 @alex-l-kong (#757)

v0.4.2

🚀 Features

• Rename notebooks folder to templates @alex-l-kong (#729)
• Speed up relabel_segmentation @alex-l-kong (#733)
• Add progress bar for the EDA calculation in spatial LDA @alex-l-kong (#680)
• Remove batching functionality from cell and pixel mask generation @alex-l-kong (#727)
• Remove cell table batching functionality @alex-l-kong (#726)
• Update kmeans neighborhood notebook take 2 @cliu72 (#714)
• Nuclear counts loading @camisowers (#715)
• Removing Unused Imports @srivarra (#698)
• Example Dataset @srivarra (#685)

🐛 Bug Fixes

• Edit spatial enrichment notebook to only load in desired features @camisowers (#730)
• validate_paths data prefix @camisowers (#728)
• Support rectangular images for pixel mask generation @alex-l-kong (#723)
• Passes distance limit argument through channel enrichment function @ackagel (#679)
• Docker local changes @ackagel (#711)
• Handle variable size images for Deepcell input generation @alex-l-kong (#706)
• Allow user to set number of cores for R functions @alex-l-kong (#705)

🧰 Maintenance

• Next Release v0.4.2 @srivarra (#721)
• Fixed FlowSOM on Docker Image @srivarra (#684)

📚️ Documentation

• Add section to troubleshoot WSL distro issues @alex-l-kong (#697)
• Added Docker Vesion Badge @srivarra (#699)
• Workshop Links @srivarra (#694)
• Update README.md @ngreenwald (#683)
  @ackagel, @alex-l-kong, @camisowers, @cliu72, @ngreenwald and @srivarra

v0.4.1

🚀 Features

Post clustering functionality @ngreenwald (#658)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?
Adds in logic to annotate the cell table following pixel clustering, as well as detailed instructions for how to use Mantis.

How did you implement your changes
A notebook with two parts. The first allows the user to select subsets of cells, give them new names, and then create a mantis directory to examine them. The second allows the user to identify specific thresholds for functional markers.

Remaining issues
For people who haven't used Mantis, is this clear enough guidance for what they need to do? Are there places where the bare code in the notebook is going to lead to problems? Given that the user is directly manipulating the pandas df there are some inelegant lines still in there, but I couldn't think of a way to avoid it that wasn't super over-engineered.

Update spatial analysis @ackagel (#451)

Purpose

The spatial analysis notebook doesn't support the current saved label format, doesn't support batching (although that might be hard to add conceptually), and is generally pretty lackluster feature/documentation wise. This PR hopes to update the notebook, clarify implementation details, and add QoL features, such as an example cell-cluster/expression image plot.

Implementation

So far just extracting tifs from the .xr labels file, and converting the notebook to load those tiffs, instead of the .xr file. More changes to come from separate branches.

Remaining issues

Only the labels loading interface has been updated so far. The rest of the features mentioned still need to be implemented.

Simplify `save_segmentation_labels` to remove label saving and default to channel overlay saving @alex-l-kong (#661)

What is the purpose of this PR?

Closes #626. There is no need to resave the segmentation labels (already done so in _feature_0.tiff), and the user should by default receive an overlay with the nuclear and membrane channels.

How did you implement your changes

1. Remove saving of segmentation labels.
2. Default channels arg to ['nuclear_channel', 'membrane_channel']. This list will be assigned in ark.settings and copied over to prevent overwriting. Changes will propagate over to Segment_Image_Data.ipynb because no channel arg is passed, meaning the nuclear and membrane channel overlay gets generated by default.

Create release-drafter.yml @srivarra (#673)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Adds Github Actions to this repo via the workflow-template made in angelolab/.github.

How did you implement your changes

Made a few adjustments to the angelolab/.github.

Remaining issues

Will have to fix the icon later.

Delete .github/ISSUE\_TEMPLATE directory @ngreenwald (#670)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?
Removes the old templates in ark so that the repo-wide files will apply

🐛 Bug Fixes

Mantis Project - Improved Filtering of user Supplied FOVs @srivarra (#667)

• investigating mask issue
• added fov mask filtering

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #655.

How did you implement your changes

Updated the test to save different masks and segmentation labels for each FOV. In addition added filtering for user supplied FOVs, so only those masks with a supplied FOV get copied over. Improved the organization and documentation for this test function.

Remaining issues

Might want to refactor it one day, incase we create more Mantis related functions.

🧰 Maintenance

Next Release - v0.4.1 @srivarra (#674)

• added development docs for creating a new release
• label documentation -> dependencies

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Creates a new Release of Ark. In addition adds instructions about creating a new release.

How did you implement your changes

Added detailed steps to go through when you create a new release.

Remaining issues

Waiting for the following to be merged to main:

• #650
• #658
• #660
• #657 - Save v0.4.2
• #675

After this is merged:

Create the new release with the tag: v0.4.1. This will push it to PyPI and create a new Docker Image.

Python 3.8 and Native Apple Silicon Support @srivarra (#650)

• bumped packages, skimage.tifffile -> tifffile
• tiffwriter adjustments
• adjusted connectivity in measure.label

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #649. Closes #504.

How did you implement your changes

Fixed API changes with respect to tifffile, and scikit-image. In addition bumped a few packages up for Apple Silicon compatibility.

Updated rtd requirements as well.

Remaining issues

• Adjusting documentation, removing Rosetta installation instructions for miniforge.

📚️ Documentation

Ark Pipeline Flowchart @srivarra (#660)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #211.

How did you implement your changes

Added a pipeline flowchart for the ark notebooks.

Remaining issues

Should we consider renaming example_spatial_analysis_script to reflect that it's using spatial pairwise enrichment?

Update README.md to account for new Docker tagging system @alex-l-kong (#675)

What is the purpose of this PR?

Although start_docker.sh will no longer require explicit updates to ensure the right image gets pulled, the user still needs to ensure the correct image gets pulled.

start_docker.sh does automatically pull the image for you, but this only applies if you don't have the image already. If you need to overwrite an existing image, you'll need to explicitly run the docker pull command with the correct tag.

How did you implement your changes

Change the Docker section of the README.md to explicitly tell the user which tag to look out for when running the docker pull command. This change will also be applied to the Docker update section.

Remaining issues

If we plan on overwriting the Docker images each time the dependencies change, then we can just allow start_docker.sh to update the image in the background without explicit input from the user.

Python 3.8 and Native Apple Silicon Support @srivarra (#650)

• bumped packages, skimage.tifffile -> tifffile
• tiffwriter adjustments
• adjusted connectivity in measure.label

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #649. Closes #504.

How did you implement your changes

Fixed API changes with respect to tifffile, and scikit-image. In addition bumped a few packages up for Apple Silicon compatibility.

Updated rtd requirements as well.

Remaining issues

• Adjusting documentation, removing Rosetta installation instructions for miniforge.

Add a section to address restarting WSL in case of mounting issues @alex-l-kong (#664)

What is the purpose of this PR?

Closes #651. Occasionally, WSL may fail to mount an external drive correctly, meaning it cannot be accessed by Docker. The process to address this should be put in windows_setup.md.

How did you implement your changes

Add a brief step-by-step process for restarting WSL, which involves running wsl --shutdown and restarting Docker using the prompt that pops up.

Notebook Reorganization @ngreenwald (#648)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Reorganizes the notebooks to be more in line with toffy. Closes #640, closes #590

Remaining issues
The links, table of contents, etc in readme will need to be updated. The testbook stuff will need to be updated

@ackagel, @alex-l-kong, @ngreenwald and @srivarra

v0.4.0

🚀 Features

Split up `som_utils.py` into separate modules for pixel and cell clustering @alex-l-kong (#662)

What is the purpose of this PR?

Closes #544. It's confusing to have both pixel and cell clustering helper functions reside in the same module, so we'll split them up.

How did you implement your changes

Reorganize the functions to pixel_cluster_utils.py and cell_cluster_utils.py, with the test modules also being moved accordingly.

Remove Rplots.pdf @camisowers (#638)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #519. Removes the pdf that is produced by R scripts in the example_pixel_clustering notebook.

Docker Tags in line with ark-analysis Version @srivarra (#636)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #634, #659.

Currently latest always points to the most recent build of the docker image, which may not be the current version of the repository. As a best practice we should tag the docker image to the same version of ark-analysis.

How did you implement your changes

Adjusted the docker build section of the .travis.yml to include the tag, and pushing it properly.

Remaining issues

None atm.

Add nuclear filtering functionality to pixel clustering pipeline @alex-l-kong (#653)

What is the purpose of this PR?

Closes #619. Prior to pixel clustering, users may need to filter out non-nuclear marker signal from the nucleus and vice-versa for nuclear marker signal.

How did you implement your changes

Add a function filter_with_nuclear_mask to som_utils which runs this process. The code in #619 provides most of the functionality, however we'll need to make two modifications:

• Ensure seg_dir is not None prior to running. Users may run pixel clustering without segmentation labels, however they will not be able to run this function if that's the case because it relies on the _feature_1.tiff files.
• Ensure channel defaults to None in the notebook and function. This is the easiest guard against errors caused by running all the cells at once in the notebook.

The cell that calls this function in example_pixel_clustering will be placed after the smoothing process.

This PR also solidifies some of the notebook testing, as it was not using the renamed channels after smoothing. This should also apply for the nuclear filtering tests too.

Added Batching Functionality to Pixel Mask Generation and FOV Saving @srivarra (#643)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #552

How did you implement your changes

Added two batching functions for the pixel and cell clustering notebooks. In addition added testbook tests.

Remaining issues

None ATM.

Create Mantis Project Function @srivarra (#641)

• added mantis function
• mantis_project made

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #563.

How did you implement your changes

Added the function which creates a mantis project directory. Added tests to the mantis project function.

Remaining issues

Need to encapsulate pixel clustering output as well. Potential reorganization of the output if necessary. Also need to add an argument which only runs over a subset of fovs supplied (so the user won't copy over their entire run contents into a mantis project).

Save fov images as int16 @camisowers (#644)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #521.
Edits the save_fov_images function so the pixel clustering notebook saves overlays images that can be loaded into photoshop.

Remaining issues

Could instead add the saving dtype as a default arg to the function, if that's something you think we'd use.

Ensure the meta cluster rename drop down menu for remapping doesn't get cut off @alex-l-kong (#642)

What is the purpose of this PR?

Closes #630. The way the toolbar is set up, the meta cluster renaming textbox/drop-down menu can get cut off on the far right. This needs to be pushed further to the center.

How did you implement your changes

Adjust the toolbar's layout.justify_content setting to 'center'.

Add option to restart pixel clustering notebook from crash @alex-l-kong (#620)

What is the purpose of this PR?

Closes #575. example_pixel_clustering.ipynb includes very expensive processes such as preprocessing, SOM training, and SOM cluster assignment. This PR provides a way to prevent these from re-running (at least in full) for items that have already been checked off on a previous run that crashed midway through.

How did you implement your changes

Add options to check for files, or specific columns in files, indicating that certain processes have finished or not. This ensures we only need to process FOVs that haven't been completed.

Removal of `force_ints` in all load functions. @srivarra (#609)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #606.

load_imgs_from_tree and load_imgs_from_dir will no longer allow any type conversions. There are significantly fewer user accessible type conversion options throughout ark.

How did you implement your changes

Removed the majority of front facing instances dtype in function arguments. In deepcell_service_utils the raw bytes output from Deepcell will now be converted from floating point to integer values.

Remaining issues

The JSON returned by Deepcell has the following metadata when running the Segment_Image_Data.ipynb notebook:
{"shape": [1, 1024, 1024]}. This is the same for this branch, master (after #605 was merged in), and with the last commit in #624. Seems like it was just something I got confused with when I was adjusting the reshaping earlier in the PR process.

Add `dtype` argument to channel and pixel norm functions @alex-l-kong (#623)

What is the purpose of this PR?

Closes #612. The dtype argument currently isn't propagated through to the channel and pixel normalization functions, causing the load_imgs_from_tree calls to attempt to load the images using the default int16 type. This will throw a warning if the images were of type float32 regardless of if the user passed float32 to create_pixel_matrix.

How did you implement your changes

Add dtype with a default argument of int16 to both calculate_channel_percentiles and calculate_pixel_intensity_percentile. Explicitly pass the dtype argument received by create_pixel_matrix to these functions.

🐛 Bug Fixes

Docker Tags in line with ark-analysis Version @srivarra (#636)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Closes #634, #659.

Currently latest always points to the most recent build of the docker image, which may not be the current version of the repository. As a best practice we should tag the docker image to the same version of ark-analysis.

How did you implement your changes

Adjusted the docker build section of the .travis.yml to include the tag, and pushing it properly.

Remaining issues

None atm.

Ensure channel norm and pixel norm files get re-computed if a new set of channels are provided @alex-l-kong (#654)

What is the purpose of this PR?

Closes #599. The underlying issue is if the user provides a different set of channels on a pixel_output_dir that has an existing channel_norm.feather computed on a different set of channels.

How did you implement your changes

We need to ensure that the set of channels provided when loading in an existing channel_norm.feather match the channels argument set-wise. If they don't, then we simply re-compute and re-save.

The same needs to be done for pixel_norm.feather. We store a checkpoint during the channel_norm.feather check to ensure this can happen seamlessly for the pixel norm too.

Pin matplotlib at 3.4.3 in requirements.txt to prevent colorbar scaling issue @alex-l-kong (#639)

What is the purpose of this PR?

Closes #614. Currently, the Python 3.7 image is built using the latest matplotlib version (3.5.x), which introduced several bugs to colorbar scaling. Several issues related to this are still open on t...

v0.3.2

🐛 Bug Fixes

PKG_FOLDER variable in `setup.py` cannot find `requirements.txt` due to filepath issue. @srivarra (#624)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

ark-analysis will not install properly via PyPI, the issue seems to reside with the PKG_FOLDER variable.

How did you implement your changes

Set PKG_FOLDER the same as it is in toffy.

Remaining issues

Need to adjust ark-analysis requirement in angelolab/toffy#132, and make sure it can install properly.

v0.3.1

🚀 Features

Place `channel_norm.feather` and `pixel_norm.feather` in the pixel directory @alex-l-kong (#616)

What is the purpose of this PR?

Closes #610. This PR saves the normalization files in the pixel directory for the clustering run to ensure it doesn't get overwritten. Additionally, it adds pixel_cluster_prefix to their names.

🐛 Bug Fixes

VERSION 0.3.00 -> 0.3.1 @srivarra (#617)

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

This fixes an issue where version 0.3.00 does not exist, and pip / conda will try to install this version and fails.

How did you implement your changes

Adjusts the version from 0.3.00 to 0.3.1. This should fix the installing it locally.

Remaining issues

None ATM.

Add back channel normalization before preprocessing @cliu72 (#613)

What is the purpose of this PR?

Add back channel normalization before pixel matrix generation, which seems to have been lost when parallelization was added. As in this commit: https://github.com/angelolab/ark-analysis/tree/370415cea71ec675c4be9f2c57a8644702390350 before the parallelization functionality was added, pixel values should be channel normalized before being fed into create_fov_pixel_data.

How did you implement your changes

Added back the code to channel normalize in preprocess_fov before calling create_fov_pixel_data.

Remaining issues

Don't think there are remaining issues.

@alex-l-kong, @cliu72 and @srivarra