Releases: angelolab/ark-analysis
v0.3.0
Major Updates
- Ark-Analysis now uses Python 3.7.
- Added FlowSOM implementation and pipeline.
🚀 Features
remove external manual clustering @camisowers (#615)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Closes #574.
Removes the example_manually_adjust_metaclusters
notebook and the metaclustering
folder from ark-analysis/data/example_dataset
.
Run expensive per-FOV pixel clustering processes in parallel @alex-l-kong (#596)
What is the purpose of this PR?
Closes #532. There are many processes for pixel clustering that run on a one-per-FOV basis that can be optimized with parallelization.
How did you implement your changes
We target four functions:
create_pixel_matrix
run_pixel_som
pixel_consensus_cluster
apply_pixel_meta_cluster_remapping
1 and 4 are done in Python and rely on the multiprocessing
module to generate separate processes in parallel. 2 and 3 are done in R, which provides a much cleaner solution using parallel
, doParallel
, and foreach
.
Added Release Drafter @srivarra (#598)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Closes #597.
Adds Release Drafter using Github Actions.
How did you implement your changes
Added:
workflows/release-drafter.yaml
: Uses Github Actions Workflows to build the release notes.release-drafter.yaml
: Organizes the version resolver, and organizes labeled PRs into the following categories:- Features
- Bug Fixes
- Maintenance
- Documentation
Remaining issues
We can add more categories as well using labels, and projects.
Remove intermediate `_clustered`/`_consensus` directories/files from pixel and cell clustering pipeline @alex-l-kong (#586)
What is the purpose of this PR?
Closes #571. There is no need to create a separate pixel_mat_clustered
directory/cell_mat_clustered
file as the data is just the channel expression/pixel cluster count expression values with an additional pixel_som_cluster
/cell_som_cluster
attached. Ditto for the pixel_mat_consensus
directory/cell_mat_consensus
file. These should be removed from the pixel and cell clustering processes.
How did you implement your changes
We should overwrite the original data with the newly created data in the cluster_pixels
/pixel_consensus_cluster
(and similarly for the cell clustering functions).
Remaining issues
There are a few extra checks we need to include for validation if the user decides to rerun the clustering pipeline in the same session.
Pix preprocessing @ngreenwald (#579)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Adds in functionality to normalize each channel of image data data separately prior to pixel clustering. This helps to make sure that markers which have different intensity values are treated equally right from the beginning of the clustering process.
In addition, it changes from removing pixels that have 0 total counts to removing pixels in the bottom 5% of total counts from the image. This better matches the format of the data following rosetta, where there are very few true zeros.
Remaining issues
The testing I put together is very basic. @alex-l-kong, if you could go in and double check that everything is working as intended, and if needed adding more testing, that would be great. Also feel free to change the organization/saving structure if you think it would be better some other way. I didn't add any new tests for create_pixel_matrix, that will likely need to be checked as well.
Added `sub_dir` option for `save_fov_images` @srivarra (#580)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Closes #533.
Allows fov_image
masks to be saved in a separate subfolder instead of at the top level.
How did you implement your changes
Added a new subdirectory parameter in save_fov_images
. Added another test as well.
Remaining issues
None atm.
Custom channel smoothing @ngreenwald (#581)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Adds in functionality for different smoothing of individual channels in pixel clustering
How did you implement your changes
Provides the user a separate step to specify individual channels for increased smoothing, along with custom smoothing coefficients.
Also adds in an error check to create_pixel_matrix
which identifies cases where the user provided the original channel name, but a modified channel name exists. This will be useful for other pre-processing steps we introduce which produce modified channels
Spatial lda @bcollica (#437)
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Creates the directories and some files for the spatial-LDA pipeline and gives an initial example of code for preprocessing steps.
How did you implement your changes
Added a function for formatting a cell table to be compatible with the existing spatial_lda library. Also added a function for checking valid arguments along with a test.
Remaining issues
Fixing code format to be pep8 compliant so builds can pass.
Update FlowSOM pipeline with updated `MapDataToCodes` optional flag in `SOM` function @alex-l-kong (#570)
What is the purpose of this PR?
Closes #568. This addresses the issue caused by the current functionality of SOM
in the FlowSOM
library which causes MapDataToCodes
to run by default.
How did you implement your changes
- We've created a fork of the main
FlowSOM
repo and added an optional flagmap
to theSOM
function which allows us to prevent the call toMapDataToCodes
. This parameter is also added to ourSOM
calls increate_pixel_som.R
andcreate_cell_som.R
. - For the time being, we add an explicit command to overwrite the default
FlowSOM
installation with our forkedFlowSOM
package. This will be removed whenever the devs of the mainFlowSOM
repo review the PR I've opened addressing themap
parameter.
Remaining issues
- It is possible there may be other memory-related issues we'll run into that will need further addressing.
- It is possible the main
FlowSOM
devs do not approve our PR. In that case, we'll have to continue piggybacking on the forkedFlowSOM
repo for the foreseeable future. That might be a good idea considering we may want to make more optimizations to the C code and what not.
Append cell meta cluster labels to cell table @alex-l-kong (#567)
What is the purpose of this PR?
Closes #551 and closes #523. The cell table is a requirement for several downstream analysis scripts, not just pixel clustering. To ensure analysis can be performed based on the cell meta cluster labels, these will need to be appended to the cell table as well.
How did you implement your changes
A function add_consensus_labels_cell_table
will run this appending process, with the call in example_cell_clustering.ipynb
placed at the end of the notebook. The original cell_table_name
will be overwritten with the same data including the 'cell_meta_cluster_rename'
column in cell_consensus_name
(simply 'cell_meta_cluster'
in the cell table).
Users will be able to save multiple times and can also re-run the cell SOM training with the newly-saved cell table if desired.
Additionally, we now allow more flexible locations for the cell table. Initially, it was forced to be in base_dir
. Now, we default the location to segmentation_dir
, with options to change the root. Note that the updated cell table will be saved in the same root directory as the original cell table.
Delete references to Docker locally rebuilding in `README.md` @alex-l-kong (#560)
What is the purpose of this PR?
There shouldn't be any more references to Docker needing to rebuild locally now that we have Dockerhub integration.
How did you implement your changes
This necessitates the removal of the line indicating that Docker would rebuild with the -u
or --update
flag pass to start_docker.sh
.
Speed up generation of pixel cluster masks overlay @alex-l-kong (#546)
What is the purpose of this PR?
Closes #535. generate_pixel_cluster_masks
in data_utils.py
runs very slowly and needs to be optimized.
How did you implement your changes
The main issue is in the way the mask for each FOV is currently indexed. Currently, each FOV image is accessed in its original 2D format. Due to ho...
Docker Hub Integration
Users should now pull pre-built images from Docker Hub instead of building locally.
Validation of inputs
This release incorporates better sanitization of input data to ensure that the functions can run as expected for generating the Mesmer figures.
Better support for MIBITIFF files
This release includes a number of bug fixes for loading and analyzing MIBITIFF files
Update requirements and add support for integrated deepcell processing
This release updates the way requirements are handled by PyPI in preparation for the 0.3 release, as well as some usability improvements
Bugfix for setup.py
Fixes a bug in the setup.py file that prevented correctly pip installation
Automated requirements parsing
Fixes bug in Travis PyPI deployment; automatically generates required packages from requirements.txt file
Automated PyPI deployment
Interim release to check if Travis is correctly grabbing releases and formatting them for pip installation
v0.2.3: travisCI integration with PyPI (#46)
for travis_CI to push to PyPI
Importable util functions
added required init.py files so that util functions can be imported into other projects