From 3216e0c1668d2689f9bdde3fa4b2c8b1cd817ade Mon Sep 17 00:00:00 2001 From: alex-l-kong Date: Tue, 27 Sep 2022 11:26:47 -0700 Subject: [PATCH 1/2] Remove _ark from the templates folder --- README.md | 16 ++++++++-------- ark/utils/io_utils_test.py | 3 +-- ark/utils/notebook_update_test.py | 6 +++--- ark/utils/notebooks_test.py | 12 ++++++------ start_docker.sh | 2 +- .../1_Segment_Image_Data.ipynb | 0 .../2_Cluster_Pixels.ipynb | 0 .../3_Cluster_Cells.ipynb | 0 .../4_Post_Clustering.ipynb | 0 .../LDA_Preprocessing.ipynb | 0 .../LDA_Training_and_Inference.ipynb | 0 .../example_neighborhood_analysis_script.ipynb | 0 .../example_pairwise_spatial_enrichment.ipynb | 0 .../example_visualization.ipynb | 0 update_notebooks.sh | 6 +++--- 15 files changed, 22 insertions(+), 23 deletions(-) rename {templates_ark => templates}/1_Segment_Image_Data.ipynb (100%) rename {templates_ark => templates}/2_Cluster_Pixels.ipynb (100%) rename {templates_ark => templates}/3_Cluster_Cells.ipynb (100%) rename {templates_ark => templates}/4_Post_Clustering.ipynb (100%) rename {templates_ark => templates}/LDA_Preprocessing.ipynb (100%) rename {templates_ark => templates}/LDA_Training_and_Inference.ipynb (100%) rename {templates_ark => templates}/example_neighborhood_analysis_script.ipynb (100%) rename {templates_ark => templates}/example_pairwise_spatial_enrichment.ipynb (100%) rename {templates_ark => templates}/example_visualization.ipynb (100%) diff --git a/README.md b/README.md index a876f2a1e..b9ef39a19 100644 --- a/README.md +++ b/README.md @@ -47,33 +47,33 @@ This repo contains tools for analyzing multiplexed imaging data. The assumption We have recorded workshop talks which complement the repository. [MIBI Workshop Playlist](https://youtube.com/playlist?list=PLjNbkEm4vA26o5YvWKeyHXF8HjTJc7yB0). #### 1. Segmentation -The [**segmentation notebook**](./templates_ark/1_Segment_Image_Data.ipynb) will walk you through the process of using [Mesmer](https://www.nature.com/articles/s41587-021-01094-0) to segment your image data. This includes selecting the appropriate channel(s) for segmentation, running your data through the network, and then extracting single-cell statistics from the resulting segmentation mask. [Workshop Talk - Session V - Part 1: Segmentation](https://youtu.be/4_AJxrxPYlk?t=231) +The [**segmentation notebook**](./templates/1_Segment_Image_Data.ipynb) will walk you through the process of using [Mesmer](https://www.nature.com/articles/s41587-021-01094-0) to segment your image data. This includes selecting the appropriate channel(s) for segmentation, running your data through the network, and then extracting single-cell statistics from the resulting segmentation mask. [Workshop Talk - Session V - Part 1: Segmentation](https://youtu.be/4_AJxrxPYlk?t=231) #### 2. Pixel clustering with Pixie -The first step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**pixel clustering notebook**](./templates_ark/2_Cluster_Pixels.ipynb). The notebook walks you through the process of generating pixel clusters for your data, and lets you specify what markers to use for the clustering, train a model, use it to classify your entire dataset, and generate pixel cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session IV - Pixel Level Analysis](https://youtu.be/e7C1NvaPLaY) +The first step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**pixel clustering notebook**](./templates/2_Cluster_Pixels.ipynb). The notebook walks you through the process of generating pixel clusters for your data, and lets you specify what markers to use for the clustering, train a model, use it to classify your entire dataset, and generate pixel cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session IV - Pixel Level Analysis](https://youtu.be/e7C1NvaPLaY) #### 3. Cell clustering with Pixie -The second step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**cell clustering notebook**](./templates_ark/3_Cluster_Cells.ipynb). This notebook will use the pixel clusters generated in the first notebook to cluster the cells in your dataset. The notebook walks you through generating cell clusters for your data and generates cell cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session V - Cell-level Analysis - Part 2: Cell Clustering](https://youtu.be/4_AJxrxPYlk?t=2704) +The second step in the [Pixie](https://www.biorxiv.org/content/10.1101/2022.08.16.504171v1) pipeline is to run the [**cell clustering notebook**](./templates/3_Cluster_Cells.ipynb). This notebook will use the pixel clusters generated in the first notebook to cluster the cells in your dataset. The notebook walks you through generating cell clusters for your data and generates cell cluster overlays. The notebook includes a GUI for manual cluster adjustment and annotation. [Workshop Talk - Session V - Cell-level Analysis - Part 2: Cell Clustering](https://youtu.be/4_AJxrxPYlk?t=2704) #### 4. Post Clustering Tasks -After the Pixie Pipeline, the user can inspect and fine tune their results with the [**post clustering notebook**](./templates_ark/4_Post_Clustering.ipynb). This notebook will go over cleaning up artifacts left from clustering, and working with functional markers. +After the Pixie Pipeline, the user can inspect and fine tune their results with the [**post clustering notebook**](./templates/4_Post_Clustering.ipynb). This notebook will go over cleaning up artifacts left from clustering, and working with functional markers. #### 5. Spatial Analysis [Workshop Talk - Session VI - Spatial Analysis - Part 1: Choosing the Right Analysis Tool](https://youtu.be/HQAkS7kbzH8). 1. **Pairwise Enrichment Analysis** - The [**pairwise enrichment notebook**](templates_ark/example_pairwise_spatial_enrichment.ipynb) allows the user to investigate the + The [**pairwise enrichment notebook**](templates/example_pairwise_spatial_enrichment.ipynb) allows the user to investigate the interaction between the phenotypes present in their data. In addition users can cluster based on phenotypes around a particular feature such as *artery* or *gland*. [Workshop Talk - Session VI - Spatial Analysis - Part 2: Pairwise Spatial Enrichment](https://youtu.be/HQAkS7kbzH8?t=619). 2. **K-means Neighborhood Analysis** - The [**neighborhood analysis notebook**](templates_ark/example_neighborhood_analysis_script.ipynb) sheds light on neighborhoods made of micro-environments which consist of a collection of cell phenotypes. [Workshop Talk - Session VI - Spatial Analysis - Part 3: K-means Neighborhood Analysis](https://youtu.be/HQAkS7kbzH8?t=2117). + The [**neighborhood analysis notebook**](templates/example_neighborhood_analysis_script.ipynb) sheds light on neighborhoods made of micro-environments which consist of a collection of cell phenotypes. [Workshop Talk - Session VI - Spatial Analysis - Part 3: K-means Neighborhood Analysis](https://youtu.be/HQAkS7kbzH8?t=2117). 1. **Spatial LDA** - The [**preprocessing**](templates_ark/LDA_Preprocessing.ipynb) and [**training / inference**](templates_ark/LDA_Training_and_Inference.ipynb) draws from language analysis, specifically topic modelling. Spatial LDA overlays a probability distribution on cells belonging to a any particular micro-environment. [Workshop Talk - Session VI - Spatial Analysis - Part 4: Spatial LDA](https://youtu.be/HQAkS7kbzH8?t=3087). + The [**preprocessing**](templates/LDA_Preprocessing.ipynb) and [**training / inference**](templates/LDA_Training_and_Inference.ipynb) draws from language analysis, specifically topic modelling. Spatial LDA overlays a probability distribution on cells belonging to a any particular micro-environment. [Workshop Talk - Session VI - Spatial Analysis - Part 4: Spatial LDA](https://youtu.be/HQAkS7kbzH8?t=3087). ### Installation Steps @@ -127,7 +127,7 @@ You can shut down the notebooks and close docker by entering `control-c` in the **REMEMBER TO DUPLICATE AND RENAME NOTEBOOKS** -If you didn't change the name of any of the notebooks within the `templates_ark` folder, they will be overwritten when you decide to update the repo. Read about updating Ark [here](#updating-the-repository) +If you didn't change the name of any of the notebooks within the `templates` folder, they will be overwritten when you decide to update the repo. Read about updating Ark [here](#updating-the-repository) ## External Tools diff --git a/ark/utils/io_utils_test.py b/ark/utils/io_utils_test.py index 66f21de4d..8e0cd7649 100644 --- a/ark/utils/io_utils_test.py +++ b/ark/utils/io_utils_test.py @@ -9,8 +9,7 @@ def test_validate_paths(): # change cwd to /scripts for more accurate testing - # we'll ignore templates_qc because that is only run on the commercial MIBI computer - os.chdir('templates_ark') + os.chdir('templates') # make a tempdir for testing with tempfile.TemporaryDirectory(dir='../data') as valid_path: diff --git a/ark/utils/notebook_update_test.py b/ark/utils/notebook_update_test.py index 495fd7d1f..95b201d8f 100644 --- a/ark/utils/notebook_update_test.py +++ b/ark/utils/notebook_update_test.py @@ -23,7 +23,7 @@ def _exec_update_notebooks(base_path, update_flag=True, bad_flag=False): if bad_flag: args.append("-g") - # attempt to copy files from base_path/templates_ark to scripts + # attempt to copy files from base_path/templates to scripts try: # we have to append /private ahead of the base_path due to the # way the temp_dir gets configured @@ -48,9 +48,9 @@ def _exec_update_notebooks(base_path, update_flag=True, bad_flag=False): def _make_dir_and_exec(base_dir, templates, scripts=None, update_flag=True, bad_flag=False): - os.mkdir(os.path.join(base_dir, "templates_ark")) + os.mkdir(os.path.join(base_dir, "templates")) for template in templates: - pathlib.Path(os.path.join(base_dir, "templates_ark", template[0])).write_text(template[1]) + pathlib.Path(os.path.join(base_dir, "templates", template[0])).write_text(template[1]) if scripts is not None: os.mkdir(os.path.join(base_dir, "scripts")) diff --git a/ark/utils/notebooks_test.py b/ark/utils/notebooks_test.py index 37bd1908f..7b818a903 100644 --- a/ark/utils/notebooks_test.py +++ b/ark/utils/notebooks_test.py @@ -12,11 +12,11 @@ SEGMENT_IMAGE_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), - '..', '..', 'templates_ark', '1_Segment_Image_Data.ipynb') + '..', '..', 'templates', '1_Segment_Image_Data.ipynb') PIXEL_CLUSTER_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), - '..', '..', 'templates_ark', '2_Cluster_Pixels.ipynb') + '..', '..', 'templates', '2_Cluster_Pixels.ipynb') CELL_CLUSTER_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), - '..', '..', 'templates_ark', '3_Cluster_Cells.ipynb') + '..', '..', 'templates', '3_Cluster_Cells.ipynb') def _exec_notebook(nb_filename, base_folder): @@ -31,15 +31,15 @@ def _exec_notebook(nb_filename, base_folder): # test runs with default inputs def test_segment_image_data(): - _exec_notebook('1_Segment_Image_Data.ipynb', 'templates_ark') + _exec_notebook('1_Segment_Image_Data.ipynb', 'templates') def test_example_pairwise_spatial_enrichment(): - _exec_notebook('example_pairwise_spatial_enrichment.ipynb', 'templates_ark') + _exec_notebook('example_pairwise_spatial_enrichment.ipynb', 'templates') def test_example_neighborhood_analysis(): - _exec_notebook('example_neighborhood_analysis_script.ipynb', 'templates_ark') + _exec_notebook('example_neighborhood_analysis_script.ipynb', 'templates') # test folder inputs for image segmentation diff --git a/start_docker.sh b/start_docker.sh index 1c59526bc..2f9233c3c 100755 --- a/start_docker.sh +++ b/start_docker.sh @@ -8,7 +8,7 @@ while test $# -gt 0 do case "$1" in -n|--develop-notebook-templates) - JUPYTER_DIR='templates_ark' + JUPYTER_DIR='templates' shift ;; -u|--update) diff --git a/templates_ark/1_Segment_Image_Data.ipynb b/templates/1_Segment_Image_Data.ipynb similarity index 100% rename from templates_ark/1_Segment_Image_Data.ipynb rename to templates/1_Segment_Image_Data.ipynb diff --git a/templates_ark/2_Cluster_Pixels.ipynb b/templates/2_Cluster_Pixels.ipynb similarity index 100% rename from templates_ark/2_Cluster_Pixels.ipynb rename to templates/2_Cluster_Pixels.ipynb diff --git a/templates_ark/3_Cluster_Cells.ipynb b/templates/3_Cluster_Cells.ipynb similarity index 100% rename from templates_ark/3_Cluster_Cells.ipynb rename to templates/3_Cluster_Cells.ipynb diff --git a/templates_ark/4_Post_Clustering.ipynb b/templates/4_Post_Clustering.ipynb similarity index 100% rename from templates_ark/4_Post_Clustering.ipynb rename to templates/4_Post_Clustering.ipynb diff --git a/templates_ark/LDA_Preprocessing.ipynb b/templates/LDA_Preprocessing.ipynb similarity index 100% rename from templates_ark/LDA_Preprocessing.ipynb rename to templates/LDA_Preprocessing.ipynb diff --git a/templates_ark/LDA_Training_and_Inference.ipynb b/templates/LDA_Training_and_Inference.ipynb similarity index 100% rename from templates_ark/LDA_Training_and_Inference.ipynb rename to templates/LDA_Training_and_Inference.ipynb diff --git a/templates_ark/example_neighborhood_analysis_script.ipynb b/templates/example_neighborhood_analysis_script.ipynb similarity index 100% rename from templates_ark/example_neighborhood_analysis_script.ipynb rename to templates/example_neighborhood_analysis_script.ipynb diff --git a/templates_ark/example_pairwise_spatial_enrichment.ipynb b/templates/example_pairwise_spatial_enrichment.ipynb similarity index 100% rename from templates_ark/example_pairwise_spatial_enrichment.ipynb rename to templates/example_pairwise_spatial_enrichment.ipynb diff --git a/templates_ark/example_visualization.ipynb b/templates/example_visualization.ipynb similarity index 100% rename from templates_ark/example_visualization.ipynb rename to templates/example_visualization.ipynb diff --git a/update_notebooks.sh b/update_notebooks.sh index 34ea4677c..b10589c82 100644 --- a/update_notebooks.sh +++ b/update_notebooks.sh @@ -24,7 +24,7 @@ then if [ $update -ne 0 ] then # check for each template's existance - for f in "$PWD"/templates_ark/*.ipynb + for f in "$PWD"/templates/*.ipynb do # get basename of notebook name=$(basename "$f") @@ -55,10 +55,10 @@ then fi done else - cp -n "$PWD/templates_ark"/*\.ipynb "$PWD/scripts/." + cp -n "$PWD/templates"/*\.ipynb "$PWD/scripts/." fi else # since there is no scripts directory, just make one and copy from templates mkdir "$PWD/scripts" - cp "$PWD"/templates_ark/*\.ipynb "$PWD/scripts/." + cp "$PWD"/templates/*\.ipynb "$PWD/scripts/." fi From 450a1851de5560916106176a173262bbfeb365e0 Mon Sep 17 00:00:00 2001 From: alex-l-kong Date: Wed, 28 Sep 2022 17:24:26 -0700 Subject: [PATCH 2/2] Fix reference to clustering notebook in post clustering notebook (should point to templates) --- templates/4_Post_Clustering.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/templates/4_Post_Clustering.ipynb b/templates/4_Post_Clustering.ipynb index c27ff4f52..14fcf5a2c 100644 --- a/templates/4_Post_Clustering.ipynb +++ b/templates/4_Post_Clustering.ipynb @@ -6,7 +6,7 @@ "metadata": {}, "source": [ "# Post-clustering tasks\n", - "This notebook allows the user to inspect and fine-tune the output of the [clustering](https://github.com/angelolab/ark-analysis/blob/main/templates_ark/3_Cluster_Cells.ipynb) notebook. There are two parts of this notebook. \n", + "This notebook allows the user to inspect and fine-tune the output of the [clustering](https://github.com/angelolab/ark-analysis/blob/main/templates/3_Cluster_Cells.ipynb) notebook. There are two parts of this notebook. \n", "1. Clustering cleanup: If there are clusters that were not properly separated during Pixie cell clustering, this provides the option of defining manual thresholds based on marker intensity to combine or separate specific clusters\n", "2. Marker thresholding: For markers that were not directly used for clustering, but whose expression is important for phenotyping individual cells, this provides the user with visualization to determine accurate thresholds for positive/negative classification" ] @@ -400,7 +400,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.13" + "version": "3.8.13" } }, "nbformat": 4,