templates/2_Pixie_Cluster_Pixels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pixie: pixel clustering notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "import"
    ]
   },
   "outputs": [],
   "source": [
    "# import required packages\n",
    "import json\n",
    "import os\n",
    "from datetime import datetime as dt\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from alpineer import io_utils, load_utils\n",
    "from matplotlib import rc_file_defaults\n",
    "\n",
    "from ark.phenotyping import (pixel_cluster_utils, pixel_meta_clustering,\n",
    "                             pixel_som_clustering, pixie_preprocessing)\n",
    "from ark.utils import data_utils, example_dataset, plot_utils\n",
    "from ark.utils.metacluster_remap_gui import (MetaClusterGui,\n",
    "                                             colormap_helper,\n",
    "                                             metaclusterdata_from_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0: Set root directory and download the example dataset\n",
    "\n",
    "Here we are using the example data located in `data/example_dataset`. To use your own data, change `base_dir` to point to your own sub-directory within the data folder.\n",
    "\n",
    "* `base_dir`: the path to all of your imaging data. This directory will also store all of the directories/files created during pixel clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "base_dir"
    ]
   },
   "outputs": [],
   "source": [
    "base_dir = \"../data/example_dataset\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test the features in ark with an example dataset, run the cell below. It will download a dataset consisting of 11 FOVs with 22 channels. You may find more information about the example dataset in the [README](../README.md#example-dataset).\n",
    "\n",
    "If you are using your own data, skip the cell below.\n",
    "\n",
    "* `overwrite_existing`: If set to `False`, it will not overwrite existing data in `data/example_dataset`. Recommended setting to `False` if you are running Notebooks 1,2,3 and 4 in succession. Set to `True` if you are just running Notebook 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "ex_data_download"
    ]
   },
   "outputs": [],
   "source": [
    "example_dataset.get_example_dataset(dataset=\"cluster_pixels\", save_dir = base_dir, overwrite_existing = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1: Set file paths and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `tiff_dir`: path to the directory containing your imaging data. Images should be single-channel TIFFs.\n",
    "* `img_sub_folder`: if `tiff_dir` contains an additional subfolder structure, override `None` with the appropriate folder name\n",
    "* `segmentation_dir`: path to the directory containing your segmentations (which can be generated using `1_Segment_Image_Data.ipynb`). Set this argument to `None` if you do not have segmentation labels or wish to run pixel clustering without them (they are required for cell clustering)\n",
    "* `seg_suffix`: the suffix plus the file extension of the segmented images for each FOV. Note that these should be the same for all FOVs. This argument will be ignored if `segmentation_dir` is set to `None`\n",
    "* `pixie_seg_dir`: the created path from the `segmentation_dir`. Is `None` if `segmentation_dir` is `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "file_path"
    ]
   },
   "outputs": [],
   "source": [
    "tiff_dir = os.path.join(base_dir, \"image_data\")\n",
    "img_sub_folder = None\n",
    "segmentation_dir = os.path.join(\"segmentation\", \"deepcell_output\")\n",
    "seg_suffix = '_whole_cell.tiff'\n",
    "\n",
    "if segmentation_dir is not None:\n",
    "    pixie_seg_dir = os.path.join(base_dir, segmentation_dir)\n",
    "else:\n",
    "    pixie_seg_dir = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `fovs` (optional): set a specific set of fovs to load, default loads all the fovs in `tiff_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "load_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# either get all fovs in the folder...\n",
    "fovs = io_utils.list_folders(tiff_dir)\n",
    "\n",
    "# ... or optionally, select a specific set of fovs manually\n",
    "# fovs = [\"fov14\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define multiprocessing parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turning on multiprocessing provides a speed boost; however, it is not always cross-platform compatible. If you receive errors such as hanging cells without progress updates, try setting `multiprocess` back to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "set_multi"
    ]
   },
   "outputs": [],
   "source": [
    "# set to True to turn on multiprocessing\n",
    "multiprocess = False\n",
    "\n",
    "# define the number of FOVs to process in parallel, ignored if multiprocessing is set to False\n",
    "batch_size = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2: Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a prefix to be applied to all data directories/files created during pixel clustering. If the prefix is not set, a default of the datetime at the start of the run is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_prefix"
    ]
   },
   "outputs": [],
   "source": [
    "# explicitly set pixel_cluster_prefix to override datetime default\n",
    "pixel_cluster_prefix = \"example\"\n",
    "\n",
    "if pixel_cluster_prefix is None:\n",
    "    pixel_cluster_prefix = dt.now().strftime('%Y-%m-%dT%H:%M:%S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following data directories/files will be created with names prefixed by `pixel_cluster_prefix`:\n",
    "\n",
    "* `pixel_output_dir`: directory name where the pixel clustering outputs are stored\n",
    "* `preprocessed_dir`: directory name where the preprocessed pixel data are stored\n",
    "* `subsetted_dir`: directory name where the subsetted pixel data are stored\n",
    "* `norm_vals_name`: file name where the values used to normalize each channel are stored"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "dir_set"
    ]
   },
   "outputs": [],
   "source": [
    "# define the output directory using the specified pixel cluster prefix\n",
    "pixel_output_dir = os.path.join(\"pixie\", \"%s_pixel_output_dir\" % pixel_cluster_prefix)\n",
    "if not os.path.exists(os.path.join(base_dir, pixel_output_dir)):\n",
    "    os.makedirs(os.path.join(base_dir, pixel_output_dir))\n",
    "\n",
    "# define the preprocessed pixel data folders\n",
    "pixel_data_dir = os.path.join(pixel_output_dir, 'pixel_mat_data')\n",
    "pixel_subset_dir = os.path.join(pixel_output_dir, 'pixel_mat_subset')\n",
    "norm_vals_name = os.path.join(pixel_output_dir, 'channel_norm_post_rownorm.feather')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For certain channels, such as membraneous tumor markers, the subcellular localization of the marker isn't important. Instead, what matters is that cells which are positive for the marker show up as positive. In these cases, we have sometimes found it useful to add additional blurring to these markers before clustering. This ensures that more of the pixels within the cell are positive for the marker, instead of only a few pixels at the border, especially for cells which are under-segmented. However, higher blur levels will also cause more of the pixels in neighboring cells to show up as positive. Therefore, this works best when you have other, robust markers (like CD45) which you can use to determine which cells are false positives for the blurred channel. If you have markers in your panel which fit this description, you can add them in the cell below. Then, when specifying the list of markers to include for clustering, make sure to add `{marker_name}_smoothed`, as that is what the TIFF will be called.\n",
    "\n",
    "Skip this cell if you don't want to add an additional blur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "smooth_channels"
    ]
   },
   "outputs": [],
   "source": [
    "# set an optional list of markers for additional blurring\n",
    "blurred_channels = [\"ECAD\"]\n",
    "smooth_vals = 6\n",
    "\n",
    "pixel_cluster_utils.smooth_channels(\n",
    "    fovs=fovs,\n",
    "    tiff_dir=tiff_dir,\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    channels=blurred_channels,\n",
    "    smooth_vals=smooth_vals,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, markers will have background staining that you'll want to filter out for the clustering process. Define the name of the marker you want to filter using `filter_channel`. If the marker is only present in the nucleus, set `exclude = False` to filter it out from the membrane. Conversely, if the marker is only present in the membrane, set `exclude = True` to filter it out from the nucleus.\n",
    "\n",
    "When specifying the list of markers to include for clustering, make sure to add `{marker_name}_nuc_exclude` or `{marker_name}_nuc_include` depending on what type of signal was filtered out.\n",
    "\n",
    "Skip this cell if you don't want to run marker filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "filter_channels"
    ]
   },
   "outputs": [],
   "source": [
    "filter_channel = 'CD163'\n",
    "nuclear_exclude = True\n",
    "\n",
    "pixel_cluster_utils.filter_with_nuclear_mask(\n",
    "    fovs=fovs,\n",
    "    tiff_dir=tiff_dir,\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    channel=filter_channel,\n",
    "    nuc_seg_suffix=\"_nuclear.tiff\",\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    exclude=nuclear_exclude\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the following arguments:\n",
    "\n",
    "* `channels`: channels to run pixel clustering on\n",
    "* `blur_factor`: sigma (standard deviation) for the Gaussian blur. Higher values are more aggressive in smoothing signal.\n",
    "* `subset_proportion`: the fraction of pixels to take from each FOV for training. Sampling is random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "channel_set"
    ]
   },
   "outputs": [],
   "source": [
    "channels = [\"CD3\", \"CD4\", \"CD8\", \"CD14\", \"CD20\", \"CD31\", \"CD45\", \"CD68\",\n",
    "            \"CD163_nuc_exclude\", \"CK17\", \"Collagen1\", \"Fibronectin\", \n",
    "            \"ECAD_smoothed\", \"HLADR\", \"SMA\", \"Vim\"]\n",
    "blur_factor = 2\n",
    "subset_proportion = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During pixel preprocessing, the following is done for each FOV:\n",
    "\n",
    "* Gaussian blur each channel separately\n",
    "* Remove empty pixels\n",
    "* For the remaining pixels, normalize each pixel by the sum of all the channels\n",
    "* Subset a `subset_proportion` fraction of non-empty, normalized pixels. This creates the subsetted dataset for training\n",
    "\n",
    "Note: if you get integer overflow errors loading in your data, try changing the `dtype` argument to a larger type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "gen_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# run pixel data preprocessing\n",
    "pixie_preprocessing.create_pixel_matrix(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    tiff_dir,\n",
    "    pixie_seg_dir,\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    seg_suffix=seg_suffix,\n",
    "    pixel_output_dir=pixel_output_dir,\n",
    "    data_dir=pixel_data_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name_post_rownorm=norm_vals_name,\n",
    "    blur_factor=blur_factor,\n",
    "    subset_proportion=subset_proportion,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Pixel clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: Train pixel SOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the pixel SOM using the subsetted data. Training is done using a self-organizing map (SOM).\n",
    "\n",
    "The following data directories/files will be created for pixel clustering:\n",
    "\n",
    "* `pixel_som_weights_name`: file name to store the pixel SOM weights\n",
    "* `pc_chan_avg_som_cluster_name`: file name to store the average channel expression across all pixel SOM clusters\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above for pixel meta clusters\n",
    "* `pixel_meta_cluster_remap_name`: file name to store the SOM cluster to meta cluster manual mappings created using the GUI below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_som_path_set"
    ]
   },
   "outputs": [],
   "source": [
    "pixel_som_weights_name = os.path.join(pixel_output_dir, 'pixel_som_weights.feather')\n",
    "pc_chan_avg_som_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_som_cluster.csv')\n",
    "pc_chan_avg_meta_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_meta_cluster.csv')\n",
    "pixel_meta_cluster_remap_name = os.path.join(pixel_output_dir, 'pixel_meta_cluster_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each channel is normalized by their 99.9% value across the entire dataset before training. These values get saved to `norm_vals_name`.\n",
    "\n",
    "For a full set of parameters you can customize for `train_pixel_som`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.train_pixel_som>pixel training docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "train_pixel_som"
    ]
   },
   "outputs": [],
   "source": [
    "# create the pixel SOM weights\n",
    "pixel_pysom = pixel_som_clustering.train_pixel_som(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name=norm_vals_name,\n",
    "    som_weights_name=pixel_som_weights_name,\n",
    "    num_passes=1,\n",
    "    seed=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: Assign pixel SOM clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the SOM weights learned from `train_pixel_som` to assign pixel clusters to the full preprocessed dataset.\n",
    "\n",
    "Note that each channel is normalized by the respective value stored in `norm_vals_name` (computed in `train_pixel_som`) prior to cluster assignment.\n",
    "\n",
    "`generate_som_avg_files` will compute the average channel expression across all pixel SOM clusters, as well as the number of pixels in each pixel SOM cluster (the data in `pc_chan_avg_som_cluster_name`). This is needed for consensus clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# use pixel SOM weights to assign pixel clusters\n",
    "pixel_som_clustering.cluster_pixels(\n",
    "    fovs,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the SOM cluster summary files\n",
    "pixel_som_clustering.generate_som_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: Run pixel consensus clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use consensus hierarchical clustering to cluster pixel SOM clusters into a user-defined number of meta clusters. The consensus clusters are trained on the average channel expression across all pixel SOM clusters (the data stored in `pc_chan_avg_som_cluster_name`). These values are z-scored and capped at the value specified in the `cap` argument prior to consensus clustering. This helps improve meta clustering performance.\n",
    "\n",
    "After consensus clustering, the following are computed by `generate_meta_avg_files`:\n",
    "\n",
    "* The average channel expression across all pixel meta clusters, and the number of pixels per meta cluster (the data in `pc_chan_avg_meta_cluster_name`)\n",
    "* The meta cluster mapping for each pixel SOM cluster in `pc_chan_avg_som_cluster_name` (data is resaved, same data except with an associated meta cluster column)\n",
    "\n",
    "For a full set of parameters you can customize for `pixel_consensus_cluster`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.pixel_consensus_cluster>pixel consensus clustering docs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `max_k`: the number of consensus clusters desired\n",
    "* `cap`: used to clip z-scored values prior to consensus clustering (in the range `[-cap, cap]`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_consensus_cluster"
    ]
   },
   "outputs": [],
   "source": [
    "max_k = 20\n",
    "cap = 3\n",
    "\n",
    "# run hierarchical clustering using average pixel SOM cluster expression\n",
    "pixel_cc = pixel_meta_clustering.pixel_consensus_cluster(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    max_k=max_k,\n",
    "    cap=cap,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the meta cluster summary files\n",
    "pixel_meta_clustering.generate_meta_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_cc,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name=pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: Interactive adjustments to relabel pixel meta clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization shows the z-scored average channel expression per pixel SOM and meta cluster. The heatmaps are faceted by pixel SOM clusters on the left and pixel meta clusters on the right.\n",
    "\n",
    "## Usage\n",
    "\n",
    "### Quickstart\n",
    "- **Select**: Left Click\n",
    "- **Remap**: **New metacluster** button or Right Click\n",
    "- **Edit Metacluster Name**: Textbox at bottom right of the heatmaps.\n",
    "\n",
    "### Selection and remapping details\n",
    "- To select a SOM cluster, click on its respective position in the **selected** bar. Click on it again to deselect.\n",
    "- To select a meta cluster, click on its corresponding color in the **metacluster** bar. Click on it again to deselect.\n",
    "- To remap the selected clusters, click the **New metacluster** button (alternatively, right click anywhere). Note that remapping an entire metacluster deletes it.\n",
    "- To clear the selected SOM/meta clusters, use the **Clear Selection** button.\n",
    "- **After remapping a meta cluster, make sure to deselect the newly created one to prevent unwanted combinations.**\n",
    "\n",
    "### Other features and notes\n",
    "- You will likely need to zoom out to see the entire visualization. To toggle Zoom, use Ctrl -/Ctrl + on Windows or ⌘ +/⌘ - on Mac.\n",
    "- The bars at the top show the number of pixels in each SOM cluster.\n",
    "- The text box at the bottom right allows you to rename a particular meta cluster. This can be useful as remapping may cause inconsistent numbering. **You cannot use the same name for different meta clusters; doing so will cause the next step to fail.**\n",
    "- Adjust the z-score limit using the slider on the bottom left to adjust your dynamic range.\n",
    "- When meta clusters are combined or a meta cluster is renamed, the change is immediately saved to `pixel_meta_cluster_remap_name`.\n",
    "- You won't be able to advance in the notebook until you've clicked `New metacluster` or renamed a meta cluster at least once. If you don't want to make changes, just click `New metacluster` to trigger a save before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_interactive"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib widget\n",
    "rc_file_defaults()\n",
    "plt.ion()\n",
    "\n",
    "pixel_mcd = metaclusterdata_from_files(\n",
    "    os.path.join(base_dir, pc_chan_avg_som_cluster_name),\n",
    "    cluster_type='pixel'\n",
    ")\n",
    "pixel_mcd.output_mapping_filename = os.path.join(base_dir, pixel_meta_cluster_remap_name)\n",
    "pixel_mcg = MetaClusterGui(pixel_mcd, width=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relabel the pixel meta clusters using the mapping, and recompute the meta cluster average files with the new meta cluster names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_apply_remap"
    ]
   },
   "outputs": [],
   "source": [
    "# rename the meta cluster values in the pixel dataset\n",
    "pixel_meta_clustering.apply_pixel_meta_cluster_remapping(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# recompute the mean channel expression per meta cluster and apply these new names to the SOM cluster average data\n",
    "pixel_meta_clustering.generate_remap_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the color scheme returned by the interactive reclustering process. This will be for visualizing the pixel phenotype maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_cmap_gen"
    ]
   },
   "outputs": [],
   "source": [
    "raw_cmap, _ = colormap_helper.generate_meta_cluster_colormap_dict(\n",
    "    pixel_mcd.output_mapping_filename,\n",
    "    pixel_mcg.im_cl.cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2: Generate pixel phenotype maps\n",
    "\n",
    "Generate pixel phenotype maps, in which each pixel in the image corresponds to its pixel meta cluster. Select a small subset of your FOVs to view within this notebook. Or if you wish to generate and save a significant amount of FOVs, the masks will be created and saved in batches.\n",
    "\n",
    "Files will be written as `{fov_name}_pixel_mask.tiff` in `pixel_output_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# select fovs to display\n",
    "subset_pixel_fovs = ['fov0', 'fov1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mask_gen_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the path to the channel file\n",
    "if img_sub_folder is None:\n",
    "    chan_file = os.path.join(\n",
    "        io_utils.list_files(os.path.join(tiff_dir, fovs[0]), substrs=['.tiff'])[0]\n",
    "    )\n",
    "else:\n",
    "    chan_file = os.path.join(\n",
    "        img_sub_folder, io_utils.list_files(os.path.join(tiff_dir, fovs[0], img_sub_folder), substrs=['.tiff'])[0]\n",
    "    )\n",
    "\n",
    "# generate and save the pixel cluster masks for each fov in subset_pixel_fovs\n",
    "data_utils.generate_and_save_pixel_cluster_masks(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    base_dir=base_dir,\n",
    "    save_dir=os.path.join(base_dir, pixel_output_dir),\n",
    "    tiff_dir=tiff_dir,\n",
    "    chan_file=chan_file,\n",
    "    pixel_data_dir=pixel_data_dir,\n",
    "    cluster_id_to_name_path=os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    pixel_cluster_col='pixel_meta_cluster',\n",
    "    sub_dir='pixel_masks',\n",
    "    name_suffix='_pixel_mask',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the colored pixel masks for each FOV in `subset_pixel_fovs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "save_pixel_masks"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.save_colored_masks(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    mask_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "    save_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_mask_colored\"),\n",
    "    cluster_id_to_name_path=os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    metacluster_colors=raw_cmap,\n",
    "    cluster_type=\"pixel\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a subset of the pixel cluster masks that you would like to preview. If the dimensions or text size of the plot need to be adjusted for optimal viewing, change the `figsize` and `dpi` parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_gen"
    ]
   },
   "outputs": [],
   "source": [
    "for pixel_fov in subset_pixel_fovs:\n",
    "    pixel_cluster_mask = load_utils.load_imgs_from_dir(\n",
    "        data_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "        files=[pixel_fov + \"_pixel_mask.tiff\"],\n",
    "        trim_suffix=\"_pixel_mask\",\n",
    "        match_substring=\"_pixel_mask\",\n",
    "        xr_dim_name=\"pixel_mask\",\n",
    "        xr_channel_names=None,\n",
    "    )\n",
    "\n",
    "    plot_utils.plot_pixel_cell_cluster(\n",
    "        pixel_cluster_mask,\n",
    "        [pixel_fov],\n",
    "        os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "        metacluster_colors=raw_cmap, \n",
    "        figsize=(5,5), \n",
    "        dpi=150\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5: Save parameters for use in cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following parameters are saved:\n",
    "\n",
    "* `fovs`: fovs in `pixel_data_dir`\n",
    "* `channels`: channels used for clustering\n",
    "* `segmentation_dir`: path to the directory containing your segmentated images for each FOV (can be generated using `1_Segment_Image_Data.ipynb`)\n",
    "* `seg_suffix`: suffix plus the file extension of the segmented images for each FOV\n",
    "* `pixel_data_dir`: name of the directory containing the full pixel data with the pixel SOM and meta cluster assignments\n",
    "* `pc_chan_avg_som_cluster_name`: name of the file containing the average channel expression per pixel SOM cluster\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above for pixel meta clusters\n",
    "\n",
    "The file will be saved to `{pixel_cluster_prefix}_cell_clustering_params.json` and will be placed in `pixel_output_dir`. Note that the `pixel_output_dir` you use in `2_Pixie_Cluster_Pixels.ipynb` should be the same as in `3_Pixie_Cluster_Cells.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_param_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the params dict\n",
    "cell_clustering_params = {\n",
    "    'fovs': io_utils.remove_file_extensions(io_utils.list_files(os.path.join(base_dir, pixel_data_dir), substrs='.feather')),\n",
    "    'channels': channels,\n",
    "    'tiff_dir': tiff_dir,\n",
    "    'img_sub_folder': img_sub_folder,\n",
    "    'segmentation_dir': segmentation_dir,\n",
    "    'seg_suffix': seg_suffix,\n",
    "    'pixel_data_dir': pixel_data_dir,\n",
    "    'pc_chan_avg_som_cluster_name': pc_chan_avg_som_cluster_name,\n",
    "    'pc_chan_avg_meta_cluster_name': pc_chan_avg_meta_cluster_name\n",
    "}\n",
    "\n",
    "# save the params dict\n",
    "with open(os.path.join(base_dir, pixel_output_dir, 'cell_clustering_params.json'), 'w') as fh:\n",
    "    json.dump(cell_clustering_params, fh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Save images for Mantis Viewer\n",
    "\n",
    "Mantis Viewer is a visualization tool for multi-dimensional imaging in pathology. Learn more about Mantis Viewer in the [README](../README.md#mantis-viewer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.create_mantis_dir(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    mantis_project_path=os.path.join(base_dir, \"mantis\"),\n",
    "    img_data_path=tiff_dir,\n",
    "    mask_output_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "    mapping = os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    seg_dir=pixie_seg_dir,\n",
    "    mask_suffix=\"_pixel_mask\",\n",
    "    seg_suffix_name=seg_suffix,\n",
    "    img_sub_folder=img_sub_folder\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "key": "version",
       "op": "remove"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "diff": [
          {
           "key": 5,
           "op": "addrange",
           "valuelist": "5"
          },
          {
           "key": 5,
           "length": 1,
           "op": "removerange"
          }
         ],
         "key": 0,
         "op": "patch"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ]
  },
  "toc-autonumbering": false,
  "toc-showcode": true,
  "toc-showmarkdowntxt": true,
  "vscode": {
   "interpreter": {
    "hash": "b0ac63cc785e7c4cb24da0642dbfe09cfb2d314979108d7192228f9c45902a86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}