templates/3_Pixie_Cluster_Cells.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Pixie: cell clustering notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: this notebook should be run after `2_Pixie_Cluster_Pixels.ipynb`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "import"
    ]
   },
   "outputs": [],
   "source": [
    "# import required packages\n",
    "import json\n",
    "import os\n",
    "from datetime import datetime as dt\n",
    "\n",
    "import feather\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from alpineer import load_utils\n",
    "from matplotlib import rc_file_defaults\n",
    "\n",
    "from ark.phenotyping import (cell_cluster_utils, cell_meta_clustering,\n",
    "                             cell_som_clustering, weighted_channel_comp)\n",
    "from ark.utils import data_utils, example_dataset, plot_utils\n",
    "from ark.utils.metacluster_remap_gui import (MetaClusterGui,\n",
    "                                             colormap_helper,\n",
    "                                             metaclusterdata_from_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 0: Set root directory and download the example dataset\n",
    "\n",
    "Here we are using the example data located in `data/example_dataset`. To use your own data, change `base_dir` to point to your own sub-directory within the data folder.\n",
    "\n",
    "* `base_dir`: the path to all of your imaging data. Should contain a directory for your images, segmentations, and cell table (which can be generated from `1_Segment_Image_Data.ipynb`). This directory will also store all of the directories/files created during cell clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "base_dir"
    ]
   },
   "outputs": [],
   "source": [
    "# define the home directory (should contain pixel_output_dir from pixel clustering notebook)\n",
    "base_dir = \"../data/example_dataset/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test the features in ark with an example dataset, run the cell below. It will download a dataset consisting of 11 FOVs with 22 channels. You may find more information about the example dataset in the [README](../README.md#example-dataset).\n",
    "\n",
    "If you are using your own data, skip the cell below.\n",
    "\n",
    "* `overwrite_existing`: If set to `False`, it will not overwrite existing data in `data/example_dataset`. Recommended setting to `False` if you are running Notebooks 1,2,3 and 4 in succession. Set to `True` if you are just running Notebook 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "ex_data_download"
    ]
   },
   "outputs": [],
   "source": [
    "example_dataset.get_example_dataset(dataset=\"cluster_cells\", save_dir = base_dir, overwrite_existing = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1: Load parameters for cell clustering (saved during `2_Pixie_Cluster_Pixels.ipynb`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cell_clustering_params_name` should be `cell_clustering_params.json` contained in `{pixel_cluster_prefix}_pixel_output_dir`. Make sure to set `base_dir` and `pixel_output_dir` to the same value used in `2_Pixie_Cluster_Pixels.ipynb`.\n",
    "\n",
    "NOTE: `{pixel_cluster_prefix}` is set in `2_Pixie_Cluster_Pixels.ipynb`. If you did not explicity set a `{pixel_cluster_prefix}` in `2_Pixie_Cluster_Pixels.ipynb`, the prefix defaults to the timestamp of the run. Please check the run directory (`base_dir` as set in `2_Pixie_Cluster_Pixels.ipynb`) to see the prefix that was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "dir_set"
    ]
   },
   "outputs": [],
   "source": [
    "# define the name of the folder containing the pixel cluster data\n",
    "pixel_output_dir = 'example_pixel_output_dir'\n",
    "\n",
    "# define the name of the cell clustering params file\n",
    "cell_clustering_params_name = 'cell_clustering_params.json'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following params are loaded:\n",
    "\n",
    "* `fovs`: subset of fovs used for pixel clustering\n",
    "* `channels`: subset of channels used for pixel clustering\n",
    "* `tiff_dir`: path to the directory containing your imaging data. Images should be single-channel TIFFs.\n",
    "* `img_sub_folder`: if `tiff_dir` contains an additional subfolder structure, set to the appropriate folder name (or `None` if no such structure present)\n",
    "* `segmentation_dir`: path to the directory containing your segmented images (can be generated from `1_Segment_Image_Data.ipynb`)\n",
    "* `seg_suffix`: suffix plus the file extension of the segmented images for each FOV\n",
    "* `pixel_data_dir`: name of the directory containing pixel data with the pixel SOM and meta cluster assignments\n",
    "* `pc_chan_avg_som_cluster_name`: name of the file containing the average channel expression per pixel SOM cluster, used for the visualization of weighted channel average per cell\n",
    "* `pc_chan_avg_meta_cluster_name`: name of the file containing the average channel expression per pixel meta cluster, used for the visualization of weighted channel average per cell\n",
    "\n",
    "Additionally, define the following parameter:\n",
    "\n",
    "* `cell_table_path`: path to the cell table where each row in the table is one cell, must contain `fov`, `label`, and `cell_size` columns. Can be created by `1_Segment_Image_Data.ipynb`, should be placed in `segmentation_dir` by default. You can use either the normalized or arcsinh versions (only columns that are used are `fov`, `label`, and `cell_size`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "param_load"
    ]
   },
   "outputs": [],
   "source": [
    "# load the params\n",
    "with open(os.path.join(base_dir, \"pixie\", pixel_output_dir, cell_clustering_params_name)) as fh:\n",
    "    cell_clustering_params = json.load(fh)\n",
    "    \n",
    "# assign the params to variables\n",
    "fovs = cell_clustering_params['fovs']\n",
    "channels = cell_clustering_params['channels']\n",
    "tiff_dir = cell_clustering_params['tiff_dir']\n",
    "img_sub_folder = cell_clustering_params['img_sub_folder']\n",
    "segmentation_dir = cell_clustering_params['segmentation_dir']\n",
    "seg_suffix = cell_clustering_params['seg_suffix']\n",
    "pixel_data_dir = cell_clustering_params['pixel_data_dir']\n",
    "pc_chan_avg_som_cluster_name = cell_clustering_params['pc_chan_avg_som_cluster_name']\n",
    "pc_chan_avg_meta_cluster_name = cell_clustering_params['pc_chan_avg_meta_cluster_name']\n",
    "\n",
    "# define the cell table path\n",
    "cell_table_path = os.path.join(base_dir, 'segmentation', 'cell_table', 'cell_table_size_normalized.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a prefix to be applied to all data directories/files created during cell clustering. If the prefix is not set, a default of the datetime at the start of the run is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_prefix"
    ]
   },
   "outputs": [],
   "source": [
    "# explicitly set cell_cluster_prefix to override datetime default\n",
    "cell_cluster_prefix = \"example\"\n",
    "\n",
    "if cell_cluster_prefix is None:\n",
    "    cell_cluster_prefix = dt.now().strftime('%Y-%m-%dT%H:%M:%S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following folders/files will be created with names prefixed by `cell_cluster_prefix`:\n",
    "\n",
    "* `cell_output_dir`: directory name where the cell clustering outputs are stored\n",
    "* `cell_som_weights_name`: file name to store the cell SOM weights\n",
    "* `cluster_counts_name`: file name to store the counts of each pixel cluster per cell\n",
    "* `cluster_counts_size_norm_name`: same as above, except with each value normalized by the respective cell's size. The data will also contain the cell SOM and meta cluster labels assigned.\n",
    "* `weighted_cell_channel_name`: file name to store the weighted cell channel expression for each cell. Refer to <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.compute_p2c_weighted_channel_avg>cell channel weighting docs</a> for how the weighting is computed.\n",
    "* `cell_som_cluster_count_avg_name`: file name to store the average number of pixel clusters per cell SOM cluster\n",
    "* `cell_meta_cluster_count_avg_name`: same as above for cell meta clusters\n",
    "* `cell_som_cluster_channel_avg_name`: file name to store the average weighted channel expression per cell SOM cluster\n",
    "* `cell_meta_cluster_channel_avg_name`: same as above for cell meta clusters\n",
    "* `cell_meta_cluster_remap_name`: file name to store the SOM cluster to meta cluster manual mappings created using the GUI below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_cluster_files"
    ]
   },
   "outputs": [],
   "source": [
    "# define the base output cell folder\n",
    "cell_output_dir = '%s_cell_output_dir' % cell_cluster_prefix\n",
    "if not os.path.exists(os.path.join(base_dir, \"pixie\", cell_output_dir)):\n",
    "    os.mkdir(os.path.join(base_dir, \"pixie\", cell_output_dir))\n",
    "    \n",
    "# define the paths to cell clustering files, explicitly set the variables to use custom names\n",
    "cell_som_weights_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_weights.feather')\n",
    "cluster_counts_name = os.path.join(\"pixie\", cell_output_dir, 'cluster_counts.feather')\n",
    "cluster_counts_size_norm_name = os.path.join(\"pixie\", cell_output_dir, 'cluster_counts_size_norm.feather')\n",
    "weighted_cell_channel_name = os.path.join(\"pixie\", cell_output_dir, 'weighted_cell_channel.feather')\n",
    "cell_som_cluster_count_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_cluster_count_avg.csv')\n",
    "cell_meta_cluster_count_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_count_avg.csv')\n",
    "cell_som_cluster_channel_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_cluster_channel_avg.csv')\n",
    "cell_meta_cluster_channel_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_channel_avg.csv')\n",
    "cell_meta_cluster_remap_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, define the following variable:\n",
    "\n",
    "* `pixel_cluster_col`: whether to use the pixel SOM or meta cluster counts for cell clustering. Must be either `'pixel_som_cluster'` or `'pixel_meta_cluster_rename'`. Note that if you did not explicitly rename your pixel meta clusters in `2_Pixie_Cluster_Pixels.ipynb`, the default numeric names will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_cluster_col"
    ]
   },
   "outputs": [],
   "source": [
    "# define the type of pixel cluster to aggregate on\n",
    "pixel_cluster_col = 'pixel_meta_cluster_rename'\n",
    "\n",
    "# depending on which pixel_cluster_col is selected, choose the pixel channel average table accordingly\n",
    "if pixel_cluster_col == 'pixel_som_cluster':\n",
    "    pc_chan_avg_name = pc_chan_avg_som_cluster_name\n",
    "elif pixel_cluster_col == 'pixel_meta_cluster_rename':\n",
    "    pc_chan_avg_name = pc_chan_avg_meta_cluster_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the input data for the cell SOM. This computes the counts of each pixel cluster per cell, both raw and normalized by cell size.\n",
    "\n",
    "If both datasets already exist, load them in instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "generate_som_input_data"
    ]
   },
   "outputs": [],
   "source": [
    "if os.path.exists(os.path.join(base_dir, cluster_counts_name)) and os.path.exists(os.path.join(base_dir, cluster_counts_size_norm_name)):\n",
    "    # load the data if it exists\n",
    "    cluster_counts = feather.read_dataframe(os.path.join(base_dir, cluster_counts_name))\n",
    "    cluster_counts_size_norm = feather.read_dataframe(os.path.join(base_dir, cluster_counts_size_norm_name))\n",
    "else:\n",
    "    # generate the preprocessed data \n",
    "    cluster_counts, cluster_counts_size_norm = cell_cluster_utils.create_c2pc_data(\n",
    "        fovs, os.path.join(base_dir, pixel_data_dir), cell_table_path, pixel_cluster_col\n",
    "    )\n",
    "\n",
    "    # write both unnormalized and normalized input data for reference\n",
    "    feather.write_dataframe(\n",
    "        cluster_counts,\n",
    "        os.path.join(base_dir, cluster_counts_name),\n",
    "        compression='uncompressed'\n",
    "    )\n",
    "    feather.write_dataframe(\n",
    "        cluster_counts_size_norm,\n",
    "        os.path.join(base_dir, cluster_counts_size_norm_name),\n",
    "        compression='uncompressed'\n",
    "    )\n",
    "    \n",
    "# define the count columns found in cluster_counts_norm\n",
    "cell_som_cluster_cols = cluster_counts_size_norm.filter(\n",
    "    regex=f'{pixel_cluster_col}.*'\n",
    ").columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the weighted cell channel expression file. This data will be needed to compute the weighted average channel expression per cell cluster (the data stored in `cell_som_cluster_channel_avg_name` and `cell_meta_cluster_channel_avg_name`). See documentation of `compute_p2c_weighted_channel_avg` for how weighted cell channel average is computed: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.compute_p2c_weighted_channel_avg>cell channel weighting docs</a>.\n",
    "\n",
    "If this file already exists, skip this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "generate_weighted_channel_data"
    ]
   },
   "outputs": [],
   "source": [
    "# depending on which pixel_cluster_col is selected, choose the pixel channel average table accordingly\n",
    "if pixel_cluster_col == 'pixel_som_cluster':\n",
    "    pc_chan_avg_name = pc_chan_avg_som_cluster_name\n",
    "elif pixel_cluster_col == 'pixel_meta_cluster_rename':\n",
    "    pc_chan_avg_name = pc_chan_avg_meta_cluster_name\n",
    "\n",
    "if not os.path.exists(os.path.join(base_dir, weighted_cell_channel_name)):\n",
    "    # generate the weighted cell channel expression data\n",
    "    pixel_channel_avg = pd.read_csv(os.path.join(base_dir, pc_chan_avg_name))\n",
    "    weighted_cell_channel = weighted_channel_comp.compute_p2c_weighted_channel_avg(\n",
    "        pixel_channel_avg,\n",
    "        channels,\n",
    "        cluster_counts,\n",
    "        fovs=fovs,\n",
    "        pixel_cluster_col=pixel_cluster_col\n",
    "    )\n",
    "\n",
    "    # write the data to weighted_cell_channel_name\n",
    "    feather.write_dataframe(\n",
    "        weighted_cell_channel,\n",
    "        os.path.join(base_dir, weighted_cell_channel_name),\n",
    "        compression='uncompressed'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: Train cell SOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the cell SOM on the size-normalized number of pixel clusters per cell (the data stored in `cluster_counts_size_norm_name`). Training is done using the self-organizing map (SOM) algorithm. Note that each of the pixel SOM/meta cluster columns are normalized by their 99.9% value prior to training.\n",
    "\n",
    "For a full set of parameters you can customize for `train_cell_som`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.train_cell_som>cell training docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "train_cell_som"
    ]
   },
   "outputs": [],
   "source": [
    "# create the cell SOM weights\n",
    "cell_pysom = cell_som_clustering.train_cell_som(\n",
    "    fovs,\n",
    "    base_dir,\n",
    "    cell_table_path=cell_table_path,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    som_weights_name=cell_som_weights_name,\n",
    "    num_passes=1,\n",
    "    seed=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: Assign cell SOM clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the weights learned from `train_cell_som` to assign cell clusters to the dataset. Note that this is done on the size-normalized pixel cluster counts table. As with `train_pixel_som`, each of the columns are normalized by their 99.9% value prior to assigning a cell SOM cluster label.\n",
    "\n",
    "`generate_som_avg_files` will compute the average number of pixel clusters per cell SOM cluster, as well as the number of cells in each cell SOM cluster (the data placed in `cell_som_cluster_count_avg_name`). This is needed for cell consensus clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_cell_data"
    ]
   },
   "outputs": [],
   "source": [
    "# use cell SOM weights to assign cell clusters\n",
    "cluster_counts_size_norm = cell_som_clustering.cluster_cells(\n",
    "    base_dir,\n",
    "    cell_pysom,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols\n",
    ")\n",
    "\n",
    "# intermediate saving of cell data with SOM labels assigned\n",
    "feather.write_dataframe(\n",
    "    cluster_counts_size_norm,\n",
    "    os.path.join(base_dir, cluster_counts_size_norm_name),\n",
    "    compression='uncompressed'\n",
    ")\n",
    "\n",
    "# generate the SOM cluster summary files\n",
    "cell_som_clustering.generate_som_avg_files(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: Run cell consensus clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use consensus hierarchical clustering to cluster cell SOM clusters into a user-defined number of meta clusters. The consensus clusters are trained on the average number of pixel clusters across all cell SOM clusters (the data stored in `cell_som_cluster_count_avg_name`). These values are z-scored and capped at the value specified in the `cap` argument prior to consensus clustering. This helps improve meta clustering performance.\n",
    "\n",
    "After consensus clustering, the following are computed by `generate_meta_avg_files`:\n",
    "\n",
    "* The average number of pixel clusters across all cell meta clusters, and the number of cells per meta cluster (the data placed in `cell_meta_cluster_count_avg_name`)\n",
    "* The meta cluster mapping for each cell SOM cluster in `cell_som_cluster_count_avg_name` (data is resaved, same data except with an associated meta cluster column)\n",
    "\n",
    "`generate_wc_avg_files` also creates the following:\n",
    "\n",
    "* The weighted channel average across all cell clusters (the data placed in `cell_som_cluster_channel_avg_name` and `cell_meta_cluster_channel_avg_name`). This will be done for both `'cell_som_cluster'` and `'cell_meta_cluster'`.\n",
    "\n",
    "For a full set of parameters you can customize for `cell_consensus_cluster`, please consult <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.cell_consensus_cluster>cell consensus clustering docs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `max_k`: the number of consensus clusters desired\n",
    "* `cap`: used to clip z-scored values prior to consensus clustering (in the range `[-cap, cap]`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_consensus_cluster",
     "cell_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "max_k = 20\n",
    "cap = 3\n",
    "\n",
    "# run hierarchical clustering using average count of pixel clusters per cell SOM cluster\n",
    "cell_cc, cluster_counts_size_norm = cell_meta_clustering.cell_consensus_cluster(\n",
    "    base_dir,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name,\n",
    "    max_k=max_k,\n",
    "    cap=cap\n",
    ")\n",
    "\n",
    "# intermediate saving of cell data with SOM and meta labels assigned\n",
    "feather.write_dataframe(\n",
    "    cluster_counts_size_norm,\n",
    "    os.path.join(base_dir, cluster_counts_size_norm_name),\n",
    "    compression='uncompressed'\n",
    ")\n",
    "\n",
    "# generate the meta cluster summary files\n",
    "cell_meta_clustering.generate_meta_avg_files(\n",
    "    base_dir,\n",
    "    cell_cc,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name,\n",
    "    cell_meta_expr_col_avg_name=cell_meta_cluster_count_avg_name\n",
    ")\n",
    "\n",
    "# generate weighted channel summary files\n",
    "weighted_channel_comp.generate_wc_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    cell_cc,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    weighted_cell_channel_name=weighted_cell_channel_name,\n",
    "    cell_som_cluster_channel_avg_name=cell_som_cluster_channel_avg_name,\n",
    "    cell_meta_cluster_channel_avg_name=cell_meta_cluster_channel_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: Interactive adjustments to relabel cell meta clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization shows the z-scored average pixel cluster count expression per cell SOM and meta cluster. The heatmaps are faceted by cell SOM clusters on the left and cell meta clusters on the right.\n",
    "\n",
    "## Usage\n",
    "\n",
    "### Quickstart\n",
    "- **Select**: Left Click\n",
    "- **Remap**: **New metacluster button** or Right Click\n",
    "- **Edit Metacluster Name**: Textbox at bottom right of the heatmaps.\n",
    "\n",
    "### Selection and Remapping details\n",
    "- To select a SOM cluster, click on its respective position in the **selected** bar. Click on it again to deselect.\n",
    "- To select a meta cluster, click on its corresponding color in the **metacluster** bar. Click on it again to deselect.\n",
    "- To remap the selected clusters, click the **New metacluster** button (alternatively, right click anywhere). Note that remapping an entire metacluster deletes it.\n",
    "- To clear the selected SOM/meta clusters, use the **Clear Selection** button.\n",
    "- **After remapping a meta cluster, make sure to deselect the newly created one to prevent unwanted combinations.**\n",
    "\n",
    "### Other features and notes\n",
    "- You will likely need to zoom out to see the entire visualization. To toggle Zoom, use Ctrl -/Ctrl + on Windows or ⌘ +/⌘ - on Mac.\n",
    "- The bars at the top show the number of cells in each SOM cluster.\n",
    "- The text box at the bottom right allows you to rename a particular meta cluster. This can be useful as remapping may cause inconsistent numbering. **You cannot use the same name for different meta clusters; doing so will cause the next step to fail.**\n",
    "- Adjust the z-score limit using the slider on the bottom left to adjust your dynamic range.\n",
    "- When meta clusters are combined or a meta cluster is renamed, the change is immediately saved to `cell_meta_cluster_remap_name`.\n",
    "- You won't be able to advance until you've clicked `New metacluster` or renamed a meta cluster at least once. If you do not want to make changes, just click `New metacluster` to trigger a save before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_interactive"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib widget\n",
    "rc_file_defaults()\n",
    "plt.ion()\n",
    "\n",
    "cell_mcd = metaclusterdata_from_files(\n",
    "    os.path.join(base_dir, cell_som_cluster_count_avg_name),\n",
    "    cluster_type='cell',\n",
    "    prefix_trim=pixel_cluster_col + '_'\n",
    ")\n",
    "cell_mcd.output_mapping_filename = os.path.join(base_dir, cell_meta_cluster_remap_name)\n",
    "cell_mcg = MetaClusterGui(cell_mcd, width=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relabel the cell meta clusters using the mapping, and recompute the meta cluster average files with the new meta cluster names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_apply_remap"
    ]
   },
   "outputs": [],
   "source": [
    "# rename the meta cluster values in the cell dataset\n",
    "cluster_counts_size_norm = cell_meta_clustering.apply_cell_meta_cluster_remapping(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name\n",
    ")\n",
    "\n",
    "# intermediate saving of cell data with SOM, raw meta, and renamed meta labels assigned\n",
    "feather.write_dataframe(\n",
    "    cluster_counts_size_norm,\n",
    "    os.path.join(base_dir, cluster_counts_size_norm_name),\n",
    "    compression='uncompressed'\n",
    ")\n",
    "\n",
    "# recompute the mean column expression per meta cluster and apply these new names to the SOM cluster average data\n",
    "cell_meta_clustering.generate_remap_avg_count_files(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name,\n",
    "    cell_som_cluster_cols,\n",
    "    cell_som_cluster_count_avg_name,\n",
    "    cell_meta_cluster_count_avg_name,\n",
    ")\n",
    "\n",
    "# recompute the mean weighted channel expression per meta cluster and apply these new names to the SOM channel average data\n",
    "weighted_channel_comp.generate_remap_avg_wc_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name,\n",
    "    weighted_cell_channel_name,\n",
    "    cell_som_cluster_channel_avg_name,\n",
    "    cell_meta_cluster_channel_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the color scheme returned by the interactive reclustering process. This will be for visualizing the weighted channel average heatmaps and the cell phenotype maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_cmap_gen"
    ]
   },
   "outputs": [],
   "source": [
    "raw_cmap, renamed_cmap = colormap_helper.generate_meta_cluster_colormap_dict(\n",
    "    cell_mcd.output_mapping_filename,\n",
    "    cell_mcg.im_cl.cmap,\n",
    "    cluster_type='cell'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2: Weighted cell SOM cluster average heatmap over channels (z-scored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_som_heatmap"
    ]
   },
   "outputs": [],
   "source": [
    "weighted_channel_comp.generate_weighted_channel_avg_heatmap(\n",
    "    os.path.join(base_dir, cell_som_cluster_channel_avg_name),\n",
    "    'cell_som_cluster',\n",
    "    channels,\n",
    "    raw_cmap,\n",
    "    renamed_cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3: Weighted cell meta cluster average heatmap over channels (z-scored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_meta_heatmap"
    ]
   },
   "outputs": [],
   "source": [
    "weighted_channel_comp.generate_weighted_channel_avg_heatmap(\n",
    "    os.path.join(base_dir, cell_meta_cluster_channel_avg_name),\n",
    "    'cell_meta_cluster_rename',\n",
    "    channels,\n",
    "    raw_cmap,\n",
    "    renamed_cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4: Generate cell phenotype maps\n",
    "\n",
    "Generate cell phenotype maps, in which each pixel in the image corresponds to its cell meta cluster. Run this cell if you wish to create cell cluster mask images for downstream analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_overlay_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# select fovs to display\n",
    "subset_cell_fovs = ['fov0', 'fov1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_mask_gen_save"
    ]
   },
   "outputs": [],
   "source": [
    "# generate and save the cell cluster masks for each fov in subset_cell_fovs\n",
    "data_utils.generate_and_save_cell_cluster_masks(\n",
    "    fovs=subset_cell_fovs,\n",
    "    save_dir=os.path.join(base_dir, \"pixie\", cell_output_dir),\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    cell_data=cluster_counts_size_norm,\n",
    "    cluster_id_to_name_path=os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "    seg_suffix=seg_suffix,\n",
    "    sub_dir='cell_masks',\n",
    "    name_suffix='_cell_mask'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the colored cell masks for each FOV in `subset_cell_fovs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "save_cell_masks"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.save_colored_masks(\n",
    "    fovs=subset_cell_fovs,\n",
    "    mask_dir=os.path.join(base_dir, \"pixie\",cell_output_dir, \"cell_masks\"),\n",
    "    save_dir=os.path.join(base_dir, \"pixie\",cell_output_dir, \"cell_mask_colored\"),\n",
    "    cluster_id_to_name_path=os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "    metacluster_colors=raw_cmap,\n",
    "    cluster_type=\"cell\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a subset of the cell cluster masks that you would like to preview. If the dimensions or text size of the plot need to be adjusted for optimal viewing, change the `figsize` and `dpi` parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_overlay_gen"
    ]
   },
   "outputs": [],
   "source": [
    "for cell_fov in subset_cell_fovs:\n",
    "    cell_cluster_mask = load_utils.load_imgs_from_dir(\n",
    "        data_dir = os.path.join(base_dir, \"pixie\", cell_output_dir, \"cell_masks\"),\n",
    "        files=[cell_fov + \"_cell_mask.tiff\"],\n",
    "        trim_suffix=\"_cell_mask\",\n",
    "        match_substring=\"_cell_mask\",\n",
    "        xr_dim_name=\"cell_mask\",\n",
    "        xr_channel_names=None,\n",
    "    )\n",
    "\n",
    "    plot_utils.plot_pixel_cell_cluster(\n",
    "        cell_cluster_mask,\n",
    "        [cell_fov],\n",
    "        os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "        metacluster_colors=raw_cmap,\n",
    "        cluster_type='cell',\n",
    "        erode=True, \n",
    "        figsize=(5,5),\n",
    "        dpi=150\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5: Append consensus cluster labels to cell table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell table with cell meta cluster labels is saved to `{cell_table_path}_cell_labels.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_append_meta"
    ]
   },
   "outputs": [],
   "source": [
    "cell_cluster_utils.add_consensus_labels_cell_table(\n",
    "    base_dir, cell_table_path, cluster_counts_size_norm\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6: Save images for Mantis Viewer\n",
    "\n",
    "Mantis Viewer is a visualization tool for multi-dimensional imaging in pathology. Learn more about Mantis Viewer in the [README](../README.md#mantis-viewer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.create_mantis_dir(\n",
    "    fovs=subset_cell_fovs,\n",
    "    mantis_project_path=os.path.join(base_dir, \"mantis\"),\n",
    "    img_data_path=tiff_dir,\n",
    "    mask_output_dir=os.path.join(base_dir, \"pixie\", cell_output_dir, \"cell_masks\"),\n",
    "    mapping = os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    cluster_type='cell',\n",
    "    mask_suffix=\"_cell_mask\",\n",
    "    seg_suffix_name=seg_suffix,\n",
    "    img_sub_folder=img_sub_folder\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "key": "version",
       "op": "remove"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "3.9.16"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ]
  },
  "vscode": {
   "interpreter": {
    "hash": "b0ac63cc785e7c4cb24da0642dbfe09cfb2d314979108d7192228f9c45902a86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}