monailabel/monailabel_wholebody_totalSegmentator_3dslicer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License. \n",
    "\n",
    "# MONAI Label - Whole Body CT TotalSegmentator\n",
    "\n",
    "***The 104 Anatomies Segmentation Workflow in MONAI Label***\n",
    "\n",
    "In this notebook, we show the MONAI Label model with the bundle app for whole body CT segmentation. The model is trained with the totalSegmentator dataset [1].\n",
    "\n",
    "MONAI Label + 3D Slicer integration is used for model training and inference. Whole body CT segmentation is a fundamental task for many downstream workflows. In MONAI Label, we provide a single model for inferencing 104 classes of segments, the model we developed used SegResNet [2] network.\n",
    "\n",
    "![workflow](./figures/monailabel_wholebody_totalSegmentator_3dslicer/totalsegmentator_monailabel.png)\n",
    "\n",
    "**Whole Body CT TotalSegmentator Bundle:**\n",
    "\n",
    "This tutorial uses the whole body ct segmentation bundle. The bundle provides two versions of models, a high-resolution model trained with (1.5 x 1.5 x 1.5)mm images, the other one is the low-resolution model trained with (3.0 x 3.0 x 3.0)mm images. MONAI Label provides functions for visualizing config options, users can select which model to use in the 3D Slicer MONAI Label plugin.\n",
    "\n",
    "![workflow](./figures/monailabel_wholebody_totalSegmentator_3dslicer/overview_classes.png)\n",
    "\n",
    "Figure source from the TotalSegmentator [1]."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "\n",
    "MONAI Label is pre-built within the MONAI Toolkit. Download monaibundle app\n",
    "\n",
    "### Prerequisites\n",
    "- **Install MONAI Label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install monailabel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download monaibundle app\n",
    "Sample-app with built-in versions are in the Toolkit, pull the monaibundle app using monailabel API "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel apps --download --name monaibundle --output apps"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download sample data\n",
    "\n",
    "The training set is the 104 whole-body structures from the TotalSegmentator released datasets. Users can find more details on the datasets at https://github.com/wasserth/TotalSegmentator. All rights and licenses are reserved for the original authors.\n",
    "\n",
    "- Target: 104 structures\n",
    "- Modality: CT\n",
    "- Source: TotalSegmentator\n",
    "- Challenge: Large volumes of structures in CT images\n",
    "\n",
    "Disclaimer: We are not the host of the data. Please make sure to read the requirements and usage policies of the data and give credit to the authors of the dataset! We acknowledge the critical role in the creation of the free publicly available TotalSegmentator Database used in this study.\n",
    "\n",
    " #### Preprocessing\n",
    "\n",
    " - If you are going to use full dataset of TotalSegmentator, please refer to the dataset link, download the data, create and preprocess the images following [this page](https://zenodo.org/record/6802614).\n",
    " \n",
    " - In this tutorial, we prepared a sample subset, resampled and ready to use. The subset is only for demonstration. Download [here](https://developer.download.nvidia.com/assets/Clara/monai/tutorials/totalSegmentator_mergedLabel_samples.zip).\n",
    "  \n",
    " To use the bundle, users need to download the data and merge all annotated labels into one NIFTI file. Each file contains 0-104 values, each value represents one anatomy class.\n",
    " \n",
    " #### Abdominal CT\n",
    " \n",
    " - In this tutorial, in order to demo the segmentation results and MONAI Label functions, we provide several automated commands with the MSD Task09 abdominal CT scans. The task is the volumetric (3D) segmentation of the spleen from CT image. The dataset is from the 2018 MICCAI challenge. \n",
    " - Users can also use the original TotalSegmentator sample dataset provided above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel datasets --download --name Task09_Spleen --output datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Starting MONAI Label Server\n",
    "\n",
    "Specify the bundle name in `--conf models <BUNDLENAME>` argument. For example: wholeBody_ct_segmentation\n",
    "\n",
    "**Note:** If you are running a MONAI Label server in this notebook and plan to close the notebook, ensure that you terminate the kernel first. Failing to do so might result in an issue where the port remains occupied from the previous session when you try to restart the server later. To prevent this, always terminate the kernel before closing the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel start_server --app apps/monaibundle --studies datasets/Task09_Spleen/imagesTs --conf models wholebody_ct_segmentation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Auto Segmentation with 3D Slicer\n",
    "\n",
    "Below is a step-by-step tutorial on using 3D Slicer for auto segmentation. Pre-trained model is provided by the bundle and loaded by MONAI Label server.\n",
    "\n",
    "\n",
    "**Start 3D Slicer and MONAI Label Plugin**: the tutorial starts with monaibundle app in the 3D Slicer, please refer to the [\"Hello World\" tutorial](monailabel_HelloWorld_radiology_3dslicer.ipynb) for detailed instructions on installing 3D Slicer and MONAI Label plugin.\n",
    "\n",
    "### 2.1 Fetch Data\n",
    "\n",
    "- Select the bundle model loaded with MONAI Label server, **wholeBody_ct_segmentation** in this tutorial. Start MONAI Label server will automatically download the bundle.\n",
    "\n",
    "\n",
    "click **next sample** to load image data from datastore.\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody1.png)\n",
    "\n",
    "### 2.2 High-Resolution and Low-Resolution Models\n",
    "\n",
    "We retrained two versions of the totalSegmentator models, following the original paper and implementation. To meet multiple demands according to computation resources and performance, we provide a 1.5 mm model and a 3.0 mm model, both models are trained with 104 foreground output channels.\n",
    "\n",
    "In this bundle, we configured a parameter called `highres`, users can set it to true when using 1.5 mm model, and set it to `false` to use the 3.0 mm model. The high-resolution model is named `model.pt` by default, the low-resolution model is named `model_lowres.pt`.\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody2.png)\n",
    "\n",
    "\n",
    "### 2.3 Run Auto Inference and Select Config Options\n",
    "\n",
    "- MONAI Label has loaded the pre-trained weights within the bundle for multi-organ segmentation task, click **run** to do inference on the currently loaded data. \n",
    "\n",
    "- Note: the pre-trained models are in the bundle's \"**models**\" folder. For instance, in this usecase, the downloaded pre-trained model is saved at \"**apps/monaibundle/model/wholeBody_ct_segmentation/models**\".\n",
    "- Users can move the axis and slices to view the correct location of segmentation.\n",
    "\n",
    "Users can monitor the logs in the MONAI Label server terminal\n",
    "\n",
    "- High Resolution Segmentation Results:\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody3.png)\n",
    "\n",
    "### 2.4 Edit Annotations and Submit Label\n",
    "\n",
    "- Users can edit the inference labels manually using MONAI Label segmentation editor, click the **Segment Editor** panel. Users can go back to MONAI Label plugin and click **Submit Label** to save the ground truth label to the file system.\n",
    "\n",
    "- Note: the final annotations will be saved to \"**labels/final**\" folder in the study dataset, for example, in this usecase, the ground truth label will be saved to \"**datasets/Task09_Spleen/imagesTs/labels/final**\"\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody5.png)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Train and Fine Tune Models\n",
    "\n",
    "Training and actively fine-tuning models with several iterations is a common labeling scenario. Users can train their models anytime when new annotated labels are saved. Click **Train** button, MONAI Label server will fetch the saved final ground truth label and fine-tune the prior model. \n",
    "\n",
    "- Training Config Options:\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody6.png)\n",
    "\n",
    "\n",
    "Users can monitor the logs in the MONAI Label server terminal\n",
    "\n",
    "![slicer1](./figures/monailabel_wholebody_totalSegmentator_3dslicer/wholeBody7.png)\n",
    "\n",
    "Same to the Radiology app, users can do auto segmentation using the latest fine-tuned model. \n",
    "The active learning process will select unlabeled batch data.\n",
    "\n",
    "The next image is selected, already trained images will be marked as labeled data, then won't be selected for the next labeling batch. Active learning strategies such as \"first/random\" will be used for selecting which unlabeled data to fetch. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Repeat Until All Data are Annotated and Trained \n",
    "\n",
    "Repeated fetch data and active learning process to section 2.2, until all unlabeled data are annotated and trained. \n",
    "\n",
    "For each training loop, the new best metric model will be saved in \"**model/model.pt**\", in this usecase, \"\"**apps/monaibundle/wholeBody_ct_segmentation/model/model.pt**\" is saved."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion\n",
    "\n",
    "This tutorial demonstrates MONAI Label usage with monaibundle app, the 104 anatomies segmentation with CT scans is introduced. Active learning process, auto inference, segmentation editor, submit labels, and save models are included with the bundle. The monaibundle app highlights the robust model deployment for MONAI Label.\n",
    "\n",
    "[1] Wasserthal, J., Meyer, M., Breit, H.C., Cyriac, J., Yang, S. and Segeroth, M., 2022. TotalSegmentator: robust segmentation of 104 anatomical structures in CT images. arXiv preprint arXiv:2208.05868.\n",
    "\n",
    "[2] Myronenko, A., Siddiquee, M.M.R., Yang, D., He, Y. and Xu, D., 2022. Automated head and neck tumor segmentation from 3D PET/CT. arXiv preprint arXiv:2209.10809."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}