tutorials/plugins/tutorial_05_survival_analysis_plugins.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "molecular-moscow",
   "metadata": {},
   "source": [
    "# Survival analysis Plugins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corrected-basis",
   "metadata": {},
   "source": [
    "\n",
    "**AutoPrognosis** provides a set of default predicton plugins and can be extended with any number of other plugins.\n",
    "\n",
    "In this tutorial, we will focus on the __survival analysis__ tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "multiple-annotation",
   "metadata": {},
   "source": [
    "### Plugins 101\n",
    "\n",
    "Every **AutoPrognosis plugin** must implement the **`Plugin`** interface provided by `autoprognosis/plugins/core/base_plugin.py`.\n",
    "\n",
    "Each **AutoPrognosis prediction plugin** must implement the **`PredictionPlugin`** interface provided by `autoprognosis/plugins/prediction/base.py`\n",
    "\n",
    "__Warning__ : If a plugin doesn't override all the abstract methods, it won't be loaded by the library.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "__API__ : Every prediction plugin must implement the following methods:\n",
    "- `name()` - a static method that returns the name of the plugin. e.g., neural_nets, perceptron, etc.\n",
    "\n",
    "- `subtype()` - a static method that returns the plugin's subtype. e.g., \"classification\", \"survival_analysis\" etc. It will be used for filtering the plugin in the optimization process.\n",
    "    \n",
    "- `hyperparameter_space()` - a static method that returns the hyperparameters that can be tuned during the optimization. The method will return a list of `skopt.space.Dimension` derived objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "auburn-hygiene",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wanted-point",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stdlib\n",
    "import sys\n",
    "import warnings\n",
    "\n",
    "# third party\n",
    "from IPython.display import HTML, display\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tabulate\n",
    "\n",
    "if not sys.warnoptions:\n",
    "    warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "devoted-console",
   "metadata": {},
   "source": [
    "### Loading the RiskEstimation plugins\n",
    "\n",
    "Make sure that you have installed AutoPrognosis in your workspace.\n",
    "\n",
    "You can do that by running `pip install .` in the root of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coated-innocent",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autoprognosis absolute\n",
    "from autoprognosis.plugins.prediction.risk_estimation import (\n",
    "    RiskEstimation,\n",
    "    RiskEstimationPlugin,\n",
    ")\n",
    "\n",
    "predictors = RiskEstimation()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-booth",
   "metadata": {},
   "source": [
    "### List the existing plugins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incident-familiar",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors.list_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominican-tulsa",
   "metadata": {},
   "source": [
    "## Benchmarks\n",
    "\n",
    "We test the prediction plugins using the [METABRIC dataset](https://github.com/jaredleekatzman/DeepSurv/tree/master/experiments/data)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advance-expert",
   "metadata": {},
   "source": [
    "### Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "preceding-vermont",
   "metadata": {},
   "outputs": [],
   "source": [
    "# third party\n",
    "import numpy as np\n",
    "from pycox import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = datasets.metabric.read_df()\n",
    "\n",
    "X = df.drop(columns=[\"duration\", \"event\"])\n",
    "T = df[\"duration\"]\n",
    "E = df[\"event\"]\n",
    "\n",
    "X = X[T > 0]\n",
    "E = E[T > 0]\n",
    "T = T[T > 0]\n",
    "\n",
    "time_horizons = np.linspace(T.min(), T.max(), 5)[1:-1]\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opened-finding",
   "metadata": {},
   "source": [
    "### Duration benchmarks\n",
    "\n",
    "__About__ : This step measures the fit_predict duration for each plugin on the dataset. The times are reported in milliseconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "recovered-tonight",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stdlib\n",
    "import time\n",
    "\n",
    "# third party\n",
    "from tqdm import tqdm\n",
    "\n",
    "duration = []\n",
    "\n",
    "plugins = predictors.list_available()\n",
    "\n",
    "for plugin in tqdm(plugins):\n",
    "    plugin_duration = [plugin]\n",
    "    ctx = predictors.get(plugin)\n",
    "\n",
    "    start = time.time() * 1000\n",
    "    ctx.fit(X, T, E)\n",
    "    ctx.predict(X, time_horizons)\n",
    "\n",
    "    plugin_duration.append(round(time.time() * 1000 - start, 4))\n",
    "\n",
    "    duration.append(plugin_duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numeric-million",
   "metadata": {},
   "source": [
    "### Duration(ms) results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "viral-suggestion",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        tabulate.tabulate(duration, headers=[\"Plugin\", \"Duration(ms)\"], tablefmt=\"html\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de21a377",
   "metadata": {},
   "source": [
    "### Prediction performance\n",
    "\n",
    "__Steps__\n",
    " - We train each prediction plugin on the dataset.\n",
    " - We report the Brier score and the C-Index metrics using cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c1ecd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autoprognosis absolute\n",
    "from autoprognosis.utils.tester import evaluate_survival_estimator\n",
    "\n",
    "\n",
    "def get_metrics(plugin):\n",
    "    score = evaluate_survival_estimator(plugin, X, T, E, time_horizons)[\"str\"]\n",
    "\n",
    "    return score[\"c_index\"], score[\"brier_score\"]\n",
    "\n",
    "\n",
    "plugins = predictors.list_available()\n",
    "\n",
    "metrics_headers = [\"Plugin\", \"C-INDEX\", \"Brier Score\"]\n",
    "test_score = []\n",
    "\n",
    "\n",
    "for plugin in plugins:\n",
    "    fproc = predictors.get(plugin)\n",
    "\n",
    "    cindex, brier_score = get_metrics(fproc)\n",
    "\n",
    "    test_score.append([plugin, cindex, brier_score])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d43c9fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(HTML(tabulate.tabulate(test_score, headers=metrics_headers, tablefmt=\"html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "neural-dutch",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards Machine learning and AI for medicine, you can do so in the following ways!\n",
    "\n",
    "### Star AutoPrognosis on GitHub\n",
    "\n",
    "The easiest way to help our community is just by starring the Repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "- [Star AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- [Star HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}