experiments/6.0. Comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "095be5a9",
   "metadata": {},
   "source": [
    "# PHIStruct: Improving phage-host interaction prediction at low sequence similarity settings using structure-aware protein embeddings\n",
    "\n",
    "<b>Mark Edward M. Gonzales<sup>1, 2</sup>, Jennifer C. Ureta<sup>1, 2, 3</sup> & Anish M.S. Shrestha<sup>1, 2</sup></b>\n",
    "\n",
    "<sup>1</sup> Bioinformatics Lab, Advanced Research Institute for Informatics, Computing and Networking, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>2</sup> Department of Software Technology, College of Computer Studies, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>3</sup> Walter and Eliza Hall Institute of Medical Research, Melbourne, Victoria, 3052, Australia\n",
    "\n",
    "✉️ gonzales.markedward@gmail.com, jennifer.ureta@gmail.com, anish.shrestha@dlsu.edu.ph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b4696a",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d9f2bf3",
   "metadata": {},
   "source": [
    "# 💡 Prerequisites\n",
    "\n",
    "The prerequisite files (i.e., the results of evaluating the models' performance) &mdash; which are saved in `temp/results` &mdash; are already included when the repository was cloned. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea978dd",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767e1bca",
   "metadata": {},
   "source": [
    "# Part I: Preliminaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61f701be",
   "metadata": {},
   "source": [
    "Import the necessary libraries and modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "970c2a40",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import ConstantsUtil\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d5a0bd4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None)\n",
    "pd.set_option(\"display.max_columns\", 50)\n",
    "\n",
    "pd.options.mode.chained_assignment = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b9b44f16",
   "metadata": {},
   "outputs": [],
   "source": [
    "constants = ConstantsUtil.ConstantsUtil()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ffed7a",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b124835",
   "metadata": {},
   "source": [
    "# Part II: Compare the performance of the classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e026a4a",
   "metadata": {},
   "source": [
    "Declare constant variables for readability of subsequent code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cabf4edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASS = 0\n",
    "PRECISION = 0\n",
    "RECALL = 1\n",
    "F1 = 2\n",
    "\n",
    "Y_TEST = -3\n",
    "Y_PRED = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9fcd5eb",
   "metadata": {},
   "source": [
    "#### Change the value of `SIMILARITY` as needed.\n",
    "\n",
    "`SIMILARITY` refers to the maximum train-versus-test sequence similarity threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7d09197a",
   "metadata": {},
   "outputs": [],
   "source": [
    "SIMILARITY = 40\n",
    "\n",
    "SOTA = [\n",
    "    f\"boeckaerts-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"phiembed-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"badam-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"blast-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"psiblast-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "]\n",
    "\n",
    "PLM = [\n",
    "    f\"prott5-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"esm2-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"esm1b-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"seqvec-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "]\n",
    "\n",
    "STRUCT = [\n",
    "    f\"prostt5_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"prostt5_3di_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"pst_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "]\n",
    "\n",
    "MASKING = [\n",
    "    f\"saprot_seq_mask_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_struct_mask_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_mask_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "]\n",
    "\n",
    "ML = [\n",
    "    f\"saprot_relaxed_r3-rf-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-svm-eskapee-smotetomek-{SIMILARITY}\",\n",
    "    f\"saprot_relaxed_r3-mlp-eskapee-smotetomek-{SIMILARITY}\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45cdae84",
   "metadata": {},
   "source": [
    "#### Change the value of `model` as needed.\n",
    "\n",
    "Choose from the constants defined in the code block above<br> (these constants pertain to our benchmarking experiments):\n",
    "- `SOTA` - state-of-the-art phage-host interaction prediction tools\n",
    "- `PLM`- sequence-only protein language models\n",
    "- `STRUCT` - structure-aware protein language models\n",
    "- `MASKING` - masking strategy (with respect to the SaProt encoding)\n",
    "- `ML` - downstream classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2f37af85",
   "metadata": {},
   "outputs": [],
   "source": [
    "models = SOTA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3283ff19",
   "metadata": {},
   "source": [
    "Load the pickled files storing the results of evaluating the models' performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8ecfd888",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_results = []\n",
    "for model in models:\n",
    "    with open(f\"{constants.TEMP_RESULTS}/{model}.pickle\", \"rb\") as f:\n",
    "        model_results.append(pickle.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6fc0a898",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = [\n",
    "    \"enterococcus\",\n",
    "    \"staphylococcus\",\n",
    "    \"klebsiella\",\n",
    "    \"acinetobacter\",\n",
    "    \"pseudomonas\",\n",
    "    \"enterobacter\",\n",
    "    \"escherichia\",\n",
    "    \"others\",\n",
    "]\n",
    "\n",
    "labels.sort()\n",
    "\n",
    "\n",
    "def get_macro_no_others(model, threshold, metric):\n",
    "    ave = 0\n",
    "    for idx, label in enumerate(labels):\n",
    "        if label != \"others\":\n",
    "            ave += model_results[models.index(model)][threshold][CLASS][metric][idx]\n",
    "\n",
    "    return ave / (len(labels) - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec4e5990",
   "metadata": {},
   "source": [
    "The shaded cells in the subsequent tables correspond to the highest scores (i.e., best performance in terms of the specified evaluation metric). \n",
    "\n",
    "⚠️ **Caveat**: The highest score is determined via lexicographical ordering of the values (i.e., the scores are treated as strings) &mdash; so do some quick sanity check! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "98ab591e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Macro Recall\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_4bb21_row0_col0, #T_4bb21_row0_col8, #T_4bb21_row1_col5, #T_4bb21_row3_col6, #T_4bb21_row3_col7, #T_4bb21_row3_col9, #T_4bb21_row5_col1, #T_4bb21_row5_col2, #T_4bb21_row5_col3, #T_4bb21_row5_col4 {\n",
       "  background-color: lightgreen;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_4bb21\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_4bb21_level0_col0\" class=\"col_heading level0 col0\" >0%</th>\n",
       "      <th id=\"T_4bb21_level0_col1\" class=\"col_heading level0 col1\" >10%</th>\n",
       "      <th id=\"T_4bb21_level0_col2\" class=\"col_heading level0 col2\" >20%</th>\n",
       "      <th id=\"T_4bb21_level0_col3\" class=\"col_heading level0 col3\" >30%</th>\n",
       "      <th id=\"T_4bb21_level0_col4\" class=\"col_heading level0 col4\" >40%</th>\n",
       "      <th id=\"T_4bb21_level0_col5\" class=\"col_heading level0 col5\" >50%</th>\n",
       "      <th id=\"T_4bb21_level0_col6\" class=\"col_heading level0 col6\" >60%</th>\n",
       "      <th id=\"T_4bb21_level0_col7\" class=\"col_heading level0 col7\" >70%</th>\n",
       "      <th id=\"T_4bb21_level0_col8\" class=\"col_heading level0 col8\" >80%</th>\n",
       "      <th id=\"T_4bb21_level0_col9\" class=\"col_heading level0 col9\" >90%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row0\" class=\"row_heading level0 row0\" >boeckaerts-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row0_col0\" class=\"data row0 col0\" >64.12%</td>\n",
       "      <td id=\"T_4bb21_row0_col1\" class=\"data row0 col1\" >48.78%</td>\n",
       "      <td id=\"T_4bb21_row0_col2\" class=\"data row0 col2\" >37.91%</td>\n",
       "      <td id=\"T_4bb21_row0_col3\" class=\"data row0 col3\" >32.31%</td>\n",
       "      <td id=\"T_4bb21_row0_col4\" class=\"data row0 col4\" >25.38%</td>\n",
       "      <td id=\"T_4bb21_row0_col5\" class=\"data row0 col5\" >20.86%</td>\n",
       "      <td id=\"T_4bb21_row0_col6\" class=\"data row0 col6\" >16.05%</td>\n",
       "      <td id=\"T_4bb21_row0_col7\" class=\"data row0 col7\" >13.36%</td>\n",
       "      <td id=\"T_4bb21_row0_col8\" class=\"data row0 col8\" >6.53%</td>\n",
       "      <td id=\"T_4bb21_row0_col9\" class=\"data row0 col9\" >2.55%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row1\" class=\"row_heading level0 row1\" >phiembed-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row1_col0\" class=\"data row1 col0\" >52.43%</td>\n",
       "      <td id=\"T_4bb21_row1_col1\" class=\"data row1 col1\" >39.69%</td>\n",
       "      <td id=\"T_4bb21_row1_col2\" class=\"data row1 col2\" >29.86%</td>\n",
       "      <td id=\"T_4bb21_row1_col3\" class=\"data row1 col3\" >21.97%</td>\n",
       "      <td id=\"T_4bb21_row1_col4\" class=\"data row1 col4\" >14.23%</td>\n",
       "      <td id=\"T_4bb21_row1_col5\" class=\"data row1 col5\" >8.70%</td>\n",
       "      <td id=\"T_4bb21_row1_col6\" class=\"data row1 col6\" >5.31%</td>\n",
       "      <td id=\"T_4bb21_row1_col7\" class=\"data row1 col7\" >2.54%</td>\n",
       "      <td id=\"T_4bb21_row1_col8\" class=\"data row1 col8\" >1.70%</td>\n",
       "      <td id=\"T_4bb21_row1_col9\" class=\"data row1 col9\" >1.24%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row2\" class=\"row_heading level0 row2\" >badam-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row2_col0\" class=\"data row2 col0\" >59.29%</td>\n",
       "      <td id=\"T_4bb21_row2_col1\" class=\"data row2 col1\" >58.45%</td>\n",
       "      <td id=\"T_4bb21_row2_col2\" class=\"data row2 col2\" >57.24%</td>\n",
       "      <td id=\"T_4bb21_row2_col3\" class=\"data row2 col3\" >56.69%</td>\n",
       "      <td id=\"T_4bb21_row2_col4\" class=\"data row2 col4\" >55.63%</td>\n",
       "      <td id=\"T_4bb21_row2_col5\" class=\"data row2 col5\" >54.87%</td>\n",
       "      <td id=\"T_4bb21_row2_col6\" class=\"data row2 col6\" >54.10%</td>\n",
       "      <td id=\"T_4bb21_row2_col7\" class=\"data row2 col7\" >53.03%</td>\n",
       "      <td id=\"T_4bb21_row2_col8\" class=\"data row2 col8\" >51.67%</td>\n",
       "      <td id=\"T_4bb21_row2_col9\" class=\"data row2 col9\" >49.70%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row3\" class=\"row_heading level0 row3\" >blast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row3_col0\" class=\"data row3 col0\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col1\" class=\"data row3 col1\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col2\" class=\"data row3 col2\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col3\" class=\"data row3 col3\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col4\" class=\"data row3 col4\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col5\" class=\"data row3 col5\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col6\" class=\"data row3 col6\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col7\" class=\"data row3 col7\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col8\" class=\"data row3 col8\" >58.14%</td>\n",
       "      <td id=\"T_4bb21_row3_col9\" class=\"data row3 col9\" >58.14%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row4\" class=\"row_heading level0 row4\" >psiblast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row4_col0\" class=\"data row4 col0\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col1\" class=\"data row4 col1\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col2\" class=\"data row4 col2\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col3\" class=\"data row4 col3\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col4\" class=\"data row4 col4\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col5\" class=\"data row4 col5\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col6\" class=\"data row4 col6\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col7\" class=\"data row4 col7\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col8\" class=\"data row4 col8\" >46.06%</td>\n",
       "      <td id=\"T_4bb21_row4_col9\" class=\"data row4 col9\" >46.06%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_4bb21_level0_row5\" class=\"row_heading level0 row5\" >saprot_relaxed_r3-mlp-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_4bb21_row5_col0\" class=\"data row5 col0\" >63.09%</td>\n",
       "      <td id=\"T_4bb21_row5_col1\" class=\"data row5 col1\" >62.60%</td>\n",
       "      <td id=\"T_4bb21_row5_col2\" class=\"data row5 col2\" >61.09%</td>\n",
       "      <td id=\"T_4bb21_row5_col3\" class=\"data row5 col3\" >60.23%</td>\n",
       "      <td id=\"T_4bb21_row5_col4\" class=\"data row5 col4\" >59.30%</td>\n",
       "      <td id=\"T_4bb21_row5_col5\" class=\"data row5 col5\" >58.52%</td>\n",
       "      <td id=\"T_4bb21_row5_col6\" class=\"data row5 col6\" >57.69%</td>\n",
       "      <td id=\"T_4bb21_row5_col7\" class=\"data row5 col7\" >56.35%</td>\n",
       "      <td id=\"T_4bb21_row5_col8\" class=\"data row5 col8\" >55.00%</td>\n",
       "      <td id=\"T_4bb21_row5_col9\" class=\"data row5 col9\" >51.95%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1e80c3e9a30>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Macro Recall\")\n",
    "\n",
    "results = []\n",
    "for model in models:\n",
    "    result = []\n",
    "    for threshold in range(0, 10):\n",
    "        metric = \"{:.2f}\".format(get_macro_no_others(model, threshold, RECALL) * 100)\n",
    "        result.append(f\"{metric}%\")\n",
    "    results.append(result)\n",
    "\n",
    "results_df = pd.DataFrame(\n",
    "    results, columns=[str(_) + \"%\" for _ in range(0, 91, 10)], index=models\n",
    ")\n",
    "results_df.style.highlight_max(color=\"lightgreen\", axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ab6fe985",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Macro Precision\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_aee45_row0_col1, #T_aee45_row0_col2, #T_aee45_row0_col3, #T_aee45_row0_col4, #T_aee45_row0_col5, #T_aee45_row1_col6, #T_aee45_row1_col7, #T_aee45_row1_col8, #T_aee45_row5_col0, #T_aee45_row5_col9 {\n",
       "  background-color: lightgreen;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_aee45\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_aee45_level0_col0\" class=\"col_heading level0 col0\" >0%</th>\n",
       "      <th id=\"T_aee45_level0_col1\" class=\"col_heading level0 col1\" >10%</th>\n",
       "      <th id=\"T_aee45_level0_col2\" class=\"col_heading level0 col2\" >20%</th>\n",
       "      <th id=\"T_aee45_level0_col3\" class=\"col_heading level0 col3\" >30%</th>\n",
       "      <th id=\"T_aee45_level0_col4\" class=\"col_heading level0 col4\" >40%</th>\n",
       "      <th id=\"T_aee45_level0_col5\" class=\"col_heading level0 col5\" >50%</th>\n",
       "      <th id=\"T_aee45_level0_col6\" class=\"col_heading level0 col6\" >60%</th>\n",
       "      <th id=\"T_aee45_level0_col7\" class=\"col_heading level0 col7\" >70%</th>\n",
       "      <th id=\"T_aee45_level0_col8\" class=\"col_heading level0 col8\" >80%</th>\n",
       "      <th id=\"T_aee45_level0_col9\" class=\"col_heading level0 col9\" >90%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row0\" class=\"row_heading level0 row0\" >boeckaerts-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row0_col0\" class=\"data row0 col0\" >52.07%</td>\n",
       "      <td id=\"T_aee45_row0_col1\" class=\"data row0 col1\" >63.82%</td>\n",
       "      <td id=\"T_aee45_row0_col2\" class=\"data row0 col2\" >79.03%</td>\n",
       "      <td id=\"T_aee45_row0_col3\" class=\"data row0 col3\" >85.65%</td>\n",
       "      <td id=\"T_aee45_row0_col4\" class=\"data row0 col4\" >89.71%</td>\n",
       "      <td id=\"T_aee45_row0_col5\" class=\"data row0 col5\" >89.68%</td>\n",
       "      <td id=\"T_aee45_row0_col6\" class=\"data row0 col6\" >69.14%</td>\n",
       "      <td id=\"T_aee45_row0_col7\" class=\"data row0 col7\" >70.45%</td>\n",
       "      <td id=\"T_aee45_row0_col8\" class=\"data row0 col8\" >56.88%</td>\n",
       "      <td id=\"T_aee45_row0_col9\" class=\"data row0 col9\" >42.86%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row1\" class=\"row_heading level0 row1\" >phiembed-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row1_col0\" class=\"data row1 col0\" >43.18%</td>\n",
       "      <td id=\"T_aee45_row1_col1\" class=\"data row1 col1\" >53.66%</td>\n",
       "      <td id=\"T_aee45_row1_col2\" class=\"data row1 col2\" >64.31%</td>\n",
       "      <td id=\"T_aee45_row1_col3\" class=\"data row1 col3\" >72.68%</td>\n",
       "      <td id=\"T_aee45_row1_col4\" class=\"data row1 col4\" >79.75%</td>\n",
       "      <td id=\"T_aee45_row1_col5\" class=\"data row1 col5\" >83.93%</td>\n",
       "      <td id=\"T_aee45_row1_col6\" class=\"data row1 col6\" >98.56%</td>\n",
       "      <td id=\"T_aee45_row1_col7\" class=\"data row1 col7\" >85.71%</td>\n",
       "      <td id=\"T_aee45_row1_col8\" class=\"data row1 col8\" >71.43%</td>\n",
       "      <td id=\"T_aee45_row1_col9\" class=\"data row1 col9\" >42.86%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row2\" class=\"row_heading level0 row2\" >badam-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row2_col0\" class=\"data row2 col0\" >46.83%</td>\n",
       "      <td id=\"T_aee45_row2_col1\" class=\"data row2 col1\" >47.26%</td>\n",
       "      <td id=\"T_aee45_row2_col2\" class=\"data row2 col2\" >47.82%</td>\n",
       "      <td id=\"T_aee45_row2_col3\" class=\"data row2 col3\" >48.60%</td>\n",
       "      <td id=\"T_aee45_row2_col4\" class=\"data row2 col4\" >49.08%</td>\n",
       "      <td id=\"T_aee45_row2_col5\" class=\"data row2 col5\" >50.25%</td>\n",
       "      <td id=\"T_aee45_row2_col6\" class=\"data row2 col6\" >51.62%</td>\n",
       "      <td id=\"T_aee45_row2_col7\" class=\"data row2 col7\" >52.62%</td>\n",
       "      <td id=\"T_aee45_row2_col8\" class=\"data row2 col8\" >54.78%</td>\n",
       "      <td id=\"T_aee45_row2_col9\" class=\"data row2 col9\" >58.35%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row3\" class=\"row_heading level0 row3\" >blast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row3_col0\" class=\"data row3 col0\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col1\" class=\"data row3 col1\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col2\" class=\"data row3 col2\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col3\" class=\"data row3 col3\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col4\" class=\"data row3 col4\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col5\" class=\"data row3 col5\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col6\" class=\"data row3 col6\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col7\" class=\"data row3 col7\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col8\" class=\"data row3 col8\" >52.97%</td>\n",
       "      <td id=\"T_aee45_row3_col9\" class=\"data row3 col9\" >52.97%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row4\" class=\"row_heading level0 row4\" >psiblast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row4_col0\" class=\"data row4 col0\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col1\" class=\"data row4 col1\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col2\" class=\"data row4 col2\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col3\" class=\"data row4 col3\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col4\" class=\"data row4 col4\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col5\" class=\"data row4 col5\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col6\" class=\"data row4 col6\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col7\" class=\"data row4 col7\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col8\" class=\"data row4 col8\" >43.96%</td>\n",
       "      <td id=\"T_aee45_row4_col9\" class=\"data row4 col9\" >43.96%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_aee45_level0_row5\" class=\"row_heading level0 row5\" >saprot_relaxed_r3-mlp-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_aee45_row5_col0\" class=\"data row5 col0\" >56.62%</td>\n",
       "      <td id=\"T_aee45_row5_col1\" class=\"data row5 col1\" >57.98%</td>\n",
       "      <td id=\"T_aee45_row5_col2\" class=\"data row5 col2\" >58.31%</td>\n",
       "      <td id=\"T_aee45_row5_col3\" class=\"data row5 col3\" >59.13%</td>\n",
       "      <td id=\"T_aee45_row5_col4\" class=\"data row5 col4\" >59.90%</td>\n",
       "      <td id=\"T_aee45_row5_col5\" class=\"data row5 col5\" >61.45%</td>\n",
       "      <td id=\"T_aee45_row5_col6\" class=\"data row5 col6\" >62.79%</td>\n",
       "      <td id=\"T_aee45_row5_col7\" class=\"data row5 col7\" >65.12%</td>\n",
       "      <td id=\"T_aee45_row5_col8\" class=\"data row5 col8\" >67.50%</td>\n",
       "      <td id=\"T_aee45_row5_col9\" class=\"data row5 col9\" >69.43%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1e81051de50>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Macro Precision\")\n",
    "\n",
    "results = []\n",
    "for model in models:\n",
    "    result = []\n",
    "    for threshold in range(0, 10):\n",
    "        metric = \"{:.2f}\".format(get_macro_no_others(model, threshold, PRECISION) * 100)\n",
    "        result.append(f\"{metric}%\")\n",
    "    results.append(result)\n",
    "\n",
    "results_df = pd.DataFrame(\n",
    "    results, columns=[str(_) + \"%\" for _ in range(0, 91, 10)], index=models\n",
    ")\n",
    "results_df.style.highlight_max(color=\"lightgreen\", axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d9a0e376",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Macro F1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_d9ad0_row1_col6, #T_d9ad0_row5_col0, #T_d9ad0_row5_col1, #T_d9ad0_row5_col2, #T_d9ad0_row5_col3, #T_d9ad0_row5_col4, #T_d9ad0_row5_col5, #T_d9ad0_row5_col7, #T_d9ad0_row5_col8, #T_d9ad0_row5_col9 {\n",
       "  background-color: lightgreen;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_d9ad0\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_d9ad0_level0_col0\" class=\"col_heading level0 col0\" >0%</th>\n",
       "      <th id=\"T_d9ad0_level0_col1\" class=\"col_heading level0 col1\" >10%</th>\n",
       "      <th id=\"T_d9ad0_level0_col2\" class=\"col_heading level0 col2\" >20%</th>\n",
       "      <th id=\"T_d9ad0_level0_col3\" class=\"col_heading level0 col3\" >30%</th>\n",
       "      <th id=\"T_d9ad0_level0_col4\" class=\"col_heading level0 col4\" >40%</th>\n",
       "      <th id=\"T_d9ad0_level0_col5\" class=\"col_heading level0 col5\" >50%</th>\n",
       "      <th id=\"T_d9ad0_level0_col6\" class=\"col_heading level0 col6\" >60%</th>\n",
       "      <th id=\"T_d9ad0_level0_col7\" class=\"col_heading level0 col7\" >70%</th>\n",
       "      <th id=\"T_d9ad0_level0_col8\" class=\"col_heading level0 col8\" >80%</th>\n",
       "      <th id=\"T_d9ad0_level0_col9\" class=\"col_heading level0 col9\" >90%</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row0\" class=\"row_heading level0 row0\" >boeckaerts-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row0_col0\" class=\"data row0 col0\" >55.55%</td>\n",
       "      <td id=\"T_d9ad0_row0_col1\" class=\"data row0 col1\" >52.63%</td>\n",
       "      <td id=\"T_d9ad0_row0_col2\" class=\"data row0 col2\" >45.49%</td>\n",
       "      <td id=\"T_d9ad0_row0_col3\" class=\"data row0 col3\" >40.69%</td>\n",
       "      <td id=\"T_d9ad0_row0_col4\" class=\"data row0 col4\" >33.29%</td>\n",
       "      <td id=\"T_d9ad0_row0_col5\" class=\"data row0 col5\" >28.17%</td>\n",
       "      <td id=\"T_d9ad0_row0_col6\" class=\"data row0 col6\" >22.70%</td>\n",
       "      <td id=\"T_d9ad0_row0_col7\" class=\"data row0 col7\" >19.77%</td>\n",
       "      <td id=\"T_d9ad0_row0_col8\" class=\"data row0 col8\" >11.02%</td>\n",
       "      <td id=\"T_d9ad0_row0_col9\" class=\"data row0 col9\" >4.65%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row1\" class=\"row_heading level0 row1\" >phiembed-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row1_col0\" class=\"data row1 col0\" >43.60%</td>\n",
       "      <td id=\"T_d9ad0_row1_col1\" class=\"data row1 col1\" >42.91%</td>\n",
       "      <td id=\"T_d9ad0_row1_col2\" class=\"data row1 col2\" >39.27%</td>\n",
       "      <td id=\"T_d9ad0_row1_col3\" class=\"data row1 col3\" >31.87%</td>\n",
       "      <td id=\"T_d9ad0_row1_col4\" class=\"data row1 col4\" >23.20%</td>\n",
       "      <td id=\"T_d9ad0_row1_col5\" class=\"data row1 col5\" >15.46%</td>\n",
       "      <td id=\"T_d9ad0_row1_col6\" class=\"data row1 col6\" >9.90%</td>\n",
       "      <td id=\"T_d9ad0_row1_col7\" class=\"data row1 col7\" >4.78%</td>\n",
       "      <td id=\"T_d9ad0_row1_col8\" class=\"data row1 col8\" >3.23%</td>\n",
       "      <td id=\"T_d9ad0_row1_col9\" class=\"data row1 col9\" >2.33%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row2\" class=\"row_heading level0 row2\" >badam-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row2_col0\" class=\"data row2 col0\" >48.12%</td>\n",
       "      <td id=\"T_d9ad0_row2_col1\" class=\"data row2 col1\" >47.93%</td>\n",
       "      <td id=\"T_d9ad0_row2_col2\" class=\"data row2 col2\" >47.78%</td>\n",
       "      <td id=\"T_d9ad0_row2_col3\" class=\"data row2 col3\" >47.96%</td>\n",
       "      <td id=\"T_d9ad0_row2_col4\" class=\"data row2 col4\" >47.86%</td>\n",
       "      <td id=\"T_d9ad0_row2_col5\" class=\"data row2 col5\" >48.18%</td>\n",
       "      <td id=\"T_d9ad0_row2_col6\" class=\"data row2 col6\" >48.61%</td>\n",
       "      <td id=\"T_d9ad0_row2_col7\" class=\"data row2 col7\" >48.68%</td>\n",
       "      <td id=\"T_d9ad0_row2_col8\" class=\"data row2 col8\" >49.03%</td>\n",
       "      <td id=\"T_d9ad0_row2_col9\" class=\"data row2 col9\" >49.46%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row3\" class=\"row_heading level0 row3\" >blast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row3_col0\" class=\"data row3 col0\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col1\" class=\"data row3 col1\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col2\" class=\"data row3 col2\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col3\" class=\"data row3 col3\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col4\" class=\"data row3 col4\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col5\" class=\"data row3 col5\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col6\" class=\"data row3 col6\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col7\" class=\"data row3 col7\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col8\" class=\"data row3 col8\" >51.35%</td>\n",
       "      <td id=\"T_d9ad0_row3_col9\" class=\"data row3 col9\" >51.35%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row4\" class=\"row_heading level0 row4\" >psiblast-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row4_col0\" class=\"data row4 col0\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col1\" class=\"data row4 col1\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col2\" class=\"data row4 col2\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col3\" class=\"data row4 col3\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col4\" class=\"data row4 col4\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col5\" class=\"data row4 col5\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col6\" class=\"data row4 col6\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col7\" class=\"data row4 col7\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col8\" class=\"data row4 col8\" >39.67%</td>\n",
       "      <td id=\"T_d9ad0_row4_col9\" class=\"data row4 col9\" >39.67%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_d9ad0_level0_row5\" class=\"row_heading level0 row5\" >saprot_relaxed_r3-mlp-eskapee-smotetomek-40</th>\n",
       "      <td id=\"T_d9ad0_row5_col0\" class=\"data row5 col0\" >57.15%</td>\n",
       "      <td id=\"T_d9ad0_row5_col1\" class=\"data row5 col1\" >57.61%</td>\n",
       "      <td id=\"T_d9ad0_row5_col2\" class=\"data row5 col2\" >57.22%</td>\n",
       "      <td id=\"T_d9ad0_row5_col3\" class=\"data row5 col3\" >57.09%</td>\n",
       "      <td id=\"T_d9ad0_row5_col4\" class=\"data row5 col4\" >57.09%</td>\n",
       "      <td id=\"T_d9ad0_row5_col5\" class=\"data row5 col5\" >57.42%</td>\n",
       "      <td id=\"T_d9ad0_row5_col6\" class=\"data row5 col6\" >57.48%</td>\n",
       "      <td id=\"T_d9ad0_row5_col7\" class=\"data row5 col7\" >57.67%</td>\n",
       "      <td id=\"T_d9ad0_row5_col8\" class=\"data row5 col8\" >57.44%</td>\n",
       "      <td id=\"T_d9ad0_row5_col9\" class=\"data row5 col9\" >56.27%</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x1e80f3a32c0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Macro F1\")\n",
    "\n",
    "results = []\n",
    "for model in models:\n",
    "    result = []\n",
    "    for threshold in range(0, 10):\n",
    "        metric = \"{:.2f}\".format(get_macro_no_others(model, threshold, F1) * 100)\n",
    "        result.append(f\"{metric}%\")\n",
    "    results.append(result)\n",
    "\n",
    "results_df = pd.DataFrame(\n",
    "    results, columns=[str(_) + \"%\" for _ in range(0, 91, 10)], index=models\n",
    ")\n",
    "results_df.style.highlight_max(color=\"lightgreen\", axis=0)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}