experiments/5.1. Benchmarking - Classifier Building & Evaluation (ProstT5 - AA Tokens).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "62ae27ce",
   "metadata": {},
   "source": [
    "# PHIStruct: Improving phage-host interaction prediction at low sequence similarity settings using structure-aware protein embeddings\n",
    "\n",
    "<b>Mark Edward M. Gonzales<sup>1, 2</sup>, Jennifer C. Ureta<sup>1, 2, 3</sup> & Anish M.S. Shrestha<sup>1, 2</sup></b>\n",
    "\n",
    "<sup>1</sup> Bioinformatics Lab, Advanced Research Institute for Informatics, Computing and Networking, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>2</sup> Department of Software Technology, College of Computer Studies, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>3</sup> Walter and Eliza Hall Institute of Medical Research, Melbourne, Victoria, 3052, Australia\n",
    "\n",
    "✉️ gonzales.markedward@gmail.com, jennifer.ureta@gmail.com, anish.shrestha@dlsu.edu.ph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "537a360b",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c99ba094",
   "metadata": {},
   "source": [
    "# 💡 Prerequisites\n",
    "\n",
    "### Option 1: Download the prerequisite files\n",
    "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook only needs `consolidated/rbp_embeddings_prostt5_relaxed_r3.csv`.\n",
    "1. Create a folder named `inphared` inside `data`, and save the extracted `consolidated` folder inside `data/inphared`. \n",
    "1. Download `fasta.tar.gz` from this [link](https://drive.google.com/file/d/1NMFR3JrrrCHLoCMQp2nia4dgtcXs5x05/view?usp=sharing), and unzip it. This should result in a folder named `fasta`. <br> Technically, this notebook only needs the `.clstr` files inside `fasta`.\n",
    "1. Save the extracted `fasta` folder inside `data/inphared`.\n",
    "\n",
    "### Option 2: Generate the prerequisite files yourself\n",
    "1. If you have run `3.1. Data Consolidation (ProstT5 - AA Tokens).ipynb`, then `data/inphared/consolidated` should have already been populated with the prerequisite files.\n",
    "1. Consolidate the sequences of the proteins with predicted structures into a single FASTA file. <br>\n",
    "   For reproducibility, we provide our consolidated FASTA file [here](https://drive.google.com/file/d/1LTZte1f4lreQ5MXWeM-y2Mtp9z96pXS7/view?usp=sharing).\n",
    "1. Generate the protein clusters by running CD-HIT on this FASTA file at a sequence similarity threshold of 100%, following the instructions [here](https://github.com/weizhongli/cdhit). \n",
    "1. Rename the resulting `.clstr` file to `complete-struct-100.fasta.clstr` and the resulting FASTA file (containing only the representative sequences) to `complete-struct-100.fasta`. \n",
    "1. Generate `complete-struct-80.fasta.clstr`, `complete-struct-60.fasta.clstr`, and `complete-struct-40.fasta.clstr` by running CD-HIT on `complete-struct-100.fasta` at sequence similarity thresholds of 80%, 60%, and 40%, respectively.\n",
    "1. Create a folder named `fasta` inside `data/inphared`, and save the four `.clstr` files inside `data/inphared/fasta`.\n",
    "\n",
    "### Resulting folder structure\n",
    "\n",
    "`experiments` (parent folder of this notebook) <br> \n",
    "↳ `data` <br>\n",
    "&nbsp; &nbsp;↳ `inphared` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `consolidated` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_embeddings_prostt5_relaxed_r3.csv` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `fasta` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-100.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-80.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-40.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-60.fasta.clstr` <br>\n",
    "↳ `5.1. Benchmarking - Classifier Building & Evaluation (ProstT5 - AA Tokens).ipynb` (this notebook) <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509b764f",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62339c3f",
   "metadata": {},
   "source": [
    "# 📁 Output files\n",
    "\n",
    "The output files (i.e., the results of evaluating the model's performance) &mdash; which are saved in `temp/results` &mdash; are already included when the repository was cloned. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "334047fa",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f274d083",
   "metadata": {},
   "source": [
    "# Part I: Preliminaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc7d25a1",
   "metadata": {},
   "source": [
    "Import the necessary libraries and modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0840061c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "\n",
    "import ConstantsUtil\n",
    "import ClassificationUtil\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "98fb65d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None)\n",
    "pd.set_option(\"display.max_columns\", 50)\n",
    "\n",
    "pd.options.mode.chained_assignment = None\n",
    "\n",
    "with warnings.catch_warnings():\n",
    "    warnings.filterwarnings(\n",
    "        \"ignore\", category=sklearn.exceptions.UndefinedMetricWarning\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e27723c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "constants = ConstantsUtil.ConstantsUtil()\n",
    "util = ClassificationUtil.ClassificationUtil()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ada3031",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9f7cc0",
   "metadata": {},
   "source": [
    "# Part II: Classifier Building and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d63c6cc",
   "metadata": {},
   "source": [
    "Train a multilayer perceptron, and evaluate its performance at different train-versus-test similarity and confidence thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d8cf9f6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** prostt5_relaxed_r3, similarity = 100% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (16942, 1024)\n",
      "Test set shape: (2340, 1024)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8588    0.6577    0.7449       111\n",
      "  enterobacter     0.3050    0.3707    0.3346       116\n",
      "  enterococcus     0.7966    0.9216    0.8545        51\n",
      "   escherichia     0.8326    0.8606    0.8463      1040\n",
      "    klebsiella     0.8146    0.8482    0.8310       461\n",
      "        others     0.0000    0.0000    0.0000        51\n",
      "   pseudomonas     0.8928    0.8701    0.8813       354\n",
      "staphylococcus     0.9226    0.9167    0.9196       156\n",
      "\n",
      "      accuracy                         0.8120      2340\n",
      "     macro avg     0.6779    0.6807    0.6765      2340\n",
      "  weighted avg     0.8003    0.8120    0.8050      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8690    0.6577    0.7487       111\n",
      "  enterobacter     0.3209    0.3707    0.3440       116\n",
      "  enterococcus     0.8070    0.9020    0.8519        51\n",
      "   escherichia     0.8393    0.8538    0.8465      1040\n",
      "    klebsiella     0.8220    0.8416    0.8317       461\n",
      "        others     0.0789    0.0588    0.0674        51\n",
      "   pseudomonas     0.9006    0.8701    0.8851       354\n",
      "staphylococcus     0.9226    0.9167    0.9196       156\n",
      "\n",
      "      accuracy                         0.8085      2340\n",
      "     macro avg     0.6951    0.6839    0.6869      2340\n",
      "  weighted avg     0.8092    0.8085    0.8079      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8795    0.6577    0.7526       111\n",
      "  enterobacter     0.3359    0.3707    0.3525       116\n",
      "  enterococcus     0.8070    0.9020    0.8519        51\n",
      "   escherichia     0.8430    0.8413    0.8422      1040\n",
      "    klebsiella     0.8301    0.8373    0.8337       461\n",
      "        others     0.0548    0.0784    0.0645        51\n",
      "   pseudomonas     0.9003    0.8672    0.8835       354\n",
      "staphylococcus     0.9226    0.9167    0.9196       156\n",
      "\n",
      "      accuracy                         0.8021      2340\n",
      "     macro avg     0.6967    0.6839    0.6875      2340\n",
      "  weighted avg     0.8131    0.8021    0.8066      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8795    0.6577    0.7526       111\n",
      "  enterobacter     0.3559    0.3621    0.3590       116\n",
      "  enterococcus     0.8214    0.9020    0.8598        51\n",
      "   escherichia     0.8500    0.8337    0.8417      1040\n",
      "    klebsiella     0.8359    0.8286    0.8322       461\n",
      "        others     0.0446    0.0980    0.0613        51\n",
      "   pseudomonas     0.9027    0.8644    0.8831       354\n",
      "staphylococcus     0.9226    0.9167    0.9196       156\n",
      "\n",
      "      accuracy                         0.7966      2340\n",
      "     macro avg     0.7016    0.6829    0.6887      2340\n",
      "  weighted avg     0.8188    0.7966    0.8065      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9012    0.6577    0.7604       111\n",
      "  enterobacter     0.3832    0.3534    0.3677       116\n",
      "  enterococcus     0.8333    0.8824    0.8571        51\n",
      "   escherichia     0.8527    0.8240    0.8381      1040\n",
      "    klebsiella     0.8381    0.8200    0.8289       461\n",
      "        others     0.0397    0.1176    0.0594        51\n",
      "   pseudomonas     0.9050    0.8616    0.8828       354\n",
      "staphylococcus     0.9286    0.9167    0.9226       156\n",
      "\n",
      "      accuracy                         0.7897      2340\n",
      "     macro avg     0.7102    0.6792    0.6896      2340\n",
      "  weighted avg     0.8237    0.7897    0.8051      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9114    0.6486    0.7579       111\n",
      "  enterobacter     0.3922    0.3448    0.3670       116\n",
      "  enterococcus     0.8491    0.8824    0.8654        51\n",
      "   escherichia     0.8540    0.8096    0.8312      1040\n",
      "    klebsiella     0.8449    0.8156    0.8300       461\n",
      "        others     0.0372    0.1373    0.0586        51\n",
      "   pseudomonas     0.9069    0.8531    0.8792       354\n",
      "staphylococcus     0.9286    0.9167    0.9226       156\n",
      "\n",
      "      accuracy                         0.7808      2340\n",
      "     macro avg     0.7155    0.6760    0.6890      2340\n",
      "  weighted avg     0.8271    0.7808    0.8017      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9103    0.6396    0.7513       111\n",
      "  enterobacter     0.4086    0.3276    0.3636       116\n",
      "  enterococcus     0.8491    0.8824    0.8654        51\n",
      "   escherichia     0.8561    0.7952    0.8245      1040\n",
      "    klebsiella     0.8552    0.8069    0.8304       461\n",
      "        others     0.0385    0.1765    0.0632        51\n",
      "   pseudomonas     0.9207    0.8531    0.8856       354\n",
      "staphylococcus     0.9346    0.9167    0.9256       156\n",
      "\n",
      "      accuracy                         0.7722      2340\n",
      "     macro avg     0.7216    0.6747    0.6887      2340\n",
      "  weighted avg     0.8333    0.7722    0.7996      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9221    0.6396    0.7553       111\n",
      "  enterobacter     0.4375    0.3017    0.3571       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8600    0.7856    0.8211      1040\n",
      "    klebsiella     0.8652    0.7939    0.8281       461\n",
      "        others     0.0458    0.2549    0.0776        51\n",
      "   pseudomonas     0.9255    0.8418    0.8817       354\n",
      "staphylococcus     0.9342    0.9103    0.9221       156\n",
      "\n",
      "      accuracy                         0.7632      2340\n",
      "     macro avg     0.7296    0.6738    0.6872      2340\n",
      "  weighted avg     0.8398    0.7632    0.7968      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9452    0.6216    0.7500       111\n",
      "  enterobacter     0.4286    0.2586    0.3226       116\n",
      "  enterococcus     0.8431    0.8431    0.8431        51\n",
      "   escherichia     0.8651    0.7587    0.8084      1040\n",
      "    klebsiella     0.8789    0.7874    0.8307       461\n",
      "        others     0.0394    0.2745    0.0690        51\n",
      "   pseudomonas     0.9335    0.8333    0.8806       354\n",
      "staphylococcus     0.9467    0.9103    0.9281       156\n",
      "\n",
      "      accuracy                         0.7457      2340\n",
      "     macro avg     0.7351    0.6609    0.6791      2340\n",
      "  weighted avg     0.8473    0.7457    0.7895      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9577    0.6126    0.7473       111\n",
      "  enterobacter     0.4364    0.2069    0.2807       116\n",
      "  enterococcus     0.8400    0.8235    0.8317        51\n",
      "   escherichia     0.8788    0.7317    0.7985      1040\n",
      "    klebsiella     0.8880    0.7397    0.8071       461\n",
      "        others     0.0371    0.3333    0.0668        51\n",
      "   pseudomonas     0.9417    0.8220    0.8778       354\n",
      "staphylococcus     0.9524    0.8974    0.9241       156\n",
      "\n",
      "      accuracy                         0.7197      2340\n",
      "     macro avg     0.7415    0.6459    0.6667      2340\n",
      "  weighted avg     0.8576    0.7197    0.7773      2340\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** prostt5_relaxed_r3, similarity = 80% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (14812, 1024)\n",
      "Test set shape: (2822, 1024)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4303    0.6514    0.5182       109\n",
      "  enterobacter     0.1618    0.1317    0.1452       167\n",
      "  enterococcus     0.5522    0.6167    0.5827        60\n",
      "   escherichia     0.6971    0.6540    0.6749      1344\n",
      "    klebsiella     0.6165    0.5492    0.5809       559\n",
      "        others     0.0000    0.0000    0.0000        60\n",
      "   pseudomonas     0.4583    0.6895    0.5506       351\n",
      "staphylococcus     0.8323    0.8081    0.8201       172\n",
      "\n",
      "      accuracy                         0.6013      2822\n",
      "     macro avg     0.4686    0.5126    0.4841      2822\n",
      "  weighted avg     0.5998    0.6013    0.5959      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4383    0.6514    0.5240       109\n",
      "  enterobacter     0.1615    0.1257    0.1414       167\n",
      "  enterococcus     0.5692    0.6167    0.5920        60\n",
      "   escherichia     0.7011    0.6421    0.6703      1344\n",
      "    klebsiella     0.6271    0.5385    0.5794       559\n",
      "        others     0.0244    0.0333    0.0282        60\n",
      "   pseudomonas     0.4647    0.6752    0.5505       351\n",
      "staphylococcus     0.8457    0.7965    0.8204       172\n",
      "\n",
      "      accuracy                         0.5914      2822\n",
      "     macro avg     0.4790    0.5099    0.4883      2822\n",
      "  weighted avg     0.6066    0.5914    0.5943      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4410    0.6514    0.5259       109\n",
      "  enterobacter     0.1626    0.1198    0.1379       167\n",
      "  enterococcus     0.5781    0.6167    0.5968        60\n",
      "   escherichia     0.7074    0.6295    0.6661      1344\n",
      "    klebsiella     0.6311    0.5295    0.5759       559\n",
      "        others     0.0267    0.0667    0.0381        60\n",
      "   pseudomonas     0.4659    0.6610    0.5465       351\n",
      "staphylococcus     0.8509    0.7965    0.8228       172\n",
      "\n",
      "      accuracy                         0.5822      2822\n",
      "     macro avg     0.4830    0.5089    0.4888      2822\n",
      "  weighted avg     0.6112    0.5822    0.5914      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4551    0.6514    0.5358       109\n",
      "  enterobacter     0.1579    0.1078    0.1281       167\n",
      "  enterococcus     0.5873    0.6167    0.6016        60\n",
      "   escherichia     0.7108    0.6198    0.6622      1344\n",
      "    klebsiella     0.6376    0.5224    0.5742       559\n",
      "        others     0.0279    0.1000    0.0436        60\n",
      "   pseudomonas     0.4679    0.6439    0.5420       351\n",
      "staphylococcus     0.8509    0.7965    0.8228       172\n",
      "\n",
      "      accuracy                         0.5741      2822\n",
      "     macro avg     0.4869    0.5073    0.4888      2822\n",
      "  weighted avg     0.6149    0.5741    0.5887      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4610    0.6514    0.5399       109\n",
      "  enterobacter     0.1619    0.1018    0.1250       167\n",
      "  enterococcus     0.6167    0.6167    0.6167        60\n",
      "   escherichia     0.7179    0.6116    0.6605      1344\n",
      "    klebsiella     0.6414    0.5152    0.5714       559\n",
      "        others     0.0215    0.1000    0.0354        60\n",
      "   pseudomonas     0.4661    0.6268    0.5346       351\n",
      "staphylococcus     0.8544    0.7849    0.8182       172\n",
      "\n",
      "      accuracy                         0.5656      2822\n",
      "     macro avg     0.4926    0.5010    0.4877      2822\n",
      "  weighted avg     0.6200    0.5656    0.5862      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4730    0.6422    0.5447       109\n",
      "  enterobacter     0.1702    0.0958    0.1226       167\n",
      "  enterococcus     0.6271    0.6167    0.6218        60\n",
      "   escherichia     0.7259    0.5990    0.6563      1344\n",
      "    klebsiella     0.6430    0.5027    0.5643       559\n",
      "        others     0.0275    0.1667    0.0472        60\n",
      "   pseudomonas     0.4781    0.6211    0.5403       351\n",
      "staphylococcus     0.8581    0.7733    0.8135       172\n",
      "\n",
      "      accuracy                         0.5563      2822\n",
      "     macro avg     0.5004    0.5022    0.4888      2822\n",
      "  weighted avg     0.6271    0.5563    0.5837      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4930    0.6422    0.5578       109\n",
      "  enterobacter     0.1882    0.0958    0.1270       167\n",
      "  enterococcus     0.6545    0.6000    0.6261        60\n",
      "   escherichia     0.7274    0.5878    0.6502      1344\n",
      "    klebsiella     0.6437    0.4848    0.5531       559\n",
      "        others     0.0287    0.2167    0.0507        60\n",
      "   pseudomonas     0.5000    0.6125    0.5506       351\n",
      "staphylococcus     0.8733    0.7616    0.8137       172\n",
      "\n",
      "      accuracy                         0.5464      2822\n",
      "     macro avg     0.5136    0.5002    0.4911      2822\n",
      "  weighted avg     0.6341    0.5464    0.5807      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5111    0.6330    0.5656       109\n",
      "  enterobacter     0.2222    0.0958    0.1339       167\n",
      "  enterococcus     0.6792    0.6000    0.6372        60\n",
      "   escherichia     0.7334    0.5751    0.6447      1344\n",
      "    klebsiella     0.6558    0.4669    0.5455       559\n",
      "        others     0.0285    0.2667    0.0515        60\n",
      "   pseudomonas     0.5137    0.5869    0.5479       351\n",
      "staphylococcus     0.8784    0.7558    0.8125       172\n",
      "\n",
      "      accuracy                         0.5340      2822\n",
      "     macro avg     0.5278    0.4975    0.4923      2822\n",
      "  weighted avg     0.6446    0.5340    0.5772      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5197    0.6055    0.5593       109\n",
      "  enterobacter     0.2963    0.0958    0.1448       167\n",
      "  enterococcus     0.7143    0.5833    0.6422        60\n",
      "   escherichia     0.7458    0.5565    0.6374      1344\n",
      "    klebsiella     0.6694    0.4419    0.5323       559\n",
      "        others     0.0284    0.3333    0.0523        60\n",
      "   pseudomonas     0.5215    0.5527    0.5367       351\n",
      "staphylococcus     0.8811    0.7326    0.8000       172\n",
      "\n",
      "      accuracy                         0.5145      2822\n",
      "     macro avg     0.5470    0.4877    0.4881      2822\n",
      "  weighted avg     0.6597    0.5145    0.5695      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5439    0.5688    0.5561       109\n",
      "  enterobacter     0.3421    0.0778    0.1268       167\n",
      "  enterococcus     0.7500    0.5500    0.6346        60\n",
      "   escherichia     0.7664    0.5223    0.6212      1344\n",
      "    klebsiella     0.6815    0.4097    0.5117       559\n",
      "        others     0.0309    0.4667    0.0580        60\n",
      "   pseudomonas     0.5526    0.5242    0.5380       351\n",
      "staphylococcus     0.9044    0.7151    0.7987       172\n",
      "\n",
      "      accuracy                         0.4869      2822\n",
      "     macro avg     0.5715    0.4793    0.4807      2822\n",
      "  weighted avg     0.6817    0.4869    0.5565      2822\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** prostt5_relaxed_r3, similarity = 60% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (13099, 1024)\n",
      "Test set shape: (3153, 1024)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4621    0.5537    0.5038       121\n",
      "  enterobacter     0.1164    0.1378    0.1262       196\n",
      "  enterococcus     0.4500    0.5510    0.4954        49\n",
      "   escherichia     0.6667    0.6532    0.6599      1589\n",
      "    klebsiella     0.4271    0.3588    0.3900       588\n",
      "        others     0.0000    0.0000    0.0000        49\n",
      "   pseudomonas     0.5071    0.6163    0.5564       404\n",
      "staphylococcus     0.6437    0.7134    0.6767       157\n",
      "\n",
      "      accuracy                         0.5490      3153\n",
      "     macro avg     0.4091    0.4480    0.4261      3153\n",
      "  weighted avg     0.5446    0.5490    0.5452      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4815    0.5372    0.5078       121\n",
      "  enterobacter     0.1189    0.1378    0.1277       196\n",
      "  enterococcus     0.4655    0.5510    0.5047        49\n",
      "   escherichia     0.6684    0.6394    0.6536      1589\n",
      "    klebsiella     0.4352    0.3486    0.3872       588\n",
      "        others     0.0211    0.0408    0.0278        49\n",
      "   pseudomonas     0.5157    0.6114    0.5595       404\n",
      "staphylococcus     0.6488    0.6943    0.6708       157\n",
      "\n",
      "      accuracy                         0.5385      3153\n",
      "     macro avg     0.4194    0.4451    0.4299      3153\n",
      "  weighted avg     0.5498    0.5385    0.5424      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4769    0.5124    0.4940       121\n",
      "  enterobacter     0.1168    0.1276    0.1220       196\n",
      "  enterococcus     0.4821    0.5510    0.5143        49\n",
      "   escherichia     0.6762    0.6281    0.6512      1589\n",
      "    klebsiella     0.4345    0.3384    0.3805       588\n",
      "        others     0.0104    0.0408    0.0166        49\n",
      "   pseudomonas     0.5216    0.5965    0.5566       404\n",
      "staphylococcus     0.6545    0.6879    0.6708       157\n",
      "\n",
      "      accuracy                         0.5271      3153\n",
      "     macro avg     0.4216    0.4353    0.4257      3153\n",
      "  weighted avg     0.5544    0.5271    0.5387      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4803    0.5041    0.4919       121\n",
      "  enterobacter     0.1186    0.1173    0.1179       196\n",
      "  enterococcus     0.5094    0.5510    0.5294        49\n",
      "   escherichia     0.6787    0.6155    0.6455      1589\n",
      "    klebsiella     0.4510    0.3367    0.3856       588\n",
      "        others     0.0070    0.0408    0.0119        49\n",
      "   pseudomonas     0.5333    0.5941    0.5621       404\n",
      "staphylococcus     0.6626    0.6879    0.6750       157\n",
      "\n",
      "      accuracy                         0.5192      3153\n",
      "     macro avg     0.4301    0.4309    0.4274      3153\n",
      "  weighted avg     0.5613    0.5192    0.5375      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4797    0.4876    0.4836       121\n",
      "  enterobacter     0.1307    0.1173    0.1237       196\n",
      "  enterococcus     0.5400    0.5510    0.5455        49\n",
      "   escherichia     0.6837    0.6054    0.6422      1589\n",
      "    klebsiella     0.4528    0.3265    0.3794       588\n",
      "        others     0.0052    0.0408    0.0093        49\n",
      "   pseudomonas     0.5442    0.5792    0.5612       404\n",
      "staphylococcus     0.6667    0.6879    0.6771       157\n",
      "\n",
      "      accuracy                         0.5097      3153\n",
      "     macro avg     0.4379    0.4245    0.4277      3153\n",
      "  weighted avg     0.5669    0.5097    0.5349      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4957    0.4711    0.4831       121\n",
      "  enterobacter     0.1220    0.1020    0.1111       196\n",
      "  enterococcus     0.5532    0.5306    0.5417        49\n",
      "   escherichia     0.6891    0.5941    0.6381      1589\n",
      "    klebsiella     0.4629    0.3180    0.3770       588\n",
      "        others     0.0124    0.1224    0.0225        49\n",
      "   pseudomonas     0.5547    0.5644    0.5595       404\n",
      "staphylococcus     0.6815    0.6815    0.6815       157\n",
      "\n",
      "      accuracy                         0.4995      3153\n",
      "     macro avg     0.4464    0.4230    0.4268      3153\n",
      "  weighted avg     0.5740    0.4995    0.5317      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5000    0.4463    0.4716       121\n",
      "  enterobacter     0.1399    0.1020    0.1180       196\n",
      "  enterococcus     0.5652    0.5306    0.5474        49\n",
      "   escherichia     0.6952    0.5828    0.6340      1589\n",
      "    klebsiella     0.4828    0.3095    0.3772       588\n",
      "        others     0.0167    0.2041    0.0309        49\n",
      "   pseudomonas     0.5653    0.5569    0.5611       404\n",
      "staphylococcus     0.6954    0.6688    0.6818       157\n",
      "\n",
      "      accuracy                         0.4910      3153\n",
      "     macro avg     0.4576    0.4251    0.4278      3153\n",
      "  weighted avg     0.5844    0.4910    0.5301      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5158    0.4050    0.4537       121\n",
      "  enterobacter     0.1280    0.0816    0.0997       196\n",
      "  enterococcus     0.6047    0.5306    0.5652        49\n",
      "   escherichia     0.7038    0.5683    0.6288      1589\n",
      "    klebsiella     0.4887    0.2942    0.3673       588\n",
      "        others     0.0153    0.2245    0.0286        49\n",
      "   pseudomonas     0.5725    0.5470    0.5595       404\n",
      "staphylococcus     0.7123    0.6624    0.6865       157\n",
      "\n",
      "      accuracy                         0.4767      3153\n",
      "     macro avg     0.4676    0.4142    0.4237      3153\n",
      "  weighted avg     0.5921    0.4767    0.5241      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5172    0.3719    0.4327       121\n",
      "  enterobacter     0.1165    0.0612    0.0803       196\n",
      "  enterococcus     0.7027    0.5306    0.6047        49\n",
      "   escherichia     0.7096    0.5412    0.6141      1589\n",
      "    klebsiella     0.5046    0.2772    0.3578       588\n",
      "        others     0.0192    0.3469    0.0364        49\n",
      "   pseudomonas     0.5874    0.5322    0.5584       404\n",
      "staphylococcus     0.7338    0.6497    0.6892       157\n",
      "\n",
      "      accuracy                         0.4567      3153\n",
      "     macro avg     0.4864    0.4139    0.4217      3153\n",
      "  weighted avg     0.6018    0.4567    0.5136      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5200    0.3223    0.3980       121\n",
      "  enterobacter     0.1528    0.0561    0.0821       196\n",
      "  enterococcus     0.7812    0.5102    0.6173        49\n",
      "   escherichia     0.7248    0.5022    0.5933      1589\n",
      "    klebsiella     0.5196    0.2483    0.3360       588\n",
      "        others     0.0188    0.4286    0.0360        49\n",
      "   pseudomonas     0.6118    0.5149    0.5591       404\n",
      "staphylococcus     0.7537    0.6433    0.6942       157\n",
      "\n",
      "      accuracy                         0.4278      3153\n",
      "     macro avg     0.5103    0.4032    0.4145      3153\n",
      "  weighted avg     0.6200    0.4278    0.4984      3153\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** prostt5_relaxed_r3, similarity = 40% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (11654, 1024)\n",
      "Test set shape: (3203, 1024)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2193    0.4607    0.2971        89\n",
      "  enterobacter     0.1267    0.1713    0.1457       216\n",
      "  enterococcus     0.2987    0.5610    0.3898        41\n",
      "   escherichia     0.7137    0.5095    0.5945      1796\n",
      "    klebsiella     0.3343    0.4611    0.3876       527\n",
      "        others     0.0000    0.0000    0.0000        41\n",
      "   pseudomonas     0.3726    0.6302    0.4683       311\n",
      "staphylococcus     0.7768    0.4780    0.5918       182\n",
      "\n",
      "      accuracy                         0.4814      3203\n",
      "     macro avg     0.3553    0.4090    0.3594      3203\n",
      "  weighted avg     0.5540    0.4814    0.4993      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2179    0.4382    0.2910        89\n",
      "  enterobacter     0.1312    0.1713    0.1486       216\n",
      "  enterococcus     0.3099    0.5366    0.3929        41\n",
      "   escherichia     0.7137    0.4983    0.5869      1796\n",
      "    klebsiella     0.3409    0.4554    0.3899       527\n",
      "        others     0.0110    0.0244    0.0152        41\n",
      "   pseudomonas     0.3762    0.6206    0.4684       311\n",
      "staphylococcus     0.7890    0.4725    0.5911       182\n",
      "\n",
      "      accuracy                         0.4724      3203\n",
      "     macro avg     0.3612    0.4022    0.3605      3203\n",
      "  weighted avg     0.5567    0.4724    0.4956      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2281    0.4382    0.3000        89\n",
      "  enterobacter     0.1295    0.1667    0.1457       216\n",
      "  enterococcus     0.3099    0.5366    0.3929        41\n",
      "   escherichia     0.7160    0.4844    0.5779      1796\n",
      "    klebsiella     0.3431    0.4459    0.3878       527\n",
      "        others     0.0054    0.0244    0.0088        41\n",
      "   pseudomonas     0.3862    0.6109    0.4732       311\n",
      "staphylococcus     0.8095    0.4670    0.5923       182\n",
      "\n",
      "      accuracy                         0.4614      3203\n",
      "     macro avg     0.3660    0.3968    0.3598      3203\n",
      "  weighted avg     0.5606    0.4614    0.4907      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2284    0.4157    0.2948        89\n",
      "  enterobacter     0.1255    0.1574    0.1396       216\n",
      "  enterococcus     0.3143    0.5366    0.3964        41\n",
      "   escherichia     0.7134    0.4699    0.5666      1796\n",
      "    klebsiella     0.3415    0.4231    0.3780       527\n",
      "        others     0.0106    0.0732    0.0185        41\n",
      "   pseudomonas     0.3908    0.5981    0.4727       311\n",
      "staphylococcus     0.8173    0.4670    0.5944       182\n",
      "\n",
      "      accuracy                         0.4477      3203\n",
      "     macro avg     0.3677    0.3926    0.3576      3203\n",
      "  weighted avg     0.5596    0.4477    0.4825      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2260    0.3708    0.2809        89\n",
      "  enterobacter     0.1288    0.1574    0.1417       216\n",
      "  enterococcus     0.3016    0.4634    0.3654        41\n",
      "   escherichia     0.7167    0.4549    0.5565      1796\n",
      "    klebsiella     0.3503    0.4175    0.3810       527\n",
      "        others     0.0124    0.1220    0.0226        41\n",
      "   pseudomonas     0.3948    0.5852    0.4715       311\n",
      "staphylococcus     0.8182    0.4451    0.5765       182\n",
      "\n",
      "      accuracy                         0.4343      3203\n",
      "     macro avg     0.3686    0.3770    0.3495      3203\n",
      "  weighted avg     0.5633    0.4343    0.4756      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2313    0.3483    0.2780        89\n",
      "  enterobacter     0.1265    0.1481    0.1365       216\n",
      "  enterococcus     0.3065    0.4634    0.3689        41\n",
      "   escherichia     0.7161    0.4382    0.5437      1796\n",
      "    klebsiella     0.3539    0.4023    0.3766       527\n",
      "        others     0.0134    0.1707    0.0248        41\n",
      "   pseudomonas     0.4050    0.5691    0.4733       311\n",
      "staphylococcus     0.8316    0.4341    0.5704       182\n",
      "\n",
      "      accuracy                         0.4196      3203\n",
      "     macro avg     0.3730    0.3718    0.3465      3203\n",
      "  weighted avg     0.5654    0.4196    0.4671      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2320    0.3258    0.2710        89\n",
      "  enterobacter     0.1198    0.1343    0.1266       216\n",
      "  enterococcus     0.2963    0.3902    0.3368        41\n",
      "   escherichia     0.7129    0.4176    0.5267      1796\n",
      "    klebsiella     0.3542    0.3871    0.3699       527\n",
      "        others     0.0156    0.2439    0.0292        41\n",
      "   pseudomonas     0.4187    0.5627    0.4801       311\n",
      "staphylococcus     0.8495    0.4341    0.5745       182\n",
      "\n",
      "      accuracy                         0.4034      3203\n",
      "     macro avg     0.3749    0.3620    0.3394      3203\n",
      "  weighted avg     0.5655    0.4034    0.4562      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2222    0.2921    0.2524        89\n",
      "  enterobacter     0.1176    0.1296    0.1233       216\n",
      "  enterococcus     0.3077    0.3902    0.3441        41\n",
      "   escherichia     0.7156    0.3909    0.5056      1796\n",
      "    klebsiella     0.3603    0.3719    0.3660       527\n",
      "        others     0.0127    0.2439    0.0242        41\n",
      "   pseudomonas     0.4289    0.5434    0.4794       311\n",
      "staphylococcus     0.8667    0.4286    0.5735       182\n",
      "\n",
      "      accuracy                         0.3825      3203\n",
      "     macro avg     0.3790    0.3488    0.3336      3203\n",
      "  weighted avg     0.5696    0.3825    0.4429      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2170    0.2584    0.2359        89\n",
      "  enterobacter     0.1166    0.1204    0.1185       216\n",
      "  enterococcus     0.3111    0.3415    0.3256        41\n",
      "   escherichia     0.7162    0.3569    0.4764      1796\n",
      "    klebsiella     0.3755    0.3605    0.3679       527\n",
      "        others     0.0113    0.2683    0.0218        41\n",
      "   pseudomonas     0.4328    0.5177    0.4714       311\n",
      "staphylococcus     0.8953    0.4231    0.5746       182\n",
      "\n",
      "      accuracy                         0.3569      3203\n",
      "     macro avg     0.3845    0.3308    0.3240      3203\n",
      "  weighted avg     0.5743    0.3569    0.4251      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.2234    0.2360    0.2295        89\n",
      "  enterobacter     0.1156    0.1065    0.1108       216\n",
      "  enterococcus     0.2703    0.2439    0.2564        41\n",
      "   escherichia     0.7150    0.3185    0.4407      1796\n",
      "    klebsiella     0.3805    0.3264    0.3514       527\n",
      "        others     0.0132    0.3902    0.0255        41\n",
      "   pseudomonas     0.4512    0.4759    0.4632       311\n",
      "staphylococcus     0.9136    0.4066    0.5627       182\n",
      "\n",
      "      accuracy                         0.3234      3203\n",
      "     macro avg     0.3853    0.3130    0.3050      3203\n",
      "  weighted avg     0.5769    0.3234    0.3993      3203\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n"
     ]
    }
   ],
   "source": [
    "models = list(constants.PROSTT5_AA_PLM.keys())\n",
    "\n",
    "for similarity in range(100, 39, -20):\n",
    "    for model in models:\n",
    "        model = model.lower()\n",
    "        df, df_all, protein_clusters = util.filter_proteins_based_on_struct_and_seq_sim(\n",
    "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_{model}.csv\",\n",
    "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_prostt5_relaxed_r3.csv\",\n",
    "            f\"{constants.INPHARED}/{constants.FASTA}/complete-struct-{similarity}.fasta.clstr\",\n",
    "        )\n",
    "\n",
    "        include_proteins_in_cluster = True\n",
    "        if similarity == 100:\n",
    "            include_proteins_in_cluster = False\n",
    "\n",
    "        print(f\"*** {model}, similarity = {similarity}% ***\")\n",
    "        util.classify(\n",
    "            df,\n",
    "            model + \"-mlp-eskapee-smotetomek\",\n",
    "            similarity,\n",
    "            genus=[\n",
    "                \"enterococcus\",\n",
    "                \"staphylococcus\",\n",
    "                \"klebsiella\",\n",
    "                \"acinetobacter\",\n",
    "                \"pseudomonas\",\n",
    "                \"enterobacter\",\n",
    "                \"escherichia\",\n",
    "            ],\n",
    "            feature_columns=[f\"s{i}\" for i in range(1, 1025)],\n",
    "            include_proteins_in_cluster=include_proteins_in_cluster,\n",
    "            rbp_embeddings_all=df_all,\n",
    "            protein_clusters=protein_clusters,\n",
    "            undersample_others=True,\n",
    "            oversample_technique=\"SMOTETomek\",\n",
    "            model=\"MLP-prot\",\n",
    "        )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}