experiments/5.0. Classifier Building & Evaluation (SaProt).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "864700f6",
   "metadata": {},
   "source": [
    "# PHIStruct: Improving phage-host interaction prediction at low sequence similarity settings using structure-aware protein embeddings\n",
    "\n",
    "<b>Mark Edward M. Gonzales<sup>1, 2</sup>, Jennifer C. Ureta<sup>1, 2, 3</sup> & Anish M.S. Shrestha<sup>1, 2</sup></b>\n",
    "\n",
    "<sup>1</sup> Bioinformatics Lab, Advanced Research Institute for Informatics, Computing and Networking, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>2</sup> Department of Software Technology, College of Computer Studies, De La Salle University, Manila 1004, Philippines <br>\n",
    "<sup>3</sup> Walter and Eliza Hall Institute of Medical Research, Melbourne, Victoria, 3052, Australia\n",
    "\n",
    "✉️ gonzales.markedward@gmail.com, jennifer.ureta@gmail.com, anish.shrestha@dlsu.edu.ph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9619d49",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eefb76b",
   "metadata": {},
   "source": [
    "# 💡 Prerequisites\n",
    "\n",
    "### Option 1: Download the prerequisite files\n",
    "1. Download `consolidated.tar.gz` from this [link](https://drive.google.com/file/d/1yQSXwlb37dm2ZLXGJHdIM5vmrzwPAwvI/view?usp=sharing), and unzip it. This should result in a folder named `consolidated`. <br> Technically, this notebook only needs `consolidated/rbp_embeddings_saprot_relaxed_r3.csv`.\n",
    "1. Create a folder named `inphared` inside `data`, and save the extracted `consolidated` folder inside `data/inphared`. \n",
    "1. Download `fasta.tar.gz` from this [link](https://drive.google.com/file/d/1NMFR3JrrrCHLoCMQp2nia4dgtcXs5x05/view?usp=sharing), and unzip it. This should result in a folder named `fasta`. <br> Technically, this notebook only needs the `.clstr` files inside `fasta`.\n",
    "1. Save the extracted `fasta` folder inside `data/inphared`.\n",
    "\n",
    "### Option 2: Generate the prerequisite files yourself\n",
    "1. If you have run `3.0. Data Consolidation (SaProt).ipynb`, then `data/inphared/consolidated` should have already been populated with the prerequisite files.\n",
    "1. Consolidate the sequences of the proteins with predicted structures into a single FASTA file. <br>\n",
    "   For reproducibility, we provide our consolidated FASTA file [here](https://drive.google.com/file/d/1LTZte1f4lreQ5MXWeM-y2Mtp9z96pXS7/view?usp=sharing).\n",
    "1. Generate the protein clusters by running CD-HIT on this FASTA file at a sequence similarity threshold of 100%, following the instructions [here](https://github.com/weizhongli/cdhit). \n",
    "1. Rename the resulting `.clstr` file to `complete-struct-100.fasta.clstr` and the resulting FASTA file (containing only the representative sequences) to `complete-struct-100.fasta`. \n",
    "1. Generate `complete-struct-80.fasta.clstr`, `complete-struct-60.fasta.clstr`, and `complete-struct-40.fasta.clstr` by running CD-HIT on `complete-struct-100.fasta` at sequence similarity thresholds of 80%, 60%, and 40%, respectively.\n",
    "1. Create a folder named `fasta` inside `data/inphared`, and save the four `.clstr` files inside `data/inphared/fasta`.\n",
    "\n",
    "### Resulting folder structure\n",
    "\n",
    "`experiments` (parent folder of this notebook) <br> \n",
    "↳ `data` <br>\n",
    "&nbsp; &nbsp;↳ `inphared` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `consolidated` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `rbp_embeddings_saprot_relaxed_r3.csv` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp; ↳ `fasta` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-100.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-80.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-40.fasta.clstr` <br>\n",
    "&nbsp; &nbsp;&nbsp; &nbsp;&nbsp; &nbsp; ↳ `complete-struct-60.fasta.clstr` <br>\n",
    "↳ `5.0. Classifier Building & Evaluation (SaProt).ipynb` (this notebook) <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3015489",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c059ce7f",
   "metadata": {},
   "source": [
    "# 📁 Output files\n",
    "\n",
    "The output files (i.e., the results of evaluating the model's performance) &mdash; which are saved in `temp/results` &mdash; are already included when the repository was cloned. <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cecf5469",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f274d083",
   "metadata": {},
   "source": [
    "# Part I: Preliminaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc7d25a1",
   "metadata": {},
   "source": [
    "Import the necessary libraries and modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0840061c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "\n",
    "import ConstantsUtil\n",
    "import ClassificationUtil\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "98fb65d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None)\n",
    "pd.set_option(\"display.max_columns\", 50)\n",
    "\n",
    "pd.options.mode.chained_assignment = None\n",
    "\n",
    "with warnings.catch_warnings():\n",
    "    warnings.filterwarnings(\n",
    "        \"ignore\", category=sklearn.exceptions.UndefinedMetricWarning\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e27723c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "constants = ConstantsUtil.ConstantsUtil()\n",
    "util = ClassificationUtil.ClassificationUtil()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ada3031",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9f7cc0",
   "metadata": {},
   "source": [
    "# Part II: Classifier Building and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53d0124d",
   "metadata": {},
   "source": [
    "Train a multilayer perceptron, and evaluate its performance at different train-versus-test similarity and confidence thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0faa8454",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** saprot_relaxed_r3, similarity = 100% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (16942, 1280)\n",
      "Test set shape: (2340, 1280)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8750    0.8198    0.8465       111\n",
      "  enterobacter     0.3543    0.3879    0.3704       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8827    0.8683    0.8754      1040\n",
      "    klebsiella     0.8200    0.8698    0.8442       461\n",
      "        others     0.0000    0.0000    0.0000        51\n",
      "   pseudomonas     0.8642    0.9350    0.8982       354\n",
      "staphylococcus     0.9383    0.9744    0.9560       156\n",
      "\n",
      "      accuracy                         0.8406      2340\n",
      "     macro avg     0.6976    0.7148    0.7056      2340\n",
      "  weighted avg     0.8247    0.8406    0.8321      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9010    0.8198    0.8585       111\n",
      "  enterobacter     0.3697    0.3793    0.3745       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8850    0.8587    0.8716      1040\n",
      "    klebsiella     0.8285    0.8698    0.8487       461\n",
      "        others     0.0278    0.0196    0.0230        51\n",
      "   pseudomonas     0.8757    0.9350    0.9044       354\n",
      "staphylococcus     0.9441    0.9744    0.9590       156\n",
      "\n",
      "      accuracy                         0.8363      2340\n",
      "     macro avg     0.7097    0.7149    0.7118      2340\n",
      "  weighted avg     0.8321    0.8363    0.8338      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9010    0.8198    0.8585       111\n",
      "  enterobacter     0.3717    0.3621    0.3668       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8884    0.8500    0.8688      1040\n",
      "    klebsiella     0.8418    0.8655    0.8535       461\n",
      "        others     0.0149    0.0196    0.0169        51\n",
      "   pseudomonas     0.8780    0.9350    0.9056       354\n",
      "staphylococcus     0.9441    0.9744    0.9590       156\n",
      "\n",
      "      accuracy                         0.8308      2340\n",
      "     macro avg     0.7108    0.7111    0.7104      2340\n",
      "  weighted avg     0.8364    0.8308    0.8331      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9100    0.8198    0.8626       111\n",
      "  enterobacter     0.3846    0.3448    0.3636       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8903    0.8346    0.8615      1040\n",
      "    klebsiella     0.8476    0.8568    0.8522       461\n",
      "        others     0.0183    0.0392    0.0250        51\n",
      "   pseudomonas     0.8827    0.9350    0.9081       354\n",
      "staphylococcus     0.9497    0.9679    0.9587       156\n",
      "\n",
      "      accuracy                         0.8214      2340\n",
      "     macro avg     0.7162    0.7076    0.7108      2340\n",
      "  weighted avg     0.8406    0.8214    0.8302      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9192    0.8198    0.8667       111\n",
      "  enterobacter     0.4124    0.3448    0.3756       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8894    0.8192    0.8529      1040\n",
      "    klebsiella     0.8615    0.8503    0.8559       461\n",
      "        others     0.0329    0.0980    0.0493        51\n",
      "   pseudomonas     0.8946    0.9350    0.9144       354\n",
      "staphylococcus     0.9554    0.9615    0.9585       156\n",
      "\n",
      "      accuracy                         0.8141      2340\n",
      "     macro avg     0.7264    0.7114    0.7159      2340\n",
      "  weighted avg     0.8472    0.8141    0.8293      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9381    0.8198    0.8750       111\n",
      "  enterobacter     0.4023    0.3017    0.3448       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8924    0.8058    0.8469      1040\n",
      "    klebsiella     0.8700    0.8416    0.8556       461\n",
      "        others     0.0303    0.1176    0.0482        51\n",
      "   pseudomonas     0.9016    0.9322    0.9167       354\n",
      "staphylococcus     0.9613    0.9551    0.9582       156\n",
      "\n",
      "      accuracy                         0.8038      2340\n",
      "     macro avg     0.7303    0.7046    0.7125      2340\n",
      "  weighted avg     0.8521    0.8038    0.8258      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9479    0.8198    0.8792       111\n",
      "  enterobacter     0.4074    0.2845    0.3350       116\n",
      "  enterococcus     0.8462    0.8627    0.8544        51\n",
      "   escherichia     0.8925    0.7904    0.8383      1040\n",
      "    klebsiella     0.8790    0.8351    0.8565       461\n",
      "        others     0.0299    0.1373    0.0491        51\n",
      "   pseudomonas     0.9036    0.9266    0.9149       354\n",
      "staphylococcus     0.9613    0.9551    0.9582       156\n",
      "\n",
      "      accuracy                         0.7944      2340\n",
      "     macro avg     0.7335    0.7014    0.7107      2340\n",
      "  weighted avg     0.8549    0.7944    0.8216      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9474    0.8108    0.8738       111\n",
      "  enterobacter     0.4677    0.2500    0.3258       116\n",
      "  enterococcus     0.8431    0.8431    0.8431        51\n",
      "   escherichia     0.8970    0.7702    0.8288      1040\n",
      "    klebsiella     0.8832    0.8200    0.8504       461\n",
      "        others     0.0270    0.1569    0.0461        51\n",
      "   pseudomonas     0.9056    0.9209    0.9132       354\n",
      "staphylococcus     0.9613    0.9551    0.9582       156\n",
      "\n",
      "      accuracy                         0.7795      2340\n",
      "     macro avg     0.7415    0.6909    0.7049      2340\n",
      "  weighted avg     0.8608    0.7795    0.8149      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9556    0.7748    0.8557       111\n",
      "  enterobacter     0.4667    0.2414    0.3182       116\n",
      "  enterococcus     0.8431    0.8431    0.8431        51\n",
      "   escherichia     0.9008    0.7510    0.8191      1040\n",
      "    klebsiella     0.9049    0.8048    0.8519       461\n",
      "        others     0.0283    0.1961    0.0495        51\n",
      "   pseudomonas     0.9183    0.9209    0.9196       354\n",
      "staphylococcus     0.9675    0.9551    0.9613       156\n",
      "\n",
      "      accuracy                         0.7667      2340\n",
      "     macro avg     0.7482    0.6859    0.7023      2340\n",
      "  weighted avg     0.8695    0.7667    0.8109      2340\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.9765    0.7477    0.8469       111\n",
      "  enterobacter     0.5610    0.1983    0.2930       116\n",
      "  enterococcus     0.8400    0.8235    0.8317        51\n",
      "   escherichia     0.9059    0.7038    0.7922      1040\n",
      "    klebsiella     0.9247    0.7722    0.8416       461\n",
      "        others     0.0295    0.2745    0.0532        51\n",
      "   pseudomonas     0.9359    0.9068    0.9211       354\n",
      "staphylococcus     0.9739    0.9551    0.9644       156\n",
      "\n",
      "      accuracy                         0.7350      2340\n",
      "     macro avg     0.7684    0.6728    0.6930      2340\n",
      "  weighted avg     0.8844    0.7350    0.7955      2340\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** saprot_relaxed_r3, similarity = 80% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (14822, 1280)\n",
      "Test set shape: (2822, 1280)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5597    0.6881    0.6173       109\n",
      "  enterobacter     0.2564    0.2395    0.2477       167\n",
      "  enterococcus     0.5455    0.8000    0.6486        60\n",
      "   escherichia     0.7437    0.7493    0.7465      1344\n",
      "    klebsiella     0.6707    0.5903    0.6280       559\n",
      "        others     0.0000    0.0000    0.0000        60\n",
      "   pseudomonas     0.5691    0.6923    0.6247       351\n",
      "staphylococcus     0.9415    0.9360    0.9388       172\n",
      "\n",
      "      accuracy                         0.6747      2822\n",
      "     macro avg     0.5358    0.5869    0.5564      2822\n",
      "  weighted avg     0.6636    0.6747    0.6671      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5682    0.6881    0.6224       109\n",
      "  enterobacter     0.2621    0.2275    0.2436       167\n",
      "  enterococcus     0.5465    0.7833    0.6438        60\n",
      "   escherichia     0.7479    0.7396    0.7437      1344\n",
      "    klebsiella     0.6751    0.5725    0.6196       559\n",
      "        others     0.0000    0.0000    0.0000        60\n",
      "   pseudomonas     0.5793    0.6866    0.6284       351\n",
      "staphylococcus     0.9471    0.9360    0.9415       172\n",
      "\n",
      "      accuracy                         0.6648      2822\n",
      "     macro avg     0.5408    0.5792    0.5554      2822\n",
      "  weighted avg     0.6688    0.6648    0.6646      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5725    0.6881    0.6250       109\n",
      "  enterobacter     0.2754    0.2275    0.2492       167\n",
      "  enterococcus     0.5465    0.7833    0.6438        60\n",
      "   escherichia     0.7468    0.7307    0.7386      1344\n",
      "    klebsiella     0.6930    0.5653    0.6227       559\n",
      "        others     0.0083    0.0167    0.0111        60\n",
      "   pseudomonas     0.5848    0.6781    0.6280       351\n",
      "staphylococcus     0.9527    0.9360    0.9443       172\n",
      "\n",
      "      accuracy                         0.6584      2822\n",
      "     macro avg     0.5475    0.5782    0.5578      2822\n",
      "  weighted avg     0.6739    0.6584    0.6636      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5952    0.6881    0.6383       109\n",
      "  enterobacter     0.2891    0.2216    0.2508       167\n",
      "  enterococcus     0.5412    0.7667    0.6345        60\n",
      "   escherichia     0.7479    0.7217    0.7346      1344\n",
      "    klebsiella     0.6996    0.5581    0.6209       559\n",
      "        others     0.0112    0.0333    0.0168        60\n",
      "   pseudomonas     0.5934    0.6695    0.6292       351\n",
      "staphylococcus     0.9699    0.9360    0.9527       172\n",
      "\n",
      "      accuracy                         0.6513      2822\n",
      "     macro avg     0.5559    0.5744    0.5597      2822\n",
      "  weighted avg     0.6795    0.6513    0.6625      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6198    0.6881    0.6522       109\n",
      "  enterobacter     0.2759    0.1916    0.2261       167\n",
      "  enterococcus     0.5488    0.7500    0.6338        60\n",
      "   escherichia     0.7478    0.7083    0.7276      1344\n",
      "    klebsiella     0.7041    0.5492    0.6171       559\n",
      "        others     0.0124    0.0500    0.0199        60\n",
      "   pseudomonas     0.6005    0.6638    0.6306       351\n",
      "staphylococcus     0.9697    0.9302    0.9496       172\n",
      "\n",
      "      accuracy                         0.6403      2822\n",
      "     macro avg     0.5599    0.5664    0.5571      2822\n",
      "  weighted avg     0.6816    0.6403    0.6575      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6218    0.6789    0.6491       109\n",
      "  enterobacter     0.2925    0.1856    0.2271       167\n",
      "  enterococcus     0.5488    0.7500    0.6338        60\n",
      "   escherichia     0.7486    0.6890    0.7176      1344\n",
      "    klebsiella     0.7180    0.5420    0.6177       559\n",
      "        others     0.0228    0.1167    0.0381        60\n",
      "   pseudomonas     0.6042    0.6610    0.6313       351\n",
      "staphylococcus     0.9697    0.9302    0.9496       172\n",
      "\n",
      "      accuracy                         0.6300      2822\n",
      "     macro avg     0.5658    0.5692    0.5580      2822\n",
      "  weighted avg     0.6865    0.6300    0.6533      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6325    0.6789    0.6549       109\n",
      "  enterobacter     0.3100    0.1856    0.2322       167\n",
      "  enterococcus     0.5488    0.7500    0.6338        60\n",
      "   escherichia     0.7523    0.6711    0.7094      1344\n",
      "    klebsiella     0.7209    0.5313    0.6117       559\n",
      "        others     0.0243    0.1500    0.0418        60\n",
      "   pseudomonas     0.6090    0.6524    0.6300       351\n",
      "staphylococcus     0.9697    0.9302    0.9496       172\n",
      "\n",
      "      accuracy                         0.6191      2822\n",
      "     macro avg     0.5709    0.5687    0.5579      2822\n",
      "  weighted avg     0.6909    0.6191    0.6487      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6435    0.6789    0.6607       109\n",
      "  enterobacter     0.3261    0.1796    0.2317       167\n",
      "  enterococcus     0.5625    0.7500    0.6429        60\n",
      "   escherichia     0.7578    0.6496    0.6995      1344\n",
      "    klebsiella     0.7442    0.5206    0.6126       559\n",
      "        others     0.0238    0.1833    0.0421        60\n",
      "   pseudomonas     0.6185    0.6467    0.6323       351\n",
      "staphylococcus     0.9755    0.9244    0.9493       172\n",
      "\n",
      "      accuracy                         0.6060      2822\n",
      "     macro avg     0.5815    0.5666    0.5589      2822\n",
      "  weighted avg     0.7013    0.6060    0.6448      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6765    0.6330    0.6540       109\n",
      "  enterobacter     0.3218    0.1677    0.2205       167\n",
      "  enterococcus     0.5513    0.7167    0.6232        60\n",
      "   escherichia     0.7636    0.6272    0.6887      1344\n",
      "    klebsiella     0.7649    0.5063    0.6093       559\n",
      "        others     0.0228    0.2167    0.0413        60\n",
      "   pseudomonas     0.6314    0.6296    0.6305       351\n",
      "staphylococcus     0.9815    0.9244    0.9521       172\n",
      "\n",
      "      accuracy                         0.5879      2822\n",
      "     macro avg     0.5892    0.5527    0.5525      2822\n",
      "  weighted avg     0.7109    0.5879    0.6376      2822\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6939    0.6239    0.6570       109\n",
      "  enterobacter     0.3425    0.1497    0.2083       167\n",
      "  enterococcus     0.5915    0.7000    0.6412        60\n",
      "   escherichia     0.7745    0.5878    0.6684      1344\n",
      "    klebsiella     0.7964    0.4758    0.5957       559\n",
      "        others     0.0254    0.3167    0.0471        60\n",
      "   pseudomonas     0.6573    0.6011    0.6280       351\n",
      "staphylococcus     0.9810    0.9012    0.9394       172\n",
      "\n",
      "      accuracy                         0.5585      2822\n",
      "     macro avg     0.6078    0.5445    0.5481      2822\n",
      "  weighted avg     0.7284    0.5585    0.6240      2822\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** saprot_relaxed_r3, similarity = 60% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (13099, 1280)\n",
      "Test set shape: (3153, 1280)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6090    0.7851    0.6859       121\n",
      "  enterobacter     0.2136    0.2245    0.2189       196\n",
      "  enterococcus     0.7193    0.8367    0.7736        49\n",
      "   escherichia     0.7362    0.7306    0.7334      1589\n",
      "    klebsiella     0.5019    0.4507    0.4749       588\n",
      "        others     0.0000    0.0000    0.0000        49\n",
      "   pseudomonas     0.6645    0.7649    0.7112       404\n",
      "staphylococcus     0.7805    0.8153    0.7975       157\n",
      "\n",
      "      accuracy                         0.6480      3153\n",
      "     macro avg     0.5281    0.5760    0.5494      3153\n",
      "  weighted avg     0.6365    0.6480    0.6410      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6225    0.7769    0.6912       121\n",
      "  enterobacter     0.2204    0.2092    0.2147       196\n",
      "  enterococcus     0.7143    0.8163    0.7619        49\n",
      "   escherichia     0.7389    0.7105    0.7244      1589\n",
      "    klebsiella     0.5088    0.4422    0.4732       588\n",
      "        others     0.0098    0.0204    0.0132        49\n",
      "   pseudomonas     0.6732    0.7599    0.7140       404\n",
      "staphylococcus     0.7853    0.8153    0.8000       157\n",
      "\n",
      "      accuracy                         0.6343      3153\n",
      "     macro avg     0.5342    0.5688    0.5491      3153\n",
      "  weighted avg     0.6415    0.6343    0.6365      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6395    0.7769    0.7015       121\n",
      "  enterobacter     0.2335    0.1990    0.2149       196\n",
      "  enterococcus     0.7407    0.8163    0.7767        49\n",
      "   escherichia     0.7423    0.6979    0.7194      1589\n",
      "    klebsiella     0.5170    0.4388    0.4747       588\n",
      "        others     0.0161    0.0612    0.0255        49\n",
      "   pseudomonas     0.6847    0.7525    0.7170       404\n",
      "staphylococcus     0.7901    0.8153    0.8025       157\n",
      "\n",
      "      accuracy                         0.6264      3153\n",
      "     macro avg     0.5455    0.5697    0.5540      3153\n",
      "  weighted avg     0.6484    0.6264    0.6357      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6408    0.7521    0.6920       121\n",
      "  enterobacter     0.2450    0.1888    0.2133       196\n",
      "  enterococcus     0.7358    0.7959    0.7647        49\n",
      "   escherichia     0.7448    0.6816    0.7118      1589\n",
      "    klebsiella     0.5267    0.4354    0.4767       588\n",
      "        others     0.0182    0.1020    0.0310        49\n",
      "   pseudomonas     0.6982    0.7500    0.7232       404\n",
      "staphylococcus     0.8050    0.8153    0.8101       157\n",
      "\n",
      "      accuracy                         0.6159      3153\n",
      "     macro avg     0.5518    0.5651    0.5528      3153\n",
      "  weighted avg     0.6547    0.6159    0.6328      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6522    0.7438    0.6950       121\n",
      "  enterobacter     0.2537    0.1735    0.2061       196\n",
      "  enterococcus     0.7358    0.7959    0.7647        49\n",
      "   escherichia     0.7477    0.6677    0.7055      1589\n",
      "    klebsiella     0.5338    0.4303    0.4765       588\n",
      "        others     0.0171    0.1224    0.0300        49\n",
      "   pseudomonas     0.7073    0.7475    0.7268       404\n",
      "staphylococcus     0.8153    0.8153    0.8153       157\n",
      "\n",
      "      accuracy                         0.6067      3153\n",
      "     macro avg     0.5579    0.5621    0.5525      3153\n",
      "  weighted avg     0.6601    0.6067    0.6299      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6642    0.7355    0.6980       121\n",
      "  enterobacter     0.2521    0.1531    0.1905       196\n",
      "  enterococcus     0.7500    0.7959    0.7723        49\n",
      "   escherichia     0.7546    0.6501    0.6984      1589\n",
      "    klebsiella     0.5363    0.4150    0.4679       588\n",
      "        others     0.0200    0.1837    0.0361        49\n",
      "   pseudomonas     0.7136    0.7401    0.7266       404\n",
      "staphylococcus     0.8205    0.8153    0.8179       157\n",
      "\n",
      "      accuracy                         0.5934      3153\n",
      "     macro avg     0.5639    0.5611    0.5510      3153\n",
      "  weighted avg     0.6657    0.5934    0.6243      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6953    0.7355    0.7149       121\n",
      "  enterobacter     0.2667    0.1429    0.1860       196\n",
      "  enterococcus     0.7647    0.7959    0.7800        49\n",
      "   escherichia     0.7602    0.6344    0.6916      1589\n",
      "    klebsiella     0.5336    0.3912    0.4514       588\n",
      "        others     0.0181    0.2041    0.0332        49\n",
      "   pseudomonas     0.7302    0.7302    0.7302       404\n",
      "staphylococcus     0.8194    0.8089    0.8141       157\n",
      "\n",
      "      accuracy                         0.5791      3153\n",
      "     macro avg     0.5735    0.5554    0.5502      3153\n",
      "  weighted avg     0.6724    0.5791    0.6185      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.7131    0.7190    0.7160       121\n",
      "  enterobacter     0.2824    0.1224    0.1708       196\n",
      "  enterococcus     0.7800    0.7959    0.7879        49\n",
      "   escherichia     0.7659    0.6136    0.6813      1589\n",
      "    klebsiella     0.5340    0.3741    0.4400       588\n",
      "        others     0.0180    0.2449    0.0336        49\n",
      "   pseudomonas     0.7372    0.7153    0.7261       404\n",
      "staphylococcus     0.8247    0.8089    0.8167       157\n",
      "\n",
      "      accuracy                         0.5623      3153\n",
      "     macro avg     0.5819    0.5493    0.5466      3153\n",
      "  weighted avg     0.6784    0.5623    0.6100      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.7565    0.7190    0.7373       121\n",
      "  enterobacter     0.3485    0.1173    0.1756       196\n",
      "  enterococcus     0.7959    0.7959    0.7959        49\n",
      "   escherichia     0.7727    0.5733    0.6582      1589\n",
      "    klebsiella     0.5364    0.3384    0.4150       588\n",
      "        others     0.0176    0.3061    0.0334        49\n",
      "   pseudomonas     0.7622    0.6980    0.7287       404\n",
      "staphylococcus     0.8301    0.8089    0.8194       157\n",
      "\n",
      "      accuracy                         0.5338      3153\n",
      "     macro avg     0.6025    0.5446    0.5454      3153\n",
      "  weighted avg     0.6918    0.5338    0.5954      3153\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.8119    0.6777    0.7387       121\n",
      "  enterobacter     0.3958    0.0969    0.1557       196\n",
      "  enterococcus     0.8409    0.7551    0.7957        49\n",
      "   escherichia     0.7856    0.5236    0.6284      1589\n",
      "    klebsiella     0.5506    0.2959    0.3850       588\n",
      "        others     0.0173    0.3878    0.0331        49\n",
      "   pseudomonas     0.7976    0.6634    0.7243       404\n",
      "staphylococcus     0.8411    0.8089    0.8247       157\n",
      "\n",
      "      accuracy                         0.4941      3153\n",
      "     macro avg     0.6301    0.5262    0.5357      3153\n",
      "  weighted avg     0.7118    0.4941    0.5733      3153\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n",
      "*** saprot_relaxed_r3, similarity = 40% ***\n",
      "Constructing training and test sets...\n",
      "Training set shape: (11662, 1280)\n",
      "Test set shape: (3203, 1280)\n",
      "Training the model...\n",
      "Saving evaluation results...\n",
      "Confidence threshold k: 0.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4394    0.6517    0.5249        89\n",
      "  enterobacter     0.4000    0.1019    0.1624       216\n",
      "  enterococcus     0.6190    0.9512    0.7500        41\n",
      "   escherichia     0.7716    0.7166    0.7431      1796\n",
      "    klebsiella     0.4231    0.4801    0.4498       527\n",
      "        others     0.0000    0.0000    0.0000        41\n",
      "   pseudomonas     0.4411    0.7460    0.5544       311\n",
      "staphylococcus     0.8696    0.7692    0.8163       182\n",
      "\n",
      "      accuracy                         0.6341      3203\n",
      "     macro avg     0.4955    0.5521    0.5001      3203\n",
      "  weighted avg     0.6416    0.6341    0.6260      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 10.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4640    0.6517    0.5421        89\n",
      "  enterobacter     0.4200    0.0972    0.1579       216\n",
      "  enterococcus     0.6500    0.9512    0.7723        41\n",
      "   escherichia     0.7734    0.6993    0.7345      1796\n",
      "    klebsiella     0.4323    0.4725    0.4515       527\n",
      "        others     0.0215    0.0488    0.0299        41\n",
      "   pseudomonas     0.4505    0.7460    0.5617       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.6232      3203\n",
      "     macro avg     0.5101    0.5538    0.5078      3203\n",
      "  weighted avg     0.6477    0.6232    0.6228      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 20.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4661    0.6180    0.5314        89\n",
      "  enterobacter     0.4000    0.0833    0.1379       216\n",
      "  enterococcus     0.6667    0.9268    0.7755        41\n",
      "   escherichia     0.7755    0.6904    0.7305      1796\n",
      "    klebsiella     0.4475    0.4611    0.4542       527\n",
      "        others     0.0275    0.1220    0.0448        41\n",
      "   pseudomonas     0.4569    0.7331    0.5630       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.6138      3203\n",
      "     macro avg     0.5136    0.5498    0.5063      3203\n",
      "  weighted avg     0.6510    0.6138    0.6198      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4602    0.5843    0.5149        89\n",
      "  enterobacter     0.4250    0.0787    0.1328       216\n",
      "  enterococcus     0.6786    0.9268    0.7835        41\n",
      "   escherichia     0.7798    0.6804    0.7267      1796\n",
      "    klebsiella     0.4624    0.4554    0.4589       527\n",
      "        others     0.0268    0.1707    0.0464        41\n",
      "   pseudomonas     0.4641    0.7267    0.5664       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.6060      3203\n",
      "     macro avg     0.5207    0.5483    0.5053      3203\n",
      "  weighted avg     0.6583    0.6060    0.6181      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4630    0.5618    0.5076        89\n",
      "  enterobacter     0.4211    0.0741    0.1260       216\n",
      "  enterococcus     0.7037    0.9268    0.8000        41\n",
      "   escherichia     0.7835    0.6670    0.7206      1796\n",
      "    klebsiella     0.4784    0.4402    0.4585       527\n",
      "        others     0.0195    0.1707    0.0350        41\n",
      "   pseudomonas     0.4745    0.7170    0.5711       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.5941      3203\n",
      "     macro avg     0.5265    0.5402    0.5040      3203\n",
      "  weighted avg     0.6640    0.5941    0.6144      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4808    0.5618    0.5181        89\n",
      "  enterobacter     0.4444    0.0741    0.1270       216\n",
      "  enterococcus     0.7308    0.9268    0.8172        41\n",
      "   escherichia     0.7881    0.6542    0.7149      1796\n",
      "    klebsiella     0.4955    0.4137    0.4509       527\n",
      "        others     0.0210    0.2439    0.0387        41\n",
      "   pseudomonas     0.4944    0.7074    0.5820       311\n",
      "staphylococcus     0.8679    0.7582    0.8094       182\n",
      "\n",
      "      accuracy                         0.5823      3203\n",
      "     macro avg     0.5404    0.5425    0.5073      3203\n",
      "  weighted avg     0.6737    0.5823    0.6115      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5000    0.5506    0.5241        89\n",
      "  enterobacter     0.4688    0.0694    0.1210       216\n",
      "  enterococcus     0.7451    0.9268    0.8261        41\n",
      "   escherichia     0.7946    0.6442    0.7116      1796\n",
      "    klebsiella     0.5121    0.4023    0.4506       527\n",
      "        others     0.0194    0.2683    0.0361        41\n",
      "   pseudomonas     0.5082    0.6977    0.5881       311\n",
      "staphylococcus     0.8662    0.7473    0.8024       182\n",
      "\n",
      "      accuracy                         0.5729      3203\n",
      "     macro avg     0.5518    0.5383    0.5075      3203\n",
      "  weighted avg     0.6837    0.5729    0.6096      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5169    0.5169    0.5169        89\n",
      "  enterobacter     0.5172    0.0694    0.1224       216\n",
      "  enterococcus     0.7917    0.9268    0.8539        41\n",
      "   escherichia     0.8010    0.6320    0.7065      1796\n",
      "    klebsiella     0.5324    0.3738    0.4392       527\n",
      "        others     0.0202    0.3415    0.0381        41\n",
      "   pseudomonas     0.5275    0.6785    0.5935       311\n",
      "staphylococcus     0.8718    0.7473    0.8047       182\n",
      "\n",
      "      accuracy                         0.5595      3203\n",
      "     macro avg     0.5723    0.5358    0.5094      3203\n",
      "  weighted avg     0.6971    0.5595    0.6058      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5610    0.5169    0.5380        89\n",
      "  enterobacter     0.5500    0.0509    0.0932       216\n",
      "  enterococcus     0.7917    0.9268    0.8539        41\n",
      "   escherichia     0.8161    0.6030    0.6936      1796\n",
      "    klebsiella     0.5768    0.3491    0.4350       527\n",
      "        others     0.0181    0.3902    0.0346        41\n",
      "   pseudomonas     0.5574    0.6559    0.6027       311\n",
      "staphylococcus     0.8718    0.7473    0.8047       182\n",
      "\n",
      "      accuracy                         0.5364      3203\n",
      "     macro avg     0.5929    0.5300    0.5070      3203\n",
      "  weighted avg     0.7192    0.5364    0.5973      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6250    0.5056    0.5590        89\n",
      "  enterobacter     0.5000    0.0278    0.0526       216\n",
      "  enterococcus     0.8222    0.9024    0.8605        41\n",
      "   escherichia     0.8268    0.5635    0.6702      1796\n",
      "    klebsiella     0.6324    0.3036    0.4103       527\n",
      "        others     0.0152    0.4146    0.0293        41\n",
      "   pseudomonas     0.5859    0.6141    0.5997       311\n",
      "staphylococcus     0.8675    0.7198    0.7868       182\n",
      "\n",
      "      accuracy                         0.4992      3203\n",
      "     macro avg     0.6094    0.5064    0.4960      3203\n",
      "  weighted avg     0.7356    0.4992    0.5767      3203\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confidence threshold k: 30.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4602    0.5843    0.5149        89\n",
      "  enterobacter     0.4250    0.0787    0.1328       216\n",
      "  enterococcus     0.6786    0.9268    0.7835        41\n",
      "   escherichia     0.7798    0.6804    0.7267      1796\n",
      "    klebsiella     0.4624    0.4554    0.4589       527\n",
      "        others     0.0268    0.1707    0.0464        41\n",
      "   pseudomonas     0.4641    0.7267    0.5664       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.6060      3203\n",
      "     macro avg     0.5207    0.5483    0.5053      3203\n",
      "  weighted avg     0.6583    0.6060    0.6181      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 40.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4630    0.5618    0.5076        89\n",
      "  enterobacter     0.4211    0.0741    0.1260       216\n",
      "  enterococcus     0.7037    0.9268    0.8000        41\n",
      "   escherichia     0.7835    0.6670    0.7206      1796\n",
      "    klebsiella     0.4784    0.4402    0.4585       527\n",
      "        others     0.0195    0.1707    0.0350        41\n",
      "   pseudomonas     0.4745    0.7170    0.5711       311\n",
      "staphylococcus     0.8688    0.7637    0.8129       182\n",
      "\n",
      "      accuracy                         0.5941      3203\n",
      "     macro avg     0.5265    0.5402    0.5040      3203\n",
      "  weighted avg     0.6640    0.5941    0.6144      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 50.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.4808    0.5618    0.5181        89\n",
      "  enterobacter     0.4444    0.0741    0.1270       216\n",
      "  enterococcus     0.7308    0.9268    0.8172        41\n",
      "   escherichia     0.7881    0.6542    0.7149      1796\n",
      "    klebsiella     0.4955    0.4137    0.4509       527\n",
      "        others     0.0210    0.2439    0.0387        41\n",
      "   pseudomonas     0.4944    0.7074    0.5820       311\n",
      "staphylococcus     0.8679    0.7582    0.8094       182\n",
      "\n",
      "      accuracy                         0.5823      3203\n",
      "     macro avg     0.5404    0.5425    0.5073      3203\n",
      "  weighted avg     0.6737    0.5823    0.6115      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 60.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5000    0.5506    0.5241        89\n",
      "  enterobacter     0.4688    0.0694    0.1210       216\n",
      "  enterococcus     0.7451    0.9268    0.8261        41\n",
      "   escherichia     0.7946    0.6442    0.7116      1796\n",
      "    klebsiella     0.5121    0.4023    0.4506       527\n",
      "        others     0.0194    0.2683    0.0361        41\n",
      "   pseudomonas     0.5082    0.6977    0.5881       311\n",
      "staphylococcus     0.8662    0.7473    0.8024       182\n",
      "\n",
      "      accuracy                         0.5729      3203\n",
      "     macro avg     0.5518    0.5383    0.5075      3203\n",
      "  weighted avg     0.6837    0.5729    0.6096      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 70.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5169    0.5169    0.5169        89\n",
      "  enterobacter     0.5172    0.0694    0.1224       216\n",
      "  enterococcus     0.7917    0.9268    0.8539        41\n",
      "   escherichia     0.8010    0.6320    0.7065      1796\n",
      "    klebsiella     0.5324    0.3738    0.4392       527\n",
      "        others     0.0202    0.3415    0.0381        41\n",
      "   pseudomonas     0.5275    0.6785    0.5935       311\n",
      "staphylococcus     0.8718    0.7473    0.8047       182\n",
      "\n",
      "      accuracy                         0.5595      3203\n",
      "     macro avg     0.5723    0.5358    0.5094      3203\n",
      "  weighted avg     0.6971    0.5595    0.6058      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 80.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.5610    0.5169    0.5380        89\n",
      "  enterobacter     0.5500    0.0509    0.0932       216\n",
      "  enterococcus     0.7917    0.9268    0.8539        41\n",
      "   escherichia     0.8161    0.6030    0.6936      1796\n",
      "    klebsiella     0.5768    0.3491    0.4350       527\n",
      "        others     0.0181    0.3902    0.0346        41\n",
      "   pseudomonas     0.5574    0.6559    0.6027       311\n",
      "staphylococcus     0.8718    0.7473    0.8047       182\n",
      "\n",
      "      accuracy                         0.5364      3203\n",
      "     macro avg     0.5929    0.5300    0.5070      3203\n",
      "  weighted avg     0.7192    0.5364    0.5973      3203\n",
      "\n",
      "===================\n",
      "Confidence threshold k: 90.0%\n",
      "                precision    recall  f1-score   support\n",
      "\n",
      " acinetobacter     0.6250    0.5056    0.5590        89\n",
      "  enterobacter     0.5000    0.0278    0.0526       216\n",
      "  enterococcus     0.8222    0.9024    0.8605        41\n",
      "   escherichia     0.8268    0.5635    0.6702      1796\n",
      "    klebsiella     0.6324    0.3036    0.4103       527\n",
      "        others     0.0152    0.4146    0.0293        41\n",
      "   pseudomonas     0.5859    0.6141    0.5997       311\n",
      "staphylococcus     0.8675    0.7198    0.7868       182\n",
      "\n",
      "      accuracy                         0.4992      3203\n",
      "     macro avg     0.6094    0.5064    0.4960      3203\n",
      "  weighted avg     0.7356    0.4992    0.5767      3203\n",
      "\n",
      "===================\n",
      "Finished\n",
      "===================\n"
     ]
    }
   ],
   "source": [
    "models = list(constants.SAPROT_PLM.keys())\n",
    "\n",
    "for similarity in range(100, 39, -20):\n",
    "    for model in models:\n",
    "        model = model.lower()\n",
    "        df, df_all, protein_clusters = util.filter_proteins_based_on_struct_and_seq_sim(\n",
    "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_{model}.csv\",\n",
    "            f\"{constants.INPHARED}/{constants.CONSOLIDATED}/rbp_embeddings_saprot_relaxed_r3.csv\",\n",
    "            f\"{constants.INPHARED}/{constants.FASTA}/complete-struct-{similarity}.fasta.clstr\",\n",
    "        )\n",
    "\n",
    "        include_proteins_in_cluster = True\n",
    "        if similarity == 100:\n",
    "            include_proteins_in_cluster = False\n",
    "\n",
    "        print(f\"*** {model}, similarity = {similarity}% ***\")\n",
    "        util.classify(\n",
    "            df,\n",
    "            model + \"-mlp-eskapee-smotetomek\",\n",
    "            similarity,\n",
    "            genus=[\n",
    "                \"enterococcus\",\n",
    "                \"staphylococcus\",\n",
    "                \"klebsiella\",\n",
    "                \"acinetobacter\",\n",
    "                \"pseudomonas\",\n",
    "                \"enterobacter\",\n",
    "                \"escherichia\",\n",
    "            ],\n",
    "            feature_columns=[f\"s{i}\" for i in range(1, 1281)],\n",
    "            include_proteins_in_cluster=include_proteins_in_cluster,\n",
    "            rbp_embeddings_all=df_all,\n",
    "            protein_clusters=protein_clusters,\n",
    "            undersample_others=True,\n",
    "            oversample_technique=\"SMOTETomek\",\n",
    "            model=\"MLP\",\n",
    "        )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}