3_BERT_multiclass_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "33a1903b-67f0-4566-b540-1f0d149d0077",
   "metadata": {},
   "source": [
    "# Using BERT to Identify the Narrators (Multiclass Classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05f5bc0b-1604-4202-a258-ff346af633cd",
   "metadata": {},
   "source": [
    "[The previous investigation indicates that binary classification is suitable for detecting pages belonging to Bill Hardy](./2_BERT_binary_classification_bill.ipynb). \n",
    "\n",
    "Can this approach be extended to all narrators using multiclass classification?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a2613187-fe07-4d6d-a411-472b4f0d0c49",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "# make logs less verbose\n",
    "import os\n",
    "\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"2\"\n",
    "\n",
    "# suppress UnparsedFlagAccessError warning\n",
    "import sys\n",
    "from absl import flags\n",
    "\n",
    "sys.argv = [\"preserve_unused_tokens=False\"]\n",
    "flags.FLAGS(sys.argv)\n",
    "\n",
    "# suppress warnings\n",
    "# (after having ensured they did not impact the analysis)\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99aa2a8f-2c77-4955-ba95-78e8027b5bae",
   "metadata": {},
   "source": [
    "## Process data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "108404f2-a14a-4dc4-af6c-5924f3c4d3b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_json(\"./data/jawbone.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6453c0e5-bd61-407c-9023-5d788086c8fe",
   "metadata": {},
   "source": [
    "Augment the data with the narrator labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "83336e1a-e7ba-4b0c-b145-7751808d8ee7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>page</th>\n",
       "      <th>text</th>\n",
       "      <th>narrator</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>I sit down alone at the appointed table and ta...</td>\n",
       "      <td>Bill Hardy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>I plunged for the last time. The few remaining...</td>\n",
       "      <td>Oscar Mills</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>At my meeting with Clement yesterday, he had b...</td>\n",
       "      <td>John Walker</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>And I really think I would have preferred the ...</td>\n",
       "      <td>Bill Hardy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>I hated my eye for being caught by what didn’t...</td>\n",
       "      <td>May Doncaster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>96</td>\n",
       "      <td>Now I think I will try a cup of what they inso...</td>\n",
       "      <td>Bill Hardy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>97</td>\n",
       "      <td>To have slept and to wake right up surrounded ...</td>\n",
       "      <td>John Walker</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>98</td>\n",
       "      <td>He stood and looked down at me; but I was not ...</td>\n",
       "      <td>Sir Paul Trinder</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>99</td>\n",
       "      <td>And she wore a mauve love-knot on her breast, ...</td>\n",
       "      <td>Henry (dog)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>100</td>\n",
       "      <td>I dimly guess why the old dead so wanted this....</td>\n",
       "      <td>Bill Hardy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    page                                               text          narrator\n",
       "0      1  I sit down alone at the appointed table and ta...        Bill Hardy\n",
       "1      2  I plunged for the last time. The few remaining...       Oscar Mills\n",
       "2      3  At my meeting with Clement yesterday, he had b...       John Walker\n",
       "3      4  And I really think I would have preferred the ...        Bill Hardy\n",
       "4      5  I hated my eye for being caught by what didn’t...     May Doncaster\n",
       "..   ...                                                ...               ...\n",
       "95    96  Now I think I will try a cup of what they inso...        Bill Hardy\n",
       "96    97  To have slept and to wake right up surrounded ...       John Walker\n",
       "97    98  He stood and looked down at me; but I was not ...  Sir Paul Trinder\n",
       "98    99  And she wore a mauve love-knot on her breast, ...       Henry (dog)\n",
       "99   100  I dimly guess why the old dead so wanted this....        Bill Hardy\n",
       "\n",
       "[100 rows x 3 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "f = open(\"./data/unofficial_solution.json\")\n",
    "possible_solutions = json.load(f)\n",
    "f.close()\n",
    "\n",
    "\n",
    "def get_narrator(id):\n",
    "    for key, ids in possible_solutions.items():\n",
    "        if id in ids:\n",
    "            return key\n",
    "    return None\n",
    "\n",
    "\n",
    "df = df.astype({\"text\": \"string\"})\n",
    "df[\"narrator\"] = df[\"page\"].apply(lambda x: get_narrator(x))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5a39083-36ff-4205-86cf-9b45ee6f1ea8",
   "metadata": {},
   "source": [
    "Combining the pages and the suggested solution results in a series of numbered pages with a label indicating the narrators.\n",
    "\n",
    "The data is now labeled."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6af53b8c-59f5-4077-bc2b-a7991f4a45c2",
   "metadata": {},
   "source": [
    "## Train BERT Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d427fb78-c405-46c5-86cc-03823d4a8d2c",
   "metadata": {},
   "source": [
    "We are still using [BERT<sub>BASE</sub> uncased](https://huggingface.co/bert-base-uncased)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3fee4c4e-c542-40dc-b70c-f4584ff63a20",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertForSequenceClassification\n",
    "from transformers import logging\n",
    "\n",
    "logging.set_verbosity_error()  # suppress untrained weight error (unconsequential)\n",
    "\n",
    "# use BERT base uncased model\n",
    "# https://huggingface.co/bert-base-uncased\n",
    "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n",
    "bert_model = BertForSequenceClassification.from_pretrained(\n",
    "    \"bert-base-uncased\", num_labels=8\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2b06bb-6e39-442e-9377-85d2ecd0f0e1",
   "metadata": {},
   "source": [
    "The number of epochs is increased (given the complexity of the classification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "aa06971b-697f-47ba-a3b3-cf890f558e47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# hyperparameters\n",
    "\n",
    "num_epochs = 100\n",
    "learning_rate = 2e-5\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a425a9e8-cd38-4657-8868-cf5eb60b6af3",
   "metadata": {},
   "source": [
    "The book is sampled for training and testing in the following proportion: **5 pages per narrator**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "54e1db73-4020-4ddb-8f28-78e91dd2cc60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             page                                               text  narrator\n",
      "narrator                                                                      \n",
      "0        0      1  I sit down alone at the appointed table and ta...         0\n",
      "         99   100  I dimly guess why the old dead so wanted this....         0\n",
      "         86    87  Of the old bold mate of Henry Morgan. My guest...         0\n",
      "         83    84  I wish she would tell me more. I wish she woul...         0\n",
      "         16    17  I am conveniently situated, with the Moon on t...         0\n",
      "1        5      6  I considered that venerable whose winter Achil...         1\n",
      "         70    71  But next day that religious fellow’s head drap...         1\n",
      "         85    86  Next day I let Caroline Jasmine—what a name!—d...         1\n",
      "         20    21  To reckon with Henry! That was never easy. Jus...         1\n",
      "         79    80  Hospitality, when I came to consider it, was i...         1\n",
      "2        63    64  Needless to say I didn’t know that that was th...         2\n",
      "         18    19  Yesterday he got in another of his own kind, w...         2\n",
      "         47    48  I had always thought that Tate essentially mea...         2\n",
      "         38    39  She said it didn’t matter what they had done, ...         2\n",
      "         78    79  It was that day my friend Sandy told me he was...         2\n",
      "3        40    41  And then with horrid clearness I had seen a wo...         3\n",
      "         6      7  I had gone to sleep the night before after rer...         3\n",
      "         14    15  Now, I considered, in my dear Lyons it would b...         3\n",
      "         81    82  I forgot why I was sitting and staring at the ...         3\n",
      "         94    95  I knew, of course, that if I got there in five...         3\n",
      "4        54    55  The picture of the Old Mill at Bramley, with i...         4\n",
      "         2      3  At my meeting with Clement yesterday, he had b...         4\n",
      "         96    97  To have slept and to wake right up surrounded ...         4\n",
      "         50    51  As I was not staying, but only passing through...         4\n",
      "         31    32  I was a little consoled for the weeping weathe...         4\n",
      "5        15    16  One’s eyebrows were one’s own, I always though...         5\n",
      "         76    77  Compact, they call it; but when I used it, I w...         5\n",
      "         80    81  The victim, for that I must now reluctantly ca...         5\n",
      "         12    13      Rintrah, where has thou hid thy bride?    ...         5\n",
      "         67    68  Out cascaded the darling young. It was no trag...         5\n",
      "6        91    92  My earlier days had been so different. There h...         6\n",
      "         1      2  I plunged for the last time. The few remaining...         6\n",
      "         13    14  My ears were becoming attuned, and for the fir...         6\n",
      "         89    90  While my mind had been thus far away, a grotes...         6\n",
      "         30    31  The others did not seem similarly impressed. P...         6\n",
      "7        49    50  The swallow, the bright Homonoea.I wondered if...         7\n",
      "         8      9  The sound of the bell, as of a boding gnat, ju...         7\n",
      "         27    28  I found myself by that one of the windows whic...         7\n",
      "         17    18  I discussed certain passages with the man, and...         7\n",
      "         77    78  Then there disappeared the last rose flivers o...         7\n",
      "Train dataset size: 32\n",
      "Test dataset size: 8\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import preprocessing\n",
    "\n",
    "label_encoder = preprocessing.LabelEncoder()\n",
    "df[\"narrator\"] = label_encoder.fit_transform(df[\"narrator\"])\n",
    "\n",
    "\n",
    "def get_datasets(df):\n",
    "    # sample 5% of the pages per narrator\n",
    "    df_sampled = df.groupby(\"narrator\").apply(lambda s: s.sample(5))\n",
    "    print(df_sampled)\n",
    "\n",
    "    # split datasets\n",
    "    return train_test_split(\n",
    "        df_sampled[\"text\"].values,\n",
    "        df_sampled[\"narrator\"].values,\n",
    "        test_size=0.2,\n",
    "    )\n",
    "\n",
    "\n",
    "train_texts, test_texts, train_labels, test_labels = get_datasets(df)\n",
    "\n",
    "print(f\"Train dataset size: {train_texts.shape[0]}\")\n",
    "print(f\"Test dataset size: {test_texts.shape[0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "31e2edc8-1183-4bc4-92a3-e7be116ee79d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, Dataset\n",
    "\n",
    "\n",
    "# tokenize and prepare the input data\n",
    "class TokenizedData(Dataset):\n",
    "    def __init__(self, texts, labels, tokenizer, max_len=128):\n",
    "        self.texts = texts\n",
    "        self.labels = labels\n",
    "        self.tokenizer = tokenizer\n",
    "        self.max_len = max_len\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.texts)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        text = str(self.texts[idx])\n",
    "        label = int(self.labels[idx])\n",
    "        inputs = self.tokenizer(\n",
    "            text,\n",
    "            truncation=True,\n",
    "            padding=\"max_length\",\n",
    "            max_length=self.max_len,\n",
    "            return_tensors=\"pt\",\n",
    "        )\n",
    "        return {\n",
    "            \"input_ids\": inputs[\"input_ids\"].flatten(),\n",
    "            \"attention_mask\": inputs[\"attention_mask\"].flatten(),\n",
    "            \"label\": torch.tensor(label, dtype=torch.long),\n",
    "        }\n",
    "\n",
    "\n",
    "# important: set a large max_len for the pages\n",
    "train_dataloader = DataLoader(\n",
    "    TokenizedData(train_texts, train_labels, tokenizer, max_len=256),\n",
    "    batch_size=2,\n",
    "    shuffle=True,\n",
    ")\n",
    "test_dataloader = DataLoader(\n",
    "    TokenizedData(test_texts, test_labels, tokenizer, max_len=256),\n",
    "    batch_size=2,\n",
    "    shuffle=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "52ca8f06-cd12-4708-8ce5-335335a921df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/100 — loss: 2.1343\n",
      "2/100 — loss: 2.0167\n",
      "3/100 — loss: 1.8633\n",
      "4/100 — loss: 1.7807\n",
      "5/100 — loss: 1.6406\n",
      "6/100 — loss: 1.4455\n",
      "7/100 — loss: 1.1702\n",
      "8/100 — loss: 0.9549\n",
      "9/100 — loss: 0.8879\n",
      "10/100 — loss: 0.7361\n",
      "11/100 — loss: 0.5977\n",
      "12/100 — loss: 0.5194\n",
      "13/100 — loss: 0.4555\n",
      "14/100 — loss: 0.3631\n",
      "15/100 — loss: 0.3056\n",
      "16/100 — loss: 0.2626\n",
      "17/100 — loss: 0.2255\n",
      "18/100 — loss: 0.2017\n",
      "19/100 — loss: 0.1683\n",
      "20/100 — loss: 0.1547\n",
      "21/100 — loss: 0.1426\n",
      "22/100 — loss: 0.1237\n",
      "23/100 — loss: 0.1106\n",
      "24/100 — loss: 0.1046\n",
      "25/100 — loss: 0.0884\n",
      "26/100 — loss: 0.0795\n",
      "27/100 — loss: 0.0765\n",
      "28/100 — loss: 0.0737\n",
      "29/100 — loss: 0.0635\n",
      "30/100 — loss: 0.0626\n",
      "31/100 — loss: 0.0499\n",
      "32/100 — loss: 0.0538\n",
      "33/100 — loss: 0.0529\n",
      "34/100 — loss: 0.0501\n",
      "35/100 — loss: 0.0447\n",
      "36/100 — loss: 0.0404\n",
      "37/100 — loss: 0.0375\n",
      "38/100 — loss: 0.0359\n",
      "39/100 — loss: 0.0347\n",
      "40/100 — loss: 0.0341\n",
      "41/100 — loss: 0.0335\n",
      "42/100 — loss: 0.0307\n",
      "43/100 — loss: 0.0295\n",
      "44/100 — loss: 0.0278\n",
      "45/100 — loss: 0.0272\n",
      "46/100 — loss: 0.0269\n",
      "47/100 — loss: 0.0257\n",
      "48/100 — loss: 0.0247\n",
      "49/100 — loss: 0.0225\n",
      "50/100 — loss: 0.0230\n",
      "51/100 — loss: 0.0224\n",
      "52/100 — loss: 0.0208\n",
      "53/100 — loss: 0.0194\n",
      "54/100 — loss: 0.0194\n",
      "55/100 — loss: 0.0184\n",
      "56/100 — loss: 0.0179\n",
      "57/100 — loss: 0.0181\n",
      "58/100 — loss: 0.0175\n",
      "59/100 — loss: 0.0165\n",
      "60/100 — loss: 0.0162\n",
      "61/100 — loss: 0.0160\n",
      "62/100 — loss: 0.0151\n",
      "63/100 — loss: 0.0145\n",
      "64/100 — loss: 0.0140\n",
      "65/100 — loss: 0.0138\n",
      "66/100 — loss: 0.0130\n",
      "67/100 — loss: 0.0133\n",
      "68/100 — loss: 0.0133\n",
      "69/100 — loss: 0.0128\n",
      "70/100 — loss: 0.0121\n",
      "71/100 — loss: 0.0127\n",
      "72/100 — loss: 0.0120\n",
      "73/100 — loss: 0.0110\n",
      "74/100 — loss: 0.0108\n",
      "75/100 — loss: 0.0110\n",
      "76/100 — loss: 0.0108\n",
      "77/100 — loss: 0.0105\n",
      "78/100 — loss: 0.0106\n",
      "79/100 — loss: 0.0100\n",
      "80/100 — loss: 0.0100\n",
      "81/100 — loss: 0.0099\n",
      "82/100 — loss: 0.0092\n",
      "83/100 — loss: 0.0092\n",
      "84/100 — loss: 0.0095\n",
      "85/100 — loss: 0.0087\n",
      "86/100 — loss: 0.0082\n",
      "87/100 — loss: 0.0088\n",
      "88/100 — loss: 0.0082\n",
      "89/100 — loss: 0.0082\n",
      "90/100 — loss: 0.0080\n",
      "91/100 — loss: 0.0084\n",
      "92/100 — loss: 0.0077\n",
      "93/100 — loss: 0.0078\n",
      "94/100 — loss: 0.0073\n",
      "95/100 — loss: 0.0070\n",
      "96/100 — loss: 0.0067\n",
      "97/100 — loss: 0.0073\n",
      "98/100 — loss: 0.0074\n",
      "99/100 — loss: 0.0069\n",
      "100/100 — loss: 0.0066\n"
     ]
    }
   ],
   "source": [
    "from transformers import AdamW\n",
    "import copy\n",
    "\n",
    "\n",
    "def train(dataloader):\n",
    "    model = copy.copy(bert_model)\n",
    "    model.to(device)\n",
    "\n",
    "    criterion = torch.nn.CrossEntropyLoss()\n",
    "    optimizer = AdamW(model.parameters(), lr=learning_rate)\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()\n",
    "        total_loss = 0.0\n",
    "\n",
    "        for batch in dataloader:\n",
    "            input_ids = batch[\"input_ids\"].to(device)\n",
    "            attention_mask = batch[\"attention_mask\"].to(device)\n",
    "            labels = batch[\"label\"].to(device)\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(input_ids, attention_mask=attention_mask)\n",
    "            loss = criterion(outputs.logits, labels)\n",
    "            loss.backward()\n",
    "\n",
    "            optimizer.step()\n",
    "\n",
    "            total_loss += loss.item()\n",
    "\n",
    "        avg_loss = total_loss / len(dataloader)\n",
    "        print(f\"{epoch + 1}/{num_epochs} — loss: {avg_loss:.4f}\")\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "model = train(train_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82f9f942-4c07-41df-8bf5-ce16f7d20ce6",
   "metadata": {},
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3804b422-60c9-4570-9690-29eae2c0a687",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.2500\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "def eval(model, dataloader):\n",
    "    model.eval()\n",
    "\n",
    "    all_labels = []\n",
    "    all_predictions = []\n",
    "\n",
    "    for batch in dataloader:\n",
    "        input_ids = batch[\"input_ids\"].to(device)\n",
    "        attention_mask = batch[\"attention_mask\"].to(device)\n",
    "        labels = batch[\"label\"].cpu().numpy()\n",
    "\n",
    "        with torch.no_grad():\n",
    "            outputs = model(input_ids, attention_mask=attention_mask)\n",
    "\n",
    "        logits = outputs.logits\n",
    "        predicted_labels = torch.argmax(logits, dim=1).cpu().numpy()\n",
    "\n",
    "        all_labels.extend(labels)\n",
    "        all_predictions.extend(predicted_labels)\n",
    "\n",
    "    accuracy = accuracy_score(all_labels, all_predictions)\n",
    "\n",
    "    return all_predictions, accuracy\n",
    "\n",
    "\n",
    "predicted_labels, accuracy = eval(model, test_dataloader)\n",
    "print(f\"Accuracy: {accuracy:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93387c4a-ea72-4ecc-9fc9-33bfdcd2b665",
   "metadata": {},
   "source": [
    "The accuracy of the model is very low."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "52edf0b0-95e5-45a4-9bd0-7c6b9d249d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_dataloader = DataLoader(\n",
    "    TokenizedData(df[\"text\"], df[\"narrator\"], tokenizer, max_len=256),\n",
    "    batch_size=2,\n",
    "    shuffle=True,\n",
    ")\n",
    "predicted_labels = eval(model, all_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d147c3c1-7ac7-4bbc-a966-c46f6ba90b5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 800x600 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAG2CAYAAACNs6TQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZsElEQVR4nO3de1xUdf4/8NcMyIDC4BUGEBBvKAoImAZmWplmZhK7pq4lXntsYnnJctlfpWaJu62aZuItNS0jszQzL7EW3tAVL+xqKonJJeWmCMNFBpiZ3x9+mZoUmGGAD2fm9exxHt/vOZzLy7NnePP5nM+cI9Pr9XoQERGRVZCLDkBERESNh4WdiIjIirCwExERWREWdiIiIivCwk5ERGRFWNiJiIisCAs7ERGRFWFhJyIisiIs7ERERFaEhZ2IiMiKsLATERE1sWXLlkEmk2HOnDl1rvfll1+iV69ecHR0RGBgIPbv32/2sVjYiYiImlBKSgrWr1+PoKCgOtdLTk7GhAkTMG3aNJw/fx6RkZGIjIzExYsXzTqejC+BISIiahqlpaUIDQ3F2rVr8e6776Jfv3744IMPHrjuuHHjUFZWhn379hmWPfzww+jXrx/WrVtn8jHtLQ0tkk6nw82bN+Hi4gKZTCY6DhERmUmv16OkpASenp6Qy5uuE7miogKVlZUW70ev199XbxQKBRQKxQPXj4mJwahRozBs2DC8++67de775MmTmDdvntGyESNGYM+ePWZllHRhv3nzJry9vUXHICIiC2VnZ6Nz585Nsu+Kigo4uXQAqsst3pezszNKS0uNli1cuBCLFi26b92EhAScO3cOKSkpJu07NzcX7u7uRsvc3d2Rm5trVkZJF3YXFxcAQOdpWyF3aC04jXk2zRggOkKDBPu2Ex3B5jz05iHRERok5d0RoiM0yH8z74iO0GCjo98THcFsem0lKi99Yvh93hQqKyuB6nIoAqIBO4eG70hbidJLnyA7OxtKpdKw+EGt9ezsbMyePRuJiYlwdHRs+DEbQNKFvaY7RO7QGnKFtAp7G2dl/Su1QL+/mKl5SO3ariHVa6WNs1Z0hAaTWVK0BGuW26n2jhadI73s3q0CpVJZ7/V99uxZ5OfnIzQ01LBMq9Xi6NGjWLNmDTQaDezs7Iy2UalUyMvLM1qWl5cHlUplVk6OiiciItsgAyCTWTCZfqgnnngCFy5cQGpqqmHq378/Jk6ciNTU1PuKOgCEh4fj8OHDRssSExMRHh5u1j9T0i12IiIik8nk9yZLtjeRi4sL+vbta7SsTZs26NChg2H5pEmT4OXlhbi4OADA7NmzMWTIECxfvhyjRo1CQkICzpw5gw0bNpgVky12IiIiAbKyspCTk2OYj4iIwI4dO7BhwwYEBwdj165d2LNnz31/INSHLXYiIrINNV3qlmxvgaSkpDrnAWDs2LEYO3asRcdhYSciItvQjF3xIkkjJREREZmELXYiIrINgrvimwsLOxER2QgLu+Il0sktjZRERERkErbYiYjINrArnoiIyIpwVDwRERFJDVvsRERkG9gVT0REZEVspCuehZ2IiGwDW+y2aXy4D8aH+8KrnRMAID2vFGsTr+JYWoHgZPX736UMfLH3OK5ev4nbd0qweP4EPDIgQHQsk2zceQQffnoY+bfV6NvDC/94fSzC+nQRHateUswt5WsckOY5l/Jns8ac6CexcNYYxH/+I/6+4ivRcagOLaJf4aOPPkKXLl3g6OiIgQMH4vTp08Ky5BZVYMX+K/jzquMYu+oETqXfxprJ/dHd3VlYJlPd1VSiWxcVXp32jOgoZvn6+7N484PdWDB9JJK2L0DfHl740ysfoaCwRHS0Okk1t5Svcamec6l+NmuEBPhg8nODcPHnX0VHsUxNV7wlkwQIT/nFF19g3rx5WLhwIc6dO4fg4GCMGDEC+fn5QvIkXc7H0SsFyLxVjoxbZVh1MA3lldUI9mknJI85Bob0xNTxwyTXEli74wdMiozAxGfD0aurB1bEjkdrRwd8uvek6Gh1kmpuKV/jUj3nUv1sAkAbJwdseGcyZi/9HEUld0XHsYxMZmFhl0ZXvPDCvmLFCsyYMQNTpkxBQEAA1q1bh9atW2Pz5s2io0EuA54O9kBrBzukZt4RHccqVVZVI/VKNoYO8Dcsk8vlGDLAHykXrgtMVjep5v4jKV3j1nLOpeb9N8bh+xMXceR0mugoZCKh99grKytx9uxZxMbGGpbJ5XIMGzYMJ0/e/xe4RqOBRqMxzKvV6ibJ1UPlgs9nRUBhL0d5pRavfHIW1/JLm+RYtu52USm0Wh06tXcxWt6pvRJXM/IEpaqfVHPXkOI1LvVzLkVRT4YhuJc3Ho/+p+gojUMuuzdZsr0ECG2x37p1C1qtFu7u7kbL3d3dkZube9/6cXFxcHV1NUze3t5NkiujoBRRK49h3IcnkHAyE3HjgtHNreXffyQyFa9xqo+Xe1vEvfYnvPTWVmgqq0XHaRw2co9dUqPiY2NjMW/ePMO8Wq1ukuJepdUj63Y5AODSDTUCvdvixcFdsOiri41+LFvXoa0z7Ozk9w1+KihUw62DUlCq+kk1dw0pXuNSP+dSE9zLB24dlEjavsCwzN7eDhEh3TBj7KNwHzQHOp1eYEKqjdDC3rFjR9jZ2SEvz7gbLS8vDyqV6r71FQoFFApFc8UzkMkAB3tp/KUmNQ6t7NGvlzeOpKRh1NBgAIBOp8PRlJ8xfeyjgtPVTqq5ayOFa9zaznlLdzQlDRHj3zNatubtF3A1Iw+rtiVKs6jze+xNz8HBAWFhYTh8+DAiIyMB3PugHj58GLNmzRKSae5Ifxy7UoCbRXfRRmGPZ0I8MaBrB8zYJO4reKa6W6HBjdxCw3xufhHSM3Lg4uwE945txQWrx8y/PI6Zi7cjpLcPQvt0QfznP6LsrgYTRz8sOlqdpJpbyte4VM+5FD+bpeUaXL6WY7Ss/G4lCovL7lsuGXzyXPOYN28eoqOj0b9/fwwYMAAffPABysrKMGXKFCF5OjgrsGx8MDopFSipqMbPOSWYsek0kq/eEpLHHGnXbuK1xb99myB+2wEAwPAhIVgQEyUqVr2ihofhVlEplq7/Dvm3SxDY0wu7Vse0+O5VqeaW8jUu1XMu1c8mSZNMr9cL709Zs2YN3n//feTm5qJfv35YvXo1Bg4cWO92arUarq6u8Hl5J+SK1s2QtPF8NjNCdIQGCfVr+d91tja9X/9OdIQGufz+KNERGuTc9Zb9tb+6PPH8W6IjmE2vrYTmwkYUFxdDqWyaP9BqaoVi6CLI7B0bvB99dQU0SYuaNGtjEN5iB4BZs2YJ63onIiIbwa54IiIiK2Ijg+ek8ecHERERmYQtdiIisg3siiciIrIi7IonIiIiqWGLnYiIbISlz3uXRluYhZ2IiGwDu+KJiIhIathiJyIi2yCTWTgqXhotdhZ2IiKyDTbydTdppCQiIiKTsMVORES2gYPniIiIrEhNV7wlkxni4+MRFBQEpVIJpVKJ8PBwHDhwoNb1t27dCplMZjQ5Opr/Njq22ImIyDY0c4u9c+fOWLZsGXr06AG9Xo9PPvkEY8aMwfnz59GnT58HbqNUKpGWlva7Q5qfl4WdiIioCYwePdpo/r333kN8fDxOnTpVa2GXyWRQqVQWHZdd8UREZBsaqSterVYbTRqNpt5Da7VaJCQkoKysDOHh4bWuV1paCl9fX3h7e2PMmDH46aefzP5nWkWLfXiELxxaO4uOYRPOXb8jOkKDhfq1Ex3Bpkj5WiEr1Uhd8d7e3kaLFy5ciEWLFj1wkwsXLiA8PBwVFRVwdnbG7t27ERAQ8MB1/f39sXnzZgQFBaG4uBj/+te/EBERgZ9++gmdO3c2OaZVFHYiIqLmkp2dDaVSaZhXKBS1ruvv74/U1FQUFxdj165diI6OxpEjRx5Y3MPDw41a8xEREejduzfWr1+PJUuWmJyPhZ2IiGxCzUhzC3YAAIZR7qZwcHBA9+7dAQBhYWFISUnBqlWrsH79+nq3bdWqFUJCQpCenm5WTN5jJyIim/DHr5I1ZLKUTqcz6Z48cO++/IULF+Dh4WHWMdhiJyIiagKxsbEYOXIkfHx8UFJSgh07diApKQmHDh0CAEyaNAleXl6Ii4sDALzzzjt4+OGH0b17dxQVFeH9999HZmYmpk+fbtZxWdiJiMg2yP5vsmR7M+Tn52PSpEnIycmBq6srgoKCcOjQITz55JMAgKysLMjlv3Wc37lzBzNmzEBubi7atWuHsLAwJCcn1zrYrjYs7EREZBMa6x67qT7++OM6f56UlGQ0v3LlSqxcudLcVPfhPXYiIiIrwhY7ERHZhOZusYvCwk5ERDaBhZ2IiMiK2Eph5z12IiIiK8IWOxER2YZm/rqbKCzsRERkE9gVT0RERJLDFjsREdmEe29ttaTF3nhZmhILOxER2QQZLH2RizQqO7viiYiIrAhb7H/wePcOCPRQws3FAVVaPTILy7HvUj4KyipFR6vX/y5l4Iu9x3H1+k3cvlOCxfMn4JEB5r08QASp5gaAjTuP4MNPDyP/thp9e3jhH6+PRVifLqJj1Wl8uA/Gh/vCq50TACA9rxRrE6/iWFqB4GR1k/J1IuXsNeZEP4mFs8Yg/vMf8fcVX4mO0yAcPNcMjh49itGjR8PT0xMymQx79uwRGQcA0K1jGyRnFGL1sQysP5kJuVyGl8J94GDX8v8HvaupRLcuKrw67RnRUcwi1dxff38Wb36wGwumj0TS9gXo28MLf3rlIxQUloiOVqfcogqs2H8Ff151HGNXncCp9NtYM7k/urs7i45WJ6leJ4C0swNASIAPJj83CBd//lV0FMvIGmGSAKEt9rKyMgQHB2Pq1KmIiooSGcVg46kso/mE8zfxzlP+6OzqhF8KywWlMs3AkJ4YGNJTdAyzSTX32h0/YFJkBCY+Gw4AWBE7Ht+f+Amf7j2JuZOHC05Xu6TL+Ubzqw6mYXy4D4J92iE9r1RQqvpJ9ToBpJ29jZMDNrwzGbOXfo75U58SHYdMILSwjxw5EiNHjhQZoV6Ore51apRXaQUnoZaksqoaqVeyjQq4XC7HkAH+SLlwXWAy88hlwFNBHmjtYIfUzDui41AL9P4b4/D9iYs4cjpN+oXdwq54vUS64iV1j12j0UCj0Rjm1Wp1kx5PBiCyjwrXb5cjt0RT7/pkO24XlUKr1aFTexej5Z3aK3E1I09QKtP1ULng81kRUNjLUV6pxSufnMW1/JbbWicxop4MQ3Avbzwe/U/RURqFpffYLRtR33wkNSo+Li4Orq6uhsnb27tJjxcVpIJKqcD2sxK/r0T0BxkFpYhaeQzjPjyBhJOZiBsXjG5uLfseOzUvL/e2iHvtT3jpra3QVFaLjtMoagq7JZMUSKrFHhsbi3nz5hnm1Wp1kxX35wJVCHB3wUcnMlBcYR0XNTWeDm2dYWcnv2+gXEGhGm4dlIJSma5Kq0fW7XtjRi7dUCPQuy1eHNwFi766KDgZtRTBvXzg1kGJpO0LDMvs7e0QEdINM8Y+CvdBc6DT6QUmpNpIqrArFAooFIomP85zgSoEqlywNjkTheVVTX48kh6HVvbo18sbR1LSMGpoMABAp9PhaMrPmD72UcHpzCeTAQ72kurAoyZ2NCUNEePfM1q25u0XcDUjD6u2JUqzqPMlMLYpKlCF0M6u2Hw6G5pqLVwUdgCAu1U6VLfwC/luhQY3cgsN87n5RUjPyIGLsxPcO7YVF6weUs098y+PY+bi7Qjp7YPQPl0Q//mPKLurwcTRD4uOVqe5I/1x7EoBbhbdRRuFPZ4J8cSArh0wY9Np0dHqJNXrBJBm9tJyDS5fyzFaVn63EoXFZfctlwpbuccutLCXlpYiPT3dMH/9+nWkpqaiffv28PHxEZJpkF97AEDMoC5GyxPO30BKdrGARKZLu3YTry3ebJiP33YAADB8SAgWxLSMrxM+iFRzRw0Pw62iUixd/x3yb5cgsKcXdq2OafFd8R2cFVg2PhidlAqUVFTj55wSzNh0GslXb4mOViepXieAtLOT9Mj0er2wZmhSUhIee+yx+5ZHR0dj69at9W6vVqvh6uqK6dv/A4fW0hr4MzHQQ3QEmxPq1050hAbp/fp3oiM0yGczI0RHsDlPPP+W6Ahm02srobmwEcXFxVAqm+aP4ppa0WnSJ5A7tG7wfnSV5SjYFt2kWRuD0Bb70KFDIfDvCiIisiG20hXP0TJERERWhIPniIjIJthKi52FnYiIbIONfN2NXfFERERWhC12IiKyCeyKJyIisiIs7ERERFbEVgo777ETERFZEbbYiYjINtjIqHgWdiIisgnsiiciIiLJYYudiIhsAlvsREREVkQGmaG4N2gy8yZ7fHw8goKCoFQqoVQqER4ejgMHDtS5zZdffolevXrB0dERgYGB2L9/v9n/ThZ2IiKiJtC5c2csW7YMZ8+exZkzZ/D4449jzJgx+Omnnx64fnJyMiZMmIBp06bh/PnziIyMRGRkJC5evGjWcVnYiYjIJljUWm9AN/7o0aPx9NNPo0ePHujZsyfee+89ODs749SpUw9cf9WqVXjqqafw+uuvo3fv3liyZAlCQ0OxZs0as47Lwk5ERLZB1ghTA2m1WiQkJKCsrAzh4eEPXOfkyZMYNmyY0bIRI0bg5MmTZh3LKgbPPd/HHW2claJj2ITPLuSIjtBgUs3+9GA/0RFsysS1yaIjNNjhnUtERzBbWakaw0M3io5hFrVabTSvUCigUCgeuO6FCxcQHh6OiooKODs7Y/fu3QgICHjgurm5uXB3dzda5u7ujtzcXLPyscVOREQ2obG64r29veHq6mqY4uLiaj2mv78/UlNT8Z///Acvv/wyoqOjcenSpSb9d1pFi52IiKg+jfV1t+zsbCiVv/US19ZaBwAHBwd0794dABAWFoaUlBSsWrUK69evv29dlUqFvLw8o2V5eXlQqVRm5WSLnYiIbIJMZvkEwPD1tZqprsL+RzqdDhqN5oE/Cw8Px+HDh42WJSYm1npPvjZssRMRETWB2NhYjBw5Ej4+PigpKcGOHTuQlJSEQ4cOAQAmTZoELy8vQ1f+7NmzMWTIECxfvhyjRo1CQkICzpw5gw0bNph1XBZ2IiKyCfda3ZZ0xZu3fn5+PiZNmoScnBy4uroiKCgIhw4dwpNPPgkAyMrKglz+W8d5REQEduzYgTfffBN///vf0aNHD+zZswd9+/Y167gs7EREZBtk5hfnP25vjo8//rjOnyclJd23bOzYsRg7dqx5B/oD3mMnIiKyImyxExGRTbCVl8CwsBMRkU2QWdgVL5G6zq54IiIia8IWOxER2QS5XAa5vOHNbr0F2zYnFnYiIrIJ7IonIiIiyWGLnYiIbAJHxRMREVkRW+mKZ2EnIiKbYCstdt5jJyIisiJssf/B/y5l4Iu9x3H1+k3cvlOCxfMn4JEBAaJjmUSK2R/v3gGBHkq4uTigSqtHZmE59l3KR0FZpehodZJqbkC62aV4fQPA+HAfjA/3hVc7JwBAel4p1iZexbG0AsHJ6ifVc14bttibQVxcHB566CG4uLjAzc0NkZGRSEtLExkJdzWV6NZFhVenPSM0R0NIMXu3jm2QnFGI1ccysP5kJuRyGV4K94GDXcv+AEk1NyDd7FK8vgEgt6gCK/ZfwZ9XHcfYVSdwKv021kzuj+7uzqKj1Uuq57w2jfU+9pZOaIv9yJEjiImJwUMPPYTq6mr8/e9/x/Dhw3Hp0iW0adNGSKaBIT0xMKSnkGNbSorZN57KMppPOH8T7zzlj86uTvilsFxQqvpJNTcg3exSvL4BIOlyvtH8qoNpGB/ug2CfdkjPKxWUyjRSPee2TmhhP3jwoNH81q1b4ebmhrNnz+LRRx8VlIpEcmx1rxOpvEorOIl5pJobkHZ2qZHLgKeCPNDawQ6pmXdEx7E5MljYFW/ue1sFaVH32IuLiwEA7du3F5yERJABiOyjwvXb5cgt0YiOYzKp5gaknV1Keqhc8PmsCCjs5Siv1OKVT87iWn7Lbq1bI37drZnpdDrMmTMHgwYNQt++fR+4jkajgUbz2y8ftVrdXPGoGUQFqaBSKrDmeIboKGaRam5A2tmlJKOgFFErj8HZ0R4jgjwQNy4Yk+JPsbhTk2gxX3eLiYnBxYsXkZCQUOs6cXFxcHV1NUze3t7NmJCa0nOBKgS4uyA+ORPFFdWi45hMqrkBaWeXmiqtHlm3y3HphhorD6QhLacELw7uIjqWzakZFW/JJAUtorDPmjUL+/btw48//ojOnTvXul5sbCyKi4sNU3Z2djOmpKbyXKAKgap7BaawvEp0HJNJNTcg7ezWQCYDHOxbxK9fm8JR8c1Ar9fjlVdewe7du5GUlAQ/P78611coFFAoFE2a6W6FBjdyCw3zuflFSM/IgYuzE9w7tm3SY1tKitmjAlUI7eyKzaezoanWwkVhBwC4W6VDtU4vOF3tpJobkG52KV7fADB3pD+OXSnAzaK7aKOwxzMhnhjQtQNmbDotOlq9pHrObZ3Qwh4TE4MdO3bgm2++gYuLC3JzcwEArq6ucHJyEpIp7dpNvLZ4s2E+ftsBAMDwISFYEBMlJJOppJh9kN+9gZIxg7oYLU84fwMp2cUCEplGqrkB6WaX4vUNAB2cFVg2PhidlAqUVFTj55wSzNh0GslXb4mOVi+pnvPa2MoDamR6vV7Yn+i1naQtW7Zg8uTJ9W6vVqvh6uqK789loI2zspHT0YN8diFHdASSiImBHqIjNMjEtcmiIzTYZzMjREcwW1mpGsNDu6C4uBhKZdP8Hq+pFaFv7YOdY8OfkaKtKMO5Jc80adbGILwrnoiIqDnYSoudozeIiIisSIv5HjsREVGTsnRkuzQa7CzsRERkG9gVT0RERJLDFjsREdkEPiueiIjIirArnoiIiCSHLXYiIrIJ7IonIiKyIuyKJyIiIslhi52IiGyCrbTYWdiJiMgm8B47ERGRFbGVFjvvsRMRETWBuLg4PPTQQ3BxcYGbmxsiIyORlpZW5zZbt241/AFSMzk6Opp1XBZ2IiKyCTVd8ZZM5jhy5AhiYmJw6tQpJCYmoqqqCsOHD0dZWVmd2ymVSuTk5BimzMxMs47LrngiIrIJzd0Vf/DgQaP5rVu3ws3NDWfPnsWjjz5a53FUKlWDMgJssRMREZlFrVYbTRqNxqTtiouLAQDt27evc73S0lL4+vrC29sbY8aMwU8//WRWPrbYySybl6wVHaHBDu9cIjqCTQn1ayc6QoM8PdhPdIQGm7g2WXQEs+k05c12LBksHBX/f//X29vbaPnChQuxaNGiOrfV6XSYM2cOBg0ahL59+9a6nr+/PzZv3oygoCAUFxfjX//6FyIiIvDTTz+hc+fOJuVkYSciIpsgl8kgt6Cy12ybnZ0NpVJpWK5QKOrdNiYmBhcvXsTx48frXC88PBzh4eGG+YiICPTu3Rvr16/HkiWmNU5Y2ImIiMygVCqNCnt9Zs2ahX379uHo0aMmt7prtGrVCiEhIUhPTzd5G95jJyIim9Dco+L1ej1mzZqF3bt344cffoCfn/m3ebRaLS5cuAAPDw+Tt2GLnYiIbEJzj4qPiYnBjh078M0338DFxQW5ubkAAFdXVzg5OQEAJk2aBC8vL8TFxQEA3nnnHTz88MPo3r07ioqK8P777yMzMxPTp083+bgs7EREZBPksnuTJdubIz4+HgAwdOhQo+VbtmzB5MmTAQBZWVmQy3/rPL9z5w5mzJiB3NxctGvXDmFhYUhOTkZAQIDJx2VhJyIiagJ6vb7edZKSkozmV65ciZUrV1p0XBZ2IiKyDTILn/cujUfFs7ATEZFtsJW3u3FUPBERkRVhi52IiGyC7P/+s2R7KWBhJyIim9Dco+JFYVc8ERGRFWGLnYiIbEJzP6BGFJMK+969e03e4bPPPtvgMERERE3FVkbFm1TYIyMjTdqZTCaDVqu1JA8RERFZwKTCrtPpmjoHERFRk2qs17a2dBbdY6+oqICjo2NjZWkR/ncpA1/sPY6r12/i9p0SLJ4/AY8MMP0ZvSJJOXuNOdFPYuGsMYj//Ef8fcVXouPUSarnW6q5a2zceQQffnoY+bfV6NvDC/94fSzC+nQRHatWj3fvgEAPJdxcHFCl1SOzsBz7LuWjoKxSdLQ6jQ/3wfhwX3i1u/eykvS8UqxNvIpjaQWCkzWcrXTFmz0qXqvVYsmSJfDy8oKzszN++eUXAMBbb72Fjz/+uNEDNre7mkp066LCq9OeER3FbFLODgAhAT6Y/NwgXPz5V9FRTCLV8y3V3ADw9fdn8eYHu7Fg+kgkbV+Avj288KdXPkJBYYnoaLXq1rENkjMKsfpYBtafzIRcLsNL4T5wsGvZVSK3qAIr9l/Bn1cdx9hVJ3Aq/TbWTO6P7u7OoqM1WM3gOUsmKTC7sL/33nvYunUr/vnPf8LBwcGwvG/fvti0aZNZ+4qPj0dQUJDhpfXh4eE4cOCAuZEa1cCQnpg6fpikWjA1pJy9jZMDNrwzGbOXfo6ikrui45hEqudbqrkBYO2OHzApMgITnw1Hr64eWBE7Hq0dHfDp3pOio9Vq46kspGQXI69Egxy1Bgnnb6J9awd0dnUSHa1OSZfzcfRKATJvlSPjVhlWHUxDeWU1gn3aiY5G9TC7sG/btg0bNmzAxIkTYWdnZ1geHByMK1eumLWvzp07Y9myZTh79izOnDmDxx9/HGPGjMFPP/1kbiySuPffGIfvT1zEkdNpoqNQC1VZVY3UK9kYOsDfsEwul2PIAH+kXLguMJl5HFvd+7VbXiWdgcZyGfB0sAdaO9ghNfOO6DgNVtMVb8kkBWbfY79x4wa6d+9+33KdToeqqiqz9jV69Gij+ffeew/x8fE4deoU+vTpY240kqioJ8MQ3Msbj0f/U3QUasFuF5VCq9WhU3sXo+Wd2itxNSNPUCrzyABE9lHh+u1y5JZoRMepVw+VCz6fFQGFvRzllVq88slZXMsvFR2rwTh4rhYBAQE4duwYfH19jZbv2rULISEhDQ6i1Wrx5ZdfoqysDOHh4Q9cR6PRQKP57cOgVqsbfDxqGbzc2yLutT8hatYaaCqrRcchalJRQSqolAqsOZ4hOopJMgpKEbXyGJwd7TEiyANx44IxKf6UpIu7LTC7sL/99tuIjo7GjRs3oNPp8PXXXyMtLQ3btm3Dvn37zA5w4cIFhIeHo6KiAs7Ozti9ezcCAh587y8uLg6LFy82+xjUcgX38oFbByWSti8wLLO3t0NESDfMGPso3AfNgU6nF5iQWooObZ1hZye/b6BcQaEabh2UglKZ7rlAFQLcXfDRiQwUV0jjj9gqrR5Zt8sBAJduqBHo3RYvDu6CRV9dFJysYWSw7JXq0mivN6CwjxkzBt9++y3eeecdtGnTBm+//TZCQ0Px7bff4sknnzQ7gL+/P1JTU1FcXIxdu3YhOjoaR44ceWBxj42Nxbx58wzzarUa3t7eZh+TWo6jKWmIGP+e0bI1b7+Aqxl5WLUtkUWdDBxa2aNfL28cSUnDqKHBAO7dAjya8jOmj31UcLq6PReoQqDKBWuTM1FYbt4ty5ZEJgMc7KX7ihE+UrYOgwcPRmJiYqMEcHBwMNyzDwsLQ0pKClatWoX169fft65CoYBCoWiU49bmboUGN3ILDfO5+UVIz8iBi7MT3Du2bdJjW0qK2UvLNbh8LcdoWfndShQWl923vKWR4vkGpJsbAGb+5XHMXLwdIb19ENqnC+I//xFldzWYOPph0dFqFRWoQmhnV2w+nQ1NtRYuinuDju9W6VDdgv9wnTvSH8euFOBm0V20UdjjmRBPDOjaATM2nRYdjerR4AfUnDlzBpcvXwZw7757WFhYowTS6XRG99GbW9q1m3ht8WbDfPy2e1+/Gz4kBAtiokTFMomUs0uRVM+3VHMDQNTwMNwqKsXS9d8h/3YJAnt6YdfqmBbdFT/Irz0AIGZQF6PlCedvICW7WEAi03RwVmDZ+GB0UipQUlGNn3NKMGPTaSRfvSU6WoPZymtbZXq93qw/GX/99VdMmDABJ06cQNu2bQEARUVFiIiIQEJCAjp37mzyvmJjYzFy5Ej4+PigpKQEO3bswD/+8Q8cOnTIpG59tVoNV1dXfH8uA22cW+4H25o88fxboiM02OGdS0RHsCmhftL8vvNrey+JjtBg+49J52t/NXSacmTFP4/i4mIolU3ze7ymVjy/4ThaOTX8ATtVd0ux86VHmjRrYzD7Zsn06dNRVVWFy5cvo7CwEIWFhbh8+TJ0Oh2mT59u1r7y8/MxadIk+Pv744knnkBKSorJRZ2IiIjuZ3ZX/JEjR5CcnAx//98eEuHv748PP/wQgwcPNmtf1vAIWiIikg6JjH+ziNmF3dvb+4EPotFqtfD09GyUUERERI3NVkbFm90V//777+OVV17BmTNnDMvOnDmD2bNn41//+lejhiMiImosNYPnLJmkwKQWe7t27Yz+UikrK8PAgQNhb39v8+rqatjb22Pq1KmIjIxskqBERERUP5MK+wcffNDEMYiIiJqWrXTFm1TYo6OjmzoHERFRk+IjZU1QUVGByspKo2Ut+bt9RERE1s7swl5WVoYFCxZg586duH379n0/12ql845hIiKyHbby2lazR8W/8cYb+OGHHxAfHw+FQoFNmzZh8eLF8PT0xLZt25oiIxERkcVkMssnKTC7xf7tt99i27ZtGDp0KKZMmYLBgweje/fu8PX1xWeffYaJEyc2RU4iIiIygdkt9sLCQnTt2hXAvfvphYX33hL1yCOP4OjRo42bjoiIqJHUjIq3ZJICswt7165dcf36vRcN9OrVCzt37gRwryVf81IYIiKilsZWuuLNLuxTpkzBf//7XwDA3/72N3z00UdwdHTE3Llz8frrrzd6QCIiIjKd2YV97ty5ePXVVwEAw4YNw5UrV7Bjxw6cP38es2fPbvSAREREjaFmVLwlkzni4uLw0EMPwcXFBW5uboiMjERaWlq923355Zfo1asXHB0dERgYiP3795v37zRr7Qfw9fVFVFQUgoKCLN0VERFRk2nurvgjR44gJiYGp06dQmJiIqqqqjB8+HCUlZXVuk1ycjImTJiAadOm4fz584iMjERkZCQuXrxo8nFNGhW/evVqk3dY05onIiJqSZr7kbIHDx40mt+6dSvc3Nxw9uxZPProow/cZtWqVXjqqacMt7aXLFmCxMRErFmzBuvWrTPpuCYV9pUrV5q0M5lMxsJORERWTa1WG80rFAooFIp6tysuLgYAtG/fvtZ1Tp48iXnz5hktGzFiBPbs2WNyPpMKe80oeGo8E9cmi47QIHdS1oiO0GCv7b0kOoJNCfVrJzqCzXl6sJ/oCGarLC/FpvjmOZYclt1/rtnW29vbaPnChQuxaNGiOrfV6XSYM2cOBg0ahL59+9a6Xm5uLtzd3Y2Wubu7Izc31+ScFj0rnoiISCoaqys+Ozvb6L0oprTWY2JicPHiRRw/frzBxzcVCzsREZEZlEqlWS88mzVrFvbt24ejR4+ic+fOda6rUqmQl5dntCwvLw8qlcrk41k8Kp6IiEgKZDJAbsFkbmNfr9dj1qxZ2L17N3744Qf4+dV/qyQ8PByHDx82WpaYmIjw8HCTj8sWOxER2YSaAm3J9uaIiYnBjh078M0338DFxcVwn9zV1RVOTk4AgEmTJsHLywtxcXEAgNmzZ2PIkCFYvnw5Ro0ahYSEBJw5cwYbNmwwPad5MYmIiMgU8fHxKC4uxtChQ+Hh4WGYvvjiC8M6WVlZyMnJMcxHRERgx44d2LBhA4KDg7Fr1y7s2bOnzgF3f9SgFvuxY8ewfv16XLt2Dbt27YKXlxe2b98OPz8/PPLIIw3ZJRERUZNq7u+x6/X6etdJSkq6b9nYsWMxduxYs471e2a32L/66iuMGDECTk5OOH/+PDQaDYB7389bunRpg4MQERE1JUvur1vajd+czC7s7777LtatW4eNGzeiVatWhuWDBg3CuXPnGjUcERERmcfsrvi0tLQHPgrP1dUVRUVFjZGJiIio0Vn66lWrfW2rSqVCenr6fcuPHz+Orl27NkooIiKixtbcb3cTxezCPmPGDMyePRv/+c9/IJPJcPPmTXz22WeYP38+Xn755abISEREZDF5I0xSYHZX/N/+9jfodDo88cQTKC8vx6OPPgqFQoH58+fjlVdeaYqMREREZCKzC7tMJsP/+3//D6+//jrS09NRWlqKgIAAODs7N0U+IiKiRmEr99gb/OQ5BwcHBAQENGYWIiKiJiOHZffJ5ZBGZTe7sD/22GN1fkn/hx9+sCgQERERNZzZhb1fv35G81VVVUhNTcXFixcRHR3dWLmIiIgaFbvia7Fy5coHLl+0aBFKS0stDiTa/y5l4Iu9x3H1+k3cvlOCxfMn4JEBLf+Ww/hwH4wP94VXu3svFkjPK8XaxKs4llYgOJlpNu48gg8/PYz822r07eGFf7w+FmF9uoiOVavHu3dAoIcSbi4OqNLqkVlYjn2X8lFQVik6Wr2knB3gtdJcpJq7Ls39EhhRGm30/gsvvIDNmzc31u6EuaupRLcuKrw67RnRUcySW1SBFfuv4M+rjmPsqhM4lX4bayb3R3f3lj+o8evvz+LND3ZjwfSRSNq+AH17eOFPr3yEgsIS0dFq1a1jGyRnFGL1sQysP5kJuVyGl8J94GDX8j/5Us7Oa6X5SDU3NWJhP3nyJBwdHRu8/bJlyyCTyTBnzpzGitQgA0N6Yur4YZJopf9e0uV8HL1SgMxb5ci4VYZVB9NQXlmNYJ92oqPVa+2OHzApMgITnw1Hr64eWBE7Hq0dHfDp3pOio9Vq46kspGQXI69Egxy1Bgnnb6J9awd0dnUSHa1eUs7Oa6X5SDV3Xe69j73hD6ex2q74qKgoo3m9Xo+cnBycOXMGb731VoNCpKSkYP369QgKCmrQ9mRMLgOeCvJAawc7pGbeER2nTpVV1Ui9ko25k4cblsnlcgwZ4I+UC9cFJjOPY6t7fyOXV2kFJzGfVLLzWhFLqrl/j/fYa+Hq6mo0L5fL4e/vj3feeQfDhw+vZavalZaWYuLEidi4cSPeffdds7en3/RQueDzWRFQ2MtRXqnFK5+cxbX8lj3u4XZRKbRaHTq1dzFa3qm9Elcz8gSlMo8MQGQfFa7fLkduiUZ0HLNIKTuvFXGkmttWmVXYtVotpkyZgsDAQLRr1zhdvDExMRg1ahSGDRtWb2HXaDSG18QCgFqtbpQM1iKjoBRRK4/B2dEeI4I8EDcuGJPiT7X44i51UUEqqJQKrDmeITqK2aScXYqker6lmvuPOHjuAezs7DB8+PBGe4tbQkICzp07h7i4OJPWj4uLg6urq2Hy9vZulBzWokqrR9btcly6ocbKA2lIyynBi4O7iI5Vpw5tnWFnJ79v8FNBoRpuHZSCUpnuuUAVAtxdEJ+cieKKatFxzCK17LxWxJBq7geRNcJ/UmD24Lm+ffvil19+sfjA2dnZmD17Nj777DOTB93FxsaiuLjYMGVnZ1ucw5rJZICDfct+bYFDK3v06+WNIylphmU6nQ5HU37GQ4F+ApPV77lAFQJV937hFZZXiY5jFilm57XS/KSauzY1LXZLJikw+x77u+++i/nz52PJkiUICwtDmzZtjH6uVJr2l/PZs2eRn5+P0NBQwzKtVoujR49izZo10Gg0sLOzM9pGoVBAoVCYG9ksdys0uJFbaJjPzS9CekYOXJyd4N6xbZMe2xJzR/rj2JUC3Cy6izYKezwT4okBXTtgxqbToqPVa+ZfHsfMxdsR0tsHoX26IP7zH1F2V4OJox8WHa1WUYEqhHZ2xebT2dBUa+GiuHet3q3SoVqnF5yublLOzmul+Ug1N5lR2N955x289tprePrppwEAzz77rNGjZfV6PWQyGbRa00ZMPvHEE7hw4YLRsilTpqBXr15YsGDBfUW9uaRdu4nXFv/2ffz4bQcAAMOHhGBBTFRtmwnXwVmBZeOD0UmpQElFNX7OKcGMTaeRfPWW6Gj1ihoehltFpVi6/jvk3y5BYE8v7Fod06K7Vwf5tQcAxAzqYrQ84fwNpGQXC0hkOiln57XSfKSauy62co9dptfrTfrTy87ODjk5Obh8+XKd6w0ZMqTBYYYOHYp+/frhgw8+MGl9tVoNV1dXfH8uA22cW+4H+0Emrk0WHaFBLr8/SnSEBntt7yXREWzK8mel9SyIGrxOmldleSk2vTgQxcXFJvf4mqumVryzLxWObVzq36AWFWUlePuZfk2atTGY3GKvqf+WFG4iIiJqWmbdY6/rrW6NISkpqUn3T0REtstWuuLNKuw9e/ast7gXFhbW+XMiIiIR+OS5B1i8ePF9T54jIiKilsOswj5+/Hi4ubk1VRYiIqImU/MyF0u2lwKTC3tT318nIiJqSrZyj93kx5KZ+K04IiIiEsjkFrtOp2vKHERERE3LwsFzEnlUvPmPlCUiIpIiOWSQW1CdLdm2ObGwExGRTbCVr7u17Fd/ERERkVnYYiciIptgK6PiWdiJiMgm2Mr32NkVT0REZEVY2ImIyCbUDJ6zZDLH0aNHMXr0aHh6ekImk2HPnj11rp+UlASZTHbflJuba9Zx2RVPREQ2QQ4Lu+LN/LpbWVkZgoODMXXqVERFRZm8XVpamtH73s19lDsLOxERURMYOXIkRo4cafZ2bm5uaNu2bYOPy654IiKyCY3VFa9Wq40mjUbTqDn79esHDw8PPPnkkzhx4oTZ21tFiz3Yt51Rt4UUfDYzQnSEBnlt7yXRERpsYqCH6AgNMnFtsugIDbL82QDRERpk85K1oiM02NS3ZoqO0KLJYVlrtmZbb29vo+ULFy7EokWLLNjzPR4eHli3bh369+8PjUaDTZs2YejQofjPf/6D0NBQk/djFYWdiIiouWRnZxs1JhUKRaPs19/fH/7+/ob5iIgIXLt2DStXrsT27dtN3g8LOxER2YSaUeaWbA8ASqWy2XqJBwwYgOPHj5u1DQs7ERHZBBkse0GbiMfTpKamwsPDvNuILOxERGQTmvvJc6WlpUhPTzfMX79+HampqWjfvj18fHwQGxuLGzduYNu2bQCADz74AH5+fujTpw8qKiqwadMm/PDDD/j+++/NOi4LOxERURM4c+YMHnvsMcP8vHnzAADR0dHYunUrcnJykJWVZfh5ZWUlXnvtNdy4cQOtW7dGUFAQ/v3vfxvtwxQs7EREZDOaszt96NCh0Ov1tf5869atRvNvvPEG3njjDYuPy8JOREQ2ge9jJyIiIslhi52IiGxCY33draVjYSciIpvQWE+ea+mkkpOIiIhMwBY7ERHZBHbFExERWREpPnmuIdgVT0REZEXYYiciIpvArngiIiIrYiuj4lnYiYjIJrDFbuM27jyCDz89jPzbavTt4YV/vD4WYX26iI5Vp/9dysAXe4/j6vWbuH2nBIvnT8AjAwJEx6rT4907INBDCTcXB1Rp9cgsLMe+S/koKKsUHa1eUjzfADA+3Afjw33h1c4JAJCeV4q1iVdxLK1AcDLTSPGz+Xtzop/EwlljEP/5j/j7iq9Ex6mVlD+btk5oz8KiRYsMf0HVTL169RIZCQDw9fdn8eYHu7Fg+kgkbV+Avj288KdXPkJBYYnoaHW6q6lEty4qvDrtGdFRTNatYxskZxRi9bEMrD+ZCblchpfCfeBg1/L/Mpbi+QaA3KIKrNh/BX9edRxjV53AqfTbWDO5P7q7O4uOVi+pfjZrhAT4YPJzg3Dx519FR6mXlD+btZE1wiQFwm8Z9OnTBzk5OYbp+PHjoiNh7Y4fMCkyAhOfDUevrh5YETserR0d8Onek6Kj1WlgSE9MHT9MEq3GGhtPZSEluxh5JRrkqDVIOH8T7Vs7oLOrk+ho9ZLi+QaApMv5OHqlAJm3ypFxqwyrDqahvLIawT7tREerl1Q/mwDQxskBG96ZjNlLP0dRyV3Rceol5c9mbWpeAmPJJAXCC7u9vT1UKpVh6tixo9A8lVXVSL2SjaED/A3L5HI5hgzwR8qF6wKT2QbHVvcuyfIqreAktkEuA54O9kBrBzukZt4RHadOUv9svv/GOHx/4iKOnE4THaVB+NmUDuH32K9evQpPT084OjoiPDwccXFx8PHxeeC6Go0GGo3GMK9Wqxs9z+2iUmi1OnRq72K0vFN7Ja5m5DX68eg3MgCRfVS4frscuSWaetenhuuhcsHnsyKgsJejvFKLVz45i2v5paJj1UnKn82oJ8MQ3Msbj0f/U3SUBrGWz6YcMsgt6FC3ZNvmJLTFPnDgQGzduhUHDx5EfHw8rl+/jsGDB6Ok5MH3y+Li4uDq6mqYvL29mzkxNaWoIBVUSgW2n2359x+lLqOgFFErj2HchyeQcDITceOC0c2t5d9jlyIv97aIe+1PeOmtrdBUVouO0yDW8tm0la54oS32kSNHGv7/oKAgDBw4EL6+vti5cyemTZt23/qxsbGYN2+eYV6tVjd6ce/Q1hl2dvL7BuMUFKrh1kHZqMei3zwXqEKAuws+OpGB4gpp/vKTkiqtHlm3ywEAl26oEejdFi8O7oJFX10UnKx2Uv1sBvfygVsHJZK2LzAss7e3Q0RIN8wY+yjcB82BTqcXmLBu/GxKj/Cu+N9r27YtevbsifT09Af+XKFQQKFQNGkGh1b26NfLG0dS0jBqaDAAQKfT4WjKz5g+9tEmPbatei5QhUCVC9YmZ6KwvEp0HJskkwEO9sKH3NRJqp/NoylpiBj/ntGyNW+/gKsZeVi1LbHFF3Vr+mzK/u8/S7aXghZV2EtLS3Ht2jW8+OKLQnPM/MvjmLl4O0J6+yC0TxfEf/4jyu5qMHH0w0Jz1eduhQY3cgsN87n5RUjPyIGLsxPcO7YVF6wOUYEqhHZ2xebT2dBUa+GisAMA3K3SoboF/8IDpHm+AWDuSH8cu1KAm0V30UZhj2dCPDGgawfM2HRadLR6SfGzWVquweVrOUbLyu9WorC47L7lLYmUP5u1sbQ7nV3xJpg/fz5Gjx4NX19f3Lx5EwsXLoSdnR0mTJggMhaihofhVlEplq7/Dvm3SxDY0wu7Vse06O4+AEi7dhOvLd5smI/fdgAAMHxICBbERImKVadBfu0BADGDuhgtTzh/AynZxQISmU6K5xsAOjgrsGx8MDopFSipqMbPOSWYsek0kq/eEh2tXlL9bEqRlD+btk5oYf/1118xYcIE3L59G506dcIjjzyCU6dOoVOnTiJjAQBeen4IXnp+iOgYZunXxw+Hdy4RHcMsr+29JDpCg0nxfAPAm1/+T3QEi0jxs/lHo/+6SnSEekn5s1kbmYWj4tkVb4KEhASRhyciIhvCrngiIiIrYiuFvWUPgyUiIiKzsMVOREQ2gV93IyIisiJy2b3Jku2lgF3xREREVoQtdiIisgnsiiciIrIiHBVPREREksMWOxER2QQZLOtOl0iDnYWdiIhsA0fFExERkeSwxU5ERDbBVkbFs8VOREQ2oWZUvCWTOY4ePYrRo0fD09MTMpkMe/bsqXebpKQkhIaGQqFQoHv37ti6davZ/04WdiIisgmyRpjMUVZWhuDgYHz00UcmrX/9+nWMGjUKjz32GFJTUzFnzhxMnz4dhw4dMuu47IonIiJqAiNHjsTIkSNNXn/dunXw8/PD8uXLAQC9e/fG8ePHsXLlSowYMcLk/bDFTkRENkEOGeQyC6b/a7Or1WqjSaPRNEq+kydPYtiwYUbLRowYgZMnT5q1H6tosT/05iHIFa1FxzDL5fdHiY7QIJ9dyBEdocGkmv3pwX6iIzTIuet3REdokKlvzRQdocGWPxsgOoLZ1Go1NjXTsRrSnf7H7QHA29vbaPnChQuxaNEiC/Z8T25uLtzd3Y2Wubu7Q61W4+7du3BycjJpP1ZR2ImIiJpLdnY2lEqlYV6hUAhMcz8WdiIisg2N1GRXKpVGhb2xqFQq5OXlGS3Ly8uDUqk0ubUOsLATEZGNaOnfYw8PD8f+/fuNliUmJiI8PNys/XDwHBERURMoLS1FamoqUlNTAdz7OltqaiqysrIAALGxsZg0aZJh/b/+9a/45Zdf8MYbb+DKlStYu3Ytdu7ciblz55p1XLbYiYjINlj42lZzG+xnzpzBY489ZpifN28eACA6Ohpbt25FTk6OocgDgJ+fH7777jvMnTsXq1atQufOnbFp0yazvuoGsLATEZGNaKxR8aYaOnQo9Hp9rT9/0FPlhg4divPnz5t5JGPsiiciIrIibLETEZFtaO4muyAs7EREZBNa+qj4xsLCTkRENqEhb2j74/ZSwHvsREREVoQtdiIisgk2coudhZ2IiGyEjVR2dsUTERFZEbbYiYjIJnBUPBERkRXhqHgiIiKSHLbY/2B8uA/Gh/vCq929d9+m55VibeJVHEsrEJzMNBt3HsGHnx5G/m01+vbwwj9eH4uwPl1Ex6rV4907INBDCTcXB1Rp9cgsLMe+S/koKKsUHa1OUs0NSDf7/y5l4Iu9x3H1+k3cvlOCxfMn4JEBAaJj1Uuq57uG1H6n1MVGxs6Jb7HfuHEDL7zwAjp06AAnJycEBgbizJkzwvLkFlVgxf4r+POq4xi76gROpd/Gmsn90d3dWVgmU339/Vm8+cFuLJg+EknbF6BvDy/86ZWPUFBYIjparbp1bIPkjEKsPpaB9SczIZfL8FK4DxzsWvZHSKq5Aelmv6upRLcuKrw67RnRUcwi1fMNSPN3Sp1kjTBJgNDCfufOHQwaNAitWrXCgQMHcOnSJSxfvhzt2rUTlinpcj6OXilA5q1yZNwqw6qDaSivrEawj7hMplq74wdMiozAxGfD0aurB1bEjkdrRwd8uvek6Gi12ngqCynZxcgr0SBHrUHC+Zto39oBnV2dREerk1RzA9LNPjCkJ6aOHyaJVvrvSfV8A9L8nUKCu+L/8Y9/wNvbG1u2bDEs8/PzE5jImFwGPBXkgdYOdkjNvCM6Tp0qq6qReiUbcycPNyyTy+UYMsAfKReuC0xmHsdW9/7WLK/SCk5iHqnmBqSdXYqkcr6t5XfK73FUfDPYu3cvRowYgbFjx+LIkSPw8vLCzJkzMWPGjAeur9FooNFoDPNqtbpJcvVQueDzWRFQ2MtRXqnFK5+cxbX80iY5VmO5XVQKrVaHTu1djJZ3aq/E1Yw8QanMIwMQ2UeF67fLkVuiqXf9lkKquQFpZ5ciKZ1va/id8kccFd8MfvnlF8THx6NHjx44dOgQXn75Zbz66qv45JNPHrh+XFwcXF1dDZO3t3eT5MooKEXUymMY9+EJJJzMRNy4YHRza/n32KUuKkgFlVKB7Wd/FR3FLFLNDUg7uxTxfItlI7fYxRZ2nU6H0NBQLF26FCEhIXjppZcwY8YMrFu37oHrx8bGori42DBlZ2c3Sa4qrR5Zt8tx6YYaKw+kIS2nBC8O7tIkx2osHdo6w85Oft+gloJCNdw6KAWlMt1zgSoEuLsgPjkTxRXVouOYTKq5AWlnlyKpnW+p/06xZUILu4eHBwICjAfC9O7dG1lZWQ9cX6FQQKlUGk3NQSYDHOyFf4GgTg6t7NGvlzeOpKQZlul0OhxN+RkPBbaccQsP8lygCoGqe7/wCsurRMcxmVRzA9LOLkVSPN9S/p1SKxtpsgu9xz5o0CCkpaUZLfv555/h6+srKBEwd6Q/jl0pwM2iu2ijsMczIZ4Y0LUDZmw6LSyTqWb+5XHMXLwdIb19ENqnC+I//xFldzWYOPph0dFqFRWoQmhnV2w+nQ1NtRYuCjsAwN0qHap1esHpaifV3IB0s9+t0OBGbqFhPje/COkZOXBxdoJ7x7bigtVDqucbkObvlLpw8FwzmDt3LiIiIrB06VI8//zzOH36NDZs2IANGzYIy9TBWYFl44PRSalASUU1fs4pwYxNp5F89ZawTKaKGh6GW0WlWLr+O+TfLkFgTy/sWh3TorvNBvm1BwDEDOpitDzh/A2kZBcLSGQaqeYGpJs97dpNvLZ4s2E+ftsBAMDwISFYEBMlKla9pHq+AWn+TiFAptfrhf7JuG/fPsTGxuLq1avw8/PDvHnzah0V/0dqtRqurq7weXkn5IrWTZy0cV1+f5ToCA3y2t5LoiOQREwM9BAdoUE+u5AjOkKDLX9WWt/xB+79Hnfv4Iri4uImu71aUyv+c+UmnF0afozSEjUG9vJs0qyNQfgjZZ955hk884y0niRFRETSw0fKEhERkeQIb7ETERE1CxtpsrOwExGRTbCVUfHsiiciIrIibLETEZFNsJVnxbOwExGRTbCRW+ws7EREZCNspLLzHjsREZEVYYudiIhsgq2MimdhJyIi22Dh4DmJ1HV2xRMREVkTttiJiMgm2MjYObbYiYjIRsgaYWqAjz76CF26dIGjoyMGDhyI06dP17ru1q1bIZPJjCZHR0ezjsfCTkRE1ES++OILzJs3DwsXLsS5c+cQHByMESNGID8/v9ZtlEolcnJyDFNmZqZZx2RhJyIimyBrhP/MtWLFCsyYMQNTpkxBQEAA1q1bh9atW2Pz5s2155TJoFKpDJO7u7tZx2RhJyIim1DzSFlLJnNUVlbi7NmzGDZsmGGZXC7HsGHDcPLkyVq3Ky0tha+vL7y9vTFmzBj89NNPZh2XhZ2IiMgMarXaaNJoNA9c79atW9Bqtfe1uN3d3ZGbm/vAbfz9/bF582Z88803+PTTT6HT6RAREYFff/3V5HxWMSp+04wBaOOsFB3DLL1f/050hAZ5erCf6AgNNjHQQ3SEBvnsQo7oCA0S6tdOdIQGker5BoBz1++IjmC2slJ1sx2rsUbFe3t7Gy1fuHAhFi1aZMGefxMeHo7w8HDDfEREBHr37o3169djyZIlJu3DKgo7ERFRvRqpsmdnZ0Op/K0xqVAoHrh6x44dYWdnh7y8PKPleXl5UKlUJh2yVatWCAkJQXp6uskx2RVPREQ2obEGzymVSqOptsLu4OCAsLAwHD582LBMp9Ph8OHDRq3yumi1Wly4cAEeHqb3OLLFTkRE1ETmzZuH6Oho9O/fHwMGDMAHH3yAsrIyTJkyBQAwadIkeHl5IS4uDgDwzjvv4OGHH0b37t1RVFSE999/H5mZmZg+fbrJx2RhJyIimyCDZc+Kb8im48aNQ0FBAd5++23k5uaiX79+OHjwoGFAXVZWFuTy3zrP79y5gxkzZiA3Nxft2rVDWFgYkpOTERAQYPIxWdiJiMgmiHqk7KxZszBr1qwH/iwpKclofuXKlVi5cmUDj3QP77ETERFZEbbYiYjIJjTkITN/3F4KWNiJiMhG2Mb73dgVT0REZEXYYiciIpvArngiIiIrYhsd8eyKJyIisipssRMRkU1gVzwREZEV+f3z3hu6vRSwsBMRkW2wkZvsvMdORERkRdhi/4P/XcrAF3uP4+r1m7h9pwSL50/AIwNMf/i+KOPDfTA+3Bde7ZwAAOl5pVibeBXH0goEJ6vb4907INBDCTcXB1Rp9cgsLMe+S/koKKsUHa1eUr1WpHzOAWDjziP48NPDyL+tRt8eXvjH62MR1qeL6Fi1kvL5luo1XhsbabCzxf5HdzWV6NZFhVenPSM6illyiyqwYv8V/HnVcYxddQKn0m9jzeT+6O7uLDpanbp1bIPkjEKsPpaB9SczIZfL8FK4DxzsWv5HSKrXipTP+dffn8WbH+zGgukjkbR9Afr28MKfXvkIBYUloqPVSsrnW6rXeG1qBs9ZMkmB0MLepUsXyGSy+6aYmBhhmQaG9MTU8cMk91dp0uV8HL1SgMxb5ci4VYZVB9NQXlmNYJ92oqPVaeOpLKRkFyOvRIMctQYJ52+ifWsHdHZ1Eh2tXlK9VqR8ztfu+AGTIiMw8dlw9OrqgRWx49Ha0QGf7j0pOlqtpHy+pXqN2zqhhT0lJQU5OTmGKTExEQAwduxYkbEkTy4Dng72QGsHO6Rm3hEdxyyOre5dkuVVWsFJbIdUznllVTVSr2Rj6AB/wzK5XI4hA/yRcuG6wGTmkcr5tkayRvhPCoTeY+/UqZPR/LJly9CtWzcMGTJEUCJp66FyweezIqCwl6O8UotXPjmLa/mlomOZTAYgso8K12+XI7dEIzqOTZDSOb9dVAqtVodO7V2Mlndqr8TVjDxBqcwjpfNtlWzkJnuLucdeWVmJTz/9FFOnToWslhsZGo0GarXaaKLfZBSUImrlMYz78AQSTmYiblwwurm17HvsvxcVpIJKqcD2s7+KjmIzeM6bF883NYcWU9j37NmDoqIiTJ48udZ14uLi4Orqapi8vb2bL6AEVGn1yLpdjks31Fh5IA1pOSV4cXAX0bFM8lygCgHuLohPzkRxRbXoODZBaue8Q1tn2NnJ7xsoV1CohlsHpaBUppPa+bZGskaYpKDFFPaPP/4YI0eOhKenZ63rxMbGori42DBlZ2c3Y0LpkckAB/sW8z9xrZ4LVCFQde8XXmF5leg4NkGK59yhlT369fLGkZQ0wzKdToejKT/joUA/gcnqJ8XzbY1sZVR8i/gee2ZmJv7973/j66+/rnM9hUIBhULRpFnuVmhwI7fQMJ+bX4T0jBy4ODvBvWPbJj22JeaO9MexKwW4WXQXbRT2eCbEEwO6dsCMTadFR6tTVKAKoZ1dsfl0NjTVWrgo7AAAd6t0qNbpBaerm1SvFSmf85l/eRwzF29HSG8fhPbpgvjPf0TZXQ0mjn5YdLRaSfl8S/Uat3UtorBv2bIFbm5uGDVqlOgoSLt2E68t3myYj992AAAwfEgIFsREiYpVrw7OCiwbH4xOSgVKKqrxc04JZmw6jeSrt0RHq9Mgv/YAgJhBXYyWJ5y/gZTsYgGJTCfVa0XK5zxqeBhuFZVi6frvkH+7BIE9vbBrdUyL7oqX8vmW6jVeO0tHtkujyS7T6/VC/2TU6XTw8/PDhAkTsGzZMrO2VavVcHV1xffnMtDGueV+sB9k4tpk0REa5OnBLbvLsy4TAz1ER2iQzy7kiI7QIMufleZ3n1/be0l0hAaT4jVeVqrG8NAuKC4uhlLZNL/Ha2pFRk6hRcdQq9Xo4tG+SbM2BuE3YP/9738jKysLU6dOFR2FiIhI8oR3xQ8fPhyCOw2IiIishvDCTkRE1BwsHdnOUfFEREQtiKWPhZXKI2WF32MnIiKixsMWOxER2QR2xRMREVkRG3kHDLviiYiIrAlb7EREZBtspMnOwk5ERDaBo+KJiIhIcthiJyIim8BR8URERFbERm6xs7ATEZGNsJHKznvsRERETeijjz5Cly5d4OjoiIEDB+L06dN1rv/ll1+iV69ecHR0RGBgIPbv32/W8VjYiYjIJsga4T9zffHFF5g3bx4WLlyIc+fOITg4GCNGjEB+fv4D109OTsaECRMwbdo0nD9/HpGRkYiMjMTFixdNPiYLOxER2YSawXOWTOZasWIFZsyYgSlTpiAgIADr1q1D69atsXnz5geuv2rVKjz11FN4/fXX0bt3byxZsgShoaFYs2aNyceU9D32mve4l5WWCE5iPp2mXHSEBqksLxUdocHKStWiIzSIVM+5Ws3z3dykeI3X/P6u+X3elCy9Jmu2/+N+FAoFFArFfetXVlbi7NmziI2NNSyTy+UYNmwYTp48+cBjnDx5EvPmzTNaNmLECOzZs8fknJIu7CUl9y6I5x4NFJzEdmyKF52g4TaJDmBjeL6bn5TPeUlJCVxdXZtk3w4ODlCpVOjh523xvpydneHtbbyfhQsXYtGiRfete+vWLWi1Wri7uxstd3d3x5UrVx64/9zc3Aeun5uba3JGSRd2T09PZGdnw8XFBbJG/oKhWq2Gt7c3srOzoVQqG3XfTUmquQHpZmfu5sXcza8ps+v1epSUlMDT07NR9/t7jo6OuH79OiorKy3el16vv6/ePKi1LpKkC7tcLkfnzp2b9BhKpVJyH0JAurkB6WZn7ubF3M2vqbI3VUv99xwdHeHo6Njkx/m9jh07ws7ODnl5eUbL8/LyoFKpHriNSqUya/0H4eA5IiKiJuDg4ICwsDAcPnzYsEyn0+Hw4cMIDw9/4Dbh4eFG6wNAYmJires/iKRb7ERERC3ZvHnzEB0djf79+2PAgAH44IMPUFZWhilTpgAAJk2aBC8vL8TFxQEAZs+ejSFDhmD58uUYNWoUEhIScObMGWzYsMHkY7Kw10KhUGDhwoUt7t5JfaSaG5BuduZuXszd/KScXbRx48ahoKAAb7/9NnJzc9GvXz8cPHjQMEAuKysLcvlvnecRERHYsWMH3nzzTfz9739Hjx49sGfPHvTt29fkY8r0zfEdAyIiImoWvMdORERkRVjYiYiIrAgLOxERkRVhYSciIrIiLOy1MPc1ey3B0aNHMXr0aHh6ekImk5n1bGFR4uLi8NBDD8HFxQVubm6IjIxEWlqa6FgmiY+PR1BQkOGhHeHh4Thw4IDoWGZZtmwZZDIZ5syZIzpKvRYtWgSZTGY09erVS3Qsk9y4cQMvvPACOnToACcnJwQGBuLMmTOiY9WpS5cu951vmUyGmJgY0dGoHizsD2Dua/ZairKyMgQHB+Ojjz4SHcVkR44cQUxMDE6dOoXExERUVVVh+PDhKCsrEx2tXp07d8ayZctw9uxZnDlzBo8//jjGjBmDn376SXQ0k6SkpGD9+vUICgoSHcVkffr0QU5OjmE6fvy46Ej1unPnDgYNGoRWrVrhwIEDuHTpEpYvX4527dqJjlanlJQUo3OdmJgIABg7dqzgZFQvPd1nwIAB+piYGMO8VqvVe3p66uPi4gSmMg8A/e7du0XHMFt+fr4egP7IkSOiozRIu3bt9Js2bRIdo14lJSX6Hj166BMTE/VDhgzRz549W3Skei1cuFAfHBwsOobZFixYoH/kkUdEx7DY7Nmz9d26ddPrdDrRUagebLH/Qc1r9oYNG2ZYVt9r9qjxFBcXAwDat28vOIl5tFotEhISUFZWZtajH0WJiYnBqFGjjK5zKbh69So8PT3RtWtXTJw4EVlZWaIj1Wvv3r3o378/xo4dCzc3N4SEhGDjxo2iY5mlsrISn376KaZOndroL9yixsfC/gd1vWbPnNfmkfl0Oh3mzJmDQYMGmfWUJZEuXLgAZ2dnKBQK/PWvf8Xu3bsREBAgOladEhIScO7cOcMjLKVi4MCB2Lp1Kw4ePIj4+Hhcv34dgwcPNry+uaX65ZdfEB8fjx49euDQoUN4+eWX8eqrr+KTTz4RHc1ke/bsQVFRESZPniw6CpmAj5SlFiMmJgYXL16UxH3TGv7+/khNTUVxcTF27dqF6OhoHDlypMUW9+zsbMyePRuJiYnN/qYrS40cOdLw/wcFBWHgwIHw9fXFzp07MW3aNIHJ6qbT6dC/f38sXboUABASEoKLFy9i3bp1iI6OFpzONB9//DFGjhzZpK9WpcbDFvsfNOQ1e2S5WbNmYd++ffjxxx+b/FW8jcnBwQHdu3dHWFgY4uLiEBwcjFWrVomOVauzZ88iPz8foaGhsLe3h729PY4cOYLVq1fD3t4eWq1WdESTtW3bFj179kR6erroKHXy8PC47w+93r17S+I2AgBkZmbi3//+N6ZPny46CpmIhf0PGvKaPWo4vV6PWbNmYffu3fjhhx/g5+cnOpJFdDodNBqN6Bi1euKJJ3DhwgWkpqYapv79+2PixIlITU2FnZ2d6IgmKy0txbVr1+Dh4SE6Sp0GDRp031c4f/75Z/j6+gpKZJ4tW7bAzc0No0aNEh2FTMSu+Aeo7zV7LVVpaalR6+X69etITU1F+/bt4ePjIzBZ7WJiYrBjxw588803cHFxMYxjcHV1hZOTk+B0dYuNjcXIkSPh4+ODkpIS7NixA0lJSTh06JDoaLVycXG5b/xCmzZt0KFDhxY/rmH+/PkYPXo0fH19cfPmTSxcuBB2dnaYMGGC6Gh1mjt3LiIiIrB06VI8//zzOH36NDZs2GDWazhF0el02LJlC6Kjo2Fvz3IhGaKH5bdUH374od7Hx0fv4OCgHzBggP7UqVOiI9Xrxx9/1AO4b4qOjhYdrVYPygtAv2XLFtHR6jV16lS9r6+v3sHBQd+pUyf9E088of/+++9FxzKbVL7uNm7cOL2Hh4fewcFB7+XlpR83bpw+PT1ddCyTfPvtt/q+ffvqFQqFvlevXvoNGzaIjmSSQ4cO6QHo09LSREchM/C1rURERFaE99iJiIisCAs7ERGRFWFhJyIisiIs7ERERFaEhZ2IiMiKsLATERFZERZ2IiIiK8LCTmShyZMnIzIy0jA/dOhQzJkzp9lzJCUlQSaToaioqNZ1ZDIZ9uzZY/I+Fy1ahH79+lmUKyMjAzKZDKmpqRbth4hMw8JOVmny5MmQyWSQyWSGF7W88847qK6ubvJjf/3111iyZIlJ65pSjImIzMGH/5LVeuqpp7BlyxZoNBrs378fMTExaNWqFWJjY+9bt7KyEg4ODo1y3Pbt2zfKfoiIGoItdrJaCoUCKpUKvr6+ePnllzFs2DDs3bsXwG/d5++99x48PT3h7+8P4N77yp9//nm0bdsW7du3x5gxY5CRkWHYp1arxbx589C2bVt06NABb7zxBv74VOY/dsVrNBosWLAA3t7eUCgU6N69Oz7++GNkZGTgscceAwC0a9cOMpkMkydPBnDv5RtxcXHw8/ODk5MTgoODsWvXLqPj7N+/Hz179oSTkxMee+wxo5ymWrBgAXr27InWrVuja9eueOutt1BVVXXfeuvXr4e3tzdat26N559/HsXFxUY/37RpE3r37g1HR0f06tULa9euNTsLETUOFnayGU5OTqisrDTMHz58GGlpaUhMTMS+fftQVVWFESNGwMXFBceOHcOJEyfg7OyMp556yrDd8uXLsXXrVmzevBnHjx9HYWEhdu/eXedxJ02ahM8//xyrV6/G5cuXsX79ejg7O8Pb2xtfffUVACAtLQ05OTmGd7nHxcVh27ZtWLduHX766SfMnTsXL7zwAo4cOQLg3h8gUVFRGD16NFJTUzF9+nT87W9/M/ucuLi4YOvWrbh06RJWrVqFjRs3YuXKlUbrpKenY+fOnfj2229x8OBBnD9/HjNnzjT8/LPPPsPbb7+N9957D5cvX8bSpUvx1ltv4ZNPPjE7DxE1AsEvoSFqEtHR0foxY8bo9Xq9XqfT6RMTE/UKhUI/f/58w8/d3d31Go3GsM327dv1/v7+ep1OZ1im0Wj0Tk5O+kOHDun1er3ew8ND/89//tPw86qqKn3nzp0Nx9Lrjd+WlpaWpgegT0xMfGDOmjfy3blzx7CsoqJC37p1a31ycrLRutOmTdNPmDBBr9fr9bGxsfqAgACjny9YsOC+ff0RAP3u3btr/fn777+vDwsLM8wvXLhQb2dnp//1118Nyw4cOKCXy+X6nJwcvV6v13fr1k2/Y8cOo/0sWbJEHx4ertfr9frr16/rAejPnz9f63GJqPHwHjtZrX379sHZ2RlVVVXQ6XT4y1/+gkWLFhl+HhgYaHRf/b///S/S09Ph4uJitJ+Kigpcu3YNxcXFyMnJwcCBAw0/s7e3R//+/e/rjq+RmpoKOzs7DBkyxOTc6enpKC8vx5NPPmm0vLKyEiEhIQCAy5cvG+UAgPDwcJOPUeOLL77A6tWrce3aNZSWlqK6uhpKpdJoHR8fH3h5eRkdR6fTIS0tDS4uLrh27RqmTZuGGTNmGNaprq6Gq6ur2XmIyHIs7GS1HnvsMcTHx8PBwQGenp6wtze+3Nu0aWM0X1pairCwMHz22Wf37atTp04NyuDk5GT2NqWlpQCA7777zqigAvfGDTSWkydPYuLEiVi8eDFGjBgBV1dXJCQkYPny5WZn3bhx431/aNjZ2TVaViIyHQs7Wa02bdqge/fuJq8fGhqKL774Am5ubve1Wmt4eHjgP//5Dx599FEA91qmZ8+eRWho6APXDwwMhE6nw5EjRzBs2LD7fl7TY6DVag3LAgICoFAokJWVVWtLv3fv3oaBgDVOnTpV/z/yd5KTk+Hr64v/9//+n2FZZmbmfetlZWXh5s2b8PT0NBxHLpfD398f7u7u8PT0xC+//IKJEyeadXwiahocPEf0fyZOnIiOHTtizJgxOHbsGK5fv46kpCS8+uqr+PXXXwEAs2fPxrJly7Bnzx5cuXIFM2fOrPM76F26dEF0dDSmTp2KPXv2GPa5c+dOAICvry9kMhn27duHgoIClJaWwsXFBfPnz8fcuXPxySef4Nq1azh37hw+/PBDw4C0v/71r7h69Spef/11pKWlYceOHdi6datZ/94ePXogKysLCQkJuHbtGlavXv3AgYCOjo6Ijo7Gf//7Xxw7dgyvvvoqnn/+eahUKgDA4sWLERcXh9WrV+Pnn3/GhQsXsGXLFqxYscKsPETUOFjYif5P69atcfToUfj4+CAqKgq9e/fGtGnTUFFRYWjBv/baa3jxxRcRHR2N8PBwuLi44Lnnnqtzv/Hx8fjzn/+MmTNnolevXpgxYwbKysoAAF5eXli8eDH+9re/wd3dHbNmzQIALFmyBG+99Rbi4uLQu3dvPPXUU/juu+/g5+cH4N5976+++gp79uxBcHAw1q1bh6VLl5r173322Wcxd+5czJo1C/369UNycjLeeuut+9br3r07oqKi8PTTT2P48OEICgoy+jrb9OnTsWnTJmzZsgWBgYEYMmQItm7dashKRM1Lpq9t1A8RERFJDlvsREREVoSFnYiIyIqwsBMREVkRFnYiIiIrwsJORERkRVjYiYiIrAgLOxERkRVhYSciIrIiLOxERERWhIWdiIjIirCwExERWREWdiIiIivy/wHqRD9ZoDFObwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def generate_confusion_matrix(model, df, predicted_labels):\n",
    "    true_labels = df[\"narrator\"].values\n",
    "\n",
    "    class_labels = sorted(df[\"narrator\"].unique())\n",
    "    cm = confusion_matrix(true_labels, predicted_labels, labels=class_labels)\n",
    "\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=class_labels)\n",
    "\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".4g\")\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "generate_confusion_matrix(model, df, predicted_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1397ad93-eb4e-4d57-8867-35d8ad62e34c",
   "metadata": {},
   "source": [
    "The confusion matrix confirms that the multiclass classification performs very poorly. \n",
    "\n",
    "The classification is almost random."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}