deep-learning-course-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep learning course project\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This Jupyter Notebook implements a deep learning project using the BERT-GAN architecture for text classification. The project involves training a BERT-based generator and discriminator to perform text classification on a SemEval dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and Environment\n",
    "\n",
    "here we implemenet the needed libraries for this project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2024-02-03T15:30:24.935782Z",
     "iopub.status.busy": "2024-02-03T15:30:24.935427Z",
     "iopub.status.idle": "2024-02-03T15:30:24.943947Z",
     "shell.execute_reply": "2024-02-03T15:30:24.942880Z",
     "shell.execute_reply.started": "2024-02-03T15:30:24.935755Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using cuda device\n",
      "using 2 GPUs!\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "import torchvision\n",
    "from torch.utils.data import Dataset, DataLoader, random_split\n",
    "import os\n",
    "\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "print(f\"using {device} device\")\n",
    "print(f\"using {torch.cuda.device_count()} GPUs!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.status.idle": "2024-02-03T14:49:07.308734Z",
     "shell.execute_reply": "2024-02-03T14:49:07.307581Z",
     "shell.execute_reply.started": "2024-02-03T14:48:54.347414Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: gdown\n",
      "Successfully installed gdown-5.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install gdown\n",
    "DOWNLOAD = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.status.idle": "2024-02-03T14:50:59.643800Z",
     "shell.execute_reply": "2024-02-03T14:50:59.642792Z",
     "shell.execute_reply.started": "2024-02-03T14:49:39.775072Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed peft-0.8.2\n"
     ]
    }
   ],
   "source": [
    "!pip install git+https://github.com/adapter-hub/adapters.git\n",
    "!pip install git+https://github.com/huggingface/peft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "in this section we create the `Dataset` and `DataLoader` classes that we will be using for this project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom `Dataset` classes\n",
    "\n",
    "This section defines two PyTorch datasets, `SemEvalDataset` and `MaskedSemEvalDataset`, tailored for handling SemEval-format datasets. The `SemEvalDataset` class initializes by optionally downloading the dataset, loading it from a specified path, and storing tokenized representations. The derived class, `MaskedSemEvalDataset`, extends the functionality by introducing a static mask for each sample based on a specified masking percentage. Both datasets enable efficient handling of raw text, labels, and tokenized representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:30:57.994178Z",
     "iopub.status.busy": "2024-02-03T15:30:57.993340Z",
     "iopub.status.idle": "2024-02-03T15:30:58.009207Z",
     "shell.execute_reply": "2024-02-03T15:30:58.008147Z",
     "shell.execute_reply.started": "2024-02-03T15:30:57.994148Z"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class SemEvalDataset(Dataset):\n",
    "    def __init__(self, path, url=None, download=False):\n",
    "        if download and url is not None:\n",
    "            import gdown\n",
    "            gdown.download(url, path, quiet=False, fuzzy=True)\n",
    "        \n",
    "        with open(path) as f:\n",
    "            self.data = [json.loads(line) for line in f]\n",
    "            \n",
    "        self.K = len(set(sample['label'] for sample in self.data))\n",
    "        self.path = path\n",
    "        \n",
    "        self.tokenized_data = []\n",
    "        self.tokenizer = None\n",
    "        self.token_opts = {}\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        if self.tokenizer is None:\n",
    "            token = []\n",
    "        elif idx < len(self.tokenized_data):\n",
    "            token = self.tokenized_data[idx]\n",
    "        else:\n",
    "            token = self.tokenizer(self.data[idx]['text'], **self.token_opts)\n",
    "            token.pop('token_type_ids', None)\n",
    "            token['input_ids'] = token['input_ids'].squeeze(dim=0)\n",
    "            token['attention_mask'] = token['attention_mask'].squeeze(dim=0)\n",
    "        return self.data[idx]['text'], self.data[idx]['label'], token\n",
    "    \n",
    "    def tokenize_data(self, tokenizer, opts):\n",
    "        from tqdm import tqdm\n",
    "        self.tokenized_data = []\n",
    "        self.tokenizer = tokenizer\n",
    "        self.token_opts = opts\n",
    "        for sample in tqdm(self.data, desc='Tokenizing data'):\n",
    "            text = sample['text']\n",
    "            token = tokenizer(text, **opts)\n",
    "#             tokens = {k: v.squeeze(dim=0) for k, v in tokens.items()}\n",
    "            token.pop('token_type_ids', None)\n",
    "            token['input_ids'] = token['input_ids'].squeeze(dim=0)\n",
    "            token['attention_mask'] = token['attention_mask'].squeeze(dim=0)\n",
    "            self.tokenized_data.append(token)\n",
    "        \n",
    "    \n",
    "class MaskedSemEvalDataset(SemEvalDataset):\n",
    "    def __init__(self, path, url=None, download=False, mask_percentage=0.5):\n",
    "        super().__init__(path, url, download)\n",
    "        self.mask_percentage = mask_percentage\n",
    "        # Generate a static mask\n",
    "        self.masks = [1 if random.random() < self.mask_percentage else 0 for _ in range(len(self))]\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        text, label, token = super().__getitem__(idx)\n",
    "        mask = self.masks[idx]\n",
    "\n",
    "        return text, label, mask, token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading the dataset\n",
    "\n",
    "in this section we download the original dataset from the given repository.\n",
    "\n",
    "equivalently one can just load the datasets via `pickle` from the later code sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T14:49:13.485508Z",
     "iopub.status.busy": "2024-02-03T14:49:13.484588Z",
     "iopub.status.idle": "2024-02-03T14:49:24.393760Z",
     "shell.execute_reply": "2024-02-03T14:49:24.392909Z",
     "shell.execute_reply.started": "2024-02-03T14:49:13.485474Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading...\n",
      "From (original): https://drive.google.com/uc?id=1k5LMwmYF7PF-BzYQNE2ULBae79nbM268\n",
      "From (redirected): https://drive.google.com/uc?id=1k5LMwmYF7PF-BzYQNE2ULBae79nbM268&confirm=t&uuid=9594dc83-4ab0-4232-a7e9-9dc09d8e160a\n",
      "To: /kaggle/working/train.json\n",
      "100%|██████████| 155M/155M [00:00<00:00, 262MB/s] \n",
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1oh9c-d0fo3NtETNySmCNLUc6H1j4dSWE\n",
      "To: /kaggle/working/test.json\n",
      "100%|██████████| 4.93M/4.93M [00:00<00:00, 195MB/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 67475\n",
      "Number of validation examples: 3552\n",
      "Number of test examples: 3000\n",
      "Number of classes: 6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# import gdown\n",
    "train_url = 'https://drive.google.com/file/d/1k5LMwmYF7PF-BzYQNE2ULBae79nbM268/view?usp=drive_link'\n",
    "test_url  = 'https://drive.google.com/file/d/1oh9c-d0fo3NtETNySmCNLUc6H1j4dSWE/view?usp=drive_link'\n",
    "\n",
    "# train_url = test_url # JUST FOR NOW!!!!\n",
    "\n",
    "train_path = 'train.json'\n",
    "test_path = 'test.json'\n",
    "\n",
    "full_dataset = SemEvalDataset(path=train_path, url=train_url, download=DOWNLOAD)\n",
    "test_dataset = SemEvalDataset(path=test_path, url=test_url,  download=DOWNLOAD)\n",
    "\n",
    "DOWNLOAD = False\n",
    "\n",
    "TOKENIZE = True\n",
    "\n",
    "num_samples = len(full_dataset)\n",
    "train_size = int(0.95 * num_samples)\n",
    "val_size = num_samples - train_size\n",
    "\n",
    "K = full_dataset.K\n",
    "\n",
    "print(f\"Number of training examples: {train_size}\")\n",
    "print(f\"Number of validation examples: {val_size}\")\n",
    "print(f\"Number of test examples: {len(test_dataset)}\")\n",
    "print(f\"Number of classes: {K}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading bert\n",
    "\n",
    "\n",
    "This section defines a function `get_bert` that loads pre-trained `BERT` or `DistilBERT` models along with their respective tokenizers. \n",
    "\n",
    "It includes an option to extend the model using the PEFT (Parameter Efficient Fine-Tuning) technique, allowing for different PEFT methods like `lora`, `adalora`, or `IA3`. Additionally, the function can print the number of trainable parameters in the loaded model when the `print_params` parameter is set to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:31:02.283928Z",
     "iopub.status.busy": "2024-02-03T15:31:02.283567Z",
     "iopub.status.idle": "2024-02-03T15:31:02.295753Z",
     "shell.execute_reply": "2024-02-03T15:31:02.294764Z",
     "shell.execute_reply.started": "2024-02-03T15:31:02.283900Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertModel, DistilBertTokenizer, DistilBertModel\n",
    "\n",
    "def print_trainable_parameters(model):\n",
    "    \"\"\"\n",
    "    Prints the number of trainable parameters in the model.\n",
    "    \"\"\"\n",
    "    trainable_params = 0\n",
    "    all_param = 0\n",
    "    for _, param in model.named_parameters():\n",
    "        all_param += param.numel()\n",
    "        if param.requires_grad:\n",
    "            trainable_params += param.numel()\n",
    "    print(\n",
    "        f\"trainable params: {trainable_params:.2e} || all params: {all_param:.2e} || trainable: {100 * trainable_params / all_param:.2f}%\"\n",
    "    )\n",
    "\n",
    "def get_bert(model=\"bert\", use_peft=False, print_params=False, peft_method=\"lora\"):\n",
    "    # Load pre-trained BERT\n",
    "    if model == \"bert\":\n",
    "        tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "        bert = BertModel.from_pretrained('bert-base-uncased')\n",
    "    elif model == \"distil\":\n",
    "        tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "        bert = DistilBertModel.from_pretrained('distilbert-base-uncased')\n",
    "    else:\n",
    "        raise ValueError(\"Invalid bert model\")\n",
    "\n",
    "    # Peft\n",
    "    if use_peft:\n",
    "        if peft_method == \"lora\":\n",
    "            from peft import LoraConfig, LoraModel\n",
    "            # Let's define the LoraConfig\n",
    "            config = LoraConfig(\n",
    "                r=16, lora_alpha=16, lora_dropout=0.01,\n",
    "                bias=\"all\",  task_type=\"SEQ_CLS\", #target_modules=[\"q\", \"v\"],,\n",
    "            )\n",
    "            bert = LoraModel(bert, config, \"default\")\n",
    "        elif peft_method == \"adalora\":\n",
    "            from peft import AdaLoraModel, AdaLoraConfig\n",
    "            config = AdaLoraConfig(\n",
    "                peft_type=\"ADALORA\", task_type=\"SEQ_CLS\", r=8, lora_alpha=32,# target_modules=[\"q\", \"v\"],\n",
    "                lora_dropout=0.01, adapter_name=\"default\",\n",
    "            )\n",
    "            bert = AdaLoraModel(bert, config, \"default\")\n",
    "        elif peft_method == \"IA3\":\n",
    "            from peft import IA3Model, IA3Config\n",
    "            config = IA3Config(\n",
    "                peft_type=\"IA3\", task_type=\"SEQ_CLS\",\n",
    "#                 target_modules=[\"k\", \"v\", \"w0\"], feedforward_modules=[\"w0\"],\n",
    "            )\n",
    "            bert = IA3Model(bert, config, \"default\")\n",
    "            \n",
    "    if print_params:\n",
    "        print_trainable_parameters(bert)\n",
    "    \n",
    "    return bert, tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization and Preprocessing\n",
    "\n",
    "in this section we preprocess the data, as in tokenizing the dataset, and finding good parameters for things such as `MAX_LEN` for tokenization, and seeing the data distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T14:51:09.497590Z",
     "iopub.status.busy": "2024-02-03T14:51:09.496593Z",
     "iopub.status.idle": "2024-02-03T14:51:13.377240Z",
     "shell.execute_reply": "2024-02-03T14:51:13.376362Z",
     "shell.execute_reply.started": "2024-02-03T14:51:09.497544Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41a3089261fa465fa204391f812fceb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f91d8c571994c5abe2f0fb1d1458b44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e498b966abf4d6fb27b77b2d879353c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e0567ab9f74c4cb49018d4f496090f7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/570 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5dcf2bff7f214a60909ed500e0ad25f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/440M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 6.45e+04 || all params: 1.10e+08 || trainable: 0.06%\n",
      "Last hidden states:  torch.Size([1, 7, 768])\n"
     ]
    }
   ],
   "source": [
    "bert, tokenizer = get_bert(model=\"bert\", use_peft=True, print_params=True, peft_method=\"IA3\")\n",
    "\n",
    "# Test\n",
    "text = \"Ali nourian save me\"\n",
    "tokens = tokenizer(text, return_tensors='pt')\n",
    "outputs = bert(**tokens)\n",
    "\n",
    "last_hidden_states = outputs.last_hidden_state\n",
    "_, _, hidden_dim = last_hidden_states.shape\n",
    "print(\"Last hidden states: \", last_hidden_states.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data distribution\n",
    "\n",
    "in this section we visualize the datasets length and class distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T15:50:54.548642Z",
     "iopub.status.busy": "2024-02-02T15:50:54.548252Z",
     "iopub.status.idle": "2024-02-02T15:50:56.307985Z",
     "shell.execute_reply": "2024-02-02T15:50:56.307003Z",
     "shell.execute_reply.started": "2024-02-02T15:50:54.548613Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 71027/71027 [00:01<00:00, 40574.39it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "class_counts = [0] * K\n",
    "sequence_lengths = []\n",
    "\n",
    "for text, label, _ in tqdm(full_dataset):\n",
    "    class_counts[label] += 1\n",
    "#     num_tokens = len(tokenizer(text)['input_ids'])\n",
    "    num_tokens = len(text.split(' '))\n",
    "    sequence_lengths.append(num_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T15:51:00.851271Z",
     "iopub.status.busy": "2024-02-02T15:51:00.850736Z",
     "iopub.status.idle": "2024-02-02T15:51:03.125861Z",
     "shell.execute_reply": "2024-02-02T15:51:03.124973Z",
     "shell.execute_reply.started": "2024-02-02T15:51:00.851225Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAHHCAYAAACiOWx7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLBklEQVR4nO3dfXyP9f////tryzZ2amqbRTPn5jSUlpPIMqdRIplIi961xehNvItQEeU8pyXUh7ciVOqNoZpYzpdzIaHYlNlmZJvt+P3Rd8fPq6HjtTZ7ye16ubwub8fz+TyO1+M4Zm/3juP5er5shmEYAgAAwHW5lHQBAAAANwNCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhNQTCpVqqSnnnqqpMsodgsWLJDNZtP27duL7JhPPfWUKlWqVGTHKyktW7ZUy5Ytb8h72Ww2xcbG3pD3Am5VhCbAQUePHtWzzz6rypUry8PDQz4+PmratKmmTp2q33//vaTLu67iCDi4cTZv3qxRo0YpLS2txGqoVKmSbDabbDabXFxc5Ofnp7p166p///7asmXL3zr22LFjtXLlyqIp9G/av3+/Ro0apZ9++qmkS4ETua2kCwBuJl988YW6desmd3d39e7dW3Xq1FF2dra+/fZbDRkyRPv27dPcuXNLukw4ibVr1xbp8TZv3qzRo0frqaeekp+fX5Ee2xENGjTQiy++KEk6f/68Dhw4oKVLl+rdd9/VoEGDNGnSpEIdd+zYsXrsscfUpUuXIqy2cPbv36/Ro0erZcuW/4i7nigahCbAomPHjqlHjx4KCQnRhg0bVL58ebMvJiZGR44c0RdffFGCFcLZuLm5lXQJxeLOO+9Ur1697NrGjx+vnj17avLkyapWrZqee+65EqoOKD48ngMsmjBhgjIzMzVv3jy7wJSvatWqGjhw4DX3T01N1b///W/VrVtXXl5e8vHxUbt27fT9998XGDt9+nTVrl1bZcqUUdmyZdW4cWMtXrzY7D9//rzi4uJUqVIlubu7KyAgQA899JB27tzp8Hk99dRT8vLy0okTJ9SxY0d5eXnpzjvv1IwZMyRJe/bs0YMPPihPT0+FhITY1XGlixcv6tlnn1W5cuXk4+Oj3r1769y5c3ZjPv30U3Xo0EHBwcFyd3dXlSpV9Nprryk3N/cv63z77bd1//33q1y5cipdurQaNWqkZcuWFRiXP7dn5cqVqlOnjtzd3VW7dm2tXr26wNhffvlF0dHRZj2hoaF67rnnlJ2dbY5JS0tTXFycKlasKHd3d1WtWlXjx49XXl7eX9b85zlNX3/9tWw2mz7++GO98cYbqlChgjw8PNS6dWsdOXLkuscaNWqUhgwZIkkKDQ01H5H9+fGR1fN++umnFRgYaI57//33//J8rqd06dL68MMP5e/vrzfeeEOGYZh9Vn52NptNFy5c0MKFC81zy58TePz4cT3//POqUaOGSpcurXLlyqlbt24Fzj0nJ0ejR49WtWrV5OHhoXLlyqlZs2aKj4+3G3fw4EE99thj8vf3l4eHhxo3bqzPPvvM7F+wYIG6desmSWrVqpVZz9dff/23rhFuftxpAiz6/PPPVblyZd1///2F2v/HH3/UypUr1a1bN4WGhiolJUVz5szRAw88oP379ys4OFiS9O6772rAgAF67LHHNHDgQF26dEm7d+/Wli1b1LNnT0nSv/71Ly1btkyxsbEKCwvT2bNn9e233+rAgQNq2LChw7Xl5uaqXbt2atGihSZMmKBFixYpNjZWnp6eevnllxUVFaVHH31Us2fPVu/evRUeHq7Q0FC7Y8TGxsrPz0+jRo3SoUOHNGvWLB0/ftwMCtIf/xh5eXlp8ODB8vLy0oYNGzRy5EhlZGTorbfeum6NU6dO1cMPP6yoqChlZ2dryZIl6tatm1atWqUOHTrYjf3222+1fPlyPf/88/L29ta0adPUtWtXnThxQuXKlZMknTp1Svfee6/S0tLUv39/1axZU7/88ouWLVumixcvys3NTRcvXtQDDzygX375Rc8++6zuuusubd68WcOHD9fp06c1ZcoUh6+1JL355ptycXHRv//9b6Wnp2vChAmKioq67pygRx99VD/88IP++9//avLkybr99tslSXfccYdD552SkqL77rvPDJd33HGH/ve//yk6OloZGRmKi4sr1DlJkpeXlx555BHNmzdP+/fvV+3atSVZ+9l9+OGHeuaZZ3Tvvfeqf//+kqQqVapIkrZt26bNmzerR48eqlChgn766SfNmjVLLVu21P79+1WmTBlJfwTLcePGmcfJyMjQ9u3btXPnTj300EOSpH379qlp06a68847NWzYMHl6eurjjz9Wly5d9Mknn+iRRx5RixYtNGDAAE2bNk3/+c9/VKtWLUky/xe3MAPAX0pPTzckGZ07d7a8T0hIiNGnTx9z+9KlS0Zubq7dmGPHjhnu7u7GmDFjzLbOnTsbtWvXvu6xfX19jZiYGMu15Js/f74hydi2bZvZ1qdPH0OSMXbsWLPt3LlzRunSpQ2bzWYsWbLEbD948KAhyXj11VcLHLNRo0ZGdna22T5hwgRDkvHpp5+abRcvXixQ07PPPmuUKVPGuHTpkl1NISEhduP+vG92drZRp04d48EHH7Rrl2S4ubkZR44cMdu+//57Q5Ixffp0s613796Gi4uL3bXIl5eXZxiGYbz22muGp6en8cMPP9j1Dxs2zHB1dTVOnDhRYN8rPfDAA8YDDzxgbn/11VeGJKNWrVpGVlaW2T516lRDkrFnz57rHu+tt94yJBnHjh0r0Gf1vKOjo43y5csbv/32m93+PXr0MHx9fa/6M7pSSEiI0aFDh2v2T548+S9/7tf62Xl6etr9zlxrf8MwjMTEREOS8cEHH5ht9evXv25thmEYrVu3NurWrWv39y0vL8+4//77jWrVqpltS5cuNSQZX3311XWPh1sLj+cACzIyMiRJ3t7ehT6Gu7u7XFz++JXLzc3V2bNn5eXlpRo1atg9VvPz89PPP/+sbdu2XfNYfn5+2rJli06dOlXoev7smWeesTt+jRo15Onpqe7du5vtNWrUkJ+fn3788ccC+/fv31+lSpUyt5977jnddttt+vLLL8220qVLm38+f/68fvvtNzVv3lwXL17UwYMHr1vflfueO3dO6enpat68+VUfSUZERJh3KSSpXr168vHxMevOy8vTypUr1alTJzVu3LjA/vl3xpYuXarmzZurbNmy+u2338xXRESEcnNzlZCQcN2ar6Vv3752852aN28uSVe9ro74q/M2DEOffPKJOnXqJMMw7M4pMjJS6enphXrEeyUvLy9Jf/x88znys7uaK/fPycnR2bNnVbVqVfn5+RX43dm3b58OHz581eOkpqZqw4YN6t69u/n377ffftPZs2cVGRmpw4cP65dffnHofHFr4fEcYIGPj48k+38IHJWXl6epU6dq5syZOnbsmN08nvxHJ5L00ksvad26dbr33ntVtWpVtWnTRj179lTTpk3NMRMmTFCfPn1UsWJFNWrUSO3bt1fv3r1VuXLlQtXm4eFh95hHknx9fVWhQgUzQFzZ/ue5SpJUrVo1u20vLy+VL1/ebt7Jvn379Morr2jDhg1mEM2Xnp5+3RpXrVql119/XUlJScrKyjLb/1yfJN11110F2sqWLWvW/euvvyojI0N16tS57nsePnxYu3fvLnBt8p05c+a6+1/Ln+srW7asJF31uv6d4+Yf+8rzTktL09y5c6/5Kc/CnlO+zMxMSfb/geHIz+5qfv/9d40bN07z58/XL7/8Yjdf6sq/N2PGjFHnzp1VvXp11alTR23bttWTTz6pevXqSZKOHDkiwzA0YsQIjRgx4qrvdebMGd15553WTxi3FEITYIGPj4+Cg4O1d+/eQh9j7NixGjFihJ5++mm99tpr8vf3l4uLi+Li4uwmFdeqVUuHDh3SqlWrtHr1an3yySeaOXOmRo4cqdGjR0uSunfvrubNm2vFihVau3at3nrrLY0fP17Lly9Xu3btHK7N1dXVofYr/9GyKi0tTQ888IB8fHw0ZswYValSRR4eHtq5c6deeuml606s3rhxox5++GG1aNFCM2fOVPny5VWqVCnNnz//qhPTi6ruvLw8PfTQQxo6dOhV+6tXr+7Q8fIV5XV15Lj517hXr17q06fPVcfmB4zCyv8dqVq1qiTHf3ZX88ILL2j+/PmKi4tTeHi4fH19ZbPZ1KNHD7u/Ny1atNDRo0f16aefau3atXrvvfc0efJkzZ49W88884w59t///rciIyOv+l75dQNXQ2gCLOrYsaPmzp2rxMREhYeHO7z/smXL1KpVK82bN8+uPS0tzZzUm8/T01OPP/64Hn/8cWVnZ+vRRx/VG2+8oeHDh8vDw0OSVL58eT3//PN6/vnndebMGTVs2FBvvPFGoUJTUTh8+LBatWplbmdmZur06dNq3769pD8+OXb27FktX75cLVq0MMcdO3bsL4/9ySefyMPDQ2vWrJG7u7vZPn/+/ELVescdd8jHx+cvQ3CVKlWUmZmpiIiIQr1PUbN6Z+Za7rjjDnl7eys3N7dYzikzM1MrVqxQxYoVzUnTjvzsrnV+y5YtU58+fTRx4kSz7dKlS1dd5NPf3199+/ZV3759lZmZqRYtWmjUqFF65plnzDuxpUqV+svz/7vXGv9MzGkCLBo6dKg8PT31zDPPKCUlpUD/0aNHNXXq1Gvu7+rqWuBOwtKlSwvMoTh79qzdtpubm8LCwmQYhnJycpSbm1vgUVZAQICCg4PtHn3caHPnzlVOTo65PWvWLF2+fNkMcfl3Qa68BtnZ2Zo5c+ZfHtvV1VU2m83ukeZPP/1U6NWjXVxc1KVLF33++edXXR09v8bu3bsrMTFRa9asKTAmLS1Nly9fLtT7F5anp6f53oXh6uqqrl276pNPPrlqYPz1118LXdvvv/+uJ598UqmpqXr55ZfN0OHIz87T0/Oq53a1353p06cXWKriz787Xl5eqlq1qvl7ERAQoJYtW2rOnDk6ffp0gfe58vz/7rXGPxN3mgCLqlSposWLF+vxxx9XrVq17FYE37x5s5YuXXrd75rr2LGjxowZo759++r+++/Xnj17tGjRogLzkNq0aaOgoCA1bdpUgYGBOnDggN555x116NBB3t7eSktLU4UKFfTYY4+pfv368vLy0rp167Rt2za7/xK/0bKzs9W6dWt1795dhw4d0syZM9WsWTM9/PDDkqT7779fZcuWVZ8+fTRgwADZbDZ9+OGHlh5JdejQQZMmTVLbtm3Vs2dPnTlzRjNmzFDVqlW1e/fuQtU7duxYrV27Vg888ID69++vWrVq6fTp01q6dKm+/fZb+fn5aciQIfrss8/UsWNHPfXUU2rUqJEuXLigPXv2aNmyZfrpp58K3CUsTo0aNZIkvfzyy+rRo4dKlSqlTp06mf/AW/Hmm2/qq6++UpMmTdSvXz+FhYUpNTVVO3fu1Lp165SamvqXx/jll1/0f//3f5L+uLu0f/9+LV26VMnJyXrxxRf17LPPmmMd+dk1atRI69at06RJkxQcHKzQ0FA1adJEHTt21IcffihfX1+FhYUpMTFR69ats5sLKElhYWFq2bKlGjVqJH9/f23fvt1cmiPfjBkz1KxZM9WtW1f9+vVT5cqVlZKSosTERP3888/mumkNGjSQq6urxo8fr/T0dLm7u+vBBx9UQECA5WuNf6CS+MgecDP74YcfjH79+hmVKlUy3NzcDG9vb6Np06bG9OnT7T7GfLUlB1588UWjfPnyRunSpY2mTZsaiYmJBT6WPmfOHKNFixZGuXLlDHd3d6NKlSrGkCFDjPT0dMMwDCMrK8sYMmSIUb9+fcPb29vw9PQ06tevb8ycOfMva7/WkgOenp4Fxj7wwANXXfrgzx85zz/mN998Y/Tv398oW7as4eXlZURFRRlnz56123fTpk3GfffdZ5QuXdoIDg42hg4daqxZs6bAR7uvtuTAvHnzjGrVqhnu7u5GzZo1jfnz5xuvvvqq8ef/G5N01eUY/vzzMAzDOH78uNG7d2/jjjvuMNzd3Y3KlSsbMTExdssBnD9/3hg+fLhRtWpVw83Nzbj99tuN+++/33j77bftlli4mmstObB06VK7cceOHTMkGfPnz7/u8Qzjj2UQ7rzzTsPFxcVu+QFHzjslJcWIiYkxKlasaJQqVcoICgoyWrdubcydO/cv3z8kJMSQZEgybDab4ePjY9SuXdvo16+fsWXLlqvuY/Vnd/DgQaNFixZG6dKlDUlm3efOnTP69u1r3H777YaXl5cRGRlpHDx4sMC5vf7668a9995r+Pn5GaVLlzZq1qxpvPHGGwV+TkePHjV69+5tBAUFGaVKlTLuvPNOo2PHjsayZcvsxr377rtG5cqVDVdXV5YfgGEYhmEzjL858xAAAOAWwJwmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGLWxaRvLw8nTp1St7e3iy/DwDATcIwDJ0/f17BwcFycbn+vSRCUxE5deqUKlasWNJlAACAQjh58qQqVKhw3TGEpiLi7e0t6Y+L7uPjU8LVAAAAKzIyMlSxYkXz3/HrITQVkfxHcj4+PoQmAABuMlam1jARHAAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFJRqaEhIS1KlTJwUHB8tms2nlypVmX05Ojl566SXVrVtXnp6eCg4OVu/evXXq1Cm7Y6SmpioqKko+Pj7y8/NTdHS0MjMz7cbs3r1bzZs3l4eHhypWrKgJEyYUqGXp0qWqWbOmPDw8VLduXX355ZfFcs4AAODmVKKh6cKFC6pfv75mzJhRoO/ixYvauXOnRowYoZ07d2r58uU6dOiQHn74YbtxUVFR2rdvn+Lj47Vq1SolJCSof//+Zn9GRobatGmjkJAQ7dixQ2+99ZZGjRqluXPnmmM2b96sJ554QtHR0dq1a5e6dOmiLl26aO/evcV38gAA4KZiMwzDKOkipD++KG/FihXq0qXLNcds27ZN9957r44fP6677rpLBw4cUFhYmLZt26bGjRtLklavXq327dvr559/VnBwsGbNmqWXX35ZycnJcnNzkyQNGzZMK1eu1MGDByVJjz/+uC5cuKBVq1aZ73XfffepQYMGmj17tqX6MzIy5Ovrq/T0dL6wFwCAm4Qj/37fVHOa0tPTZbPZ5OfnJ0lKTEyUn5+fGZgkKSIiQi4uLtqyZYs5pkWLFmZgkqTIyEgdOnRI586dM8dERETYvVdkZKQSExOL+YwAAMDN4raSLsCqS5cu6aWXXtITTzxhJsHk5GQFBATYjbvtttvk7++v5ORkc0xoaKjdmMDAQLOvbNmySk5ONtuuHJN/jKvJyspSVlaWuZ2RkVH4kwMAAE7vpghNOTk56t69uwzD0KxZs0q6HEnSuHHjNHr06Bv2fpWGfXHD3suZ/PRmh0Lve6teM+nvXTcUzq369+3v/l3juuFm4vSP5/ID0/HjxxUfH2/3vDEoKEhnzpyxG3/58mWlpqYqKCjIHJOSkmI3Jn/7r8bk91/N8OHDlZ6ebr5OnjxZ+JMEAABOz6lDU35gOnz4sNatW6dy5crZ9YeHhystLU07duww2zZs2KC8vDw1adLEHJOQkKCcnBxzTHx8vGrUqKGyZcuaY9avX2937Pj4eIWHh1+zNnd3d/n4+Ni9AADAP1eJhqbMzEwlJSUpKSlJknTs2DElJSXpxIkTysnJ0WOPPabt27dr0aJFys3NVXJyspKTk5WdnS1JqlWrltq2bat+/fpp69at2rRpk2JjY9WjRw8FBwdLknr27Ck3NzdFR0dr3759+uijjzR16lQNHjzYrGPgwIFavXq1Jk6cqIMHD2rUqFHavn27YmNjb/g1AQAAzqlEQ9P27dt199136+6775YkDR48WHfffbdGjhypX375RZ999pl+/vlnNWjQQOXLlzdfmzdvNo+xaNEi1axZU61bt1b79u3VrFkzuzWYfH19tXbtWh07dkyNGjXSiy++qJEjR9qt5XT//fdr8eLFmjt3rurXr69ly5Zp5cqVqlOnzo27GAAAwKmV6ETwli1b6nrLRFlZQsrf31+LFy++7ph69epp48aN1x3TrVs3devW7S/fDwAA3Jqcek4TAACAsyA0AQAAWHBTrNMEAAD+cKuubSWV/PpW3GkCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAFfowLAzq36FQ0l/fUMAJwfd5oAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWFCioSkhIUGdOnVScHCwbDabVq5caddvGIZGjhyp8uXLq3Tp0oqIiNDhw4ftxqSmpioqKko+Pj7y8/NTdHS0MjMz7cbs3r1bzZs3l4eHhypWrKgJEyYUqGXp0qWqWbOmPDw8VLduXX355ZdFfr4AAODmVaKh6cKFC6pfv75mzJhx1f4JEyZo2rRpmj17trZs2SJPT09FRkbq0qVL5pioqCjt27dP8fHxWrVqlRISEtS/f3+zPyMjQ23atFFISIh27Niht956S6NGjdLcuXPNMZs3b9YTTzyh6Oho7dq1S126dFGXLl20d+/e4jt5AABwU7mtJN+8Xbt2ateu3VX7DMPQlClT9Morr6hz586SpA8++ECBgYFauXKlevTooQMHDmj16tXatm2bGjduLEmaPn262rdvr7ffflvBwcFatGiRsrOz9f7778vNzU21a9dWUlKSJk2aZIarqVOnqm3bthoyZIgk6bXXXlN8fLzeeecdzZ49+wZcCQAA4Oycdk7TsWPHlJycrIiICLPN19dXTZo0UWJioiQpMTFRfn5+ZmCSpIiICLm4uGjLli3mmBYtWsjNzc0cExkZqUOHDuncuXPmmCvfJ39M/vtcTVZWljIyMuxeAADgn8tpQ1NycrIkKTAw0K49MDDQ7EtOTlZAQIBd/2233SZ/f3+7MVc7xpXvca0x+f1XM27cOPn6+pqvihUrOnqKAADgJuK0ocnZDR8+XOnp6ebr5MmTJV0SAAAoRk4bmoKCgiRJKSkpdu0pKSlmX1BQkM6cOWPXf/nyZaWmptqNudoxrnyPa43J778ad3d3+fj42L0AAMA/l9OGptDQUAUFBWn9+vVmW0ZGhrZs2aLw8HBJUnh4uNLS0rRjxw5zzIYNG5SXl6cmTZqYYxISEpSTk2OOiY+PV40aNVS2bFlzzJXvkz8m/30AAABKNDRlZmYqKSlJSUlJkv6Y/J2UlKQTJ07IZrMpLi5Or7/+uj777DPt2bNHvXv3VnBwsLp06SJJqlWrltq2bat+/fpp69at2rRpk2JjY9WjRw8FBwdLknr27Ck3NzdFR0dr3759+uijjzR16lQNHjzYrGPgwIFavXq1Jk6cqIMHD2rUqFHavn27YmNjb/QlAQAATqpElxzYvn27WrVqZW7nB5k+ffpowYIFGjp0qC5cuKD+/fsrLS1NzZo10+rVq+Xh4WHus2jRIsXGxqp169ZycXFR165dNW3aNLPf19dXa9euVUxMjBo1aqTbb79dI0eOtFvL6f7779fixYv1yiuv6D//+Y+qVaumlStXqk6dOjfgKgAAgJtBiYamli1byjCMa/bbbDaNGTNGY8aMueYYf39/LV68+LrvU69ePW3cuPG6Y7p166Zu3bpdv2AAAHDLcto5TQAAAM6E0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAodD08mTJ/Xzzz+b21u3blVcXJzmzp1bpIUBAAA4E4dDU8+ePfXVV19JkpKTk/XQQw9p69atevnllzVmzJgiLxAAAMAZOBya9u7dq3vvvVeS9PHHH6tOnTravHmzFi1apAULFhR1fQAAAE7B4dCUk5Mjd3d3SdK6dev08MMPS5Jq1qyp06dPF211AAAATsLh0FS7dm3Nnj1bGzduVHx8vNq2bStJOnXqlMqVK1fkBQIAADgDh0PT+PHjNWfOHLVs2VJPPPGE6tevL0n67LPPzMd2AAAA/zS3ObpDy5Yt9dtvvykjI0Nly5Y12/v3768yZcoUaXEAAADOolDrNBmGoR07dmjOnDk6f/68JMnNzY3QBAAA/rEcvtN0/PhxtW3bVidOnFBWVpYeeugheXt7a/z48crKytLs2bOLo04AAIAS5fCdpoEDB6px48Y6d+6cSpcubbY/8sgjWr9+fZEWBwAA4CwcvtO0ceNGbd68WW5ubnbtlSpV0i+//FJkhQEAADgTh+805eXlKTc3t0D7zz//LG9v7yIpCgAAwNk4HJratGmjKVOmmNs2m02ZmZl69dVX1b59+6KsDQAAwGk4HJomTpyoTZs2KSwsTJcuXVLPnj3NR3Pjx48v0uJyc3M1YsQIhYaGqnTp0qpSpYpee+01GYZhjjEMQyNHjlT58uVVunRpRURE6PDhw3bHSU1NVVRUlHx8fOTn56fo6GhlZmbajdm9e7eaN28uDw8PVaxYURMmTCjScwEAADc3h+c0VahQQd9//72WLFmi3bt3KzMzU9HR0YqKirKbGF4Uxo8fr1mzZmnhwoWqXbu2tm/frr59+8rX11cDBgyQJE2YMEHTpk3TwoULFRoaqhEjRigyMlL79++Xh4eHJCkqKkqnT59WfHy8cnJy1LdvX/Xv31+LFy+WJGVkZKhNmzaKiIjQ7NmztWfPHj399NPy8/NT//79i/ScAADAzcnh0CRJt912m3r16lXUtRSwefNmde7cWR06dJD0x2Tz//73v9q6daukP+4yTZkyRa+88oo6d+4sSfrggw8UGBiolStXqkePHjpw4IBWr16tbdu2qXHjxpKk6dOnq3379nr77bcVHBysRYsWKTs7W++//77c3NxUu3ZtJSUladKkSYQmAAAgyWJo+uyzzywfMP8LfIvC/fffr7lz5+qHH35Q9erV9f333+vbb7/VpEmTJEnHjh1TcnKyIiIizH18fX3VpEkTJSYmqkePHkpMTJSfn58ZmCQpIiJCLi4u2rJlix555BElJiaqRYsWdp8IjIyM1Pjx43Xu3Dm7lc/zZWVlKSsry9zOyMgosvMGAADOx1Jo6tKli6WD2Wy2q36yrrCGDRumjIwM1axZU66ursrNzdUbb7yhqKgoSVJycrIkKTAw0G6/wMBAsy85OVkBAQF2/bfddpv8/f3txoSGhhY4Rn7f1ULTuHHjNHr06CI4SwAAcDOwNBE8Ly/P0qsoA5Mkffzxx1q0aJEWL16snTt3auHChXr77be1cOHCIn2fwhg+fLjS09PN18mTJ0u6JAAAUIwKNafpRhkyZIiGDRumHj16SJLq1q2r48ePa9y4cerTp4+CgoIkSSkpKSpfvry5X0pKiho0aCBJCgoK0pkzZ+yOe/nyZaWmppr7BwUFKSUlxW5M/nb+mD9zd3eXu7v73z9JAABwUyjUF/auX79eHTt2VJUqVVSlShV17NhR69atK+radPHiRbm42Jfo6uqqvLw8SVJoaKiCgoLsvr4lIyNDW7ZsUXh4uCQpPDxcaWlp2rFjhzlmw4YNysvLU5MmTcwxCQkJysnJMcfEx8erRo0aV300BwAAbj0Oh6aZM2eqbdu28vb21sCBAzVw4ED5+Pioffv2mjFjRpEW16lTJ73xxhv64osv9NNPP2nFihWaNGmSHnnkEUl/zKGKi4vT66+/rs8++0x79uxR7969FRwcbM7DqlWrltq2bat+/fpp69at2rRpk2JjY9WjRw8FBwdLknr27Ck3NzdFR0dr3759+uijjzR16lQNHjy4SM8HAADcvBx+PDd27FhNnjxZsbGxZtuAAQPUtGlTjR07VjExMUVW3PTp0zVixAg9//zzOnPmjIKDg/Xss89q5MiR5pihQ4fqwoUL6t+/v9LS0tSsWTOtXr3aXKNJkhYtWqTY2Fi1bt1aLi4u6tq1q6ZNm2b2+/r6au3atYqJiVGjRo10++23a+TIkSw3AAAATA6HprS0NLVt27ZAe5s2bfTSSy8VSVH5vL29NWXKFLuvbfkzm82mMWPGaMyYMdcc4+/vby5keS316tXTxo0bC1sqAAD4h3P48dzDDz+sFStWFGj/9NNP1bFjxyIpCgAAwNk4fKcpLCxMb7zxhr7++mtzsvV3332nTZs26cUXX7R77JX/VScAAAA3O4dD07x581S2bFnt379f+/fvN9v9/Pw0b948c9tmsxGaAADAP4bDoenYsWPFUQcAAIBTK9Q6TQAAALcah+80GYahZcuW6auvvtKZM2fMhSbzLV++vMiKAwAAcBYOh6a4uDjNmTNHrVq1UmBgoGw2W3HUBQAA4FQcDk0ffvihli9frvbt2xdHPQAAAE7J4TlNvr6+qly5cnHUAgAA4LQcDk2jRo3S6NGj9fvvvxdHPQAAAE7J4cdz3bt313//+18FBASoUqVKKlWqlF3/zp07i6w4AAAAZ+FwaOrTp4927NihXr16MREcAADcMhwOTV988YXWrFmjZs2aFUc9AAAATsnhOU0VK1aUj49PcdQCAADgtBwOTRMnTtTQoUP1008/FUM5AAAAzsnhx3O9evXSxYsXVaVKFZUpU6bARPDU1NQiKw4AAMBZOByapkyZUgxlAAAAOLdCfXoOAADgVuNwaLrSpUuXlJ2dbdfGJHEAAPBP5PBE8AsXLig2NlYBAQHy9PRU2bJl7V4AAAD/RA6HpqFDh2rDhg2aNWuW3N3d9d5772n06NEKDg7WBx98UBw1AgAAlDiHH899/vnn+uCDD9SyZUv17dtXzZs3V9WqVRUSEqJFixYpKiqqOOoEAAAoUQ7faUpNTVXlypUl/TF/KX+JgWbNmikhIaFoqwMAAHASDoemypUr69ixY5KkmjVr6uOPP5b0xx0oPz+/Ii0OAADAWTgcmvr27avvv/9ekjRs2DDNmDFDHh4eGjRokIYMGVLkBQIAADgDh+c0DRo0yPxzRESEDhw4oJ07d6pq1aqqV69ekRYHAADgLP7WOk2SVKlSJVWqVKkISgEAAHBelh/PJSYmatWqVXZtH3zwgUJDQxUQEKD+/fsrKyuryAsEAABwBpZD05gxY7Rv3z5ze8+ePYqOjlZERISGDRumzz//XOPGjSuWIgEAAEqa5dCUlJSk1q1bm9tLlixRkyZN9O6772rw4MGaNm2a+Uk6AACAfxrLoencuXMKDAw0t7/55hu1a9fO3L7nnnt08uTJoq0OAADASVgOTYGBgeb6TNnZ2dq5c6fuu+8+s//8+fMqVapU0VcIAADgBCyHpvbt22vYsGHauHGjhg8frjJlyqh58+Zm/+7du1WlSpViKRIAAKCkWV5y4LXXXtOjjz6qBx54QF5eXlq4cKHc3NzM/vfff19t2rQpliIBAABKmuXQdPvttyshIUHp6eny8vKSq6urXf/SpUvl5eVV5AUCAAA4A4cXt/T19b1qu7+//98uBgAAwFk5/N1zAAAAtyJCEwAAgAWEJgAAAAsshaaGDRvq3Llzkv74OpWLFy8Wa1EAAADOxlJoOnDggC5cuCBJGj16tDIzM4u1KAAAAGdj6dNzDRo0UN++fdWsWTMZhqG33377mssLjBw5skgLBAAAcAaWQtOCBQv06quvatWqVbLZbPrf//6n224ruKvNZiM0AQCAfyRLoalGjRpasmSJJMnFxUXr169XQEBAsRYGAADgTBxe3DIvL6846gAAAHBqDocmSTp69KimTJmiAwcOSJLCwsI0cOBAvrAXAAD8Yzm8TtOaNWsUFhamrVu3ql69eqpXr562bNmi2rVrKz4+vjhqBAAAKHEOh6Zhw4Zp0KBB2rJliyZNmqRJkyZpy5YtiouL00svvVTkBf7yyy/q1auXypUrp9KlS6tu3bravn272W8YhkaOHKny5curdOnSioiI0OHDh+2OkZqaqqioKPn4+MjPz0/R0dEFlk3YvXu3mjdvLg8PD1WsWFETJkwo8nMBAAA3L4dD04EDBxQdHV2g/emnn9b+/fuLpKh8586dU9OmTVWqVCn973//0/79+zVx4kSVLVvWHDNhwgRNmzZNs2fP1pYtW+Tp6anIyEhdunTJHBMVFaV9+/YpPj5eq1atUkJCgvr372/2Z2RkqE2bNgoJCdGOHTv01ltvadSoUZo7d26Rng8AALh5OTyn6Y477lBSUpKqVatm156UlFTkn6gbP368KlasqPnz55ttoaGh5p8Nw9CUKVP0yiuvqHPnzpKkDz74QIGBgVq5cqV69OihAwcOaPXq1dq2bZsaN24sSZo+fbrat2+vt99+W8HBwVq0aJGys7P1/vvvy83NTbVr11ZSUpImTZpkF64AAMCty+E7Tf369VP//v01fvx4bdy4URs3btSbb76pZ599Vv369SvS4j777DM1btxY3bp1U0BAgO6++269++67Zv+xY8eUnJysiIgIs83X11dNmjRRYmKiJCkxMVF+fn5mYJKkiIgIubi4aMuWLeaYFi1ayM3NzRwTGRmpQ4cOmV8fAwAAbm0O32kaMWKEvL29NXHiRA0fPlySFBwcrFGjRmnAgAFFWtyPP/6oWbNmafDgwfrPf/6jbdu2acCAAXJzc1OfPn2UnJwsSQoMDLTbLzAw0OxLTk4ucAfstttuk7+/v92YK+9gXXnM5ORku8eB+bKyspSVlWVuZ2Rk/M2zBQAAzszh0GSz2TRo0CANGjRI58+flyR5e3sXeWHSH2tCNW7cWGPHjpUk3X333dq7d69mz56tPn36FMt7WjVu3DiNHj26RGsAAAA3jsOP567k7e1dbIFJksqXL6+wsDC7tlq1aunEiROSpKCgIElSSkqK3ZiUlBSzLygoSGfOnLHrv3z5slJTU+3GXO0YV77Hnw0fPlzp6enm6+TJk4U5RQAAcJP4W6GpuDVt2lSHDh2ya/vhhx8UEhIi6Y9J4UFBQVq/fr3Zn5GRoS1btig8PFySFB4errS0NO3YscMcs2HDBuXl5alJkybmmISEBOXk5Jhj4uPjVaNGjas+mpMkd3d3+fj42L0AAMA/l1OHpkGDBum7777T2LFjdeTIES1evFhz585VTEyMpD8eFcbFxen111/XZ599pj179qh3794KDg5Wly5dJP1xZ6pt27bq16+ftm7dqk2bNik2NlY9evRQcHCwJKlnz55yc3NTdHS09u3bp48++khTp07V4MGDS+rUAQCAkynU16jcKPfcc49WrFih4cOHa8yYMQoNDdWUKVMUFRVljhk6dKguXLig/v37Ky0tTc2aNdPq1avl4eFhjlm0aJFiY2PVunVrubi4qGvXrpo2bZrZ7+vrq7Vr1yomJkaNGjXS7bffrpEjR7LcAAAAMDkUmnJyctS2bVvNnj27wDpNxaVjx47q2LHjNfttNpvGjBmjMWPGXHOMv7+/Fi9efN33qVevnjZu3FjoOgEAwD+bQ4/nSpUqpd27dxdXLQAAAE7L4TlNvXr10rx584qjFgAAAKfl8Jymy5cv6/3339e6devUqFEjeXp62vVPmjSpyIoDAABwFg6Hpr1796phw4aS/vj4/5VsNlvRVAUAAOBkHA5NX331VXHUAQAA4NQKvU7TkSNHtGbNGv3++++SJMMwiqwoAAAAZ+NwaDp79qxat26t6tWrq3379jp9+rQkKTo6Wi+++GKRFwgAAOAMHA5NgwYNUqlSpXTixAmVKVPGbH/88ce1evXqIi0OAADAWTg8p2nt2rVas2aNKlSoYNderVo1HT9+vMgKAwAAcCYO32m6cOGC3R2mfKmpqXJ3dy+SogAAAJyNw6GpefPm+uCDD8xtm82mvLw8TZgwQa1atSrS4gAAAJyFw4/nJkyYoNatW2v79u3Kzs7W0KFDtW/fPqWmpmrTpk3FUSMAAECJc/hOU506dfTDDz+oWbNm6ty5sy5cuKBHH31Uu3btUpUqVYqjRgAAgBLn8J0mSfL19dXLL79c1LUAAAA4rUKFpnPnzmnevHk6cOCAJCksLEx9+/aVv79/kRYHAADgLBx+PJeQkKBKlSpp2rRpOnfunM6dO6dp06YpNDRUCQkJxVEjAABAiXP4TlNMTIwef/xxzZo1S66urpKk3NxcPf/884qJidGePXuKvEgAAICS5vCdpiNHjujFF180A5Mkubq6avDgwTpy5EiRFgcAAOAsHA5NDRs2NOcyXenAgQOqX79+kRQFAADgbCw9ntu9e7f55wEDBmjgwIE6cuSI7rvvPknSd999pxkzZujNN98snioBAABKmKXQ1KBBA9lsNhmGYbYNHTq0wLiePXvq8ccfL7rqAAAAnISl0HTs2LHirgMAAMCpWQpNISEhxV0HAACAUyvU4panTp3St99+qzNnzigvL8+ub8CAAUVSGAAAgDNxODQtWLBAzz77rNzc3FSuXDnZbDazz2azEZoAAMA/ksOhacSIERo5cqSGDx8uFxeHVywAAAC4KTmcei5evKgePXoQmAAAwC3F4eQTHR2tpUuXFkctAAAATsvhx3Pjxo1Tx44dtXr1atWtW1elSpWy6580aVKRFQcAAOAsChWa1qxZoxo1akhSgYngAAAA/0QOh6aJEyfq/fff11NPPVUM5QAAADgnh+c0ubu7q2nTpsVRCwAAgNNyODQNHDhQ06dPL45aAAAAnJbDj+e2bt2qDRs2aNWqVapdu3aBieDLly8vsuIAAACchcOhyc/PT48++mhx1AIAAOC0HA5N8+fPL446AAAAnBrLegMAAFjg8J2m0NDQ667H9OOPP/6tggAAAJyRw6EpLi7ObjsnJ0e7du3S6tWrNWTIkKKqCwAAwKk4HJoGDhx41fYZM2Zo+/btf7sgAAAAZ1Rkc5ratWunTz75pKgOBwAA4FSKLDQtW7ZM/v7+RXU4AAAAp+Lw47m7777bbiK4YRhKTk7Wr7/+qpkzZxZpcQAAAM7C4dDUpUsXu20XFxfdcccdatmypWrWrFlUdQEAADgVh0PTq6++Whx1AAAAODUWtwQAALDA8p0mFxeX6y5qKUk2m02XL1/+20UBAAA4G8uhacWKFdfsS0xM1LRp05SXl1ckRQEAADgby4/nOnfuXOBVs2ZNLViwQG+//ba6deumQ4cOFWetevPNN2Wz2exWJb906ZJiYmJUrlw5eXl5qWvXrkpJSbHb78SJE+rQoYPKlCmjgIAADRkypMAdsa+//loNGzaUu7u7qlatqgULFhTruQAAgJtLoeY0nTp1Sv369VPdunV1+fJlJSUlaeHChQoJCSnq+kzbtm3TnDlzVK9ePbv2QYMG6fPPP9fSpUv1zTff6NSpU3r00UfN/tzcXHXo0EHZ2dnavHmzFi5cqAULFmjkyJHmmGPHjqlDhw5q1aqVkpKSFBcXp2eeeUZr1qwptvMBAAA3F4dCU3p6ul566SVVrVpV+/bt0/r16/X555+rTp06xVWfJCkzM1NRUVF69913VbZsWbt65s2bp0mTJunBBx9Uo0aNNH/+fG3evFnfffedJGnt2rXav3+//u///k8NGjRQu3bt9Nprr2nGjBnKzs6WJM2ePVuhoaGaOHGiatWqpdjYWD322GOaPHlysZ4XAAC4eVgOTRMmTFDlypW1atUq/fe//9XmzZvVvHnz4qzNFBMTow4dOigiIsKufceOHcrJybFrr1mzpu666y4lJiZK+mO+Vd26dRUYGGiOiYyMVEZGhvbt22eO+fOxIyMjzWNcTVZWljIyMuxeAADgn8vyRPBhw4apdOnSqlq1qhYuXKiFCxdeddzy5cuLrDhJWrJkiXbu3Klt27YV6EtOTpabm5v8/Pzs2gMDA5WcnGyOuTIw5ffn911vTEZGhn7//XeVLl26wHuPGzdOo0ePLvR5AQCAm4vl0NS7d++/XHKgqJ08eVIDBw5UfHy8PDw8buh7/5Xhw4dr8ODB5nZGRoYqVqxYghUBAIDiZDk0lcSnyXbs2KEzZ86oYcOGZltubq4SEhL0zjvvaM2aNcrOzlZaWprd3aaUlBQFBQVJkoKCgrR161a74+Z/uu7KMX/+xF1KSop8fHyuepdJktzd3eXu7v63zxEAANwcnHpF8NatW2vPnj1KSkoyX40bN1ZUVJT551KlSmn9+vXmPocOHdKJEycUHh4uSQoPD9eePXt05swZc0x8fLx8fHwUFhZmjrnyGPlj8o8BAADg8HfP3Uje3t4FPpnn6empcuXKme3R0dEaPHiw/P395ePjoxdeeEHh4eG67777JElt2rRRWFiYnnzySU2YMEHJycl65ZVXFBMTY94p+te//qV33nlHQ4cO1dNPP60NGzbo448/1hdffHFjTxgAADgtpw5NVkyePFkuLi7q2rWrsrKyFBkZqZkzZ5r9rq6uWrVqlZ577jmFh4fL09NTffr00ZgxY8wxoaGh+uKLLzRo0CBNnTpVFSpU0HvvvafIyMiSOCUAAOCEbrrQ9PXXX9tte3h4aMaMGZoxY8Y19wkJCdGXX3553eO2bNlSu3btKooSAQDAP5BTz2kCAABwFoQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAqcOTePGjdM999wjb29vBQQEqEuXLjp06JDdmEuXLikmJkblypWTl5eXunbtqpSUFLsxJ06cUIcOHVSmTBkFBARoyJAhunz5st2Yr7/+Wg0bNpS7u7uqVq2qBQsWFPfpAQCAm4hTh6ZvvvlGMTEx+u677xQfH6+cnBy1adNGFy5cMMcMGjRIn3/+uZYuXapvvvlGp06d0qOPPmr25+bmqkOHDsrOztbmzZu1cOFCLViwQCNHjjTHHDt2TB06dFCrVq2UlJSkuLg4PfPMM1qzZs0NPV8AAOC8bivpAq5n9erVdtsLFixQQECAduzYoRYtWig9PV3z5s3T4sWL9eCDD0qS5s+fr1q1aum7777Tfffdp7Vr12r//v1at26dAgMD1aBBA7322mt66aWXNGrUKLm5uWn27NkKDQ3VxIkTJUm1atXSt99+q8mTJysyMvKGnzcAAHA+Tn2n6c/S09MlSf7+/pKkHTt2KCcnRxEREeaYmjVr6q677lJiYqIkKTExUXXr1lVgYKA5JjIyUhkZGdq3b5855spj5I/JP8bVZGVlKSMjw+4FAAD+uW6a0JSXl6e4uDg1bdpUderUkSQlJyfLzc1Nfn5+dmMDAwOVnJxsjrkyMOX35/ddb0xGRoZ+//33q9Yzbtw4+fr6mq+KFSv+7XMEAADO66YJTTExMdq7d6+WLFlS0qVIkoYPH6709HTzdfLkyZIuCQAAFCOnntOULzY2VqtWrVJCQoIqVKhgtgcFBSk7O1tpaWl2d5tSUlIUFBRkjtm6davd8fI/XXflmD9/4i4lJUU+Pj4qXbr0VWtyd3eXu7v73z43AABwc3DqO02GYSg2NlYrVqzQhg0bFBoaatffqFEjlSpVSuvXrzfbDh06pBMnTig8PFySFB4erj179ujMmTPmmPj4ePn4+CgsLMwcc+Ux8sfkHwMAAMCp7zTFxMRo8eLF+vTTT+Xt7W3OQfL19VXp0qXl6+ur6OhoDR48WP7+/vLx8dELL7yg8PBw3XfffZKkNm3aKCwsTE8++aQmTJig5ORkvfLKK4qJiTHvFP3rX//SO++8o6FDh+rpp5/Whg0b9PHHH+uLL74osXMHAADOxanvNM2aNUvp6elq2bKlypcvb74++ugjc8zkyZPVsWNHde3aVS1atFBQUJCWL19u9ru6umrVqlVydXVVeHi4evXqpd69e2vMmDHmmNDQUH3xxReKj49X/fr1NXHiRL333nssNwAAAExOfafJMIy/HOPh4aEZM2ZoxowZ1xwTEhKiL7/88rrHadmypXbt2uVwjQAA4Nbg1HeaAAAAnAWhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABYQmgAAACwgNAEAAFhAaAIAALCA0AQAAGABoQkAAMACQhMAAIAFhCYAAAALCE0AAAAWEJoAAAAsIDQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBo+pMZM2aoUqVK8vDwUJMmTbR169aSLgkAADgBQtMVPvroIw0ePFivvvqqdu7cqfr16ysyMlJnzpwp6dIAAEAJIzRdYdKkSerXr5/69u2rsLAwzZ49W2XKlNH7779f0qUBAIASRmj6f7Kzs7Vjxw5FRESYbS4uLoqIiFBiYmIJVgYAAJzBbSVdgLP47bfflJubq8DAQLv2wMBAHTx4sMD4rKwsZWVlmdvp6emSpIyMjGKpLy/rYrEc19n9net5q14zietWGH/3d5frVjhcN8fdqtdMKp5/Y/OPaRjGX44lNBXSuHHjNHr06ALtFStWLIFq/rl8p5R0BTcnrpvjuGaFw3UrHK5b4RTndTt//rx8fX2vO4bQ9P/cfvvtcnV1VUpKil17SkqKgoKCCowfPny4Bg8ebG7n5eUpNTVV5cqVk81mK/Z6b5SMjAxVrFhRJ0+elI+PT0mXc9PgujmOa1Y4XLfC4boVzj/xuhmGofPnzys4OPgvxxKa/h83Nzc1atRI69evV5cuXST9EYTWr1+v2NjYAuPd3d3l7u5u1+bn53cDKi0ZPj4+/5hfkBuJ6+Y4rlnhcN0Kh+tWOP+06/ZXd5jyEZquMHjwYPXp00eNGzfWvffeqylTpujChQvq27dvSZcGAABKGKHpCo8//rh+/fVXjRw5UsnJyWrQoIFWr15dYHI4AAC49RCa/iQ2Nvaqj+NuVe7u7nr11VcLPIrE9XHdHMc1KxyuW+Fw3QrnVr9uNsPKZ+wAAABucSxuCQAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITbimGTNmqFKlSvLw8FCTJk20devWki7J6SUkJKhTp04KDg6WzWbTypUrS7okpzdu3Djdc8898vb2VkBAgLp06aJDhw6VdFlOb9asWapXr565yGB4eLj+97//lXRZN5U333xTNptNcXFxJV2KUxs1apRsNpvdq2bNmiVdVokgNOGqPvroIw0ePFivvvqqdu7cqfr16ysyMlJnzpwp6dKc2oULF1S/fn3NmDGjpEu5aXzzzTeKiYnRd999p/j4eOXk5KhNmza6cOFCSZfm1CpUqKA333xTO3bs0Pbt2/Xggw+qc+fO2rdvX0mXdlPYtm2b5syZo3r16pV0KTeF2rVr6/Tp0+br22+/LemSSgRLDuCqmjRponvuuUfvvPOOpD++UqZixYp64YUXNGzYsBKu7uZgs9m0YsUK82t5YM2vv/6qgIAAffPNN2rRokVJl3NT8ff311tvvaXo6OiSLsWpZWZmqmHDhpo5c6Zef/11NWjQQFOmTCnpspzWqFGjtHLlSiUlJZV0KSWOO00oIDs7Wzt27FBERITZ5uLiooiICCUmJpZgZbgVpKenS/ojAMCa3NxcLVmyRBcuXFB4eHhJl+P0YmJi1KFDB7v/j8P1HT58WMHBwapcubKioqJ04sSJki6pRLAiOAr47bfflJubW+DrYwIDA3Xw4MESqgq3gry8PMXFxalp06aqU6dOSZfj9Pbs2aPw8HBdunRJXl5eWrFihcLCwkq6LKe2ZMkS7dy5U9u2bSvpUm4aTZo00YIFC1SjRg2dPn1ao0ePVvPmzbV37155e3uXdHk3FKEJgNOIiYnR3r17b9n5Eo6qUaOGkpKSlJ6ermXLlqlPnz765ptvCE7XcPLkSQ0cOFDx8fHy8PAo6XJuGu3atTP/XK9ePTVp0kQhISH6+OOPb7lHwYQmFHD77bfL1dVVKSkpdu0pKSkKCgoqoarwTxcbG6tVq1YpISFBFSpUKOlybgpubm6qWrWqJKlRo0batm2bpk6dqjlz5pRwZc5px44dOnPmjBo2bGi25ebmKiEhQe+8846ysrLk6upaghXeHPz8/FS9enUdOXKkpEu54ZjThALc3NzUqFEjrV+/3mzLy8vT+vXrmS+BImcYhmJjY7VixQpt2LBBoaGhJV3STSsvL09ZWVklXYbTat26tfbs2aOkpCTz1bhxY0VFRSkpKYnAZFFmZqaOHj2q8uXLl3QpNxx3mnBVgwcPVp8+fdS4cWPde++9mjJlii5cuKC+ffuWdGlOLTMz0+6/vo4dO6akpCT5+/vrrrvuKsHKnFdMTIwWL16sTz/9VN7e3kpOTpYk+fr6qnTp0iVcnfMaPny42rVrp7vuukvnz5/X4sWL9fXXX2vNmjUlXZrT8vb2LjBXztPTU+XKlWMO3XX8+9//VqdOnRQSEqJTp07p1Vdflaurq5544omSLu2GIzThqh5//HH9+uuvGjlypJKTk9WgQQOtXr26wORw2Nu+fbtatWplbg8ePFiS1KdPHy1YsKCEqnJus2bNkiS1bNnSrn3+/Pl66qmnbnxBN4kzZ86od+/eOn36tHx9fVWvXj2tWbNGDz30UEmXhn+Yn3/+WU888YTOnj2rO+64Q82aNdN3332nO+64o6RLu+FYpwkAAMAC5jQBAABYQGgCAACwgNAEAABgAaEJAADAAkITAACABYQmAAAACwhNAAAAFhCaANyUbDabVq5cWdJlFNpPP/0km82mpKSkki4FgEWEJgBOJzk5WS+88IIqV64sd3d3VaxYUZ06dbL7PsSS1LJlS8XFxZV0GQBuML5GBYBT+emnn9S0aVP5+fnprbfeUt26dZWTk6M1a9YoJiZGBw8eLOkSAdyiuNMEwKk8//zzstls2rp1q7p27arq1aurdu3aGjx4sL777rtr7vfSSy+pevXqKlOmjCpXrqwRI0YoJyfH7P/+++/VqlUreXt7y8fHR40aNdL27dslScePH1enTp1UtmxZeXp6qnbt2vryyy8t11ypUiWNHTtWTz/9tLy9vXXXXXdp7ty5dmO2bt2qu+++Wx4eHmrcuLF27dpV4Dh79+5Vu3bt5OXlpcDAQD355JP67bffJElff/213NzctHHjRnP8hAkTFBAQoJSUFMu1Aig8QhMAp5GamqrVq1crJiZGnp6eBfr9/Pyuua+3t7cWLFig/fv3a+rUqXr33Xc1efJksz8qKkoVKlTQtm3btGPHDg0bNkylSpWSJMXExCgrK0sJCQnas2ePxo8fLy8vL4dqnzhxohmGnn/+eT333HM6dOiQJCkzM1MdO3ZUWFiYduzYoVGjRunf//633f5paWl68MEHdffdd2v79u1avXq1UlJS1L17d0n//yPBJ598Uunp6dq1a5dGjBih9957jy/SBm4UAwCcxJYtWwxJxvLly/9yrCRjxYoV1+x/6623jEaNGpnb3t7exoIFC646tm7dusaoUaMs1/nAAw8YAwcONLdDQkKMXr16mdt5eXlGQECAMWvWLMMwDGPOnDlGuXLljN9//90cM2vWLEOSsWvXLsMwDOO1114z2rRpY/c+J0+eNCQZhw4dMgzDMLKysowGDRoY3bt3N8LCwox+/fpZrhnA38ecJgBOwzCMQu/70Ucfadq0aTp69KgyMzN1+fJl+fj4mP2DBw/WM888ow8//FARERHq1q2bqlSpIkkaMGCAnnvuOa1du1YRERHq2rWr6tWr59D7XzneZrMpKChIZ86ckSQdOHBA9erVk4eHhzkmPDzcbv/vv/9eX3311VXvcB09elTVq1eXm5ubFi1apHr16ikkJMTuThqA4sfjOQBOo1q1arLZbA5P9k5MTFRUVJTat2+vVatWadeuXXr55ZeVnZ1tjhk1apT27dunDh06aMOGDQoLC9OKFSskSc8884x+/PFHPfnkk9qzZ48aN26s6dOnO1RD/qO+fDabTXl5eZb3z8zMVKdOnZSUlGT3Onz4sFq0aGGO27x5s6Q/HmWmpqY6VCOAv4fQBMBp+Pv7KzIyUjNmzNCFCxcK9KelpV11v82bNyskJEQvv/yyGjdurGrVqun48eMFxlWvXl2DBg3S2rVr9eijj2r+/PlmX8WKFfWvf/1Ly5cv14svvqh33323yM6rVq1a2r17ty5dumS2/XlSe8OGDbVv3z5VqlRJVatWtXvlz+86evSoBg0apHfffVdNmjRRnz59HApmAP4eQhMApzJjxgzl5ubq3nvv1SeffKLDhw/rwIEDmjZtWoFHWvmqVaumEydOaMmSJTp69KimTZtm3kWSpN9//12xsbH6+uuvdfz4cW3atEnbtm1TrVq1JElxcXFas2aNjh07pp07d+qrr74y+4pCz549ZbPZ1K9fP+3fv19ffvml3n77bbsxMTExSk1N1RNPPKFt27bp6NGjWrNmjfr27avc3Fzl5uaqV69eioyMVN++fTV//nzt3r1bEydOLLI6AVwfoQmAU6lcubJ27typVq1a6cUXX1SdOnX00EMPaf369Zo1a9ZV93n44Yc1aNAgxcbGqkGDBtq8ebNGjBhh9ru6uurs2bPq3bu3qlevru7du6tdu3YaPXq0JCk3N1cxMTGqVauW2rZtq+rVq2vmzJlFdk5eXl76/PPPtWfPHt199916+eWXNX78eLsxwcHB2rRpk3Jzc9WmTRvVrVtXcXFx8vPzk4uLi9544w0dP35cc+bMkSSVL19ec+fO1SuvvKLvv/++yGoFcG024+/MvAQAALhFcKcJAADAAkITAACABYQmAAAACwhNAAAAFhCaAAAALCA0AQAAWEBoAgAAsIDQBAAAYAGhCQAAwAJCEwAAgAWEJgAAAAsITQAAABb8f1w6sPOgQasIAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABSMAAAHWCAYAAACMppwFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABo0UlEQVR4nO3dd3gU1f7H8c+SkEBIoyUhAglFehUUIyAISCiCCFcFUcpFUG6QakmuSlNpVxRRBCsgoghcRQQB6QjSFVBAEKUJhICUUBNIzu8Pf9nLkgSyYVs279fz7KM7c3bmO5PN7ObDmXMsxhgjAAAAAAAAAHCyAu4uAAAAAAAAAED+QBgJAAAAAAAAwCUIIwEAAAAAAAC4BGEkAAAAAAAAAJcgjAQAAAAAAADgEoSRAAAAAAAAAFyCMBIAAAAAAACASxBGAgAAAAAAAHAJwkgAAAAAAAAALkEYCQAAJEnDhw+XxWJxyb6aNm2qpk2bWp+vWrVKFotFc+fOdcn+e/TooejoaJfsK7fOnz+vJ598UhEREbJYLBo4cKC7S0IecuDAAVksFr3++utO39e//vUv3X///U7fT163ePFiBQYG6sSJE+4uBQAAtyKMBADAC02bNk0Wi8X6KFSokCIjIxUbG6uJEyfq3LlzDtnP0aNHNXz4cG3bts0h23MkT64tJ0aNGqVp06apb9++mjFjhp544ols26ampuqtt95S3bp1FRwcrNDQUFWvXl19+vTRr7/+6sKqvU/Tpk1Vo0YNd5eRrW+//VbDhw932/7379+vDz/8UP/+978l/X2+rr32ZPdwRc1XrlzRiBEjVL58efn7+6t8+fJ69dVXdfXq1UxtU1JS9MILLygyMlKFCxdWgwYNtHTp0kzt3nvvPZUrV07FihXTE088oeTkZJv16enpqlu3rkaNGpXpta1atVLFihU1evRoxx0kAAB5kK+7CwAAAM4zcuRIlStXTleuXFFiYqJWrVqlgQMH6o033tD8+fNVq1Yta9uXXnpJ8fHxdm3/6NGjGjFihKKjo1WnTp0cv+67776zaz+5caPaPvjgA6Wnpzu9hluxYsUK3X333Ro2bNhN23bq1EmLFi1Sly5d1Lt3b125ckW//vqrFixYoHvuuUdVqlRxQcVwh2+//VaTJk1yWyD51ltvqVy5crrvvvskSS+++KKefPJJ6/rNmzdr4sSJ+ve//62qVatal1977XGWxx9/XHPmzNE///lP1a9fXxs2bNDLL7+sQ4cO6f3337dp26NHD82dO1cDBw7U7bffrmnTpqlNmzZauXKlGjVqJElau3at+vbtq/79+6t8+fIaPXq0nnvuOb333nvW7XzwwQc6e/ashgwZkmVNTz31lJ599lmNGDFCQUFBzjt4AAA8GGEkAABerHXr1qpfv771eUJCglasWKEHHnhA7du31+7du1W4cGFJkq+vr3x9nfvV4OLFiwoICJCfn59T93MzBQsWdOv+cyIpKUnVqlW7abvNmzdrwYIFeu2116y90zK88847OnPmjJMqRH535coVzZw5U08//bR12fW3axcqVEgTJ07U/fffbzM0g7Nt3rxZs2fP1ssvv6yRI0dKkp5++mmVKFFCb7zxhvr162cNRDdt2qRZs2bpP//5j5599llJUrdu3VSjRg09//zz+uGHHyRJCxYsUNOmTTVhwgRJUnBwsBISEqxh5JkzZ/TSSy/pvffek7+/f5Z1derUSc8884w1JAUAID/iNm0AAPKZZs2a6eWXX9bBgwf16aefWpdnNWbk0qVL1ahRI4WGhiowMFCVK1e2Bl6rVq3SnXfeKUnq2bOn9fbLadOmSfrf7a1bt27Vvffeq4CAAJtbObMKJtLS0vTvf/9bERERKlKkiNq3b6/Dhw/btImOjlaPHj0yvfbabd6stqzGjLxw4YKGDBmiMmXKyN/fX5UrV9brr78uY4xNO4vFon79+mnevHmqUaOG/P39Vb16dS1evDjrE36dpKQk9erVS+Hh4SpUqJBq166t6dOnW9dnjJ+5f/9+LVy40Fr7gQMHstze77//Lklq2LBhpnU+Pj4qXry4zbIjR47on//8p8LDw621f/zxx5le++eff6pDhw4qUqSIwsLCNGjQIC1ZskQWi0WrVq2ytsvJzyNDSkqKhg0bpooVK8rf319lypTR888/r5SUFJt29pzjI0eOqFevXoqMjJS/v7/KlSunvn37KjU11drmzJkzGjhwoPVnW7FiRY0dO9ahvWMXLVqkxo0bq0iRIgoKClLbtm21c+dOmzY9evRQYGCgjhw5og4dOigwMFAlS5bUs88+q7S0NJu2f/31l5544gnrbffdu3fX9u3bM72PJ02aZD1nGY/rvf/++6pQoYL8/f115513avPmzTbrExMT1bNnT5UuXVr+/v4qVaqUHnzwwWzfcxnWrl2rkydPqkWLFnaeLendd99V9erV5e/vr8jISMXFxWUKzq+9htxzzz0qXLiwypUrpylTptx0+99//70kqXPnzjbLO3fuLGOMvvjiC+uyuXPnysfHR3369LEuK1SokHr16qX169dbr0GXLl1S0aJFrW2KFSumixcvWp8PHz5cNWvWVMeOHbOtKywsTLVq1dLXX39902MAAMBb0TMSAIB86IknntC///1vfffdd+rdu3eWbXbu3KkHHnhAtWrV0siRI+Xv7699+/Zp3bp1kqSqVatq5MiRGjp0qPr06aPGjRtLku655x7rNv766y+1bt1anTt31uOPP67w8PAb1vXaa6/JYrHohRdeUFJSkiZMmKAWLVpo27Zt1h6cOZGT2q5ljFH79u21cuVK9erVS3Xq1NGSJUv03HPP6ciRI3rzzTdt2q9du1Zffvml/vWvfykoKEgTJ05Up06ddOjQoUzh37UuXbqkpk2bat++ferXr5/KlSunOXPmqEePHjpz5owGDBigqlWrasaMGRo0aJBKly5tvd2zZMmSWW4zKipKkjRz5kw1bNjwhr1bjx8/rrvvvtsa9pUsWVKLFi1Sr169lJycbJ0k59KlS2revLkOHTqk/v37KzIyUjNmzNCKFSuy3fbNpKenq3379lq7dq369OmjqlWr6ueff9abb76pvXv3at68eTbtc3KOjx49qrvuuktnzpxRnz59VKVKFR05ckRz587VxYsX5efnp4sXL6pJkyY6cuSInnrqKZUtW1Y//PCDEhISdOzYMWsvt1sxY8YMde/eXbGxsRo7dqwuXryoyZMnq1GjRvrpp59sgu+0tDTFxsaqQYMGev3117Vs2TKNHz9eFSpUUN++fa3nql27dtq0aZP69u2rKlWq6Ouvv1b37t1t9vvUU0/p6NGjWrp0qWbMmJFlbZ999pnOnTunp556ShaLRePGjVPHjh31xx9/WHsId+rUSTt37tQzzzyj6OhoJSUlaenSpTp06NANJ3r64YcfZLFYVLduXbvO1/DhwzVixAi1aNFCffv21Z49ezR58mRt3rxZ69ats+m5fPr0abVp00aPPPKIunTpotmzZ6tv377y8/O7Yc/CjID7+utGQECAJGnr1q3WZT/99JMqVaqk4OBgm7Z33XWXJGnbtm0qU6aM7rzzTn344Yf67rvvVK5cOY0fP97aZteuXZoyZYo2bdp00+OvV69epvc7AAD5igEAAF5n6tSpRpLZvHlztm1CQkJM3bp1rc+HDRtmrv1q8OabbxpJ5sSJE9luY/PmzUaSmTp1aqZ1TZo0MZLMlClTslzXpEkT6/OVK1caSea2224zycnJ1uWzZ882ksxbb71lXRYVFWW6d+9+023eqLbu3bubqKgo6/N58+YZSebVV1+1afePf/zDWCwWs2/fPusyScbPz89m2fbt240k8/bbb2fa17UmTJhgJJlPP/3Uuiw1NdXExMSYwMBAm2OPiooybdu2veH2jDEmPT3deq7Dw8NNly5dzKRJk8zBgwczte3Vq5cpVaqUOXnypM3yzp07m5CQEHPx4kWbOmfPnm1tc+HCBVOxYkUjyaxcudKmzpz8PGbMmGEKFChgvv/+e5t2U6ZMMZLMunXrrMtyeo67detmChQokOX7PD093RhjzCuvvGKKFCli9u7da7M+Pj7e+Pj4mEOHDmV67fXHUb169WzXnzt3zoSGhprevXvbLE9MTDQhISE2y7t3724kmZEjR9q0rVu3rqlXr571+X//+18jyUyYMMG6LC0tzTRr1izTezouLs5k9ZV+//79RpIpXry4OXXqlHX5119/bSSZb775xhhjzOnTp40k85///OeG5yErjz/+uClevPgN28yZM8fmPZOUlGT8/PxMy5YtTVpamrXdO++8YySZjz/+2Los4309fvx467KUlBRTp04dExYWZlJTU7Pdb8Y5nDFjhs3yjPdbjRo1rMuqV69umjVrlmkbO3futLmGXb161XTs2NFIMpJMmTJlzI4dO4wxxrRs2dI8/fTTNzwXGUaNGmUkmePHj+eoPQAA3obbtAEAyKcCAwNvOKt2aGioJOnrr7/O9e2s/v7+6tmzZ47bd+vWzWZSh3/84x8qVaqUvv3221ztP6e+/fZb+fj4qH///jbLhwwZImOMFi1aZLO8RYsWqlChgvV5rVq1FBwcrD/++OOm+4mIiFCXLl2sywoWLKj+/fvr/PnzWr16td21WywWLVmyRK+++qqKFi2qzz//XHFxcYqKitKjjz5qvfXVGKP//ve/ateunYwxOnnypPURGxurs2fP6scff7TWWapUKf3jH/+w7icgIMDmNlZ7zZkzR1WrVlWVKlVs9t2sWTNJ0sqVK23a3+wcp6ena968eWrXrp3NuKjXnpeM/TZu3FhFixa12W+LFi2UlpamNWvW5PqYpL+HMjhz5oy6dOlis30fHx81aNAg03FJshljUZIaN25s895ZvHixChYsaNNruUCBAoqLi7O7vkcffdTm1uKMXsIZ+ytcuLD8/Py0atUqnT592q5t//XXXzbbzolly5YpNTVVAwcOVIEC//tTpHfv3goODtbChQtt2vv6+uqpp56yPvfz89NTTz2lpKQkm96N12vTpo2ioqL07LPP6ssvv9TBgwc1e/Zsvfjii/L19dWlS5esbS9dupTlGI+FChWyrpf+Hvbgv//9r3777Tdt2bJFe/fuVc2aNTV//nxt2rRJr7zyio4cOaJ27dopMjJS7dq109GjRzNtN+OcnTx5MienDAAAr0MYCQBAPnX+/Pkbzub66KOPqmHDhnryyScVHh6uzp07a/bs2XYFk7fddptdk9XcfvvtNs8tFosqVqx407HrbtXBgwcVGRmZ6XxkzP578OBBm+Vly5bNtI2iRYveNMw5ePCgbr/9dpsQ5kb7ySl/f3+9+OKL2r17t44eParPP/9cd999t2bPnq1+/fpJkk6cOKEzZ87o/fffV8mSJW0eGYFxUlKStY6KFStmGn+wcuXKuapPkn777Tft3Lkz074rVapks+8MNzvHJ06cUHJysmrUqHHT/S5evDjTfjPGObx+v7k5LunvsViv38d3332XafuFChXKdMv99e+dgwcPqlSpUtZbijNUrFjR7vquP48ZQVjG/vz9/TV27FgtWrRI4eHhuvfeezVu3DglJibmaPvmujFVbybjPX79e8nPz0/ly5fP9DsQGRmpIkWK2CzLeM/c6LpQqFAhLVy4UMWLF1enTp0UHR2tbt26aejQoSpWrJgCAwOtbQsXLpxp3FJJunz5snX9tSpWrKh69eqpUKFCSk1N1ZAhQzRs2DCVKFFCnTt3VuHChfXNN9+oUKFCeuyxxzJtN+OcZTW+JwAA+QFjRgIAkA/9+eefOnv27A3DjcKFC2vNmjVauXKlFi5cqMWLF+uLL75Qs2bN9N1338nHx+em+7FnnMecyu4P+LS0tBzV5AjZ7cfeYMYZSpUqpc6dO6tTp06qXr26Zs+erWnTpllD5McffzzT2IMZMmYXtkdOfx7p6emqWbOm3njjjSzblylTxua5o85xenq67r//fj3//PNZrs8ItnIr47zOmDFDERERmdZfP4anq96jN9vftedx4MCBateunebNm6clS5bo5Zdf1ujRo7VixYobjgdZvHhxu3tTulL16tX1yy+/aNeuXTp9+rSqVaumwoULa9CgQWrSpIm1XalSpXTkyJFMrz927JikvwPR7Lz55pvy9fVVv379dPjwYa1du1b79+9XdHS0xo0bp/Lly+vPP/9U6dKlra/JOGclSpRw1KECAJCnEEYCAJAPZUx2ERsbe8N2BQoUUPPmzdW8eXO98cYbGjVqlF588UWtXLlSLVq0cHjPnoxeZhmMMdq3b59NSFa0aNFMs+5Kf/e4Kl++vPW5PbVFRUVp2bJlOnfunE3vyF9//dW63hGioqK0Y8cOpaen2/SOdPR+pL9v/65Vq5Z+++03nTx5UiVLllRQUJDS0tJuOvtxVFSUfvnlFxljbM7jnj17MrXN6c+jQoUK2r59u5o3b+6Q903JkiUVHBysX3755YbtKlSooPPnz+dqxuecyLiVPCwszGH7iIqK0sqVK3Xx4kWb3pH79u3L1NZRv4MVKlTQkCFDNGTIEP3222+qU6eOxo8fr08//TTb11SpUkUzZ87U2bNnFRISkqP9ZLzH9+zZY/P+SE1N1f79+zOdw6NHj+rChQs2vSP37t0rSTecXCeDxWJR9erVrc+//fZbpaen2+ynTp06WrlypZKTk20msdm4caN1fVaOHTumV199VXPmzJGvr6/1luyM8DLjv0eOHLEJI/fv368SJUpkOykVAADejtu0AQDIZ1asWKFXXnlF5cqVU9euXbNtd+rUqUzLMv4oz7ilMSMgyCqMyo1PPvnEZhzLuXPn6tixY2rdurV1WYUKFbRhwwalpqZaly1YsECHDx+22ZY9tbVp00ZpaWl65513bJa/+eabslgsNvu/FW3atFFiYqK++OIL67KrV6/q7bffVmBgoE1vrZz67bffdOjQoUzLz5w5o/Xr16to0aIqWbKkfHx81KlTJ/33v//NMsA7ceKETZ1Hjx7V3LlzrcsuXryo999/P9PrcvrzeOSRR3TkyBF98MEHmbZx6dIlXbhwIWcH/P8KFCigDh066JtvvtGWLVsyrc/o+ffII49o/fr1WrJkSaY2Z86c0dWrV+3a7/ViY2MVHBysUaNG6cqVK5nWX3te7dnmlStXbM5Venq6Jk2alKntrf4OXrx40Xo7coYKFSooKCgoy1uXrxUTEyNjzA3HbrxeixYt5Ofnp4kTJ9r0zvzoo4909uxZtW3b1qb91atX9d5771mfp6am6r333lPJkiVVr169HO9X+vt99vLLL6tUqVI247b+4x//UFpams37OyUlRVOnTlWDBg0y9drNEB8fr3vvvVetWrWSJIWHh0v63z8u7N69W5Iy9ZjdunWrYmJi7KodAABvQs9IAAC82KJFi/Trr7/q6tWrOn78uFasWKGlS5cqKipK8+fPt07QkJWRI0dqzZo1atu2raKiopSUlKR3331XpUuXVqNGjST9HVqEhoZqypQpCgoKUpEiRdSgQQOVK1cuV/UWK1ZMjRo1Us+ePXX8+HFNmDBBFStWtJnI48knn9TcuXPVqlUrPfLII/r999/16aef2kx2Ym9t7dq103333acXX3xRBw4cUO3atfXdd9/p66+/1sCBAzNtO7f69Omj9957Tz169NDWrVsVHR2tuXPnat26dZowYcINx/DMzvbt2/XYY4+pdevWaty4sYoVK6YjR45o+vTpOnr0qCZMmGC9VXfMmDFauXKlGjRooN69e6tatWo6deqUfvzxRy1btswaQPfu3VvvvPOOunXrpq1bt6pUqVKaMWNGpjEMpZz/PJ544gnNnj1bTz/9tFauXKmGDRsqLS1Nv/76q2bPnq0lS5ZkORHNjYwaNUrfffedmjRpoj59+qhq1ao6duyY5syZo7Vr1yo0NFTPPfec5s+frwceeEA9evRQvXr1dOHCBf3888+aO3euDhw4cNPbZU+cOKFXX3010/KMQH/y5Ml64okndMcdd6hz584qWbKkDh06pIULF6phw4aZQu6b6dChg+666y4NGTJE+/btU5UqVTR//nzrz+fa3pAZgVz//v0VGxsrHx8fde7cOcf72rt3r5o3b65HHnlE1apVk6+vr7766isdP378pttp1KiRihcvrmXLllknIrqZkiVLKiEhQSNGjFCrVq3Uvn177dmzR++++67uvPNOPf744zbtIyMjNXbsWB04cECVKlXSF198oW3btun9999XwYIFb7ivRx55RJGRkapWrZqSk5P18ccf648//tDChQttftcaNGighx9+WAkJCUpKSlLFihU1ffp0HThwQB999FGW2960aZO++OIL7dixw7osOjpa9evXV48ePdSrVy99+OGHatCggU2P56SkJO3YsSNXkxEBAOA13DKHNwAAcKqpU6caSdaHn5+fiYiIMPfff7956623THJycqbXDBs2zFz71WD58uXmwQcfNJGRkcbPz89ERkaaLl26mL1799q87uuvvzbVqlUzvr6+RpKZOnWqMcaYJk2amOrVq2dZX5MmTUyTJk2sz1euXGkkmc8//9wkJCSYsLAwU7hwYdO2bVtz8ODBTK8fP368ue2224y/v79p2LCh2bJlS6Zt3qi27t27m6ioKJu2586dM4MGDTKRkZGmYMGC5vbbbzf/+c9/THp6uk07SSYuLi5TTVFRUaZ79+5ZHu+1jh8/bnr27GlKlChh/Pz8TM2aNa11Xb+9tm3b5mh7Y8aMMU2aNDGlSpUyvr6+pmjRoqZZs2Zm7ty5WbaPi4szZcqUMQULFjQRERGmefPm5v3337dpd/DgQdO+fXsTEBBgSpQoYQYMGGAWL15sJJmVK1fatM3pzyM1NdWMHTvWVK9e3fj7+5uiRYuaevXqmREjRpizZ89a29lzjg8ePGi6detmSpYsafz9/U358uVNXFycSUlJsbY5d+6cSUhIMBUrVjR+fn6mRIkS5p577jGvv/66SU1NveH5bdKkic3v0rWP5s2bW9utXLnSxMbGmpCQEFOoUCFToUIF06NHD7NlyxZrm+7du5siRYpk2sf1v3vGGHPixAnz2GOPmaCgIBMSEmJ69Ohh1q1bZySZWbNmWdtdvXrVPPPMM6ZkyZLGYrFYt7N//34jyfznP//JtD9JZtiwYcYYY06ePGni4uJMlSpVTJEiRUxISIhp0KCBmT179g3PS4b+/fubihUrZrt+zpw5Wb5n3nnnHVOlShVTsGBBEx4ebvr27WtOnz5t0ybjGrJlyxYTExNjChUqZKKiosw777yTo9rGjh1rqlSpYgoVKmSKFi1q2rdvb3766acs2166dMk8++yzJiIiwvj7+5s777zTLF68OMu26enppkGDBmbw4MGZ1u3bt8/ce++9JjAw0Nx7773m999/t1k/efJkExAQkOU1GACA/MJijAeMtA4AAACPt2rVKt13331auXKlmjZt6u5y8p158+bpoYce0tq1a9WwYUN3lyNJ+uOPP1SlShUtWrRIzZs3d+i2mzZtqpMnT950XNC8pG7dumratKnefPNNd5cCAIDbMGYkAAAA4GEuXbpk8zwtLU1vv/22goODdccdd7ipqszKly+vXr16acyYMe4uxeMtXrxYv/32mxISEtxdCgAAbsWYkQAAAICHeeaZZ3Tp0iXFxMQoJSVFX375pX744QeNGjVKhQsXdnd5NiZPnuzuEvKEVq1a6fz58+4uAwAAtyOMBAAAADxMs2bNNH78eC1YsECXL19WxYoV9fbbb6tfv37uLg0AAOCWMGYkAAAAAAAAAJdgzEgAAAAAAAAALkEYCQAAAAAAAMAlGDNSUnp6uo4ePaqgoCBZLBZ3lwMAAAAAAADkKcYYnTt3TpGRkSpQIPv+j4SRko4ePaoyZcq4uwwAAAAAAAAgTzt8+LBKly6d7XrCSElBQUGS/j5ZwcHBbq4GAAAAAAAAyFuSk5NVpkwZa86WHcJIyXprdnBwMGEkAAAAAAAAkEs3GwKRCWwAAAAAAAAAuARhJAAAAAAAAACXIIwEAAAAAAAA4BKEkQAAAAAAAABcgjASAAAAAAAAgEsQRgIAAAAAAABwCcJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAQAAAAAAALgEYSQAAAAAAAAAlyCMBAAAAAAAAOAShJEAAAAAAAAAXIIwEgAAAAAAAIBLEEYCAAAAAAAAcAnCSAAAAAAAAAAuQRgJAA4SHb/Q3SUAAAAAAODRCCMBAAAAAAAAuARhJAAAAAAAAACXIIwEAAAAAAAA4BKEkQAAAAAAAABcgjASAAAAAAAAgEsQRgIAAAAAAABwCcJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAcBFouMXursEAAAAAADcijASAAAAAAAAgEsQRgIAAAAAAABwCcJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAQAAAAAAALgEYSQAAAAAAAAAlyCMBAAAAAAAAOAShJEAAAAAAAAAXIIwEgCcJDp+YY7a5KSdI7l6fwAAAAAAZCCMBAAPRGAIAAAAAPBGhJEAAAAAAAAAXIIwEgBcwJE9Hek1CQAAAADIqwgjAQAAAAAAALgEYSQAAAAAAAAAlyCMBAAP4YiZtbmFGwAAAADgyQgjAcBNCA4BAAAAAPkNYSQAuAFBJAAAAAAgP3JrGDl58mTVqlVLwcHBCg4OVkxMjBYtWmRdf/nyZcXFxal48eIKDAxUp06ddPz4cZttHDp0SG3btlVAQIDCwsL03HPP6erVq64+FAD5mCNurwYAAAAAID9waxhZunRpjRkzRlu3btWWLVvUrFkzPfjgg9q5c6ckadCgQfrmm280Z84crV69WkePHlXHjh2tr09LS1Pbtm2VmpqqH374QdOnT9e0adM0dOhQdx0SAAAAAAAAgGz4unPn7dq1s3n+2muvafLkydqwYYNKly6tjz76SJ999pmaNWsmSZo6daqqVq2qDRs26O6779Z3332nXbt2admyZQoPD1edOnX0yiuv6IUXXtDw4cPl5+fnjsMCAAAAAAAAkAWPGTMyLS1Ns2bN0oULFxQTE6OtW7fqypUratGihbVNlSpVVLZsWa1fv16StH79etWsWVPh4eHWNrGxsUpOTrb2rsxKSkqKkpOTbR4AAMfi1nUAAAAAwPXcHkb+/PPPCgwMlL+/v55++ml99dVXqlatmhITE+Xn56fQ0FCb9uHh4UpMTJQkJSYm2gSRGesz1mVn9OjRCgkJsT7KlCnj2IMCkG/cLHBz93iSBIIAAAAAAE/i9jCycuXK2rZtmzZu3Ki+ffuqe/fu2rVrl1P3mZCQoLNnz1ofhw8fdur+AAAAAAAAALh5zEhJ8vPzU8WKFSVJ9erV0+bNm/XWW2/p0UcfVWpqqs6cOWPTO/L48eOKiIiQJEVERGjTpk0228uYbTujTVb8/f3l7+/v4CMBAAAAAAAAcCNu7xl5vfT0dKWkpKhevXoqWLCgli9fbl23Z88eHTp0SDExMZKkmJgY/fzzz0pKSrK2Wbp0qYKDg1WtWjWX1w4AGbK7PdqRt227+xZwAAAAAADs5daekQkJCWrdurXKli2rc+fO6bPPPtOqVau0ZMkShYSEqFevXho8eLCKFSum4OBgPfPMM4qJidHdd98tSWrZsqWqVaumJ554QuPGjVNiYqJeeuklxcXF0fMRAG5RdPxCHRjT1t1lAAAAAAC8iFvDyKSkJHXr1k3Hjh1TSEiIatWqpSVLluj++++XJL355psqUKCAOnXqpJSUFMXGxurdd9+1vt7Hx0cLFixQ3759FRMToyJFiqh79+4aOXKkuw4JAAAAAAAAQDbcGkZ+9NFHN1xfqFAhTZo0SZMmTcq2TVRUlL799ltHlwYAXi3j9m56PgIAAAAAXMnjxowEAHiW68emZKxKAAAAAEBuuX02bQBA1nLae5FgEAAAAACQV9AzEgAAAAAAAIBLEEYCAHLMHb0w6fkJAAAAAN6DMBIAAAAAAACASxBGAgCs6IUIAAAAAHAmwkgAsIOzwjpPCgGZLRsAAAAA4CyEkQCQj3hSyOhJtQAAAAAAXIMwEgAAAAAAAIBLEEYCgJejByIAAAAAwFMQRgIAHCInoSfjUQIAAABA/kYYCQC5RKgGAAAAAIB9CCMBwMu4MiS1Z1+EtwAAAAAAwkgAyCHCNAAAAAAAbg1hJAAAAAAAAACXIIwEAGQrr/YGZaIcAAAAAPBMhJEAAAAAAAAAXMLX3QUAAJyPXoIAAAAAAE9Az0gAAAAAAAAALkHPSABAjmTXuzI3vS7pqQkAAAAA+RM9IwEAAAAAAAC4BGEkAHgYb+g16A3HAAAAAABwPMJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAQAAAAAAALgEs2kDgBfytDEbPa0eAAAAAIB7EEYCgJ3cFax5Y6CXcUwHxrR1cyUAAAAAAFfgNm0AQK54YzgKAAAAAHAuwkgAgEch5AQAAAAA70UYCQD5HOEfAAAAAMBVGDMSAOAwBJsAAAAAgBuhZyQA5EGEfgAAAACAvIgwEgA8HMFj1qLjF9p9bjiXAAAAAOBehJEAAAAAAAAAXIIwEgDyAHr0AQAAAAC8ARPYAEA+5uyQ057tE7gCAAAAgPejZyQAXINADAAAAAAA56FnJADcBAElAAAAAACOQc9IAAAAAAAAAC5Bz0gAcCB6UQIAAAAAkD16RgIA3I4QFwAAAADyB8JIALhOdPxCwjEAAAAAAJyAMBIAAAAAAACASxBGAgA8Dj1TAQAAAMA7EUYCAAAAAAAAcAm3hpGjR4/WnXfeqaCgIIWFhalDhw7as2ePTZumTZvKYrHYPJ5++mmbNocOHVLbtm0VEBCgsLAwPffcc7p69aorDwUAAAAAAADATbg1jFy9erXi4uK0YcMGLV26VFeuXFHLli114cIFm3a9e/fWsWPHrI9x48ZZ16Wlpalt27ZKTU3VDz/8oOnTp2vatGkaOnSoqw8HAOBm3N4NAAAAAJ7N1507X7x4sc3zadOmKSwsTFu3btW9995rXR4QEKCIiIgst/Hdd99p165dWrZsmcLDw1WnTh298soreuGFFzR8+HD5+fk59RgAAM5xbbAYHb9QB8a0dWM1AAAAAABH8KgxI8+ePStJKlasmM3ymTNnqkSJEqpRo4YSEhJ08eJF67r169erZs2aCg8Pty6LjY1VcnKydu7cmeV+UlJSlJycbPMAAAAAAAAA4Fxu7Rl5rfT0dA0cOFANGzZUjRo1rMsfe+wxRUVFKTIyUjt27NALL7ygPXv26Msvv5QkJSYm2gSRkqzPExMTs9zX6NGjNWLECCcdCQAAAAAAAICseEwYGRcXp19++UVr1661Wd6nTx/r/9esWVOlSpVS8+bN9fvvv6tChQq52ldCQoIGDx5sfZ6cnKwyZcrkrnAAAAAAAAAAOeIRYWS/fv20YMECrVmzRqVLl75h2wYNGkiS9u3bpwoVKigiIkKbNm2yaXP8+HFJynacSX9/f/n7+zugcgDwDkz8AgAAAABwBbeOGWmMUb9+/fTVV19pxYoVKleu3E1fs23bNklSqVKlJEkxMTH6+eeflZSUZG2zdOlSBQcHq1q1ak6pGwAAAAAAAID93BpGxsXF6dNPP9Vnn32moKAgJSYmKjExUZcuXZIk/f7773rllVe0detWHThwQPPnz1e3bt107733qlatWpKkli1bqlq1anriiSe0fft2LVmyRC+99JLi4uLo/QgA+cD1vTrp5QkAAAAAnsutYeTkyZN19uxZNW3aVKVKlbI+vvjiC0mSn5+fli1bppYtW6pKlSoaMmSIOnXqpG+++ca6DR8fHy1YsEA+Pj6KiYnR448/rm7dumnkyJHuOiwAAAAAAAAAWXDrmJHGmBuuL1OmjFavXn3T7URFRenbb791VFkAAAAAAAAAnMCtPSMBwNNxyy8AAAAAAI5DGAkAAAAAAADAJQgjAQB5Cr1VAQAAACDvIowEAAAAAAAA4BKEkQAAAAAAAABcgjASAAAAAAAAgEsQRgIAAAAAAABwCcJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAQD5WnT8QneXAAAAAAD5BmEkAAAAAAAAAJfwdXcBAOCp6DHnWfh5AAAAAEDeR89IAAAAAAAAAC5hdxh5+PBh/fnnn9bnmzZt0sCBA/X+++87tDAAAAAAAAAA3sXuMPKxxx7TypUrJUmJiYm6//77tWnTJr344osaOXKkwwsEAAAAAAAA4B3sDiN/+eUX3XXXXZKk2bNnq0aNGvrhhx80c+ZMTZs2zdH1AQAAAAAAAPASdk9gc+XKFfn7+0uSli1bpvbt20uSqlSpomPHjjm2OgAAHIDJbwAAAADAM9jdM7J69eqaMmWKvv/+ey1dulStWrWSJB09elTFixd3eIEAAAAAAAAAvIPdYeTYsWP13nvvqWnTpurSpYtq164tSZo/f7719m0AAJzJ0T0do+MX0nsSAAAAAFzA7tu0mzZtqpMnTyo5OVlFixa1Lu/Tp48CAgIcWhwAAJ4iOn6hDoxp6+4yAAAAACBPs7tnpCQZY7R161a99957OnfunCTJz8+PMBIA4PXoRQkAAAAAuWd3GHnw4EHVrFlTDz74oOLi4nTixAlJf9++/eyzzzq8QAAAcovQEAAAAAA8i91h5IABA1S/fn2dPn1ahQsXti5/6KGHtHz5cocWBwAAAAAAAMB72D1m5Pfff68ffvhBfn5+Nsujo6N15MgRhxUGAAAAAAAAwLvY3TMyPT1daWlpmZb/+eefCgoKckhRAAAAAAAAALyP3WFky5YtNWHCBOtzi8Wi8+fPa9iwYWrTpo0jawMAwKNlNyZlVssZvxIAAAAAchFGjh8/XuvWrVO1atV0+fJlPfbYY9ZbtMeOHeuMGgEAQDYIOQEAAADkJXaPGVm6dGlt375ds2bN0o4dO3T+/Hn16tVLXbt2tZnQBgAATxQdv1AHxrR1dxkAAAAAkC/ZHUZKkq+vrx5//HFH1wIALkcwlT/lpDch7w0AAAAAcLwchZHz58/P8Qbbt2+f62IAAAAAAAAAeK8chZEdOnTI0cYsFkuWM20DAIDMMnpo0gMTAAAAQH6Rowls0tPTc/QgiAQAeJMb3c6d3YzZuZlQhkloAAAAAOQXds+mDQAAAAAAAAC5kaswcvny5XrggQdUoUIFVahQQQ888ICWLVvm6NoAAMgzHNm7kZ6SAAAAALyV3WHku+++q1atWikoKEgDBgzQgAEDFBwcrDZt2mjSpEnOqBEAAAAAAACAF8jRBDbXGjVqlN58803169fPuqx///5q2LChRo0apbi4OIcWCAAAAAAAAMA72N0z8syZM2rVqlWm5S1bttTZs2cdUhQAAI7CLc8AAAAA4DnsDiPbt2+vr776KtPyr7/+Wg888IBDigIAwB0cOUM2AAAAACAzu2/Trlatml577TWtWrVKMTExkqQNGzZo3bp1GjJkiCZOnGht279/f8dVCgAAAAAAACBPszuM/Oijj1S0aFHt2rVLu3btsi4PDQ3VRx99ZH1usVgIIwEALkHPRQAAAADIG+wOI/fv3++MOgAAAAAAAAB4ObvHjAQAAPa5vuemPT056fUJAAAAwJvY3TPSGKO5c+dq5cqVSkpKUnp6us36L7/80mHFAQAAAAAAAPAedoeRAwcO1Hvvvaf77rtP4eHhslgszqgLAAAAAAAAgJexO4ycMWOGvvzyS7Vp08YZ9QAAAAAAAADwUnaPGRkSEqLy5cs7ZOejR4/WnXfeqaCgIIWFhalDhw7as2ePTZvLly8rLi5OxYsXV2BgoDp16qTjx4/btDl06JDatm2rgIAAhYWF6bnnntPVq1cdUiMAAAAAAAAAx7A7jBw+fLhGjBihS5cu3fLOV69erbi4OG3YsEFLly7VlStX1LJlS124cMHaZtCgQfrmm280Z84crV69WkePHlXHjh2t69PS0tS2bVulpqbqhx9+0PTp0zVt2jQNHTr0lusDACC3nD3xTHT8Qia3AQAAAJDn2H2b9iOPPKLPP/9cYWFhio6OVsGCBW3W//jjjzne1uLFi22eT5s2TWFhYdq6davuvfdenT17Vh999JE+++wzNWvWTJI0depUVa1aVRs2bNDdd9+t7777Trt27dKyZcsUHh6uOnXq6JVXXtELL7yg4cOHy8/Pz95DBAAAAAAAAOAEdoeR3bt319atW/X44487fAKbs2fPSpKKFSsmSdq6dauuXLmiFi1aWNtUqVJFZcuW1fr163X33Xdr/fr1qlmzpsLDw61tYmNj1bdvX+3cuVN169bNtJ+UlBSlpKRYnycnJzvsGAAAAAAAAABkze4wcuHChVqyZIkaNWrk0ELS09M1cOBANWzYUDVq1JAkJSYmys/PT6GhoTZtw8PDlZiYaG1zbRCZsT5jXVZGjx6tESNGOLR+AIB34NZnAAAAAHAeu8eMLFOmjIKDgx1eSFxcnH755RfNmjXL4du+XkJCgs6ePWt9HD582On7BADAUQhMAQAAAORVdoeR48eP1/PPP68DBw44rIh+/fppwYIFWrlypUqXLm1dHhERodTUVJ05c8am/fHjxxUREWFtc/3s2hnPM9pcz9/fX8HBwTYPAPkXE4EAAAAAAOAadoeRjz/+uFauXKkKFSooKChIxYoVs3nYwxijfv366auvvtKKFStUrlw5m/X16tVTwYIFtXz5cuuyPXv26NChQ4qJiZEkxcTE6Oeff1ZSUpK1zdKlSxUcHKxq1arZe3gAAGTLG0JrbzgGAAAAAHmX3WNGTpgwwWE7j4uL02effaavv/5aQUFB1jEeQ0JCVLhwYYWEhKhXr14aPHiwihUrpuDgYD3zzDOKiYnR3XffLUlq2bKlqlWrpieeeELjxo1TYmKiXnrpJcXFxcnf399htQIAAAAAAAC4NbmaTdtRJk+eLElq2rSpzfKpU6eqR48ekqQ333xTBQoUUKdOnZSSkqLY2Fi9++671rY+Pj5asGCB+vbtq5iYGBUpUkTdu3fXyJEjHVYnAAC5cX0vRHolAgAAAMjv7A4jr3X58mWlpqbaLLNn/EVjzE3bFCpUSJMmTdKkSZOybRMVFaVvv/02x/sFAMBRCBgBAAAAIOfsHjPywoUL6tevn8LCwlSkSBEVLVrU5gEAeRWhEgAAAAAAzmV3GPn8889rxYoVmjx5svz9/fXhhx9qxIgRioyM1CeffOKMGgEA8Dq5Db/teZ29+yCQBwAAAOBsdt+m/c033+iTTz5R06ZN1bNnTzVu3FgVK1ZUVFSUZs6cqa5duzqjTgAAkEOEkAAAAAA8ld09I0+dOqXy5ctL+nt8yFOnTkmSGjVqpDVr1ji2OgAA8hmCQQAAAADezO4wsnz58tq/f78kqUqVKpo9e7akv3tMhoaGOrQ4AADwt+j4hQSVAAAAAPI8u2/T7tmzp7Zv364mTZooPj5e7dq10zvvvKMrV67ojTfecEaNAADkOwSPAAAAALyR3WHkoEGDrP/fokUL7d69Wz/++KMqVqyoWrVqObQ4AAAAAAAAAN7D7jDyetHR0YqOjnZAKQAAIK+Kjl+oA2PaursMAAAAAB4ux2NGrl+/XgsWLLBZ9sknn6hcuXIKCwtTnz59lJKS4vACAcDRuP0VAAAAAAD3yHEYOXLkSO3cudP6/Oeff1avXr3UokULxcfH65tvvtHo0aOdUiQAALh1BPEAAAAA3C3HYeS2bdvUvHlz6/NZs2apQYMG+uCDDzR48GBNnDjROrM2AADwDASQAAAAADxJjseMPH36tMLDw63PV69erdatW1uf33nnnTp8+LBjqwMAJyGgAQAAAADA9XLcMzI8PFz79++XJKWmpurHH3/U3XffbV1/7tw5FSxY0PEVAgAAAAAAAPAKOQ4j27Rpo/j4eH3//fdKSEhQQECAGjdubF2/Y8cOVahQwSlFAgAAAAAAAMj7cnyb9iuvvKKOHTuqSZMmCgwM1PTp0+Xn52dd//HHH6tly5ZOKRIAAHi+jOEPDoxp6+ZKAAAAAHiqHIeRJUqU0Jo1a3T27FkFBgbKx8fHZv2cOXMUGBjo8AIBAAAAAAAAeIcch5EZQkJCslxerFixWy4GAAC4lr2TOV3bnh6QAAAAAOxldxgJAAAcz5UzvHM7NQAAAAB3yfEENgAAwHNFxy90aaBpD0+tCwAAAIDrEUYCAAAAAAAAcIkchZF33HGHTp8+LUkaOXKkLl686NSiAACAe3hyD0sAAAAAeV+Owsjdu3frwoULkqQRI0bo/PnzTi0KAAB4L8JOAAAAIP/K0QQ2derUUc+ePdWoUSMZY/T6668rMDAwy7ZDhw51aIEAAODGCPcAAAAA5BU5CiOnTZumYcOGacGCBbJYLFq0aJF8fTO/1GKxEEYCAICbio5fyGzeAAAAQD6UozCycuXKmjVrliSpQIECWr58ucLCwpxaGAAA+Js7ez5m7NsRwSE9OAEAAADkKIy8Vnp6ujPqAAAAAAAAAODl7A4jJen333/XhAkTtHv3bklStWrVNGDAAFWoUMGhxQEAAAAAAADwHjmaTftaS5YsUbVq1bRp0ybVqlVLtWrV0saNG1W9enUtXbrUGTUCAAAPxG3XAAAAAOxld8/I+Ph4DRo0SGPGjMm0/IUXXtD999/vsOIAAAAAAAAAeA+7e0bu3r1bvXr1yrT8n//8p3bt2uWQogAAgOfx5J6Q0fELPbo+AAAAAH+zO4wsWbKktm3blmn5tm3bmGEbAAAAAAAAQLbsvk27d+/e6tOnj/744w/dc889kqR169Zp7NixGjx4sMMLBABHoucU8D/8PgAAAABwNbvDyJdffllBQUEaP368EhISJEmRkZEaPny4+vfv7/ACAQCA+xFcAgAAAHAEu2/TtlgsGjRokP7880+dPXtWZ8+e1Z9//qkBAwbIYrE4o0YAAJBDnhIaekodAAAAADyL3T0jrxUUFOSoOgAAAAAAAAB4Obt7RgIAAAAAAABAbhBGAgAAt4uOX8it3QAAAEA+cEu3aQMAAM+Xm5CPYBAAAACAM9jVM/LKlStq3ry5fvvtN2fVAwAORaACAAAAAIDnsKtnZMGCBbVjxw5n1QIAAPIYAn8AAAAA9rB7zMjHH39cH330kTNqAQAAAAAAAODF7B4z8urVq/r444+1bNky1atXT0WKFLFZ/8YbbzisOAAAAGeKjl+oA2PaursMAAAAIN+wO4z85ZdfdMcdd0iS9u7da7POYrE4pioAAAAAAAAAXsfuMHLlypXOqAMAAHgJxpEEAAAAkB27x4zMsG/fPi1ZskSXLl2SJBljHFYUAAAAAAAAAO9jdxj5119/qXnz5qpUqZLatGmjY8eOSZJ69eqlIUOG2LWtNWvWqF27doqMjJTFYtG8efNs1vfo0UMWi8Xm0apVK5s2p06dUteuXRUcHKzQ0FD16tVL58+ft/ewAAAAAAAAADiZ3WHkoEGDVLBgQR06dEgBAQHW5Y8++qgWL15s17YuXLig2rVra9KkSdm2adWqlY4dO2Z9fP755zbru3btqp07d2rp0qVasGCB1qxZoz59+th3UAAAIM/gNnAAAAAg77J7zMjvvvtOS5YsUenSpW2W33777Tp48KBd22rdurVat259wzb+/v6KiIjIct3u3bu1ePFibd68WfXr15ckvf3222rTpo1ef/11RUZG2lUPAO9FeAEAAAAAgPvZ3TPywoULNj0iM5w6dUr+/v4OKepaq1atUlhYmCpXrqy+ffvqr7/+sq5bv369QkNDrUGkJLVo0UIFChTQxo0bs91mSkqKkpOTbR4AAAAAAAAAnMvuMLJx48b65JNPrM8tFovS09M1btw43XfffQ4trlWrVvrkk0+0fPlyjR07VqtXr1br1q2VlpYmSUpMTFRYWJjNa3x9fVWsWDElJiZmu93Ro0crJCTE+ihTpoxD6wYAAAAAAACQmd23aY8bN07NmzfXli1blJqaqueff147d+7UqVOntG7dOocW17lzZ+v/16xZU7Vq1VKFChW0atUqNW/ePNfbTUhI0ODBg63Pk5OTCSQBAAAAAAAAJ7O7Z2SNGjW0d+9eNWrUSA8++KAuXLigjh076qefflKFChWcUaNV+fLlVaJECe3bt0+SFBERoaSkJJs2V69e1alTp7IdZ1L6exzK4OBgmwcA78DYkIB3uZXf6axeyzUCAAAAcC+7e0ZKUkhIiF588UVH13JTf/75p/766y+VKlVKkhQTE6MzZ85o69atqlevniRpxYoVSk9PV4MGDVxeHwAAcC7CRAAAACBvy1UYefr0aX300UfavXu3JKlatWrq2bOnihUrZtd2zp8/b+3lKEn79+/Xtm3bVKxYMRUrVkwjRoxQp06dFBERod9//13PP/+8KlasqNjYWElS1apV1apVK/Xu3VtTpkzRlStX1K9fP3Xu3JmZtAEAAAAAAAAPY/dt2mvWrFF0dLQmTpyo06dP6/Tp05o4caLKlSunNWvW2LWtLVu2qG7duqpbt64kafDgwapbt66GDh0qHx8f7dixQ+3bt1elSpXUq1cv1atXT99//73NrN0zZ85UlSpV1Lx5c7Vp00aNGjXS+++/b+9hAQAAAAAAAHAyu3tGxsXF6dFHH9XkyZPl4+MjSUpLS9O//vUvxcXF6eeff87xtpo2bSpjTLbrlyxZctNtFCtWTJ999lmO9wkAADwft2MDAAAA3snunpH79u3TkCFDrEGkJPn4+Gjw4ME2t1wDAADkBkEkAAAA4L3sDiPvuOMO61iR19q9e7dq167tkKIAAED+cbPwkXASAAAA8B45uk17x44d1v/v37+/BgwYoH379unuu++WJG3YsEGTJk3SmDFjnFMlAAAAAAAAgDwvR2FknTp1ZLFYbMZ3fP755zO1e+yxx/Too486rjoAyCV6UgHuR49HAAAAANfLURi5f/9+Z9cBAAC8mKuCx+j4hTowpq1L9gUAAADAfjkKI6OiopxdBwAAgFPRExMAAABwvxyFkdc7evSo1q5dq6SkJKWnp9us69+/v0MKA4CcoicUAAAAAAB5g91h5LRp0/TUU0/Jz89PxYsXl8Visa6zWCyEkQAAAAAAAACyZHcY+fLLL2vo0KFKSEhQgQIFnFETADgUt2YCAAAAAOAZ7E4TL168qM6dOxNEAgAAh+MfDwAAAADvZnei2KtXL82ZM8cZtQAAAAAAAADwYnbfpj169Gg98MADWrx4sWrWrKmCBQvarH/jjTccVhwAAIA7MDEWAAAA4By5CiOXLFmiypUrS1KmCWwAAAA8BaEiAAAA4FnsDiPHjx+vjz/+WD169HBCOQAAAAAAAAC8ld1jRvr7+6thw4bOqAUAAMAuTHgDAAAA5C12h5EDBgzQ22+/7YxaAAAA7EYgCQAAAOQddt+mvWnTJq1YsUILFixQ9erVM01g8+WXXzqsOAAAAAAAAADew+4wMjQ0VB07dnRGLQCQa/SMAgAAAADA89kdRk6dOtUZdQAAAOSYq/4BImM/zMgNAAAAOIbdY0YCAAAAAAAAQG7Y3TOyXLlyslgs2a7/448/bqkgAAAATxQdv5AekgAAAMAtsjuMHDhwoM3zK1eu6KefftLixYv13HPPOaouAAAAl2P8WQAAAMC57A4jBwwYkOXySZMmacuWLbdcEAAAwPUICQEAAADv4LAxI1u3bq3//ve/jtocAACAQ0THL3R6mElYCgAAAOSM3T0jszN37lwVK1bMUZsDAADweISQAAAAgH3sDiPr1q1rM4GNMUaJiYk6ceKE3n33XYcWBwAA4GjZBYgEiwAAAIDz2R1GdujQweZ5gQIFVLJkSTVt2lRVqlRxVF0AAAAAAAAAvIzdYeSwYcOcUQcAAAAAAAAAL+ewCWwAAADys2snyuGWbwAAACBrOQ4jCxQoIB8fnxs+fH0dNh8OAGSLP/IBuFpurjuumMUbAAAAyGtynB5+9dVX2a5bv369Jk6cqPT0dIcUBQAAkN9kBJcHxrR1cyUAAACA8+Q4jHzwwQczLduzZ4/i4+P1zTffqGvXrho5cqRDiwMAAAAAAADgPXI1ZuTRo0fVu3dv1axZU1evXtW2bds0ffp0RUVFObo+APkUtzYC8Hbcxg0AAID8yK5BHs+ePatRo0bp7bffVp06dbR8+XI1btzYWbUBAAB4jWuDR27FBgAAQH6V4zBy3LhxGjt2rCIiIvT5559neds2AACAu+W13obR8QsJJwEAAJBv5DiMjI+PV+HChVWxYkVNnz5d06dPz7Ldl19+6bDiAAAAAAAAAHiPHIeR3bp1k8VicWYtAAAAAAAAALxYjsPIadOmObEMAHCcvHaLJgAAAAAA+UWuZtMGAADIj5gBGwAAALg1hJEAPBZ/9APwBlzHAAAAgP8hjAQAAAAAAADgEoSRAAAATnZ970h6SwIAACC/IowEAADIBQJFAAAAwH6EkQDyJEIAAN6K6xsAAAC8GWEkAACAAxEmAgAAANlzaxi5Zs0atWvXTpGRkbJYLJo3b57NemOMhg4dqlKlSqlw4cJq0aKFfvvtN5s2p06dUteuXRUcHKzQ0FD16tVL58+fd+FRAAAAAAAAAMgJt4aRFy5cUO3atTVp0qQs148bN04TJ07UlClTtHHjRhUpUkSxsbG6fPmytU3Xrl21c+dOLV26VAsWLNCaNWvUp08fVx0CAAAAAAAAgBzydefOW7durdatW2e5zhijCRMm6KWXXtKDDz4oSfrkk08UHh6uefPmqXPnztq9e7cWL16szZs3q379+pKkt99+W23atNHrr7+uyMhIlx0LANfjVkgAjpCbawnXHwAAACB3PHbMyP379ysxMVEtWrSwLgsJCVGDBg20fv16SdL69esVGhpqDSIlqUWLFipQoIA2btyY7bZTUlKUnJxs8wDg+aLjFxIAAMh3uO4BAADAm3hsGJmYmChJCg8Pt1keHh5uXZeYmKiwsDCb9b6+vipWrJi1TVZGjx6tkJAQ66NMmTIOrh4AAMDxCCYBAACQ13lsGOlMCQkJOnv2rPVx+PBhd5cEAAAAAAAAeD2PDSMjIiIkScePH7dZfvz4ceu6iIgIJSUl2ay/evWqTp06ZW2TFX9/fwUHB9s8AAAAAAAAADiXx4aR5cqVU0REhJYvX25dlpycrI0bNyomJkaSFBMTozNnzmjr1q3WNitWrFB6eroaNGjg8poBAAAAAAAAZM+ts2mfP39e+/btsz7fv3+/tm3bpmLFiqls2bIaOHCgXn31Vd1+++0qV66cXn75ZUVGRqpDhw6SpKpVq6pVq1bq3bu3pkyZoitXrqhfv37q3LkzM2kDAAAAAAAAHsatYeSWLVt03333WZ8PHjxYktS9e3dNmzZNzz//vC5cuKA+ffrozJkzatSokRYvXqxChQpZXzNz5kz169dPzZs3V4ECBdSpUydNnDjR5ccCAAAAAAAA4MbcGkY2bdpUxphs11ssFo0cOVIjR47Mtk2xYsX02WefOaM8AAAAt2L2bAAAAHgbjx0zEgAAwJ0IAgEAAADHc2vPSADICQIBAAAAAAC8Az0jAQAA3MwT/tHFE2oAAACA96NnJAAAgIchGAQAAIC3omckAABAHhcdv/CmASYBJwAAADwBYSQAAEAe5oyQkeASAAAAzkIYCQAAkEfZGxoSMgIAAMDdCCMBAAAAAAAAuARhJAAAAAAAAACXYDZtAAAAD8ft1QAAAPAW9IwEAAAAAAAA4BKEkQAAAAAAAABcgjASAAAAduPWcQAAAOQGYSQAAAAAAAAAlyCMBAAAyGOi4xfmumfita+ldyMAAABcjTASAADASzgrXHR2aEkoCgAAkH8QRgIAAHgRZ/Z8zE2PTIJGAAAAXIswEgAAAAAAAIBLEEYCcKtbGfcMAOCZuK4DAAAgO77uLgAAAAA556qgLyf7yWhzYExbZ5cDAAAAL0HPSAC3JKd/FNvzxzM9agDA+bjWAgAAwB0IIwEAAAAAAAC4BGEkAAAAAAAAAJdgzEgAAABI4tZtAAAAOB9hJAC3uf6PXv4IBgDP5c5rNBPlAAAAeA9u0wZwy6LjFxIkAkA+xiRlAAAAyCnCSABuwR+jAAAAAADkP4SRAAAAAAAAAFyCMBIeJbvbfelF5xn4OQBA3nOzoTQYagMAAACuRBgJINdy88crgTMA5D9c4wEAAJCBMBJuR48M78TPFAAAAAAAXI8wEgAAAAAAAIBLEEYCAAAAAAAAcAnCSAAuw63bAAAAAADkb4SRyFMYXxIAgLzh+s/s3HyGe8pnvqfUAQAA4A0IIwHYxZ4/yPjjDQC8kyOu7474B8abvT5jH3weAQAAeA7CSAAAAAAAAAAuQRgJAAAAt6HXIgAAQP5CGAkgW/yBCACAd+G2dQAA4G6EkfBIfEnOu/gjBwBwqzz9s8STawMAAPB0hJEAbPAHFgDAW/EZBwAA4H6EkUAe5mk9RzypFgCAe+T2syC3n2m3sj9HbzMvy4/HDAAA3IMwEgAAAAAAAIBLEEYiT/O0noEAAHgKV38+2vOZnBc/u/NizQAAAJ6IMBIeiy/9AADAG/CdBgAA4H88OowcPny4LBaLzaNKlSrW9ZcvX1ZcXJyKFy+uwMBAderUScePH3djxQAAAHA2R90ZQUgIAADgeh4dRkpS9erVdezYMetj7dq11nWDBg3SN998ozlz5mj16tU6evSoOnbs6MZqAQAAAAAAAGTH190F3Iyvr68iIiIyLT979qw++ugjffbZZ2rWrJkkaerUqapatao2bNigu+++29WlwsXozZC16PiFOjCm7S1vAwAA5IwjPnvt3Z8kl+4TAADAUTy+Z+Rvv/2myMhIlS9fXl27dtWhQ4ckSVu3btWVK1fUokULa9sqVaqobNmyWr9+/Q23mZKSouTkZJsH3I8AzHMxURAAwFWc/Xnjis80PjMBAACy59FhZIMGDTRt2jQtXrxYkydP1v79+9W4cWOdO3dOiYmJ8vPzU2hoqM1rwsPDlZiYeMPtjh49WiEhIdZHmTJlnHgUAAAAcKasAsb8Egjml+MEAADew6PDyNatW+vhhx9WrVq1FBsbq2+//VZnzpzR7Nmzb2m7CQkJOnv2rPVx+PBhB1UMeIas/jBxZE8QekoCANyBzx7X4DwDAABn8ugw8nqhoaGqVKmS9u3bp4iICKWmpurMmTM2bY4fP57lGJPX8vf3V3BwsM0DuBZhGwAA+QOf+bnDOQMAALnl8RPYXOv8+fP6/fff9cQTT6hevXoqWLCgli9frk6dOkmS9uzZo0OHDikmJsbNlQKud6M/Ctz5BwN/rAAA8gp7P7Oc8Rl3K5PhXFtPbrbBZzYAAHAFjw4jn332WbVr105RUVE6evSohg0bJh8fH3Xp0kUhISHq1auXBg8erGLFiik4OFjPPPOMYmJimEk7j3D1zJO4Of4IAQB4Ov6BzX2YxRsAADiCR9+m/eeff6pLly6qXLmyHnnkERUvXlwbNmxQyZIlJUlvvvmmHnjgAXXq1En33nuvIiIi9OWXX7q5asCzeUKvDwAA8D/ZfdZmLM/NreQ3a+/Iz3d7tuWstgAAIO/w6J6Rs2bNuuH6QoUKadKkSZo0aZKLKgIAAAAAAACQWx4dRgLImVvpOUCvAwAAsuaIz8ib9Xp0RQ0Z23Hl7dXc0g0AALLj0bdpA7md4ZKADQCA/MlV3x2ub+/ttx/ntmZ7QldntAUAAJ6HnpFAPsWXeABAfuKKzz1H9mJ0x37tdau9PgEAQP5Ez0jkC678UuzIf63nX/4BAMi5vPCZydAqnoPzCQCAe9AzEnnGtV8Ys/ry6KoeD94w9hFfvgEAeQWfWdnLyblx5aza9nD1mJLZfYdjbEsAAFyPMBJe4UZfpLMLLu350nl9EMoXVgAAkNc5Ooh0123qAAAgb+E2bbgVXzZzz9WzcAIAgLzlVibZyWjvad8jbvWYAACA+9EzEvnWtbflOOIWHXpMAgCADM4KyQjf7MP5AuBuDAcBZEYYCa+V3b+c5+UPAW84BgAAkJknh2aO7o1I70YAAPI3btMGAAAAAAAA4BL0jASuY8/MlNfe4g0AAJCdvPZ9IS/Vy50jjsGQQwAAV6FnJHANZ3/xvpXtXz+jd07a56U/JAAAgHfK+E6S1fcSe76r3GpbvhcBAOAZCCOR7zjrS29uXsuXYgAAPBOf0Z7DEUGmp3HUPxrn5XMAAMi/CCMBAAAAeAQmtwEAwPsRRgJOxBdoAACAvMHTv7d5en0AAOQUE9jA5Tzti5Qj67F38psbvZYBxAEAQF7hKd/vclqHq+rNr5PCMKkQkDv59ZqB/IeekYCLOHv8SQAAADiGo7+3MbFg3sbPLjNvf097+/EB7kYYCdwCPqAAAAA8E9/TnIuwBgCQW4SRAAAAAAAAAFyCMBJws+z+VZl/aQYAAMg5Zw+Jc6OegPbs2xO+492sBk+oEfAk9vQEptcwcHOEkcBN3OoHCR9GAAAAeUNW39tc/V3OHd8dPeW7qqfUAcdyxHvane+N3O6b9zOux3vifwgj4VL88gEAAMBdPC0UydhWTuty5YSI3vgP6s46phsF2NeuyyvnM6/UmRfkh3N5o2PMD8eP3CGMBNyECzMAAEDex+zZzsW5zH/cfUu0N77fvPGYkLcRRgIAAADAdW4Wcjg7JMvN9j09uLOnJ6g7jsPTz19OuGNYAWdvP6//TG7GVXMI5MVzmRdrRs4QRgIAAAAAnCa725azeu4pHDVhUU72k1+5czxWd513R+3XHePKuvsfZ+BdCCMBAAAAII/JzR/njgoMXBUM2BNc5peA0FUB3q3sx1U9/RzJFb2Qb+V31l29XT35Z4a8jTASAAAAAOzgrD/QHRUwXn87tDtv2/W2yS3snXQop9v09IDxVvZ5o+e5bZvd6739tl5P6N3pKHm9ftwawkgAAAAAuEWe1hvMkT0nPTU0uFH45IljUt6MMydDupVejva0cVevuhudO1eMa+nK191sm/YeszPryIvyat15DWEkAAAAAAAAAJcgjITL8C8MAAAAwK1xV4+jvNzTSXLfOJcZy3Jza21enezkVjjqtn4mU/mfvHouXNGjNa8NteBNfN1dALwfv5QAAACAd34vdvQx5XR72YV+B8a0zdTm2mW3ul9HvS43+7n+2HJyXDfbpvS/85PdseRkX64Oe2/12LPapqcPUXCzoNaec3Ir7/dbfS84c9/IOwgj4VSedPEGAAAA8hpPmizHE/fhKrkJWJzxmpyyJ7jKqyGPu3oI2/M8J9uw57Xu4MoxN3MadNr7fr2VEPVGr82rvzuegDASAAAAAJClW53h2BPdygzP7uTIWjzp55rbyZZcuT9ncWQgfSu9ip21r2vbOjq0y2lImFXP15zW4knvFW9DGAkAAAAAeVxe/qP5VnqT5WfXhyuu6u3qqNvD4TmyG+LAGfu50fPcbi+r96Q94Wxuelrmti78jTASAAAAANyEYAZwPn7Pbs4RPSvdNVGUveGfvXUSLjoeYSSchgs+AAAAAEdy5OQY7vp7JS/+nZQXa84PPCEYzClX3EbuivEt7R17kiAza4SRAAAAAACv5WmhjDPkh2N0NGeOwYn8g5997hRwdwEAAAAAAAAA8gfCSAAAAAAAkCP0BIOz5IX3Vl4YCiIv4DZtAAAAAAAA5EuOmuWb8DHn6BkJAAAAAAAAwCUII+Ew/CsAAAAAAAAAboTbtOFwhJIAAAAAAAB/IyexRc9IAAAAAAAAAC5Bz0jcEkcN9AoAAAAAAADv5zU9IydNmqTo6GgVKlRIDRo00KZNm9xdkleLjl9I8AgAAAAAAAC7eEUY+cUXX2jw4MEaNmyYfvzxR9WuXVuxsbFKSkpyd2l5TlYh47XPCSABAAAAAACQW14RRr7xxhvq3bu3evbsqWrVqmnKlCkKCAjQxx9/7O7SAAAAAAAAAPy/PD9mZGpqqrZu3aqEhATrsgIFCqhFixZav359lq9JSUlRSkqK9fnZs2clScnJyc4t1o1qDFuiX0bE3vD5tcoOmpPt8+vXAQAAAAAA4Ma8OXeS/nd8xpgbtsvzYeTJkyeVlpam8PBwm+Xh4eH69ddfs3zN6NGjNWLEiEzLy5Qp45QaPUXIhBs/BwAAAAAAgHPklxzm3LlzCgkJyXZ9ng8jcyMhIUGDBw+2Pk9PT9epU6dUvHhxWSwWN1bmeMnJySpTpowOHz6s4OBgd5cDIA/h+gEgt7h+ALgVXEMA5BbXD/cyxujcuXOKjIy8Ybs8H0aWKFFCPj4+On78uM3y48ePKyIiIsvX+Pv7y9/f32ZZaGios0r0CMHBwfwiAsgVrh8AcovrB4BbwTUEQG5x/XCfG/WIzJDnJ7Dx8/NTvXr1tHz5cuuy9PR0LV++XDExMW6sDAAAAAAAAMC18nzPSEkaPHiwunfvrvr16+uuu+7ShAkTdOHCBfXs2dPdpQEAAAAAAAD4f14RRj766KM6ceKEhg4dqsTERNWpU0eLFy/ONKlNfuTv769hw4Zlui0dAG6G6weA3OL6AeBWcA0BkFtcP/IGi7nZfNsAAAAAAAAA4AB5fsxIAAAAAAAAAHkDYSQAAAAAAAAAlyCMBAAAAAAAAOAShJEAAAAAAAAAXIIw0otNmjRJ0dHRKlSokBo0aKBNmza5uyQAbjZ69GjdeeedCgoKUlhYmDp06KA9e/bYtLl8+bLi4uJUvHhxBQYGqlOnTjp+/LhNm0OHDqlt27YKCAhQWFiYnnvuOV29etWVhwLAzcaMGSOLxaKBAwdal3H9AJCdI0eO6PHHH1fx4sVVuHBh1axZU1u2bLGuN8Zo6NChKlWqlAoXLqwWLVrot99+s9nGqVOn1LVrVwUHBys0NFS9evXS+fPnXX0oAFwsLS1NL7/8ssqVK6fChQurQoUKeuWVV3TtfMxcQ/IWwkgv9cUXX2jw4MEaNmyYfvzxR9WuXVuxsbFKSkpyd2kA3Gj16tWKi4vThg0btHTpUl25ckUtW7bUhQsXrG0GDRqkb775RnPmzNHq1at19OhRdezY0bo+LS1Nbdu2VWpqqn744QdNnz5d06ZN09ChQ91xSADcYPPmzXrvvfdUq1Ytm+VcPwBk5fTp02rYsKEKFiyoRYsWadeuXRo/fryKFi1qbTNu3DhNnDhRU6ZM0caNG1WkSBHFxsbq8uXL1jZdu3bVzp07tXTpUi1YsEBr1qxRnz593HFIAFxo7Nixmjx5st555x3t3r1bY8eO1bhx4/T2229b23ANyWMMvNJdd91l4uLirM/T0tJMZGSkGT16tBurAuBpkpKSjCSzevVqY4wxZ86cMQULFjRz5syxttm9e7eRZNavX2+MMebbb781BQoUMImJidY2kydPNsHBwSYlJcW1BwDA5c6dO2duv/12s3TpUtOkSRMzYMAAYwzXDwDZe+GFF0yjRo2yXZ+enm4iIiLMf/7zH+uyM2fOGH9/f/P5558bY4zZtWuXkWQ2b95sbbNo0SJjsVjMkSNHnFc8ALdr27at+ec//2mzrGPHjqZr167GGK4heRE9I71Qamqqtm7dqhYtWliXFShQQC1atND69evdWBkAT3P27FlJUrFixSRJW7du1ZUrV2yuH1WqVFHZsmWt14/169erZs2aCg8Pt7aJjY1VcnKydu7c6cLqAbhDXFyc2rZta3OdkLh+AMje/PnzVb9+fT388MMKCwtT3bp19cEHH1jX79+/X4mJiTbXj5CQEDVo0MDm+hEaGqr69etb27Ro0UIFChTQxo0bXXcwAFzunnvu0fLly7V3715J0vbt27V27Vq1bt1aEteQvMjX3QXA8U6ePKm0tDSbL/qSFB4erl9//dVNVQHwNOnp6Ro4cKAaNmyoGjVqSJISExPl5+en0NBQm7bh4eFKTEy0tsnq+pKxDoD3mjVrln788Udt3rw50zquHwCy88cff2jy5MkaPHiw/v3vf2vz5s3q37+//Pz81L17d+vvf1bXh2uvH2FhYTbrfX19VaxYMa4fgJeLj49XcnKyqlSpIh8fH6Wlpem1115T165dJYlrSB5EGAkA+VRcXJx++eUXrV271t2lAMgDDh8+rAEDBmjp0qUqVKiQu8sBkIekp6erfv36GjVqlCSpbt26+uWXXzRlyhR1797dzdUB8HSzZ8/WzJkz9dlnn6l69eratm2bBg4cqMjISK4heRS3aXuhEiVKyMfHJ9PslcePH1dERISbqgLgSfr166cFCxZo5cqVKl26tHV5RESEUlNTdebMGZv2114/IiIisry+ZKwD4J22bt2qpKQk3XHHHfL19ZWvr69Wr16tiRMnytfXV+Hh4Vw/AGSpVKlSqlatms2yqlWr6tChQ5L+9/t/o79fIiIiMk3GefXqVZ06dYrrB+DlnnvuOcXHx6tz586qWbOmnnjiCQ0aNEijR4+WxDUkLyKM9EJ+fn6qV6+eli9fbl2Wnp6u5cuXKyYmxo2VAXA3Y4z69eunr776SitWrFC5cuVs1terV08FCxa0uX7s2bNHhw4dsl4/YmJi9PPPP9t8mC9dulTBwcGZ/tAA4D2aN2+un3/+Wdu2bbM+6tevr65du1r/n+sHgKw0bNhQe/bssVm2d+9eRUVFSZLKlSuniIgIm+tHcnKyNm7caHP9OHPmjLZu3Wpts2LFCqWnp6tBgwYuOAoA7nLx4kUVKGAbX/n4+Cg9PV0S15A8yd0z6MA5Zs2aZfz9/c20adPMrl27TJ8+fUxoaKjN7JUA8p++ffuakJAQs2rVKnPs2DHr4+LFi9Y2Tz/9tClbtqxZsWKF2bJli4mJiTExMTHW9VevXjU1atQwLVu2NNu2bTOLFy82JUuWNAkJCe44JABudO1s2sZw/QCQtU2bNhlfX1/z2muvmd9++83MnDnTBAQEmE8//dTaZsyYMSY0NNR8/fXXZseOHebBBx805cqVM5cuXbK2adWqlalbt67ZuHGjWbt2rbn99ttNly5d3HFIAFyoe/fu5rbbbjMLFiww+/fvN19++aUpUaKEef75561tuIbkLYSRXuztt982ZcuWNX5+fuauu+4yGzZscHdJANxMUpaPqVOnWttcunTJ/Otf/zJFixY1AQEB5qGHHjLHjh2z2c6BAwdM69atTeHChU2JEiXMkCFDzJUrV1x8NADc7fowkusHgOx88803pkaNGsbf399UqVLFvP/++zbr09PTzcsvv2zCw8ONv7+/ad68udmzZ49Nm7/++st06dLFBAYGmuDgYNOzZ09z7tw5Vx4GADdITk42AwYMMGXLljWFChUy5cuXNy+++KJJSUmxtuEakrdYjDHGnT0zAQAAAAAAAOQPjBkJAAAAAAAAwCUIIwEAAAAAAAC4BGEkAAAAAAAAAJcgjAQAAAAAAADgEoSRAAAAAAAAAFyCMBIAAAAAAACASxBGAgAAAAAAAHAJwkgAAAAAAAAALkEYCQAAgHzNYrFo3rx5dr9uz549ioiI0Llz525p/9OmTVNoaOgtbeNWLF68WHXq1FF6errbagAAAPkHYSQAAMAtOnHihPr27auyZcvK399fERERio2N1bp169xdmsfIbeDnSMOHD1edOnUctr2EhAQ988wzCgoKkuT+UDG3WrVqpYIFC2rmzJnuLgUAAOQDhJEAAAC3qFOnTvrpp580ffp07d27V/Pnz1fTpk31119/ubs0OMmhQ4e0YMEC9ejRw92lOESPHj00ceJEd5cBAADyAcJIAACAW3DmzBl9//33Gjt2rO677z5FRUXprrvuUkJCgtq3b2/T7sknn1TJkiUVHBysZs2aafv27TbbGjNmjMLDwxUUFKRevXopPj7epidf06ZNNXDgQJvXdOjQwSYQS0lJ0bPPPqvbbrtNRYoUUYMGDbRq1Srr+ozee0uWLFHVqlUVGBioVq1a6dixYzbb/fjjj1W9enX5+/urVKlS6tevn13HYq8PP/xQVatWVaFChVSlShW9++671nUHDhyQxWLRl19+qfvuu08BAQGqXbu21q9fb7ONDz74QGXKlFFAQIAeeughvfHGG9aeitOmTdOIESO0fft2WSwWWSwWTZs2zfrakydP6qGHHlJAQIBuv/12zZ8//4b1zp49W7Vr19Ztt90mSVq1apV69uyps2fPWrc/fPhwSdLp06fVrVs3FS1aVAEBAWrdurV+++23bLd94sQJ1a9fXw899JBSUlKUnp6u0aNHq1y5cipcuLBq166tuXPnWtuvWrVKFotFy5cvV/369RUQEKB77rlHe/bssbbZvn277rvvPgUFBSk4OFj16tXTli1brOvbtWunLVu26Pfff7/hcQMAANwqwkgAAIBbEBgYqMDAQM2bN08pKSnZtnv44YeVlJSkRYsWaevWrbrjjjvUvHlznTp1StLf4dbw4cM1atQobdmyRaVKlbIJ5HKqX79+Wr9+vWbNmqUdO3bo4YcfVqtWrWzCr4sXL+r111/XjBkztGbNGh06dEjPPvusdf3kyZMVFxenPn366Oeff9b8+fNVsWLFHB+LvWbOnKmhQ4fqtdde0+7duzVq1Ci9/PLLmj59uk27F198Uc8++6y2bdumSpUqqUuXLrp69aokad26dXr66ac1YMAAbdu2Tffff79ee+0162sfffRRDRkyRNWrV9exY8d07NgxPfroo9b1I0aM0COPPKIdO3aoTZs26tq16w2P5/vvv1f9+vWtz++55x5NmDBBwcHB1u1nnNMePXpoy5Ytmj9/vtavXy9jjNq0aaMrV65k2u7hw4fVuHFj1ahRQ3PnzpW/v79Gjx6tTz75RFOmTNHOnTs1aNAgPf7441q9enWm8zN+/Hht2bJFvr6++uc//2ld17VrV5UuXVqbN2/W1q1bFR8fr4IFC1rXly1bVuHh4fr+++9v+LMCAAC4ZQYAAAC3ZO7cuaZo0aKmUKFC5p577jEJCQlm+/bt1vXff/+9CQ4ONpcvX7Z5XYUKFcx7771njDEmJibG/Otf/7JZ36BBA1O7dm3r8yZNmpgBAwbYtHnwwQdN9+7djTHGHDx40Pj4+JgjR47YtGnevLlJSEgwxhgzdepUI8ns27fPun7SpEkmPDzc+jwyMtK8+OKLWR5rTo4lK5LMV199leW6ChUqmM8++8xm2SuvvGJiYmKMMcbs37/fSDIffvihdf3OnTuNJLN7925jjDGPPvqoadu2rc02unbtakJCQqzPhw0bZnM+r63tpZdesj4/f/68kWQWLVqU7fHUrl3bjBw50mbZ1KlTbfZnjDF79+41ksy6deusy06ePGkKFy5sZs+ebfO6X3/91ZQpU8b079/fpKenG2OMuXz5sgkICDA//PCDzXZ79eplunTpYowxZuXKlUaSWbZsmXX9woULjSRz6dIlY4wxQUFBZtq0adkejzHG1K1b1wwfPvyGbQAAAG4VPSMBAABuUadOnXT06FHNnz9frVq10qpVq3THHXdYbwPevn27zp8/r+LFi1t7UgYGBmr//v3W22J3796tBg0a2Gw3JibGrjp+/vlnpaWlqVKlSjb7Wb16tc3ttwEBAapQoYL1ealSpZSUlCRJSkpK0tGjR9W8efMs95GTY7HHhQsX9Pvvv6tXr14223v11Vczba9WrVo2NWfUK/09s/Vdd91l0/765zdy7baLFCmi4OBg67azcunSJRUqVOim2929e7d8fX1tfrbFixdX5cqVtXv3bpvtNW7cWB07dtRbb70li8UiSdq3b58uXryo+++/3+b8fPLJJ3adn8GDB+vJJ59UixYtNGbMmCx/VoULF9bFixdvekwAAAC3wtfdBQAAAHiDQoUK6f7779f999+vl19+WU8++aSGDRumHj166Pz58ypVqpTN2I0Z7Jl9uUCBAjLG2Cy79lbf8+fPy8fHR1u3bpWPj49Nu8DAQOv/X3t7rvT3TNcZ2y1cuPANa3DUsVy7Penv8R6vD2OvP4Zr684I69LT0+3eZ1ayOic32naJEiV0+vRph+xbkvz9/dWiRQstWLBAzz33nHUsyozzs3DhQuuya19zrRudn+HDh+uxxx7TwoULtWjRIg0bNkyzZs3SQw89ZH3NqVOnVLJkSYcdEwAAQFYIIwEAAJygWrVqmjdvniTpjjvuUGJionx9fRUdHZ1l+6pVq2rjxo3q1q2bddmGDRts2pQsWdJmopm0tDT98ssvuu+++yRJdevWVVpampKSktS4ceNc1R0UFKTo6GgtX77cut1r5eRY7BEeHq7IyEj98ccf6tq1a663U7lyZW3evNlm2fXP/fz8lJaWlut9XKtu3bratWvXTbdftWpVXb16VRs3btQ999wjSfrrr7+0Z88eVatWzdquQIECmjFjhh577DHdd999WrVqlSIjI1WtWjX5+/vr0KFDatKkyS3VXKlSJVWqVEmDBg1Sly5dNHXqVGsYefnyZf3++++qW7fuLe0DAADgZggjAQAAbsFff/2lhx9+WP/85z9Vq1YtBQUFacuWLRo3bpwefPBBSVKLFi0UExOjDh06aNy4capUqZKOHj2qhQsX6qGHHlL9+vU1YMAA9ejRQ/Xr11fDhg01c+ZM7dy5U+XLl7fuq1mzZho8eLAWLlyoChUq6I033tCZM2es6ytVqqSuXbuqW7duGj9+vOrWrasTJ05o+fLlqlWrltq2bZujYxo+fLiefvpphYWFqXXr1jp37pzWrVunZ555JkfHkp39+/dr27ZtNstuv/12jRgxQv3791dISIhatWqllJQUbdmyRadPn9bgwYNzVPMzzzyje++9V2+88YbatWunFStWaNGiRdYegpIUHR1traF06dIKCgrK1Lswp2JjY/Xkk08qLS3N2oMzOjpa58+f1/Lly1W7dm3rzNwPPvigevfurffee09BQUGKj4/XbbfdZn1/ZPDx8dHMmTPVpUsXNWvWTKtWrVJERISeffZZDRo0SOnp6WrUqJHOnj2rdevWKTg4WN27d79prZcuXdJzzz2nf/zjHypXrpz+/PNPbd68WZ06dbK22bBhg/z9/e0eGgAAAMBu7h60EgAAIC+7fPmyiY+PN3fccYcJCQkxAQEBpnLlyuall14yFy9etLZLTk42zzzzjImMjDQFCxY0ZcqUMV27djWHDh2ytnnttddMiRIlTGBgoOnevbt5/vnnbSZcSU1NNX379jXFihUzYWFhZvTo0TYT2GS0GTp0qImOjjYFCxY0pUqVMg899JDZsWOHMSbrSVa++uorc/3XwilTppjKlStbt/HMM8/YdSzXk5Tl4/vvvzfGGDNz5kxTp04d4+fnZ4oWLWruvfde8+WXXxpj/jeBzU8//WTd3unTp40ks3LlSuuy999/39x2222mcOHCpkOHDubVV181ERERNj+rTp06mdDQUCPJTJ061Vrb9ZPrhISEWNdn5cqVKyYyMtIsXrzYZvnTTz9tihcvbiSZYcOGGWOMOXXqlHniiSdMSEiIKVy4sImNjTV79+61vub6n8mVK1dMx44dTdWqVc3x48dNenq6mTBhgvXnUbJkSRMbG2tWr15tjPnfBDanT5+2buOnn34yksz+/ftNSkqK6dy5sylTpozx8/MzkZGRpl+/ftbJbYwxpk+fPuapp57K9ngBAAAcxWLMdQMPAQAAwCMMHz5c8+bNy9SbEDnTu3dv/frrr/r++++dsv1JkyZp/vz5WrJkiVO27yonT55U5cqVtWXLFpUrV87d5QAAAC/HbdoAAADwCq+//rruv/9+FSlSRIsWLdL06dP17rvvOm1/Tz31lM6cOaNz584pKCjIaftxtgMHDujdd98liAQAAC5BGAkAAACvsGnTJo0bN07nzp1T+fLlNXHiRD355JNO25+vr69efPFFp23fVerXr3/DsT4BAAAcidu0AQAAAAAAALhEAXcXAAAAAAAAACB/IIwEAAAAAAAA4BKEkQAAAAAAAABcgjASAAAAAAAAgEsQRgIAAAAAAABwCcJIAAAAAAAAAC5BGAkAAAAAAADAJQgjAQAAAAAAALjE/wG/dy1CT1PlNwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1600x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "829.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualize class imbalance\n",
    "plt.bar(range(K), class_counts, tick_label=range(K))\n",
    "plt.xlabel('Class Index')\n",
    "plt.ylabel('Number of Samples')\n",
    "plt.title('Class Imbalance in the Dataset')\n",
    "plt.show()\n",
    "\n",
    "# Calculate the threshold for the top 95%\n",
    "threshold_length = np.percentile(sequence_lengths, 95)\n",
    "\n",
    "# Filter sequence lengths\n",
    "filtered_lengths = [length for length in sequence_lengths if length <= threshold_length]\n",
    "\n",
    "# Plot the distribution of sequence lengths for the top 90%\n",
    "plt.figure(figsize=(16,5))\n",
    "plt.hist(filtered_lengths, bins=range(min(filtered_lengths), max(filtered_lengths) + 1, 1))\n",
    "plt.xlabel('Sequence Length (tokens)')\n",
    "plt.ylabel('Number of Samples')\n",
    "plt.title('Distribution of Sequence Lengths (Top 90%)')\n",
    "plt.show()\n",
    "\n",
    "threshold_length\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as we can see the data is pretty balanced class wise, so no special preprocessing is required, \n",
    "and by the length distribution and hardware limitations we choose `MAX_LEN = 256` to be adequite for classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data tokenization\n",
    "\n",
    "with the found parameters, here we tokenize the dataset, one can also load saved tokenized datasets via `pickele`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T15:51:08.828292Z",
     "iopub.status.busy": "2024-02-02T15:51:08.827912Z",
     "iopub.status.idle": "2024-02-02T16:08:47.459722Z",
     "shell.execute_reply": "2024-02-02T16:08:47.458820Z",
     "shell.execute_reply.started": "2024-02-02T15:51:08.828262Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Tokenizing data: 100%|██████████| 71027/71027 [17:06<00:00, 69.21it/s] \n",
      "Tokenizing data: 100%|██████████| 3000/3000 [00:32<00:00, 92.83it/s] \n"
     ]
    }
   ],
   "source": [
    "MAX_LEN = 256\n",
    "\n",
    "token_opts = {\n",
    "    \"return_tensors\":'pt',\n",
    "    \"truncation\": True,\n",
    "    \"padding\": \"max_length\",\n",
    "    \"max_length\": MAX_LEN\n",
    "}\n",
    "\n",
    "if TOKENIZE:\n",
    "    full_dataset.tokenize_data(tokenizer, token_opts)\n",
    "    test_dataset.tokenize_data(tokenizer, token_opts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T16:09:28.370935Z",
     "iopub.status.busy": "2024-02-02T16:09:28.370016Z",
     "iopub.status.idle": "2024-02-02T16:09:41.577968Z",
     "shell.execute_reply": "2024-02-02T16:09:41.577058Z",
     "shell.execute_reply.started": "2024-02-02T16:09:28.370893Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open( \"full_dataset.p\", \"wb\" ) as f:\n",
    "    pickle.dump(full_dataset, f)\n",
    "with open( \"test_dataset.p\", \"wb\" ) as f:\n",
    "    pickle.dump(test_dataset, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:31:09.904918Z",
     "iopub.status.busy": "2024-02-03T15:31:09.904543Z",
     "iopub.status.idle": "2024-02-03T15:31:26.589759Z",
     "shell.execute_reply": "2024-02-03T15:31:26.588697Z",
     "shell.execute_reply.started": "2024-02-03T15:31:09.904889Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "MAX_LEN = 256\n",
    "\n",
    "token_opts = {\n",
    "    \"return_tensors\":'pt',\n",
    "    \"truncation\": True,\n",
    "    \"padding\": \"max_length\",\n",
    "    \"max_length\": MAX_LEN\n",
    "}\n",
    "\n",
    "with open( \"full_dataset.p\", \"rb\" ) as f:\n",
    "    full_dataset = pickle.load(f)\n",
    "with open( \"test_dataset.p\", \"rb\" ) as f:\n",
    "    test_dataset = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoaders\n",
    "\n",
    "here we define a simple function `get_loader` that helps us create dataloaders suitable for each task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:31:30.707133Z",
     "iopub.status.busy": "2024-02-03T15:31:30.706770Z",
     "iopub.status.idle": "2024-02-03T15:31:31.252530Z",
     "shell.execute_reply": "2024-02-03T15:31:31.251705Z",
     "shell.execute_reply.started": "2024-02-03T15:31:30.707102Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_loader(dataset, batch_size=24, shuffle=True):\n",
    "    return DataLoader(\n",
    "        dataset, batch_size=batch_size, shuffle=shuffle, num_workers=os.cpu_count(), pin_memory=False\n",
    "    )\n",
    "num_samples = len(full_dataset)\n",
    "train_size = int(0.95 * num_samples)\n",
    "val_size = num_samples - train_size\n",
    "\n",
    "# batch_size = 1 # online\n",
    "train_dataset, val_dataset = random_split(full_dataset, [train_size, val_size])\n",
    "train_loader = get_loader(train_dataset)\n",
    "test_loader  = get_loader(test_dataset)\n",
    "val_loader   = get_loader(val_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:31:33.751042Z",
     "iopub.status.busy": "2024-02-03T15:31:33.750697Z",
     "iopub.status.idle": "2024-02-03T15:31:33.758978Z",
     "shell.execute_reply": "2024-02-03T15:31:33.757929Z",
     "shell.execute_reply.started": "2024-02-03T15:31:33.751015Z"
    }
   },
   "outputs": [],
   "source": [
    "def plot_results(metrics_by_percentage):\n",
    "    plt.figure(figsize=(12, 6))\n",
    "\n",
    "    percentages = list(metrics_by_percentage.keys())\n",
    "    num_percentages = len(percentages)\n",
    "    \n",
    "    fig, axs = plt.subplots(1, 2, figsize=(12, 6))\n",
    "\n",
    "    losses = []\n",
    "    acc = []\n",
    "    \n",
    "    for i, (percentage, metrics) in enumerate(metrics_by_percentage.items(), 1):\n",
    "        acc.append(metrics['accuracy'])\n",
    "        losses.append(metrics['loss'])\n",
    "\n",
    "    axs[0].plot(percentages, losses)\n",
    "    axs[0].set_xlabel('Percentage')\n",
    "    axs[0].set_ylabel('Loss')\n",
    "    axs[0].set_title('Training Loss Over Percentages')\n",
    "\n",
    "    axs[1].plot(percentages, acc)\n",
    "    axs[1].set_xlabel('Percentage')\n",
    "    axs[1].set_ylabel('Accuracy')\n",
    "    axs[1].set_title('Test Accuracy Over Percentages')\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "# Example usage\n",
    "# plot_results(metrics_by_percentage)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bert with Linear probing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T14:52:51.924557Z",
     "iopub.status.busy": "2024-02-03T14:52:51.923834Z",
     "iopub.status.idle": "2024-02-03T14:52:52.936926Z",
     "shell.execute_reply": "2024-02-03T14:52:52.935938Z",
     "shell.execute_reply.started": "2024-02-03T14:52:51.924522Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 6.93e+05 || all params: 1.10e+08 || trainable: 0.63%\n",
      "Last hidden states:  torch.Size([1, 256, 768])\n"
     ]
    }
   ],
   "source": [
    "bert, tokenizer = get_bert(model=\"bert\")\n",
    "\n",
    "# Test\n",
    "text = \"Ali nourian save me\"\n",
    "tokens = tokenizer(text, **token_opts)\n",
    "outputs = bert(**tokens)\n",
    "last_hidden_states = outputs.last_hidden_state\n",
    "_, _, hidden_dim = last_hidden_states.shape\n",
    "\n",
    "print(\"Last hidden states: \", last_hidden_states.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T16:58:12.293623Z",
     "iopub.status.busy": "2024-02-02T16:58:12.292621Z",
     "iopub.status.idle": "2024-02-02T16:58:12.317252Z",
     "shell.execute_reply": "2024-02-02T16:58:12.316205Z",
     "shell.execute_reply.started": "2024-02-02T16:58:12.293586Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using 2 GPUs!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DataParallel(\n",
       "  (module): BertLinear(\n",
       "    (bert): BertModel(\n",
       "      (embeddings): BertEmbeddings(\n",
       "        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "        (position_embeddings): Embedding(512, 768)\n",
       "        (token_type_embeddings): Embedding(2, 768)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (encoder): BertEncoder(\n",
       "        (layer): ModuleList(\n",
       "          (0-11): 12 x BertLayer(\n",
       "            (attention): BertAttention(\n",
       "              (self): BertSelfAttention(\n",
       "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "              )\n",
       "              (output): BertSelfOutput(\n",
       "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "              )\n",
       "            )\n",
       "            (intermediate): BertIntermediate(\n",
       "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "              (intermediate_act_fn): GELUActivation()\n",
       "            )\n",
       "            (output): BertOutput(\n",
       "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (pooler): BertPooler(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (activation): Tanh()\n",
       "      )\n",
       "    )\n",
       "    (fc): Linear(in_features=768, out_features=6, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class BertLinear(nn.Module):\n",
    "    def __init__(self, bert_model, num_labels, has_pooler=False):\n",
    "        super(BertLinear, self).__init__()\n",
    "        self.bert = bert_model\n",
    "        self.fc = nn.Linear(bert_model.config.hidden_size, num_labels)\n",
    "        self.has_pooler = has_pooler\n",
    "\n",
    "    def forward(self, input_ids, attention_mask):\n",
    "        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)\n",
    "        h = outputs.pooler_output if self.has_pooler else outputs.last_hidden_state[:, 0]\n",
    "        logits = self.fc(h)\n",
    "        return logits\n",
    "    \n",
    "bert_model = BertLinear(bert, K)\n",
    "    \n",
    "if torch.cuda.device_count() > 1:\n",
    "    print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "    bert_model = nn.DataParallel(bert_model)\n",
    "    \n",
    "bert_model = bert_model.to(device)\n",
    "    \n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(bert_model.parameters(), lr=2e-5)\n",
    "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)\n",
    "\n",
    "bert_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir linear_probe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-02T17:03:10.419577Z",
     "iopub.status.busy": "2024-02-02T17:03:10.419193Z",
     "iopub.status.idle": "2024-02-02T17:03:12.790245Z",
     "shell.execute_reply": "2024-02-02T17:03:12.789190Z",
     "shell.execute_reply.started": "2024-02-02T17:03:10.419545Z"
    }
   },
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "# Define the percentages to use\n",
    "train_percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "\n",
    "num_epochs = 3 # Same as the paper\n",
    "print_every = 1\n",
    "\n",
    "bert_metrics_by_percentage = {}\n",
    "\n",
    "for train_percentage in train_percentages:\n",
    "    print(f\"\"\"\n",
    "=================================================================================================\n",
    "\n",
    "                                Training on {100 * train_percentage} % of the dataset\n",
    "    \n",
    "=================================================================================================\n",
    "\n",
    "    \"\"\")\n",
    "    \n",
    "    # Creating model\n",
    "    bert_model = BertLinear(bert, K)\n",
    "    \n",
    "    if torch.cuda.device_count() > 1:\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bert_model = nn.DataParallel(bert_model)\n",
    "\n",
    "    bert_model = bert_model.to(device)\n",
    "\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.Adam(bert_model.parameters(), lr=2e-5)\n",
    "    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)\n",
    "\n",
    "    num_train_batches = int(len(train_loader) * train_percentage)\n",
    "    \n",
    "    metrics = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': []}\n",
    "    \n",
    "\n",
    "    # batch_size = 1 # online\n",
    "    train_size = len(train_dataset)\n",
    "    num_train_batches = int(train_size * train_percentage)\n",
    "    train_loader_percented, _ = random_split(train_dataset, [num_train_batches, train_size - num_train_batches])\n",
    "    train_loader_percented = get_loader(train_loader_percented)\n",
    "    \n",
    "    # Train the model\n",
    "    for epoch in range(num_epochs):\n",
    "        bert_model.train()\n",
    "\n",
    "        # Training\n",
    "        train_loss = 0.0\n",
    "        correct_train = 0\n",
    "        total_train = 0\n",
    "        for inputs, labels, token in tqdm(train_loader_percented, desc=f'Train - epoch {epoch + 1}/{num_epochs}'):\n",
    "            token, labels = token.to(device), labels.to(device)\n",
    "            optimizer.zero_grad()\n",
    "            outputs = bert_model(**token)\n",
    "            loss = criterion(outputs, labels)\n",
    "            \n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            train_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs, 1)\n",
    "            total_train += labels.size(0)\n",
    "            correct_train += (predicted == labels).sum().item()\n",
    "            \n",
    "\n",
    "        train_acc = correct_train / total_train\n",
    "\n",
    "        # Validation\n",
    "        bert_model.eval()\n",
    "        val_loss = 0.0\n",
    "        total_val = 0\n",
    "        correct_val = 0\n",
    "        with torch.no_grad():\n",
    "            for inputs, labels, token in tqdm(val_loader, desc=f'Validation - epoch {epoch + 1}/{num_epochs}'):\n",
    "                token, labels = token.to(device), labels.to(device)\n",
    "                outputs = bert_model(**token)\n",
    "                loss = criterion(outputs, labels)\n",
    "\n",
    "                val_loss += loss.item()\n",
    "\n",
    "                # Calculate accuracy\n",
    "                _, predicted = torch.max(outputs, 1)\n",
    "                total_val += labels.size(0)\n",
    "                correct_val += (predicted == labels).sum().item()\n",
    "\n",
    "        val_acc = correct_val / total_val\n",
    "\n",
    "        metrics['train_loss'].append(train_loss / num_train_batches)\n",
    "        metrics['val_loss'].append(val_loss / len(val_loader))\n",
    "        metrics['train_acc'].append(train_acc)\n",
    "        metrics['val_acc'].append(val_acc)\n",
    "\n",
    "        scheduler.step()\n",
    "\n",
    "        if (epoch + 1) % print_every == 0:\n",
    "            print(f\"\"\"\n",
    "            Training loss: {metrics['train_loss'][-1]:.4f}, Validation loss: {metrics['val_loss'][-1]:.4f},\n",
    "            Training accuracy: {metrics['train_acc'][-1] * 100:.2f}%, Validation accuracy: {metrics['val_acc'][-1] * 100:.2f}%\n",
    "            \"\"\")\n",
    "#                 Model is too large, we won't save it\n",
    "    # Save model checkpoint\n",
    "    checkpoint = {\n",
    "        'epoch': epoch + 1,\n",
    "        'model_state_dict': bert_model.state_dict(),\n",
    "        'optimizer_state_dict': optimizer.state_dict(),\n",
    "        'scheduler_state_dict': scheduler.state_dict(),\n",
    "        'metrics': metrics,\n",
    "    }\n",
    "    torch.save(checkpoint, f'./linear_probe/model_checkpoint_{int(train_percentage * 100)}%.pth')\n",
    "    # Save metrics for the current percentage\n",
    "    bert_metrics_by_percentage[f\"{int(train_percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "percentages = list(bert_metrics_by_percentage.keys())\n",
    "\n",
    "fig, axs = plt.subplots(1, 2, figsize=(12, 5))\n",
    "\n",
    "# Plot training and validation loss\n",
    "axs[0].plot(percentages, [metrics['train_loss'][-1] for metrics in bert_metrics_by_percentage.values()], label='Train Loss')\n",
    "axs[0].plot(percentages, [metrics['val_loss'][-1] for metrics in bert_metrics_by_percentage.values()], label='Validation Loss')\n",
    "axs[0].set_title(\"Loss as a function of dataset percentage\")\n",
    "axs[0].set_xlabel('Percentage of the dataset')\n",
    "axs[0].set_ylabel('Loss')\n",
    "axs[0].legend()\n",
    "\n",
    "# Plot training and validation accuracy\n",
    "axs[1].plot(percentages, [metrics['train_acc'][-1] * 100 for metrics in bert_metrics_by_percentage.values()], label='Train Accuracy')\n",
    "axs[1].plot(percentages, [metrics['val_acc'][-1] * 100 for metrics in bert_metrics_by_percentage.values()], label='Validation Accuracy')\n",
    "axs[1].set_title(\"Accuracy as a function of dataset percentage\")\n",
    "axs[1].set_xlabel('Percentage of the dataset')\n",
    "axs[1].set_ylabel('Accuracy (%)')\n",
    "axs[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "metrics_by_percentage = {}\n",
    "for percentage in percentages:\n",
    "    # Define model\n",
    "    bert_model = BertLinear(bert, K)\n",
    "    \n",
    "    if torch.cuda.device_count() > 1:\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bert_model = nn.DataParallel(bert_model)\n",
    "\n",
    "    bert_model = bert_model.to(device)\n",
    "\n",
    "    metrics = {'loss': [], 'accuracy': []}\n",
    "\n",
    "    # Load model\n",
    "    path = f'./linear_probe/model_checkpoint_{int(percentage * 100)}%.pth'\n",
    "    checkpoint = torch.load(path)\n",
    "    bert_state_dict = checkpoint['model_state_dict']\n",
    "    bert_model.load_state_dict(bert_state_dict)\n",
    "    # Testing\n",
    "    bert_model.eval()\n",
    "    test_loss = 0.0\n",
    "    all_predicts_test = np.array([])\n",
    "    all_labels_test = np.array([])\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels, token in tqdm(test_loader, desc=f'Testing'):\n",
    "            token, labels = token.to(device), labels.to(device)\n",
    "            outputs = bert_model(**token)\n",
    "            loss = criterion(outputs, labels)\n",
    "\n",
    "            test_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs, 1)\n",
    "            all_predicts_test = np.append(all_predicts_test, predicted.detach().cpu().numpy())\n",
    "            all_labels_test = np.append(all_labels_test, labels.detach().cpu().numpy())\n",
    "\n",
    "    test_acc = (all_predicts_test == all_labels_test).sum().item() / len(all_predicts_test)\n",
    "\n",
    "    metrics['loss'].append(test_loss / len(test_loader))\n",
    "    metrics['accuracy'].append(test_acc)\n",
    "    \n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_results(metrics_by_percentage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using adapters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer, BertModel, DistilBertTokenizer, DistilBertModel\n",
    "# add adapters to BERT for classification task\n",
    "from adapters import AutoAdapterModel\n",
    "\n",
    "\n",
    "def print_trainable_parameters(model):\n",
    "    \"\"\"\n",
    "    Prints the number of trainable parameters in the model.\n",
    "    \"\"\"\n",
    "    trainable_params = 0\n",
    "    all_param = 0\n",
    "    for _, param in model.named_parameters():\n",
    "        all_param += param.numel()\n",
    "        if param.requires_grad:\n",
    "            trainable_params += param.numel()\n",
    "    print(\n",
    "        f\"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}\"\n",
    "    )\n",
    "\n",
    "def get_bert(model=\"bert\", use_peft=False, print_params=False, peft_method=\"lora\", num_labels=None):\n",
    "    # Load pre-trained BERT\n",
    "    if num_labels == None:\n",
    "        if model == \"bert\":\n",
    "            tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "            bert = BertModel.from_pretrained('bert-base-uncased')\n",
    "        elif model == \"distil\":\n",
    "            tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
    "            bert = DistilBertModel.from_pretrained('distilbert-base-uncased')\n",
    "        else:\n",
    "            raise ValueError(\"Invalid bert model\")\n",
    "\n",
    "        # Peft\n",
    "        if use_peft:\n",
    "            if peft_method == \"lora\":\n",
    "                from peft import LoraConfig, LoraModel\n",
    "                # Let's define the LoraConfig\n",
    "                config = LoraConfig(\n",
    "                    r=16, lora_alpha=16, lora_dropout=0.01,\n",
    "                    bias=\"all\",  task_type=\"SEQ_CLS\", #target_modules=[\"q\", \"v\"],,\n",
    "                )\n",
    "                bert = LoraModel(bert, config, \"default\")\n",
    "            elif peft_method == \"adalora\":\n",
    "                from peft import AdaLoraModel, AdaLoraConfig\n",
    "                config = AdaLoraConfig(\n",
    "                    peft_type=\"ADALORA\", task_type=\"SEQ_CLS\", r=8, lora_alpha=32,# target_modules=[\"q\", \"v\"],\n",
    "                    lora_dropout=0.01, adapter_name=\"default\",\n",
    "                )\n",
    "                bert = AdaLoraModel(bert, config, \"default\")\n",
    "            elif peft_method == \"IA3\":\n",
    "                from peft import IA3Model, IA3Config\n",
    "                config = IA3Config(\n",
    "                    peft_type=\"IA3\", task_type=\"SEQ_CLS\",\n",
    "    #                 target_modules=[\"k\", \"v\", \"w0\"], feedforward_modules=[\"w0\"],\n",
    "                )\n",
    "                bert = IA3Model(bert, config, \"default\")\n",
    "    else:\n",
    "        # load pre-trained BERT model\n",
    "        tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "        bert = AutoAdapterModel.from_pretrained(\"bert-base-uncased\")\n",
    "        # By default, this model doesn’t have any heads yet. We add a new one in the next step\n",
    "        bert.add_classification_head(\"mrpc\", num_labels = num_labels)\n",
    "        # add a new adapter with proper config \n",
    "        bert.add_adapter(\"mrpc\", config=\"double_seq_bn\")\n",
    "        bert.set_active_adapters(\"mrpc\")\n",
    "        # Enable adapter training\n",
    "        bert.train_adapter(\"mrpc\")\n",
    "        bert.set_active_adapters(\"mrpc\")\n",
    "    if print_params:\n",
    "        print_trainable_parameters(bert)\n",
    "    \n",
    "    return bert, tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model = get_bert(num_labels=K)\n",
    "print_trainable_parameters(bert_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if torch.cuda.device_count() > 1:\n",
    "    print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "    bert_model = nn.DataParallel(bert_model)\n",
    "    \n",
    "bert_model = bert_model.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(bert_model.parameters(), lr=1e-4) # change lr\n",
    "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1) #CosineAnnealingLR\n",
    "\n",
    "bert_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "import time\n",
    "# Define the percentages to use\n",
    "train_percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "\n",
    "num_epochs = 3 # Same as the paper + 1 (adapters need more epoches)\n",
    "print_every = 1\n",
    "\n",
    "#bert_model = bert_model.to(device)\n",
    "bert_metrics_by_percentage = {}\n",
    "\n",
    "for train_percentage in train_percentages:\n",
    "    print(f\"\"\"\n",
    "=================================================================================================\n",
    "\n",
    "                                Training on {100 * train_percentage} % of the dataset\n",
    "    \n",
    "=================================================================================================\n",
    "\n",
    "    \"\"\")\n",
    "    \n",
    "    num_train_batches = int(len(train_loader) * train_percentage)\n",
    "    \n",
    "    metrics = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': []}\n",
    "    bert_model = get_bert(num_labels=K)\n",
    "    if torch.cuda.device_count() > 1:\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bert_model = nn.DataParallel(bert_model)\n",
    "    bert_model = bert_model.to(device)\n",
    "    \n",
    "    # Train the model\n",
    "    for epoch in range(num_epochs):\n",
    "        \n",
    "        # Freeze all parameters in BERT and only fine-tune the adapter's parameters\n",
    "        bert_model.train_adapter(\"mrpc\")\n",
    "        \n",
    "\n",
    "        # Training\n",
    "        train_loss = 0.0\n",
    "        correct_train = 0\n",
    "        total_train = 0\n",
    "        batch_idx = 0\n",
    "        for inputs, labels, token in tqdm(train_loader, desc=f'Train - epoch {epoch + 1}/{num_epochs}'):\n",
    "            start_time = time.time()\n",
    "            batch_idx += 1\n",
    "            if batch_idx >= num_train_batches:\n",
    "                break\n",
    "            encoded = tokenizer(inputs, **token_opts) if not TOKENIZE else token\n",
    "            encoded.pop('token_type_ids', None)\n",
    "            encoded, labels = encoded.to(device), labels.to(device)\n",
    "            encoded = dict([(k, v.squeeze(dim=1)) for (k, v) in encoded.items()])\n",
    "            optimizer.zero_grad()\n",
    "            outputs = bert_model(**encoded)\n",
    "            loss = criterion(outputs.logits, labels)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            train_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs.logits, 1)\n",
    "            total_train += labels.size(0)\n",
    "            correct_train += (predicted == labels).sum().item()\n",
    "\n",
    "        train_acc = correct_train / total_train\n",
    "\n",
    "        # Validation\n",
    "        bert_model.eval()\n",
    "        val_loss = 0.0\n",
    "        total_val = 0\n",
    "        correct_val = 0\n",
    "        with torch.no_grad():\n",
    "            for inputs, labels, token in tqdm(val_loader, desc=f'Validation - epoch {epoch + 1}/{num_epochs}'):\n",
    "                encoded = tokenizer(inputs, **token_opts) if not TOKENIZE else token\n",
    "                encoded.pop('token_type_ids')\n",
    "                encoded, labels = encoded.to(device), labels.to(device)\n",
    "                encoded = dict([(k, v.squeeze(dim=1)) for (k, v) in encoded.items()])\n",
    "                outputs = bert_model(**encoded)\n",
    "                loss = criterion(outputs.logits, labels)\n",
    "\n",
    "                val_loss += loss.item()\n",
    "\n",
    "                # Calculate accuracy\n",
    "                _, predicted = torch.max(outputs.logits, 1)\n",
    "                total_val += labels.size(0)\n",
    "                correct_val += (predicted == labels).sum().item()\n",
    "\n",
    "        val_acc = correct_val / total_val\n",
    "\n",
    "        metrics['train_loss'].append(train_loss / num_train_batches)\n",
    "        metrics['val_loss'].append(val_loss / len(val_loader))\n",
    "        metrics['train_acc'].append(train_acc)\n",
    "        metrics['val_acc'].append(val_acc)\n",
    "\n",
    "        scheduler.step()\n",
    "\n",
    "        if (epoch + 1) % print_every == 0:\n",
    "            print(f\"\"\"\n",
    "            Training loss: {metrics['train_loss'][-1]:.4f}, Validation loss: {metrics['val_loss'][-1]:.4f},\n",
    "            Training accuracy: {metrics['train_acc'][-1] * 100:.2f}%, Validation accuracy: {metrics['val_acc'][-1] * 100:.2f}%\n",
    "            \"\"\")\n",
    "    path = f'./bert_model_adapters_checkpoint_{int(train_percentage * 100)}%.pth'\n",
    "    #bert_model.save_adapter(path, \"mrpc\", with_head=True)\n",
    "    #with open(path, 'wb') as handle:\n",
    "     #   pickle.dump(bert_model, handle, protocol=pickle.HIGHEST_PROTOCOL)\n",
    "    # Save model checkpoint\n",
    "    checkpoint = {\n",
    "         'epoch': epoch + 1,\n",
    "         'model_state_dict': bert_model.state_dict(),\n",
    "         'optimizer_state_dict': optimizer.state_dict(),\n",
    "         'scheduler_state_dict': scheduler.state_dict(),\n",
    "         'metrics': metrics,\n",
    "    }\n",
    "    torch.save(checkpoint, path)\n",
    "    \n",
    "        \n",
    "    # Save metrics for the current percentage\n",
    "    bert_metrics_by_percentage[f\"{int(train_percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "percentages = list(bert_metrics_by_percentage.keys())\n",
    "\n",
    "def plot_results(metrics_by_percentage):\n",
    "    plt.figure(figsize=(12, 6))\n",
    "\n",
    "    percentages = list(metrics_by_percentage.keys())\n",
    "    num_percentages = len(percentages)\n",
    "    \n",
    "    fig, axs = plt.subplots(1, 2, figsize=(12, 6))\n",
    "\n",
    "    losses = []\n",
    "    acc = []\n",
    "    \n",
    "    for i, (percentage, metrics) in enumerate(metrics_by_percentage.items(), 1):\n",
    "        acc.append(metrics['accuracy'])\n",
    "        losses.append(metrics['loss'])\n",
    "\n",
    "    axs[0].plot(percentages, losses)\n",
    "    axs[0].set_xlabel('Percentage')\n",
    "    axs[0].set_ylabel('Loss')\n",
    "    axs[0].set_title('Training Loss Over Percentages')\n",
    "\n",
    "    axs[1].plot(percentages, acc)\n",
    "    axs[1].set_xlabel('Percentage')\n",
    "    axs[1].set_ylabel('Accuracy')\n",
    "    axs[1].set_title('Test Accuracy Over Percentages')\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "\n",
    "fig, axs = plt.subplots(1, 2, figsize=(12, 5))\n",
    "\n",
    "# Plot training and validation loss\n",
    "axs[0].plot(percentages, [metrics['train_loss'][-1] for metrics in bert_metrics_by_percentage.values()], label='Train Loss')\n",
    "axs[0].plot(percentages, [metrics['val_loss'][-1] for metrics in bert_metrics_by_percentage.values()], label='Validation Loss')\n",
    "axs[0].set_title(\"Loss as a function of dataset percentage\")\n",
    "axs[0].set_xlabel('Percentage of the dataset')\n",
    "axs[0].set_ylabel('Loss')\n",
    "axs[0].legend()\n",
    "\n",
    "# Plot training and validation accuracy\n",
    "axs[1].plot(percentages, [metrics['train_acc'][-1] * 100 for metrics in bert_metrics_by_percentage.values()], label='Train Accuracy')\n",
    "axs[1].plot(percentages, [metrics['val_acc'][-1] * 100 for metrics in bert_metrics_by_percentage.values()], label='Validation Accuracy')\n",
    "axs[1].set_title(\"Accuracy as a function of dataset percentage\")\n",
    "axs[1].set_xlabel('Percentage of the dataset')\n",
    "axs[1].set_ylabel('Accuracy (%)')\n",
    "axs[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "metrics_by_percentage = {}\n",
    "for percentage in percentages:\n",
    "    # Define model\n",
    "    # Load model\n",
    "    path = f'./bert_model_adapters_checkpoint_{int(percentage * 100)}%.pth'\n",
    "    checkpoint = torch.load(path)\n",
    "    bert_state_dict = checkpoint['model_state_dict']\n",
    "    bert_model.load_state_dict(bert_state_dict)\n",
    "    \n",
    "    if torch.cuda.device_count() > 1:\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bert_model = nn.DataParallel(bert_model)\n",
    "\n",
    "    bert_model = bert_model.to(device)\n",
    "\n",
    "    metrics = {'loss': [], 'accuracy': []}\n",
    "\n",
    "    # Testing\n",
    "    bert_model.eval()\n",
    "    test_loss = 0.0\n",
    "    all_predicts_test = np.array([])\n",
    "    all_labels_test = np.array([])\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels, token in tqdm(test_loader, desc=f'Testing'):\n",
    "            token, labels = token.to(device), labels.to(device)\n",
    "            outputs = bert_model(**token)\n",
    "            loss = criterion(outputs.logits, labels)\n",
    "\n",
    "            test_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs.logits, 1)\n",
    "            all_predicts_test = np.append(all_predicts_test, predicted.detach().cpu().numpy())\n",
    "            all_labels_test = np.append(all_labels_test, labels.detach().cpu().numpy())\n",
    "\n",
    "    test_acc = (all_predicts_test == all_labels_test).sum().item() / len(all_predicts_test)\n",
    "\n",
    "    metrics['loss'].append(test_loss / len(test_loader))\n",
    "    metrics['accuracy'].append(test_acc)\n",
    "    \n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics\n",
    "plot_results(metrics_by_percentage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BertGan Implementation\n",
    "\n",
    "in this section we implement the `BertGan` architechture that tries to learn the task adversially, including a generator model that helps our typical discriminator model in classification, in a `GAN` fashion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## G1 generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FF_Generator Class\n",
    "\n",
    "this class implements the `G1` model specified in the paper, this is a simple `MLP` model, that tries to imitate the manifolds generated by the real data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:32:24.630651Z",
     "iopub.status.busy": "2024-02-03T15:32:24.629686Z",
     "iopub.status.idle": "2024-02-03T15:32:24.643722Z",
     "shell.execute_reply": "2024-02-03T15:32:24.642766Z",
     "shell.execute_reply.started": "2024-02-03T15:32:24.630615Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 768])\n"
     ]
    }
   ],
   "source": [
    "class FF_Generator(nn.Module):\n",
    "    def __init__(self, in_features, hidden_dim=256, out_features=768, dropout_prob=0.15):\n",
    "        super(FF_Generator, self).__init__()\n",
    "        # Parameters\n",
    "        self.latent_size = in_features\n",
    "        self.hidden_size = hidden_dim\n",
    "        self.out_features = out_features\n",
    "        # Layers\n",
    "        self.hidden = nn.Linear(in_features=in_features, out_features=hidden_dim)\n",
    "        self.act = nn.LeakyReLU(negative_slope=0.2)\n",
    "        self.dropout = nn.Dropout(p=dropout_prob)\n",
    "        self.head = nn.Linear(in_features=hidden_dim, out_features=out_features)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # hidden layer\n",
    "        h = self.act(self.hidden(x))\n",
    "        h = self.dropout(h)\n",
    "        # output layer\n",
    "        y = self.head(h)\n",
    "        \n",
    "        return y\n",
    "\n",
    "# Testing\n",
    "latent_size = 100\n",
    "generator = FF_Generator(in_features=latent_size)\n",
    "\n",
    "num_samples = 5\n",
    "random_latents = torch.randn((num_samples, latent_size))\n",
    "\n",
    "generated_samples = generator(random_latents)\n",
    "\n",
    "print(generated_samples.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discriminator class\n",
    "\n",
    "this class is a simple Feed-Forward architechture for our discriminator, which does the classification task on the given data generated by either the generator or the bert model processing real data, trying to classify the data in a supervised way, while learning the representations adversially in an unsupervised manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:32:26.843948Z",
     "iopub.status.busy": "2024-02-03T15:32:26.843568Z",
     "iopub.status.idle": "2024-02-03T15:32:26.865373Z",
     "shell.execute_reply": "2024-02-03T15:32:26.864495Z",
     "shell.execute_reply.started": "2024-02-03T15:32:26.843916Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probs: torch.Size([5, 7])\n",
      "Logits: torch.Size([5, 7])\n",
      "Hidden States: torch.Size([5, 128])\n"
     ]
    }
   ],
   "source": [
    "class Discriminator(nn.Module):\n",
    "    def __init__(self, k, in_features, hidden_dims=[512, 256, 128], dropout_prob=0.15):\n",
    "        super(Discriminator, self).__init__()\n",
    "        # Parameters\n",
    "        self.k = k\n",
    "        self.in_feature_size = in_features\n",
    "\n",
    "        # Define the discriminator layers based on hidden_dims\n",
    "        layers = []\n",
    "        layers.append(nn.Dropout(p=dropout_prob))\n",
    "        for i in range(len(hidden_dims)):\n",
    "            layers.extend([\n",
    "                nn.Linear(in_features=in_features if i == 0 else hidden_dims[i-1], out_features=hidden_dims[i]),\n",
    "                nn.LeakyReLU(negative_slope=0.2),\n",
    "                nn.Dropout(p=dropout_prob)\n",
    "            ])\n",
    "\n",
    "        # Sequential discriminator layers\n",
    "        self.discriminator = nn.Sequential(*layers)\n",
    "\n",
    "        # Head layer\n",
    "        self.head = nn.Linear(in_features=hidden_dims[-1], out_features=k + 1)\n",
    "        self.softmax = nn.Softmax(dim=-1)\n",
    "\n",
    "    def forward(self, x, return_logits=True):\n",
    "        # Forward pass through the sequential layers\n",
    "        h = self.discriminator(x)\n",
    "        # Output layer\n",
    "        logits = self.head(h)\n",
    "        # Softmax\n",
    "        probs = self.softmax(logits)\n",
    "\n",
    "        if return_logits:\n",
    "            return probs, logits, h\n",
    "        return probs, h\n",
    "\n",
    "# Testing\n",
    "k_value = K\n",
    "in_features = generator.out_features\n",
    "discriminator = Discriminator(k=k_value, in_features=in_features)\n",
    "\n",
    "batch_size = 5\n",
    "input_data = torch.randn((batch_size, in_features))\n",
    "\n",
    "probs, logits, hidden_states = discriminator(input_data)\n",
    "\n",
    "print(\"Probs:\", probs.shape)\n",
    "print(\"Logits:\", logits.shape)\n",
    "print(\"Hidden States:\", hidden_states.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BertGan implementation\n",
    "\n",
    "this model is just combining the `generator` and `discriminator` along with the `bert` model, and helps us in training the bertGan, as it computes the neccesary losses, and performing the forward pass alongside adverserial generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:32:30.267977Z",
     "iopub.status.busy": "2024-02-03T15:32:30.267639Z",
     "iopub.status.idle": "2024-02-03T15:32:30.283748Z",
     "shell.execute_reply": "2024-02-03T15:32:30.282756Z",
     "shell.execute_reply.started": "2024-02-03T15:32:30.267954Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class BertGan(nn.Module):\n",
    "    def __init__(self, Generator, Discriminator, Bert, fast_gen_grad=False, has_pooler=False, no_grad_bert=False):\n",
    "        super(BertGan, self).__init__()\n",
    "        \n",
    "        self.generator = Generator\n",
    "        self.discriminator = Discriminator\n",
    "        self.bert = Bert\n",
    "        \n",
    "        self.k = self.discriminator.k\n",
    "        self.epsilon = 1e-8\n",
    "        \n",
    "        self.gen_log_grad = fast_gen_grad\n",
    "        \n",
    "        self.has_pooler = has_pooler\n",
    "        self.no_grad_bert = no_grad_bert\n",
    "\n",
    "    def forward(self, inputs, z=None, labels=None, label_mask=None):\n",
    "        # Bert real data forward pass\n",
    "        bert_outputs = self.bert(**inputs) \n",
    "        x_r = bert_outputs.pooler_output if self.has_pooler else bert_outputs.last_hidden_state[:, 0]\n",
    "        if self.no_grad_bert:\n",
    "            x_r = x_r.detach()\n",
    "            \n",
    "        # Discriminator forward pass on real data\n",
    "        D_real_prob, D_real_logits, D_real_features = self.discriminator(x_r)\n",
    "        \n",
    "        # Supervised loss\n",
    "        real_logits = D_real_logits[:, :-1]\n",
    "        real_prob   = F.softmax(real_logits, dim=-1)\n",
    "        d_loss_sup = 0\n",
    "        if labels is not None:\n",
    "            if label_mask is None:\n",
    "                label_mask = torch.ones_like(labels)\n",
    "            ce = F.cross_entropy(real_logits, labels, reduction='none')\n",
    "            masked_ce = ce * label_mask\n",
    "            d_loss_sup = masked_ce.sum() / max(label_mask.sum(), 1) \n",
    "        \n",
    "        # Inference mode\n",
    "        if z is None:\n",
    "            # If latent (z) is None, skip the generator and set g_loss to None\n",
    "            d_loss = -torch.mean(\n",
    "                torch.log(1 - D_real_prob[:, -1] + self.epsilon)\n",
    "            ) + d_loss_sup\n",
    "            return {\n",
    "                'probs': real_prob,\n",
    "                'logits': real_logits,\n",
    "                'd_loss': d_loss,\n",
    "                'g_loss': None,\n",
    "            }\n",
    "\n",
    "        # Generator forward pass\n",
    "        x_g = self.generator(z)\n",
    "\n",
    "        # Discriminator forward pass on fake data\n",
    "        D_fake_prob, D_fake_logits, D_fake_features = self.discriminator(x_g)\n",
    "\n",
    "        # Discriminator loss\n",
    "        # Unsupervised loss\n",
    "        d_loss_unsup = -torch.mean(\n",
    "            torch.log(1 - D_real_prob[:, -1] + self.epsilon)\n",
    "        ) - torch.mean(\n",
    "            torch.log(D_fake_prob[:, -1] + self.epsilon)\n",
    "        )\n",
    "\n",
    "        d_loss = d_loss_unsup + d_loss_sup\n",
    "\n",
    "        # Generator loss\n",
    "        g_loss_unsup = -torch.mean(\n",
    "            torch.log(1 - D_fake_prob[:, -1] + self.epsilon) \n",
    "            if not self.gen_log_grad else\n",
    "            - torch.log(D_fake_prob[:, -1] + self.epsilon)\n",
    "        )\n",
    "        g_loss_feature_match = torch.mean(\n",
    "            (torch.mean(D_real_features, dim=0) - torch.mean(D_fake_features, dim=0)) ** 2\n",
    "        )\n",
    "\n",
    "        g_loss = g_loss_unsup + g_loss_feature_match\n",
    "        \n",
    "        output = {\n",
    "            'probs': real_prob,\n",
    "            'logits': real_logits,\n",
    "            'd_loss': d_loss,\n",
    "            'g_loss': g_loss,\n",
    "        }\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adverserial training\n",
    "\n",
    "the adverserial training step is defined as follows:\n",
    "\n",
    "$$\\quad L_{\\mathcal{D}}=L_{\\mathcal{D}_{\\text {sup. }}}+L_{\\mathcal{D}_{\\text {unsup. }}}$$\n",
    "$$\n",
    "\\begin{gather*}\n",
    "L_{\\mathcal{D}_{\\text {sup. }}}  =-\\mathbb{E}_{x, y \\sim p_d} \\log \\left[p_{\\mathrm{m}}(\\hat{y}=y \\mid x, y \\in(1, \\ldots, k))\\right] \\\\\n",
    "L_{\\mathcal{D}_{\\text {unsup. }}}  =-\\mathbb{E}_{x \\sim p_d} \\log \\left[1-p_{\\mathrm{m}}(\\hat{y}=y \\mid x, y=k+1)\\right] -\\mathbb{E}_{x \\sim \\mathcal{G}} \\log \\left[p_{\\mathrm{m}}(\\hat{y}=y \\mid x, y=k+1)\\right] \\\\\n",
    "\\rightarrow  L_{\\mathcal{D}_{\\text {unsup. }}}  =-\\mathbb{E}_{x \\sim p_d} [\\log (\\mathcal{D}(x))] -\\mathbb{E}_{x \\sim \\mathcal{G}} \n",
    "[\\log (1-\\mathcal{D}(x))]\n",
    "\\end{gather*}\n",
    "$$\n",
    "\n",
    "$$L_{\\mathcal{G}_{\\text {unsup. }}}=-\\mathbb{E}_{x \\sim \\mathcal{G}} \n",
    "\\log \\left[1-p_m(\\hat{y}=y \\mid x, y=k+1)\\right]$$\n",
    "$$ \n",
    "L_{\\mathcal{G}_{\\text {feature matching }}} = \\frac{1}{\\text{hidden\\_dim}}||\\mathbb{E}_{x \\sim p_d} f(x) \n",
    "- \\mathbb{E}_{x \\sim \\mathcal{G}} f(x) ||_2^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this block simply tests the defined bertGan model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:32:33.165404Z",
     "iopub.status.busy": "2024-02-03T15:32:33.164738Z",
     "iopub.status.idle": "2024-02-03T15:32:50.473409Z",
     "shell.execute_reply": "2024-02-03T15:32:50.472511Z",
     "shell.execute_reply.started": "2024-02-03T15:32:33.165371Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training:\n",
      "\n",
      "Discriminator Loss: 3.8696823120117188\n",
      "Generator Loss: 0.1505683958530426\n",
      "Logits: torch.Size([5, 6])\n",
      "Probabilities: torch.Size([5, 6])\n",
      "\n",
      "Inference:\n",
      "\n",
      "Discriminator Loss: 0.15020646154880524\n",
      "Generator Loss: None\n",
      "Logits: torch.Size([5, 6])\n",
      "Probabilities: torch.Size([5, 6])\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "bert, tokenizer = get_bert()\n",
    "bert_gan = BertGan(Generator=generator, Discriminator=discriminator, Bert=bert)\n",
    "\n",
    "batch_size = 5\n",
    "latent_batch_size = 10\n",
    "random_latents = torch.randn((latent_batch_size, latent_size))\n",
    "random_inputs = tokenizer(\n",
    "    [\"ali nourian save me\"] * batch_size, return_tensors='pt', truncation=True, padding=\"max_length\"\n",
    ")\n",
    "random_labels = torch.randint(0, discriminator.k, (batch_size,))\n",
    "random_mask   = torch.randint(0, 2, (batch_size,))\n",
    "\n",
    "# Train\n",
    "output = bert_gan(random_inputs, random_latents, random_labels, random_mask)\n",
    "# Inference\n",
    "output_inf = bert_gan(random_inputs)\n",
    "\n",
    "print(\"Training:\\n\")\n",
    "print(\"Discriminator Loss:\", output['d_loss'].item())\n",
    "print(\"Generator Loss:\", output['g_loss'].item())\n",
    "print(\"Logits:\", output['logits'].shape)\n",
    "print(\"Probabilities:\", output['probs'].shape)\n",
    "print(\"\\nInference:\\n\")\n",
    "print(\"Discriminator Loss:\", output_inf['d_loss'].item())\n",
    "print(\"Generator Loss:\", output_inf['g_loss'])\n",
    "print(\"Logits:\", output_inf['logits'].shape)\n",
    "print(\"Probabilities:\", output_inf['probs'].shape)\n",
    "\n",
    "# Testing good grads\n",
    "output['g_loss'].backward(retain_graph=True)\n",
    "output['d_loss'].backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### model implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in this part we simply visualize what the model looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:32:59.413108Z",
     "iopub.status.busy": "2024-02-03T15:32:59.412268Z",
     "iopub.status.idle": "2024-02-03T15:33:00.023674Z",
     "shell.execute_reply": "2024-02-03T15:33:00.022811Z",
     "shell.execute_reply.started": "2024-02-03T15:32:59.413074Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using 2 GPUs!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BertGan(\n",
       "  (generator): DataParallel(\n",
       "    (module): FF_Generator(\n",
       "      (hidden): Linear(in_features=100, out_features=768, bias=True)\n",
       "      (act): LeakyReLU(negative_slope=0.2)\n",
       "      (dropout): Dropout(p=0.15, inplace=False)\n",
       "      (head): Linear(in_features=768, out_features=768, bias=True)\n",
       "    )\n",
       "  )\n",
       "  (discriminator): DataParallel(\n",
       "    (module): Discriminator(\n",
       "      (discriminator): Sequential(\n",
       "        (0): Dropout(p=0.15, inplace=False)\n",
       "        (1): Linear(in_features=768, out_features=512, bias=True)\n",
       "        (2): LeakyReLU(negative_slope=0.2)\n",
       "        (3): Dropout(p=0.15, inplace=False)\n",
       "        (4): Linear(in_features=512, out_features=256, bias=True)\n",
       "        (5): LeakyReLU(negative_slope=0.2)\n",
       "        (6): Dropout(p=0.15, inplace=False)\n",
       "        (7): Linear(in_features=256, out_features=128, bias=True)\n",
       "        (8): LeakyReLU(negative_slope=0.2)\n",
       "        (9): Dropout(p=0.15, inplace=False)\n",
       "      )\n",
       "      (head): Linear(in_features=128, out_features=7, bias=True)\n",
       "      (softmax): Softmax(dim=-1)\n",
       "    )\n",
       "  )\n",
       "  (bert): DataParallel(\n",
       "    (module): BertModel(\n",
       "      (embeddings): BertEmbeddings(\n",
       "        (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "        (position_embeddings): Embedding(512, 768)\n",
       "        (token_type_embeddings): Embedding(2, 768)\n",
       "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (encoder): BertEncoder(\n",
       "        (layer): ModuleList(\n",
       "          (0-11): 12 x BertLayer(\n",
       "            (attention): BertAttention(\n",
       "              (self): BertSelfAttention(\n",
       "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "              )\n",
       "              (output): BertSelfOutput(\n",
       "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "                (dropout): Dropout(p=0.1, inplace=False)\n",
       "              )\n",
       "            )\n",
       "            (intermediate): BertIntermediate(\n",
       "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "              (intermediate_act_fn): GELUActivation()\n",
       "            )\n",
       "            (output): BertOutput(\n",
       "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1, inplace=False)\n",
       "            )\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "      (pooler): BertPooler(\n",
       "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "        (activation): Tanh()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "latent_size = 100\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "bert = BertModel.from_pretrained('bert-base-uncased')\n",
    "\n",
    "generator = FF_Generator(\n",
    "    in_features=latent_size, hidden_dim=hidden_dim, out_features=hidden_dim\n",
    ")\n",
    "discriminator = Discriminator(\n",
    "    k=K, in_features=hidden_dim\n",
    ")\n",
    "bertGan = BertGan(\n",
    "    generator, discriminator, bert, fast_gen_grad=True\n",
    ")\n",
    "\n",
    "#models parameters\n",
    "bert_vars = [i for i in bertGan.bert.parameters()]\n",
    "d_vars = bert_vars + [v for v in bertGan.discriminator.parameters()]\n",
    "g_vars = bertGan.generator.parameters()\n",
    "\n",
    "#optimizer\n",
    "d_lr = 5e-4\n",
    "g_lr = 5e-4\n",
    "n_critic = 5\n",
    "dis_optimizer = optim.AdamW(d_vars, lr=d_lr)\n",
    "gen_optimizer = optim.AdamW(g_vars, lr=g_lr) \n",
    "\n",
    "if torch.cuda.device_count() > 1: # Since we are using different ootimizers\n",
    "    print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "    bertGan.bert = nn.DataParallel(bertGan.bert)\n",
    "    bertGan.discriminator = nn.DataParallel(bertGan.discriminator)\n",
    "    bertGan.generator = nn.DataParallel(bertGan.generator)\n",
    "    \n",
    "bertGan = bertGan.to(device)\n",
    "\n",
    "bertGan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BertGan training\n",
    "\n",
    "this function implement the training procedure of the BertGan model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:33:04.121438Z",
     "iopub.status.busy": "2024-02-03T15:33:04.120671Z",
     "iopub.status.idle": "2024-02-03T15:33:04.143828Z",
     "shell.execute_reply": "2024-02-03T15:33:04.142781Z",
     "shell.execute_reply.started": "2024-02-03T15:33:04.121404Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch.nn.utils import clip_grad_norm_\n",
    "from tqdm import tqdm\n",
    "import time\n",
    "\n",
    "def train_bertgan(\n",
    "    bertgan, \n",
    "    dis_optimizer, gen_optimizer, bert_optimizer,\n",
    "    masked_train_loader, val_loader, \n",
    "    dis_scheduler=None, gen_scheduler=None, bert_scheduler=None,\n",
    "    num_epochs=3, print_every=1, c=1, n_critic=5, latent_dim=100, spherical=True):\n",
    "\n",
    "    metrics = {'train_loss': [], 'val_loss': [], 'train_acc': [], 'val_acc': []}\n",
    "    \n",
    "    for epoch in range(num_epochs):\n",
    "        bertgan.train()\n",
    "\n",
    "        if gen_scheduler is not None:\n",
    "            gen_scheduler.step()\n",
    "        if dis_scheduler is not None:\n",
    "            dis_scheduler.step()\n",
    "        if bert_scheduler is not None:\n",
    "            bert_scheduler.step()\n",
    "            print(f\"Current bert lr: {bert_scheduler.get_last_lr()}\")\n",
    "        \n",
    "        # Training\n",
    "        train_loss = 0.0\n",
    "        batch_idx = 0\n",
    "        all_predicts_train = np.array([])\n",
    "        all_labels_train = np.array([])\n",
    "        \n",
    "        for inputs, labels, mask, token in tqdm(masked_train_loader, desc=f'Train - epoch {epoch + 1}/{num_epochs}'):\n",
    "            batch_idx += 1\n",
    "            token, labels, mask = token.to(device), labels.to(device), mask.to(device)\n",
    "            z = torch.zeros(train_loader.batch_size, latent_dim, device=device).uniform_(0, 1)\n",
    "            if spherical:\n",
    "                theta = 0\n",
    "            \n",
    "            bertGan.bert.zero_grad()\n",
    "            bertGan.discriminator.zero_grad()\n",
    "            \n",
    "            outputs = bertgan(token, z=z, labels=labels, label_mask=mask)\n",
    "            d_loss = outputs['d_loss']\n",
    "            g_loss = outputs['g_loss']\n",
    "            \n",
    "            dis_optimizer.zero_grad()\n",
    "            gen_optimizer.zero_grad()\n",
    "            \n",
    "            # retain_graph=True is required since the underlying graph will be deleted after backward\n",
    "            g_loss.backward(retain_graph=True)\n",
    "            d_loss.backward()\n",
    "            \n",
    "            # Lipchitz continious\n",
    "#             clip_grad_norm_(bertgan.discriminator.parameters(), c)\n",
    "#             clip_grad_norm_(bertgan.generator.parameters(), c)\n",
    "            if (batch_idx + 1) % n_critic == 0:\n",
    "                gen_optimizer.step() \n",
    "            dis_optimizer.step()\n",
    "            bert_optimizer.step()\n",
    "\n",
    "            train_loss += g_loss.item() + d_loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs['probs'], 1)\n",
    "            all_predicts_train = np.append(all_predicts_train, predicted.detach().cpu().numpy())\n",
    "            all_labels_train = np.append(all_labels_train, labels.detach().cpu().numpy())\n",
    "\n",
    "        train_acc = (all_predicts_train == all_labels_train).sum().item() / len(all_predicts_train)\n",
    "\n",
    "        # Validation\n",
    "        bertgan.eval()\n",
    "        val_loss = 0.0\n",
    "        all_predicts_val = np.array([])\n",
    "        all_labels_val = np.array([])\n",
    "        with torch.no_grad():\n",
    "            for inputs, labels, _, token in tqdm(masked_val_loader, desc=f'Validation - epoch {epoch + 1}/{num_epochs}'):\n",
    "                token, labels = token.to(device), labels.to(device)\n",
    "                outputs = bertgan(token, labels=labels)\n",
    "                loss = outputs['d_loss']\n",
    "\n",
    "                val_loss += loss.item()\n",
    "\n",
    "                # Calculate accuracy\n",
    "                _, predicted = torch.max(outputs['probs'], 1)\n",
    "                all_predicts_val = np.append(all_predicts_val, predicted.detach().cpu().numpy())\n",
    "                all_labels_val = np.append(all_labels_val, labels.detach().cpu().numpy())\n",
    "\n",
    "        val_acc = (all_predicts_val == all_labels_val).sum().item() / len(all_predicts_val)\n",
    "\n",
    "        metrics['train_loss'].append(train_loss / len(train_loader))\n",
    "        metrics['val_loss'].append(val_loss / len(val_loader))\n",
    "        metrics['train_acc'].append(train_acc)\n",
    "        metrics['val_acc'].append(val_acc)\n",
    "\n",
    "        if (epoch + 1) % print_every == 0:\n",
    "            print(f\"\"\"\n",
    "            Training loss: {metrics['train_loss'][-1]:.4f}, Validation loss: {metrics['val_loss'][-1]:.4f},\n",
    "            Training accuracy: {metrics['train_acc'][-1] * 100:.2f}%, Validation accuracy: {metrics['val_acc'][-1] * 100:.2f}%\n",
    "            \"\"\")\n",
    "\n",
    "    return metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T09:36:49.271000Z",
     "iopub.status.busy": "2024-02-03T09:36:49.270669Z",
     "iopub.status.idle": "2024-02-03T09:36:50.373454Z",
     "shell.execute_reply": "2024-02-03T09:36:50.372309Z",
     "shell.execute_reply.started": "2024-02-03T09:36:49.270974Z"
    }
   },
   "outputs": [],
   "source": [
    "!mkdir bertgan_G1_with_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:34:30.861197Z",
     "iopub.status.busy": "2024-02-03T15:34:30.860791Z",
     "iopub.status.idle": "2024-02-03T15:34:30.868192Z",
     "shell.execute_reply": "2024-02-03T15:34:30.867195Z",
     "shell.execute_reply.started": "2024-02-03T15:34:30.861163Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_masked_loader(path, percentage, full_dataset, train_size, val_size, batch_size=128):\n",
    "    masked_full_dataset = MaskedSemEvalDataset(path, mask_percentage=percentage)\n",
    "    masked_full_dataset.tokenized_data = full_dataset.tokenized_data\n",
    "    masked_full_dataset.tokenizer = full_dataset.tokenizer\n",
    "    masked_full_dataset.data = full_dataset.data\n",
    "    masked_train_dataset, masked_val_dataset = random_split(masked_full_dataset, [train_size, val_size])\n",
    "    masked_train_loader = get_loader(masked_train_dataset, batch_size=batch_size)\n",
    "    masked_val_loader = get_loader(masked_val_dataset, batch_size=batch_size)\n",
    "    \n",
    "    return masked_train_loader, masked_val_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training loop\n",
    "\n",
    "in this loop we create a new bertGan model for each percentage of dataset visible for the model, and then return the metrics of the training loop in the end, saving the metrics by percentage for future plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:42:58.035055Z",
     "iopub.status.busy": "2024-02-03T15:42:58.034618Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup, get_constant_schedule_with_warmup\n",
    "\n",
    "percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "metrics_by_percentage = {}\n",
    "\n",
    "for percentage in percentages:\n",
    "    print(f\"\"\"\n",
    "=================================================================================================\n",
    "\n",
    "                                Training on {100 * percentage} % of the dataset\n",
    "    \n",
    "=================================================================================================\n",
    "\n",
    "    \"\"\")\n",
    "    # Define model\n",
    "    bert, _ = get_bert(model=\"distil\")\n",
    "\n",
    "    generator = FF_Generator(\n",
    "        in_features=latent_size, hidden_dim=hidden_dim, out_features=hidden_dim\n",
    "    )\n",
    "    discriminator = Discriminator(\n",
    "        k=K, in_features=hidden_dim\n",
    "    )\n",
    "    bertGan = BertGan(\n",
    "        generator, discriminator, bert, fast_gen_grad=False,# no_grad_bert=True,\n",
    "    )\n",
    "\n",
    "    if torch.cuda.device_count() > 1: # Since we are using different ootimizers\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bertGan.bert = nn.DataParallel(bertGan.bert)\n",
    "        bertGan.discriminator = nn.DataParallel(bertGan.discriminator)\n",
    "        bertGan.generator = nn.DataParallel(bertGan.generator)\n",
    "        \n",
    "    bertGan = bertGan.to(device)\n",
    "\n",
    "    # hyper-parameters\n",
    "    d_lr = 5e-4\n",
    "    g_lr = 5e-4\n",
    "    batch_size = 128\n",
    "    num_epochs = 3\n",
    "    warmup_ratio = 0.1\n",
    "    num_training_steps = int(train_size / batch_size * num_epochs)\n",
    "    num_warmup_steps   = int(num_training_steps * warmup_ratio)\n",
    "    \n",
    "    #optimizer\n",
    "    bert_optimizer = optim.AdamW(bertGan.bert.parameters(), lr=d_lr) \n",
    "    dis_optimizer = optim.AdamW(bertGan.discriminator.parameters(), lr=d_lr)\n",
    "    gen_optimizer = optim.AdamW(bertGan.generator.parameters(), lr=g_lr) \n",
    "    \n",
    "    #scheduler\n",
    "    bert_scheduler = get_linear_schedule_with_warmup(\n",
    "        bert_optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps,\n",
    "    )\n",
    "    dis_scheduler = None\n",
    "    gen_scheduler = None\n",
    "    \n",
    "    # Training\n",
    "    masked_train_loader, masked_val_loader = get_masked_loader(\n",
    "        train_path, percentage, full_dataset, train_size, val_size, batch_size=batch_size,\n",
    "    )\n",
    "    metrics = train_bertgan(\n",
    "        bertGan, \n",
    "        dis_optimizer, gen_optimizer, bert_optimizer,\n",
    "        dis_scheduler=dis_scheduler, gen_scheduler=gen_scheduler, bert_scheduler=bert_scheduler,\n",
    "        masked_train_loader=masked_train_loader, val_loader=masked_val_loader, n_critic=1, num_epochs=num_epochs,\n",
    "    )\n",
    "    \n",
    "    # Save model checkpoint\n",
    "    checkpoint = {\n",
    "        'percentage': percentage,\n",
    "        'model_state_dict': bertGan.state_dict(),\n",
    "        'metrics': metrics,\n",
    "        'disc_optimizer_state_dict': dis_optimizer.state_dict(),\n",
    "        'gen_optimizer_state_dict': gen_optimizer.state_dict(),\n",
    "    }\n",
    "    \n",
    "    torch.save(checkpoint, f'./bertgan_G1_with_grad/model_checkpoint_{int(percentage * 100)}%.pth')\n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "percentages = list(metrics_by_percentage.keys())\n",
    "\n",
    "fig, axs = plt.subplots(1, 2, figsize=(12, 5))\n",
    "\n",
    "# Plot training and validation loss\n",
    "axs[0].plot(percentages, [metrics['train_loss'][-1] for metrics in metrics_by_percentage.values()], label='Train Loss')\n",
    "axs[0].plot(percentages, [metrics['val_loss'][-1] for metrics in metrics_by_percentage.values()], label='Validation Loss')\n",
    "axs[0].set_title(\"Loss as a function of dataset percentage\")\n",
    "axs[0].set_xlabel('Percentage of the dataset')\n",
    "axs[0].set_ylabel('Loss')\n",
    "axs[0].legend()\n",
    "\n",
    "# Plot training and validation accuracy\n",
    "axs[1].plot(percentages, [metrics['train_acc'][-1] * 100 for metrics in metrics_by_percentage.values()], label='Train Accuracy')\n",
    "axs[1].plot(percentages, [metrics['val_acc'][-1] * 100 for metrics in metrics_by_percentage.values()], label='Validation Accuracy')\n",
    "axs[1].set_title(\"Accuracy as a function of dataset percentage\")\n",
    "axs[1].set_xlabel('Percentage of the dataset')\n",
    "axs[1].set_ylabel('Accuracy (%)')\n",
    "axs[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T09:26:50.858833Z",
     "iopub.status.busy": "2024-02-03T09:26:50.858446Z",
     "iopub.status.idle": "2024-02-03T09:26:50.863181Z",
     "shell.execute_reply": "2024-02-03T09:26:50.862242Z",
     "shell.execute_reply.started": "2024-02-03T09:26:50.858802Z"
    }
   },
   "outputs": [],
   "source": [
    "train_path = '/kaggle/working/train.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:42:53.718695Z",
     "iopub.status.busy": "2024-02-03T15:42:53.717914Z",
     "iopub.status.idle": "2024-02-03T15:42:54.721759Z",
     "shell.execute_reply": "2024-02-03T15:42:54.720691Z",
     "shell.execute_reply.started": "2024-02-03T15:42:53.718659Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5647"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gc\n",
    "torch.cuda.empty_cache()\n",
    "del bertGan , bert, g_vars, d_vars\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing G1\n",
    "\n",
    "here we use the test dataset and check the accuracy of each trained model, with respect the percentage of dataset used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:01:09.583516Z",
     "iopub.status.busy": "2024-02-03T15:01:09.582787Z",
     "iopub.status.idle": "2024-02-03T15:01:38.260046Z",
     "shell.execute_reply": "2024-02-03T15:01:38.258774Z",
     "shell.execute_reply.started": "2024-02-03T15:01:09.583483Z"
    }
   },
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "percentages = [0.01, 0.05, 0.1, 0.5]\n",
    "acc_per_percent = {}\n",
    "for percentage in percentages:\n",
    "    # Define model\n",
    "    bert, _ = get_bert(model=\"distil\")\n",
    "\n",
    "    generator = FF_Generator(\n",
    "        in_features=latent_size, hidden_dim=hidden_dim, out_features=hidden_dim\n",
    "    )\n",
    "    discriminator = Discriminator(\n",
    "        k=K, in_features=hidden_dim\n",
    "    )\n",
    "    bertgan = BertGan(\n",
    "        generator, discriminator, bert, fast_gen_grad=False,# no_grad_bert=True,\n",
    "    )\n",
    "\n",
    "    if torch.cuda.device_count() > 1: # Since we are using different ootimizers\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bertgan.bert = nn.DataParallel(bertgan.bert)\n",
    "        bertgan.discriminator = nn.DataParallel(bertgan.discriminator)\n",
    "        bertgan.generator = nn.DataParallel(bertgan.generator)\n",
    "    \n",
    "    metrics = {'loss': [], 'accuracy': []}\n",
    "    bertgan = bertgan.to(device)\n",
    "    # Load model\n",
    "    path = f'./bertgan_G1_with_grad/model_checkpoint_{int(percentage * 100)}%.pth'\n",
    "    checkpoint = torch.load(path)\n",
    "    bertgan_state_dict = checkpoint['model_state_dict']\n",
    "    bertgan.load_state_dict(bertgan_state_dict)\n",
    "    # Testing\n",
    "    bertgan.eval()\n",
    "    test_loss = 0.0\n",
    "    all_predicts_test = np.array([])\n",
    "    all_labels_test = np.array([])\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels, token in tqdm(test_loader, desc=f'Testing'):\n",
    "            token, labels = token.to(device), labels.to(device)\n",
    "            outputs = bertgan(token, labels=labels)\n",
    "            loss = outputs['d_loss']\n",
    "\n",
    "            test_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs['probs'], 1)\n",
    "            all_predicts_test = np.append(all_predicts_test, predicted.detach().cpu().numpy())\n",
    "            all_labels_test = np.append(all_labels_test, labels.detach().cpu().numpy())\n",
    "\n",
    "    test_acc = (all_predicts_test == all_labels_test).sum().item() / len(all_predicts_test)\n",
    "\n",
    "    metrics['loss'].append(test_loss / len(test_loader))\n",
    "    metrics['accuracy'].append(test_acc)\n",
    "    \n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-02-03T15:10:47.774998Z",
     "iopub.status.busy": "2024-02-03T15:10:47.774016Z",
     "iopub.status.idle": "2024-02-03T15:10:47.980413Z",
     "shell.execute_reply": "2024-02-03T15:10:47.979536Z",
     "shell.execute_reply.started": "2024-02-03T15:10:47.774961Z"
    }
   },
   "outputs": [],
   "source": [
    "plot_results(metrics_by_percentage)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## G2 generator\n",
    "\n",
    "in this section we use the same bertGan architechture, but use a Bert based generator instead, using a random text generated by `BOW` as the latent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.random import default_rng\n",
    "from time import time\n",
    "from collections import Counter\n",
    "import os\n",
    "import re\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "\n",
    "class BERT_Generator_dataset(SemEvalDataset):\n",
    "    def __init__(self, path, tokenizer, url=None, download=False, sentence_length=512):\n",
    "        super().__init__(path, url, download)\n",
    "        self.word_dict=Counter({})\n",
    "        self.regex='[ ,:,\\\\n,\\\\t,?,\\.,\\(,\\),\",“,”,!,--,},{]'\n",
    "        self.avg_length=0\n",
    "        self.lengths=[]\n",
    "        self.rng=default_rng()\n",
    "        intermediate_dict=Counter({})\n",
    "        i=0\n",
    "        for data in tqdm(self.data):\n",
    "            text=data['text']\n",
    "            split_array=re.split(self.regex,data['text'])\n",
    "            self.avg_length+=len(split_array)/len(self)\n",
    "            self.lengths.append(len(split_array))\n",
    "            intermediate_dict += Counter(split_array)\n",
    "            i+=1\n",
    "            if i%200==0:\n",
    "                self.word_dict+=intermediate_dict\n",
    "                intermediate_dict=Counter({})\n",
    "        self.word_dict+=intermediate_dict\n",
    "        \n",
    "        self.sentence_length=512 #As the histogram says\n",
    "        self.word_dict.pop('')\n",
    "        values=np.array(list(self.word_dict.values()))\n",
    "        self.p=values/np.sum(values)\n",
    "        self.tokenizer=tokenizer\n",
    "        self.arange_arr=np.arange(len(values))\n",
    "        keys=list(self.word_dict.keys())\n",
    "        \n",
    "        tokens=[]\n",
    "        for text in tqdm(keys):\n",
    "            token_dict=tokenizer(text, return_tensors='pt', add_special_tokens = False)\n",
    "            tokens.append(token_dict['input_ids'][0])\n",
    "        s=tokenizer('')['input_ids']\n",
    "        self.tokenizer_eos=torch.tensor([s[1]])\n",
    "        self.tokenizer_bos=torch.tensor([s[0]])\n",
    "        self.tokens=tokens\n",
    "        self.full_attention=torch.ones(self.sentence_length, dtype=torch.long)\n",
    "        \n",
    "    def __getitem__(self, idx):\n",
    "        sample_indices=self.rng.choice(self.arange_arr,size=self.sentence_length, replace=True, p=self.p)\n",
    "        \n",
    "        \n",
    "        \n",
    "        my_tensor=self.tokenizer_bos\n",
    "        i=0\n",
    "        temporary_tensor=torch.tensor([0])#It cannot concat empty tensors\n",
    "        for index in sample_indices:\n",
    "            temporary_tensor=torch.cat([temporary_tensor,self.tokens[index]],dim=0)\n",
    "            i+=1\n",
    "            \n",
    "            if i%25==0:\n",
    "\n",
    "                my_tensor=torch.cat([my_tensor,temporary_tensor[1:]],dim=0)\n",
    "                \n",
    "                temporary_tensor=torch.tensor([0])\n",
    "        my_tensor=torch.cat([my_tensor,temporary_tensor,self.tokenizer_eos],dim=0)\n",
    "        \n",
    "        n=len(my_tensor)\n",
    "        if n>=self.sentence_length:\n",
    "            my_tensor=torch.cat([my_tensor[:self.sentence_length-1],self.tokenizer_eos],dim=0).type(torch.long)\n",
    "            attention=torch.ones(self.sentence_length, dtype=torch.long)\n",
    "        else:\n",
    "            pad=torch.zeros(self.sentence_length-n, dtype=torch.long)\n",
    "            my_tensor=torch.cat([my_tensor,pad],dim=0)\n",
    "            attention=torch.cat([torch.ones(n,dtype=torch.long),pad],dim=0)\n",
    "        out_dict={'input_ids':my_tensor, \n",
    "                  'attention_mask': attention} #'token_type_ids': torch.zeros_like(my_tensor), \n",
    "        \n",
    "        return out_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset=BERT_Generator_dataset(tokenizer=tokenizer, path=train_path, url=train_url, download=DOWNLOAD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BERT_Generator(nn.Module):\n",
    "    def __init__(self, bert, dataset, device, has_pooler=False, batch_size=24):\n",
    "        super(BERT_Generator, self).__init__()\n",
    "        self.bert = bert.to(device)\n",
    "        self.has_pooler = has_pooler\n",
    "        self.batch_size=batch_size\n",
    "        self.dataset=dataset\n",
    "        self.dataloader=DataLoader(self.dataset, batch_size=batch_size, shuffle=False)\n",
    "        self.iter=iter(self.dataloader)\n",
    "        self.device=device\n",
    "        \n",
    "        for parameters in bert.embeddings.parameters():\n",
    "            parameters.requires_grad=False\n",
    "        for parameters in bert.encoder.parameters():\n",
    "            parameters.requires_grad=False\n",
    "    def forward(self, _):\n",
    "        try:\n",
    "            bag_of_words=next(self.iter)\n",
    "        except:\n",
    "            self.iter=iter(self.dataloader)\n",
    "            bag_of_words=next(self.iter)\n",
    "        outputs = self.bert(input_ids=bag_of_words['input_ids'].to(device), attention_mask=bag_of_words['attention_mask'].to(device))\n",
    "        y = outputs.pooler_output if self.has_pooler else outputs.last_hidden_state[:, 0]\n",
    "        \n",
    "        return y.to(self.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code for run, with minor changes from the previous parts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup, get_cosine_schedule_with_warmup\n",
    "percentages = [0.01]\n",
    "metrics_by_percentage = {}\n",
    "for percentage in percentages:\n",
    "    gen_bert, tokenizer = get_bert(model=\"bert\")\n",
    "\n",
    "    generator = BERT_Generator(\n",
    "        gen_bert, dataset, has_pooler=False, batch_size=24, device=device)\n",
    "    discriminator = Discriminator(\n",
    "        k=K, in_features=hidden_dim\n",
    "    )\n",
    "    bertGan = BertGan(\n",
    "        generator, discriminator, bert, fast_gen_grad=True, no_grad_bert=True,\n",
    "    )\n",
    "\n",
    "    if torch.cuda.device_count() > 1: # Since we are using different optimizers\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bertGan.bert = nn.DataParallel(bertGan.bert)\n",
    "        bertGan.discriminator = nn.DataParallel(bertGan.discriminator)\n",
    "        bertGan.generator = nn.DataParallel(bertGan.generator)\n",
    "    bertGan = bertGan.to(device)\n",
    "\n",
    "    # hyper-parameters\n",
    "    d_lr = 5e-4\n",
    "    g_lr = 5e-4\n",
    "    batch_size = 24\n",
    "    num_epochs = 1\n",
    "    warmup_ratio = 0.1\n",
    "    num_training_steps = int(train_size / batch_size * num_epochs)\n",
    "    num_warmup_steps   = int(num_training_steps * warmup_ratio)\n",
    "    \n",
    "    #optimizer\n",
    "    bert_optimizer = optim.AdamW(bertGan.bert.parameters(), lr=d_lr) \n",
    "    dis_optimizer = optim.AdamW(bertGan.discriminator.parameters(), lr=d_lr)\n",
    "    gen_optimizer = optim.AdamW(bertGan.generator.parameters(), lr=g_lr) \n",
    "    \n",
    "    #scheduler\n",
    "    bert_scheduler = get_cosine_schedule_with_warmup(\n",
    "        bert_optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps,\n",
    "    )\n",
    "#     dis_scheduler = get_linear_schedule_with_warmup(\n",
    "#         dis_optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps,\n",
    "#     )\n",
    "    dis_scheduler = None\n",
    "    gen_scheduler = get_cosine_schedule_with_warmup(\n",
    "        gen_optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps,\n",
    "    )\n",
    "    \n",
    "    # Training\n",
    "    masked_train_loader, masked_val_loader = get_masked_loader(\n",
    "        train_path, percentage, full_dataset, train_size, val_size, batch_size=batch_size,\n",
    "    )\n",
    "    print('----Before Train----')\n",
    "    metrics = train_bertgan(\n",
    "        bertGan, \n",
    "        dis_optimizer, gen_optimizer, bert_optimizer,\n",
    "        dis_scheduler=dis_scheduler, gen_scheduler=gen_scheduler, bert_scheduler=bert_scheduler,\n",
    "        masked_train_loader=masked_train_loader, val_loader=masked_val_loader, n_critic=1, num_epochs=num_epochs,\n",
    "    )\n",
    "    \n",
    "    # Save model checkpoint\n",
    "    checkpoint = {\n",
    "        'percentage': percentage,\n",
    "        'model_state_dict': bertGan.state_dict(),\n",
    "        'metrics': metrics,\n",
    "        'disc_optimizer_state_dict': dis_optimizer.state_dict(),\n",
    "        'gen_optimizer_state_dict': gen_optimizer.state_dict(),\n",
    "    }\n",
    "    \n",
    "    torch.save(checkpoint, f'./bertgan_G2_without_most_grad/model_checkpoint_{int(percentage * 100)}%.pth')\n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mixture generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for the bonus section, we propose the MixtureBertGan architecture. in this architecture we will be trying to create a strong \n",
    "\n",
    "in this proposal the generator model is a mixture of experts architecture, each expert specializing in replicating embedding from one of the labels.\n",
    "\n",
    "in effect the generator becomes conditioned on the label.\n",
    "\n",
    "we also modify the feature matching of the loss, making it be class-wise.\n",
    "\n",
    "$$\n",
    "L_{\\mathcal{G}_{\\text {feature matching }}} = \\frac{1}{K}\\sum_{y=1}^{k}\\frac{1}{\\text{hidden\\_dim}}||\\mathbb{E}_{x \\sim p_d} f(x|y)- \\mathbb{E}_{x \\sim \\mathcal{G}} f(x|y) ||_2^2\n",
    "$$\n",
    "\n",
    "in effect, instead of the generator just trying to create a _real_ embedding, it will try to fake a _real chat gpt_ embedding. which would help in the adversarial training process.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "class MixtureGenerator(nn.Module):\n",
    "    def __init__(self, latent_size, num_labels, hidden_dim=256, out_features=768, dropout_prob=0.15):\n",
    "        super(MixtureGenerator, self).__init__()\n",
    "        self.latent_size = latent_size\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "        # Main generator\n",
    "        self.main_generator = FF_Generator(in_features=latent_size, hidden_dim=hidden_dim, out_features=out_features, dropout_prob=dropout_prob)\n",
    "\n",
    "        # Create generators for each label\n",
    "        self.label_generators = nn.ModuleList([\n",
    "            FF_Generator(in_features=latent_size, hidden_dim=hidden_dim, out_features=out_features, dropout_prob=dropout_prob)\n",
    "            for _ in range(num_labels)\n",
    "        ])\n",
    "\n",
    "    def forward(self, x, labels):\n",
    "        # Generate the main embedding\n",
    "        main_embedding = self.main_generator(x)\n",
    "\n",
    "        # Generate embeddings for each label\n",
    "        label_embeddings = torch.stack([\n",
    "            generator(x) for generator in self.label_generators\n",
    "        ])\n",
    "        \n",
    "        # Use the main embedding for labels with -1 (indicating the main generator)\n",
    "        selected_embeddings = label_embeddings[labels, range(len(labels))]\n",
    "        \n",
    "        # Combine the selected embeddings\n",
    "        combined_embeddings = main_embedding + selected_embeddings\n",
    "\n",
    "        return combined_embeddings\n",
    "\n",
    "\n",
    "# Testing\n",
    "latent_size = 100\n",
    "num_labels = 6\n",
    "mixture_generator = MixtureGenerator(latent_size=latent_size, num_labels=num_labels, out_features=768)\n",
    "\n",
    "num_samples = 40\n",
    "random_latents = torch.randn((num_samples, latent_size))\n",
    "labels = torch.randint(0, num_labels, (num_samples,))\n",
    "\n",
    "generated_samples = mixture_generator(random_latents, labels)\n",
    "\n",
    "print(generated_samples.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MixtureBertGan\n",
    "\n",
    "same logic, just used for correct loss calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MixtureBertGan(nn.Module):\n",
    "    def __init__(self, MixtureGenerator, Discriminator, Bert, fast_gen_grad=False, has_pooler=False, no_grad_bert=False):\n",
    "        super(MixtureBertGan, self).__init__()\n",
    "\n",
    "        self.generator = MixtureGenerator\n",
    "        self.discriminator = Discriminator\n",
    "        self.bert = Bert\n",
    "        num_labels = self.discriminator.k\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "        self.k = self.discriminator.k\n",
    "        self.epsilon = 1e-8\n",
    "\n",
    "        self.gen_log_grad = fast_gen_grad\n",
    "        self.has_pooler = has_pooler\n",
    "        self.no_grad_bert = no_grad_bert\n",
    "\n",
    "    def forward(self, inputs, z=None, labels=None, label_mask=None):\n",
    "        # Bert real data forward pass\n",
    "        bert_outputs = self.bert(**inputs)\n",
    "        x_r = bert_outputs.pooler_output if self.has_pooler else bert_outputs.last_hidden_state[:, 0]\n",
    "        if self.no_grad_bert:\n",
    "            x_r = x_r.detach()\n",
    "\n",
    "        # Discriminator forward pass on real data\n",
    "        D_real_prob, D_real_logits, D_real_features = self.discriminator(x_r)\n",
    "\n",
    "        # Supervised loss\n",
    "        real_logits = D_real_logits[:, :-1]\n",
    "        real_prob = F.softmax(real_logits, dim=-1)\n",
    "        d_loss_sup = 0\n",
    "        if labels is not None:\n",
    "            if label_mask is None:\n",
    "                label_mask = torch.ones_like(labels)\n",
    "            ce = F.cross_entropy(real_logits, labels, reduction='none')\n",
    "            masked_ce = ce * label_mask\n",
    "            d_loss_sup = masked_ce.sum() / max(label_mask.sum(), 1)\n",
    "\n",
    "        # Inference mode\n",
    "        if z is None:\n",
    "            # If latent (z) is None, skip the generator and set g_loss to None\n",
    "            d_loss = -torch.mean(\n",
    "                torch.log(1 - D_real_prob[:, -1] + self.epsilon)\n",
    "            ) + d_loss_sup\n",
    "            return {\n",
    "                'probs': real_prob,\n",
    "                'logits': real_logits,\n",
    "                'd_loss': d_loss,\n",
    "                'g_loss': None,\n",
    "            }\n",
    "\n",
    "        # Generator forward pass\n",
    "        x_g = self.generator(z, labels)\n",
    "\n",
    "        # Discriminator forward pass on fake data\n",
    "        D_fake_prob, D_fake_logits, D_fake_features = self.discriminator(x_g)\n",
    "\n",
    "        # Discriminator loss\n",
    "        # Unsupervised loss\n",
    "        d_loss_unsup = -torch.mean(\n",
    "            torch.log(1 - D_real_prob[:, -1] + self.epsilon)\n",
    "        ) - torch.mean(\n",
    "            torch.log(D_fake_prob[:, -1] + self.epsilon)\n",
    "        )\n",
    "\n",
    "        d_loss = d_loss_unsup + d_loss_sup\n",
    "\n",
    "        # Generator loss\n",
    "        g_loss_unsup = -torch.mean(\n",
    "            torch.log(1 - D_fake_prob[:, -1] + self.epsilon)\n",
    "            if not self.gen_log_grad else\n",
    "            - torch.log(D_fake_prob[:, -1] + self.epsilon)\n",
    "        )\n",
    "\n",
    "        g_loss_feature_match = 0.0\n",
    "\n",
    "        for label_idx in range(self.num_labels):\n",
    "            real_features_label = D_real_features[labels == label_idx]\n",
    "            fake_features_label = D_fake_features[labels == label_idx]\n",
    "\n",
    "            if len(real_features_label) > 0 and len(fake_features_label) > 0:\n",
    "                mean_real_features_label = torch.mean(real_features_label, dim=0)\n",
    "                mean_fake_features_label = torch.mean(fake_features_label, dim=0)\n",
    "\n",
    "                g_loss_feature_match += torch.mean((mean_real_features_label - mean_fake_features_label) ** 2)\n",
    "\n",
    "        g_loss = g_loss_unsup + g_loss_feature_match\n",
    "        output = {\n",
    "            'probs': real_prob,\n",
    "            'logits': real_logits,\n",
    "            'd_loss': d_loss,\n",
    "            'g_loss': g_loss,\n",
    "        }\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test\n",
    "bert, tokenizer = get_bert()\n",
    "bert_gan = MixtureBertGan(MixtureGenerator=mixture_generator, Discriminator=discriminator, Bert=bert)\n",
    "\n",
    "batch_size = 5\n",
    "latent_batch_size = 5\n",
    "random_latents = torch.randn((latent_batch_size, latent_size))\n",
    "random_inputs = tokenizer(\n",
    "    [\"ali nourian save me\"] * batch_size, return_tensors='pt', truncation=True, padding=\"max_length\"\n",
    ")\n",
    "random_labels = torch.randint(0, discriminator.k, (batch_size,))\n",
    "random_mask   = torch.randint(0, 2, (batch_size,))\n",
    "\n",
    "# Train\n",
    "output = bert_gan(random_inputs, random_latents, random_labels, random_mask)\n",
    "# Inference\n",
    "output_inf = bert_gan(random_inputs)\n",
    "\n",
    "print(\"Training:\\n\")\n",
    "print(\"Discriminator Loss:\", output['d_loss'].item())\n",
    "print(\"Generator Loss:\", output['g_loss'].item())\n",
    "print(\"Logits:\", output['logits'].shape)\n",
    "print(\"Probabilities:\", output['probs'].shape)\n",
    "print(\"\\nInference:\\n\")\n",
    "print(\"Discriminator Loss:\", output_inf['d_loss'].item())\n",
    "print(\"Generator Loss:\", output_inf['g_loss'])\n",
    "print(\"Logits:\", output_inf['logits'].shape)\n",
    "print(\"Probabilities:\", output_inf['probs'].shape)\n",
    "\n",
    "# Testing good grads\n",
    "output['g_loss'].backward(retain_graph=True)\n",
    "output['d_loss'].backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir mixture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training MixtureBertGan model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import get_linear_schedule_with_warmup, get_constant_schedule_with_warmup\n",
    "\n",
    "percentages = [1]\n",
    "metrics_by_percentage = {}\n",
    "\n",
    "for percentage in percentages:\n",
    "    print(f\"\"\"\n",
    "=================================================================================================\n",
    "\n",
    "                                Training on {100 * percentage} % of the dataset\n",
    "    \n",
    "=================================================================================================\n",
    "\n",
    "    \"\"\")\n",
    "    # Define model\n",
    "    bert, _ = get_bert(model=\"distil\")\n",
    "\n",
    "    generator = MixtureGenerator(\n",
    "        latent_size=latent_size, hidden_dim=hidden_dim, out_features=hidden_dim, num_labels=K,\n",
    "    )\n",
    "    discriminator = Discriminator(\n",
    "        k=K, in_features=hidden_dim\n",
    "    )\n",
    "    bertGan = MixtureBertGan(\n",
    "        generator, discriminator, bert, fast_gen_grad=False,# no_grad_bert=True,\n",
    "    )\n",
    "\n",
    "    if torch.cuda.device_count() > 1: # Since we are using different ootimizers\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bertGan.bert = nn.DataParallel(bertGan.bert)\n",
    "        bertGan.discriminator = nn.DataParallel(bertGan.discriminator)\n",
    "        bertGan.generator = nn.DataParallel(bertGan.generator)\n",
    "        \n",
    "    bertGan = bertGan.to(device)\n",
    "\n",
    "    # hyper-parameters\n",
    "    d_lr = 5e-4\n",
    "    g_lr = 5e-4\n",
    "    batch_size = 32\n",
    "    num_epochs = 3\n",
    "    warmup_ratio = 0.1\n",
    "    num_training_steps = int(train_size / batch_size * num_epochs)\n",
    "    num_warmup_steps   = int(num_training_steps * warmup_ratio)\n",
    "    \n",
    "    #optimizer\n",
    "    bert_optimizer = optim.AdamW(bertGan.bert.parameters(), lr=d_lr) \n",
    "    dis_optimizer = optim.AdamW(bertGan.discriminator.parameters(), lr=d_lr)\n",
    "    gen_optimizer = optim.AdamW(bertGan.generator.parameters(), lr=g_lr) \n",
    "    \n",
    "    #scheduler\n",
    "    bert_scheduler = get_linear_schedule_with_warmup(\n",
    "        bert_optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps,\n",
    "    )\n",
    "    dis_scheduler = None\n",
    "    gen_scheduler = None\n",
    "    \n",
    "    # Training\n",
    "    masked_train_loader, masked_val_loader = get_masked_loader(\n",
    "        train_path, percentage, full_dataset, train_size, val_size, batch_size=batch_size,\n",
    "    )\n",
    "    metrics = train_bertgan(\n",
    "        bertGan, \n",
    "        dis_optimizer, gen_optimizer, bert_optimizer,\n",
    "        dis_scheduler=dis_scheduler, gen_scheduler=gen_scheduler, bert_scheduler=bert_scheduler,\n",
    "        masked_train_loader=masked_train_loader, val_loader=masked_val_loader, n_critic=1, num_epochs=num_epochs,\n",
    "    )\n",
    "    \n",
    "    # Save model checkpoint\n",
    "    checkpoint = {\n",
    "        'percentage': percentage,\n",
    "        'model_state_dict': bertGan.state_dict(),\n",
    "        'metrics': metrics,\n",
    "        'disc_optimizer_state_dict': dis_optimizer.state_dict(),\n",
    "        'gen_optimizer_state_dict': gen_optimizer.state_dict(),\n",
    "    }\n",
    "    \n",
    "    torch.save(checkpoint, f'./mixture/model_checkpoint_{int(percentage * 100)}%.pth')\n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "percentages = list(metrics_by_percentage.keys())\n",
    "\n",
    "fig, axs = plt.subplots(1, 2, figsize=(12, 5))\n",
    "\n",
    "# Plot training and validation loss\n",
    "axs[0].plot(percentages, [metrics['train_loss'][-1] for metrics in metrics_by_percentage.values()], label='Train Loss')\n",
    "axs[0].plot(percentages, [metrics['val_loss'][-1] for metrics in metrics_by_percentage.values()], label='Validation Loss')\n",
    "axs[0].set_title(\"Loss as a function of dataset percentage\")\n",
    "axs[0].set_xlabel('Percentage of the dataset')\n",
    "axs[0].set_ylabel('Loss')\n",
    "axs[0].legend()\n",
    "\n",
    "# Plot training and validation accuracy\n",
    "axs[1].plot(percentages, [metrics['train_acc'][-1] * 100 for metrics in metrics_by_percentage.values()], label='Train Accuracy')\n",
    "axs[1].plot(percentages, [metrics['val_acc'][-1] * 100 for metrics in metrics_by_percentage.values()], label='Validation Accuracy')\n",
    "axs[1].set_title(\"Accuracy as a function of dataset percentage\")\n",
    "axs[1].set_xlabel('Percentage of the dataset')\n",
    "axs[1].set_ylabel('Accuracy (%)')\n",
    "axs[1].legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "percentages = [1]\n",
    "acc_per_percent = {}\n",
    "for percentage in percentages:\n",
    "    # Define model\n",
    "    bert, _ = get_bert(model=\"distil\")\n",
    "\n",
    "    generator = MixtureGenerator(\n",
    "        latent_size=latent_size, hidden_dim=hidden_dim, out_features=hidden_dim, num_labels=K,\n",
    "    )\n",
    "    discriminator = Discriminator(\n",
    "        k=K, in_features=hidden_dim\n",
    "    )\n",
    "    bertgan = MixtureBertGan(\n",
    "        generator, discriminator, bert, fast_gen_grad=False,# no_grad_bert=True,\n",
    "    )\n",
    "\n",
    "    if torch.cuda.device_count() > 1: # Since we are using different ootimizers\n",
    "        print(\"using\", torch.cuda.device_count(), \"GPUs!\")\n",
    "        bertgan.bert = nn.DataParallel(bertgan.bert)\n",
    "        bertgan.discriminator = nn.DataParallel(bertgan.discriminator)\n",
    "        bertgan.generator = nn.DataParallel(bertgan.generator)\n",
    "        \n",
    "    bertgan = bertgan.to(device)\n",
    "    \n",
    "    metrics = {'loss': [], 'accuracy': []}\n",
    "    # Load model\n",
    "    path = f'./mixture/model_checkpoint_{int(percentage * 100)}%.pth'\n",
    "    checkpoint = torch.load(path)\n",
    "    bertgan_state_dict = checkpoint['model_state_dict']\n",
    "    bertgan.load_state_dict(bertgan_state_dict)\n",
    "    # Testing\n",
    "    bertgan.eval()\n",
    "    test_loss = 0.0\n",
    "    all_predicts_test = np.array([])\n",
    "    all_labels_test = np.array([])\n",
    "    with torch.no_grad():\n",
    "        for inputs, labels, token in tqdm(test_loader, desc=f'Testing'):\n",
    "            token, labels = token.to(device), labels.to(device)\n",
    "            outputs = bertgan(token, labels=labels)\n",
    "            loss = outputs['d_loss']\n",
    "\n",
    "            test_loss += loss.item()\n",
    "\n",
    "            # Calculate accuracy\n",
    "            _, predicted = torch.max(outputs['probs'], 1)\n",
    "            all_predicts_test = np.append(all_predicts_test, predicted.detach().cpu().numpy())\n",
    "            all_labels_test = np.append(all_labels_test, labels.detach().cpu().numpy())\n",
    "\n",
    "    test_acc = (all_predicts_test == all_labels_test).sum().item() / len(all_predicts_test)\n",
    "\n",
    "    metrics['loss'].append(test_loss / len(test_loader))\n",
    "    metrics['accuracy'].append(test_acc)\n",
    "    \n",
    "    metrics_by_percentage[f\"{int(percentage * 100)}%\"] = metrics\n",
    "    \n",
    "metrics_by_percentage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gc\n",
    "torch.cuda.empty_cache()\n",
    "del bertGan , bert\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data distribution analysis\n",
    "\n",
    "in this section we analyse the distribution of the test and train datasets, we claim that the datasets are not equivalent, hence why we are not getting good results from the test sets, even though we are getting good results on the cross validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding principal components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "\n",
    "dist_train_val_loader = get_loader(full_dataset, batch_size=16)\n",
    "dist_test_loader = get_loader(test_dataset, batch_size=16)\n",
    "\n",
    "num_labels = K\n",
    "\n",
    "all_embeddings_train = {label: [] for label in range(num_labels)}\n",
    "all_embeddings_test = {label: [] for label in range(num_labels)}\n",
    "\n",
    "with torch.no_grad():\n",
    "    for _, labels, token in tqdm(dist_train_val_loader, desc='finding train set distribution'):\n",
    "        token = token.to(device)\n",
    "        embedding = bert(**token).last_hidden_state[:, 0].detach().cpu().numpy()\n",
    "\n",
    "        for label, emb in zip(labels.numpy(), embedding):\n",
    "            all_embeddings_train[label].append(emb)\n",
    "            \n",
    "        break\n",
    "with torch.no_grad():\n",
    "    for _, labels, token in tqdm(dist_test_loader, desc='finding test set distribution'):\n",
    "        token = token.to(device)\n",
    "        embedding = bert(**token).last_hidden_state[:, 0].detach().cpu().numpy()\n",
    "\n",
    "        for label, emb in zip(labels.numpy(), embedding):\n",
    "            all_embeddings_test[label].append(emb)\n",
    "            \n",
    "            \n",
    "\n",
    "# Convert the lists of embeddings to tensors\n",
    "for label in range(num_labels):\n",
    "    all_embeddings_train[label] = torch.from_numpy(np.array(all_embeddings_train[label]))\n",
    "    all_embeddings_test[label] = torch.from_numpy(np.array(all_embeddings_test[label]))\n",
    "    \n",
    "pca_train = {label: PCA(n_components=2).fit_transform(all_embeddings_train[label]) for label in range(num_labels)}\n",
    "pca_test = {label: PCA(n_components=2).fit_transform(all_embeddings_test[label]) for label in range(num_labels)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distributions for each class overlayed in subplots\n",
    "plt.figure(figsize=(12, 9))\n",
    "\n",
    "# Plot for each class\n",
    "for label in range(num_labels):\n",
    "    plt.subplot(2, 3, label + 1) \n",
    "    plt.scatter(pca_train[label][:, 0], pca_train[label][:, 1], label='Train Set', alpha=0.5)\n",
    "    plt.scatter(pca_test[label][:, 0], pca_test[label][:, 1], label='Test Set', alpha=0.5)\n",
    "    plt.xlabel('Principal Component 1')\n",
    "    plt.ylabel('Principal Component 2')\n",
    "    plt.title(f'Distribution of Embeddings (Class {label})')\n",
    "    plt.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as seen from the results, the two datasets clearly have different distributions thus explaining why our models are learning the task well on the training and validation datasets,\n",
    "but not on the test/dev dataset."
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "nvidiaTeslaT4",
   "dataSources": [],
   "dockerImageVersionId": 30635,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}