08_introduction_to_nlp_in_tensorflow.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Blurr99/tensorflow_deepLearning/blob/main/08_introduction_to_nlp_in_tensorflow.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ayTVo1r1Nbge"
      },
      "source": [
        "# Introduction to NLP Fundamentals in TensorFlow\n",
        "\n",
        "NLP has the goal of deriving information out of natural language (could be sequences, text or speech)\n",
        "\n",
        "Another common term for NLP problems is sequence to sequence problems (seq2seq)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7asIQln9N6sK"
      },
      "source": [
        "## Check for GPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nTy399kSOBsC",
        "outputId": "beb83c07-7b9d-463b-b6f6-6a4040fb5cc3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU 0: Tesla T4 (UUID: GPU-df6bf92f-cacb-b40c-a7e1-b9c5ce3b5293)\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi -L"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCxgTOurOMgl"
      },
      "source": [
        "## Getting helper function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pDOgdjlnOVQE",
        "outputId": "46b345a9-a6dc-4ac4-96dd-6e9faeaff3fa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-02-20 04:49:02--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10246 (10K) [text/plain]\n",
            "Saving to: ‘helper_functions.py’\n",
            "\n",
            "\rhelper_functions.py   0%[                    ]       0  --.-KB/s               \rhelper_functions.py 100%[===================>]  10.01K  --.-KB/s    in 0s      \n",
            "\n",
            "2024-02-20 04:49:02 (97.8 MB/s) - ‘helper_functions.py’ saved [10246/10246]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
        "\n",
        "# Import a series of helper functions for the notebook\n",
        "from helper_functions import unzip_data, create_tensorboard_callback, plot_loss_curves, compare_historys"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LYCTCmf4OzGS"
      },
      "source": [
        "## Get the  text dataset\n",
        "\n",
        "The dataset that is going to be used in this notebook is Kaggle's introduction to NLP dataset (text samples of Tweets labelled as disaster or not disaster).\n",
        "> LINK: https://www.kaggle.com/competitions/nlp-getting-started"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "usrL3SEdPIhy",
        "outputId": "a5b3493b-ccbb-4d18-f5ef-e77d303f3ed5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-02-20 04:49:08--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.207, 74.125.137.207, 2607:f8b0:4023:c03::cf, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.207|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip.1’\n",
            "\n",
            "\r          nlp_getti   0%[                    ]       0  --.-KB/s               \rnlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.004s  \n",
            "\n",
            "2024-02-20 04:49:08 (135 MB/s) - ‘nlp_getting_started.zip.1’ saved [607343/607343]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
        "\n",
        "# Unzip the data\n",
        "unzip_data(\"nlp_getting_started.zip\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHOgJHFuP5jq"
      },
      "source": [
        "## Visualize the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "US_spsyHQiqP",
        "outputId": "31ef2723-9880-4ab1-beef-d7714b4e1daa"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   id keyword location                                               text  \\\n",
              "0   1     NaN      NaN  Our Deeds are the Reason of this #earthquake M...   \n",
              "1   4     NaN      NaN             Forest fire near La Ronge Sask. Canada   \n",
              "2   5     NaN      NaN  All residents asked to 'shelter in place' are ...   \n",
              "3   6     NaN      NaN  13,000 people receive #wildfires evacuation or...   \n",
              "4   7     NaN      NaN  Just got sent this photo from Ruby #Alaska as ...   \n",
              "\n",
              "   target  \n",
              "0       1  \n",
              "1       1  \n",
              "2       1  \n",
              "3       1  \n",
              "4       1  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3e98e101-a369-4888-9241-d6a50d477bc6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3e98e101-a369-4888-9241-d6a50d477bc6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3e98e101-a369-4888-9241-d6a50d477bc6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3e98e101-a369-4888-9241-d6a50d477bc6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-328c29e2-8cd3-4ee6-a18a-e6ebdc035a83\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-328c29e2-8cd3-4ee6-a18a-e6ebdc035a83')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-328c29e2-8cd3-4ee6-a18a-e6ebdc035a83 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "train_df",
              "summary": "{\n  \"name\": \"train_df\",\n  \"rows\": 7613,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3137,\n        \"min\": 1,\n        \"max\": 10873,\n        \"samples\": [\n          3796,\n          3185,\n          7769\n        ],\n        \"num_unique_values\": 7613,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"keyword\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"injury\",\n          \"nuclear%20reactor\",\n          \"engulfed\"\n        ],\n        \"num_unique_values\": 221,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"Oklahoma\",\n          \"Starling City\",\n          \"Trinidad and Tobago\"\n        ],\n        \"num_unique_values\": 3341,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"Three Homes Demolished in Unrecognized Arab Village - International Middle East Media Center http://t.co/ik8m4Yi9T4\",\n          \"Reid Lake fire prompts campground evacuation order http://t.co/jBODKM6rBU\",\n          \"FAAN orders evacuation of abandoned aircraft at MMA http://t.co/dEvYbnVXGQ via @todayng\"\n        ],\n        \"num_unique_values\": 7503,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "train_df = pd.read_csv(\"/content/train.csv\")\n",
        "test_df = pd.read_csv(\"/content/test.csv\")\n",
        "\n",
        "train_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "juGptPHJRG4N",
        "outputId": "9b9fd7b2-96a6-4dbd-b427-f4ebad822cfc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Forest fire near La Ronge Sask. Canada'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "train_df[\"text\"][1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ZaQEH5yKRRNu",
        "outputId": "cf1cfa1f-6e4b-4a3a-8ac5-050d30fe8739"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        id      keyword               location  \\\n",
              "2644  3796  destruction                    NaN   \n",
              "2227  3185       deluge                    NaN   \n",
              "5448  7769       police                     UK   \n",
              "132    191   aftershock                    NaN   \n",
              "6845  9810       trauma  Montgomery County, MD   \n",
              "\n",
              "                                                   text  target  \n",
              "2644  So you have a new weapon that can cause un-ima...       1  \n",
              "2227  The f$&amp;@ing things I do for #GISHWHES Just...       0  \n",
              "5448  DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...       1  \n",
              "132   Aftershock back to school kick off was great. ...       0  \n",
              "6845  in response to trauma Children of Addicts deve...       0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cf44d146-56ff-4ece-92e1-d2f61d8ae1f2\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2644</th>\n",
              "      <td>3796</td>\n",
              "      <td>destruction</td>\n",
              "      <td>NaN</td>\n",
              "      <td>So you have a new weapon that can cause un-ima...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2227</th>\n",
              "      <td>3185</td>\n",
              "      <td>deluge</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The f$&amp;amp;@ing things I do for #GISHWHES Just...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5448</th>\n",
              "      <td>7769</td>\n",
              "      <td>police</td>\n",
              "      <td>UK</td>\n",
              "      <td>DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>191</td>\n",
              "      <td>aftershock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Aftershock back to school kick off was great. ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6845</th>\n",
              "      <td>9810</td>\n",
              "      <td>trauma</td>\n",
              "      <td>Montgomery County, MD</td>\n",
              "      <td>in response to trauma Children of Addicts deve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cf44d146-56ff-4ece-92e1-d2f61d8ae1f2')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-cf44d146-56ff-4ece-92e1-d2f61d8ae1f2 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-cf44d146-56ff-4ece-92e1-d2f61d8ae1f2');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8b2e2431-0d03-4774-b2bd-c65ca5619339\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8b2e2431-0d03-4774-b2bd-c65ca5619339')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8b2e2431-0d03-4774-b2bd-c65ca5619339 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "train_df_shuffled",
              "summary": "{\n  \"name\": \"train_df_shuffled\",\n  \"rows\": 7613,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3137,\n        \"min\": 1,\n        \"max\": 10873,\n        \"samples\": [\n          7061,\n          843,\n          10603\n        ],\n        \"num_unique_values\": 7613,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"keyword\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"blazing\",\n          \"emergency\",\n          \"rescue\"\n        ],\n        \"num_unique_values\": 221,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"617-BTOWN-BEATDOWN\",\n          \"Newcastle Upon Tyne, England\",\n          \"Federal Capital Territory\"\n        ],\n        \"num_unique_values\": 3341,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"This week first responders and DART members are participating in a four day intensive Technical Large Animal... http://t.co/tL93AOd3ER\",\n          \"My mom is watching a show about bridges breaking/falling and the people on them drowning in their cars aka one of my biggest fears ????\",\n          \"@NickLee8  i went to school in a bombed out East End of London3 families to one house no bathroom outside loo &amp; poor so whats yr point\"\n        ],\n        \"num_unique_values\": 7503,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "# Shuffle the train dataset\n",
        "train_df_shuffled = train_df.sample(frac = 1, random_state = 42)\n",
        "train_df_shuffled.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "8-G-tWoiSIGu",
        "outputId": "dabccad5-b4e3-4718-a89b-0aa3b03a4181"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3d0aee28-686b-4206-9be6-cebb7f584c5e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3d0aee28-686b-4206-9be6-cebb7f584c5e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-3d0aee28-686b-4206-9be6-cebb7f584c5e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-3d0aee28-686b-4206-9be6-cebb7f584c5e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b1d47226-e38a-4101-969a-8d5b41b1cca5\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b1d47226-e38a-4101-969a-8d5b41b1cca5')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b1d47226-e38a-4101-969a-8d5b41b1cca5 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "test_df",
              "summary": "{\n  \"name\": \"test_df\",\n  \"rows\": 3263,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3146,\n        \"min\": 0,\n        \"max\": 10875,\n        \"samples\": [\n          8051,\n          425,\n          1330\n        ],\n        \"num_unique_values\": 3263,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"keyword\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"injury\",\n          \"nuclear%20reactor\",\n          \"engulfed\"\n        ],\n        \"num_unique_values\": 221,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"UAE\",\n          \"Tokio / Tokyo\",\n          \"Texas\"\n        ],\n        \"num_unique_values\": 1602,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"Latest: USA: Huge sinkhole swallows up Brooklyn intersection http://t.co/vspKHg3nZy\",\n          \"I liked a @YouTube video http://t.co/a5YTAw9Vih S.O.S. Rona Guide - The Red Whirlwind\",\n          \"HitchBot travels Europe and greeted with open arms. Gets destroyed after two weeks in america. There's a lesson to be learned here.\"\n        ],\n        \"num_unique_values\": 3243,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "# Structure of the test dataset\n",
        "test_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mpgGMifpSROD",
        "outputId": "b840fde1-d3be-4af1-dde5-9bcba425a4a0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# Number of examples of each class\n",
        "train_df.target.value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5PZ68-atSbCI",
        "outputId": "67708583-5bd5-4306-edfc-a7381b5292cd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(7613, 3263)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "# Number of total samples\n",
        "len(train_df), len(test_df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HXPl_Y3gSxta",
        "outputId": "92567a7d-d8bf-4ecc-f28d-b651e39520f5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 0 (not a real disaster)\n",
            "Text: \n",
            "Roger Goodell's Fatal Mistake: Tom Brady An Innocent Man http://t.co/UCNcKrnLow\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not a real disaster)\n",
            "Text: \n",
            ".POTUS #StrategicPatience is a strategy for #Genocide; refugees; IDP Internally displaced people; horror; etc. https://t.co/rqWuoy1fm4\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not a real disaster)\n",
            "Text: \n",
            "@BrandonMulcahy @fpine Here's the story http://t.co/TgXutUoyHl\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real disaster)\n",
            "Text: \n",
            "Young dancer moves about 300 youth in attendance at the GMMBC Youth Explosion this past Saturday. Inspiring! http://t.co/TMmOrvxsWz\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real disaster)\n",
            "Text: \n",
            "SpaceX Founder Musk: Structural Failure Took Down Falcon 9 http://t.co/LvIzO9CSSR\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Visualizing some random training examples\n",
        "import random\n",
        "random_index = random.randint(0, len(train_df)-5)\n",
        "for row in train_df_shuffled[[\"text\", \"target\"]][random_index : random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target: {target}\", \"(real disaster)\" if target > 0 else \"(not a real disaster)\")\n",
        "  print(f\"Text: \\n{text}\\n\")\n",
        "  print(\"---\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HdFf2LuNT8YX"
      },
      "source": [
        "### Splitting the data into train and validation sets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NbwOeFmtlf16"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_sentences, val_sentences,  train_labels, val_labels = train_test_split(train_df_shuffled[\"text\"].to_numpy(),\n",
        "                                                                             train_df_shuffled[\"target\"].to_numpy(),\n",
        "                                                                             test_size = 0.1,\n",
        "                                                                             random_state = 42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o5NTgq2qmQWU",
        "outputId": "c76eaa1f-cfa3-475d-b4a0-016c924ecaf1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6851, 6851, 762, 762)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "# Checking the lengths\n",
        "len(train_sentences), len(train_labels), len(val_sentences), len(val_labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f6ehQ_RTmYF9",
        "outputId": "eb78fdf4-19cb-4931-ed51-47f82f498098"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "('@mogacola @zamtriossu i screamed after hitting tweet', 0)\n",
            "\n",
            "('Imagine getting flattened by Kurt Zouma', 0)\n",
            "\n",
            "('@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....', 1)\n",
            "\n",
            "(\"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\", 0)\n",
            "\n",
            "('Somehow find you and I collide http://t.co/Ee8RpOahPk', 0)\n",
            "\n",
            "('@EvaHanderek @MarleyKnysh great times until the bus driver held us hostage in the mall parking lot lmfao', 1)\n",
            "\n",
            "('destroy the free fandom honestly', 1)\n",
            "\n",
            "('Weapons stolen from National Guard Armory in New Albany still missing #Gunsense http://t.co/lKNU8902JE', 0)\n",
            "\n",
            "('@wfaaweather Pete when will the heat wave pass? Is it really going to be mid month? Frisco Boy Scouts have a canoe trip in Okla.', 1)\n",
            "\n",
            "('Patient-reported outcomes in long-term survivors of metastatic colorectal cancer - British Journal of Surgery http://t.co/5Yl4DC1Tqt', 1)\n",
            "\n",
            "('Lava Dragon Breeder! I just bred a Lava Dragon in DragonVale! Visit my park to check it out! http://t.co/QGum9xHEOs', 0)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for i in range(11):\n",
        "  print(f\"{train_sentences[i], train_labels[i]}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OthVDAH9mf5C"
      },
      "source": [
        "## Converting text into numbers\n",
        "\n",
        "When working with a text problem, one of the first things that needs to be done is to convert the text into numbers.\n",
        "\n",
        "Ways to do this:\n",
        "* Tokenization: direct mapping of token (a token can be a work or a character) to number.\n",
        "* Embedding: create a matrix of feature vector of each token (the size of the feature vector can be defined and this embedidng can be learned)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vbSK6fPl3Y4-"
      },
      "source": [
        "### Text vectorization (tokenzation)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DSBM5pdO5U67"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "# Using the default parameters\n",
        "text_vectorization = TextVectorization(max_tokens = None,\n",
        "                                       standardize = \"lower_and_strip_punctuation\",\n",
        "                                       split = \"whitespace\",\n",
        "                                       ngrams = None,\n",
        "                                       output_mode = \"int\",\n",
        "                                       output_sequence_length = None,\n",
        "                                       pad_to_max_tokens = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tFcBa_Rh569p",
        "outputId": "3727ed84-02f7-422a-8011-4b458e37016e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HsLK2rqJV_ml"
      },
      "outputs": [],
      "source": [
        "# Setting up TextVectorization variables\n",
        "max_vocab_length = 10000 # max number of words in the vocabulary\n",
        "max_length = 15 # max number of words the sequences will be\n",
        "\n",
        "text_vectorization = TextVectorization(max_tokens = max_vocab_length,\n",
        "                                       output_mode = \"int\",\n",
        "                                       output_sequence_length = max_length)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I0QqCKgeWy4N"
      },
      "outputs": [],
      "source": [
        "# Fit the text vectorizer to the training data\n",
        "text_vectorization.adapt(train_sentences)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VfJxpI3hXnOw",
        "outputId": "dd591e7a-8a2a-45bc-8b05-d4814156bc59"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[264,   3, 232,  17,  13, 698,   0,   0,   0,   0,   0,   0,   0,\n",
              "          0,   0]])>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "# Create a sample sentence and vectorize it\n",
        "sample_sentence = \"Theres a flood at my street!\"\n",
        "text_vectorization([sample_sentence])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rMPgoSQNX4V_",
        "outputId": "9e0d32f2-9c7e-40c1-93f6-115a384021b6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original Text: \n",
            "I'm awful at painting.. why did I agree to do an A3 landscape in bloody oils of all paints ??      \n",
            "\n",
            " Vectorized Version: \n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[  32, 2186,   17, 2815,   91,  145,    8, 1331,    5,   68,   39,\n",
              "           1, 5252,    4,  261]])>"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "# Choose a random sentence from the training data and tokenize it\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original Text: \\n{random_sentence}\\\n",
        "      \\n\\n Vectorized Version: \\n\")\n",
        "text_vectorization([random_sentence])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lGmEF_iGYryy",
        "outputId": "ec2f455b-74c4-4b61-e6b9-87399010b50d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of words in vocab: 10000\n",
            "5 most common words: ['', '[UNK]', 'the', 'a', 'in']\n",
            "5 least common words: ['pages', 'paeds', 'pads', 'padres', 'paddytomlinson1']\n"
          ]
        }
      ],
      "source": [
        "# Get the unique words in the vocabulary\n",
        "words_in_vocab = text_vectorization.get_vocabulary()\n",
        "top_5_words = words_in_vocab[:5]\n",
        "bottom_5_words = words_in_vocab[-5:]\n",
        "print(f\"Number of words in vocab: {len(words_in_vocab)}\")\n",
        "print(f\"5 most common words: {top_5_words}\")\n",
        "print(f\"5 least common words: {bottom_5_words}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSzIbeoAZaTz"
      },
      "source": [
        "### Creating an embedding using an Embedding Layer\n",
        "Most important parameters:\n",
        "* `input_dim` = the size of the vocabulary\n",
        "* `output_dim` = the size of the output embedding vector, for e.g., a value of 100 would mean each token gets represented by a vector 100 long.\n",
        "* `input_length` = length of sequences being passed to embedding layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mM7rnpbRadLe",
        "outputId": "6dc040c3-5564-4603-9b46-ff0832356423"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.src.layers.core.embedding.Embedding at 0x7e843d360370>"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "embedding = layers.Embedding(input_dim = max_vocab_length,\n",
        "                             output_dim = 128,\n",
        "                             embeddings_initializer = \"uniform\",\n",
        "                             input_length = max_length,\n",
        "                             )\n",
        "embedding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dU66KCuWeJTp",
        "outputId": "ba2c4d66-a50a-4c8d-f9de-2026723f3655"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original Text: \n",
            " this is about to be a bomb ass firework picture http://t.co/lr4BTvuEoM        \n",
            "\n",
            " Embedded Version: \n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              "array([[[-0.00523932, -0.03772547, -0.04767262, ..., -0.02875663,\n",
              "         -0.02041202,  0.02420417],\n",
              "        [-0.00016909, -0.0418968 , -0.0444315 , ..., -0.04619812,\n",
              "         -0.03843952, -0.0309157 ],\n",
              "        [-0.01654409,  0.01404966, -0.01004831, ...,  0.02406291,\n",
              "          0.00199614, -0.01266981],\n",
              "        ...,\n",
              "        [ 0.04053747,  0.04513217,  0.03193649, ..., -0.03861039,\n",
              "          0.00243799,  0.02379228],\n",
              "        [ 0.04053747,  0.04513217,  0.03193649, ..., -0.03861039,\n",
              "          0.00243799,  0.02379228],\n",
              "        [ 0.04053747,  0.04513217,  0.03193649, ..., -0.03861039,\n",
              "          0.00243799,  0.02379228]]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "# Get a random sentence from the training set\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original Text: \\n {random_sentence}\\\n",
        "        \\n\\n Embedded Version: \\n\")\n",
        "\n",
        "# Embed the random sentence into dense vectors of fixed size\n",
        "sample_embed = embedding(text_vectorization([random_sentence]))\n",
        "sample_embed"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eSBh2Q8zeunQ",
        "outputId": "2151a87d-5ca9-41bf-b133-033cb47070da"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(128,), dtype=float32, numpy=\n",
              " array([ 0.0350269 ,  0.04763344,  0.04934568,  0.0034021 ,  0.03701004,\n",
              "        -0.04460923, -0.03470205,  0.03025014, -0.04959337,  0.04013282,\n",
              "         0.01214186, -0.02739686,  0.00498893, -0.00905039,  0.01669471,\n",
              "         0.03963857, -0.03075154, -0.00073842, -0.01363566, -0.02480757,\n",
              "        -0.01684182,  0.0208205 , -0.00849099,  0.03930781, -0.01727662,\n",
              "        -0.01563343, -0.0467287 , -0.00870993,  0.025607  , -0.03916845,\n",
              "         0.01945375, -0.03604817,  0.02880149,  0.01160804, -0.04446768,\n",
              "        -0.02362078,  0.03859422,  0.01201778, -0.00110849,  0.00805342,\n",
              "        -0.03543192, -0.0373893 ,  0.01997868,  0.0027064 ,  0.01861105,\n",
              "        -0.01578895, -0.01857527, -0.02942636, -0.0159477 ,  0.02946586,\n",
              "         0.02896955,  0.04428423, -0.00286146,  0.01835997,  0.01665803,\n",
              "         0.0400846 ,  0.03139669,  0.0209065 ,  0.02690455,  0.01836878,\n",
              "        -0.03138734, -0.04929837,  0.01571831, -0.01214593, -0.0223027 ,\n",
              "         0.03687838,  0.03395427,  0.01955571, -0.01431947, -0.01276154,\n",
              "         0.00720291, -0.02933925,  0.00034816,  0.00971787, -0.02015063,\n",
              "         0.0086964 , -0.01440753,  0.04419701,  0.03343232, -0.00470984,\n",
              "         0.02299437, -0.049084  ,  0.00173912,  0.00375051, -0.04655395,\n",
              "        -0.04680942,  0.01328206, -0.04658737,  0.00106432, -0.00160247,\n",
              "        -0.00351219,  0.04732225, -0.04242171, -0.0335142 , -0.00590802,\n",
              "         0.03436777, -0.01913055, -0.00176324, -0.02904797, -0.01812392,\n",
              "        -0.04786165,  0.00941641,  0.01644273,  0.04093572, -0.02230378,\n",
              "        -0.02012756, -0.0171454 ,  0.013727  , -0.047542  , -0.0483891 ,\n",
              "        -0.04361072,  0.0028353 , -0.0444309 ,  0.02246484, -0.02632786,\n",
              "         0.03786108, -0.02695428, -0.04461036,  0.01024354,  0.04559335,\n",
              "        -0.00395872,  0.01212181,  0.02000136, -0.00685424, -0.04779175,\n",
              "         0.0448277 , -0.02549235, -0.04406649], dtype=float32)>,\n",
              " TensorShape([128]),\n",
              " \"Kakeru Teduka: Bfore 70years of today in Hiroshima it's exploded the one atomic bomd. It is so sad day.http://t.co/8Vzl1ns2iO\")"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "# Checking out a single token's embedding\n",
        "sample_embed[0][0], sample_embed[0][0].shape, random_sentence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAzZct5ufH7n"
      },
      "source": [
        "## Modelling a text dataset (running a series of experiments)\n",
        "\n",
        "* Model 0: Naive Bayes (Baseline)\n",
        "* Model 1: Feed forward neural network (dense model)\n",
        "* Model 2: LSTM model (RNN)\n",
        "* Model 3: GRU model (RNN)\n",
        "* Model 4: Bidirectional-LSTM model (RNN)\n",
        "* Model 5: 1D ConvolutionalNeural Network (CNN)\n",
        "* Model 6: TensorFlow Hub pretrained Feature Extractor (using transfer learning for NLP)\n",
        "* Model 7: Same as Model 6 with 10% of training data\n",
        "\n",
        "\n",
        "Steps to approach:\n",
        "* Create a model\n",
        "* Build a model\n",
        "* Fit a model\n",
        "* Evaluate a model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sLtb-gAakDzJ"
      },
      "source": [
        "### Model 0: Getting a baseline\n",
        "\n",
        "Creating a baseline model so there's a benchmark for future experiments to build upon.\n",
        "\n",
        "Using sklearn's Multinomial Naive Bayes for creating a baseline using the TF-IDF formula to convert the words to numbers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "7fBncCgDkSPi",
        "outputId": "9f5afe52-619e-42d1-e162-f81012de8116"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('tfidf', TfidfVectorizer()), ('clf', MultinomialNB())])"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;, TfidfVectorizer()), (&#x27;clf&#x27;, MultinomialNB())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;tfidf&#x27;, TfidfVectorizer()), (&#x27;clf&#x27;, MultinomialNB())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MultinomialNB</label><div class=\"sk-toggleable__content\"><pre>MultinomialNB()</pre></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "model_0 = Pipeline([\n",
        "    (\"tfidf\", TfidfVectorizer()),\n",
        "    (\"clf\", MultinomialNB())\n",
        "])\n",
        "\n",
        "# Fit the pipeline to the training data\n",
        "model_0.fit(train_sentences, train_labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fh5M27qQm9ID",
        "outputId": "b6b393ff-b7ee-426d-d6fc-02007d58bedf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The baseline model achieves an accuracy of: 79.27%\n"
          ]
        }
      ],
      "source": [
        "# Evaluating the baseline model\n",
        "baseline_score = model_0.score(val_sentences, val_labels)\n",
        "print(f\"The baseline model achieves an accuracy of: {baseline_score*100:.2f}%\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uImtZAtTplX1",
        "outputId": "97e461bd-0f42-4d3a-d261-52bcd97238d2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ],
      "source": [
        "train_df.target.value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZXjgSLN3pqXY",
        "outputId": "69b4aecf-8f7e-4f9e-c0b1-ee628a928a91"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ],
      "source": [
        "# Making predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds[:20]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JE3tKJX8rWIf"
      },
      "source": [
        "### Creating an evaluation function for a model's experiments\n",
        "\n",
        "Creating a function to compare a model's predictions with the turth labels using the following metrics:\n",
        "* Accuracy\n",
        "* Precision\n",
        "* Recall\n",
        "* F1 score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ctR-McJ5r3mf"
      },
      "outputs": [],
      "source": [
        "def get_evaluation_dict(model):\n",
        "  dict = {}\n",
        "  y_preds = model.predict(val_sentences)\n",
        "  dict[\"Accuracy\"] = f\"{accuracy_score(val_labels, y_preds)*100:.2f}%\"\n",
        "  dict[\"Precision\"] = precision_score(val_labels, y_preds)\n",
        "  dict[\"Recall\"] = recall_score(val_labels, y_preds)\n",
        "  dict[\"F1-score\"] = f1_score(val_labels, y_preds)\n",
        "\n",
        "  return dict"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ndge7bXOsb0S"
      },
      "outputs": [],
      "source": [
        "# Function to evaluate: accuracy, precision, recall, f1-score\n",
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def calculate_results(y_true, y_pred):\n",
        "  \"\"\"\n",
        "    Calculates model accuracy, precision, recall and f1-score for binary classification model.\n",
        "  \"\"\"\n",
        "\n",
        "  # Calculate model accuracy\n",
        "  model_accuracy = accuracy_score(y_true, y_pred) * 100\n",
        "  # Calculate model precision, recall, f1-score using \"weighted average\"\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average = \"weighted\")\n",
        "  model_results = {\n",
        "      \"Accuracy\" : model_accuracy,\n",
        "      \"Precision\" : model_precision,\n",
        "      \"Recall\" : model_recall,\n",
        "      \"F1 score\" : model_f1\n",
        "  }\n",
        "\n",
        "  return model_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "om747b8CuHMh",
        "outputId": "87497fd1-dd1b-43c6-c2a0-ffecb5f074f8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 79.26509186351706,\n",
              " 'Precision': 0.8111390004213173,\n",
              " 'Recall': 0.7926509186351706,\n",
              " 'F1 score': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ],
      "source": [
        "baseline_results = calculate_results(val_labels, baseline_preds)\n",
        "baseline_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5jQujHkJukpn"
      },
      "source": [
        "### Model 1: A simple dense model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YwpSNDO9wS-a"
      },
      "outputs": [],
      "source": [
        "# Create a tensorboard callback\n",
        "from helper_functions import create_tensorboard_callback\n",
        "\n",
        "# Create a directory to save TensorBoard logs\n",
        "SAVE_DIR = \"model_logs\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "al9yDMkTxGgh"
      },
      "outputs": [],
      "source": [
        "# Build model with Functional API\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "inputs = layers.Input(shape = (1,), dtype = tf.string)\n",
        "x = text_vectorization(inputs)  # turn inuput text into numbers\n",
        "x = embedding(x)  # create an embedding of the numebrized inputs\n",
        "x = layers.GlobalAveragePooling1D()(x)\n",
        "outputs = layers.Dense(1, activation = \"sigmoid\")(x)\n",
        "model_1 = tf.keras.Model(inputs, outputs, name = \"model_1_dense\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kPPshjrzx0GT",
        "outputId": "13111349-3b5a-40f8-d73a-1539286b08ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " global_average_pooling1d (  (None, 128)               0         \n",
            " GlobalAveragePooling1D)                                         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1280129 (4.88 MB)\n",
            "Trainable params: 1280129 (4.88 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model_1.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QhyBpPwGx7qb"
      },
      "outputs": [],
      "source": [
        "# Compile the model\n",
        "model_1.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cLWqGiGiyUgM",
        "outputId": "129a5b0a-2bd4-495f-96d8-23bab5ace26e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_1_dense/20240219-085911\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 20s 74ms/step - loss: 0.6150 - accuracy: 0.6920 - val_loss: 0.5390 - val_accuracy: 0.7507\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 20ms/step - loss: 0.4425 - accuracy: 0.8192 - val_loss: 0.4677 - val_accuracy: 0.7861\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.3473 - accuracy: 0.8600 - val_loss: 0.4569 - val_accuracy: 0.7900\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.2845 - accuracy: 0.8905 - val_loss: 0.4655 - val_accuracy: 0.7874\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.2380 - accuracy: 0.9114 - val_loss: 0.4777 - val_accuracy: 0.7782\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_1_history = model_1.fit(x = train_sentences,\n",
        "                              y = train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(dir_name = SAVE_DIR,\n",
        "                                                                       experiment_name = \"model_1_dense\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JDdZNeoLyxar",
        "outputId": "22626327-a47d-4816-a175-7c5ae3e28940"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 3ms/step - loss: 0.4777 - accuracy: 0.7782\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4776516556739807, 0.778215229511261]"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ],
      "source": [
        "# Checking the results\n",
        "model_1.evaluate(val_sentences, val_labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Exu9Tz5v0MEf",
        "outputId": "7e37fede-b812-465d-99b8-e7033537d192"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 2ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(762, 1)"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ],
      "source": [
        "# Making predictions and evaluating them\n",
        "model_1_pred_probs = model_1.predict(val_sentences)\n",
        "model_1_pred_probs.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y1gKKZZZ0kgx",
        "outputId": "af04592b-88ee-474e-a7f0-11f9e14ea6f5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.3805468], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ],
      "source": [
        "# Looking at a single prediction\n",
        "model_1_pred_probs[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0A4MI1xE1STY",
        "outputId": "5b46c145-5df7-4c33-c96b-19debc511bd8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.3805468 ],\n",
              "       [0.83576655],\n",
              "       [0.99774325],\n",
              "       [0.12597875],\n",
              "       [0.10358561],\n",
              "       [0.9433119 ],\n",
              "       [0.9286019 ],\n",
              "       [0.9921393 ],\n",
              "       [0.96620834],\n",
              "       [0.28312448]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ],
      "source": [
        "#Looking at the first 10 predictions\n",
        "model_1_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o6KyoIRn1WVh",
        "outputId": "2ab41d4d-e44d-41b9-9cf5-f7d90f003131"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(762,), dtype=float32, numpy=\n",
              "array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 1., 1., 0., 0., 0., 1., 1., 0., 0., 0., 0., 1., 0., 1., 0.,\n",
              "       0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 1., 1., 0., 1., 0., 1., 0.,\n",
              "       0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 1., 1., 1., 0.,\n",
              "       0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0.,\n",
              "       0., 0., 0., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
              "       0., 1., 1., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 1.,\n",
              "       0., 1., 0., 0., 1., 0., 0., 1., 0., 0., 1., 0., 1., 1., 1., 1., 0.,\n",
              "       1., 0., 0., 0., 1., 0., 0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0.,\n",
              "       0., 0., 1., 1., 0., 1., 0., 1., 0., 0., 1., 0., 0., 1., 0., 1., 0.,\n",
              "       1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 1., 1., 1., 1., 0., 0.,\n",
              "       1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 1., 0.,\n",
              "       0., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0., 0., 1., 1., 1., 0., 1.,\n",
              "       0., 0., 0., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 0.,\n",
              "       0., 0., 0., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 1.,\n",
              "       0., 1., 1., 0., 0., 0., 0., 0., 1., 1., 1., 1., 0., 0., 0., 0., 0.,\n",
              "       1., 0., 0., 1., 0., 0., 1., 0., 0., 0., 1., 1., 1., 1., 1., 0., 1.,\n",
              "       0., 1., 0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 1., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 1., 0., 0., 1., 0., 1., 0., 0., 0., 0., 0., 1.,\n",
              "       0., 1., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 1., 0., 0., 1., 1.,\n",
              "       0., 0., 1., 1., 1., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.,\n",
              "       1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 1., 1., 1., 0., 1., 0., 1.,\n",
              "       0., 0., 1., 0., 1., 0., 0., 0., 0., 1., 0., 1., 1., 0., 1., 0., 0.,\n",
              "       0., 1., 1., 0., 0., 0., 1., 1., 0., 0., 1., 0., 1., 1., 1., 1., 0.,\n",
              "       0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 1.,\n",
              "       0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 0., 0., 1.,\n",
              "       1., 1., 0., 0., 0., 1., 1., 0., 0., 1., 0., 0., 1., 0., 0., 1., 0.,\n",
              "       0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 1.,\n",
              "       0., 0., 0., 0., 1., 0., 1., 1., 1., 1., 1., 0., 0., 0., 1., 0., 0.,\n",
              "       0., 0., 0., 0., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0.,\n",
              "       0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 1., 1., 0., 0., 1., 0.,\n",
              "       0., 0., 0., 1., 1., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 1., 0., 1., 0., 1., 1., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0.,\n",
              "       1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 1., 1., 1., 0., 0., 1., 1., 1., 0., 1., 0., 1., 1., 0., 1.,\n",
              "       0., 1., 1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 1.,\n",
              "       0., 0., 1., 1., 1., 1., 0., 0., 0., 0., 0., 1., 0., 1., 1., 0., 0.,\n",
              "       0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.,\n",
              "       0., 1., 0., 0., 0., 1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1.,\n",
              "       0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 1., 1., 1., 0., 1., 0., 1.,\n",
              "       0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,\n",
              "       0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0.,\n",
              "       1., 1., 1., 1., 1., 0., 0., 0., 1., 1., 1., 0., 1., 1., 0., 0., 0.,\n",
              "       0., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 1., 0., 0.],\n",
              "      dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ],
      "source": [
        "# Convert model prediction probabilities to label formats\n",
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_probs))\n",
        "model_1_preds"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_7M26j1avmEc",
        "outputId": "fa4faabe-1c75-4fb0-aa85-55e663cf1ae2"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 79.26509186351706,\n",
              " 'Precision': 0.8111390004213173,\n",
              " 'Recall': 0.7926509186351706,\n",
              " 'F1 score': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ],
      "source": [
        "baseline_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O0g-Ogx01sKT",
        "outputId": "2583b17c-5843-4a31-bcbe-7ff225d9743c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 77.82152230971128,\n",
              " 'Precision': 0.7817665184655122,\n",
              " 'Recall': 0.7782152230971129,\n",
              " 'F1 score': 0.7754656354277033}"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ],
      "source": [
        "# Calculating the model 1's results\n",
        "model_1_results = calculate_results(val_labels, model_1_preds)\n",
        "model_1_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cebHtBkw16D_",
        "outputId": "eb3e5925-efd4-41e5-806d-678286834411"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([False, False, False, False])"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ],
      "source": [
        "import numpy as np\n",
        "np.array(list(model_1_results.values())) > np.array(list(baseline_results.values()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ITBg1DZg2GmU"
      },
      "source": [
        "## Visualize Learned embeddings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "00gJxAU0xat8",
        "outputId": "41424a62-57fa-47f9-9d0e-f72005f0163c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, ['', '[UNK]', 'the', 'a', 'in', 'to', 'of', 'and', 'i', 'is'])"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ],
      "source": [
        "# Get the vocabulary from the text vectorization layer\n",
        "words_in_vocab = text_vectorization.get_vocabulary()\n",
        "len(words_in_vocab), words_in_vocab[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dvmCNlYix2T-",
        "outputId": "936fac32-494b-4595-9047-3d4077c6acb4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " global_average_pooling1d (  (None, 128)               0         \n",
            " GlobalAveragePooling1D)                                         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1280129 (4.88 MB)\n",
            "Trainable params: 1280129 (4.88 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# Model 1 summary\n",
        "model_1.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lv-XOiu5x77b",
        "outputId": "0abe38a4-93b6-48fa-b9e7-7e9720dd830b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(128,)\n"
          ]
        }
      ],
      "source": [
        "# Get the weight matrix of embedding layer\n",
        "embed_weights = model_1.get_layer(\"embedding\").get_weights()[0]\n",
        "print(embed_weights[0].shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PTANqSzHyVbn"
      },
      "outputs": [],
      "source": [
        "# Create embedding files\n",
        "import io\n",
        "\n",
        "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
        "\n",
        "for index, word in enumerate(words_in_vocab):\n",
        "  if index == 0:\n",
        "    continue  # skip 0, it's padding.\n",
        "  vec = embed_weights[index]\n",
        "  out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
        "  out_m.write(word + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "3F1nI6K-zpFi",
        "outputId": "bb4f6639-638a-472f-9eaf-2a32bd8606e5"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_3b48d7ef-c2f9-47ca-9335-b2038bd236ba\", \"vectors.tsv\", 15368433)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_7f920f23-932f-4f20-b093-5d05923af580\", \"metadata.tsv\", 80388)"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Download files from colab to upload to projector\n",
        "try:\n",
        "  from google.colab import files\n",
        "  files.download('vectors.tsv')\n",
        "  files.download('metadata.tsv')\n",
        "except Exception:\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E0Zw32ar0QFW"
      },
      "source": [
        "### Model 2: LSTM\n",
        "\n",
        "Structure of an RNN usually looks like:\n",
        "\n",
        "``` Input (text) -> Tokenize -> Embedding -> Layers (RNN/Dense) -> Output (label probability)```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hpihlluuEpOC"
      },
      "outputs": [],
      "source": [
        "# Create an LSTM model\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape = (1,), dtype = tf.string)\n",
        "x = text_vectorization(inputs)\n",
        "x = embedding(x)\n",
        "# print(x.shape)\n",
        "# x = layers.LSTM(units = 64, return_sequences = True)(x) # While stacking RNN cells together, return_sequences need to be True\n",
        "# print(x.shape)\n",
        "x = layers.LSTM(64)(x)\n",
        "# print(x.shape)\n",
        "outputs = layers.Dense(1, activation = \"sigmoid\")(x)\n",
        "model_2 = tf.keras.Model(inputs, outputs, name = \"model_2_LSTM\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fK5quLS_F2Wb",
        "outputId": "1912895f-701f-430b-c728-bb239bb25c53"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_2_LSTM\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_2 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " lstm (LSTM)                 (None, 64)                49408     \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1329473 (5.07 MB)\n",
            "Trainable params: 1329473 (5.07 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model_2.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QfwAjoxFHF-t"
      },
      "outputs": [],
      "source": [
        "#Compile the model\n",
        "model_2.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fXSHRbrSHUc-",
        "outputId": "56abb859-55dd-47de-de24-bbbce8c64a4a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_2_LSTM/20240219-085957\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 11s 35ms/step - loss: 0.2178 - accuracy: 0.9254 - val_loss: 0.5651 - val_accuracy: 0.7808\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 12ms/step - loss: 0.1562 - accuracy: 0.9418 - val_loss: 0.6753 - val_accuracy: 0.7795\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 16ms/step - loss: 0.1276 - accuracy: 0.9550 - val_loss: 0.6531 - val_accuracy: 0.7848\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 12ms/step - loss: 0.1035 - accuracy: 0.9597 - val_loss: 0.7266 - val_accuracy: 0.7756\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 17ms/step - loss: 0.0867 - accuracy: 0.9672 - val_loss: 0.7967 - val_accuracy: 0.7717\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_2_history = model_2.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"model_2_LSTM\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ySz0mRK0HrMb",
        "outputId": "958bb24b-6b9a-4624-84f3-a16dbd62a79c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 3ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.02599768],\n",
              "       [0.8833593 ],\n",
              "       [0.99970716],\n",
              "       [0.07771336],\n",
              "       [0.00155785],\n",
              "       [0.99806565],\n",
              "       [0.7181698 ],\n",
              "       [0.9998234 ],\n",
              "       [0.99963796],\n",
              "       [0.43661162]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ],
      "source": [
        "# Make predictions with LSTM model\n",
        "model_2_pred_probs = model_2.predict(val_sentences)\n",
        "model_2_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5DA-zQOfIJxm",
        "outputId": "069cf68e-7cee-4c1d-9e28-cebbba76314d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ],
      "source": [
        "model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))\n",
        "model_2_preds[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xi81YAMgIUzo",
        "outputId": "beb323ab-5dd1-4b99-f9ad-2da31035eed1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 77.16535433070865,\n",
              " 'Precision': 0.773559668677779,\n",
              " 'Recall': 0.7716535433070866,\n",
              " 'F1 score': 0.7694523344399302}"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ],
      "source": [
        "# Calculate model 2 results\n",
        "model_2_results = calculate_results(val_labels, model_2_preds)\n",
        "model_2_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jRzGur3HIeDD",
        "outputId": "abe7acdd-d96a-411d-d22e-38ed976750cd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 79.26509186351706,\n",
              " 'Precision': 0.8111390004213173,\n",
              " 'Recall': 0.7926509186351706,\n",
              " 'F1 score': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ],
      "source": [
        "baseline_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r5Ky_KfqIg1X"
      },
      "source": [
        "### Model 3: GRU\n",
        "\n",
        "GRU is another popular and effective RNN component. (also known as Gated Recurrent Unit).\n",
        "\n",
        "The GRU cell has similar features to LSTM cell but has less parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EZpLiyRpJJom"
      },
      "outputs": [],
      "source": [
        "# Creating model_3\n",
        "inputs = layers.Input(shape = (1,), dtype = \"string\")\n",
        "x = text_vectorization(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.GRU(64)(x)\n",
        "# x = layers.GRU(units = 64, return_sequences = True)(x) # While stacking RNN cells together, return_sequences need to be True\n",
        "# print(x.shape)\n",
        "# x = layers.LSTM(42, return_sequences = True)(x)\n",
        "# print(x.shape)\n",
        "# x = layers.GRU(99)(x)\n",
        "# print(x.shape)\n",
        "# x = layers.Dense(64, activation = \"relu\")(x)\n",
        "# x = layers.GlobalAveragePooling1D()(x)\n",
        "outputs = layers.Dense(1, activation = \"sigmoid\")(x)\n",
        "model_3 = tf.keras.Model(inputs, outputs, name = \"model_3_GRU\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KofXDr8ZJy_u",
        "outputId": "03bb26f2-3846-43ff-8586-bf231f4d6f1c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_3_GRU\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_3 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " gru (GRU)                   (None, 64)                37248     \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1317313 (5.03 MB)\n",
            "Trainable params: 1317313 (5.03 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model_3.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lFqnHsS_Jz70"
      },
      "outputs": [],
      "source": [
        "# Compile the model\n",
        "model_3.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pId2jdAFMQY_",
        "outputId": "63361504-3068-4e4e-9dbe-1d706c73cb0c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_3_GRU/20240219-090041\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 11s 41ms/step - loss: 0.1547 - accuracy: 0.9399 - val_loss: 0.7898 - val_accuracy: 0.7756\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0825 - accuracy: 0.9693 - val_loss: 0.8903 - val_accuracy: 0.7874\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0723 - accuracy: 0.9729 - val_loss: 0.9247 - val_accuracy: 0.7730\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0637 - accuracy: 0.9747 - val_loss: 1.0952 - val_accuracy: 0.7664\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.0546 - accuracy: 0.9768 - val_loss: 1.2528 - val_accuracy: 0.7782\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_3_history = model_3.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"model_3_GRU\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QeTlkXFOMfvG",
        "outputId": "50da4cb3-0372-4711-a24f-baee3d4a64a9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 3ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4.97849716e-04],\n",
              "       [8.51698875e-01],\n",
              "       [9.99935031e-01],\n",
              "       [8.50020051e-02],\n",
              "       [1.11222405e-04],\n",
              "       [9.99861121e-01],\n",
              "       [8.84690702e-01],\n",
              "       [9.99964952e-01],\n",
              "       [9.99931455e-01],\n",
              "       [9.98613715e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ],
      "source": [
        "# Making some predictions with model 3\n",
        "model_3_pred_probs = model_3.predict(val_sentences)\n",
        "model_3_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iOqQ-IrkM3qe",
        "outputId": "e9727b1e-ae0c-47a4-e445-096e8b8b9c9d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ],
      "source": [
        "model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))\n",
        "model_3_preds[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JeHQPzEgM_ok",
        "outputId": "0f83e13a-8c88-423c-ab19-4c9ede6b7fe9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 77.82152230971128,\n",
              " 'Precision': 0.7796479981380524,\n",
              " 'Recall': 0.7782152230971129,\n",
              " 'F1 score': 0.7763875633348853}"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ],
      "source": [
        "# Calculate mdoel 3 results\n",
        "model_3_results = calculate_results(val_labels, model_3_preds)\n",
        "model_3_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i0ydGdHnNIY1"
      },
      "source": [
        "### Model 4: Bidirectional\n",
        "\n",
        "Normal RNN's go from left to right (just like an english sentence), however, a bidirectional RNN goes from left to right as well as from right to left."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_6Wc0Q-3NqyE"
      },
      "outputs": [],
      "source": [
        "# Create model 4\n",
        "from tensorflow.keras import layers\n",
        "x = layers.Input(shape = (1,), dtype = \"string\")\n",
        "x = text_vectorization(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.Bidirectional(layers.LSTM(64, return_sequences = True))(x)\n",
        "x = layers.Bidirectional(layers.LSTM(64))(x)\n",
        "outputs = layers.Dense(1, activation = \"sigmoid\")(x)\n",
        "model_4 = tf.keras.Model(inputs, outputs, name = \"model_4_bidirectional\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oEW2tXSrPjYc",
        "outputId": "618435ee-b496-41d5-a545-b115e65b26f2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_4_bidirectional\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_3 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " bidirectional (Bidirection  (None, 128)               98816     \n",
            " al)                                                             \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1378945 (5.26 MB)\n",
            "Trainable params: 1378945 (5.26 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model_4.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yIn0S7AwPmxP"
      },
      "outputs": [],
      "source": [
        "# Compile the model\n",
        "model_4.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SVfs14oSQJ2N",
        "outputId": "586a7550-188e-4dd9-f6e8-7285df90c80d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_4_bidirectional/20240219-090101\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 13s 42ms/step - loss: 0.1013 - accuracy: 0.9724 - val_loss: 0.9794 - val_accuracy: 0.7703\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 5s 25ms/step - loss: 0.0552 - accuracy: 0.9780 - val_loss: 1.1208 - val_accuracy: 0.7717\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 11ms/step - loss: 0.0478 - accuracy: 0.9791 - val_loss: 1.2178 - val_accuracy: 0.7795\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.0465 - accuracy: 0.9781 - val_loss: 1.3091 - val_accuracy: 0.7703\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.0427 - accuracy: 0.9803 - val_loss: 1.1986 - val_accuracy: 0.7612\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_4_history = model_4.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"model_4_bidirectional\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RLtiP667QnLH",
        "outputId": "fe2fdeb1-8459-483b-8c3e-c6a31c4e4784"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 4ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[9.9247687e-02],\n",
              "       [9.2212003e-01],\n",
              "       [9.9991667e-01],\n",
              "       [1.1342701e-01],\n",
              "       [7.1940558e-05],\n",
              "       [9.9937916e-01],\n",
              "       [8.0773932e-01],\n",
              "       [9.9998391e-01],\n",
              "       [9.9994445e-01],\n",
              "       [9.9842787e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ],
      "source": [
        "# Making predictions on model 4\n",
        "model_4_pred_probs = model_4.predict(val_sentences)\n",
        "model_4_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zE7WFmpaQ4h4",
        "outputId": "e8cb7988-cd85-4936-ab68-d27c59947a71"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ],
      "source": [
        "model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))\n",
        "model_4_preds[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hv5nyWIDQ_3h",
        "outputId": "b1f044d3-42ce-4c19-8358-68a9d8d4eaf8"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 76.11548556430446,\n",
              " 'Precision': 0.7612627209133627,\n",
              " 'Recall': 0.7611548556430446,\n",
              " 'F1 score': 0.7598431815524319}"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ],
      "source": [
        "# Calculating model 4 results\n",
        "model_4_results = calculate_results(val_labels, model_4_preds)\n",
        "model_4_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nIeUKVVhRGKM"
      },
      "source": [
        "## Convolutional Neural Networks for Text\n",
        "\n",
        "The typical structure of Conv1D model for sequences:\n",
        "```\n",
        "  Inputs(text) -> Tokenization -> Embedding -> Layer(s) (typically Conv1D or pooling) -> Outputs (class probabilities)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9GZLBTZ0x4F9"
      },
      "source": [
        "### Model 5: Conv1D"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ijJjuatIx8JJ",
        "outputId": "fa59bfb4-54b6-4028-80d9-a8a3041774cb"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([1, 15, 128]), TensorShape([1, 11, 32]), TensorShape([1, 32]))"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ],
      "source": [
        "# Testing out the embedding layer, Conv1D layer and the max pooling layer\n",
        "from tensorflow.keras import layers\n",
        "embedding_test = embedding(text_vectorization([\"this is a test sentence\"]))\n",
        "conv_1d = layers.Conv1D(filters = 32,\n",
        "                        kernel_size = 5, # also referred to as an ngram of 5 (looks at 5 words at a time)\n",
        "                        strides = 1,\n",
        "                        activation = \"relu\",\n",
        "                        padding = \"valid\")\n",
        "conv_1d_output = conv_1d(embedding_test)\n",
        "max_pool = layers.GlobalMaxPooling1D()\n",
        "max_pool_output = max_pool(conv_1d_output)\n",
        "\n",
        "embedding_test.shape, conv_1d_output.shape, max_pool_output.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9kx_uuVYzr-A",
        "outputId": "ecd6a402-3a35-4584-b60f-2dfeb3532026"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_5_conv1D\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_5 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (Text  (None, 15)                0         \n",
            " Vectorization)                                                  \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " conv1d_1 (Conv1D)           (None, 11, 64)            41024     \n",
            "                                                                 \n",
            " global_max_pooling1d_1 (Gl  (None, 64)                0         \n",
            " obalMaxPooling1D)                                               \n",
            "                                                                 \n",
            " dense_4 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1321089 (5.04 MB)\n",
            "Trainable params: 1321089 (5.04 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# Create a 1-dimensional convolutional layer to model sequences\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape = (1,), dtype = tf.string)\n",
        "x = text_vectorization(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.Conv1D(filters = 64,\n",
        "                  kernel_size = 5,\n",
        "                  strides = 1,\n",
        "                  activation = \"relu\",\n",
        "                  padding = \"valid\")(x)\n",
        "x = layers.GlobalMaxPooling1D()(x)\n",
        "outputs = layers.Dense(1, activation = \"sigmoid\")(x)\n",
        "model_5 = tf.keras.Model(inputs, outputs, name = \"model_5_conv1D\")\n",
        "\n",
        "# Compile the Conv1D model\n",
        "model_5.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])\n",
        "\n",
        "# Summary of the model\n",
        "model_5.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GXH8yk-rrvdB",
        "outputId": "eec9badd-7cf3-45eb-b19f-0506101e1fa2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/Conv1D/20240219-090129\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 10s 38ms/step - loss: 0.1247 - accuracy: 0.9546 - val_loss: 0.9196 - val_accuracy: 0.7769\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.0750 - accuracy: 0.9721 - val_loss: 1.0702 - val_accuracy: 0.7703\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.0626 - accuracy: 0.9755 - val_loss: 1.1059 - val_accuracy: 0.7677\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.0561 - accuracy: 0.9772 - val_loss: 1.2051 - val_accuracy: 0.7572\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.0505 - accuracy: 0.9791 - val_loss: 1.2532 - val_accuracy: 0.7612\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_5_history = model_5.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"Conv1D\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_AC8NAqisEvW",
        "outputId": "745c6f49-9694-4c2f-c554-f49265048b77"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 2ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1.3999943e-01],\n",
              "       [9.7192818e-01],\n",
              "       [9.9994516e-01],\n",
              "       [4.8758712e-02],\n",
              "       [2.7119063e-08],\n",
              "       [9.9450433e-01],\n",
              "       [7.8593647e-01],\n",
              "       [9.9999225e-01],\n",
              "       [9.9999940e-01],\n",
              "       [9.7717834e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ],
      "source": [
        "# Making some predictions\n",
        "model_5_pred_probs = model_5.predict(val_sentences)\n",
        "model_5_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Npp9-KvfsXjJ",
        "outputId": "065682b7-9a73-44db-f2fa-3ba90c8cb300"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ],
      "source": [
        "# Converting pred probs to preds\n",
        "model_5_predictions = tf.squeeze(tf.round(model_5_pred_probs))\n",
        "model_5_predictions[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-5jjQFV1slsc",
        "outputId": "b152b395-60fa-4a47-a58f-e693174067ab"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 76.11548556430446,\n",
              " 'Precision': 0.7615445184736523,\n",
              " 'Recall': 0.7611548556430446,\n",
              " 'F1 score': 0.7596168376680188}"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ],
      "source": [
        "# Evaluate mode 5 predictions\n",
        "model_5_results = calculate_results(val_labels, model_5_predictions)\n",
        "model_5_results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1HAGeE6fswbF",
        "outputId": "4c15f464-4016-48a0-d2fc-86f7351b86f5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 79.26509186351706,\n",
              " 'Precision': 0.8111390004213173,\n",
              " 'Recall': 0.7926509186351706,\n",
              " 'F1 score': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ],
      "source": [
        "baseline_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMFHNGyGs2GD"
      },
      "source": [
        "## Model 6: Tensorflow Hub Pretrained Sentence Encoder\n",
        "\n",
        "Using transfer learning, specifically TensorFlow Hub's Universal Sentence Encoder: https://www.kaggle.com/models/google/universal-sentence-encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b55KCM8qvh3R",
        "outputId": "8aa4fadc-0ac5-4002-8862-6d534d282ea4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tensorflow_text\n",
            "  Downloading tensorflow_text-2.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.2/5.2 MB\u001b[0m \u001b[31m45.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tensorflow-hub>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow_text) (0.16.1)\n",
            "Requirement already satisfied: tensorflow<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow_text) (2.15.0)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.4.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (23.5.26)\n",
            "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (0.5.4)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (0.2.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (3.9.0)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (16.0.6)\n",
            "Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (0.2.0)\n",
            "Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.25.2)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (3.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (23.2)\n",
            "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (3.20.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (67.7.2)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.16.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (2.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (4.9.0)\n",
            "Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.14.1)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (0.36.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (1.60.1)\n",
            "Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (2.15.2)\n",
            "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (2.15.0)\n",
            "Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.16,>=2.15.0->tensorflow_text) (2.15.0)\n",
            "Requirement already satisfied: tf-keras>=2.14.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-hub>=0.13.0->tensorflow_text) (2.15.0)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow<2.16,>=2.15.0->tensorflow_text) (0.42.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (2.27.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (1.2.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (3.5.2)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (2.31.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (3.0.1)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (5.3.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (1.3.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (2024.2.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (2.1.5)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (0.5.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow<2.16,>=2.15.0->tensorflow_text) (3.2.2)\n",
            "Installing collected packages: tensorflow_text\n",
            "Successfully installed tensorflow_text-2.15.0\n"
          ]
        }
      ],
      "source": [
        "!pip install tensorflow_text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "5mLXlFxGpjDC",
        "outputId": "4e98e737-5693-4d51-df55-5319f06582ad"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Theres a flood at my street!'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 79
        }
      ],
      "source": [
        "sample_sentence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FyL-5hCWtLMM",
        "outputId": "2fffb66e-4e2e-4e39-83d8-06ce4707a216"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[ 0.0248296  -0.03637185  0.04463828 -0.11526786  0.10707054 -0.00070173\n",
            " -0.0989859  -0.05026377  0.08818121  0.00720744  0.1009884  -0.10928837\n",
            "  0.03227057  0.00738077 -0.04685839  0.06913014 -0.08000515  0.06578954\n",
            " -0.06255247 -0.03652623  0.0235874   0.00640677 -0.17568503  0.08474603\n",
            "  0.03024769 -0.06156315 -0.01483053  0.07346989  0.01629649 -0.03841049\n",
            " -0.02170764 -0.17450276 -0.14372426  0.01268796 -0.18157813 -0.01743637\n",
            " -0.02416911  0.05488009 -0.00423118 -0.00302255  0.02020528 -0.06483547\n",
            " -0.0756525   0.07647108  0.01962425 -0.02268834  0.09726764 -0.04308834\n",
            "  0.08747728 -0.05536968], shape=(50,), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "import tensorflow_hub as hub\n",
        "import tensorflow as tf\n",
        "import tensorflow_text as text\n",
        "\n",
        "english_sentences = tf.constant([sample_sentence, \"When the universal sentence encoder is called on a sentence, it turns it into numbers.\"])\n",
        "\n",
        "preprocessor = hub.KerasLayer(\n",
        "    \"https://kaggle.com/models/tensorflow/bert/frameworks/TensorFlow2/variations/en-uncased-preprocess/versions/3\")\n",
        "encoder = hub.KerasLayer(\n",
        "    \"https://www.kaggle.com/models/google/universal-sentence-encoder/frameworks/TensorFlow2/variations/cmlm-en-base/versions/1\")\n",
        "\n",
        "english_embeds = encoder(preprocessor(english_sentences))[\"default\"]\n",
        "\n",
        "print (english_embeds[0][:50])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vSxL-ccp3o3W",
        "outputId": "298dd3ac-c9b1-4c7c-9373-1b20eda1e103"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([768])"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ],
      "source": [
        "english_embeds[0].shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yGGxOTFDydRj"
      },
      "outputs": [],
      "source": [
        "# Create a Keras layer using the USE pretrained layer from Kaggle\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[], # shape of inputs coming to our model\n",
        "                                        dtype=tf.string, # data type of inputs coming to the USE layer\n",
        "                                        trainable=False, # keep the pretrained weights (we'll create a feature extractor)\n",
        "                                        name=\"USE\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xB-iMOtN5Ax2",
        "outputId": "d80c1e0d-f352-4a96-b5ed-e4cfb66e3511"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[-0.00438637  0.02435914  0.02274885 -0.0141593   0.02424628  0.0889047\n",
            "  0.02888083  0.06567273 -0.00610242 -0.00664645  0.00411185  0.00201372\n",
            " -0.02119484  0.09455124  0.06445288 -0.04171616  0.01693997 -0.04375052\n",
            " -0.00695595 -0.08056634 -0.05382917  0.02640241  0.0149875   0.01279173\n",
            " -0.01601805 -0.06174997  0.03324892 -0.00981064 -0.0380497  -0.01027906\n",
            " -0.0378894   0.06094595 -0.03173684  0.0007248   0.03728033 -0.06037606\n",
            "  0.01713017  0.02897402  0.0068134  -0.09162798  0.02461487 -0.00304058\n",
            " -0.01315405  0.05872618 -0.10164236 -0.03168903  0.0131347  -0.01563225\n",
            " -0.03587101  0.03673211], shape=(50,), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "import tensorflow_hub as hub\n",
        "embed = hub.load(\"https://tfhub.dev/google/universal-sentence-encoder/4\") # load Universal Sentence Encoder\n",
        "embed_samples = embed([sample_sentence,\n",
        "                      \"When you call the universal sentence encoder on a sentence, it turns it into numbers.\"])\n",
        "\n",
        "print(embed_samples[0][:50])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PBk6Wd5x6Mae",
        "outputId": "bd7cb534-b1ae-4fdb-b562-e4f93c467448"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([512])"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ],
      "source": [
        "embed_samples[0].shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GKArmOuQrl1m"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mAqes9x36cC6",
        "outputId": "7e64587e-9ed8-4d9e-a278-f5ccd6c4df33"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " USE (KerasLayer)            (None, 512)               256797824 \n",
            "                                                                 \n",
            " dense_5 (Dense)             (None, 128)               65664     \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 64)                8256      \n",
            "                                                                 \n",
            " output_layer (Dense)        (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 256871809 (979.89 MB)\n",
            "Trainable params: 73985 (289.00 KB)\n",
            "Non-trainable params: 256797824 (979.61 MB)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# Create model using the Sequential API\n",
        "model_6 = tf.keras.Sequential([\n",
        "  sentence_encoder_layer, # take in sentences and then encode them into an embedding\n",
        "  layers.Dense(128, activation=\"relu\"),\n",
        "  layers.Dense(64, activation = \"relu\"),\n",
        "  layers.Dense(1, activation=\"sigmoid\", name = \"output_layer\")\n",
        "], name=\"model_6_USE\")\n",
        "\n",
        "# Compile model\n",
        "model_6.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "model_6.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k5s2w3Djq5af",
        "outputId": "c1d142c3-63dc-4d2b-d058-c17e4a73b705"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/tf_hub_sentence_encoder/20240219-090336\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 7s 14ms/step - loss: 0.4733 - accuracy: 0.7884 - val_loss: 0.4396 - val_accuracy: 0.8071\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 12ms/step - loss: 0.3949 - accuracy: 0.8286 - val_loss: 0.4234 - val_accuracy: 0.8150\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 13ms/step - loss: 0.3660 - accuracy: 0.8437 - val_loss: 0.4112 - val_accuracy: 0.8136\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.3345 - accuracy: 0.8608 - val_loss: 0.4341 - val_accuracy: 0.8176\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 3s 13ms/step - loss: 0.2923 - accuracy: 0.8835 - val_loss: 0.4512 - val_accuracy: 0.8281\n"
          ]
        }
      ],
      "source": [
        "# Training the model\n",
        "model_6_history = model_6.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"tf_hub_sentence_encoder\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CZelHCrAuo9p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af33deb4-32ac-44b0-b8ba-39237b0a50bf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 24ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.07147994],\n",
              "       [0.7978028 ],\n",
              "       [0.9997229 ],\n",
              "       [0.16383699],\n",
              "       [0.611284  ],\n",
              "       [0.9436232 ],\n",
              "       [0.9961588 ],\n",
              "       [0.99917716],\n",
              "       [0.99736845],\n",
              "       [0.06414556]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 165
        }
      ],
      "source": [
        "# Make predictions with USE TF Hub\n",
        "model_6_pred_probs = model_6.predict(val_sentences)\n",
        "model_6_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rFBPmy17vf9a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "42e5dfe3-b06e-47ba-aa69-c7ffae3803a3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 166
        }
      ],
      "source": [
        "# Convert prediction probabilities to predicted classes\n",
        "model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))\n",
        "model_6_preds[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fvxc2kkYvqFZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0658847c-8b70-4f96-8943-e5ccd6e92bde"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 82.80839895013123,\n",
              " 'Precision': 0.8321679242685085,\n",
              " 'Recall': 0.8280839895013123,\n",
              " 'F1 score': 0.8262746345531319}"
            ]
          },
          "metadata": {},
          "execution_count": 167
        }
      ],
      "source": [
        "# Calculate model 6 results\n",
        "model_6_results = calculate_results(val_labels, model_6_preds)\n",
        "model_6_results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-dUFOMzlvx4i"
      },
      "source": [
        "## Model 7: TF Hub pretrained USE but with 10% of the data\n",
        "Transfer learning is really helpful with a small dataset. To see how model 6 performs with 10% of the data, replicating it and experimenting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gdcpVL1Bx0Mh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa9b80c2-3127-497f-a4bc-356322ff5219"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(761, 761)"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ],
      "source": [
        "# Create subsets of 10% of the training data\n",
        "train_10_percent = train_df_shuffled[[\"text\", \"target\"]].sample(frac = 0.1, random_state = 42)\n",
        "train_sentences_10_percent = train_10_percent[\"text\"].to_list()\n",
        "train_labels_10_percent = train_10_percent[\"target\"].to_list()\n",
        "len(train_sentences_10_percent), len(train_labels_10_percent)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bx3V8sgryMrR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba267cd0-909e-41e8-e300-7b160c19a47e"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0    413\n",
              " 1    348\n",
              " Name: target, dtype: int64,\n",
              " 0    4342\n",
              " 1    3271\n",
              " Name: target, dtype: int64)"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ],
      "source": [
        "# Checking the number of target classes\n",
        "train_10_percent[\"target\"].value_counts(), train_df_shuffled[\"target\"].value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fy9WjjZwy6Nv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5118a2e7-dbec-45bc-bd12-0419fd8f8056"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_7_USE\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " USE (KerasLayer)            (None, 512)               256797824 \n",
            "                                                                 \n",
            " dense_8 (Dense)             (None, 64)                32832     \n",
            "                                                                 \n",
            " output_layer (Dense)        (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 256830721 (979.73 MB)\n",
            "Trainable params: 32897 (128.50 KB)\n",
            "Non-trainable params: 256797824 (979.61 MB)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# Creating model 7\n",
        "# model_7 = tf.keras.models.clone_model(model_6)\n",
        "model_7 = tf.keras.Sequential([\n",
        "    sentence_encoder_layer,\n",
        "    # layers.Dense(128, activation = \"relu\"),\n",
        "    layers.Dense(64, activation = \"relu\"),\n",
        "    layers.Dense(1, activation = \"sigmoid\", name = \"output_layer\")\n",
        "], name = \"model_7_USE\")\n",
        "\n",
        "# Compile the model\n",
        "model_7.compile(loss = \"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics = [\"accuracy\"])\n",
        "\n",
        "# Summary of the model\n",
        "model_7.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1JvCFOk40cs_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "574f0611-936f-499c-b06b-ec56990d6f57"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_7_USE_correct_split/20240219-093447\n",
            "Epoch 1/5\n",
            "22/22 [==============================] - 2s 49ms/step - loss: 0.6763 - accuracy: 0.6628 - val_loss: 0.6510 - val_accuracy: 0.7612\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 1s 27ms/step - loss: 0.6105 - accuracy: 0.8000 - val_loss: 0.5958 - val_accuracy: 0.7677\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 0s 21ms/step - loss: 0.5369 - accuracy: 0.8131 - val_loss: 0.5407 - val_accuracy: 0.7769\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 0s 21ms/step - loss: 0.4747 - accuracy: 0.8146 - val_loss: 0.5036 - val_accuracy: 0.7782\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 1s 27ms/step - loss: 0.4294 - accuracy: 0.8263 - val_loss: 0.4901 - val_accuracy: 0.7769\n"
          ]
        }
      ],
      "source": [
        "# Fit the model\n",
        "model_7_history = model_7.fit(train_10_percent,\n",
        "                              train_labels_10_percent,\n",
        "                              epochs = 5,\n",
        "                              validation_data = (val_sentences,val_labels),\n",
        "                              callbacks = [create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                       \"model_7_USE_correct_split\")])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VG-njue60rZ0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a6debcc8-f665-4731-9903-136babc85b86"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 10ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.20878984],\n",
              "       [0.5959316 ],\n",
              "       [0.9154499 ],\n",
              "       [0.36151952],\n",
              "       [0.5476825 ],\n",
              "       [0.7077107 ],\n",
              "       [0.8778115 ],\n",
              "       [0.8243677 ],\n",
              "       [0.84349006],\n",
              "       [0.1706455 ]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 107
        }
      ],
      "source": [
        "# Making predictions with model 7\n",
        "model_7_pred_probs = model_7.predict(val_sentences)\n",
        "model_7_pred_probs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9IHtc4R5040P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f7240bb-d404-4c2a-a4e0-6671e738505c"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 108
        }
      ],
      "source": [
        "# Converting prediction probabilities to predicted classes\n",
        "model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))\n",
        "model_7_preds[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CDHd-eez1Cb0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de4469ba-8a33-4c85-ec41-39b176f8bddc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 77.69028871391076,\n",
              " 'Precision': 0.7779652793701148,\n",
              " 'Recall': 0.7769028871391076,\n",
              " 'F1 score': 0.7752416323647977}"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ],
      "source": [
        "# Calucalting model 7 results\n",
        "model_7_results = calculate_results(val_labels, model_7_preds)\n",
        "model_7_results"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uk2DjlapuA8j",
        "outputId": "bb1bd148-0417-4b8b-8671-3e57c31ddde9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 82.80839895013123,\n",
              " 'Precision': 0.8321679242685085,\n",
              " 'Recall': 0.8280839895013123,\n",
              " 'F1 score': 0.8262746345531319}"
            ]
          },
          "metadata": {},
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qqibvEE61MJ5"
      },
      "outputs": [],
      "source": [
        "# Making a better data split (with no data leakage)\n",
        "train_10_percent_split = int(0.1 * len(train_sentences))\n",
        "train_10_percent = train_sentences[:train_10_percent_split]\n",
        "train_labels_10_percent = train_labels[:train_10_percent_split]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.Series(np.array(train_labels_10_percent)).value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SOly3biCszKs",
        "outputId": "eea3fcfb-866f-4af6-afe5-84163ab438a9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    406\n",
              "1    279\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Comparing the performance of each of the models"
      ],
      "metadata": {
        "id": "JbgVcok-s-vG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Combine all model results\n",
        "all_model_results = pd.DataFrame({\"0_baseline\" : baseline_results,\n",
        "                                  \"1_simple_dense\" : model_1_results,\n",
        "                                  \"2_LSTM\": model_2_results,\n",
        "                                  \"3_gru\": model_3_results,\n",
        "                                  \"4_bidirectional_LSTM\": model_4_results,\n",
        "                                  \"5_conv1d\": model_5_results,\n",
        "                                  \"6_tf_hub_use_encoder\": model_6_results,\n",
        "                                  \"7_tf_hub_use_encoder_10_percent\": model_7_results})\n",
        "all_model_results = all_model_results.T"
      ],
      "metadata": {
        "id": "PYjyD72OuXjG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "all_model_results[\"Accuracy\"] = all_model_results[\"Accuracy\"]/100\n",
        "all_model_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "hXliZzqqutfA",
        "outputId": "5093de57-d192-4e79-a5ae-bd21ee755d2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                 Accuracy  Precision    Recall  F1 score\n",
              "0_baseline                       0.792651   0.811139  0.792651  0.786219\n",
              "1_simple_dense                   0.778215   0.781767  0.778215  0.775466\n",
              "2_LSTM                           0.771654   0.773560  0.771654  0.769452\n",
              "3_gru                            0.778215   0.779648  0.778215  0.776388\n",
              "4_bidirectional_LSTM             0.761155   0.761263  0.761155  0.759843\n",
              "5_conv1d                         0.761155   0.761545  0.761155  0.759617\n",
              "6_tf_hub_use_encoder             0.828084   0.832168  0.828084  0.826275\n",
              "7_tf_hub_use_encoder_10_percent  0.776903   0.777965  0.776903  0.775242"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ba72e4e0-3d3d-4073-af67-6faa19b68bef\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Accuracy</th>\n",
              "      <th>Precision</th>\n",
              "      <th>Recall</th>\n",
              "      <th>F1 score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0_baseline</th>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1_simple_dense</th>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.781767</td>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.775466</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2_LSTM</th>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.773560</td>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.769452</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3_gru</th>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.779648</td>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.776388</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4_bidirectional_LSTM</th>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.761263</td>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.759843</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5_conv1d</th>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.761545</td>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.759617</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6_tf_hub_use_encoder</th>\n",
              "      <td>0.828084</td>\n",
              "      <td>0.832168</td>\n",
              "      <td>0.828084</td>\n",
              "      <td>0.826275</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7_tf_hub_use_encoder_10_percent</th>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.777965</td>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.775242</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ba72e4e0-3d3d-4073-af67-6faa19b68bef')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ba72e4e0-3d3d-4073-af67-6faa19b68bef button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ba72e4e0-3d3d-4073-af67-6faa19b68bef');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c2d99233-05d2-4c14-ac96-971db3877835\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c2d99233-05d2-4c14-ac96-971db3877835')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c2d99233-05d2-4c14-ac96-971db3877835 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_3fa93b20-df1e-44f7-8ed7-744ee5b11acc\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('all_model_results')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_3fa93b20-df1e-44f7-8ed7-744ee5b11acc button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('all_model_results');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "all_model_results",
              "summary": "{\n  \"name\": \"all_model_results\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": \"Accuracy\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.02158027508365884,\n        \"min\": 0.7611548556430446,\n        \"max\": 0.8280839895013123,\n        \"samples\": [\n          0.7926509186351706,\n          0.7782152230971129,\n          0.7769028871391076\n        ],\n        \"num_unique_values\": 6,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Precision\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.02461426504064846,\n        \"min\": 0.7612627209133627,\n        \"max\": 0.8321679242685085,\n        \"samples\": [\n          0.7817665184655122,\n          0.7615445184736523,\n          0.8111390004213173\n        ],\n        \"num_unique_values\": 8,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Recall\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.02158027508365884,\n        \"min\": 0.7611548556430446,\n        \"max\": 0.8280839895013123,\n        \"samples\": [\n          0.7926509186351706,\n          0.7782152230971129,\n          0.7769028871391076\n        ],\n        \"num_unique_values\": 6,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1 score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.02122251031228979,\n        \"min\": 0.7596168376680188,\n        \"max\": 0.8262746345531319,\n        \"samples\": [\n          0.7754656354277033,\n          0.7596168376680188,\n          0.7862189758049549\n        ],\n        \"num_unique_values\": 8,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot and compare all model's results\n",
        "all_model_results.plot(kind = \"bar\", figsize = (10,7)).legend(bbox_to_anchor = (1.0, 1.0));"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 825
        },
        "id": "-YTxSg14v3aD",
        "outputId": "b1b18336-7880-4615-cbab-8bb16640d5ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7MAAAMoCAYAAAAdv7fvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACBD0lEQVR4nOzdfXyN9ePH8ffZ2GbsBmPuxjC5ndshEWIhcv/t6zYL6VtMshLKfTFflSRFue9GRJIiqeV2TRhbkrsxhsxtyGRj2+8Pv86300Y2O7tc57yej8d5tF3X55zz3k6bvc/1uT6XJTMzM1MAAAAAAJiIi9EBAAAAAADIKcosAAAAAMB0KLMAAAAAANOhzAIAAAAATIcyCwAAAAAwHcosAAAAAMB0KLMAAAAAANOhzAIAAAAATKeA0QHuREZGhn799Vd5eXnJYrEYHQcAAACAQTIzM/X777+rTJkycnHh2JwzM0WZ/fXXXxUQEGB0DAAAAAD3iOPHj6tcuXJGx4CBTFFmvby8JN38H9bb29vgNAAAAACMcvnyZQUEBFg7ApyXKcrsn1OLvb29KbMAAAAAOP0QLAAFAAAAADAfyiwAAAAAwHQoswAAAAAA0zHFObMAAAAAcKcyMjKUlpZmdAzkUMGCBeXq6nrH4ymzAAAAABxGWlqaEhMTlZGRYXQU5IKvr69KlSp1Rwt8UWYBAAAAOITMzEydOnVKrq6uCggIkIsLZ1WaRWZmpq5evaozZ85IkkqXLv2P96HMAgAAAHAIN27c0NWrV1WmTBl5enoaHQc5VKhQIUnSmTNnVLJkyX+ccsxbFQAAAAAcQnp6uiTJzc3N4CTIrT/fhLh+/fo/jqXMAgAAAHAod3K+Je5NOXntKLMAAAAAANOhzAIAAADAPSAmJkaurq7q0KGD0VFMgQWgAAAAADi0wFFr8vX5jk7NXRmdP3++hg4dqvnz5+vXX39VmTJl8jjZnUlLSzPFecccmQUAAAAAg125ckXLli3TM888ow4dOmjRokU2+7/88ks1bNhQHh4e8vPzU9euXa37UlNTNXLkSAUEBMjd3V1BQUGaP3++JGnRokXy9fW1eaxVq1bZnJs6YcIE1a1bV/PmzVPFihXl4eEhSVq3bp2aNWsmX19fFS9eXI8++qgOHz5s81gnTpxQr169VKxYMRUuXFghISH68ccfdfToUbm4uGjnzp0242fMmKEKFSrkyXWAKbMAAAAAYLBPP/1U1apVU9WqVdW3b18tWLBAmZmZkqQ1a9aoa9euat++vXbv3q2oqCg1atTIet9+/frpk08+0cyZM7Vv3z699957KlKkSI6ePyEhQZ999plWrlypuLg4SVJKSooiIiK0c+dORUVFycXFRV27drUW0StXrqhFixY6efKkVq9erfj4eL344ovKyMhQYGCgQkNDtXDhQpvnWbhwoZ544ok8uQYw04wBAAAAwGDz589X3759JUnt2rXTpUuXtGnTJrVs2VKTJ09Wz549NXHiROv4OnXqSJIOHjyoTz/9VN9++61CQ0MlSZUqVcrx86elpemDDz5QiRIlrNu6d+9uM2bBggUqUaKEfvnlF9WqVUtLlizR2bNntWPHDhUrVkySFBQUZB3/5JNP6umnn9b06dPl7u6uXbt2ac+ePfriiy9ynC87HJkFAAAAAAMdOHBA27dvV69evSRJBQoUUI8ePaxThePi4tS6dets7xsXFydXV1e1aNHirjJUqFDBpshK0qFDh9SrVy9VqlRJ3t7eCgwMlCQlJSVZn7tevXrWIvt3Xbp0kaurqz7//HNJN6c8P/TQQ9bHuVscmQUAAAAAA82fP183btywWfApMzNT7u7umjVrlgoVKnTL+95unyS5uLhYpyv/6fr161nGFS5cOMu2jh07qkKFCpo7d67KlCmjjIwM1apVS2lpaXf03G5uburXr58WLlyobt26acmSJXrrrbdue5+c4MgsAAAAABjkxo0b+uCDD/TGG28oLi7OeouPj1eZMmX0ySefqHbt2oqKisr2/sHBwcrIyNCmTZuy3V+iRAn9/vvvSklJsW7785zY2zl//rwOHDigMWPGqHXr1qpevbp+++03mzG1a9dWXFycLly4cMvHefLJJ/Xdd9/p3Xff1Y0bN9StW7d/fO47xZFZAAAAADDIV199pd9++00DBw6Uj4+Pzb7u3btr/vz5eu2119S6dWtVrlxZPXv21I0bN7R27VqNHDlSgYGBCgsL04ABAzRz5kzVqVNHx44d05kzZ/Tvf/9bjRs3lqenp1566SU9++yz+vHHH7OslJydokWLqnjx4nr//fdVunRpJSUladSoUTZjevXqpSlTpqhLly6KjIxU6dKltXv3bpUpU0ZNmjSRJFWvXl3333+/Ro4cqQEDBvzj0dyc4MgsAAAAABhk/vz5Cg0NzVJkpZtldufOnSpWrJiWL1+u1atXq27dumrVqpW2b99uHTd79mz961//0uDBg1WtWjUNGjTIeiS2WLFi+uijj7R27VoFBwfrk08+0YQJE/4xl4uLi5YuXarY2FjVqlVLw4cP12uvvWYzxs3NTevXr1fJkiXVvn17BQcHa+rUqXJ1dbUZN3DgQKWlpWnAgAG5+A7dmiXz7xOo70GXL1+Wj4+PLl26JG9vb6PjAAAAADDI7brBtWvXlJiYaHOtVBjvlVde0fLly/XTTz/949icvIYcmQUAAAAA5LkrV67o559/1qxZszR06NA8f3zKLAAAAAAgz4WHh6tBgwZq2bJlnk8xllgACgAAAHdjQtbz/G4//pJ9cgC45yxatOiOFpvKLY7MAgAAAABMhyOzAAAAsAoctSZH44/mcI2d4MXBObuDpD1he3J8HwCOjzILAACAe9q+atVzNL76/n12SgLgXsI0YwAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAnIzFYtGqVavyfGx+oswCAAAAgIGeeOIJWSwWWSwWubm5KSgoSJMmTdKNGzfs9pynTp3SI488kudj8xOrGQMAAABwbBN88vn5LuX4Lu3atdPChQuVmpqqtWvXasiQISpYsKBGjx5tMy4tLU1ubm53HbFUqVJ2GZufODILAAAAAAZzd3dXqVKlVKFCBT3zzDMKDQ3V6tWr9cQTT6hLly6aPHmyypQpo6pVq0qSjh8/rn//+9/y9fVVsWLF1LlzZx09etTmMRcsWKCaNWvK3d1dpUuXVnh4uHXfX6cOp6WlKTw8XKVLl5aHh4cqVKigyMjIbMdK0p49e9SqVSsVKlRIxYsX11NPPaUrV65Y9/+Z+fXXX1fp0qVVvHhxDRkyRNevX8/T7xllFgAAAADuMYUKFVJaWpokKSoqSgcOHNC3336rr776StevX1fbtm3l5eWlLVu2KDo6WkWKFFG7du2s95k9e7aGDBmip556Snv27NHq1asVFBSU7XPNnDlTq1ev1qeffqoDBw7o448/VmBgYLZjU1JS1LZtWxUtWlQ7duzQ8uXL9d1339kUZUnasGGDDh8+rA0bNmjx4sVatGiRFi1alGffH4lpxgAAAABwz8jMzFRUVJS++eYbDR06VGfPnlXhwoU1b9486/Tijz76SBkZGZo3b54sFoskaeHChfL19dXGjRvVpk0bvfrqq3r++ec1bNgw62M3bNgw2+dMSkpSlSpV1KxZM1ksFlWoUOGW+ZYsWaJr167pgw8+UOHChSVJs2bNUseOHfXf//5X/v7+kqSiRYtq1qxZcnV1VbVq1dShQwdFRUVp0KBBefJ9kiizuZfTefe5mDcPAAAAwDl89dVXKlKkiK5fv66MjAz17t1bEyZM0JAhQxQcHGxznmx8fLwSEhLk5eVl8xjXrl3T4cOHdebMGf36669q3br1HT33E088oYcfflhVq1ZVu3bt9Oijj6pNmzbZjt23b5/q1KljLbKS1LRpU2VkZOjAgQPWMluzZk25urpax5QuXVp79uy54+/HnaDMAgAAAIDBHnroIc2ePVtubm4qU6aMChT4X1X7a3GUpCtXrqhBgwb6+OOPszxOiRIl5OKSs7NJ69evr8TERH399df67rvv9O9//1uhoaFasWJF7r4YSQULFrT53GKxKCMjI9ePl51cnTP7zjvvKDAwUB4eHmrcuLG2b99+2/EzZsxQ1apVVahQIQUEBGj48OG6du1argIDAAAAgKMpXLiwgoKCVL58eZsim5369evr0KFDKlmypIKCgmxuPj4+8vLyUmBgoKKiou74+b29vdWjRw/NnTtXy5Yt02effaYLFy5kGVe9enXFx8crJSXFui06OlouLi7WxanyS47L7LJlyxQREaHx48dr165dqlOnjtq2baszZ85kO37JkiUaNWqUxo8fr3379mn+/PlatmyZXnrppbsODwAAAADOpk+fPvLz81Pnzp21ZcsWJSYmauPGjXr22Wd14sQJSdKECRP0xhtvaObMmTp06JB27dqlt99+O9vHmz59uj755BPt379fBw8e1PLly1WqVCn5+vpm+9weHh4KCwvTzz//rA0bNmjo0KF6/PHHrVOM80uOy+z06dM1aNAg9e/fXzVq1NCcOXPk6empBQsWZDv+hx9+UNOmTdW7d28FBgaqTZs26tWr1z8ezQUAAAAAZOXp6anNmzerfPny6tatm6pXr66BAwfq2rVr8vb2liSFhYVpxowZevfdd1WzZk09+uijOnToULaP5+XlpWnTpikkJEQNGzbU0aNHtXbt2mynK3t6euqbb77RhQsX1LBhQ/3rX/9S69atNWvWLLt+zdmxZGZmZt7p4LS0NHl6emrFihXq0qWLdXtYWJguXryoL774Ist9lixZosGDB2v9+vVq1KiRjhw5og4dOujxxx+/5dHZ1NRUpaamWj+/fPmyAgICdOnSJeuLYzgWgAIAAA4ocNSaHI0/6tE7R+ODK5bP0XhJ+jTyRo7GV9+/L8fPAfO4fPmyfHx8su0G165dU2JioipWrCgPDw+DEuJu5OQ1zNECUOfOnVN6enqWw8f+/v7av39/tvfp3bu3zp07p2bNmikzM1M3btzQ008/fdtpxpGRkZo4cWJOogEAAAAAnEiuFoDKiY0bN2rKlCl69913tWvXLq1cuVJr1qzRK6+8csv7jB49WpcuXbLejh8/bu+YAAAAAAATydGRWT8/P7m6uur06dM220+fPq1SpUple5+xY8fq8ccf15NPPilJCg4OVkpKip566im9/PLL2c7Ddnd3l7u7e06iAQAAAACcSI6OzLq5ualBgwY2SzxnZGQoKipKTZo0yfY+V69ezVJY/7x4bg5O1wUAAAAAwCpHR2YlKSIiQmFhYQoJCVGjRo00Y8YMpaSkqH///pKkfv36qWzZsoqMjJQkdezYUdOnT1e9evXUuHFjJSQkaOzYserYsaO11AIAAAAAkBM5LrM9evTQ2bNnNW7cOCUnJ6tu3bpat26ddVGopKQkmyOxY8aMkcVi0ZgxY3Ty5EmVKFFCHTt21OTJk/Puq8gDOV+5L2ePH7w4OEfj94TtydkTAAAAAIATyXGZlaTw8HCFh4dnu2/jxo22T1CggMaPH6/x48fn5qkAAAAAAMjC7qsZAwAAAACQ1yizAAAAAADTydU0Y9jfvmrVczS++v59dkoCAAAAwNFZLBZ9/vnn6tKli44ePaqKFStq9+7dqlu3rtHRbokjswAAAABgoCeeeEIWi0UWi0UFCxZUxYoV9eKLL+ratWtGR7uncWQWAAAAgEPL6ZVF7lZurkzSrl07LVy4UNevX1dsbKzCwsJksVj03//+1w4JHQNHZmFeE3xydgMAAADuUe7u7ipVqpQCAgLUpUsXhYaG6ttvv5UkZWRkKDIyUhUrVlShQoVUp04drVixwub+e/fu1aOPPipvb295eXnpwQcf1OHDhyVJO3bs0MMPPyw/Pz/5+PioRYsW2rVrV75/jXmNI7MAYGY5faNmwiX75AAAAHnm559/1g8//KAKFSpIkiIjI/XRRx9pzpw5qlKlijZv3qy+ffuqRIkSatGihU6ePKnmzZurZcuW+v777+Xt7a3o6GjduHFDkvT7778rLCxMb7/9tjIzM/XGG2+offv2OnTokLy8vIz8Uu8KZRb3jMBRa3I0/qhHzh4/p9NLcjM9BLhb/BwAAHKMNzYdwldffaUiRYroxo0bSk1NlYuLi2bNmqXU1FRNmTJF3333nZo0aSJJqlSpkrZu3ar33ntPLVq00DvvvCMfHx8tXbpUBQsWlCTdd9991sdu1aqVzXO9//778vX11aZNm/Too4/m3xeZxyizAIBbYmV1AMh/vLHpnB566CHNnj1bKSkpevPNN1WgQAF1795de/fu1dWrV/Xwww/bjE9LS1O9evUkSXFxcXrwwQetRfbvTp8+rTFjxmjjxo06c+aM0tPTdfXqVSUlJdn967InyixwC/wRDwAAkPO/iST+LsqNwoULKygoSJK0YMEC1alTR/Pnz1etWrUkSWvWrFHZsmVt7uPu7i5JKlSo0G0fOywsTOfPn9dbb72lChUqyN3dXU2aNFFaWpodvpL8Q5kFkHtMawIAAMhzLi4ueumllxQREaGDBw/K3d1dSUlJatGiRbbja9eurcWLF+v69evZHp2Njo7Wu+++q/bt20uSjh8/rnPnztn1a8gPlFkAVkxrAgAAuDc89thjGjFihN577z298MILGj58uDIyMtSsWTNdunRJ0dHR8vb2VlhYmMLDw/X222+rZ8+eGj16tHx8fLRt2zY1atRIVatWVZUqVfThhx8qJCREly9f1ogRI/7xaK4ZUGYB3LOY6g0AAJxVgQIFFB4ermnTpikxMVElSpRQZGSkjhw5Il9fX9WvX18vvfSSJKl48eL6/vvvNWLECLVo0UKurq6qW7eumjZtKkmaP3++nnrqKdWvX18BAQGaMmWKXnjhBSO/vDxBmQUAAADg0O712V6LFi3KdvuoUaM0atQoSdKwYcM0bNiwWz5G7dq19c0332S7r169etqxY4fNtn/96182n2dmZlo/DgwMtPn8XuVidAAAAAAAAHKKMgsAAAAAMB3KLAAAAADAdDhnFgCA/5fTFb0l6ahH7xyND65YPkfj7/XzvPJazldVz9n3X+I1AABHQZkFAOAexqrexuM1AIB7E9OMAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAAM98cQTslgsWW4JCQmSpM2bN6tjx44qU6aMLBaLVq1aZWzgewSX5gEAAADg0HJ6ia27lZtLdLVr104LFy602VaiRAlJUkpKiurUqaMBAwaoW7dueZIxL2VmZio9PV0FCuRvveTILAAAAAAYzN3dXaVKlbK5ubq6SpIeeeQRvfrqq+ratesdP158fLweeugheXl5ydvbWw0aNNDOnTut+6Ojo9WyZUt5enqqaNGiatu2rX777TdJUmpqqp599lmVLFlSHh4eatasmXbs2GG978aNG2WxWPT111+rQYMGcnd319atW5WRkaHIyEhVrFhRhQoVUp06dbRixYo8+g5lRZkFAAAAAAfTp08flStXTjt27FBsbKxGjRqlggULSpLi4uLUunVr1ahRQzExMdq6das6duyo9PR0SdKLL76ozz77TIsXL9auXbsUFBSktm3b6sKFCzbPMWrUKE2dOlX79u1T7dq1FRkZqQ8++EBz5szR3r17NXz4cPXt21ebNm2yy9fINGMAAAAAMNhXX32lIkWKWD9/5JFHtHz58lw/XlJSkkaMGKFq1apJkqpUqWLdN23aNIWEhOjdd9+1bqtZs6akm1OaZ8+erUWLFumRRx6RJM2dO1fffvut5s+frxEjRljvM2nSJD388MOSbh7NnTJlir777js1adJEklSpUiVt3bpV7733nlq0aJHrr+VWKLMAAAAAYLCHHnpIs2fPtn5euHDhu3q8iIgIPfnkk/rwww8VGhqqxx57TJUrV5Z088jsY489lu39Dh8+rOvXr6tp06bWbQULFlSjRo20b5/tucAhISHWjxMSEnT16lVruf1TWlqa6tWrd1dfy61QZgEAAADAYIULF1ZQUFCePd6ECRPUu3dvrVmzRl9//bXGjx+vpUuXqmvXripUqFCePMdfC/eVK1ckSWvWrFHZsmVtxrm7u+fJ8/0d58wCAAAAgAO67777NHz4cK1fv17dunWzrpZcu3ZtRUVFZXufypUry83NTdHR0dZt169f144dO1SjRo1bPleNGjXk7u6upKQkBQUF2dwCAgLy9gv7fxyZBQAAAIB72JUrV6zXnJWkxMRExcXFqVixYipfvnyW8X/88YdGjBihf/3rX6pYsaJOnDihHTt2qHv37pKk0aNHKzg4WIMHD9bTTz8tNzc3bdiwQY899pj8/Pz0zDPPaMSIEdbHnzZtmq5evaqBAwfeMqOXl5deeOEFDR8+XBkZGWrWrJkuXbqk6OhoeXt7KywsLM+/L5RZAAAAALiH7dy5Uw899JD184iICElSWFiYFi1alGW8q6urzp8/r379+un06dPy8/NTt27dNHHiREk3j9iuX79eL730kho1aqRChQqpcePG6tWrlyRp6tSpysjI0OOPP67ff/9dISEh+uabb1S0aNHb5nzllVdUokQJRUZG6siRI/L19VX9+vX10ksv5dF3whZlFgAAAIBDq75/3z8PMlB2hfSvWrZsqczMzDt+PDc3N33yySe3HdOiRQubqcR/5eHhoZkzZ2rmzJk5ymOxWDRs2DANGzbsjrPeDc6ZBQAAAACYDmUWAAAAAGA6lFkAAAAAgOlQZgEAAAAApkOZBQAAAACYDmUWAAAAgEPJycq/uLfk5LWjzAIAAABwCK6urpKktLQ0g5Mgt65evSpJKliw4D+O5TqzAAAAABxCgQIF5OnpqbNnz6pgwYJyceHYnVlkZmbq6tWrOnPmjHx9fa1vTNwOZRYAAACAQ7BYLCpdurQSExN17Ngxo+MgF3x9fVWqVKk7GkuZBQAAAOAw3NzcVKVKFaYam1DBggXv6IjsnyizAAAAAByKi4uLPDw8jI4BO2MSOQAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdHJVZt955x0FBgbKw8NDjRs31vbt2285tmXLlrJYLFluHTp0yHVoAAAAAIBzy3GZXbZsmSIiIjR+/Hjt2rVLderUUdu2bXXmzJlsx69cuVKnTp2y3n7++We5urrqscceu+vwAAAAAADnlOMyO336dA0aNEj9+/dXjRo1NGfOHHl6emrBggXZji9WrJhKlSplvX377bfy9PSkzAIAAAAAci1HZTYtLU2xsbEKDQ393wO4uCg0NFQxMTF39Bjz589Xz549Vbhw4VuOSU1N1eXLl21uAAAAAAD8KUdl9ty5c0pPT5e/v7/Ndn9/fyUnJ//j/bdv366ff/5ZTz755G3HRUZGysfHx3oLCAjISUwAAAAAgIPL19WM58+fr+DgYDVq1Oi240aPHq1Lly5Zb8ePH8+nhAAAAAAAMyiQk8F+fn5ydXXV6dOnbbafPn1apUqVuu19U1JStHTpUk2aNOkfn8fd3V3u7u45iQYAAAAAcCI5OjLr5uamBg0aKCoqyrotIyNDUVFRatKkyW3vu3z5cqWmpqpv3765SwoAAAAAwP/L0ZFZSYqIiFBYWJhCQkLUqFEjzZgxQykpKerfv78kqV+/fipbtqwiIyNt7jd//nx16dJFxYsXz5vkAAAAAACnleMy26NHD509e1bjxo1TcnKy6tatq3Xr1lkXhUpKSpKLi+0B3wMHDmjr1q1av3593qQGAAAAADi1HJdZSQoPD1d4eHi2+zZu3JhlW9WqVZWZmZmbpwIAAAAAIIt8Xc0YAAAAAIC8QJkFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJhOrsrsO++8o8DAQHl4eKhx48bavn37bcdfvHhRQ4YMUenSpeXu7q777rtPa9euzVVgAAAAAAAK5PQOy5YtU0REhObMmaPGjRtrxowZatu2rQ4cOKCSJUtmGZ+WlqaHH35YJUuW1IoVK1S2bFkdO3ZMvr6+eZEfAAAAAOCEclxmp0+frkGDBql///6SpDlz5mjNmjVasGCBRo0alWX8ggULdOHCBf3www8qWLCgJCkwMPDuUgMAAAAAnFqOphmnpaUpNjZWoaGh/3sAFxeFhoYqJiYm2/usXr1aTZo00ZAhQ+Tv769atWppypQpSk9Pv7vkAAAAAACnlaMjs+fOnVN6err8/f1ttvv7+2v//v3Z3ufIkSP6/vvv1adPH61du1YJCQkaPHiwrl+/rvHjx2d7n9TUVKWmplo/v3z5ck5iAgAAAAAcnN1XM87IyFDJkiX1/vvvq0GDBurRo4defvllzZkz55b3iYyMlI+Pj/UWEBBg75gAAAAAABPJUZn18/OTq6urTp8+bbP99OnTKlWqVLb3KV26tO677z65urpat1WvXl3JyclKS0vL9j6jR4/WpUuXrLfjx4/nJCYAAAAAwMHlqMy6ubmpQYMGioqKsm7LyMhQVFSUmjRpku19mjZtqoSEBGVkZFi3HTx4UKVLl5abm1u293F3d5e3t7fNDQAAAACAP+V4mnFERITmzp2rxYsXa9++fXrmmWeUkpJiXd24X79+Gj16tHX8M888owsXLmjYsGE6ePCg1qxZoylTpmjIkCF591UAAAAAAJxKji/N06NHD509e1bjxo1TcnKy6tatq3Xr1lkXhUpKSpKLy/86ckBAgL755hsNHz5ctWvXVtmyZTVs2DCNHDky774KAAAAAIBTyXGZlaTw8HCFh4dnu2/jxo1ZtjVp0kTbtm3LzVMBAAAAAJCF3VczBgAAAAAgr1FmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJhOrsrsO++8o8DAQHl4eKhx48bavn37LccuWrRIFovF5ubh4ZHrwAAAAAAA5LjMLlu2TBERERo/frx27dqlOnXqqG3btjpz5swt7+Pt7a1Tp05Zb8eOHbur0AAAAAAA55bjMjt9+nQNGjRI/fv3V40aNTRnzhx5enpqwYIFt7yPxWJRqVKlrDd/f/+7Cg0AAAAAcG45KrNpaWmKjY1VaGjo/x7AxUWhoaGKiYm55f2uXLmiChUqKCAgQJ07d9bevXtv+zypqam6fPmyzQ0AAAAAgD/lqMyeO3dO6enpWY6s+vv7Kzk5Odv7VK1aVQsWLNAXX3yhjz76SBkZGXrggQd04sSJWz5PZGSkfHx8rLeAgICcxAQAAAAAODi7r2bcpEkT9evXT3Xr1lWLFi20cuVKlShRQu+9994t7zN69GhdunTJejt+/Li9YwIAAAAATKRATgb7+fnJ1dVVp0+fttl++vRplSpV6o4eo2DBgqpXr54SEhJuOcbd3V3u7u45iQYAAAAAcCI5OjLr5uamBg0aKCoqyrotIyNDUVFRatKkyR09Rnp6uvbs2aPSpUvnLCkAAAAAAP8vR0dmJSkiIkJhYWEKCQlRo0aNNGPGDKWkpKh///6SpH79+qls2bKKjIyUJE2aNEn333+/goKCdPHiRb322ms6duyYnnzyybz9SgAAAAAATiPHZbZHjx46e/asxo0bp+TkZNWtW1fr1q2zLgqVlJQkF5f/HfD97bffNGjQICUnJ6to0aJq0KCBfvjhB9WoUSPvvgoAAAAAgFPJcZmVpPDwcIWHh2e7b+PGjTafv/nmm3rzzTdz8zQAAAAAAGTL7qsZAwAAAACQ1yizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEyHMgsAAAAAMB3KLAAAAADAdCizAAAAAADTocwCAAAAAEwnV2X2nXfeUWBgoDw8PNS4cWNt3779ju63dOlSWSwWdenSJTdPCwAAAACApFyU2WXLlikiIkLjx4/Xrl27VKdOHbVt21Znzpy57f2OHj2qF154QQ8++GCuwwIAAAAAIOWizE6fPl2DBg1S//79VaNGDc2ZM0eenp5asGDBLe+Tnp6uPn36aOLEiapUqdJdBQYAAAAAIEdlNi0tTbGxsQoNDf3fA7i4KDQ0VDExMbe836RJk1SyZEkNHDjwjp4nNTVVly9ftrkBAAAAAPCnHJXZc+fOKT09Xf7+/jbb/f39lZycnO19tm7dqvnz52vu3Ll3/DyRkZHy8fGx3gICAnISEwAAAADg4Oy6mvHvv/+uxx9/XHPnzpWfn98d32/06NG6dOmS9Xb8+HE7pgQAAAAAmE2BnAz28/OTq6urTp8+bbP99OnTKlWqVJbxhw8f1tGjR9WxY0frtoyMjJtPXKCADhw4oMqVK2e5n7u7u9zd3XMSDQAAAADgRHJ0ZNbNzU0NGjRQVFSUdVtGRoaioqLUpEmTLOOrVaumPXv2KC4uznrr1KmTHnroIcXFxTF9GAAAAACQKzk6MitJERERCgsLU0hIiBo1aqQZM2YoJSVF/fv3lyT169dPZcuWVWRkpDw8PFSrVi2b+/v6+kpSlu0AAAAAANypHJfZHj166OzZsxo3bpySk5NVt25drVu3zrooVFJSklxc7HoqLgAAAADAyeW4zEpSeHi4wsPDs923cePG29530aJFuXlKAAAAAACsOIQKAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHQoswAAAAAA06HMAgAAAABMhzILAAAAADAdyiwAAAAAwHRyVWbfeecdBQYGysPDQ40bN9b27dtvOXblypUKCQmRr6+vChcurLp16+rDDz/MdWAAAAAAAHJcZpctW6aIiAiNHz9eu3btUp06ddS2bVudOXMm2/HFihXTyy+/rJiYGP3000/q37+/+vfvr2+++eauwwMAAAAAnFOOy+z06dM1aNAg9e/fXzVq1NCcOXPk6empBQsWZDu+ZcuW6tq1q6pXr67KlStr2LBhql27trZu3XrX4QEAAAAAzilHZTYtLU2xsbEKDQ393wO4uCg0NFQxMTH/eP/MzExFRUXpwIEDat68ec7TAgAAAAAgqUBOBp87d07p6eny9/e32e7v76/9+/ff8n6XLl1S2bJllZqaKldXV7377rt6+OGHbzk+NTVVqamp1s8vX76ck5gAAAAAAAeXozKbW15eXoqLi9OVK1cUFRWliIgIVapUSS1btsx2fGRkpCZOnJgf0QAAAAAAJpSjMuvn5ydXV1edPn3aZvvp06dVqlSpW97PxcVFQUFBkqS6detq3759ioyMvGWZHT16tCIiIqyfX758WQEBATmJCgAAAABwYDk6Z9bNzU0NGjRQVFSUdVtGRoaioqLUpEmTO36cjIwMm2nEf+fu7i5vb2+bGwAAAAAAf8rxNOOIiAiFhYUpJCREjRo10owZM5SSkqL+/ftLkvr166eyZcsqMjJS0s0pwyEhIapcubJSU1O1du1affjhh5o9e3befiUAAAAAAKeR4zLbo0cPnT17VuPGjVNycrLq1q2rdevWWReFSkpKkovL/w74pqSkaPDgwTpx4oQKFSqkatWq6aOPPlKPHj3y7qsAAAAAADiVXC0AFR4ervDw8Gz3bdy40ebzV199Va+++mpungYAAAAAgGzl6JxZAAAAAADuBZRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKZDmQUAAAAAmA5lFgAAAABgOpRZAAAAAIDpUGYBAAAAAKaTqzL7zjvvKDAwUB4eHmrcuLG2b99+y7Fz587Vgw8+qKJFi6po0aIKDQ297XgAAAAAAP5JjsvssmXLFBERofHjx2vXrl2qU6eO2rZtqzNnzmQ7fuPGjerVq5c2bNigmJgYBQQEqE2bNjp58uRdhwcAAAAAOKccl9np06dr0KBB6t+/v2rUqKE5c+bI09NTCxYsyHb8xx9/rMGDB6tu3bqqVq2a5s2bp4yMDEVFRd11eAAAAACAc8pRmU1LS1NsbKxCQ0P/9wAuLgoNDVVMTMwdPcbVq1d1/fp1FStWLGdJAQAAAAD4fwVyMvjcuXNKT0+Xv7+/zXZ/f3/t37//jh5j5MiRKlOmjE0h/rvU1FSlpqZaP798+XJOYgIAAAAAHFy+rmY8depULV26VJ9//rk8PDxuOS4yMlI+Pj7WW0BAQD6mBAAAAADc63JUZv38/OTq6qrTp0/bbD99+rRKlSp12/u+/vrrmjp1qtavX6/atWvfduzo0aN16dIl6+348eM5iQkAAAAAcHA5KrNubm5q0KCBzeJNfy7m1KRJk1veb9q0aXrllVe0bt06hYSE/OPzuLu7y9vb2+YGAAAAAMCfcnTOrCRFREQoLCxMISEhatSokWbMmKGUlBT1799fktSvXz+VLVtWkZGRkqT//ve/GjdunJYsWaLAwEAlJydLkooUKaIiRYrk4ZcCAAAAAHAWOS6zPXr00NmzZzVu3DglJyerbt26WrdunXVRqKSkJLm4/O+A7+zZs5WWlqZ//etfNo8zfvx4TZgw4e7SAwAAAACcUo7LrCSFh4crPDw8230bN260+fzo0aO5eQoAAAAAAG4pX1czBgAAAAAgL1BmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJhOrsrsO++8o8DAQHl4eKhx48bavn37Lcfu3btX3bt3V2BgoCwWi2bMmJHbrAAAAAAASMpFmV22bJkiIiI0fvx47dq1S3Xq1FHbtm115syZbMdfvXpVlSpV0tSpU1WqVKm7DgwAAAAAQI7L7PTp0zVo0CD1799fNWrU0Jw5c+Tp6akFCxZkO75hw4Z67bXX1LNnT7m7u991YAAAAAAAclRm09LSFBsbq9DQ0P89gIuLQkNDFRMTk2ehUlNTdfnyZZsbAAAAAAB/ylGZPXfunNLT0+Xv72+z3d/fX8nJyXkWKjIyUj4+PtZbQEBAnj02AAAAAMD87snVjEePHq1Lly5Zb8ePHzc6EgAAAADgHlIgJ4P9/Pzk6uqq06dP22w/ffp0ni7u5O7uzvm1AAAAAIBbytGRWTc3NzVo0EBRUVHWbRkZGYqKilKTJk3yPBwAAAAAANnJ0ZFZSYqIiFBYWJhCQkLUqFEjzZgxQykpKerfv78kqV+/fipbtqwiIyMl3Vw06pdffrF+fPLkScXFxalIkSIKCgrKwy8FAAAAAOAsclxme/ToobNnz2rcuHFKTk5W3bp1tW7dOuuiUElJSXJx+d8B319//VX16tWzfv7666/r9ddfV4sWLbRx48a7/woAAAAAAE4nx2VWksLDwxUeHp7tvr8X1MDAQGVmZubmaQAAAAAAyNY9uZoxAAAAAAC3Q5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJgOZRYAAAAAYDqUWQAAAACA6VBmAQAAAACmQ5kFAAAAAJhOrsrsO++8o8DAQHl4eKhx48bavn37bccvX75c1apVk4eHh4KDg7V27dpchQUAAAAAQMpFmV22bJkiIiI0fvx47dq1S3Xq1FHbtm115syZbMf/8MMP6tWrlwYOHKjdu3erS5cu6tKli37++ee7Dg8AAAAAcE45LrPTp0/XoEGD1L9/f9WoUUNz5syRp6enFixYkO34t956S+3atdOIESNUvXp1vfLKK6pfv75mzZp11+EBAAAAAM4pR2U2LS1NsbGxCg0N/d8DuLgoNDRUMTEx2d4nJibGZrwktW3b9pbjAQAAAAD4JwVyMvjcuXNKT0+Xv7+/zXZ/f3/t378/2/skJydnOz45OfmWz5OamqrU1FTr55cuXZIkXb58OSdxcyQj9WqOxl+2ZOZofPof6TkafyU9Z+Pt+b3JL7wGxuM1MB6vgbFy+v2XeA3ymr1/BiReg39yr/0ekngN/sm99ntIsu9r8OdjZ2bm/OcfjiVHZTa/REZGauLEiVm2BwQEGJAmez45vse+HI1ulNOH98l5IrPjNTAer4HxeA2Mx2tgrNx9tbwGecnePwMSr8E/ued+D0n58hr8/vvv8nGy1xq2clRm/fz85OrqqtOnT9tsP336tEqVKpXtfUqVKpWj8ZI0evRoRUREWD/PyMjQhQsXVLx4cVkslpxEvidcvnxZAQEBOn78uLy9vY2O45R4DYzHa2A8XgPj8RoYj9fAWHz/jecIr0FmZqZ+//13lSlTxugoMFiOyqybm5saNGigqKgodenSRdLNohkVFaXw8PBs79OkSRNFRUXpueees2779ttv1aRJk1s+j7u7u9zd3W22+fr65iTqPcnb29u0vzQcBa+B8XgNjMdrYDxeA+PxGhiL77/xzP4acEQWUi6mGUdERCgsLEwhISFq1KiRZsyYoZSUFPXv31+S1K9fP5UtW1aRkZGSpGHDhqlFixZ644031KFDBy1dulQ7d+7U+++/n7dfCQAAAADAaeS4zPbo0UNnz57VuHHjlJycrLp162rdunXWRZ6SkpLk4vK/RZIfeOABLVmyRGPGjNFLL72kKlWqaNWqVapVq1befRUAAAAAAKeSqwWgwsPDbzmteOPGjVm2PfbYY3rsscdy81QOwd3dXePHj88ydRr5h9fAeLwGxuM1MB6vgfF4DYzF9994vAZwJJZM1rQGAAAAAJiMyz8PAQAAAADg3kKZBQAAAACYDmUWAAAAAGA6lFkAAAAAgOlQZgEAAGAX169f14ABA5SYmGh0FAAOiNWM7SwtLU2JiYmqXLmyChTI1ZWQANP54IMP7mhcv3797JwEAGA0Hx8fxcXFqWLFikZHAeBgKLN2cvXqVQ0dOlSLFy+WJB08eFCVKlXS0KFDVbZsWY0aNcrghM7j8OHDWrhwoQ4fPqy33npLJUuW1Ndff63y5curZs2aRsdzSC4uLipSpIgKFCigW/2KsVgsunDhQj4nA/LHgAED7mjcggUL7JzEea1evfqOx3bq1MmOSRAWFqa6detq+PDhRkdxWq6urjp16pRKlixps/38+fMqWbKk0tPTDUoG3B0OFdrJ6NGjFR8fr40bN6pdu3bW7aGhoZowYQJlNp9s2rRJjzzyiJo2barNmzdr8uTJKlmypOLj4zV//nytWLHC6IgOqXr16jp9+rT69u2rAQMGqHbt2kZHcloVK1aUxWK55f4jR47kYxrnsWjRIlWoUEH16tW75Rs6sK8uXbrYfG6xWGxei7/+XPCHvH1VqVJFkyZNUnR0tBo0aKDChQvb7H/22WcNSuY8bvV7KDU1VW5ubvmcBsg7lFk7WbVqlZYtW6b777/f5h/MmjVr6vDhwwYmcy6jRo3Sq6++qoiICHl5eVm3t2rVSrNmzTIwmWPbu3evfvzxRy1YsEDNmzdXUFCQBg4cqD59+sjb29voeE7lueees/n8+vXr2r17t9atW6cRI0YYE8oJPPPMM/rkk0+UmJio/v37q2/fvipWrJjRsZxKRkaG9ePvvvtOI0eO1JQpU9SkSRNJUkxMjMaMGaMpU6YYFdFpzJ8/X76+voqNjVVsbKzNPovFQpm1o5kzZ0q6+X2eN2+eihQpYt2Xnp6uzZs3q1q1akbFA+4a04ztxNPTUz///LMqVaokLy8vxcfHq1KlSoqPj1fz5s116dIloyM6hSJFimjPnj2qWLGizetw9OhRVatWTdeuXTM6osP7448/tHz5ci1cuFDbt29Xly5dtGDBArm7uxsdzam988472rlzpxYuXGh0FIeVmpqqlStXasGCBfrhhx/UoUMHDRw4UG3atLnt0XLkvVq1amnOnDlq1qyZzfYtW7boqaee0r59+wxKBtjXn+cpHzt2TOXKlZOrq6t1n5ubmwIDAzVp0iQ1btzYqIjAXWE1YzsJCQnRmjVrrJ//+YfLvHnzrO8Kw/58fX116tSpLNt3796tsmXLGpDI+RQqVEj9+vXTxIkT1ahRIy1dulRXr141OpbTe+SRR/TZZ58ZHcOhubu7q1evXvr222/1yy+/qGbNmho8eLACAwN15coVo+M5lcOHD8vX1zfLdh8fHx09ejTf8zirtLQ0HThwQDdu3DA6itNITExUYmKiWrRoofj4eOvniYmJOnDggL755huKLEyNMmsnU6ZM0UsvvaRnnnlGN27c0FtvvaU2bdpo4cKFmjx5stHxnEbPnj01cuRIJScny2KxKCMjQ9HR0XrhhRdYSTcfnDx5UlOmTFGVKlXUs2dPNWzYUHv37lXRokWNjub0VqxYwbTXfOTi4mI9Z5PzM/Nfw4YNFRERodOnT1u3nT59WiNGjFCjRo0MTOYcrl69qoEDB8rT01M1a9ZUUlKSJGno0KGaOnWqwemcw4YNG/i3Fw6JacZ2dPjwYU2dOlXx8fG6cuWK6tevr5EjRyo4ONjoaE4jLS1NQ4YM0aJFi5Senq4CBQooPT1dvXv31qJFi2ym2yDvfPrpp1q4cKE2bdqktm3bqn///urQoQPfbwPUq1fPZkprZmamkpOTdfbsWb377rt66qmnDEzn2P46zXjr1q169NFH1b9/f7Vr104uLryXnJ8SEhLUtWtXHTx4UAEBAZKk48ePq0qVKlq1apWCgoIMTujYhg0bpujoaM2YMUPt2rXTTz/9pEqVKumLL77QhAkTtHv3bqMjOrz09HQtWrRIUVFROnPmjM055ZL0/fffG5QMuDuUWTiF48ePa8+ePbpy5Yrq1aunKlWqGB3Jobm4uKh8+fLq06eP/P39bzmORT/sb+LEiTafu7i4qESJEmrZsiWLftjR4MGDtXTpUgUEBGjAgAHq06eP/Pz8jI7l1DIzM/Xtt99q//79km6uuh4aGsr5y/mgQoUK1kUx/7p+RUJCgurXr6/Lly8bHdHhhYeHa9GiRerQoYNKly6d5f/7N99806BkwN2hzNpRRkaGEhISsn0HrHnz5galcm7p6enas2ePKlSowHQbOwoMDPzHPxAtFguXhbGzGzduaMmSJWrbtu1t31RA3vvzDZ2/Hxn/u5UrV+ZjKsAYLIppPD8/P33wwQdq37690VGAPMWleexk27Zt6t27t44dO5bl2l4Wi4VzpvLJc889p+DgYA0cOFDp6elq0aKFfvjhB3l6euqrr75Sy5YtjY7okFhQ5d5QoEABPf3006zUaoB+/fpxxM8ETp8+rffee0/jxo0zOopD+3NRzKFDh0piUUwjuLm5MZ0eDokjs3ZSt25d3XfffZo4cWK20zl8fHwMSuZcypUrp1WrVikkJESrVq3S4MGDtXHjRn344Yf6/vvvFR0dbXREh9SqVSutXLky29VDkb9atmyp5557Tl26dDE6CnDPiY+PV/369XmD2c62bt2qRx55RH379tWiRYv0n//8R7/88ot++OEHbdq0SQ0aNDA6osN74403dOTIEc2aNYs32uBQKLN2UrhwYcXHx/MumME8PDyUkJCgcuXK6amnnpKnp6dmzJihxMRE1alTh/N07MTFxUXJyckqWbKk0VGc3qeffqrRo0dr+PDhatCggQoXLmyzv3bt2gYlc2yurq46deoUPwMG++mnn267f//+/erVqxdlNh+wKKaxunbtqg0bNqhYsWKqWbOmChYsaLOfUx5gVkwztpPGjRsrISGBMmswf39//fLLLypdurTWrVun2bNnS7p5mQBW1oUz6NmzpyTbxbb+vEQMpzzYD+8T3xvq1q1r/f/97/76cwD7q1y5subOnWt0DKfl6+urrl27Gh0DyHOUWTsZOnSonn/+eSUnJys4ODjLO2AcDckf/fv317///W/rVO/Q0FBJ0o8//shKrnb2yy+/KDk5+bZj+Dmwv8TERKMjAIYpVqyYpk2bptatW2e7f+/everYsWM+p3IOOZn55O3tbcckkKSFCxcaHQGwC6YZ20l21xDkaIgxVqxYoePHj+uxxx5TuXLlJEmLFy+Wr6+vOnfubHA6x+Ti4nJHR0P4OYCjcnFx0auvvqoiRYrcdhyXp7Kvtm3b6sEHH9SYMWOy3R8fH6969eplueIA7t6f/w7cCf4tyB83btzQxo0bdfjwYfXu3VteXl769ddf5e3t/Y+/q4B7FWXWTo4dO3bb/RUqVMinJED+c3Fx0fbt21WiRInbjuPnwP5Wr16d7XaLxSIPDw8FBQWpYsWK+ZzK8bm4uKhcuXK3PZ2By1PZ3+eff66UlBT17ds32/2//fabVq9erbCwsHxO5vg2bdpk/fjo0aMaNWqUnnjiCevqxTExMVq8eLEiIyP5/ueDY8eOqV27dkpKSlJqaqoOHjyoSpUqadiwYUpNTdWcOXOMjgjkCmUWDi8qKkpRUVHZXu93wYIFBqVybCwAde+41VHyvx4hb9asmVatWsW1l/MQPwPA/7Ru3VpPPvmkevXqZbN9yZIlev/997Vx40ZjgjmRLl26yMvLS/Pnz1fx4sWt1/rduHGjBg0apEOHDhkdEciVrHNhkWurV6/W9evXrR/f7ob8MXHiRLVp00ZRUVE6d+6cfvvtN5sb4Oi+/fZbNWzYUN9++60uXbqkS5cu6dtvv1Xjxo311VdfafPmzTp//rxeeOEFo6M6FBYVurds3brV6AhOLSYmRiEhIVm2h4SEaPv27QYkcj5btmzRmDFj5ObmZrM9MDBQJ0+eNCgVcPdYACoPdenSxfpO/O2u6ci5gvlnzpw5WrRokR5//HGjoziVFi1aZPkHE8YYNmyY3n//fT3wwAPWba1bt5aHh4eeeuop7d27VzNmzNCAAQMMTOl4mPR0b2nVqpXKli2rXr16qW/fvqpRo4bRkZxKQECA5s6dq2nTptlsnzdvngICAgxK5VwyMjKy/dvzxIkT8vLyMiARkDcos3nor1NYWUzi3pCWlmbzRzzyx4YNG7Jsu3btmpYtW6aUlBQ9/PDDqlKligHJnM/hw4ezXSnU29vber5mlSpVdO7cufyO5tDGjx/Pgir3kF9//VVLly7VJ598oqlTp6p27drq06ePevXqZV0YEPbz5ptvqnv37vr666/VuHFjSdL27dt16NAhffbZZwancw5t2rTRjBkz9P7770u6eWDlypUrGj9+vNq3b29wOiD3OGcWDm3kyJEqUqSIxo4da3QUpxIREaHr16/r7bfflnTzTYXGjRtr79698vT01I0bN/Ttt99aFwKB/TRr1kxeXl764IMPrAtynT17Vv369VNKSoo2b96s7777TkOGDNGBAwcMTus4zp07p5SUFJtFzvbu3avXX39dKSkp6tKli3r37m1gQueVmJioJUuW6JNPPtH+/fvVvHlzff/990bHcngnTpzQu+++q/3790uSqlevrqeffpojs/nkxIkTatu2rTIzM3Xo0CGFhITo0KFD8vPz0+bNmzm/H6ZFmc1DM2fOvOOxXI4hfwwbNkwffPCBateurdq1a2e53u/06dMNSubYatWqpSlTpqhTp06Sbl7f7vnnn9fu3btVvnx5DRgwQGfOnNGaNWsMTur4Dhw4oM6dOysxMdH6R+Px48dVqVIlffHFF7rvvvu0atUq/f7770zHz0O9evVSmTJl9MYbb0iSzpw5o2rVqqlMmTKqXLmyvv76a82fP5/vuUHS09P19ddfa+zYsfrpp5849QdO4caNG1q2bJni4+N15coV1a9fX3369FGhQoWMjgbkGmU2D93p5S24HEP+eeihh265z2Kx8G68nXh7e2vXrl0KCgqSdPMPey8vL+v0pri4OLVv316//vqrkTGdRkZGhtavX6+DBw9KkqpWraqHH3442+thI29UrFhRixYtUosWLSRJr7/+uubMmaP9+/erQIECev3117VixQpt27bN4KTOJTo6Wh9//LFWrFiha9euqXPnzurTp4/atWtndDSHd/HiRc2fP1/79u2TJNWsWVMDBgyQj4+PwckAmBllFkCe8/X11Y4dO6znxVasWFFjx461LjJ09OhRVa9eXX/88YeRMfEXwcHBWrt2LVP+8kihQoW0f/9+6zTj9u3bq1atWtYFcA4ePKgmTZro/PnzRsZ0GqNHj9bSpUv166+/6uGHH1afPn3UuXNneXp6Gh3NKezcuVNt27ZVoUKF1KhRI0nSjh079Mcff2j9+vWqX7++wQkdX2RkpPz9/bMs9rdgwQKdPXtWI0eONCgZcHd4W97O0tLSdODAAd24ccPoKE4tISFB33zzjbU88R6OfVWvXl1ffvmlpJvnCSYlJdkcJT927Jj8/f2NiodsHD161HppMdw9b29vXbx40fr59u3brQvfSDdnhqSmphqQzDlt3rxZI0aM0MmTJ/XVV1+pV69eFNl8NHz4cHXq1ElHjx7VypUrtXLlSiUmJurRRx/Vc889Z3Q8p/Dee++pWrVqWbbXrFlTc+bMMSARkDcos3Zy9epVDRw4UJ6enqpZs6aSkpIkSUOHDtXUqVMNTuc8zp8/r9atW+u+++5T+/btderUKUnSwIED9fzzzxucznG9+OKLGj16tFq3bq3WrVurffv2NtPw165da313HnBE999/v2bOnKmMjAytWLFCv//+u1q1amXdf/DgQY6C56Po6GgNHjxYfn5+RkdxSjt37tTIkSNVoMD/LqJRoEABvfjii9q5c6eByZxHcnKySpcunWV7iRIlrH8bAWbEpXnsZPTo0YqPj9fGjRttzsUJDQ3VhAkTNGrUKAPTOY/hw4erYMGCSkpKUvXq1a3be/TooYiICOviLMhbXbt21dq1a/XVV1+pTZs2Gjp0qM1+T09PDRkyxKB0gP298sorat26tT766CPduHFDL730kooWLWrdv3TpUuv5tMgfhw4d0oYNG3TmzJksl88bN26cQamcg7e3t5KSkrIcGTx+/DjXOM0nAQEBio6OzrK+S3R0tMqUKWNQKuDuUWbtZNWqVVq2bJnuv/9+WSwW6/aaNWvq8OHDBiZzLuvXr9c333yT5TqCVapU0bFjxwxK5Rz+PCqbnWHDhmnt2rX5nAjIP7Vr19a+ffsUHR2tUqVK2UwxlqSePXuqZs2aBqVzPnPnztUzzzwjPz8/lSpVyubfZYvFQpm1sx49emjgwIF6/fXXrdd+j46O1ogRI9SrVy+D0zmHQYMG6bnnntP169ets0SioqL04osvMlMNpkaZtZOzZ89me82ulJQUm39EYV8pKSnZnhd14cIFubu7G5AI0s1zZh9//HGuswmH5ufnp86dO2e7r06dOpo0aZJ1hW/Y16uvvqrJkyezyI1BXn/9dVksFvXr18+6hkjBggX1zDPPcOpVPhkxYoTOnz+vwYMHKy0tTZLk4eGhkSNHavTo0QanA3KPc2btJCQkxOYamn8W2Hnz5qlJkyZGxXI6Dz74oD744APr5xaLRRkZGZo2bdptL9sDAPZ0/vx5zZ8/3+gYTuO3337TY489ZnQMp+Xm5qa33npLv/32m+Li4hQXF6cLFy7ozTff5I3lfJCenq4tW7Zo1KhROnv2rLZt26b4+HhduHCBWQkwPY7M2smUKVP0yCOP6JdfftGNGzf01ltv6ZdfftEPP/ygTZs2GR3PaUybNk2tW7fWzp07lZaWphdffFF79+7VhQsXFB0dbXQ8wK727dunbdu2qUmTJqpWrZr279+vt956S6mpqerbt6/NgkTvvfceK0zDYT322GNav369nn76aaOjOKVLly4pPT1dxYoVU3BwsHX7hQsXVKBAAXl7exuYzvG5urqqTZs22rdvnypWrKiGDRsaHQnIM5RZO2nWrJni4uI0depUBQcHW6+jFhMTY/OLHPZVq1YtHTx4ULNmzZKXl5euXLmibt26aciQIdmu6gc4inXr1qlz584qUqSIrl69qs8//1z9+vVTnTp1lJGRoTZt2mj9+vXWQsuUbziyoKAgjR07Vtu2bVNwcLAKFixos//ZZ581KJlz6Nmzpzp27KjBgwfbbP/000+1evVq1lDIB7Vq1dKRI0eyLAAFmJ0lkwtuAshjM2fOvO3+kydP6vXXX1d6eno+JXI+DzzwgFq1aqVXX31VS5cu1eDBg/XMM89o8uTJkm6uuB4bG6v169cbnNQ5xcfHq379+vwM5JPb/QFvsVh05MiRfEzjfIoVK6bo6GibqwpI0v79+9W0aVOdP3/eoGTOY926dRo9erReeeUVNWjQQIULF7bZz9FxmBVl1k527dqlggULWo/CfvHFF1q4cKFq1KihCRMmyM3NzeCEjuunn36647G1a9e2YxLndafv/CYmJto5ifPy8fFRbGysgoKClJGRIXd3d23fvl316tWTJP38888KDQ1VcnKywUkdU7du3W67/+LFi9q0aRNlFk6hcOHC1qPif7Vnzx41btxYV69eNSiZ83Bx+d8yOX9diDQzM1MWi4XfRTAtphnbyX/+8x+NGjVKwcHBOnLkiHr06KFu3bpp+fLlunr1qmbMmGF0RIdVt25dWSwW6y/oP/35vs1ft/HL2z4oqfeGP/9fd3FxkYeHh3x8fKz7vLy8dOnSJaOiOby/fq9vtb9fv375lAZ/ld2/BbCvRo0a6f3339fbb79ts33OnDlq0KCBQamcy4YNG4yOANgFZdZODh48qLp160qSli9frhYtWmjJkiWKjo5Wz549KbN29NcitXv3br3wwgsaMWKEdRXpmJgYvfHGG5o2bZpREfE3wcHBWrt2rQICAoyO4jACAwN16NAhVa5cWdLN/+/Lly9v3Z+UlMR543a0cOHCHI0/ceKEypQpY3P0BHnrgw8+0GuvvaZDhw5Jku677z6NGDFCjz/+uMHJHN+rr76q0NBQxcfHW68/HhUVpR07dnCqQz5p0aKF0REAu6DM2klmZqYyMjIkSd99950effRRSVJAQIDOnTtnZDSHV6FCBevHjz32mGbOnKn27dtbt9WuXVsBAQEaO3asunTpYkBC/N3Ro0d1/fp1o2M4lGeeecZm5kGtWrVs9n/99dc2qxnDWDVq1FBcXJwqVapkdBSHNH36dI0dO1bh4eFq2rSpJGnr1q16+umnde7cOQ0fPtzghI6tadOmiomJ0WuvvaZPP/1UhQoVUu3atTV//nxVqVLF6HhOY8uWLXrvvfd05MgRLV++XGXLltWHH36oihUrqlmzZkbHA3KFc2btpFWrVgoICFBoaKgGDhyoX375RUFBQdq0aZPCwsJ09OhRoyM6hUKFCmnXrl1ZFp3Yt2+f6tevrz/++MOgZPgrLy8vxcfH84c8nBY/A/ZVsWJFTZw4McvU7sWLF2vChAmcGgGH99lnn+nxxx9Xnz599OGHH+qXX35RpUqVNGvWLK1du5YVpWFazGeykxkzZmjXrl0KDw/Xyy+/rKCgIEnSihUr9MADDxicznlUr15dkZGRSktLs25LS0tTZGRkloILAHBMp06dyvbf3gceeECnTp0yIJHzycjI0MGDB7V161Zt3rzZ5gb7e/XVVzVnzhzNnTvX5tJUTZs21a5duwxMBtwdphnbSe3atbVnz54s21977TW5uroakMg5zZkzRx07dlS5cuWsKxf/9NNPslgs+vLLLw1OBwDID0FBQfr000/10ksv2WxftmwZ01zzwbZt29S7d28dO3ZMf58QyEq6+ePAgQNq3rx5lu0+Pj66ePFi/gcC8ghlNp95eHgYHcGpNGrUSEeOHNHHH3+s/fv3S5J69Oih3r17Z7nGGgDAMU2cOFE9evTQ5s2brefMRkdHKyoqSp9++qnB6Rzf008/rZCQEK1Zs0alS5dmJWkDlCpVSgkJCQoMDLTZvnXrVk5vgKlRZu0kPT1db775pj799FMlJSXZTHOVpAsXLhiUzPkULlxYTz311G3HdOjQQfPmzWN1VwCG4I97++revbt+/PFHvfnmm1q1apWkm6eh/PXay7CfQ4cOacWKFdZTrpD/Bg0apGHDhmnBggWyWCz69ddfFRMToxdeeEFjx441Oh6Qa5RZO5k4caLmzZun559/XmPGjNHLL7+so0ePatWqVRo3bpzR8fA3mzdvZjEoA7333nvy9/c3OgZgGNZitL8GDRroo48+MjqGU2rcuLESEhIoswYaNWqUMjIy1Lp1a129elXNmzeXu7u7XnjhBQ0dOtToeECusZqxnVSuXFkzZ85Uhw4d5OXlpbi4OOu2bdu2acmSJUZHxF+wkmje++OPPxQbG6tixYqpRo0aNvuuXbumTz/9NMvKooCzOn78uMqUKcOaCnaydu1aubq6qm3btjbbv/nmG2VkZOiRRx4xKJlz+PzzzzVmzBiNGDFCwcHBNgsQSbKuaQH7S0tLU0JCgq5cuaIaNWqoSJEiRkcC7gpl1k4KFy6sffv2qXz58ipdurTWrFmj+vXr68iRI6pXr54uXbpkdET8BWU2bx08eFBt2rRRUlKSLBaLmjVrpqVLl1qncZ8+fVplypRh0Q84pG7dut3x2JUrV9oxCf5Uu3ZtTZ061eaa45K0bt06jRw5UvHx8QYlcw4uLlkvnmGxWJSZmckCUAY4fvy4JCkgIMDgJMDd49I8dlKuXDnrcv+VK1fW+vXrJUk7duyQu7u7kdEAuxs5cqRq1aqlM2fO6MCBA/Ly8lLTpk2VlJRkdDTA7nx8fO74hvxx6NChLDNEJKlatWpKSEgwIJFzSUxMzHI7cuSI9b+wvxs3bmjs2LHy8fFRYGCgAgMD5ePjozFjxuj69etGxwNyjXNm7aRr166KiopS48aNNXToUPXt21fz589XUlKShg8fbnQ8wK5++OEHfffdd/Lz85Ofn5++/PJLDR48WA8++KA2bNjAStJwaAsXLjQ6Av7Gx8dHR44cybKSa0JCAr+P8kGFChWMjuD0hg4dqpUrV2ratGlq0qSJJCkmJkYTJkzQ+fPnNXv2bIMTArnDNON8EhMTo5iYGFWpUkUdO3Y0Og7+hmnGecvb21s//vijqlevbrM9PDxcX3zxhZYsWaKWLVsytQxAvvjPf/6jmJgYff7556pcubKkm0W2e/fuatiwoebNm2dwQsf34Ycfas6cOUpMTFRMTIwqVKigGTNmqGLFiurcubPR8Ryej4+Pli5dmuX88LVr16pXr16c/gbT4shsPmnSpIn1nTDce1566SUVK1bM6BgOo1q1atq5c2eWMjtr1ixJUqdOnYyIBRhixYoVt7xM265duwxK5VymTZumdu3aqVq1aipXrpwk6cSJE3rwwQf1+uuvG5zO8c2ePVvjxo3Tc889p8mTJ1vfyPT19dWMGTMos/nA3d09y8wESapYsaLc3NzyPxCQRzhn1o4OHDig8PBwtW7dWq1bt1Z4eLgOHDhgdCyn8+GHH6pp06YqU6aMjh07JkmaMWOGvvjiC+uY0aNHy9fX16CEjqdr16765JNPst03a9Ys9erVi0uRwCnMnDlT/fv3l7+/v3bv3q1GjRqpePHiOnLkCCvo5iMfHx/98MMPWrNmjQYPHqznn39eUVFR+v777/ndnw/efvttzZ07Vy+//LLNit0hISHas2ePgcmcR3h4uF555RWlpqZat6Wmpmry5MkKDw83MBlwd5hmbCefffaZevbsqZCQEOsR2W3btmnHjh1aunSpunfvbnBC5/D3d4N//vlnVapUSYsWLdLixYu1YcMGoyMCcGDVqlXT+PHj1atXL5vTGcaNG6cLFy5YZyvg3hAcHKy1a9eyymseK1SokPbv368KFSrY/BwcOnRItWvX5jrv+eDPtVzc3d1Vp04dSVJ8fLzS0tLUunVrm7Gssg4zYZqxnbz44osaPXq0Jk2aZLN9/PjxevHFFymz+eTPd4O7dOmiqVOnWreHhITohRdeMDAZAGeQlJSkBx54QNLNP+h///13SdLjjz+u+++/nzJ7jzl69Cgru9pBxYoVFRcXl2UhqHXr1mU5HQX24evrm+VvT960gSOgzNrJqVOn1K9fvyzb+/btq9dee82ARM4pMTFR9erVy7Ld3d1dKSkpBiQC4ExKlSqlCxcuqEKFCipfvry2bdumOnXqKDExkan2cBoREREaMmSIrl27pszMTG3fvl2ffPKJIiMjWXwrn9zpKuvR0dFKTU3lMpIwDcqsnbRs2VJbtmxRUFCQzfatW7fqwQcfNCiV8+HdYABGatWqlVavXq169eqpf//+Gj58uFasWKGdO3eqW7duRscD8sWTTz6pQoUKacyYMbp69ap69+6tMmXK6K233lLPnj2Njoe/eOSRRxQXF8fVHWAalNk8tHr1auvHnTp10siRIxUbG6v7779f0s1zZpcvX66JEycaFdHp8G4wACO9//77ysjIkCQNGTJExYsX1w8//KBOnTrpP//5j8HpgPzTp08f9enTR1evXtWVK1dUsmTJLGOio6MVEhLCUUEDMWMEZsMCUHnIxeXOFoe2WCxcXzMfffzxx5owYYIOHz4sSSpTpowmTpyogQMHGpwMAHAv4ZrjxvL29uaooMH4GYDZcGQ2D/357jvuLXfybjAA2MvFixe1fft2nTlzJsu/E9mtrQA4K46vAMgpyqzBuAxA/vH09JSnp6fRMQA4kS+//FJ9+vTRlStX5O3tLYvFYt1nsVgos/ksJSVFn376qRISElS6dGn16tVLxYsXt+5/77335O/vb2BCAEBOMM3YYEznyHv16tWz+YPxdnbt2mXnNACc2X333af27dtrypQpvJlmgBo1amjr1q0qVqyYjh8/rubNm+u3337Tfffdp8OHD6tAgQLatm2bKlasaHRUiL+J7gVM9YbZcGQWDqdLly5GRwAASdLJkyf17LPPUmQNsn//ft24cUOSNHr0aJUpU0ZxcXHy8fHRlStX1LVrV7388stasmSJwUmBewPHuGA2lFk4nPHjxxsdAQAkSW3bttXOnTs5ynEPiImJ0Zw5c+Tj4yNJKlKkiCZOnMilYe4hdzqrCrlz6dIlJScnS7p5Dew/fxb+6vfff8/vWMBdoczCKezcuVP79u2TdHPaWYMGDQxOBMAZdOjQQSNGjNAvv/yi4OBgFSxY0GZ/p06dDErmPP4sSNeuXVPp0qVt9pUtW1Znz541IhaywVFB+5g3b56mT5+uAwcO2GyvWrWqnn/+ea7uAFOjzMKhnThxQr169VJ0dLR8fX0l3VxZ9IEHHtDSpUtVrlw5YwMCcGiDBg2SJE2aNCnLPi7Tlj9at26tAgUK6PLlyzpw4IBq1apl3Xfs2DGbBaBgPzdu3NDGjRt1+PBh9e7dW15eXvr111/l7e2tIkWKSOKooD289tprmjBhgp599lm1bdvWusDZ6dOntX79eg0bNky//fabXnjhBYOTArlDmYVDe/LJJ3X9+nXt27dPVatWlSQdOHBA/fv315NPPql169YZnBCAI+OSbcb6+2knf5amP3355Zd68MEH8zOSUzp27JjatWunpKQkpaam6uGHH5aXl5f++9//KjU1VXPmzDE6osOaNWuWFi5cqH//+98226tXr66WLVuqTp06GjFiBGUWpsVqxnZw7tw5LViwQDExMTbnJjzwwAN64oknVKJECevYJUuWqHPnzipcuLBRcR1aoUKF9MMPP6hevXo222NjY/Xggw/q6tWrBiUDAMA5dOnSRV5eXpo/f76KFy9uXbF448aNGjRokA4dOmR0RIdVqFAh7dq1S9WrV892/y+//KKQkBD+HoJpuRgdwNHs2LFD9913n2bOnCkfHx81b95czZs3l4+Pj2bOnKlq1app586d1vG9e/emyNpRQECArl+/nmV7enq6ypQpY0AiAM5m06ZN6tixo4KCghQUFKROnTppy5YtRscC8s2WLVs0ZswYubm52WwPDAzUyZMnDUrlHBo2bKipU6daV/X+q/T0dP33v/9Vw4YNDUgG5A2mGeexoUOH6rHHHtOcOXOyrMqXmZmpp59+WkOHDlVMTIxBCZ3La6+9pqFDh+qdd95RSEiIpJuLQQ0bNkyvv/66wekAOLqPPvpI/fv3V7du3fTss89KkqKjo9W6dWstWrRIvXv3NjghYH8ZGRnZnh9+4sQJeXl5GZDIecyaNUtt27ZVqVKl1Lx5c5tzZjdv3iw3NzetX7/e4JRA7jHNOI8VKlRIu3fvVrVq1bLdv3//ftWrV09//PFHPidzTkWLFtXVq1d148YNFShw872bPz/++xHxCxcuGBERgAOrXr26nnrqKQ0fPtxm+/Tp0zV37lzrKuuAI+vRo4d8fHz0/vvvy8vLSz/99JNKlCihzp07q3z58lq4cKHRER3a77//ro8++kjbtm2zOf2tSZMm6t27t7y9vQ1OCOQeZTaPVaxYURMnTlS/fv2y3f/BBx9o3LhxOnr0aP4Gc1KLFy++47FhYWF2TALAGbm7u2vv3r0KCgqy2Z6QkKBatWrp2rVrBiUD8s+JEyfUtm1bZWZm6tChQwoJCdGhQ4fk5+enzZs3q2TJkkZHBGBSTDPOYy+88IKeeuopxcbGqnXr1jbTOaKiojR37lymt+YjCioAIwUEBCgqKipLmf3uu+8UEBBgUCogf5UrV07x8fFatmyZ4uPjdeXKFQ0cOFB9+vRRoUKFjI7n1K5fv65Tp06pfPnyRkcBcoUjs3awbNkyvfnmm4qNjbWeI+Lq6qoGDRooIiIiy/LosL8zZ87ozJkzWS6TUbt2bYMSAXAGs2fP1nPPPacBAwbogQcekHTznNlFixbprbfe0n/+8x+DEwJwZvHx8apfvz7XvIZpUWbt6Pr16zp37pwkyc/PTwULFjQ4kfOJjY1VWFiY9u3bp7//r26xWPjlDcDuPv/8c73xxhvW82OrV6+uESNGqHPnzgYnA/LH4sWL5efnpw4dOkiSXnzxRb3//vuqUaOGPvnkE1WoUMHghM6LMguzo8zCodWpU0eVK1fWyJEj5e/vn2WFaf4BBQDAvqpWrarZs2erVatWiomJUevWrTVjxgx99dVXKlCggFauXGl0RIdVv3792+7/448/dPDgQcosTIsyC4fm5eWl3bt3ZzlfDQAA5A9PT0/t379f5cuX18iRI3Xq1Cl98MEH2rt3r1q2bKmzZ88aHdFheXh4qGfPnqpYsWK2+0+dOqW5c+dSZmFaLAAFh9a6dWvFx8dTZgHkm2LFiungwYPy8/NT0aJFs8wI+SsuCQZnUKRIEZ0/f17ly5fX+vXrFRERIelm0eJShfZVq1YtNW7cWM8880y2++Pi4jR37tx8TgXkHcosHNq8efMUFhamn3/+WbVq1cpy3nKnTp0MSgbAUb355pvy8vKyfny7Mgs4g4cfflhPPvmk6tWrp4MHD6p9+/aSpL179yowMNDYcA6uadOmOnDgwC33e3l5qXnz5vmYCMhbTDOGQ/vyyy/1+OOP6/Lly1n2sQAUAAD2d/HiRY0ZM0bHjx/XM888o3bt2kmSxo8fLzc3N7388ssGJwRgVpRZOLTAwEA9+uijGjt2rPWavwCQX1xdXXXq1CmVLFnSZvv58+dVsmRJ3lADcE8ZPHiwJk2aJD8/P6OjAHeEMguH5uXlpbi4OFWuXNnoKACckIuLi5KTk7OU2V9//VWVK1fmfEE4hc2bN992P9Nc7x3e3t6Ki4tTpUqVjI4C3BHOmYVD69atmzZs2ECZBZCvZs6cKenm6Qzz5s1TkSJFrPvS09O1efNmVatWzah4QL5q2bJllm1/PZecGQr3Do5xwWwos3Bo9913n0aPHq2tW7cqODg4ywJQzz77rEHJADiyN998U9LNPwznzJkjV1dX6z43NzcFBgZqzpw5RsUD8tVvv/1m8/n169e1e/dujR07VpMnTzYoFQBHwDRjOLRbXVdNuvmu8JEjR/IxDQBn89BDD2nlypUqWrSo0VGAe86mTZsUERGh2NhYo6Pg/3l5eSk+Pp5pxjANjszCoSUmJhodAYAT27Bhg9ERgHuWv7//bS8bAwD/hDILAICddO/eXY0aNdLIkSNttk+bNk07duzQ8uXLDUoG5J+ffvrJ5vPMzEydOnVKU6dOVd26dY0JBcAhUGbhcCIiIvTKK6+ocOHCioiIuO3Y6dOn51MqAM5o8+bNmjBhQpbtjzzyiN544438DwQYoG7durJYLFkWF7r//vu1YMECg1I5jxs3bmjKlCkaMGCAypUrd9uxffv2lbe3dz4lA+4eZRYOZ/fu3bp+/br141v560qKAGAPV65ckZubW5btBQsW1OXLlw1IBOS/v5/y4+LiohIlSsjDw8OgRM6lQIECeu2119SvX79/HDt79ux8SATkHcosHM5fz1HjfDUARgoODtayZcs0btw4m+1Lly5VjRo1DEoF5K8KFSrc0bjg4GCtXbtWAQEBdk7kfFq1aqVNmzYpMDDQ6ChAnqLMwqlcvnxZ33//vapVq8Y1HgHY3dixY9WtWzcdPnxYrVq1kiRFRUXpk08+4XxZ4G+OHj1qnVmFvPXII49o1KhR2rNnjxo0aKDChQvb7O/UqZNByYC7w6V54ND+/e9/q3nz5goPD9cff/yhOnXq6OjRo8rMzNTSpUvVvXt3oyMCcHBr1qzRlClTFBcXp0KFCql27doaP368WrRoYXQ04J7CZWHsx8XF5Zb7LBaL0tPT8zENkHc4MguHtnnzZr388suSpM8//1yZmZm6ePGiFi9erFdffZUyC8DuOnTooA4dOhgdA4ATy8jIMDoCYBe3fpsGcACXLl1SsWLFJEnr1q1T9+7d5enpqQ4dOujQoUMGpwPgDC5evKh58+bppZde0oULFyRJu3bt0smTJw1OBsAZXbt2zegIQJ6hzMKhBQQEKCYmRikpKVq3bp3atGkjSfrtt99YRRGA3f3000+677779N///levvfaaLl68KElauXKlRo8ebWw4AE4jPT1dr7zyisqWLasiRYroyJEjkm6e1z9//nyD0wG5R5mFQ3vuuefUp08flStXTmXKlFHLli0l3Zx+HBwcbGw4AA4vIiJCTzzxhA4dOmTzBlr79u21efNmA5MBcCaTJ0/WokWLNG3aNJvLhdWqVUvz5s0zMBlwd1gACg4vNjZWSUlJevjhh1WkSBFJNxdk8fX1VdOmTQ1OB8CR+fj4aNeuXapcubLN4jbHjh1T1apVme4H/MWSJUvUuXPnLCvt4u4FBQXpvffeU+vWrW1+F+3fv19NmjTRb7/9ZnREIFc4MguH16BBA3Xt2tVaZKWbC7L8tch6e3tbp9wAQF5xd3fX5cuXs2w/ePCgSpQoYUAiwBhRUVF69NFHVblyZVWuXFmPPvqovvvuO5sxvXv3psjaycmTJxUUFJRle0ZGBpdDgqlRZgFJTFAAYA+dOnXSpEmTrH8sWiwWJSUlaeTIkaymDqfx7rvvql27dvLy8tKwYcM0bNgweXt7q3379nrnnXeMjucUatSooS1btmTZvmLFCtWrV8+AREDe4NI8AADYyRtvvKF//etfKlmypP744w+1aNFCycnJatKkiSZPnmx0PCBfTJkyRW+++abCw8Ot25599lk1bdpUU6ZM0ZAhQwxM5xzGjRunsLAwnTx5UhkZGVq5cqUOHDigDz74QF999ZXR8YBc45xZQFyoHYB9RUdHKz4+XleuXFH9+vUVGhpqdCQg3xQpUkRxcXFZprkeOnRI9erV05UrVwxK5ly2bNmiSZMm2fwuGjdunPVKD4AZUWYBUWYB5L3r16+rUKFCiouLU61atYyOAximd+/eqlevnkaMGGGz/fXXX9fOnTu1dOlSg5IBMDumGQO6eR4bAOSlggULqnz58kpPTzc6CpDvZs6caf24Ro0amjx5sjZu3KgmTZpIkrZt26bo6Gg9//zzRkUE4AA4MguII7MA7GP+/PlauXKlPvzwQxUrVszoOEC+qVix4h2Ns1gsXE3ATooWLXrHb9ZfuHDBzmkA++DILJzS8ePHNX78eC1YsECS9PXXX6ts2bIGpwLgaGbNmqWEhASVKVNGFSpUyHLZkV27dhmUDLCvxMREoyM4vRkzZlg/Pn/+vF599VW1bdvWenQ8JiZG33zzjcaOHWtQQuDucWQWTik+Pl7169dn+h8Au5o4ceJt948fPz6fkgBwZt27d9dDDz1ks6K0dPMNt++++06rVq0yJhhwlyizcEirV6++7f4jR47o+eefp8wCAGBnAwYMuO3+P2dJwX5utaJ0QkKC6taty4rSMC2mGcMhdenSRRaLRbd7r4ZFnwAAsL/ffvvN5vPr16/r559/1sWLF9WqVSuDUjmX4sWL64svvsiy4NYXX3yh4sWLG5QKuHuUWTik0qVL691331Xnzp2z3R8XF6cGDRrkcyoAzqBYsWI6ePCg/Pz8/nEBFhZdgTP4/PPPs2zLyMjQM888o8qVKxuQyPlMnDhRTz75pDZu3KjGjRtLkn788UetW7dOc+fONTgdkHuUWTikBg0aKDY29pZl9p+O2gJAbr355pvy8vKSZLsAC4D/cXFxUUREhFq2bKkXX3zR6DgO74knnlD16tU1c+ZMrVy5UpJUvXp1bd261VpuATPinFk4pC1btiglJUXt2rXLdn9KSop27typFi1a5HMyAAAgSWvXrlVYWJjOnj1rdBQAJkWZBQAgD12+fPmOx3p7e9sxCXBviIiIsPk8MzNTp06d0po1axQWFqZZs2YZlMy5pKena9WqVdq3b58kqWbNmurUqZNcXV0NTgbkHmUWAIA85OLicscLzLGiOpzBQw89ZPO5i4uLSpQooVatWmnAgAEqUICz3uwtISFBHTp00IkTJ1S1alVJ0oEDBxQQEKA1a9Zw7jJMizILAEAe2rRpk/Xjo0ePatSoUXriiSfUpEkTSVJMTIwWL16syMhIhYWFGRUTgBNp3769MjMz9fHHH6tYsWKSpPPnz6tv375ycXHRmjVrDE4I5A5lFgAAO2ndurWefPJJ9erVy2b7kiVL9P7772vjxo3GBAPgVAoXLqxt27YpODjYZnt8fLyaNm3KdWZhWi5GBwAAwFHFxMQoJCQky/aQkBBt377dgERA/jt9+rQef/xxlSlTRgUKFJCrq6vNDfbn7u6u33//Pcv2K1euyM3NzYBEQN7gJAUAAOwkICBAc+fO1bRp02y2z5s3TwEBAQalAvLXE088oaSkJI0dO1alS5e+43PKkXceffRRPfXUU5o/f74aNWok6eZ1Zp9++ml16tTJ4HRA7jHNGAAAO1m7dq26d++uoKAg67Uct2/frkOHDumzzz5T+/btDU4I2J+Xl5e2bNmiunXrGh3FaV28eFFhYWH68ssvVbBgQUnSjRs31KlTJy1atEg+Pj4GJwRyhzILAIAdHT9+XLNnz9b+/fslSdWrV9fTTz/NkVk4jRo1aujjjz9WvXr1jI7i9BISEqyX5qlevbqCgoIMTgTcHcosAAAA7Gb9+vV644039N577ykwMNDoOAAcCGUWAIA89NNPP6lWrVpycXHRTz/9dNuxtWvXzqdUQP4qWrSozbmxKSkpunHjhjw9Pa3TXP904cKF/I7ndLp3765GjRpp5MiRNtunTZumHTt2aPny5QYlA+4OZRYAgDzk4uKi5ORklSxZUi4uLrJYLMrun1qLxaL09HQDEgL2t3jx4jsey/WW7a9EiRL6/vvvs1yaZ8+ePQoNDdXp06cNSgbcHVYzBgAgDyUmJqpEiRLWjwFnlJuCOnXqVD399NPy9fXN+0BO7laX4ClYsKAuX75sQCIgb3BkFgAAAIbz9vZWXFycKlWqZHQUh9OoUSM9+uijGjdunM32CRMm6Msvv1RsbKxByYC7w5FZAADs6MCBA3r77bdtVhAdOnSoqlatanAy4N7C8RX7GTt2rLp166bDhw+rVatWkqSoqCh98sknnC8LU3MxOgAAAI7qs88+U61atRQbG6s6deqoTp062rVrl2rVqqXPPvvM6HgAnETHjh21atUqJSQkaPDgwXr++ed14sQJfffdd+rSpYvR8YBcY5oxAAB2UrlyZfXp00eTJk2y2T5+/Hh99NFHOnz4sEHJgHuPl5eX4uPjmWYM4I5RZgEAsBNPT0/99NNPCgoKstl+6NAh1alTR1evXjUoGXDvoczaX1pams6cOaOMjAyb7eXLlzcoEXB3OGcWAAA7admypbZs2ZKlzG7dulUPPvigQakAOJtDhw5pwIAB+uGHH2y2Z2ZmcpkwmBplFgCAPLR69Wrrx506ddLIkSMVGxur+++/X5K0bds2LV++XBMnTjQqImB3EREReuWVV1S4cGFt3rxZDzzwgAoUuP2fnQ8++KAKFSqUTwmdyxNPPKECBQroq6++UunSpWWxWIyOBOQJphkDAJCHXFzubG1FjobAkRUsWFAnTpyQv7+/XF1dderUKZUsWdLoWE6rcOHCio2NVbVq1YyOAuQpjswCAJCH/n4uGuCMAgMDNXPmTLVp00aZmZmKiYlR0aJFsx3bvHnzfE7nfGrUqKFz584ZHQPIcxyZBQDAYMHBwVq7dq0CAgKMjgLkiVWrVunpp5/WmTNnZLFYbnkNWWYo5I/vv/9eY8aM0ZQpUxQcHKyCBQva7Pf29jYoGXB3KLMAABiMVVzhqK5cuSJvb28dOHDgltOMfXx88jmV8/nz9Ie/nyvLAlAwO6YZAwAAwC6KFCmiDRs2qGLFiv+4ABTsZ8OGDUZHAOyCI7MAABiMI7NwZLdaAOr8+fMqWbIkRwUB5NqdLbkIAAAA5MKtjpukpqbKzc0tn9M4ry1btqhv37564IEHdPLkSUnShx9+qK1btxqcDMg95nsAAAAgz82cOVPSzfM0582bpyJFilj3paena/PmzVwqJp989tlnevzxx9WnTx/t2rVLqampkqRLly5pypQpWrt2rcEJgdxhmjEAAPnkz8VW/o5pxnBEFStWlCQdO3ZM5cqVk6urq3Wfm5ubAgMDNWnSJDVu3NioiE6jXr16Gj58uPr162fz+2b37t165JFHlJycbHREIFc4MgsAQD5xd3dXfHy8qlevbrP9vffek7+/v0GpAPtITEyUJD300ENauXLlLa8zC/s7cOBAttfz9fHx0cWLF/M/EJBHKLMAAOSxiIiIbLenp6dr6tSpKl68uCRp+vTpkqTevXvnWzYgv93pSrre3t6Ki4tjhoIdlCpVSgkJCQoMDLTZvnXrVr7fMDXKLAAAeWzGjBmqU6eOfH19bbZnZmZq3759Kly4cLbTjQFnxplv9jNo0CANGzZMCxYskMVi0a+//qqYmBi98MILGjt2rNHxgFyjzAIAkMemTJmi999/X2+88YZatWpl3V6wYEEtWrRINWrUMDAdAGczatQoZWRkqHXr1rp69aqaN28ud3d3vfDCCxo6dKjR8YBcYwEoAADsYMeOHerbt686duyoyMhIFSxYUAULFlR8fDxlFsgGC6HZX1pamhISEnTlyhXVqFHDZoVpSTpx4oTKlCkjFxeu3glz4P9UAADsoGHDhoqNjdXZs2cVEhKin3/+manFAAzl5uamGjVqqFGjRlmKrCTVqFFDR48ezf9gQC4xzRgAADspUqSIFi9erKVLlyo0NFTp6elGRwLuWbzZYzwmbMJsKLMAANhZz5491axZM8XGxqpChQpGxwHuSRQpADlFmQUAIB+UK1dO5cqVMzoGcM/6+uuvVbZsWaNjADARzpkFAABAntu1a5cSExOtn3/44Ydq2rSpAgIC1KxZMy1dutRmfLNmzeTu7p7fMQGYGGUWAAAAea5///46fPiwJGnevHn6z3/+o5CQEL388stq2LChBg0apAULFhicEn/FecswG6YZAwAAIM8dOnRIVapUkSS9++67euuttzRo0CDr/oYNG2ry5MkaMGCAURHxN5y3DLPhyCwAAADynKenp86dOydJOnnypBo1amSzv3HjxjbTkGF/CQkJ+uabb/THH39Iylpef/nlFxapg6lQZgEAAJDnHnnkEc2ePVuS1KJFC61YscJm/6effqqgoCAjojmd8+fPKzQ0VPfdd5/at2+vU6dOSZIGDhyo559/3jouICBArq6uRsUEcsySyXwCAAAA5LFff/1VTZs2Vfny5RUSEqLZs2erQYMGql69ug4cOKBt27bp888/V/v27Y2O6vD69eunM2fOaN68eapevbri4+NVqVIlffPNN4qIiNDevXuNjgjkCufMAgAAIM+VKVNGu3fv1tSpU/Xll18qMzNT27dv1/Hjx9W0aVNFR0crJCTE6JhOYf369frmm2+yXB6sSpUqOnbsmEGpgLtHmQUAAIBd+Pr6aurUqZo6darRUZxaSkqKPD09s2y/cOECl0OCqXHOLAAAAODAHnzwQX3wwQfWzy0WizIyMjRt2jQ99NBDBiYD7g7nzAIAAAAO7Oeff1br1q1Vv359ff/99+rUqZP27t2rCxcuKDo6WpUrVzY6IpArlFkAAADAwV26dEmzZs1SfHy8rly5ovr162vIkCEqXbq00dGAXKPMAgAA4P/au7/QLMs+DuDf53ErKCpbubJyTmdIoS3PAgsiKGql2VkYUVCnOxlEVGBZFEF/CCuiiCgSKmonkvSHCZkeROIWeRCVtTXLRSflqpWs+bwHL68wzZexPe7mvv18jnbd93PwPdyX33VdN0DpODMLAAAV9uGHH2b37t1H1y+++GKuvPLKbNy4Mb/++muByWBulFkAAKiw++67L+Pj40mSffv2pa+vLz09PRkeHk5fX1/B6WD2fJoHAAAqbHh4OJdffnmSpL+/P+vWrcsTTzyRwcHB9PT0FJwOZs9kFgAAKuy0007LxMREkmRgYCA33HBDkqStre3oxBbKyGQWAAAq7Oqrr05fX1/Wrl2bzz//PO+8806S5Jtvvskll1xScDqYPZNZAACosBdeeCEtLS1577338tJLL+Xiiy9OknzwwQe58cYbC04Hs+fTPAAAAJSObcYAAFBho6Oj//d9R0fHPCWB5jKZBQCACqvX66nVaid8PzU1NY9poHlMZgEAoMKGhoamrScnJzM0NJRnn302jz/+eEGpYO5MZgEA4BS0ffv2PPXUU/nkk0+KjgKz4jZjAAA4Ba1cuTJ79uwpOgbMmm3GAABQYePj49PWjUYjY2NjeeSRR3LppZcWlArmTpkFAIAKW7hw4XEXQDUajSxZsiRvv/12Qalg7pyZBQCACtu5c+e0db1ez6JFi7JixYq0tJhtUV7KLAAAkJtvvjmvvvpqFi9eXHQUmBEXQAEAAPn000/z119/FR0DZkyZBQAAoHSUWQAAAEpHmQUAAKB0lFkAAABKR5kFAACgdJRZAAAgDz74YNra2oqOATPmO7MAAFBxX3/9dZ5//vl89dVXSZLLLrssvb29WblyZcHJYPZMZgEAoML6+/uzatWq7N27N93d3enu7s7g4GBWrVqV/v7+ouPBrJnMAgBAhXV1deWOO+7Io48+Ou35ww8/nK1bt+a7774rKBnMjTILAAAVdsYZZ+TLL7/MihUrpj3/9ttv093dnYmJiYKSwdzYZgwAABV27bXXZteuXcc93717d6655poCEkFztBQdAAAAaK5t27Yd/Xv9+vW5//77s3fv3lx11VVJks8++yzvvvtuNm/eXFREmDPbjAEAoGLq9ZltwKzVapmamjrJaeDkUGYBAAAoHWdmAQAAKB1nZgEAoMKO/STPsTZt2jRPSaC5bDMGAIAKW7NmzbT15ORkhoeH09LSkq6urgwODhaUDObGZBYAACpsaGjouGfj4+O5++67c9tttxWQCJrDZBYAAE5B+/bty7p16zIyMlJ0FJgVF0ABAMAp6NChQzl06FDRMWDWbDMGAIAK27Jly7R1o9HI2NhY3nzzzdx0000FpYK5s80YAAAqbNmyZdPW9Xo9ixYtynXXXZcHHnggZ511VkHJYG6UWQAAAErHmVkAAABKx5lZAACosD///DNPPvlkduzYkV9++SVHjhyZ9v77778vKBnMjTILAAAVdu+992bnzp258847s3jx4tRqtaIjQVM4MwsAABW2cOHCbN++PWvXri06CjSVM7MAAFBh5557btra2oqOAU2nzAIAQIU99thj2bRpUyYmJoqOAk1lmzEAAFTMmjVrpp2N3b9/fxqNRjo7O9Pa2jrtt4ODg/MdD5rCBVAAAFAxGzZsKDoCnHQmswAAQN56662sX78+Z555ZtFRYEaUWQAAIGeffXa++OKLLF++vOgoMCMugAIAAGLGRdkoswAAAJSOMgsAAEDpKLMAAACUjjILAABA6SizAABQMVu2bMnff/+dJBkdHZ3R5U5Lly5Na2vryY4GTePTPAAAUDEtLS05ePBg2tvbs2DBgoyNjaW9vb3oWNBULUUHAAAAmuuiiy5Kf39/enp60mg08uOPPx6d1B6ro6NjntNBc5jMAgBAxbzyyivp7e3NP//8c8LfNBqN1Gq1TE1NzWMyaB5lFgAAKuj333/PDz/8kCuuuCIDAwM577zz/vV33d3d85wMmkOZBQCACnvjjTdy++235/TTTy86CjSV24wBAKDCNm/enD/++OO457/99luWL19eQCJoDmUWAAAqbGRk5F/PxR4+fDg//fRTAYmgOdxmDAAAFbRt27ajf3/00Uc555xzjq6npqayY8eOdHZ2FpAMmsOZWQAAqKB6/b+bMGu1Wo79l7+1tTWdnZ155plncssttxQRD+ZMmQUAgApbtmxZ9uzZk/PPP7/oKNBUzswCAECFDQ8Pz6jIrl69OgcOHJiHRNAcyiwAAJCRkZFMTk4WHQNmTJkFAACgdJRZAAAASkeZBQAAoHSUWQAAAEpHmQUAAKB0lFkAAKig3t7e7Nq1a8a/f/nll3PBBRecxETQXLVGo9EoOgQAANBc9Xo9tVotXV1dueeee3LXXXflwgsvLDoWNI3JLAAAVNTHH3+cnp6ePP300+no6Mitt96a999/P0eOHCk6GsyZMgsAABW1evXqPPfcczl48GC2bt2aw4cPZ8OGDVmyZEkeeuih7N+/v+iIMGu2GQMAQAXV6/X8/PPPaW9vn/Z8dHQ0r732Wl5//fUcOHAgU1NTBSWEuVFmAQCggk5UZv+n0WhkYGAg119//Twng+awzRgAACpo6dKlWbBgwQnf12o1RZZSM5kFAACgdExmAQAAKB1lFgAAgNJRZgEAACgdZRYAAIDSUWYBAAAoHWUWAACA0lFmAQAAKB1lFgAAgNL5D5Oqao0dH4wmAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(by = \"F1 score\", ascending = False)[\"F1 score\"].plot(kind = \"bar\", figsize = (10,7));"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 825
        },
        "id": "0fEJgfLowShB",
        "outputId": "57c2603b-4ba0-49c5-ff49-ef096e026812"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzoAAAMoCAYAAAAHr5UkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABq20lEQVR4nO3deVhV5eL28RtQUBRQQ3FCcSolZ0lTc0hJS3Oofh3HNDQ7ZZhFmVoOaSkeMyWPJuWQNpiWmQ2aWZQzHhMUyxknnMBZEwsUeP/wbdcONDHWXvbs7+e69hX7WWvDjTuRe61nPcsjJycnRwAAAABgEE+7AwAAAABAQaPoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDiF7A5wPbKzs3X06FH5+fnJw8PD7jgAAAAAbJKTk6Off/5Z5cuXl6fn1c/b/COKztGjRxUcHGx3DAAAAAA3iUOHDqlixYpX3f6PKDp+fn6Srnwz/v7+NqcBAAAAYJfz588rODjY0RGu5h9RdH6brubv70/RAQAAAPCXl7SwGAEAAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjFPI7gA3i5BhS+2O8LccmNDR7ggAAADATYMzOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgnBsqOtOnT1dISIiKFCmiJk2aaOPGjdfcPyYmRrfddpuKFi2q4OBgPfvss/r1119vKDAAAAAA/JV8F52FCxcqKipKo0ePVmJiourVq6f27dvr+PHjee4/f/58DRs2TKNHj9aOHTs0e/ZsLVy4UC+++OLfDg8AAAAAecl30Zk8ebIGDBigiIgIhYaGKjY2Vr6+vpozZ06e+69fv17NmzdXz549FRISonbt2qlHjx5/eRYIAAAAAG5UvopOZmamEhISFB4e/vsn8PRUeHi44uPj83xNs2bNlJCQ4Cg2+/bt07Jly9ShQ4e/ERsAAAAArq5QfnY+efKksrKyFBQU5DQeFBSknTt35vmanj176uTJk7rrrruUk5Ojy5cv64knnrjm1LWMjAxlZGQ4np8/fz4/MQEAAAC4OctXXVu5cqXGjx+vN998U4mJiVq8eLGWLl2qV1555aqviY6OVkBAgOMRHBxsdUwAAAAABsnXGZ3AwEB5eXkpLS3NaTwtLU1ly5bN8zUjR47UI488oscee0ySVKdOHaWnp+vxxx/XSy+9JE/P3F1r+PDhioqKcjw/f/48ZQcAAADAdcvXGR1vb281atRIcXFxjrHs7GzFxcWpadOmeb7m4sWLucqMl5eXJCknJyfP1/j4+Mjf39/pAQAAAADXK19ndCQpKipKffv2VVhYmBo3bqyYmBilp6crIiJCktSnTx9VqFBB0dHRkqROnTpp8uTJatCggZo0aaLk5GSNHDlSnTp1chQeAAAAAChI+S463bp104kTJzRq1Cilpqaqfv36Wr58uWOBgpSUFKczOCNGjJCHh4dGjBihI0eOqHTp0urUqZPGjRtXcN8FAAAAAPyBR87V5o/dRM6fP6+AgACdO3fOsmlsIcOWWvJ5XeXAhI52RwAAAAAsd73dwPJV1wAAAADA1Sg6AAAAAIyT72t0AKswfRAAAAAFhTM6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGYXlpAA4s8W0/3gN7/dP//KV//nsAAAWFMzoAAAAAjEPRAQAAAGAcig4AAAAA43CNDgAAuGlwnRSAgsIZHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcVl0DAACAAyvf2Y/3oGBwRgcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMM4NFZ3p06crJCRERYoUUZMmTbRx48ar7tu6dWt5eHjkenTs2PGGQwMAAADAteS76CxcuFBRUVEaPXq0EhMTVa9ePbVv317Hjx/Pc//Fixfr2LFjjsdPP/0kLy8vPfzww387PAAAAADkJd9FZ/LkyRowYIAiIiIUGhqq2NhY+fr6as6cOXnuX6pUKZUtW9bx+Oabb+Tr60vRAQAAAGCZfBWdzMxMJSQkKDw8/PdP4Omp8PBwxcfHX9fnmD17trp3765ixYpddZ+MjAydP3/e6QEAAAAA1ytfRefkyZPKyspSUFCQ03hQUJBSU1P/8vUbN27UTz/9pMcee+ya+0VHRysgIMDxCA4Ozk9MAAAAAG7OpauuzZ49W3Xq1FHjxo2vud/w4cN17tw5x+PQoUMuSggAAADABIXys3NgYKC8vLyUlpbmNJ6WlqayZcte87Xp6elasGCBxo4d+5dfx8fHRz4+PvmJBgAAAAAO+Tqj4+3trUaNGikuLs4xlp2drbi4ODVt2vSar/3444+VkZGh3r1731hSAAAAALhO+TqjI0lRUVHq27evwsLC1LhxY8XExCg9PV0RERGSpD59+qhChQqKjo52et3s2bPVtWtX3XLLLQWTHAAAAACuIt9Fp1u3bjpx4oRGjRql1NRU1a9fX8uXL3csUJCSkiJPT+cTRbt27dLatWu1YsWKgkkNAAAAANeQ76IjSZGRkYqMjMxz28qVK3ON3XbbbcrJybmRLwUAAAAA+ebSVdcAAAAAwBUoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYJwbKjrTp09XSEiIihQpoiZNmmjjxo3X3P/s2bN66qmnVK5cOfn4+OjWW2/VsmXLbigwAAAAAPyVQvl9wcKFCxUVFaXY2Fg1adJEMTExat++vXbt2qUyZcrk2j8zM1P33HOPypQpo0WLFqlChQo6ePCgSpQoURD5AQAAACCXfBedyZMna8CAAYqIiJAkxcbGaunSpZozZ46GDRuWa/85c+bo9OnTWr9+vQoXLixJCgkJ+XupAQAAAOAa8jV1LTMzUwkJCQoPD//9E3h6Kjw8XPHx8Xm+5vPPP1fTpk311FNPKSgoSLVr19b48eOVlZV11a+TkZGh8+fPOz0AAAAA4Hrlq+icPHlSWVlZCgoKchoPCgpSampqnq/Zt2+fFi1apKysLC1btkwjR47U66+/rldfffWqXyc6OloBAQGOR3BwcH5iAgAAAHBzlq+6lp2drTJlyujtt99Wo0aN1K1bN7300kuKjY296muGDx+uc+fOOR6HDh2yOiYAAAAAg+TrGp3AwEB5eXkpLS3NaTwtLU1ly5bN8zXlypVT4cKF5eXl5RirVauWUlNTlZmZKW9v71yv8fHxkY+PT36iAQAAAIBDvs7oeHt7q1GjRoqLi3OMZWdnKy4uTk2bNs3zNc2bN1dycrKys7MdY7t371a5cuXyLDkAAAAA8Hfle+paVFSUZs6cqXnz5mnHjh168sknlZ6e7liFrU+fPho+fLhj/yeffFKnT5/W4MGDtXv3bi1dulTjx4/XU089VXDfBQAAAAD8Qb6Xl+7WrZtOnDihUaNGKTU1VfXr19fy5csdCxSkpKTI0/P3/hQcHKyvv/5azz77rOrWrasKFSpo8ODBGjp0aMF9FwAAAADwB/kuOpIUGRmpyMjIPLetXLky11jTpk21YcOGG/lSAAAAAJBvlq+6BgAAAACuRtEBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADj3FDRmT59ukJCQlSkSBE1adJEGzduvOq+c+fOlYeHh9OjSJEiNxwYAAAAAP5KvovOwoULFRUVpdGjRysxMVH16tVT+/btdfz48au+xt/fX8eOHXM8Dh48+LdCAwAAAMC15LvoTJ48WQMGDFBERIRCQ0MVGxsrX19fzZkz56qv8fDwUNmyZR2PoKCgvxUaAAAAAK4lX0UnMzNTCQkJCg8P//0TeHoqPDxc8fHxV33dhQsXVLlyZQUHB6tLly7atm3bNb9ORkaGzp8/7/QAAAAAgOuVr6Jz8uRJZWVl5TojExQUpNTU1Dxfc9ttt2nOnDn67LPP9P777ys7O1vNmjXT4cOHr/p1oqOjFRAQ4HgEBwfnJyYAAAAAN2f5qmtNmzZVnz59VL9+fbVq1UqLFy9W6dKl9dZbb131NcOHD9e5c+ccj0OHDlkdEwAAAIBBCuVn58DAQHl5eSktLc1pPC0tTWXLlr2uz1G4cGE1aNBAycnJV93Hx8dHPj4++YkGAAAAAA75OqPj7e2tRo0aKS4uzjGWnZ2tuLg4NW3a9Lo+R1ZWln788UeVK1cuf0kBAAAA4Drl64yOJEVFRalv374KCwtT48aNFRMTo/T0dEVEREiS+vTpowoVKig6OlqSNHbsWN15552qXr26zp49q9dee00HDx7UY489VrDfCQAAAAD8f/kuOt26ddOJEyc0atQopaamqn79+lq+fLljgYKUlBR5ev5+oujMmTMaMGCAUlNTVbJkSTVq1Ejr169XaGhowX0XAAAAAPAH+S46khQZGanIyMg8t61cudLp+ZQpUzRlypQb+TIAAAAAcEMsX3UNAAAAAFyNogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMa5oaIzffp0hYSEqEiRImrSpIk2btx4Xa9bsGCBPDw81LVr1xv5sgAAAABwXfJddBYuXKioqCiNHj1aiYmJqlevntq3b6/jx49f83UHDhzQ888/rxYtWtxwWAAAAAC4HvkuOpMnT9aAAQMUERGh0NBQxcbGytfXV3PmzLnqa7KystSrVy+NGTNGVatW/VuBAQAAAOCv5KvoZGZmKiEhQeHh4b9/Ak9PhYeHKz4+/qqvGzt2rMqUKaP+/ftf19fJyMjQ+fPnnR4AAAAAcL3yVXROnjyprKwsBQUFOY0HBQUpNTU1z9esXbtWs2fP1syZM6/760RHRysgIMDxCA4Ozk9MAAAAAG7O0lXXfv75Zz3yyCOaOXOmAgMDr/t1w4cP17lz5xyPQ4cOWZgSAAAAgGkK5WfnwMBAeXl5KS0tzWk8LS1NZcuWzbX/3r17deDAAXXq1Mkxlp2dfeULFyqkXbt2qVq1arle5+PjIx8fn/xEAwAAAACHfJ3R8fb2VqNGjRQXF+cYy87OVlxcnJo2bZpr/5o1a+rHH3/Uli1bHI/OnTvr7rvv1pYtW5iSBgAAAMAS+TqjI0lRUVHq27evwsLC1LhxY8XExCg9PV0RERGSpD59+qhChQqKjo5WkSJFVLt2bafXlyhRQpJyjQMAAABAQcl30enWrZtOnDihUaNGKTU1VfXr19fy5csdCxSkpKTI09PSS38AAAAA4JryXXQkKTIyUpGRkXluW7ly5TVfO3fu3Bv5kgAAAABw3Tj1AgAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADDODRWd6dOnKyQkREWKFFGTJk20cePGq+67ePFihYWFqUSJEipWrJjq16+v995774YDAwAAAMBfyXfRWbhwoaKiojR69GglJiaqXr16at++vY4fP57n/qVKldJLL72k+Ph4bd26VREREYqIiNDXX3/9t8MDAAAAQF7yXXQmT56sAQMGKCIiQqGhoYqNjZWvr6/mzJmT5/6tW7fWAw88oFq1aqlatWoaPHiw6tatq7Vr1/7t8AAAAACQl3wVnczMTCUkJCg8PPz3T+DpqfDwcMXHx//l63NychQXF6ddu3apZcuW+U8LAAAAANehUH52PnnypLKyshQUFOQ0HhQUpJ07d171defOnVOFChWUkZEhLy8vvfnmm7rnnnuuun9GRoYyMjIcz8+fP5+fmAAAAADcXL6Kzo3y8/PTli1bdOHCBcXFxSkqKkpVq1ZV69at89w/OjpaY8aMcUU0AAAAAAbKV9EJDAyUl5eX0tLSnMbT0tJUtmzZq77O09NT1atXlyTVr19fO3bsUHR09FWLzvDhwxUVFeV4fv78eQUHB+cnKgAAAAA3lq9rdLy9vdWoUSPFxcU5xrKzsxUXF6emTZte9+fJzs52mpr2Zz4+PvL393d6AAAAAMD1yvfUtaioKPXt21dhYWFq3LixYmJilJ6eroiICElSnz59VKFCBUVHR0u6Mg0tLCxM1apVU0ZGhpYtW6b33ntPM2bMKNjvBAAAAAD+v3wXnW7duunEiRMaNWqUUlNTVb9+fS1fvtyxQEFKSoo8PX8/UZSenq6BAwfq8OHDKlq0qGrWrKn3339f3bp1K7jvAgAAAAD+4IYWI4iMjFRkZGSe21auXOn0/NVXX9Wrr756I18GAAAAAG5Ivm8YCgAAAAA3O4oOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAY54aKzvTp0xUSEqIiRYqoSZMm2rhx41X3nTlzplq0aKGSJUuqZMmSCg8Pv+b+AAAAAPB35bvoLFy4UFFRURo9erQSExNVr149tW/fXsePH89z/5UrV6pHjx76/vvvFR8fr+DgYLVr105Hjhz52+EBAAAAIC/5LjqTJ0/WgAEDFBERodDQUMXGxsrX11dz5szJc/8PPvhAAwcOVP369VWzZk3NmjVL2dnZiouL+9vhAQAAACAv+So6mZmZSkhIUHh4+O+fwNNT4eHhio+Pv67PcfHiRV26dEmlSpXKX1IAAAAAuE6F8rPzyZMnlZWVpaCgIKfxoKAg7dy587o+x9ChQ1W+fHmnsvRnGRkZysjIcDw/f/58fmICAAAAcHMuXXVtwoQJWrBggT799FMVKVLkqvtFR0crICDA8QgODnZhSgAAAAD/dPkqOoGBgfLy8lJaWprTeFpamsqWLXvN106aNEkTJkzQihUrVLdu3WvuO3z4cJ07d87xOHToUH5iAgAAAHBz+So63t7eatSokdNCAr8tLNC0adOrvm7ixIl65ZVXtHz5coWFhf3l1/Hx8ZG/v7/TAwAAAACuV76u0ZGkqKgo9e3bV2FhYWrcuLFiYmKUnp6uiIgISVKfPn1UoUIFRUdHS5L+85//aNSoUZo/f75CQkKUmpoqSSpevLiKFy9egN8KAAAAAFyR76LTrVs3nThxQqNGjVJqaqrq16+v5cuXOxYoSElJkafn7yeKZsyYoczMTP3f//2f0+cZPXq0Xn755b+XHgAAAADykO+iI0mRkZGKjIzMc9vKlSudnh84cOBGvgQAAAAA3DCXrroGAAAAAK5A0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAOPcUNGZPn26QkJCVKRIETVp0kQbN2686r7btm3TQw89pJCQEHl4eCgmJuZGswIAAADAdcl30Vm4cKGioqI0evRoJSYmql69emrfvr2OHz+e5/4XL15U1apVNWHCBJUtW/ZvBwYAAACAv5LvojN58mQNGDBAERERCg0NVWxsrHx9fTVnzpw897/jjjv02muvqXv37vLx8fnbgQEAAADgr+Sr6GRmZiohIUHh4eG/fwJPT4WHhys+Pr7AwwEAAADAjSiUn51PnjyprKwsBQUFOY0HBQVp586dBRYqIyNDGRkZjufnz58vsM8NAAAAwHw35apr0dHRCggIcDyCg4PtjgQAAADgHyRfRScwMFBeXl5KS0tzGk9LSyvQhQaGDx+uc+fOOR6HDh0qsM8NAAAAwHz5Kjre3t5q1KiR4uLiHGPZ2dmKi4tT06ZNCyyUj4+P/P39nR4AAAAAcL3ydY2OJEVFRalv374KCwtT48aNFRMTo/T0dEVEREiS+vTpowoVKig6OlrSlQUMtm/f7vj4yJEj2rJli4oXL67q1asX4LcCAAAAAFfku+h069ZNJ06c0KhRo5Samqr69etr+fLljgUKUlJS5On5+4mio0ePqkGDBo7nkyZN0qRJk9SqVSutXLny738HAAAAAPAn+S46khQZGanIyMg8t/25vISEhCgnJ+dGvgwAAAAA3JCbctU1AAAAAPg7KDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGAcig4AAAAA41B0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIxD0QEAAABgHIoOAAAAAONQdAAAAAAYh6IDAAAAwDgUHQAAAADGoegAAAAAMA5FBwAAAIBxKDoAAAAAjEPRAQAAAGCcGyo606dPV0hIiIoUKaImTZpo48aN19z/448/Vs2aNVWkSBHVqVNHy5Ytu6GwAAAAAHA98l10Fi5cqKioKI0ePVqJiYmqV6+e2rdvr+PHj+e5//r169WjRw/1799fmzdvVteuXdW1a1f99NNPfzs8AAAAAOQl30Vn8uTJGjBggCIiIhQaGqrY2Fj5+vpqzpw5ee7/xhtv6N5779WQIUNUq1YtvfLKK2rYsKGmTZv2t8MDAAAAQF7yVXQyMzOVkJCg8PDw3z+Bp6fCw8MVHx+f52vi4+Od9pek9u3bX3V/AAAAAPi7CuVn55MnTyorK0tBQUFO40FBQdq5c2eer0lNTc1z/9TU1Kt+nYyMDGVkZDienzt3TpJ0/vz5/MTNl+yMi5Z9blew8s/GVXgP7Md7YD/eA3v90//8Jd6DmwHvgf14D+xn5Xvw2+fOycm55n75KjquEh0drTFjxuQaDw4OtiHNP0NAjN0JwHtgP94D+/Ee2I/3wH68B/bjPbCfK96Dn3/+WQEBAVfdnq+iExgYKC8vL6WlpTmNp6WlqWzZsnm+pmzZsvnaX5KGDx+uqKgox/Ps7GydPn1at9xyizw8PPIT+aZw/vx5BQcH69ChQ/L397c7jlviPbAf74H9eA/sx3tgP94De/Hnbz8T3oOcnBz9/PPPKl++/DX3y1fR8fb2VqNGjRQXF6euXbtKulJC4uLiFBkZmedrmjZtqri4OD3zzDOOsW+++UZNmza96tfx8fGRj4+P01iJEiXyE/Wm5O/v/4/9H8oUvAf24z2wH++B/XgP7Md7YC/+/O33T38PrnUm5zf5nroWFRWlvn37KiwsTI0bN1ZMTIzS09MVEREhSerTp48qVKig6OhoSdLgwYPVqlUrvf766+rYsaMWLFigTZs26e23387vlwYAAACA65LvotOtWzedOHFCo0aNUmpqqurXr6/ly5c7FhxISUmRp+fvi7k1a9ZM8+fP14gRI/Tiiy+qRo0aWrJkiWrXrl1w3wUAAAAA/MENLUYQGRl51alqK1euzDX28MMP6+GHH76RL2UEHx8fjR49Otd0PLgO74H9eA/sx3tgP94D+/Ee2Is/f/u503vgkfNX67IBAAAAwD9Mvm4YCgAAAAD/BBQdAAAAAMah6AAAAAAwDkUHAAAAgHEoOha4dOmS+vXrp/3799sdBQAAAHBLrLpmkYCAAG3ZskVVqlSxO4rby8zM1P79+1WtWjUVKnRDK6oDAP6B3n333evar0+fPhYnAWAHio5F+vbtq/r16+vZZ5+1O4rbunjxogYNGqR58+ZJknbv3q2qVatq0KBBqlChgoYNG2ZzQsA19u7dq3feeUd79+7VG2+8oTJlyuirr75SpUqVdPvtt9sdz3heXl46duyYypQp4zR+6tQplSlTRllZWTYlM5+np6eKFy+uQoUK6Wq/7nh4eOj06dMuTuYe+vXrd137zZkzx+Ik7unzzz+/7n07d+5sYRL7cHjbIjVq1NDYsWO1bt06NWrUSMWKFXPa/vTTT9uUzH0MHz5cSUlJWrlype69917HeHh4uF5++WWKjgtUqVJFHh4eV92+b98+F6ZxT6tWrdJ9992n5s2ba/Xq1Ro3bpzKlCmjpKQkzZ49W4sWLbI7ovGu9gt2RkaGvL29XZzGvdSqVUtpaWnq3bu3+vXrp7p169odya3MnTtXlStXVoMGDa769wDW6dq1q9NzDw8Pp/fhj/8+m3rAhaJjkdmzZ6tEiRJKSEhQQkKC0zYPDw+KjgssWbJECxcu1J133un0l/n222/X3r17bUzmPp555hmn55cuXdLmzZu1fPlyDRkyxJ5QbmbYsGF69dVXFRUVJT8/P8d4mzZtNG3aNBuTmW/q1KmSrvzMnzVrlooXL+7YlpWVpdWrV6tmzZp2xXML27Zt0//+9z/NmTNHLVu2VPXq1dW/f3/16tVL/v7+dscz3pNPPqkPP/xQ+/fvV0REhHr37q1SpUrZHcttZGdnOz7+9ttvNXToUI0fP15NmzaVJMXHx2vEiBEaP368XREtx9Q1GMvX11c//fSTqlatKj8/PyUlJalq1apKSkpSy5Ytde7cObsjuq3p06dr06ZNeuedd+yOYrzixYvrxx9/VJUqVZz+Hhw4cEA1a9bUr7/+andEY/12jebBgwdVsWJFeXl5ObZ5e3srJCREY8eOVZMmTeyK6FZ++eUXffzxx3rnnXe0ceNGde3aVXPmzJGPj4/d0YyWkZGhxYsXa86cOVq/fr06duyo/v37q127dtc844+CVbt2bcXGxuquu+5yGl+zZo0ef/xx7dixw6Zk1mLVNYtlZmZq165dunz5st1R3E5YWJiWLl3qeP7bD9RZs2Y5jmbAHvfdd58++eQTu2O4hRIlSujYsWO5xjdv3qwKFSrYkMh97N+/X/v371erVq2UlJTkeL5//37t2rVLX3/9NSXHhYoWLao+ffpozJgxaty4sRYsWKCLFy/aHct4Pj4+6tGjh7755htt375dt99+uwYOHKiQkBBduHDB7nhuY+/evSpRokSu8YCAAB04cMDleVyFomORixcvqn///vL19dXtt9+ulJQUSdKgQYM0YcIEm9O5h/Hjx+vFF1/Uk08+qcuXL+uNN95Qu3bt9M4772jcuHF2x3NrixYtYvqCi3Tv3l1Dhw5VamqqPDw8lJ2drXXr1un5559npSkX+f7771WyZEm7Y7i1I0eOaPz48apRo4a6d++uO+64Q9u2beN9cTFPT0/HdSKmXhNys7rjjjsUFRWltLQ0x1haWpqGDBmixo0b25jMWkxds8jgwYO1bt06xcTE6N5779XWrVtVtWpVffbZZ3r55Ze1efNmuyO6hb1792rChAlKSkrShQsX1LBhQw0dOlR16tSxO5pbaNCggdPUhJycHKWmpurEiRN688039fjjj9uYzj1kZmbqqaee0ty5c5WVlaVChQopKytLPXv21Ny5c52mU8EaWVlZmjt3ruLi4nT8+HGnefOS9N1339mUzHwfffSR3nnnHa1atUrt27dXRESEOnbsyP/3LvTHqWtr167V/fffr4iICN17773y9OR4u6skJyfrgQce0O7duxUcHCxJOnTokGrUqKElS5aoevXqNie0BkXHIpUrV3ZcCP/HefHJyclq2LChzp8/b3dEwHJjxoxxeu7p6anSpUurdevWXITtYocOHdKPP/6oCxcuqEGDBqpRo4bdkdxGZGSk5s6dq44dO6pcuXK5rkuYMmWKTcnM5+npqUqVKqlXr14KCgq66n4sEGSNgQMHasGCBQoODla/fv3Uq1cvBQYG2h3LbeXk5Oibb77Rzp07JV1ZlTA8PNzoa6UoOhbhQvibQ3Z2tpKTk/M8itqyZUubUrmHy5cva/78+Wrfvv01f8GAa2VlZenHH39U5cqVmbbjIoGBgXr33XfVoUMHu6O4nZCQkL/8Jc7Dw4Ol7i3yW9H889n9P1u8eLELU8GdsLy0RX67EH7QoEGSuBDeDhs2bFDPnj118ODBXOv3e3h4MD/YYoUKFdITTzxh7Eou/xTPPPOM6tSpo/79+ysrK0utWrXS+vXr5evrqy+//FKtW7e2O6LxvL29jZ0WcrMz+SLrf4I+ffoYfbbABGlpaXrrrbc0atQou6NYgjM6Flm7dq3uu+8+9e7dW3PnztW///1vbd++XevXr9eqVavUqFEjuyMar379+rr11ls1ZsyYPKeLBAQE2JTMfbRu3VrPPPNMrpuWwXUqVqyoJUuWKCwsTEuWLNHAgQO1cuVKvffee/ruu++0bt06uyMa7/XXX9e+ffs0bdo0fulzsTZt2mjx4sV5rjYFQEpKSlLDhg2NPfhL0bEQF8Lbq1ixYkpKSuJIqo0++ugjDR8+XM8++6waNWqkYsWKOW3nLuXWK1KkiJKTk1WxYkU9/vjj8vX1VUxMjPbv36969epxvaALPPDAA/r+++9VqlQp3X777SpcuLDTdqbtWMfT01OpqakqU6aM3VHckpeXl44dO8afv422bt16ze07d+5Ujx49jC06TF2zULVq1TRz5ky7Y7itJk2aKDk5maJjo+7du0tyvtD3t6VFmT7oGkFBQdq+fbvKlSun5cuXa8aMGZKuLIHPylOuUaJECT3wwAN2xwBcjmPp9qtfv77j390/++O/x6ai6BSg/BwZ9ff3tzAJpCv3LHruueeUmpqqOnXq5DqKytkE6+3fv9/uCG4vIiJC//rXvxzTN8PDwyVJ//vf/1j5zkXeeecduyO4te3btys1NfWa+/DvAUxVqlQpTZw4UW3bts1z+7Zt29SpUycXp3Idpq4VoN9uhHU9OJJtvbzW5+dsAtzRokWLdOjQIT388MOqWLGiJGnevHkqUaKEunTpYnM693D58mWtXLlSe/fuVc+ePeXn56ejR4/K399fxYsXtzuesf54g8o/498D63l6eurVV1/9y//HWd7bOu3bt1eLFi00YsSIPLcnJSWpQYMGuVamNQVFpwCtWrXK8fGBAwc0bNgwPfroo45V1uLj4zVv3jxFR0erb9++dsV0GwcPHrzm9sqVK7soifv6/PPP8xz38PBQkSJFVL16dVWpUsXFqQDXOnjwoO69916lpKQoIyNDu3fvVtWqVTV48GBlZGQoNjbW7ojG8vT01MaNG1W6dOlr7se/B9bw9PRUxYoVrzlNluW9rfXpp58qPT1dvXv3znP7mTNn9Pnnnxv7eylFxyJt27bVY489ph49ejiNz58/X2+//bZWrlxpTzDAha52NPWPR1LvuusuLVmyhHu6WCguLk5xcXF53k9qzpw5NqVyH127dpWfn59mz56tW265xXFftZUrV2rAgAHas2eP3RGNxWIE9uLPH3bLPbcHBSI+Pl5hYWG5xsPCwrRx40YbErmHzz//XJcuXXJ8fK0HrPfNN9/ojjvu0DfffKNz587p3Llz+uabb9SkSRN9+eWXWr16tU6dOqXnn3/e7qjGGjNmjNq1a6e4uDidPHlSZ86ccXrAemvWrNGIESPk7e3tNB4SEqIjR47YlAqwnskXuf/TrF271u4ItmAxAosEBwdr5syZmjhxotP4rFmzFBwcbFMq83Xt2tVx9Oha925hTrZrDB48WG+//baaNWvmGGvbtq2KFCmixx9/XNu2bVNMTIz69etnY0qzxcbGau7cuXrkkUfsjuK2srOz8/x5c/jwYfn5+dmQyH20atUqV8GE6zBp6ObRpk0bVahQQT169FDv3r0VGhpqdySXoOhYZMqUKXrooYf01VdfqUmTJpKkjRs3as+ePfrkk09sTmeuP07LMfXCun+SvXv35rnCoL+/v2NOdo0aNXTy5ElXR3MbmZmZTkUTrteuXTvFxMTo7bfflnTlQMuFCxc0evRodejQweZ0Zvv+++9zjf36669auHCh0tPTdc8996hGjRo2JHMPo0ePZrGNm8TRo0e1YMECffjhh5owYYLq1q2rXr16qUePHo5FakzENToWOnz4sN58803t3LlTklSrVi098cQTnNGB27jrrrvk5+end99913Ex8IkTJ9SnTx+lp6dr9erV+vbbb/XUU09p165dNqc109ChQ1W8eHGNHDnS7ihu6/Dhw2rfvr1ycnK0Z88ehYWFac+ePQoMDNTq1au5fsFCUVFRunTpkv773/9KulL8mzRpom3btsnX11eXL1/WN99841g0CAXr5MmTSk9Pd1rsYdu2bZo0aZLS09PVtWtX9ezZ08aE7mn//v2aP3++PvzwQ+3cuVMtW7bUd999Z3csS1B0YJSpU6de974sZ2m9Xbt2qUuXLtq/f7+j4B86dEhVq1bVZ599pltvvVVLlizRzz//zNQqiwwePFjvvvuu6tatq7p16+a6n9TkyZNtSuZeLl++rIULFyopKUkXLlxQw4YN1atXLxUtWtTuaEarXbu2xo8fr86dO0u6ck+j5557Tps3b1alSpXUr18/HT9+XEuXLrU5qZl69Oih8uXL6/XXX5ckHT9+XDVr1lT58uVVrVo1ffXVV5o9ezY//22QlZWlr776SiNHjtTWrVuNnc5P0bHQ2bNnNXv2bO3YsUOSdPvtt6tfv34KCAiwOZm5rnepYpazdJ3s7GytWLFCu3fvliTddtttuueee/K8zxEK3t13333VbR4eHsYexQOkK9NkExMTVb16dUlXfvH28/NzTCPcsmWLOnTooKNHj9oZ01hVqlTR3Llz1apVK0nSpEmTFBsbq507d6pQoUKaNGmSFi1apA0bNtic1H2sW7dOH3zwgRYtWqRff/1VXbp0Ua9evXTvvffaHc0SFB2LbNq0Se3bt1fRokXVuHFjSdIPP/ygX375RStWrFDDhg1tTgjcPOrUqaNly5YxrRNGio6OVlBQUK5FN+bMmaMTJ05o6NChNiUzX4kSJfTDDz84rsOpUqWKRo4c6XgvDhw4oFq1aumXX36xM6axihYtqp07dzqmrnXo0EG1a9d2LNS0e/duNW3aVKdOnbIzplsYPny4FixYoKNHj+qee+5Rr1691KVLF/n6+todzVIcUrXIs88+q86dO+vAgQNavHixFi9erP379+v+++/XM888Y3c8t5KZmaldu3bp8uXLdkfBVRw4cMCxLDiskZycrK+//trxCx3HuFznrbfeUs2aNXON33777dws1GK1atXSF198IenKtSEpKSlOZzkPHjyooKAgu+IZz9/fX2fPnnU837hxo2OBJunKWeWMjAwbkrmf1atXa8iQITpy5Ii+/PJL9ejRw/iSI1F0LLNp0yYNHTpUhQr9vrBdoUKF9MILL2jTpk02JnMfFy9eVP/+/eXr66vbb79dKSkpkqRBgwZpwoQJNqcDXOPUqVNq27atbr31VnXo0EHHjh2TJPXv31/PPfeczencQ2pqqsqVK5drvHTp0o73A9Z44YUXNHz4cLVt21Zt27ZVhw4dnKY4L1u2zDHrAgXvzjvv1NSpU5Wdna1Fixbp559/Vps2bRzbd+/ezZl8F1m3bp0GDhyowMBAu6O4FMtLW8Tf318pKSm5juIdOnSI+ya4yPDhw5WUlKSVK1c6zT0NDw/Xyy+/rGHDhtmYDnCNZ599VoULF1ZKSopq1arlGO/WrZuioqIcFwnDOsHBwVq3bl2uawjXrVun8uXL25TKPTzwwANatmyZvvzyS7Vr106DBg1y2u7r66unnnrKpnTme+WVV9S2bVu9//77unz5sl588UWVLFnSsX3BggWO63dgvT179uj777/X8ePHc92CY9SoUTalshZFxyLdunVT//79NWnSJMc9LNatW6chQ4aoR48eNqdzD0uWLNHChQt15513Ot2d+fbbb9fevXttTAa4zooVK/T111/nuk9CjRo1dPDgQZtSuZcBAwbomWee0aVLlxxHs+Pi4vTCCy9wVs0Ffjubk5fBgwdr2bJlLk7kPurWrasdO3Zo3bp1Klu2rNO0NUnq3r27br/9dpvSuZeZM2fqySefVGBgoMqWLev0e5GHhwdFB/kzadIkeXh4qE+fPo5rQwoXLqwnn3ySaVMucuLEiTzvT5Genu70FxwwWXp6ep7zsE+fPi0fHx8bErmfIUOG6NSpUxo4cKAyMzMlSUWKFNHQoUM1fPhwm9O5t4MHD+qRRx7hXi4WCgwMVJcuXfLcVq9ePY0dO9axCh6s8+qrr2rcuHFut/gJ1+hYxNvbW2+88YbOnDmjLVu2aMuWLTp9+rSmTJnCLxcuEhYW5nRvhN/KzaxZs7g5HNxGixYt9O677zqee3h4KDs7WxMnTrzm0tMoGFlZWVqzZo2GDRumEydOaMOGDUpKStLp06eNPYIKXK9Tp05p9uzZdsdwC2fOnNHDDz9sdwyX44yORc6dO6esrCyVKlVKderUcYyfPn1ahQoVkr+/v43p3MP48eN13333afv27bp8+bLeeOMNbd++XevXr9eqVavsjme8HTt2aMOGDWratKlq1qypnTt36o033lBGRoZ69+7tdEHqW2+9xcpHFpk4caLatm2rTZs2KTMzUy+88IK2bdum06dPa926dXbHM56Xl5fatWunHTt2qEqVKrrjjjvsjgTADT388MNasWKFnnjiCbujuBRFxyLdu3dXp06dNHDgQKfxjz76SJ9//jlzgl3grrvu0pYtWzRhwgTVqVPHcf+i+Ph4p/KJgrd8+XJ16dJFxYsX18WLF/Xpp5+qT58+qlevnrKzs9WuXTutWLHCUXaYNmKd2rVra/fu3Zo2bZr8/Px04cIFPfjgg3rqqafyXAkMBa927drat2/fdd/QGAAKWvXq1TVy5Eht2LBBderUUeHChZ22P/300zYlsxY3DLVIqVKltG7dOqdVjiRp586dat68OTfHgtGaNWumNm3a6NVXX9WCBQs0cOBAPfnkkxo3bpykKyviJSQkaMWKFTYnBay3fPlyDR8+XK+88ooaNWqkYsWKOW3nDL91pk6des3tR44c0aRJk5SVleWiRPijpKQkNWzYkD9/F7jWgRYPDw/t27fPhWlch6JjkWLFijla8x/9+OOPatKkiS5evGhTMveRmJiowoULO96Dzz77TO+8845CQ0P18ssvy9vb2+aE5goICFBCQoKqV6+u7Oxs+fj4aOPGjWrQoIEk6aefflJ4eLhSU1NtTmqmrVu3Xve+devWtTAJJMnT8/fLYf+4EEpOTo48PDz4Jc9C13sWbf/+/RYncU8PPvjgNbefPXtWq1at4u8ALMPUNYs0btxYb7/9tv773/86jcfGxqpRo0Y2pXIv//73vzVs2DDVqVNH+/btU7du3fTggw/q448/1sWLFxUTE2N3RKP99gudp6enihQpooCAAMc2Pz8/nTt3zq5oxqtfv748PDwcv0j/5rfjWn8c4xcM633//fd2R3BbFBh7/fHn/tW29+nTx0Vp8Ju8/i0wFUXHIq+++qrCw8OVlJTkWL8/Li5OP/zwA9N1XGT37t2qX7++JOnjjz9Wq1atNH/+fK1bt07du3en6FgoJCREe/bsUbVq1SRJ8fHxqlSpkmN7SkoK14dY6I+/3G3evFnPP/+8hgwZ4lhtMD4+Xq+//romTpxoV0S3wg0R/znq1KmjZcuWKTg42O4oRnjnnXfytf/hw4dVvnx5p7OgKDjvvvuuXnvtNe3Zs0eSdOutt2rIkCF65JFHbE5mHYqORZo3b674+Hi99tpr+uijj1S0aFHVrVtXs2fPVo0aNeyO5xZycnIcd/799ttvdf/990u6cpfykydP2hnNeE8++aTTmYLatWs7bf/qq6+cVl1DwapcubLj44cfflhTp05Vhw4dHGN169ZVcHCwRo4cqa5du9qQ0P2sWbNGb731lvbt26ePP/5YFSpU0HvvvacqVarorrvusjse/r8DBw7o0qVLdsdwW6GhodqyZYuqVq1qdxTjTJ48WSNHjlRkZKSaN28uSVq7dq2eeOIJnTx5Us8++6zNCa3BNTowVps2bRQcHKzw8HD1799f27dvV/Xq1bVq1Sr17dtXBw4csDsiYLmiRYsqMTEx18IoO3bsUMOGDfXLL7/YlMx9fPLJJ3rkkUfUq1cvvffee9q+fbuqVq2qadOmadmyZazCeRPx8/NTUlISv2jbhD9/61SpUkVjxozJNVVw3rx5evnll42d5sm5QQtlZ2dr9+7dWrt2rVavXu30gPViYmKUmJioyMhIvfTSS6pevbokadGiRWrWrJnN6QDXqFWrlqKjo5WZmekYy8zMVHR0dK7yA2u8+uqrio2N1cyZM52WdG3evLkSExNtTAbAXRw7dizP332aNWumY8eO2ZDINZi6ZpENGzaoZ8+eOnjwoP580oxVdlyjbt26+vHHH3ONv/baa/Ly8rIhEeB6sbGx6tSpkypWrOhYYW3r1q3y8PDQF198YXM697Br1y61bNky13hAQIDOnj3r+kAA3E716tX10Ucf6cUXX3QaX7hwodGXVFB0LPLEE08oLCxMS5cuVbly5dxiZYt/iiJFitgdAXCZxo0ba9++ffrggw+0c+dOSVK3bt3Us2fPXPdzgTXKli2r5ORkhYSEOI2vXbuWKToAXGLMmDHq1q2bVq9e7bhGZ926dYqLi9NHH31kczrrUHQssmfPHi1atMgxXQqul5WVpSlTpuijjz5SSkqK09QdSTp9+rRNyQDXKlasmB5//PFr7tOxY0fNmjWL1fAsMGDAAA0ePFhz5syRh4eHjh49qvj4eD3//PMaOXKk3fGAmwYHha3z0EMP6X//+5+mTJmiJUuWSLoytfmP97gzEUXHIk2aNFFycjJFx0ZjxozRrFmz9Nxzz2nEiBF66aWXdODAAS1ZskSjRo2yOx5wU1m9ejULE1hk2LBhys7OVtu2bXXx4kW1bNlSPj4+ev755zVo0CC74+EP3nrrLQUFBdkdw22xPpa1GjVqpPfff9/uGC7FqmsW+fTTTzVixAgNGTJEderUcboAVeJu5K5QrVo1TZ06VR07dpSfn5+2bNniGNuwYYPmz59vd0TgpsFqR9bLzMxUcnKyLly4oNDQUBUvXtzuSG7hl19+UUJCgkqVKqXQ0FCnbb/++qs++ugjblp5kzh06JDKly/PdbQWWLZsmby8vNS+fXun8a+//lrZ2dm67777bEpmLYqORfK62dUf71TOYgTWK1asmHbs2KFKlSqpXLlyWrp0qRo2bKh9+/apQYMGOnfunN0RgZsGRcc1Dh06JEnckNJFdu/erXbt2iklJUUeHh666667tGDBAscUzbS0NJUvX55/ky3w4IMPXve+ixcvtjAJpCsH2CdMmOB0TzVJWr58uYYOHaqkpCSbklmL5aUtsn///lyPffv2Of4L61WsWNGxZGK1atW0YsUKSdIPP/wgHx8fO6MBcCOXL1/WyJEjFRAQoJCQEIWEhCggIEAjRozg5pQWGzp0qGrXrq3jx49r165d8vPzU/PmzZWSkmJ3NOMFBARc9wPW27NnT64zmpJUs2ZNJScn25DINbhGxyJ/vDM57PHAAw8oLi5OTZo00aBBg9S7d2/Nnj1bKSkpxt4BGMDNZ9CgQVq8eLEmTpyopk2bSpLi4+P18ssv69SpU5oxY4bNCc21fv16ffvttwoMDFRgYKC++OILDRw4UC1atND333/PyoMWeuedd+yOgD8ICAjQvn37cq3+mJycbPTfA6auWei9995TbGys9u/fr/j4eFWuXFkxMTGqUqWKunTpYnc8txMfH6/4+HjVqFFDnTp1sjsOcFNh6pp1AgICtGDBglxz4JctW6YePXowjdZC/v7++t///pfr5riRkZH67LPPNH/+fLVu3ZqpazDev//9b8XHx+vTTz9VtWrVJF0pOQ899JDuuOMOzZo1y+aE1uCMjkVmzJihUaNG6ZlnntG4ceMcP0RLlCihmJgYio4NmjZt6jiaCsDZiy++qFKlStkdw0g+Pj65jqJKUpUqVeTt7e36QG6kZs2a2rRpU66iM23aNElS586d7YjllhYtWnTV2z0kJibalMp9TJw4Uffee69q1qypihUrSpIOHz6sFi1aaNKkSTansw7X6Fjkv//9r2bOnKmXXnrJafWQsLAw/fjjjzYmcy+7du1SZGSk2rZtq7Zt2yoyMlK7du2yOxbgUu+9956aN2+u8uXL6+DBg5KkmJgYffbZZ459hg8frhIlStiU0GyRkZF65ZVXlJGR4RjLyMjQuHHjFBkZaWMy8z3wwAP68MMP89w2bdo09ejRgyWNXWDq1KmKiIhQUFCQNm/erMaNG+uWW27Rvn37jF3t62YTEBCg9evXa+nSpRo4cKCee+45xcXF6bvvvjP6Zz9T1yxStGhR7dy5U5UrV3aaErJnzx7VrVuX+1W4wCeffKLu3bsrLCzMcSZnw4YN+uGHH7RgwQI99NBDNicErPfns8s//fSTqlatqrlz52revHn6/vvv7Y5ovN+uF/Tx8VG9evUkSUlJScrMzFTbtm2d9mX1KZioZs2aGj16tHr06OH0O9GoUaN0+vRpxxk22K9OnTpatmyZMStDMnXNIlWqVNGWLVtyLUqwfPnyXKfQYY0XXnhBw4cP19ixY53GR48erRdeeIGiA7fw29nlrl27asKECY7xsLAwPf/88zYmcx8lSpTI9fPGlF8igOuRkpKiZs2aSbpyIPjnn3+WJD3yyCO68847KTo3kQMHDhi1GiRFxyJRUVF66qmn9OuvvyonJ0cbN27Uhx9+qOjoaGMv+LrZHDt2LM+bwPXu3VuvvfaaDYkA19u/f78aNGiQa9zHx0fp6ek2JHI/17v61Lp165SRkcHy9zBO2bJldfr0aVWuXFmVKlXShg0bVK9ePe3fv5+pg7AURccijz32mIoWLaoRI0bo4sWL6tmzp8qXL6833nhD3bt3tzueW2jdurXWrFmj6tWrO42vXbtWLVq0sCkV4FqcXf7nuO+++7RlyxZWvoNx2rRpo88//1wNGjRQRESEnn32WS1atEibNm3K141Fgfyi6FioV69e6tWrly5evKgLFy6oTJkyufZZt26dwsLCOIJXQD7//HPHx507d9bQoUOVkJCgO++8U9KVa3Q+/vhjjRkzxq6IgEtxdvmfgyPbMNXbb7+t7OxsSdJTTz2lW265RevXr1fnzp3173//2+Z0MBmLEdjM39+fI3gFyNPz+hYS9PDw4L4JcBsffPCBXn75Ze3du1eSVL58eY0ZM0b9+/e3ORn+iHsZAbCbaT+HOKNjM3pmwfrtiBGA313P2WUAsNLZs2e1ceNGHT9+PNe/1XldTwsUBIoO3J5pSykCV+Pr6ytfX1+7YwBwM1988YV69eqlCxcuyN/fXx4eHo5tHh4eFB0XSk9P10cffaTk5GSVK1dOPXr00C233OLY/tZbbykoKMjGhAWLqWs2M+0U4T8R7wFM06BBA6dfJK6FO5LfPJjKDFPdeuut6tChg8aPH8/BFhcLDQ3V2rVrVapUKR06dEgtW7bUmTNndOutt2rv3r0qVKiQNmzYoCpVqtgd1RKc0QEAw3Tt2tXuCLgBHHeEqY4cOaKnn36akmODnTt36vLly5Kk4cOHq3z58tqyZYsCAgJ04cIFPfDAA3rppZc0f/58m5Nag6Jjs+s96goA12v06NF2R8CfnDt3TqmpqZKu3FMkICAg1z6/3UQRME379u21adMmzlbaLD4+XrGxsY6fP8WLF9eYMWOMvu0JRcdmHMED4AqbNm3Sjh07JF2ZytCoUSObE7mHWbNmafLkydq1a5fT+G233abnnnuOle/gFjp27KghQ4Zo+/btqlOnjgoXLuy0vXPnzjYlcw+/HVT/9ddfVa5cOadtFSpU0IkTJ+yI5RIUHQtdvnxZK1eu1N69e9WzZ0/5+fnp6NGj8vf3V/HixSVxBA+AtQ4fPqwePXpo3bp1KlGihKQrqx81a9ZMCxYsUMWKFe0NaLDXXntNL7/8sp5++mm1b9/ecYFvWlqaVqxYocGDB+vMmTN6/vnnbU4KWGvAgAGSpLFjx+baxu0erNe2bVsVKlRI58+f165du1S7dm3HtoMHDzotRmAaio5FDh48qHvvvVcpKSnKyMjQPffcIz8/P/3nP/9RRkaGYmNj7Y4IwA089thjunTpknbs2KHbbrtNkrRr1y5FREToscce0/Lly21OaK5p06bpnXfe0b/+9S+n8Vq1aql169aqV6+ehgwZQtGB8bj1g33+PJX5twPtv/niiy/UokULV0ZyKVZds0jXrl3l5+en2bNn65ZbbnGs6rVy5UoNGDBAe/bssTui0U6ePKk5c+YoPj7eaV58s2bN9Oijj6p06dKOfefPn68uXbqoWLFidsUFLFO0aFGtX79eDRo0cBpPSEhQixYtdPHiRZuSma9o0aJKTExUrVq18ty+fft2hYWF8R4AgEWu7zbyyLc1a9ZoxIgR8vb2dhoPCQnRkSNHbErlHn744Qfdeuutmjp1qgICAtSyZUu1bNlSAQEBmjp1qmrWrKlNmzY59u/ZsyclB8YKDg7WpUuXco1nZWWpfPnyNiRyH3fccYcmTJjgWPHoj7KysvSf//xHd9xxhw3JANdbtWqVOnXqpOrVq6t69erq3Lmz1qxZY3csGI6paxbJzs7Oc87p4cOH5efnZ0Mi9zFo0CA9/PDDio2NzbWqXU5Ojp544gkNGjRI8fHxNiUEXOe1117ToEGDNH36dIWFhUm6sjDB4MGDNWnSJJvTmW3atGlq3769ypYtq5YtWzpdo7N69Wp5e3trxYoVNqcErPf+++8rIiJCDz74oJ5++mlJ0rp169S2bVvNnTtXPXv2tDkhTMXUNYt069ZNAQEBevvtt+Xn56etW7eqdOnS6tKliypVqqR33nnH7ojGKlq0qDZv3qyaNWvmuX3nzp1q0KCBfvnlFxcnA1yvZMmSunjxoi5fvqxCha4c2/rt4z+fyTx9+rQdEY32888/6/3339eGDRucptE2bdpUPXv2lL+/v80JAevVqlVLjz/+uJ599lmn8cmTJ2vmzJmOFSGBgkbRscjhw4fVvn175eTkaM+ePQoLC9OePXsUGBio1atXq0yZMnZHNFaVKlU0ZswY9enTJ8/t7777rkaNGqUDBw64Nhhgg3nz5l33vn379rUwCQB35ePjo23btql69epO48nJyapdu7Z+/fVXm5LBdExds0jFihWVlJSkhQsXKikpSRcuXFD//v3Vq1cvFS1a1O54Rnv++ef1+OOPKyEhQW3btnWaLhIXF6eZM2cyZQdug/Jy87p06ZKOHTumSpUq2R0FsFRwcLDi4uJyFZ1vv/1WwcHBNqWCO+CMDoy0cOFCTZkyRQkJCY5rpby8vNSoUSNFRUXlWu4VMN3x48d1/PjxXMu81q1b16ZESEpKUsOGDbmHCIw3Y8YMPfPMM+rXr5+aNWsm6co1OnPnztUbb7yhf//73zYnhKkoOhaZN2+eAgMD1bFjR0nSCy+8oLfffluhoaH68MMPVblyZZsTuodLly7p5MmTkqTAwMBcd2MGTJeQkKC+fftqx44d+vOPe27UZy+KDtzJp59+qtdff91xPU6tWrU0ZMgQdenSxeZkMBlFxyK33XabZsyYoTZt2ig+Pl5t27ZVTEyMvvzySxUqVEiLFy+2OyIAN1CvXj1Vq1ZNQ4cOVVBQUK6VCDnoYp2GDRtec/svv/yi3bt3U3QAwCIUHYv4+vpq586dqlSpkoYOHapjx47p3Xff1bZt29S6dWudOHHC7ogA3ICfn582b96ca248rFekSBF1795dVapUyXP7sWPHNHPmTIoOAFiExQgsUrx4cZ06dUqVKlXSihUrFBUVJenKP3wsawzAVdq2baukpCSKjg1q166tJk2a6Mknn8xz+5YtWzRz5kwXpwJco1SpUtq9e7cCAwNVsmTJXGeT/4il7WEVio5F7rnnHj322GNq0KCBdu/erQ4dOkiStm3bppCQEHvDAXAbs2bNUt++ffXTTz+pdu3aua5T69y5s03JzNe8eXPt2rXrqtv9/PzUsmVLFyYCXGfKlCmOG6RPmTLlmkUHsApT1yxy9uxZjRgxQocOHdKTTz6pe++9V5I0evRoeXt766WXXrI5IQB38MUXX+iRRx7R+fPnc21jMQIAgMkoOgBgsJCQEN1///0aOXKk455SuDkNHDhQY8eOVWBgoN1RgALl5eWlY8eO5bpZ+qlTp1SmTBkOuMAyFB2LrF69+prbma4AwBX8/Py0ZcsWVatWze4o+Av+/v7asmWLqlatancUoEB5enoqNTU1V9E5evSoqlWrxrXLsAzX6FikdevWucb+OD+VoxcAXOHBBx/U999/T9H5B+C4I0wzdepUSVd+/5k1a5aKFy/u2JaVlaXVq1erZs2adsWDG6DoWOTMmTNOzy9duqTNmzdr5MiRGjdunE2pALibW2+9VcOHD9fatWtVp06dXIsRPP300zYlA2C6KVOmSLpS4mNjY+Xl5eXY5u3trZCQEMXGxtoVD26AqWsutmrVKkVFRSkhIcHuKADcwNXu4SJdOcq6b98+F6bBtfj5+SkpKYmpazDO3XffrcWLF6tkyZJ2R4Gb4YyOiwUFBV1zuVEAKEj79++3OwIAN/f999/bHQFuiqJjka1btzo9z8nJ0bFjxzRhwgTVr1/fnlAAAAAu9tBDD6lx48YaOnSo0/jEiRP1ww8/6OOPP7YpGUxH0bFI/fr15eHhkevi0jvvvFNz5syxKRUAdxAVFaVXXnlFxYoVU1RU1DX3nTx5sotSuafLly9r/Pjx6tevnypWrHjNfXv37i1/f38XJQNcZ/Xq1Xr55Zdzjd933316/fXXXR8IboOiY5E/Txfx9PRU6dKlVaRIEZsSAXAXmzdv1qVLlxwfXw13KrdeoUKF9Nprr6lPnz5/ue+MGTNckAhwvQsXLsjb2zvXeOHChfO8mTFQUCg6FqlcufJ17VenTh0tW7ZMwcHBFicC4C7+OB+eufH2a9OmjVatWqWQkBC7owC2qFOnjhYuXKhRo0Y5jS9YsEChoaE2pYI7oOjY7MCBA44jrwBgtfPnz+u7775TzZo1uX+Fi9x3330aNmyYfvzxRzVq1EjFihVz2t65c2ebkgGuMXLkSD344IPau3ev2rRpI0mKi4vThx9+yPU5sBTLS9uM5UQBWOlf//qXWrZsqcjISP3yyy+qV6+eDhw4oJycHC1YsEAPPfSQ3RGN5+npedVtHh4e3EAabmHp0qUaP368tmzZoqJFi6pu3boaPXq0WrVqZXc0GIwzOgBgsNWrV+ull16SJH366afKycnR2bNnNW/ePL366qsUHRfIzs62OwJgu44dO6pjx452x4CbufphJgDAP965c+dUqlQpSdLy5cv10EMPydfXVx07dtSePXtsTud+fv31V7sjALY4e/asZs2apRdffFGnT5+WJCUmJurIkSM2J4PJKDoAYLDg4GDFx8crPT1dy5cvV7t27SRJZ86cYRVIF8nKytIrr7yiChUqqHjx4tq3b5+kK9ctzJ492+Z0gPW2bt2qW2+9Vf/5z3/02muv6ezZs5KkxYsXa/jw4faGg9EoOgBgsGeeeUa9evVSxYoVVb58ebVu3VrSlSltderUsTecmxg3bpzmzp2riRMnOi2xW7t2bc2aNcvGZIBrREVF6dFHH9WePXucDrB06NBBq1evtjEZTMdiBDabP3++unTpkmsVHgAoKAkJCUpJSdE999yj4sWLS7pyYXCJEiXUvHlzm9OZr3r16nrrrbfUtm1bpwVodu7cqaZNm+rMmTN2RwQsFRAQoMTERFWrVs3p78DBgwd12223MaUTluGMjoXi4uJ0//33q1q1aqpWrZruv/9+ffvtt0779OzZk5IDwFKNGjXSAw884Cg50pULg/9Ycvz9/R1TqlCwjhw5ourVq+caz87O5vYCcAs+Pj553hh09+7dKl26tA2J4C4oOhZ58803de+998rPz0+DBw/W4MGD5e/vrw4dOmj69Ol2xwMAJ5zct05oaKjWrFmTa3zRokVq0KCBDYkA1+rcubPGjh3rKPYeHh5KSUnR0KFDWfkRlmJ5aYuMHz9eU6ZMUWRkpGPs6aefVvPmzTV+/Hg99dRTNqYDALjKqFGj1LdvXx05ckTZ2dlavHixdu3apXfffVdffvml3fEAy73++uv6v//7P5UpU0a//PKLWrVqpdTUVDVt2lTjxo2zOx4MxjU6FilevLi2bNmSa7rCnj171KBBA124cMGmZACQGzcvttaaNWs0duxYJSUl6cKFC2rYsKFGjRrlWAUPcAfr1q1z+jsQHh5udyQYjqJjkZ49e6pBgwYaMmSI0/ikSZO0adMmLViwwKZkAJAbRQeAFS5duqSiRYtqy5Ytql27tt1x4GaYulaApk6d6vg4NDRU48aN08qVK9W0aVNJ0oYNG7Ru3To999xzdkUEgDx5eHjYHQGAgQoXLqxKlSopKyvL7ihwQ5zRKUBVqlS5rv08PDxY3QjATYUzOgWrZMmS110ef7tLPGCq2bNna/HixXrvvfdUqlQpu+PAjXBGpwDt37/f7ggAcF0OHTqk0aNHa86cOZKkr776ShUqVLA5lTliYmIcH586dUqvvvqq2rdv7zjDHx8fr6+//lojR460KSHgOtOmTVNycrLKly+vypUr57qtRmJiok3JYDrO6ACAG0pKSlLDhg2ZTuICDz30kO6++26nVTilK7/8ffvtt1qyZIk9wQAXGTNmzDW3jx492kVJ4G4oOhbp16/fNbf/dhQVAKzw+eefX3P7vn379Nxzz1F0XOBqq3AmJyerfv36rMIJABZh6ppFzpw54/T80qVL+umnn3T27Fm1adPGplQA3EXXrl3l4eFxzRuBsgCBa9xyyy367LPPci1E89lnn+mWW26xKRUAmI+iY5FPP/0011h2draefPJJVatWzYZEANxJuXLl9Oabb6pLly55bt+yZYsaNWrk4lTuacyYMXrssce0cuVKNWnSRJL0v//9T8uXL9fMmTNtTgdYo1SpUtq9e7cCAwP/cnEOFuSAVSg6LuTp6amoqCi1bt1aL7zwgt1xABisUaNGSkhIuGrR+auzPSg4jz76qGrVqqWpU6dq8eLFkqRatWpp7dq1juIDmGbKlCny8/OT5Lw4B+BKXKPjYsuWLVPfvn114sQJu6MAMNiaNWuUnp6ue++9N8/t6enp2rRpk1q1auXiZAAAuAZFxyJRUVFOz3NycnTs2DEtXbpUffv21bRp02xKBgBwtaysLC1ZskQ7duyQJN1+++3q3LmzvLy8bE4GWOP8+fPXva+/v7+FSeDOKDoWufvuu52ee3p6qnTp0mrTpo369eunQoWYNQgA7iA5OVkdO3bU4cOHddttt0mSdu3apeDgYC1dupTrNmEkT0/P617whNUfYRWKDgAAFurQoYNycnL0wQcfOO4Kf+rUKfXu3Vuenp5aunSpzQmBgrdq1SrHxwcOHNCwYcP06KOPOt00d968eYqOjlbfvn3tignDUXQAALBQsWLFtGHDBtWpU8dpPCkpSc2bN+c+OjBe27Zt9dhjj6lHjx5O4/Pnz9fbb7+tlStX2hMMxvO0O4Cp0tLS9Mgjj6h8+fIqVKiQvLy8nB4AAPfg4+Ojn3/+Odf4hQsX5O3tbUMiwLXi4+MVFhaWazwsLEwbN260IRHcBReKWOTRRx9VSkqKRo4cqXLlynFjPgBwU/fff78ef/xxzZ49W40bN5Z05T46TzzxhDp37mxzOsB6wcHBmjlzpiZOnOg0PmvWLAUHB9uUCu6AqWsW8fPz05o1a1S/fn27owAAbHT27Fn17dtXX3zxhQoXLixJunz5sjp37qy5c+cqICDA5oSAtZYtW6aHHnpI1atXd9w7auPGjdqzZ48++eQTdejQweaEMBVFxyKhoaH64IMP1KBBA7ujAABuAsnJyY7lpWvVqqXq1avbnAhwnUOHDmnGjBnauXOnpCt/B5544gnO6MBSFB2LrFixQq+//rreeusthYSE2B0HAAAAcCsUnQJUsmRJp2tx0tPTdfnyZfn6+jqmK/zm9OnTro4HALDBQw89pMaNG2vo0KFO4xMnTtQPP/ygjz/+2KZkgHW2bt2q2rVry9PTU1u3br3mvnXr1nVRKrgbik4Bmjdv3nXvy5rxAOAeSpcure+++y7X8tI//vijwsPDlZaWZlMywDqenp5KTU1VmTJlHDcPzetXTg8PD24YCsuw6loBupHyMmHCBD3xxBMqUaJEwQcCANjuastIFy5cWOfPn7chEWC9/fv3q3Tp0o6PATtwRsdm/v7+2rJli6pWrWp3FACABRo3bqz7779fo0aNchp/+eWX9cUXXyghIcGmZABgNs7o2IyeCQBmGzlypB588EHt3btXbdq0kSTFxcXpww8/5PocuI1du3bpv//9r9PKg4MGDdJtt91mczKYzNPuAAAAmKxTp05asmSJkpOTNXDgQD333HM6fPiwvv32W3Xt2tXueIDlPvnkE9WuXVsJCQmqV6+e6tWrp8TERNWuXVuffPKJ3fFgMKau2czPz09JSUlMXQMAAEaqVq2aevXqpbFjxzqNjx49Wu+//7727t1rUzKYjqJjM4oOALiHzMxMHT9+XNnZ2U7jlSpVsikR4Bq+vr7aunVrrpvk7tmzR/Xq1dPFixdtSgbTcY0OAAAW2rNnj/r166f169c7jefk5LC0LtxC69attWbNmlxFZ+3atWrRooVNqeAOKDoFKCoqSq+88oqKFSum1atXq1mzZipU6Np/xC1atFDRokVdlBAA4GqPPvqoChUqpC+//FLlypVzurE0YKrPP//c8XHnzp01dOhQJSQk6M4775QkbdiwQR9//LHGjBljV0S4AaauFaDChQvr8OHDCgoKkpeXl44dO6YyZcrYHQsAYKNixYopISFBNWvWtDsK4DKente33hVnNWElzugUoJCQEE2dOlXt2rVTTk6O4uPjVbJkyTz3bdmypYvTAQDsEBoaqpMnT9odA3CpP1+LBtiBMzoFaMmSJXriiSd0/PhxeXh4XPUeORy9AAD38d1332nEiBEaP3686tSpo8KFCztt9/f3tykZcHOpU6eOli1bpuDgYLujwBAUHQtcuHBB/v7+2rVr11WnrgUEBLg4FQDADr9N4fnztTksRgA4YyVaFDSmrlmgePHi+v7771WlSpW/XIwAAGC277//3u4IAOCWOKNjkastRnDq1CmVKVOGI3gAAAB/wBkdFLTrWxID+Xa1/piRkSFvb28XpwEA2GnNmjXq3bu3mjVrpiNHjkiS3nvvPa1du9bmZABgLuZVFbCpU6dKujIXe9asWSpevLhjW1ZWllavXs0SowDgRj755BM98sgj6tWrlxITE5WRkSFJOnfunMaPH69ly5bZnBAAzMTUtQJWpUoVSdLBgwdVsWJFeXl5ObZ5e3srJCREY8eOVZMmTeyKCABwoQYNGujZZ59Vnz59nKbmbN68Wffdd59SU1Ptjgi41G8LcfwZU9dQ0DijU8D2798vSbr77ru1ePHiq95HBwDgHnbt2pXnvdMCAgJ09uxZ1wcCbObj46OkpCTVqlXLafytt95SUFCQTalgIoqORa53lR1/f39t2bKFoxcAYKiyZcsqOTlZISEhTuNr167lZz+MFhUVled4VlaWJkyYoFtuuUWSNHnyZElSz549XZYN7oGiYzNmDgKA2QYMGKDBgwdrzpw58vDw0NGjRxUfH6/nn39eI0eOtDseYJmYmBjVq1dPJUqUcBrPycnRjh07VKxYsTynsAEFhaIDAICFhg0bpuzsbLVt21YXL15Uy5Yt5ePjo+eff16DBg2yOx5gmfHjx+vtt9/W66+/rjZt2jjGCxcurLlz5yo0NNTGdHAHLEZgMy68AwD3kJmZqeTkZF24cEGhoaFOq3JK0uHDh1W+fHl5enLnB5jjhx9+UO/evdWpUydFR0ercOHCKly4sJKSkig6sBw/TQEAcAFvb2+FhoaqcePGuUqOJIWGhurAgQOuDwZY6I477lBCQoJOnDihsLAw/fTTT0xXg8swdc1m/GUHAEhcswlzFS9eXPPmzdOCBQsUHh6urKwsuyPBTVB0bMY/bAAAwB10795dd911lxISElS5cmW748ANUHRs9tVXX6lChQp2xwAAALBcxYoVVbFiRbtjwE1wjU4BS0xMdNw0VJLee+89NW/eXMHBwbrrrru0YMECp/3vuusu+fj4uDomAAAAYDSKTgGLiIjQ3r17JUmzZs3Sv//9b4WFhemll17SHXfcoQEDBmjOnDk2pwQA3Gy4ZhMAChZT1wrYnj17VKNGDUnSm2++qTfeeEMDBgxwbL/jjjs0btw49evXz66IAICbENdsAkDB4oxOAfP19dXJkyclSUeOHFHjxo2dtjdp0sRpahsAwD0kJyfr66+/1i+//CIpd7HZvn07F2gDQAGi6BSw++67TzNmzJAktWrVSosWLXLa/tFHH6l69ep2RAMA2ODUqVMKDw/Xrbfeqg4dOujYsWOSpP79++u5555z7BccHCwvLy+7YgKAcTxyOFdeoI4eParmzZurUqVKCgsL04wZM9SoUSPVqlVLu3bt0oYNG/Tpp5+qQ4cOdkcFALhAnz59dPz4cc2aNUu1atVSUlKSqlatqq+//lpRUVHatm2b3REBwEhco1PAypcvr82bN2vChAn64osvlJOTo40bN+rQoUNq3ry51q1bp7CwMLtjAgBcZMWKFfr6669zLalbo0YNHTx40KZUAGA+io4FSpQooQkTJmjChAl2RwEA2Cw9PV2+vr65xk+fPs3tBQDAQlyjAwCAhVq0aKF3333X8dzDw0PZ2dmaOHGi7r77bhuTAYDZuEYHAAAL/fTTT2rbtq0aNmyo7777Tp07d9a2bdt0+vRprVu3TtWqVbM7IgAYiaIDAIDFzp07p2nTpikpKUkXLlxQw4YN9dRTT6lcuXJ2RwMAY1F0AAAAABiHa3QAALDQ8uXLtXbtWsfz6dOnq379+urZs6fOnDljYzIAMBtFBwAACw0ZMkTnz5+XJP3444+KiopShw4dtH//fkVFRdmcDgDMxfLSAABYaP/+/QoNDZUkffLJJ+rUqZPGjx+vxMREbh4NABbijA4AABby9vbWxYsXJUnffvut2rVrJ0kqVaqU40wPAKDgcUYHAAAL3XXXXYqKilLz5s21ceNGLVy4UJK0e/duVaxY0eZ0AGAuzugAAGChadOmqVChQlq0aJFmzJihChUqSJK++uor3XvvvTanAwBzsbw0AAAAAOMwdQ0AAAulpKRcc3ulSpVclAQA3AtndAAAsJCnp6c8PDyuuj0rK8uFaQDAfXBGBwAAC23evNnp+aVLl7R582ZNnjxZ48aNsykVAJiPMzoAANhg6dKleu2117Ry5Uq7owCAkVh1DQAAG9x222364Ycf7I4BAMZi6hoAABb6801Bc3JydOzYMb388suqUaOGTakAwHwUHQAALFSiRIlcixHk5OQoODhYCxYssCkVAJiPa3QAALDQqlWrnJ57enqqdOnSql69ugoV4ngjAFiFogMAwE2gY8eOmjVrlsqVK2d3FAAwAosRAABwE1i9erV++eUXu2MAgDEoOgAAAACMQ9EBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAG4CL774okqVKmV3DAAwBvfRAQDAYrt27dJ///tf7dixQ5JUq1YtDRo0SLfddpvNyQDAXJzRAQDAQp988olq166thIQE1atXT/Xq1VNiYqJq166tTz75xO54AGAszugAAGChatWqqVevXho7dqzT+OjRo/X+++9r7969NiUDALNRdAAAsJCvr6+2bt2q6tWrO43v2bNH9erV08WLF21KBgBmY+oaAAAWat26tdasWZNrfO3atWrRooUNiQDAPRSyOwAAAKb5/PPPHR937txZQ4cOVUJCgu68805J0oYNG/Txxx9rzJgxdkUEAOMxdQ0AgALm6Xl9EyY8PDyUlZVlcRoAcE8UHQAAAADG4RodAAAAAMbhGh0AACz052Wl/2zUqFEuSgIA7oWpawAAWKhBgwZOzy9duqT9+/erUKFCqlatmhITE21KBgBm44wOAAAW2rx5c66x8+fP69FHH9UDDzxgQyIAcA+c0QEAwAY//vijOnXqpAMHDtgdBQCMxGIEAADY4Ny5czp37pzdMQDAWExdAwDAQlOnTnV6npOTo2PHjum9997TfffdZ1MqADAfU9cAALBQlSpVnJ57enqqdOnSatOmjYYPHy4/Pz+bkgGA2Sg6AAAAAIzDNToAAAAAjMM1OgAAWCg9PV0TJkxQXFycjh8/ruzsbKft+/btsykZAJiNogMAgIUee+wxrVq1So888ojKlSsnDw8PuyMBgFvgGh0AACxUokQJLV26VM2bN7c7CgC4Fa7RAQDAQiVLllSpUqXsjgEAboeiAwCAhV555RWNGjVKFy9etDsKALgVpq4BAFDAGjRo4HQtTnJysnJychQSEqLChQs77ZuYmOjqeADgFliMAACAAta1a1e7IwCA2+OMDgAAN4EPP/xQnTt3VrFixeyOAgBGoOgAAHAT8Pf315YtW1S1alW7owCAEViMAACAmwDHHQGgYFF0AAAAABiHogMAAADAOBQdAAAAAMah6AAAAAAwDkUHAIACNnXqVP3666+SpJSUlOtaaKBy5cq5biYKALhxLC8NAEABK1SokI4ePaoyZcrIy8tLx44dU5kyZeyOBQBupZDdAQAAME358uX1ySefqEOHDsrJydHhw4cdZ3j+rFKlSi5OBwDugTM6AAAUsLfffluDBg3S5cuXr7pPTk6OPDw8lJWV5cJkAOA+KDoAAFjg559/1sGDB1W3bl19++23uuWWW/Lcr169ei5OBgDugaIDAICF5s2bp+7du8vHx8fuKADgVlh1DQAAC40ZM0YXLlzINX727FlVrVrVhkQA4B4oOgAAWOjAgQN5XoeTkZGhI0eO2JAIANwDq64BAGCBzz//3PHx119/rYCAAMfzrKwsxcXFKSQkxIZkAOAeuEYHAAALeHpemTTh4eGR64ahhQsXVkhIiF5//XXdf//9dsQDAONRdAAAsFCVKlX0ww8/KDAw0O4oAOBWuEYHAAAL7d+//7pKTp06dXTo0CEXJAIA90DRAQDgJnDgwAFdunTJ7hgAYAyKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAAsMGjRIa9asue7933rrLQUFBVmYCADcCzcMBQDAAp6envLw8FC1atXUv39/9e3bV2XLlrU7FgC4Dc7oAABgkRUrVqhDhw6aNGmSKlWqpC5duujLL79Udna23dEAwHgUHQAALFKnTh3FxMTo6NGjev/995WRkaGuXbsqODhYL730kpKTk+2OCADGYuoaAAAW8PT0VGpqqsqUKeM0npKSojlz5mju3Lk6dOiQsrKybEoIAGaj6AAAYIGrFZ3f5OTk6Ntvv9U999zj4mQA4B6YugYAgAUqV64sLy+vq2738PCg5ACAhTijAwAAAMA4nNEBAAAAYByKDgAAAADjUHQAAAAAGIeiAwAAAMA4FB0AAAAAxqHoAAAAADAORQcAAACAcSg6AAAAAIzz/wC+YZ+3kbiRdwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Saving and loading a trained model"
      ],
      "metadata": {
        "id": "5y-iAxJQz3yM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save TF Hub Sentence Encoder model to HDF5 format\n",
        "model_6.save(\"model_6.h5\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MtNqh6yc2ScR",
        "outputId": "85bbb274-ed75-41a6-baa0-7421a0d1f622"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
            "  saving_api.save_model(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Load model with custom Hub Layer (req when using HDF5 format)\n",
        "import tensorflow_hub as hub\n",
        "loaded_model_6 = tf.keras.models.load_model(\"model_6.h5\",\n",
        "                                            custom_objects = {\"KerasLayer\": hub.KerasLayer})"
      ],
      "metadata": {
        "id": "HsWRcJj62pPL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluating the loaded model\n",
        "loaded_model_6.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yAIIDTc13G0S",
        "outputId": "13d3d491-859d-4879-9ef2-5486ad6dd279"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 19ms/step - loss: 0.4512 - accuracy: 0.8281\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.45119187235832214, 0.8280839920043945]"
            ]
          },
          "metadata": {},
          "execution_count": 149
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Saving to the `SavedModel` format"
      ],
      "metadata": {
        "id": "XM__-1Ra3c0D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save TH Hub Sentence Encoder Model to SavedModel format (default)\n",
        "model_6.save(\"model_6_SavedModel_format\")"
      ],
      "metadata": {
        "id": "bfC4cUO-3s0n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Load in a model from the SavedModel format\n",
        "loaded_model_6_SavedModel_format = tf.keras.models.load_model(\"model_6_SavedModel_format\")"
      ],
      "metadata": {
        "id": "1wrOn3r931_D"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate model in SavedModel format\n",
        "loaded_model_6_SavedModel_format.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x5rti65s4N4N",
        "outputId": "c3723746-189a-4de1-92c5-4fd37a8168c0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 2s 31ms/step - loss: 0.4512 - accuracy: 0.8281\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.45119187235832214, 0.8280839920043945]"
            ]
          },
          "metadata": {},
          "execution_count": 154
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C4xKEQIV4W1q",
        "outputId": "8b8b1036-331b-4b04-edd8-770fd1dc9700"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " USE (KerasLayer)            (None, 512)               256797824 \n",
            "                                                                 \n",
            " dense_5 (Dense)             (None, 128)               65664     \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 64)                8256      \n",
            "                                                                 \n",
            " output_layer (Dense)        (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 256871809 (979.89 MB)\n",
            "Trainable params: 73985 (289.00 KB)\n",
            "Non-trainable params: 256797824 (979.61 MB)\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Finding the most wrong examples"
      ],
      "metadata": {
        "id": "kfaOyLWM4yTT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download a pretrained model from Google Storage\n",
        "!wget https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
        "!unzip 08_model_6_USE_feature_extractor.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b5q4TF8a6o9O",
        "outputId": "a07068a3-0203-4f2c-c116-a4364fb7b63b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-02-20 04:48:25--  https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.141.207, 142.251.2.207, 142.250.101.207, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.141.207|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 960779165 (916M) [application/zip]\n",
            "Saving to: ‘08_model_6_USE_feature_extractor.zip’\n",
            "\n",
            "08_model_6_USE_feat 100%[===================>] 916.27M  83.3MB/s    in 11s     \n",
            "\n",
            "2024-02-20 04:48:36 (82.1 MB/s) - ‘08_model_6_USE_feature_extractor.zip’ saved [960779165/960779165]\n",
            "\n",
            "Archive:  08_model_6_USE_feature_extractor.zip\n",
            "   creating: 08_model_6_USE_feature_extractor/\n",
            "   creating: 08_model_6_USE_feature_extractor/assets/\n",
            "   creating: 08_model_6_USE_feature_extractor/variables/\n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.data-00000-of-00001  \n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.index  \n",
            "  inflating: 08_model_6_USE_feature_extractor/saved_model.pb  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Import previously trained model\n",
        "model_6_pretrained = tf.keras.models.load_model(\"08_model_6_USE_feature_extractor\")\n",
        "model_6_pretrained.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9BAaT4mV7Iie",
        "outputId": "7eacd515-5367-4be7-c54f-a386be753c3c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named \"keras_metadata.pb\" in the SavedModel directory.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 9s 18ms/step - loss: 0.4272 - accuracy: 0.8163\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.42723122239112854, 0.8162729740142822]"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions using the pretrained model\n",
        "model_6_pretrained_pred_probs = model_6_pretrained.predict(val_sentences)\n",
        "model_6_pretrained_preds = tf.squeeze(tf.round(model_6_pretrained_pred_probs))\n",
        "model_6_pretrained_preds[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v_02d13h7lNa",
        "outputId": "eddf1c98-2967-43e8-e2f0-90a7a38e29dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 9ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tf.squeeze(val_sentences).shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WqSvgSK69apc",
        "outputId": "b29d8b59-4292-408e-87f2-4ba157ec022d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([762])"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create DataFrame with validation sentences, labels and best performing model prediction labels + probabilities\n",
        "val_df = pd.DataFrame({\"text\" : val_sentences,\n",
        "                       \"target\" : val_labels,\n",
        "                       \"pred\" : model_6_pretrained_preds,\n",
        "                       \"pred_prob\" : tf.squeeze(model_6_pretrained_pred_probs)})"
      ],
      "metadata": {
        "id": "YjLA1aRV45BO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "val_df.head()"
      ],
      "metadata": {
        "id": "uZfXRkcE5qeE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "19bc5b0d-dffd-4be5-864a-2decfa68a29c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text  target  pred  pred_prob\n",
              "0  DFR EP016 Monthly Meltdown - On Dnbheaven 2015...       0   0.0   0.159757\n",
              "1  FedEx no longer to transport bioterror germs i...       0   1.0   0.747162\n",
              "2  Gunmen kill four in El Salvador bus attack: Su...       1   1.0   0.988749\n",
              "3  @camilacabello97 Internally and externally scr...       1   0.0   0.196229\n",
              "4  Radiation emergency #preparedness starts with ...       1   1.0   0.707808"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-929393bf-b9ac-48ab-9942-d4307d0e9f12\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>DFR EP016 Monthly Meltdown - On Dnbheaven 2015...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.159757</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>FedEx no longer to transport bioterror germs i...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.747162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Gunmen kill four in El Salvador bus attack: Su...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.988749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@camilacabello97 Internally and externally scr...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.196229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Radiation emergency #preparedness starts with ...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.707808</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-929393bf-b9ac-48ab-9942-d4307d0e9f12')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-929393bf-b9ac-48ab-9942-d4307d0e9f12 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-929393bf-b9ac-48ab-9942-d4307d0e9f12');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7ce214d0-b2dc-408d-a895-72401322935d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7ce214d0-b2dc-408d-a895-72401322935d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7ce214d0-b2dc-408d-a895-72401322935d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "val_df",
              "summary": "{\n  \"name\": \"val_df\",\n  \"rows\": 762,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"collapsed the moment i got home last night lol\",\n          \"Reality Training: Train falls off elevated tracks during windstorm http://t.co/qzRciNaF5z\",\n          \"Storm headed towards Idaho Falls with blowing dust &amp; winds to 60 mph.  US HWY 20 &amp; I15 look out.  #idwx http://t.co/0cR74m1Uxm\"\n        ],\n        \"num_unique_values\": 761,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          1.0,\n          0.0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred_prob\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          0.11760178953409195,\n          0.948488175868988\n        ],\n        \"num_unique_values\": 762,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "most_wrong = val_df[val_df[\"pred\"] != val_df[\"target\"]].sort_values(by = \"pred_prob\", ascending = False)\n",
        "most_wrong[:10]"
      ],
      "metadata": {
        "id": "vr2xJ2La-Ahv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "e36a01ee-e5cd-4338-f614-d476b48ba2c9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  text  target  pred  \\\n",
              "31   ? High Skies - Burning Buildings ? http://t.co...       0   1.0   \n",
              "759  FedEx will no longer transport bioterror patho...       0   1.0   \n",
              "628  @noah_anyname That's where the concentration c...       0   1.0   \n",
              "209  Ashes 2015: AustraliaÛªs collapse at Trent Br...       0   1.0   \n",
              "251  @AshGhebranious civil rights continued in the ...       0   1.0   \n",
              "393  @SonofLiberty357 all illuminated by the bright...       0   1.0   \n",
              "109  [55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...       0   1.0   \n",
              "49   @madonnamking RSPCA site multiple 7 story high...       0   1.0   \n",
              "119  @freefromwolves GodsLove &amp; #thankU brother...       0   1.0   \n",
              "344  Air Group is here to the rescue! We have 24/7 ...       0   1.0   \n",
              "\n",
              "     pred_prob  \n",
              "31    0.910196  \n",
              "759   0.876982  \n",
              "628   0.852300  \n",
              "209   0.835454  \n",
              "251   0.827213  \n",
              "393   0.814816  \n",
              "109   0.810840  \n",
              "49    0.803122  \n",
              "119   0.766901  \n",
              "344   0.766625  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-18c8284f-f6d4-4396-81fa-3a562435bb81\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>? High Skies - Burning Buildings ? http://t.co...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.910196</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>759</th>\n",
              "      <td>FedEx will no longer transport bioterror patho...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.876982</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>@noah_anyname That's where the concentration c...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.852300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>209</th>\n",
              "      <td>Ashes 2015: AustraliaÛªs collapse at Trent Br...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.835454</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251</th>\n",
              "      <td>@AshGhebranious civil rights continued in the ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.827213</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>393</th>\n",
              "      <td>@SonofLiberty357 all illuminated by the bright...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.814816</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.810840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>@madonnamking RSPCA site multiple 7 story high...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.803122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>119</th>\n",
              "      <td>@freefromwolves GodsLove &amp;amp; #thankU brother...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.766901</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>344</th>\n",
              "      <td>Air Group is here to the rescue! We have 24/7 ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.766625</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-18c8284f-f6d4-4396-81fa-3a562435bb81')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-18c8284f-f6d4-4396-81fa-3a562435bb81 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-18c8284f-f6d4-4396-81fa-3a562435bb81');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f241e1a3-2266-4c67-b002-877b5f3bc6ba\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f241e1a3-2266-4c67-b002-877b5f3bc6ba')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f241e1a3-2266-4c67-b002-877b5f3bc6ba button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"most_wrong[:10]\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"@freefromwolves GodsLove &amp; #thankU brother Danny for RT of NEW VIDEO http://t.co/cybKsXHF7d The Coming Apocalyptic US Earthquake &amp; Tsunami\",\n          \"FedEx will no longer transport bioterror pathogens in wake of anthrax lab mishaps http://t.co/lHpgxc4b8J\",\n          \"@SonofLiberty357 all illuminated by the brightly burning buildings all around the town!\"\n        ],\n        \"num_unique_values\": 10,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"samples\": [\n          0\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          1.0\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred_prob\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          0.766900897026062\n        ],\n        \"num_unique_values\": 10,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "most_wrong.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 259
        },
        "id": "bnZcbS1G4bIo",
        "outputId": "c861cace-9996-4ace-87b9-e55daf63bb28"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  text  target  pred  \\\n",
              "411  @SoonerMagic_ I mean I'm a fan but I don't nee...       1   0.0   \n",
              "233                    I get to smoke my shit in peace       1   0.0   \n",
              "38   Why are you deluged with low self-image? Take ...       1   0.0   \n",
              "244  Reddit Will Now QuarantineÛ_ http://t.co/pkUA...       1   0.0   \n",
              "23   Ron &amp; Fez - Dave's High School Crush https...       1   0.0   \n",
              "\n",
              "     pred_prob  \n",
              "411   0.043919  \n",
              "233   0.042087  \n",
              "38    0.038998  \n",
              "244   0.038949  \n",
              "23    0.037186  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-40b50c3a-9312-45ae-9b12-2bec8acfe6cf\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>411</th>\n",
              "      <td>@SoonerMagic_ I mean I'm a fan but I don't nee...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.043919</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>233</th>\n",
              "      <td>I get to smoke my shit in peace</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.042087</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Why are you deluged with low self-image? Take ...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>244</th>\n",
              "      <td>Reddit Will Now QuarantineÛ_ http://t.co/pkUA...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038949</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Ron &amp;amp; Fez - Dave's High School Crush https...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.037186</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-40b50c3a-9312-45ae-9b12-2bec8acfe6cf')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-40b50c3a-9312-45ae-9b12-2bec8acfe6cf button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-40b50c3a-9312-45ae-9b12-2bec8acfe6cf');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-143df083-b57e-4c99-b3c4-75351c7ef90c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-143df083-b57e-4c99-b3c4-75351c7ef90c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-143df083-b57e-4c99-b3c4-75351c7ef90c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"most_wrong\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"I get to smoke my shit in peace\",\n          \"Ron &amp; Fez - Dave's High School Crush https://t.co/aN3W16c8F6 via @YouTube\",\n          \"Why are you deluged with low self-image? Take the quiz: http://t.co/XsPqdOrIqj http://t.co/CQYvFR4UCy\"\n        ],\n        \"num_unique_values\": 5,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          0.0\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pred_prob\",\n      \"properties\": {\n        \"dtype\": \"float32\",\n        \"samples\": [\n          0.042086850851774216\n        ],\n        \"num_unique_values\": 5,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualizing all the false positives\n",
        "for row in most_wrong[:10].itertuples():\n",
        "  _, txt, targ, predic, pred_proba = row\n",
        "  print(f\"Target: {targ}, Prediction: {predic}, Prediction Probability: {pred_proba}\")\n",
        "  print(f\"Text: \\n {txt}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ia-dMOAP48vW",
        "outputId": "16bc1078-e69d-4a61-814b-05402d87f946"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.9101957678794861\n",
            "Text: \n",
            " ? High Skies - Burning Buildings ? http://t.co/uVq41i3Kx2 #nowplaying\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8769821524620056\n",
            "Text: \n",
            " FedEx will no longer transport bioterror pathogens in wake of anthrax lab mishaps http://t.co/lHpgxc4b8J\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8523001074790955\n",
            "Text: \n",
            " @noah_anyname That's where the concentration camps and mass murder come in. \n",
            " \n",
            "EVERY. FUCKING. TIME.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8354543447494507\n",
            "Text: \n",
            " Ashes 2015: AustraliaÛªs collapse at Trent Bridge among worst in history: England bundled out Australia for 60 ... http://t.co/t5TrhjUAU0\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8272132873535156\n",
            "Text: \n",
            " @AshGhebranious civil rights continued in the 60s. And what about trans-generational trauma? if anything we should listen to the Americans.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.814815878868103\n",
            "Text: \n",
            " @SonofLiberty357 all illuminated by the brightly burning buildings all around the town!\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8108396530151367\n",
            "Text: \n",
            " [55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES WITH MAGNE-TRACTION INSTRUCTIONS http://t.co/xEZBs3sq0y http://t.co/C2x0QoKGlY\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.8031218647956848\n",
            "Text: \n",
            " @madonnamking RSPCA site multiple 7 story high rise buildings next to low density character residential in an area that floods\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.766900897026062\n",
            "Text: \n",
            " @freefromwolves GodsLove &amp; #thankU brother Danny for RT of NEW VIDEO http://t.co/cybKsXHF7d The Coming Apocalyptic US Earthquake &amp; Tsunami\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Prediction: 1.0, Prediction Probability: 0.7666252255439758\n",
            "Text: \n",
            " Air Group is here to the rescue! We have 24/7 Emergency Service! Learn more about it here - http://t.co/9lyx7zMtHE http://t.co/5PbC96rTMJ\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualizing all the false negatives\n",
        "for row in most_wrong[-10:].itertuples():\n",
        "  _, txt, targ, predic, pred_proba = row\n",
        "  print(f\"Target: {targ}, Prediction: {predic}, Prediction Probability: {pred_proba}\")\n",
        "  print(f\"Text: \\n {txt}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VZXd6a6N6Cmo",
        "outputId": "55431da0-8a5c-42bf-c2b5-98b96f30f742"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.06730346381664276\n",
            "Text: \n",
            " @DavidVonderhaar At least you were sincere ??\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.05507583171129227\n",
            "Text: \n",
            " @willienelson We need help! Horses will die!Please RT &amp; sign petition!Take a stand &amp; be a voice for them! #gilbert23 https://t.co/e8dl1lNCVu\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.05460337549448013\n",
            "Text: \n",
            " Lucas Duda is Ghost Rider. Not the Nic Cage version but an actual 'engulfed in flames' badass. #Mets\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.05459701269865036\n",
            "Text: \n",
            " going to redo my nails and watch behind the scenes of desolation of smaug ayyy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.04963727295398712\n",
            "Text: \n",
            " You can never escape me. Bullets don't harm me. Nothing harms me. But I know pain. I know pain. Sometimes I share it. With someone like you.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.04391850158572197\n",
            "Text: \n",
            " @SoonerMagic_ I mean I'm a fan but I don't need a girl sounding off like a damn siren\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.042086850851774216\n",
            "Text: \n",
            " I get to smoke my shit in peace\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.03899793699383736\n",
            "Text: \n",
            " Why are you deluged with low self-image? Take the quiz: http://t.co/XsPqdOrIqj http://t.co/CQYvFR4UCy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.03894945606589317\n",
            "Text: \n",
            " Reddit Will Now QuarantineÛ_ http://t.co/pkUAMXw6pm #onlinecommunities #reddit #amageddon #freespeech #Business http://t.co/PAWvNJ4sAP\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Prediction: 0.0, Prediction Probability: 0.03718579187989235\n",
            "Text: \n",
            " Ron &amp; Fez - Dave's High School Crush https://t.co/aN3W16c8F6 via @YouTube\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "test_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 774
        },
        "id": "XvOgVRHF64fX",
        "outputId": "b953cc55-8437-4c03-d286-44a6f6403ff6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         id keyword location  \\\n",
              "0         0     NaN      NaN   \n",
              "1         2     NaN      NaN   \n",
              "2         3     NaN      NaN   \n",
              "3         9     NaN      NaN   \n",
              "4        11     NaN      NaN   \n",
              "...     ...     ...      ...   \n",
              "3258  10861     NaN      NaN   \n",
              "3259  10865     NaN      NaN   \n",
              "3260  10868     NaN      NaN   \n",
              "3261  10874     NaN      NaN   \n",
              "3262  10875     NaN      NaN   \n",
              "\n",
              "                                                   text  \n",
              "0                    Just happened a terrible car crash  \n",
              "1     Heard about #earthquake is different cities, s...  \n",
              "2     there is a forest fire at spot pond, geese are...  \n",
              "3              Apocalypse lighting. #Spokane #wildfires  \n",
              "4         Typhoon Soudelor kills 28 in China and Taiwan  \n",
              "...                                                 ...  \n",
              "3258  EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...  \n",
              "3259  Storm in RI worse than last hurricane. My city...  \n",
              "3260  Green Line derailment in Chicago http://t.co/U...  \n",
              "3261  MEG issues Hazardous Weather Outlook (HWO) htt...  \n",
              "3262  #CityofCalgary has activated its Municipal Eme...  \n",
              "\n",
              "[3263 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1a75671e-1c50-448c-9d4a-06128f96bda7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3258</th>\n",
              "      <td>10861</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3259</th>\n",
              "      <td>10865</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Storm in RI worse than last hurricane. My city...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3260</th>\n",
              "      <td>10868</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Green Line derailment in Chicago http://t.co/U...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3261</th>\n",
              "      <td>10874</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>MEG issues Hazardous Weather Outlook (HWO) htt...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3262</th>\n",
              "      <td>10875</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>#CityofCalgary has activated its Municipal Eme...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3263 rows × 4 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1a75671e-1c50-448c-9d4a-06128f96bda7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1a75671e-1c50-448c-9d4a-06128f96bda7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1a75671e-1c50-448c-9d4a-06128f96bda7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ce51a2b7-0ec3-4535-a0c3-42669fedc774\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ce51a2b7-0ec3-4535-a0c3-42669fedc774')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ce51a2b7-0ec3-4535-a0c3-42669fedc774 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_3b5db096-5dc5-4d9f-bd1a-61dd80bb738d\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('test_df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_3b5db096-5dc5-4d9f-bd1a-61dd80bb738d button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('test_df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "test_df",
              "summary": "{\n  \"name\": \"test_df\",\n  \"rows\": 3263,\n  \"fields\": [\n    {\n      \"column\": \"id\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3146,\n        \"min\": 0,\n        \"max\": 10875,\n        \"samples\": [\n          8051,\n          425,\n          1330\n        ],\n        \"num_unique_values\": 3263,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"keyword\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"injury\",\n          \"nuclear%20reactor\",\n          \"engulfed\"\n        ],\n        \"num_unique_values\": 221,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"samples\": [\n          \"UAE\",\n          \"Tokio / Tokyo\",\n          \"Texas\"\n        ],\n        \"num_unique_values\": 1602,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"text\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"samples\": [\n          \"Latest: USA: Huge sinkhole swallows up Brooklyn intersection http://t.co/vspKHg3nZy\",\n          \"I liked a @YouTube video http://t.co/a5YTAw9Vih S.O.S. Rona Guide - The Red Whirlwind\",\n          \"HitchBot travels Europe and greeted with open arms. Gets destroyed after two weeks in america. There's a lesson to be learned here.\"\n        ],\n        \"num_unique_values\": 3243,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Making predictions on the test dataset"
      ],
      "metadata": {
        "id": "KysR3NO47urS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_sentences = test_df[\"text\"].to_list()\n",
        "test_samples = random.sample(test_sentences, 10)\n",
        "\n",
        "for test_sample in test_samples:\n",
        "  pred_proba = tf.squeeze(model_6_pretrained.predict([test_sample]))\n",
        "  predic = tf.round(pred_proba)\n",
        "  print(f\"Pred: {int(predic)}, Prob: {pred_proba}\")\n",
        "  print(f\"Text: \\n{test_sample}\")\n",
        "  print(\"---\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tf1_MBA38zPs",
        "outputId": "10e4b6cd-8c4b-49d0-be41-d247ba21ad9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 33ms/step\n",
            "Pred: 1, Prob: 0.9352068305015564\n",
            "Text: \n",
            "India: Militant 'Overpowered' By Hostages After Attack http://t.co/mkjszGbaWR\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 36ms/step\n",
            "Pred: 1, Prob: 0.9686601758003235\n",
            "Text: \n",
            "Rare photographs show the nightmare aftermath of #Hiroshima | #NoNukes #Amerikkka #WhiteTerrorism #Nuclear #Disaster  http://t.co/8tWLAKdaBf\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 34ms/step\n",
            "Pred: 0, Prob: 0.4745257794857025\n",
            "Text: \n",
            "#LocalEventCountdown:\n",
            "8. Curtain opens on The Black Cyclone in Shelby tickets &amp; info: http://t.co/v7dCz5DJDj #rstoday\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 33ms/step\n",
            "Pred: 0, Prob: 0.46143704652786255\n",
            "Text: \n",
            "Enter the world of extreme diving ÛÓ 9 stories up and into the Volga River http://t.co/gptysALfKi\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 33ms/step\n",
            "Pred: 0, Prob: 0.05172423645853996\n",
            "Text: \n",
            "Mental health is real..?? Self harm is real..??Depression is real..?? feeling alone..?? no where to fit in..?? one day it could be you???\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 33ms/step\n",
            "Pred: 0, Prob: 0.07006745785474777\n",
            "Text: \n",
            "@MeekMill I think its time you consulted @kendricklamar cause @Drake done body bagging you\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 35ms/step\n",
            "Pred: 0, Prob: 0.29719728231430054\n",
            "Text: \n",
            "@MWCahill5 also mcutchen crushed it\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 34ms/step\n",
            "Pred: 1, Prob: 0.9084580540657043\n",
            "Text: \n",
            "Hailstorm cleanup begins in Calgary http://t.co/DhH5jz2w49\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 35ms/step\n",
            "Pred: 0, Prob: 0.07738707959651947\n",
            "Text: \n",
            "so privileged and proud to wear this uniform.?? #NHS #Ambulance #GayUK #Uniform #Proud #Privileged #WhatsYourEmergency http://t.co/0BkmuhYSFx\n",
            "---\n",
            "\n",
            "1/1 [==============================] - 0s 36ms/step\n",
            "Pred: 0, Prob: 0.027656061574816704\n",
            "Text: \n",
            "@TopherBreezy it was my fav too!!! Amazing! Michael and I screamed the whole time ??\n",
            "---\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The Speed/Score tradeoff"
      ],
      "metadata": {
        "id": "-g0Dr3gM87xd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Making a fucntion to measure the time of prediction\n",
        "import time\n",
        "def pred_timer(model,samples):\n",
        "  \"\"\"\n",
        "    Times how long a model takes to make predictions on a sample.\n",
        "  \"\"\"\n",
        "  start_time = time.perf_counter()\n",
        "  model.predict(samples)\n",
        "  end_time = time.perf_counter()\n",
        "  total_time = end_time - start_time\n",
        "  time_per_prediction = total_time / len(samples)\n",
        "\n",
        "  return total_time, time_per_prediction"
      ],
      "metadata": {
        "id": "iO9BLOVR-ZgA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate TF Hub Sentence Encoder time per pred\n",
        "model_6_total_time, model_6_time_per_pred = pred_timer(model_6_pretrained, val_sentences)\n",
        "model_6_total_time, model_6_time_per_pred"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BKC3aAM4_NPS",
        "outputId": "b0add61e-3dea-45c5-c9d4-37df9e869bf7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 8ms/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.3570796399999381, 0.0004686084514434883)"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate thwe baseline model's time per pred\n",
        "baseline_total_time, baseline_time_per_pred = pred_timer(model_0, val_sentences)\n",
        "baseline_total_time, baseline_time_per_pred"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L4AaCCXF_dmw",
        "outputId": "463e55bf-1536-4b93-f13d-4a7a6810fd3e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.01549743799978387, 2.0337845144073318e-05)"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_pretrained_results = calculate_results(val_labels, model_6_pretrained_preds)\n",
        "model_6_pretrained_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Njau1LeJ_y_N",
        "outputId": "16693ac7-d5fd-4f68-b139-f58993965dc8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Accuracy': 81.62729658792651,\n",
              " 'Precision': 0.818446310697231,\n",
              " 'Recall': 0.8162729658792651,\n",
              " 'F1 score': 0.8148082644367335}"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize = (10,7))\n",
        "plt.scatter(baseline_time_per_pred, baseline_results['F1 score'], label = 'baseline')\n",
        "plt.scatter(model_6_time_per_pred, model_6_pretrained_results['F1 score'], label = 'tf_hub_sentence_encoder')\n",
        "plt.legend()\n",
        "plt.title(\"F1-score Vs time per prediction\")\n",
        "plt.xlabel('Time per prediction')\n",
        "plt.ylabel('F1-score');"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 641
        },
        "id": "RaCqjJ6GA6r6",
        "outputId": "d0530c9e-d1cc-44ab-ad2d-df047bacd249"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA18AAAJwCAYAAAB2y9s+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABin0lEQVR4nO3de3zPdeP/8ednmx3tgM0OGhNyKCbTFjmVZQ4toggxckilsA6szDGUq6Sry6GrhK4Sl1I6SMqhkMhhJCxEDtfMKRtzGNvr94efz7ePHcxs7w2P++32vvF5vV/v1+G9t+vas/f7/frYjDFGAAAAAIBi5VTSAwAAAACAmwHhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAHDTGDVqlGw2W0kPA5fZu3evbDabZs2aZS8r6p/VihUrZLPZtGLFiiJrEwCuFuELAK7BrFmzZLPZct2GDRtmr7dkyRL16dNHd9xxh5ydnRUWFlZygy4hDz74oDw9PXXy5Mk863Tv3l2urq46duxYofs5ffq0Ro0axS/ZN6mpU6c6hDgAKE1cSnoAAHAjGDNmjKpWrepQdscdd9j/PmfOHM2bN08NGjRQSEiI1cMrFbp3764vv/xSn332mXr27Jlj/+nTp7Vw4UK1bt1aFSpUKHQ/p0+f1ujRoyVJLVq0cNg3fPhwh1CM0quwP6upU6fK399fvXr1cihv1qyZzpw5I1dX1yIaIQBcPcIXABSBNm3aqGHDhnnuHz9+vN59912VKVNGDzzwgLZu3Wrh6IpGRkaGvLy8Cn38gw8+KG9vb82ZMyfX8LVw4UJlZGSoe/fu1zLMfLm4uMjF5eb7v75r/dnlxRijs2fPysPDo8jbLuqflZOTk9zd3YusPQAoDB47BAALhISEqEyZMoU+/uTJkxo8eLDCwsLk5uamihUr6v7779fGjRsd6q1du1Zt27ZVuXLl5OXlpXr16umtt95yqLNs2TI1bdpUXl5e8vPzU/v27bV9+3aHOpfet9m2bZu6deumcuXKqUmTJvb9H374oSIiIuTh4aHy5cvr0Ucf1f79+/Odg4eHhzp27KilS5fq8OHDOfbPmTNH3t7eevDBB+1lc+fOVUREhLy9veXj46O6devmmM/f7d27VwEBAZKk0aNH2x8BHTVqlMO8/s5ms2ngwIGaP3++6tSpIw8PDzVq1Ei//vqrJOmdd95R9erV5e7urhYtWmjv3r05+l27dq1at24tX19feXp6qnnz5lq9enW+50P6v/eQ5s2bp5deeklBQUHy8vLSgw8+mOv5LEg/V/rZXe7So7M//vijnnjiCVWoUEE+Pj7q2bOn/vrrL4e6YWFheuCBB/Ttt9+qYcOG8vDw0DvvvCNJOnHihAYPHqzQ0FC5ubmpevXqeu2115Sdne3QxokTJ9SrVy/5+vrKz89PcXFxOnHiRI5x5fXO14cffqjIyEh5enqqXLlyatasmZYsWWIf32+//aYffvjB/rO/dPczr3e+5s+fb7+W/f399dhjj+ngwYMOdXr16qWyZcvq4MGD6tChg8qWLauAgAA9//zzysrKyvPcAsDlbr7//AcAxSAtLU1Hjx51KPP39y+y9gcMGKBPPvlEAwcOVJ06dXTs2DGtWrVK27dvV4MGDSRJ3333nR544AEFBwdr0KBBCgoK0vbt2/XVV19p0KBBkqTvv/9ebdq00a233qpRo0bpzJkzevvtt3XPPfdo48aNOd5Fe+SRR1SjRg2NHz9exhhJ0rhx45SYmKjOnTurb9++OnLkiN5++201a9ZMmzZtkp+fX57z6N69u2bPnq3//ve/GjhwoL38+PHj+vbbb9W1a1f7XZTvvvtOXbt2VcuWLfXaa69JkrZv367Vq1fb53O5gIAATZs2TU8++aQeeughdezYUZJUr169fM/vypUr9cUXX+jpp5+WJE2YMEEPPPCAXnzxRU2dOlVPPfWU/vrrL02cOFGPP/64li1bZj922bJlatOmjSIiIjRy5Eg5OTlp5syZuu+++7Ry5UpFRkbm2/elc2qz2TR06FAdPnxYkydPVnR0tJKSkuzn42r7ye1nl5+BAwfKz89Po0aNUnJysqZNm6Y///zTHlouSU5OVteuXfXEE0+oX79+qlmzpk6fPq3mzZvr4MGDeuKJJ1S5cmX99NNPSkhIUEpKiiZPnizp4p2y9u3ba9WqVRowYIBq166tzz77THFxcVccn3QxUI8aNUqNGzfWmDFj5OrqqrVr12rZsmVq1aqVJk+erGeeeUZly5bVyy+/LEkKDAzMs71Zs2apd+/euuuuuzRhwgSlpqbqrbfe0urVq3Ncy1lZWYqJiVFUVJRef/11ff/993rjjTdUrVo1PfnkkwUaPwDIAAAKbebMmUZSrlte2rVrZ6pUqXJV/fj6+pqnn346z/0XLlwwVatWNVWqVDF//fWXw77s7Gz73+vXr28qVqxojh07Zi/bvHmzcXJyMj179rSXjRw50kgyXbt2dWhr7969xtnZ2YwbN86h/NdffzUuLi45ynMbZ3BwsGnUqJFD+fTp040k8+2339rLBg0aZHx8fMyFCxfybfNyR44cMZLMyJEjc+y7NK+/k2Tc3NzMnj177GXvvPOOkWSCgoJMenq6vTwhIcFIstfNzs42NWrUMDExMQ7n+fTp06Zq1arm/vvvz3esy5cvN5JMpUqVHPr573//aySZt95666r7yetnl5dL13BERITJzMy0l0+cONFIMgsXLrSXValSxUgyixcvdmhj7NixxsvLy/z+++8O5cOGDTPOzs5m3759xhhjPv/8cyPJTJw40V7nwoULpmnTpkaSmTlzZo55XLJz507j5ORkHnroIZOVleXQz9/Pye23326aN2+eY56XzvXy5cuNMcZkZmaaihUrmjvuuMOcOXPGXu+rr74yksyIESPsZXFxcUaSGTNmjEObd955p4mIiMjRFwDkhccOAaAITJkyRd99953DVpT8/Py0du1a/e9//8t1/6ZNm7Rnzx4NHjw4x52nS3ctUlJSlJSUpF69eql8+fL2/fXq1dP999+vRYsW5Wh3wIABDp8XLFig7Oxsde7cWUePHrVvQUFBqlGjhpYvX57vPJydnfXoo49qzZo1Do/vzZkzR4GBgWrZsqXDnDMyMor8XOamZcuWDnf9oqKiJEmdOnWSt7d3jvI//vhDkpSUlKSdO3eqW7duOnbsmP18ZGRkqGXLlvrxxx9zPHaXm549ezr08/DDDys4ONj+MylMP5f/7K6kf//+Do/GPvnkk3JxcclxXVStWlUxMTEOZfPnz1fTpk1Vrlw5h+siOjpaWVlZ+vHHHyVJixYtkouLi8OdImdnZz3zzDNXHN/nn3+u7OxsjRgxQk5Ojr++FGZJ+vXr1+vw4cN66qmnHN4Fa9eunWrVqqWvv/46xzGXn9OmTZvarwUAKAgeOwSAIhAZGZnvghsFkZWVpSNHjjiUlS9fXq6urpo4caLi4uIUGhqqiIgItW3bVj179tStt94qSdq9e7ckxxUWL/fnn39KkmrWrJljX+3atfXtt9/mWJjh8hUcd+7cKWOMatSokWsfBXmvrXv37nrzzTc1Z84cvfTSSzpw4IBWrlypZ599Vs7OzvZ6Tz31lP773/+qTZs2qlSpklq1aqXOnTurdevWV+zjalWuXNnhs6+vryQpNDQ01/JL70Lt3LlTkvJ9bC4tLU3lypXLt//Lz6fNZlP16tXtAbUw/Vz+s7uSy8dQtmxZBQcH53jHLbd2d+7cqS1bttjft7vcpXf8/vzzTwUHB6ts2bIO+3O7Ji+3e/duOTk5qU6dOlesWxD5/XuoVauWVq1a5VDm7u6eY37lypXL8V4cAOSH8AUApcT+/ftz/GK7fPlytWjRQp07d1bTpk312WefacmSJfrHP/6h1157TQsWLFCbNm2KbUyXr2KXnZ0tm82mb775xiEoXXL5L9W5iYiIUK1atfTxxx/rpZde0scffyxjTI5VDitWrKikpCR9++23+uabb/TNN99o5syZ6tmzp2bPnn1tE7tMbnPJr9z8/3eoLt1t+sc//qH69evnWrcg5+RKCtNPcaxAmFe72dnZuv/++/Xiiy/mesxtt91WLGOxUl7XAgBcDcIXAJQSQUFBOR6xCw8Pt/89ODhYTz31lJ566ikdPnxYDRo00Lhx49SmTRtVq1ZNkrR161ZFR0fn2n6VKlUkXVww4XI7duyQv7//FZcjr1atmowxqlq16jX9Qt29e3clJiZqy5YtmjNnjmrUqKG77rorRz1XV1fFxsYqNjZW2dnZeuqpp/TOO+8oMTFR1atXz7XtwjyCVliXzruPj0+e570gLt3ZusQYo127dtkXCimqfq40hnvvvdf++dSpU0pJSVHbtm2veGy1atV06tSpK46tSpUqWrp0qU6dOuUQFnO7JnPrIzs7W9u2bcszgEoF//n//d/Dfffd57AvOTnZvh8AihLvfAFAKeHu7q7o6GiHrVy5csrKylJaWppD3YoVKyokJETnzp2TJDVo0EBVq1bV5MmTcyzbfekuTXBwsOrXr6/Zs2c71Nm6dauWLFlSoF+yO3bsKGdnZ40ePTrHCnrGGB07dqxAc710l2vEiBFKSkrK9bu9Lm/LycnJHkYuzTs3np6ekpTr8uVFLSIiQtWqVdPrr7+uU6dO5dh/+WOkefnggw908uRJ++dPPvlEKSkp9ruaRdVPfv7973/r/Pnz9s/Tpk3ThQsXCnRntXPnzlqzZo2+/fbbHPtOnDihCxcuSJLatm2rCxcuaNq0afb9WVlZevvtt6/YR4cOHeTk5KQxY8bkeL/t79eil5dXgX72DRs2VMWKFTV9+nSH6+mbb77R9u3b1a5duyu2AQBXiztfAGCBLVu26IsvvpAk7dq1S2lpaXrllVckXby7FRsbm+exJ0+e1C233KKHH35Y4eHhKlu2rL7//nv98ssveuONNyRdDCbTpk1TbGys6tevr969eys4OFg7duzQb7/9Zv+l+B//+IfatGmjRo0aqU+fPval5n19fe3fhZWfatWq6ZVXXlFCQoL27t2rDh06yNvbW3v27NFnn32m/v376/nnn79iO1WrVlXjxo21cOFCSco1fPXt21fHjx/Xfffdp1tuuUV//vmn3n77bdWvX1+1a9fOs20PDw/VqVNH8+bN02233aby5cvrjjvuyPd9uMJycnLSe++9pzZt2uj2229X7969ValSJR08eFDLly+Xj4+Pvvzyyyu2U758eTVp0kS9e/dWamqqJk+erOrVq6tfv35F2k9+MjMz1bJlS3Xu3FnJycmaOnWqmjRp4vC9a3l54YUX9MUXX+iBBx5Qr169FBERoYyMDP3666/65JNPtHfvXvn7+ys2Nlb33HOPhg0bpr1796pOnTpasGBBjv+4kJvq1avr5Zdf1tixY9W0aVN17NhRbm5u+uWXXxQSEqIJEyZIuhhUp02bpldeeUXVq1dXxYoVc9zZki6+n/jaa6+pd+/eat68ubp27Wpfaj4sLExDhgy5+pMIAFdSYussAsAN4NIy3b/88kuB6uW2xcXF5XvsuXPnzAsvvGDCw8ONt7e38fLyMuHh4Wbq1Kk56q5atcrcf//99nr16tUzb7/9tkOd77//3txzzz3Gw8PD+Pj4mNjYWLNt2zaHOpeW+T5y5EiuY/r0009NkyZNjJeXl/Hy8jK1atUyTz/9tElOTs53Ln83ZcoUI8lERkbmuv+TTz4xrVq1MhUrVjSurq6mcuXK5oknnjApKSlXbPunn34yERERxtXV1WHZ+byWmr98Gf89e/YYSeYf//iHQ/ml5crnz5/vUL5p0ybTsWNHU6FCBePm5maqVKliOnfubJYuXZrvOC+19/HHH5uEhARTsWJF4+HhYdq1a2f+/PPPHPUL0s+VfnaXu3Rt/vDDD6Z///6mXLlypmzZsqZ79+4OX0lgzMWl5tu1a5drOydPnjQJCQmmevXqxtXV1fj7+5vGjRub119/3WEJ+2PHjpkePXoYHx8f4+vra3r06GE2bdp0xaXmL3n//ffNnXfeadzc3Ey5cuVM8+bNzXfffWfff+jQIdOuXTvj7e1tJNmXnb98qflL5s2bZ2+vfPnypnv37ubAgQMOdeLi4oyXl1eOseQ1RgDIi82YAnzzIgAAKHIrVqzQvffeq/nz5+vhhx8ukTFc+qLhX3755ZpX7AQA5I93vgAAAADAAoQvAAAAALAA4QsAAAAALMA7XwAAAABgAe58AQAAAIAFCF8AAAAAYAG+ZLmQsrOz9b///U/e3t6y2WwlPRwAAAAAJcQYo5MnTyokJEROTnnf3yJ8FdL//vc/hYaGlvQwAAAAAJQS+/fv1y233JLnfsJXIXl7e0u6eIJ9fHxKeDQAAAAASkp6erpCQ0PtGSEvhK9CuvSooY+PD+ELAAAAwBVfR2LBDQAAAACwAOELAAAAACxA+AIAAAAAC/DOVzEyxujChQvKysoq6aEAN6UyZcrI2dm5pIcBAAAgifBVbDIzM5WSkqLTp0+X9FCAm5bNZtMtt9yismXLlvRQAAAACF/FITs7W3v27JGzs7NCQkLk6urKFzEDFjPG6MiRIzpw4IBq1KjBHTAAAFDiCF/FIDMzU9nZ2QoNDZWnp2dJDwe4aQUEBGjv3r06f/484QsAAJQ4FtwoRk5OnF6gJHHHGQAAlCakAwAAAACwAOELAAAAACxA+IKDFi1aaPDgwSXWf69evdShQ4dSMx4AAACgqLDgBkq1BQsWqEyZMiU9DAAAAOCaEb5Ksaxso3V7juvwybOq6O2uyKrl5ex0cy0gUL58+ZIeAgAAAFAkeOywlFq8NUVNXlumru/+rEFzk9T13Z/V5LVlWrw1pdj7vnDhggYOHChfX1/5+/srMTFRxhhJ0n/+8x81bNhQ3t7eCgoKUrdu3XT48GH7sX/99Ze6d++ugIAAeXh4qEaNGpo5c6Z9//79+9W5c2f5+fmpfPnyat++vfbu3ZvnWC5/7DAsLEzjx4/X448/Lm9vb1WuXFn//ve/HY652j4AAAAAK5R4+JoyZYrCwsLk7u6uqKgorVu3Lt/6kydPVs2aNeXh4aHQ0FANGTJEZ8+ete//8ccfFRsbq5CQENlsNn3++ec52ujVq5dsNpvD1rp166KeWqEt3pqiJz/cqJS0sw7lh9LO6skPNxZ7AJs9e7ZcXFy0bt06vfXWW5o0aZLee+89SdL58+c1duxYbd68WZ9//rn27t2rXr162Y9NTEzUtm3b9M0332j79u2aNm2a/P397cfGxMTI29tbK1eu1OrVq1W2bFm1bt1amZmZBR7fG2+8oYYNG2rTpk166qmn9OSTTyo5OblI+wAAAACKWok+djhv3jzFx8dr+vTpioqK0uTJkxUTE6Pk5GRVrFgxR/05c+Zo2LBhev/999W4cWP9/vvv9iA1adIkSVJGRobCw8P1+OOPq2PHjnn23bp1a4c7Mm5ubkU/wULIyjYa/eU2mVz2GUk2SaO/3Kb76wQV2yOIoaGhevPNN2Wz2VSzZk39+uuvevPNN9WvXz89/vjj9nq33nqr/vnPf+quu+7SqVOnVLZsWe3bt0933nmnGjZsKOninapL5s2bp+zsbL333nv271+aOXOm/Pz8tGLFCrVq1apA42vbtq2eeuopSdLQoUP15ptvavny5apZs2aR9QEAAAAUtRINX5MmTVK/fv3Uu3dvSdL06dP19ddf6/3339ewYcNy1P/pp590zz33qFu3bpIu/mLftWtXrV271l6nTZs2atOmzRX7dnNzU1BQUBHNpOis23M8xx2vvzOSUtLOat2e42pUrUKxjOHuu+92+HLaRo0a6Y033lBWVpaSkpI0atQobd68WX/99Zeys7MlSfv27VOdOnX05JNPqlOnTtq4caNatWqlDh06qHHjxpKkzZs3a9euXfL29nbo7+zZs9q9e3eBx1evXj373202m4KCguyPPhZVHwAAACjFsrOkP3+STqVKZQOlKo0lJ+eSHtUVlVj4yszM1IYNG5SQkGAvc3JyUnR0tNasWZPrMY0bN9aHH36odevWKTIyUn/88YcWLVqkHj16XHX/K1asUMWKFVWuXDndd999euWVV1ShQt5h5ty5czp37pz9c3p6+lX3WRCHT+YdvApTryidPXtWMTExiomJ0UcffaSAgADt27dPMTEx9kf62rRpoz///FOLFi3Sd999p5YtW+rpp5/W66+/rlOnTikiIkIfffRRjrYDAgIKPI7LVz+02Wz2EFhUfQAAAKCU2vaFtHiolP6//yvzCZFavybVebDkxlUAJRa+jh49qqysLAUGBjqUBwYGaseOHbke061bNx09elRNmjSRMUYXLlzQgAED9NJLL11V361bt1bHjh1VtWpV7d69Wy+99JLatGmjNWvWyNk598Q8YcIEjR49+qr6KYyK3u5FWq8w/n4nUZJ+/vln1ahRQzt27NCxY8f06quvKjQ0VJK0fv36HMcHBAQoLi5OcXFxatq0qV544QW9/vrratCggebNm6eKFSvKx8enWMZuRR8AAAAoIdu+kP7bU7r8JZ30lIvlnT8o1QGsxBfcuBorVqzQ+PHjNXXqVG3cuFELFizQ119/rbFjx15VO48++qgefPBB1a1bVx06dNBXX32lX375RStWrMjzmISEBKWlpdm3/fv3X+NschdZtbyCfd2V19tcNknBvheXnS8u+/btU3x8vJKTk/Xxxx/r7bff1qBBg1S5cmW5urrq7bff1h9//KEvvvgix7kfMWKEFi5cqF27dum3337TV199pdq1a0uSunfvLn9/f7Vv314rV67Unj17tGLFCj377LM6cOBAkYzdij4AAABQArKzLt7xynN1BEmLh12sV0qVWPjy9/eXs7OzUlNTHcpTU1PzfBcrMTFRPXr0UN++fVW3bl099NBDGj9+vCZMmGB/7Kwwbr31Vvn7+2vXrl151nFzc5OPj4/DVhycnWwaGVtHknIEsEufR8bWKdbv++rZs6fOnDmjyMhIPf300xo0aJD69++vgIAAzZo1S/Pnz1edOnX06quv6vXXX3c41tXVVQkJCapXr56aNWsmZ2dnzZ07V5Lk6empH3/8UZUrV1bHjh1Vu3Zt9enTR2fPni2y82lFHwAAACgBf/7k+KhhDkZKP3ixXilVYo8durq6KiIiQkuXLlWHDh0kSdnZ2Vq6dKkGDhyY6zGnT5+Wk5NjXrz0mOCl76EqjAMHDujYsWMKDg4udBtFqfUdwZr2WAON/nKbw+IbQb7uGhlbR63vKL5x/v3u37Rp03Ls79q1q7p27epQ9vdzP3z4cA0fPjzP9oOCgjR79uw898+aNSvP8UjK9fu6kpKSrqoPAAAAXIdOpV65ztXUKwElutphfHy84uLi1LBhQ0VGRmry5MnKyMiwr37Ys2dPVapUSRMmTJAkxcbGatKkSbrzzjsVFRWlXbt2KTExUbGxsfYQdurUKYc7WHv27FFSUpLKly+vypUr69SpUxo9erQ6deqkoKAg7d69Wy+++KKqV6+umJgY609CHlrfEaz76wRp3Z7jOnzyrCp6X3zUsDjveAEAAAClVtnAK9e5mnoloETDV5cuXXTkyBGNGDFChw4dUv369bV48WL7Ihz79u1zuNM1fPhw2Ww2DR8+XAcPHlRAQIBiY2M1btw4e53169fr3nvvtX+Oj4+XJMXFxWnWrFlydnbWli1bNHv2bJ04cUIhISFq1aqVxo4dW2q+6+sSZydbsS0nDwAAAFxXqjS+uKpheopyf+/LdnF/lcZWj6zAbOZante7iaWnp8vX11dpaWk53iU6e/as9uzZo6pVq8rdvfhWJQSQP/4tAgBwg7Gvdig5BrD//3RYCa12mF82+LvrarVDAAAAADexOg9eDFg+l62B4BNS6peZl0r4sUMAAAAAuCp1HpRqtbu4quGp1IvveFVpLDnl/n29pQnhCwAAAMD1xclZqtq0pEdx1XjsEAAAAAAsQPgCAAAAAAsQvgAAAADAAoQvXNHq1atVt25dlSlTRh06dMi37ooVK2Sz2XTixIlr6rNFixYaPHjwNbWB60dRXTcAAAClGeELDnILPfHx8apfv7727NmjWbNmlci4SptRo0apfv36JT0MAAAAXEcIX6VZdpa0Z6X06ycX/8zOKpFh7N69W/fdd59uueUW+fn5lcgYgCvJzMws6SEAAADki/BVWm37Qpp8hzT7AenTPhf/nHzHxfJi0qtXL/3www966623ZLPZ7NuxY8f0+OOPy2azFfjO14YNG9SwYUN5enqqcePGSk5Odujn8scXBw8erBYtWjiUXbhwQQMHDpSvr6/8/f2VmJgoY4wKYurUqapRo4bc3d0VGBiohx9+2L4vOztbEyZMUNWqVeXh4aHw8HB98skn9v2XHoFbunRprnOYNWuWRo8erc2bN9vP0aXzcuLECfXt21cBAQHy8fHRfffdp82bN9vbvnTH7D//+Y/CwsLk6+urRx99VCdPnnQY38SJE1W9enW5ubmpcuXKGjdunH3//v371blzZ/n5+al8+fJq37699u7dW6DzIknvvfeeateuLXd3d9WqVUtTp06179u7d69sNpsWLFige++9V56engoPD9eaNWsc2li9erVatGghT09PlStXTjExMfrrr78kSefOndOzzz6rihUryt3dXU2aNNEvv/zicPyiRYt02223ycPDQ/fee2+u41+1apWaNm0qDw8PhYaG6tlnn1VGRoZ9f1hYmMaOHauePXvKx8dH/fv3L/A5AAAAKAmEr9Jo2xfSf3tK6f9zLE9PuVheTAHsrbfeUqNGjdSvXz+lpKTowIEDOnDggHx8fDR58mSlpKSoS5cuBWrr5Zdf1htvvKH169fLxcVFjz/++FWPZ/bs2XJxcdG6dev01ltvadKkSXrvvfeueNz69ev17LPPasyYMUpOTtbixYvVrFkz+/4JEybogw8+0PTp0/Xbb79pyJAheuyxx/TDDz8UaA5dunTRc889p9tvv10pKSkO5+WRRx7R4cOH9c0332jDhg1q0KCBWrZsqePHj9vb3b17tz7//HN99dVX+uqrr/TDDz/o1Vdfte9PSEjQq6++qsTERG3btk1z5sxRYGCgJOn8+fOKiYmRt7e3Vq5cqdWrV6ts2bJq3bp1ge78fPTRRxoxYoTGjRun7du3a/z48UpMTNTs2bNzzP35559XUlKSbrvtNnXt2lUXLlyQJCUlJally5aqU6eO1qxZo1WrVik2NlZZWRfvzL744ov69NNPNXv2bG3cuFHVq1dXTEyM/Rzs379fHTt2VGxsrJKSktS3b18NGzbMof/du3erdevW6tSpk7Zs2aJ58+Zp1apVGjhwoEO9119/XeHh4dq0aZMSExOvOH8AAIASZVAoaWlpRpJJS0vLse/MmTNm27Zt5syZM1ffcNYFY96oZcxInzw2X2PeqH2xXjFo3ry5GTRokEOZr6+vmTlzZoGOX758uZFkvv/+e3vZ119/bSTZz0dcXJxp3769w3GDBg0yzZs3dxhH7dq1TXZ2tr1s6NChpnbt2lccw6effmp8fHxMenp6jn1nz541np6e5qeffnIo79Onj+natWuB5zBy5EgTHh7u0MbKlSuNj4+POXv2rEN5tWrVzDvvvGM/ztPT02FsL7zwgomKijLGGJOenm7c3NzMu+++m+vc/vOf/5iaNWs6nJdz584ZDw8P8+233+Z7Xi6NZc6cOQ5lY8eONY0aNTLGGLNnzx4jybz33nv2/b/99puRZLZv326MMaZr167mnnvuybX9U6dOmTJlypiPPvrIXpaZmWlCQkLMxIkTjTHGJCQkmDp16jgcN3ToUCPJ/PXXX8aYiz+P/v37O9RZuXKlcXJysv8MqlSpYjp06JDvfK/p3yIAAEAB5ZcN/s6lxFIfcvfnTznveDkwUvrBi/VK8bd616tXz/734OBgSdLhw4dVuXLlArdx9913y2az2T83atRIb7zxhrKysuTs7Jzncffff7+qVKmiW2+9Va1bt1br1q310EMPydPTU7t27dLp06d1//33OxyTmZmpO++885rmsHnzZp06dUoVKlRwKD9z5ox2795t/xwWFiZvb2+Htg8fPixJ2r59u86dO6eWLVvm2ceuXbscjpeks2fPOvSRm4yMDO3evVt9+vRRv3797OUXLlyQr6+vQ9285l6rVi0lJSXpkUceybWP3bt36/z587rnnnvsZWXKlFFkZKS2b99un2NUVJTDcY0aNcoxzy1btuijjz6ylxljlJ2drT179qh27dqSpIYNG+Y7ZwAAgNKE8FXanEot2nolpEyZMva/XwpQ2dnZkiQnJ6cc726dP3++yPr29vbWxo0btWLFCi1ZskQjRozQqFGj9Msvv+jUqVOSpK+//lqVKlVyOM7Nza3Ac8jNqVOnFBwcrBUrVuTY9/eFSv7e7qW2L7Xr4eGR79xOnTqliIgIh1BySUBAwBWPlaR33303R/i5PMzmN/crjbEonDp1Sk888YSeffbZHPv+Hn69vLyKfSwAAABFhfBV2pQNLNp6V8nV1dX+7k5xCQgI0NatWx3KkpKScoSStWvXOnz++eefVaNGjXzvel3i4uKi6OhoRUdHa+TIkfLz89OyZct0//33y83NTfv27VPz5s0LPYfczlODBg106NAhubi4KCwsrFDt1qhRQx4eHlq6dKn69u2bY3+DBg00b948VaxYUT4+PlfVdmBgoEJCQvTHH3+oe/fuhRqfdPGu2NKlSzV69Ogc+6pVqyZXV1etXr1aVapUkXQxWP/yyy/2rzCoXbu2vvjC8b3Fn3/+2eFzgwYNtG3bNlWvXr3Q4wQAAChtWHCjtKnSWPIJkWTLo4JN8ql0sV4xCAsL09q1a7V3714dPXo03zs9hXXfffdp/fr1+uCDD7Rz506NHDkyRxiTpH379ik+Pl7Jycn6+OOP9fbbb2vQoEFXbP+rr77SP//5TyUlJenPP//UBx98oOzsbNWsWVPe3t56/vnnNWTIEM2ePVu7d+/Wxo0b9fbbb+dYdCI/YWFh2rNnj5KSknT06FGdO3dO0dHRatSokTp06KAlS5Zo7969+umnn/Tyyy9r/fr1BWrX3d1dQ4cO1YsvvqgPPvhAu3fv1s8//6wZM2ZIkrp37y5/f3+1b99eK1eu1J49e7RixQo9++yzOnDgwBXbHz16tCZMmKB//vOf+v333/Xrr79q5syZmjRpUoHnnpCQoF9++UVPPfWUtmzZoh07dmjatGk6evSovLy89OSTT+qFF17Q4sWLtW3bNvXr10+nT59Wnz59JEkDBgzQzp079cILLyg5OVlz5szJsYrm0KFD9dNPP2ngwIFKSkrSzp07tXDhwhwLbgAAAFxPCF+ljZOz1Pq1///h8gD2/z+3fvVivWLw/PPPy9nZWXXq1FFAQID27dtX5H3ExMQoMTFRL774ou666y6dPHlSPXv2zFGvZ8+eOnPmjCIjI/X0009r0KBBBVpO3M/PTwsWLNB9992n2rVra/r06fr44491++23S5LGjh2rxMRETZgwQbVr11br1q319ddfq2rVqgWeQ6dOndS6dWvde++9CggI0McffyybzaZFixapWbNm6t27t2677TY9+uij+vPPP+2rFRZEYmKinnvuOY0YMUK1a9dWly5d7O+EeXp66scff1TlypXVsWNH1a5dW3369NHZs2cLdCesb9++eu+99zRz5kzVrVtXzZs316xZs65q7rfddpuWLFmizZs3KzIyUo0aNdLChQvl4nLxRvqrr76qTp06qUePHmrQoIF27dqlb7/9VuXKlZN08bHBTz/9VJ9//rnCw8M1ffp0jR8/3qGPevXq6YcfftDvv/+upk2b6s4779SIESMUEhJS4HECAACUNjZz+cs3KJD09HT5+voqLS0txy+9Z8+e1Z49e1S1alW5u7sXroNtX0iLhzouvuFT6WLwqvPgNYwcuHkUyb9FAACAK8gvG/wd73yVVnUelGq1u7iq4anUi+94VWlcbHe8AAAAABQvHjsszZycLy4nX/fhi3+WguA1YMAAlS1bNtdtwIABloxh5cqVeY6hbNmyloyhtMrvvKxcubKkhwcAAHBT484XrsqYMWP0/PPP57rvalffK6yGDRsqKSnJkr6uN/mdl8uX1gcAAIC1CF+4KhUrVlTFihVLdAweHh4sQZ4HzgsAAEDpxWOHxYi1TICSxb9BAABQmhC+isGlLws+ffp0CY8EuLllZmZKUoG+mBsAAKC48dhhMXB2dpafn5/DdzPZbHl9aTKA4pCdna0jR47I09PT/h1kAAAAJYnfSIpJUFCQJNkDGADrOTk5qXLlyvzHDwAAUCoQvoqJzWZTcHCwKlasqPPnz5f0cICbkqurq5yceLoaAACUDoSvYubs7Mz7JgAAAABYcAMAAAAArED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAiUevqZMmaKwsDC5u7srKipK69aty7f+5MmTVbNmTXl4eCg0NFRDhgzR2bNn7ft//PFHxcbGKiQkRDabTZ9//nmONowxGjFihIKDg+Xh4aHo6Gjt3LmzqKcGAAAAAHYlGr7mzZun+Ph4jRw5Uhs3blR4eLhiYmJ0+PDhXOvPmTNHw4YN08iRI7V9+3bNmDFD8+bN00svvWSvk5GRofDwcE2ZMiXPfidOnKh//vOfmj59utauXSsvLy/FxMQ4hDgAAAAAKEo2Y4wpqc6joqJ011136V//+pckKTs7W6GhoXrmmWc0bNiwHPUHDhyo7du3a+nSpfay5557TmvXrtWqVaty1LfZbPrss8/UoUMHe5kxRiEhIXruuef0/PPPS5LS0tIUGBioWbNm6dFHHy3Q2NPT0+Xr66u0tDT5+PhczbQBAAAA3EAKmg1K7M5XZmamNmzYoOjo6P8bjJOToqOjtWbNmlyPady4sTZs2GB/NPGPP/7QokWL1LZt2wL3u2fPHh06dMihX19fX0VFReXZrySdO3dO6enpDhsAAAAAFJRLSXV89OhRZWVlKTAw0KE8MDBQO3bsyPWYbt266ejRo2rSpImMMbpw4YIGDBjg8NjhlRw6dMjez+X9XtqXmwkTJmj06NEF7gcAAAAA/q7EF9y4GitWrND48eM1depUbdy4UQsWLNDXX3+tsWPHFnvfCQkJSktLs2/79+8v9j4BAAAA3DhK7M6Xv7+/nJ2dlZqa6lCempqqoKCgXI9JTExUjx491LdvX0lS3bp1lZGRof79++vll1+Wk9OVs+SltlNTUxUcHOzQb/369fM8zs3NTW5ubldsHwAAAAByU2J3vlxdXRUREeGweEZ2draWLl2qRo0a5XrM6dOncwQsZ2dnSRcX0iiIqlWrKigoyKHf9PR0rV27Ns9+AQAAAOBaldidL0mKj49XXFycGjZsqMjISE2ePFkZGRnq3bu3JKlnz56qVKmSJkyYIEmKjY3VpEmTdOeddyoqKkq7du1SYmKiYmNj7SHs1KlT2rVrl72PPXv2KCkpSeXLl1flypVls9k0ePBgvfLKK6pRo4aqVq2qxMREhYSEOKyKCAAAAABFqUTDV5cuXXTkyBGNGDFChw4dUv369bV48WL7Yhj79u1zuNM1fPhw2Ww2DR8+XAcPHlRAQIBiY2M1btw4e53169fr3nvvtX+Oj4+XJMXFxWnWrFmSpBdffNH+uOKJEyfUpEkTLV68WO7u7hbMGgAAAMDNqES/5+t6xvd8AQAAAJCug+/5AgAAAICbCeELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxQKsLXlClTFBYWJnd3d0VFRWndunX51p88ebJq1qwpDw8PhYaGasiQITp79uxVtdmiRQvZbDaHbcCAAUU+NwAAAACQSkH4mjdvnuLj4zVy5Eht3LhR4eHhiomJ0eHDh3OtP2fOHA0bNkwjR47U9u3bNWPGDM2bN08vvfTSVbfZr18/paSk2LeJEycW61wBAAAA3LxKPHxNmjRJ/fr1U+/evVWnTh1Nnz5dnp6eev/993Ot/9NPP+mee+5Rt27dFBYWplatWqlr164Od7YK2qanp6eCgoLsm4+PT7HOFQAAAMDNq0TDV2ZmpjZs2KDo6Gh7mZOTk6Kjo7VmzZpcj2ncuLE2bNhgD1t//PGHFi1apLZt2151mx999JH8/f11xx13KCEhQadPn85zrOfOnVN6errDBgAAAAAF5VKSnR89elRZWVkKDAx0KA8MDNSOHTtyPaZbt246evSomjRpImOMLly4oAEDBtgfOyxom926dVOVKlUUEhKiLVu2aOjQoUpOTtaCBQty7XfChAkaPXr0tUwXAAAAwE2sRMNXYaxYsULjx4/X1KlTFRUVpV27dmnQoEEaO3asEhMTC9xO//797X+vW7eugoOD1bJlS+3evVvVqlXLUT8hIUHx8fH2z+np6QoNDb22yQAAAAC4aZRo+PL395ezs7NSU1MdylNTUxUUFJTrMYmJierRo4f69u0r6WJwysjIUP/+/fXyyy8Xqk1JioqKkiTt2rUr1/Dl5uYmNze3q5ofAAAAAFxSou98ubq6KiIiQkuXLrWXZWdna+nSpWrUqFGux5w+fVpOTo7DdnZ2liQZYwrVpiQlJSVJkoKDgws7HQAAAADIU4k/dhgfH6+4uDg1bNhQkZGRmjx5sjIyMtS7d29JUs+ePVWpUiVNmDBBkhQbG6tJkybpzjvvtD92mJiYqNjYWHsIu1Kbu3fv1pw5c9S2bVtVqFBBW7Zs0ZAhQ9SsWTPVq1evZE4EAAAAgBtaiYevLl266MiRIxoxYoQOHTqk+vXra/HixfYFM/bt2+dwp2v48OGy2WwaPny4Dh48qICAAMXGxmrcuHEFbtPV1VXff/+9PZSFhoaqU6dOGj58uLWTBwAAAHDTsBljTEkP4nqUnp4uX19fpaWl8f1gAAAAwE2soNmgxL9kGQAAAABuBoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAAC1xT+Nq1a5e+/fZbnTlzRpJkjCmSQQEAAADAjaZQ4evYsWOKjo7WbbfdprZt2yolJUWS1KdPHz333HNFOkAAAAAAuBEUKnwNGTJELi4u2rdvnzw9Pe3lXbp00eLFi4tscAAAAABwo3ApzEFLlizRt99+q1tuucWhvEaNGvrzzz+LZGAAAAAAcCMp1J2vjIwMhztelxw/flxubm7XPCgAAAAAuNEUKnw1bdpUH3zwgf2zzWZTdna2Jk6cqHvvvbfIBgcAAAAAN4pCPXY4ceJEtWzZUuvXr1dmZqZefPFF/fbbbzp+/LhWr15d1GMEAAAAgOteoe583XHHHfr999/VpEkTtW/fXhkZGerYsaM2bdqkatWqFfUYAQAAAOC6d9V3vs6fP6/WrVtr+vTpevnll4tjTAAAAABww7nqO19lypTRli1bimMsAAAAAHDDKtRjh4899phmzJhR1GMBAAAAgBtWoRbcuHDhgt5//319//33ioiIkJeXl8P+SZMmFcngAAAAAOBGUajwtXXrVjVo0ECS9Pvvvzvss9ls1z4qAAAAALjBFCp8LV++vKjHAQAAAAA3tEK98/V3Bw4c0IEDB4piLAAAAABwwypU+MrOztaYMWPk6+urKlWqqEqVKvLz89PYsWOVnZ1d1GMEAAAAgOteoR47fPnllzVjxgy9+uqruueeeyRJq1at0qhRo3T27FmNGzeuSAcJAAAAANc7mzHGXO1BISEhmj59uh588EGH8oULF+qpp57SwYMHi2yApVV6erp8fX2VlpYmHx+fkh4OAAAAgBJS0GxQqMcOjx8/rlq1auUor1Wrlo4fP16YJgEAAADghlao8BUeHq5//etfOcr/9a9/KTw8/JoHBQAAAAA3mkK98zVx4kS1a9dO33//vRo1aiRJWrNmjfbv369FixYV6QABAAAA4EZQqDtfzZs3V3Jysh566CGdOHFCJ06cUMeOHZWcnKymTZsW9RgBAAAA4LpXqAU3wIIbAAAAAC4q1gU3Zs6cqfnz5+conz9/vmbPnl2YJgEAAADghlao8DVhwgT5+/vnKK9YsaLGjx9/zYMCAAAAgBtNocLXvn37VLVq1RzlVapU0b59+655UAAAAABwoylU+KpYsaK2bNmSo3zz5s2qUKHCNQ8KAAAAAG40hQpfXbt21bPPPqvly5crKytLWVlZWrZsmQYNGqRHH320qMcIAAAAANe9Qn3P19ixY7V37161bNlSLi4Xm8jOzlbPnj155wsAAAAAcnFNS83v3LlTSUlJ8vDwUN26dVWlSpWiHFupxlLzAAAAAKSCZ4NC3fm6pEaNGqpRo4aysrL066+/ysfHR+XKlbuWJgEAAADghlSod74GDx6sGTNmSJKysrLUvHlzNWjQQKGhoVqxYsVVtzdlyhSFhYXJ3d1dUVFRWrduXb71J0+erJo1a8rDw0OhoaEaMmSIzp49e1Vtnj17Vk8//bQqVKigsmXLqlOnTkpNTb3qsQMAAABAQRQqfH3yyScKDw+XJH355Zf6448/tGPHDg0ZMkQvv/zyVbU1b948xcfHa+TIkdq4caPCw8MVExOjw4cP51p/zpw5GjZsmEaOHKnt27drxowZmjdvnl566aWranPIkCH68ssvNX/+fP3www/63//+p44dOxbibAAAAADAlRXqnS93d3ft2rVLt9xyi/r37y9PT09NnjxZe/bsUXh4uNLT0wvcVlRUlO666y7961//knRx4Y7Q0FA988wzGjZsWI76AwcO1Pbt27V06VJ72XPPPae1a9dq1apVBWozLS1NAQEBmjNnjh5++GFJ0o4dO1S7dm2tWbNGd9999xXHzTtfAAAAAKSCZ4NC3fkKDAzUtm3blJWVpcWLF+v++++XJJ0+fVrOzs4FbiczM1MbNmxQdHT0/w3IyUnR0dFas2ZNrsc0btxYGzZssD9G+Mcff2jRokVq27ZtgdvcsGGDzp8/71CnVq1aqly5cp79njt3Tunp6Q4bAAAAABRUoRbc6N27tzp37qzg4GDZbDZ7iFm7dq1q1apV4HaOHj2qrKwsBQYGOpQHBgZqx44duR7TrVs3HT16VE2aNJExRhcuXNCAAQPsjx0WpM1Dhw7J1dVVfn5+OeocOnQo134nTJig0aNHF3huAAAAAPB3hbrzNWrUKL333nvq37+/Vq9eLTc3N0mSs7Nzro8KFqUVK1Zo/Pjxmjp1qjZu3KgFCxbo66+/1tixY4u134SEBKWlpdm3/fv3F2t/AAAAAG4shV5q/tK7UgcOHFB2dracnJwUFxd3VW34+/vL2dk5xyqDqampCgoKyvWYxMRE9ejRQ3379pUk1a1bVxkZGerfv79efvnlArUZFBSkzMxMnThxwuHuV379urm52UMmAAAAAFytQt35+rs6depo7969hTrW1dVVERERDotnZGdna+nSpWrUqFGux5w+fVpOTo7DvvSemTGmQG1GRESoTJkyDnWSk5O1b9++PPsFAAAAgGtxTV+yLF0MPNciPj5ecXFxatiwoSIjIzV58mRlZGSod+/ekqSePXuqUqVKmjBhgiQpNjZWkyZN0p133qmoqCjt2rVLiYmJio2NtYewK7Xp6+urPn36KD4+XuXLl5ePj4+eeeYZNWrUqEArHQIAAADA1brm8HWtunTpoiNHjmjEiBE6dOiQ6tevr8WLF9sXzNi3b5/Dna7hw4fLZrNp+PDhOnjwoAICAhQbG6tx48YVuE1JevPNN+Xk5KROnTrp3LlziomJ0dSpU62bOAAAAICbSqG+5+vvJkyYoCeffDLHyoE3Or7nCwAAAIBU8GxwzeHrZkX4AgAAACAV85cs52X//v16/PHHi7JJAAAAALghFGn4On78uGbPnl2UTQIAAADADeGqFtz44osv8t3/xx9/XNNgAAAAAOBGdVXhq0OHDrLZbPkuL2+z2a55UAAAAABwo7mqxw6Dg4O1YMECZWdn57pt3LixuMYJAAAAANe1qwpfERER2rBhQ577r3RXDAAAAABuVlf12OELL7ygjIyMPPdXr15dy5cvv+ZBAQAAAMCN5qrCV6VKlVS1atU893t5eal58+bXPCgAAAAAuNFc1WOHNWrU0JEjR+yfu3TpotTU1CIfFAAAAADcaK4qfF3+PteiRYvyfQwRAAAAAHBRkX7JMgAAAAAgd1cVvmw2W47v8eJ7vQAAAADgyq5qwQ1jjHr16iU3NzdJ0tmzZzVgwAB5eXk51FuwYEHRjRAAAAAAbgBXFb7i4uIcPj/22GNFOhgAAAAAuFFdVfiaOXNmcY0DAAAAAG5oLLgBAAAAABYgfAEAAACABQhfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQAAAIAFCF8AAAAAYAHCFwAAAABYgPAFAAAAABYgfAEAAACABQhfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQAAAIAFCF8AAAAAYAHCFwAAAABYgPAFAAAAABYgfAEAAACABQhfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQAAAIAFCF8AAAAAYAHCFwAAAABYgPAFAAAAABYgfAEAAACABQhfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQAAAIAFCF8AAAAAYAHCFwAAAABYgPAFAAAAABYgfAEAAACABQhfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQAAAIAFCF8AAAAAYIFSEb6mTJmisLAwubu7KyoqSuvWrcuzbosWLWSz2XJs7dq1s9dJTU1Vr169FBISIk9PT7Vu3Vo7d+68YjsDBgwotjkCAAAAuLmVePiaN2+e4uPjNXLkSG3cuFHh4eGKiYnR4cOHc62/YMECpaSk2LetW7fK2dlZjzzyiCTJGKMOHTrojz/+0MKFC7Vp0yZVqVJF0dHRysjIcGirX79+Dm1NnDix2OcLAAAA4OZU4uFr0qRJ6tevn3r37q06depo+vTp8vT01Pvvv59r/fLlyysoKMi+fffdd/L09LSHr507d+rnn3/WtGnTdNddd6lmzZqaNm2azpw5o48//tihLU9PT4e2fHx8in2+AAAAAG5OJRq+MjMztWHDBkVHR9vLnJycFB0drTVr1hSojRkzZujRRx+Vl5eXJOncuXOSJHd3d4c23dzctGrVKodjP/roI/n7++uOO+5QQkKCTp8+nWc/586dU3p6usMGAAAAAAVVouHr6NGjysrKUmBgoEN5YGCgDh06dMXj161bp61bt6pv3772slq1aqly5cpKSEjQX3/9pczMTL322ms6cOCAUlJS7PW6deumDz/8UMuXL1dCQoL+85//6LHHHsuzrwkTJsjX19e+hYaGFmLGAAAAAG5WLiU9gGsxY8YM1a1bV5GRkfayMmXKaMGCBerTp4/Kly8vZ2dnRUdHq02bNjLG2Ov179/f/ve6desqODhYLVu21O7du1WtWrUcfSUkJCg+Pt7+OT09nQAGAAAAoMBKNHz5+/vL2dlZqampDuWpqakKCgrK99iMjAzNnTtXY8aMybEvIiJCSUlJSktLU2ZmpgICAhQVFaWGDRvm2V5UVJQkadeuXbmGLzc3N7m5uRVkWgAAAACQQ4k+dujq6qqIiAgtXbrUXpadna2lS5eqUaNG+R47f/58nTt3Lt9HBX19fRUQEKCdO3dq/fr1at++fZ51k5KSJEnBwcFXNwkAAAAAKIASf+wwPj5ecXFxatiwoSIjIzV58mRlZGSod+/ekqSePXuqUqVKmjBhgsNxM2bMUIcOHVShQoUcbc6fP18BAQGqXLmyfv31Vw0aNEgdOnRQq1atJEm7d+/WnDlz1LZtW1WoUEFbtmzRkCFD1KxZM9WrV6/4Jw0AAADgplPi4atLly46cuSIRowYoUOHDql+/fpavHixfRGOffv2ycnJ8QZdcnKyVq1apSVLluTaZkpKiuLj45Wamqrg4GD17NlTiYmJ9v2urq76/vvv7UEvNDRUnTp10vDhw4tvogAAAABuajbz91UoUGDp6eny9fVVWloa3w8GAAAA3MQKmg1K/EuWAQAAAOBmQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAqUifE2ZMkVhYWFyd3dXVFSU1q1bl2fdFi1ayGaz5djatWtnr5OamqpevXopJCREnp6eat26tXbu3OnQztmzZ/X000+rQoUKKlu2rDp16qTU1NRimyMAAACAm1uJh6958+YpPj5eI0eO1MaNGxUeHq6YmBgdPnw41/oLFixQSkqKfdu6daucnZ31yCOPSJKMMerQoYP++OMPLVy4UJs2bVKVKlUUHR2tjIwMeztDhgzRl19+qfnz5+uHH37Q//73P3Xs2NGSOQMAAAC4+diMMaYkBxAVFaW77rpL//rXvyRJ2dnZCg0N1TPPPKNhw4Zd8fjJkydrxIgRSklJkZeXl37//XfVrFlTW7du1e23325vMygoSOPHj1ffvn2VlpamgIAAzZkzRw8//LAkaceOHapdu7bWrFmju+++O0c/586d07lz5+yf09PTFRoaqrS0NPn4+BTFqQAAAABwHUpPT5evr+8Vs0GJ3vnKzMzUhg0bFB0dbS9zcnJSdHS01qxZU6A2ZsyYoUcffVReXl6SZA9I7u7uDm26ublp1apVkqQNGzbo/PnzDv3WqlVLlStXzrPfCRMmyNfX176FhoZe3WQBAAAA3NRKNHwdPXpUWVlZCgwMdCgPDAzUoUOHrnj8unXrtHXrVvXt29dedilEJSQk6K+//lJmZqZee+01HThwQCkpKZKkQ4cOydXVVX5+fgXuNyEhQWlpafZt//79VzlbAAAAADezEn/n61rMmDFDdevWVWRkpL2sTJkyWrBggX7//XeVL19enp6eWr58udq0aSMnp8JP183NTT4+Pg4bAAAAABRUiYYvf39/OTs751hlMDU1VUFBQfkem5GRoblz56pPnz459kVERCgpKUknTpxQSkqKFi9erGPHjunWW2+VJAUFBSkzM1MnTpy46n4BAAAAoDBKNHy5uroqIiJCS5cutZdlZ2dr6dKlatSoUb7Hzp8/X+fOndNjjz2WZx1fX18FBARo586dWr9+vdq3by/pYjgrU6aMQ7/Jycnat2/fFfsFAAAAgMJwKekBxMfHKy4uTg0bNlRkZKQmT56sjIwM9e7dW5LUs2dPVapUSRMmTHA4bsaMGerQoYMqVKiQo8358+crICBAlStX1q+//qpBgwapQ4cOatWqlaSLoaxPnz6Kj49X+fLl5ePjo2eeeUaNGjXKdaVDAAAAALhWJR6+unTpoiNHjmjEiBE6dOiQ6tevr8WLF9sX4di3b1+Od7WSk5O1atUqLVmyJNc2U1JSFB8fr9TUVAUHB6tnz55KTEx0qPPmm2/KyclJnTp10rlz5xQTE6OpU6cWzyQBAAAA3PRK/Hu+rlcFXcsfAAAAwI3tuvieLwAAAAC4WRC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAoQvAAAAALAA4QsAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAs4FLSA8C1yco2WrfnuA6fPKuK3u6KrFpezk62kh4WAAAAgMsQvq5ji7emaPSX25SSdtZeFuzrrpGxddT6juASHBkAAACAy/HY4XVq8dYUPfnhRofgJUmH0s7qyQ83avHWlBIaGQAAAIDcEL6uQ1nZRqO/3CaTy75LZaO/3Kas7NxqAAAAACgJhK/r0Lo9x3Pc8fo7Iykl7azW7Tlu3aAAAAAA5IvwdR06fDLv4FWYegAAAACKH+HrOlTR271I6wEAAAAofoSv61Bk1fIK9nVXXgvK23Rx1cPIquWtHBYAAACAfBC+rkPOTjaNjK0jSTkC2KXPI2Pr8H1fAAAAQClC+LpOtb4jWNMea6AgX8dHC4N83TXtsQZ8zxcAAABQyvAly9ex1ncE6/46QVq357gOnzyrit4XHzXkjhcAAABQ+hC+rnPOTjY1qlahpIcBAAAA4Ap47BAAAAAALED4AgAAAAALEL4AAAAAwAKELwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAuUivA1ZcoUhYWFyd3dXVFRUVq3bl2edVu0aCGbzZZja9eunb3OqVOnNHDgQN1yyy3y8PBQnTp1NH369Cu2M2DAgGKbIwAAAICbm0tJD2DevHmKj4/X9OnTFRUVpcmTJysmJkbJycmqWLFijvoLFixQZmam/fOxY8cUHh6uRx55xF4WHx+vZcuW6cMPP1RYWJiWLFmip556SiEhIXrwwQft9fr166cxY8bYP3t6ehbTLAEAAADc7Er8ztekSZPUr18/9e7d236HytPTU++//36u9cuXL6+goCD79t1338nT09MhfP3000+Ki4tTixYtFBYWpv79+ys8PDzHHTVPT0+Htnx8fIp1rgAAAABuXiUavjIzM7VhwwZFR0fby5ycnBQdHa01a9YUqI0ZM2bo0UcflZeXl72scePG+uKLL3Tw4EEZY7R8+XL9/vvvatWqlcOxH330kfz9/XXHHXcoISFBp0+fzrOfc+fOKT093WEDAAAAgIIq0ccOjx49qqysLAUGBjqUBwYGaseOHVc8ft26ddq6datmzJjhUP7222+rf//+uuWWW+Ti4iInJye9++67atasmb1Ot27dVKVKFYWEhGjLli0aOnSokpOTtWDBglz7mjBhgkaPHl2IWQIAAABAKXjn61rMmDFDdevWVWRkpEP522+/rZ9//llffPGFqlSpoh9//FFPP/20QkJC7HfZ+vfvb69ft25dBQcHq2XLltq9e7eqVauWo6+EhATFx8fbP6enpys0NLSYZgYAAADgRlOi4cvf31/Ozs5KTU11KE9NTVVQUFC+x2ZkZGju3LkOC2ZI0pkzZ/TSSy/ps88+s6+AWK9ePSUlJen11193eMTx76KioiRJu3btyjV8ubm5yc3NrcBzAwAAAIC/K9Hw5erqqoiICC1dulQdOnSQJGVnZ2vp0qUaOHBgvsfOnz9f586d02OPPeZQfv78eZ0/f15OTo6vszk7Oys7OzvP9pKSkiRJwcHBBRq7MUaSePcLAAAAuMldygSXMkKeTAmbO3eucXNzM7NmzTLbtm0z/fv3N35+fubQoUPGGGN69Ohhhg0bluO4Jk2amC5duuTaZvPmzc3tt99uli9fbv744w8zc+ZM4+7ubqZOnWqMMWbXrl1mzJgxZv369WbPnj1m4cKF5tZbbzXNmjUr8Lj3799vJLGxsbGxsbGxsbGxsRlJZv/+/flmiBJ/56tLly46cuSIRowYoUOHDql+/fpavHixfRGOffv25biLlZycrFWrVmnJkiW5tjl37lwlJCSoe/fuOn78uKpUqaJx48bZv0TZ1dVV33//vSZPnqyMjAyFhoaqU6dOGj58eIHHHRISov3798vb21snT55UaGio9u/fz3L1sMSldw655mAlrjtYjWsOVuOaQ2EZY3Ty5EmFhITkW89mzJXujeFK0tPT5evrq7S0NP6hwhJccygJXHewGtccrMY1h+JW4l+yDAAAAAA3A8IXAAAAAFiA8FUE3NzcNHLkSJaih2W45lASuO5gNa45WI1rDsWNd74AAAAAwALc+QIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAscFOGrylTpigsLEzu7u6KiorSunXr8q0/f/581apVS+7u7qpbt64WLVrksN8YoxEjRig4OFgeHh6Kjo7Wzp07HeocP35c3bt3l4+Pj/z8/NSnTx+dOnXKoc6WLVvUtGlTubu7KzQ0VBMnTiyaCaPElcZr7uzZs+rVq5fq1q0rFxcXdejQocjmi5JXGq+5FStWqH379goODpaXl5fq16+vjz76qOgmjRJVGq+55ORk3XvvvQoMDJS7u7tuvfVWDR8+XOfPny+6iaNElcbr7u927dolb29v+fn5XdM8cQMxN5m5c+caV1dX8/7775vffvvN9OvXz/j5+ZnU1NRc669evdo4OzubiRMnmm3btpnhw4ebMmXKmF9//dVe59VXXzW+vr7m888/N5s3bzYPPvigqVq1qjlz5oy9TuvWrU14eLj5+eefzcqVK0316tVN165d7fvT0tJMYGCg6d69u9m6dav5+OOPjYeHh3nnnXeK72TAEqX1mjt16pQZMGCA+fe//21iYmJM+/bti+0cwFql9ZobN26cGT58uFm9erXZtWuXmTx5snFycjJffvll8Z0MWKK0XnO7d+8277//vklKSjJ79+41CxcuNBUrVjQJCQnFdzJgmdJ63V2SmZlpGjZsaNq0aWN8fX2LfP64Pt104SsyMtI8/fTT9s9ZWVkmJCTETJgwIdf6nTt3Nu3atXMoi4qKMk888YQxxpjs7GwTFBRk/vGPf9j3nzhxwri5uZmPP/7YGGPMtm3bjCTzyy+/2Ot88803xmazmYMHDxpjjJk6daopV66cOXfunL3O0KFDTc2aNa9xxihppfWa+7u4uDjC1w3kerjmLmnbtq3p3bv31U8Spcr1dM0NGTLENGnS5OoniVKntF93L774onnsscfMzJkzCV+wu6keO8zMzNSGDRsUHR1tL3NyclJ0dLTWrFmT6zFr1qxxqC9JMTEx9vp79uzRoUOHHOr4+voqKirKXmfNmjXy8/NTw4YN7XWio6Pl5OSktWvX2us0a9ZMrq6uDv0kJyfrr7/+usaZo6SU5msON6br7ZpLS0tT+fLlr36iKDWup2tu165dWrx4sZo3b164yaLUKO3X3bJlyzR//nxNmTLl2ieLG8pNFb6OHj2qrKwsBQYGOpQHBgbq0KFDuR5z6NChfOtf+vNKdSpWrOiw38XFReXLl3eok1sbf+8D15/SfM3hxnQ9XXP//e9/9csvv6h3794FnB1Ko+vhmmvcuLHc3d1Vo0YNNW3aVGPGjLnKWaK0Kc3X3bFjx9SrVy/NmjVLPj4+hZwhblQ3VfgCAJQOy5cvV+/evfXuu+/q9ttvL+nh4AY3b948bdy4UXPmzNHXX3+t119/vaSHhBtYv3791K1bNzVr1qykh4JS6KYKX/7+/nJ2dlZqaqpDeWpqqoKCgnI9JigoKN/6l/68Up3Dhw877L9w4YKOHz/uUCe3Nv7eB64/pfmaw43perjmfvjhB8XGxurNN99Uz549r3KGKG2uh2suNDRUderUUdeuXfXqq69q1KhRysrKusqZojQpzdfdsmXL9Prrr8vFxUUuLi7q06eP0tLS5OLiovfff7+QM8aN4qYKX66uroqIiNDSpUvtZdnZ2Vq6dKkaNWqU6zGNGjVyqC9J3333nb1+1apVFRQU5FAnPT1da9eutddp1KiRTpw4oQ0bNtjrLFu2TNnZ2YqKirLX+fHHHx2Wv/3uu+9Us2ZNlStX7hpnjpJSmq853JhK+zW3YsUKtWvXTq+99pr69+9/7RNGiSvt19zlsrOzdf78eWVnZ1/9ZFFqlObrbs2aNUpKSrJvY8aMkbe3t5KSkvTQQw8VzQnA9aukV/yw2ty5c42bm5uZNWuW2bZtm+nfv7/x8/Mzhw4dMsYY06NHDzNs2DB7/dWrVxsXFxfz+uuvm+3bt5uRI0fmuiypn5+fWbhwodmyZYtp3759rsuS3nnnnWbt2rVm1apVpkaNGg7Lkp44ccIEBgaaHj16mK1bt5q5c+caT09Plpq/AZTWa84YY3777TezadMmExsba1q0aGE2bdpkNm3aVLwnBMWutF5zy5YtM56eniYhIcGkpKTYt2PHjllwVlCcSus19+GHH5p58+aZbdu2md27d5t58+aZkJAQ0717dwvOCopbab3uLsdqh/i7my58GWPM22+/bSpXrmxcXV1NZGSk+fnnn+37mjdvbuLi4hzq//e//zW33XabcXV1Nbfffrv5+uuvHfZnZ2ebxMREExgYaNzc3EzLli1NcnKyQ51jx46Zrl27mrJlyxofHx/Tu3dvc/LkSYc6mzdvNk2aNDFubm6mUqVK5tVXXy3aiaPElNZrrkqVKkZSjg3Xv9J4zcXFxeV6vTVv3rzI5w/rlcZrbu7cuaZBgwambNmyxsvLy9SpU8eMHz/e4RdpXN9K43V3OcIX/s5mjDElddcNAAAAAG4WN9U7XwAAAABQUghfAAAAAGABwhcAAAAAWIDwBQAAAAAWIHwBAAAAgAUIXwAAAABgAcIXAAAAAFiA8AUAAAAAFiB8AQCKRa9evdShQ4eSHsYNx2az6fPPP5ck7d27VzabTUlJSYVuryjaAAAUjEtJDwAAcP2x2Wz57h85cqTeeustGWMsGtHNKTQ0VCkpKfL39y9Q/V69eunEiRP28FaYNgAAhUf4AgBctZSUFPvf582bpxEjRig5OdleVrZsWZUtW7YkhlZszp8/rzJlypSqtpydnRUUFFTibQAACobHDgEAVy0oKMi++fr6ymazOZSVLVs2x2OHLVq00DPPPKPBgwerXLlyCgwM1LvvvquMjAz17t1b3t7eql69ur755huHvrZu3ao2bdqobNmyCgwMVI8ePXT06NE8xzZr1iz5+fnp888/V40aNeTu7q6YmBjt37/fod7ChQvVoEEDubu769Zbb9Xo0aN14cIF+36bzaZp06bpwQcflJeXl8aNG5drf2FhYRo7dqy6du0qLy8vVapUSVOmTHGok1dbVxrDzp071axZM7m7u6tOnTr67rvvHNrN7ZHB3377TQ888IB8fHzk7e2tpk2bavfu3Ro1apRmz56thQsXymazyWazacWKFbm28cMPPygyMlJubm4KDg7WsGHDHMbVokULPfvss3rxxRdVvnx5BQUFadSoUXn+TAAAFxG+AACWmT17tvz9/bVu3To988wzevLJJ/XII4+ocePG2rhxo1q1aqUePXro9OnTkqQTJ07ovvvu05133qn169dr8eLFSk1NVefOnfPt5/Tp0xo3bpw++OADrV69WidOnNCjjz5q379y5Ur17NlTgwYN0rZt2/TOO+9o1qxZOQLWqFGj9NBDD+nXX3/V448/nmd///jHPxQeHq5NmzZp2LBhGjRoUI6gdHlbVxpDdna2OnbsKFdXV61du1bTp0/X0KFD8533wYMH1axZM7m5uWnZsmXasGGDHn/8cV24cEHPP/+8OnfurNatWyslJUUpKSlq3Lhxrm20bdtWd911lzZv3qxp06ZpxowZeuWVVxzqzZ49W15eXlq7dq0mTpyoMWPG5JgzAOAyBgCAazBz5kzj6+ubozwuLs60b9/e/rl58+amSZMm9s8XLlwwXl5epkePHvaylJQUI8msWbPGGGPM2LFjTatWrRza3b9/v5FkkpOT8xyPJPPzzz/by7Zv324kmbVr1xpjjGnZsqUZP368w3H/+c9/THBwsP2zJDN48OArzN6YKlWqmNatWzuUdenSxbRp0ybftq40hm+//da4uLiYgwcP2vd/8803RpL57LPPjDHG7Nmzx0gymzZtMsYYk5CQYKpWrWoyMzNzHevlP5Pc2njppZdMzZo1TXZ2tr3OlClTTNmyZU1WVpYxJufP0hhj7rrrLjN06NBc+wUAXMQ7XwAAy9SrV8/+d2dnZ1WoUEF169a1lwUGBkqSDh8+LEnavHmzli9fnuv7Y7t379Ztt92Waz8uLi6666677J9r1aolPz8/bd++XZGRkdq8ebNWr17tcKcrKytLZ8+e1enTp+Xp6SlJatiwYYHm1ahRoxyfJ0+e7FB2eVtXGsP27dsVGhqqkJCQPPu5XFJSkpo2bXpN75Nt375djRo1clhU5Z577tGpU6d04MABVa5cWZLjz1KSgoOD7T83AEDuCF8AAMtcHgpsNptD2aVf+LOzsyVJp06dUmxsrF577bUcbQUHBxd6HKdOndLo0aPVsWPHHPvc3d3tf/fy8ip0H5e7vK2CjuFqeHh4FOq4wsjtZ3np5wYAyB3hCwBQajVo0ECffvqpwsLC5OJS8P/LunDhgtavX6/IyEhJUnJysk6cOKHatWvb201OTlb16tWLZJw///xzjs+X+srLlcZQu3Zt7d+/XykpKfageXk/l6tXr55mz56d52qKrq6uysrKyreN2rVr69NPP5Uxxh6GV69eLW9vb91yyy35HgsAyB8LbgAASq2nn35ax48fV9euXfXLL79o9+7d+vbbb9W7d+98Q0SZMmX0zDPPaO3atdqwYYN69eqlu+++2x7GRowYoQ8++ECjR4/Wb7/9pu3bt2vu3LkaPnx4oca5evVqTZw4Ub///rumTJmi+fPna9CgQfkec6UxREdH67bbblNcXJw2b96slStX6uWXX863zYEDByo9PV2PPvqo1q9fr507d+o///mP/WsAwsLCtGXLFiUnJ+vo0aM6f/58jjaeeuop7d+/X88884x27NihhQsXauTIkYqPj5eTE782AMC14H9FAQClVkhIiFavXq2srCy1atVKdevW1eDBg+Xn55dvEPD09NTQoUPVrVs33XPPPSpbtqzmzZtn3x8TE6OvvvpKS5Ys0V133aW7775bb775pqpUqVKocT733HNav3697rzzTr3yyiuaNGmSYmJi8j3mSmNwcnLSZ599pjNnzigyMlJ9+/bNc7n7SypUqKBly5bp1KlTat68uSIiIvTuu+/a74L169dPNWvWVMOGDRUQEKDVq1fnaKNSpUpatGiR1q1bp/DwcA0YMEB9+vQpdDAFAPwfmzHGlPQgAAAoKrNmzdLgwYN14sQJS/oLCwvT4MGDNXjwYEv6AwBcv7jzBQAAAAAWIHwBAAAAgAV47BAAAAAALMCdLwAAAACwAOELAAAAACxA+AIAAAAACxC+AAAAAMAChC8AAAAAsADhCwAAAAAsQPgCAAAAAAsQvgAAAADAAv8P8OhV6JiqQL8AAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "A8lMhwdzBzGC"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNKvhiyLgK+yYOdZ931fSZR",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}