DataExploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cbcf10b4-b8ba-4232-bf71-5b0b6d81b513",
   "metadata": {},
   "source": [
    "# Part 1: Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6830ba-ece4-45f3-9707-c132f0f0cf20",
   "metadata": {},
   "source": [
    "In this notebook you will perform your:\n",
    "- data cleaning\n",
    "- data exploration\n",
    "- feature engineering\n",
    "\n",
    "There is no one, correct, way to structure this notebook and so you may build this notebook out in whatever way is most conducive towards your preferences. At the very minimum, notebook should read in the raw data and output to .csv format your cleaned and prepared data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c57f70-6eb2-487f-a5a1-cd6adf7f0ed9",
   "metadata": {},
   "source": [
    "To start you off, the variable df has been pre-loaded to have your training dataset. **This is the data that you will use to train and test your model. The file labled \"test.tsv\" should only be used at the very end of the challenge for your final performance evaluation!** If you would like to know more about this data you can go [here](https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/overview). We encourage you to look at the 'Overview' and 'Data' tabs; however, we encourage you also to not look at community submissions and to develop your own unique solution from your own research!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37517385-a40d-4e72-8918-b5b989302223",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PhraseId</th>\n",
       "      <th>SentenceId</th>\n",
       "      <th>Phrase</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>A series of escapades demonstrating the adage ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>A series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>series</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PhraseId  SentenceId                                             Phrase  \\\n",
       "0         1           1  A series of escapades demonstrating the adage ...   \n",
       "1         2           1  A series of escapades demonstrating the adage ...   \n",
       "2         3           1                                           A series   \n",
       "3         4           1                                                  A   \n",
       "4         5           1                                             series   \n",
       "\n",
       "   Sentiment  \n",
       "0          1  \n",
       "1          2  \n",
       "2          2  \n",
       "3          2  \n",
       "4          2  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import spacy\n",
    "from tqdm import tqdm\n",
    "%matplotlib inline\n",
    "os.system(\"python -m spacy download en_core_web_lg\")\n",
    "\n",
    "# when it comes time to evaluate the model for final scoring, just change this to test.tsv!\n",
    "df = pd.read_csv('train.csv', index_col=0)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d931f409-01c6-4c03-ad3a-5cd30edbfede",
   "metadata": {
    "tags": []
   },
   "source": [
    "One final note: if you are looking for files you export from this notebook, by default (if you don't otherwise assign the path) any files output from this notebook will be placed in the code directory! Good luck on BYTES Challenge II!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7c029c3a-0768-4fbd-abeb-c680edff5eda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Sentences: 4877\n",
      "Number of Phrases: 93644\n",
      "Number of words:\n",
      "Min: 1 Mean: 19.044084478162805 Max: 52\n",
      "Number of times a word occurs only once: 6440\n",
      "Total number of unique words: 12201\n",
      "Known words ratio: 0.4721744119334481\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Distribution of number of words in a sentence')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFwCAYAAACGt6HXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABEAElEQVR4nO3deXxV9Zn48c+TPUBIIAmQhB3CEhACIi64r4AKOp12tHVpa3+OrXabtqO2nf7s9qvtTDenVmtbp9o6tVptRYtSxeK+gCxhh7AmEMi+EbI/vz/OCV6vN8kN3Jtzl+f9et1X7tmfc5fnfvM93/P9iqpijDFm8CV4HYAxxsQrS8DGGOMRS8DGGOMRS8DGGOMRS8DGGOMRS8DGGOMRS8C9EJEHReQ/QrSv8SLSLCKJ7vQaEflMKPbt7u95Ebk5VPsbwHG/JyLVInJksI/tF8eFIlLu4fGvFZEy9z2e52Ec+0Xk0gFuc56I7AxXTKZvSV4H4AUR2Q+MBjqBLmAb8CjwkKp2A6jqbQPY12dU9aXe1lHVg8CwU4v6xPHuAaaq6g0++18Sin0PMI5xwFeACapaOdjHjzD/Bdyhqs94HchAqeprwHSv4zhZIvI7oFxVv+l1LCcjnkvAV6tqBjABuBe4E/htqA8iIrH6IzcBqIm15HuS79cEYGuoY+lNDH+m4o+qxt0D2A9c6jdvIdANzHanfwd8z32eAzwH1AO1wGs4P16/d7c5DjQD/w5MBBS4BTgIvOozL8nd3xrgB8C7QAPwDDDSXXYhzi/6h+IFFgPtQId7vE0++/uM+zwB+CZwAKjEKdlnust64rjZja0a+EYfr1Omu32Vu79vuvu/1D3nbjeO3wXY9kKgHKeUXAlUAJ/yWX4iZnf6k8DrPtMKfA7YDTQB3wWmAG8BjcATQIrfsb7untN+4BM++0rFKaUeBI4CDwLpftveCRwBfh/gXAK+pu5+m91YjwF7Amz7beC/3efJ7no/cqfTgVZghDu9DCeR17uvz0y/z8CdQAnQhvPf641uTDXAN/D5XON8nte5r9VR4Ce9vMcX4vN5c/fxVfc4DcCfgLRetp0CvOwevxp4DMjqZV0Bfuq+fg3u/nu+a8G8Px/6HAG34nwX2t334Vl3fj7wFM7ndh/wBZ847sH57DyK87naCizwWT4OeNrdtgb4hc+yTwPbgTpgFc5/f6eWi7xOhl48CJCA3fkHgc+6z3/H+wn4B+6HItl9nAdIoH3xfpJ7FBiK8yXrmeebgA8Bs911ngL+EOgL4X8M9wP0B7/la3g/AX8aKAUm41R7PI2bVHzi+LUb11ycL/PMXl6nR3F+HDLcbXcBt/QWp9+2F+JU8XzHfc2WAi28n2xOxOxOf5IPJ+AVwHBglhvnave8MnGqjW72O9ZPcL7MF+Akuunu8p+5+xrpnsuzwA/8tv2hu216gHPp9TX1iXVqL6/DxcBm9/k5wB7gHZ9lPT+i09yYL3Nfr393j9nzI7Mf2IiTINKBIpykc74b90/c8+j5nLwF3Og+Hwac1cf75J+A38VJYiNxEs5tvWw71Y03FcjFKWz8rJd1rwDeA7JwkvFMIG8A709vn6Pf4X5P3ekE9zjfAlLc92wvcIXP96fV3U8iznf7bXdZIrAJ54diKJAGnOsuu8Z9P2bi/Ph9E3jzlHORFwnQ6we9J+C3cUuEfDABfwcnEX3oS+a/L95PcpMDzPNNwPf6LC/C+RVP9P9C+B+D/hPwauBzPsum45QSknziGOuz/F3gugDnlYiT9Ip85v0rsMbni9FfAj7ec87uvErcREBwCXiRz/R7wJ0+0z/G/bLz/pd0qM/yJ4D/wPmyHwOm+Cw7G9jns207vZTy+ntNfWLtLQH3lHKzgbtwSunlOEnx28B97nr/ATzhs10Czo/0hT6fgU/7LP8W8LjP9FD3PHo+J6+6+8/p57vwgffRPc4NPtM/Ah4M8nt1DbChl2UX4/yAnwUk+MwP5v3p63P0Oz6YgM8EDvod+27gf3y+Py/5ffeO+xy3yvdYPus9j1v48Hl/WjjFUnA81wEHUoBTxeDvP3F+/f4uIntF5K4g9lU2gOUHcH7dc4KKsm/57v58952Ec9Gxh2+rhRYCXyDMwSlB+O+rYACx1KhqZxDH6s1Rn+fHA0z77qtOVY/5TB/AeS1ygSHAeyJSLyL1wAvu/B5VqtraRxzBvKYBqepxnKqAC3BKq68AbwKL3HmvBDqGOheDy/jg6+37mcn3nXbPvcZn+S04peodIrJWRK7qL1YfwXw+EJFRIvK4iBwSkUbgD/TyGVbVl4FfAPcDR0XkIREZTnDvz0A+RxOA/J59ufv7On1//tPcevVxwAG/Y/nu9+c++6zF+fEYyPfhQywBu0TkDJwX83X/ZarapKpfUdXJwNXAv4nIJT2Le9llb/N7jPN5Ph6nRFWNUxoY4hNXIh/8MPa338M4HxbffXfyweQVjGo3Jv99HRrgfnrzgfMExpzi/kaIyFCf6fE4r0U1TrKepapZ7iNTVX2/wOF+TV/BKQHOA9a601fg1NO+GugYIiI4nxHf19s3zgp8PkMiMgSnlO2sqLpbVa8HRuFUr/zZ7/UJhR+4Mc1R1eHADThJKSBVvU9VT8epUpoGfI3g3p+++L93ZTil5yyfR4aqLg1iX2XA+F4ucpYB/+q333RVfTPIOAOK+wQsIsPd0sHjOP/abw6wzlUiMtX9UjTiNF3rchcfxalnGqgbRKTI/eJ8B/izqnbh/JuWJiJXikgyTl1Tqs92R4GJItLbe/dH4MsiMklEhgH/D/hTL7/qvXJjeQL4vohkiMgE4N9wSjmhsBH4JxEZIiJTcUpsp+rbIpIiIucBVwFPuiXJXwM/FZFRACJSICJXDGC/p/qavgLcBGxT1Xbc6hecRFHlrvMEcKWIXOK+71/BqQLq7Qv+Z+AqETlXRFJwPkMnPhMicoOI5LrnX+/O7vrwbk5JBk49dL2IFOAk1IBE5AwROdM9t2M41TJdIXh//L9/7wKNInKniKSLSKKIzHYLWP15F+eH7V4RGSoiaSKyyF32IHC3iMxyY8wUkY8GGWOv4jkBPysiTTi/bN/AuYjxqV7WLQRewvmwvQX8UlXXuMt+AHzT/dfkqwM4/u9x6q+O4FT2fwFAVRtwrv7/Bqf0cwynzrDHk+7fGhFZH2C/D7v7fhXnCnAr8PkBxOXr8+7x9+L8Z/C/7v5D4ac4dZZHgUdwrqCfiiM4V6cPu/u6TVV3uMvuxKlCetv9V/klBtb29VRf0zdx6oJ7Srvb3H30TKOqO3FKkP+NUyq8GqepZHugHarqVuB2nPekAufcfT8ni4GtItIM/Bynnr+vapaT8W1gPk6rhr/hXJzszXCcRFvH+y03/stddirvz2+BIvf791e34HA1UIzzXlXjfJcy+9uRz7ZTcS7IlwP/4i77C85/Eo+7MW4BTrn9fc+VfGOMMYMsnkvAxhjjKUvAxhjjEUvAxhjjEUvAxhjjkbjo1GPx4sX6wgsveB2GMSb29doOOpC4KAFXV1d7HYIxxnxIXCRgY4yJRJaAjTHGI5aAjTHGI5aAjTHGI5aAjTHGI5aAjTHGI5aAjTHGI5aAjTHGI2FNwCKyWER2ikhpoGF8xHGfu7xEROb7LU8UkQ0i8pzPvJEi8qKI7Hb/jgjnORhjTLiELQG7Q+ncj9NpcRFwvYgU+a22BKez80KcIaYf8Fv+RZxRWX3dBaxW1UKcwRKDGZ/NGGMiTjhLwAuBUlXd6/bq/ziw3G+d5cCj6ngbyBKRPAARGQtcidObvf82j7jPH8EZidUYY6JOOBNwAR8cxbWcD48g2tc6PwP+Hej222a0qlYAuH9HBTq4iNwqIutEZF1VVVWgVYwxxlPh7A0tUK9A/uMfBVzHHSSzUlXfE5ELT+bgqvoQ8BDAggULbNylCFPRcJxXd1WxqbyBysY2hqUmMrsgk2Vz8xk1PM3r8IwZFOFMwOV8cOj1sTgDJgazzj8Dy0RkKc6AlcNF5A+qegNwVETyVLXCra6oDNsZmJDq7lZW76jk4df38dbeGgAy05PJy0yjqbWTv248zH+u2slXL5/OLedOIiFhQD37GRN1wpmA1wKFIjIJZ3Tf64CP+62zArhDRB4HzgQa3GqFu90Hbgn4q27y7dnmZuBe9+8zYTwHEwKqynMlFfz0pV3srTpGQVY6/3bZNBbPHkPhqGGIOIl2T1UzP1i5g++v3M6+mmN8/5rZJ5YZE4vCloBVtVNE7gBWAYnAw6q6VURuc5c/CKwEluIMSd1C78PC+7oXeEJEbsEZOvqj4YjfhMb+6mP8xzNbeG13NTPzhnPf9fNYOnsMSYkfvvwwJXcYv77pdP5z1U5+uWYPI4ek8NUrBjJ6vDHRJS6GpV+wYIGuW7fO6zDizp/fK+cbf9lMcmICX7tiOjecNYHEIKoVVJU7nyrhyffKeewzZ3LOlJxBiNaYkLARMYy3VJV7n9/BV5/cxPzxI1j9lQu4+ZyJQSVfABHhnmWzmJQ9lK89WUJrR1eYIzbGG5aATUipKt96ZisPvrKHT5w5nt/fspDRJ9GqYUhKEt+/9jQO1R/nN6/tDUOkxnjPErAJqV+8XMrv3z7AredP5nvXzA5Y1xuss6dkc8Ws0fxyzR6qmtpCGKUxkcESsAmZJ9eV8eMXd/FP8wq4e8mMkLRguHPxDFo7unj4jX0hiNCYyGIJ2ITEW3tquPvpzZw7NYd7PzInZM3HJucOY+lpefz+rQM0HO8IyT6NiRSWgM0pq2pq4wuPb2B89hAeuGE+KUmh/Vh97sKpNLd18r/vHAzpfo3xmiVgc0q6upUv/2kjjcc7+OUn5pORlhzyYxTlD+fsydn877sH6O6O/WaTJn5YAjan5Bcvl/J6aTXfWT6LGWOGh+04Hz9zPGW1x3mttDpsxzBmsFkCNidt/cE6fr56F9fOK+BjC8b1v8EpuGLWGLKHpvDY2wfCehxjBpMlYHNSjrd38dUnNpGXmc53ls8Ke58NKUkJfHTBOFbvqORIQ2tYj2XMYLEEbE7KD1/Ywd7qY/znR+eEpd43kOsXjqOrW/nT2rL+VzYmClgCNgP2Rmk1v3tzP588Z+Kg9tMwIXso5xXm8Pjag3TZxTgTAywBmwFpbO3ga09uYnLOUO5cPGPQj3/dGeOpaGjlbbc/YWOimSVgMyDffXYbRxpb+fHH5pKekjjox79k5igyUpP4y4ZDg35sY0LNErAJ2hul1Tz5Xjm3XTCFeeNHeBJDWnIii2eP4YUtR6yXNBP1LAGboLR2dPGNv2xmYvYQvnBJoaexXDuvgOa2Tl7aftTTOIw5VZaATVB+8XIp+2ta+P61p5GWPPhVD77OnJzNmOFp/NWqIUyUswRs+rX7aBO/enUP/zSvgEVTvR+dIjFBWFacz5qdVdQea/c6HGNOmiVg0697n99BWnIi37hyptehnHBNcQGd3crfNld4HYoxJ80SsOnTO3trWL2jks9dOJXsYaleh3PCzLwMpo/OsGoIE9UsAZteqSr3vrCDMcPT+NSiiV6H8wEiwvJ5+bx3oI6DNS1eh2PMSbEEbHq1autRNhys58uXFXp+4S2Q5cUFADyz0UrBJjpZAjYBdXUr/7lqB1NHDeMj88d6HU5ABVnpnDlpJH/ZeAhVuzXZRB9LwCagF7cdYU/VMb50aeEpDawZbsuLC9hbdYythxu9DsWYAYvcb5bxjKry4Ct7GT9yCEtm53kdTp+WzB5DUoLw7KbDXodizIBZAjYf8t6BOjaW1fN/zptEYkJ4+/k9VSOGpnDBtFxWbDpswxWZqGMJ2HzIH94+QEZqEh85PTLrfv0tK86noqGVdQfqvA7FmAGxBGw+oPZYOyu3HOHa+QUMSUnyOpygXDpzNOnJidYawkSdsCZgEVksIjtFpFRE7gqwXETkPnd5iYjMd+enici7IrJJRLaKyLd9trlHRA6JyEb3sTSc5xBvnl5fTntnNx8/c7zXoQRtaGoSlxaNZuXmCjq6ur0Ox5ighS0Bi0gicD+wBCgCrheRIr/VlgCF7uNW4AF3fhtwsarOBYqBxSJyls92P1XVYvexMlznEI+eWn+IueOywjrCcTgsm5tPXUsHr9uoySaKhLMEvBAoVdW9qtoOPA4s91tnOfCoOt4GskQkz51udtdJdh92hSXMdh5pYntFI9cW53sdyoBdMC2XzPRkVmy01hAmeoQzARcAvqMnlrvzglpHRBJFZCNQCbyoqu/4rHeHW2XxsIgE7BlcRG4VkXUisq6qquoUTyU+/HXjIRIThKvmRl8CTklKYMnsMfx96xGOt1tH7SY6hDMBB2q/5F+K7XUdVe1S1WJgLLBQRGa7yx8ApuBUTVQAPw50cFV9SFUXqOqC3NzcgUcfZ1SVFRsPc15hDjkR1OnOQCybm8+x9i5W77CO2k10CGcCLgfG+UyPBfz/P+x3HVWtB9YAi93po25y7gZ+jVPVYU7R1sONHKo/ztIIv/GiL2dOzmZURqpVQ5ioEc4EvBYoFJFJIpICXAes8FtnBXCT2xriLKBBVStEJFdEsgBEJB24FNjhTvtmiGuBLWE8h7ixausREgQuLRrtdSgnLTFBuGqO01F7w/EOr8Mxpl9hS8Cq2gncAawCtgNPqOpWEblNRG5zV1sJ7AVKcUqzn3Pn5wH/EJESnET+oqo+5y77kYhsdpddBHw5XOcQT17YcoSFk0YycmiK16GckmXF+bR3dbNqyxGvQzGmX2Ftae82EVvpN+9Bn+cK3B5guxJgXi/7vDHEYca9/dXH2F3ZzMfP9G8lGH3mjs1kQvYQVmw6zMfOGNf/BsZ4yO6EM7y622klcvGMUR5HcupEhGVz83lzTzWVTa1eh2NMnywBG17dVc34kUOYkD3U61BCYtncfLoV/lZi48WZyGYJOM51dHXz1p5qziv0frTjUCkcncHMvOGssC4qTYSzBBznNhys51h7F+cVxlZb6WVz89lwsN7GizMRzRJwnHttdxWJCcLZU7K9DiWkrp7rtFZ8tsRKwSZyWQKOc6/urqZ4XBaZ6clehxJSY0cMYcGEEXZThololoDjWH1LOyXl9TFV/+trWXE+O482seOIjRdnIpMl4Dj2RmkNqsRc/W+PpaflkZggVgo2EcsScBx7bXcVGWlJzB2b6XUoYZEzLJVFU3NYsemwDVtvIpIl4Dj21t4azpqcHdHDzp+qZXPzKa87zvqD9V6HYsyHxO43z/SpsrGVAzUtLJw40utQwuqKWaNJSUqwYetNRLIEHKfW7ndGED5jUmwn4Iy0ZC6ZMYrnSg7TaePFmQhjCThOrd1fS3pyIrPyo2vst5OxbG4+1c3tvLW3xutQjPkAS8Bx6t19tcyfkEVyDNf/9rhoxigyUpOsNYSJOLH/7TMf0tjawfYjjZwR4/W/PdKSE7l81hhe2HKE1g4bL85EDkvAcei9A3WoEvMX4HwtK86nqa2TNTttgFYTOSwBx6G1+2pJShCKx2d5HcqgWTQlm+yhKazYdMjrUIw5wRJwHFq7v5ZZBZkMSQnrgCgRJSkxgSvn5LF6eyVNrTZenIkMloDjTGtHF5vKGlg4cYTXoQy6ZXPzaevs5u9bbdh6ExksAceZzYcaaO/qjpsLcL7mjx9BfmYaz9uAnSZCWAKOM+8dcG7AOH1C/JWAExKEy2eN4bXdVbS0d3odjjGWgOPNhoN1TMweQvawVK9D8cTls0bT1tnNq7uqvQ7FGEvA8URVWX+wnnnj46/022PhxJFkpifz921WDWG8Zwk4jhyqP05VUxvz46j5mb+kxAQumTmK1dsrrW8I4zlLwHGkp0vGeC4BA1xeNIaG4x28u7/W61BMnLMEHEc2HKwjLTmBGWMyvA7FU+dPyyE1KcGaoxnPWQKOIxsO1jNnbFZMd8AejCEpSZw/LZcXtx21kTKMp+L7mxhHWju62Hq4gXlxXP/r67Ki0RyqP862Chuw03gnrAlYRBaLyE4RKRWRuwIsFxG5z11eIiLz3flpIvKuiGwSka0i8m2fbUaKyIsistv9G98VmkHaeriRji5lfpzX//a4aPooAP6xo9LjSEw8C1sCFpFE4H5gCVAEXC8iRX6rLQEK3cetwAPu/DbgYlWdCxQDi0XkLHfZXcBqVS0EVrvTph8bDjo3YFgJ2JGbkcrcsZm8bAnYeCicJeCFQKmq7lXVduBxYLnfOsuBR9XxNpAlInnudLO7TrL7UJ9tHnGfPwJcE8ZziBkbDtYzdkQ6ozLSvA4lYlw4fRQbyuqpPdbudSgmToUzARcAZT7T5e68oNYRkUQR2QhUAi+q6jvuOqNVtQLA/Tsq0MFF5FYRWSci66qqrA/Y9Qfr4r75mb+LZ4xCFV7dZZ8P441wJmAJMM//knOv66hql6oWA2OBhSIyeyAHV9WHVHWBqi7Izc0dyKYxp6LhOBUNrcwbl+V1KBHltIJMcoal8I+dVg1hvBHOBFwOjPOZHgv4D8rV7zqqWg+sARa7s46KSB6A+9e+Pf3Y6N6AMT8OO+DpS0KCcMG0Ubyyq4qubmuOZgZfOBPwWqBQRCaJSApwHbDCb50VwE1ua4izgAZVrRCRXBHJAhCRdOBSYIfPNje7z28GngnjOcSE9QfrSElKoCgv9kdAHqiLZuRS39Jx4iKlMYMpbEMiqGqniNwBrAISgYdVdauI3OYufxBYCSwFSoEW4FPu5nnAI25LigTgCVV9zl12L/CEiNwCHAQ+Gq5ziBUbDtZzWkEmKUnW7NvfeYW5JCYI/9hZyYI47CPZeCusY9Ko6kqcJOs770Gf5wrcHmC7EmBeL/usAS4JbaSxq72zm5JDDdx01gSvQ4lImenJnD5hBC/vqOJrV8zwOhwTZ6xIFOO2VzTS3tltLSD6cPGMUWyvaORIQ6vXoZg4Ywk4xtkNGP07cVectYYwg8wScIzbUFbP6OGp5GXaDRi9mTZ6GAVZ6XZXnBl0loBj3MayeuaNG4FIoCbXBkBEuHB6Lm+UVtPW2eV1OCaOWAKOYTXNbRyoabHqhyBcOH0ULe1drD9Q73UoJo5YAo5hG8vqARsBIxhnTR5JUoLw6m67LdkMHkvAMWzDwXoSE4TTCjK9DiXiZaQlM3/8CF6zBGwGkSXgGLaxrJ4ZYzJIT0n0OpSocF5hDlsONVLT3OZ1KCZOWAKOUV3d6lyAs/rfoJ0/zem06fXSao8jMfHCEnCM2lPVTHNbJ/PGWf1vsGYXZJI1JJnXdlsCNoPDEnCM6rkBo9hKwEFLTBAWTc3htd1VNlinGRSWgGPUhoP1ZKYnMyl7qNehRJXzC3M42tjGrqPN/a9szCmyBByjNpbVUzwui4QEuwFjIM4tdOqBrTWEGQyWgGNQc1snO4822QW4k1CQlc6U3KG8avXAZhBYAo5BJWX1qNoNGCfr/Gm5vLO3htYOuy3ZhJcl4Bi0wb0DrnhslqdxRKvzC3Np6+xm3X4bJcOElyXgGLThYB2Tc4eSOSTZ61Ci0pmTR5KcaLclm/CzBBxjVPVED2jm5AxJSWLBhJE2XL0JO0vAMaa87jjVze12Ae4UnTcthx1HmqhstFEyTPhYAo4x63tuwBiX5W0gUe78Qrst2YSfJeAYs+FgPWnJCcwYk+F1KFGtKG842UNT7LZkE1aWgGPMxrJ65ozNIinR3tpTkZAgnFvo3Jbc3W23JZvwsG9pDGnr7GLb4Uar/w2R8wpzqW5uZ/uRRq9DMTHKEnAM2Xq4kfaubmsBESLnTs0B4M3SGo8jMbHKEnAM2XCwHrAh6ENlTGYak3OH8sYeqwc24WEJOIZsOFhHfmYao4fbEPShsmhKDu/uq6W9s9vrUEwMsgQcQ5wRMKz6IZQWTc2mpb3rxACnxoSSJeAYUdnUSnndcat+CLGzJ+cgAm9Ye2ATBmFNwCKyWER2ikipiNwVYLmIyH3u8hIRme/OHyci/xCR7SKyVUS+6LPNPSJySEQ2uo+l4TyHaLHRrf+1GzBCK3NIMqcVZPKm1QObMAhbAhaRROB+YAlQBFwvIkV+qy0BCt3HrcAD7vxO4CuqOhM4C7jdb9ufqmqx+1gZrnOIJhvK6klKEGbbEPQhd86UHDYcrOdYW6fXoZgYE84S8EKgVFX3qmo78Diw3G+d5cCj6ngbyBKRPFWtUNX1AKraBGwHCsIYa9TbVFbPzLzhpCXbEPShtmhqNp3dyrv7a70OxcSYcCbgAqDMZ7qcDyfRftcRkYnAPOAdn9l3uFUWD4tIwKtOInKriKwTkXVVVbHdq1V3t7L5UANzxlrpNxwWTBhJSmICb1o9sAmxcCbgQIOR+d/T2ec6IjIMeAr4kqr23I70ADAFKAYqgB8HOriqPqSqC1R1QW5u7gBDjy4Haltoau20BBwm6SmJzJ+QxRt2Q4YJsXAm4HJgnM/0WOBwsOuISDJO8n1MVZ/uWUFVj6pql6p2A7/GqeqIayXl9QCcVpDlaRyxbNGUHLZVNFLT3OZ1KCaGhDMBrwUKRWSSiKQA1wEr/NZZAdzktoY4C2hQ1QoREeC3wHZV/YnvBiKS5zN5LbAlfKcQHUrKG0hNSmDa6GFehxKzznFvS35rr5WCTeiELQGraidwB7AK5yLaE6q6VURuE5Hb3NVWAnuBUpzS7Ofc+YuAG4GLAzQ3+5GIbBaREuAi4MvhOodosbm8gVn5w60HtDCaOzaTYalJVg1hQiopnDt3m4it9Jv3oM9zBW4PsN3rBK4fRlVvDHGYUa2rW9lyuIGPLRjX/8rmpCUlJnDW5JHWHtiElBWZotyeqmZa2rs4zdr/ht05U3I4UNNCeV2L16GYGGEJOMqVlDcAMHecJeBwW2TdU5oQswQc5TaX1zM0JZFJOXYBLtymjR5GzrBU657ShIwl4ChXcqiBWQWZJCYErDI3ISQinDMlmzdKa3AuXxhzaiwBR7GOrm62HW5kjtX/DppFU7Opbm5j19Fmr0MxMcAScBTbdbSJts5u5lgPaIPmnClOPbB1T2lCwRJwFNt22Lk7e1b+cI8jiR/jRg5hQvYQa45mQsIScBTbXtFEWnICE7OHeh1KXDlnSg7v7K2ls8uGKTKnJqgELCJPiciVImIJO4Jsr2hk+pjhdgFukC2amk1TWyclhxq8DsVEuWAT6gPAx4HdInKviMwIY0wmCKrK9iONFOVleB1K3Dl7cjaAdU9pTllQCVhVX1LVTwDzgf3AiyLypoh8yu21zAyyioZW6ls6mJln9b+DLXtYKjPzhvO6JWBzioKuUhCRbOCTwGeADcDPcRLyi2GJzPRpe4VzAc4SsDfOK8xh/YF6WtptmCJz8oKtA34aeA0YAlytqstU9U+q+nnAbsHyQE8CnjHGqiC8sGhqDu1d3azdX+d1KCaKBVsC/o2qFqnqD1S1AkBEUgFUdUHYojO92l7RxLiR6WSkWQ2QF86YOIKUxARrD2xOSbAJ+HsB5r0VykDMwGyvaGTmGKt+8MqQlCTmT8ji9d2WgM3J6zMBi8gYETkdSBeReSIy331ciFMdYTzQ0t7JvppjVv/rsXOn2jBF5tT0VwK+AvgvnLHafoIzAOaPgX8Dvh7e0Exvdh5pQtUuwHntRPeUe6x7SnNy+hwRQ1UfAR4RkY+o6lODFJPpx/aKJgCKLAF7as7YLDLSknijtJqr5+Z7HY6JQn0mYBG5QVX/AEwUkX/zX+4/YKYZHNsrGhmWmsTYEelehxLXEhOc7ilf212NquKMJWtM8PqrgujpZGAYkBHgYTywvaKRGWMySLBbkD137tQcDtUf52CtDVNkBq6/KohfuX+/PTjhmP6oKruONnGV/csbEXrqgV/bXc0E6xTJDFCwN2L8SESGi0iyiKwWkWoRuSHcwZkPq2puo7G1k8JRdv9LJJiUM5T8zDRrD2xOSrDtgC9X1UbgKqAcmAZ8LWxRmV6VuiMxTLUEHBFEhEVTc3hzTw1d3TZMkRmYYBNwz+1WS4E/qmptmOIx/SitchJw4Sirgo8U5xbm0HC8g62HrXtKMzDBJuBnRWQHsABYLSK5QGv4wjK9Ka1sZlhqEqOHp3odinH1DFP0mt0VZwYo2O4o7wLOBhaoagdwDFgezsBMYLuPNjN11DBr8hRBcjNSKcobziu7qrwOxUSZPltB+JmJ0x7Yd5tHQxyP6UdpVTMXTMv1Ogzj58Lpufzq1b00tnYw3DpIMkEKthXE73FuST4XOMN99NsLmogsFpGdIlIqIncFWC4icp+7vERE5rvzx4nIP0Rku4hsFZEv+mwzUkReFJHd7t8RQZ5r1Gto6aCqqc0uwEWgi2aMoqtbrXMeMyDBloAXAEWqGvRlXhFJBO4HLsNpObFWRFao6jaf1ZYAhe7jTJyhj84EOoGvqOp6EckA3hORF91t7wJWq+q9blK/C7gz2LiiWWmVcwuyNUGLPPPGZTE8LYk1OytZelqe1+GYKBHsRbgtwJgB7nshUKqqe1W1HXicD9cbLwceVcfbQJaI5KlqhaquB1DVJmA7UOCzzSPu80eAawYYV9QqrbQmaJEqKTGB86blsmZnFQMop5g4F2wCzgG2icgqEVnR8+hnmwKgzGe6nPeTaNDriMhEYB7wjjtrdE+n8O7fUUGeQ9QrrWwmJSmBsSOsJ9BIdNH0UVQ2tbHNHa3EmP4EWwVxz0nsO9Blev+iQZ/riMgw4CngS+6NIMEfXORW4FaA8ePHD2TTiLW7spkpucNsGPoI1XNxdM3OKmblZ3ocjYkGwTZDewVnNORk9/laYH0/m5UD43ymxwKHg13HHW35KeAxVX3aZ52jIpLnrpMHVPYS80OqukBVF+TmxkargdLKZqt+iGC5GamcVpDJmp0BP5LGfEiwrSD+D/Bn4FfurALgr/1sthYoFJFJIpICXAf4V1usAG5yW0OcBTSoaoU4jVx/C2wP0OXlCuBm9/nNwDPBnEO0a2nvpLzuuF2Ai3AXTc/lvQN1NLR0eB2KiQLB1gHfDiwCGgFUdTf91L2qaidwB7AK5yLaE6q6VURuE5Hb3NVWAnuBUuDXwOfc+YuAG4GLRWSj+1jqLrsXuExEduO0sLg3yHOIanurjgF2AS7SXTxzNN0K/7BSsAlCsHXAbara3nP3lXszRr+XelV1JU6S9Z33oM9zxUnu/tu9TuD6YVS1BrgkyLhjhrWAiA5zCjIZPTyVv287wjXz/K85G/NBwZaAXxGRr+MMznkZ8CTwbPjCMv72VjWTIDAh21pARLKEBOHSmaNZs7OK1o4ur8MxES7YBHwXUAVsBv4Vp1T7zXAFZT5sf00L+VnppCYleh2K6cfls8bQ0t7Fm3vsrjjTt6CqIFS1W0T+CvxVVa3HEQ/srznGpBwbcSEanD05m4zUJP6+9SgXzxjtdTgmgvVZAnZbJ9wjItXADmCniFSJyLcGJzwDzjBE+6qPMdGGvIkKKUkJXDhjFC9tP2qdtJs+9VcF8SWcFglnqGq2qo7E6athkYh8OdzBGUddSwdNrZ1W/xtFLi8aTXVzOxsO1nkdiolg/SXgm4DrVXVfzwxV3Qvc4C4zg2B/jdMEzaogoseF03NJThRWbT3idSgmgvWXgJNV9UNXEtx6YOv0dJDsr3YSsI26Gz0y0pJZNDWH57ccsc55TK/6S8DtJ7nMhND+mhYSBMaNTPc6FDMAV56WR3ndcTaV21hxJrD+EvBcEWkM8GgCThuMAI1TArYmaNHn8lljSElM4LlN/l2gGOPoMwGraqKqDg/wyFBVq4IYJAesCVpUykxP5vxpOfxtcwXd1hrCBBDsjRjGI9YELbpdNSefioZW1ltrCBOAJeAIV9/SQaM1QYtalxaNJjUpgedKKrwOxUQgS8ARbp81QYtqw1KTuGj6KP62ucJuyjAfYgk4wlkTtOh31dw8qpraeHdfrdehmAhjCTjCWRO06HfxjFGkJyfyXIm1hjAfZAk4wlkTtOg3JCWJS2aO4vktR+js6vY6HBNBLAFHOGuCFhuumpNP7bF23txT43UoJoJYAo5g1gQtdlw4PZdhqUlWDWE+wBJwBGs4bk3QYkVaciKXzxrN81uO0NZpI2UYhyXgCFZedxyAsSMsAceC5cUFNLV2smanjWlgHJaAI1hZbQtgLSBixaIp2WQPTWHFRquGMA5LwBGsrK4nAVsJOBYkJSZw5Zw8Xtp+lOa2Tq/DMRHAEnAEK6s9TmZ6MsPTrN+jWLG8OJ+2zm7+bh21GywBR7SyuhbGjrDqh1gyf/wICrLSWWFdVBosAUe0stoWxtkFuJgiIiwrzue13dXUNLd5HY7xmCXgCKWqlNcdtwtwMWh5cT5d3crKzdZDWryzBByhqpraaOvstgtwMWjGmOFMGz3MqiGMJeBIdaIFhFVBxKTlxQWs3V9Hufs+m/gU1gQsIotFZKeIlIrIXQGWi4jc5y4vEZH5PsseFpFKEdnit809InJIRDa6j6XhPAevlNU6N2FYFURsWjY3H4BnN1k1RDwLWwIWkUTgfmAJUARcLyJFfqstAQrdx63AAz7Lfgcs7mX3P1XVYvexMqSBR4iemzDsLrjYNG7kEOaNz7JqiDgXzhLwQqBUVfeqajvwOLDcb53lwKPqeBvIEpE8AFV9FYjbHqzL6lrIzUglLdm6oYxVy+fms72ikV1Hm7wOxXgknAm4ACjzmS535w10nUDucKssHhaREYFWEJFbRWSdiKyrqoq+e+/L644zztoAx7Qr5+STINityXEsnAlYAszzHxQrmHX8PQBMAYqBCuDHgVZS1YdUdYGqLsjNze1nl5GnrK7FWkDEuNyMVBZNzWHFpsOo2nhx8SicCbgcGOczPRbw/6kPZp0PUNWjqtqlqt3Ar3GqOmJKZ1c3h+tb7S64OLBsbj4Ha1vYWFbvdSjGA+FMwGuBQhGZJCIpwHXACr91VgA3ua0hzgIaVLXPy8I9dcSua4Etva0brSoaWunqVmuCFgeumD2GlKQEnrFqiLgUtgSsqp3AHcAqYDvwhKpuFZHbROQ2d7WVwF6gFKc0+7me7UXkj8BbwHQRKReRW9xFPxKRzSJSAlwEfDlc5+AV6wUtfgxPS+bi6aN4rsSGrY9HSeHcudtEbKXfvAd9nitwey/bXt/L/BtDGWMkKu9pA2wl4LiwvDifF7Ye4a09NZxbmON1OGYQ2Z1wEaiszhmKPi8rzetQzCC4aMYohqUm8czGQ16HYgaZJeAIVFbbQl5mOsmJ9vbEg7TkRK6YNYYXth6htcPGi4sn9g2PQGXWC1rcWV6cb+PFxSFLwBHI+gGOP+dMySZnWAorNlk1RDyxBBxhWju6qGxqsxYQcSYpMYErT8tj9fZKmlo7vA7HDBJLwBGmZyh6q4KIP8uKC9zx4o56HYoZJJaAI4z1Axy/5o/PYuyIdJ6xHtLihiXgCNNTArZuKOOPiLBsbj5vlFZTbePFxQVLwBGmvLaFlKQERmWkeh2K8cDy4gIbLy6OWAKOMGV1LYzNSichIVBHcSbWTR+TwfTRGdZFZZywBBxhymqPM9ZaQMS1ZcX5rDtQd2JUFBO7LAFHmLK6FuuIPc6dGC+uxErBsc4ScARpau2gvqXD2gDHuXEjhzB/fJZVQ8QBS8AR5P0WEFYCjnfLiwvYcaSJnUdsvLhYZgk4gpy4CcOaoMW9paflOePF2a3JMc0ScAR5fyh6KwHHOxsvLj5YAo4g5XXHGZKSyMihKV6HYiLA8uICymqPs8HGi4tZloAjSHldC2NHpCNibYANXDFrNClJCXYxLoZZAo4gZXXH7RZkc0JGWjKXzHDGi+vs6vY6HBMGloAjSLm1ATZ+lhfnU93cxlt7a7wOxYSBJeAI0dDSQVNrp5WAzQdcOH0UGalJNmx9jLIEHCHeH4reSsDmfWnJiVwxewyrtth4cbHIEnCEKK/raYJmJWDzQcuL82lq62TNzkqvQzEhZgk4QthNGKY3Z092xouzaojYYwk4QpTVtpCRmsTw9CSvQzERJikxgavm5LN6h40XF2ssAUeI8jqnG0prA2wCuXpuPu2d3ayy8eJiiiXgCFFed9xuQTa9OjFe3EbrGyKWWAKOAKrq9gNs9b8mMBFheXE+b+6poarJxouLFWFNwCKyWER2ikipiNwVYLmIyH3u8hIRme+z7GERqRSRLX7bjBSRF0Vkt/t3RDjPYTDUtXTQ0t5lJWDTp2Vzbby4WBO2BCwiicD9wBKgCLheRIr8VlsCFLqPW4EHfJb9DlgcYNd3AatVtRBY7U5HtZ5e0KwjdtOXnvHinrORMmJGOEvAC4FSVd2rqu3A48Byv3WWA4+q420gS0TyAFT1VaA2wH6XA4+4zx8BrglH8IPJOmI3wbp6bh5r99dxuP6416GYEAhnAi4Aynymy915A13H32hVrQBw/44KtJKI3Coi60RkXVVV1YACH2xlddYPsAnOVXOc8eL+VmLVELEgnAk4UHsq/56lg1nnpKjqQ6q6QFUX5ObmhmKXYVNe10LWkGQy0pK9DsVEuIk5QzmtINMG7IwR4UzA5cA4n+mxgP+nJph1/B3tqaZw/0b9/ZlltdYEzQTv6rl5lJQ3sL/6mNehmFMUzgS8FigUkUkikgJcB6zwW2cFcJPbGuIsoKGneqEPK4Cb3ec3A8+EMmgvlFsTNDMAV7rVEHYxLvqFLQGraidwB7AK2A48oapbReQ2EbnNXW0lsBcoBX4NfK5nexH5I/AWMF1EykXkFnfRvcBlIrIbuMydjlqqajdhmAEpyEpnwYQRPGf1wFEvrB0PqOpKnCTrO+9Bn+cK3N7Lttf3Mr8GuCSEYXqqqrmNts5ua4JmBuTqufn83xVb2XW0iWmjM7wOx5wkuxPOY9YEzZyMJaeNIUHguU1WDRHNLAF77MRNGFYHbAZgVEYaZ03O5tmSChu2PopZAvZYTwm4wErAZoCunpvPvupjbD3c6HUo5iRZAvZYeV0LOcNSGJJi/QCbgVk8awxJCcKzVg0RtSwBe6y87jgFVv1gTsKIoSmcV5jDc1YNEbUsAXvsQE0LE6wFhDlJV8/N51D9cdYfrPc6FHMSLAF7qL2zm/K6FiZmWwI2J+eyotGkJCWwwjpqj0qWgD1UXtdCt8KE7KFeh2KiVEZaMpfNHM2zJRW0d3Z7HY4ZIEvAHjpQ4zRBm5hjJWBz8v5pfgG1x9p5ZVdk9/pnPswSsIf21zidqVgJ2JyK86flkjMshafeK/c6FDNAloA9dKDGGYo+e2iK16GYKJacmMCyuQWs3nGU+pZ2r8MxA2AJ2EP7qo8xIceGojen7iOnF9DRpTxrHfREFUvAHjpQc8yqH0xIFOUNZ8aYDKuGiDKWgD3S0dVNed1xa4JmQkJE+Mj8sWwsq6e0ssnrcEyQLAF75HD9cTq71UrAJmSumVdAUoLw+Ltl/a9sIoIlYI/sc4eTmZRjCdiERm5GKlfMGsNT68tp7ejyOhwTBEvAHulpAzzBqiBMCF2/cDx1LR2s2nrE61BMECwBe2R/zTGGpCSSOyzV61BMDDlnSjbjRw7hj+8e9DoUEwRLwB45UNPChOyh1gTNhFRCgnDdwnG8vbeWPVXNXodj+mEJ2CP7q49ZCwgTFv98+liSEoQ/vmOl4EhnCdgDnV3dlNW1MNEuwJkwGJWRxhWzxvDke+W0tHd6HY7pgyVgD1Q0tNLRpVYCNmHz6XMn0nC8w27MiHCWgD1gnfCYcJs/fgRzx2Xx8Bv76e620TIilSVgD+ytsjbAJrxEhFvOncS+6mO8vKPS63BMLywBe6C0spmM1CRGZVgTNBM+S2aPIT8zjd++vs/rUEwvLAF7oLSymamjh1kTNBNWyYkJ3HzORN7aW8OWQw1eh2MCsATsgd2VzUzNHeZ1GCYOXLdwPBmpSfzi5VKvQzEBWAIeZA0tHVQ3tzF1lCVgE36Z6cl8+txJvLD1CFsPWyk40oQ1AYvIYhHZKSKlInJXgOUiIve5y0tEZH5/24rIPSJySEQ2uo+l4TyHUCutcroKLBxtCdgMjk+fO4nhaUn87KXdXodi/IQtAYtIInA/sAQoAq4XkSK/1ZYAhe7jVuCBILf9qaoWu4+V4TqHcCitdG4PnZqb4XEkJl5kpifzmfMm8+K2o2wut1JwJAlnCXghUKqqe1W1HXgcWO63znLgUXW8DWSJSF6Q20al0spmUpMSKBiR7nUoJo58atFEMtOT+elLu7wOxfgIZwIuAHx7hi535wWzTn/b3uFWWTwsIiMCHVxEbhWRdSKyrqoqcobr3nm0mSm5w0hMsBYQZvBkpCVz2wVTeHlHpQ1fH0HCmYADZRj/W3J6W6evbR8ApgDFQAXw40AHV9WHVHWBqi7Izc0NKuDBsKOikZl5w70Ow8ShT587kUk5Q7lnxVbaOq3D9kgQzgRcDozzmR4LHA5ynV63VdWjqtqlqt3Ar3GqK6JCTXMblU1tzMyz+l8z+FKTErln2Sz2VR/j16/u9TocQ3gT8FqgUEQmiUgKcB2wwm+dFcBNbmuIs4AGVa3oa1u3jrjHtcCWMJ5DSO044rSAsBKw8coF03JZMnsMv/hHKWW1LV6HE/fCloBVtRO4A1gFbAeeUNWtInKbiNzmrrYS2AuU4pRmP9fXtu42PxKRzSJSAlwEfDlc5xBq2ysaAZgxxkrAxjv/cVURCSLc/fRm66jHY0nh3LnbRGyl37wHfZ4rcHuw27rzbwxxmINme0UTuRmpZNswRMZD+VnpfPPKIr7+l808/MY+PnPeZK9Dilt2J9wg2m4X4EyEuH7hOC4rGs29z+/gzdJqr8OJW5aAB0lbZxellc3MtOoHEwFEhB9/bC6Tcoby2cfWs6/6mNchxSVLwINkR0UT7V3dzB2X5XUoxgAwPC2Z3958BgkCt/xuLQ0tHV6HFHcsAQ+SkvJ6AOaMzfQ2EGN8jM8ewq9uXEBZXQuf+O3b1B5r9zqkuGIJeJBsKm8ge2gKBVl2C7KJLAsnjeShmxaw+2gz1z30FpWNrV6HFDcsAQ+STWX1zBmbaZ2wm4h00fRR/M+nzqC87jj//OBb7Dra5HVIccES8CBobuuktKrZ6n9NRDtnSg6PfeZMWtq7uOb+N3h+c4XXIcU8S8CDoKS8HlWYOzbL61CM6dO88SN47vPnMn1MBp99bD3fe26b9RsRRpaAB8G7+2oRgfkTAnbcZkxEGZOZxuO3nsWNZ03gN6/vY/kv3mDHkUavw4pJloAHwTt7aynKG05merLXoRgTlNSkRL57zWwe/uQCqpvbWPaLN/jNa3vt1uUQswQcZm2dXaw/WMeZk7K9DsWYAbt4xmhWfel8LpiWy/f+tp0bH36HiobjXocVMywBh1lJeQNtnd0snDTS61CMOSnZw1J56MbTufefTmPDwXoW/+w1u0AXIpaAw+ztPTUAloBNVBMRrls4nr994TwmZA/hs4+t5+6nS2hp7/Q6tKhmCTjM1uyqYnbBcEYOTfE6FGNO2aScofz5tnP47IVTeHxtGVf99+tsOWQDfZ4sS8BhVHusnQ0H67h4xmivQzEmZFKSErhz8Qweu+VMjrV18pEH3uTP75V7HVZUsgQcRq/sqqRb4ZIZo7wOxZiQO2dqDn/7wnnMG5/FV5/cxDf/upn2zm6vw4oqloDDaPX2SnKGpXJagXXAY2JTzrBU/nDLmdx6/mT+8PZBrnvoLY40WF8SwbIEHCYt7Z28vKOSS2eOIsGGoDcxLCkxga8vncn9H5/PjiNNXPXfr/H23hqvw4oKloDD5O9bj9LS3sW18wq8DsWYQXHlnDyeuX0Rw9OT+cRv3uGBNXvsxo1+WAIOk6fWlzN2RDpnTLTmZyZ+FI7O4JnbF7Fk9hh++MIObnlkLXXWx3CvLAGHwaH647xRWs218wqs+sHEnYy0ZP77+nl8d/ks3iit4cr7XmPt/lqvw4pIloDD4Dev7SVBhOsXjvc6FGM8ISLcePZEnvrsOSQlJvDRB9/i7qc327BHfiwBh1jdsXYef7eMZcX55NvoFybOnTY2k+e/eB6fOXcSf1p7kEt+sobfvLaXY212Bx1YAg65X64p5XhHF7ddMMXrUIyJCENTk/jmVUWsuONcCkdl8L2/bWfRD1/mhy/sYMuhBlTj90KdxMPJL1iwQNetWxf24+w80sTS+17jo6eP5d6PzAn78YyJRusP1vHAmj28vKOSrm5l3Mh0Fk3JYf6EEcwbl8Xk3GEkRu+1kwEFbgk4RFo7uviXX73FgdoWXv7Khdb3gzH9qD3WzovbjrBq61HW7a+lsdWplkhLTmD6mOEU5WVQlDecovzhTB8znGGpSR5HHBRLwP7CnYC7u5WvPLmJv2w4xEM3ns7ls8aE7VjGxKLubmVvdTMbyxrYXtHI9opGtlU0Uu9z0W5i9hBm5g0/kZTnjssiZ1iqh1EHNKAEHBU/KZGstaOLr/25hGc3HeYrl02z5GvMSUhIEKaOymDqqIwT81SVioZWJxkfdhLy9opGnt9y5MQ6E7OHcPqEkZw+YQSnTxhB4ahhUdX0M6wlYBFZDPwcSAR+o6r3+i0Xd/lSoAX4pKqu72tbERkJ/AmYCOwHPqaqdX3FEY4ScFe38uK2o/zg+e0cqGnh7iUzuPX8yTbsvDFh1tzWybbDjWw4WMd7B5xHjXuzR0ZaEvPGj+D08U5CLh6fNdhVF5FRBSEiicAu4DKgHFgLXK+q23zWWQp8HicBnwn8XFXP7GtbEfkRUKuq94rIXcAIVb2zr1hOJgGrKh1dSntXN60dXdQea+doYytltccpKa9nzc4qjjS2MilnKN+/djbnTMkZ0P6NMaGhqhysbWHd/jreO1jH+gN17DzahCokCEzMGcqEkUOYkD2UidlDGD08jcwhyWSlp5A1JJn05ESSEoXkxARSEhNOtQQdMVUQC4FSVd0LICKPA8uBbT7rLAceVedX4G0RyRKRPJzSbW/bLgcudLd/BFgD9JmAB2rB916k5lg7vf02ZaQlce7UHJbNzeeyotEkJVprPmO8IiJMyB7KhOyhfOT0sQA0tnaw8WA96w7UsftoE/trWnh3Xy3H2rv63V9igpCYIAxJSWTjty4Pa+zhTMAFQJnPdDlOKbe/dQr62Xa0qlYAqGqFiATsbFdEbgVudSebRWTnyZxEb7YAD/a/Wg5QHcrjhonFGVoWZ2h5Fqf83wGtngO8oKqLg90gnAk4UFHcv0zZ2zrBbNsnVX0IeGgg24SaiKxT1QVexhAMizO0LM7QirI4g06+EN474cqBcT7TY4HDQa7T17ZH3WoK3L+VIYzZGGMGTTgT8FqgUEQmiUgKcB2wwm+dFcBN4jgLaHCrF/radgVws/v8ZuCZMJ6DMcaETdiqIFS1U0TuAFbhNCV7WFW3isht7vIHgZU4LSBKcZqhfaqvbd1d3ws8ISK3AAeBj4brHELA0yqQAbA4Q8viDK2YjTMu7oQzxphIZO2njDHGI5aAjTHGI5aAw0REFovIThEpde/Yiwgi8rCIVIrIFp95I0XkRRHZ7f4d4WWMbkzjROQfIrJdRLaKyBcjMVYRSRORd0VkkxvntyMxTjemRBHZICLPRWqMACKyX0Q2i8hGEVnnzou4WN0bx/4sIjvcz+nZA43TEnAYuLdS3w8sAYqA60WkyNuoTvgd4N9W8S5gtaoWAqvdaa91Al9R1ZnAWcDt7msYabG2ARer6lygGFjstuiJtDgBvghs95mOxBh7XKSqxT7tfyMx1p/j3HgxA5iL89oOLE5VtUeIH8DZwCqf6buBu72OyyeeicAWn+mdQJ77PA/Y6XWMAWJ+BqdvkIiNFRgCrMe5azOi4sRpS78auBh4LpLfd5xOtnL85kVUrMBwYB9uQ4aTjdNKwOHR2y3WkeoDt3cDAW/v9oqITATmAe8QgbG6/9pvxLkp6EVVjcQ4fwb8O9DtMy/SYuyhwN9F5D23SwGIvFgnA1XA/7jVOr8RkaEMME5LwOFxyrdSG4eIDAOeAr6kqo1exxOIqnapajFOKXOhiMz2OKQPEJGrgEpVfc/rWIK0SFXn41Th3S4i53sdUABJwHzgAVWdBxzjJKpFLAGHRzC3YUeSiLy9W0SScZLvY6r6tDs7ImMFUNV6nN75FhNZcS4ClonIfuBx4GIR+QORFeMJqnrY/VsJ/AWnZ8VIi7UcKHf/2wH4M05CHlCcloDDI5jbsCNJxN3eLSIC/BbYrqo/8VkUUbGKSK6IZLnP04FLgR1EUJyqereqjlXViTifxZdV9QYiKMYeIjJURDJ6ngOX43Q+GFGxquoRoExEpruzLsHpLndgcXpd4R6rD5xbrHcBe4BveB2PT1x/BCqADpxf8VuAbJwLNLvdvyMjIM5zcaptSoCN7mNppMUKzAE2uHFuAb7lzo+oOH3ivZD3L8JFXIw4daub3MfWnu9OhMZaDKxz3/u/AiMGGqfdimyMMR6xKghjjPGIJWBjjPGIJWBjjPGIJWBjjPGIJWBjjPGIJWAz6ETkG27PYSVuj1f+o2UHu59iEVka6viCPPZE3x7lwnSMrw/m8czgswRsBpWInA1cBcxX1Tk4Ny6U9b1Vr4px2gbHqq/3v4qJZpaAzWDLA6pVtQ1AVavVvfVURE4XkVfcTlhW+dzSuUZEfuj2u7tLRM5z7zD8DvAvbin6X9y7qB4WkbVuBynL3e0/KSJPi8gLbj+tP+oJRpx+m9e7/fmuducF3E8wBnIO7vwhIvKE+9/An0TkHRFZICL3AunuuT3m7j5RRH7t/vfwd/fOOxPNvL6bxB7x9QCG4dzVtgv4JXCBOz8ZeBPIdaf/BWcwVnD6V/ix+3wp8JL7/JPAL3z2/f+AG9znWe4xhrrr7QUygTTgAE5fHbk4pe9J7jYj+9qP33lMxKdLz1M4h68Cv3Kfz8bpB3mBO93sd7xOoNidfqInRntE7yNsoyIbE4iqNovI6cB5wEXAn8QZMWQdTgJ60ekGgkScW6Z79HTG8x5OMgrkcpxOZ77qTqcB493nq1W1AUBEtgETcG4dfVVV97mx1fazH9/OzAOZfhLncC5Ox96o6hYRKelj//tUdWOAfZgoZQnYDDpV7cIpEa4Rkc04nZa8B2xV1bN72azN/dtF759bAT6iqjs/MNO5yNfmM6tnH0LgbkID7icIwsDPIVDXpb3xPwergohyVgdsBpWITBeRQp9ZxThVAjuBXPciHSKSLCKz+tldE5DhM70K+LzbkxoiMq+f7d8CLhCRSe76I09yPz1O5hxeBz7mrl8EnOazrMPtktPEKEvAZrANAx4RkW3uv9tFwD2q2g78M/BDEdmEU098Tj/7+gdQ1HMRDvguTj1sidtk67t9bayqVcCtwNPuMf/kLgp2P9NFpLznASw/iXP4JU7SLgHuxOlZq8Fd9pAbw2O9bWyim/WGZoyHxBnANVlVW0VkCk4XhtPcHyQT46wO2BhvDQH+4VY1CPBZS77xw0rAxhjjEasDNsYYj1gCNsYYj1gCNsYYj1gCNsYYj1gCNsYYj/x/hEm3Gn9RSdQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Print number of samples\n",
    "print('Number of Sentences:', len(df['SentenceId'].value_counts()))\n",
    "print('Number of Phrases:', len(df))\n",
    "\n",
    "# Filter out repeats\n",
    "reduced = df[['SentenceId', 'Phrase', 'Sentiment']].groupby(by='SentenceId').apply(lambda group: group[group['Phrase'].str.len()==group['Phrase'].str.len().max()])\n",
    "# Look at length of average sample\n",
    "reduced['lengths'] = reduced['Phrase'].str.split().str.len()\n",
    "reduced = reduced.set_index(keys=np.arange(len(reduced)))\n",
    "print('Number of words:')\n",
    "print('Min:', reduced['lengths'].min(), 'Mean:', reduced['lengths'].mean(), 'Max:', reduced['lengths'].max())\n",
    "\n",
    "# Print the size of the vocabulary\n",
    "words = reduced['Phrase'].str.split(expand=True).melt(value_name='Word').dropna()\n",
    "words = words[words['Word'].str.isalpha()]['Word'].value_counts()\n",
    "print('Number of times a word occurs only once:', len(words[words==1]))\n",
    "print('Total number of unique words:', len(words))\n",
    "print('Known words ratio:', 1-len(words[words==1])/len(words))\n",
    "\n",
    "sns.displot(reduced['lengths'], kind='kde') # Weird distrbuiton in bars\n",
    "plt.xlabel('Sentence Length')\n",
    "plt.title('Distribution of number of words in a sentence')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a7f39d04-e306-4275-b55c-a3d06afc5a3a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Distribution of ratings')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFtCAYAAAA5/7CSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdGElEQVR4nO3dfbildV3v8ffH4dGDBMRoMDMI5BxzoKPJSICVFJ4gU6EuifFSQcMwQ/OpDMIkz2lOnuqYaaFyKYEP8ZDZAS1MIsk6CjSSCAOikyhsZ2QGTAUf0MHv+eO+53K52Xtm7Zm11m/27Pfruta17/W7n777N3s++96/+2GlqpAkTd4jWhcgSQuVASxJjRjAktSIASxJjRjAktSIASxJjRjA2iFJ3p7k90a0rUOSPJBkUf/+uiQvHsW2++1dneSMUW1vDvv9gyT3JvnyGPfxQJLDx7V9jUe8DlizSfIF4DHAZuAh4Dbg3cCFVfW97djWi6vqH+ewznXAe6vqnXPZV7/u7wOPq6rnz3XdUUqyDPgs8Niq2jiibV7HdvaLdi4eAWtbnlVVjwIeC7wR+B3gXaPeSZLdRr3NncRjgfuGDd9duB80k6ry5WvGF/AF4OnT2o4Gvgcc2b+/GPiDfvpA4EPAV4GvAP9C90v+Pf063wIeAF4LHAoUcCZwF/Cxgbbd+u1dB/whcCPwNeBK4IB+3vHA1Ez1AicB3wG+2+/v5oHtvbiffgTwOuCLwEa6I/sf6udtqeOMvrZ7gfO20k8/1K+/qd/e6/rtP73/nr/X13HxDOseD0zR/WL7ct9X+/f9uAn4z356ab/8arq/Rr7db/PP+/aiO+Lf8m/yF8DfAfcDNwA/OrDPnwfu6Pv0AuCfB/rlcf37r/Xf9+Wtfw535ZdHwJqTqrqRLjB+eobZr+nnLaYbuvjdbpV6AV2QPauq9qmqPxpY52nAE4ATZ9nl6cCvAgfTDYW8ZYgaPwz8L7rw2KeqnjjDYi/sXz8LHA7sA/z5tGV+Cng8cALw+iRPmGWXb6UL4cP77+d04EXVDbf8ArC+r+OFs6z/I8ABdEfLZ9GF91/27w+hC/E/77+38+h+sb2s3+bLZtnmc4E30IX5OrrgJsmBwPuBc4Efpgvi4wbW+5/AR/r1lvbfm8bEANb2WE8XGNN9FziIbrzzu1X1L9UfVm3F71fVN6rqW7PMf09V3VpV3wB+D/iVLSfpdtDzgDdV1eer6gG6QFo1bQjgDVX1raq6GbgZeFiQ97WcBpxbVfdX1ReA/wO8YA61fA84v6oe7Pd3X1X9TVV9s6rupwvPp83x+/tAVd1YVZuB9wFP6tufAaytqg/0895Cd+S9xXfpgv/gqvp2Vf3rHPerOTCAtT2W0A0xTPfHdEdbH0ny+STnDLGtu+cw/4vA7nRDHTvq4H57g9veje7IfYvBYPom3VHydAcCe8ywrSVzqGVTVX17y5skj0zyjiRfTPJ1uuGZ/eb4i2e22g9moE/7X5BTA8u+FghwY5K1SX51DvvUHBnAmpMkT6ELl4cdGfVHgK+pqsOBZwGvTnLCltmzbHJbR8jLBqYPoTtCuxf4BvDIgboW0Q19DLvd9XRHeoPb3gzcs431pruX7x81Dm7rS3PYxvRaX0M39PGTVbUv8DN9e2ZZfi420A0tdBtMMvi+qr5cVb9WVQcDLwEuSPK4HdiftsIA1lCS7JvkmcBldJdA3TLDMs9M8rj+P/XX6U4WPdTPvodujHSunp9kRZJHAv8DeH9VPUR3addeSX4xye50J772HFjvHuDQJLP9jF8KvCrJYUn24ftjxpvnUlxfyxXA6iSPSvJY4NXAe+eynWkeRTfu+9UkBwDnT5u/vX0J3Ym5H09ySj/ccjbdGDQASU5NsiWQ/5Mu7B96+GY0CgawtuWDSe6n+7P1POBNwItmWXY58I90Z+c/AVxQVdf18/4QeF2Sryb5rTns/z10Z/W/DOwF/CZAVX0N+A3gnXRHm9/gB/+U/uv+631Jbpphuxf12/4YcCfdVQUvn0Ndg17e7//zdH8Z/FW//e31ZmBvuqPr64EPT5v/Z8Bzkvxnkm2elBxUVfcCpwJ/BNwHrADWAA/2izwFuCHJA8BVwCuq6s7t/D60Dd6IIS1g/V8IU8DzquqjretZaDwClhaYJCcm2S/JnnSXCobuSFsTZgBLC8+xwH/QDXE8CzhlK5cBaozGFsBJLkqyMcmtM8z7rSTVXxS+pe3cJOuS3JHkxIH2o5Lc0s97S3+CR9J2qqrfr6ofrqpHVdVPVtUNrWtaqMZ5BHwx3S2hP6B/OMl/p7szakvbCmAVcES/zgUD1zy+je7uoOX962HblKT5aGwBXFUfY+aL9f+U7mLvwbN/JwOX9XcC3Ul3Mf/RSQ4C9q2qT/QXjL8bOGVcNUvSJE30yUtJng18qapunjaSsIQfPAkw1bd9lx+8tGhL+2zbP4vuaJkVK1YctXbt2hFVLkk7ZMah04mdhOsvpD8PeP1Ms2doq620z6iqLqyqlVW1cu+9996+QiVpQiZ5BPyjwGHAlqPfpcBNSY6mO7IdvOV0Kd2tolMM3CY50C5J897EjoCr6paqenRVHVpVh9KF65Or6st0d9ysSrJnksPoTrbdWFUbgPuTHNNf/XA63TNhJWneG+dlaJfS3Y76+CRTSc6cbdmqWkt3P/1tdLddnt3fYw/wUrrbTdfRXbt49bhqlqRJ2mVvRV65cmWtWbOmdRmSBK1PwkmSfpABLEmNGMCS1IgBLEmNGMCS1IgBLEmNGMCS1IgBLEmNGMDSArRk2SEk2e7XkmWHtP4WdgkTfRylpJ3D+qm7Oe0dH9/u9S9/yXEjrGbh8ghYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYkhoxgCWpEQNYPhtWasTnActnw0qNeAQsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUyNgCOMlFSTYmuXWg7Y+TfCbJp5P8bZL9Buadm2RdkjuSnDjQflSSW/p5b0mScdUsSZM0ziPgi4GTprVdAxxZVf8N+CxwLkCSFcAq4Ih+nQuSLOrXeRtwFrC8f03fpiTNS2ML4Kr6GPCVaW0fqarN/dvrgaX99MnAZVX1YFXdCawDjk5yELBvVX2iqgp4N3DKuGqWpElqOQb8q8DV/fQS4O6BeVN925J+enq7JM17TQI4yXnAZuB9W5pmWKy20j7bds9KsibJmk2bNu14oZI0RhMP4CRnAM8EntcPK0B3ZLtsYLGlwPq+fekM7TOqqguramVVrVy8ePFoC5ekEZtoACc5Cfgd4NlV9c2BWVcBq5LsmeQwupNtN1bVBuD+JMf0Vz+cDlw5yZolaVzG9jzgJJcCxwMHJpkCzqe76mFP4Jr+arLrq+rXq2ptkiuA2+iGJs6uqof6Tb2U7oqKvenGjK9GknYBYwvgqnruDM3v2sryq4HVM7SvAY4cYWmStFPwTjhJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJasQAlqRGDGBJamRsAZzkoiQbk9w60HZAkmuSfK7/uv/AvHOTrEtyR5ITB9qPSnJLP+8tSTKumiVpksZ5BHwxcNK0tnOAa6tqOXBt/54kK4BVwBH9OhckWdSv8zbgLGB5/5q+TUmal8YWwFX1MeAr05pPBi7ppy8BThlov6yqHqyqO4F1wNFJDgL2rapPVFUB7x5YR5LmtUmPAT+mqjYA9F8f3bcvAe4eWG6qb1vST09vn1GSs5KsSbJm06ZNIy1ckkZtZzkJN9O4bm2lfUZVdWFVrayqlYsXLx5ZcZI0DpMO4Hv6YQX6rxv79ilg2cByS4H1ffvSGdolad6bdABfBZzRT58BXDnQvirJnkkOozvZdmM/THF/kmP6qx9OH1hHkua13ca14SSXAscDByaZAs4H3ghckeRM4C7gVICqWpvkCuA2YDNwdlU91G/qpXRXVOwNXN2/JGneG1sAV9VzZ5l1wizLrwZWz9C+BjhyhKVJ0k5hZzkJJ0kLjgEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwJLUiAEsSY0YwFrQliw7hCTb9Vqy7JDW5WueG9udcNJ8sH7qbk57x8e3a93LX3LciKvRQuMRsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiMGsCQ1YgBLUiNNAjjJq5KsTXJrkkuT7JXkgCTXJPlc/3X/geXPTbIuyR1JTmxRsySN2sQDOMkS4DeBlVV1JLAIWAWcA1xbVcuBa/v3JFnRzz8COAm4IMmiSdctSaPWaghiN2DvJLsBjwTWAycDl/TzLwFO6adPBi6rqger6k5gHXD0ZMuVpNGbeABX1ZeAPwHuAjYAX6uqjwCPqaoN/TIbgEf3qywB7h7YxFTf9jBJzkqyJsmaTZs2jetbkKSRaDEEsT/dUe1hwMHAf0ny/K2tMkNbzbRgVV1YVSurauXixYt3vFhJGqMWQxBPB+6sqk1V9V3gA8BxwD1JDgLov27sl58Clg2sv5RuyEKS5rUWAXwXcEySRyYJcAJwO3AVcEa/zBnAlf30VcCqJHsmOQxYDtw44ZolaeR2m/QOq+qGJO8HbgI2A/8OXAjsA1yR5Ey6kD61X35tkiuA2/rlz66qhyZdtySN2sQDGKCqzgfOn9b8IN3R8EzLrwZWj7suSZok74STpEYMYElqxACWpEaGCuAkTx2mTZI0vGGPgN86ZJskaUhbvQoiybF0N0ksTvLqgVn70j1ER5K0nbZ1GdoedNfn7gY8aqD968BzxlWUJC0EWw3gqvpn4J+TXFxVX5xQTZK0IAx7I8aeSS4EDh1cp6p+bhxFSdJCMGwA/zXwduCdgLcBS9IIDBvAm6vqbWOtRJImYMmyQ1g/dfe2F5zBwUuX8aW77xpZLcMG8AeT/Abwt3TPbACgqr4yskokaQLWT93Nae/4+Hate/lLjhtpLcMG8JbHRP72QFsBh4+0GklaQIYK4Ko6bNyFSNJCM1QAJzl9pvaqevdoy5GkhWPYIYinDEzvRffc3psAA1iSttOwQxAvH3yf5IeA94ylIklaILb3cZTfpPtsNknSdhp2DPiDfP+j4BcBTwCuGFdRkrQQDDsG/CcD05uBL1bV1BjqkaQFY6ghiP6hPJ+heyLa/sB3xlmUJC0Ew34ixq8AN9J9VPyvADck8XGUkrQDhh2COA94SlVtBEiyGPhH4P3jKkySdnXDXgXxiC3h27tvDutKkmYw7BHwh5P8A3Bp//404O/HU5IkLQzb+ky4xwGPqarfTvLLwE8BAT4BvG8C9UnSLmtbwwhvBu4HqKoPVNWrq+pVdEe/bx5vaZK0a9tWAB9aVZ+e3lhVa+g+nkiStJ22FcB7bWXe3qMsRJIWmm0F8L8l+bXpjUnOBD45npIkaWHY1lUQrwT+Nsnz+H7grgT2AH5pjHVJ0i5vqwFcVfcAxyX5WeDIvvnvquqfxl6ZJO3ihn0e8EeBj465FklaULybTZIaaRLASfZL8v4kn0lye5JjkxyQ5Jokn+u/7j+w/LlJ1iW5I8mJLWqWpFFrdQT8Z8CHq+rHgCcCtwPnANdW1XLg2v49SVYAq4AjgJOAC5IsalK1JI3QxAM4yb7AzwDvAqiq71TVV4GTgUv6xS4BTumnTwYuq6oHq+pOYB1w9DhqW7LsEJJs92vJskPGUZakXdSwD+MZpcOBTcBfJnki3eVtr6B75sQGgKrakOTR/fJLgOsH1p/q20Zu/dTdnPaOj2/3+pe/5LgRViNpV9diCGI34MnA26rqJ4Bv0A83zCIztNUMbSQ5K8maJGs2bdq045VK0hi1COApYKqqbujfv58ukO9JchBA/3XjwPLLBtZfCqyfacNVdWFVrayqlYsXLx5L8ZI0KhMP4Kr6MnB3ksf3TScAtwFXAWf0bWcAV/bTVwGrkuyZ5DBgOd3HI0nSvNZiDBjg5cD7kuwBfB54Ed0vgyv650zcRff5c1TV2iRX0IX0ZuDsqnqoTdmSNDpNAriqPkX3TInpTphl+dXA6nHWJEmT5p1wktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktRIswBOsijJvyf5UP/+gCTXJPlc/3X/gWXPTbIuyR1JTmxVsySNUssj4FcAtw+8Pwe4tqqWA9f270myAlgFHAGcBFyQZNGEa5WkkWsSwEmWAr8IvHOg+WTgkn76EuCUgfbLqurBqroTWAccPaFSJWlsWh0Bvxl4LfC9gbbHVNUGgP7ro/v2JcDdA8tN9W0Pk+SsJGuSrNm0adPIi5akUZp4ACd5JrCxqj457CoztNVMC1bVhVW1sqpWLl68eLtrlKRJ2K3BPp8KPDvJM4C9gH2TvBe4J8lBVbUhyUHAxn75KWDZwPpLgfUTrViSxmDiR8BVdW5VLa2qQ+lOrv1TVT0fuAo4o1/sDODKfvoqYFWSPZMcBiwHbpxw2ZI0ci2OgGfzRuCKJGcCdwGnAlTV2iRXALcBm4Gzq+qhdmVK0mg0DeCqug64rp++DzhhluVWA6snVpgkTYB3wklSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDViAEtSIwawJDUy8QBOsizJR5PcnmRtklf07QckuSbJ5/qv+w+sc26SdUnuSHLipGuWpHFocQS8GXhNVT0BOAY4O8kK4Bzg2qpaDlzbv6eftwo4AjgJuCDJogZ1S9JITTyAq2pDVd3UT98P3A4sAU4GLukXuwQ4pZ8+Gbisqh6sqjuBdcDREy1aksag6RhwkkOBnwBuAB5TVRugC2ng0f1iS4C7B1ab6ttm2t5ZSdYkWbNp06ax1S1Jo9AsgJPsA/wN8Mqq+vrWFp2hrWZasKourKqVVbVy8eLFoyhTksamSQAn2Z0ufN9XVR/om+9JclA//yBgY98+BSwbWH0psH5StUrSuLS4CiLAu4Dbq+pNA7OuAs7op88ArhxoX5VkzySHAcuBGydVrySNy24N9vlU4AXALUk+1bf9LvBG4IokZwJ3AacCVNXaJFcAt9FdQXF2VT008aolacQmHsBV9a/MPK4LcMIs66wGVo+tKElqwDvhJKkRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGjGAJakRA1iSGpk3AZzkpCR3JFmX5JzW9UjSjpoXAZxkEfAXwC8AK4DnJlnRtipJ2jHzIoCBo4F1VfX5qvoOcBlwcuOaJGmHzJcAXgLcPfB+qm+TpHkrVdW6hm1KcipwYlW9uH//AuDoqnr5tOXOAs7q3z4euGOOuzoQuHcHy91R1rDz1AA7Rx3WMP9ruLeqTpreuNuO1zMRU8CygfdLgfXTF6qqC4ELt3cnSdZU1crtXX8UrGHnqWFnqcMadt0a5ssQxL8By5MclmQPYBVwVeOaJGmHzIsj4KranORlwD8Ai4CLqmpt47IkaYfMiwAGqKq/B/5+zLvZ7uGLEbKGzs5QA+wcdVhDZ5erYV6chJOkXdF8GQOWpF3Oggzgbd3WnM5b+vmfTvLkBjUcn+RrST7Vv14/4v1flGRjkltnmT/2PhiyjnH3w7IkH01ye5K1SV4xwzKT+HkYpo5x98VeSW5McnNfwxtmWGasfTFkDWPth4H9LEry70k+NMO80fRDVS2oF91JvP8ADgf2AG4GVkxb5hnA1UCAY4AbGtRwPPChMfbDzwBPBm6dZf5Y+2AOdYy7Hw4CntxPPwr47KR/HuZQx7j7IsA+/fTuwA3AMZPsiyFrGGs/DOzn1cBfzbSvUfXDQjwCHua25pOBd1fnemC/JAdNuIaxqqqPAV/ZyiLj7oNh6xirqtpQVTf10/cDt/PwuyzH3hdD1jFW/ff3QP929/41/STRWPtiyBrGLslS4BeBd86yyEj6YSEG8DC3NY/71udht39s/6fY1UmOGOH+h7Ez3f49kX5IcijwE3RHXYMm2hdbqQPG3Bf9n92fAjYC11TVxPtiiBpg/D8TbwZeC3xvlvkj6YeFGMCZoW36b9hhlhl3DTcBj62qJwJvBf7vCPc/jHH3wbAm0g9J9gH+BnhlVX19+uwZVhlLX2yjjrH3RVU9VFVPorvb9OgkR04vcabVJlzDWPshyTOBjVX1ya0tNkPbnPthIQbwMLc1D3Xr8zhrqKqvb/lTrLproHdPcuAIa9iWcffBUCbRD0l2pwu991XVB2ZYZCJ9sa06JvkzUVVfBa4Dpj+/YGI/F7PVMIF+eCrw7CRfoBse/Lkk7522zEj6YSEG8DC3NV8FnN6f6TwG+FpVbZhkDUl+JEn66aPp/q3uG2EN2zLuPhjKuPuh3/a7gNur6k2zLDb2vhimjgn0xeIk+/XTewNPBz4zbbGx9sUwNYy7H6rq3KpaWlWH0v3f/Keqev60xUbSD/PmTrhRqVlua07y6/38t9PdcfcMYB3wTeBFDWp4DvDSJJuBbwGrqj/9OgpJLqU7m3xgkingfLoTHhPpgznUMdZ+oDvaeQFwSz/uCPC7wCEDNUyiL4apY9x9cRBwSboPQHgEcEVVfWiS/zeGrGHc/TCjcfSDd8JJUiMLcQhCknYKBrAkNWIAS1IjBrAkNWIAS1IjBrDmtSTnpXtq1qfTPRnrJ7djG09K8oyB98/ODE+oG6V0T/Q6bpz70M5vwV0HrF1HkmOBZ9I9RezB/m6oPbZjU08CVtJ/4kpVXcX4P3PweOAB4ONj3o92Yl4HrHkryS8DL6qqZ01rPwp4E7AP3UeIv7CqNiS5ju4BNz8L7Aec2b9fB+wNfAn4w356ZVW9LMnFdBf7/xjwWLoL7s8AjqV7BOEL+33+PPAGYE+6R42+qKoe6G9nvQR4Ft0NJqcC3wauBx4CNgEvr6p/GWnnaF5wCELz2UeAZUk+m+SCJE/rn6fwVuA5VXUUcBGwemCd3arqaOCVwPn940BfD1xeVU+qqstn2M/+wM8BrwI+CPwpcATw4/3wxYHA64CnV9WTgTV0z5Ld4t6+/W3Ab1XVF4C3A3/a79PwXaAcgtC81R9hHgX8NN1R7eXAHwBHAtf0jwtYBAzeo7/lITefBA4dclcfrKpKcgtwT1XdApBkbb+NpcAK4P/1+9wD+MQs+/zl4b9D7eoMYM1rVfUQ3ROzrusD8mxgbVUdO8sqD/ZfH2L4n/8t63xvYHrL+936bV1TVc8d4T61ADgEoXkryeOTLB9oehLdJ0ks7k/QkWT3IR7YfT/dxwBtr+uBpyZ5XL/PRyb5r2Pep3YBBrDms33onpx1W5JP0w0DvJ7uaVn/O8nNwKeAbV3u9VFgRX8Z22lzLaKqNgEvBC7t67ie7qTd1nwQ+KV+nz89131q1+BVEJLUiEfAktSIASxJjRjAktSIASxJjRjAktSIASxJjRjAktSIASxJjfx/0nd22MQP2EcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.displot(reduced['Sentiment'])\n",
    "plt.xlabel('Sentiment')\n",
    "plt.title('Distribution of ratings')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5bd97e5c-f0ec-45fa-b7d5-a21d750a0e28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 93644 entries, 0 to 93643\n",
      "Data columns (total 4 columns):\n",
      " #   Column      Non-Null Count  Dtype \n",
      "---  ------      --------------  ----- \n",
      " 0   PhraseId    93644 non-null  int64 \n",
      " 1   SentenceId  93644 non-null  int64 \n",
      " 2   Phrase      93644 non-null  object\n",
      " 3   Sentiment   93644 non-null  int64 \n",
      "dtypes: int64(3), object(1)\n",
      "memory usage: 3.6+ MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 62416 entries, 93644 to 156059\n",
      "Data columns (total 4 columns):\n",
      " #   Column      Non-Null Count  Dtype \n",
      "---  ------      --------------  ----- \n",
      " 0   PhraseId    62416 non-null  int64 \n",
      " 1   SentenceId  62416 non-null  int64 \n",
      " 2   Phrase      62416 non-null  object\n",
      " 3   Sentiment   62416 non-null  int64 \n",
      "dtypes: int64(3), object(1)\n",
      "memory usage: 2.4+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Confirm that we don't need dropna or anything\n",
    "print(pd.read_csv('train.csv', index_col=0).info())\n",
    "print(pd.read_csv('test.csv', index_col=0).info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6dffc107-4adc-4734-bae3-d49fcfcb9b82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\nConclusions from exploratory analysis:\\nBecause each sentence comes with Sentiment ratings for each separate phrase, that could be used for learning. However, this is unlikely to be\\neffective since it is does not account for meaning at all.\\nThere's not a ton of samples, so if I train a neural network, I need to be extremely cautious to not overfit.\\nThe length of each sentence indicates that I could use average word vectors or bag of words representation.\\nThe vocabulary is relatively large, meaning I should use a full vocabulary import while preprocessing the data.\\nOnly about half of the words are repeated, so it is unlikely that bag of words would be very effective.\\n\\nSteps forward:\\nFor this dataset, the obvious choice as to how to preprocess this data is to use average word vectors for the phrase and the length of the\\nphrase in number of words. I'm interested in comparing the performance of a Linear SVM, a Radial SVM, and PyTorch model. I expect all to perform\\nrather similarly, since these word vectors have 200 dimensions.\\n\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Conclusions from exploratory analysis:\n",
    "Because each sentence comes with Sentiment ratings for each separate phrase, that could be used for learning. However, this is unlikely to be\n",
    "effective since it is does not account for meaning at all.\n",
    "There's not a ton of samples, so if I train a neural network, I need to be extremely cautious to not overfit.\n",
    "The length of each sentence indicates that I could use average word vectors or bag of words representation.\n",
    "The vocabulary is relatively large, meaning I should use a full vocabulary import while preprocessing the data.\n",
    "Only about half of the words are repeated, so it is unlikely that bag of words would be very effective.\n",
    "\n",
    "Steps forward:\n",
    "For this dataset, the obvious choice as to how to preprocess this data is to use average word vectors for the phrase and the length of the\n",
    "phrase in number of words. I'm interested in comparing the performance of a Linear SVM, a Radial SVM, and PyTorch model. I expect all to perform\n",
    "rather similarly, since these word vectors have 200 dimensions.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4ae1dc9f-bb76-4238-bae9-299b05327e34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell will contain my function definitions for feature cleaning\n",
    "\n",
    "def avg_vector(data, nlp):\n",
    "    data['lengths'] = data['Phrase'].str.split().str.len()\n",
    "    \n",
    "    with nlp.disable_pipes():\n",
    "        doc_vectors = np.array([nlp(text).vector for text in tqdm(data['Phrase'])])\n",
    "    doc_vectors = pd.DataFrame(doc_vectors, index=data.index)\n",
    "    \n",
    "    data = data.merge(doc_vectors, left_index=True, right_index=True)\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "aedfcc6a-f84b-473a-a62e-d200edaa64ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 93644/93644 [10:09<00:00, 153.54it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished saving preprocessed training data\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    nlp = spacy.load(\"en_core_web_lg\")\n",
    "    train_data = pd.read_csv('train.csv', index_col=0)\n",
    "    preprocessed_train = avg_vector(train_data, nlp)\n",
    "    preprocessed_train.to_csv('preprocessed_train.csv', index=False)\n",
    "    print('Finished saving preprocessed training data')\n",
    "except OSError:\n",
    "    print('You need to run the import box again. The language module was not downloaded properly.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2528b5cc-7551-490e-9925-6a8e153f1a9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 62416/62416 [11:09<00:00, 93.18it/s] \n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    nlp = spacy.load(\"en_core_web_lg\")\n",
    "    test_data = pd.read_csv('test.csv', index_col=0)\n",
    "    preprocessed_test = avg_vector(test_data, nlp)\n",
    "    preprocessed_test.to_csv('preprocessed_test.csv', index=False)\n",
    "    print('Finished saving preprocessed testing data')\n",
    "except OSError:\n",
    "    print('You need to run the import box again. The language module was not downloaded properly.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c418eb-959f-4aa2-a3ea-31808adec97d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}