Text_Classification_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Text_Classification_NLP.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNJXXSdX5hxr+g930hWMWad"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_AJkqDYTS2f-",
        "colab_type": "text"
      },
      "source": [
        "# What to Expect\n",
        "This notebook wil demo various machine learning and deep learning techniques for text classification problems. Initially, I will explore a couple of datasets and decide which one to deep dive into. The first dataset is scientific paper reviews' data and the second dataset is Amazon product reviews data. If the quality of data is good enough, I will carry out sentiment analysis on it using various algorithms. Otherwise I will explore another data set. I will select the data set with higher data quality."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCRjFD5O8wY1",
        "colab_type": "text"
      },
      "source": [
        "# Initial Data Exploration\n",
        "The main goal of this step is to skim different datasets containing text, examine data quality and finally select the dataset that I will use for demonstrating various machine learning algorithms in action. The main steps will be:\n",
        "\n",
        "\n",
        "*   Load the dataset into local memory\n",
        "*   Parse the file into the desired format like a pandas dataframe (this might be a multistep process depending on the level of granularity we need)\n",
        "*   Examine the basics like the main features, missing values etc\n",
        "*   If data quality is satisfactory, proceed to machine learning, else move on to exploration of the next dataset\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5UuKgU0DTuRz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 228
        },
        "outputId": "18ca9679-d273-4413-c5f9-fd0ae8c61138"
      },
      "source": [
        "#Loading the first dataset\n",
        "!wget 'https://archive.ics.uci.edu/ml/machine-learning-databases/00410/reviews.json'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-07-06 18:08:41--  https://archive.ics.uci.edu/ml/machine-learning-databases/00410/reviews.json\n",
            "Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252\n",
            "Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 593600 (580K) [application/x-httpd-php]\n",
            "Saving to: ‘reviews.json’\n",
            "\n",
            "reviews.json        100%[===================>] 579.69K  3.20MB/s    in 0.2s    \n",
            "\n",
            "2020-07-06 18:08:41 (3.20 MB/s) - ‘reviews.json’ saved [593600/593600]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0kHejsKfUPQC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "623ffd88-66c3-4fee-8b00-e565c7c8fef2"
      },
      "source": [
        "#Importing python libraries\n",
        "import numpy as np\n",
        "import pandas as pd #to work with csv files\n",
        "\n",
        "#matplotlib imports are used to plot confusion matrices for the classifiers\n",
        "import matplotlib as mpl \n",
        "import matplotlib.cm as cm \n",
        "import matplotlib.pyplot as plt \n",
        "\n",
        "#import feature extraction methods \n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.feature_extraction import stop_words\n",
        "\n",
        "#pre-processing of text\n",
        "import string\n",
        "import re\n",
        "import json\n",
        "import gzip\n",
        "from urllib.request import urlopen\n",
        "\n",
        "#import classifiers \n",
        "import sklearn\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import LinearSVC\n",
        "\n",
        "#import different evaluation metrics \n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import confusion_matrix \n",
        "from sklearn import metrics\n",
        "\n",
        "#import time function to track the training duration\n",
        "from time import time\n",
        "\n",
        "#System libraries\n",
        "import os\n",
        "import sys"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.feature_extraction.stop_words module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.feature_extraction.text. Anything that cannot be imported from sklearn.feature_extraction.text is now part of the private API.\n",
            "  warnings.warn(message, FutureWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4IgawsF8Xt8",
        "colab_type": "text"
      },
      "source": [
        "The json file is nested. Hence, I will visualize all levels (top down approach)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g-13_iZ2Un31",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Explore dataset (top most level in the nested file)\n",
        "df = pd.read_json(r'reviews.json')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cZ7ywClnU1W8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "outputId": "8f0f2d40-eebf-453b-f1ae-81080ff0ff03"
      },
      "source": [
        "df.loc[0,'paper'] #The json file ahs 3 main keys id, preliminary decision and review"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': 1,\n",
              " 'preliminary_decision': 'accept',\n",
              " 'review': [{'confidence': '4',\n",
              "   'evaluation': '1',\n",
              "   'id': 1,\n",
              "   'lan': 'es',\n",
              "   'orientation': '0',\n",
              "   'remarks': '',\n",
              "   'text': '- El artículo aborda un problema contingente y muy relevante, e incluye tanto un diagnóstico nacional de uso de buenas prácticas como una solución (buenas prácticas concretas). - El lenguaje es adecuado.  - El artículo se siente como la concatenación de tres artículos diferentes: (1) resultados de una encuesta, (2) buenas prácticas de seguridad, (3) incorporación de buenas prácticas. - El orden de las secciones sería mejor si refleja este orden (la versión revisada es #2, #1, #3). - El artículo no tiene validación de ningún tipo, ni siquiera por evaluación de expertos.',\n",
              "   'timespan': '2010-07-05'},\n",
              "  {'confidence': '4',\n",
              "   'evaluation': '1',\n",
              "   'id': 2,\n",
              "   'lan': 'es',\n",
              "   'orientation': '1',\n",
              "   'remarks': '',\n",
              "   'text': 'El artículo presenta recomendaciones prácticas para el desarrollo de software seguro. Se describen las mejores prácticas recomendadas para desarrollar software que sea proactivo ante los ataques, y se realiza un análisis de costos de estas prácticas en desarrollo de software. Todo basado en una revisión de prácticas propuestas en la bibliografía y su contraste con datos obtenidos de una encuesta en empresas. Finalmente se recomienda una guía.  Sería ideal aplicar la guía propuesta a empresas no involucradas en la encuesta que sirvió para originarla de modo de poder evaluar su efectividad en forma independiente.',\n",
              "   'timespan': '2010-07-05'},\n",
              "  {'confidence': '5',\n",
              "   'evaluation': '1',\n",
              "   'id': 3,\n",
              "   'lan': 'es',\n",
              "   'orientation': '1',\n",
              "   'remarks': '',\n",
              "   'text': '- El tema es muy interesante y puede ser de mucha ayuda una guía para incorporar prácticas de seguridad. - La presentación (descripción, etapa y uso) de las 9 prácticas para el desarrollo de software seguro.  - El “estado real del desarrollo de software en Chile” (como lo indica en su paper) no se puede lograr con solamente 22 encuestas de un total de 50. - Presenta nueve tablas que corresponden a las prácticas para el desarrollo de software seguro, pero la guía presenta 10 prácticas. ¿explica por qué? - Sugiero mejorar la guía, el mayor aporte está en la secuencia de incorporación que propone.  Además, no debería explicar la práctica en Observaciones ni diferenciarla con otras prácticas en esa columna, sino que debería dar sugerencias de cómo aplicarla. - En el texto indica “Más adelante, se presentan además tres prácticas extras…” ¿cuáles son o no leí correctamente? - De acuerdo a formato, poner como mínimo 5 palabras clave. - Sugiero mencionar las prácticas antes de mostrar cada tabla. - Algunas referencias están incompletas, por ejemplo, falta año en referencia 17, falta año y tipo de evento en referencia 11, falta editorial en referencia 19 (¿es un libro?) - Algunos títulos llevan una coma dentro de las comillas, ejemplo, referencia 1',\n",
              "   'timespan': '2010-07-05'}]}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 168
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mlfLJvptU2ja",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Extracting the main features of each research paper\n",
        "paper_df = pd.json_normalize(df[\"paper\"])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUYM7KDNcgh1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 226
        },
        "outputId": "4b01edb1-e196-4fdf-a07a-9c8f2d393361"
      },
      "source": [
        "paper_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>preliminary_decision</th>\n",
              "      <th>review</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>accept</td>\n",
              "      <td>[{'confidence': '4', 'evaluation': '1', 'id': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>accept</td>\n",
              "      <td>[{'confidence': '4', 'evaluation': '2', 'id': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>accept</td>\n",
              "      <td>[{'confidence': '4', 'evaluation': '2', 'id': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>accept</td>\n",
              "      <td>[{'confidence': '4', 'evaluation': '2', 'id': ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>accept</td>\n",
              "      <td>[{'confidence': '4', 'evaluation': '2', 'id': ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id preliminary_decision                                             review\n",
              "0   1               accept  [{'confidence': '4', 'evaluation': '1', 'id': ...\n",
              "1   2               accept  [{'confidence': '4', 'evaluation': '2', 'id': ...\n",
              "2   3               accept  [{'confidence': '4', 'evaluation': '2', 'id': ...\n",
              "3   4               accept  [{'confidence': '4', 'evaluation': '2', 'id': ...\n",
              "4   5               accept  [{'confidence': '4', 'evaluation': '2', 'id': ..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 170
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9v797fSuYaNK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Extracting the main features of review per research paper\n",
        "review_df = pd.json_normalize(paper_df[\"review\"][0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RCTFEb6qjoZZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 216
        },
        "outputId": "9156e645-4d84-4ce4-ff9b-c3a9f5e7a006"
      },
      "source": [
        "review_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>confidence</th>\n",
              "      <th>evaluation</th>\n",
              "      <th>id</th>\n",
              "      <th>lan</th>\n",
              "      <th>orientation</th>\n",
              "      <th>remarks</th>\n",
              "      <th>text</th>\n",
              "      <th>timespan</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>es</td>\n",
              "      <td>0</td>\n",
              "      <td></td>\n",
              "      <td>- El artículo aborda un problema contingente y...</td>\n",
              "      <td>2010-07-05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>es</td>\n",
              "      <td>1</td>\n",
              "      <td></td>\n",
              "      <td>El artículo presenta recomendaciones prácticas...</td>\n",
              "      <td>2010-07-05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>es</td>\n",
              "      <td>1</td>\n",
              "      <td></td>\n",
              "      <td>- El tema es muy interesante y puede ser de mu...</td>\n",
              "      <td>2010-07-05</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  confidence  ...    timespan\n",
              "0          4  ...  2010-07-05\n",
              "1          4  ...  2010-07-05\n",
              "2          5  ...  2010-07-05\n",
              "\n",
              "[3 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 173
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XY2hO_aeKhxP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        },
        "outputId": "caaf2ef2-e0a7-4ba6-f7d3-c4a556de0a5a"
      },
      "source": [
        "len(df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "101"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ByIObh_35x8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Extracting the main features that I am primarily interested in and combining into final df\n",
        "final_df =pd.concat([review_df.text, paper_df.preliminary_decision,review_df.confidence], axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jQlakJjo6ATK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "outputId": "98551077-79ee-4c57-9d6a-0befd5c47330"
      },
      "source": [
        "final_df.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 172 entries, 0 to 171\n",
            "Data columns (total 3 columns):\n",
            " #   Column                Non-Null Count  Dtype \n",
            "---  ------                --------------  ----- \n",
            " 0   text                  3 non-null      object\n",
            " 1   preliminary_decision  172 non-null    object\n",
            " 2   confidence            3 non-null      object\n",
            "dtypes: object(3)\n",
            "memory usage: 4.2+ KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SzrTrNHX_m7d",
        "colab_type": "text"
      },
      "source": [
        "I am primarily interested in analysing the text data but ~98% data is null values. Hence, I will not proceed with this dataset. Let's look at the next dataset. It is a very large data file which contains millions of Amazon product reviews. I will be analysing the first 1 million reviews."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R5Rnm02o6mkh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 208
        },
        "outputId": "6bdb58ff-77a3-4593-ddcc-53a69db5d40f"
      },
      "source": [
        "#Loading the second dataset\n",
        "!wget http://snap.stanford.edu/data/amazon/productGraph/item_dedup.json.gz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-07-14 14:07:03--  http://snap.stanford.edu/data/amazon/productGraph/item_dedup.json.gz\n",
            "Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80\n",
            "Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 19233006068 (18G) [application/x-gzip]\n",
            "Saving to: ‘item_dedup.json.gz’\n",
            "\n",
            "item_dedup.json.gz  100%[===================>]  17.91G  7.00MB/s    in 53m 41s \n",
            "\n",
            "2020-07-14 15:00:44 (5.69 MB/s) - ‘item_dedup.json.gz’ saved [19233006068/19233006068]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K1MNSB-RvAKX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "f = gzip.open('item_dedup.json.gz')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fyOAemIK7b9d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Since, the file size is very big, parsing the entire file crashes the session. \n",
        "# I will unwrap first million records\n",
        "\n",
        "i = 0\n",
        "data = []\n",
        "for line in f:\n",
        "  i+=1\n",
        "  data.append(json.loads(line))\n",
        "  if i>1000000:\n",
        "    break\n",
        "\n",
        "f.close()\n",
        "  \n",
        "df = pd.DataFrame(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DDuqP8ALCcM3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        },
        "outputId": "c11211a5-2853-408a-a0a9-a40b2fec0846"
      },
      "source": [
        "df.isnull().sum()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "reviewerID            0\n",
              "asin                  0\n",
              "helpful               0\n",
              "reviewText            0\n",
              "overall               0\n",
              "summary               0\n",
              "unixReviewTime        0\n",
              "reviewTime            0\n",
              "reviewerName      59931\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nYwepicSEYya",
        "colab_type": "text"
      },
      "source": [
        "I want to analyse text- hence, I am interested in the reviewText feature. It has 0 null values. That's great! So I will finalise this dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9LkgwkE9__58",
        "colab_type": "text"
      },
      "source": [
        "The text classification throughout this notebook will be sentiment analysis. We can see the review and star rating for each product. I will add one more feature (the output class) called 'Sentiment'. If the rating is 3, the sentiment is 'OK', if it >3 the sentiment is 'Positive' else it is 'Negative'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cWZCq5RI6aNB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#df.to_csv('df_mil.csv', header=None, index=False, columns=['reviewerID','asin','helpful','reviewText','overall','summary','unixReviewTime','reviewTime','reviewerName'] )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dn7C8h4k50f6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 293
        },
        "outputId": "08136230-3243-4ff4-e39c-7a14b1b15fa4"
      },
      "source": [
        "df = pd.read_csv('df_mil.csv')\n",
        "df.columns = ['reviewerID','asin','helpful','reviewText','overall','summary','unixReviewTime','reviewTime','reviewerName']\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>reviewerID</th>\n",
              "      <th>asin</th>\n",
              "      <th>helpful</th>\n",
              "      <th>reviewText</th>\n",
              "      <th>overall</th>\n",
              "      <th>summary</th>\n",
              "      <th>unixReviewTime</th>\n",
              "      <th>reviewTime</th>\n",
              "      <th>reviewerName</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AH2L9G3DQHHAJ</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[5, 5]</td>\n",
              "      <td>Interesting Grisham tale of a lawyer that take...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Show me the money!</td>\n",
              "      <td>1019865600</td>\n",
              "      <td>04 27, 2002</td>\n",
              "      <td>chris</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A2IIIDRK3PRRZY</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>The thumbnail is a shirt.  The product shown i...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Listing is all screwed up</td>\n",
              "      <td>1395619200</td>\n",
              "      <td>03 24, 2014</td>\n",
              "      <td>Helene</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1TADCM7YWPQ8M</td>\n",
              "      <td>0000000868</td>\n",
              "      <td>[10, 10]</td>\n",
              "      <td>I'll be honest. I work for a large online reta...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not a Bad Translation</td>\n",
              "      <td>1031702400</td>\n",
              "      <td>09 11, 2002</td>\n",
              "      <td>Joel@AWS</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>AWGH7V0BDOJKB</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>It had all the songs I wanted but I had ordere...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not the large print</td>\n",
              "      <td>1383177600</td>\n",
              "      <td>10 31, 2013</td>\n",
              "      <td>Barbara Marshall</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A3UTQPQPM4TQO0</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>We have many of the old, old issue. But the nu...</td>\n",
              "      <td>5.0</td>\n",
              "      <td>I was disappointed that you would only allow m...</td>\n",
              "      <td>1374883200</td>\n",
              "      <td>07 27, 2013</td>\n",
              "      <td>betty burnett</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       reviewerID        asin  ...   reviewTime      reviewerName\n",
              "0   AH2L9G3DQHHAJ  0000000116  ...  04 27, 2002             chris\n",
              "1  A2IIIDRK3PRRZY  0000000116  ...  03 24, 2014            Helene\n",
              "2  A1TADCM7YWPQ8M  0000000868  ...  09 11, 2002          Joel@AWS\n",
              "3   AWGH7V0BDOJKB  0000013714  ...  10 31, 2013  Barbara Marshall\n",
              "4  A3UTQPQPM4TQO0  0000013714  ...  07 27, 2013     betty burnett\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RL_IoGdWC8ii",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df['sentiment'] = pd.Series(df['overall'].apply(lambda x: 1 if x == 3 or x > 3 else(-1) ))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n9n6_IrsFkmM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "outputId": "66f51f81-790e-499a-f69f-d07a738e46c7"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>reviewerID</th>\n",
              "      <th>asin</th>\n",
              "      <th>helpful</th>\n",
              "      <th>reviewText</th>\n",
              "      <th>overall</th>\n",
              "      <th>summary</th>\n",
              "      <th>unixReviewTime</th>\n",
              "      <th>reviewTime</th>\n",
              "      <th>reviewerName</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AH2L9G3DQHHAJ</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[5, 5]</td>\n",
              "      <td>Interesting Grisham tale of a lawyer that take...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Show me the money!</td>\n",
              "      <td>1019865600</td>\n",
              "      <td>04 27, 2002</td>\n",
              "      <td>chris</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A2IIIDRK3PRRZY</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>The thumbnail is a shirt.  The product shown i...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Listing is all screwed up</td>\n",
              "      <td>1395619200</td>\n",
              "      <td>03 24, 2014</td>\n",
              "      <td>Helene</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1TADCM7YWPQ8M</td>\n",
              "      <td>0000000868</td>\n",
              "      <td>[10, 10]</td>\n",
              "      <td>I'll be honest. I work for a large online reta...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not a Bad Translation</td>\n",
              "      <td>1031702400</td>\n",
              "      <td>09 11, 2002</td>\n",
              "      <td>Joel@AWS</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>AWGH7V0BDOJKB</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>It had all the songs I wanted but I had ordere...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not the large print</td>\n",
              "      <td>1383177600</td>\n",
              "      <td>10 31, 2013</td>\n",
              "      <td>Barbara Marshall</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A3UTQPQPM4TQO0</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>We have many of the old, old issue. But the nu...</td>\n",
              "      <td>5.0</td>\n",
              "      <td>I was disappointed that you would only allow m...</td>\n",
              "      <td>1374883200</td>\n",
              "      <td>07 27, 2013</td>\n",
              "      <td>betty burnett</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       reviewerID        asin  ...      reviewerName sentiment\n",
              "0   AH2L9G3DQHHAJ  0000000116  ...             chris         1\n",
              "1  A2IIIDRK3PRRZY  0000000116  ...            Helene        -1\n",
              "2  A1TADCM7YWPQ8M  0000000868  ...          Joel@AWS         1\n",
              "3   AWGH7V0BDOJKB  0000013714  ...  Barbara Marshall         1\n",
              "4  A3UTQPQPM4TQO0  0000013714  ...     betty burnett         1\n",
              "\n",
              "[5 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WzkB13_FfE29",
        "colab_type": "text"
      },
      "source": [
        "# Shallow Machine Learning for Text Classification\n",
        "\n",
        "### Naive Bayes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fuYB5BsoEPr1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Function to clean data (remove stopwords, punctuation and numbers)\n",
        "stopwords = stop_words.ENGLISH_STOP_WORDS\n",
        "def clean(doc): #doc is a string of text\n",
        "    doc = \"\".join([char for char in doc if char not in string.punctuation and not char.isdigit()])\n",
        "    doc = \" \".join([token for token in doc.split() if token not in stopwords])\n",
        "    return doc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ViIRtAVWETIR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "d73c0508-c46d-4e15-d30b-ed8e7eefe9c4"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "#Saving data into x and y variables\n",
        "X = df.reviewText\n",
        "y = df.sentiment\n",
        "print('shape of X: ', X.shape)\n",
        "print('shape of y: ',y.shape)\n",
        "\n",
        "#Splitting data into train and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
        "print('shape of X_train: ', X_train.shape)\n",
        "print('shape of y_train: ',y_train.shape)\n",
        "print('shape of X_test: ', X_test.shape)\n",
        "print('shape of y_test: ',y_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "shape of X:  (1000000,)\n",
            "shape of y:  (1000000,)\n",
            "shape of X_train:  (750000,)\n",
            "shape of y_train:  (750000,)\n",
            "shape of X_test:  (250000,)\n",
            "shape of y_test:  (250000,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "df2BTLfFGJk-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "c7f4a2a4-3905-46e6-8de9-7d14746b32f4"
      },
      "source": [
        "#Preprocess train and test data (vectorization)\n",
        "vect = CountVectorizer(preprocessor=clean) \n",
        "X_train_dtm = vect.fit_transform(X_train)\n",
        "#transform testing data (using training data's features)\n",
        "X_test_dtm = vect.transform(X_test)\n",
        "print(X_train_dtm.shape, X_test_dtm.shape)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(750000, 1095433) (250001, 1095433)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZGj_Ln9WGrlv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "478171b5-6061-4050-b5f1-c19001503786"
      },
      "source": [
        "#Train the Naive Bayes Classifier\n",
        "nb = MultinomialNB() #instantiate a Multinomial Naive Bayes model\n",
        "%time nb.fit(X_train_dtm, y_train)\n",
        "y_pred_class = nb.predict(X_test_dtm)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 656 ms, sys: 2.96 ms, total: 659 ms\n",
            "Wall time: 660 ms\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sL_4MFEGZYfS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6d2e9936-d0ab-43a4-d1d8-83f2f051d9f6"
      },
      "source": [
        "set(y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{-1, 1}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xZRLxRXVHBwo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 478
        },
        "outputId": "ad5c04b5-6f86-4803-e3be-ab50f61b9de7"
      },
      "source": [
        "#Evaluate the model and visualize the results in a confusion matrix\n",
        "import itertools\n",
        "\n",
        "def plot_confusion_matrix(cm, classes,\n",
        "                          normalize=False,\n",
        "                          title='Confusion matrix',\n",
        "                          cmap=plt.cm.Blues):\n",
        "    '''\n",
        "    This function prints and plots the confusion matrix.\n",
        "    \n",
        "    '''\n",
        "    if normalize:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=45)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "\n",
        "    fmt = '.2f' if normalize else 'd'\n",
        "    thresh = cm.max() / 2.\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i, format(cm[i, j], fmt),\n",
        "                 horizontalalignment=\"center\",\n",
        "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.ylabel('True label',fontsize=15)\n",
        "    plt.xlabel('Predicted label',fontsize=15)\n",
        "    \n",
        "    \n",
        "#Print accuracy:\n",
        "print(\"Accuracy: \", metrics.accuracy_score(y_test, y_pred_class))\n",
        "\n",
        "    \n",
        "# print the confusion matrix\n",
        "cnf_matrix = confusion_matrix(y_test, y_pred_class)\n",
        "plt.figure(figsize=(8,6))\n",
        "plot_confusion_matrix(cnf_matrix, classes=['Negative','Positive'],normalize=True,\n",
        "                      title='Confusion matrix with all features')\n",
        "\n",
        "y_pred_prob = nb.predict_proba(X_test_dtm)[:, 1]\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.9166603333586666\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAG7CAYAAAA12GczAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxd8/348dc7CRFbSOzEvm8NguKr9NvaVaii6EJbrbZ036iqaim68Wtpiy+6qF3tFVpLVYvEUiq2VKgkSGJJgghJ3r8/zpn0ZEwyM8nM3Dk3r6fHfcw953zuOe97c817Psv5fCIzkSRJ9dan0QFIkqSFZ0KXJKkJmNAlSWoCJnRJkpqACV2SpCZgQpckqQmY0FVLETEgIq6PiCkRccVCnOfwiLilK2NrlIjYOSKe6OZrvBYR687n+DMR8f4uutacc0XESRHx+/mU/WxEvFjGN7grri/VjQld3SoiDouIUeUv2ucj4k8R8T9dcOoPASsDgzPzoAU9SWZenJm7d0E83SoiMiLWn1+ZzLwrMzfqzjgyc+nMfLqM6aKI+EF3Xq8jImIx4KfA7mV8Ly3EudYuP+t+XReh1DNM6Oo2EfEV4EzgVIrkuyZwDjC8C06/FvBkZs7sgnPV3iKegFYGlgAebXQgUfD3qhrCL566RUQMBE4GPp+ZV2fm65n5dmZen5lfL8v0j4gzI2JC+TgzIvqXx3aNiHER8dWImFjW7o8sj30POBE4pKz5f7J1k2zrmlZEHBERT0fEtIgYGxGHV/b/rfK6HSNiZNmUPzIidqwcuyMivh8Rd5fnuSUiVpjH+2+J/xuV+PePiL0j4smIeDkijq+U3y4i/hERr5ZlfxERi5fH/loW+2f5fg+pnP+bEfECcGHLvvI165XX2LrcXi0iJkXErm3EemREXF/ZfqrajRERz0XE0PJ5RsT6EfFp4HDgG2VM11dOOTQiHi4/w8siYol5fEbrRcRtEfFSREyOiIsjYrm2ys5LRGwItHQzvBoRt5X7N46IW8vP4ImIOLjymn0i4sGImFq+t5Mqp/xr5VyvRcQOHfhu3RERp0TE3cAbwLrtXH/viBhdfofGR8TXOvOepXnKTB8+uvwB7AnMBPrNp8zJwD3ASsCKwN+B75fHdi1ffzKwGLA3xS/L5cvjJwG/r5yr9fbaQAL9gKWAqcBG5bFVgc3K50cAfyufDwJeAT5avu7QcntwefwO4N/AhsCAcvu0eby3lvhPLOM/CpgE/AFYBtgMmA6sU5bfBnh3ed21gceAL1XOl8D6bZz/dKB/Gc+uwLhKmaOA0cCSwAjgx/OIdV3gVYo/8FcDnm05T3nsFaBP6ziAi4AftDrXM8B95XkGle/j6Hlcd31gtzL+FSmS6ZmtzvX+tv59W51nzr91ub0U8BxwZPl5bgVMBjatfHZblO93S+BFYP+2ztXed6vyvfhP+W/aDxjYzvWfB3Yuny8PbN3o/199NMfDGrq6y2Bgcs6/Sfxw4OTMnJiZk4DvUSTTFm+Xx9/OzJuA14AF7SOeDWweEQMy8/nMbKt5dh/gqcz8XWbOzMxLgMeBD1TKXJiZT2bmdOByYOh8rvk2cEpmvg1cCqwAnJWZ08rrjwbeBZCZ92fmPeV1nwF+DezSgff03cycUcYzl8w8DxgD3EvxR8y32zpJFn3i08r38h6K5D8hIjYuY7grM2e3E0vV/8vMCZn5MnA98/iMMnNMZt5axj+Joh+8vffcEfsCz2TmheXn+SBwFXBQed07MvORzJydmQ8Dl3TBdS/KzEfL7/ue87s+xfdi04hYNjNfycwHFvLaEmCTu7rPS8AKMf++3ZbaYItny31zztHqD4I3gKU7G0hmvg4cAhwNPB8RN5bJqr14WmJavbL9QifieSkzZ5XPWxLui5Xj01teHxEbRsQNEfFCREylGHfQZnN+xaTMfLOdMucBmwM/z8wZ8yl3J0XN9T3l8zsoktwu5XZndOgzioiVI+LSstl5KvB72n/PHbEWsH3ZffFqRLxK8cfjKuV1t4+I28suiCkU34uFve5zHb0+cCBFi9OzEXFnROywkNeWABO6us8/gBnA/vMpM4Hil1+LNct9C+J1iqblFqtUD2bmiMzcjaKm+jhFomsvnpaYxi9gTJ3xS4q4NsjMZYHjgWjnNfNdKjEilqYYlPh/wEkRMWg+xVsS+s7l8ztpP6Ev7FKNp5bn2KJ8zx+h/ffcEc8Bd2bmcpXH0pn52fL4H4DrgCGZORD4VeW6bb2n+X632njdfK+fmSMzczhFV9M1FC090kIzoatbZOYUiv7js8vBYEtGxGIRsVdEnFEWuwQ4ISJWLAeXnUhRS1sQDwHviYg1oxiQd1zLgbImODwilqL4I+M1iubq1m4CNoziVrt+EXEIsClwwwLG1BnLUPTzv1a2Hny21fEXKfqzO+MsYFRmfgq4kSJxzcudwHuBAZk5DriLoul4MPDgPF6zIDFVLUPxbzElIlYHvr4Q56q6geLf8aPld26xiNg2IjapXPflzHwzIrYDDqu8dhLFd6P6vub53ers9SNi8SjmPhhYdsVMpe3votRpJnR1m8z8CfAV4ASKX5TPAcdQ1EoAfgCMAh4GHgEeKPctyLVuBS4rz3U/cyfhPmUcE4CXKWqdrRMmWdy/vC/wVYoug28A+2bm5AWJqZO+RpFYplG0HlzW6vhJwG/KJtyDaUdEDKdIyC3v8yvA1lGO7m8tM5+kSK53ldtTgaeBuyvdBq39H0Vf8KsRcc08yszP94CtgSkUf3BcvQDneIfMnAbsDnyY4t/8Bf47eBDgc8DJETGN4o/IyyuvfQM4Bbi7fF/vbue7tSDX/yjwTNnNcDRFc7y00CJzYVvNJElSo1lDlySpCZjQJUlqAiZ0SZKagAldkqQmsCgv6LBQll1+UK646pBGhyF1yotT25uHRuqdXh//5OTMXLG7r9N32bUyZ75j4sVOy+mTRmTmnl0QUoeZ0BfQiqsO4fQ//KnRYUid8uObn2p0CNICufe4XVvP4tgtcuZ0+m/U7p2h7XrzobO7YtbDTjGhS5I0R0BNV8CtZ9SSJGku1tAlSWoRQHTFkgI9z4QuSVJVTZvcTeiSJFXVtIZezz9DJEnSXKyhS5I0R31HuZvQJUmqqmmTuwldkqQWQW1r6PWMWpIkzcUauiRJc4RN7pIkNQWb3CVJUqNYQ5ckqcomd0mS6s770CVJqr8aL85Szz9DJEnSXKyhS5JUZZO7JEl1Zx+6JEnNoY996JIkqUGsoUuS1KLGi7OY0CVJqqrpbWsmdEmS5qjvoLh6Ri1JkuZiDV2SpCqb3CVJagI1bXI3oUuS1CKitjX0ev4ZIkmS5mINXZKkKpvcJUlqAja5S5KkRrGGLknSHPWdWMaELklSVU2b3E3okiS1qPHiLPWMWpIkzcUauiRJc9iHLklSc7APXZKkJlDTGno9o5YkSXOxhi5JUpVN7pIk1Vw4KE6SpOZQ0xp6Pf8MkSRJc7GGLklSRdS0hm5ClySpFNQ3odvkLklSE7CGLklSiygfNWRClyRpjqhtk7sJXZKkiromdPvQJUlqAtbQJUmqqGsN3YQuSVKFCV2SpLqr8Sh3+9AlSWoC1tAlSSqFt61JktQcTOiSJDWBuiZ0+9AlSWoC1tAlSaqoaw3dhC5JUgtvW5MkSY1kQpckqSIiFvrRgWvsGRFPRMSYiPhWG8fXjIjbI+LBiHg4IvZu75w2uUuSVOqJ+9Ajoi9wNrAbMA4YGRHXZeboSrETgMsz85cRsSlwE7D2/M5rQpckqaIHBsVtB4zJzKfL610KDAeqCT2BZcvnA4EJ7Z3UhC5JUtdbISJGVbbPzcxzy+erA89Vjo0Dtm/1+pOAWyLiWGAp4P3tXdCELklSVddU0Cdn5rCFeP2hwEWZ+ZOI2AH4XURsnpmz5/UCE7okSS2iR5rcxwNDKttrlPuqPgnsCZCZ/4iIJYAVgInzOqmj3CVJquiBUe4jgQ0iYp2IWBz4MHBdqzL/Ad5XxrMJsAQwaX4nNaFLktSDMnMmcAwwAniMYjT7oxFxckTsVxb7KnBURPwTuAQ4IjNzfue1yV2SpIqemPo1M2+iuBWtuu/EyvPRwE6dOacJXZKkkuuhS5LULOqZz+1DlySpGVhDlySpRc/cttYtTOiSJFXUNaHb5C5JUhOwhi5JUkVda+gmdPWoB+++nQt/dCKzZ8/mffsfygGfOGau47dc8Vtuvvw39OnThyWWXIrPnHAGQ9bbcM7xSc+P58sH7srBR3+V/T52dE+Hr0XUu9dZni+/f3369Amu++fz/O6e595R5n0br8in/mctMuGpia/x3esfB2DlZftz/F4bsvIy/UngK1c8wvNTZvTwO1Cn1DOfm9DVc2bNmsX/nfZtvvPLSxi08qocd/jeDNtl97kS9v/sdQC7H/QxAEbecQu/+en3OOHsi+cc/81PTmKrnd7b47Fr0dUn4Gu7b8AXLn2YidNmcOERW3PXUy/xzEtvzCkzZPkBfGyHIXz6dw8xbcZMll9ysTnHvrvvxlz09/9w3zOvMGCxPsye71xf6g3qWkO3D109Zsy/HmSVIWuz8hprsdhii7PTHsMZdceIucosufQyc57PmP4GUflT+b7bb2al1ddkyHob9VjM0qarLsu4V6YzYcqbzJyd3Dp6Iu/ZYPBcZYa/a1Wuun8C02bMBOCVN94GYO3BS9I3gvueeQWA6W/PZsbMeS6WJS0Ua+jqMS9PfIHBK682Z3vQyqvy1L8efEe5my+7iBt+fy4z336L7/76cgCmv/E611x4Nt/51aVc/9tf9VjM0orLLM7Eaf9tIp84bQabrbbsXGWGDBoAwLkfGUqfCM7/2zPcM/YV1hw0gGkzZnLaAZuy6nJLMPKZVznnjqetpfdiHVxcpVfq0Rp6RGRE/KSy/bWIOKkbrnN8q+2/d/U11H32POQIfnH93zn8i9/mqvPPAuCKX/2EfT9yFAOWXKrB0Unv1LdPsMagAXz2D//kO9c9xnF7bcjS/fvSt08wdI2B/L/bnuYTFz3A6sstwT5brNLocNWOHlhtrVv0dA19BvDBiPhhZk7uxuscD5zaspGZO3bjtdRBg1ZahZdenDBn++UXn2fwivP+5bbTHsM579TjAHjqXw9yz59v5PdnnsLr06YSffqw2OL92evDR3Z73Fq0TZr2Fist03/O9krL9GfStLkHtU2cNoNHJ0xl1uzk+Slv8p+XpzNk+SWZOG0GT058jQlT3gTgzicns/lqy3J9j74DdZY19I6ZCZwLfLn1gYhYMSKuioiR5WOnyv5bI+LRiDg/Ip6NiBXKY9dExP3lsU+X+04DBkTEQxFxcbnvtfLnpRGxT+WaF0XEhyKib0T8qLzuwxHxmW7/JBZB6282lOf/M5YXx/+Ht99+i7tHXMuwXXefq8zzzz495/kDd/2ZVYesA8D3L/gj59x0L+fcdC/7HP4pPvjJY03m6hGPPT+VIYMGsOrAJejXJ9ht05W4a8xLc5X565OT2XrN5QAYOKAfaw4awPhXp/PY89NYZol+LDegGCQ3bK3lGVsZTCd1pUb0oZ8NPBwRZ7Tafxbws8z8W0SsSbFO7CbAd4HbMvOHEbEn8MnKaz6RmS9HxABgZERclZnfiohjMnNoG9e+DDgYuLFcVP59wGfLc07JzG0joj9wd0Tckplju/KNL+r69uvHJ7/5A0753GHMnj2b9w4/hCHrbcSl5/yI9TZ9F9vuujt/uuwiHrn3Lvr268fSyw7kmO+f2eiwtYiblfDjW8Zw1iFb0CeCGx5+gbGT3+Condfm8eencdeYl7hn7Ctsv84gLvnUMGbNTn5++9NMfbMYIPfz257mF4duCcATL77GtQ8938i3o46oZwWdaGe99K69WMRrmbl0RJwMvA1MB5bOzJMiYiIwoVJ8RWAj4G/AAS3JNSJeBjbMzMll//sBZfm1gT0y856W67Rx3SWAJ4ENgD2BgzPz8Ii4EtgSaPnTeSDwmcy8pVX8nwY+DbDCqqtv88ub7uuiT0bqGT+++alGhyAtkHuP2/X+zBzW3dfpv/IGufrhZy30ecb+bJ8eibeqUaPczwQeAC6s7OsDvDsz36wWnFdfRkTsCrwf2CEz34iIO4Al5nfRzHyzLLcHcAhwacvpgGMzc8S8Xlu+/lyKLgPW2/RdjlOVpGZT48VZGnIfema+DFzO3M3ntwDHtmxEREuT+d0UzeRExO7A8uX+gcArZTLfGHh35VxvR8RitO0y4EhgZ+Dmct8I4LMtr4mIDSPC4dSSpNpo5MQyPwFWqGx/ARhWDkobDbTM6/k9YPeI+BdwEPACMI0iGfeLiMeA04B7Kuc6l6Kf/mLe6RZgF+DPmflWue98YDTwQHmdX+M9+pK0yAkgYuEfjdCjSavar52ZLwJLVrYnUzSDtzaFom98ZkTsAGybmS33jOw1j+t8E/jmPK77NjCoVfnZFLe6zXX/uiRpUVPfiWXqUAtdE7g8IvoAbwFHNTgeSVITq2k+7/0JPTOfArZqdBySJPVmvT6hS5LUk2xylySp7ho4qG1huXyqJElNwBq6JEmlAPr0qWcV3YQuSVJFXZvcTeiSJFXUdVCcfeiSJDUBa+iSJLWo8Sh3E7okSaViLvd6ZnQTuiRJc9R3Lnf70CVJagLW0CVJqqhpBd2ELklSVV2b3E3okiS1qPEod/vQJUlqAtbQJUkqeduaJElNoqb53CZ3SZKagTV0SZIqbHKXJKkJ1DSfm9AlSZoj6ltDtw9dkqQmYA1dkqRScdtao6NYMCZ0SZLmqO9qayZ0SZIqaprP7UOXJKkZWEOXJKnCJndJkuquxqutmdAlSSrVeXEW+9AlSWoC1tAlSaqoaw3dhC5JUkVN87lN7pIkNQNr6JIkVdjkLklS3XnbmiRJ9Rc1nsvdPnRJkpqANXRJkipqWkE3oUuSVNWnphndhC5JUkVN87l96JIkNQMTuiRJpYjiPvSFfbR/ndgzIp6IiDER8a15lDk4IkZHxKMR8Yf2zmmTuyRJFX26uck9IvoCZwO7AeOAkRFxXWaOrpTZADgO2CkzX4mIldo7rwldkqSKHrgPfTtgTGY+XV7vUmA4MLpS5ijg7Mx8BSAzJ7Z3UpvcJUnqeitExKjK49OVY6sDz1W2x5X7qjYENoyIuyPinojYs70LWkOXJKmiiyrokzNz2EK8vh+wAbArsAbw14jYIjNfnd8L2hQRk4Ds6JUzs932fUmSerOgmP61m40HhlS21yj3VY0D7s3Mt4GxEfEkRYIfOa+Tzq+GfjadSOiSJKlDRgIbRMQ6FIn8w8BhrcpcAxwKXBgRK1A0wT89v5POM6Fn5kkLE60kSXXU3aPcM3NmRBwDjAD6Ahdk5qMRcTIwKjOvK4/tHhGjgVnA1zPzpfmdt1N96BGxPLA5RVPBn8qh9EsAb2Xm7M6/LUmSepEO3ke+sDLzJuCmVvtOrDxP4Cvlo0M6lNAjoh9wKvB5YABFU/y2wCvAVcAo4LsdvagkSb1Vs0/9egrFPXHHAOvCXCMGrgU+0MVxSZKkTuhok/vHgG9l5oXlDDdV/6ZI8pIk1VrQ/KutLUeRuNuyOEWnviRJtVfTfN7hJvd/UUxL15a9gAe6JhxJkhqrJxZn6Q4draH/ALgqIgYAV1AMihsaEQcAnwH266b4JElSB3QooWfmtRFxGHAG8Ily9/kUN8R/NDNHdFN8kiT1mGL51EZHsWA6fB96Zl4OXB4RGwGDgZeBJ8p75SRJagrNPihujsx8ojsCkSSpN6hnOu/E8qkRsUVE/CEixkTE6+XPP0TElt0ZoCRJal9HZ4rbH7ic4ta1K4GJwEoUI99HRcTBmXlNt0UpSVIPadQo9YXV0Sb30ylmhDu42mceEcdRjHo/nWJlGEmSaquYWKbRUSyYjja5DwHObz0Artw+j7nXdZUkqZ664B70RtXwO5rQRwGbzePY5jixjCRJDTXPJveIWLKy+RXg0ohYjKJpvaUP/QDgUxSLs0uSVHs17UKfbx/6axQzwrUI4IcUy6hW9wHci/O5S5KaQDMOivsEcyd0SZLUS80zoWfmRT0YhyRJDVfnUe6dnilOkqRm1oxN7nOJiEOAo4ANgSVaH8/MlbowLkmSGqKe6byDt62VK639BhgDrAFcB9xQvn4q8IvuClCSJLWvo/ehfx34PvD5cvuczPwEsA4wGXijG2KTJKlHRRSrrS3soxE6mtA3AO7OzFnALGBZgMycRjHt6zHdE54kST2rZU30hXk0QkcT+lSgf/l8PLBJ5VhQrI8uSVLt1XXq144OihsJbAmMoOg/PzEiZgJvAScC93RPeJIkqSM6mtB/CKxVPj+xfP5Lihr+SOAzXR+aJEk9r6Z3rXUsoWfmPZS18Mx8FRgeEf2B/pk5tRvjkySpxwSNG9S2sBZ4YpnMnAHM6MJYJElqrAYOaltY81tt7YxOnCcz85tdEI8kSVoA86uhH9SJ8ySwSCX05QYsxr6br9boMKRO+eiRp7ZfSFrENd3Ur5m5Tk8GIklSb9DR+7l7m7rGLUmSKlxtTZKkUtCETe6SJC2KXA9dkqQmUNeEbh+6JElNoFM19Cg6FtYAhgD/zMzXuyUqSZIaoFgtrZ5V9A7X0CPicxQrrT0L3AVsVO6/OiK+1D3hSZLUs/rEwj8aEndHCkXE14GfAucB/0sxELDFHcAhXR6ZJEkNUNf10Dva5P554MTMPCMi+rY69gSwYdeGJUmSOqOjCX0V4P55HJsNLNE14UiS1DgBtV1traN96GOAXeZx7D3A6K4JR5KkxurTBY9G6GgN/UzgnIh4C7iy3LdSRHwS+ApwVHcEJ0lST6tpBb1jCT0zz4+I5YETge+Vu28C3gBOysw/dFN8kiSpAzp8H3pm/igifgXsCAwGXgb+kZlTuis4SZJ6UkTUtg+9UxPLZOY0YEQ3xSJJUsPVNJ93LKGXk8rMV2aes/DhSJKkBdHRGvov5nMsy58mdElS7TX14iyZ2af1AxgEHAr8E9i0O4OUJKkntNyHvrCPRljg5VMz81XgsogYCPwa2LWrgpIkqVHq2ofeFfe/jwWGdcF5JEnSAlrgGjpARKwKfJUiqUuSVG8NXC1tYXV0lPsk/jv4rcXiwDLAm8AHuzguSZIaIqhnRl+YUe5vAuOAmzPzpa4LSZKkxigGxTU6igXTbkKPiMWAPwNjM3NC94ckSZI6qyOD4mYBtwEbd3MskiQ1XJ9Y+EcjtFtDz8zZEfEUxZrokiQ1tajpfWsd7UP/NnB6RDySmY90Z0CSJDVKU/ahR8R7gAcy8zXgBIoV1h6KiPHAi7Qa9Z6Z23VnoJIkad7mV0O/HdgBuA/4V/mQJKl5RX1niptfQp/zljLzyB6IRZKkhqvreuhdMfWrJElqsPYGxe0dER26XS0zf9sF8UiS1DA9NSguIvYEzgL6Audn5mnzKHcgcCWwbWaOmt8520voJ3YwtgRM6JKk2uvuFveI6AucDexGMePqyIi4LjNHtyq3DPBF4N6OnLe9Jvf3UszX3t5j2Q6/E0mSeq2gTxc82rEdMCYzn87Mt4BLgeFtlPs+cDrFVOvtai+hT8/M1zvy6MjFJEkSqwPPVbbHlfvmiIitgSGZeWNHT7pQy6dKktRMgi5rcl8hIqp93udm5rkdiiGiD/BT4IjOXNCELklSi66bi31yZg6bx7HxwJDK9hrlvhbLAJsDd5TT0K4CXBcR+81vYNw8E3pmekubJGmR0wP3oY8ENoiIdSgS+YeBw1oOZuYUYIWW7Yi4A/hae6PcTdqSJPWgzJwJHAOMAB4DLs/MRyPi5IjYb0HPa5O7JEmlLuxDn6/MvAm4qdW+Nm8Vz8xdO3JOE7okSRV1nfrVhC5JUkVN87l96JIkNQNr6JIklYL61nRN6JIktQiImra5m9AlSaqoZzqvb8uCJEmqsIYuSVKpWA+9nnV0E7okSRX1TOc2uUuS1BSsoUuSVFHTFncTuiRJ/xXetiZJUt3VeWKZusYtSZIqrKFLklRhk7skSU2gnunchC5J0n/VeC53+9AlSWoC1tAlSSrVeZS7CV2SpIq6Nrmb0CVJqqhnOq9vy4IkSaqwhi5JUkVNW9xN6JIktSgGxdUzo9vkLklSE7CGLklShU3ukiTVXhA1bXI3oUuSVFHXGrp96JIkNQFr6JIkleo8yt2ELklSi6hvk7sJXZKkiromdPvQJUlqAtbQJUmq8LY1SZJqLoA+9cznJnRJkqrqWkO3D12SpCZgDV2SpIq6jnI3oUuSVGGTu9QBt4y4mS0324jNNl6fH51x2juOz5gxg48cdgibbbw+O++4Pc8+8wwAf/nzrey43TYMG7oFO263DXfcflsPR65F2W47bsI///gd/nXtd/nakbu94/iaqy7PTb86lvsuO44R532R1Vdabs6xU744nPuv/DYPXnUCP/nGh3oybC1iTOjqMbNmzeJLX/g8117/Jx58eDRXXHoJj40ePVeZiy74P5ZfbnkefXwMx37xy3z7+G8CMHjwClx5zfWMeugRzrvgN3ziiI824i1oEdSnT3Dmtw5m+DHnsNWBP+CgPbdh43VXmavMD798ABffeB/bHfJDTj33T5x87H4AvPtd67DD0HXZ9uBT2eagU9hms7XYeZsNGvE21EEto9wX9tEIJnT1mJH33cd6663POuuuy+KLL85Bh3yYG66/dq4yN1x/LYd/9OMAfPDAD3HHbX8hMxm61VasttpqAGy62Wa8OX06M2bM6PH3oEXPtpuvzb+fm8wz41/i7ZmzuGLEA+y765Zzldl43VW5874nALhz5JPsu+sWAGRC/8UXY/HF+tF/8X7069eXiS9P7fH3oM6ILvmvEUzo6jETJoxnjTWGzNleffU1GD9+/DvLDCnK9OvXj2UHDuSll16aq8wfr76KoVttTf/+/bs/aC3yVltpIONefGXO9vgXX2H1FQfOVeaRJ8cz/H+HAjD8f9/FsksPYNDApbj34bH8ddRTjL31FMbecip//vtjPDH2xR6NX51UzuW+sI9G6DUJPSJmRcRDEfGviLgiIpbs5OtXi4gry+dDI2LvyrH9IuJbXR2zet7oRx/lhOO/yS/O+XWjQ5HmOO5nf2TnbdbnH9xX5FUAABUQSURBVJd8k523WZ/xL77CrFmzWXfICmy0zsqsv8cJrLfHt9l1uw3Zaav1Gh2umlSvSejA9MwcmpmbA28BR3fmxZk5ITNbRpwMBfauHLsuM985Aks9arXVVmfcuOfmbI8fP47VV1/9nWWeK8rMnDmTqVOmMHjwYADGjRvHIQcdwPkX/JZ11/OXonrGhIlTWGPl5edsr77y8oyfNGWuMs9PmsKHv3Y+Oxx6Ot/9xfUATHltOsPf+y7ue+QZXp/+Fq9Pf4sRdz/K9luu06Pxq/OiCx6N0JsSetVdwPoRMSgiromIhyPinojYEiAidilr8w9FxIMRsUxErF3W7hcHTgYOKY8fEhFHRMQvImJgRDwbEX3K8ywVEc9FxGIRsV5E3BwR90fEXRGxcQPff1Matu22jBnzFM+MHctbb73FFZddyj777jdXmX323Y+Lf/cbAK6+6kp2ee//EhG8+uqrfHC/ffj+Kaex4047NSJ8LaJGPfos66+5ImutNpjF+vXloD225sY7Hp6rzODlliLKdtavf2IPfnPtPQA898Ir7LzN+vTt24d+/fqw89Yb8PjYF3r8PajjikFxsdCPRuh1CT0i+gF7AY8A3wMezMwtgeOB35bFvgZ8PjOHAjsD01ten5lvAScCl5U1/ssqx6YADwG7lLv2BUZk5tvAucCxmblNef5z2ojt0xExKiJGTZo8qSvf9iKhX79+/OysX/CBffZg6BabcOBBB7PpZptx8kkncsP11wFwxCc+yUsvv8RmG6/P/zvzp/zglKJh5Vfn/IJ//3sMP/zByWy/zVC232YoEydObOTb0SJi1qzZfPn0y7n+nM/z0NUncNUtD/LY0y/wnc/uwz67FIPf3jNsAx6+5js8fM2JrDR4GU4/fwQAV//5QZ5+bjKjLj+e+y47jkeeGs9Nf/1XI9+OOqCuNfTIzAZdem4RMYsiiUNRQ/8qcC9wYGY+XZZ5DtgM+BxwAHAxcHVmjouItYEbMnPziDgCGJaZx5Svm7MdEYcB78nMoyPijxSJ+x/AJOCJSkj9M3OTecW7zTbD8u57R3XJe5d6yvLbHtPoEKQF8uZDZ9+fmcO6+zqbbLFVXvjH2xf6PDtssHyPxFvVm2aKm17WuOeIeTRbZOZpEXEjRT/53RGxB/BmB69zHXBqRAwCtgFuA5YCXm19fUnSIqieE8X1vib3Vu4CDgeIiF2ByZk5NSLWy8xHMvN0YCTQur97GrBMWyfMzNfK15xFUaOflZlTgbERcVB5rYiId3XLO5Ik9Wreh949TgK2iYiHgdOAj5f7v1QOgHsYeBv4U6vX3Q5s2jIoro3zXgZ8pPzZ4nDgkxHxT+BRYHjXvQ1JUl3U9T70XtPknplLt7HvZWD/NvYf28YpngE2r7xu21bHL6q8/kpaNapk5lhgz06GLUlSr9BrErokSb1BTbvQTeiSJM2lphm9t/ehS5KkDrCGLklSqZgYpp5VdBO6JEktGjhKfWGZ0CVJqqhpPrcPXZKkZmANXZKkqppW0U3okiTN0bipWxeWCV2SpIq6DoqzD12SpCZgQpckqRRd9Gj3OhF7RsQTETEmIr7VxvGvRMToiHg4Iv4SEWu1d04TuiRJVd2c0SOiL3A2sBewKXBoRGzaqtiDwLDM3BK4EjijvbBN6JIkVfTAeujbAWMy8+nMfAu4lFZLdmfm7Zn5Rrl5D7BGeyc1oUuS1LNWB56rbI8r983LJ4E/tXdSR7lLklTRRaPcV4iIUZXtczPz3M7HEh8BhgG7tFfWhC5JUkUX3bU2OTOHzePYeGBIZXuNct/ccUS8H/g2sEtmzmjvgja5S5LUomeGuY8ENoiIdSJiceDDwHVzhRGxFfBrYL/MnNiR0E3okiT1oMycCRwDjAAeAy7PzEcj4uSI2K8s9iNgaeCKiHgoIq6bx+nmsMldkqSKnpj6NTNvAm5qte/EyvP3d/acJnRJkkqBU79KkqQGsoYuSVJFTSvoJnRJkuZS04xuQpckqaKu66Hbhy5JUhOwhi5JUkVdR7mb0CVJqqhpPjehS5I0l5pmdPvQJUlqAtbQJUkqFWur1LOKbkKXJKlFOChOkqSmUNN8bh+6JEnNwBq6JElVNa2im9AlSZojajsoziZ3SZKagDV0SZIqHOUuSVLNBbXtQjehS5I0l5pmdPvQJUlqAtbQJUmqqOsodxO6JEkVDoqTJKkJ1DSf24cuSVIzsIYuSVILV1uTJKlZ1DOjm9AlSSoF9a2h24cuSVITsIYuSVJFTSvoJnRJkqpscpckSQ1jDV2SpAqnfpUkqRnUM5+b0CVJqqppPrcPXZKkZmANXZKkUjj1qyRJzcFBcZIkNYN65nP70CVJagbW0CVJqqhpBd2ELklSlYPiJEmqvajtoDj70CVJagLW0CVJKgX1bXK3hi5JUhMwoUuS1ARscpckqaKuTe4mdEmSKuo6yt2ELklSixovzmIfuiRJTcAauiRJpcCpXyVJag41zegmdEmSKuo6KM4+dEmSmoA1dEmSKuo6yt2ELklSRU3zuQldkqS51DSj24cuSVITsIYuSVJFXUe5m9AlSSrVeT30yMxGx1BLETEJeLbRcTSpFYDJjQ5CWgB+d7vPWpm5YndfJCJupvh3XFiTM3PPLjhPh5nQ1etExKjMHNboOKTO8rurRnJQnCRJTcCELklSEzChqzc6t9EBSAvI764axj50SZKagDV0SZKagAldkqQmYEKXJKkJmNAlSWoCJnRJ6gERdZ1QVHVhQldttfyCjIhVI2K1RscjzUtERJa3FEXEJhExxASvruZta6q1iNgf+BIwBXgc+HlmjmtsVFLbIuIY4GDgIWBz4H3pL2F1EWvoqq2I2AL4CrAvcB/wXorELvU6EbEHcACwDzAVmNnYiNRsTOiqs1nADcBBFL8kP5yZ0yJis8aGJbXpVYqZ5D4FbAfsm5kZEbs3Niw1C9dDV+1ExKbAZsCDwM7AKsChmfl0ROwFfCciPpiZLzQyTgkgIo4EFgP+AtwEPJ2Z25bHjgD2joh7M9PWJS0Ua+iqo52AL2fmGIpfkk8Bu0bEYcCPgVNN5mqUiGj9e/UZYD9gPPBZYLWIODwivg18ATjZZK6u4KA49XotI4Qjol9mziz3XQzck5k/j4hPAWsBg4BrM/OW6qhiqZEiYjBwMnB5Zt4ZER8CtgcSuCAzH29ogGoaJnT1WhGxIfCuzLwiIrahGPQ2JjOviYj3A3tk5tcr5RfLzLcbFa8WbeXYjW0y87cRsS/wOeDLwNPAh4DjgXdn5usNDFNNzCZ39WZ9gIkRsQwwDlgc+HxE/JxihPBeEfHRSnlHDashymb2wcBNEbEOcDvwCHAs8BtgFHAnsGfDglTTs4auXi0i+gGTgW9m5q8jYgDwE+BZil+WjwP7Z+ZrDQxTi7CIWDwz3yqfrwF8D/hnZv6/iFge+BhwCLAm8DeKAZz+4lWXM6GrV4mIJYHdMvPaiNgeeAsI4GbglMw8q6wNrUIxQcdTmXlj4yLWoiwiBlIM0vwrsCPFaPbFgfcBY4EzM3NWeWfGuygS/ehGxavmZkJXrxMRFwHDgDeBozLzwYjYGvgzcEJmntOqvAPg1OPK1qO+wBEUtfDBwCblAM4PAHtQtCT9rGUwp9Sd7ENXr1GZ2/qHFCPWZ2bmgwCZ+QDwfuCsiPhi9XUmc/W0iNgYOCczZ1DM+rYN8A+KpA5wK8U95xsDxzQkSC1yrKGrV6jcmtYHWBpYHrgAeDsz96yU2wBYOzNvbVCoEhHRl+I7uj7wGLAqMBxYgyLRPxYRGwEbUdxeObFhwWqRYUJXw1WS+e7Au4EXMvPc8thtwOvAD4AzgAMy82Wb2dUIEdEnM2dXts8DNgX2BpYFPlP+nAKsSDGY00lj1CNsclfDlcl8T+BnwF3AyRFxdkQMysz/BV6jGDn8k8x8ueU1jYtYi6Lyj8jZ5fM9yj70oylGrv8RmAacDfwb+B/gbJO5epI1dDVU2cS+DMW9ut8BVgZ+RDFN5qvAsZn5SkQsl5mvWjNXo0XE5ylumdy7XD+gD0Xr0VCKBYImR8QSmflmQwPVIseEroaoNLMvmZlvlNNjDqJI7DsDA4AXgJ9TzHU9vYHhSgBExM7AWcCemTmxnMHwBYo/Pk8D1gM+AMz2D0/1NFdbU4+rJPPtgXMi4ojMfCQiVqK473x5itHCtwFXm8zVKG20CL1NMQvc4RGxGrAX8BxwXGYeGxErZ+asRsQq2YeuHlfpMz+K4j7eERGxRWY+BtwHXExxy8/ZmTmygaFqEVZN5hGxRkSsQPH9nAFsCFyfmZtT3Gs+DCAzX2xUvJJN7upx5VzXNwNHZubfI+JEisk59qEYUDSM4h70+xoXpRZVrWvlEfEF4DCKuy2eAo6prPp3AHAicFC5nK/UMNbQ1QgvAfdSrEJFZp5MMVJ4BLByZv7dZK4GmtMVWfaZHwF8kKJvfE2KFiTKVqZjgI+ZzNUbmNDV7VpmgIuIgRExMDOnUtyr+8FKsYuBScC1EbF0A8KUiIjdgN9GxLfKJXonAfcAz2fmG5m5N7BWRBxIsXraoZn5SANDluZwUJy6XWVu668Ar0TEPcC3gEvK1ammUyT3Iykm5liK4t5zqceUNe6Tgd8BK1E0sz9AMX3rFsDDZdHbKL7W0ym+u1KvYEJXt2g1oOjdwPHAQcBHKBZcOSMiDqGYn31NisUtVqBYuWp222eVukdEDKIYiDk8M6+PiDUp7i1/CHgDODci/kAxZ8IHgIsaFas0LyZ0dbmIWBHYPyIuKdcpX5xiwZUdKOa73r0s+lbLymkRsSNwHsXa5pMaELYWYeV0wh8AzoiIOzPzPxGRFKunnRcRUynmaV+ZYgDckw0NWGqDCV3dYSdge6B/uRRqX4qE/hKwVznj227A0RFxdLn/WeB9mflsg2LWIi4zb4yI2cD9ETEC6A/8oTx2ZUODkzrA29bUZSKib2bOKlei2h/YFRidmb+MiO8DB1A0u29JcavPNzLzxoYFLLWhHAx3C7BKORvcACc3Uh2Y0NUlyqUiP0Xxi/CvmTkjIvaimElrdGb+KiJOolhmcjnggswc4dzs6o3K7+6Pgfe69KnqwoSuLhERu1BMifkUcDmwLsUiK7tR9KFPAC4qR7y7cIV6vYgYDnyXYqKj9A9P9XYmdHWZiPgf4AaK/vMDKeZkPwAYB6wPnARcAFBdU1rqrSJi6XJgp9TrOShOXSYz/xYRhwJXAjtm5rSIuIHiHt5PA2NN5KoTk7nqxBq6ulxE7E2x7Om2mflyua9lhTX7zCWpG1hDV5fLzJvK238ej4iNMvOVliRuMpek7mENXd0mIvYBXs/MOxodiyQ1OxO6up3N7JLU/UzokiQ1AZdPlSSpCZjQJUlqAiZ0SZKagAldaiUiToqIrDwmRMRVEbFeN15z3/Jaa5fba5fb+3biHAdHxBFdGNPSZQzzPOeCxFm+7qKIGLXQQRbnuiMiXA1NizzvQ5faNgXYs3y+LvB94C8RsVlmvt4D13+eYv34xzvxmoOBFYCLuiMgSb2bCV1q28zMvKd8fk9E/Ae4C9gbuKJ14a5eYjMzZwD3tFtQkko2uUsdc3/5c22AiHgmIn4SEd+JiHHA1HJ/n4j4VkSMiYgZEfFkRHy8eqIonBQREyNiWkT8Fli2VZk2m7Ij4qiIeCQi3oyIFyPiyogYGBEXUSyIs0ulq+CkyuuGR8So8nUvRMQZEbFYq3MfWMY7PSL+Cmy8IB9URHwsIv4WES9HxCsRcXtEDJtH2f0j4vEyrr9FxKatjrf7eUoqWEOXOmbt8ucLlX2HAY8Cn+O//y/9HPg4cDLwAMXysRdExEuZeUNZ5gvAicCpFLX+DwJntBdARJxQnvcc4OvAksA+wNIUXQJrUqw1/7nyJePK1x0MXAL8GjgeWA/4IcUf9F8ry2wNXAb8EfgisDnFMrgLYm3gt8C/KZbOPRS4q+yueLpSbi3gp8B3gOnA94AREbFBZXndjnyekgAy04cPH5UHxTKvkymSdD9gQ4q13qcCq5ZlnqHo516i8rr1gdnAx1ud77fAyPJ5X4q14X/ZqsytQAJrl9trl9v7ltvLAW8AP51P3FcCd7TaF8CzwIWt9n+CIokOLrcvB0ZTTjZV7vt2GcMR87nmXHG2cbxP+Rk+DpxY2X9R+bodK/vWAmYCR3f08yy37wCubPT3xoePRj9scpfaNhh4u3w8QTEw7pDMfL5S5i/535okwPsoEtAfI6JfywP4CzA0IvoCQ4BVgWtbXe/qduLZARgAXNjJ97EhRc398lYx3QYsQVETB9gOuC4zq1NHthdTmyJik4j4Y0S8CMyi+Aw3KmOpmpiZf2/ZyMxnKbo2tit3deTzlFSyyV1q2xTg/RS1yBeACa2SHcCLrbZXoKiBT5nHOVcFVimfT2x1rPV2a4PLn8/Pt9Q7rVD+vGkex4eUP1dZgJjeISKWAW6h+Gy+QtE68CZwPsUfEO2dfyLF5wQd+zzHdTZGqVmZ0KW2zczM9u6Tbp3gX6ZoMt6JombZ2kT++//cSq2Otd5u7aXy56oU3QEd9XL589PAg20cH1v+fGEBYmrLDsAawG6ZOeeWu4gY2EbZts6/EsW4BOjY5ympZEKXus5tFDXKgZl5a1sFIuI5iuQ5HLi5cuiD7Zz7HxR93h+nHMjWhrd4Zy34CWA8Rd/8efM5/0hgv4g4rtIS0V5MbRlQ/pzRsiMidqToa7+/VdmVImLHlmb3iFgT2Jr/diu0+3lK+i8TutRFMvOJiPgVcGlEnAGMokiwmwEbZuanMnNWeezHETGZYpT7gcAm7Zz71Yj4PnBKRCxO0YTen2KU+/cyczzFwLPhEbE/RVP0hMycEBFfBX4XEcsCf6JI/OsC+wMfysw3gNOBeyn62v+Pom/9kwvwMdwDvAacV77PNSgGGY5vo+xk4Pfl6P2WUe4TKSfG6cjnuQDxSU3LQXFS1/o8xS1kH6NIuhdRJN2/VsqcSXHL2tHAVRS3nX2jvRNn5g+Bz1L07V9LcRvacsC0ssg5FP3XF1DUuD9dvu4yihaBoRST4lxNcWvbAxTJnbJ74cPAVsA1FMn+kM6++cx8ETiIok/+WuBL5fsc00bxZylaG04CLi3fxx6tBhp25POUhOuhS5LUFKyhS5LUBEzokiQ1ARO6JElNwIQuSVITMKFLktQETOiSJDUBE7okSU3AhC5JUhP4/3fkmHdhh0kGAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 576x432 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aZP3zTtFZnh-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "bca165e4-7215-485c-ff89-4c03ae5ba455"
      },
      "source": [
        "print(\"ROC_AOC_Score: \", metrics.roc_auc_score(y_test, y_pred_prob))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ROC_AOC_Score:  0.8243181378594006\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U4jZtE8NLr4_",
        "colab_type": "text"
      },
      "source": [
        "### Support Vector Machines"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4shh5DvOKr5y",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 550
        },
        "outputId": "bd456eaf-35e9-435c-dc03-e11a38a5c99e"
      },
      "source": [
        "from sklearn.svm import LinearSVC\n",
        "\n",
        "vect = CountVectorizer(preprocessor=clean, max_features=1000) \n",
        "X_train_dtm = vect.fit_transform(X_train)\n",
        "X_test_dtm = vect.transform(X_test)\n",
        "\n",
        "classifier = LinearSVC(class_weight='balanced') \n",
        "classifier.fit(X_train_dtm, y_train) #fit the model with training data\n",
        "\n",
        "#Make predictions on test data\n",
        "y_pred_class = classifier.predict(X_test_dtm)\n",
        "\n",
        "#calculate evaluation measures:\n",
        "print(\"Accuracy: \", metrics.accuracy_score(y_test, y_pred_class))\n",
        "print(\"AUC: \", metrics.roc_auc_score(y_test, y_pred_prob))\n",
        "cnf_matrix = confusion_matrix(y_test, y_pred_class)\n",
        "plt.figure(figsize=(8,6))\n",
        "plot_confusion_matrix(cnf_matrix, classes=['Negative','Positive'],normalize=True,\n",
        "                      title='Confusion matrix with normalization')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/svm/_base.py:947: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
            "  \"the number of iterations.\", ConvergenceWarning)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Accuracy:  0.8318646725413098\n",
            "AUC:  0.8243181378594006\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAG7CAYAAAA12GczAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5hdZbmw8ftJQiD0EmoCBJBepAQUBMEjSChSVJpYQAVRsKEewBIBC+XY+DygBg9ERaUpEDAaVEQBKYlIkR6aJKEkdEIJSZ7vj7VmWBlmMpNpe9bO/eNaV/bqz94zzLPfst43MhNJklRvgxodgCRJ6jkTuiRJTcCELklSEzChS5LUBEzokiQ1ARO6JElNwISuASsihkXElRHxfERc0oPrHB4RV/dmbI0SEbtExH19fI+XImL9hex/JCJ278sY+kNEnBwRF5Sv1ynf9+Bevkef/7ykFiZ09VhEfDAippR/EB+PiD9ExM69cOkPAKsDq2TmQd29SGb+KjPf0wvx9KmIyIh4y8KOyczrMnPjvowjM5fNzIfKmMZHxLf68n4DQWb+p3zf83pynbY/w/74eUktTOjqkYg4Hvgh8B2K5LsOcA6wfy9cfl3g/syc2wvXqr2IGNLoGBppcX//Uqcy08WlWwuwAvAScNBCjlmSIuHPKJcfAkuW+3YDpgFfBJ4CHgeOLPedAswBXi/v8XHgZOCCyrVHAQkMKdePAB4CXgQeBg6vbL++ct5OwGTg+fLfnSr7rgW+CdxQXudqYHgH760l/v+uxH8AsDdwP/AM8JXK8TsANwLPlcf+LzC03Pf38r3MLt/vIZXrnwA8AfyyZVt5zgblPbYt19cCZgK7tRPrkcCVlfUHgEsq648BW5evE3gLcHT5+c8pY7qy3P8I8CXgjvIzvAhYqoPP6AjgeuC7wLPlz2Wvyv61gAnl+5gKHFXZdzJwKXAB8ALwifLn8y3gHy0xAasAvyqPmQyMqlzjrPK9vQD8E9ilzfUvaPu7BOxYXrtleRV4pCc/w8o9Ny3fw3PAXcB+lX3jgbOB31P87t0MbNDo/89d6rM0PACX+i7AGGAuZULt4JhTgZuA1YBVyz/E3yz37VaefyqwBEUifBlYqdzf+ge3g/XqH+Flyj/aG5f71gQ2L18fQZnQgZXLxPLh8rzDyvVVyv3XAg8CGwHDyvXTO3hvLfGPLeM/iiKh/hpYDtgceAVYrzx+O+Dt5X1HAfcAn69cL4G3tHP9Myi+GA1rJ0EcBdwNLA1MAr7bQazrl0lkEEUSfZQ3vhisX34Gg9rGUSaZb7W51iPALeV1Vi7fxzEd3PcIii8FRwGDgU9RfLGLcv/fKWp0lgK2Lj+//6r8vF+n+JI0qPLzmErxZWaF8r3fD+xefq6/AM6v3P9DFAl/CMUXxycov3zQQUJvE/8SwN+A03rwM5xWudZU4CvAUOC/KBL3xpXP+mmKLw1DKL6kXNjo/89d6rNY5a6eWAWYlQuvEj8cODUzn8rMmRQl7w9X9r9e7n89MydSlGy62+Y4H9giIoZl5uOZeVc7x+wDPJCZv8zMuZn5G+Be4L2VY87PzPsz8xXgYopE05HXgW9n5uvAhcBw4KzMfLG8/93AWwEy85+ZeVN530eAnwK7duE9fSMzXyvjWUBmnkuRJG6m+BLz1fYukkWb+Ivle3knRfKfERGblDFcl5nzO4ml6v9l5ozMfIailLywz+jRzDw3i/bpn5dxrh4RawPvAE7IzFcz8zbgZ8BHKufemJmXZ+b8yvs/PzMfzMzngT8AD2bmn8vfw0uAbSrv+4LMfLr8zL9H8cVoUX6//h/F5/bV8nrd+Rm2eDuwLMUXxDmZeQ1wFcWXyhaXZeYt5Xv5FQv/XKUFmNDVE08Dwztp22wpDbZ4tNzWeo02Xwhepvijt0gyczZFFecxwOMR8fsyWXUWT0tMIyrrTyxCPE/nGx2pWhLOk5X9r7ScHxEbRcRVEfFERLxA0e9g+EKuDTAzM1/t5JhzgS2AH2Xmaws57m8UJcZ3lq+vpUhGu5bri2JRPqPWYzPz5fLlshQ/i2cy88XKsW1/Fo+1c722n2+7nzdARHwpIu4pn5R4jqJU39ln3nLuJyk+rw+2fNnp5s+wxVrAY22+OPXkd09agAldPXEj8BpFlWhHZlB0bmuxTrmtO2ZTVC23WKO6MzMnZeYeFCXAeykSXWfxtMQ0vZsxLYofU8S1YWYuT1H1Gp2cs9DpECNiWYp+Cf8HnBwRKy/k8JaEvkv5+m90ntD7cjrGGcDKEbFcZVvbn0W37x8Ru1D0bziYohlnRYo2/84+85Zzvwnsn5kvVHZ152fYYgawdkRU/+721++eFgMmdHVbWeU5Fjg7Ig6IiKUjYomI2CsiziwP+w3wtYhYNSKGl8df0M1b3ga8s3xmeAXgpJYdEbF6ROwfEctQfMl4iaK6uq2JwEblo3ZDIuIQYDOKqs++thxFO/9LZe3Bp9rsf5KiPXtRnAVMycxPUHSm+slCjv0b8C5gWGZOA66j6AexCvCvDs7pTkxdkpmPUfSpOC0iloqIrSg6P3b396Ot5Sj6IMwEhkTEWGD5zk4qmwIuBj6Smfe3c83u/gxvpih1/3f5/8luFE09F3bt7UgLZ0JXj5TtkscDX6P4w/kYcBxweXnIt4ApFD2i7wRuLbd1515/ouhRfQdFj+VqEh5UxjGDosf0rrz5jy2Z+TSwL0UHqacpSnD7Zuas7sS0iL4EfJCiTfZcivdSdTLw84h4LiIO7uxiEbE/RUJueZ/HA9tGxOHtHV8mp5coEjllyfMh4Ibs+Pnr/wM2K2O6vINjeuIwis5lM4DLKPoL/LmXrj0J+CNFp7lHKXqrt1eF39a7KR7BvLQcW+GliGjpj9Htn2FmzqFI4HsBsyg6A34kM+/txnuT3qSlp6kkSaoxS+iSJDUBE7okSU3AhC5JUhMwoUuS1ASc7KCbYsnlMpZepdFhSItk83X9nVU9/fv2W2dl5qp9fZ/By6+bOfdNgzIusnxl5qTMHNMLIXWZCb2bYulVWHK3dkfZlAasCed+uPODpAFo/VWHtR3hsU/k3FdYcuNOnxrt1Ku3nd3VEQR7jQldkqRWAVHP1uh6Ri1JkhZgCV2SpBYBRFeH5x9YTOiSJFXVtMrdhC5JUlVNS+j1/BoiSZIWYAldkqRW9e3lbkKXJKmqplXuJnRJkloEtS2h1zNqSZK0AEvokiS1CqvcJUlqCla5S5KkRrGELklSlVXukiTVnc+hS5JUfzWenKWeX0MkSaqxiBgTEfdFxNSIOLGd/etExF8j4l8RcUdE7N3ZNS2hS5JU1cdV7hExGDgb2AOYBkyOiAmZeXflsK8BF2fmjyNiM2AiMGph1zWhS5LUql/a0HcApmbmQwARcSGwP1BN6AksX75eAZjR2UVN6JIkVQ3qlTb04RExpbI+LjPHla9HAI9V9k0D3tbm/JOBqyPiM8AywO6d3dCELklS75uVmaN7cP5hwPjM/F5E7Aj8MiK2yMz5HZ1gQpckqUX/TM4yHVi7sj6y3Fb1cWAMQGbeGBFLAcOBpzq6qL3cJUmqiuj5snCTgQ0jYr2IGAocCkxoc8x/gHcX4cSmwFLAzIVd1BK6JEmt+r5TXGbOjYjjgEnAYOC8zLwrIk4FpmTmBOCLwLkR8QWKDnJHZGYu7LomdEmS+llmTqR4FK26bWzl9d3AOxblmiZ0SZKqajpSnAldkqQqx3KXJKnmutapbUCq59cQSZK0AEvokiRVWeUuSVITsMpdkiQ1iiV0SZJa9ctsa33ChC5JUlVNq9xN6JIkteifyVn6RD2jliRJC7CELklSK9vQJUlqDrahS5LUBGpaQq9n1JIkaQGW0CVJqrLKXZKkmgs7xUmS1BxqWkKv59cQSZK0AEvokiRVRE1L6CZ0SZJKQX0TulXukiQ1AUvokiS1iHKpIRO6JEmtorZV7iZ0SZIq6prQbUOXJKkJWEKXJKmiriV0E7okSRUmdEmS6q7GvdxtQ5ckqQlYQpckqRQ+tiZJUnMwoUuS1ATqmtBtQ5ckqQlYQpckqaKuJXQTuiRJLXxsTZIkNZIldEmSKqxylySp5nwOXZKkJlHXhG4buiRJTcASuiRJVfUsoFtClySpVRRV7j1dOr1NxJiIuC8ipkbEie3s/0FE3FYu90fEc51d0xK6JEkVfd2GHhGDgbOBPYBpwOSImJCZd7cck5lfqBz/GWCbzq5rCV2SpP61AzA1Mx/KzDnAhcD+Czn+MOA3nV3UErokSRW9VEIfHhFTKuvjMnNc+XoE8Fhl3zTgbR3Esi6wHnBNZzc0oUuSVOrF59BnZeboXrjOocClmTmvswOtcpckqSp6YVm46cDalfWR5bb2HEoXqtvBhC5JUn+bDGwYEetFxFCKpD2h7UERsQmwEnBjVy5qlbskSS2i73u5Z+bciDgOmAQMBs7LzLsi4lRgSma2JPdDgQszM7tyXRO6JEkV/TH0a2ZOBCa22Ta2zfrJi3JNq9wlSWoCltAlSaqo6+QsJnT1qz22Gcl3P74jgwcF4/98H9/93e0L7D/zyLfzzi3XAmDpJYew6gpLseaHfgHA2sOX4Zxj38nI4cuQCQd884/8Z+ZL/f4etPj521+u5tSvfon58+Zx8IeO4FOf+/IC+3/247O4+ILxDB4yhJVXGc6ZZ/2EEWuv27r/xRdfYM93bMMee72XU874YX+Hr0VVz3xuQlf/GTQo+OHR72Cfkycy/enZXH/mAVx1y6PcO+2NIYr/+/ybWl9/au/Neev6q7Su/+xzu3HGpbdxze3TWWapIcyf36V+IlKPzJs3j2+c+Hl+ccnvWWOtERzwnp3Zfcy+bLjxpq3HbL7l1lzxpxsYtvTSXHD+OE4/5av86GcXtO7/wWmnsP2OOzcifHVDXUvotqGr32y/4ao8+PgLPPLki7w+dz6XXP8g++6wbofHH7zLBlx83YMAbDJyRYYMHsQ1txePas5+dS6vzOl0nAWpx26/dTLrjtqAdUatx9ChQ9n3gIP40x+uWuCYHXfelWFLLw3ANtvtwBMz3nik+M7bb2XWzKfYZbfd+zVuLX5M6Oo3a628DNNmvVFFPv3p2YxYZZl2j11n1WVZd7XluPbOGQBsuNYKPDd7DheesDs3fu9AvvPRHRg0qJ7folUvTzw+gzVHjGxdX3OtETz5eEdjgMDFvxrPru/eE4D58+fznbEnctIpp/V1mOolvTHTWqNK+P2a0CMiI+J7lfUvRcTJfXCfr7RZ/0dv30N966CdN+DyGx9urVYfMngQ79h0DU4cfzM7f/ly1lt9eT78ro0aHKW0oMsv+Q133n4rRx1XTJR1wXk/Zbfd92TNtUZ2cqYGkrom9P5uQ38NeF9EnJaZs/rwPl8BvtOykpk79eG91EUznpnNyOHLtq6PWGUZpj89u91jP7Dz+nxh3A2t69Ofns0djzzNI0++CMCEmx9hh41X4+d/6duYpTXWXIvHp09rXX98xnRWX3PEm467/m/XcPYPzuA3V1zNkksuCcCtU25m8k03cMH543h59mxenzOHpZdZlhPGfqvf4teisw29a+YC44AvtN0REatGxG8jYnK5vKOy/U8RcVdE/CwiHo2I4eW+yyPin+W+o8ttpwPDyknhf1Vue6n898KI2Kdyz/ER8YGIGBwR/1Pe946I+GSffxKLoSkPzOQtay7PuqstxxJDBnHQzhvw+8n/edNxG41YgZWWXZKb7nvqjXOnzmSFpYcyfPmlANhty7W497Hn3nSu1Nu22mY0jzw8lccefYQ5c+Zw1eWXsPuYfRY45q47buNrXzqOcb+8lOGrrta6/Yc/Gc8Ntz3Adbfex0knn8aBB3/QZK4+04he7mcDd0TEmW22nwX8IDOvj4h1KIbE2xT4BnBNZp4WEWOAj1fO+VhmPhMRwygmiP9tZp4YEcdl5tbt3Psi4GDg9+X4ue8GPlVe8/nM3D4ilgRuiIirM/Ph3nzji7t585MvnPsPrvzGXgweFPz8L/dxz2PP8vXDtuPWqTNbk/tBO2/AJdc/uMC58+cnJ/38ZiaesjcRwb8enMV5f7q3EW9Di5khQ4Zw8mk/4KMHv5f58+dx0GEfZaNNNuMHp5/Klltvy+5j9uW0U77C7NmzOe7jhwOw1si1OfeCSxscubqtngV0ootDxPbOzSJeysxly/FqXwdeAZbNzJMj4ilgRuXwVYGNgeuBA1uSa0Q8A2yUmbPK9vcDy+NHAXtm5k0t92nnvksB9wMbAmOAgzPz8Ii4FNgKeLk8ZQXgk5l5dZv4jwaOBmDYytsttefpvfPBSP3k7nM/3OgQpG5Zf9Vh/+yl6UgXasnVN8wRh5/V4+s8/IN9+iXeqkY9h/5D4Fbg/Mq2QcDbM/PV6oEdtWVExG7A7sCOmflyRFwLLLWwm2bmq+VxewKHABe2XA74TGZO6uT8cRRNBgxaaZQPQUtSs+mHyVn6SkMeW8vMZ4CLWbD6/GrgMy0rEdFSZX4DRTU5EfEeiqnkoChFP1sm802At1eu9XpELNHB7S8CjgR2Af5YbpsEfKrlnIjYKCLaf55KkqQBqJHPoX8PGF5Z/ywwuuyUdjdwTLn9FOA9EfFv4CDgCeBFimQ8JCLuAU4HbqpcaxxFO/2v2rnv1cCuwJ8zc0657WfA3cCt5X1+iqPoSdJiJ4CIni+N0K9Jq9qunZlPAktX1mdRVIO39TxF2/jciNgR2D4zXyv37dXBfU4ATujgvq8DK7c5fj7Fo24LPL8uSVrcNO458p6qQyl0HeDiiBgEzAGOanA8kqQmVtN8PvATemY+AGzT6DgkSRrIBnxClySpP1nlLklS3TWwU1tPOduaJElNwBK6JEmlgNpOzWxClySpoq5V7iZ0SZIq6topzjZ0SZKagCV0SZJa1LiXuwldkqRSMZZ7PTO6CV2SpFb1HcvdNnRJkpqAJXRJkipqWkA3oUuSVFXXKncTuiRJLWrcy902dEmSmoAldEmSSj62JklSk6hpPrfKXZKkZmAJXZKkCqvcJUlqAjXN5yZ0SZJaRX1L6LahS5LUBCyhS5JUKh5ba3QU3WNClySpVX1nWzOhS5JUUdN8bhu6JEn9LSLGRMR9ETE1Ik7s4JiDI+LuiLgrIn7d2TUtoUuSVNHXVe4RMRg4G9gDmAZMjogJmXl35ZgNgZOAd2TmsxGxWmfXtYQuSVKLcra1ni6d2AGYmpkPZeYc4EJg/zbHHAWcnZnPAmTmU51d1IQuSVKpZXKWni6dGAE8VlmfVm6r2gjYKCJuiIibImJMZxe1yl2SpN43PCKmVNbHZea4RTh/CLAhsBswEvh7RGyZmc8t7ARJklTqpTb0WZk5uoN904G1K+sjy21V04CbM/N14OGIuJ8iwU/u6IZWuUuSVNEPbeiTgQ0jYr2IGAocCkxoc8zlFKVzImI4RRX8Qwu7qAldkqR+lJlzgeOAScA9wMWZeVdEnBoR+5WHTQKejoi7gb8CX87Mpxd2XavcJUmq6I+R4jJzIjCxzbaxldcJHF8uXWJClySpRdeqzAckE7okSaWo8VjutqFLktQELKFLklRR0wK6CV2SpKpBNc3oJnRJkipqms9tQ5ckqRlYQpckqVSM9FbPIroJXZKkikH1zOcmdEmSqupaQrcNXZKkJmAJXZKkipoW0DtO6BExE8iuXigzV+uViCRJapCgGP61jhZWQj+bRUjokiSpcTpM6Jl5cj/GIUnSgLBY9HKPiJWALYC1gT9k5rMRsRQwJzPn90WAkiT1m6jvbGtdSugRMQT4DnAsMIyiKn574Fngt8AU4Bt9FKMkSf2mpvm8y4+tfRs4CjgOWB8W6DFwBfDeXo5LkiQtgq5WuX8EODEzz4+IwW32PUiR5CVJqrWg+WdbW5EicbdnKNA2yUuSVEs1zeddrnL/N7B/B/v2Am7tnXAkSWqsKDvG9WRphK6W0L8F/DYihgGXUHSK2zoiDgQ+CezXR/FJkqQu6FJCz8wrIuKDwJnAx8rNPwOmAx/OzEl9FJ8kSf2mmD610VF0T5efQ8/Mi4GLI2JjYBXgGeC+zHQ0OUlS02j2TnGtMvO+vghEkqSBoJ7pfBGmT42ILSPi1xExNSJml//+OiK26ssAJUlS57o6UtwBwMUUj65dCjwFrEbR831KRBycmZf3WZSSJPWTph76FTiDYkS4g6tt5hFxEkWv9zMAE7okqdaKgWUaHUX3dLXKfW3gZ207wJXr55b7JUmqt154Br1RJfyuJvQpwOYd7NsCB5aRJKmhOqxyj4ilK6vHAxdGxBIUVestbegHAp8ADu3LICVJ6i81bUJfaBv6SxQjwrUI4DSKaVSr2wBuxvHcJUlNoBk7xX2MBRO6JEkaoDpM6Jk5vh/jkCSp4ercy32RR4qTJKmZNWOV+wIi4hDgKGAjYKm2+zNztV6MS5KkhqhnOu/iY2vlTGs/B6YCI4EJwFXl+S8A/9tXAUqSpM519Tn0LwPfBI4t18/JzI8B6wGzgJf7IDZJkvpVRDHbWk+XRuhqQt8QuCEz5wHzgOUBMvNFimFfj+ub8CRJ6l8tc6L3ZGmErib0F4Aly9fTgU0r+4JifnRJkmqvrkO/drVT3GRgK2ASRfv52IiYC8wBxgI39U14kiSpK7qa0E8D1i1fjy1f/5iihD8Z+GTvhyZJUv+r6VNrXatyz8ybMvOi8vVzmbk/sAywYma+LTMf6ssgJUnqD0HPO8R1pVNcRIyJiPsiYmpEnNjO/iMiYmZE3FYun+jsmt0eWCYzXwNe6+75kiQNOP3QqS0iBgNnA3sA04DJETEhM+9uc+hFmdnlTucLm23tzEWILzPzhEU4XpKkxdUOwNSW2u2IuBDYH2ib0BfJwkroBy3CdRJYrBL6NusP54ZLjmp0GNIiWWl7nzCVOtMPvdRHAI9V1qcBb2vnuPdHxDuB+4EvZOZj7RzTamGTs6zXnSglSaqzrj7P3YnhETGlsj4uM8ctwvlXAr/JzNci4pMUo7X+18JOcHIWSZJ636zMHN3BvunA2pX1keW2Vpn5dGX1Z0CnzeC99EVEkqT6C/plYJnJwIYRsV5EDAUOpRjj5Y04ItasrO4H3NPZRS2hS5JU0dfzoWfm3Ig4jmKwtsHAeZl5V0ScCkzJzAnAZyNiP2Au8AxwRGfXNaFLklTR1wkdIDMnAhPbbBtbeX0ScNKiXNMqd0mSmsAildCjaBgYSdGYf3tmzu6TqCRJaoBitrR6jv3a5RJ6RHyaohfeo8B1wMbl9t9FxOf7JjxJkvrXoOj50pC4u3JQRHwZ+D5wLsVzcNVwrwUO6fXIJElqgLrOh97VKvdjgbGZeWY5Bm3VfcBGvRuWJElaFF1N6GsA/+xg33xgqd4JR5Kkxgno0mxpA1FX29CnArt2sO+d9HBAeUmSBopBvbA0QldL6D8EzomIOcCl5bbVIuLjwPGAs5RIkppCTQvoXUvomfmziFgJGAucUm6eCLwMnJyZv+6j+CRJUhd0+Tn0zPyfiPgJsBOwCsVQdDdm5vN9FZwkSf0pImrbhr5IA8tk5osUY89KktSUaprPu5bQy0FlFiozz+l5OJIkqTu6WkL/34Xsy/JfE7okqfYaNdJbT3Wpd31mDmq7ACsDhwG3A5v1ZZCSJPWHlufQe7o0QrenT83M54CLImIF4KfAbr0VlCRJjVLXNvTeeP79YWB0L1xHkiR1U7dL6AARsSbwRYqkLklSvTVwtrSe6mov95m80fmtxVBgOeBV4H29HJckSQ0R1DOj96SX+6vANOCPmfl074UkSVJjFJ3iGh1F93Sa0CNiCeDPwMOZOaPvQ5IkSYuqK53i5gHXAJv0cSySJDXcoOj50gidltAzc35EPEAxJ7okSU0tavrcWlfb0L8KnBERd2bmnX0ZkCRJjdKUbegR8U7g1sx8CfgaxQxrt0XEdOBJ2vR6z8wd+jJQSZLUsYWV0P8K7AjcAvy7XCRJal5R35HiFpbQW99SZh7ZD7FIktRwdZ0PvTeGfpUkSQ3WWae4vSOiS4+rZeYveiEeSZIapik7xZXGdvE6CZjQJUm1V9Ma904T+ruAKf0RiCRJjRcMatKx3F/JzNn9EokkSeq2Hk2fKklSMwmat8pdkqTFRzPOh56ZPtImSVrs+By6JElqGKvcJUkq2YYuSVKTqGuVuwldkqSKmuZz29AlSWoGltAlSSoF9S3pmtAlSWoREDWtczehS5JUUc90Xt+aBUmSVGFClySpVMyHHj1eOr1PxJiIuC8ipkbEiQs57v0RkRExurNrmtAlSaqIXlgWev2IwcDZwF7AZsBhEbFZO8ctB3wOuLkrcZvQJUnqXzsAUzPzocycA1wI7N/Ocd8EzgBe7cpFTeiSJFVE9HwBhkfElMpydOUWI4DHKuvTym2VGGJbYO3M/H1X47aXuyRJraK3HlublZmdtnu3G0HEIOD7wBGLcp4JXZKkUj8NLDMdWLuyPrLc1mI5YAvg2vLLxRrAhIjYLzOndHRRq9wlSepfk4ENI2K9iBgKHApMaNmZmc9n5vDMHJWZo4CbgIUmc7CELknSAvp6pLjMnBsRxwGTgMHAeZl5V0ScCkzJzAkLv0L7TOiSJFX0x0hxmTkRmNhm29gOjt2tK9c0oUuS1KLGY7nbhi5JUhOwhC5JUsnpUyVJahJ1rXI3oUuSVFHPdF7fmgVJklRhCV2SpIqa1rib0CVJalF0iqtnRrfKXZKkJmAJXZKkCqvcJUmqvSBqWuVuQpckqaKuJXTb0CVJagKW0CVJKtW5l7sJXZKkFlHfKncTuiRJFXVN6LahS5LUBCyhS5JU4WNrkiTVXACD6pnPTeiSJFXVtYRuG7okSU3AErokSRV17eVuQpckqcIqd6kLrp70R7bafGM23+Qt/M+Zp79p//XX/Z0dt9+WZZcawu9+e+kC+/7zn/+w717vYestN2WbrTbj0Uce6aeotbjbY6dNuf2yr/PvK77Bl47c4037115jJf447rPc+JsTuOWik9hz583etH/mDd/j8x9+d3+FrMWQCV39Zt68eXz+s8dyxZV/4F933M0lF/6Ge+6+e4Fj1l57Hcb933gOOfSDbzr/E0d+hC988cvcduc9XPePW1h1tdX6K3QtxmkC6kgAABc3SURBVAYNCn544sHsf9w5bPP+b3HQmO3YZP01FjjmhE+M4bd/upUdDzuDj5x0PmeddMgC+8/44vu4+oa7+jNsdVNLL/eeLo1glbv6zeRbbmGDDd7CeuuvD8BBhxzKVVdewaabvVGaWXfUKAAGDVrwu+Y9d9/N3LlzeffuRelo2WWX7Z+gtdjbfotRPPjYLB6Z/jQAl0y6lX1324p7H3qi9ZjMZPlllgJghWWH8fjM51v3vXe3rXhk+tPMfmVO/waubqrv9KmW0NVvZsyYzsiRa7eujxgxkunTp3fp3AceuJ8VV1yRQw56H28fvQ0nnfBl5s2b11ehSq3WWm0Fpj35bOv69CefZcSqKyxwzLd/OpFD996BqX/8Jpf96FMcf8YlACwzbChfPHIPvv3Tif0as3qgHMu9p0sjDJiEHhHzIuK2iPh3RFwSEUsv4vlrRcSl5eutI2Lvyr79IuLE3o5Z/Wfu3LnccP11nH7Gd7n+psk8/PBD/PLn4xsdlgTAwWNGc8GVN/GWMV/nwM/8mP/71keICL52zD786IJrLJ2rXwykKvdXMnNrgIj4FXAM8P2unpyZM4APlKtbA6OBieW+CcCEXo1Wi2yttUYwbdpjrevTp09jxIgRXTp3xIiRbPXWrVur6/fb7wBuufkm4ON9EarUasZTzzNy9ZVa10esvhLTK1XqAB89YEf2P/ZsAG6+42GWGroEw1dchu23WJcDd9+ab3/+AFZYbhjz5yevznmdn1z09359D1o09axwH1gJveo6YKuIWBk4D1gfeBk4OjPviIhdgbPKYxN4J7AKcBWwLXAqMCwidgZOA4ZRJPivAncA62Xm/IhYBri3vP46wNnAquW9jsrMe/vjzS4uRm+/PVOnPsAjDz/MWiNGcMlFFzL+l7/u8rnPP/ccM2fOZNVVV+Xav17DttuN7uOIJZhy16O8ZZ1VWXetVZjx1HMctOe2HHHS+AWOeeyJZ9hth4254Mqb2Xi91VlqySWY+exL7P7xH7Ye89VP7s3sl18zmQ9wRae4eqb0AVPl3iIihgB7AXcCpwD/ysytgK8AvygP+xJwbFmi3wV4peX8zJwDjAUuysytM/Oiyr7ngduAXctN+wKTMvN1YBzwmczcrrz+Oe3EdnRETImIKTNnzezNt71YGDJkCD8463957z57svWWm/L+gw5ms80359STx3LVlUUFypTJk9lg1Eh+99tL+MynP8m2b90cgMGDB3Pamd9l7/e8m9Fbb0lm8rFPHNXIt6PFxLx58/nCGRdz5TnHctvvvsZvr/4X9zz0BF//1D7ss+uWAJz4/cv42Pt24uaLTuTnpx3JUWN/2eCo1RPRC0sjRGY26NYLioh5FEkcihL6F4Gbgfdn5kPlMY8BmwOfBg4EfgX8LjOnRcQo4KrM3CIijgBGZ+Zx5Xmt6xHxQeCdmXlMRFxGkbhvBGYC91VCWjIzN+0o3u22G5033DylV9671F9W2v64Rocgdcurt539z8zs82q5TbfcJs+/7K89vs6OG67UL/FWDaQq99Y29BbRQbVHZp4eEb8H9gZuiIg9gVe7eJ8JwHfK6vztgGuAZYDn2t5fkrQYqmeN+8Crcm/jOuBwgIjYDZiVmS9ExAaZeWdmngFMBjZpc96LwHLtXTAzXyrPOYuiRD8vM18AHo6Ig8p7RUS8tU/ekSRpQIte+K8RBnpCPxnYLiLuAE4HPlpu/3z5eNsdwOvAH9qc91dgs/IxuEN4s4uAD5X/tjgc+HhE3A7cBezfe29DklQXdX0OfcBUuWfmm4b+ysxngAPa2f6Zdi7xCLBF5bzt2+wfXzn/UtpUqmTmw8CYRQxbkqQBYcAkdEmSBoKaNqGb0CVJWkBNM/pAb0OXJEldYAldkqRSMTBMPYvoJnRJklo0sJd6T5nQJUmqqGk+tw1dkqT+FhFjIuK+iJja3vTeEXFMRNxZjqdyfURs1tk1TeiSJFX18ewsETGYYnbPvYDNgMPaSdi/zswtyyHJz6QL04mb0CVJatUbA792Wmm/AzA1Mx8qZwi9kDajk5ZDkrdYhmKq8IWyDV2SpIpe6hQ3PCKqU3KOy8xx5esRwGOVfdOAt705jjgWOB4YCvxXZzc0oUuS1Ptm9XT61Mw8Gzi7nPb7a7wxn0m7rHKXJKnUG83nXSjgTwfWrqyPLLd15ELamdekLRO6JElVfZ/RJwMbRsR6ETEUOBSYsEAIERtWVvcBHujsola5S5JU0dcjxWXm3Ig4DpgEDAbOy8y7IuJUYEpmTgCOi4jdKaYIf5ZOqtvBhC5JUr/LzInAxDbbxlZef25Rr2lClySpwqFfJUlqAjXN5yZ0SZJadbGb+kBkL3dJkpqAJXRJkiqcD12SpJoL6tspzip3SZKagCV0SZIqalpAN6FLkrSAmmZ0E7okSRV17RRnG7okSU3AErokSRV17eVuQpckqaKm+dyELknSAmqa0W1DlySpCVhClySpVMzNUs8iugldkqQWYac4SZKaQk3zuW3okiQ1A0vokiRV1bSIbkKXJKlV1LZTnFXukiQ1AUvokiRV2MtdkqSaC2rbhG5ClyRpATXN6LahS5LUBCyhS5JUUdde7iZ0SZIq7BQnSVITqGk+tw1dkqRmYAldkqQWzrYmSVKzqGdGN6FLklQK6ltCtw1dkqQmYAldkqSKmhbQTeiSJFVZ5S5JkhrGErokSRUO/SpJUjOoZz43oUuSVFXTfG4buiRJzcASuiRJpajx0K+W0CVJqohe+K/Te0SMiYj7ImJqRJzYzv7jI+LuiLgjIv4SEet2dk0TuiRJVdELy8IuHzEYOBvYC9gMOCwiNmtz2L+A0Zm5FXApcGZnYZvQJUnqXzsAUzPzocycA1wI7F89IDP/mpkvl6s3ASM7u6gJXZKkil4qoA+PiCmV5ejKLUYAj1XWp5XbOvJx4A+dxW2nOEmSKnqpU9yszBzd04tExIeA0cCunR1rQpckqVXXOrX10HRg7cr6yHLbgpFE7A58Fdg1M1/r7KJWuUuS1L8mAxtGxHoRMRQ4FJhQPSAitgF+CuyXmU915aKW0CVJKgV9/xx6Zs6NiOOAScBg4LzMvCsiTgWmZOYE4H+AZYFLogjoP5m538Kua0KXJKmfZeZEYGKbbWMrr3df1Gta5S5JUhOwhC5JUkVdh341oUuSVOF86JIk1Z2Ts0iSpEayhC5JUqkLc6sMWCZ0SZKqaprRTeiSJFXUtVOcbeiSJDUBS+iSJFXUtZe7CV2SpIqa5nMTuiRJC6hpRrcNXZKkJmAJXZKkirr2cjehS5JU6o/50PtKZGajY6iliJgJPNroOJrUcGBWo4OQusHf3b6zbmau2tc3iYg/Uvwce2pWZo7phet0mQldA05ETMnM0Y2OQ1pU/u6qkewUJ0lSEzChS5LUBEzoGojGNToAqZv83VXD2IYuSVITsIQuSVITMKFLktQETOiSJDUBE7okSU3AhC5J/SCirgOKqi5M6Kqtlj+QEbFmRKzV6HikjkREZPlIUURsGhFrm+DV23xsTbUWEQcAnweeB+4FfpSZ0xobldS+iDgOOBi4DdgCeHf6R1i9xBK6aisitgSOB/YFbgHeRZHYpQEnIvYEDgT2AV4A5jY2IjUbE7rqbB5wFXAQxR/JQzPzxYjYvLFhSe16jmIkuU8AOwD7ZmZGxHsaG5aahfOhq3YiYjNgc+BfwC7AGsBhmflQROwFfD0i3peZTzQyTgkgIo4ElgD+AkwEHsrM7ct9RwB7R8TNmWntknrEErrq6B3AFzJzKsUfyQeA3SLig8B3ge+YzNUoEdH27+ojwH7AdOBTwFoRcXhEfBX4LHCqyVy9wU5xGvBaeghHxJDMnFtu+xVwU2b+KCI+AawLrAxckZlXV3sVS40UEasApwIXZ+bfIuIDwNuABM7LzHsbGqCahgldA1ZEbAS8NTMviYjtKDq9Tc3MyyNid2DPzPxy5fglMvP1RsWrxVvZd2O7zPxFROwLfBr4AvAQ8AHgK8DbM3N2A8NUE7PKXQPZIOCpiFgOmAYMBY6NiB9R9BDeKyI+XDneXsNqiLKafRVgYkSsB/wVuBP4DPBzYArwN2BMw4JU07OErgEtIoYAs4ATMvOnETEM+B7wKMUfy3uBAzLzpQaGqcVYRAzNzDnl65HAKcDtmfn/ImIl4CPAIcA6wPUUHTj9w6teZ0LXgBIRSwN7ZOYVEfE2YA4QwB+Bb2fmWWVpaA2KAToeyMzfNy5iLc4iYgWKTpp/B3ai6M0+FHg38DDww8ycVz6Z8VaKRH93o+JVczOha8CJiPHAaOBV4KjM/FdEbAv8GfhaZp7T5ng7wKnflbVHg4EjKErhqwCblh043wvsSVGT9IOWzpxSX7INXQNGZWzr0yh6rM/NzH8BZOatwO7AWRHxuep5JnP1t4jYBDgnM1+jGPVtO+BGiqQO8CeKZ843AY5rSJBa7FhC14BQeTRtELAssBJwHvB6Zo6pHLchMCoz/9SgUCUiYjDF7+hbgHuANYH9gZEUif6eiNgY2Jji8cqnGhasFhsmdDVcJZm/B3g78ERmjiv3XQPMBr4FnAkcmJnPWM2uRoiIQZk5v7J+LrAZsDewPPDJ8t/ngVUpOnM6aIz6hVXuargymY8BfgBcB5waEWdHxMqZ+V/ASxQ9h7+Xmc+0nNO4iLU4Kr9Ezi9f71m2oR9D0XP9MuBF4GzgQWBn4GyTufqTJXQ1VFnFvhzFs7pfB1YH/odimMzngM9k5rMRsWJmPmfJXI0WEcdSPDK5dzl/wCCK2qOtKSYImhURS2Xmqw0NVIsdE7oaolLNvnRmvlwOj7kyRWLfBRgGPAH8iGKs61caGK4EQETsApwFjMnMp8oRDJ+g+PJ5OrAB8F5gvl881d+cbU39rpLM3wacExFHZOadEbEaxXPnK1H0Fr4G+J3JXI3STo3Q6xSjwB0eEWsBewGPASdl5mciYvXMnNeIWCXb0NXvKm3mR1E8xzspIrbMzHuAW4BfUTzyc3ZmTm5gqFqMVZN5RIyMiOEUv5+vARsBV2bmFhTPmo8GyMwnGxWvZJW7+l051vUfgSMz8x8RMZZicI59KDoUjaZ4Bv2WxkWpxVXbUnlEfBb4IMXTFg8Ax1Vm/TsQGAscVE7nKzWMJXQ1wtPAzRSzUJGZp1L0FJ4ErJ6Z/zCZq4FamyLLNvMjgPdRtI2vQ1GDRFnLdBzwEZO5BgITuvpcywhwEbFCRKyQmS9QPKv7vsphvwJmAldExLINCFMiIvYAfhERJ5ZT9M4EbgIez8yXM3NvYN2IeD/F7GmHZeadDQxZamWnOPW5ytjWxwPPRsRNwInAb8rZqV6hSO5HUgzMsQzFs+dSvylL3KcCvwRWo6hmv5Vi+NYtgTvKQ6+h+LV+heJ3VxoQTOjqE206FL0d+ApwEPAhiglXzoyIQyjGZ1+HYnKL4RQzV81v/6pS34iIlSk6Yu6fmVdGxDoUz5bfBrwMjIuIX1OMmfBeYHyjYpU6YkJXr4uIVYEDIuI35TzlQykmXNmRYrzr95SHzmmZOS0idgLOpZjbfGYDwtZirBxO+L3AmRHxt8z8T0Qkxexp50bECxTjtK9O0QHu/oYGLLXDhK6+8A7gbcCS5VSogykS+tPAXuWIb3sAx0TEMeX2R4F3Z+ajDYpZi7nM/H1EzAf+GRGTgCWBX5f7Lm1ocFIX+Niaek1EDM7MeeVMVAcAuwF3Z+aPI+KbwIEU1e5bUTzq89+Z+fuGBSy1o+wMdzWwRjka3DAHN1IdmNDVK8qpIj9B8Yfw75n5WkTsRTGS1t2Z+ZOIOJlimskVgfMyc5Jjs2sgKn93vwu8y6lPVRcmdPWKiNiVYkjMB4CLgfUpJlnZg6INfQYwvuzx7sQVGvAiYn/gGxQDHaVfPDXQmdDVayJiZ+Aqivbz91OMyX4gMA14C3AycB5AdU5paaCKiGXLjp3SgGenOPWazLw+Ig4DLgV2yswXI+Iqimd4jwYeNpGrTkzmqhNL6Op1EbE3xbSn22fmM+W2lhnWbDOXpD5gCV29LjMnlo//3BsRG2fmsy1J3GQuSX3DErr6TETsA8zOzGsbHYskNTsTuvqc1eyS1PdM6JIkNQGnT5UkqQmY0CVJagImdEmSmoAJXWojIk6OiKwsMyLitxGxQR/ec9/yXqPK9VHl+r6LcI2DI+KIXoxp2TKGDq/ZnTjL88ZHxJQeB1lc69qIcDY0LfZ8Dl1q3/PAmPL1+sA3gb9ExOaZObsf7v84xfzx9y7COQcDw4HxfRGQpIHNhC61b25m3lS+viki/gNcB+wNXNL24N6eYjMzXwNu6vRASSpZ5S51zT/Lf0cBRMQjEfG9iPh6REwDXii3D4qIEyNiakS8FhH3R8RHqxeKwskR8VREvBgRvwCWb3NMu1XZEXFURNwZEa9GxJMRcWlErBAR4ykmxNm10lRwcuW8/SNiSnneExFxZkQs0eba7y/jfSUi/g5s0p0PKiI+EhHXR8QzEfFsRPw1IkZ3cOwBEXFvGdf1EbFZm/2dfp6SCpbQpa4ZVf77RGXbB4G7gE/zxv9LPwI+CpwK3Eoxfex5EfF0Zl5VHvNZYCzwHYpS//uAMzsLICK+Vl73HODLwNLAPsCyFE0C61DMNf/p8pRp5XkHA78Bfgp8BdgAOI3iC/2XymO2BS4CLgM+B2xBMQ1ud4wCfgE8SDF17mHAdWVzxUOV49YFvg98HXgFOAWYFBEbVqbX7crnKQkgM11cXCoLxTSvsyiS9BBgI4q53l8A1iyPeYSinXupynlvAeYDH21zvV8Ak8vXgynmhv9xm2P+BCQwqlwfVa7vW66vCLwMfH8hcV8KXNtmWwCPAue32f4xiiS6Srl+MXA35WBT5bavljEcsZB7LhBnO/sHlZ/hvcDYyvbx5Xk7VbatC8wFjunq51muXwtc2ujfGxeXRi9WuUvtWwV4vVzuo+gYd0hmPl455i/5RkkS4N0UCeiyiBjSsgB/AbaOiMHA2sCawBVt7ve7TuLZERgGnL+I72MjipL7xW1iugZYiqIkDrADMCEzq0NHdhZTuyJi04i4LCKeBOZRfIYbl7FUPZWZ/2hZycxHKZo2dig3deXzlFSyyl1q3/PA7hSlyCeAGW2SHcCTbdaHU5TAn+/gmmsCa5Svn2qzr+16W6uU/z6+0KPebHj578QO9q9d/rtGN2J6k4hYDria4rM5nqJ24FXgZxRfIDq7/lMUnxN07fOctqgxSs3KhC61b25mdvacdNsE/wxFlfE7KEqWbT3FG//PrdZmX9v1tp4u/12Tojmgq54p/z0a+Fc7+x8u/32iGzG1Z0dgJLBHZrY+chcRK7RzbHvXX42iXwJ07fOUVDKhS73nGooS5QqZ+af2DoiIxyiS5/7AHyu73tfJtW+kaPP+KGVHtnbM4c2l4PuA6RRt8+cu5PqTgf0i4qRKTURnMbVnWPnvay0bImInirb2f7Y5drWI2Kml2j0i1gG25Y1mhU4/T0lvMKFLvSQz74uInwAXRsSZwBSKBLs5sFFmfiIz55X7vhsRsyh6ub8f2LSTaz8XEd8Evh0RQymq0Jek6OV+SmZOp+h4tn9EHEBRFT0jM2dExBeBX0bE8sAfKBL/+sABwAcy82XgDOBmirb2/6NoW/94Nz6Gm4CXgHPL9zmSopPh9HaOnQVcUPbeb+nl/hTlwDhd+Ty7EZ/UtOwUJ/WuYykeIfsIRdIdT5F0/1455ocUj6wdA/yW4rGz/+7swpl5GvApirb9KygeQ1sReLE85ByK9uvzKErcR5fnXURRI7A1xaA4v6N4tO1WiuRO2bxwKLANcDlFsj9kUd98Zj4JHETRJn8F8PnyfU5t5/BHKWobTgYuLN/Hnm06Gnbl85SE86FLktQULKFLktQETOiSJDUBE7okSU3AhC5JUhMwoUuS1ARM6JIkNQETuiRJTcCELklSE/j/nJIB0SqLKFIAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 576x432 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vxpn48p3NQP-",
        "colab_type": "text"
      },
      "source": [
        "# Deep Learning for Text Classification\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jv35ZMrkNoEe",
        "colab_type": "text"
      },
      "source": [
        "### FastText"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YFH1m0iXZTJ2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Fasttext expects a single file with features and labels. Hence, combining into single dataframe\n",
        "df_train = pd.concat([X_train, y_train], axis = 1)\n",
        "df_test = pd.concat([X_test, y_test], axis = 1)\n",
        "\n",
        "#Adding label prefix to satisfy fasttext requirements\n",
        "df_train['sentiment'] = '__label__' + df_train['sentiment'].astype(str) + ' '\n",
        "df_test['sentiment'] = '__label__' + df_test['sentiment'].astype(str) + ' '\n",
        "\n",
        "#FastText API reads files from disk. Hence saving training data to disk\n",
        "train_file = 'train_file.csv'\n",
        "df_train.to_csv(train_file, header=None, index=False, columns=['reviewText','sentiment'] )\n",
        "\n",
        "test_file = 'test_file.csv'\n",
        "df_test.to_csv(test_file, header=None, index=False, columns=['reviewText','sentiment'] )\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPw-V331vndQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "outputId": "869ef8f8-fa1d-4336-ede0-8abe4928db2f"
      },
      "source": [
        "!head train_file.csv"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\"I LOVE the character of Miss Marple.  Someday I hope to be as observant and thoughtful as she!  Seeing is not the only way to observe, listen and smell!\",__label__1 \n",
            "\"I cannot say enough about Paullina's Trilogy - each book is excellent with the last one a brilliant capstone.  I like Alexander's point of view in this one... they all resonate deeply with me, in different ways.... although this is not my favorite book of the trilogy, it is still 5 stars (so that tells you what I think regardless) - my favorite is the final installment The Summer Garden, because it is better than any conclusion I have ever read!.  They all wear you out emotionally, but also fill you with joy - Paullina's mantra must be \"\"we shall overcome!\"\" and they do... which makes me happy!!!!\",__label__1 \n",
            "\"This is one of the most amazing books I've ever read.  I savored each and every page.  The characters are well drawn and the stories told from each one's point of view are riveting.  Each member of the family is so different from the other, as children in most families are.  The story of political turmoil and upheaval told through their eyes over a period of years is just stunning.\",__label__1 \n",
            "\"I liked that this was a no-brainer, easy to read, relaxing, take your mind off of stress, and funny kind of a book.  Easy quick read.\",__label__1 \n",
            "\"There are a lot of guidebooks out there for every major city in North America, but I always find the Access Guides useful, as well as helpful and up to date. So many books are geared toward the foreign tourist with information that you can hardly use. At last here is a book that give honest looks at San Francisco hotels, shops, and restaurants. Keep up the good work..\",__label__1 \n",
            "\"As they say in Hollywood, &quot;What a concept!&quot;  Kevin Murphy decided to see a movie in a theater every day in 2001, and write a book about it.  The result, _A Year at the Movies: One Man's Filmgoing Odyssey_ (HarperCollins) is, as you might expect from Murphy, a good deal of fun.  He is, after all, the guy who tells us, &quot;over the course of ten years on the TV series _Mystery Science Theater 3000_, I hefted a fifteen-pound plastic puppet over my head for an obscene amount of money.&quot;  Murphy was not just puppeteer but a writer of the show, in which he (as robot Tom Servo) and his pals made wonderful jokes while really bad films played on the screen.  The bad part of watching all those bad movies was that he started losing his taste for movies in general.  Worse, he noticed that multiplexes were erupting everywhere, and lots of people (including himself) were exiting the movies less happy than when they went in.  &quot;Something had to be done, and I was the man to do it.&quot;To do it, he went all over the world, to metroplexes as well as film festivals and hole-in-the-wall theaters.  He rode theme rides at a Hollywood movie park.  He went to the same date movie six times, with six different women, one of whom was his wife, and another his mother-in-law.  He worked at a multiplex.  He ate nothing but movie theater food for a week.  He even watched the wretched _Corky Romano_ (but he did call upon fellow MST3K buddy Michael J. Nelson, and beer, to help get him through that one.)  He knows plenty about movies, but this isn't really a book about movies; it is a book about watching movies.  If you want authoritative words on why movies these days can be wonderful sometimes but usually aren't, or how to sneak a full Thanksgiving dinner into a theater, or how to pick your seat, or the best theater in Hollywood, or how to creep unauthorized into Cannes Film Festival screenings, this is your book.  Much of the book reads like a very quirky travelogue.  Visit a theater made of snow and ice in Quebec, or the world's smallest commercial theater (22 seats, in Australia, and a moviegoer's paradise), go to the best film festival ever in Lapland, where movies are shown all day, and daylight is 24 hours.Plenty of the metroplex experiences were painful, mentally and physically (Murphy has many complaints about Hollywood), but much that happened to him in his moviegoing year was fun and even sweet.  On 11 September 2001, he was at Raratonga in the Cook Islands, thinking about human suffering and in comparison &quot;my silly little movie project,&quot; and worrying about how his friends in New York were.  The proprietor of the Empire Cinema found the only two American flags and put them out on the street, to show they were mourning, too.  &quot;And the Empire Cinema was fuller than usual, with people looking for some togetherness and a little healing.&quot;  Movies can do it, the good ones (the 11 September showing happened to be the lovely Irish film _Waking Ned Devine_).  And it is pleasing to know that despite all its crankiness against Hollywood and the metroplex system, Murphy's book documents his return to a love for going to films; his epilogue begins, &quot;Wait a minute.  It's over?  So soon?  No, please, so many more things to see, places to see them!&quot;  Murphy's original and funny book is recommended if you love going to the movies, and want to read about someone who did it lots more than you ever will, and who can write with enthusiasm and joking breeziness about a full year of doing so.\",__label__1 \n",
            "\"The fact that her husband tells her he is leaving her the day after the birth of their daughter is the only original thing I could found in the book. For the rest... a shallow, selfish and immature woman's ego trip. And the end! Her husband cheats on her because he loves her too much? Oh, boy! And predictably, she has fallen in love with someone else... in a month!\",__label__-1 \n",
            "\"Vivian and Nowell moved to the country to prepare his grandmother's house for sale. Nowell moved ahead of her since he was an author working on his second novel. Vivian arrived as Nowell's brother, Lonnie, was leaving. Soon after her arrival, a 17-year-old girl was found dead in the woods. The death is ruled an accident but as Vivian gets to know her neighbors, she is puzzled by some of their strange behavior. Her husband, Nowell, is also acting strangely and is growing more distant. Lonnie returns and their life becomes more strained. Will she be able to figure out what really happened?This book moves slowly. For the most part, I was okay with that. I thought there was a lot of character development and I felt like we were given a lot of insight into Vivian and Nowell's lives. I was, however, disappointed with the mystery storyline. It was mentioned from time to time but there wasn't much done to move that storyline forward. The reveal at the end was anti-climatic. I would have liked to have had more answers given along the way, and had a few more loose ends tied up when all was said and done.I enjoyed the different characters that were introduced. Katherine drove out to meet her when Vivian had lived there for a week and they became fast friends. There was mystery surrounding some of them--Mrs. Brodie, Mr. Stokes, and the road crew. Nowell, Lonnie and Dot each added tension to the story and there were some interesting family dynamics between all of them.I would recommend this to someone looking for a good literary read and Ms. White delivered in the secrets department, but don't read it for the mystery or you will most likely be disappointed. Overall, I enjoyed Ms. White's debut novel and I look forward to reading more of her work in the future!I received a copy of this book to review. My opinion is 100% my own.\",__label__1 \n",
            "\"If you like Games of Thrones you're going to like this a lot. Good story, a genuine anti-hero and very realistic.\",__label__1 \n",
            "\"I bought this book based largely on the review written by Molly. It has not disappointed. I have never read a book quite like Stephenson's, but I look forward to reading more of his work. I like the &#34;meatiness&#34; of it all and the fact that it's a book that I will re-read more than once. I consider &#34;re-readingness&#34; to be the hallmark of an excellent book.\",__label__1 \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s0XqXw6rddnM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 263
        },
        "outputId": "675b4285-aefb-4e92-88b3-e487a5a14620"
      },
      "source": [
        "!pip install fasttext"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fasttext\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f8/85/e2b368ab6d3528827b147fdb814f8189acc981a4bc2f99ab894650e05c40/fasttext-0.9.2.tar.gz (68kB)\n",
            "\r\u001b[K     |████▊                           | 10kB 21.1MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 20kB 2.2MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 30kB 2.8MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 40kB 3.1MB/s eta 0:00:01\r\u001b[K     |███████████████████████▉        | 51kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▋   | 61kB 2.8MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 71kB 2.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.6/dist-packages (from fasttext) (2.5.0)\n",
            "Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from fasttext) (49.1.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from fasttext) (1.18.5)\n",
            "Building wheels for collected packages: fasttext\n",
            "  Building wheel for fasttext (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fasttext: filename=fasttext-0.9.2-cp36-cp36m-linux_x86_64.whl size=3015571 sha256=85c675006915f2f4648f21a94c05706a9af97a3768659bef53eb939eccde457c\n",
            "  Stored in directory: /root/.cache/pip/wheels/98/ba/7f/b154944a1cf5a8cee91c154b75231136cc3a3321ab0e30f592\n",
            "Successfully built fasttext\n",
            "Installing collected packages: fasttext\n",
            "Successfully installed fasttext-0.9.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UTynEn5TcxQp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Using fastText for feature extraction and training\n",
        "import fasttext\n",
        "from fasttext import supervised\n",
        "model = fasttext.train_supervised(train_file)\n",
        "results = model.test(test_file)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ln0N_5n3thTm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "7156f30c-5284-4dc4-d4a8-ab14a03e5de4"
      },
      "source": [
        "results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, nan, nan)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rm3xdeESNywp",
        "colab_type": "text"
      },
      "source": [
        "### RNN"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aaEKOQy8ZTyh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Import Libraries\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "from keras.utils import to_categorical\n",
        "from keras.layers import Dense, Input, GlobalMaxPooling1D\n",
        "from keras.layers import Conv1D, MaxPooling1D, Embedding, LSTM\n",
        "from keras.models import Model, Sequential\n",
        "from keras.initializers import Constant"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KP_vxF6m1oQE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Load pretrained word embeddings"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UHizX-6A1PVn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Prepare train and test data\n",
        "\n",
        "#Tokenization\n",
        "\n",
        "#Embedding"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Yytn1zq_1avI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Initialize and Train model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fjPaN85K1e-i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Evaluate Model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2Ns84GhNtZ2",
        "colab_type": "text"
      },
      "source": [
        "### BERT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2-6y6A9V5X5K",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 293
        },
        "outputId": "180464ac-42bf-4c40-f622-ec9b221f8b15"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>reviewerID</th>\n",
              "      <th>asin</th>\n",
              "      <th>helpful</th>\n",
              "      <th>reviewText</th>\n",
              "      <th>overall</th>\n",
              "      <th>summary</th>\n",
              "      <th>unixReviewTime</th>\n",
              "      <th>reviewTime</th>\n",
              "      <th>reviewerName</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AH2L9G3DQHHAJ</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[5, 5]</td>\n",
              "      <td>Interesting Grisham tale of a lawyer that take...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Show me the money!</td>\n",
              "      <td>1019865600</td>\n",
              "      <td>04 27, 2002</td>\n",
              "      <td>chris</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A2IIIDRK3PRRZY</td>\n",
              "      <td>0000000116</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>The thumbnail is a shirt.  The product shown i...</td>\n",
              "      <td>1.0</td>\n",
              "      <td>Listing is all screwed up</td>\n",
              "      <td>1395619200</td>\n",
              "      <td>03 24, 2014</td>\n",
              "      <td>Helene</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>A1TADCM7YWPQ8M</td>\n",
              "      <td>0000000868</td>\n",
              "      <td>[10, 10]</td>\n",
              "      <td>I'll be honest. I work for a large online reta...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not a Bad Translation</td>\n",
              "      <td>1031702400</td>\n",
              "      <td>09 11, 2002</td>\n",
              "      <td>Joel@AWS</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>AWGH7V0BDOJKB</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>It had all the songs I wanted but I had ordere...</td>\n",
              "      <td>4.0</td>\n",
              "      <td>Not the large print</td>\n",
              "      <td>1383177600</td>\n",
              "      <td>10 31, 2013</td>\n",
              "      <td>Barbara Marshall</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A3UTQPQPM4TQO0</td>\n",
              "      <td>0000013714</td>\n",
              "      <td>[0, 0]</td>\n",
              "      <td>We have many of the old, old issue. But the nu...</td>\n",
              "      <td>5.0</td>\n",
              "      <td>I was disappointed that you would only allow m...</td>\n",
              "      <td>1374883200</td>\n",
              "      <td>07 27, 2013</td>\n",
              "      <td>betty burnett</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       reviewerID        asin  ...   reviewTime      reviewerName\n",
              "0   AH2L9G3DQHHAJ  0000000116  ...  04 27, 2002             chris\n",
              "1  A2IIIDRK3PRRZY  0000000116  ...  03 24, 2014            Helene\n",
              "2  A1TADCM7YWPQ8M  0000000868  ...  09 11, 2002          Joel@AWS\n",
              "3   AWGH7V0BDOJKB  0000013714  ...  10 31, 2013  Barbara Marshall\n",
              "4  A3UTQPQPM4TQO0  0000013714  ...  07 27, 2013     betty burnett\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JNO6qF2x5ZQe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}