lstm_and_gru/lstm_and_gru_fastai.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "lstm_and_gru_fastai.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "ZIC0nEEjEjGG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# LSTM\n",
        "\n",
        "In this notebook, we will go through basics of LSTM on IMDB dataset. \n",
        "\n",
        "Here we will use [fastai](https://github.com/fastai/course-v3  \"Fastai Tutorial\").\n",
        "\n",
        "\n",
        "Everything is explained in-detail in [blog post](https://dudeperf3ct.github.io/lstm/gru/nlp/2019/01/28/Force-of-LSTM-and-GRU/). This is notebook which replicates the result of blog and runs in colab. Enjoy!\n",
        "\n",
        "Let's see at [nlpprogess](http://nlpprogress.com/english/sentiment_analysis.html) what is the current state-of-the-art in sentiment analysis.\n",
        "\n",
        "\n",
        "Model | Accuracy | Paper |\n",
        "----- | -------- | ------|\n",
        "ULMFit| 95.4     | [Universal Language Model Fine-tuning for Text Classification](https://arxiv.org/abs/1801.06146)|\n",
        "Block-sparse LSTM| 94.99 | [GPU Kernels for Block-Sparse Weights](https://s3-us-west-2.amazonaws.com/openai-assets/blocksparse/blocksparsepaper.pdf)|\n",
        "oh-LSTM | 94.1 | [Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings](https://arxiv.org/abs/1602.02373) |\n",
        "Virtual adversarial training  | 94.1 | [Adversarial Training Methods for Semi-Supervised Text Classification](https://arxiv.org/abs/1605.07725) |\n",
        "BCN+Char+CoVe | 91.8 | [Learned in Translation: Contextualized Word Vectors](https://arxiv.org/abs/1708.00107) |\n",
        "\n",
        "\n",
        "#### Run in Colab\n",
        "\n",
        "You can run this notebook in google colab.\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dudeperf3ct/DL_notebooks/blob/master/lstm_and_gru/lstm_and_gru_fastai.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "CPX0rk2CUG-f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## IMDB data\n",
        "\n",
        "Code Adapted from: [link](https://github.com/fastai/course-v3/blob/master/nbs/dl1/lesson3-imdb.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "pNkINkerEmzY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from fastai.text import *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hMYhISbwwNG8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Preparing the data\n",
        "\n",
        "First let's download the dataset we are going to study. The dataset has been curated by Andrew Maas et al. and contains a total of 100,000 reviews on IMDB. 25,000 of them are labelled as positive and negative for training, another 25,000 are labelled for testing (in both cases they are highly polarized). The remaning 50,000 is an additional unlabelled data (but we will find a use for it nonetheless).\n",
        "\n",
        "We will use sample and show different steps involved in converting text data usable for classifier"
      ]
    },
    {
      "metadata": {
        "id": "Wo8Hb0KUwBJP",
        "colab_type": "code",
        "outputId": "8181486a-d961-4c57-81d1-7a9eadd76c67",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "path = untar_data(URLs.IMDB_SAMPLE)\n",
        "path.ls()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[PosixPath('/root/.fastai/data/imdb_sample/texts.csv')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "metadata": {
        "id": "bUq3yHm1wBM1",
        "colab_type": "code",
        "outputId": "59409fa7-45dd-4b12-ed90-a21fa739aa94",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "cell_type": "code",
      "source": [
        "df = pd.read_csv(path/'texts.csv')\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>is_valid</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>negative</td>\n",
              "      <td>Un-bleeping-believable! Meg Ryan doesn't even ...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>positive</td>\n",
              "      <td>This is a extremely well-made film. The acting...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>negative</td>\n",
              "      <td>Every once in a long while a movie will come a...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>positive</td>\n",
              "      <td>Name just says it all. I watched this movie wi...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>negative</td>\n",
              "      <td>This movie succeeds at being one of the most u...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      label                                               text  is_valid\n",
              "0  negative  Un-bleeping-believable! Meg Ryan doesn't even ...     False\n",
              "1  positive  This is a extremely well-made film. The acting...     False\n",
              "2  negative  Every once in a long while a movie will come a...     False\n",
              "3  positive  Name just says it all. I watched this movie wi...     False\n",
              "4  negative  This movie succeeds at being one of the most u...     False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "metadata": {
        "id": "BKflY4lRwBQ-",
        "colab_type": "code",
        "outputId": "f08aa6c5-a073-40bc-c6ae-98cc3807d4a1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        }
      },
      "cell_type": "code",
      "source": [
        "df['text'][1]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'This is a extremely well-made film. The acting, script and camera-work are all first-rate. The music is good, too, though it is mostly early in the film, when things are still relatively cheery. There are no really superstars in the cast, though several faces will be familiar. The entire cast does an excellent job with the script.<br /><br />But it is hard to watch, because there is no good end to a situation like the one presented. It is now fashionable to blame the British for setting Hindus and Muslims against each other, and then cruelly separating them into two countries. There is some merit in this view, but it\\'s also true that no one forced Hindus and Muslims in the region to mistreat each other as they did around the time of partition. It seems more likely that the British simply saw the tensions between the religions and were clever enough to exploit them to their own ends.<br /><br />The result is that there is much cruelty and inhumanity in the situation and this is very unpleasant to remember and to see on the screen. But it is never painted as a black-and-white case. There is baseness and nobility on both sides, and also the hope for change in the younger generation.<br /><br />There is redemption of a sort, in the end, when Puro has to make a hard choice between a man who has ruined her life, but also truly loved her, and her family which has disowned her, then later come looking for her. But by that point, she has no option that is without great pain for her.<br /><br />This film carries the message that both Muslims and Hindus have their grave faults, and also that both can be dignified and caring people. The reality of partition makes that realisation all the more wrenching, since there can never be real reconciliation across the India/Pakistan border. In that sense, it is similar to \"Mr & Mrs Iyer\".<br /><br />In the end, we were glad to have seen the film, even though the resolution was heartbreaking. If the UK and US could deal with their own histories of racism with this kind of frankness, they would certainly be better off.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "IfxA8zQRrE2Y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Method 1: Text DataBunch"
      ]
    },
    {
      "metadata": {
        "id": "JbutxqVowBU_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# create text databunch\n",
        "\n",
        "data_lm = TextDataBunch.from_csv(path, 'texts.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cFfUSWWzpFRZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " A text is composed of words, and we can't apply mathematical functions to them directly. We first have to convert them to numbers. This is done in two differents steps: tokenization and numericalization. A TextDataBunch does all of that behind the scenes for you."
      ]
    },
    {
      "metadata": {
        "id": "VuJcaZQ0o9b3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# # save databunch\n",
        "# data_lm.save()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0q00NvUWo9gC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# # load the saved databunch instead of runnning all steps above\n",
        "# data = load_data(path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KZ4xhbo-pjeh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Tokenization\n",
        "\n",
        "The first step of processing we make the texts go through is to split the raw sentences into words, or more exactly tokens. The easiest way to do this would be to split the string on spaces, but we can be smarter:\n",
        "\n",
        "*    we need to take care of punctuation\n",
        "*   some words are contractions of two different words, like isn't (is n't) or don't (do n't)\n",
        "*   we may need to clean some parts of our texts, if there's HTML code for instance\n",
        "\n",
        "\n",
        "To see what the tokenizer had done behind the scenes, let's have a look at a few texts in a batch.\n"
      ]
    },
    {
      "metadata": {
        "id": "OmsMuqg9o9jp",
        "colab_type": "code",
        "outputId": "f8383c99-f1bf-4e55-abf7-693c68958755",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "cell_type": "code",
      "source": [
        "data = TextClasDataBunch.from_csv(path, 'texts.csv')\n",
        "data.show_batch()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>xxbos xxup the xxup shop xxup around xxup the xxup corner is one of the sweetest and most feel - good romantic comedies ever made . xxmaj there 's just no getting around that , and it 's hard to actually put one 's feeling for this film into words . xxmaj it 's not one of those films that tries too hard , nor does it come up with</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj now that xxmaj che(2008 ) has finished its relatively short xxmaj australian cinema run ( extremely limited xxunk screen in xxmaj xxunk , after xxunk ) , i can xxunk join both xxunk of \" xxmaj at xxmaj the xxmaj movies \" in taking xxmaj steven xxmaj soderbergh to task . \\n \\n  xxmaj it 's usually satisfying to watch a film director change his style /</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos \\n \\n  i 'm sure things did n't exactly go the same way in the real life of xxmaj homer xxmaj hickam as they did in the film adaptation of his book , xxmaj rocket xxmaj boys , but the movie \" xxmaj october xxmaj sky \" ( an xxunk of the book 's title ) is good enough to stand alone . i have not read xxmaj</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj to review this movie , i without any doubt would have to quote that memorable scene in xxmaj tarantino 's \" xxmaj pulp xxmaj fiction \" ( xxunk ) when xxmaj jules and xxmaj vincent are talking about xxmaj mia xxmaj wallace and what she does for a living . xxmaj jules tells xxmaj vincent that the \" xxmaj only thing she did worthwhile was pilot \" .</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj the trouble with the book , \" xxmaj memoirs of a xxmaj geisha \" is that it had xxmaj japanese surfaces but underneath the surfaces it was all an xxmaj american man 's way of thinking . xxmaj reading the book is like watching a magnificent ballet with great music , sets , and costumes yet performed by xxunk animals dressed in those xxunk far from xxmaj japanese</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "yCnpFNE9qB50",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "The texts are truncated at 100 tokens for more readability. We can see that it did more than just split on space and punctuation symbols:\n",
        "\n",
        "  * the \"'s\" are grouped together in one token\n",
        "  * the contractions are separated like this: \"did\", \"n't\"\n",
        "  * content has been cleaned for any HTML symbol and lower cased\n",
        "  * there are several special tokens (all those that begin by xx), to replace unknown tokens (see below) or to introduce different text fields (here we only have one).\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "9sEqO3TLqTOn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "### Numericalization\n",
        "\n",
        "Once we have extracted tokens from our texts, we convert to integers by creating a list of all the words used. We only keep the ones that appear at least twice with a maximum vocabulary size of 60,000 (by default) and replace the ones that don't make the cut by the unknown token UNK.\n",
        "\n",
        "The correspondance from ids to tokens is stored in the vocab attribute of our datasets, in a dictionary called itos (for int to string).\n"
      ]
    },
    {
      "metadata": {
        "id": "Ux1Q-eVqo9ny",
        "colab_type": "code",
        "outputId": "830b7782-08ac-4b14-d5af-6d41d9972510",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 196
        }
      },
      "cell_type": "code",
      "source": [
        "# mapping from int to string\n",
        "data.vocab.itos[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['xxunk',\n",
              " 'xxpad',\n",
              " 'xxbos',\n",
              " 'xxfld',\n",
              " 'xxmaj',\n",
              " 'xxup',\n",
              " 'xxrep',\n",
              " 'xxwrep',\n",
              " 'the',\n",
              " ',']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "2HYxqDl_qZeD",
        "colab_type": "code",
        "outputId": "b06ff5ad-964f-41fe-a876-b35c98c39494",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "# sample training example \n",
        "# tokenization\n",
        "data.train_ds[0][0]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<fastai.text.data.Text at 0x7f86a3a54b00>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "npjbwFisqZiH",
        "colab_type": "code",
        "outputId": "14537e92-14b4-4872-cb98-6babab2b33e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "# numericalization\n",
        "data.train_ds[0][0].data[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([   2,    4,   13,    8,  287, 4651,   49,    4, 6033,    4])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "J6O7r114cHRh"
      },
      "cell_type": "markdown",
      "source": [
        "We will combine both steps i.e creating databunch and tokenization and numericalization\n",
        "\n",
        "We will use the complete dataset to train our classifier"
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "EQ0kBLmycHRc",
        "outputId": "25cdfaf2-542c-45b0-fda8-e26891e3dc2e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        }
      },
      "cell_type": "code",
      "source": [
        "path = untar_data(URLs.IMDB)\n",
        "path.ls()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[PosixPath('/root/.fastai/data/imdb/train'),\n",
              " PosixPath('/root/.fastai/data/imdb/test'),\n",
              " PosixPath('/root/.fastai/data/imdb/imdb.vocab'),\n",
              " PosixPath('/root/.fastai/data/imdb/unsup'),\n",
              " PosixPath('/root/.fastai/data/imdb/tmp_clas'),\n",
              " PosixPath('/root/.fastai/data/imdb/README'),\n",
              " PosixPath('/root/.fastai/data/imdb/tmp_lm')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "BVXKo3QLcHRS",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#batch size\n",
        "bs = 32\n",
        "\n",
        "# datablock for language model\n",
        "data_lm = (TextList.from_folder(path)\n",
        "           #Inputs: all the text files in path\n",
        "            .filter_by_folder(include=['train', 'test', 'unsup']) \n",
        "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
        "            .random_split_by_pct(0.1)\n",
        "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
        "            .label_for_lm()           \n",
        "           #We want to do a language model so we label accordingly\n",
        "            .databunch(bs=bs))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "raG2RJPIcHRK",
        "outputId": "f0f3a545-31e0-4643-a570-b76428e93da3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 328
        }
      },
      "cell_type": "code",
      "source": [
        "data_lm.show_batch()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>idx</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>great fighting and a lot of the locations were beautifully shot , great effects , and a lot of sword play . xxmaj another solid effort by xxmaj tadanobu xxmaj asano in my opinion . xxmaj well i really ca n't say anymore about the movie , but if you 're only outlook on xxmaj asian cinema is xxmaj crouching xxmaj tiger xxmaj hidden xxmaj dragon or xxmaj house of</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>help . xxmaj joycelyn works in an up - scale dress shop but dreams about becoming a singer . xxmaj in a very poignant scene in her bedroom , she pretends to be talking to her fans , then lies down in bed to recite one of her poems . xxmaj she is close to xxmaj lanisha and xxmaj maria at the beginning but drifts off to make friends outside</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>ca n't act , they yell and scream every time they do something important . xxmaj in one scene people try to hang the xxmaj barbarian xxmaj brothers , and they escape very extraordinary way . \\n \\n  xxmaj bad acting , bad special effects , very stupid story , bad direction , actually everything is bad in this movie . i ca n't describe how much i laughed</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>. xxmaj lois xxmaj ramsey and xxmaj kati xxmaj edwards give delightful performances in supporting roles as the friendly xxmaj margaret and xxmaj mary . xxmaj they also add a subtle humor to the bittersweet story . \\n \\n  xxmaj an anecdote from xxmaj xxunk ' childhood , the story of loss and discovery is remarkably captured in ' xxmaj tulip ' . xxmaj beautifully shot and superbly acted</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>a xxmaj crackerjack ring that is rumored to kill you if you 're punched with it . xxmaj but not immediately . xxmaj no , the death ring unleashes a poison that kills you in the eight grade . \\n \\n  xxmaj joe and his axis of evil welcome xxmaj billy by smuggling a handful of slimy worms into his thermos . xxmaj once discovered , xxmaj billy plays</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "YG6W-_mBcHQ0",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# #Since above step can be a bit time-consuming, it's best to save the result.\n",
        "# #This will create a 'tmp' directory where all the computed stuff will be stored\n",
        "# data_lm.save('data_lm_export.pkl')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "rQYbZouGcHQd",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# # then reload those results using load_data\n",
        "# data_lm = load_data(path, fname='data_lm_export.pkl')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iQNFFTmDwSn6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Classifier"
      ]
    },
    {
      "metadata": {
        "id": "sXta9ZsKwUXb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# datablock for classifier\n",
        "\n",
        "data_clas = (TextList.from_folder(path, vocab=data_lm.vocab)\n",
        "             #grab all the text files in path\n",
        "             .split_by_folder(valid='test')\n",
        "             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)\n",
        "             .label_from_folder(classes=['neg', 'pos'])\n",
        "             #label them all with their folders\n",
        "             .databunch(bs=bs))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "p_GNKllOwVq3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# data_clas.save('data_clas.pkl')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fVaoIv_xwV_p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# data_cls = data_clas.load('data_clas.pkl', bs=bs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YtwAn7-YwWCX",
        "colab_type": "code",
        "outputId": "a59abe97-fb89-470a-a3b8-b594e4171866",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 345
        }
      },
      "cell_type": "code",
      "source": [
        "data_clas.show_batch()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj match 1 : xxmaj tag xxmaj team xxmaj table xxmaj match xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley vs xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit xxmaj bubba xxmaj ray and xxmaj spike xxmaj dudley started things off with a xxmaj tag xxmaj team xxmaj table xxmaj match against xxmaj eddie xxmaj guerrero and xxmaj chris xxmaj benoit . xxmaj according to the rules</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj to be a xxmaj buster xxmaj keaton fan is to have your heart broken on a regular basis . xxmaj most of us first encounter xxmaj keaton in one of the brilliant feature films from his great period of independent production : ' xxmaj the xxmaj general ' , ' xxmaj the xxmaj navigator ' , ' xxmaj sherlock xxmaj jnr ' . xxmaj we recognise him as</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj as with all of xxmaj angelopoulos ' films , \" xxmaj the xxmaj suspended xxmaj step of the xxmaj stork \" implicitly demands a close and intimate participation on the part of the viewer , a fact that has certainly contributed to the limited popularity of his work . xxmaj dialogues are sparing , with no monologues or exchanges xxunk the characters ' inner conflicts , doubts ,</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj if you 've ever been harassed on the xxmaj underground by a xxmaj christian who says , \" xxmaj jesus is the answer . xxmaj what 's the question ? \" , then perhaps you should thank xxmaj god if you 've never met a xxmaj lacanian . xxmaj slavoj xxmaj zizek , the most evangelical of xxmaj xxunk , would surely exchange the word \" xxmaj jesus</td>\n",
              "      <td>neg</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>xxbos xxmaj many neglect that this is n't just a classic due to the fact that it 's the first xxup 3d game , or even the first shoot-'em - up . xxmaj it 's also one of the first stealth games , one of the xxunk definitely the first ) truly claustrophobic games , and just a pretty well - rounded gaming experience in general . xxmaj with graphics</td>\n",
              "      <td>pos</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "w36iuFp4wrbV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Byn4qfX3wrYM",
        "colab_type": "code",
        "outputId": "05ac14c5-e5f7-4550-e42c-290f551dd85b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 380
        }
      },
      "cell_type": "code",
      "source": [
        "learn.lr_find()\n",
        "learn.recorder.plot(skip_end=15)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAFZCAYAAABwne0JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xt8nOV54P3fMzPSaDQanc+WZFuS\nZRv5AOZoDAFcOwQaknZbHLdZU6f07WazL9sN8C7FberkTSBd2m2XfJK0L23Isg4EQ4NJSEOABEwC\nOHaMjQ8yPts6H2ak0WhOmuPz/jF6RseRRtIcpNH1/XzyiSU9M3P7wZpr7vu+7utSVFVVEUIIIURG\n0qV7AEIIIYRIHgn0QgghRAaTQC+EEEJkMAn0QgghRAaTQC+EEEJkMAn0QgghRAYzJPPJn3zySU6c\nOIGiKOzZs4cNGzYA0Nvby6OPPhq9rr29nUceeYT77rsPAJvNxj333MO3v/1tbr75Znbt2oXH4yE3\nNxeAxx57jHXr1iVz6EIIIURGSFqgP3LkCK2trezfv59Lly6xZ88e9u/fD0BFRQX79u0DIBgMsmvX\nLrZu3Rp97FNPPUVtbe245/vmN79JU1NTXK9ttToT9LeYv6KiXOx2T7qHseDJfZqZ3KP4yH2amdyj\n+Cy2+1RWZpny+0lbuj906BDbtm0DoKGhAYfDgcvlmnTdgQMHuPvuuzGbzdHHmc3muIP6Qmcw6NM9\nhEVB7tPM5B7FR+7TzOQexSdT7lPSZvQ2m43m5ubo18XFxVitVvLy8sZd9/LLL/Pss88C4Pf7+c53\nvsN3v/tdnnzyyXHXfetb38Jut9PQ0MCePXvIycmJ+dpFRbkL6j9QrE9ZYjy5TzOTexQfuU8zk3sU\nn0y4T0ndox9rqkq7x48fp76+Phr8n3nmGe6//37y8/PHXffAAw+wevVq6urq2Lt3L88//zwPPvhg\nzNdaSEstZWWWBbWVsFDJfZqZ3KP4yH2amdyj+Cy2+xTrQ0nSAn15eTk2my36dV9fH2VlZeOuOXjw\nIJs3b45+/d577xEOh3n++edpa2vj5MmTPP3002zfvj16zdatW/nZz36WrGELIYQQGSVpe/Rbtmzh\njTfeAKClpYXy8vJJy/anTp1izZo10a9ffPFFXnrpJV566SXuvPNO9u7dS2NjI7t372ZoaAiAw4cP\ns2rVqmQNWwghhMgoSZvRb9q0iebmZnbu3ImiKOzdu5dXXnkFi8USnaFbrVZKSkqmfR5FUdixYwe7\nd+/GZDJRUVHBQw89lKxhCyGEEBlFycQ2tQtpT2Wx7fGki9ynmck9io/cp5nJPYrPYrtPKT9eJ4QQ\nQoj0k0AvhBBCZDAJ9EIIIUQGk0AvhBBCZDAJ9EIIIUQKeX1BDp3uIRgKp+T1JNALIYQQKfTrk938\ny0/PcKHDkZLXk0AvhBBCpJBt0AuAyZianiwS6IUQQogUsrt8ABTlGVPyehLohRBCiBQadPrQ6xQs\n5uyUvJ4EeiGEECKFBpw+CvOy0SlKSl5PAr0QQgiRIuGwisPlp9CSmmV7kEAvhBBCpMyQx09YVVO2\nPw8S6IUQQoiUsTsjiXgyoxdCCCEy0OBIoC+25KTsNSXQCyGEECkyEJ3RpybjHiTQCyGEECkzmOIz\n9CCBXgghhEgZbY++SPbohRBCiMwTTcaTGb0QQgiReQZdPvJMWWRnpabOPUigF0IIIVLG7vSldDYP\nEuiFEEKIlPD6ggz7QyndnwcJ9EIIIURKjCbipe5oHUigF0IIIVJCa08rS/dCCCFEBopWxctPXVU8\nkEAvhBBCpEQ6jtaBBHohhBAiJdJRLAck0AshhBApIYFeCCGEyGB2lw+DXoc5x5DS15VAL4QQQqTA\noNNHscWIoigpfV0J9EIIIUSSBUNhhtx+ClO8bA8S6IUQQoikG3L7UUn9/jxIoBdCCCGSbsCZ+j70\nGgn0QgghRJJpxXJk6V4IIYTIQOk6WgcS6IUQQoik0+rcS6AXQgghMtCg7NELIYQQC1swFOY7B07x\n/qnuWT92wOlDAQryUtuiFiTQCyGEEHHptLr58JyVfz/UOuvHDjp9WMzZGPSpD7sS6IUQQog4tPU6\nAegZ8NA74In7caqqYnf50rJsDxLohRBCiLi09bqifz5x0Rb349zDQQLBcFoS8UACvRBCCBGXtj4n\nWpn6E5f6437cYBqP1gGktoWOEEIIsQiFVZX2PhdVJWayDTrOtw/iGQ6SG0cnuoE0FssBmdELIYQQ\nM7INehn2h6grz2NjYymhsErL1YG4HjvoSt/ROpBAL4QQQsxI25+vrchjY2MJEP8+fTqr4oEEeiGE\nEGJGbX2RjPu6Cgt1FRYK8rI5eamfcFid8bES6IUQQogFLjqjL89DpyhsbCjB5Q1wuXtoxscOprH8\nLUigF0IIIWbU1uukyGIkPzdS2W5jYykQ3/K93enDmK3HZExP/rsEeiGEEGIaQ24/gy4/teV50e9d\ns7wYg17HiYszH7OzO9NXLAck0AshhBDTau+LLNvXVVii3zNm61m7vIgOq4t+x3DMxwaCIVzeQNqW\n7UEC/aLzvZ+e4bUPrqZ7GEIIsWREE/HGzOiBaPb9yUuxl+/tLj+Qvv15kEC/qASCYd4/3cMvj7aj\nqjNnegohhJg/LRGvrmJ8oN/QMHLMbpoqeemuigcS6BcVlzcAwJAnQP9Q7KUiIYQQidPW6yQnW09p\noWnc90sLTNSUmTlz1Y7PH5rysdrRusJM3aN/8skn+dznPsfOnTs5efJk9Pu9vb3s2rUr+r8777yT\n1157Lfpzm83GjTfeyOHDhwE4e/YsO3fuZOfOnezduzeZQ17QnB5/9M9Xup1pHIkQQiwNvkCIngEP\ndSPH6iba2FhKMBTm41b7lI9P9xl6SGKgP3LkCK2trezfv58nnniCJ554IvqziooK9u3bx759+/j+\n979PVVUVW7dujf78qaeeora2Nvr1E088wZ49e3jxxRdxuVy8++67yRr2gjY0NtB3zXx2UwghxPx0\nWF2oKtSOScQba2PDyDG7GPv0GR3oDx06xLZt2wBoaGjA4XDgcrkmXXfgwAHuvvtuzGZz9HFms5mm\npiYA/H4/nZ2dbNiwAYC77rqLQ4cOJWvYC5rTE4j+OZ4iDUIIIeanXdufn5CIp6mvzifPlMWJi7Yp\nc6fsaS6WA0nsXmez2Whubo5+XVxcjNVqJS9v/M16+eWXefbZZ4FIUP/Od77Dd7/7XZ588kkA7HY7\n+fn50etLSkqwWq3TvnZRUS4Ggz5Rf5V5Kyub+pPgbKm6vuif23qdFJfkoddNXkparBJ1nzKZ3KP4\nyH2amdyj+FiHIoF645qKmPfsxmsqeOfDDob8YRprCsf9zD0cRKdTqF9ekrb365SV6Znqk87x48ep\nr6+PBv9nnnmG+++/f1xgj+d5JrLbPXMfaIKVlVmwWhOzn949csSjvMhEn93LybM91JRN/SlzsUnk\nfcpUco/iI/dpZnKP4lNWZuF86wB6nYJJr8S8Z6trCnjnww7+9cBJ/uy+ZgrM2dGf9Q14KDBnM9A/\neUU7GeOdStKW7svLy7HZRvcs+vr6KCsrG3fNwYMH2bx5c/Tr9957j+eff54dO3Zw8OBBvva1r2G3\n2xkcHIxe09vbS3l5ebKGvaBpyXjakQ7ZpxdCiOQJhVXarZEe9FmG2OHyulWlrF1eRMtVO1/518Mc\n+bgXiPSwH3T50rpsD0kM9Fu2bOGNN94AoKWlhfLy8knL9qdOnWLNmjXRr1988UVeeuklXnrpJe68\n80727t3LmjVrqK+v5+jRowC8+eab3H777cka9oKm7dFryR9XZJ9eCCGSpsvqwh8ITzo/P1GWQc8j\nO6/lj7atwh8I8c8/buGfXj1Nd7+HUFhNa/lbSOLS/aZNm2hubmbnzp0oisLevXt55ZVXsFgsbN++\nHQCr1UpJScmMz7Vnzx7+5m/+hnA4zMaNG7n11luTNewFzekJoFMUmmoLMeh1kpAnhBBJdKXLAYwv\nfRuLTlHYfkMt6+tL+N6/n+G3Z/s4OVJIJ90z+qTu0T/66KPjvh47ewfGnZ2f6G//9m+jf25sbOSF\nF15I7OAWoSGPn7zcLLIMOpZX5nGly4k/ECI7a+EkHgohRKa43DkS6GNk3E+lsjiXxz9/PW/8to0D\nv7oCZHigF4nl9AQozo/8g1lZlc+lziHael001hSkeWRCCJF5tEBfO8PS/UQ6ncI9Ny9nQ0MpB491\nctPaimQML/7xpPXVRdyCoTBeXzDaC7m+KnIyQZbvhRAi8VRV5XKXg5L8HMw5WXN6jmWlZj7/ySZK\nCnISPLrZkUC/SGiJeJbcyD+4ldWRQC8JeUIIkXgOtx+Hyz9jIt5iIIF+kdCO1llMkRl9eaEJc45B\njtgJIUQStPWOtKaNIxFvoZNAv0hEZ/TmyIxeURRWVuXTN+iNdrUTQgiRGG0zlL5dTCTQLxJaQxtL\n7mjFpRVVsnwvhBDJ0NYXCfSzTcRbiCTQLxLRGb1pNCmkXgK9EEIkRVuvkzxTFiX56U2kSwQJ9IuE\ntkefP6aG8sqqyN6R7NMLIUTihMJhrINeaissKFP0oF9sJNAvEhOz7gEK8oyU5Bu50j0UV7MfIYQQ\nM3N5AqgqFGfAbB4k0C8azin26CFSOGfIE6B/aDgdwxJCiIzjcEfebwvTXNEuUSTQLxJanfvcnPHF\nDEfP00vLSSGESIQhCfQiHZwjde51E/aLogl5sk8vhBAJoc3o012jPlEk0C8SQ57AuP15zfJKC4oi\npXCFECJRtOPMhWluL5soEugXAa3O/dijdZqcbAPVpWau9gwRCofTMDohhMgsDpcs3YsU0zLuxx6t\nG2tlVT7+QJhumyeVwxJCiIykzeiLLJJ1L1JkYp37iaSTnRBCJI4k44mUm+oM/VgrtUAvCXlCCDFv\nDrcfk1FPdpY+3UNJCAn0i0B0Rh9j6X5ZmZlsg04CvRBCJMCQ20++OTNm8yCBflEYmqLO/VgGvY4V\nlRY6bS68vmAqhyaEEBklFA7j8gQoiLGCuhhJoF8ERqvixf6H17CsAFWVBjdCCDEfTk8AldjJz4uR\nBPpFYKase4D66gIALsnyvRBCzJmWiCeBXqRUrDr3YzUuiyTkXep0pGRMQgiRibRAXyCBXqRSrDr3\nYxXkGSktyOFyl3SyE0KIuXLIjF6kQ6w69xM1LCvA5Q3QZ/emaGRCCJFZZOlepIUzRp37iepHOtld\nlOV7IYSYE0d06V6O14kUCYbCeGLUuZ+ocZkk5AkhxHxo5W/zzXK8TqRIPBn3mtryPLIMOknIE0KI\nOdIa2kgynkiZmercj6UVzumwuhj2S+EcIYSYrSGPH5PRQJYhM8rfggT6BW+mOvcTNVRrhXOcyRyW\nEEJkpEj528yZzYME+gVvpjr3EzXIeXohhJiTTCx/CxLoFzznDHXuJ9Iq5EmDGyGEmJ1o+du8zMm4\nBwn0C95QHHXuxyqyGCnJN3Kx0yGFc4QQYhaiVfGmqUK6GEmgX+BG9+jj/4cXLZwzKIVzhBAiXqNV\n8WTpXqSQ0zP7Kk0NWoMb2acXQoi4RWf0snQvEunAry7z9y8eJxxjmd3pnbnO/UQNUjhHCCFmLVr+\nNsOW7uOPHiLhXN4Arx9uIxgK02V1U1OeN+kap9tPnskwY537seoq8jDopXCOEELMRiY2tAGZ0afV\neye7CYbCAJzvGJzyGqcnEPfROk20cE6fG58/NO9xCiHEUpCJLWpBAn3ahFWVg8c7ozP18+2TA/1s\n6txP1LAsn7CqcrVHlu+FECIekownEurMlQH6Br3cuq6S/NwsLnRMPg43l4x7jZaQJ53shBAiPkPu\nzCt/CxLo0+ad450A3LVpGatqC7E7fdgcw+OuiWbczyXQawl5nTKjF0KIeDjc/oxbtgcJ9GkxMDTM\nRxdtLK+0sLIqn6aaQmDy8r3TO7s692MVWYwU5xu51CWFc4QQYibBUBi3N5BxiXgggT4t3v2oC1WF\nu65bBkBTbSTQX5iQkOd0z64q3kQN1QU4PQGsE1YKhBBCjBctfyuBXsxXMBTmVye6MBkN3Ly2AoCa\ncjPGbD3n28fvp89njx6goVoa3AghRDwytfwtSKBPueMXbDjcfrasq8SYHUn40Ot0NC4roGfAE/3H\nBuD0zm9GrzW4ae2RlrVCCDEdra9Ifp4EejFP7xzrAODOkWV7TVNNJCiPXb4fcs9vRl9amANEcgKE\nEELElqln6EECfUp197s52zbImrpCqkvN436m7dOPXb6fS537sfLN2eh1CgNO3xxHLIQQS4MjQ8vf\nggT6lNKO1E2czQOsrMpHr1PGVcibS537sXSKQpHFKDN6IYSYwWhDGwn0Yo58gRAfnOqhwJzNpqay\nST/PztKzsiqftl4nXl8QiCTjzbbO/UTF+Tk4XP5oqV0hhBCTZWpDG5BAnzJHzvTi8QW5fWM1Bv3U\nt31VbQGqCpe6Isv3Trd/1nXuJyrON6ICgy5ZvhdCiFgytfwtSKBPmTOtdgC2rKuMec1o4RzHvOrc\nj1Vs0RLyJNALIUQsQ24/uRlY/hYk0KfMwNAwijKaCT+VVTUFKMCF9kFc3vll3GuK842R13fKPr0Q\nQsTicPszslgOSKBPGbvTR2GeEb0u9i3PzcliWVkel7uHsI9kys/1DL1Gm9HbZUYvhBBTCobCuDK0\n/C3A3NK54/Tkk09y4sQJFEVhz549bNiwAYDe3l4effTR6HXt7e088sgj3HrrrTz22GP4fD4CgQCP\nP/44GzduZNeuXXg8HnJzcwF47LHHWLduXTKHnlBhVcXu9LG80jLjtU21BXRYXZy63A/MPzEkOqOX\nQC+EEFPSqpBm4hl6SGKgP3LkCK2trezfv59Lly6xZ88e9u/fD0BFRQX79u0DIBgMsmvXLrZu3cpL\nL73EZz/7We677z6OHDnC008/zbPPPgvAN7/5TZqampI13KRyegKEwipFFuOM1zbVFvL2sU4+PGcF\nEjCjzx/Zo5eleyGEmFI0414C/ewcOnSIbdu2AdDQ0IDD4cDlcpGXlzfuugMHDnD33XdjNpv5whe+\nEP1+d3c3FRUVyRpeStlHgmw8gX7VSEJee58LmP8evTnHQLZBJzN6IYSIwSGBfm5sNhvNzc3Rr4uL\ni7FarZMC/csvvxydtQNYrVa++MUv4na7ee6556Lf/9a3voXdbqehoYE9e/aQkxM7qa2oKBfDAsqc\nDCmRffnaygLKyqZfvi8rs1BZkktPvyfymOqZHzOTsiITg27fvJ8n2Rb6+BYCuUfxkfs0M7lHo9Qr\nkVNRNZX5k+5LJtynpO7RjzVVT/Tjx49TX18/LviXlZXxox/9iHfffZfHH3+cZ599lgceeIDVq1dT\nV1fH3r17ef7553nwwQdjvpbd7knK32EuysosXB2pdpetA6t15gYzDVX50UAf8gfjesx08nOz6bS6\n6eoeXLBHR8rKLPP+e2Y6uUfxkfs0M7lH43X2DgGghMPj7stiu0+xPpQkLeu+vLwcm80W/bqvr4+y\nsvEV4Q4ePMjmzZujXx85cgSHI1Is5o477qClpQWA7du3U1dXB8DWrVs5f/58soadFNr+uJYYN5NV\nI3XvYf579GNfV2reCyHEZA5X5ja0gSQG+i1btvDGG28A0NLSQnl5+aRl+1OnTrFmzZro12+++SYH\nDhwA4Ny5c1RVVaGqKrt372ZoKPKJ6/Dhw6xatSpZw04K7ahcPHv0MNrgRlHAPM+COSBFc4QQYjpa\ni9pMDfRJW7rftGkTzc3N7Ny5E0VR2Lt3L6+88goWi4Xt27cDkf34kpKS6GO+9KUv8Zd/+Ze89dZb\n+P1+vvrVr6IoCjt27GD37t2YTCYqKip46KGHkjXspLAP+VCAwrz4An1FkYmCvGx0ijKvOvea0SN2\nknkvhBATaVn3801+XqiSukc/9qw8MG72DvDaa6+N+7q4uJhnnnlm0vPce++93HvvvYkfYIrYnT7y\nzdkxa9xPpCgKD/2HDYTCiWlEM3rETmb0QggxkSNa/jYza8ilLBlvqVJVlQGnj5oy88wXj1FfnZ+w\nMRSPbBnYZUYvhBCTDLn9GdmeVpOZH18WkCF3pEVsvPvzySAzeiGEmFq0/G2GLtuDBPqksw16gdGE\nuHQwGQ2YjHrZoxdCiAm08reZWiwHJNAnXb9jdkfrkqXYkiNZ90IIMUGml78FCfRJZ3NEZvTpXLoH\nKMo34vEFGfYH0zoOIYRYSLTyt5l6tA4k0CedtnSf7kAvZ+mFEGIyhzvynigzejFn2tJ9UX769uhh\nbHU82acXQgiNLN2LeYvO6NN8dENm9EIIMdmQO7N70YME+qTrd3ix5GalvZmMVMcTQojJMr38LUig\nTypVVbEODqd9fx7kLL0QQkzF4Yq8J2Zq+VuQQJ9U7uEg/kAorWfoNdqHDZnRCyHEqEwvfwtxBvrT\np0/zzjvvAPCP//iP/Mmf/AlHjx5N6sAyQbRrXZrP0AMYs/TkmbJkj14IIUaoqkq/Y5jSgvRPxpIp\nrkD/jW98g5UrV3L06FFOnTrFV77yFb71rW8le2yLnl3rQ78Alu4hMo4B5zCqqqZ7KEIIkXaDLj/+\nYJjyIlO6h5JUcQV6o9HIihUr+OUvf8mOHTtobGxEp8vcZY5EGZhlH/pkK87PwR8I4x6WojlCCNFn\n9wBQJoEevF4vr7/+Or/4xS+47bbbGBwcZGhoKNljW/TsQ1qgXxjLQkWSeS+EEFF9I8efK4py0zyS\n5Ior0D/88MO89tprfPnLXyYvL499+/axe/fuJA9t8RtYYEv3JZJ5L4QQUX32SKAvK8zsGX1c/ehv\nueUW1q1bR15eHjabjc2bN7Np06Zkj23R05LxChdIoJe+9EIIMUoL9BWydA9f//rXef311xkcHGTn\nzp384Ac/4Ktf/WqSh7b42Z0+LLnZGLPSWyxHI2fphRBiVN+gF4Net2AmY8kSV6A/c+YM999/P6+/\n/jq///u/z//6X/+L1tbWZI9tUVNVlYEhH6WFC2N/HkZn9LJHL4RY6lRVpc/upawwB52ipHs4SRVX\noNeOYx08eJCtW7cC4Pf7kzeqDOD1hfAFQpQULJwloUKLEQWpdy+EEO7hIF5fMOMT8SDOQL9y5Uru\nvfde3G43a9eu5dVXX6WgoCDZY1vUtDP0pQsoycOg15Gfly0d7IQQS17vyNG6TD9DD3Em433jG9/g\n/PnzNDQ0ANDY2MhTTz2V1IEtdloi3kKruFRsyaG9z0lYVTN+uUoIIWJZKhn3EGegHx4e5u233+bp\np59GURSuvfZaGhsbkz22RU1LeFtIS/cQ6WJ3pXsIpyeQ0d2ahBBiOtYlknEPcS7df+UrX8HlcrFz\n50527NiBzWbjr//6r5M9tkVNS3hbSMl4MLYvvSzfCyGWrl5tRr8EAn1cM3qbzcY//MM/RL++6667\n2LVrV9IGlQmiS/cLbFlotC+9j5VVaR6MEEKkSd+gB52iRAuJZbK4S+B6vd7o1x6PB59PMrenY1+w\nS/faWXqZ0Qshli6r3UtpQQ4Gfeb3bYlrRv+5z32Oe+65h3Xr1gHQ0tLCX/zFXyR1YIud3ekj12jA\nZDTgSvdgxhitjrdwPqi9c7yT9avKKc3LSvdQhBBLgNcXZMgToLbCku6hpERcgf4P//AP2bJlCy0t\nLSiKwle+8hX27duX7LEtagNOX3SZfCFZaDP6XruHfW+cY/3Ffr58/4Z0D0cIsQRYR5rZLIWjdRBn\noAeoqqqiqmp0U/fkyZNJGVAm8PoihRiKLPnpHsokBeZs9DplwRTNOX15AIDufneaRyKEWCqiNe4X\nWA5Vssx5c0Krlicm0/bnF0rXurF0OoXCBVQ059TlfgD6HV4CwXCaRyOEWAp6l0gfes2cA70ixVZi\n0gL9QulDP1FRfg6DTj/hcHo/rAWCIc622QFQVbA5vDM8Qggh5m906T7zy9/CDEv3d9xxx5QBXVVV\n7HZ70ga12C20PvQTFVuMXFRVBl2+6J59OpzvcOAPhMky6AgEw1gHh6kqMadtPEKIpSFaFW+BVS5N\nlmkD/QsvvJCqcWSU6Ix+ASbjwfh2tekM9KdHlu1vXlvBe6e6o5+yhRAimfoGvRRZjGQvkBbiyTZt\noF+2bFmqxpFRFvrS/bh2tcvS15zo9OUBsg06tqyvlEAvhEgJfyDEwJCPNXWF6R5KymR+pYA0WMjJ\neAClI0V80hlYB4aG6bS5WbO8iOpSc9rHI4RYGqyOyNbqUmhmo5FAnwQDQ8PkZOsxGeM+vZhS1WWR\nwNppTd+RttNXIsfq1q0sJs+UhclokEAvhEg6rZnNUjlDDxLok8Lu9FG0QGfzEGmda8zS02FNX80+\nbX9+fX0JiqJQWZKLdXBYjm0KIZKqb+RoXcUSybgHCfQJ5/OHcA8H05rkNhOdorCszEx3v4dgKPVn\n10PhMC1X7ZQV5kQ/VVeWmPEFQgx5AikfjxBi6egbXDp96DUS6BPM7tIS8RbujB6gpsxMKKzSM+BJ\n+Wtf7hrC6wuybmVJ9PhmZYns0wshkq9Plu7FfNmHFvYZes2ysjyAtCzfnxope7uuvjj6vcqSyDKa\ntn8mhBDJ0Gf3YsnNWrA5VMkggT7BBpyLZUYfCfTpSMg7fbkfvU5hTV1R9HsyoxdCJFswFKZ/aHhJ\nzeZBAn3CaUvhC/UMvaZmJPO+oy+1M/oht5+rPU5W1RSM+0QdndFLoBdCJMnA0DChsEp54dJJxAMJ\n9AkVCIb49cluTEY9q2rSV4gmHpbcbArM2XSkeEbfcjWybL++vmTc98sKc1GU0UQZIYRItKW4Pw8S\n6BPqUEsvQ24/d167bFHs/ywrM9M/NIzXF0zZa2rH6tZNCPRZBh3FlhyZ0QshkqZvifWh10igT5Cw\nqvLGkTb0OoVtN9Smezhxie7T21Izqw+rKqevDFCQlx3dOhirvMjEoMuPPxBKyXiEEEuLzOjFvJy8\n2E93v4dbrqlY8Il4mmXaPn2KMu/bep04PQHWjzlWN1ZZYSSvQStRKYQQiRQN9EvoDD1IoE+Ynx9p\nA+Dum+rSPJL4RWf0famZ0Z+e4ljdWFoBC1m+F0IkQ9+gF5PRQJ4pK91DSSkJ9AlwuWuI8+2DrFtZ\nTE15XrqHE7fqUjMKqZvRn74yYbRlAAAgAElEQVTcj6LANSsk0AshUiusqvTZvZQXmqZcUcxkEugT\nQJvNf+rmxTObBzBm6SkvMtFhdSW9xnxYVbnc7aS2PC/mp+looJ9j0ZxwWOrkCyGmNuj0EQyFl9z+\nPEign7e+QS8fnuujrjyPtcuLZn7AAlNTlod7OMigy5/U17EPRX7JKotjn1+dz4z+46sDfOkf3+U3\nZ3rmPEYhROZaqol4IIF+3t460o6qRmbzi3E5aFm0ZW1yl+974+gYlWfKItdomFMy3plWO/5AmP/9\ns7O09TpnvN7h8vHxyJl+IUTmix6tW2KJeCCBfl5c3gC/PtVFcb6RG9aUp3s4c1ITrXmf3IQ87dN0\nRfH0v2RlhSasg95ZbyV090c+SPiDYb79yilc3thd8HoHPPy/zx3l7178iH7J8BdiSVjKM/qkVnV5\n8sknOXHiBIqisGfPHjZs2ABAb28vjz76aPS69vZ2HnnkEW699VYee+wxfD4fgUCAxx9/nI0bN3L2\n7Fm++tWvArB69Wq+9rWvJXPYcXvnWAf+QJhP3l6LQb84PzOl6oidNqMvn6EHdFlhDq29TgZd/lkd\nU+zud2MyGth+Qw0/ef8q/99PWvjy/RvR6cavsnTZ3Pzdi8dxjGxVdFhdlBQs7HLFQoj564vzPSgT\nJS06HTlyhNbWVvbv388TTzzBE088Ef1ZRUUF+/btY9++fXz/+9+nqqqKrVu38pOf/ITPfvaz7Nu3\nj4cffpinn34agCeeeII9e/bw4osv4nK5ePfdd5M17LgFgiF++WEHJqOB2zdWp3s4c1ZRlEuWQZf8\nQD8wMqOf4dN0WdHs9+mDoTB9di/VJbl85raVbGwooeXKAK/86vK46zqsLp564RgOl58NDZHKfNpK\ngBAis3X3ezBm6ynIy073UFIuaYH+0KFDbNu2DYCGhgYcDgcu1+RgcuDAAe6++27MZjNf+MIXuO++\n+wDo7u6moqICv99PZ2dndDXgrrvu4tChQ8kadtxOXOxnyBPgzmurF0W521h0OoXqEjNdNk9Ss9bj\nPb86l4S8PruXUFilqsSMTlH4v+67hooiEz/7TStHz/YB0Nrj5KkXjjPkCfDA3au5/65GALr6U9+9\nTwiRWoFgmO5+DzWlkfeIpSZpEcpms9Hc3Bz9uri4GKvVSl7e+HPmL7/8Ms8++2z0a6vVyhe/+EXc\nbjfPPfccdrud/Pz86M9LSkqwWq3TvnZRUS4Ggz5Bf5Op+c5EAsh1ayspK7NMe+1MP0+3htpCWnud\nBBSFmiSMNRxWsQ56WV6VT3l5fszrysosrFoeOWPv9ofjvm8XuiPJd6uWF0Uf85U/u4VHn/4Vz/7s\nY4ymbJ559RSe4QD/dce1bL95OYFgGJ1OoX/It+D/+4y1mMaaTnKfZraU7tGVLgdhVaWxrmjWf+9M\nuE8pm4pOlVx1/Phx6uvrxwX/srIyfvSjH/Huu+/y+OOP881vfnPG55nIbk/+cmyvLbI6EfIHsFpj\nZ3mXlVmm/flCUDqyF37qXB/GJHzY7XcMEwiGKbEYY94L7T5ljbx+a9dg3Pft7JVIoxxLjiH6mFy9\nwp/eu5bvvnqaf/zhMRQF/ux3r+Ha+uLoNWWFJtp6hujrG1oUJyYWw7+lhUDu08yW2j06dS4yMSud\n5j1oKovtPsX6UJK0pfvy8nJsNlv0676+PsrKysZdc/DgQTZv3hz9+siRIzgcDgDuuOMOWlpaKC4u\nZnBwMHpNb28v5eXpz3B3D0eyus0ZUEqxpjy5CXnRRLw4jrWU5BvRKQrWwfiz4XtGlt+rS8Yn2dyw\nppzPbFlBdpaO//SZZjavqxz38+qSXNzDQYY8sTP0hRCLn/beNlUzraUgaYF+y5YtvPHGGwC0tLRQ\nXl4+adn+1KlTrFmzJvr1m2++yYEDBwA4d+4cVVVVZGVlUV9fz9GjR6PX3H777ckadtzcI8e3zDkZ\nEOiTfMSuN86jdQB6nY6SAuOs+tJ39Xsw6HWUFkx+/t+7vZ5v/7dPcNPaikk/qyqJ/NJ3p6h7nxAi\nPdpHAv2yssVTojyRkrZ0v2nTJpqbm9m5cyeKorB3715eeeUVLBYL27dvByL78SUlo33Jv/SlL/GX\nf/mXvPXWW/j9/uiRuj179vA3f/M3hMNhNm7cyK233pqsYcdNC/R5psWbiKcpMGeTZ8pK2oy+L45i\nOWOVFZo4c9WOzx/CmD19rkVYVenp91BZnDvpKJ0m1tHHqpEVgO4BD2sWYVVDIUR8Oq1uiizGJdfM\nRpPUKDX2rDwwbvYO8Nprr437uri4mGeeeWbS8zQ2NvLCCy8kfoDz4BoOkp2lIyvJSX+poCgKNWVm\nzrUNxhVcZ0s7WhdvoYryQhNnsGN1eKOrDbHYh3z4AiGqS2d/NlZm9EJkPvdwALvTF7Nr5lKwOKu8\nLABubyAjlu01y0rzUEnOcbNeu4fcWbSGnE1zm+6R8WpBezaiM3o5YidExuro0/bnl+ayPUignzP3\ncDCzAr2WkNeX2OV77WhdeVH8rSFnc5a+a6TgTVXJ7Gf0JqOBIosx+hxCiMyj5R7VSqAXsxEKh/H6\nghmxP69JVkLegHOYYEilYpqudRONBvqZM++7oxn3c8umrSrJxe704fUF5/R4IcTC1hFNxFuaGfcg\ngX5O3MORoJBRM/rSkS52tsTO6KPNbGbRSEIL9PFk3nfb3ChKfBn9U6kqjvy9ewZkVi9EJuqwutAp\nypy29zKFBPo5iB6ty6AZvclooLQgJ+Ez+t45dIzKzTFgzjHEvXRfVmCac1JkVans0wuRqcKqSofV\nTWVJpKfHUrV0/+bzEJ3RZ9hRjZqyPIbcfoY8/oQ9Z+/A7I7WacoKTdgcXsLTVEJ0evy4vIE57c9r\nopn3Cdyn9wVC/M/9H/Heye6EPacQYvb6HcP4/KElWyhHI4F+DqJn6DNo6R5G97DaE5iQN9ce0OVF\nJoIhlUGnL+Y1WnCuKp37L7FWTa8rgUfsTly00XJlgLePdSTsOYUQszdaEW/pJuKBBPo5yaTyt2PV\nV0cazlxoH5zhyvjN9midJp7M+67o0bq5z+jzzdnkGg0J3aM/fKYXgNZepyT5CZFG2lakBHoxa26v\nloyXOXv0AKtrC1EUONtqT8jzaUfrKorjP1qniSchr9sWCc5zzbiHSLGgqpJc+uxegqHwnJ9H4xkO\ncOpypMmOqsKlTse8n1MIMTejZ+hl6V7MkiuD6tyPlZuTRV2FhUtdQ/gCoRmv/+kHV/na93+LP8a1\n2tG68lnuz0N8M/r5FMsZq6rETCisRrcZ5uPDc1aCIZWm2kIAziVwdUQIMTsdVhc52XpKCnLSPZS0\nkkA/B5m6dA+wdnkRobDKxRlmoqqq8stjHbT2Omm5OjDlNb1zOFqnWVZqRlGg5Urs1YXufjcFednk\nznNlJZGZ94c/jizb//G2VShKYrdBhBDxCwTD9A5EymgvhjbUySSBfg60rPtMbJCwpi7S3GWm5fu2\nXhcOVyQ7/9g565TXzDURDyJ75+vrS7jSPUTnFM12hv1B+od881q212grAtNVyNv35jme/dnHqNOc\nAnC4fHzcaqehOp+6Cgt15RYudw8RCM68OiKESKzufjdhVV3yy/YggX5ORlvUZtYePcCqmgJ0ijJj\noD9xyRb980cXbVPub8/1aJ3mtvVVALx3avIxNS15bj6JeBrtOXpizOj7Br28c6yT9052c/rK1KsX\nAEfO9qGqcNM1kZa4TbWFBEMql7uG5j1GIcTsaKeHasqXdiIeSKCfE/dwgCyDjuysxd+5biKT0cDK\nagtXuqfPGD9xsR+9TmFzcwXu4eCUe9HRqnizKH871sbGUvJMWXxwumfSB4no0boEzOjLCkwY9LqY\nM/r3x5yHf/mdSzHP9h8504uiwE1rygFoqi0A4Lws3wuRcp2ScR8lgX4O3N5gRs7mNWvqigirKhc6\npg5QDrefq91DrKop4LYN1cDUy/fa0bq53qssg45bmitwegKcvNQ/7mejNe7nP6PX6RQqi0309Hsm\nBfFwWOX9093kZOu5YU05HVYXh1t6Jz2HddDLpa4h1i4voiDPCMCqkYS88x2SeS9EqkmN+1ES6OfA\n5Q1kZCKeZu1ybZ9+6kB/6lI/KrChoZSm2gLyTFkcu2AdFyTnc7RurOjy/YQqc9rRuvkUyxmrqsSM\nLxDCPjS+QM+Z1gEGhnzctLaCHXc1YNArvPKrywSC41cYjowk4d28tiL6vfzcbKpKcrnY4SAUnv/R\nPSFE/NqtLoosxow7HTUXEuhnKRxW8fiCGVcVb6zGZQUY9Aofx9inPzmyP7+hoQS9Tse1jaU4XP5x\ne9HRrnVz3J/X1FVYWF5h4eSlfhyu0SDc1e/GZDRQYM6e1/Nror3pB8bv02sfMG7fUEVpgYmtm2ro\nHxrmneOd4647fKYXg17h+tVl476/urYQXyBEW29imwUJsVipqspvWnpmPNkzHy5vAIfLT63szwMS\n6GfN48vMOvdjZWfpqa8uoK3XGT1KqAmGwrRcHaCsMCcaHDeNBLexy/dzaWYTy20bqgirKh+09ETH\n0Gf3Ul2Sm7BjM9Ga97bRfXr3cIBj521UleRGqwZ++tYVmIx6fvrBVTwjpy86rS46rG7W15eQO+ED\nYPQ8fZvs0wsBkQ/Fz7x2hif3fciT+z7k2HnrtD0t5kIrlCPL9hES6GcpkzPux1q7vAgVOD8hQF1o\nH8TrC7GhoTQaZJtXFGHM1vPh+b7o8bO+eWbcj3XzNRUY9ArvnexGVSNbAqGwmtC2k9EZ/ZjM+8Nn\negmGwty2oSr6d80zZXHvLctxeQP8/Ehr5Dpt2f6aCibSAn2sfAchlhKvL8j+ty+SZdCxvr6Ei50O\nvv3KKf7qXw5z8KPOhB1FlRr340mgnyVXBhfLGWtNXSRATVy+PzGSFLexoST6vSyDno0NJVgHh6O1\npRM5o88zZbGpqYzufg+Xu4boiu7Pz/9DhKayOBeF8Wfpf32yG52icGtz5bhrt91QS0FeNm8eacfu\n9HH4TC/GLD0bG0snPW9xfg6lBTmcbx+MOWtRVZUr3UOEw4md1Qix0Pz4vSs43H5+d/NyvrxjI1//\ns5u5bUMV/Q4v/+fn5/h/vvtBzAJcsyGBfjwJ9LO0VGb09dUFZBl0nG0bH+hPXurHmKVn9cgHAc2m\npsjy/Yfn+oD5H62bSEvK+/XJ7oSVvh0rO0tPaWFO9Lnb+1y09jjZ0FASzaLXGLP0/N5tK/EHw3z3\n1VNYB4e5blUpxhjHLZtqC3EPB2N2yHvzt+18/bmjfHC6J2F/HyEWmo4+F7842kF5oYl7bq4DIhUw\n//TetTz1n2/l3luW4/GF+KcDp+m1z6/JVIfVjV6nJKTORiaQQD9L0YY2GT6jzzLoWFVTQIfVHe1P\n32v30DPg4ZoVRWQZxge19fUlGPQKx85bo9eac2bftS6Wa1YUU2QxcuTjXlp7nEBijtaNVVVixukJ\n4PIGokl4W0Y+YEx024YqKotzudQZSUC8aYple422fD/VeXq708er710BZHlfZC5VVdn35jnCqsof\nb2+a9P5RmGfkD+9s4E8+tRqPL8h3XjmFzz+3ZfywqtJpdVNZkotBLyEOJNDPmrZ0n8lZ9xqtHK6W\nSHbyYmTZfsOYZXuNyWigeUUxHVY33f1urIPehCzba3Q6hS3rqxj2hzh2wYpBr6O0IHHPD6P79O19\nLg619GDJzWJj4+S/K4Bep+MP7mgAIqs761YWx3ze1dME+v1vX4i+oV0d+QAjRKY51NLDhQ4H160q\nnfL9Q7NlfRVbNy2jw+rm+69PX3I6FptjGF8gJMv2Y0ign6Xo0n2Gz+hh7Hn6yPL9ieixusl70TCa\nff+LDzsScrRuotvWR/bKVRUqi03odIltVKFtBfz8cBsub4DNzZXTzgg2NZVy90217NjaOO115UUm\n8s3ZnG8fHPfG9fHVAY583MfKqnxWVuXTZXNLXXyRcTzDAV56+yLZBh1/tG3VjNfv/J1VNNYUcOTj\nPt78bfusX09a004mgX6WtIY2mb5HD7C80oIxW8/HrXa8viDn2gapq8ijyGKc8vprG0tRlNGz54mc\n0UeeLzc6O07k/rxGa5Cj9ZO/bcPUy/YaRVH43NZV3D5SHXC665pqCxl0+aNtd4OhMD946zwKsOvu\nJlZWWQiFVdr75t9BT4iF5NVfX2HIE+DTt66IaxXOoNfxpd9bR0FeNi+/cylmPY9YJBFvMgn0s6Sd\nK8/EznUTGfQ6mmoK6Rnw8MHpHkJhNeZsHsCSm83q2sJo1bhEz+gBbt8YCb7JaFRROWbPf2WVJaFv\nFKsn9Kd/62g73f0e7rxuGSsq81leaQGgtUca4IjM0dbr5JfHOqgoMnH3TXVxP64wz8h/+b31KAr8\n06un6XcMx/1YrZmNFMsZJYF+llzRrPvMD/Qwunz/k/cjCWOx9qw1168uj/65vDixM3qAW5or+c+/\nt45t19ck/LnzTFnk50b+u94WIwlvrsYm5A0MDfOT966SZ8ri9z9RD8DKykhBHtmnF5lCVVV+8NZ5\nVBU+/8kmsgyzCzeNNQX88bZVuLwBvn3gVNzbWm29TvJMWTFXHpciCfSz5PYGMegVsrOWxq1bszwS\noJyeAJbcLFZW5U97/XWrRmf8yZjR6xSFG9eUYzImZ+ukvroAk1E/bRb9XCwrM5NrNHCh3cH+ty/i\nC4S4/86G6MpQVWku2QadBHqRMVp7nVzscLCxoYR1K6efIMRy53XL2LK+ktYeJ7+ZopnURO7hANbB\nYZZX5CWsamYmWBrRKoHcw5GGNkvlH1FduYXckaC6vr4E3Qx/7+L8HJpXFFFRnLsotzce/PRavvan\nNyV8xUanKKyqKaBv0Mtvz/ZRX53PljE5AHqdjtryPEnIExlDC8yf2Dh9Dst0FEVh+w21AFzqmrk2\nvtZTom5kK0xESKCfJbc3sCSO1ml0OiVaHGeqym9TeegPNvDV3Tcmc1hJY87JSvixPU3TyH1UgF2f\nXD3pQ9PySknIE5khHFY5/HEv5hwD66c5TheP6lIz2QYdl7tmXu3Samwsr5BAP5YE+lkIqyqe4czu\nRT+Ve25Zzic2Vo0rezud7Cw9xuypq8QtZetXlqAAW6+viSbfjSUJeSJTnG2z43D5uX51+byL1hj0\nOuoqLXTZ3DMW0Wnrk0A/laUVsebJMxxEZWmcoR+rcVkBjcsK0j2MRa+mPI//8cXNFBfkTPlzScgT\nmeI3ZyLL9pubE5PrUl+Vz8UOB629zmhi61Rae5zkZOspS/DR3sVOZvSzoB2tWyoZ9yLxSgtNMfMc\nJCFPZIJAMMSH5/ooshhZNU1Qno0VVZEZ+pXu2KtdPn+Inn4PdRWWGXOJlhoJ9LOg1blfjElmYuGT\nhDyRCU5c7MfrC3HzNRUJC7j1I6d9pgv07X0uVGTZfioS6GchOqM3yY6HSA5JyBOL3eGRZftbEnhE\ntazQhDnHMG2gb+0d2Z+vlEI5E0mgnwX3EiuWI1Jvxcg+vSTkicXIMxzgxCUby0rNCa1MpygKK6vy\nsQ4O4xzppjmRFujrZEY/iQT6WYjWuZele5EkK0Yy76/IPr1YhI6esxIMqdx8TUXCa41oxbpi5bC0\n9TjJMuikB/0UJNDPwmj5W1m6F8mhJeS1SqAXi1Aylu01K6tH9um7Jq92BYJhOm1uasvz0OskrE0k\nd2QWZOleJJsk5InFyu70cbbVTmNNAaWFiT/eps3oL0+xT99lcxMKq5KIF4ME+llYSp3rRPpIQp5Y\njA6f6UUlObN5gAJzNiX5Rq50D6Gq6rifje7PSyLeVCTQz8LoHr0s3YvkkYQ8sRgdPtOLXhdpOpUs\nK6vycXoC9A+Nb1sbLX0rNe6nJIF+FtzeAHqdgjFLyruK5JGEPLHYdPe7ae110ryyGEtudtJeJ7pP\n3z3+d6O114lep7CsVGb0U5FAPwsu79LqXCfSQxLyxGJzaKRT3S0JKnkbi1YmemxCXigcpr3PxbJS\n86x73i8Vcldmwb0EG9qI1JOEPLGYqKrK4TM9GLP0XNdYltTXWl5pQWF8hbyefg+BYFha005DAn2c\nwqoa7UUvRLKtqMyXhDyxKFzsdGAdHGZTU2nSu1aajAaqS81c7XESDkcS8qIV8STjPiYJ9HEa9gVR\nVZZUL3qRPtKyViwWh073AHDruqqUvN6KKgu+QIiu/siH4NYeFyCBfjoS6OPkkox7kUKSkCcWg0Aw\nxJGP+yjMy2bt8qKUvObEBjetvU4USGjJ3UwjgT5OUixHpJIk5InF4MTFfjy+ILc0V6LTpSZJeWzm\nfVhVae9zUlmSm/Rtg8VMAn2cooFe9uhFCkhCnlgMPogu21em7DVryvIw6BWudA1hHfTi9YVk2X4G\nEujj5NKq4knWvUgRLSHvUqfs04uFx+nxc+pyP3XledSUpW7Z3KDXUVdhocPq4mKHA5COdTORQB8n\nt1c614nU2rQ6clTpB2+dxx+QWb1YWI583EcorKZ0Nq9ZWRX5EPzeyW5AKuLNRAJ9nLQ69xLoRaqs\nXV7E72yqocvm5uV3LiX8+YOhME6Pf1LdcCHi8cHpbhQFbk5SbfvpaAl559oHAVguNe6nldR16Cef\nfJITJ06gKAp79uxhw4YNAPT29vLoo49Gr2tvb+eRRx7hnnvu4a/+6q9oa2sjFArx3//7f+eGG25g\n165deDwecnMjfYYfe+wx1q1bl8yhT6LN6OV4nUil++9q4OM2O7881sFtm2pYXpqYXtudVhd/v/8j\nHC4/JqOB8iITFUUmyotMlBfmsqmpjNxFtk2lqio/ef8q61aV0SBv/EnV3e/mSreT9fUlFOQZU/76\nK6pGZ/BlhTnkyvvytJL2m3zkyBFaW1vZv38/ly5dYs+ePezfvx+AiooK9u3bB0AwGGTXrl1s3bqV\nH//4x5hMJn74wx9y4cIFHn/8cf7t3/4NgG9+85s0NTUla7gzis7oF9mbn1jcsrP0/Pl91/D1547y\n9P7jfHX3jeSb51dLvMPq4u9+eBynJ0DzymIGnT66bO5xGf5vH+vgrx+4IWWZ1LF097vxB8JxLc3+\n4sMOfvzeFY5dsPK1L9yUgtFN74W3znOh08FfP3B9xvVI15LwNq9L/WweoKI4F5PRgNcXlP35OCQt\nah06dIht27YB0NDQgMPhwOVykZc3/pP2gQMHuPvuuzGbzXzmM5/h05/+NADFxcUMDg4ma3iz5pKs\ne5EmdRUW/uCOBl565yL/+/WzPPQH6+fcb6Gjz8VTPzyOyxvggbtXc+d1y4BI5cdBp49eu5e3ftvO\nRxdtvHuii7tGfp5qFzoGef03bXx00YZep/DfdmykeUVxzOs7+lzR7Y32Xhee4UBaZ3kft9r5xYcd\nAHT3e1KarJZsYVXlNy095GTruW5VckvexqJTFFZWWThz1S4Z93FIWqC32Ww0NzdHvy4uLsZqtU4K\n9C+//DLPPvssAFlZo7+Yzz33XDToA3zrW9/CbrfT0NDAnj17yMnJifnaRUW5GAyJPVPpD4bR6xRq\nlxXO+k22rEz+IcZD7lNsn7/3Gs62D/LRRRsfXuznnltXzvo5rnQ5+Pv9H+HyBvgvf7iRT21eMe7n\nFeWwGljXVM4X//aXHPjVZe6+dWXKlmZVVeXox73829sXOHNlAICmukIudw7xz6+e5u/+6yeoneJN\n3RcI8b3//VuCoTCrlxdxrtXOgCfI8trYHwySKRAM88Nnj0S/tnsCXLcA/23P9fft1EUb/UM+tt9U\nR011YYJHFb+NTeWcuWpn09rKpL53ZML7UsrWoadK+Dl+/Dj19fWTgv/zzz9PS0sL//zP/wzAAw88\nwOrVq6mrq2Pv3r08//zzPPjggzFfy273JHbwwKDThznHgM3mmtXjysosWK1S9GQmcp9m9uU/2sT/\n/Xdv868/Ps2yYhNVJea4H9vW6+TvX4wE+d33rOH6xpJp7/dnt6zgxbcv8swrJ9l9z5pEDH9aFzsd\nPPfzs3RaI2VNNzSUcO8ty2mqLeTQ6R7+5adn2PvMB/zVAzeQP6EN6vNvnae1x8ldm5axbmUx51rt\nHPu4h5piU9LHPZWffnCVjj4XDdX5XOoa4vQFG+tTVDUuXvP5ffvZ+5cBuK5h+n9DyfaJdZWU5Rup\nKjQmbRyL7X0p1oeSpG0clZeXY7PZol/39fVRVjZ+mefgwYNs3rx53Pdefvll3n77bb773e9GZ/jb\nt2+nrq4OgK1bt3L+/PlkDTsmaWgj0q200MSffGoN/mCYZ35yhlA4HNfj2nqd/N0Pj+P2BvjCPWv4\nxMbqGR+z9foalpWa+fWJrnGdwpLB5w/xT6+epsvm5pbmCr72pzfx3+7fSFNtZLa4eV0l9926Auvg\nMN9+5dS4AkInL9n45YcdVJXk8rm7GqmvLgDgcld6ag9YB7289sFV8s3ZfOn316Mw2nQlE/gCIY6e\n7aMk30hTXfpm8wDGbD3XNpZK2/A4JC3Qb9myhTfeeAOAlpYWysvLJ83cT506xZo1o7OF9vZ2Xnzx\nRb797W9jNEaWC1VVZffu3QwNRX5xDx8+zKpVq5I17CmpqorbG5TytyLtblhTzo1rymntdXK1O74A\n8tzPz+IZDrL73jXcHkeQh0hRkv/4ySZU4AdvniOcxCN4Pz10FbvTx+9uXs6f39c8Zc3y37t9JTet\nLedih4Pv/+wsqqricPt59t8/xqBX+E+faSY7S0+BOZuK4lwudTpSfmxQVVWef+s8gWCYz21tpMhi\npLIkl7ZeZ1LvXyp9dMHGsD8UKXkrAXbRSNrS/aZNm2hubmbnzp0oisLevXt55ZVXsFgsbN++HQCr\n1UpJSUn0MS+//DKDg4P8+Z//efR73/ve99ixYwe7d+/GZDJRUVHBQw89lKxhT8nrCxFWVcm4FwvC\nNSuK+O3ZPrpsbhqWFUx7bSgcpr3PxYoqC7dviC/Ia1bXFXHzNRUcPtPLr090cce1iU/M6x3w8MaR\nNorzjfzuLStiXqcoCngaHE0AABpNSURBVA/+7lr6h4b5zZleyotMXO1xMuQJsHNr47jM69XLi/jV\n8U767F4qihNzHDEex87bOHmpn7XLi7hl5Gz58koL3f0erINeKopSN5ZkOXymF0htyVsxf0mNXGPP\nygPjZu8Ar7322rivH374YR5++OFJz3Pvvfdy7733Jn6AcZJiOWIhqS6N7M1rbTqnYx0cJhhSqZ7F\nfv5YO+5q5KOLNn707mWuX11OXgJ/B1RV5YVfXCAYUtm5ddWMTUmyDHoe+g8b+Mb/OcpP3r8KQPPK\nYrbdWDvuOi3QX+pypCzQD/uDvPCL8+h1Cv/xk03R5eS6cgu/aemltceZEYH+cvcQpQU5s8oPEemX\nWYc7k2T0DL0EepF+2ptsl23mpNMuW+TDgPbhYLaKLEY+u2UlLm+AV351eU7PEcuJi/2cutzPNSuK\nuH51fMe08s3Z/MX9GzEZ9eSZsnjwd9dOWkJeszySbX8phfv0P3kvsv1wzy1144Kgdv4/E/bphzx+\nhtz+jDoquFRIoI9DtCqe9KIXC0CeKYsCczbdcczoO+cZ6AG23VBDdamZd493crUnMcEzEAxFZ8B/\nvK1pVglVy0rNfOPPbuHrD95E4RRH/1ZWF2DQ6+ackHeuzc4Pf3EBh8sX1/UdfS7e/G07pQU5fHrC\nkUWtNGtbBrQb7rLO/9+SSA8J9HGQpXux0FSXmrE5hvH5p292052AQG/Q6/j89khi3nOvn4s72386\nrx9uw+YYZvsNtXMaW5HFGPN8f5ZBx/LKPDr6XPhm0QzoXJudp144xv944ThvHW3n1feuxPW4f3v3\nEmFV5fPbm8jOGr/9kJuTRVlhDq29rkXfU0D70LisTAL9YiOBPg7RXvSydC8WCG3PvXtg+ll9l81N\ndpaOkoLYBabisXZ5EVvWVdLa6+TN37bP67lsg17+/VArBXnZ3LdlxbyeK5aG6gJCYXVcWd9Yxgb4\ns22DrKsvpiQ/h/dP9TDk9k/72A6ri5OX+mlcVsDGxtIpr6mrsODyBrA741shWKg6rZEaIstkRr/o\nSKCPw2j5W1m6FwtD9UhzG20PfirhsEr3gIeqEnNCjkJ97ndWkZ+bxau/vkLvPIpS7X/7IoFgmB13\nNWIyJud3qr460t1suuV7ry/I/3zx+LgA/1e7rufhHdfyqZvrCIbCvH2sY9rXeeNwGwD33FIX8xqt\nRGs8HzoWsk6bG52iUFWy+JMKlxoJ9HFwD4/0opcZvVggopn30yTkWR1eAsHwnDPuJ8ozZfHH25sI\nBMM89/rZGZeiQ+EwNoeXc2123j/VzU/ev8K/vNbCh+etNNUURI+gJYMW6C91OWJec/CjTlqu2llT\nVxgN8NpxxdvWV2HOMfD2sc6Yy/8DI0f9qkpyY87mIfUJeVe6hwiHE7tNoKoqnVY3FcUmshJcXlwk\nn0xR46At3SfyaJEQ81EVDfSxZ/SjGfeJm4HduKac37T08tFFG78+2T1llT1VVfnZb1r58XtXCYYm\n7+fnmbL4j59cndSKZiX5ORSYs2PO6MNhlXeOdZJt0PFf/sP6SR/ijdl67tpUw08/uMoHp7q5a1PN\npOd462g7obDKp26qm3bFRDvj39Y7u/LZc3Hioo2n/+0kn9/exO9cP3nMczXo8uPxBVm7YmGV8hXx\nkUAfB5nRi4UmPzebPFPWtGfp53u0biqKEjknfrbNzv63L7KhoWRc5rvXF+R7//4xx85bKTBns3Z5\nESUFOZQU5FCaP/L/BTlJnxUqikJ9dT7HL9gYGBqmOH98jsLJy/3YHMN8YmN1zN/r37m+hp8fbuWN\n37Zzx7XLxrXsdQ8HOPhRF4V52dzSPH3xmAJzNoV52SmZ0X90MVJ2/ONWe0IDvezPL26ydB8H13AA\nnaJgMsqSlVg4qkvNWAe9+GMsLWuBPtFvzsX5Odx/VyNeX5Dn3xztO9Flc/P1545y7LyV1bWFfPVP\nb+LPP9PMH9zRwJ3XLmNdfQlVJeaULf1qy/BTzerfHmkhu3VT7Gp/BeZsbl1XSZ/dy/ELtnE/O3i8\nE58/xPYba8kyzPw2urzCgt3pmzG5bz5UVeX05UjXv4sdgwnN8u8YOVonZ+gXJwn0cXB7A+TmGKR5\nglhQqkvNqCr0DEy9T99l85Bl0FFakPgubndcW01TTQEfnrfy4bk+PjzXx9f/z1F6Bjx88sZaHv2j\naykwZ8/8REnUEGOfvmfAw+krA6yqKRhXOncqn7wxkmT3xpG26PcCwRBvHe3AZNRzx8b4ygKPLt8n\nb1bfM+Chf2gYgCFPAOugN2HP3SVH6xY1CfRxcHulc51YeKpHsp+nWr4Pqyrd/W6qinPHLTknik5R\n+JN71mDQ6/jXf/+Y7xw4jaqq/KfPNLPzd1ah16X/rWVFZT6KMnlGr2XSx7O0XV1qZmNDCRc7HVzs\niHxg+OB05NjdndctIzfO/hepSMhruRKZzWuz7gsdsRMRZ6vT5sKgVygvSk/rXzE/6f9tXOBUVcU9\nHCRPGtqIBaZqmsx7m2MYfzCc1CpmVSVmPrNlBT5/iPIiE3+96wZuTmIm/WwZs/XUluVxtccZTQoc\n9gd5/1Q3BXnZbGqKr+zup26OzOp/fqSNcFjl50faMegVtl1fO8MjR6XiiN3pkUD/2dtWAnCpMzGB\nPqyqdNrcVJWYF8QHODF7Er1mMOwPEQqrMqMXC060aM4UmffJSMSbyr2bl1NXkUfjssK4Z7epVL+s\ngLY+Fx1WFysq8znU0ovXF+KTN9Zh0McXtJpqC1lZZeH4eStvHGmjd8DD7RuqKLJMXZlvKsX5RvJM\nWUmb0QeCYc622UeO+pWQnaXjQoICvc0xjD8QlkS8RUw+ns1AGtqIhaowLxuT0TDl0n0iSt/GQ6co\nbGgoXZBBHqC+amSfvnMIVVV5+1gHep3CHdfG37JXURTuvqkOFXj54CUURmf5s3mOuoo8rIPDeEbe\nUxLpYqcDfyBM88piDHod9VX5dFndCXmtaMa97M8vWhLoZ6A1tJGqeGKhURSF6tJc+uzeSefVE9HM\nJhM0LNMq5Dk43z5Ip9XN9avLpmyGM53rV5dROlJG+NpVpXNq07o8iefptf35dStLAGisKUAlMR38\nRk9vSMb9/9/enQdFfW0JHP/+upumoRFokEXEfR8EoxiTuMfkvcRJSiMaUB8kJE4WTTRWzKj11Gip\nbxL3FGrU8lkhEbRcXhYddWLMjJPRGJOJThCj4o60gCCbotL2Mn9gdyTu0E3bv5xPlVXS9A/uPdX0\n6Xt/597rqyTR34P9xhKVsCYN2ytcCE+ICTdiszsoLq9bYX2+tBqdVkNE6B/7dRsVFkigv46T56v4\n1rWk7sHXl2s1Gob0aYNOq+H53q3r1RZPFuTlnr6ITqvQqUUoAO1vLC10R0GeuUQq7n2dDFPvoVV0\nEyalPOIaGQjxMHGO2AtLq133UGsr7q8QHRb4hy+e0tzYOCf3dBmlFddoERlEh9iQev2svgnN6N01\nut6rGDy1xK6y2kJ+8WW6tDLhr6/do8C5h4A7CvIKSqrx99M2+GAk4T1/7HeB+6BRFOLahGHQy2ci\n8fCJuc1WuGVV16i5bnPr1re+zLnvvd3h4KnE2Abth9GQpYqRpgD89VrO3mHqvua67bZbBt/Lr2ec\n0/ZhrseMBj9imho5db6qQccKW212isqqiWka6JaDkYR3SPYSwoc5K+9vLshrrIp7X+Ec3Qb667y6\n/E+jKLSKDOK4uZIai801+gbIOVnK3//9CFqtwtC+beiX0Oy+Z2Ocu+HF3ZTooXb6/nxpNQUXql23\nDR5Ubf2HQ+7P+zgZ0Qvhw8KC/fHXa+uM6J3r6mU5VK32zUOIDgvkud6t8Pfz7jbWLaOa4HDAuRuV\n7Da7nU27T/DRphyuWWxcrbHy2X8cY8bff+TnYyX33MbW4XBw+EwZwUY9sZF1k/Fv9+kr6t1e2RFP\nHWREL4QPUxSFmPBAzl24jM1uR6vRyIj+dwL8dfzb6497uxnAbwV5+cWXCA82sPKrXI4XVBJpCmDs\n0K6EBOnZsuc03/1SyPIvDtG+eQgvPtmODrGht/155y5cpqrawhNx0bdMrTtrEU6YK3m65/1v7nOz\nAllapwqS6IXwcTHhRk4XXqKk4hrRYYGYS6vRahQiQmW70oeNc4nd97lFfPk/p7l89To9O0fyyuDO\nBPjXvh2/9Gxn/vRoC/7x36c4kFfCB1kHeLJHc/7yp463JPPDt7k/7xRpCqBJoB8nGlCQZ5aldaog\nU/dC+LibC/IcDgfnL1YTHR543zu/icbTrGkgfjoNp85Xcc1iJfXPHRk7NM6V5F3PCzfydlI8f01N\nJDbCyH8dMJO1M++WqXzn/fl/uk2iVxSF9s1DKKuqoezGYTcPylxSjdGgIzTIuwcUiYaRdwIhfFyz\nmxJ9+aUaaiw2V5GeeLhoNRoS2oYTZQrgr2mJDOpx91UA7WNDmDy6By0ig9h90My6b467kn3NdRvH\nCypoGRl0x5MC2980ff+grlttFJdfIaapUU7u9HGS6IXwca4R/cVquT/vA8YN68rfXn+c1tH3tzdH\nUIAf7418hNgII98eKGD9t7XJ/lh+BVabg7i2t47mnZwFeSfusHHO/50oZfkXh6isttzyvcKLV3A4\noLmcQe/z5B69ED6uabABP52GwtIrsvWtD1AUhQcdHzcJ1PPeyO7MX3+QXf9bu1+/zV47su/a+s6J\nvnV0E3Ra5bYH3JwtusTKL3OxWO3UWGxMTO5Wpwbgt/vz8lrydTKiF8LHaTQKzcICKbxY7dquVBK9\n+gQb9fzrqO40Cw/k6x/PsfugGb2fhvZ3qMgH8NNpaR0dzLniy9RYbK7Hyy9dI+MfOVy32omNMJJ7\nuoxvfjpX51rnaylWKu59niR6IVQgpqkRi9XOodMX0WoUokxSca9GITeSfVRYIFabg84tTfjp7v42\n3r55CHaHg1OFtQfcXLfa+SDzJ8ov1TCsf1smjexOsFHP5t0nOVv02/a8zlPr5EOj75NEL4QKOAvy\nKi9biDQFSMW9ioUG+TN5VHcSO0XwzKP3Xh/vKsgrqMDhcJD9zTGOnCmjV5dInnuiFSFGPf/yXBds\ndgcrtxzmmqX2xE5zaTXBRj1NAqXi3tfJu4EQKnBzlb3cU1U/UxN/3hoWT5e73J93cm4BfMJcxX8e\nMPPdL4W0bR7CK//cxVVN37VtOH9+tAXFZVdYt+s41yxWSiuvyWtJJaQYTwgVuPkAG5lqFTcLMeqJ\nNAVwLL+cw6fLCA70Y9orvVCstjrPGz6gHUfzy9mTU4jRUJsaZEc8dZARvRAqEGkKQHvjZDVJ9OL3\n2jcPwWK1oygwblg8kaZbTzb002l4Y0gcej8NX/9YW5gXK0vrVEESvRAqoNVoiA6vffOWzXLE7yW0\nC0cB0p7pRMcWd67SbxZu5C9Pd3R9LVP36iBT90KoxCPtm2KzOVwJXwinRztH0rVNGIEGv3s+t29C\nM/LOVZB7ukxG9CohiV4IlRg+oB3DB7TzdjPEQ0hRlPtK8s7nvvpcF+wOB1qNTPqqgSR6IYQQdSiK\nglb2t1cN+bgmhBBCqJgkeiGEEELFJNELIYQQKiaJXgghhFAxSfRCCCGEikmiF0IIIVRMEr0QQgih\nYpLohRBCCBWTRC+EEEKomCR6IYQQQsUk0QshhBAqpjgcDoe3GyGEEEIIz5ARvRBCCKFikuiFEEII\nFZNEL4QQQqiYJHohhBBCxSTRCyGEEComiV4IIYRQMUn0QgghhIpJoq+HvLw8nn76abKysu77msLC\nQtLS0hg9ejTvvPMOFosFgKNHj5KUlERSUhLLly/3VJO9wp1xiouLIy0tzfXPZrN5qtmNyp0xcnr3\n3XeZOnWqu5vqVe6M07Jly0hJSSE5OZmPP/7YU01udO6M0fbt2xkxYgTJycksWbLEU032CnfGqbKy\nkjFjxjBhwgRPNdctJNE/oCtXrjBnzhyeeOKJB7ouIyOD0aNHs27dOlq1asXmzZsBmDFjBnPmzGHz\n5s2cPHmSq1eveqLZjc7dcQoKCmLt2rWuf1qt1hPNblTujhHA3r17yc/Pd3dTvcqdcSooKCAvL48N\nGzawfv16vvzyS4qLiz3U8sbjzhhdvXqVhQsXkpmZyYYNG/j+++85ceKEh1reuNz9Nzdz5kwSExM9\n0VS3kkT/gPR6PatXryYyMtL12IkTJ3jppZd4+eWXGTduHFVVVbdct3//fp566ikAnnzySfbt20dp\naSlXrlwhLi4OjUbD4sWLCQgIaLS+eJI746RW7o6RxWJhxYoVjB07tnE60EjcGafY2FgyMjKA2tGY\noigEBQU1Tkc8yJ0xCggIYMuWLQQFBaEoCqGhoVRUVDRaXzzJ3X9zc+fOlUSvRjqdDoPBUOexOXPm\nMHv2bD799FP69OlDdnb2LdddvXoVvV4PQHh4OCUlJZjNZkJCQpg6dSojR44kMzOzMbrQKNwZJ6hN\nYpMmTWLkyJF88sknnu9AI3B3jFatWsWoUaNUkbhu5u44Qe0b9PPPP8+4ceMwGo2e7UAjcHeMnK+h\nY8eOYTab6datm4d70Dg8FaeHnc7bDVCDnJwcZsyYAdQmpPj4+Ls+33m8gMPhoKCggOXLl2MwGEhJ\nSaFPnz506NDB4232hvrGCWDy5MkMGTIERVFITU2lZ8+e97zeF9U3RmfOnCE3N5fx48ezf/9+j7fT\n2xryWgKYPn0648ePJy0tjR49etCiRQuPtdVbGhqjM2fO8N5777Fo0SL8/Pw81k5va2icfIEkejcI\nCAjgs88+Q1EU12MHDx5k8eLFACxcuJDAwECuXbuGwWCguLiYyMhIwsPD6dChAyaTCYDExESOHz+u\n2kRf3zgBjBo1ynXN448/Tl5enioTfX1jtHv3bs6fP09ycjKXL1+mrKyM1atX89prr3mrKx5V3zgV\nFhZSWlpKfHw8ISEh9OjRg0OHDqky0Tfk762oqIi33nqL+fPn06VLF6+0v7E0JE6+Qqbu3aBz5858\n9913AGzbto19+/bRvXt3V+FYVFQUvXv35uuvvwZg586d9OvXjxYtWlBdXU1FRQV2u50jR47Qtm1b\nb3bFo+obp1OnTjFp0iQcDgdWq5UDBw6o9sNQfWOUnp7O1q1b2bhxIzNnzmTgwIGqTfJQ/ziVlZUx\na9YsrFYrNpuNw4cP06ZNG292xWPqGyOAadOmMWvWLOLi4rzW/sbSkDj5Cjmm9gHl5uYyb948zGYz\nOp2OqKgoJk6cyKJFi9BoNPj7+7No0SJCQ0PrXHfhwgWmTJlCTU0NMTExfPDBB/j5+fHLL78wd+5c\nFEWhX79+jB8/3ks9cy93x2nBggX88MMPaDQaBg0apIqCM3fHyGn//v188cUXfPjhh43dJY9wd5xW\nrVrFrl27cDgcDBw4kLfffttLPXMfd8aooKCAF154gYSEBNfz0tPTXcVovsydcdJoNKSnp1NVVUVx\ncTEdOnRg3LhxD1zR3xgk0QshhBAqJlP3QgghhIpJohdCCCFUTBK9EEIIoWKS6IUQQggVk0QvhBBC\nqJgkeiFUoKCggP79+zfq73TXKYKdOnUiNTXVdTJhcnIyO3fuvOd1W7duxW63N/j3C6F2sjOeEKJe\n1q5d67aflZmZiU5X+3ZUWlrK0KFD6dWr1y3rmW+2dOlSBg8ejEYj4xUh7kYSvRAqt337drKysnA4\nHISFhTF37lxMJhPr1q3jq6++ws/PD39/f5YsWUJwcDCDBg1i8ODBnDt3jsmTJzN27Fj69u1LTk4O\n1dXVrFq1iqioKDp16sThw4dZsWIFFRUVFBUVcfbsWR577DFmzJhBTU0NU6ZMwWw2Ex0djVarpU+f\nPrz44ot3bW/Tpk2JiIggPz+f4OBgZs6cyalTp7BYLHTr1o3p06eTkZHB2bNnSU9PZ9myZRw9epTl\ny5fjcDjQ6XTMmTNHldvaClEf8lFYCBUrLCxk5cqVZGZmsn79enr16sWqVasAqKmpYc2aNWRlZdG8\neXO2bNniuq5169au41xPnjxJUlIS2dnZdOnShR07dtzye3799VcyMjLYvHkzn3/+OZWVlWzZsgWr\n1cqmTZt4//332bt37321OTc3lwsXLtCuXTsqKyvp1KkT2dnZbNq0iT179pCXl8eECROA2pkAf39/\nZs6cydKlS8nKyiI1NZX58+c3NHRCqIaM6IVQsYMHD1JSUsKYMWOA2tO5YmNjAQgNDeX1119Ho9Fg\nNpuJiIhwXde9e3fX/00mk+tsgZiYmNueTZ6YmIhWq0Wr1WIymaisrOTIkSP06tULgIiIiLue252e\nno6iKJSWlmIwGFi5ciVGoxGDwUBhYSEpKSno9XpKSkooLy+vc+3x48cpKSlxbR9ts9nqHFAixB+d\nJHohVEyv15OQkOAaxTsVFRUxb948tm3bRnh4OPPmzavz/Zv3ztdqtXW+d7tds2/3HLvdXuf++d3u\npTvv0efk5DBlyhQ6duwI1B4ycujQIbKzs9HpdCQlJd22jzExMW6tGRBCTWTqXggVi4+PJycnh5KS\nEgB27NjBrl27uHjxIiaTifDwcCoqKtizZw8Wi8Wtv7tt27YcPHgQgIsXL/Lzzz/f85qEhAT69u3L\nRx995LquTZs26HQ6cnNzyc/Pd7VTURSsViutW7emvLycvLw8AH766Sc2bNjg1r4I4ctkRC+ESpSV\nlZGWlub6Oj4+nsmTJzNt2jTeeOMNAgICMBgMzJs3j7CwMFq1asWIESNo2bIlEyZMYNasWQwYMMBt\n7UlKSmL37t2kpKQQGxtLz549bxn5387EiRMZMmQIzzzzDM8++yxvvvkmqamp9OjRg1dffZW5c+ey\nceNG+vXrx/Dhw1mxYgULFixg2rRp+Pv7AzB79my39UMIXyen1wkhPKK4uJgDBw4wePBg7HY7w4YN\nY9asWXXu/wshPE9G9EIIj2jSpAnbt29nzZo1KIpC//79JckL4QUyohdCCCFUTIrxhBBCCBWTRC+E\nEEKomCR6IYQQQsUk0QshhBAqJoleCCGEULH/B7xZndsBU2HsAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 576x396 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "ESTKDFlMw2oG",
        "colab_type": "code",
        "outputId": "04c4d713-ab87-475b-edad-2d91ff522e12",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "cell_type": "code",
      "source": [
        "learn.fit_one_cycle(1, 3e-2, moms=(0.8,0.7))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "Total time: 16:09 <p><table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.691735</td>\n",
              "      <td>0.697038</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>16:09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "-gfLZ4jFw_GA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Gradual Unfreezing"
      ]
    },
    {
      "metadata": {
        "id": "FOyWNosgw2wv",
        "colab_type": "code",
        "outputId": "ab9a3f99-8db1-4824-c48d-6d947bc8d1e3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "cell_type": "code",
      "source": [
        "#unfreeze last 1 layer and train\n",
        "learn.freeze_to(-2)\n",
        "learn.fit_one_cycle(1, slice(1e-2/(2.6**4), 1e-2), moms=(0.8,0.7))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "Total time: 17:34 <p><table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.672092</td>\n",
              "      <td>0.671172</td>\n",
              "      <td>0.583720</td>\n",
              "      <td>17:34</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "gx0-axnvw21W",
        "colab_type": "code",
        "outputId": "072801a4-901e-48e1-9b22-3a013b1eedce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "cell_type": "code",
      "source": [
        "#unfreeze last 2 layers and train\n",
        "learn.freeze_to(-3)\n",
        "learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "Total time: 24:19 <p><table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.588279</td>\n",
              "      <td>0.546222</td>\n",
              "      <td>0.723240</td>\n",
              "      <td>24:19</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "GfKH0d5GxD78",
        "colab_type": "code",
        "outputId": "b49e690a-3bc9-41e9-b938-df8c412cf496",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 135
        }
      },
      "cell_type": "code",
      "source": [
        "#unfreeze whole model and train\n",
        "learn.unfreeze()\n",
        "learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "Total time: 59:50 <p><table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.487773</td>\n",
              "      <td>0.754746</td>\n",
              "      <td>0.625760</td>\n",
              "      <td>29:50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.407351</td>\n",
              "      <td>0.370769</td>\n",
              "      <td>0.832560</td>\n",
              "      <td>29:59</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "CBpiGZOZxTGo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Prediction"
      ]
    },
    {
      "metadata": {
        "id": "K-eB4HUXxP3w",
        "colab_type": "code",
        "outputId": "cd017f19-943a-47c0-bb02-9d83595c08f6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "learn.predict(\"Five medical students (Kevin Bacon, David Labraccio; William Baldwin, Dr. Joe Hurley; Oliver Platt, Randy Steckle; Julia Roberts, Dr. Rachel Mannus; Kiefer Sutherland, Nelson) experiment with clandestine near death & afterlife experiences, (re)searching for medical & personal enlightenment. One by one, each medical student's heart is stopped, then revived.<br /><br />Under temporary death spells each experiences bizarre visions, including forgotten childhood memories. Their flashbacks are like children's nightmares. The revived students are disturbed by remembering regretful acts they had committed or had done against them. As they experience afterlife, they bring real life experiences back into the present. As they continue to experiment, their remembrances dramatically intensify; so much so, some are physically overcome. Thus, they probe & transcend deeper into the death-afterlife experiences attempting to find a cure.<br /><br />Even though the DVD was released in 2007, this motion picture was released in 1990. Therefore, Kevin Bacon, William Baldwin, Julia Roberts & Kiefer Sutherland were in the early stages of their adult acting careers. Besides the plot being extremely intriguing, the suspense building to a dramatic climax & the script being tight & convincing, all of the young actors make \\\"Flatliners,\\\" what is now an all-star cult semi-sci-fi suspense. Who knew 17 years ago that the film careers of this young group of actors would skyrocket? I suspect that director Joel Schumacher did.\")\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<fastai.core.Category at 0x7f869b99b588>, tensor(1), tensor([0.4656, 0.5344]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "metadata": {
        "id": "UPfQNXi2xP0e",
        "colab_type": "code",
        "outputId": "f501b680-9a14-4084-d001-25b4772ccc76",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "learn.predict(\"This has to be one of the biggest misfires ever...the script was nice and could have ended a lot better.the actors should have played better and maybe then i would have given this movie a slightly better grade. maybe Hollywood should remake this movie with some little better actors and better director.sorry guys for disappointment but the movie is bad.<br /><br />If i had to re-watch it it would be like torture. I don't want to spoil everyone's opinion with mine so..my advice is watch the movie first..see if u like it and after vote(do not vote before you watch it ! ) and by the way... Have fun watching it ! Don't just peek...watch it 'till the end :))))))))) !!\")\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<fastai.core.Category at 0x7f869b996908>, tensor(0), tensor([0.9640, 0.0360]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "metadata": {
        "id": "AgnAd-1xS4mc",
        "colab_type": "code",
        "outputId": "4b15dd60-ceb0-428c-fd40-cfe7c80cc8d7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "learn.predict(\"A very accurate depiction of small time mob life filmed in New Jersey. The story, characters and script are believable but the acting drops the ball. Still, it's worth watching, especially for the strong images, some still with me even though I first viewed this 25 years ago.<br /><br />A young hood steps up and starts doing bigger things (tries to) but these things keep going wrong, leading the local boss to suspect that his end is being skimmed off, not a good place to be if you enjoy your health, or life.<br /><br />This is the film that introduced Joe Pesce to Martin Scorsese. Also present is that perennial screen wise guy, Frank Vincent. Strong on characterizations and visuals. Sound muddled and much of the acting is amateurish, but a great story.\")\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<fastai.core.Category at 0x7f869b9932b0>, tensor(1), tensor([0.0347, 0.9653]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    }
  ]
}