nn-spanish.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Espanish ULMFiT from scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bs=48\n",
    "# bs=24\n",
    "bs=128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# torch.cuda.set_device(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/datafast/javiber/.fastai/data')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = Config.data_path(); data_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a `viwiki` folder, containing a `viwiki` text file with the wikipedia contents. (For other languages, replace `vi` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang = 'es'\n",
    "# lang = 'zh'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/datafast/javiber/.fastai/data/eswiki')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "lm_fns = [f'{lang}_wt', f'{lang}_wt_vocab']\n",
    "path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Espanish wikipedia model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nlputils import split_wiki,get_wiki"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/datafast/javiber/.fastai/data/eswiki/eswiki already exists; not downloading\n"
     ]
    }
   ],
   "source": [
    "get_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/datafast/javiber/.fastai/data/eswiki/log'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/eswiki'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/models'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/eswiki-latest-pages-articles.xml'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/docs'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/wikiextractor'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/eswiki-latest-pages-articles.xml.bz2')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"7\" url=\"https://es.wikipedia.org/wiki?curid=7\" title=\"Andorra\">\r\n",
      "Andorra\r\n",
      "\r\n",
      "Andorra, oficialmente Principado de Andorra (), es un pequeño país soberano del suroeste de Europa. Constituido en Estado independiente, de derecho, democrático y social, cuya forma de gobierno es el coprincipado parlamentario. Su territorio está organizado en siete parroquias, con una población total de 78 282 habitantes. Su capital es Andorra la Vieja.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path}/{name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function splits the single wikipedia file into a separate file per article. This is often easier to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/datafast/javiber/.fastai/data/eswiki/docs already exists; not splitting\n"
     ]
    }
   ],
   "source": [
    "dest = split_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/datafast/javiber/.fastai/data/eswiki/docs/Ion Luca Caragiale.txt'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/docs/Juan José Montes de Oca (médico).txt'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/docs/Pedro Camilo Franco.txt'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/docs/Distrito de Boquete.txt'),\n",
       " PosixPath('/datafast/javiber/.fastai/data/eswiki/docs/El Carbayón.txt')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dest.ls()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dest = Path('/datafast/javiber/.fastai/data/eswiki/docs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this to convert Chinese traditional to simplified characters\n",
    "# ls *.txt | parallel -I% opencc -i % -o ../zhsdocs/% -c t2s.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = (TextList.from_folder(dest)\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "            .label_for_lm()           \n",
    "            .databunch(bs=bs, num_workers=8))\n",
    "\n",
    "data.save(f'{lang}_databunch')\n",
    "len(data.vocab.itos),len(data.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path/'docs', f'{lang}_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 40530)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample = (TextList.from_folder(dest)\n",
    "            .filter_by_rand(0.1, seed=42)\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "            .label_for_lm()           \n",
    "            .databunch(bs=bs, num_workers=16))\n",
    "\n",
    "sample.save(f'{lang}_databunch_sample')\n",
    "len(sample.vocab.itos),len(sample.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(sample, AWD_LSTM, drop_mult=1, pretrained=False).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-2\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.878435</td>\n",
       "      <td>3.773497</td>\n",
       "      <td>0.321659</td>\n",
       "      <td>15:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.976288</td>\n",
       "      <td>3.852319</td>\n",
       "      <td>0.314584</td>\n",
       "      <td>15:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.996570</td>\n",
       "      <td>3.891692</td>\n",
       "      <td>0.310930</td>\n",
       "      <td>15:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.972999</td>\n",
       "      <td>3.841241</td>\n",
       "      <td>0.315664</td>\n",
       "      <td>15:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.921818</td>\n",
       "      <td>3.773446</td>\n",
       "      <td>0.319703</td>\n",
       "      <td>15:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.815292</td>\n",
       "      <td>3.674266</td>\n",
       "      <td>0.328842</td>\n",
       "      <td>15:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.726619</td>\n",
       "      <td>3.568360</td>\n",
       "      <td>0.337504</td>\n",
       "      <td>15:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.617370</td>\n",
       "      <td>3.455657</td>\n",
       "      <td>0.348079</td>\n",
       "      <td>15:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.547157</td>\n",
       "      <td>3.365438</td>\n",
       "      <td>0.356938</td>\n",
       "      <td>15:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.501343</td>\n",
       "      <td>3.336197</td>\n",
       "      <td>0.360241</td>\n",
       "      <td>15:42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(10, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(f'{mdl_path/lm_fns[0]}_sample', with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/f'{lm_fns[1]}_sample.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Full data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# learn = language_model_learner(data, AWD_LSTM, drop_mult=0.5, pretrained=False).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "\n",
    "learn = language_model_learner(data, AWD_LSTM, drop_mult=1, pretrained=False)\n",
    "learn.load(f'{mdl_path/lm_fns[0]}_sample');\n",
    "learn = learn.to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU5bn/8c+VfU+AhMgqi4CisqOACu6ttlVptVpt3dqi1mq3s7Snv985nqOtx1p/rXVHqtbWaivaVq1raxUBBUE2BWQJW5AlIYTs61y/P2aAmAYIkNky3/frNS9mnrmf57lmmMw19/Lct7k7IiKSuJKiHYCIiESXEoGISIJTIhARSXBKBCIiCU6JQEQkwaVEO4DDVVhY6IMGDYp2GCIicWXx4sXl7l7U0XNxlwgGDRrEokWLoh2GiEhcMbNNB3pOTUMiIglOiUBEJMEpEYiIJDglAhGRBKdEICKS4JQIREQSnBKBiEiCUyIQEYkDv/zbGt5ZWxaWYysRiIjEOHfnV39fy4KSirAcX4lARCTG1Ta1EnDIywzPZBBKBCIiMa6qvhmAvIzUsBxfiUBEJMZVNYQSQaYSgYhIQqqqbwFUIxARSVj7mobURyAikpj2NQ2pRiAikpj21wiUCEREElJVQ7CPIDdDTUMiIgmpqr6ZrLRkUpPD85WtRCAiEuOqGprD1j8ASgQiIjGvqr4lbCOGQIlARCTmqUYgIpLgqhqawzZiCJQIRERiXlV9C3lhGjEESgQiIjFPNQIRkQTm7lTVq49ARCRhhXstAlAiEBGJaeFeiwCUCEREYtreCedylQhERBLTvrUI4rFpyMxGmNnSNrcqM/tuuzJmZr8ys3VmttzMxoUrHhGReBSJpqGwpRh3/xgYA2BmycBW4E/til0ADAvdTgUeCv0rIiKEf5lKiFzT0DnAenff1G77xcCTHvQeUGBmfSIUk4hIzNtfI4jDpqF2rgCe7mB7P2BLm8eloW2fYmYzzGyRmS0qKysLU4giIrFn/1oEcVwjMLM04CLg2SM9hrvPdPcJ7j6hqKio64ITEYlxVfXNZKYmk5YSvq/rSNQILgA+cPcdHTy3FRjQ5nH/0DYREWHv9BLhaxaCyCSCr9BxsxDAC8DVodFDk4A97r4tAjGJiMSF4IRz4WsWgjCOGgIws2zgPOCGNttuBHD3h4GXgQuBdUAdcF044xERiTfhnnAOwpwI3L0W6NVu28Nt7jtwczhjEBGJZ1UNzRTlpIf1HLqyWEQkhgWXqQxvjUCJQEQkhlWHeZlKUCIQEYlZ7k5VQ3gXrgclAhGRmFXX1EprwFUjEBFJVJGYZwiUCEREYta+KahVIxARSUz7awTqIxARSUiRWIsAlAhERGKW+ghERBLc/j4CNQ2JiCSkvU1D4VyLAJQIRERiVlVD+NciACUCEZGYFZxnKLzNQqBEICISs6oiMM8QKBGIiMSsSKxFAEoEIiIxK7g6mZqGREQSlmoEIiIJrqpefQQiIgkrUmsRgBKBiEhMitRaBKBEICISkyI1zxAoEYiIxKRIrUUASgQiIjEpUmsRgBKBiEhMitRaBKBEICISk9RHICKS4CK1FgEoEYiIxKRIrUUASgQiIjEpUmsRgBKBiEhMqqxrJj8C/QOgRCAiEpPKahopyk2PyLnCmgjMrMDMZpvZajNbZWaT2z2fb2YvmtkyM/vIzK4LZzwiIvFiZ1UjvbtDIgDuBV519+OB0cCqds/fDKx099HAmcA9ZpYW5phERGJeJGsEYRuXZGb5wFTgWgB3bwKa2hVzINfMDMgBKoCWcMUkIhIPWgPOrm7SNDQYKAMeN7MlZjbLzLLblbkfOAH4BFgBfMfdA+0PZGYzzGyRmS0qKysLY8giItG3q7aRgNMtEkEKMA54yN3HArXAD9uV+QywFOgLjAHuN7O89gdy95nuPsHdJxQVFYUxZBGR6CurbgToFn0EpUCpuy8IPZ5NMDG0dR3wvAetAzYAx4cxJhGRmLc3EcR9jcDdtwNbzGxEaNM5wMp2xTaHtmNmxcAIoCRcMYmIxIOdexNBTkZEzhfuSSxuAZ4KjQQqAa4zsxsB3P1h4HbgCTNbARjw7+5eHuaYRERiWqRrBGFNBO6+FJjQbvPDbZ7/BDg/nDGIiMSbsupGctNTyExLjsj5dGWxiEiMKauO3NBRUCIQEYk5SgQiIgkuklcVgxKBiEjM2VnVoEQgIpKoahtbqG1qpXduZIaOghKBiEhMKa+J7NBRUCIQEYkpOyN8DQEoEYiIxJR9F5PlKBGIiCSkfRPO5SkRiIgkpJ3VDSQnGT2yIrdGlxKBiEgMKatupFd2GslJFrFzKhGIiMSQsurGiDYLgRKBiEhM2VndGNGOYlAiEBGJKZGeZwiUCEREYkZrwNlV2xTRq4pBiUBEJGbsrmuiNeCqEYiIJKqdVZG/qhiUCEREYkZZaJ6h3koEIiKJKdJrFe+lRCAiEiN2VjcAUKjhoyIiiamsupHstGSy01Miel4lAhGRGBG8qjiyQ0ehk4nAzIaaWXro/plmdquZFYQ3NBGRxBKNq4qh8zWC54BWMzsOmAkMAH4ftqhERBJQeRSuKobOJ4KAu7cA04H73P1fgT7hC0tEJPFEY3oJ6HwiaDazrwDXAC+FtqWGJyQRkcRT39RKdWNLTCeC64DJwE/cfYOZDQZ+G76wREQSS7SuIQDo1Bgld18J3ApgZj2AXHe/K5yBiYgkkm176gEojuFRQ2+ZWZ6Z9QQ+AB41s/8X3tBERBLHhvJaAAb3yo74uTvbNJTv7lXAF4En3f1U4NxD7WRmBWY228xWm9kqM5vcQZkzzWypmX1kZm8fXvgiIt1DSXktaSlJ9OuRGfFzd/bytRQz6wN8GfjxYRz/XuBVd7/UzNKArLZPhq5FeBD4rLtvNrPeh3FsEZFuo6SshkG9siK6VvFena0R/A/wGrDe3d83syHA2oPtYGb5wFTg1wDu3uTule2KXQk87+6bQ2V2Hk7wIiLdRUlZLUMKc6Jy7k4lAnd/1t1HuftNoccl7v6lQ+w2GCgDHjezJWY2y8zaN34NB3qE+iAWm9nVHR3IzGaY2SIzW1RWVtaZkEVE4kZza4DNFXUMKYp8/wB0vrO4v5n9ycx2hm7PmVn/Q+yWAowDHnL3sUAt8MMOyowHPgd8Bvi/Zja8/YHcfaa7T3D3CUVFRZ0JWUQkbmyuqKMl4AwpiuEaAfA48ALQN3R7MbTtYEqBUndfEHo8m2BiaF/mNXevdfdyYA4wupMxiYh0CyVlwRFDMV0jAIrc/XF3bwndngAO+tPc3bcDW8xsRGjTOcDKdsX+ApxuZilmlgWcCqzqfPgiIvGvpKwGgKFR6iPo7KihXWb2VeDp0OOvALs6sd8twFOhEUMlwHVmdiOAuz/s7qvM7FVgORAAZrn7h4f1CkRE4lxJWS29stPIz4rOzD2dTQTXA/cBvwAcmA9ce6id3H0pMKHd5ofblbkbuLuTcYiIdDsl5TVRaxaCzo8a2uTuF7l7kbv3dvdLgEONGhIRkU6I5tBROLoVyr7fZVGIiCSoPXXN7Kptiv0awQFE/vI3EZFuZn15qKM4SkNH4egSgXdZFCIiCSraQ0fhEJ3FZlZNx1/4BkR+ZiQRkW6mpKyGlCRjQM+sQxcOk4MmAnfPjVQgIiKJqKSsloG9skhNPpoGmqMTvTOLiEhw6GgURwyBEoGISNS0BpyNu+oYGsX+Aej8BWWSQAIBZ31ZDSu3VVHX1EpTS4DGllaSzMjLTKUgM5WCrDSGF+dQkJUW7XBF4tbW3fU0tQSi2lEMSgQJIRBw6ppbqWlooaaxhfSUJDLTkslOS6ElEGBDeS3ry2pYv7OW5Vv3sHTzbqoaWg55XDMY2SePyUN6MWpAAY3NrVQ1tFBV30xjSwAPjTMwjKLcdPoVZNC3IJOe2Wm0Bpzm1gDNrU5tYwtVDc3sqW+mrqmVfgWZDCnMoV+PzKgs0iESKXuHjkZr1tG9lAhiSE1jC6u2VfHR1j2s3FYFQJ/8TPoVZFKUm05VQzNl1Y2U1TQSCDjH5GfSNz+DY/IzaG51Kmob2VXbRFl1I5sr6ti8q45NFXWU1zTinRjsm5xkDOudw+dG9WXcwAJG9S8gLzOF9JRk0lKSaA04VfXBL+xdtU0s3VzJuyXlPPnuJprmbvjUsdKSk8CCw8sC7jS3Hv5o47TkJEb2zWP62H5cNLovPbJV+5DuZd/Q0ULVCCJi065a7np1NTOmDmXMgIKInbexpZWK2iZ21TSxq7aJitpGKmqbQ/82sbOqke1VDeyoaqS8pnHffr2y00hJNnZW//OXeGqykWRGY0vggOftk5/BwJ5ZnDWiiOK8DHIzUshJTyU7PZmmlgD1za3UNbUCMLgwm6FF2QzsmU1aysG7jfIzUxkQuj9teBHfYRgNza1sKK8lJz2FvIxUcjJSPvVL3t3ZU9/M1sp6tu6uZ3ddEylJSaSmJJGaZGSlp5CfmUp+ZioZqUmU7q5nQ1kt68treGdNOf/1wkfc8deVnH18b6YOL2J0/wJGHJMb1VEWIl2hpKyG/MxUekb5R07CJIKPt1fzztpyXl6xnVMH9+SGaUM4c3hvkrq46aGpJcCSzbuZs7aMOWvK+fCTPR3+Gk9OMnpkpVKUm8Exeemc3C+ffgWZjOybx0n98umdm46Z0dQSYEdVAzurG8nPTKEwJ538zOAMhbvrmvmksp4dVQ2kpSTRMzuNXtnp9MhOJT0luUtf18FkpCZzQp+8Az5vZhRkpVGQlcaJffMPebw++ZlMHNQTgB9dAKu2VfHc4lL+suwTXvtoBwBpKUmc0CePoUXZDCnMZnBhDoMLsxlUmEVWWsJ8rCXOlZTVMqQoG7PoNoGad6bNIIZMmDDBFy1adET71jS28MzCzfx67ga27WmgX0Emk4f2YvKQXkwa2otj8jI63Sbt7iwv3cPzH5SydEsllfXNVNY1U9XQjHvwi37cwAImD+lFn1C7eGFOGj2ygl/WuRkpXZ6Eujt3Z0tFPctKK1leWslHn1SxobyWbXsaPlWuT35GqJaTw7DiHI4rymFYcS5FuelRilzkn7k74+/4G+ee0JufXRr+9bjMbLG7t58NGkigGgFATnoK3zhjCNdMGcRfl2/j1Q+387dVO5i9uHRfmYzUJLLTUijKTWfa8CLOHVnMuIE9SE4yKmqbWLWtisWbdvPnpVspKaslLSWJUwb1ZFBhNgWh5o0T++UzeWgv8jKiM7d4d2VmDOyVxcBeWXxhdN992+ubgk1TG3fV7uv4Limr5c9Lt1LdptO7MCedkX3zOKFPLiOKcxlSlMOQomz9P0lUlO6up6K2iZP7R66p+kASKhHslZqcxCVj+3HJ2H4EAs7q7dUs2lRBRW0TdU2t1Da2sGlXHY/N28Ajc0romZ1GarKxo2p/G/4pg3sy44whXHByn31NNRIdmWnJjOybx8i+n26ecnd2VjeybmcNq7dXs2pbFau2VfH43F00te7vXzkmL4PPnFjM9HH9Gd0/P+rVdEkMK7buAWB0/0M3l4ZbQiaCtpKSrMMvEYDqhmbmrCnn76t34B4cKnlCn+Avyl45amaIdWZGcV4GxXkZnHZc4b7tza0BNlfUsX5nDevLalm2pZKn39/Cb97dxJCibC4e3Y/zRhZzQp9cJQUJm2WllaQlJzHimOjP5JNQfQQiB7KnvplXVmzj+SVbeX9jBe7QryCT80YWM2ZAAUNDzUjZ6Qn/20m6yFdmvkdtUwsvfPv0iJxPfQQih5CfmcoVpwzkilMGUlbdyJurd/DGyh08vXAzT8zfuK9ccV46A3pkMaBnFgN6ZDL8mFxOGdyT3rkZ0Qte4k4g4Hy4dQ8Xj+176MIRoEQg0k5RbjqXTxzI5RMH0tQSYNOuWtaXBTuhN5TXsqWijoUbKvjL0noCoQr1kMJsTh3SkylDCzntuMKojwuX2FZSXkt1YwujYqCjGJQIRA4qLSWJYcW5DCv+53bcppYAK7dVsaBkFws2VPDSsm08vXALZnBS33zOPr43V00aqNqC/JMVWysBGK1EIBLf0lKSGDOggDEDCrhh2lBaWgMs37qHuWvLeWdtGb96cy0PvbWeL47rxzfOGMJxvaM7n4zEjmVb9pCZmhz1WUf3UiIQ6SIpyUmMG9iDcQN7cOs5w9hYXsusuSU8u6iUZ97fwnkji7n5rOMiOsWJxKblpZWc1C+PlBiZJiU2ohDphgYVZnPHJScz/4dnc+s5w1i4oYJLHpjHVbPeY966cuJtxJ50jebWAB99UhUz/QOgRCASdr1y0vn+ecOZ98Oz+Y8Lj2fNjhqumrWAi+6fxwvLPqGl9cCTB0r3s3ZHDY0tAUbFwIVkeykRiERITnoKM6YO5Z1/O4ufTj+Z2qYWbn16CdPufovH5m6gPjQbrHRvy0uDHcWqEYgksIzUZK48dSB/+940Hr16An0LMvifl1Zy+l1v8tBb66luaI52iBJGy0r3kJeRwqBeWdEOZR8lApEoSUoyzhtZzLM3TuGPN0zmxH753PXqak6/6x/c/+ZaahsPvUqcxJ/lpZWM6l8QU9OXKBGIxIBTBvfkyetP4S83n8bEQT34+etrmHb3P3hi3gYaW9Rk1F00NLfy8fbqmOofACUCkZgyekABs66ZyHM3TWFoUQ63vbiSc+55m+cWl9Ia0CijeLdqWxUtAU+sRGBmBWY228xWm9kqM5t8gHITzazFzC4NZzwi8WL8sT14ZsYkfnP9KRRkpfKDZ5dxwb1zeGPlDg07jWPLtsReRzGEv0ZwL/Cqux8PjAZWtS9gZsnAXcDrYY5FJK6YGdOGF/HCzafzwJXjaG51vvnkIq5+bCG7a5uiHZ4cgbnryhnQM5M++bE17UjYEoGZ5QNTgV8DuHuTu1d2UPQW4DlgZ7hiEYlnSUnG50b14fXvTeW2L4xkQUkFFz0wl5WfVEU7NDkMjS2tzF+/izOH946pjmIIb41gMFAGPG5mS8xslpl9amINM+sHTAceOtiBzGyGmS0ys0VlZWXhi1gkhqUmJ3HtaYP5442TaW5xvvhQ8II0iQ+LNu6mrqmVacOLoh3KPwlnIkgBxgEPuftYoBb4YbsyvwT+3d0Pemmlu8909wnuPqGoKPbeRJFIGjOggBduOY2T++Vz69NL+NHzK6jRUNOY9/aaMtKSk5g8tFe0Q/kn4UwEpUCpuy8IPZ5NMDG0NQF4xsw2ApcCD5rZJWGMSaRb6J2bwVPfmMQN04bwzPub+ewv5zB/fXm0w5KDeOvjnUwc3CMmV7kLWyJw9+3AFjMbEdp0DrCyXZnB7j7I3QcRTBTfcvc/hysmke4kLSWJH11wArNvnExKknHlowv47xc/oqlFcxfFmk8q61mzo4Yzh/eOdigdCveooVuAp8xsOTAG+KmZ3WhmN4b5vCIJY/yxPXnlO1O5dsogHp+3katmvUdZdWO0w5I23l4T7NucNiI2m7bDWkdx96UEm3/aevgAZa8NZywi3VlmWjK3XXQi447twb/NXsYX7pvLI18bz2itfRAT3v64jL75GQyL0cWJdGWxSDdy0ei+PHfTFJKTjMseeZfnPyiNdkgJr7k1wLx15UwbURRzw0b3UiIQ6WZO7JvPi7eczriBBXz/j8u48+VVmp4iij7YtJvqxhamxWj/ACgRiHRLPbPT+O3XT+WrkwbyyJwSvvnkIk1vHSVvrSkjJck47bjYGza6lxKBSDeVmpzEHZeczO2XnMTba8r44oPz+Xh7dbTDSjhvf1zG+GN7kJuRGu1QDkiJQKSb+9qkY/nt109hd10TX7h/LrPeKSGgpqKI2FHVwMptVTE7WmgvJQKRBDBlaCGvfncqU4cVccdfV3HVrAVsrayPdljd3hsrdwBw3gnFUY7k4JQIRBJEYU46j149nru+dDLLSiu58N53mLNGc3eF0+srdzC4MJvjYnTY6F5KBCIJxMy4fOJAXr71DPrkZ3DN4wt54B/rtMZBGFQ1NPPu+nLOG1kcs8NG91IiEElAgwqzef5bU/j8qL7c/drH3Pi7xZq4rou99XEZza3O+SNju1kIlAhEElZWWgq/umIM/+dzJ/C3VTuZ/sA8NpbXRjusbuP1j7ZTmJPG2IE9oh3KISkRiCQwM+MbZwzht9efQllNIxfdP3ffvDhy5BpbWnnr4zLOPaGY5KTYbhYCJQIRAaYcV8iL3z6dvgWZXPf4Qh55e736DY7CeyUV1DS2cF4cNAuBEoGIhAzomcXz35rCBSf14c5XVvPNJxdTWae1kY/E6x9tJystmdOOK4x2KJ2iRCAi+2SlpXD/lWP5z8+P5O01O7nw3ndYtLEi2mHFlUDAeWPlDqYNLyIjNTna4XSKEoGIfIqZcf3pg3nupimkJCdx+cz3mDlHTUWdtay0kp3VjZx/Ynw0C4ESgYgcwKj+Bbx06+l85sRifvryan7w7DIaW1qjHVbMe33lDpKTjLNHKBGISDeQl5HKA1eO47vnDuP5D7Zy1aMLKK/R6mcH84/VO5k4qAf5WbE7yVx7SgQiclBmxnfPHc4DV47jw0/2cPH98zSL6QFs39PA6u3VMb32QEeUCESkUz43qg9/vGEyza0BLn1ovuYp6sCctaG1iYfH9myj7SkRiEinjepfwJ9vPo1+PTK57on3eXrh5miHFFPmrCmjKDedE/rkRjuUw6JEICKHpW9BJrNvmsLUYYX86PkV3PnyKq1vALQGnLnryjljWGHMTzLXnhKBiBy2nPQUHr16AldPPpZH5pTwrac+oL4psUcUrdi6h8q65rhrFgIlAhE5QinJSfz3RSfyn58fyWsrt3P5zHfZWdUQ7bCiZs6aMszg9Di5mrgtJQIROWJ7Lz6b+bUJrN1RwyUPzOPDrXuiHVZUzFlTxkl98+mVkx7tUA6bEoGIHLXzRhbz7I2TCThMf3AeD721ntYE6jeoamhmyZZKpg6Pv9oAKBGISBc5qV8+L3/nDM45vpi7Xl3NV2a+x5aKumiHFRHz15XTGnCmDou//gFQIhCRLtQzO42HvjqOn182mpXbqrjw3nd47aPt0Q4r7N5eU05Oegrjjo39RWg6okQgIl3KzLh0fH9e+c4ZDCnK5obfLuae1z/utkNM3Z05a8qYPLQXqcnx+ZUan1GLSMwb0DOLP9wwmcvG9+e+N9fx9d+8z5765miH1eVKymvZWlnP1DgcNrqXEoGIhE1GajI/u3QUt19yEu+sLefi++d2u3mKfvvuJpIMzhqhRNAhMysws9lmttrMVpnZ5HbPX2Vmy81shZnNN7PR4YxHRCLPzPjapGN5esYkaptamf7gPF5a/km0w+oSm3bV8tSCTVw+cSD9e2RFO5wjFu4awb3Aq+5+PDAaWNXu+Q3ANHc/GbgdmBnmeEQkSiYO6slLt5zO8cfk8u3fL+HOl1fR0hqIdlhH5e7XPiYlKYnvnTss2qEclbAlAjPLB6YCvwZw9yZ3r2xbxt3nu/vu0MP3gP7hikdEoq84L4NnZkzmq5MG8sicEr755CJqGluiHdYRWbqlkpeWb+ObZwymd15GtMM5KuGsEQwGyoDHzWyJmc0ys+yDlP868EpHT5jZDDNbZGaLyso09a1IPEtLSeKOS07mJ9NPYs7acr788Lts3xNfU1O4O3e+vIpe2WnMmDY02uEctXAmghRgHPCQu48FaoEfdlTQzM4imAj+vaPn3X2mu09w9wlFRfHbISMi+1116rH8+poJbNpVy/QH57FqW1W0Q+q0N1fvZMGGCr577jBy0lOiHc5RC2ciKAVK3X1B6PFsgonhU8xsFDALuNjdd4UxHhGJMWeO6M2zN07BHS57+F0Wb9p96J2ibP76cm578SMGF2ZzxSkDox1OlwhbInD37cAWMxsR2nQOsLJtGTMbCDwPfM3d14QrFhGJXSP75vGnm6dQmJPGtY8tZOmWykPvFAUrP6nimscWcuWjC2hpde760qi4vYCsPXMP39V+ZjaG4K/9NKAEuA64HMDdHzazWcCXgE2hXVrcfcLBjjlhwgRftGhR2GIWkejYtqeeK2a+R0VtE7/7+qmMHlAQ7ZCAYH/AL95Yw33/WEdeRio3nzWUqycPIiM1OdqhHRYzW3yg79ewJoJwUCIQ6b4+qazn8pnvUlnXzFPfOJVR/aOfDO77+1rueWMNXxzXj//6/InkZ6VGO6QjcrBE0D3qNSLSLfQtyOTpb04iPzOVq2YtYMnm6PYZzHqnJJgExvbj55eOjtskcChKBCISU/r3yOKZGZPomZ3GV2ctYOGGiqjE8fsFm7njr6u48ORj+Nmlo0hKiq91iA+HEoGIxJz+PbL4w4zJFOdncM1jC5m/rjyi5//13A38+M8rOPv43vzy8rGkdJNO4QPp3q9OROLWMfkZ/GHGZAb2zOK6J96PyLoGrQHnthc+4vaXVvLZE4/hwavGkZbS/b8mu/8rFJG4VZSbztMzJnF8nzxu/N1iHnl7PeEa4FLf1MpNv1vME/M38o3TB/PAlePibmTQkVIiEJGY1jM7jT/MmMSFJ/fhzldW88PnVtDU0rWT1bUGnOueWMgbq3Zw2xdG8n8+P7Jb9wm0F//XRotIt5eRmsx9V4xlSGE29725ji2763joq+PJz+yaUTyz3inhvZIKfvalUXx54oAuOWY8UY1AROJCUpLxg/NHcM9lo1m4oYIvP/wu2/bUH/Vx1+yo5p7X1/CZE4u5bEJiToCsRCAiceVL4/vzxHWnsLWyni8+OP+oVjxrbg3w/T8uJScjhZ9MPxmzxGkOakuJQETizunDCvnDDZNoDTiXPjyfV1ZsO6JO5Af+sY4Pt1bx0+knUZiTHoZI44MSgYjEpRP75vOnm0+jX0EmNz31AZc+/C6LNnbu4rPWgPOXpVu5/811XDKmL589qU+Yo41tmmtIROJaS2uAPy4q5Zd/W8PO6kbOGlHE2ScUM3ZAAccfk/upi8Hqm1qZvXgLs+ZuYNOuOoYX5/DsDVO67dQRbWnSORHp9uqaWnh83kYen7eR8ppGADJTkynOS6ehOUBDSyt1ja00tQYYM6CAG6YO4fwTjyE5QYaJKhGISMJwd0p317NkSyVLNu9mV00TGalJpKckk5WWzDknFDNxUI+E6xg+WECWx1wAAAgjSURBVCLQdQQi0q2YGQN6ZjGgZxYXje4b7XDigjqLRUQSnBKBiEiCUyIQEUlwSgQiIglOiUBEJMEpEYiIJDglAhGRBKdEICKS4OLuymIzKwMqgT3tnso/xLZD3d/7byFwJCtld3T+zjzffvvBHrePte22I4k7kjG3vR+N91qfD30+DvZ8PH4+DidmgGHunt/h0d097m7AzMPddqj7bf5d1FUxdeb59tsP9rh9rEcbdyRjjvZ7rc+HPh/d7fNxODEf6hzx2jT04hFsO9T9jvY/2pg683z77Qd73FGsRxN3JGNuez8a77U+H4dPn4/O34/1mA96jrhrGgo3M1vkB5iYKZbFY9yKOXLiMW7FHDnxWiMIp5nRDuAIxWPcijly4jFuxRwhqhGIiCQ41QhERBKcEoGISILr1onAzB4zs51m9uER7DvezFaY2Toz+5W1Wc7IzG4xs9Vm9pGZ/axrow5P3GZ2m5ltNbOloduFsR5zm+d/YGZuZoVdF3HY3ufbzWx56D1+3cy6dGWUMMV8d+jzvNzM/mRmBV0Zcxjjviz0Nxgwsy7roD2aWA9wvGvMbG3odk2b7Qf93EfUkYx5jZcbMBUYB3x4BPsuBCYBBrwCXBDafhbwNyA99Lh3nMR9G/Av8fReh54bALwGbAIKYz1mIK9NmVuBh+Mg5vOBlND9u4C74uHzAZwAjADeAiZEO9ZQHIPabesJlIT+7RG63+Ngrysat25dI3D3OUBF221mNtTMXjWzxWb2jpkd334/M+tD8A/6PQ/+jz0JXBJ6+ibgf929MXSOnXESd1iFMeZfAP8GdPmohnDE7O5VbYpmd3XcYYr5dXdvCRV9D+jflTGHMe5V7v5xrMR6AJ8B3nD3CnffDbwBfDaaf6sd6daJ4ABmAre4+3jgX4AHOyjTDyht87g0tA1gOHCGmS0ws7fNbGJYo93vaOMG+Hao+v+YmfUIX6j7HFXMZnYxsNXdl4U70DaO+n02s5+Y2RbgKuA/wxjrXl3x2djreoK/TiOhK+MOt87E2pF+wJY2j/fGHyuvC0iwxevNLAeYAjzbpjku/TAPk0KwmjcJmAj80cyGhLJ6WHRR3A8BtxP8hXo7cA/BP/qwONqYzSwL+A+CzRYR0UXvM+7+Y+DHZvYj4NvAf3VZkO10VcyhY/0YaAGe6proDnquLos73A4Wq5ldB3wntO044GUzawI2uPv0SMd6pBIqERCsAVW6+5i2G80sGVgcevgCwS/NttXj/sDW0P1S4PnQF/9CMwsQnGiqLJbjdvcdbfZ7FHgpjPHC0cc8FBgMLAv98fUHPjCzU9x9e4zG3N5TwMuEMRHQRTGb2bXA54Fzwvmjpo2ufq/DqcNYAdz9ceBxADN7C7jW3Te2KbIVOLPN4/4E+xK2Ev3XtV+0OicidQMG0abTB5gPXBa6b8DoA+zXviPnwtD2G4H/Cd0fTrDaZ3EQd582Zb4HPBPrMbcrs5Eu7iwO0/s8rE2ZW4DZcRDzZ4GVQFFXxxqJzwdd3Fl8pLFy4M7iDQQ7inuE7vfs7Oc+UreonDRiLw6eBrYBzQR/yX+d4K/MV4FloQ//fx5g3wnAh8B64H72X4WdBvwu9NwHwNlxEvdvgRXAcoK/tPrEesztymyk60cNheN9fi60fTnBSb76xUHM6wj+oFkaunXpSKcwxj09dKxGYAfwWjRjpYNEENp+feg9Xgdcdzif+0jdNMWEiEiCS8RRQyIi0oYSgYhIglMiEBFJcEoEIiIJTolARCTBKRFIt2BmNRE+3ywzG9lFx2q14GylH5rZi4ea/dPMCszsW11xbhHQCmXSTZhZjbvndOHxUnz/RGxh1TZ2M/sNsMbdf3KQ8oOAl9z9pEjEJ92fagTSbZlZkZk9Z2bvh26nhbafYmbvmtkSM5tvZiNC2681sxfM7E3g72Z2ppm9ZWazLThf/1N754wPbZ8Qul8TmmhumZm9Z2bFoe1DQ49XmNkdnay1vMv+SfdyzOzvZvZB6BgXh8r8LzA0VIu4O1T2X0OvcbmZ/XcXvo2SAJQIpDu7F/iFu08EvgTMCm1fDZzh7mMJzg760zb7jAMudfdpocdjge8CI4EhwGkdnCcbeM/dRwNzgG+2Of+97n4yn55pskOheXbOIXjlN0ADMN3dxxFcB+OeUCL6IbDe3ce4+7+a2fnAMOAUYAww3symHup8Insl2qRzkljOBUa2mTEyLzSTZD7wGzMbRnA21tQ2+7zh7m3nol/o7qUAZraU4Bw0c9udp4n9k/gtBs4L3Z/M/jnmfw/8/ABxZoaO3Q9YRXDOegjOQfPT0Jd6IPR8cQf7nx+6LQk9ziGYGOYc4Hwin6JEIN1ZEjDJ3RvabjSz+4F/uPv0UHv7W22erm13jMY291vp+G+m2fd3th2ozMHUu/uY0NTbrwE3A78iuJ5BETDe3ZvNbCOQ0cH+Btzp7o8c5nlFADUNSff2OsEZQAEws73TCOezf8rfa8N4/vcINkkBXHGowu5eR3B5yx+YWQrBOHeGksBZwLGhotVAbptdXwOuD9V2MLN+Zta7i16DJAAlAukussystM3t+wS/VCeEOlBXEpxCHOBnwJ1mtoTw1oq/C3zfzJYTXLRkz6F2cPclBGcu/QrB9QwmmNkK4GqCfRu4+y5gXmi46d3u/jrBpqd3Q2Vn8+lEIXJQGj4qEiahpp56d3czuwL4irtffKj9RCJNfQQi4TMeuD800qeSMC4NKnI0VCMQEUlw6iMQEUlwSgQiIglOiUBEJMEpEYiIJDglAhGRBPf/AeBbL0jPPTgMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-2\n",
    "# lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='9' class='' max='10', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      90.00% [9/10 23:24:40<2:36:04]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.492150</td>\n",
       "      <td>3.365658</td>\n",
       "      <td>0.357493</td>\n",
       "      <td>2:35:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.562045</td>\n",
       "      <td>3.456946</td>\n",
       "      <td>0.348355</td>\n",
       "      <td>2:35:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.605615</td>\n",
       "      <td>3.508311</td>\n",
       "      <td>0.343915</td>\n",
       "      <td>2:35:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.586841</td>\n",
       "      <td>3.492753</td>\n",
       "      <td>0.345466</td>\n",
       "      <td>2:35:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.554735</td>\n",
       "      <td>3.454936</td>\n",
       "      <td>0.348489</td>\n",
       "      <td>2:35:56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.540326</td>\n",
       "      <td>3.403521</td>\n",
       "      <td>0.353333</td>\n",
       "      <td>2:35:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.504061</td>\n",
       "      <td>3.340822</td>\n",
       "      <td>0.359617</td>\n",
       "      <td>2:35:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.415219</td>\n",
       "      <td>3.263391</td>\n",
       "      <td>0.367790</td>\n",
       "      <td>2:36:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.369590</td>\n",
       "      <td>3.197876</td>\n",
       "      <td>0.375321</td>\n",
       "      <td>2:37:40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='54576' class='' max='58919', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      92.63% [54576/58919 2:20:10<11:09 3.3962]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n",
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from fastai.callbacks import SaveModelCallback\n",
    "# learn.unfreeze()\n",
    "learn.fit_one_cycle(\n",
    "    10, lr, moms=(0.8,0.7),  \n",
    "    callbacks=[\n",
    "    SaveModelCallback(\n",
    "            learn, every='epoch', monitor='accuracy', name='esLM')\n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the pretrained model and vocab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vietnamese sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Data](https://github.com/ngxbac/aivivn_phanloaisacthaibinhluan/tree/master/data)\n",
    "- [Competition details](https://www.aivivn.com/contests/1)\n",
    "- Top 3 f1 scores: 0.900, 0.897, 0.897"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>train_000000</td>\n",
       "      <td>Dung dc sp tot cam on \\nshop Đóng gói sản phẩm...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>train_000001</td>\n",
       "      <td>Chất lượng sản phẩm tuyệt vời . Son mịn nhưng...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>train_000002</td>\n",
       "      <td>Chất lượng sản phẩm tuyệt vời nhưng k có hộp ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>train_000003</td>\n",
       "      <td>:(( Mình hơi thất vọng 1 chút vì mình đã kỳ vọ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>train_000004</td>\n",
       "      <td>Lần trước mình mua áo gió màu hồng rất ok mà đ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             id                                            comment  label\n",
       "0  train_000000  Dung dc sp tot cam on \\nshop Đóng gói sản phẩm...      0\n",
       "1  train_000001   Chất lượng sản phẩm tuyệt vời . Son mịn nhưng...      0\n",
       "2  train_000002   Chất lượng sản phẩm tuyệt vời nhưng k có hộp ...      0\n",
       "3  train_000003  :(( Mình hơi thất vọng 1 chút vì mình đã kỳ vọ...      1\n",
       "4  train_000004  Lần trước mình mua áo gió màu hồng rất ok mà đ...      1"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = pd.read_csv(path/'train.csv')\n",
    "train_df.loc[pd.isna(train_df.comment),'comment']='NA'\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>test_000000</td>\n",
       "      <td>Chưa dùng thử nên chưa biết</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>test_000001</td>\n",
       "      <td>Không đáng tiềnVì ngay đợt sale nên mới mua n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>test_000002</td>\n",
       "      <td>Cám ơn shop. Đóng gói sản phẩm rất đẹp và chắc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>test_000003</td>\n",
       "      <td>Vải đẹp.phom oki luôn.quá ưng</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>test_000004</td>\n",
       "      <td>Chuẩn hàng đóng gói đẹp</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            id                                            comment\n",
       "0  test_000000                        Chưa dùng thử nên chưa biết\n",
       "1  test_000001   Không đáng tiềnVì ngay đợt sale nên mới mua n...\n",
       "2  test_000002  Cám ơn shop. Đóng gói sản phẩm rất đẹp và chắc...\n",
       "3  test_000003                      Vải đẹp.phom oki luôn.quá ưng\n",
       "4  test_000004                            Chuẩn hàng đóng gói đẹp"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.read_csv(path/'test.csv')\n",
    "test_df.loc[pd.isna(test_df.comment),'comment']='NA'\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([train_df,test_df], sort=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_df(df, path, cols='comment')\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, pretrained_fnames=lm_fns, drop_mult=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.975080</td>\n",
       "      <td>4.138585</td>\n",
       "      <td>0.317773</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.408635</td>\n",
       "      <td>4.025489</td>\n",
       "      <td>0.326423</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.142114</td>\n",
       "      <td>3.928278</td>\n",
       "      <td>0.336230</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.010835</td>\n",
       "      <td>3.793583</td>\n",
       "      <td>0.349972</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.873617</td>\n",
       "      <td>3.694702</td>\n",
       "      <td>0.357240</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.761377</td>\n",
       "      <td>3.632186</td>\n",
       "      <td>0.364648</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.679017</td>\n",
       "      <td>3.595601</td>\n",
       "      <td>0.366964</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.614548</td>\n",
       "      <td>3.576386</td>\n",
       "      <td>0.369224</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.575895</td>\n",
       "      <td>3.567496</td>\n",
       "      <td>0.370285</td>\n",
       "      <td>00:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.560278</td>\n",
       "      <td>3.566525</td>\n",
       "      <td>0.370173</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(8, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = (TextList.from_df(train_df, path, vocab=data_lm.vocab, cols='comment')\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols='label')\n",
    "    .databunch(bs=bs, num_workers=1))\n",
    "\n",
    "data_clas.save(f'{lang}_textlist_class')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, f'{lang}_textlist_class', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "@np_func\n",
    "def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc')\n",
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=2e-2\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>_inner</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.338150</td>\n",
       "      <td>0.275298</td>\n",
       "      <td>0.899876</td>\n",
       "      <td>0.878430</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.302302</td>\n",
       "      <td>0.245949</td>\n",
       "      <td>0.902985</td>\n",
       "      <td>0.877226</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>_inner</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.321768</td>\n",
       "      <td>0.255457</td>\n",
       "      <td>0.899254</td>\n",
       "      <td>0.871367</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.305934</td>\n",
       "      <td>0.250888</td>\n",
       "      <td>0.894901</td>\n",
       "      <td>0.872021</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>_inner</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.300939</td>\n",
       "      <td>0.261080</td>\n",
       "      <td>0.893657</td>\n",
       "      <td>0.866201</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.263790</td>\n",
       "      <td>0.220207</td>\n",
       "      <td>0.906716</td>\n",
       "      <td>0.886115</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>_inner</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.282888</td>\n",
       "      <td>0.238203</td>\n",
       "      <td>0.905473</td>\n",
       "      <td>0.886483</td>\n",
       "      <td>00:04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.248599</td>\n",
       "      <td>0.216489</td>\n",
       "      <td>0.918532</td>\n",
       "      <td>0.901550</td>\n",
       "      <td>00:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>_inner</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.201508</td>\n",
       "      <td>0.217176</td>\n",
       "      <td>0.911070</td>\n",
       "      <td>0.890084</td>\n",
       "      <td>00:05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Competition top 3 f1 scores: 0.90, 0.89, 0.89. Winner used an ensemble of 4 models: TextCNN, VDCNN, HARNN, and SARNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, f'{lang}_textlist_class', bs=bs, num_workers=1)\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load(f'{lang}clas', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9111), tensor(0.8952))"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds,targs = learn_c.get_preds(ordered=True)\n",
    "accuracy(preds,targs),f1(preds,targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_bwd = load_data(path, f'{lang}_textlist_class_bwd', bs=bs, num_workers=1, backwards=True)\n",
    "learn_c_bwd = text_classifier_learner(data_clas_bwd, AWD_LSTM, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c_bwd.load(f'{lang}clas_bwd', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9092), tensor(0.8957))"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_b,targs_b = learn_c_bwd.get_preds(ordered=True)\n",
    "accuracy(preds_b,targs_b),f1(preds_b,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_avg = (preds+preds_b)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9154), tensor(0.9016))"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(preds_avg,targs_b),f1(preds_avg,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}