notebooks/neural_nets/MXNet basics.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd, sym, mod\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from src.fetch_mnist import preprocessed_mnist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Useful links\n",
    "\n",
    "[A Practitioner’s Guide to MXNet](http://home.cse.ust.hk/~xshiab/data/MXNet.pdf)\n",
    "\n",
    "[An introduction to the MXNet API](https://becominghuman.ai/an-introduction-to-the-mxnet-api-part-1-848febdcf8ab) blog post series on Medium\n",
    "\n",
    "[MXNet the straight dope](https://github.com/zackchase/mxnet-the-straight-dope) - jupyter-notebook based deep learning textbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preparing data\n",
    "\n",
    "We'll use MNIST dataset. Examples are standardised according to training data.\n",
    "\n",
    "Note that MXNet uses its own NDArrays, so `numpy` arrays need to be converted to that. We also use `mxnet.io` NDArrayIter class that splits data into batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/semantive/Projects/notebooks/examples-counterexamples/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:475: DataConversionWarning: Data with input dtype uint8 was converted to float64 by StandardScaler.\n",
      "  warnings.warn(msg, DataConversionWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = preprocessed_mnist(random_state=0)\n",
    "X_test, X_valid, y_test, y_valid = train_test_split(X_test, y_test, test_size=0.5) \n",
    "category_count = len(np.unique(y_train))\n",
    "\n",
    "def data_iter(X, y, batch_size=500):\n",
    "  batch_size = min(X.shape[0], batch_size)\n",
    "  return mx.io.NDArrayIter(\n",
    "    data=nd.array(X),\n",
    "    label=nd.array(y),\n",
    "    batch_size=batch_size)\n",
    "\n",
    "train_iter = data_iter(X_train, y_train)\n",
    "valid_iter = data_iter(X_valid, y_valid)\n",
    "test_iter = data_iter(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic regression with softmax using `module` API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`mxnet.sym` is a module for symbols - nodes in computation graph (they correspond to tensors from Theano/Tensorflow).\n",
    "\n",
    "The module also contains utilities that can be used for defining layers.\n",
    "\n",
    "`mxnet.mod` contains classes for *Modules* - they provide high-level functionalities a'la `scikit-learn` estimators. Modules can be built with symbols, and they automatically infer weights from layers.\n",
    "\n",
    "Below is an example of simple *Module* that performs logistic regression (it uses minibatch gradient descent for optimization)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = sym.Variable('data')\n",
    "softmax_input = sym.FullyConnected(data, name='softmax_input', num_hidden=category_count)\n",
    "softmax_output = sym.SoftmaxOutput(softmax_input, name='softmax')\n",
    "\n",
    "logistic_classifier = mod.Module(symbol=softmax_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell contains code that can be passed to our Module's `fit` method. It is used to save per-batch metric (here accuracy is used)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mxnet.callback import LogValidationMetricsCallback\n",
    "\n",
    "def log_to_list(lst, period=None, data_type='Train', log_output=False):\n",
    "    def condition(nbatch):\n",
    "      if period == None:\n",
    "        return nbatch == 0\n",
    "      else:\n",
    "        return nbatch % period == 0\n",
    "  \n",
    "    def _callback(param):\n",
    "        \"\"\"The checkpoint function.\"\"\"\n",
    "        if condition(param.nbatch) and param.eval_metric is not None:\n",
    "            name_value = param.eval_metric.get_name_value()\n",
    "            for name, value in name_value:\n",
    "                message_args = (param.epoch, param.nbatch, data_type, name, value)\n",
    "                message = 'Iter[{}] Batch[{}] {}-{}={}'.format(*message_args)\n",
    "                if log_output:\n",
    "                  print(message)\n",
    "                lst.append((message, message_args))\n",
    "    return _callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_log_list = []\n",
    "eval_log_list = []\n",
    "\n",
    "num_epoch = 100 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 51.9 s, sys: 1.9 s, total: 53.8 s\n",
      "Wall time: 22.5 s\n"
     ]
    }
   ],
   "source": [
    "%time logistic_classifier.fit(\\\n",
    "  train_iter,\\\n",
    "  eval_data=valid_iter,\\\n",
    "  eval_metric='ce',\\\n",
    "  batch_end_callback=log_to_list(train_log_list),\\\n",
    "  eval_batch_end_callback=log_to_list(eval_log_list, data_type='Valid'),\\\n",
    "  num_epoch=num_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_test_pred_proba = logistic_classifier.predict(test_iter)\n",
    "y_test_pred = nd.argmax(y_test_pred_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "def evaluate_classifier(clf, round_to=3):\n",
    "  datasets = [data_iter(X_train, y_train), data_iter(X_valid, y_valid), data_iter(X_test, y_test)]\n",
    "  label_sets = [y_train, y_valid, y_test]\n",
    "  for (set_label, dataset_iter, labels) in zip(['train', 'validation', 'test'], datasets, label_sets):\n",
    "    y_pred_proba = clf.predict(dataset_iter)\n",
    "    y_pred = nd.argmax(y_pred_proba, axis=1)\n",
    "\n",
    "    accuracy = accuracy_score(labels, y_pred.asnumpy())\n",
    "\n",
    "    print(\n",
    "      'accuracy on {} set:\\n{}'.format(\n",
    "        set_label,\n",
    "        round(accuracy, round_to)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy on train set:\n",
      "0.929\n",
      "accuracy on validation set:\n",
      "0.924\n",
      "accuracy on test set:\n",
      "0.919\n"
     ]
    }
   ],
   "source": [
    "evaluate_classifier(logistic_classifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_loss_from_logs(logs):\n",
    "  return [msg[1][4] for msg in logs]\n",
    "\n",
    "train_loss = get_loss_from_logs(train_log_list)\n",
    "eval_loss = get_loss_from_logs(eval_log_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation vs training loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_train_vs_validation_loss(train_loss, eval_loss, offset=0):\n",
    "  assert len(train_loss) == len(eval_loss)\n",
    "  l = len(train_loss)\n",
    "  rng = np.arange(start=offset, stop=l)\n",
    "  \n",
    "  plt.plot(rng, train_loss[offset:], label='train')\n",
    "  plt.plot(rng, eval_loss[offset:], label='validation')\n",
    "  plt.legend()\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmQHOWd5vHvL6uqb0nduk+QbGMQOkBSA7IBGwzh5bCxsbkcZmxYY+0QnhXeGMcG44kw9q4d69nwYIwPWGzja20YVhiDJ/AwPiAwwylhkAUCJA6hlpDUulvqs6p++0dmVVd3V3W3pO4uZfXzicioPN7MelMJT7791ltZ5u6IiEhlCcpdARERGXkKdxGRCqRwFxGpQAp3EZEKpHAXEalACncRkQqkcBcRqUAKdxGRCqRwFxGpQMlyvfHUqVN9/vz55Xp7EZFYWrdu3W53nzZUubKF+/z581m7dm253l5EJJbMbMtwyqlbRkSkAincRUQqkMJdRKQCla3PXUQqS09PDy0tLXR2dpa7KhWhpqaGuXPnkkqljmp/hbuIjIiWlhYmTJjA/PnzMbNyVyfW3J09e/bQ0tLCggULjuoY6pYRkRHR2dnJlClTFOwjwMyYMmXKMf0VpHAXkRGjYB85x/pvGbtwf3VHG//876+y51BXuasiInLcil24b951iO/+aTO7D3WXuyoichzZv38/P/jBD454v0suuYT9+/ePQo3KK3bhnkyEf6r0ZLJlromIHE9KhXs6nR50v4cffpjGxsbRqlbZxG60TCoK93TWy1wTETme3Hzzzbz++uucfvrppFIpampqaGpq4pVXXuG1117j4x//OFu3bqWzs5ObbrqJVatWAb2PQjl06BAXX3wx55xzDk8++SRz5szhwQcfpLa2tsxndnRiF+7JIPxjI62Wu8hx62u/fYmXtx8c0WOeOnsit3x0Ucnt3/zmN9mwYQMvvPACjz32GJdeeikbNmzIDyW8++67mTx5Mh0dHZxxxhl88pOfZMqUKX2OsWnTJu655x5++MMfctVVV3H//fdz7bXXjuh5jJX4hXu+W0YtdxEp7cwzz+wzRvz222/ngQceAGDr1q1s2rRpQLgvWLCA008/HYAVK1bw1ltvjVl9R1rswj2ViFruWbXcRY5Xg7Wwx0p9fX1+/rHHHuMPf/gDTz31FHV1dZx33nlFx5BXV1fn5xOJBB0dHWNS19EQvw9Ug6jPXS13ESkwYcIE2traim47cOAATU1N1NXV8corr/D000+Pce3GXmxb7hotIyKFpkyZwtlnn83ixYupra1lxowZ+W0XXXQRd955JwsXLuTkk09m5cqVZazp2IhduCc1WkZESvjVr35VdH11dTW/+93vim7L9atPnTqVDRs25Nd/6UtfGvH6jaUYdsuo5S4iMpTYhXt+nLv63EVESopduCc1WkZEZEixC/dUoHHuIiJDiV2451vu6nMXESkphuGu0TIiIkOJXbin8qNlFO4icvQaGhoA2L59O1dccUXRMueddx5r164d9Di33XYb7e3t+eXj5RHCsQv3fMtd3TIiMgJmz57NmjVrjnr//uF+vDxCOH7hnvtAVd0yIlLg5ptv5vvf/35++atf/Spf//rXueCCC1i+fDlLlizhwQcfHLDfW2+9xeLFiwHo6OjgmmuuYeHChVx++eV9ni1z44030tzczKJFi7jllluA8GFk27dv5/zzz+f8888HwkcI7969G4Bbb72VxYsXs3jxYm677bb8+y1cuJDPf/7zLFq0iA9/+MOj8gyb2H1D1cxIBqaWu8jx7Hc3w46/juwxZy6Bi79ZcvPVV1/NF7/4Rb7whS8AcN999/HII4+wevVqJk6cyO7du1m5ciWXXXZZyd8nveOOO6irq2Pjxo2sX7+e5cuX57d94xvfYPLkyWQyGS644ALWr1/P6tWrufXWW3n00UeZOnVqn2OtW7eOn/zkJzzzzDO4O2eddRYf/OAHaWpqGpNHC8eu5Q5h14w+UBWRQsuWLWPXrl1s376dF198kaamJmbOnMmXv/xlli5dyoUXXsi2bdvYuXNnyWM8/vjj+ZBdunQpS5cuzW+77777WL58OcuWLeOll17i5ZdfHrQ+TzzxBJdffjn19fU0NDTwiU98gj//+c/A2DxaOHYtdwg/VNXjB0SOY4O0sEfTlVdeyZo1a9ixYwdXX301v/zlL2ltbWXdunWkUinmz59f9FG/Q3nzzTf51re+xXPPPUdTUxPXXXfdUR0nZyweLRzflrtGy4hIP1dffTX33nsva9as4corr+TAgQNMnz6dVCrFo48+ypYtWwbd/wMf+ED+4WMbNmxg/fr1ABw8eJD6+nomTZrEzp07+zyErNSjhs8991x+85vf0N7ezuHDh3nggQc499xzR/BsBxfLlnsyoZa7iAy0aNEi2tramDNnDrNmzeLTn/40H/3oR1myZAnNzc2ccsopg+5/4403cv3117Nw4UIWLlzIihUrADjttNNYtmwZp5xyCvPmzePss8/O77Nq1SouuugiZs+ezaOPPppfv3z5cq677jrOPPNMAG644QaWLVs2Zr/uZO7laQE3Nzf7UONHS3n///oj73v3VP75qtNGuFYicrQ2btzIwoULy12NilLs39TM1rl781D7xrRbJtCDw0REBhHTcFefu4jIYGIZ7hotI3J8Klc3byU61n/LWIa7xrmLHH9qamrYs2ePAn4EuDt79uyhpqbmqI8x5GgZM5sH/ByYAThwl7t/p18ZA74DXAK0A9e5+/NHXashaLSMyPFn7ty5tLS00NraWu6qVISamhrmzp171PsPZyhkGvh7d3/ezCYA68zs9+5e+PWsi4GTouks4I7odVSkAvW5ixxvUqkUCxYsKHc1JDJkt4y7v5Nrhbt7G7ARmNOv2MeAn3voaaDRzGaNeG0jYbeMWu4iIqUcUZ+7mc0HlgHP9Ns0B9hasNzCwBvAiEklAj3PXURkEMMOdzNrAO4HvujuB4/mzcxslZmtNbO1x9IvlwzUchcRGcywwt3MUoTB/kt3/3WRItuAeQXLc6N1fbj7Xe7e7O7N06ZNO5r6AmHLXX3uIiKlDRnu0UiYHwMb3f3WEsUeAj5joZXAAXd/ZwTr2UdKo2VERAY1nNEyZwN/A/zVzF6I1n0ZOAHA3e8EHiYcBrmZcCjk9SNf1V4a5y4iMrghw93dnwCK/2xJbxkHvjBSlRpKMlC3jIjIYGL5DdVUwtQtIyIyiFiGu7plREQGF89w14PDREQGFctwT+mRvyIig4pluOvHOkREBhfLcE8FRk/G9WhREZESYhnuyURY7Yw+VBURKSqm4R4Ou9eIGRGR4mIZ7qkgrLZGzIiIFBfLcM+33DViRkSkqJiGe9Ry14gZEZGiYhnuqUAtdxGRwcQy3HMtd4W7iEhxsQz3VNTnrm4ZEZHiYhnuyUAtdxGRwcQz3HMtdw2FFBEpKpbhntKXmEREBhXLcO/tllHLXUSkmHiGe75bRi13EZFiYhnuqdxQSI2WEREpKpbhntSXmEREBhXLcM+13DVaRkSkuFiGux75KyIyuHiGux75KyIyqFiGe0qP/BURGVQswz2p0TIiIoOKZbjnHvmrce4iIsXFMtx7H/mrlruISDExDXeNlhERGUwsw733B7IV7iIixcQy3Ht/IFvdMiIixcQz3HMfqKpbRkSkqFiGu5mRDEwtdxGREmIZ7hB2zegDVRGR4mIb7qkg0OMHRERKiG24JxOmxw+IiJQQ43AP9PgBEZESYhvuqcA0zl1EpIQhw93M7jazXWa2ocT288zsgJm9EE1fGflqDpRMBBotIyJSQnIYZX4KfA/4+SBl/uzuHxmRGg1TMmEa5y4iUsKQLXd3fxzYOwZ1OSKpQC13EZFSRqrP/X1m9qKZ/c7MFpUqZGarzGytma1tbW09pjdMJtTnLiJSykiE+/PAie5+GvBd4DelCrr7Xe7e7O7N06ZNO6Y3TSY0zl1EpJRjDnd3P+juh6L5h4GUmU095poNIRVonLuISCnHHO5mNtPMLJo/MzrmnmM97lDCxw+o5S4iUsyQo2XM7B7gPGCqmbUAtwApAHe/E7gCuNHM0kAHcI27j3qTOpUI6OxJj/bbiIjE0pDh7u6fGmL79wiHSo6pZKCWu4hIKbH9hmr4JSb1uYuIFBPbcE8lTKNlRERKiG24J4NAz3MXESkhvuGuR/6KiJQU23DXj3WIiJQW23DXz+yJiJQW23BP6fEDIiIlxTbck3r8gIhISfENd/3MnohISbEN91T0yN8xeNKBiEjsxDbck0FY9Yw+VBURGSC+4Z4wAI2YEREpIrbhnorCXSNmREQGinG4h1XXiBkRkYFiG+7JKNx7NGJGRGSA2IZ7Koj63NVyFxEZILbhnlS3jIhISbEN9/wHquqWEREZILbhnhvnrpa7iMhA8Q13DYUUESkptuGe0peYRERKim2493bLqOUuItJffMM93y2jlruISH+xDff8N1Q1WkZEZIDYhntSX2ISESkptuGea7lrtIyIyECxDXc98ldEpLT4hnuglruISCmxDff8OHf1uYuIDBDbcE9qtIyISEmxDffcI381zl1EZKDYhnvvI3/VchcR6S/G4a7RMiIipcQ23FP50TIKdxGR/mIb7vmWu7plREQGiG+45z5QVbeMiMgAsQ13MyMZmFruIiJFDBnuZna3me0ysw0ltpuZ3W5mm81svZktH/lqFpdMmD5QFREpYjgt958CFw2y/WLgpGhaBdxx7NUanlQQ6PEDIiJFDBnu7v44sHeQIh8Dfu6hp4FGM5s1UhUcTDJhevyAiEgRI9HnPgfYWrDcEq0bdclEoMcPiIgUMaYfqJrZKjNba2ZrW1tbj/l4qcA0zl1EpIiRCPdtwLyC5bnRugHc/S53b3b35mnTph3zGycTgUbLiIgUMRLh/hDwmWjUzErggLu/MwLHHVIyYRrnLiJSRHKoAmZ2D3AeMNXMWoBbgBSAu98JPAxcAmwG2oHrR6uy/aUCtdxFRIoZMtzd/VNDbHfgCyNWoyOg0TIiIsXF9huqEPa5q1tGRGSgWId7So8fEBEpKtbhrm4ZEZHiYh3uqURAj77EJCIyQKzDPRmYni0jIlJEvMM9EahbRkSkiFiHeyqhlruISDGxDvdkEOh57iIiRcQ73DVaRkSkqFiHu36sQ0SkuFiHu35mT0SkuFiHeyqhlruISDGxDvdkoD53EZFi4h3u+pk9EZGiYh3u4Th3J3zqsIiI5MQ63JNBWP2MPlQVEekj3uGeMACNmBER6SfW4Z6Kwl0jZkRE+op1uOe6ZTRiRkSkr1iHe77lrhEzIiJ9xDrckwm13EVEiol3uAfRB6oKdxGRPmId7qmo5a5uGRGRvmId7vmhkGq5i4j0Ee9wj0bLaCikiEhfsQ73lL7EJCJSVKzDvXe0jFruIiKFYh3uTXUpAHa1dZW5JiIix5dYh/spMydSlQz4y9v7yl0VEZHjSqzDvSoZsGTOJP7y9v5yV0VE5LgS63AHWDavkb9uO0B3Wv3uIiI58Qz3TDo/u+yEJrrSWV7ZcbCMFRIROb7EL9xffhC+OQ8OtACw/MRGAJ7fon53EZGc+IV74wnQ0w5vPw3ArEm1zJxYw1+2qt9dRCQnfuE+Ywmk6mHrM/lVy05o1IeqIiIF4hfuiSTMXdEn3Jef0MTbe9vZfUjj3UVEII7hDjBvJezYAF2HgLDlDqj1LiISiWm4nwWegW1rAVg8ZxLJwPRlJhGRyLDC3cwuMrNXzWyzmd1cZPt1ZtZqZi9E0w0jX9UC884ADN4Ou2ZqUglOnT1RLXcRkciQ4W5mCeD7wMXAqcCnzOzUIkX/xd1Pj6YfjXA9+6qZBNNPHdDv/mLLfj1ETESE4bXczwQ2u/sb7t4N3At8bHSrNQwnnAUtz0E2A4T97u3dGV7d2VbmiomIlN9wwn0OsLVguSVa198nzWy9ma0xs3kjUrvBzDsLug7Cro0AnLVgCsnAuPfZrUPsKCJS+UbqA9XfAvPdfSnwe+BnxQqZ2SozW2tma1tbW4/tHeedFb5uDb/MNHNSDVefMY97nn2bt/e0H9uxRURibjjhvg0obInPjdblufsed88NMv8RsKLYgdz9LndvdvfmadOmHU19ezXNh4YZsPXZ/Kr/+qGTSATGbX947diOLSISc8MJ9+eAk8xsgZlVAdcADxUWMLNZBYuXARtHroolmIWt9+gxBBC23q97/3weeGEbr6nvXUTGsSHD3d3TwN8BjxCG9n3u/pKZ/Q8zuywqttrMXjKzF4HVwHWjVeE+5p0F+7fAwXfyq/72g++moSrJtx55dUyqICJyPBpWn7u7P+zu73X3d7v7N6J1X3H3h6L5f3D3Re5+mruf7+6vjGal8979ofD1ud6Rl031VXz+A+/i31/eyTNv7BmTaoiIHG/i+Q3VnBmnwpKr4Mnvwv6386v/8zkLOGFyHX/7f9exedehMlZQRKQ84h3uABfeAhbA72/Jr2qoTvKLz51JIjA+e/ez7DjQWcYKioiMvfiH+6S5cPZN8NKv+3y4euKUen56/Zkc6Ojhs3c/y/727jJWUkRkbMU/3AHOXg0TZsG/3QzZ3scPLJ4zibv+ZgVv7j7MR777BC/oBz1EZJyojHCvqocLvwrb/wJ//Bq45ze9/z1Tufe/rMQdrrjjSX74+Btks17yUCIilaAywh1g6dWw4jr4j9vg91/pE/DLT2ji4dXncsHC6Xzj4Y1c+X+eYu1be8tXVxGRUVY54W4Gl34bzrgBnrwdHvnHPgE/qS7Fndeu4H9/cilb97ZzxZ1PccPP1rJh24EyVlpEZHSYe3m6KJqbm33t2rUjf2D3sO/9mTvhpP8EH7k1/NC1QEd3hrv/403ufOx12rrSLDuhkWvPOpFLl86iJpUY+TqJiIwQM1vn7s1Dlqu4cIcw4J++A/70P8ES4XDJ5s9B0PcPlQMdPfz6+RZ+8fQW3mg9TH1VggsWzuCSJTP54HunU1uloBeR48v4DvecfW/Bb2+CNx6Dqe+F96+GpVdBsrpPMXfnqTf28NsXt/PISzvZe7ibqkRA8/wmzjlpKu971xQWzZ5EVbJyerFEJJ4U7jnu8NID8MS3Ycd6aJgJyz8DS66AaScPKJ7OZHnmzb089uou/rxpN6/sCB9AVp0MWDp3EqfPa2TR7Eksmj2Rd01rIBHY6J+DiEhE4d6fe9iCf+p78PqfwLMwYwks/Ci85wKYvQyCgd0wu9o6WfvWPp7fso91b+/jpe0H6U6HY+mrkgHvntbASdMbeM/0BuZPrWfBlHpOnFrHxJrU2J2biIwbCvfBtO0MW/Mb7g9/qg+HmkaYfw7MPQPmnQmzToequgG79mSyvN56iJe3H2TjOwfZvOsQm3YdomVfR59yk2pTzJtcy9zGOmY11jBrUg2zJtUyY2INMyZWM31Cjfr0ReSIKdyH6/AeePMx2PwnePsp2Pt6uN4CmPIemLkEZiyCqSeH/faTF0BiYKu8ozvDlr2HeWt3O1v2HKZlXwct+9rZuq+Dd/Z3cLg7M2CfhuokUxuqmNpQzZSGKibXVzOlvorJ9VU01adoqquisa6KxtoUk2pTTKxNqRtIZJxTuB+tw7vDX3d65wXY8ddwOlDwu6yWgMZ50LQg/DWoxnkw6YRwuOXE2eFjEJJVfQ7p7rR1pXlnfyc7D3ayq62LnQc72X2oi9a2LnYf6mLPoW72Hu5mX3s3g32BdkJ1kom1KSbUJJlYE75OqEnSUJNkQk2Khuok9VUJ6quT4Xx1kvrqBHVVSeqrktRWJairSlCbShDoRiESOwr3kdR5EHZvgt2vwZ5NsPdN2Pcm7NsCHUW+6Vo3FSbMhIbpUD8dGqZBfTTVTYmmyVA7Gaon9hmimc06Bzt78kF/oKOH/e3hdLCzhwMd4XSwI01bZw9tnWnauno43JWhrbOHnszwr2dNKqCuKkltKkFtFPi1qQQ1VQlqkgE1qQQ1qdxruK46mq9OBuEUzVfllpMB1ckEVcmAqkS4Pj8lwkk3FZGjN9xwT45FZWKvZiLMXRFO/XUdggMt4dS2PfxVqLbtcGgXHNoJra9B+25Il3jssAVhf39tI9Q0EtRMorFmEo01E8Pgr5kUvtZPgMkToLoBqnKvDeFzdaoa8n8tdPZkONyV5nBXhkNdadq709FrhvbuDB3daQ5H8509Gdq703R0Z+noSefXHezoYWd3hs50uNzZk6UrHb6OhGRgVCUDUone0E8ljFQiiKaC+WRAKgiXk/n1RjIRrk/m1ge925PR+lTCSAa5ZSMRRMsJIxmEy6lEEK3v3R6uL1hOGAkLlxP5cn1fzXTDkuOLwv1YVTfA9FPCqRR36D4Eh1uhfR+07wkDv2NfOLXvhc4D0LkfOvbDwW3hXwudByDdUfq4hYIkVNVTk6qnpqqOKam6MPhTtZCqi16j+WRNOF8bveaWk9XhfLIakrXhDSO3nKjGk9V0eZIuUnRlA7rSTle6N/i701m6M1m6ejJ0Z6LlaF13OktXtNyT6V3fk/H8cjqbpTvtdGeypDNZ0hmnvaMnP9+TydKTzdKTdtLZcN90JktPNnwt5/PgAqM3/M0ICoK//7og6L1ZBLnXwEgYJIOAIKDPtty+ufcICvYN5+m3bJiRfw+LyhRuD4zomH2PH5bp3da7ruC4BdvM+pYr3G4F++bKJoK+ZftvL3w/M4YuE4DRd5++x2bc3ngV7mPBDKonhNPkI9w30wNdbdB1MPwroastvFF0tUH34Whqg+72cL7ncDjfEy13HgxHB/Uchp7O8GbR0wGZI3++vQE10QSWD30SqWg+FS1XhTeGRFW0rmA+yC0nwzLVVeGNKb8tGb0WW072Tvnl6vx8lgRpS5AmQcYDegjIEJD2RDhZgrQbPdmAjAX0ZAN6PCDjRsaddNZJZ5xMNlswH07prJNxJ5MJt2X7lO9dzubKFqzLZMJ9swXHyRaUyWSdjJNfl8mGN7I+27OOO/l9s547DtH+jhcpm5vPlS9TL2zZFd5QzKzghhAt97spGL03ksIbDIS9qEbvTYbCGwn93qOgbOH7XL5sDteuPHFUz1nhfrxLpML++bojvSsMIZsJu4p6OsLXdFc03xUtd0C6u3dbpqtgW3fvcqYnmu8ObxgDpp7wppTtCefTXZBNR/t19533gSOKjkQAVEXTke+cDD8sD5Lh9x2CRN9lS4T/p+bLJcIutfy2RPH1fcoEkCi2PQhfc8v57UXmB5QLSm8z67stSOAYbgFZN7IYTviaJSyfccMJcMivd6L1ZgX7BWQxMmEMhmULj+m9+ziQ8d4y4XsYWSDr0XaMTBawgKznyoN7QAby9cxkPV+fTLRv1o2sQ9Ys2hfcc+UJ65C70RH+Ie0e3iDDG19484PwNbfeCW+44byHxyLckHWP3jva1wfu6957Y832W5ccg8+dFO7jVZCI+uvry12TXu5h0OduBNl07w0gm4ZMOtyWn0/3lvVMwfZMtC0Trs/dQDxbMJ8pXi6bGWS533r3cL7Ptug90l3h++W3ZXvL9Jn3guWC8nh4rFz53JTbdpQsmsbfgzSs94aH9d74cvN9XqO+npLb+q/r/9p/3/77AOnPAn83qmescJfjh1n0wfBRtb3Hj7Dp2Tf4s/1uArmbRv4GUnDT8Gz0i2WFx+lfPjoG3vemUvT4hNvyZQrKDdjHC9aXmi987yJ1GXLeh1mm4L1z/67FyvV/7bOf957/cPbJvU/DjNH77yOicBeJm9wnjeOw/S3Dp/86REQqkMJdRKQCKdxFRCqQwl1EpAIp3EVEKpDCXUSkAincRUQqkMJdRKQCle157mbWCmw5yt2nArtHsDpxMR7PezyeM4zP8x6P5wxHft4nuvu0oQqVLdyPhZmtHc7D6ivNeDzv8XjOMD7PezyeM4zeeatbRkSkAincRUQqUFzD/a5yV6BMxuN5j8dzhvF53uPxnGGUzjuWfe4iIjK4uLbcRURkELELdzO7yMxeNbPNZnZzueszGsxsnpk9amYvm9lLZnZTtH6ymf3ezDZFr03lrutoMLOEmf3FzP41Wl5gZs9E1/xfzKyifs3DzBrNbI2ZvWJmG83sfePhWpvZf4v++95gZveYWU0lXmszu9vMdpnZhoJ1Ra+vhW6Pzn+9mS0/2veNVbibWQL4PnAxcCrwKTM7tby1GhVp4O/d/VRgJfCF6DxvBv7o7icBf4yWK9FNwMaC5X8Cvu3u7wH2AZ8rS61Gz3eAf3P3U4DTCM+9oq+1mc0BVgPN7r4YSADXUJnX+qfARf3Wlbq+FwMnRdMq4I6jfdNYhTtwJrDZ3d9w927gXuBjZa7TiHP3d9z9+Wi+jfB/9jmE5/qzqNjPgI+Xp4ajx8zmApcCP4qWDfgQsCYqUlHnbWaTgA8APwZw92533884uNaEvwRXa2ZJoA54hwq81u7+OLC33+pS1/djwM899DTQaGazjuZ94xbuc4CtBcst0bqKZWbzgWXAM8AMd38n2rQDGP0fYhx7twH/HchGy1OA/e6ejpYr7ZovAFqBn0RdUT8ys3oq/Fq7+zbgW8DbhKF+AFhHZV/rQqWu74hlXNzCfVwxswbgfuCL7n6wcJt77td3K4eZfQTY5e7ryl2XMZQElgN3uPsy4DD9umAq9Fo3EbZSFwCzgXoGdl2MC6N1feMW7tuAeQXLc6N1FcfMUoTB/kt3/3W0emfuT7TodVe56jdKzgYuM7O3CLvcPkTYH90Y/ekOlXfNW4AWd38mWl5DGPaVfq0vBN5091Z37wF+TXj9K/laFyp1fUcs4+IW7s8BJ0WfqFcRfgDzUJnrNOKifuYfAxvd/daCTQ8Bn43mPws8ONZ1G03u/g/uPtfd5xNe2z+5+6eBR4EromIVdd7uvgPYamYnR6suAF6mwq81YXfMSjOri/57z513xV7rfkpd34eAz0SjZlYCBwq6b46Mu8dqAi4BXgNeB/6x3PUZpXM8h/DPtPXAC9F0CWH/8x+BTcAfgMnlruso/hucB/xrNP8u4FlgM/D/gOpy12+Ez/V0YG10vX8DNI2Haw18DXgF2AD8AqiuxGsN3EP4uUIP4V9qnyt1fQEjHBH4OvBXwtFER/W++oaPwO54AAAAOElEQVSqiEgFilu3jIiIDIPCXUSkAincRUQqkMJdRKQCKdxFRCqQwl1EpAIp3EVEKpDCXUSkAv1/EKz70lnaVJwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f65c073fe10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train_vs_validation_loss(train_loss, eval_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validation vs training loss without first 10 steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd4VVW6+PHvm957TwihBAiEHpqIgKBiAzt2nVGZUfk5Xqdcx7nTnPGO43gd9eroYJ1xLKPYsCDXAtgACb0TSiC9kl5IWb8/9gkEJCTASXZyzvt5nv2cs9s5b85z8q591lp7LTHGoJRSyj142B2AUkqpnqNJXyml3IgmfaWUciOa9JVSyo1o0ldKKTeiSV8ppdyIJn2llHIjmvSVUsqNaNJXSik34mV3AMeLiooyKSkpdoehlFJ9yrp160qNMdGdHdfrkn5KSgqZmZl2h6GUUn2KiBzoynFavaOUUm5Ek75SSrkRTfpKKeVGel2dvlLKtTQ1NZGbm0tDQ4PdobgEPz8/kpKS8Pb2Pq3zNekrpbpVbm4uwcHBpKSkICJ2h9OnGWMoKysjNzeXAQMGnNZraPWOUqpbNTQ0EBkZqQnfCUSEyMjIM/rVpElfKdXtNOE7z5l+li6T9A/VHuaJz7LYll9pdyhKKdVruUzS9/AQnvwii4+3FNgdilKqF6moqOBvf/vbKZ930UUXUVFR0Q0R2ctlkn6ovzfj+4ezfGeJ3aEopXqRjpJ+c3PzSc/7+OOPCQsL666wbOMySR9g5tAYthdUUVipXcOUUpb777+fvXv3MmbMGCZMmMC0adOYO3cuw4cPB+Cyyy5j/PjxjBgxgkWLFh05LyUlhdLSUrKzs0lLS+OOO+5gxIgRnH/++dTX19v155wxl+qyOXNYNH/+ZCcrdxczf0Ky3eEopY7z+w+2sT2/yqmvOTwhhN9eOqLD/Q8//DBbt25l48aNrFixgosvvpitW7ce6fL44osvEhERQX19PRMmTODKK68kMjLymNfIysri9ddf57nnnuOaa67h7bff5sYbb3Tq39FTXOpKf2hsMPGhfqzYpVU8SqkTmzhx4jF93J988klGjx7N5MmTycnJISsr63vnDBgwgDFjxgAwfvx4srOzeypcp+vSlb6IzAGeADyB540xDx+3/8fA3UALUAMsMMZsF5HzgIcBH+Aw8HNjzBdOjP/4OJkxNJoPNhXQ1NKKt6dLlWlK9XknuyLvKYGBgUeer1ixgs8++4xVq1YREBDAjBkzTtgH3tfX98hzT0/PPl2902lWFBFP4GngQmA4cJ2IDD/usNeMMSONMWOAR4DHHNtLgUuNMSOBW4BXnBZ5B2YMjaGmsZnM7EPd/VZKqT4gODiY6urqE+6rrKwkPDycgIAAdu7cyerVq3s4up7XlSv9icAeY8w+ABF5A5gHbG87wBjTvpIuEDCO7Rvabd8G+IuIrzGm8UwD78jUwVF4ewordhUzZVBk5ycopVxaZGQkU6dOJT09HX9/f2JjY4/smzNnDs8++yxpaWkMHTqUyZMn2xhpz+hK0k8Ectqt5wKTjj9IRO4G7sOqyjn3BK9zJbC+OxM+QJCvFxMHRLB8VzG/vCitO99KKdVHvPbaayfc7uvry9KlS0+4r63ePioqiq1btx7Z/rOf/czp8fUkp1V6G2OeNsYMAv4T+K/2+0RkBPBn4EcnOldEFohIpohklpSceSPszKEx7C6qIfdQ3Rm/llJKuZKuJP08oF+79STHto68AVzWtiIiScC7wM3GmL0nOsEYs8gYk2GMyYiO7nSKx07NGGq9hvbiUUqpY3Ul6a8FUkVkgIj4ANcCS9ofICKp7VYvBrIc28OAj4D7jTHfOCfkzg2KDiIp3J/lO4t76i2VUqpP6DTpG2OagYXAMmAH8KYxZpuIPCgicx2HLRSRbSKyEate/5a27cBg4DcistGxxDj/zziWiDA7LZav95RS03jyW62VUsqddKmfvjHmY+Dj47b9pt3zn3Rw3h+BP55JgKfropHxvPxtNp/vKGLemEQ7QlBKqV7HZe9eyugfTkywr466qZRS7bhs0vfwEC4aGc/yXSVaxaOU6rKgoCAA8vPzueqqq054zIwZM8jMzDzp6zz++OPU1R3tQdhbhmp22aQPcPGoeA43t/L5jiK7Q1FK9TEJCQksXrz4tM8/Pun3lqGaXTrpj08OJzbEl482axWPUu7q/vvv5+mnnz6y/rvf/Y4//vGPzJo1i3HjxjFy5Ejef//9752XnZ1Neno6APX19Vx77bWkpaVx+eWXHzP2zp133klGRgYjRozgt7/9LWAN4pafn8/MmTOZOXMmcHSoZoDHHnuM9PR00tPTefzxx4+8X08M4exSQysfz8NDuDA9nte+O0h1QxPBft52h6SUe1t6PxRuce5rxo2ECx/ucPf8+fO59957ufvuuwF48803WbZsGffccw8hISGUlpYyefJk5s6d2+H8s8888wwBAQHs2LGDzZs3M27cuCP7HnroISIiImhpaWHWrFls3ryZe+65h8cee4zly5cTFRV1zGutW7eOl156iTVr1mCMYdKkSUyfPp3w8PAeGcLZpa/0AS5xVPF8oX32lXJLY8eOpbi4mPz8fDZt2kR4eDhxcXE88MADjBo1itmzZ5OXl0dRUcfVwF9++eWR5Dtq1ChGjRp1ZN+bb77JuHHjGDt2LNu2bWP79u0dvQwAX3/9NZdffjmBgYEEBQVxxRVX8NVXXwE9M4SzS1/pA4xLDicuxI8PNxdo102l7HaSK/LudPXVV7N48WIKCwuZP38+r776KiUlJaxbtw5vb29SUlJOOKRyZ/bv38+jjz7K2rVrCQ8P59Zbbz2t12nTE0M4u/yVvoeHcOHIOFbuLqG6ocnucJRSNpg/fz5vvPEGixcv5uqrr6ayspKYmBi8vb1Zvnw5Bw4cOOn555xzzpFB27Zu3crmzZsBqKqqIjAwkNDQUIqKio4ZvK2jIZ2nTZvGe++9R11dHbW1tbz77rtMmzbNiX/tybl80ge4ZFQCh5tbWbql0O5QlFI2GDFiBNXV1SQmJhIfH88NN9xAZmYmI0eO5J///CfDhg076fl33nknNTU1pKWl8Zvf/Ibx48cDMHr0aMaOHcuwYcO4/vrrmTp16pFzFixYwJw5c4405LYZN24ct956KxMnTmTSpEncfvvtjB071vl/dAfEGNNjb9YVGRkZprP+r6fKGMOs/1lJZJAPb/34LKe+tlLq5Hbs2EFamg5z7kwn+kxFZJ0xJqOzc93iSl9EuDqjH2uzD7GvpMbucJRSyjZukfQBrhyXiKeH8Na6XLtDUUop27hN0o8J8WPm0GjeXpdLc0ur3eEo5VZ6WzVyX3amn6XbJH2AqzP6UVzdyMrdOrmKUj3Fz8+PsrIyTfxOYIyhrKwMPz+/034Nl++n3965w2KICvLhzcwcZqXFdn6CUuqMJSUlkZubizOmQlVWIZqUlHTa57tV0vf29OCKcUm8+PV+SmsaiQry7fwkpdQZ8fb2ZsCAAXaHoRzcqnoH4OrxSTS3Gt5df7JpfpVSyjW5XdJPjQ1mfP9wXvvuIK2tWseolHIvbpf0AW45K4X9pbWs2K2DsCml3ItbJv0L0+OIC/HjpW+y7Q5FKaV6lFsmfW9PD26a0p+vskrZXfT9AZGUUspVuWXSB7h+YjK+Xh56ta+Ucitum/TDA324Ylwi727I5VDtYbvDUUqpHuG2SR/g1rMG0NDUyutrD9odilJK9Qi3TvpD44KZOjiSV1YdoEnH41FKuQG3TvoAt589kILKBt7boDdrKaVcn9sn/RlDo0lPDOHp5Xt09E2llMtz+6QvIiycmUp2WR0fbi6wOxyllOpWbp/0Ac4fHsvQ2GCeWr5Hh2ZQSrm0LiV9EZkjIrtEZI+I3H+C/T8WkS0islFEvhaR4e32/dJx3i4RucCZwTuLh4ew8NzB7CmuYelWnTxdKeW6Ok36IuIJPA1cCAwHrmuf1B1eM8aMNMaMAR4BHnOcOxy4FhgBzAH+5ni9XueikfEMig7kf7/I0qt9pZTL6sqV/kRgjzFmnzHmMPAGMK/9AcaYqnargUBb1pwHvGGMaTTG7Af2OF6v1/F0XO3vLKzm0x1FdoejlFLdoitJPxHIabee69h2DBG5W0T2Yl3p33Mq5/YWl45KYGBUII8u26U9eZRSLslpDbnGmKeNMYOA/wT+61TOFZEFIpIpIpl2Tqnm5enBzy8YSlZxDW+vz7UtDqWU6i5dSfp5QL9260mObR15A7jsVM41xiwyxmQYYzKio6O7EFL3mZMex9jkMB77dDf1h1tsjUUppZytK0l/LZAqIgNExAerYXZJ+wNEJLXd6sVAluP5EuBaEfEVkQFAKvDdmYfdfUSEBy5Ko6iqkRe/2W93OEop5VSdJn1jTDOwEFgG7ADeNMZsE5EHRWSu47CFIrJNRDYC9wG3OM7dBrwJbAc+Ae42xvT6y+cJKRGcNzyWZ1bspaym0e5wlFLKacSY3tU9MSMjw2RmZtodBnuKa7jg8S+5aXJ/fjd3hN3hKKXUSYnIOmNMRmfH6R25HRgcE8T8Cf341+oDZOnsWkopF6FJ/yR+et4QAn29+PX7W+ltv4iUUup0aNI/icggX35+wVBW7ytnyaZ8u8NRSqkzpkm/E9dNTGZUUigPfbSD6oYmu8NRSqkzokm/E54ewh/mpVNS08hfP83q/ASllOrFNOl3weh+YVw3MZl/rMpme35Vp8crpVRvpUm/i35xwVDCA7z5xdubdD5dpVSfpUm/i8ICfPjjZelszavi7yv32h2OUkqdFk36p2BOejyXjIrnic+z2FWoffeVUn2PJv1T9Pu5Iwjx8+Znb23S4ZeVUn2OJv1TFBnky4Pz0tmSV8nfv9xndzhKKXVKNOmfhotHxXPxyHj++uluNuVU2B2OUkp1mSb90/TQ5enEBPtyzxsbqGlstjscpZTqEk36pykswIfHrx1LTnkdv3lvq93hKKVUl2jSPwMTB0Rwz6xU3tmQxzs6vaJSqg/QpH+GFs4czMSUCH793lb2ltTYHY5SSp2UJv0z5OXpwePXjsHX25MfvbJO6/eVUr2aJn0nSAjz56nrxrKvpIafv7VJx95XSvVamvSd5KzBUdx/4TCWbi3k2ZXaf18p1Tu5TtJvqIRvnoBS+4Y/vmPaQC4ZFc9flu3ky90ltsWhlFIdcZ2k33wYPv8DZL5oWwgiwiNXjWJIbDB3v7Ze59ZVSvU6rpP0g6Ih7RLY+Bo01dsWRoCPF8/fkoGvlyc/eHktJdWNtsWilFLHc52kDzD+B9BQAduX2BpGUngAL9ySQWlNI3f8M5OGphZb41FKqTaulfRTpkHEQFj3kt2RMLpfGI/PH8um3Arue3Mjra3ao0cpZT/XSvoeHjD+Vji4Cop32h0Nc9LjeODCND7eUshvl2zTrpxKKdu5VtIHGH09eHjD+n/YHQkAd5wzkB+dM5BXVh/gr5/pxOpKKXu5XtIPioa0S21v0G3v/guHcU1GEk9+nsVL3+y3OxyllBtzvaQPVhVPL2jQbSMi/PflI7lgRCy//2A7b2Xm2B2SUspNuWbSH3CO1aCb+YLdkRzh5enBE9eOZVpqFL94ezOL1+monEqpntelpC8ic0Rkl4jsEZH7T7D/PhHZLiKbReRzEenfbt8jIrJNRHaIyJMiIs78AzoIGCYugJw1kLO229+uq/y8PXnu5gymDori54s38bYmfqVUD+s06YuIJ/A0cCEwHLhORIYfd9gGIMMYMwpYDDziOPcsYCowCkgHJgDTnRb9yYy9CfxC4dsneuTtuqot8Z81KJKfaeJXSvWwrlzpTwT2GGP2GWMOA28A89ofYIxZboypc6yuBpLadgF+gA/gC3gDRc4IvFO+QZBxG+z4EMr29shbdpW/jyfP3zzhSOJ/ZfUBu0NSSrmJriT9RKB9y2OuY1tHbgOWAhhjVgHLgQLHsswYs+P0Qj0Nk34Ent6w6qkee8uu8vfx5IVbJjBrWAy/fm8rf1uxx+6QlFJuwKkNuSJyI5AB/MWxPhhIw7ryTwTOFZFpJzhvgYhkikhmSYkTR6cMjoPR11rdN2t636iXft6ePHPjeOaNSeCRT3bxp6U79AYupVS36krSzwP6tVtPcmw7hojMBn4FzDXGtI0ydjmw2hhTY4ypwfoFMOX4c40xi4wxGcaYjOjo6FP9G05uyv+D5gZY+5xzX9dJvD09+Os1Y7hxcjJ/X7mPny/eTFNLq91hKaVcVFeS/logVUQGiIgPcC1wTAd4ERkL/B0r4Re323UQmC4iXiLijdWI23PVOwDRQ2DoRfDdc3C4tkffuqs8PIQ/zEvn3tmpLF6Xyw9fXkt1Q5PdYSmlXFCnSd8Y0wwsBJZhJew3jTHbRORBEZnrOOwvQBDwlohsFJG2QmExsBfYAmwCNhljPnD2H9GpqT+B+nJbx9rvjIhw7+wh/OWqUazaW8bVz66ioLJ33FGslHId0tvqkDMyMkxmZqbzX/ifl0HhZvjJJvANdv7rO9FXWSXc+a/1BPh4sujmDMb0C7M7JKVULyci64wxGZ0d55p35J7Iub+GujJY86zdkXRqWmo0b995Fj5eHlzz91W8u0H78iulnMN9kn7SeKtu/5v/hfpDdkfTqaFxwSxZeDZj+4XxH//exJ+W7qBFx+RXSp0h90n6ADMfgMZK+Lb39ds/kYhAH/51+yRumGT17Ln5xTWU1ej0i0qp0+deST9uJIy4HFY/A7WldkfTJd6eHjx0+UgeuXIUmdmHuOR/v2b9wd7/S0Up1Tu5V9IHmPEANNfDV4/ZHckpuWZCP96+8yy8PT2Y//dVvPD1fr2RSyl1ytwv6UcPgTE3wHeLoLRvDX2QnhjKBwvPZsbQGP7w4XZu+0emVvcopU6J+yV9sHryePnB//3K7khOWWiAN4tuGs+D80bw9Z5SLnziK77Z0zeqqpRS9nPPpB8cC9N/Drs/gazP7I7mlIkIN09J4f27pxLi780Nz6/hwQ+209DUYndoSqlezj2TPsCkH1uzay37JbT0zSEP0uJD+GDh2dwypT8vfrOfi5/8is25FXaHpZTqxdw36Xv5wgX/DaW7Ye3zdkdz2vx9PPn9vHReuW0itY0tXP63b3l02S4am/WqXyn1fe6b9AGGzIFB58LyP0FNcefH92LTUqNZdu85zBuTwFPL93Dxk1+z7oB27VRKHcu9k74IzPmz1YVz6X/aHc0ZCw3w5rFrxvDyDyZQf7iFq579lt8t2aYjdiqljnDvpA9WF85zfgHb3oFdS+2OxilmDI1h2X+cw02T+/OPVdnMfmwlH28p0H79SilN+oA19HLMcPjop9BQZXc0ThHk68WD89J5966pRAX5cter67n1pbXsL+2dcwoopXqGJn0ALx+Y+79QlQ+f/97uaJxqTL8w3r97Kr+5ZDjrDhzi/L+u5OGlO6lpbLY7NKWUDTTpt0nKsLpxrn0eDqyyOxqn8vL04IdnD+CLn01n3phEnl25l3MfXcHb63Jp1ZE7lXIrmvTbO/e/ICwZ3vsxNFbbHY3TxQT78ejVo3n3rrOID/Xjp29t4tKnvubbvXpHr1LuQpN+e75BcPkiqDjoEr15OjI2OZx375rKE9eOoaKuieufW8NtL69lV6HrFXRKqWNp0j9e/ykw7aew8VXY9p7d0XQbDw9h3phEPv/pdH4xZyjf7S9nzhNfct+bG8kpr7M7PKVUN3GfOXJPRUsTvHgBlO2Fu1ZBSIK98fSAQ7WHeXblXl7+NptWY7huYjJ3zRhMXKif3aEppbqgq3PkatLvSNleeHaaNc3iTe+Bh6fdEfWIgsp6nvx8D29l5uDhIVw/MZm7ZgwiJkSTv1K9mU6MfqYiB8FFj8D+L2HFw3ZH02PiQ/350xUjWf6zGVw+JpFXVh9g2iPL+e37W8mvqLc7PKXUGdIr/c68fzds+Bdc/yYMucDuaHrcgbJanlmxl7fX5wJw5bgkFpwzkIHRQTZHppRqT6t3nKWpHl44Dypy4EcrITzF7ohskVdRz6KVe3l9bQ5NLa1cMDyOH00fyNjkcLtDU0qhSd+5yvfDoukQ1h9u+z/w9rc7ItuUVDfyj2+z+eeqbKoampk4IILbzh7A7LRYPD3E7vCUclua9J1t11J4/VoYeTVc8Zw1Qqcbq2ls5o3vDvLSN9nkVdTTPzKAH5yVwlUZ/Qjy9bI7PKXcjib97vDlo/DFH2Dmr2D6L+yOpldobmll2bYiXvh6H+sPVhDk68VV45O4aUp/Bmm9v1I9RpN+dzAG3v0xbH4DrnoJ0q+wO6JeZWNOBf/4NpsPN+fT1GKYlhrFDZOSmZUWi7endhRTqjtp0u8uzY3wj7lQsBFu/djqx6+OUVLdyOvfHeT17w5SUNlAbIgv8yckc01GEknhAXaHp5RLcmrSF5E5wBOAJ/C8Mebh4/bfB9wONAMlwA+NMQcc+5KB54F+gAEuMsZkd/RevT7pA9SWwnPnQlMd/HCZ1adffU9zSyvLd5Xwr9UH+DKrBLCmdbxuQj9mpcXi46VX/0o5i9OSvoh4AruB84BcYC1wnTFme7tjZgJrjDF1InInMMMYM9+xbwXwkDHmUxEJAlqNMR0O7tInkj5AaZY1VINPkNWjJzjO7oh6tZzyOt5al8tbmTkUVDYQEejDvDEJXDU+iREJoXaHp1Sf58ykPwX4nTHmAsf6LwGMMX/q4PixwFPGmKkiMhxYZIw5u6uB95mkD5C3Dl6+FCIGwK0fgX+Y3RH1ei2thi+zSlicmcun24s43NJKWnwIV4xNZN6YBB3uQanT5MykfxUwxxhzu2P9JmCSMWZhB8c/BRQaY/4oIpdhVfscBgYAnwH3G2NajjtnAbAAIDk5efyBAwc6i7v32PsFvHoNJE2AG98GH62z7qqKusMs2ZTP2+vz2JRTgYfA2anRXDYmgfNHxGnXT6VOgS1JX0RuBBYC040xjY5zXwDGAgeBfwMfG2Ne6Oj9+tSVfput78DiH8LA6XDdG25989bp2ltSw3sb8nh3Qx65h+rx9fJgdlosl46OZ8bQGPy83WPAO6VOV1eTflcupfKwGmHbJDm2Hf+Gs4Ff4Uj4js25wEZjzD7HMe8Bk7EKAteRfgU0N8B7d8EbN8C1r4G3VlOcikHRQfz0/KHcd94Q1h+sYMnGPD7cXMBHWwoI9PFk9vBYLhmVwLTUKC0AlDoDXbnS98JqyJ2FlezXAtcbY7a1O2YssBjrF0FWu+2ewHpgtjGmREReAjKNMU939H598kq/zfpXYMlCSD0f5v8LvHztjqhPa25pZdW+Mj7aXMAn2wqpqGsi0MeTWWmxXDQyjulDYvD30QJAKXB+l82LgMexumy+aIx5SEQexErgS0TkM2AkUOA45aAxZq7j3POA/wEEWAcsMMYc7ui9+nTSB8h8CT68FwafB/Nf0aoeJ2lqaeWbPaV8srWQ/9teRHntYfy8PZiWGs0FI+KYNSyG8EAfu8NUyjZ6c5ad1r0MH9wL/afCda+DX4jdEbmU5pZWvttfzrJtVgFQUNmAp4eQ0T+c84bHMjstlpSoQLvDVKpHadK325bF8O6PIG4k3PgOBETYHZFLMsawJa+ST7cX8en2InY6JncfFB3IrLRYzh0Ww/j+4ToMhHJ5mvR7g12fwJs3W/34b3wbQpPsjsjl5ZTX8dmOIr7YWcyafeUcbmkl2M+LaalRzBgSw/Sh0cTqvQDKBWnS7y32f2n16PEJghvegrh0uyNyGzWNzXydVcqKXcWs2FVCYVUDAMPigpk+JJpzhkSTkRKOr5c2Bqu+T5N+b1K4FV69Gg7XWI27A2fYHZHbMcaws7CaFbtK+HJ3CZkHymlqMfh5ezBxQCTnpEZxdmoUQ2ODETefK0H1TZr0e5vKPHj1KijdDZc+AWNvtDsit1bb2MzqfWV8lVXKV1kl7C2pBSAy0IcpgyKZOjiKswZFkhwRoIWA6hM06fdGDZVWHf++FTBlIZz3IHho1UJvkF9Rzzd7Svl2bxnf7CmluNq6vzAh1I/JgyKZMjCSyQMj6Rehw2yo3kmTfm/V0gzLHoDv/g6DZ8OVL+hAbb2MMYa9JTWs2lvGqn1lrN5XTnmtdWtJYpg/kwZEMNGxDIgK1F8CqlfQpN/brXsZPvophKdYd+/GpNkdkepAa6shq7iG1fvKWLO/jDX7yilzFAJRQb5MSAknIyWCiSkRpMUH46XdQ5UNNOn3BdnfwFu3Wg28lz4Jo662OyLVBdYvgVrWZpfz3f5y1maXk3uoHoAAH0/G9AtjfP9wxvUPZ1y/cEIDvG2OWLkDTfp9RXWhlfgProIJd8AFD+mYPX1QQWU9mdmHyMwuZ93BQ+woqKal1frfGhQdyLhkqxAYmxxGakwwnh5aJaScS5N+X9LSBJ/9DlY9BfGj4coXIWqw3VGpM1Db2MymnArWHzzEhoPW46G6JgACfTwZlRTG6H5hjOkXyuh+YcSF+GnbgDojmvT7op0fwft3Q/NhuPhRGH0daCJwCcYYssvq2JhziI0HK9iQU8GOgiqaWqz/v+hgX0YnhTIqKYyRSaGMSgwlMkh/8amu06TfV1XmwTsL4MDXkH4lXPSojtvjohqaWthRUMWmnAo251ayKbeCfaW1tP1LJob5k54YwsjEUNIdS5QWBKoDmvT7stYW+OoxWPkwBEbDvKes7p3K5VU3NLElr5KteZVsyatiS24F2WV1R/bHhfgxIiGEEQkhDE8IZURCCEnh/lo1pDTpu4T8DfDuj6FkJ2T80LqZyzfY7qhUD6tqaGJ7fhVbHYXBtvwq9pbU4GgnJsTPi+EJIQyPDyUtPpi0+BBSY4N0TCE3o0nfVTQ1wBd/gFVPW6N0XvI4pOpVv7urP9zCjsIqtudXsb3AetxZWEVDUysAnh7CoOhAhsWFMCw+mGFxwQyNCyEhVBuMXZUmfVdzcI01FWPpbhh1Lcz5k9b1q2O0tBqyy2rZUVDFjoIqdhVWs6OgmryK+iPHBPt5MTQ2mKFx1jIk1loidNaxPk+TvitqboQvH4WvHwPfEDj/DzDmBu3ho06qsr6J3UXV7CysZldhFTsLqtldVE1VQ/ORY6KCfEiNCWZ091j/AAAR9klEQVRIbBCpscGkxgQxJDZYp6DsQzTpu7KibfDhfZCzGpLPgkse02Ec1CkxxlBU1ciuomqyiqxCYFdRDXuKqqk93HLkuKggHwZFB5EaG8Tg6CAGxwQzKCZQ7yvohTTpu7rWVtj4Knz6a2iogokLYMb9OnibOiPGGPIrG8gqqiarqIY9xTVkFVezp7jmmF8GQb5eDIwOZFB0EIOiAxkYHcTA6EBSIgPx89YGZDto0ncXtWWw/I+Q+ZJVx3/ur2HczTpks3IqYwwlNY3sKa5hb7FVGOwrrWVvcQ35lQ1HjhOx7i8YGB3EwKhABkYHMiDKWhJC/fHQ4Se6jSZ9d1OwGZb+Jxz8FmJGwPkPat9+1SNqG5vZX1rLvtJa9pXUsK+k1lovqTmmqsjHy4OUyABSIq1CICXK+mWQEhVAbLCfFghnSJO+OzIGtr9njeNzKBsGzrT69sePsjsy5YaMMRRXN7K/tPbIsq+kluyyWg6W1XG4pfXIsX7eHiRHBNA/MpCUyACSIwPpH2EVEAlhfjpcdRdo0ndnzY2w9gVY+WdoqID0q2DmAxA5yO7IlAKs7qX5FfUcKKsju6yW7NJaDpTXcaCslgNldTQ2Hy0QvDyExHB/kiMCjln6RQSQHBlAiJ8OXQ2a9BVA/SH45klY86xVEIy7Cc75uXWTl1K9VGuroai6gQNldRx0FAoHy+uOLBWO0UrbhPp7OwoBf/qFB5AUEUC/cH/6RQSQGObvNg3LmvTVUdVF8NWjVmMvWA290+7T5K/6pMr6JnLK68gpr+OA4zHnUD255XXkHqo/ptoIICbYl34RASSF+5MU7k9imPU8MdzfpQoFTfrq+ypyrBu71r9irY+9Eab+BCIG2BuXUk7S9ishp7yenPI68irqyT1UR055PbkVdRRUNNDcemzOiwryJTHcn6SwowVBQpj1mBjmT4i/V5+4J0GTvupYW/Lf8C9rRM+RV8HZ/6E3eCmX19JqKKpqaFcg1JN3qJ68iqPL4eZjfykE+XoRH+pHgqMwSHA8jw/zIzHMn7hQv14xuJ0mfdW5qgJrtq7MF6GpDobMgbP+H/SfqkM7KLdkjKG05jD5bYXAoXryK+vJr6gnv6KBvIp6ymsPf++8qCAf4kP9iQ/1s5Yw63lciFVAxIT4dnvB4NSkLyJzgCcAT+B5Y8zDx+2/D7gdaAZKgB8aYw602x8CbAfeM8YsPNl7adK3QV05fLfIWurKIGEcTLkbhs8DT+0ZoVR7DU0tFFQ2OAqCegoqGyiotAqFwsoG8ivrqW5393KbyEAf4hwFwTGPjuexoX4E+55+VZLTkr6IeAK7gfOAXGAtcJ0xZnu7Y2YCa4wxdSJyJzDDGDO/3f4ngGigXJN+L9ZUDxtfs4ZxLt8LwQkw8XYY/wMd0VOpU1DT2ExhpaNAqGigsKqBgsqGI9uKqhqOzJnc3rjkMN65a+ppvWdXk75XF15rIrDHGLPP8cJvAPOwrtwBMMYsb3f8auDGdoGMB2KBT4BOA1I28vaHCbdZST7r/2DNM/D5g7DyEWvqxgm3QeJ4u6NUqtcL8vVicEwwg2M6nvSooamFoirr10FhlVUQBPl2/y/rriT9RCCn3XouMOkkx98GLAUQEQ/gf7AKgQ7HBBCRBcACgOTk5C6EpLqVhwcMnWMtRdth7XOw6d/WAG8J46xZvNKvAJ9AuyNVqs/y8/akf2Qg/SN79v/Iqfc2i8iNWFfzf3Fsugv42BiTe7LzjDGLjDEZxpiM6OhoZ4akzlTscLjkr/DTnXDhX6wG3yUL4X+GWcM7F2yyO0Kl1CnoypV+HtCv3XqSY9sxRGQ28CtgujGm0bF5CjBNRO4CggAfEakxxtx/ZmGrHucXApMWwMQ74OBqWPey1eUz8wWIGwljb4KRV2vdv1K9XFcacr2wGnJnYSX7tcD1xpht7Y4ZCywG5hhjsjp4nVuBDG3IdSF15bD1bdjwinXF7+ljdfscfZ01wqeXzrqkVE9xWkOuMaZZRBYCy7C6bL5ojNkmIg8CmcaYJVjVOUHAW47uRgeNMXPP6C9QvV9AhHXlP/EOKNxi9fzZ8hbsWAL+EVbj78irod9E7fevVC+hN2cp52ppgr3LYdNrsGspNDdAWLJVAIy4wqoK0gJAKafTO3KV/RqrYedH1tX/3uVgWiBysJX8R1wGMcO1AFDKSTTpq96lthR2fADb3oHsr8G0QsQgGD4X0uZCwlgtAJQ6A5r0Ve9VUww7P4TtS2D/l9YvgJBEGHoRDLvYGvtHG4GVOiWa9FXfUFcOuz+xqoH2fA7N9eAbAoNnWT2BUs/XbqBKdYEzh2FQqvsERMCY663lcB3sW241AO9eBtveBfGApAmQep5VAMSN0mogpc6AXumr3qm1FfI3QNYyaxyg/A3W9sAY61fA4NkwcAYERtkZpVK9hlbvKNdSUwx7PrOqgPZ+AfXl1va4UTBoJgycCcmTrUHjlHJDmvSV62ptgfyNsO8LqytoznfQ2gSevtaNYAOnQ8o5Vo8gbRBWbkKTvnIfjTVw4FvYvxL2rYSiLdZ27wDr6r//VEg521EI+Nobq1LdRBtylfvwDYIh51sLQG0ZHPjauh9g/1fwxR+s7V5+VqNw8hToP8V67tvxeOdKuSK90leur7YUDq6yfg0c+MYaJ8i0Wj2D4kZCv0mOZSKE9tPeQapP0uodpTrSWG21AxxcBTlrIHcdNNVa+4LiICnDKgASMyBhjE4Wo/oErd5RqiO+wY5un7Os9ZZmKNpqFQS5a61l54fWPvG0xghKHGctCeMgJk0njFd9ll7pK3UitaWQtw5yMyEvE/LWQ0OFtc/Lz6oWih9j/RKIHwPRQ7UgULbS6h2lnMkYKN9n3SSWtx4KNloTxxyusfZ7+kLsCIgfZd07EDfKmmpSq4ZUD9HqHaWcSQQiB1nLyKusba2tULbHSv6Fm6zHbe9aU0mC1VAcMQji0iE23fp1EDMcQpO0sVjZRpO+UqfLwwOih1jLqKutbcZAZY7VQ6hgs9VWkLfeKgza+IZavwJihlvtA7EjrEf/cHv+DuVWNOkr5Uwi1kxhYcnWMNFtGiqhaDsUb4Mix7JlMTRWHj0mKA5ihkF0mtVGED3MetRRRpUTadJXqif4hVo3hPWfcnSbMVCVD8XbHctOKNkB6/8BTXVHjwuMhqihEJVqFQJRqRCZat1T4OHR83+L6tM06StlFxEITbSW1POObm9ttaqISndD8Q7rsXS3VUXU1oMIrF5EEYMgarBVCEQOdiyD9NeB6pAmfaV6Gw8PCO9vLe0LA2OsrqRlWY6CIMtaCrfCjg+tGcja+IVZyT/C0fgcMRDCB0DEAAiI1IZkN6ZJX6m+QgSCoq2l/1nH7ms+DBUHoGyv1aOofK/1/OBqa2J62nXN9g2B8BRriRhgFQZt66FJer+Bi9Okr5Qr8PKx6vqjUr+/r6nBKhDK91v3GpTvg0PZVjvCrqXWsNRtxNOarzi8P4T1P9ooHd7fakMISQAPzx77s5TzadJXytV5+zl6Aw39/r7WFqsxueKAVRAcyoZDB6z1PZ9BTeGxx3t4WYk/NBnC+lkFQWjS0echieAT0BN/lTpNmvSVcmcenlbCDutnzTlwvKYGqMy1CoGKg1YDc8VBa9n/JVQXWCOWtucfYRUEoUlWIRCaaD2GJFoFRkiCzmtgI036SqmOeftZvYOiBp94f0uT9UuhMtex5FiPVXnWr4bsb469F6FNQNTRAiAkAYITICQegh1LSLzVGK0Nzk6nSV8pdfo8vY/2NOpIYzVUFUBVLlTmWb8OqvKtgqEyzxrVtK7s++d5+UFwnHXTWnC7JSgOgmOtx6BYq3uqFg5dpklfKdW9fIMhOtgarqIjTQ1WYVBdCNX5ViFRU+hYL7TuYN7zORyu/v65Ht5W8g+KaffoeB4YfXRbYLQVi5sXEF1K+iIyB3gC8ASeN8Y8fNz++4DbgWagBPihMeaAiIwBngFCgBbgIWPMv50Yv1LKFXj7Wd1HIwac/LjDtVYhUFN09LGmCKodj5U51pDYtSUc0021jZcfBMZY3V4DoyEwyloPjDq6HhB19NHLp1v+XDt1mvRFxBN4GjgPyAXWisgSY8z2dodtADKMMXUicifwCDAfqANuNsZkiUgCsE5ElhljKlBKqVPlE3h0tNOTaWm2qoxqi6Gm2CoEaood6yXWY1W+NShebcmx3Vbb8w2FwMh2BUHk0ceAtsdIq4opILJP/JLoypX+RGCPMWYfgIi8AcwDjiR9Y8zydsevBm50bN/d7ph8ESkGogFN+kqp7uPpZdX7B8d2fqwx1vAWtWVWAVBbbN35XNe2Xgp1pVZX1rx11vbW5hO/lof30QLAPwICwh2PEcc++oc7nodbSw/eENeVpJ8I5LRbzwUmneT424Clx28UkYmAD7D3VAJUSqluJXI0+XbUS6k9Y6CxyioM6g85CodSqC+3ntc5HusPQekex/PyjgsKAJ9g6/37TYCrXnTe33YCTm3IFZEbgQxg+nHb44FXgFuMOb5TL4jIAmABQHJysjNDUkop5xKxRk31C+36OcZYvZjqy61Cof7QiZeQhO6L26ErST8P6NduPcmx7RgiMhv4FTDdGNPYbnsI8BHwK2PM6hO9gTFmEbAIrOkSuxy9Ukr1BSLgF2It4Sm2htKVwbjXAqkiMkBEfIBrgSXtDxCRscDfgbnGmOJ2232Ad4F/GmMWOy9spZRSp6PTpG+MaQYWAsuAHcCbxphtIvKgiMx1HPYXIAh4S0Q2ikhboXANcA5wq2P7Rkc3TqWUUjYQY3pXbUpGRobJzMy0OwyllOpTRGSdMSajs+N0rjWllHIjmvSVUsqNaNJXSik3oklfKaXciCZ9pZRyI72u946IlAAH7I7DCaKAUruD6EX08/g+/UyOpZ/HsU718+hvjInu7KBel/RdhYhkdqX7lLvQz+P79DM5ln4ex+quz0Ord5RSyo1o0ldKKTeiSb/7LLI7gF5GP4/v08/kWPp5HKtbPg+t01dKKTeiV/pKKeVGNOmfIRHpJyLLRWS7iGwTkZ84tkeIyKcikuV4DLc71p4kIp4iskFEPnSsDxCRNSKyR0T+7Rh2222ISJiILBaRnSKyQ0SmuPN3RET+w/H/slVEXhcRP3f7jojIiyJSLCJb22074XdCLE86PpvNIjLudN9Xk/6ZawZ+aowZDkwG7haR4cD9wOfGmFTgc8e6O/kJ1lDcbf4M/NUYMxg4hDWtpjt5AvjEGDMMGI312bjld0REEoF7gAxjTDrgiTVPh7t9R14G5hy3raPvxIVAqmNZADxz2u9qjNHFiQvwPnAesAuId2yLB3bZHVsPfgZJji/sucCHgGDdZOLl2D8FWGZ3nD34eYQC+3G0obXb7pbfEY7Oux2BNXvfh8AF7vgdAVKArZ19J7AmqbruRMed6qJX+k4kIinAWGANEGuMKXDsKgRibQrLDo8DvwDa5kOOBCqMNSEPQC7WP767GACUAC85qryeF5FA3PQ7YozJAx4FDgIFQCWwDvf+jrTp6DvRVlC2Oe3PR5O+k4hIEPA2cK8xpqr9PmMVzW7RTUpELgGKjTHr7I6lF/ECxgHPGGPGArUcV5XjZt+RcGAeVmGYAATy/WoOt9dd3wlN+k4gIt5YCf9VY8w7js1FIhLv2B8PFHd0vouZCswVkWzgDawqnieAMBHxchyTBOTZE54tcoFcY8wax/pirELAXb8js4H9xpgSY0wT8A7W98advyNtOvpO5AH92h132p+PJv0zJCICvADsMMY81m7XEuAWx/NbsOr6XZ4x5pfGmCRjTApW49wXxpgbgOXAVY7D3ObzADDGFAI5IjLUsWkWsB03/Y5gVetMFpEAx/9P2+fhtt+Rdjr6TiwBbnb04pkMVLarBjolenPWGRKRs4GvgC0crcN+AKte/00gGWvU0GuMMeW2BGkTEZkB/MwYc4mIDMS68o8ANgA3GmMa7YyvJ4nIGOB5wAfYB/wA66LLLb8jIvJ7YD5W77cNwO1YddRu8x0RkdeBGVijaRYBvwXe4wTfCUfh+BRWNVgd8ANjzGlNJq5JXyml3IhW7yillBvRpK+UUm5Ek75SSrkRTfpKKeVGNOkrpZQb0aSvlFJuRJO+Ukq5EU36SinlRv4/LKxjJN4ImlYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f65c06d2a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train_vs_validation_loss(train_loss, eval_loss, offset=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_hidden = 64\n",
    "\n",
    "def mlp_out(num_hidden):\n",
    "  data = sym.Variable('data')\n",
    "\n",
    "  hid_in = sym.FullyConnected(\n",
    "    data,\n",
    "    name='hidden_in',\n",
    "    num_hidden=num_hidden)\n",
    "  hid_out = sym.Activation(\n",
    "    hid_in,\n",
    "    act_type='relu',\n",
    "    name='hidden_out')\n",
    "\n",
    "  softmax_in = sym.FullyConnected(\n",
    "    data=hid_out,\n",
    "    name='softmax_in',\n",
    "    num_hidden=category_count)\n",
    "  return sym.SoftmaxOutput(softmax_in, name='softmax')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = mod.Module(symbol=mlp_out(num_hidden))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"214pt\" height=\"442pt\"\n",
       " viewBox=\"0.00 0.00 214.00 442.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 438)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-438 210,-438 210,4 -4,4\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"47\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- hidden_in -->\n",
       "<g id=\"node2\" class=\"node\"><title>hidden_in</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-152 -7.10543e-15,-152 -7.10543e-15,-94 94,-94 94,-152\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-126.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-111.8\" font-family=\"Times,serif\" font-size=\"14.00\">64</text>\n",
       "</g>\n",
       "<!-- hidden_in&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>hidden_in&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-83.7443C47,-75.2043 47,-66.2977 47,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-93.8971 42.5001,-83.897 47,-88.8971 47.0001,-83.8971 47.0001,-83.8971 47.0001,-83.8971 47,-88.8971 51.5001,-83.8971 47,-93.8971 47,-93.8971\"/>\n",
       "</g>\n",
       "<!-- hidden_out -->\n",
       "<g id=\"node3\" class=\"node\"><title>hidden_out</title>\n",
       "<polygon fill=\"#ffffb3\" stroke=\"black\" points=\"94,-246 -7.10543e-15,-246 -7.10543e-15,-188 94,-188 94,-246\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-220.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-205.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- hidden_out&#45;&gt;hidden_in -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>hidden_out&#45;&gt;hidden_in</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-177.744C47,-169.204 47,-160.298 47,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-187.897 42.5001,-177.897 47,-182.897 47.0001,-177.897 47.0001,-177.897 47.0001,-177.897 47,-182.897 51.5001,-177.897 47,-187.897 47,-187.897\"/>\n",
       "</g>\n",
       "<!-- softmax_in -->\n",
       "<g id=\"node4\" class=\"node\"><title>softmax_in</title>\n",
       "<polygon fill=\"#fb8072\" stroke=\"black\" points=\"94,-340 -7.10543e-15,-340 -7.10543e-15,-282 94,-282 94,-340\"/>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"47\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">10</text>\n",
       "</g>\n",
       "<!-- softmax_in&#45;&gt;hidden_out -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>softmax_in&#45;&gt;hidden_out</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M47,-271.744C47,-263.204 47,-254.298 47,-246.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"47,-281.897 42.5001,-271.897 47,-276.897 47.0001,-271.897 47.0001,-271.897 47.0001,-271.897 47,-276.897 51.5001,-271.897 47,-281.897 47,-281.897\"/>\n",
       "</g>\n",
       "<!-- softmax_label -->\n",
       "<g id=\"node5\" class=\"node\"><title>softmax_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"159\" cy=\"-311\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"159\" y=\"-307.3\" font-family=\"Times,serif\" font-size=\"14.00\">softmax_label</text>\n",
       "</g>\n",
       "<!-- softmax -->\n",
       "<g id=\"node6\" class=\"node\"><title>softmax</title>\n",
       "<polygon fill=\"#fccde5\" stroke=\"black\" points=\"150,-434 56,-434 56,-376 150,-376 150,-434\"/>\n",
       "<text text-anchor=\"middle\" x=\"103\" y=\"-401.3\" font-family=\"Times,serif\" font-size=\"14.00\">softmax</text>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;softmax_in -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>softmax&#45;&gt;softmax_in</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M80.5686,-367.148C75.113,-358.186 69.3635,-348.74 64.1943,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"85.8939,-375.897 76.8505,-369.695 83.2941,-371.626 80.6944,-367.355 80.6944,-367.355 80.6944,-367.355 83.2941,-371.626 84.5383,-365.015 85.8939,-375.897 85.8939,-375.897\"/>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;softmax_label -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>softmax&#45;&gt;softmax_label</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.377,-367.237C131.257,-357.578 137.481,-347.353 142.955,-338.36\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"120.106,-375.897 121.462,-365.015 122.706,-371.626 125.306,-367.355 125.306,-367.355 125.306,-367.355 122.706,-371.626 129.15,-369.695 120.106,-375.897 120.106,-375.897\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f65c065c908>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mx.viz.plot_network(mlp.symbol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 21s, sys: 3.28 s, total: 1min 25s\n",
      "Wall time: 37.6 s\n"
     ]
    }
   ],
   "source": [
    "mlp_train_log_list = []\n",
    "mlp_eval_log_list = []\n",
    "\n",
    "%time mlp.fit(\\\n",
    "  train_iter,\\\n",
    "  eval_data=valid_iter,\\\n",
    "  eval_metric='ce',\\\n",
    "  batch_end_callback=log_to_list(mlp_train_log_list),\\\n",
    "  eval_batch_end_callback=log_to_list(mlp_eval_log_list, data_type='Valid'),\\\n",
    "  num_epoch=num_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation vs training loss for MLP "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp_train_loss = [msg[1][4] for msg in mlp_train_log_list]\n",
    "mlp_eval_loss = [msg[1][4] for msg in mlp_eval_log_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmUHXWd9/H39+69L+nu7BsQSCchZCPEyYAB1ImooCICR2cGzyjP8eEZdM4sh3HOEWee8Tw+8yiDzjj6oCPqDIPyxA0dUAHDICpLghBDEkgCCel00t1Zeu+7/54/6vaWdNIhud03XfV5nVOn7q2qW/dXKfjUr3/3V78y5xwiIuIvoVIXQEREik/hLiLiQwp3EREfUriLiPiQwl1ExIcU7iIiPqRwFxHxIYW7iIgPKdxFRHwoUqovbmhocAsWLCjV14uITElbt2494pxrHG+7koX7ggUL2LJlS6m+XkRkSjKz/WeynZplRER8SOEuIuJDCncRER8qWZu7iPhLJpOhpaWFZDJZ6qL4QiKRYM6cOUSj0bP6vMJdRIqipaWFqqoqFixYgJmVujhTmnOOo0eP0tLSwsKFC89qH2qWEZGiSCaTTJs2TcFeBGbGtGnTzumvIIW7iBSNgr14zvXfcsqF+5HeFP/zJzvo7E+XuigiIuetKRfuv957lPt/9TpXf/5JHnh2P7m8ngErItDZ2cm//Mu/vOnPXXfddXR2dk5AiUpryoX79dNaeXHxv7OsMcLf/GA77/3yr+jqz5S6WCJSYqcK92w2e9rPPfLII9TW1k5UsUpmyoU76T6qX3+Eb189wOdvuozfHezix9taS10qESmxu+66i71797JixQouv/xyrrzySq6//nqWLFkCwHvf+15Wr17N0qVLue+++4Y+t2DBAo4cOcK+fftobm7mYx/7GEuXLuUd73gHAwMDpTqcczb1ukLOewvEKrHdj3Hju6/ji0+8yuZd7Xx43fxSl0xECv72xy+zo7W7qPtcMquau9+z9JTrP/e5z7F9+3ZefPFFnnzySd71rnexffv2oa6E3/jGN6ivr2dgYIDLL7+cG2+8kWnTpo3ax+7du3nwwQf52te+xgc/+EG+973v8eEPf7ioxzFZpl7NPRKDCzbAnscx4NrF03l6zxEG0rkSF0xEzidr164d1Uf8S1/6Epdddhnr1q3jwIED7N69+6TPLFy4kBUrVgCwevVq9u3bN1nFLbqpV3MHWPR22PUT6NjFNYub+Oav9/Gb145wzeLppS6ZiMBpa9iTpaKiYuj1k08+yeOPP85vfvMbysvL2bBhw5h9yOPx+NDrcDg8pZtlpl7NHeCit3vz3T/nigvqKY+FeWJne2nLJCIlVVVVRU9Pz5jrurq6qKuro7y8nF27dvHMM89Mcukm39SsudfMhunLYPdjxNd/gisXNbB5VzvOOd1EIRJQ06ZNY/369SxbtoyysjKmTx/+S37jxo189atfpbm5mUsuuYR169aVsKSTY2qGO3hNM7/+J0h2ce3i6fzs5TZ2He6heWZ1qUsmIiXyH//xH2Muj8fjPProo2OuG2xXb2hoYPv27UPL/+Iv/qLo5ZtMU7NZBmDROyCfhdeeZMNi74lTv9ilphkREZjK4T5nLcRrYPfPaapKsHxODU/sbCt1qUREzgtTN9zDEbjoGtj9ODjHNYub+O2BTo71acwZEZGpG+7g9ZrpPQztO7jq4kacg+deP1bqUomIlNzUDvfp3m3FHN/H3LpyANp79BQYEZGpHe6Vha5Ove3UV8QIGXT0pEpbJhGR88DUDvcKr5cMve2EQ8a0yjhHehXuIjK+yspKAFpbW/nABz4w5jYbNmxgy5Ytp93PvffeS39//9D782UI4akd7uEolNVDn9cFsrEyrpq7iLwps2bNYtOmTWf9+RPD/XwZQnhqhztAZRP0el0gG6sU7iJBddddd/HlL3956P1nPvMZ/v7v/55rr72WVatWcemll/KjH/3opM/t27ePZcuWATAwMMAtt9xCc3Mz73vf+0aNLfPxj3+cNWvWsHTpUu6++27AG4ystbWVq6++mquvvhoYHkIY4J577mHZsmUsW7aMe++9d+j7JmNo4al7h+qgyibo7QC8cN/dNvbYEiIyiR69Cw7/rrj7nHEpvPNzp1x9880388lPfpI77rgDgIceeoif/exn3HnnnVRXV3PkyBHWrVvH9ddff8phSr7yla9QXl7Ozp072bZtG6tWrRpa99nPfpb6+npyuRzXXnst27Zt48477+See+5h8+bNNDQ0jNrX1q1buf/++3n22WdxznHFFVfw1re+lbq6ukkZWnjq19wrTqi596ZwTo/eEwmalStX0t7eTmtrKy+99BJ1dXXMmDGDT33qUyxfvpy3ve1tHDx4kLa2U9/s+NRTTw2F7PLly1m+fPnQuoceeohVq1axcuVKXn75ZXbs2HHa8jz99NO8733vo6KigsrKSt7//vfzy1/+EpicoYX9UXPvK9TcK+Nkco6ugQy15bESF0wkwE5Tw55IN910E5s2beLw4cPcfPPNPPDAA3R0dLB161ai0SgLFiwYc6jf8bz++ut8/vOf5/nnn6euro7bbrvtrPYzaDKGFp76NffKJkj3QrqPxirvH0zt7iLBdPPNN/Od73yHTZs2cdNNN9HV1UVTUxPRaJTNmzezf//+037+qquuGhp8bPv27Wzbtg2A7u5uKioqqKmpoa2tbdQgZKcaavjKK6/khz/8If39/fT19fGDH/yAK6+8sohHe3pTv+Ze0eTNe9tprPJGhOzoSbFoelUJCyUipbB06VJ6enqYPXs2M2fO5EMf+hDvec97uPTSS1mzZg2LFy8+7ec//vGP85GPfITm5maam5tZvXo1AJdddhkrV65k8eLFzJ07l/Xr1w995vbbb2fjxo3MmjWLzZs3Dy1ftWoVt912G2vXrgXgox/9KCtXrpy0pztZqdqn16xZ48brP3pGdj8OD9wIf/IYexNLuPYL/8UXb1nBDStmn/u+ReSM7dy5k+bm5lIXw1fG+jc1s63OuTXjfXbcZhkzm2tmm81sh5m9bGafGGMbM7MvmdkeM9tmZqvG2teEqBy8kalNzTIiIgVn0iyTBf7cOfeCmVUBW83sMefcyJ+K3wksKkxXAF8pzCfeiGaZqniEWCSkcBeRwBu35u6cO+Sce6HwugfYCZzY5nED8G3neQaoNbOZRS/tWCoaAIPedsxMd6mKlJC6IRfPuf5bvqneMma2AFgJPHvCqtnAgRHvWzj5AjAxwlEoHzEEQaGvu4hMrkQiwdGjRxXwReCc4+jRoyQSibPexxn3ljGzSuB7wCedc91n82VmdjtwO8C8efPOZhdjq5wOvcPhfuBY/zgfEJFimzNnDi0tLXR0dJS6KL6QSCSYM2fOWX/+jMLdzKJ4wf6Ac+77Y2xyEJg74v2cwrJRnHP3AfeB11vmTZf2VCoaR4X7C/uPF23XInJmotEoCxcuLHUxpOBMessY8K/ATufcPafY7GHgjwq9ZtYBXc65Q0Us5+lVNo0aGfJYf5pMLj9pXy8icr45k5r7euAPgd+Z2YuFZZ8C5gE4574KPAJcB+wB+oGPFL+op3FCs4xzcKwvzfTqs2+vEhGZysYNd+fc08DYQ6gNb+OAO4pVqDetohEy/ZDqHdXXXeEuIkE19ceWgRGP29ONTCIi4JtwL9yl2tdBY6XCXUTEH+E+avCwQrirr7uIBJg/wn1Es0wiGqYqEVHNXUQCzR/hXj4NsOGHduguVREJOH+EezjijTEz+Lg9jS8jIgHnj3CHwrNUh2vuRxTuIhJg/gn3ysahmnuDau4iEnA+Cvfpo0aG7EllGUjnSlwoEZHS8E+4Dw4e5txQd8gj+lFVRALKP+FeOR2ySUj1DIV7u5pmRCSgfBTuhRuZdJeqiIgPw33E+DJqlhGRoPJPuFcMjy9TUxYFoGsgU8ICiYiUjn/CPVHrzQc6SUTDJKIhOvvTpS2TiEiJ+CfcywrhnuwEoLYsppq7iASWf8I9VgkWhoFCuJdH6exXuItIMPkn3M282nuh5l5TFqVTNXcRCSj/hDt47e7JLsCruXep5i4iAeWzcK8ZbpYpi9E5oB9URSSY/BXuI5pl1OYuIkHmr3BP1A7V3GvKo6SyeZIZDR4mIsHjr3AvG9HmXhYDUO1dRALJX+GeqPGaZZyjtty7S1Xt7iISRD4L91rIZyHdR21hCALV3EUkiPwV7iPuUq0pV7iLSHD5K9xHjC9TW+61uXepWUZEAshf4T5Uc+/SyJAiEmj+CvdEjTdPdlIRCxMJmZplRCSQfBbuw80yZubdyKSau4gEkL/C/YRhf2vKNL6MiASTv8I9XgPYiGF/Nb6MiASTv8I9FIJE9Yi7VDW+jIgEk7/CHYbvUsUbX0bhLiJB5MNwrx017K+6QopIEPkv3E8Y9rc3lSWTy5e4UCIik2vccDezb5hZu5ltP8X6DWbWZWYvFqZPF7+Yb8IJT2MC3cgkIsFzJjX3bwIbx9nml865FYXp7869WOdgxNOYajR4mIgE1Ljh7px7Cjg2CWUpjlHNMhpfRkSCqVht7m8xs5fM7FEzW1qkfZ6dRC1kk5BJDg37q2YZEQmaSBH28QIw3znXa2bXAT8EFo21oZndDtwOMG/evCJ89RhG3KVaW14FqFlGRILnnGvuzrlu51xv4fUjQNTMGk6x7X3OuTXOuTWNjY3n+tVjSwyPDKlH7YlIUJ1zuJvZDDOzwuu1hX0ePdf9nrURg4dVJSKYocHDRCRwxm2WMbMHgQ1Ag5m1AHcDUQDn3FeBDwAfN7MsMADc4pxzE1bi8YxolgmFjOpElK5+/aAqIsEybrg7524dZ/0/A/9ctBKdqxE1d0DD/opIIPnzDlUY7g6pwcNEJID8F+5DT2Py7lKtKY+p5i4igeO/cA9HIVoxYvAwtbmLSPD4L9zhpMHDVHMXkaDxZ7iPGvY3StdAhny+dB14REQmmz/Dvax2VJu7c9CTzJa4UCIik8ef4T7iaUyD48voWaoiEiQ+DffaUf3cQYOHiUiw+DPcT/hBFTS+jIgEiz/DPVEL6V7IZagZHDxMNXcRCRB/hvvQXardw80y6usuIgHiz3Afuku1k9qyKGbQ0atwF5Hg8Gm4Dw8eFgmHmF6V4FDnQGnLJCIyifwZ7kPNMscBmFWboLVL4S4iweHPcD9h2N+ZtWW0diZLWCARkcnlz3Avq/PmA17NfXZtGa2dA5TyGSIiIpPJn+Fe0QAWhp7DAMyqSZDK5jnWpx9VRSQY/BnuoTBUTh8O99oyADXNiEhg+DPcAapmQM8hYDjcD6rHjIgEhI/DfeZJNfdD6jEjIgHh43AfrrnXlUdJREO0quYuIgHh73AfOAbZFGbGrBp1hxSR4PB3uMOophm1uYtIUPg43Gd686FwT6jNXUQCw8fhXqi59w7X3Nt7UqSz+RIWSkRkcvg43E+oudeU4Ry0davdXUT8z7/hXlYPoaj6uotIIPk33EOhQnfI4TZ3UF93EQkG/4Y7jOrrPrNGQxCISHAEINy9mntZLEx9RUzNMiISCP4O98rhmjsUHtqhcBeRAPB3uFfNgGQXpPsBr8fMITXLiEgA+DzcC90hR/R1V81dRILA5+F+4hAECXpSWbqTmRIWSkRk4vk83E8cgmCwx4xq7yLibz4P95MHDwPU7i4ivufvcC+rg3B8+C7VGt2lKiLBMG64m9k3zKzdzLafYr2Z2ZfMbI+ZbTOzVcUv5lkyG9XXvbEqTjRstBxXuIuIv51Jzf2bwMbTrH8nsKgw3Q585dyLVURVM4dq7uGQcUFDJa+29ZS4UCIiE2vccHfOPQUcO80mNwDfdp5ngFozm1msAp6zETV3gCWzqtnR2l3CAomITLxitLnPBg6MeN9SWHYSM7vdzLaY2ZaOjo4ifPUZOCHcm2dWcbg7yfG+9OR8v4hICUzqD6rOufucc2ucc2saGxsn50urZkC6B1JeU0zzzGoAdh5S7V1E/KsY4X4QmDvi/ZzCsvPDUF/3NmA43Hco3EXEx4oR7g8Df1ToNbMO6HLOHRrvQ5PmhMftNVTGaaqKK9xFxNci421gZg8CG4AGM2sB7gaiAM65rwKPANcBe4B+4CMTVdizcsJdquDV3nceUo8ZEfGvccPdOXfrOOsdcEfRSlRsgzX37uGWoiWzqvn1L18jnc0Ti/j7Pi4RCSb/J1uiBiqaoOPVoUXNM6vJ5Bx72ntLWDARkYnj/3AHaGqG9peH3i6ZWQWox4yI+Fcwwn36UmjfBfk8AAsbKklEQ/pRVUR8Kxjh3tQM2QHo3Ad4wxBcMr1KNXcR8a2AhPsSb962Y2iR12OmG+/3YBERfwlGuDcu9ubtO4cWLZlVzfH+DIe7Nba7iPhPMMI9Xgm180f9qKphCETEz4IR7lD4UXW45r54xmCPGd3MJCL+E5xwb2qGI7shmwKgKhFl/rRyftfSVeKCiYgUX4DCfQm4nBfwBSvn1vLCG8f1o6qI+E6wwh1GNc2snFdHe0+K1i79qCoi/hKccJ92EYQio35UXTWvDoAX9h8vValERCZEcMI9EoOGi0f/qDqzikQ0xG/f6CxhwUREii844Q7ej6ojbmSKhkMsn+21u4uI+EnAwn0JdL0ByeG+7Svn1/JyaxfJTK6EBRMRKa7ghTtAx66hRavm1ZHJOV5u1c1MIuIfAQv3Zm/ePtw0s3JeLQC/VdOMiPhIsMK9dj7EquDQtqFFTVUJ5tSVqd1dRHwlWOEeCsGc1XDguVGLV86rU48ZEfGVYIU7wNwrvL7uI35UXTWvlkNdSQ51DZSwYCIixRPMcHd5OLhlaNHwzUyqvYuIPwQv3OdcDhaCN54dWtQ8s5p4JKR2dxHxjeCFe6IampbCgWeGFsUiIVbMreXXe4+WsGAiIsUTvHAHmLsWWrZAfvjGpWsWN7HzUDetnWp3F5GpL5jhPm8dpHuhbXgQsWubmwD4xa72UpVKRKRoghnuc6/w5geG290vbKxkXn25wl1EfCGY4V47DypnjAp3M+OaxU38as8RBtIaZ0ZEprZghrsZzLtiVI8Z8JpmUtk8v957pEQFExEpjmCGO8Dcdd4Ikd2tQ4uuWDiNiliYx3eqaUZEprbghvu8k9vdY5EQV13cyC92tem5qiIypQU33Gcsh0gZ7P/NqMXXLG6irTulIYBFZEoLbriHo7DwKnj1URhRS99wSRNm6hIpIlNbcMMdoPk90PkGHHppaFFjVZwVc2v5z22H1DQjIlNWsMP9kuvAwrDzx6MW33L5XF5p6+HZ14+VqGAiIucm2OFeMQ0WrD8p3G9YMZu68ij3/+r1EhVMROTcBDvcAZqvhyOvQMcrQ4sS0TC3rp3HYzvaOHCsv4SFExE5O2cU7ma20cxeMbM9ZnbXGOtvM7MOM3uxMH20+EWdIIvf5c13Pjxq8YfXzcfM+Pdn9pegUCIi52bccDezMPBl4J3AEuBWM1syxqbfdc6tKExfL3I5J071LG+M9xOaZmbVlrFx6QwefO4N+tPZEhVOROTsnEnNfS2wxzn3mnMuDXwHuGFiizXJmq/3eswcH11L/8j6BXQns/zgtwdLVDARkbNzJuE+Gzgw4n1LYdmJbjSzbWa2yczmFqV0k6X53d58109GLV49v45ls6v52lOvkcpqMDERmTqK9YPqj4EFzrnlwGPAt8bayMxuN7MtZralo6OjSF9dBPUXeHesvvBvkM8PLTYz/vIPFrPvaD9f/6V6zojI1HEm4X4QGFkTn1NYNsQ5d9Q5lyq8/TqweqwdOefuc86tcc6taWxsPJvyTpzfuxM6dp5Ue3/rxY1cd+kMvvTEbvWcEZEp40zC/XlgkZktNLMYcAswqmuJmc0c8fZ6YGfxijhJlr0fpl0ET/3DqOEIAD797qVEQsbdD7+su1ZFZEoYN9ydc1ngfwA/wwvth5xzL5vZ35nZ9YXN7jSzl83sJeBO4LaJKvCECYXhyj+Hw7+DV386atWMmgR/9vaL+cWudh7b0VaiAoqInDkrVU10zZo1bsuWLSX57lPKZeCf10BZHXxss/dQj4JMLs97/ulpOvszPPyn62mqSpSwoCISVGa21Tm3ZrztdIfqSOGoV3tv/S3seXzUqmg4xBc+eBldAxlu//ZWkhn1nhGR85fC/UTLb4GaefD4ZyCTHLVq6awa/vHmFbx4oJO/2rRN7e8ict5SuJ8oEoPr/g+0bYfHPn3S6o3LZvCXf3AJD7/Uyhef2F2CAoqIjE/hPpZLNsK6O+C5/ws7f3LS6v++4ULev2o29z6+my/8/BXV4EXkvBMpdQHOW2+7G/b/Cn50B8y8DGqHu/qbGf9w43KioRD/9Is9tHUn+ez7LiUa1rVSRM4PSqNTicThpvshn4PvfhgGjo9eHQ7xuRsv5RPXLuKhLS189FtbONaXLlFhRURGU7ifTv0FcOPXoX0HfPM90Dt6yAQz48/efjH/6/2X8uu9R/iDe59is569KiLnAYX7eC7ZCLc+CEf3wP3vhK6TR4i8de08fnjHeurLY3zkm8/z19//HZ39qsWLSOko3M/ERW+DP/wB9LbB198Grz150iZLZ9Xw8J+u579ddQHfff4NrvqHzRpNUkRKRuF+pua/BW77T4hVwLdvgJ9+6qR+8PFImL++rplHPnElK+fV8dlHdnLtF/6Lb/9mnx74ISKTSsMPvFnpfnj8bnjuPpi2CN7+t3DJdaOGKhj09O4jfP7nr/DigU5qyqJ86Ip53Lp2HnPry0tQcBHxgzMdfkDhfrb2PAGP/pXXFj/ncrj207DgyjFDfuv+Y9z31Gv8fEcbzsG6C+r5wOq5vGPpdKoT0RIUXkSmKoX7ZMhl4cUH4MnPQU+r98CPtbfDpR+AaNlJmx/sHOAHL7SwaWsL+472EwuHWH/RNDYum8HVi5s0GJmIjEvhPpkyA/DSg/Dc17xuk4laWHKDF/Lz13vDCY/gnOOFNzr56fZDPLr9MC3HBwBonlnNVRc38HsXNrB6fh2Vcd1jJiKjKdxLwTnY9zS88C3Y9Qhk+qByOix6hzddsAES1Sd8xLHjUDf/9WoHT73awdb9x8nkHOGQsXRWNavn17Fibi0r59Yxt74MG6PZR0SCQ+Feaul+76EfO34EezdDqgtCEZi10qvNL/h9r62+rHbUx/rTWV7Y38lzrx/lmdePsa2lk2TGe65rXXmUZbNrWDqrhuaZVSyeUc0FjRUa9kAkQBTu55NcBg48540Rv/9XcHAr5AtdIxsuhtlrYNYKmHEpTF82qnafzeV5pa2H377RyfaDXWxv7eKVwz1kct55i4aNCxsrubCpkosaK7moqZKFDRUsbKigQs06Ir6jcD+fpfug5fnCtBUOboG+EUMb1MyDpsXQuNgL/4ZFXrfL8nowI5XNsbe9j1fautl1uIc9bb3sbu/lwPH+UY9/baqKM39aOfPqK5hXX87c+jLm1JUzp66Mpqo4EdX4RaYchftU4hz0HPae33r4JWjfBR274MirkBsxjEG8BuoXelPdAqidB7XzvXn1bJIW5/Ujfew70sdrhfn+Y/0cONbPoa7RN1yFQ8aM6gSzahPMqCljZk2CGdUJZtQkmF4dZ3p1gqaqBLGILgAi5xOFux/kstC5H47uhaO7vfnx1+HY69B1YLhpZ1D5NKieDTVzoHoWVM0sTDNIlTVxKF/DG/0JWjqTtHYO0No1wMHjA7R1J2ntSpLO5k8qQn1FjKaqOI2DU2Wchso4DVUxGirjTKuI01AZo64iprZ/kUlwpuGuRtnzWTgC0y70Jt4xel0+Bz2H4Ph+6Grxwr6rxZs634A3fjNqmOI4sABYEI55PXgqGr1pRiNc2Igrn0ZfpJ6jrpL2fBWH0uUcTJXR0mu09aTp6E3xWkcfHT0p0rmTLwIANWVR6iti1JV789ryWGEepa7cW15b7r2vLfPmiWh4zH2JyLlRuE9VobBXQ6+Zc+pt0v3Qe9hr8uk5BD1t3vvedm/qaYVDL0H/ESyfpRKoBOaP3Ec4BmX1Xnv/9HrcgloysRr6wtX0UkEXlRzLV3A0W0Z7NsHhdILWZIyW43Febu3mWF+a1Bh/EQyKR0JDYV9TFqWmPOrNC1N1IkJNeZTqRJTqssF5hOpElPJYWF1DRU5B4e5nsXJvTPr6C06/nXOQ7IS+o9A/OB2B/mMwcKwwPw4Dx7Gje4klO4kNHKcum2Tu6fYbrYC6avLxarKRSlKRSlKhCvpDFfRRRo9L0JOP05lLcCwb41gqxpHeKB2pKDtTETrSMfqJ00+C/Bhj3IVDRmU8QlUiQmXcC/yqRITKhLesKhGlMh4ZmioK21bEI1TGw1QUllXEIoRDukiIvyjcxRsPp6zOm7jozD+XGYCBTu/CMDhPdnmvU91Dr0OpbmKpbmLJbqoGWiHdC6keb34qIWDEaAz5cIJspIxsuJx0qIyUJUhZnAES9BGnLxujtztGT2eM7lyU7myUrmyE9lyMAWIMECdJjKQbfB1lwBWWEcOicSpiUcrjYSpiEcpjXviXx7z3ZSPee1OEirg3H3w/uK5s8DPRMCFdNKREFO5y9qJl3lQ98+w+n8953UJHhn26D1K9w8vTvZDuJ5TuJZbuJZbupzzTV1jfD5nuwrwf8v3e63zG23+4MJ2hTD5GOhknnYqRLoR+iihJF6E/H2PARejPR7zlLkonMdqJkCJKynnbpoiSJkrKRUkTwYXjWDQOkQShSJxQLEE4miAcjRONx4lEE0RiZcTiCeKxOIm4d1Eoi3oXiUTUm4aXhYbfx8IkIrqAyNgU7lI6obB3w9YJQzKcs1zG+6si01+YF6ZsYZ7ug2xy9PJsimhmgGg2SUU26Y3Vnx2AbLrwuSRkk7hsHy4zgMumIJvCsgNYLo1xml5neSBdmPpPX/TBi4I3Rcm4sDcnQooIx4mQcZGhZWnC5C1CPhTFhaK4UAwX9uYWjni/mYRjhCJRQpEYFol5F5lIjFA0QTgaK1xsYkRihYtNNE4sFicSixOPx4nGyoglEsRiceLxMuKxqC4oU4DCXfwnHPWmYl80ACtMozjnXVCySe++hGwSCuE/9DqXKlwoBrcpLMtlhl67bIpwJkU0nSSUSRH0PDmyAAAGZ0lEQVTNpMlnkuSzaVw2hcumcbm0t59cGssNYPkMls8QymcIuSwhlyGS9l7HyBT9+AFyzkgSIUuErHnznI2e8hYlH4qSD0UKFx1vzuA8HINwFAtHsHB06AJk4SihoYtRlHAkRjgSIxSJEYlGiUSihCNRorFY4XXc61UWikBh/4TC3jw8+H7EFI4Orx/6jD+78CrcRc6VGURi3nQuu8H7H7Jo/1M65zV95TPeBSWXKUwp8pkU6XSKdCpJOjVANp0mk0mRTSfJZtJk0ylymST5bIpcJk0+myKfzeCy3mvy2cJFxntthf1bPoO5rHexyWUIZbOEXZKwyw5NEbKEyRH1LgtEyRXmWaI2+Y+ldBh5C5O3MM4i3jwUhsJ7N/KCYYMXjghWuEBYKIyFI4QG5+EooXDE+8vJwt5nB+eD+7nwGlj8rgk9LoW7iF+ZebXacOSk5wsM/l5dqicI5PKOdDZPKpsjmc2TyuZJ5/KkMznSmTSZtDdls0my6TTpTIZcJu1deLIZctkMuWyaXDZDPpcln02Ty2ULF6AMLp8tzL315LO4fNa7MTA/YnI5LJ/FclnMeReZCDnC5AmTI0KesHlz76LkCA+9zhAmRcRyhMh72w59Nk+ULGFzRMxbFyosD1uevUcirFa4i4jfhENGWaFn0fkin3feBSaXJzN4scnmyeS8C1E6lydbWJbK5UkOrsvlvPWFi1Qm58gMfXZ4P9nB5bk8b18yndUTfDwKdxERIBQyEqGwb+6a9ucvCSIiAadwFxHxIYW7iIgPKdxFRHxI4S4i4kMKdxERH1K4i4j4kMJdRMSHSvYMVTPrAPaf5ccbgCNFLM5UEcTjDuIxQzCPO4jHDG/+uOc75xrH26hk4X4uzGzLmTwg1m+CeNxBPGYI5nEH8Zhh4o5bzTIiIj6kcBcR8aGpGu73lboAJRLE4w7iMUMwjzuIxwwTdNxTss1dREROb6rW3EVE5DSmXLib2UYze8XM9pjZXaUuz0Qws7lmttnMdpjZy2b2icLyejN7zMx2F+Z1pS7rRDCzsJn91sx+Uni/0MyeLZzz75rZuT3P7jxjZrVmtsnMdpnZTjN7SxDOtZn9WeG/7+1m9qCZJfx4rs3sG2bWbmbbRywb8/ya50uF499mZqvO9nunVLibWRj4MvBOYAlwq5ktKW2pJkQW+HPn3BJgHXBH4TjvAp5wzi0Cnii896NPADtHvP/fwD865y4CjgN/UpJSTZwvAj91zi0GLsM7dl+fazObDdwJrHHOLQPCwC3481x/E9h4wrJTnd93AosK0+3AV872S6dUuANrgT3Oudecc2ngO8ANJS5T0TnnDjnnXii87sH7n3023rF+q7DZt4D3lqaEE8fM5gDvAr5eeG/ANcCmwia+Om4zqwGuAv4VwDmXds51EoBzjfckuDIziwDlwCF8eK6dc08Bx05YfKrzewPwbed5Bqg1s5ln871TLdxnAwdGvG8pLPMtM1sArASeBaY75w4VVh0GppeoWBPpXuCvgHzh/TSg0zmXLbz32zlfCHQA9xeaor5uZhX4/Fw75w4CnwfewAv1LmAr/j7XI53q/BYt46ZauAeKmVUC3wM+6ZzrHrnOed2cfNXVyczeDbQ757aWuiyTKAKsAr7inFsJ9HFCE4xPz3UdXi11ITALqODkpotAmKjzO9XC/SAwd8T7OYVlvmNmUbxgf8A59/3C4rbBP9EK8/ZSlW+CrAeuN7N9eE1u1+C1R9cW/nQH/53zFqDFOfds4f0mvLD3+7l+G/C6c67DOZcBvo93/v18rkc61fktWsZNtXB/HlhU+EU9hvcDzMMlLlPRFdqZ/xXY6Zy7Z8Sqh4E/Lrz+Y+BHk122ieSc+2vn3Bzn3AK8c/sL59yHgM3ABwqb+eq4nXOHgQNmdklh0bXADnx+rvGaY9aZWXnhv/fB4/btuT7Bqc7vw8AfFXrNrAO6RjTfvDnOuSk1AdcBrwJ7gb8pdXkm6Bh/H+/PtG3Ai4XpOrz25yeA3cDjQH2pyzqB/wYbgJ8UXl8APAfsAf4fEC91+Yp8rCuALYXz/UOgLgjnGvhbYBewHfg3IO7Hcw08iPe7QgbvL7U/OdX5BQyvR+Be4Hd4vYnO6nt1h6qIiA9NtWYZERE5Awp3EREfUriLiPiQwl1ExIcU7iIiPqRwFxHxIYW7iIgPKdxFRHzo/wNj+9D6MIBn0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f65c0633518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train_vs_validation_loss(mlp_train_loss, mlp_eval_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VPW9//HXJyH7vu87ARLWQNgFsaDivlet1rVSrf607fXea/3d1tZb76+rV9tat6rdXIq4UTfqAooiSFgFAoQkhCyQnZB9/f7+OBMICGSSTDJh5vN8POYxc/ZvhuF9zvme7/keMcaglFLKPXg4uwBKKaVGjoa+Ukq5EQ19pZRyIxr6SinlRjT0lVLKjWjoK6WUG9HQV0opN6Khr5RSbkRDXyml3MgYZxfgRJGRkSY1NdXZxVBKqTPKpk2baowxUf3NN+pCPzU1lby8PGcXQymlzigiUmLPfFq9o5RSbkRDXyml3IhdoS8iS0Vkj4jsE5EHTjPfVSJiRCTXNpwqIq0istX2espRBVdKKTVw/dbpi4gn8ARwLlAGbBSRlcaYXSfMFwTcB2w4YRWFxphpDiqvUuoM09nZSVlZGW1tbc4uikvw9fUlMTERLy+vQS1vz4XcWcA+Y0wRgIi8AlwG7Dphvv8Gfgn8+6BKopRySWVlZQQFBZGamoqIOLs4ZzRjDLW1tZSVlZGWljaoddhTvZMAlPYZLrONO0pEpgNJxph3TrJ8mohsEZFPRGTBoEqplDpjtbW1ERERoYHvACJCRETEkM6ahtxkU0Q8gEeBW04y+SCQbIypFZEZwJsiMtEYc+SEdSwDlgEkJycPtUhKqVFGA99xhvpd2nOkXw4k9RlOtI3rFQRMAtaIyH5gDrBSRHKNMe3GmFoAY8wmoBAYd+IGjDHPGGNyjTG5UVH93ltwUg2tnTz24V62lR4e1PJKKeUO7An9jUCmiKSJiDdwHbCyd6IxpsEYE2mMSTXGpALrgUuNMXkiEmW7EIyIpAOZQJHD/wrAQ+CxDwv4oqh2OFavlDpDHT58mD/+8Y8DXu7CCy/k8GHXO4jsN/SNMV3APcAqIB9YbozZKSIPi8il/Sy+ENguIluBFcCdxpi6oRb6ZIJ8vQgP8KaktmU4Vq+UOkOdKvS7urpOu9y7775LaGjocBXLaeyq0zfGvAu8e8K4n5xi3kV9Pr8GvDaE8g1Icrg/B+qaR2pzSqkzwAMPPEBhYSHTpk3Dy8sLX19fwsLC2L17N3v37uXyyy+ntLSUtrY27rvvPpYtWwYc6xKmqamJCy64gLPOOot169aRkJDAW2+9hZ+fn5P/ssEZdX3vDEVqhD95JfXOLoZS6hR+9s+d7Ko40v+MA5AdH8xDl0w85fRf/OIX7Nixg61bt7JmzRouuugiduzYcbTJ4/PPP094eDitra3MnDmTq666ioiIiOPWUVBQwMsvv8yzzz7LN7/5TV577TVuvPFGh/4dI8WlumFIjgig4nArHV09zi6KUmqUmjVr1nFt3H/3u98xdepU5syZQ2lpKQUFBV9bJi0tjWnTrHtMZ8yYwf79+0equA7nUkf6KeH+9Bgoq28hPSrQ2cVRSp3gdEfkIyUgIODo5zVr1vDhhx/yxRdf4O/vz6JFi07aBt7Hx+foZ09PT1pbW0ekrMPBpY70UyL8ASip04u5SilLUFAQjY2NJ53W0NBAWFgY/v7+7N69m/Xr149w6UaeSx3pJ9tC/4C24FFK2URERDB//nwmTZqEn58fMTExR6ctXbqUp556iqysLMaPH8+cOXOcWNKR4VKhHxXog7+3J/trtQWPUuqYl1566aTjfXx8eO+99046rbfePjIykh07dhwdf//99zu8fCPJpap3RMRqtqlH+kopdVIuFfpg1etrnb5SSp2cC4Z+AAfqWujpMc4uilJKjTouGPr+dHT1cOiIPrBBKaVO5HqhH261wdU+eJRS6utcL/R7m21qHzxKKfU1Lhf6cSG+jPEQPdJXSg1KYKB1N39FRQVXX331SedZtGgReXl5p13PY489RkvLsRwaLV01u1zoj/H0ICncX0NfKTUk8fHxrFixYtDLnxj6o6WrZpcLfbC6WC7R6h2lFFbXyk888cTR4Z/+9Kf8/Oc/Z/HixUyfPp3Jkyfz1ltvfW25/fv3M2nSJABaW1u57rrryMrK4oorrjiu75277rqL3NxcJk6cyEMPPQRYnbhVVFRwzjnncM455wBWV801NTUAPProo0yaNIlJkybx2GOPHd1eVlYWd9xxBxMnTuS8884blj5+XOqO3F4pEf5sPlCPMUafzanUaPLeA3DoK8euM3YyXPCLU06+9tpr+f73v8/dd98NwPLly1m1ahX33nsvwcHB1NTUMGfOHC699NJT5sWTTz6Jv78/+fn5bN++nenTpx+d9sgjjxAeHk53dzeLFy9m+/bt3HvvvTz66KOsXr2ayMjI49a1adMmXnjhBTZs2IAxhtmzZ3P22WcTFhY2Il04u+yRfmNbF/Utnc4uilLKyXJycqiqqqKiooJt27YRFhZGbGwsDz74IFOmTGHJkiWUl5dTWVl5ynV8+umnR8N3ypQpTJky5ei05cuXM336dHJycti5cye7du06bXk+++wzrrjiCgICAggMDOTKK69k7dq1wMh04eySR/qpEb3NNpsJD/B2cmmUUked5oh8OF1zzTWsWLGCQ4cOce211/Liiy9SXV3Npk2b8PLyIjU19aRdKvenuLiY3/zmN2zcuJGwsDBuueWWQa2n10h04eySR/rHmm3qxVyllFXF88orr7BixQquueYaGhoaiI6OxsvLi9WrV1NSUnLa5RcuXHi007YdO3awfft2AI4cOUJAQAAhISFUVlYe13nbqbp0XrBgAW+++SYtLS00NzfzxhtvsGDBAgf+tafnkkf6SeG2fvW1BY9SCpg4cSKNjY0kJCQQFxfHDTfcwCWXXMLkyZPJzc1lwoQJp13+rrvu4tZbbyUrK4usrCxmzJgBwNSpU8nJyWHChAkkJSUxf/78o8ssW7aMpUuXEh8fz+rVq4+Onz59OrfccguzZs0C4Dvf+Q45OTkj9jQuMWZ09VGTm5tr+mv/ao85//MR88ZG8Og3pzmgVEqpwcrPzycrK8vZxXApJ/tORWSTMSa3v2VdsnoHrCoe7WJZKaWOZ1foi8hSEdkjIvtE5IHTzHeViBgRye0z7ke25faIyPmOKLQ90qMCKahqYrSdySillDP1G/oi4gk8AVwAZAPXi0j2SeYLAu4DNvQZlw1cB0wElgJ/tK1v2GXHBdHQ2snBBu1tUyln04Mvxxnqd2nPkf4sYJ8xpsgY0wG8Alx2kvn+G/gl0DdlLwNeMca0G2OKgX229Q27rLhgAPIPHhmJzSmlTsHX15fa2loNfgcwxlBbW4uvr++g12FP650EoLTPcBkwu+8MIjIdSDLGvCMi/37CsutPWDbhxA2IyDJgGUBycrJ9Je/HhD6hvzgrpp+5lVLDJTExkbKyMqqrq51dFJfg6+tLYmLioJcfcpNNEfEAHgVuGew6jDHPAM+A1XpnqGUCCPQZQ3K4P/kHv95OVik1cry8vEhLS3N2MZSNPaFfDiT1GU60jesVBEwC1tj6rYgFVorIpXYsO6yy4oK0ekcppfqwp05/I5ApImki4o11YXZl70RjTIMxJtIYk2qMScWqzrnUGJNnm+86EfERkTQgE/jS4X/FKWTFBVNc20xLR9dIbVIppUa1fkPfGNMF3AOsAvKB5caYnSLysO1o/nTL7gSWA7uA94G7jTHdQy+2fbLigjEGdh/SKh6llAI76/SNMe8C754w7ienmHfRCcOPAI8MsnxDkt3nYu705DBnFEEppUYVl70jFyAxzI8gnzFar6+UUjYuHfoiQlZcsLbgUUopG5cOfbBa8Ow+eISeHr0xRCml3CD0g2nu6Ka0XjtfU0optwh90O4YlFIK3CD0x8cG4SGwq0JDXymlXD70fb08SYsMYJdezFVKKdcPfcDWgkeP9JVSyi1CPzs+mPLDrTS0djq7KEop5VRuEfq9F3O1Xl8p5e7cIvSnJYYCsKmkzsklUUop53KL0A8L8GZcTCBf7q93dlGUUsqp3CL0AWamhrO5pJ5uvTNXKeXG3Cb0Z6WF09Tepa14lFJuzW1Cf2ZqOAAb92u9vlLKfblN6MeH+pEQ6qehr5Rya24T+gAzU8P4srgeY7ReXynlntwr9NPCqWlqZ3+t9riplHJPbhX6s7ReXynl5twq9MdGBxLm78XGYg19pZR7cqvQFxFyU8P1SF8p5bbsCn0RWSoie0Rkn4g8cJLpd4rIVyKyVUQ+E5Fs2/hUEWm1jd8qIk85+g8YqJmpYeyvbaGqsc3ZRVFKqRHXb+iLiCfwBHABkA1c3xvqfbxkjJlsjJkG/Ap4tM+0QmPMNNvrTkcVfLB62+vnaZcMSik3ZM+R/ixgnzGmyBjTAbwCXNZ3BmNM39tcA4BR2yZyUkIIfl6efKn1+kopN2RP6CcApX2Gy2zjjiMid4tIIdaR/r19JqWJyBYR+UREFgyptA7g5elBbmoYnxZUO7soSik14hx2IdcY84QxJgP4T+C/bKMPAsnGmBzgh8BLIhJ84rIiskxE8kQkr7p6+MN48YRoiqqbKa5pHvZtKaXUaGJP6JcDSX2GE23jTuUV4HIAY0y7MabW9nkTUAiMO3EBY8wzxphcY0xuVFSUvWU/XnsTbH0Zagr6nXVxVgwAH++uGty2lFLqDGVP6G8EMkUkTUS8geuAlX1nEJHMPoMXAQW28VG2C8GISDqQCRQ5ouBf09UGb94J+f/sd9akcH/GxQTy8e7KYSmKUkqNVmP6m8EY0yUi9wCrAE/geWPMThF5GMgzxqwE7hGRJUAnUA/cbFt8IfCwiHQCPcCdxpjhuYIaEAmR4+DAF3bN/o0JMfxpbRGNbZ0E+XoNS5GUUmq06Tf0AYwx7wLvnjDuJ30+33eK5V4DXhtKAQckeS7sfBN6esDj9Ccxi7OieeqTQtYW1HDh5LgRKqBSSjmXa92RmzwX2hugale/s+YkhRLq78WH+VrFo5RyH64V+ilzrXc7qnjGeHqwaFwUa/ZU6yMUlVJuw7VCPzQFguLsrtdfnBVDXXMHW0sPD3PBlFJqdHCt0BexqnhKvgA7HpSycFwUnh6irXiUUm7DtUIfrNBvrIDDB/qdNcTPi5mpYXyUr+31lVLuwfVC/2i9/nq7Zl88IYbdhxopqdW7c5VSrs/1Qj86G3yC4cA6u2a/aEocIvDGltPdZKyUUq7B9ULfwxOSZtt9pB8f6se8jAje2FKuD0xXSrk81wt9gOQ5UL0bWuy7+feKnERKalvYfED72FdKuTbXDP2Ueda7nUf7SyfF4uflyWubtYpHKeXaXDP046eDp7fd9fqBPmNYOimWt7dV0NbZPcyFU0op53HN0PfytYK/xL7QB7giJ4EjbV3a3bJSyqW5ZugDZJwD5Zuhyb4Qnz82kuggH17XKh6llAtz3dAffyFgYO/7ds3u6SFcnpPAmj1V1Da1D2/ZlFLKSVw39GMnQ0gy7H63/3ltrpyeQFeP4a2tFcNYMKWUch7XDX0RGH8BFK2GDvvutp0QG8zUpFD+tr6EHu15Uynlglw39AEmXGg9RrFwtd2L3DY/leKaZtbs1Qu6SinX49qhnzIffENgj/1VPBdOjiM22JfnPisexoIppZRzuHboe3pB5vnWxdwe+9rfe3l6cNO8FD7fV8vuQ0eGuYBKKTWyXDv0wariaamF0g12L3L9zGR8vTx44bP9w1cupZRyAtcP/bFLrLtzd79j9yJhAd5cOT2RN7aWa/NNpZRLcf3Q9wmCtIVWvf4AetG8bX4qHV09vLSh/4exKKXUmcKu0BeRpSKyR0T2icgDJ5l+p4h8JSJbReQzEcnuM+1HtuX2iMj5jiy83cZfCHVFVs+bdhobHcTZ46L4yxcl2h+PUspl9Bv6IuIJPAFcAGQD1/cNdZuXjDGTjTHTgF8Bj9qWzQauAyYCS4E/2tY3srIuAY8xsPWlAS32vUUZ1DS18/f1JcNUMKWUGln2HOnPAvYZY4qMMR3AK8BlfWcwxvRt5hIA9NajXAa8YoxpN8YUA/ts6xtZgdEwbilsexm6O+1ebHZ6BGeNjeTJNYU0t3cNYwGVUmpk2BP6CUBpn+Ey27jjiMjdIlKIdaR/70CWHRHTb4Lmarv74un1g3PHUdvcwV++2D8sxVJKqZHksAu5xpgnjDEZwH8C/zWQZUVkmYjkiUhedXW1o4p0vIzFEBQHm/82oMVmpISxaHwUz3xaRGOb/WcJSik1GtkT+uVAUp/hRNu4U3kFuHwgyxpjnjHG5BpjcqOiouwo0iB4joFp34J9H8CRgXWo9sNzx3G4pZMXPt8/PGVTSqkRYk/obwQyRSRNRLyxLsyu7DuDiGT2GbwIKLB9XglcJyI+IpIGZAJfDr3Yg5RzI5ge2PrigBabkhjKkqwYnl1bREOLHu0rpc5c/Ya+MaYLuAdYBeQDy40xO0XkYRG51DbbPSKyU0S2Aj8EbrYtuxNYDuwC3gfuNsY4r/1jeDqkLoAtf4eengEt+sNzx9HU3sXvPy7of2allBqlxAzghqWRkJuba/Ly8oZvA9uXw+t3wE0rIf3sAS36wGvbWbGpjPfuW0BmTNAwFVAppQZORDYZY3L7m8/178g9UdYlVs+bm14Y8KL/fv54/L09+ek/dzLadpZKKWUP9wt9Lz+YcQvsegtqCwe0aESgD/efP57P99Xy3o5Dw1M+pZQaRu4X+gBz7gYPL/j88QEvesPsFLLjgvn527to6dAbtpRSZxb3DP2gGKslz7aXB9x809NDePiyiVQ0tPGHj/cNUwGVUmp4uGfoA8y/13qwyhdPDHjR3NRwrpqeyDOfFrGjvGEYCqeUUsPDfUM/LBUmXw15L0BL3YAX/8nF2YQHePNvy7fR3qW9cCqlzgzuG/oAZ/0AOpvhy2cGvGiIvxe/vGoKeyobefxDbbuvlDozuHfoR2fB+Itg/ZPQNvBqmnMmRPPN3ESe+qSQLQfqh6GASinlWO4d+gBn/4cV+Gt/O6jF/+vibGKDffm3V7fR2qHVPEqp0U1DP36a1RHb+ietp2sNULCvF7++ZirFNc08tHLHMBRQKaUcR0Mf4Bs/ttrtf/CTQS0+f2wk95wzluV5ZbyaV9r/Akop5SQa+gDBcbDgB5D/TyheO6hVfH/JOOZlRPDjt3aw+9CR/hdQSikn0NDvNfceCEmG939ktd8fIE8P4fHrcgj29eJ7f9+sD1xRSo1KGvq9vPzg3J9B5Vew6c+DWkVUkA+/vz6HkroW7n91Gz092imbUmp00dDva+IVVn/7HzwEhwdXNz87PYIHL8xi1c5KfvOvPQ4uoFJKDY2Gfl8icNkfrKdr/fM+GGT3ybfNT+X6Wcn8cU0hKzaVObiQSik1eBr6JwpLtap5Cj+ynrA1CCJWp2zzMiL40evb+bJ44N08KKXUcNDQP5nc2yHlLFj1IDQM7kjdy9ODJ2+YQVKYP9/9Wx77qpocXEillBo4Df2T8fCwqnl6umDlvQN+nm6vEH8vnr9lJp4eHnz7uQ2U1bc4uKBKKTUwGvqnEp4G5/23Vc2zbuAPW+mVGhnAX2+bRXN7F99+7ktqmtodWEillBoYDf3Tyb3datHz0cOw/7NBryY7PpgXbp3JwYZWbnruSxpatQ2/Uso5NPRPRwQu/T2Ep8OK26CxctCrmpESztPfzqWgqpGbnttAQ4sGv1Jq5NkV+iKyVET2iMg+EXngJNN/KCK7RGS7iHwkIil9pnWLyFbba6UjCz8ifILgm3+FtiPw2u2Dulu319njonjyhhnkH2zkW39aT31zhwMLqpRS/es39EXEE3gCuADIBq4XkewTZtsC5BpjpgArgF/1mdZqjJlme13qoHKPrJiJcNFvYf9aWPV/h7SqJdkxPH3TDAqqmrj+2fXUah2/UmoE2XOkPwvYZ4wpMsZ0AK8Al/WdwRiz2hjT2zRlPZDo2GKOAjk3wJzvwYYnYf1TQ1rVOeOjef7mmeyvbebaZ9ZTcbjVQYVUSqnTsyf0E4C+fRKU2cadyu3Ae32GfUUkT0TWi8jlJ1tARJbZ5smrrq62o0hOct7PYcLF8P4DsPudIa3qrMxI/nzrLCob2rjqyXXsrWx0UCGVUurUHHohV0RuBHKBX/cZnWKMyQW+BTwmIhknLmeMecYYk2uMyY2KinJkkRzLwxOufBbic2DF7VC+aUirm5MewT++O5fuHsPVT64jb7/euauUGl72hH45kNRnONE27jgisgT4v8ClxpijFdXGmHLbexGwBsgZQnmdz9sfvvUPCIyGF6+Byl1DWl12fDCv3TWPyCAfbvjTBt7eXuGggiql1NfZE/obgUwRSRMRb+A64LhWOCKSAzyNFfhVfcaHiYiP7XMkMB8YWkqOBoHR8O03wNMb/nIJVO0e0uqSwv1Zcec8JieEcM9LW3j0g73aLbNSalj0G/rGmC7gHmAVkA8sN8bsFJGHRaS3Nc6vgUDg1ROaZmYBeSKyDVgN/MIYc+aHPkBEBtz8tlXl85dLoHrvkFYXHuDNi3fM5poZifzuowLufmkzLR1dDiqsUkpZxAyy++Dhkpuba/Ly8pxdDPtV74U/XwTiATe9BdEThrQ6YwzPfVbM/7ybz7iYIJ68cQZpkQEOKqxSylWJyCbb9dPT0jtyhypqHNy8EjDwwlIoG9oOS0T4zoJ0Xrh1FoeOtHHJ7z/jva8OOqasSim3p6HvCNFZcNsq8A2Fv1wK+z4a8irPHhfFO/cuICM6kLte3MzD/9xFR9fgevtUSqleGvqOEp5mBX94Orx0LWxfPuRVJoT68ep353Lz3BSe/7yYK5/8XPvlV0oNiYa+IwXFwC1vQ9JseP0O+PiRQffF38t7jAc/u2wST397BuX1rVz8+7W8tOEAo+1ajFLqzKCh72h+oVZzzmk3wqe/ghW3QMfQH55y/sRY3v/+QnJTwnnwja+44695VB1pG3p5lVJuRUN/OIzxtp68dd7PYddK6wJvfcmQVxsT7Mtfb5vFjy/OZm1BDef+76e8uaVcj/qVUnbT0B8uIjDv/8D1r0BdMTy9EPa8P+TVengIt5+Vxrv3LSAjKoDv/2Mr3/3bJir1qF8pZQcN/eE2fil89xMITYaXr4UPHoLuod90lREVyKt3zuPBCyfwyd5qlvz2E/62vkTv5FVKnZaG/kgIT4fbP4AZt8Dnj8GfL4T6/UNeraeHsGxhBv/6wUKmJIXw4zd3cM3TX7DnkPbYqZQ6OQ39keLlC5c8Dlc9B1X58NQC2P6qQ1adEhHA32+fzW+vmUpRdRMX/m4tj7yzi6Z27cZBKXU8Df2RNvlquPMz64au178Dr90BrfVDXq2IcNWMRD7+t0V8MzeRZ9cWs/i3a1i5rUIv9CqljtLQd4awFLjlXVj0I9j5OjwxG3a/65hVB3jz/66cwhvfm0dkoA/3vryFa576gm2lhx2yfqXUmU07XHO2g9vgzbuh8iuYfA0s/SUERDhk1d09hlfzSvnNv/ZQ09TBldMTuP+88cSH+jlk/Uqp0cPeDtc09EeDrg747FH49NfgE2S17592g9Xs0wEa2zr545pCnltbDAK3zkvle4vGEuLv5ZD1K6WcT0P/TFSVD2//AA58ASnz4aJHh9xVc19l9S08+sFe3thSTpDPGO5aNJab56Xg7z3GYdtQSjmHhv6ZqqcHtv4d/vVj6GiC2XfC2f8JvsEO20T+wSP86v3drN5TTWSgN3ctGssNs5Px9fJ02DaUUiNLQ/9M11wDHz0Mm/9qPZ5xyc9gyrXg4bhr75tK6vjtv/ayrrCWmGAfli3M4FuzkvHz1vBX6kyjoe8qyjfBu/8B5XkQOwXO/RlkfMOhm1hXWMPjHxawobiOiABvbjsrjW/PTSHYV+v8lTpTaOi7kp4e2LECPv5vOHwA0s+BJQ9BfI5DN5O3v44/rN7Hmj3VBPqM4YbZydx2Vhoxwb4O3Y5SyvE09F1RVzts/JPVyqe1HiZcbLX1j53k0M3sKG/g6U+LeGd7BZ4ewuXTErhjYTrjYoIcuh2llONo6LuytgZY/yR88QS0H4Hsy2Hhvzs8/EvrWnh2bRHL80pp6+zh7HFRLFuYzryMCMRBzUmVUo7h0NAXkaXA44An8CdjzC9OmP5D4DtAF1AN3GaMKbFNuxn4L9usPzfG/OV029LQH4DWelj3B9jwNHQ0wvgLYcH9kDjDoZupb+7g7+tL+MsXJdQ0tTMhNohb56dy2bQEbfGj1CjhsNAXEU9gL3AuUAZsBK43xuzqM885wAZjTIuI3AUsMsZcKyLhQB6QCxhgEzDDGHPKzmY09AehtR42PAMbnrQ+py6AefdC5rkOu8ELoK2zm5VbK3j+82J2H2okzN+L62clc+OcFL3LVyknc2TozwV+aow53zb8IwBjzP87xfw5wB+MMfNF5HqsHcB3bdOeBtYYY14+1fY09IegvRHyXrCqfhorICoL5t5tde/g5biLscYY1hfV8cLnxXyYXwnAedmx3DQvhbnpWvWjlDPYG/r23IqZAJT2GS4DZp9m/tuB906zbIId21SD4RME8++1buja+Tp8/jtYeQ98+FPIvRVmfgeCYoe8GRFhbkYEczMiKK1r4cUNB3hl4wHe33mIzOhAvj03hStyEgjSJp9KjToO7WVTRG7Eqsr59QCXWyYieSKSV11d7cgiuacx3jD1Orjrc7jpLUicCZ/+Bv53Eqy4HUq/BAddwE8K9+eBCyaw/keL+fXVU/Dz9uQnb+1k9v98xINvfMWO8gaHbEcp5RgOq94RkSXA74GzjTFVtnFavTNa1BbCl8/C1hetFj9xU60j/0lXgXeAQze1rfQwf/2ihLe3V9De1cPkhBCum5XEpVPj9ehfqWHiyDr9MVgXchcD5VgXcr9ljNnZZ54cYAWw1BhT0Gd8ONbF2+m2UZuxLuTWnWp7GvrDrL0Jtv/D2gFU54NPsHVWMONWiMl26KYaWjt5a2s5L204wO5Djfh5eXLRlDium5nEjJQwrftXyoEc3WTzQuAxrCabzxtjHhGRh4E8Y8xKEfl22R/pAAASYElEQVQQmAwctC1ywBhzqW3Z24AHbeMfMca8cLptaeiPEGPgwHrIex52vQndHVY1UM63YdKV1vUBh23KsK2sgX9sLGXl1nKaO7pJjwzgqhmJXDk9gbgQbfmj1FDpzVnKfs21sO1l2PI3qN4NXgEw8XKY9i1InufQTt5aOrp4Z/tBXt1UxpfFdXgIzB8byeXTEjh/UiyBPtrNs1KDoaGvBs4YKNto9ey5803rhq/QFKv6Z9LVEDXOoZsrqW3mtU1lvL6lnLL6Vny9PFiSFcNl0xI4e1wU3mP0aZ5K2UtDXw1NRzPkv21d+C3+FDAQO9kK/4lXWM/5dRBjDJsP1PPmlgre3l5BfUsnIX5eXDAplkunxjM7PQJPD63/V+p0NPSV4xw5CDvfsHr6LN9kjYvPgezLrFd4usM21dndw2cFNazcVsGqnYdo6egmKsiHiybHccnUOHKSwvDQHYBSX6Ohr4ZHXTHsest6VWy2xkVnWz1+Zl1s9fnvoFY5rR3drN5TxcqtFXy8p4qOrh5ig305f2IMF0yOY2ZquJ4BKGWjoa+GX30J7H4Hdr9tPdfX9EBQPIw7DzLPh/SzHXYPQGNbJx/mV/LeV4f4ZG817V09hAd4syQrmvMnxjJ/bKR2/qbcmoa+GlnNNbD3fdi7Cgo/tp7v6+kDqWfBuPOtzt8cVA3U3N7Fmj3VrNp5iNW7q2hs78Lf25MFmZEsyYrhGxOiiQj0cci2lDpTaOgr5+nqgJLPoeADKFgFtfus8eEZMHaJ9Uqd75CzgI6uHtYV1vBhfiUf7qri0JE2RGBaUijfGB/NOROimRgfrDeCKZenoa9Gj9pCawdQ+BEUr4WuVvD0hqTZkHGO9fjHuKngMbTqGWMMOyuO8GF+Jat3V7GtzOr3JzrIh0Xjo1g0PpqzMiP12b/KJWnoq9Gpsw0OrIPC1dar8itrvG8opC2A9EWQdjZEjB3yBeHqxnbW7KlizZ5qPi2oprGtC08PIScplLPHRbFwXBSTE0K0NZByCRr66szQVAVFn0DxGihcA0fKrPGBsZC20KoGSpk/5J1AV3cPmw8c5pO9VXy6t4avbL1/hgd4syAzkkXjo1iQGUWkXgtQZygNfXXmMQbqiqybwfavtd6bbV1tB0RB8hxInmu9x04Bz8FX09Q2tbO2oIZP91bzyd5qaps7AJiUEMzCTGsHMCMlTO8KVmcMDX115jPGughcss56HVgHhw9Y07z8IWGGtQNImgNJM8E3ZFCb6emxrgWs2VPF2oIaNh+op6vHEODtydyMCBZkRnFWZiTpkQF6QViNWhr6yjUdqbB6By3dYL0f+gpMNyAQNR4ScyFxlrVDiM4a1MXhxrZO1hXW8llBDWsLqtlf2wJAXIgv8zIimT82gvljI4kJdtwjKJUaKg195R7am6yuIUq/tDqLK9sIrbbHNXgFQPw0q8uIhOnWe1jagK8NHKht4bN9NXy+r4Z1hTXUt3QCkBEVwPyxkczLiGBOegSh/t6O/uuUspuGvnJPvdcFyjdZr7I862ygu92a7htqNQ+NnwZx06z3AewIenoMuw4eYV1hDesKa/myuI6Wjm5EYGJ8MHPTrWcHz0wN16eEqRGloa9Ur64O6ylhFVugfDMc3AqVu6DHOmLHJwTiplg7g7ipVm+iEZng2X/f/h1dPWwvO8y6wlrWFdawueQwHd09eAhMTghhTnoEs9LCyU0NJ8RPdwJq+GjoK3U6Xe1QtQsObjv2OrTj2BnBGF+rI7nYSRAz2doRxEwE3+DTrrats5vNB+pZX1TH+sJatpZaOwERyIoNZlZaOLPSwpmZGk5UkDYPVY6joa/UQHV3Qc1eqzro0Hbb+1fHrhGA9VCZ2MkQM8naCcRMtKqHTvF0sbbObrYcOMyGYqsqaPOBeto6ewBIjfAnNzWcmalhzEgJJyNKWwepwdPQV8oRjIHGg8d2AJU7rDOCukKrV1Gwmo9GTbDODKInQFSW9R6c8LVrBZ3dPewob+DL4jrySurJ21939MJwmL8XM1KsHcCstDAmJYTgM0Z7DlX20dBXajh1tFjPE67cAVX5VlVR5S5orjo2j3eQ1Yw0aoL1qMmoCRA5DkKTjzYlNcZQWN3MppI68vbXs6mknqKaZgB8xngwNSmUmalh5KaEMz0lTK8LqFPS0FfKGZprrZ3B0dce69V06Ng8Y3ytbiUiMyFyvO090xrnHUBNUzt5+62zgI3769hRcYTuHoMIjIsOYnpKKDnJYUxPDtMqIXWUhr5So0lrvRX+NXutV7Xt/XDJsWoisB5CEznW2gFEjIXwDFqDU9jaGEJeaTN5JfVsOVDPkbYuAEL8vJieHMr05DCmp4QxJTFEm4q6KYeGvogsBR4HPIE/GWN+ccL0hcBjwBTgOmPMij7TugFbV4ocMMZcerptaegrt9LZZnU1UbsPagugxvZeWwhth4/NJ54QmgThGZiwNGq948lvj2BjQygfV/qzo9q6LtB7NpCTHMq0pFCmJYeSGR2kj5V0Aw4LfRHxBPYC5wJlwEbgemPMrj7zpALBwP3AyhNCv8kYE2hvwTX0lbJpqYOaAutms7pCa0dQVwT1xdDWcNysPYGxHPFLokJi2N0eTl5DCPntkRSZODq9Q5iUEMLUxBAmJ4YyJSGElAh/rRZyMfaGfv93n8AsYJ8xpsi24leAy4CjoW+M2W+b1nOyFSilBsE/HJJnW68TtdRZD6mvL4a6YjzqiwmtKyb08Bayj1RwJQZstwE0e4ZQWh3PrvIodq+L5T0TR5V3IiHx4xmfHMPkhBAmJ4SQGOanOwI3YE/oJwClfYbLgJP8Ck/JV0TygC7gF8aYNwewrFLqZPzDrVfijK9P62yzeiO1nR0E1BYwobaQ8bUFSOOnx+argIPl4RT3xLLWxHJwTDyEZxCUMIH4tGyykyJJjQjQh8y4GHtCf6hSjDHlIpIOfCwiXxljCvvOICLLgGUAycnJI1AkpVyYl6+tiei440YLWB3U1e6z7RCKiK4pILhyL9PrN+Pb+THUAXXQvV0oN5F8IXE0+idDeBoBceOJTc0iZWw23r7+zvjLlAPYE/rlQFKf4UTbOLsYY8pt70UisgbIAQpPmOcZ4Bmw6vTtXbdSaoB8Am09j04DrJYZRx9P31oPtYV0VRdQV5oPh/aSeriY0NbVBJS/bf2vz4MeI1R7RHDEL4GekGT8YsYSkZiJX1Q6hKVCYMwp71BWzmdP6G8EMkUkDeuf/TrgW/asXETCgBZjTLuIRALzgV8NtrBKqWHkFwaJuYxJzCU6p894Y+hpruNg8U4qS3bTfHAv1BcT0FJOXPPnxB38J2w9NnuXeNMWmIBnWCq+0elIWIrVfUVosvXuHz7k5x+rwbO3yeaFWE0yPYHnjTGPiMjDQJ4xZqWIzATeAMKANuCQMWaiiMwDngZ6AA/gMWPMc6fblrbeUerMYIyhurGd/NJqyvbvoaGigI6aYvyby0iUapKkimSpJkSaj1/OKwAJTbbtBJIgJMn2bhsOiNYzhUHQm7OUUk7R2tHNnspG8g8eIf/gEUoqDtFUWUhEx0ESpYYEqSHDq5a0MTVE91Tj1914/Ao8fSAkAUISrR1CSKLVj1FIgvUenNBvb6fuyJFNNpVSym5+3p7WjWFJobYxkzBmMRUNbew5dIS9lU2sPNTInspG9lU14dXVRILUkOhRwyT/I0zwO0wqdUQfriG46iPGNFcinHBw6h0IQXEQHH/s1Tvc+x4QNajHZbo6DX2l1LATERJC/UgI9eMbE2KOju/uMRyoazm6M9hb2cj7lU0UVTbR2W0FvY90kRPWxvTQFrIDGknzbiDOo57Qrho8GiugeK3Vt1FP1wkb9bQuKgfF2nYEccc+B8VCYKz17hfuVtVJWr2jlBp1Ort72F/TzN7KJvZUNlJQ2cjeykb217bQ3WNllqeHkBYZQGZ0IJnRAUwKaSfTv5FEz8N4NR+CIweh8ZDVNXbvq7X+6xvz8LLtHGJsO4KTvcdY1xrseJqas2idvlLK5bR3dVNY1UxBlbUT2FvZxL6qJkpqm7HtC/AQSA73JyMqkLHRgWREBZIRHcjYqEBCvLqts4LGvq+D0FRpfe597/vgnKME/CNsZwnR1k4gMMr2HtPnc7Q13whXLWmdvlLK5fiM8SQ7Ppjs+OMv5LZ1dlNUbe0MCqubKayydgZrC2ro6D7WO0xkoDfpUbYdQdQUMqLmkZEZSEKY3/Gd0nW1Q1OVtRM4ukOosnYYTdXWuJp91nvvIzb7Eg8r+AOiISDStpOIsr0iwT/S9h5hvXxDRqwZq4a+UuqM5+t18p1Bd4+htK6FfVVNFNU0UVjVTGF1E+/tOMhh2xPLALzHeJAWEUBGdAAZUYGkRwWQERVEelQcgYmniUljoP2ItUNorj7hvQqaa6zPZXnW+I6mk6/HY4x1bSF5Dlz7N0d8Jaekoa+UclmeHkJqZACpkQFAzHHT6po7KKpuorC6iaLqZgqrm9l9sJFVOyuPXjcAiAn2ObojSI/s3SEEEh9qOzvwDbFekZn9F6ijBVpqoKXWeuBO7+eWWmsHERTr4G/g6zT0lVJuKTzAm/CAcHJTw48b39HVw4G6ZvbZzgp6dwpvba2gse1YC6G+ZwfpkYF9zhICCfQ5RbR6+4O37cY0J9HQV0qpPrzHeDA2Ooix0UHHjTfGUNPUQXFNM0XVTRTZ3vNPcnYQG+x7dCcwNjrw6BlCXIiv07uv1tBXSik7iAhRQT5EBfkwK62fs4OqJgprmnl9czlN7cfODvy9PUmLDCAtMoD0yADSogJIjbCGQ/29R+Tv0NBXSqkhOt3ZQVVje5/rBtb7V+UNvPvVQfqcHBDm78VZmVH8/vochpOGvlJKDRMRISbYl5hgX+ZlRB43rb2rm9K6FoprWthf00xRTTPhAcP/UHsNfaWUcgKfMZ4nPTsYbu7T4YRSSikNfaWUcica+kop5UY09JVSyo1o6CullBvR0FdKKTeioa+UUm5EQ18ppdzIqHtylohUAyXOLocDRAI1zi7EKKLfx9fpd3I8/T6ON9DvI8UYE9XfTKMu9F2FiOTZ8+gyd6Hfx9fpd3I8/T6ON1zfh1bvKKWUG9HQV0opN6KhP3yecXYBRhn9Pr5Ov5Pj6fdxvGH5PrROXyml3Ige6SullBvR0B8iEUkSkdUisktEdorIfbbx4SLygYgU2N7DnF3WkSQiniKyRUTetg2nicgGEdknIv8QkZF5NtwoISKhIrJCRHaLSL6IzHXn34iI/MD2/2WHiLwsIr7u9hsRkedFpEpEdvQZd9LfhFh+Z/tutovI9MFuV0N/6LqAfzPGZANzgLtFJBt4APjIGJMJfGQbdif3Afl9hn8J/K8xZixQD9zulFI5z+PA+8aYCcBUrO/GLX8jIpIA3AvkGmMmAZ7Adbjfb+TPwNITxp3qN3EBkGl7LQOeHPRWjTH6cuALeAs4F9gDxNnGxQF7nF22EfwOEm0/2G8AbwOCdZPJGNv0ucAqZ5dzBL+PEKAY2zW0PuPd8jcCJAClQDjW0/veBs53x98IkArs6O83ATwNXH+y+Qb60iN9BxKRVCAH2ADEGGMO2iYdAmKcVCxneAz4D6DHNhwBHDbGdNmGy7D+47uLNKAaeMFW5fUnEQnATX8jxphy4DfAAeAg0ABswr1/I71O9Zvo3VH2GvT3o6HvICISCLwGfN8Yc6TvNGPtmt2imZSIXAxUGWM2Obsso8gYYDrwpDEmB2jmhKocN/uNhAGXYe0M44EAvl7N4faG6zehoe8AIuKFFfgvGmNet42uFJE42/Q4oMpZ5Rth84FLRWQ/8ApWFc/jQKiIjLHNkwiUO6d4TlEGlBljNtiGV2DtBNz1N7IEKDbGVBtjOoHXsX437vwb6XWq30Q5kNRnvkF/Pxr6QyQiAjwH5BtjHu0zaSVws+3zzVh1/S7PGPMjY0yiMSYV6+Lcx8aYG4DVwNW22dzm+wAwxhwCSkVkvG3UYmAXbvobwarWmSMi/rb/P73fh9v+Rvo41W9iJXCTrRXPHKChTzXQgOjNWUMkImcBa4GvOFaH/SBWvf5yIBmr19BvGmPqnFJIJxGRRcD9xpiLRSQd68g/HNgC3GiMaXdm+UaSiEwD/gR4A0XArVgHXW75GxGRnwHXYrV+2wJ8B6uO2m1+IyLyMrAIqzfNSuAh4E1O8puw7Rz/gFUN1gLcaozJG9R2NfSVUsp9aPWOUkq5EQ19pZRyIxr6SinlRjT0lVLKjWjoK6WUG9HQV0opN6Khr5RSbkRDXyml3Mj/B3wC/HRimqHuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f65c060d908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_train_vs_validation_loss(mlp_train_loss, mlp_eval_loss, offset=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy on train set:\n",
      "0.96\n",
      "accuracy on validation set:\n",
      "0.954\n",
      "accuracy on test set:\n",
      "0.951\n"
     ]
    }
   ],
   "source": [
    "evaluate_classifier(mlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different optimizers/hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "customized_mlp = mod.Module(symbol=mlp_out(num_hidden))\n",
    "\n",
    "customized_mlp.bind(data_shapes=train_iter.provide_data, label_shapes=train_iter.provide_label)\n",
    "customized_mlp.init_params(initializer=mx.init.Xavier(magnitude=2.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Already bound, ignoring bind()\n",
      "/home/semantive/Projects/notebooks/examples-counterexamples/venv/lib/python3.5/site-packages/mxnet/module/base_module.py:464: UserWarning: Parameters already initialized and force_init=False. init_params call ignored.\n",
      "  allow_missing=allow_missing, force_init=force_init)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 43.2 s, sys: 2.14 s, total: 45.3 s\n",
      "Wall time: 19.6 s\n"
     ]
    }
   ],
   "source": [
    "customized_mlp_train_log_list = []\n",
    "customized_mlp_eval_log_list = []\n",
    "\n",
    "%time customized_mlp.fit(\\\n",
    "  train_iter,\\\n",
    "  eval_data=valid_iter,\\\n",
    "  optimizer='adam',\\\n",
    "  optimizer_params={'learning_rate': 0.001, 'wd': 1e-3},\\\n",
    "  eval_metric='ce',\\\n",
    "  batch_end_callback=log_to_list(customized_mlp_train_log_list),\\\n",
    "  eval_batch_end_callback=log_to_list(customized_mlp_eval_log_list, data_type='Valid'),\\\n",
    "  num_epoch=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "customized_mlp_train_loss = [msg[1][4] for msg in customized_mlp_train_log_list]\n",
    "customized_mlp_eval_loss = [msg[1][4] for msg in customized_mlp_eval_log_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XuYHHW95/H3ty/TPZNJyBUISTDRw0pIuCQMMS4iQZQnooIXbj7qEfcoz+HgIrvq2cgf4nEPZzm7LgcRL4uKl128sCgX98FV1PCAe4RDwgkhEJCAQUJIMgnkxly7+7t/VHV3TU/3zJDpmU5VPq/n6aequqq7f9Vd/alf/7qqfubuiIhIsqRaXQAREWk+hbuISAIp3EVEEkjhLiKSQAp3EZEEUriLiCSQwl1EJIEU7iIiCaRwFxFJoEyrXnj27Nm+cOHCVr28iEgsrV+/fre7zxltuZaF+8KFC1m3bl2rXl5EJJbM7IWxLKdmGRGRBFK4i4gkkMJdRCSBWtbmLiLJMjg4yLZt2+jr62t1URIhn88zf/58stnsIT1e4S4iTbFt2zamTp3KwoULMbNWFyfW3J09e/awbds2Fi1adEjPoWYZEWmKvr4+Zs2apWBvAjNj1qxZ4/oVpHAXkaZRsDfPeN/L2IX7MzsO8N9//Qx7Dva3uigiIoet2IX7c90H+drvttCtcBeRiL179/KNb3zjdT/u/PPPZ+/evRNQotaKXbjns0GReweKLS6JiBxOGoV7oVAY8XH33Xcf06dPn6hitcyo4W5mC8xsrZk9ZWZPmtln6iyzysz2mdmG8PbFiSku5LNpAPoGSxP1EiISQ2vWrOG5557jtNNO44wzzuCss87iggsu4KSTTgLg/e9/P6effjpLlizh1ltvrTxu4cKF7N69m61bt7J48WI+9alPsWTJEs477zx6e3tbtTrjNpZDIQvAZ939MTObCqw3s/vd/ama5R5y9/c2v4hDVcK9oJq7yOHq737xJE9t39/U5zzpuGlc974lDeffcMMNbNq0iQ0bNvDAAw/wnve8h02bNlUOJbztttuYOXMmvb29nHHGGXzoQx9i1qxZQ57j2Wef5cc//jHf/va3ueSSS/jZz37GRz/60aaux2QZtebu7i+7+2Ph+AFgMzBvogvWSD4ThHv/oMJdRBpbsWLFkGPEb775Zk499VRWrlzJiy++yLPPPjvsMYsWLeK0004D4PTTT2fr1q2TVdyme10nMZnZQmAZ8Eid2W81s8eB7cDn3P3JcZeujnKbu5plRA5fI9WwJ8uUKVMq4w888AC/+c1v+MMf/kBHRwerVq2qewx5LperjKfT6cQ3ywBgZp3Az4Br3L3299ZjwBvc/aCZnQ/cDZxQ5zmuAK4AOP744w+pwOVmmV7V3EUkYurUqRw4cKDuvH379jFjxgw6Ojp4+umnefjhhye5dJNvTEfLmFmWINhvd/ef18539/3ufjAcvw/ImtnsOsvd6u5d7t41Z86o15qvq73yh6rCXUSqZs2axZlnnsnSpUv5/Oc/P2Te6tWrKRQKLF68mDVr1rBy5coWlXLyjFpzt+A0qe8Cm939xgbLHAvsdHc3sxUEO409TS1pSEfLiEgjP/rRj+ren8vl+OUvf1l3Xrldffbs2WzatKly/+c+97mml28yjaVZ5kzgY8ATZrYhvO9a4HgAd/8WcBFwpZkVgF7gMnf3CSgvuUy5zV01dxGRRkYNd3f/PTDiRQ7c/RbglmYVaiSplNGWSSncRURGELszVCFod1e4i4g0Fstwz2dTanMXERlBTMM9rTNURURGEM9wz6hZRkRkJPEM92yKXjXLiMg4dHZ2ArB9+3YuuuiiususWrWKdevWjfg8N910Ez09PZXpw+USwjENd9XcRaQ5jjvuOO68885DfnxtuB8ulxCObbjrwmEiErVmzRq+/vWvV6a/9KUv8fd///ece+65LF++nJNPPpl77rln2OO2bt3K0qVLAejt7eWyyy5j8eLFfOADHxhybZkrr7ySrq4ulixZwnXXXQcEFyPbvn0755xzDueccw5QvYQwwI033sjSpUtZunQpN910U+X1JuPSwq/rwmGHCx0tI3KY++Ua2PFEc5/z2JPh3Tc0nH3ppZdyzTXXcNVVVwFwxx138Ktf/Yqrr76aadOmsXv3blauXMkFF1zQsH/Sb37zm3R0dLB582Y2btzI8uXLK/Ouv/56Zs6cSbFY5Nxzz2Xjxo1cffXV3Hjjjaxdu5bZs4decWX9+vV873vf45FHHsHdectb3sLZZ5/NjBkzJuXSwrGtuevCYSIStWzZMnbt2sX27dt5/PHHmTFjBsceeyzXXnstp5xyCu985zt56aWX2LlzZ8PnePDBByshe8opp3DKKadU5t1xxx0sX76cZcuW8eSTT/LUU7VdWgz1+9//ng984ANMmTKFzs5OPvjBD/LQQw8Bk3Np4VjW3HUSk8hhboQa9kS6+OKLufPOO9mxYweXXnopt99+O93d3axfv55sNsvChQvrXup3NH/605/4yle+wqOPPsqMGTO4/PLLD+l5yibj0sKxrbkr3EWk1qWXXspPfvIT7rzzTi6++GL27dvH0UcfTTabZe3atbzwwgsjPv7tb3975eJjmzZtYuPGjQDs37+fKVOmcNRRR7Fz584hFyFrdKnhs846i7vvvpuenh5ee+017rrrLs4666wmru3IYllzz2VT9BXU5i4iQy1ZsoQDBw4wb9485s6dy0c+8hHe9773cfLJJ9PV1cWJJ5444uOvvPJKPvGJT7B48WIWL17M6aefDsCpp57KsmXLOPHEE1mwYAFnnnlm5TFXXHEFq1ev5rjjjmPt2rWV+5cvX87ll1/OihUrAPjkJz/JsmXLJq13J5ugizeOqqury0c7frSRf7r/j3z1t8/y/D+cTyo14jXNRGSSbN68mcWLF7e6GIlS7z01s/Xu3jXaY2PbLAPqJFtEpJFYhnu7+lEVERlRLMM9r672RA5LrWrmTaLxvpcKdxFpinw+z549exTwTeDu7Nmzh3w+f8jPEcujZfJqlhE57MyfP59t27bR3d3d6qIkQj6fZ/78+Yf8+FiGey6suessVZHDRzabZdGiRa0uhoRi2SzTHoa7Lh4mIlJfLMNdh0KKiIwspuGuNncRkZHEM9wzYZv7gGruIiL1xDLc29vULCMiMpJYhnu55q5mGRGR+mIZ7rlKm7tq7iIi9cQz3DMpzHQopIhII7EMdzMjl0npJCYRkQZiGe5Q7mpPbe4iIvXENtzV1Z6ISGOjhruZLTCztWb2lJk9aWafqbOMmdnNZrbFzDaa2fKJKW5VPptWV3siIg2M5cJhBeCz7v6YmU0F1pvZ/e7+VGSZdwMnhLe3AN8MhxMml0mp5i4i0sCoNXd3f9ndHwvHDwCbgXk1i10I/NADDwPTzWxu00sb0d6mZhkRkUZeV5u7mS0ElgGP1MyaB7wYmd7G8B0AZnaFma0zs3XjveZzPqNwFxFpZMzhbmadwM+Aa9x9/6G8mLvf6u5d7t41Z86cQ3mKinw2paNlREQaGFO4m1mWINhvd/ef11nkJWBBZHp+eN+E0dEyIiKNjeVoGQO+C2x29xsbLHYv8JfhUTMrgX3u/nITyzlMPpvWSUwiIg2M5WiZM4GPAU+Y2YbwvmuB4wHc/VvAfcD5wBagB/hE84s6VF4nMYmINDRquLv77wEbZRkHrmpWocYin03p2jIiIg3E+wxVXc9dRKSu+IZ7Js1g0SkU1TQjIlIrtuHe3hZe012XIBARGSa24Z7PlntjUtOMiEit+IZ7RuEuItJIbMO92tWemmVERGrFNtzVLCMi0lhsw71d4S4i0lBsw71ac1ezjIhIrRiHe7nNXTV3EZFaMQ73sOaus1RFRIaJb7iHh0L2DijcRURqxTfcdYaqiEhD8Q33sFlGV4YUERkuvuGuM1RFRBqKbbhn00bKdCikiEg9sQ13M6NdXe2JiNQV23AHdZItItJIAsJdzTIiIrViHe65bEonMYmI1BHrcM9n0vTpJCYRkWFiHe7tbeokW0SknliHez6bUpu7iEgd8Q73jI6WERGpJ97hrkMhRUTqSkC4q1lGRKRWzMM9pZq7iEgdMQ93NcuIiNQT83BP0Vco4e6tLoqIyGEl3uGeSVMsOYNFhbuISNSo4W5mt5nZLjPb1GD+KjPbZ2YbwtsXm1/M+trb1I+qiEg9Y6m5fx9YPcoyD7n7aeHty+Mv1tjksuqwQ0SknlHD3d0fBF6ZhLK8bvlMUPx+HQ4pIjJEs9rc32pmj5vZL81sSaOFzOwKM1tnZuu6u7vH/aJ51dxFROpqRrg/BrzB3U8Fvgbc3WhBd7/V3bvcvWvOnDnjfuH2MNzVG5OIyFDjDnd33+/uB8Px+4Csmc0ed8nGoFpzV7OMiEjUuMPdzI41MwvHV4TPuWe8zzsW+WxQfDXLiIgMlRltATP7MbAKmG1m24DrgCyAu38LuAi40swKQC9wmU/SWUVqcxcRqW/UcHf3D48y/xbglqaV6HWo1NwLapYREYmK9xmq5Zq7utoTERkiGeGuM1RFRIZIRrirzV1EZIh4h3umfLSM2txFRKJiHe6ZdIpMynQSk4hIjViHOwRnqapZRkRkqNiHe079qIqIDBP7cM9nU/Sr5i4iMkQCwj2tQyFFRGrEPtzbs2l6dRKTiMgQsQ/3fDalNncRkRoJCHc1y4iI1Ip9uOcyOlpGRKRW7MNdR8uIiAwX+3Bvz6Z1hqqISI3Yh3teZ6iKiAyTgHDX0TIiIrUSEO7B0TKT1LOfiEgsJCLc3aFfXe2JiFQkItwB+tU0IyJSkYBwL3eSrT9VRUTK4h/uGXW1JyJSK/7hXulHVc0yIiJlsQ/39rZgFXQik4hIVezDXc0yIiLDxT7cc1mFu4hIrdiHe+VoGbW5i4hUxD7c21VzFxEZJvbhnle4i4gMM2q4m9ltZrbLzDY1mG9mdrOZbTGzjWa2vPnFbEzhLiIy3Fhq7t8HVo8w/93ACeHtCuCb4y/W2FXPUFWbu4hI2ajh7u4PAq+MsMiFwA898DAw3czmNquAo9GhkCIiwzWjzX0e8GJkelt436RIpYy2TEonMYmIREzqH6pmdoWZrTOzdd3d3U173nwmpatCiohENCPcXwIWRKbnh/cN4+63unuXu3fNmTOnCS8dUFd7IiJDNSPc7wX+MjxqZiWwz91fbsLzjpnCXURkqMxoC5jZj4FVwGwz2wZcB2QB3P1bwH3A+cAWoAf4xEQVthH1oyoiMtSo4e7uHx5lvgNXNa1Eh6A9m9YfqiIiEbE/QxWCi4epWUZEpCoR4Z7PpnUSk4hIRDLCPZOiXzV3EZGKRIR7e5va3EVEohIR7vmM2txFRKKSEe46FFJEZIiEhLtq7iIiUYkI91w2TX+hRKnkrS6KiMhhIRHhXu5qr1+HQ4qIAAkJ92on2WqaERGBxIR72GFHQeEuIgKJCfdyzV3NMiIikJBwb1cn2SIiQyQi3HNhuOssVRGRQCLCXZ1ki4gMlYxwD9vc1Y+qiEggIeGumruISFQiwr1dbe4iIkMkItyrNXc1y4iIQGLCXWeoiohEJSTcdYaqiEhUIsI9l9EZqiIiUYkIdzMLO+xQzV1EBBIS7qAOO0REopIT7upHVUSkIjnhrn5URUQqEhTuaZ3EJCISSlS4q1lGRCSQoHBP6cJhIiKhBIV7WicxiYiExhTuZrbazJ4xsy1mtqbO/MvNrNvMNoS3Tza/qCPT0TIiIlWZ0RYwszTwdeBdwDbgUTO7192fqln0p+7+6Qko45i0t+kPVRGRsrHU3FcAW9z9eXcfAH4CXDixxXr9dCikiEjVWMJ9HvBiZHpbeF+tD5nZRjO708wWNKV0r0NOzTIiIhXN+kP1F8BCdz8FuB/4Qb2FzOwKM1tnZuu6u7ub9NKBfDato2VEREJjCfeXgGhNfH54X4W773H3/nDyO8Dp9Z7I3W919y5375ozZ86hlLeh9myagWKJYsmb+rwiInE0lnB/FDjBzBaZWRtwGXBvdAEzmxuZvADY3Lwijo067BARqRr1aBl3L5jZp4FfAWngNnd/0sy+DKxz93uBq83sAqAAvAJcPoFlrivaSfaU3KirJSKSaGNKQXe/D7iv5r4vRsa/AHyhuUV7fSo194La3UVEEnWGKqhZRkQEEhjuvQMKdxGRxIV7v64vIyKSoHBXJ9kiIhXJCXe1uYuIVCQm3NvbyuGumruISHLCPay5dx/oa3FJRERaLzHhPm96O0uOm8Z3/9+fGNCx7iJyhEtMuKdSxt+uPpEXX+nlR4+80OriiIi0VGLCHeDtJ8zmrW+cxdd+t4WD/YVWF0dEpGXiF+4v/gvc9m7o3Ttslpnxn959InteG+DbDz7fgsKJiBwe4hfuqQy8+DDc/8W6s09bMJ3zTz6W7zz0PLsP9tddRkQk6eIX7vOWw1uvgsd+AH96qO4inzvvzfQVStzyuy2TXDgRkcND/MIdYNW1MGMR/OJqGOwdNvuNczq5pGsBtz/yAn/e09OCAoqItFY8w72tA973VXjleXjghrqLXPPOE0injBvvf2aSCyci0nrxDHeAN54Nyz4G//w12L5h2OxjpuX5xJmLuOfx7Ty1fX8LCigi0jrxDXeA8/4zTJkN9/57KA4/9PGvz34T0/JZ/uuvnm5B4UREWife4d4+A87/b7BjI/zhlmGzj2rP8jer3sQDz3Tz0LPdLSigiEhrxDvcAU66EE58LzzwX2DPc8Nmf/zfLmTBzHb+6vvr+OYDz1Eo6tIEIpJ88Q93gPO/Aukc/OIz4D5kVj6b5q6/OZNzFx/NP/7fp7noW39gy66DLSqoiMjkSEa4T5sL530Ztj4EP7wQtvx2SMjP7szxjY8s5+YPL2Prntc4/+aH+PaDz1Ms+QhPKiISX8kId4DlH4fzrofuZ+B/fRC+9TZ4/CdQHASCSxNccOpx/Po/vJ2z/80crr9vM5f8jz/wx50HWlxwEZHmM/fW1F67urp83bp1zX/iQj888b+DQyS7n4Zp8+Atfw1LPxiMm+Hu3LNhO9fd+yT7egdZfvx0Prh8Pu89ZS7TO9qaXyYRkSYxs/Xu3jXqcokL97JSCbb8Bv755qC5BiA/HY49Obgds5RXp72Zn7+Q56cbdvPHXa/Rlk5x7uKj+dDy+Zz95jlk08n5YSMiyaBwj9rxBPz54WC4cxPsfAoK1csWeCZPITeD3aVO/tybZ1dxCgfTR9Ex/VhmHX0cC+YvYP68BaQ7ZwfH1eemQSYHZpNTfhGR0FjDPTMZhWm5cm29rFQMLl2w4wnY+wLW8wrZnleY27OHY3p207d3B/Q+ScfeA7AX+OPwp/RUFnJTsdzUIOxzU4PAT2ch3RYMU5HxyrxccF+mLRimMpBKB8umMtXpyuOzQ8fLy0THU5nq66TbquPRnY97sN6lQnCD+suNVakExX4o9A07QqkilYZsR/AaIuPR80pwqHPP7nAbDrdlL1WH0e9D9LtUHITiQDjsD8cL0DYFOmZBx8zqMHcUpJLxi/3ICPdaqTTMPiG41c4COsoTxUG6d73Mpmef508vbGX79m307+9mKr10Wi8zS/0cxyBzfIAZg/10pHppsyJtFEh5IbJBDUBhIJzuDzbEyZAO/z8ob/x1WbjjyVV3OGVDQtuD/zOKA8GwNDj2cqQyQchn8pBtD4apNFgquKXSYOF0I2bV5S1Vncaq88vrUy5vcXDoDq28U8tNHX5r6wzW14vBYyrDUuObpcIAyUZ2yJng/uhrVspQrK5ref3LATRkPaxmnWvXvcF6l5dLpYeHXLlMxYFquJXH6z1PZViHl6rPURoM3+dowNZ5/Uy5UlPe1sJpfPj7VByEAztgzxZ45bkg1PuG998wISwdlM09KFt5GJ1f2W5T1c+wUqkqV9zC9StF3qNyFpQK0PXv4Kz/OKGrcmSG+1ils8yZezznzD2ec8K79vUM8szOAzy9Yz8bdxzgjpf388edB4f1/DS9I8vco9o5bmaeY4/Kc8y0PMdOy3P0tBzHdGY5dooxPeeYe/XLEb0VB8ONIvwSlsfLG0upMHQ6uvEU+4PxQn/4hU8PrcmkMgThFwZ1NLSLg9V8BIZMRL+Y0S9rqsFmVBwMmr8Gy7eeYFjoC2tcpZoQLVLz4hEeLuPVx5aK1XmVxSLj5bDNtFXX3x0GDsL+7dB/oHorRq79b6lIAJeHVidcozuQyGdYfo7aGqSlquta2YEU6q9HXFjkV6alIusUeS8O1bT5MOtNwcEQM98UjHceM3xbLn8mQ3akkR1G7S/a8jY7cBB69gS/CnpeCcf3RLaFmh1deduL3qI7pPJ3rxD5DlZ+oUd+hafbYOYbx/vOj0rh/jod1ZFlxaKZrFg0s3Kfu/PS3l62vdrLy/t62b63rzJ8aW8vj/35VV7tGV7TzaSM6R1Zpne0MaNmOC2fY1p7J9PyWaa1Z4LhlCyduQyd+QydbRlSKbX5N02xMLSmfKg8rO2N96e9R2qN9QIlWKi6bHm8vMMcVmEo1m+6izaZNaqtDmORJsUR1rPSFDhYU4EoVyr66tTywx3qlNnBr7yJNGU2zFg4sa/RQgr3JjAz5s/oYP6MjobL9BeK7Nrfz64Dfezc38/O/X10H+jn1Z5B9vYMsLdnkBdf6eGJbYO82jNAf2H0ppvOXIbOXIYpuTRTchny2TQdbWnas2nay8Nsmnw2TT6bCofBLVgmFRmvLptJGZlUinTayKSMdCoYWpL/QE436atgIzRnHPLzpMf/fK1gFryv6czEB7UMM6Yt2sxWA18l2Mq+4+431MzPAT8ETgf2AJe6+9bmFjXecpk0C2Z2sGBm4x1AVH+hyIG+Avt7B9lfGQ5ysK/Awf4CB8JhebpnoEDPQJFXXxvgpYEiPQNFegeL9IW3ZpyMmzLIpFOVwM+mU6RTRls6RVsmVR2G45nIziFl4TDcUWRSKdoywTCbTpFNG5m0kbZgmcowHK/Ny/KOJh0pU3UYvF69iC26M1AoMVAo0R8OB4olBoslOtqCnWRn5DYllwlacopFBgoeLBs+xiAoeyYof3n90ynDsKBJFiNVyemwKYdq61H5Y7FwnYJlg2EqXEcL77PyeMOmq+rnFH2OVGro+1f7/3fKqHxGlc/JeN07c3cPfnCE4yUHJ7ivWRUEd6dQcgrF4LMoFEsUSk6x5MFnHr7X5XWJbrPl7fVIMWq4m1ka+DrwLmAb8KiZ3evuT0UW+yvgVXf/CzO7DPhH4NKJKPCRIpdJk+tMM7szN+7ncg++CH2DJfoHi/QNlirB3ztYpHegOuwrFCmGX55iycMvTonBhtOlSugNFIqVsBwolOgZdErhF6/kwbAYDgtFZzAM1coXNVxWkqu8IynvtKG8M6juCKo7vqE7C2DclZRy2LelU8HOL1XdCZR3iO7B65S8ut2Wy1R+fLn85XUp85rmLCPc4RDurMMdz2VnLOCTZ01su/tYau4rgC3u/jyAmf0EuBCIhvuFwJfC8TuBW8zMvFUH0csQZhbsLDJpaD/8D0ssRXYC5S8XVGu5la3Kg5p4oVSq7IwGi6XKTqSelAU17Fy2+ksjlwmaonoGi7wW/ip6rT/8ZdRfCB4T1s5zmfIvjaCtubyDGihUd1aDxaDNuhwSTliLda/UwGFoLTy6TCmsARdLYVR4bdCNcCBLzesF72E1HMvKNehybbvkwXsW7IwZ9v6Npb7rhL8aIr9Wyq8T/UwLpeB1CuHnWvlFUq55U/2lEf21Ul4mG/5Ky6aD2nj5F59D5b0r/3IohhWGgbASMVgsMVgqMVgov88eCfKgnKlUNeiDX41BgFfep0glp1iCYqk05BeJ1bwnlTKF4zhNqbSNZizhPg94MTK9DXhLo2XcvWBm+4BZwO7oQmZ2BXAFwPHHH3+IRZakS6WMFEZ2kpuay00xx0yb3NcVmQiTerS+u9/q7l3u3jVnzpzJfGkRkSPKWML9JWBBZHp+eF/dZcwsAxxF8MeqiIi0wFjC/VHgBDNbZGZtwGXAvTXL3At8PBy/CPid2ttFRFpn1Db3sA3908CvCA6FvM3dnzSzLwPr3P1e4LvA/zSzLcArBDsAERFpkTEd5+7u9wH31dz3xch4H3Bxc4smIiKHKhmXPxMRkSEU7iIiCaRwFxFJoJb1xGRm3cALh/jw2dScIHUEOVLXXet9ZNF6N/YGdx/1RKGWhft4mNm6sXQzlURH6rprvY8sWu/xU7OMiEgCKdxFRBIoruF+a6sL0EJH6rprvY8sWu9ximWbu4iIjCyuNXcRERlB7MLdzFab2TNmtsXM1rS6PBPFzG4zs11mtily30wzu9/Mng2HM1pZxolgZgvMbK2ZPWVmT5rZZ8L7E73uZpY3s38xs8fD9f678P5FZvZIuL3/NLx4X+KYWdrM/tXM/k84nfj1NrOtZvaEmW0ws3XhfU3bzmMV7pEu/94NnAR82MxOam2pJsz3gdU1960BfuvuJwC/DaeTpgB81t1PAlYCV4WfcdLXvR94h7ufCpwGrDazlQRdVv6Tu/8F8CpBl5ZJ9Blgc2T6SFnvc9z9tMjhj03bzmMV7kS6/HP3AaDc5V/iuPuDBFfYjLoQ+EE4/gPg/ZNaqEng7i+7+2Ph+AGCL/w8Er7uHjgYTmbDmwPvIOi6EhK43gBmNh94D/CdcNo4Ata7gaZt53EL93pd/s1rUVla4Rh3fzkc3wEc08rCTDQzWwgsAx7hCFj3sGliA7ALuB94Dtjr7oVwkaRu7zcBfwuUwulZHBnr7cCvzWx92AUpNHE7H9Mlf+Xw4+5uZok91MnMOoGfAde4+/5oB8RJXXd3LwKnmdl04C7gxBYXacKZ2XuBXe6+3sxWtbo8k+xt7v6SmR0N3G9mT0dnjnc7j1vNfSxd/iXZTjObCxAOd7W4PBPCzLIEwX67u/88vPuIWHcAd98LrAXeCkwPu66EZG7vZwIXmNlWgmbWdwBfJfnrjbu/FA53EezMV9DE7Txu4T6WLv+SLNqd4ceBe1pYlgkRtrd+F9js7jdGZiV63c1sTlhjx8zagXcR/N+wlqDrSkjgerv7F9x9vrsvJPg+/87dP0LC19vMppjZ1PI4cB6wiSZu57Fh+eR2AAAAs0lEQVQ7icnMzidooyt3+Xd9i4s0Iczsx8AqgqvE7QSuA+4G7gCOJ7ii5iXuXvuna6yZ2duAh4AnqLbBXkvQ7p7YdTezUwj+QEsTVLrucPcvm9kbCWq0M4F/BT7q7v2tK+nECZtlPufu7036eofrd1c4mQF+5O7Xm9ksmrSdxy7cRURkdHFrlhERkTFQuIuIJJDCXUQkgRTuIiIJpHAXEUkghbuISAIp3EVEEkjhLiKSQP8fWEAnh+/a7o8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f65c05c1828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy on train set:\n",
      "0.996\n",
      "accuracy on validation set:\n",
      "0.976\n",
      "accuracy on test set:\n",
      "0.972\n"
     ]
    }
   ],
   "source": [
    "plot_train_vs_validation_loss(customized_mlp_train_loss, customized_mlp_eval_loss)\n",
    "evaluate_classifier(customized_mlp)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}