ml_diagnosis_confusion_matrix_and_worst_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diagnosis of a bad Digits classification\n",
    "\n",
    "Make a Naive Bayes classifier for the Digits task, this uses an 8x8 greyscale of the digits 0-9, the task is to predict the correct label from the image.\n",
    "\n",
    "This Notebook is a part of my [Data Science Delivered](https://github.com/ianozsvald/data_science_delivered) notes. License: CC-By-Attribution.\n",
    "\n",
    "Generally we do pretty well even just with Naive Bayes. Below we use a Confusion Matrix to identify the classes that are hard to predict (1s get mis-classified a portion of the time). Next we'll pull out an example that's misclassified.\n",
    "\n",
    "A simple debugging process would be:\n",
    "\n",
    "* run a classifier using 10-fold cross validation, note the results\n",
    "* use a Confusion Matrix to identify areas needing improvement\n",
    "* investigate the highest-probability wrong answers\n",
    "* ponder on why the classifier is making these mistakes\n",
    " * consider adding new features or cleaning your data or trying a new classifier or tuning your hyperparameters\n",
    "* repeat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "\n",
    "from sklearn import naive_bayes\n",
    "\n",
    "# let's plot inline\n",
    "%matplotlib inline\n",
    "\n",
    "# load the digits dataset\n",
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-10-04 \n",
      "\n",
      "CPython 3.6.2\n",
      "IPython 6.1.0\n",
      "\n",
      "numpy 1.13.1\n",
      "matplotlib 2.0.2\n",
      "sklearn 0.18.2\n",
      "seaborn 0.8.0\n",
      "pandas 0.20.3\n",
      "\n",
      "compiler   : GCC 4.8.2 20140120 (Red Hat 4.8.2-15)\n",
      "system     : Linux\n",
      "release    : 4.9.45-040945-generic\n",
      "machine    : x86_64\n",
      "processor  : x86_64\n",
      "CPU cores  : 8\n",
      "interpreter: 64bit\n",
      "Git hash   : 322359a1d2032562ecc693433647bd6332f23e84\n"
     ]
    }
   ],
   "source": [
    "# install watermark using\n",
    "# %install_ext https://raw.githubusercontent.com/rasbt/watermark/master/watermark.py\n",
    "%load_ext watermark\n",
    "# show a watermark for this environment\n",
    "%watermark -d -m -v -p numpy,matplotlib,sklearn,seaborn,pandas -g"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# start a separate QTconsole for interactive coding\n",
    "#%qtconsole"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X has shape (1797, 64), y has shape (1797,)\n"
     ]
    }
   ],
   "source": [
    "# flatten and binarise the 2D input image - this is our basic feature set\n",
    "# we could improve the features here by adding new columns to X\n",
    "arr_digits_greyscale = [digit.flatten() for digit in digits.images]\n",
    "arr_digits = [digit.flatten()>=8 for digit in digits.images]\n",
    "X_greyscale = np.vstack(arr_digits_greyscale)\n",
    "X = np.vstack(arr_digits)\n",
    "y = digits.target\n",
    "print(\"X has shape {}, y has shape {}\".format(X.shape, y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualise some of our examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y (target label) =  0\n",
      "Image for X[0] = \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAECCAYAAADesWqHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC0lJREFUeJzt3d+rZXd5x/H3p5MZx/yQgEklyQTTQDsQxM6EkFYGQptR\nk1RJb3qRgEKlZbxoJaEF0d4U/wGxF0WQJDZgEtGYgRLajKEagtBGJ5OJJplRTIhkOsaJFckPaX75\n9OLskXE67Vknnu86P573Czaz9zlr9vOcc/js79p7r72eVBWSevmttW5A0vwMvtSQwZcaMvhSQwZf\nasjgSw2ti+AnuT7J95P8MMmnBte6I8mJJE+MrHNKvUuTfDPJkSRPJrllcL3tSb6d5PFFvc+MrLeo\nuSXJY0nuH11rUe/ZJN9LcjjJwcG1zk9yb5Kji7/h+wbW2rn4mU5eXkxy65BiVbWmF2AL8DRwObAN\neBy4YmC9a4ArgSdm+vkuAq5cXD8P+MHgny/AuYvrW4FHgD8c/DP+DXA3cP9Mv9NngQtmqnUn8JeL\n69uA82equwV4Hnj3iPtfDyv+1cAPq+qZqnoN+DLwp6OKVdXDwM9G3f8Z6v24qg4trr8EHAEuGViv\nqurlxc2ti8uwo7SS7AA+BNw2qsZaSfIOlhaK2wGq6rWq+vlM5fcCT1fVj0bc+XoI/iXAc6fcPsbA\nYKylJJcBu1lahUfW2ZLkMHACeLCqRtb7HPBJ4JcDa5yugK8neTTJvoF1LgdeAL64eCpzW5JzBtY7\n1U3APaPufD0EP2f42qY7jjjJucDXgFur6sWRtarqzaraBewArk7ynhF1knwYOFFVj464///Hnqq6\nErgB+Ksk1wyqcxZLTws/X1W7gVeAoa9BASTZBtwIfHVUjfUQ/GPApafc3gEcX6NehkiylaXQ31VV\n981Vd7Fb+hBw/aASe4AbkzzL0lO0a5N8aVCtX6mq44t/TwD7WXq6OMIx4Ngpe0z3svRAMNoNwKGq\n+smoAush+N8BfjfJ7ywe6W4C/nmNe1o1ScLSc8QjVfXZGepdmOT8xfW3A+8Hjo6oVVWfrqodVXUZ\nS3+3b1TVR0bUOinJOUnOO3kd+CAw5B2aqnoeeC7JzsWX9gJPjah1mpsZuJsPS7sya6qq3kjy18AB\nll7JvKOqnhxVL8k9wB8BFyQ5Bvx9Vd0+qh5Lq+JHge8tnncD/F1V/cugehcBdybZwtID+1eqapa3\n2WbyLmD/0uMpZwF3V9UDA+t9ArhrsSg9A3xsYC2SnA18APj40DqLtw4kNbIedvUlzczgSw0ZfKkh\ngy81ZPClhtZV8AcffrlmtaxnvfVWb10FH5jzlzvrH9J61ltP9dZb8CXNYMgBPNvyttrOyj/E9Dqv\nspW3rXo/a13rN633e+/9xYr/zwv/9SYXvnPLW6r3Vvwm9X7w3bNX/H820t9vznr/zSu8Vq+e6YNv\nv2bIIbvbOYc/yN4Rd93SgQOHl99oA7vu4l1r3cKm8Uj926Tt3NWXGjL4UkMGX2rI4EsNGXypIYMv\nNWTwpYYMvtTQpODPOeJK0njLBn9x0sZ/ZOmUv1cANye5YnRjksaZsuLPOuJK0nhTgt9mxJXUxZQP\n6UwacbU4ccA+gO2s/NNWkuYzZcWfNOKqqr5QVVdV1VVzfnxR0spNCf6mHnEldbTsrv7cI64kjTfp\nRByLOW+jZr1JmplH7kkNGXypIYMvNWTwpYYMvtSQwZcaMvhSQwZfamjIJJ3N7sDxzT3ZZm5z/z6d\n3OOKL7Vk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYamjNC6I8mJJE/M0ZCk8aas+P8E\nXD+4D0kzWjb4VfUw8LMZepE0E5/jSw2t2sdynZ0nbRyrtuI7O0/aONzVlxqa8nbePcC/AzuTHEvy\nF+PbkjTSlKGZN8/RiKT5uKsvNWTwpYYMvtSQwZcaMvhSQwZfasjgSw0ZfKkhZ+dtAJt91tvcs/Oc\n1eeKL7Vk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYamnGzz0iTfTHIkyZNJbpmjMUnj\nTDlW/w3gb6vqUJLzgEeTPFhVTw3uTdIgU2bn/biqDi2uvwQcAS4Z3ZikcVb0HD/JZcBu4JERzUia\nx+SP5SY5F/gacGtVvXiG7zs7T9ogJq34SbayFPq7quq+M23j7Dxp45jyqn6A24EjVfXZ8S1JGm3K\nir8H+ChwbZLDi8ufDO5L0kBTZud9C8gMvUiaiUfuSQ0ZfKkhgy81ZPClhgy+1JDBlxoy+FJDBl9q\nyOBLDRl8qSGDLzVk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNTTlLLvbk3w7yeOL2XmfmaMx\nSeNMGajxKnBtVb28OL/+t5L8a1X9x+DeJA0y5Sy7Bby8uLl1camRTUkaa+oknS1JDgMngAerytl5\n0gY2KfhV9WZV7QJ2AFcnec/p2yTZl+RgkoOv8+pq9ylpFa3oVf2q+jnwEHD9Gb7n7Dxpg5jyqv6F\nSc5fXH878H7g6OjGJI0z5VX9i4A7k2xh6YHiK1V1/9i2JI005VX97wK7Z+hF0kw8ck9qyOBLDRl8\nqSGDLzVk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQwZfasjgSw0Z\nfKmhycFfDNV4LIkn2pQ2uJWs+LcAR0Y1Imk+U0do7QA+BNw2th1Jc5i64n8O+CTwy4G9SJrJlEk6\nHwZOVNWjy2zn7Dxpg5iy4u8BbkzyLPBl4NokXzp9I2fnSRvHssGvqk9X1Y6qugy4CfhGVX1keGeS\nhvF9fKmhKUMzf6WqHmJpTLakDcwVX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQys6gEfaDK67\neNdat7DmXPGlhgy+1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzVk8KWGDL7U0KRDdhen1n4JeBN4o6qu\nGtmUpLFWcqz+H1fVT4d1Imk27upLDU0NfgFfT/Jokn0jG5I03tRd/T1VdTzJbwMPJjlaVQ+fusHi\nAWEfwHbOXuU2Ja2mSSt+VR1f/HsC2A9cfYZtnJ0nbRBTpuWek+S8k9eBDwJPjG5M0jhTdvXfBexP\ncnL7u6vqgaFdSRpq2eBX1TPA78/Qi6SZ+Hae1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzVk8KWGnJ2n\n/+XA8cNr3YIGc8WXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQ5OCn+T8JPcmOZrk\nSJL3jW5M0jhTj9X/B+CBqvqzJNvAiRnSRrZs8JO8A7gG+HOAqnoNeG1sW5JGmrKrfznwAvDFJI8l\nuW0xWOPXJNmX5GCSg6/z6qo3Kmn1TAn+WcCVwOerajfwCvCp0zdyhJa0cUwJ/jHgWFU9srh9L0sP\nBJI2qGWDX1XPA88l2bn40l7gqaFdSRpq6qv6nwDuWryi/wzwsXEtSRptUvCr6jBw1eBeJM3EI/ek\nhgy+1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzXk7Ly34LqLd81ab7PPspv79ylXfKklgy81ZPClhgy+\n1JDBlxoy+FJDBl9qyOBLDRl8qaFlg59kZ5LDp1xeTHLrHM1JGmPZQ3ar6vvALoAkW4D/BPYP7kvS\nQCvd1d8LPF1VPxrRjKR5rDT4NwH3jGhE0nwmB39xTv0bga/+H993dp60Qaxkxb8BOFRVPznTN52d\nJ20cKwn+zbibL20Kk4Kf5GzgA8B9Y9uRNIepI7R+AbxzcC+SZuKRe1JDBl9qyOBLDRl8qSGDLzVk\n8KWGDL7UkMGXGjL4UkOpqtW/0+QF4K18Zv8C4Ker3M56qGU9681V791VdeFyGw0J/luV5GBVXbXZ\nalnPeuutnrv6UkMGX2povQX/C5u0lvWst67qravn+JLmsd5WfEkzMPhSQwZfasjgSw0ZfKmh/wHU\n94mMRQPo4AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9031244710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_example(idx, X, y):\n",
    "    \"\"\"Show an example as an image from X\"\"\"\n",
    "    print(\"y (target label) = \", y[idx])\n",
    "    print(\"Image for X[{}] = \".format(idx))\n",
    "    plt.matshow(X[idx].reshape((8,8)))\n",
    "    \n",
    "show_example(0, X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y (target label) =  1\n",
      "Image for X[11] = \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAECCAYAAADesWqHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACy5JREFUeJzt3e+r3vV9x/HnazFp6o8iVFeskTphC0jZjAS3EpDN9Ieu\nxd3ZDYUWVjayG1tRNijt7oz+A6W7MQpF7YSqpbUGhmymslaksNkmMbZq0lLFYpba2JXij7JE7Xs3\nzpWShmzne7Lz+Z5z8n4+4CLXdc73XO/3yeF1fb7XdX2v7ztVhaRefmOtG5A0P4MvNWTwpYYMvtSQ\nwZcaMvhSQ+si+EluTvL9JD9M8qnBte5JcjzJ0yPrnFbvqiTfTHI4yTNJ7hhcb2uSbyd5alHvMyPr\nLWpuSvJkkodH11rUeyHJ95IcSrJ/cK1LkzyY5Mjib/i+gbW2L36nU5dXktw5pFhVrekF2AQ8B1wD\nbAGeAq4dWO9G4Hrg6Zl+vyuA6xfXLwF+MPj3C3Dx4vpm4AngDwb/jn8D3A88PNP/6QvAZTPVuhf4\ni8X1LcClM9XdBLwEvGfE/a+HFf8G4IdV9XxVnQS+DPzJqGJV9Tjws1H3f5Z6P66qg4vrrwKHgSsH\n1quqem1xc/PiMuworSTbgA8Dd42qsVaSvIOlheJugKo6WVU/n6n8buC5qvrRiDtfD8G/EnjxtNtH\nGRiMtZTkamAHS6vwyDqbkhwCjgOPVtXIep8DPgn8cmCNMxXw9SQHkuwZWOca4GXgi4unMncluWhg\nvdPdBjww6s7XQ/Bzlq+dd8cRJ7kY+BpwZ1W9MrJWVb1VVdcB24Abkrx3RJ0kHwGOV9WBEff/f9hV\nVdcDtwB/leTGQXUuYOlp4eeragfwOjD0NSiAJFuAW4GvjqqxHoJ/FLjqtNvbgGNr1MsQSTazFPr7\nquqhueoudksfA24eVGIXcGuSF1h6inZTki8NqvUrVXVs8e9xYC9LTxdHOAocPW2P6UGWHghGuwU4\nWFU/GVVgPQT/O8BvJ/mtxSPdbcA/r3FPqyZJWHqOeLiqPjtDvcuTXLq4/nbg/cCREbWq6tNVta2q\nrmbp7/aNqvroiFqnJLkoySWnrgMfBIa8Q1NVLwEvJtm++NJu4NkRtc5wOwN382FpV2ZNVdWbSf4a\n2MfSK5n3VNUzo+oleQD4Q+CyJEeBv6+qu0fVY2lV/BjwvcXzboC/q6p/GVTvCuDeJJtYemD/SlXN\n8jbbTN4F7F16POUC4P6qemRgvU8A9y0WpeeBjw+sRZILgQ8Afzm0zuKtA0mNrIddfUkzM/hSQwZf\nasjgSw0ZfKmhdRX8wYdfrlkt61lvvdVbV8EH5vzPnfUPaT3rrad66y34kmYw5ACeLXlbbWXlH2J6\ngxNs5m2r3s9a19po9X7nd3+x4p95+b/e4vJ3bjqneufi/1PvB9+9cMU/s1H+fv/N65ysE2f74Nuv\nGXLI7lYu4veze8Rdawb79h1afqMN7EPvvm6tWxjmifq3Sdu5qy81ZPClhgy+1JDBlxoy+FJDBl9q\nyOBLDRl8qaFJwZ9zxJWk8ZYN/uKkjf/I0il/rwVuT3Lt6MYkjTNlxZ91xJWk8aYEv82IK6mLKR/S\nmTTianHigD0AW1n5p58kzWfKij9pxFVVfaGqdlbVzjk/vihp5aYE/7wecSV1tOyu/twjriSNN+lE\nHIs5b6NmvUmamUfuSQ0ZfKkhgy81ZPClhgy+1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzVk8KWGDL7U\nkMGXGjL4UkMGX2rI4EsNGXypIYMvNTRlhNY9SY4neXqOhiSNN2XF/yfg5sF9SJrRssGvqseBn83Q\ni6SZ+BxfamjSefWncHaetHGs2orv7Dxp43BXX2poytt5DwD/DmxPcjTJn49vS9JIU4Zm3j5HI5Lm\n466+1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzVk8KWGVu1DOhpn37FDa92CzjOu+FJDBl9qyOBLDRl8\nqSGDLzVk8KWGDL7UkMGXGjL4UkMGX2poysk2r0ryzSSHkzyT5I45GpM0zpRj9d8E/raqDia5BDiQ\n5NGqenZwb5IGmTI778dVdXBx/VXgMHDl6MYkjbOi5/hJrgZ2AE+MaEbSPCZ/LDfJxcDXgDur6pWz\nfN/ZedIGMWnFT7KZpdDfV1UPnW0bZ+dJG8eUV/UD3A0crqrPjm9J0mhTVvxdwMeAm5IcWlz+eHBf\nkgaaMjvvW0Bm6EXSTDxyT2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQ87OOwfOstNG54ovNWTw\npYYMvtSQwZcaMvhSQwZfasjgSw0ZfKkhgy81ZPClhqacZXdrkm8neWoxO+8zczQmaZwpx+qfAG6q\nqtcW59f/VpJ/rar/GNybpEGmnGW3gNcWNzcvLjWyKUljTZ2ksynJIeA48GhVOTtP2sAmBb+q3qqq\n64BtwA1J3nvmNkn2JNmfZP8bnFjtPiWtohW9ql9VPwceA24+y/ecnSdtEFNe1b88yaWL628H3g8c\nGd2YpHGmvKp/BXBvkk0sPVB8paoeHtuWpJGmvKr/XWDHDL1ImolH7kkNGXypIYMvNWTwpYYMvtSQ\nwZcaMvhSQwZfasjgSw0ZfKkhgy81ZPClhgy+1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzU0OfiLoRpP\nJvFEm9IGt5IV/w7g8KhGJM1n6gitbcCHgbvGtiNpDlNX/M8BnwR+ObAXSTOZMknnI8DxqjqwzHbO\nzpM2iCkr/i7g1iQvAF8GbkrypTM3cnaetHEsG/yq+nRVbauqq4HbgG9U1UeHdyZpGN/HlxqaMjTz\nV6rqMZbGZEvawFzxpYYMvtSQwZcaMvhSQwZfasjgSw0ZfKkhgy81tKIDeLTkQ+++bq1bGGrfsUNr\n3YIGc8WXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQ5MO2V2cWvtV4C3gzaraObIp\nSWOt5Fj9P6qqnw7rRNJs3NWXGpoa/AK+nuRAkj0jG5I03tRd/V1VdSzJbwKPJjlSVY+fvsHiAWEP\nwFYuXOU2Ja2mSSt+VR1b/Hsc2AvccJZtnJ0nbRBTpuVelOSSU9eBDwJPj25M0jhTdvXfBexNcmr7\n+6vqkaFdSRpq2eBX1fPA783Qi6SZ+Hae1JDBlxoy+FJDBl9qyOBLDRl8qSGDLzVk8KWGDL7UkMGX\nGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQ5OCn+TSJA8mOZLkcJL3jW5M0jhTB2r8\nA/BIVf1pki3gxAxpI1s2+EneAdwI/BlAVZ0ETo5tS9JIU3b1rwFeBr6Y5Mkkdy0Ga/yaJHuS7E+y\n/w1OrHqjklbPlOBfAFwPfL6qdgCvA586cyNHaEkbx5TgHwWOVtUTi9sPsvRAIGmDWjb4VfUS8GKS\n7Ysv7QaeHdqVpKGmvqr/CeC+xSv6zwMfH9eSpNEmBb+qDgE7B/ciaSYeuSc1ZPClhgy+1JDBlxoy\n+FJDBl9qyOBLDRl8qSGDLzVk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYaWDX6S7UkO\nnXZ5JcmdczQnaYxlz7lXVd8HrgNIsgn4T2Dv4L4kDbTSXf3dwHNV9aMRzUiax0qDfxvwwIhGJM1n\ncvAX59S/Ffjq//J9Z+dJG8RKVvxbgINV9ZOzfdPZedLGsZLg3467+dJ5YVLwk1wIfAB4aGw7kuYw\ndYTWL4B3Du5F0kw8ck9qyOBLDRl8qSGDLzVk8KWGDL7UkMGXGjL4UkMGX2ooVbX6d5q8DJzLZ/Yv\nA366yu2sh1rWs95c9d5TVZcvt9GQ4J+rJPurauf5Vst61ltv9dzVlxoy+FJD6y34XzhPa1nPeuuq\n3rp6ji9pHuttxZc0A4MvNWTwpYYMvtSQwZca+h/QhH9mQMtUUQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f903123a940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_example(11, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Make a DummyClassifier\n",
    "\n",
    "The goal here is to make a basic no-intelligence classifier, anything we do after this point must show an improvement or else we're not learning anything! This classifier makes a classificiation based on the *dominant class* in the training set (for each fold)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Scores: 0.111 +/- 0.038 (2 s.d.)'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = DummyClassifier()\n",
    "cv = StratifiedKFold(n_splits=10, shuffle=True)\n",
    "\n",
    "scores = cross_val_score(clf, X, y, cv=cv)\n",
    "\n",
    "\"Scores: {:0.3f} +/- {:0.3f} (2 s.d.)\".format(scores.mean(), scores.std()*2)\n",
    "# We expect a 10% score if we're guessing the majority class for a \n",
    "# 10 class problem when we have an example number of examples per class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bernoulli NaiveBayes\n",
    "\n",
    "Now we'll make a proper (if simple) classifier and confirm that it outperforms the DummyClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Scores: 0.891 +/- 0.035 (2 s.d.)'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = naive_bayes.BernoulliNB()\n",
    "# you'll improve things by trying a new classifier here\n",
    "\n",
    "cv = StratifiedKFold(n_splits=10, shuffle=True)\n",
    "\n",
    "scores = cross_val_score(clf, X, y, cv=cv)\n",
    "\n",
    "\"Scores: {:0.3f} +/- {:0.3f} (2 s.d.)\".format(scores.mean(), scores.std()*2)\n",
    "# We expect approximately 88% accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next we'll Train on 66% of the Data and Test on 33%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score on the training data:\n",
      "0.901080631754\n",
      "Score on the test split:\n",
      "0.897306397306\n"
     ]
    }
   ],
   "source": [
    "# We'll fix a random seed so the results are the same on each run,\n",
    "# you wouldn't do this in practice but it does help when you're debugging\n",
    "random_state = 47\n",
    "test_size = 0.33\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=random_state)\n",
    "_, X_test_greyscale, _, _ = train_test_split(X_greyscale, y, test_size=test_size, random_state=random_state)\n",
    "\n",
    "print(\"Score on the training data:\")\n",
    "clf.fit(X_train, y_train)\n",
    "print(clf.score(X_train, y_train))\n",
    "\n",
    "print(\"Score on the test split:\")\n",
    "print(clf.score(X_test, y_test))\n",
    "# We expect both train and test scores to be pretty similar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Precision and Recall to Identify Badly Performing Classes\n",
    "\n",
    "```\n",
    "\"The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.\n",
    "\n",
    "\"The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.\"\n",
    "```\n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html\n",
    "\n",
    "```\n",
    "... precision (also called positive predictive value) is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved.\n",
    "```\n",
    "https://en.wikipedia.org/wiki/Precision_and_recall\n",
    "\n",
    "Class 1's Precision is worst (Class 0 is the best), we should probably fix this first. It predicts a 1 for most of class 1 items, but it also mis-classifies other examples as class 1 (oftens class 8s).\n",
    "\n",
    "Class 8 has the worst Recall. When the classifier predicts an 8 it is generally right, but often true 8s get classified as other classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEICAYAAACwDehOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFaxJREFUeJzt3X+0ZWV93/H3R34OAmIyQyvzgwEZMRNqhExRQxehBS0Q\nAza1LWSRNC6EdFUaU5O02BiCtCZNYmJMgolEDGoVCiTi1E6CYgTTtChDAMMMEgdEuAJhVBARIj/8\n9o+9Z+d4uXPvvnfunnPv5f1a66w5e5/nOfs5d55zPud59j57p6qQJAngeeNugCRp4TAUJEkdQ0GS\n1DEUJEkdQ0GS1DEUJEkdQ2EJS7IlyQkzlFmT5LEke+ymZkm7XZITkkyMLN+T5KRxtmmhMhTGpO2U\nT7QfyH+b5I+S7D+f26iq76+q62coc29V7V9Vz8zntqXpTOr/Dya5bL77v+bGUBivH62q/YFjgH8M\nvG30wTT8P9JStaP/vxw4GnjrmNsjDIUFoaq+AvwpcFSS65O8I8lfAo8Dhyd5QZJLkzyQ5CtJ/tvo\ndE+Sc5LckeSbSbYmOaZd3w2RkxybZHOSR9uRyW+169cmqSR7tsuHJNmY5OtJtiU5Z2Q7Fya5MskH\n221tSbJh9/2ltBRV1YPAtTThQJJ9krwzyb1tX/2DJMt2lE9yepJb2758V5KT2/VvGHkf3J3kp8fz\nihY3Q2EBSLIaOBW4pV31E8C5wAHAl4EPAE8DR9B8o3oN8Ma27r8CLgR+EjgQOA342hSbeTfw7qo6\nEHgxcOVOmnM5MAEcArwe+JUkJ448fhpwBXAQsBH4vdm+XmlUklXAKcC2dtWvAS+hCYkjgJXABW3Z\nY4EPAr9A0wePB+5p6z0EvJbmffAG4F07viBpFqrK2xhuNB35MeARmg/+9wDLgOuBi0bK/QPg28Cy\nkXVnAp9u718LvHmabZzU3v8M8HZg+aQya4EC9gRWA88AB4w8/qvAZe39C4HrRh5bDzwx7r+lt8V3\nG+n/32z736doPuQDfAt48UjZVwFfau+/F3hXz21cs+O9AZwATEza/knj/jssxJsjhfF6XVUdVFWH\nVtW/r6on2vX3jZQ5FNgLeCDJI0keoXljHNw+vhq4q8e2zqb59vWFJDclee0UZQ4Bvl5V3xxZ92Wa\nb2o7PDhy/3Fg3x1TT9Isva6qDqD5wH4psBxYAewH3DzS3/+sXQ/T9PckpyS5sZ36fIRm9L184New\n5PhmXphGT117H81IYXlVPT1F2ftopoOmf8KqLwJntjuufwy4Osn3Tip2P/A9SQ4YCYY1wFdm+wKk\nvqrqhiSXAe+k6ZtPAN9fzb62yabs70n2Af6YZhr1Y1X1VJJraEYemgVHCgtcVT0AfAL4zSQHJnle\nkhcn+eG2yPuAn0/yg+3RSkckOXTy8yQ5K8mKqvoOzZQVNFNFo9u6D/i/wK8m2TfJy2hGGB8e6vVJ\nrd8GXg28DPhDmv0BBwMkWZnkn7flLgXekOTE9r2wMslLgb2BfYDtwNNJTqHZ96ZZMhQWh5+k6fRb\ngYeBq4EXAVTVVcA7gI/QzM9eA3zPFM9xMrAlyWM0O53PqKq/m6LcmTT7Ge4HPgr8clV9cj5fjDRZ\nVW2n2YH8S8B/ptnpfGOSR4HrgCPbcp+j3YkMfAO4ATi0Hdn+DM0BFA8DP05zIIRmKe1OF0mSHClI\nkv6eoSBJ6hgKkqSOoSBJ6iy63yksX7681q5dO+5maIm6+eabv1pVK2YuOf/s2xpS37696EJh7dq1\nbN68edzN0BKV5Mvj2rZ9W0Pq27edPpIkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVJnsFBI8v4kDyW5\nfSePJ8nvtNcB/ryXzZOk8RtypHAZzemad+YUYF17Oxf4/QHbIknqYbBQqKrPAF+fpsjpwAercSNw\nUJIXDdUeSdLMxvmL5pV897WIJ9p1D0wumORcmtEEa9as2ekTrj3/f/fe+D3//Ud6l5WG0rdva2GY\nzWcMLM7PmXHuaJ7q2qlTXvGnqi6pqg1VtWHFirGclkYahH1bC804Q2ECWD2yvIrmEpCSpDEZ5/TR\nRuC8JFcArwC+0V6kXtIkTo1qdxksFJJcDpwALE8yAfwysBdAVf0BsAk4leYC3Y/TXIxbkjRGg4VC\nVZ05w+MFvGmo7UuSZs9fNEuSOovuIjvPZc+Fw+EkjZcjBUlSx1CQJHWcPtK0PBRSS5FTsTtnKDxH\n+OEuLU67+73r9JEkqWMoSJI6Th9JGjunNxcORwqSpI4jhTHwyAdpfjjCmH+OFCRJHUcKu8Bv/JKW\nGkcKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSORx9pEB4/Li1OjhQkSR1DQZLUcfpI0rM4/ffcZShoSfBD\nTJofTh9JkjqOFCTNG0dsi58jBUlSx1CQJHUMBUlSx1CQJHXc0awFxR2V0ng5UpAkdQwFSVJn0FBI\ncnKSO5NsS3L+FI+vSfLpJLck+XySU4dsjyRpeoOFQpI9gIuBU4D1wJlJ1k8q9jbgyqo6GjgDeM9Q\n7ZEkzWzIHc3HAtuq6m6AJFcApwNbR8oUcGB7/wXA/QO2R3rOcce9ZmvI6aOVwH0jyxPtulEXAmcl\nmQA2Af9hqidKcm6SzUk2b9++fYi2SmNh39ZCM2QoZIp1NWn5TOCyqloFnAp8KMmz2lRVl1TVhqra\nsGLFigGaKo2HfVsLzZChMAGsHllexbOnh84GrgSoqv8H7AssH7BNkqRpDBkKNwHrkhyWZG+aHckb\nJ5W5FzgRIMn30YSCY2hJGpPBQqGqngbOA64F7qA5ymhLkouSnNYW+zngnCS3AZcDP1VVk6eYJEm7\nyaCnuaiqTTQ7kEfXXTByfytw3JBtkCT15y+aJUkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkd\nL8cpSbvBYjljrSMFSVLHUJAkdQwFSVLHUJAkdQwFSVLHo49YPEcFSNLQHClIkjqGgiSpYyhIkjqG\ngiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSpYyhIkjqGgiSp\nYyhIkjqGgiSpM2goJDk5yZ1JtiU5fydl/nWSrUm2JPnIkO2RJE1vsGs0J9kDuBh4NTAB3JRkY1Vt\nHSmzDngrcFxVPZzk4KHaI0ma2ZAjhWOBbVV1d1U9CVwBnD6pzDnAxVX1MEBVPTRgeyRJM+gVCkk+\n1WfdJCuB+0aWJ9p1o14CvCTJXya5McnJfdojSRrGtNNHSfYF9gOWJ3khkPahA4FDZnjuTLGuptj+\nOuAEYBXwF0mOqqpHJrXjXOBcgDVr1sywWWnxsG9roZlppPDTwM3AS9t/d9w+RrO/YDoTwOqR5VXA\n/VOU+VhVPVVVXwLupAmJ71JVl1TVhqrasGLFihk2Ky0e9m0tNNOGQlW9u6oOA36+qg6vqsPa2w9U\n1e/N8Nw3AeuSHJZkb+AMYOOkMtcA/xQgyXKa6aS75/RKJEm7rNfRR1X1u0l+CFg7WqeqPjhNnaeT\nnAdcC+wBvL+qtiS5CNhcVRvbx16TZCvwDPALVfW1Ob8aSdIu6RUKST4EvBi4lebDG5r9AzsNBYCq\n2gRsmrTugpH7BbylvUmSxqzv7xQ2AOvbD3FJ0hLV93cKtwP/cMiGSJLGr+9IYTmwNcnngG/vWFlV\npw3SKknSWPQNhQuHbIQkaWHoe/TRDUkOBdZV1XVJ9qM5okiStIT0Pc3FOcDVwHvbVStpfmMgSVpC\n+u5ofhNwHPAoQFV9EfCMppK0xPQNhW+3ZzoFIMmePPs8RpKkRa5vKNyQ5L8Ay5K8GrgK+F/DNUuS\nNA59Q+F8YDvw1zQnydsEvG2oRkmSxqPvIanLaM5d9IfQXVVtGfD4UA2TJO1+fUcKn6IJgR2WAdfN\nf3MkSePUNxT2rarHdiy09/cbpkmSpHHpGwrfSnLMjoUkPwg8MUyTJEnj0nefwpuBq5LsuHLai4B/\nM0yTJEnjMmMoJHkesDfNJTmPpLn28heq6qmB2yZJ2s1mDIWq+k6S36yqV9GcQluStET13afwiST/\nMkkGbY0kaaz67lN4C/B84JkkT9BMIVVVHThYyyRJu13fU2cfMHRDJEnj1/fU2UlyVpJfapdXJzl2\n2KZJkna3vvsU3gO8Cvjxdvkx4OJBWiRJGpu++xReUVXHJLkFoKoeTrL3gO2SJI1B35HCU+1J8Aog\nyQrgO4O1SpI0Fn1D4XeAjwIHJ3kH8H+AXxmsVZKkseh79NGHk9wMnEhzOOrrquqOQVsmSdrtpg2F\nJPsC/w44guYCO++tqqd3R8MkSbvfTNNHHwA20ATCKcA7B2+RJGlsZpo+Wl9V/wggyaXA54ZvkiRp\nXGYaKXRnQnXaSJKWvplGCj+Q5NH2foBl7bLnPpKkJWjaUKiqPXZXQyRJ49f3dwqSpOeAQUMhyclJ\n7kyyLcn505R7fZJKsmHI9kiSpjdYKLSnxbiY5lDW9cCZSdZPUe4A4GeAzw7VFklSP0OOFI4FtlXV\n3VX1JHAFcPoU5f4r8OvA3w3YFklSD0OGwkrgvpHliXZdJ8nRwOqq+vh0T5Tk3CSbk2zevn37/LdU\nGhP7thaaIUNhqus5V/dg8jzgXcDPzfREVXVJVW2oqg0rVqyYxyZK42Xf1kIzZChMAKtHllcB948s\nHwAcBVyf5B7glcBGdzZL0vgMGQo3AeuSHNZekOcMYOOOB6vqG1W1vKrWVtVa4EbgtKraPGCbJEnT\nGCwU2tNinAdcC9wBXFlVW5JclOS0obYrSZq7vpfjnJOq2gRsmrTugp2UPWHItkiSZuYvmiVJHUNB\nktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQx\nFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJHUNBktQxFCRJ\nHUNBktQxFCRJHUNBktQxFCRJHUNBktQZNBSSnJzkziTbkpw/xeNvSbI1yeeTfCrJoUO2R5I0vcFC\nIckewMXAKcB64Mwk6ycVuwXYUFUvA64Gfn2o9kiSZjbkSOFYYFtV3V1VTwJXAKePFqiqT1fV4+3i\njcCqAdsjSZrBkKGwErhvZHmiXbczZwN/OtUDSc5NsjnJ5u3bt89jE6Xxsm9roRkyFDLFupqyYHIW\nsAH4jaker6pLqmpDVW1YsWLFPDZRGi/7thaaPQd87glg9cjyKuD+yYWSnAT8IvDDVfXtAdsjSZrB\nkCOFm4B1SQ5LsjdwBrBxtECSo4H3AqdV1UMDtkWS1MNgoVBVTwPnAdcCdwBXVtWWJBclOa0t9hvA\n/sBVSW5NsnEnTydJ2g2GnD6iqjYBmyatu2Dk/klDbl+SNDv+olmS1DEUJEkdQ0GS1DEUJEkdQ0GS\n1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEU\nJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkd\nQ0GS1DEUJEkdQ0GS1Bk0FJKcnOTOJNuSnD/F4/sk+Z/t459NsnbI9kiSpjdYKCTZA7gYOAVYD5yZ\nZP2kYmcDD1fVEcC7gF8bqj2SpJkNOVI4FthWVXdX1ZPAFcDpk8qcDnygvX81cGKSDNgmSdI0UlXD\nPHHyeuDkqnpju/wTwCuq6ryRMre3ZSba5bvaMl+d9FznAue2i0cCd86yOcuBr85YynoLtd7u3Oah\nVbViDtuZk13s2/7/WW82evXtPefQmL6m+sY/OYH6lKGqLgEumXNDks1VtcF6i7PeuLa5O+xK3/b/\nz3qzrdfHkNNHE8DqkeVVwP07K5NkT+AFwNcHbJMkaRpDhsJNwLokhyXZGzgD2DipzEbg37b3Xw/8\neQ01nyVJmtFg00dV9XSS84BrgT2A91fVliQXAZuraiNwKfChJNtoRghnDNScuU49WW9h1BvXNhc6\n//+sN+8G29EsSVp8/EWzJKljKEiSOks6FGY6zcY09d6f5KH2dxR966xO8ukkdyTZkuTNs6i7b5LP\nJbmtrfv2WdTdI8ktST7et05b754kf53k1iSbZ1HvoCRXJ/lC+1pf1aPOke12dtweTfKzPbf3H9u/\nye1JLk+yb896b27rbOm7rcVkLn17Lv26rTenvr0r/bqtP+u+Pdd+3da1bwNU1ZK80ezcvgs4HNgb\nuA1Y37Pu8cAxwO2z2N6LgGPa+wcAfzOL7QXYv72/F/BZ4JU9674F+Ajw8Vn+fe4Bls/h7/oB4I3t\n/b2Bg+bw//IgzQ9pZiq7EvgSsKxdvhL4qR71jgJuB/ajOZjiOmDduPvkfN3m2rfn0q/benPq27vS\nr9s6s+7bc+3XbV37dtWSHin0Oc3GlKrqM8zy9xJV9UBV/VV7/5vAHTT/8X3qVlU91i7u1d5mPAIg\nySrgR4D3zaatc5XkQJoPlksBqurJqnpklk9zInBXVX25Z/k9gWXt71j249m/dZnK9wE3VtXjVfU0\ncAPwL2bZzoVsTn17Lv26rTenvj3Xfg327WkM3reXciisBO4bWZ6g54f0rkpzttejab4Z9a2zR5Jb\ngYeAT1ZVn7q/Dfwn4DtzaGYBn0hyc3uqhT4OB7YDf9QO69+X5Pmz3O4ZwOW9Glj1FeCdwL3AA8A3\nquoTPareDhyf5HuT7Aecynf/kHKxWzR9e479Gubet+fSr8G+3VnKodDrFBrzvtFkf+CPgZ+tqkf7\n1quqZ6rq5TS//D42yVEzbOe1wENVdfMcm3pcVR1DcxbbNyU5vkedPWmmH36/qo4GvgXMZl/N3sBp\nwFU9y7+Q5hvwYcAhwPOTnDVTvaq6g+aMu58E/oxmeuXpvu1cBBZN355tv263syt9ey79GuzbnaUc\nCn1OszGvkuxF86b5cFX9yVyeox2yXg+cPEPR44DTktxDM33wz5L8j1ls5/7234eAj9JMScxkApgY\n+bZ3Nc0bqa9TgL+qqr/tWf4k4EtVtb2qngL+BPihPhWr6tKqOqaqjqeZMvniLNq50C26vj2Lfg27\n0Lfn2K/Bvt1ZyqHQ5zQb8yZJaOYj76iq35pl3RVJDmrvL6PpMF+Yrk5VvbWqVlXVWprX9udVNeM3\njXYbz09ywI77wGtohqXTqqoHgfuSHNmuOhHY2mebrTPpObxu3Qu8Msl+7d/3RJr57BklObj9dw3w\nY7Pc7kK3KPr2XPo1zL1vz7Vft9u0b+8wn3utF9qNZr7tb2iO1PjFWdS7nGae7ymabxBn96jzT2iG\n8J8Hbm1vp/bc3suAW9q6twMXzPJ1nsDsjtA4nGbYeRuwZZZ/m5cDm9u2XgO8sGe9/YCvAS+Y5Wt7\nO80Hye3Ah4B9etb7C5o39W3AiePui/N9m0vfnku/buvNqW/var9un6N3396Vft3Wt29XeZoLSdLf\nW8rTR5KkWTIUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1Pn/yzVLKlOnDiIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9030ed7048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predicted_y_test = clf.predict(X_test)\n",
    "precision, recall, fbeta_score, support = precision_recall_fscore_support(y_test, predicted_y_test)\n",
    "\n",
    "f, (ax1, ax2) = plt.subplots(1, 2, sharey=True)  #, fig_kw = {\"figsize\":(6,4)})\n",
    "#ax1.plot(x, y)\n",
    "ax1.set_title('Precision')\n",
    "ax1.set_ylabel(\"Percent\")\n",
    "ax2.set_title('Recall')\n",
    "\n",
    "# plot bars, set 10 class labels\n",
    "classes = np.arange(precision.shape[0])\n",
    "_ = ax1.bar(classes, precision)\n",
    "ax1.set_xticklabels([str(c) for c in classes])\n",
    "_ = ax1.set_xticks(classes)\n",
    "_ = ax2.bar(classes, recall)\n",
    "ax2.set_xticklabels([str(c) for c in classes])\n",
    "_ = ax2.set_xticks(classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Plot Confusion Matrix to Visualise Incorrect Classifications\n",
    "\n",
    "We see that the diagonal is pretty good (each class is often correctly predicted). Class 1 is sometimes mis-predicted, let's diagnose this.\n",
    "\n",
    "For random_seed=47 we see that Class 1 is often mislabelled (for 8s and 2s), also Class 8 is mislabelled for Class 1s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAEYCAYAAAB82RxTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXecFdX5xr/P7tKXKogIKF1FBYINKxawF7C32CWmSBR7\nSVBjizG2qEE0tthj4k9iiFGjKBopFhQRpdmQLr3vsu/vj5nVBbfc3Z2Zu3d4v3zmw71zZ85zptx3\nzz1zzvvIzHAcx3Fyh7xsV8BxHMepHh64HcdxcgwP3I7jODmGB27HcZwcwwO34zhOjuGB23EcJ8fw\nwF0HkdRI0j8lLZP0t1qUc5qkV6KsW7aQtK+kz2u473aSPpS0QtLQGOq2v6TZZd5PkbR/hvtmvG1S\nSPpS0oBs18OpmIJsVyCXkXQqMAzYHlgBTAJuMrO3a1n08UBbYAszK65pIWb2JPBkLesSO5IM6G5m\nMyraxszGAtvVUOJyYIyZ/aSG+1cLM9uxJttKug7oZmanV1ezNvs6uYe3uGuIpGHAXcDNBEF2G+B+\n4JgIit8WmFaboJ0mJNW2gbEtMCVL2o4TPWbmSzUXoDmwEjihkm0aEAT2OeFyF9Ag/Gx/YDZwCbAA\nmAucHX52PbAeKAo1zgWuA54oU3YnwICC8P1ZwCyCVv8XwGll1r9dZr+9gInAsvD/vcp8Ngb4HfBO\nWM4rQOsKjq20/peXqf8g4HBgGrAYuLrM9rsD7wJLw23vBeqHn70VHsuq8HhPKlP+FcA84K+l68J9\nuoYafcP3WwOLgP3LqevrwAZgbVh+j/D6PQ4sBL4CrgXyypyzd4A7Q40byymzEfAosAT4FListG7h\n518CA8ps+1i47dTwnP1oW+DQTa77R5Vd203qU9G+Z4eaK8IyflZmn9bAS+E1WQyMLXMOytZ/+1D3\n5Gx/73wpc82zXYFcXMIvSjFh4KxgmxuAccCWQBvgf8Dvws/2D/e/AahHEPBWAy3Dz69j40C96ftO\nhIEbaAIsB7YLP2sH7Bi+PoswcAOtwuDx03C/U8L3W4SfjwFmEgS2RuH7Wys4ttL6/zas//kEQfAp\noCmwI0Gg7BJuvwvQL9TtFAaTi8qUZwQ/8zct//cEfwAbUSZwh9ucH5bTGPgPcHsl12IMcF6Z948D\nL4Z17UTwx+bcMuesGLgwrG+jcsq7lSDQtQI6Ap9QceC+FXgTaAl0AD6uZNtNr3OF17acOm20b7ju\nCII/cgL6E9xjpX/sbgFGhNevHrAvoLJ1AvoCXwNHZvs758vGi3eV1IwtgEVWeVfGacANZrbAzBYS\ntKR/WubzovDzIjMbTdBSqmkfbgmwk6RGZjbXzMrrFjgCmG5mfzWzYjN7GvgMOKrMNo+Y2TQzWwM8\nB/SpRLOIoD+/CHiGoAV3t5mtCPWnAL0AzOx9MxsX6n4JPEAQSKo6puFmti6sz0aY2YPAdGA8QUC7\nporyAJCUT9Cqvyqs65fAH9n42swxsz+F9f2RNnBieOyLzewb4J5KJE8EbjazJWY2u4ptyyOTa1su\nZvYvM5tpAW8S/IraN/y4iOC8bRveg2PNrGzion2BUcCZZvZSNevsxIwH7prxHdC6iv7PrQl+hpfy\nVbju+zI2CfyrgcLqVsTMVhEEoguAuZL+JWn7DOpTWqf2Zd7Pq0Z9vjOzDeHr0uA2v8zna0r3l9RD\n0kuS5klaTvBcoHUlZQMsNLO1VWzzILAT8CczW1fFtqW0Burz42tT9jx8U0UZW2+yzabntbJtqyr7\ne6pxbctF0mGSxklaLGkpwS+70vP+B2AG8IqkWZKu3GT3C4D/mdkbmeo5yeGBu2a8S9AVMKiSbeYQ\nPBQrZZtwXU1YRdAlUMpWZT80s/+Y2UCCFtRnBAGtqvqU1unbGtapOvyZoF7dzawZcDXBz/fKqDRt\npaRCgucGfwGuk9Qqw7osImhtbnptyp6HqlJmziXoIim7f2XbdijzvmNFG5anm+G1/dG+khoAfwdu\nB9qaWQtgNOF5D39tXGJmXQh+dQ2TdFCZIi4AtpF0ZyX1dbKEB+4aYGbLCPp375M0SFJjSfXCFs5t\n4WZPA9dKaiOpdbj9EzWUnATsJ2kbSc2Bq0o/kNRW0tGSmgDrCLpcNpRTxmigh6RTJRVIOgnoSfCA\nKm6aEvTVrgxbjD/f5PP5QJdqlnk38L6ZnQf8i6C/tkrCXwnPATdJaippW4IhndW5Ns8BV0lqKakD\nQX94Jtu2B35VybbzgU6S8qBa1/ZH+xL8qmhA8OyhWNJhwMGlG0s6UlI3SSK4Nhs2KXsFwbOc/STd\nWkmdnSzggbuGmNkdBF/4awm+HN8QfCn/L9zkRuA9godRk4EPwnU10XoVeDYs6302DrZ5BKNT5hCM\nDugP/KKcMr4Djgy3/Y5gdMORZraoJnWqJpcCpxIEgwcJjqUs1wGPSVoq6cSqCpN0DEFQuSBcNQzo\nK+m0DOtzIcGvmFnA2wQPVR/OcF8Inld8RTDa4hWCUS8VcQPBCJkvgNeA5wmCcHmUTrb6TtIHZHht\ny9vXzFYAQwn+cCwhOP+jymzfPazPSoJfkPeb2ZiyBZrZUmAgcJik31VyjE7CaOPnEY7jxImknxMM\nravq4azjVIi3uB0nRiS1k7S3pDxJ2xG0oF/Idr2c3MYDt+PES32C4Y8rCCYDvUgww9ZJMZIelrRA\n0idl1rWS9Kqk6eH/LcP1knSPpBmSPpbUt6ryPXA7ToyY2VdmtpOZNTGz9uFIjvXZrpcTO48SPIcp\ny5XAf82sO/Df8D3AYQTPHLoDQwhGYVWKB27HcZyIMbO3CB4ol+UYgvQHhP8PKrP+8XCi1DighaR2\nlZXvgdtxHCcZ2prZXIDw/y3D9e3ZeGLWbDaeEPYj6mzms6NHTkxkuMtz5+yWhIyTY5QkONoqT1XN\nRXKqomFBlRO6qqTRT36V8UVfO+m+nxF0a5Qy0sxG1lC6vLpXWpc6G7gdx3ESRZl3QIRBurqBer6k\ndmY2N+wKWRCun83GM2o7UMUsa+8qcRzHAZAyX2rGKODM8PWZBCOMStefEY4u6QcsK+1SqQhvcTuO\n40C1WtxVFiU9TZCKuHVoazecIMXvc5LOJUiXe0K4+WiCBGAzCJK7nV1V+R64HcdxoDYt6R9hZqdU\n8NFBm64I0+n+sjrle+B2HMcByMvPdg0yJqf7uB88pRf3HL8jdx27I38c3BOATq0acdsxO3DP8Tty\n7SHdaVQv+kN8Z+xbHH3EIRx56ED+8mBNHyTXLa00HlOSWtddezUH7rcXxw86quqNI8DvixhQXuZL\nlsl+DWrJNf/8nIv+MYVLXvgUgAv368xjE2Yz9PkpjPtyCcf2rnQce7XZsGEDN990A/ePeIgXRv2L\nl0e/xMwZFZqT54RWGo8paa2jBg3mvhEVpcqOFr8vYiL+h5OREVvglrS9pCvCOfh3h693iEuvlPYt\nGjJl7goAJs1ezp6dW0Za/ieTP6Zjx23p0LEj9erX59DDj2DMG/+NVCNprTQeU9Jau+y6G82bN4+l\n7E3x+yImNvcWt6QrCHwIBUwgcBQX8HQ5Fkk1x+CGI3pwx+CeHLJ9GwC+WryGPbZtAcDeXVrSukn9\nyOQAFsyfz1btfjCg2bJtW+bPn1/JHnVfK43HlLRWkvh9ERM51OKO6+HkuQRu1EVlV0q6g8BENhJH\njStGTWXx6iKaNyzghiO2Y/bSNdzz5hcM2XsbTuq7NRO+WkpxSbQz4KycCU2K6UImpZXGY0paK0n8\nvogJfzhJCRsb45bSLvysXCQNkfSepPe+eqvqlMWLVwd/F5atLWbcl0vovmUh3y5by/DR0xj2wqe8\nNfM75i2vym+2erRtuxXz5v7gqbtg/ny23HLLSvao+1ppPKaktZLE74uY2Ny7SoCLgP9K+rekkeHy\nMkEqw19XtJOZjTSzXc1s1233G1ypQIOCvO9HjDQoyKNP++Z8vXg1zRsGPyIEnPiTrXl56sKIDilg\nx5125uuvv2T27G8oWr+el0f/i/4HHBipRtJaaTympLWSxO+LmMihwB1LV4mZvSypB7A7QZYrEczH\nnxiatdaaFo3qcfXB3QDIl3hz5nd8MHs5R+3UlsN7Bn+p3/1yCa99Hq2lYkFBAVdd81t+PuQ8Sko2\nMGjwcXTr1j1SjaS10nhMSWtdedkw3p84kaVLl3DIQf254BcXMvi442PR8vsiJvJypxutznpOenZA\nJ5t4dsDcIpLsgAfelPFFX/P6NVm9aD5z0nEcB+rEaJFM8cDtOI4DOTWqxAO34zgO1ImHjpnigdtx\nHAe8q8RxHCfn8Ba34zhOjuEt7tqT1DC9m16bnogOwOX7d01Mq15B7rQeqkOSw/SSIsljWl9c4cTl\nyKmf6D0YQdD1FrfjOE6O4aNKHMdxcgxvcTuO4+QY3sftOI6TY3iL23EcJ8fwFrfjOE6OkUMPJ3Pn\nt0EVJOEQXVKygVduG8rYB64HwMyY/NLjjL5xCP+++QKmvTkqUr1169ZxxqkncsoJgzhx8JE8cP+f\nIi2/LGl0807SeT2tWhAY+f70pGMZduHPY9VJ+rg2RVLGS7ZJReBOyiF6+pujaNa24/fvvxz/GquX\nLuSwq0dw2NUj2KbvfpHq1a9fnxEPPcLTf/s/nnruBf73zttM/nhSpBqQXjfvJJ3X06oF8OxTf6VT\n5/jnICR9XJvigTthknCIXr10EXOnTKTzngd/v27mO6PpecgpKC84jQ2btohUUxKNGzcBoLi4mOLi\nIhTFRINNSKubd5LO62nVmj9/Hu+MfZNjjj0udq0kj6tcVI0lyyQeuCWdHXWZSThET/rHSHodc85G\nf21XLprHNx+O5dXbL+KtEcNZseDbSDUhaKGeeuJgBh6wD3v024udevWOXGOzc/N2MubOP9zKry66\nFOXQiIua4i3uyrm+og/KmgVXp+8zbofoOZ9MoEFhC1p17LbR+pLiIvIL6jHw0rvosuchTHz67sg0\nS8nPz+ep515g9CtvMOWTycyYPi1yjc3OzdvJiLffGkOrlq3YoeeO2a5KIuRS4I5lVImkjyv6CGhb\n0X5mNhIYCbC2uJxveAXE7RC96ItPmfPJeOZOfY+SovUUrV3DuMdvp1GL1nTovTcA7XvtycSn7opM\nc1OaNmvGLrvtzrv/e5tu3XtEWvZm5+btZMRHkz7grTff4H9vv8W69etYtWoVw6++nOtvvi3bVYuF\nvLzc+VURV03bAmcAR5WzfBe1WNwO0b2OOoujbniMI4c/TL8zL2fL7r3od8altN+5H/OnfwTAwhmT\nKWzTPjJNgCWLF7Ni+XIA1q5dy4Rx79KpU+dINWAzdPN2MuKXQ4fx0itv8H//fo0bb/0ju+62R2qD\nNpBTfdxxjeN+CSg0sx8NgZA0JmqxbDlEbz/geMb/9Xamj3mRggYN2e2UCyMtf9GihQy/9ipKSjZQ\nUlLCwIMPZd/+B0SqAel1807SeT2tWkmS7eOqC10gmVJnXd6r01VSGzyta26RxrSuSZLWtK6N69U+\n6rY8/cmMb64lT5zmLu+O4zjZJpda3B64Hcdx8MDtOI6TcygvdwJ3OjtCHcdxqkmU47glXSxpiqRP\nJD0tqaGkzpLGS5ou6VlJ9WtaVw/cjuM4RBe4JbUHhgK7mtlOQD5wMvB74E4z6w4sAc6taV09cDuO\n40DU47gLgEaSCoDGwFzgQOD58PPHgEE1repm38d9Sf8uiWkNG/VpYlp/OnanxLTSSF4OPaiqDgU5\nNDswaarzcFLSEGBImVUjw5nfmNm3km4HvgbWAK8A7wNLzaw43H42UOMZe5t94HYcx4HqTXkvm55j\nUyS1BI4BOgNLgb8Bh5VXTPVrGeCB23Ech0iHAw4AvjCzhWG5/wD2AlpIKghb3R2AOTUV8N9NjuM4\nEGUf99dAP0mNFfw1OAj4FHgDKJ3DfybwYk2r6oHbcRyH6EaVmNl4goeQHwCTCeLsSOAKYJikGcAW\nwF9qWlfvKnEcxyHamZNmNhwYvsnqWcDuUZTvgdtxHIfcmvKemq6SJF3KIV7n61uO6MHwg7vx24Fd\nuWZAkFHw+F5tueHQ7gw/uBu/2GsbGtWL9tK5y3vtSds5nDdvLkPOPYPjjjmcEwYfyVNPPB6LTilZ\nd3nPU8ZLtklF4E7SObyUuJ2v/zjmC254dSY3vTYTgE/nr+K6/0zn+ldmMH/lOg7foU1kWu7yXnvS\neA7z8/O5+JIr+PuLo3n0iWf427NPMmtmfN8rd3nPnNgCt6TtJR0kqXCT9YdGrZWkczgk63xdyqfz\nV1ISjvqc9d1qWjaqF1nZ7vJee9J4Dtu02fJ7v8kmTQrp3LkrCxbEZ+ycbZf3zT5wSxpKMNTlQuAT\nSceU+fjmqPWSdg6P3fna4KL+nbh2QFf27dLyRx/v3bklk+euiEzOXd5rT9rP4ZxvZ/PZZ1PZaefe\nsepkk80+cAPnA7uY2SBgf+A3kn4dflbhUddVl/eyJOF8fevrs7jx1ZncPfZLDujWiu6tG3//2eE7\ntKGkBMZ/vSwyPXd5rz1pPoerV6/ismFDufTyqygsLKx6h1zFPSfJN7OVAGb2paT9geclbUslh11X\nXd7LkoTz9bK1QTqDFes28OG3K+i8RSOmL1rNntu2oFe7ptzx5heRaYG7vEdBWs9hUVERlw0bymFH\nHMWBAw6ORaOu4C7vME9Sn9I3YRA/EmgN7By1WJLO4XE7X9fPFw1Cr776+aJn20K+XbaOHbcq5NDt\nW3PvO1+xfkO0vovu8l570ngOzYzfDb+Wzp27cvoZZ0defl1DynzJNnG1uM8AisuuCOfnnyHpgajF\nsuXyHgfNGhbwi723ASBfYvzXy5gybyU3Hdadgvw8hu3XCYBZi9fwxPs1TnWwEe7yXnvSeA4nffgB\n/3rpRbp178EpJwQZSH859GL22bd/5FrgLu/VYbN3eV9btCEJGQAu++fUxLTSmtY1KZf3tKZ1LY74\n11plJNnzEIXLe4/LX8745Ey77VB3eXccx8k2udTi9sDtOI5D3ei7zhQP3I7jOEB+fu5Ebg/cjuM4\neFeJ4zhOzpFDcdsDt+M4DniLO6eoX5DcmKUkh+jd9Nr0xLSuGZDcmPk0DtNLcoheksN/S0py61p5\n4HYcx8kxcihue+B2HMcByKsDBgmZ4oHbcRwH7ypxHMfJOXIobnvgdhzHgdxqcedOAtoqcAPamlFS\nsoFXbhvK2AeuB4JRB5NfepzRNw7h3zdfwLQ3R0WumTZT3aS1kjLxXbduHWeceiKnnDCIEwcfyQP3\n/ykWHUjemLg8PK1rwpSapz7w4CO0bduWU086nv0POJCu3bpFrnXUoMGcdOpp/ObqKyMve1OSOK7p\nb46iWduOFK1dDcCX419j9dKFHHb1CJSXx9oVSyPTgmSvVVq1Sk18d+i5I6tWreT0k4+j35570aVr\ntFr169dnxEOP0LhxE4qLijj3rNPZa5992blXn6p3riZJHVNl5NLDyVS0uN2AtmasXrqIuVMm0nnP\nH5xNZr4zmp6HnILCnJwNm7aITA/SaaqbtFZSJr6SaNy4CQDFxcUUFxehmHy7kjYmLg/3nAQk7S5p\nt/B1T0nDJB0eh5Yb0NaMSf8YSa9jztnoRly5aB7ffDiWV2+/iLdGDGfFgm8j04P0mupm6x6M28R3\nw4YNnHriYAYesA979NuLnXrFbxacLWPiXOoqicvlfThwD/BnSbcA9wKFwJWSrolazw1oq8+cTybQ\noLAFrTpu/FO0pLiI/IJ6DLz0LrrseQgTn747Er1S0mqqm417MAkT3/z8fJ567gVGv/IGUz6ZzIzp\n02LRKSWbxsS51OKOq4/7eKAP0ACYB3Qws+WS/gCMB24qbydJQ4AhAPfe/wDnnj8kIzE3oK0+i774\nlDmfjGfu1PcoKVpP0do1jHv8dhq1aE2H3nsD0L7Xnkx86q5I9EpJq6lu0vdg0ia+TZs1Y5fddufd\n/71Nt+49YtHItjFxHYjHGRNXV0mxmW0ws9XATDNbDmBma4CSinYys5FmtquZ7Zpp0AY3oK0JvY46\ni6NueIwjhz9MvzMvZ8vuveh3xqW037kf86d/BMDCGZMpbNM+Er1S0miqm7RWUia+SxYvZsXy5QCs\nXbuWCePepVOnzrFo1QVjYm9xw3pJjcPAvUvpSknNqSRw1xQ3oI2O7Qccz/i/3s70MS9S0KAhu51y\nYaTlp9FUN2mtpEx8Fy1ayPBrr6KkZAMlJSUMPPhQ9u1/QKQapSRtTFweuTSqJBazYEkNzGxdOetb\nA+3MbHJVZSRlFpyU+Swkm9kurdkB00haswMm2TItbFB7sf3v+l/GJ2fMRXulzyy4vKAdrl8ELIpD\n03EcpzbUgR6QjEnFBBzHcZzaUhf6rjPFA7fjOA651eJOxcxJx3Gc2pInZbxUhaQWkp6X9JmkqZL2\nlNRK0quSpof/t6xxXWu6o+M4TprIy1PGSwbcDbxsZtsDvYGpwJXAf82sO/Df8H2NqLCrRFKzynYs\nHZvtOI6TBqIaDRjGzv2AswDMbD3BEOljgP3DzR4DxgBX1ESjsj7uKYDBRlllSt8bsE1NBOsaeVKi\nQwKTIskheuc981FiWg+dnGz+irSRn59cR26uGTtH+HCyC7AQeERSb+B94NdAWzObC2BmcyXVeGpt\nhYHbzDrWtNBcIo1B23Gc6lOduF02PUfISDMrTcJeAPQFLjSz8ZLuphbdIuWR0agSSScDXczsZkkd\nCP5yvB9lRRzHcbJJdVLWhkG6IreM2cBsMxsfvn+eIHDPl9QubG23AxbUtK5VPpyUdC9wAPDTcNVq\nYERNBR3Hceoi+XnKeKkMM5sHfCNpu3DVQcCnwCjgzHDdmcCLNa1rJi3uvcysr6QPw0otllS/poKO\n4zh1kYi75C8Engxj5SzgbIKG8nOSzgW+Bk6oaeGZBO4iSXkEDySRtAUxJIpyHMfJJlE+TDWzScCu\n5Xx0UBTlZzKO+z7g70AbSdcDbwO/j0LccRynrpAqBxwzexy4FrgdWAycYGbPxF2x6uIu73Vb585B\nO3DLET246fAe3HBYMFTx2F5tuefYntx0eLC+99ZNI9d1l/fakeT9nuS1Ko805uPOB4oIukvq3GxL\nd3nPDZ2bXpvJynUbNlr38tSFjJ66MFKdUtzlvfYkdb8nef4qog7E44zJZFTJNcDTwNZAB+ApSVdV\nV0hSbM0Cd3nPDZ2kcZf32pPU/V4X7sF8KeMl22TS4j4d2CV0s0HSTQQzgW6paAdJozZdBRwgqQWA\nmR1ds+qWT3kO25M//jhKiayQ1HEloWMYVx7UBTN4ffp3vDFjMQADt2vNPl1a8sV3a3jygzmsXr+h\nipIyJ8n7Ilv3YLYc0aOmLnyH60IXSKZkEri/2mS7AoLhLZXRgWDc4kP8ME1+V+CPNahjlbjLe93X\nueE/M1i6pphmDQq4YkAX5ixfx2vTvuOFyfPB4PjeW3Fa3615cNw3kWm6y3vuUBe+wznkXFZxV4mk\nOyXdQTDhZoqkhyQ9CEwGllZR7q4ErfJrgGVmNgZYY2ZvmtmblWgOkfSepPeq83DCXd7rvs7SNcUA\nLF9XzPvfLKPrFo1ZvrYYs+Av+xszvqNL60aRarrLe+5QF77DufRwsrI+7k8IEk39C7gOeBcYB9wA\nvF5ZoWZWYmZ3Egw6vyacfVll695d3jcmqeOKW6dBfh4NC/K+f71Tu6bMXrqWFo1+uCV27dic2UvX\nRqYJ7vKeS9SF73AuDQesLMnUX2pbuJnNBk6QdAQQWxpYd3mv2zrNGhVwUf9OQPAA6H9fLuHjuSu4\nYK+ObNuyEQYsWrWeh8fPjkwT3OU9CpK635M8fxVRF1rSmVKly7ukrsBNQE+gYel6M+sRZ8Xc5T13\n8LSutSNJl/e8BAfzJnm/NyyoRoaoCjj7mckZX4hHTt45q1/mTC7jo8AjBA8YDwOeA+rcBBzHcZza\noGos2SaTwN3YzP4DYGYzzexagmyBjuM4qSFKz8m4yWQ44DoFnT8zJV0AfAvk/pANx3GcMtSBeJwx\nmQTui4FCYChBX3dz4Jw4K+U4jpM0ufRwMpMheqUuDiv4wUzBcRwnVVRlkFCXqMzl/QWoeGSHmR0b\nS40cx3GyQA41uCttcd+bWC2ySJIPGtJqTDzypF6Jae3xu2QSD43/TST57jMirUP0khzmSEHtjysV\nXSVmlvvp4RzHcTKkzuWrroRM83E7juOkmlS0uB3HcTYncujZZOaBW1IDM1sXZ2Ucx3GyRS6NKsnE\nAWd3SZOB6eH73pL+FHvNHMdxEiRPmS/ZJpP++HuAI4HvAMzsI3zKu+M4KSOX0rpmErjzzOyrTdZF\n5y8VEWl0807SYTstWttu0ZhnL9j9++Wdq/pzWr+ONGtUwIgz+jBq6J6MOKMPTRtG/3gnjfdFGp3r\nKyKXcpVkEri/kbQ7YJLyJV0ETIu5XtWi1CH6/hEP8cKof/Hy6JeYOWNGzmsdNWgw9414MJay06r1\n1XerOWnEBE4aMYFTHpjA2qINvD51Iefs04kJs5Zw9D3vMmHWEs7dd9tIddN4XyR5TKXO9X9/cTSP\nPvEMf3v2SWbNjEerIvKqsWSbTOrwc2AYsA0wH+gXrssYSftIGiYpFo+ltLp5J+kon0atPbq04psl\na5i7bC0HbN+aUZPmAjBq0lwO2L5NpFppvC/S6lxfEfl5ynjJNlUGbjNbYGYnm1nrcDnZzBZVto+k\nCWVen08wC7MpMFzSlbWu9SaU5xA9f348Fz1JLad2HLpTW16eHFybVk3qs2jlegAWrVxPqyb1I9VK\n432RrWPKlnN9LvVxV9nRFxoE/2juqplVZgpZr8zrIcBAM1so6XYC38pbK9AaEm7Pvfc/QKa+k2l3\n83aqT0G+6L9da+5+bWYiemm8LzY35/o60JDOmEye0LxW5nVDYDDwTRX75ElqSdCil5ktBDCzVZKK\nK9rJzEYCI6F61mVpdvN2asY+3bbgs7krWLwqaGUvXrWe1oVBq7t1Yf3v10dFGu+Lzc25vi48dMyU\nTLpKni2zPAYcS+A/WRnNgfeB94BWkrYCkFRIDM4/aXXzdmrOYTtvxb8n//Czfsznizi6TzsAju7T\njjc+q7QBqTedAAAaBUlEQVS3r9qk8b7Y3JzrU9VVUg6dgUofyZtZpwo+KiFosUdKWt28k3SUT5NW\nw3p59Ovait/9c+r36x4e+yV/OHFnBvXdmnnL1nLpc5Mj04N03hdpda6viFzqKsnE5X0JP/Rx5wGL\ngSvN7Lk4K5aUy3uSpDWta5LseePriegkmdY1yfsirWldCxvU/sBufX1mxhW+8sCuWQ3zlba4Q6/J\n3gQ+kwAlVlWkdxzHyUFyqcVdaR93GKRfMLMN4eJB23GcVCIp4yXbZDIBZ4KkvrHXxHEcJ4ukIsmU\npNJulH0Igvfnkj6Q9KGkD5KpnuM4TjJEPaokTBHyoaSXwvedJY2XNF3Ss5JqPAussj7uCUBfYFBN\nC3ccx8kVYnhw+2tgKtAsfP974E4ze0bSCOBc4M81KbiyrhIBmNnM8paaiDmO49RV8vMyX6pCUgfg\nCOCh8L2AA4Hnw00eoxaN4spa3G0kDavoQzO7o6aidYkkhywV5NeBzrEYSHI4W1LD9I77y4SqN4qI\nv5+7e2JaSZJr93teNeYGlk3PETIynPldyl3A5QQ5mgC2AJaaWenM8dlA+5rWtbLAnQ/EMtPRcRyn\nrlGdnpKy6Tl+XI6OBBaY2fuS9i9dXV4x1azi91QWuOea2Q01LdhxHCeXiHC0yN7A0ZIOJ8jv1Iyg\nBd5CUkHY6u4AzKmpQJV93I7jOJsDUTngmNlVZtYhTP1xMvC6mZ0GvAGU5iY4E3ixxnWt5LPk5vw6\njuNkmQSMFK4AhkmaQdDn/ZeaFlRhV4mZLa5poY7jOLlGHBMizWwMMCZ8PQuI5El09I6pjuM4OUhd\n8JLMlFyqa6Wk1Y06qeNKSgfS5VLepH4+Vw3sxogTd2bEiTuzfdtCChvkc+MR2zHy5F7ceMR2FNbP\nj1w3jfdFklrlkbZcJXWetLpRJ3VcSZ4/SJdL+ZC9tuX9b5ZxwXOT+dXzn/DNkjWc0GdrPvp2OUOe\n+ZiPvl3OCT9pF6lmGu+LpO/B8lA1lmwTS+CWtIekZuHrRpKul/RPSb+XFLk9dVrdqJM6riTPH6TH\npbxRvTx2ateUVz5bCEBxibFq/Qb6dWrBa9MCh53Xpi2iX6eWkWlCOu+LpO/B8ohqVEkidY2p3IeB\n1eHruwmszH4frnskarG0ulEndVxpdCiH+I+rXbOGLFtbxMX7d+ae43Zk6H6daFCQR4tG9ViyugiA\nJauLaNGoXhUlVY803hd14R5MRXbA2pZbZmrnrmZ2kZm9bWbXA10q2knSEEnvSXqvOn1caXWjTuq4\n0uhQDvEfV55Et9ZNGP3pAob+fQpri0s4oU+03SLlkcb7oi7cg7nUxx3XqJJPJJ1tZo8AH0na1cze\nk9QDKKpop1xweYfk3KiTOq40OpRD/Mf13ar1LFq1ns8XrALgnVmLOaHP1ixdU0TLxkGru2Xjeixd\nU+EtXyPSeF/UhXswlx74xVXX84D+kmYSOMK/K2kW8GD4WaSk1Y06qeNKo0M5xH9cS9YUsXDleto3\nbwhA7/bN+XrpGsZ/tZQBPVoDMKBHa8Z9uTQyTUjnfVEX7sHNvsVtZsuAsyQ1JegaKQBmm1ksnVZp\ndaNO6riSPH+QLpfyB975issO6kpBnpi3fB13jZmFJK4c2JWB27dh4cp13PJqtKMj0nhfJH0Plkf2\nw3HmVOnyni2Scnn3tK61J40u5Z7WNbdoWFD7uPv8R3MzvpGP792u7rq8O47jbC7k14EukEzxwO04\njkNudZV44HYcxyGeJFNx4YHbcRyH6lmXZRsP3I7jOHiLO6dI60iPJKkLuRuiJumRHje9Nj0RnWsG\nJDvELpfIpft4sw/cjpNtkgraTuV4V4njOE6OkUMNbg/cjuM44IHbcRwn55B3lTiO4+QWdSHPdqZ4\n4HYcxyG3RpXkUgraSkmrqambwrpWKSUlG3jltqGMfeB6IEgxPPmlxxl94xD+ffMFTHtzVOSaaTp/\nVaFq/Ms2qQjcaTU1dVNY1yrL9DdH0axtx+/ffzn+NVYvXchhV4/gsKtHsE3f/SLVS9v5qwq3LkuY\ntJqauimsa5Wyeuki5k6ZSOc9f3BcmvnOaHoecgrKC77GDZu2iEwP0nX+MmGzb3FLGiqpY9VbRkNa\nTU3dFNa1Spn0j5H0OuacjdxXVi6axzcfjuXV2y/irRHDWbHg28j0IF3nLxOkzJdsE1eL+3fAeElj\nJf1CUpuYdID0mpq6KaxrAcz5ZAINClvQqmO3jdaXFBeRX1CPgZfeRZc9D2Hi03dHoldKWs5fpqga\nS7aJK3DPAjoQBPBdgE8lvSzpzNDOrFxq6vKeVlNTN4V1LYBFX3zKnE/G89L15zDusdtYMP1jxj1+\nO41atKZD770BaN9rT5bN+TISvVLScv4yJV/KeMk2cQVuM7MSM3vFzM4FtgbuBw4lCOoV7TTSzHY1\ns13PPX9IxmJpNTV1U1jXAuh11FkcdcNjHDn8YfqdeTlbdu9FvzMupf3O/Zg//SMAFs6YTGGb9pHo\nlZKW85cxOdTkjmsc90aHZmZFwChglKRGUYul1dTUTWFdqzK2H3A84/96O9PHvEhBg4bsdsqFkZaf\n9vO3KXXhoWOmxGIWLKmHmU2rTRlJmQU7TrZJMjtgWtO6RmEWPGHWsoxjzu5dmqfPLLi2QdtxHCdp\ncqe97VPeHcdxgORHsdQGD9yO4zjUjfHZmZKKmZOO4zi1JapBJZI6SnpD0lRJUyT9OlzfStKrkqaH\n/7esaV09cDuO40CUwwGLgUvMbAegH/BLST2BK4H/mll34L/h+xrhgdtxHIfocpWY2Vwz+yB8vQKY\nCrQHjgEeCzd7DBhU47rGMRwwCpIaDrhg+bokZABo3bR+Ylq5lFu4OhRvSOZ+zUuwSZPktfrNy58n\npjV8YI/EtAob1P4kTvp6RcY3V59tmmakJ6kT8BawE/C1mbUo89kSM6tRd4m3uB3Hcahekqmy6TnC\n5UdTvSUVAn8HLjKz5VHW1UeVOI7jUL2Zk2Y2EqgwoZKkegRB+0kz+0e4er6kdmY2V1I7YEFN6+ot\nbsdxHKJL66pgQPhfgKlmdkeZj0YBZ4avzwRerGldvcXtOI5DpDMn9wZ+CkyWNClcdzVwK/CcpHOB\nr4ETairggdtxHAcii9xm9nYlpR0UhYYHbsdxHHJrJFZq+rjjdIi+/cbfcsLh/Tn/tMHfr3vzv69w\n3qmDOXiv3nw+dUqkeqVcd+3VHLjfXhw/6KhYyi9LGt28582by5Bzz+C4Yw7nhMFH8tQTj8emleS1\ngvjPoZVsYMwff834h24AYOG0j3jzjosY88df8/afrmDlojmR6iV5rSoih9JxpyNwx+0QffARR3Pz\nnX/eaF2nrt0Yfssd7Nxnl8h0NuWoQYO5b8SDsZVfSlrdvPPz87n4kiv4+4ujefSJZ/jbs08ya2Y8\nWkldK0jmHM4a+0+alnGU//jvf6bvaZew/yV3075vf6a/+lykekleqwrJocgdl1lwfUlnSBoQvj9V\n0r2SfhkOk4mUuB2ie/1kV5o2a77Rum07daHjtp0j0yiPXXbdjebNm1e9YS1Jq5t3mzZbskPPHQFo\n0qSQzp27smBBPAa0SV0riP8crlm6iPmfvsc2ewz8YaVE8drVABSvXUXD5q0i04Nkr1VF5JLLe1x9\n3I+EZTeWdCZQCPyDoGN+d34YEhMJ5TlET/744yglUk2S5y9b12rOt7P57LOp7LRz79i14ibuc/jJ\niw/R88izKF635vt1fU78FeMeuoH8evUpaNiYfYf+ITK9TcnWtcqhLu7YAvfOZtZLUgHwLbC1mW2Q\n9ATwUdRidcEhOpdJu5v36tWruGzYUC69/CoKCwtj1UqCOM/hvE8n0qCwOS06dmPRjMnfr5/51ov0\nO++3tNx2O2a88Q+mvPgX+pwUrVUaZPda5VLEiCtw50mqDzQBGgPNgcVAA6DCrpJw2ugQgHvvf4BM\nDYPrgkN0LpNmN++ioiIuGzaUw444igMHHBybTpLEeQ4Xf/Ep86ZMYP7U9ykpXk/x2tWMe+gGVi6Y\nTctttwNg6z77Mu7B6yLRK0u2r1UuNfbiejj5F+AzYBJwDfA3SQ8CE4FnKtopF1ze00ha3bzNjN8N\nv5bOnbty+hlnx6KRDeI8hz2POJODf/sIA699iF1Ov4zW3Xqx+9nXULxmFSsXfgvAwmkf0nTLDpHo\nlVIXrlVUMycTqWtc2QElbQ1gZnMktQAGEGTHmpDJ/tXNDjj2rTe57dabv3eIPv9nP89ov0yyA970\n28v5+IP3WLZ0KS1bteKM835B02bNue+OW1i2dAlNCpvStcf23HrXiErLqW52wCsvG8b7EyeydOkS\nWm2xBRf84kIGH3d8RvtWd0xqTc9fTaiNVnWyA374wfucd9ZpdOveg7ww3d8vh17MPvv2r3Lf6mYH\nTPJaQc3PYXWyAy6aMZmZY15gj/N+y9zJ7/LZy08hiXqNC+lz0lCabLFVpftXJztgba4VRJMdcOaC\nNRnfXF23bJTV8O1pXT2ta07haV1rh6d1rZiZC6sRuNtkN3D7zEnHcRyqlx0w23jgdhzHoW70XWeK\nB27HcRw8cDuO4+Qc3lXiOI6TY3iL23EcJ8fIobjtwwHTSlFxSWJa9QpSkWTSiZiLX/w0Ma0/H9ez\n1nF39pJ1GcecDi0b+HBAx3GcbJNLU949cDuO45BbXSUeuB3HcfCHk47jODmHDwd0HMfJNXInbnvg\ndhzHgZyK2+kwC4Z0upQnqbVu3TrOOPVETjlhECcOPpIH7v9TbFppPH9p1Ypb58ZDu3HtgC5cfVAX\nrjww8HDt274pvxnYhfuO3YFtWjSMXLMi8qSMl2yTihZ3qev1Aw8+Qtu2bTn1pOPZ/4AD6dqtm2tl\nSP369Rnx0CM0btyE4qIizj3rdPbaZ1927tUnUp20nr80aiWlc+dbX7Fq/Ybv389Zvo6R787m1L7t\nItWpkuzH44yJrcUtqaukSyXdLemPki6QFIsNdlpdypPUkkTjxk0AKC4upri4KJaHNWk9f2nUSvKY\nyjJvxXrmr1wfu86mqBpLtoklcEsaCowAGgK7AY2AjsC7kvaPWq881+v58+dHLZNqLQhaWKeeOJiB\nB+zDHv32Yqde0btsp/X8pVErCR0Dhu6zDVcd2Jl9OreItOzqkkvWZXG1uM8HDjWzGwksy3qa2TXA\nocCdFe0kaYik9yS9V53+tLS6lCftiJ6fn89Tz73A6FfeYMonk5kxfVrkGmk9f2nUSkLn9jFfcsvr\nX3DvO1/Tv0srurVuHGn51UHV+Jdt4uzjLgA2EDi7NwUws68lVejybmYjgZFQvVwlaXUpz5Z7fdNm\nzdhlt915939v0617tPZTaT1/adRKQmfZ2mIAVqzbwKQ5K+jUshEzFq2OVCNT6kJLOlPianE/BEyU\nNBJ4F7gXQFIbYHHUYml1KU9Sa8nixaxYvhyAtWvXMmHcu3Tq1DlynbSevzRqxa1TP180CBOU1c8X\nO7RtwpzlayMrv7rkUldJLC1uM7tb0mvADsAdZvZZuH4hsF/UegUFBVx1zW/5+ZDzvne97tate9Qy\nqdZatGghw6+9ipKSDZSUlDDw4EPZt/8Bkeuk9fylUStunWYNC/hZv45AYM488evlfDp/Fb23bspJ\nvbeisEE+v9x7G2YvW8uf3v46Mt2KqAtdIJniaV1Tiqd1dbJNrqV1Xb62JOOY06xhnqd1dRzHyTa5\n0972wO04jhOQQ5HbA7fjOA7UiansmeKdk47jOEQ7c1LSoZI+lzRD0pVR19UDt+M4DkQWuSXlA/cB\nhwE9gVMk9Yyyqh64HcdxiHTm5O7ADDObZWbrgWeAYyKtrJmlagGGpEnHtXJLK43HlGat2tQReK/M\nMqTMZ8cDD5V5/1Pg3ij109jiHpIyHdfKLa00HlOatWqEmY00s13LLGWTK5XXJI90XkoaA7fjOE42\nmU2QDbWUDsCcKAU8cDuO40TLRKC7pM6S6gMnA6OiFEjjOO54PaOS13Gt3NJK4zGlWStyzKxY0q+A\n/wD5wMNmNiVKjTqbq8RxHMcpH+8qcRzHyTE8cDuO4+QYHridrKA4PdiygKQmCWptlbbz51SPnA/c\nkraTtKekeuFU07j1YtcIdbpJ2lVSgwS0dpTUX9IWMevsI+mnAGZmcQYfSUdJ+nVc5W+idQzwe0mx\n+8pJOgR4gY2Hm8Wh00/ST8P/68es1T281/OT+n7lOjkduCUdC7wI3Aj8BfilpGYxafUAMLMNcd9c\nko4E/gH8AXi0VDsmrcOAp4GLgcclbVXFLjXRyJNUCDwAXCXpAvg+eEd+D0o6GPgdEHsmf0n9gd8D\nL5rZgpi1Dg612gGXxKhzNMHIjgHApcC2MWoNAp4HrgLuAH6W5K+XXCVnA3doOnwScK6ZHUQQwDsC\nl0cdvMNAOknSUxBv8Ja0F3A7cKaZHQAsASLPLhZq7Q/cDZxnZoOA9cBOUeuYWYmZrQQeI/gDu5ek\ni0s/i1IrPH9/JZiC/Kqk5pK2lRSXffguBNObX5W0taSBkvaQ1DxKEUkDgPuB04DuwA6SIrcBDH91\n/RI41czOBJYDfSRtKalhDFo/A04xs+OAj4CzgYslNY1SK23kbOAOaUZwE0Pw8/EloD5walQ/w8O/\n/r8CLgLWS3oCYm9532pmH4avhwOtYuoymQ/8zMwmhC3tPYBfSXpA0vExdGUUE/xxfQzYXdIdkm5R\nQFT34ndAEdAuDAz/B/yZ4JdLXMdUyvPAOQT3y32SWkaokw+cEY4HbgJ8DuwIkT8vKAYaAduHDaD9\ngTOAu4BrI24NFwOFwFYAZvYw8BXQBjgyQp30ke1kLbVM9DKQYEbSvuH7fOBU4AnCMeoR6WxNcIO1\nJvhyPhHjMeUDzcq87gB8CLQJ120Rk+41wLXh67OBZ0s1I9ToClwZvr4EWA3cF8Ox9AZmEUw9Pp+g\ngXIOQZdQq4i1diIIos8AZ4frugAjgENiOLa88P9DgXnAzjFoHA+8D4wDfhOuOxB4FOgdsdYFBL+Q\nfgrcFH53f0YwaSXS40rTkust7rHAK8BPJe1nZhvM7CmCQNs7KhEzm2NmK81sEcFN1ai05S2pr6Tt\nI9TaYGbLw7cClgKLzWyhpNOAGyU1ikqvjO5NZnZj+PoRoCnRPwBbA2wn6XyCL+ytwDaSfhaliJl9\nRNBiu8XMHrSgq+ZhoCWwTcRanxD0A+8BdA7XzSL4o9smSq2w7JLw/5cJ+qGPjPgXC2b2PEH/9liC\nRgNm9jrBPRF1f/fTwMsEfxgam9npZvYAsGVcz6vSQE5PeTeztZKeJMi8dVUYQNcBbYG5MWl+Fwaa\nP0j6jOALekBMWsXASknfSLoFOBg4y8zWRKkjSRY2f8L3xxGcw0gT45jZHEnfAL8Bfmlm/5R0ADAj\nSp1Q61PKPJwMj6kN8dwX/ybo0rpO0lfhup8Q/GGKk48IHirfZmYboizYzJZIeh04UdJ6oCHBH6aP\nI9ZZBjwp6enSP0qSzgBaAZEeU5pIxZT3cLjS3gSt4bXA3fZDH3FcmhcDVwADzWxyTBoC6gFTw/8P\nMrPpcWiFeg2A04FhwElhazJqjY7Almb2fvg+zyJ+QLmJngi6fi4FTrCIc0ZsotWXoJuhAfBoXPfF\nJprPAZeb2ZcxlN2CoH/7OILv1eXhr5nYkHQOwbU6KYnzl6ukInCXEj4stDgDQajTEngOuMTMIm2B\nVKB3FjAxzqAT6tQjeG4w08w+j1lro1Z+nDpAf2CemX0Wt15SJHX+Qq2mBLFieZUb115rW6CemUX+\nKyxNpCpwJ4mkhma2NiGtxL6kjuPUfTxwO47j5Bi5PqrEcRxns8MDt+M4To7hgdtxHCfH8MDtVIqk\nDZImSfpE0t9qk/ND0v6SXgpfHy2pwhwsklpI+kUNNK6TdGmm6zfZ5lFJx1dDq5OkyIdMOk5VeOB2\nqmKNmfUxs50IklBdUPbDms7aM7NRZlbZBJUWQLUDt+NsDnjgdqrDWKBb2NKcKul+4AOgo6SDJb0r\n6YOwZV4IIOlQSZ9Jehs4trQgSWdJujd83VbSC5I+Cpe9CGYddg1b+38It7tM0kRJH0u6vkxZ10j6\nXNJrwHZVHYSk88NyPpL0901+RQyQNFbSNAVZIVGQJ/oPZbQjnaLvONXFA7eTEZIKgMOA0tls2wGP\nm9lPgFXAtcAAM+sLvAcMU5AG9EHgKGBfwixw5XAP8KaZ9Qb6AlMIUtnODFv7lynIRd0d2B3oA+wi\naT9JuwAnE0wxPxbYLYPD+YeZ7RbqTQXOLfNZJ4IJO0cAI8JjOBdYZma7heWfL6lzBjqOEws5navE\nSYRGkiaFr8cS5NPeGvjKzMaF6/sBPYF3gomK1AfeBbYHviidpq8gMdeQcjQOJJhaTZhzY5l+nBL1\n4HApTWVQSBDImwIvmNnqUGNUBse0k6QbCbpjCoH/lPnsuXDm7XRJs8JjOBjoVab/u3moPS0DLceJ\nHA/cTlWsMbM+ZVeEwXlV2VXAq2Z2yibb9SFIABYFIsj298AmGhfVQONRYJCZfRSmE9i/zGeblmWh\n9oVmVjbAI6lTNXUdJxK8q8SJgnHA3pK6AUhqrMBu7TOgs6Su4XanVLD/f4Gfh/vmK0jnuYKgNV3K\nf4BzyvSdt1fg8fgWMFhSozCnxlEZ1LcpMDfMzXLaJp+doMBqrStBXu3PQ+2fh9sjqYfcXsvJIt7i\ndmpNmCv8LOBp/eDUc62ZTZM0BPiXpEXA25RvjfZrYKSkcwlSef7czN6V9E443O7fYT/3DsC7YYt/\nJXC6mX0g6VlgEoF7ytgMqvwbYHy4/WQ2/gPxOfAmQVrbC8LUwQ8R9H1/ECatWggMyuzsOE70eK4S\nx3GcHMO7ShzHcXIMD9yO4zg5hgdux3GcHMMDt+M4To7hgdtxHCfH8MDtOI6TY3jgdhzHyTE8cDuO\n4+QY/w/KudV70his3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9030d64710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_confusion_matrix(df, cmap=plt.cm.Blues, title=\"Confusion matrix\"):\n",
    "    \"\"\"Use Seaborn to plot the confusion matrix\"\"\"\n",
    "    sns.heatmap(df, annot=True, vmin=0, vmax=100, cmap=cmap)\n",
    "    plt.ylabel('True label')\n",
    "    plt.yticks(rotation=90)\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.xticks(rotation=45)\n",
    "    plt.title(title)\n",
    "    \n",
    "cm = confusion_matrix(y_test, clf.predict(X_test))\n",
    "\n",
    "col_names = [str(n) for n in range(10)]\n",
    "cm_df = pd.DataFrame(data=cm, columns=col_names, index=col_names)\n",
    "plot_confusion_matrix(cm_df, title=\"Confusion matrix for digits task\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show probabilities for each example of this class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get the indices for class 1 for our diagnosis\n",
    "target_class = 1 # class we want to investigate\n",
    "true_labels_1s_indices = y_test == target_class\n",
    "predictions_with_1s_as_truth = clf.predict(X_test)[true_labels_1s_indices]\n",
    "# Get the class predictions for every example of Class 1\n",
    "predictions_probabilities_with_1s_as_truth = clf.predict_proba(X_test)[true_labels_1s_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We hope each Class 1 example to be labelled a 1\n",
    "\n",
    "Each mis-prediction means that another class had a higher probability of being chosen than Class 1, for this Class 1 example. We'll draw a matrix of examples (rows) and the 10 class probabilities to see which ones are more likely to be chosen for a set of Class 1 examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAEnCAYAAADo7YtpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucHFWZxvHfkyGQAAkEwi0kXIQIhKzcBFREQC6CK+Aq\nKoggFwleIoqoC4qKgKKyeGGBlaAYFQUBRYOCyK6iwHIJKqgE0RjEhMByGwgIgYS8+8c5A5XJ9ExP\n0lXdNXm++fQn3V3V9Z6qqX771Kk6pxQRmJlZeYa1uwBmZkOdE62ZWcmcaM3MSuZEa2ZWMidaM7OS\nOdGamZWs9olW0mmSLlnOzx4l6aZ+pl8r6d19zSvpaUkv6+ezd0vac3nKtTwkjZR0taQnJV1RUczN\nJIWkVTphOYOIN13Smcv52a9L+lSryzRAzOUub5PLf3Ff7r0fSTpc0i+Wc7n9fr9WJpXs2L1J+juw\nAfAC8E/gGuCDEfF0O8rTSEQc0M+0NXueS5oOzIuIUwvTty23dMs4hLRN142IxRXHXmlExHvbXYZW\nK+7L9L0ffa/6Ug0t7azRHpj/wDsCOwOn9p5BSe1r3RXZFPiLk6ytIO9HJWh7EouIB4BrgckAkm6Q\n9DlJNwPPAC+TNE7SDEmPS5ot6bheixkh6QeSnpL0O0nb9UyQdLKkv+VpsyT9W6/PStJ/5kOlP0va\nuzDhBknv6avc+VB3S0lTgMOBj+dDsKvz9L9L2ic/H1Yox2OSLpe0Tp42QtIl+f0nJM2UtEGDmNvk\nMj2RmyYOyu9/Fvg08I5chmP7+GzDMuTpV0h6KG+H30jatjBtpKRzJN2fp98kaWRh8YdL+oekRyV9\nsq+yN7mcnvmOlnRP/pvNkXR8YdpYST/N2+BxSTf2/BhL+ndJD+TP3Vv8W/ZhrKTr87y/lrRpIcbW\nedrjeTlvL0x78TBe0p6S5kk6SdLDkh6UdHRh3nWVDsMX5L/rmf0dSkt6raT/zes2V9JRfcwzJq//\nI5K68/PxhelH5W32lKT7JB2e398yr+eT+e/0g8JnevblZfYjLdtk1t+2WVfpe7pA0u3AFv1s/5VL\nRFT+AP4O7JOfTwDuBs7Ir28A/gFsS2raGA78GrgAGAFsDzwC7J3nPw1YRDrkGQ58FLgPGJ6nvw0Y\nR/pReQepqWKjPO0oYDFwYv7sO4AngXUKZXlPYd6bCusQwJb5+XTgzH7W8cPArcB4YDXgQuDSPO14\n4GpgdaAL2AkY3cc2Gw7MBj4BrAq8HngK2KqwHS7pZ5s3LEOefgwwKk/7KnBnYdr5eVtsnMv4mjzf\nZnk7XASMBLYDngO2aVCGgZazSp7vX0lfUgF7kH5wd8zTzgK+nrfHcGD3PN9WwFxgXJ5vM2CLBuWY\nnrfd63L8r/X8bYE18nKOJu1/OwKPAtv2/lsDe5L2n9NzWd6YyzomT78sP1YHJuXl3tSgTJvkMh2W\nl7UusH0fMdcF3pqXOQq4AvhxoewLCvvERoVyXwp8kvQ9GAG8tsG+fBqF/YjCft/EtrkMuDzPNxl4\noNH6rmyPdibap4EngPtJSXRknnYDcHph3gmkttxRhffOAqYXdoxbC9OGAQ8CuzeIfSdwcGEnmg+o\nMP124IhCWVqRaO8h/zDk1xuRfhxWISW4/wVeMcA22x14CBhWeO9S4LTCdugv0TYsQx/zrp3Xb628\nPZ8Ftutjvs3yfON7bb9D+5i3meUsU5Y8/cfAh/Lz04Gf9Gz7wjxbAg8D+5B/ZPvZFtOBywqv18z7\n2ATSj+2Nvea/EPhM7781KdE+Wyx3LsOrSD8ki8hJL087k8aJ9hTgqn7Ke2aDadsD3fn5GqTv1FvJ\n36fCfN8BphX/Vg325aX2I5ZOtA23TWF9ty5M+3yj9V3ZHu1sOnhzRKwdEZtGxPsj4tnCtLmF5+OA\nxyPiqcJ795NqRcvMHxFLgHn5c0g6UtKd+XDsCdIv7djCZx+IvFcUlj1uhdZsWZsCVxXKcA/pi70B\n8F3gOuAySfMlfUnS8D6WMQ6Ym9evWNaN+5h3UGWQ1CXpC7lZYQHpRwLSdhpLqgH9rZ9lP1R4/gwp\ncfXWzHIAkHSApFvz4ekTpJpiz9/sbFLN/hf5EPlkgIiYTaq1nwY8LOkySf39HYv7zNPA46RtvCmw\na892yvEPBzZssJzHYun2zJ71X4/0Q1rcl4vPe5tAc9tmdUkX5uaXBcBvgLUldUXEP0nJ8L3Ag5J+\nJmnr/NGPk2r+tys1Ox0zUKw+9Ldt+lrf+5cjxpDU9jbaBoqJbz6wjqRRhfc2IR2W9JjQ8yS3140H\n5ud2t4uAqaSzqGsDfyLtcD02llR8vUmOubzl7ctc4ID8w9LzGBERD0TEooj4bERMIh1Kvwk4so9l\nzAcmaOmTg723w3KVAXgncDCpNrgWqYYJaTs9CixkxdvbmlqOpNWAHwL/AWyQ/2bX5LIQEU9FxEkR\n8TLgQOAjPW2xEfH9iHgtKSEE8MV+QhX3mTWBdUjbeC7w617bac2IeN8g1/cRUrPC+MJ7ExrMS47b\nzDY+idRMsmtEjCY1f8BL2+e6iNiXdMTyZ9L+T0Q8FBHHRcQ4UnPVBZK2HMT69JSx0bbpWd/iOm4y\nyOUPWZ2aaF8UEXNJh9ZnKZ04egVwLEtfcrKTpLcoXYf5YVI74a2kQ6kg7QTkExWTe4VYHzhB0nBJ\nbwO2IX2xB+P/gIbX1JLaFD/Xc8JF0nqSDs7P95L0L5K6SO1ri0g1zd5uI7UvfzyXdU9SormsyTI2\nLAOpre854DFS29/nez6Ua9AXA19WOinZJenVOSE2bRDLWZXUbvoIsFjSAcB+PRMlvSmfuBFpe70A\nvCBpK0mvz8tbSDqk72s79nhjPvm0KnAGcFve134KvFzSEXk7D5e0s6RtBrm+LwA/Ak7LtdCt6fsH\ntMf3gH0kvV3SKvnE0vZ9zDcqr9sTSiczP9MzQdIGkg6StAbp7/k0eRtIepteOmnWTfpe9Ld9+tJw\n2/SxvpOAdw9y+UNWxyfa7DBSLWs+cBWpvez6wvSfkA6ZuoEjgLfkmuIs4BzgFlIy/Bfg5l7Lvg2Y\nSKpxfQ44JCIeG2T5vglMyodTP+5j+teAGaTD3adIPwK75mkbAleSksY9pBN/y3TAiIjngYOAA3JZ\nLwCOjIg/N1nG/srwHdJh3gPArDyt6KPAH4GZpEPsL7J8+86Ay8lNRCeQTqp0k2rbMwqzTAT+m5RE\nbgEuiIgbSMn5C6Rt8xDpB/QT/ZTl+6Qk9TjpBOThhfj7AYeS9reHcjkH9cOSTSUdITxEaiK6lJQA\nlxER/yA1kZyUy3Qn6eRib18lnXh8lPR3+nlh2rD8+fl5GXsA78/TdgZuk/Q0aXt+KCLuG8zKNLFt\nppKaTR4itSt/azDLH8q0dPOkmZVF0heBDSPCNb2VTF1qtGa1k685fYWSXUhNXle1u1xWvbZ0wTVb\nSYwiNReMI132dQ6pmctWMm46MDMrmZsOzMxK5kRbgnyp0e+V+puf0ILlNT0UpFZsCMBl+vEv53KW\n+7N1pIqHebT6caItx8eBGyJiVESc2+7CdBpJZ0j6o6TFkk5rd3nMyuZEW45NSQPlDNpKUiuaTfox\n+lm7C2JWBSfaFpP0S2Av4DyloeZeLmktSd9RGtrufkmn6qWh/Y6SdLOkr0h6nNRXf6AYDYc0zJZr\nCMABYo6T9MO8DvcVm0SUhj+crjRs3yzSxfENRcS3I+Ja0mhVA8Xtb4jJdyiNdzA6vz4gb5f18uuv\nKQ03uEDSbyXtXljuaXk7XpK30x/z3+oUpSEP50oq9ki7QdJZkm7P2/0nKgwz2avMa0n6ptKwiQ8o\nDY/Ylac1HK7Qhi4n2haLiNcDNwJTcz/wvwD/Seoh9DJSb50jSUPN9dgVmEPqzfS5JsJcS+ohtT7w\nO5YdAf9wUrfSsaQeRt8DyF0zryf1ilqf1OPugj4S9VLyj8LVwF2kQWz2Bj4s6Q15ls+Q+ulvAbyB\n1na9PAF4M2m7jSP1FjsfICJ+QOoddq6kdUk99N4TEY/kz84kjW61Dmmdr5A0orDsA0k9tsYAvycN\n7jMsr+PppJGpio4kjbY2jtSvv1Gz0Lfz9C2BHUi9qXrGNT4D+EWOOZ60b9hQ1+7hw4big6WHV+wi\ndbucVJh+PKkNF9IwdP8YYHmn0WAIRApDGubX02ndEIDz8vNde5eRNKzft/LzOcD+hWlTej47wHpd\nQh7msZ95+h3eMa//P0hdey8cYFnd5GEa8za9vjDtQFK33q78elTermsX/qZfKMw/CXg+/303y/Ou\nQhqR7TkKwxSSftB+lZ83HK7Qj6H7cI22fGNJA6UUh4xrOMzjQNT/kIbLLC9WbAjAHpsC43p97hOk\npEJedlnD4/U3xCQR8QRp8OvJpA4BL1K688E9+TD9CdJRRXE7/V/h+bPAo5EGR+l5DUsP+dh7HYf3\nWl5PeYeThinsKfOFpCMIaM1whVYzK8OJl3Z7lFQD25Q0YAssO7zhYHqNFIc0/DspeXSz9NCPAw0B\nuO+g1iB97r6ImNhg+oO8dKcMaO3weHOBYyKi92BAACiNcHUMqQfWucD++f3dgX8nNXPcHRFLJPXe\nToPVewjARaS/b/H9uaQa7djo475bEfEQcFwu42uB/5b0m0jj6doQ5RptyXIN6XLSEIWj8ompj9DH\nCF1NajikYUGrhwC8HVigdE+ukblWPVlSz0mvy4FTlO5nNR74YH8Ly3FHkPa/VZSGv+xqMHt/Q0yO\nIG3HT5DavDeW1DNa1ShSO+kjOcangdEDrOdA3iVpkqTVSW24VxZqwABExIOkNthzJI3OJ/O2kLRH\nLnMrhiu0mnGircYHSWPJzgFuIp2YuXg5lzXQkIbQ4iEAczI5kHRi6T5SLe4bpNo0wGdzme4jJZnv\nDrAOF5EOzQ8j3cfqWdLwln3pb3jHs0htwf8VEc8B7wLOlDSRdGLrWuAvuWwLGUQTTQPfJbVjP0S6\nW0SjzihHkpqLZpGS6ZWktmVowXCFVj8e68CsCZJuIJ2Q/Ea7y2L14xqtmVnJnGjNzAokXZw7rfyp\nwXRJOlfSbEl/kLTjQMt0ojVrQkTs6WaDlcZ08tUrDRxA6jA0kXTN+H8NtEAnWjOzgoj4DelEciMH\nA9+J5FbS7d436md+J1ozs0HamKWvYJnH0h2QllFKh4WFiwd1Af5yG/OqE6sIA8Bjt3y5sljDtCLX\n1JvVy4hVVqgTCQAjd5jadM5ZeOf5x5MO+XtMi4hpgwjXV3n7je+eYWZWf2r+4Dwn1cEk1t7msXRv\nwPGk69IbctOBmdWf1Pxjxc0AjsxXH7wKeDL3CGyo3jXaFxZVFsqH82YdbBA12gEXJV1KGr1urNIt\nmT5DGiiIiPg6cA3wRtIA9s+w9JCnfap3ojUzg1bVVAGIiMMGmB7ABwazTCdaM6u/YY3GJOoMTrRm\nVn8tbDooQ60T7RXf/VRlsRY8W1178OiRwyuLZfVx0IV9DdRWjhnHv6qyWC3R4edQap1ozcwA12jN\nzErnGq2ZWcl8Mqw8bzvijMpidc88r7JYZn2pXbtpldx0YGZWMidaM7OSDXMbrZlZuVyjLdHIFb17\ntFl9PLdoSWWxVhve2YlrGb7qwMysZL7qwMysZG46KE/3bz7PiT+ZVUms82+eU0kcgA/s9rLKYlXp\nqYWLK4kzakStd+uGhq/S2YfHbeWmg/JUlWTNrMO5RmtmVjLXaM3MSuYabXmmn3lBZbHcBXfFDdW2\n06r4dkr98FUHZmYlc43WzKxkHV7bd6I1s/pzjdbMrGSu0ZqZlcwnw8zMyiXXaM3MyuVEa2ZWts7O\ns060ZlZ/rtGamZXMidbMrGTDhvk62tI8dtt/VhZrzKs/Ulms7lu+XFkssyGhsyu09U60ZmbgpgMz\ns9I50ZZo2DAxZueplcTyMIlmncuJtkRVJVkz62xOtGZmJdOwzk60nX1NhJlZEyQ1/WhiWftLulfS\nbEkn9zF9E0m/kvR7SX+Q9MaBlulEa2a116pEK6kLOB84AJgEHCZpUq/ZTgUuj4gdgEOBAe+p5URr\nZvWnQTz6twswOyLmRMTzwGXAwb3mCWB0fr4WMH+ghbqN1sxqbzAnwyRNAaYU3poWEdPy842BuYVp\n84Bdey3iNOAXkj4IrAHsM1BMJ1ozq73BdMHNSXVag8l9Zezo9fowYHpEnCPp1cB3JU2OiCWNYtY7\n0a63abtLYFaZJ55ZVFmstVcfXlmsVmjh5V3zgAmF1+NZtmngWGB/gIi4RdIIYCzwcKOFuo3WzOqv\ndW20M4GJkjaXtCrpZNeMXvP8A9gbQNI2wAjgkf4WWu8arZkZravRRsRiSVOB64Au4OKIuFvS6cAd\nETEDOAm4SNKJpGaFoyKid/PCUpxozaz2WtkzLCKuAa7p9d6nC89nAbsNZpn1TrTdD7a7BGaVGT2i\n3l/XMrkLrplZyTq9C64TrZnVnmu0ZmYlc6It0wuL212C2uv/XGlrdfh3oeMN6/DD43ZyojUzK1tn\n51knWjOrP98F18ysZB3ecuBEu7Lr9B3UrBluozUzK1mH51knWjOrP9doS9R9+7mMec1H212MWvPl\nXStmSYUbcNhQ3IAt0umbptaJ1knWzAC6ujo709Y60ZqZgZsOzMxK1+F5tuaJdtzL212C2uv0HXR5\nVNnuXGW76cMLnqss1vqjV6ssViu4RmtmVjInWjOzknV4nnWiNbP66/SRzeqdaO//Q7tLYB2o02s3\ny6tu7aZVctOBmVnJOjzPOtGaWf25RmtmVrIOz7P1TrR//eU5lcVasqTCPu0d3rDf6cYcMq2yWN1X\nTqksljXW6d+ZWidaMzNw04GZWek6PM860ZpZ/blGW6KJrz+psljdM8+rLJatGLebrnw6PM/WO9Ga\nmYFrtGZmpev0qw46+2boA/DhvJlBqtE2+2iHWtdox+w8td1FMLMO0OEtB/VOtGZm4DZaM7PSdXie\nrXeiHbvbvpXFWvTCkspiDe+qddN52907/6nKYm01blRlsayxVt5SSNL+wNeALuAbEfGFPuZ5O3Aa\nEMBdEfHO/pZZ60RrZgatu+pAUhdwPrAvMA+YKWlGRMwqzDMROAXYLSK6Ja0/YPlaUjozszYapuYf\nA9gFmB0RcyLieeAy4OBe8xwHnB8R3QAR8fCA5Rv8KpmZdZYWXt61MTC38Hpefq/o5cDLJd0s6dbc\n1NCvppoOJH0I+BbwFPANYAfg5Ij4RTOfL8tfvtL7h6Y86+x6QmWxum8/t7JYQ5HbTVc+g2milTQF\nKPbTnhYRPWNr9rWk3mOkrgJMBPYExgM3SpocEU80itlsG+0xEfE1SW8A1gOOJiXetiZaMzMA9Zkf\n+5aTaqNBi+cBEwqvxwPz+5jn1ohYBNwn6V5S4p3ZKGazTQc9a/FG4FsRcRd9Z34zs8p1DVPTjwHM\nBCZK2lzSqsChwIxe8/wY2AtA0lhSU8Kc/hbabKL9raRfkBLtdZJGAdVd72Rm1g+p+Ud/ImIxMBW4\nDrgHuDwi7pZ0uqSD8mzXAY9JmgX8CvhYRDzWb/kiBr5Fi6RhwPbAnIh4QtK6wMYR0ef9vhcuXqZN\noxRjdvtYFWEA6L757MpimbVbldeNj1ptxa/Ness3f9t0zvnRsTtVfjTebI02gElAzxmhNYARpZTI\nzGyQWlWjLUuzifYC4NXAYfn1U6SLes3M2m6ojN61a0TsKOn3ALk3xKollsvMrGlDZayDRblrWgBI\nWo8OOBk2bJNJ7S6CreSWNHGOo1UeeuK5ymKNG1OvlsGuDs+0zSbac4GrgPUlfQ44BDi1tFKZmQ3C\nkBgmMSK+J+m3wN6k62ffHBH3lFoyM7MmdfidbPpPtJLWKbx8GLi0OC0iHi+rYM147NKjK7zLwtEV\nxbE6aeXwfAOp2+F8lepeo/0tqV22Uf/fl7W8RIPgW9mYGdT8ZFhEbF5VQczMllfda7QvkvQW4LWk\nmuyNEfHj0kplZjYITYxh0FbNDpN4AbAlL7XRvlfSvhHxgdJK1oTumedxzwPV3bZkKFrw7KLKYo0e\nObyyWEPRTp+pbrC83352v8pitUJnp9nma7R7AJMjD4wg6dvAH0srVZOcZM0Mqj0puTya7YJ7L7BJ\n4fUEoM8BZczMqtbpYx00W6NdF7hH0u359c7ALZJmAETEQQ0/aWZWsqFyMuzTpZZiOb3msLMqi9X9\nm89XFqtKbjddMc8tqq4net3aTas0JE6GRcSvASSNLn6m3R0WzMyg5tfR9sg3MzsDeJY0mIzogA4L\nZmYwdJoOPgZsGxGPllkYM7Pl0exZ/XZpNtH+DXimzIIsl2cXtLsEtpJbbXinf8VXDkOlRnsK8L+S\nbgNeHBQzIk5o/BEzs2p0+LmwphPthcAvSZ0U2j7gt5lZ0ZC46gBYHBEfKbUkZmbLqcPzbNOJ9lf5\nyoOrWbrpwJd3mVnbdXgTbdOJ9p35/1MK7/nyLjPrCJ0+1kGzHRY8Lq2ZdaxOv/ZjMOPRTgYmAS/e\nTyMivlNGoczMBmNInAyT9BlgT1KivQY4ALgJcKI1s7br8JaDpmvch5DugPtQRBwNbAesVlqpzMwG\nYZiaf7RDs00Hz0bEEkmL88AyD+MTYWbWIYbEyTDgDklrAxeR7oz7NHB7/x8pX/fM89j5s9e3uxgt\nl+5jUY0O3z+tTW6eXd2wJntvPXaFl9Hp+3GzVx28Pz/9uqSfA6Mjou13WBiKSdbMBq/Dz4U110Yr\n6die5xHxd+DufILMzKztuqSmH+3Q7MmwvSVdI2mjfJnXrcCoEstlZta0IXEyLCLeKekdpEFlngEO\ni4ibSy1ZEyZPXK/dRShFp7c3dbqzb5hdWayP7bllZbGqtNuWK95uWqVOHyax2aaDicCHgB8CfweO\nkLR6ieUyM2tap9dom206uBr4VEQcD+wB/BWYWVqpzMwGoZW3G5e0v6R7Jc2WdHI/8x0iKSS9cqBl\nNnt51y4RsQAgIgI4p+dW42Zm7daq62gldQHnA/sC84CZkmZExKxe840CTgBua2a5/SZaSR+PiC9F\nxAJJb4uIKwqTjwY+MZiVaLWP7O6xbmxZQ7XddOKHf1JZrL9+9eDKYrVCV+tGldkFmB0RcwAkXQYc\nDMzqNd8ZwJeAjzaz0IGKd2jh+Sm9pu3fTAAzs7INQ00/JE2RdEfhMaWwqI2BuYXX8/J7L5K0AzAh\nIn7abPkGajpQg+d9vTYza4vBtBxExDRgWqNF9fWRl+JoGPAV4KjmIw6caKPB875em5m1RQuvJpgH\nTCi8Hg/ML7weBUwGbsiXlG0IzJB0UETc0WihAyXa7SQtIGX5kfk5+fWIxh+rxsjhXe0uglllFv5z\nYbuL0LFaOKjMTGCipM2BB0jNpz13mCEingRevMhY0g3AR/tLsjBAoo0IZzIz63itGvg7IhZLmgpc\nB3QBF0fE3ZJOB+6IiOW62qrpOyyYmXWqVnYMi4hrSDc4KL736Qbz7tnMMp1ozaz2hsw9wzrRzgc2\n7LTRct0zz6sslllf5l70jnYXoWN1+lgHtU60ZmbQ+deaOtGaWe0NlVvZdKTumefxx7lPtrsYthLb\n7axfVRbrZye8trJYa68xvLJYrdDpd1iodaJ1kjUzcButmVnpfNWBmVnJXKMt0esO6fMa4lJ03/a1\nymJZfdx8yl7tLkIpllR5z/sWXDPQ2Wm25onWzAxcozUzK127biPeLCdaM6u9zk6zNU+0bjetlzE7\nT60kjrtLr7hO7wDQW6cXt9aJ1swM0q1sOpkTrZnVnmu0ZmYl6/Smjlon2qra/MDtfq3gbWhlcdOB\nmVnJOrxC60RrZvXnRGtmVjK56cDMrFwej9bMrGS+6sDMrGRuOjAzK5mbDszMSuYarZlZyTq8idaJ\n1szqr8PzbL0TbffM8xiz+8ntLoZZJZ5fvKSyWKuu0um3O1yaB/4ukZOsmQEdX6WtdaI1MwOfDDMz\nK12HtxzUPNEufLrdJTCrTN3aTavU4Xm25onWzAzfbtzMrHQdnmfxsYiZ1Z4G8RhwWdL+ku6VNFvS\nMpc2SfqIpFmS/iDpfyRtOtAynWjNrP5alGkldQHnAwcAk4DDJE3qNdvvgVdGxCuAK4EvDVQ8J1oz\nqz0N4t8AdgFmR8SciHgeuAw4uDhDRPwqIp7JL28Fxg+0UCdaM6s9qfnHADYG5hZez8vvNXIscO1A\nC/XJMDOrvcGcDJM0BZhSeGtaREzrmdzHR6LBct4FvBLYY6CYTrRmVnuD6RmWk+q0BpPnARMKr8cD\n85eJJ+0DfBLYIyKeGyimmw7MrPZa2HQwE5goaXNJqwKHAjOWjqUdgAuBgyLi4WbK50RrZrXXqsu7\nImIxMBW4DrgHuDwi7pZ0uqSD8mxnA2sCV0i6U9KMBot7kZsOzKz+WthhISKuAa7p9d6nC8/3Gewy\n651ou4ZXFmrJkj7bw0sxrNNvgGTWYXwXXDOzknV2mnWiNbOhoMMzba0TbfetX2HMzlMrieXDebPO\n5YG/S1RVkjWzztbhTbT1TrRmZtDxLQdOtGZWfx74u0wjR7e7BGaVeW5RdbcbX214vfoydXierXmi\nNTPDTQdmZuXr8EzrRGtmtefLu8q04RbtLoFZZapsN33wiYWVxdp87IgVXobbaM3MSuZEa2ZWMjcd\nmJmVzDXaMs29u90lqL0XKhz+scvjRdTGRmuveLtplTp9z6p3ojUzwzVaM7PSuQuumVnJOjvN1j3R\nLn6+3SWoPbeb2lDQ4RXamidaMzN8eZeZWfk6O8/WO9F2zzyPj119T7uLYTbknFTh9+r8f9tmhZfR\n4Xm23onWSdbMwLcbNzMrX2fnWSdaM6u/Ds+zKKL1XTAXLqayfp1V3Qm3e+Z5lcQxW9mMWGXF8+Rj\n/1zcdM5Zd41VKs/Lta7R+nbjZga+vMvMrHQdfi7MidbM6s+Jtkyrr9XuEphVZuGiFyqLNWJ4V2Wx\nWsFNB2ZmJXON1sysZB2eZ51ozWwI6PBMW+vraBe9sKSKMACs/7p/ryxW981nVxbLrN1acR3tP59v\nPpGtsWr1DQ3V3SjezKwkGsRjwGVJ+0u6V9JsSSf3MX01ST/I02+TtNlAy3SiNbP6a1GmldQFnA8c\nAEwCDpMe7pOVAAAJU0lEQVQ0qddsxwLdEbEl8BXgiwMVz4nWzGpPg/g3gF2A2RExJyKeBy4DDu41\nz8HAt/PzK4G9NdBNyyKiYx7AlKEUx7HqFWsortNQjrUiZQTuKDymFKYdAnyj8PoI4Lxen/8TML7w\n+m/A2P5idlqNdsoQi+NY9Yo1FNdpKMdaLhExLSJeWXhMK0zuq2ba+0RbM/MspdMSrZlZO80DJhRe\njwfmN5pH0irAWsDj/S3UidbM7CUzgYmSNpe0KnAoMKPXPDOAd+fnhwC/jNyG0EindViYNvAstYrj\nWPWKNRTXaSjHarmIWCxpKnAd0AVcHBF3SzoduCMiZgDfBL4raTapJnvoQMstpcOCmZm9xE0HZmYl\nc6I1MyuZE60N2oAXZ9eMpDUqjLXhUNt+NrC2JlpJW0l6taThuetb2fEqGc1Y0paSXilptQpibStp\nD0nrlhzntZKOAIiIKDNZSDpQ0ofKWn6vWAcDX5S0fgWx3gBcxdKXD5UR51WSjsj/r1pyrIl5X++q\n6vtVR21LtJLeAvwEOJN0Fu8DkkaXFOvlABHxQtk7g6Q3AT8Czgam98QuKdYBwKXAicB3JG1YQoxh\nktYELgROkfReeDHZtnz/kbQfcAYwq9XL7iPWHqR+6j+JiIdLjrVfjrURcFKJcQ4infnfB/gosGmJ\nsd5M6oJ6CvBl4Pgqjw7qpC2JVtJw4B3AsRGxNynhTgA+3upkmxPfnZK+D+UmW0mvAf4DeHdE7AV0\nA8uM/tOiWHsCXwPeExFvBp4HJrc6TkQsiYinSX27vwm8RtKJPdNaGStvv++SukReL2ktSZtKWr2V\ncQp2InW3vF7SOEn7StpVUkvvkSRpH+AC4HBgIrCNpNe1MkaOsy7wAeCdEfFuYAGwvaT1JY0oIdbx\nwGER8VbgLuBo4ERJo1oZayhoZ9PBaNJOB+lw6qfAqsA7W3VYmn9dpwIfBp6XdAmUXrP9QkT8Pj//\nDLBOSU0I/wccHxG355rsrsBUSRdKOqSEQ/vFpB/DbwO7SPqypLOUtGo/egxYBGyUv8g/Bv6LdGRQ\n1jr1uBI4hrS/nC9pTAvjdAFHRsTdwBrAvcC20PL27sXASGDrXGHZEzgS+Cpwaotrm4uBNYENASLi\nYuB+YD3gTS2MMzS0cWCHfUk9LHbPr7uAdwKXkK/vbVGccaQdYizpy3RJievUBYwuPB8P/B5YL7+3\nbklxPwmcmp8fDfygJ2YLY2wBnJyfnwQ8A5xfwrpsB8whdXM8jlQZOIbURLJOi2NNJiW9y4Cj83sv\nA74OvKGEdRuW/98feAj4lxJiHAL8FrgV+FR+7/XAdGC7Fsd6L+kI5Ajgc/m7ezzpIv+WrlfdH+2s\n0d4I/AI4QtLrIuKFiPg+KTFu16ogETE/Ip6OiEdJO8HInpqtpB0lbd3CWC9ExIL8UsATwOMR8Yik\nw4EzJY1sVbxC3M9FxJn5+beAUbT+hMuzwFaSjiN9wb4AbCLp+FYGiYi7SDWisyLiokhNFxcDY4BN\nWhzrT6R2zF2BzfN7c0g/kuu1MlZe9pL8/89J7ahvavERARFxJal99kbSjzwR8UvSPtHq9tpLgZ+T\nEvnqEfGuiLgQWL+s8y111bYuuBGxUNL3SKPenJIT3nPABsCDJcV8LCeGsyX9mfSF2qukWIuBpyXN\nlXQWsB9wVEQ828o4khS5epFfv5W0DXsPhLFCImK+pLnAp4APRMTVkvYCZrcyTo41i8LJsLxO61HO\nfnEtqYnnNEn35/d2IP2QlOku0knML0VES+8jHhHdkn4JvF3S88AI0g/JH1oc50nge5Iu7fkRkXQk\nsA5Q3b3R66DdVWpSu+xepMO36cAOFcQ8kZIO3QoxlNftb8A/gIklr9NqpJHf7wYmlxRjArBT4fWw\nktdJpGaDWcC2JcfaEfg8cE6Z+0WvmJcDm5W07LWBE4Bfk/rtt7TZoEHMnr9VJduvTo+OGesgn5yK\naPGZ7D7ijCHt4CdFREt/4RvEOwqYGelESJlxhpPavf8WEfeWHGupWnSZcYA9gIci4s9lx6tKVdsv\nxxpFOuexYMCZVzzWpsDwiGj5UU7ddUyirZKkERGxsKJYlX2pzKwzrZSJ1sysSh7rwMysZE60ZmYl\nc6I1MyuZE+0QJukFSXdK+pOkK1ZkzABJe0r6aX5+kKSGYzhIWlvS+5cjxmmSPtrs+73mmS7pkEHE\n2kzSnwZbRrPl4UQ7tD0bEdtHxGTSoDPvLU5c3l5JETEjIvq7oH9tYNCJ1myocqJdedwIbJlrcvdI\nugD4HTBB0n6SbpH0u1zzXRNA0v6S/izpJuAtPQuSdJSk8/LzDSRdJemu/HgNqVfVFrk2fXae72OS\nZkr6g6TPFpb1SUn3SvpvYKuBVkLScXk5d0n6Ya9a+j6SbpT0F6VR21AaJ/XsQuyWdhk2a4YT7UpA\n6d7zBwB/zG9tBXwnInYA/gmcCuwTETsCdwAfURpW7yLgQGB38ihNfTgX+HVEbEfqXXU3aWjIv+Xa\n9MeUxmKdCOwCbA/sJOl1knYi3UF0B1Ii37mJ1flRROyc491D6g3XYzNSB4d/Bb6e1+FY4MmI2Dkv\n/zhJmzcRx6xlOu1249ZaIyXdmZ/fSBpPdhxwf0Tcmt9/FTAJuDmP2LcqcAuwNXBfRPwVIA/EM6WP\nGK8nDcVHpD77T/YxxOB++dEzfOSapMQ7CrgqIp7JMWY0sU6TJZ1Jap5Yk9S9tMfluWfhXyXNyeuw\nH/CKQvvtWjn2X5qIZdYSTrRD27MRsX3xjZxM/1l8C7g+Ig7rNd/2pAF/WkGk0bgu7BXjw8sRYzrw\n5oi4K3dv3rMwrfeyIsf+YEQUEzKSNhtkXLPl5qYDuxXYTdKWAJJWV7r9zp+BzSVtkec7rMHn/wd4\nX/5sVx4e7ylSbbXHdcAxhbbfjZXu0fUb4N8kjcx98g9soryjgAfz2A6H95r2NqVb72xBGlf23hz7\nfXl+JL1cvt2KVcw12pVcpLFyjwIu1Ut3gjg1Iv4iaQrwM0mPAjfR961yPgRMk3QsaWi890XELZJu\nzpdPXZvbabcBbsk16qeBd0XE7yT9ALiTNDr/jU0U+VPAbXn+P7J0Qr+XNFrVBsB7Iw3F+Q1S2+3v\n8iA1jwBvbm7rmLWGxzowMyuZmw7MzErmRGtmVjInWjOzkjnRmpmVzInWzKxkTrRmZiVzojUzK5kT\nrZlZyf4fznxuPwortKMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9030fd0a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Convert the 2D array of probabilities into a dictionary of column vectors    \n",
    "cols_to_vectors_dict = {c:predictions_probabilities_with_1s_as_truth[:,c] for c in range(predictions_probabilities_with_1s_as_truth.shape[1])}\n",
    "df_cols_to_vectors = pd.DataFrame(cols_to_vectors_dict)\n",
    "    \n",
    "def plot_classification_probabilities(df, title, cmap=plt.cm.Blues):\n",
    "    \"\"\"Plot 10 columns of class probabilities for all the examples in our dataframe\"\"\"\n",
    "    sns.heatmap(df, annot=False, vmin=0, vmax=1, cmap=cmap, yticklabels=False)\n",
    "    plt.ylabel('Examples')\n",
    "    #plt.yticks(rotation=90)\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.xticks(rotation=45)\n",
    "    plt.title(title)\n",
    "plot_classification_probabilities(df_cols_to_vectors, title=\"Probabilities of each class being classified\\nfor labelled {} examples\".format(target_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index-0 least confident Class 1 prediction is on row 39 of predictions_with_1s_as_truth\n"
     ]
    }
   ],
   "source": [
    "# use argsort to return an array of indices where the 0th item (index_to_investigate) represents the \n",
    "# index of the lowest item in predictions_probabilities_with_1s_as_truth (and the 1st item \n",
    "# is the next-lowest etc) and select the index of the least-confident prediction of class 1\n",
    "index_to_investigate = 0  # you can ask for 0 to 9th least-confident (where 9==most confident) item\n",
    "idx_least_confident_prediction_for_class_1 = predictions_probabilities_with_1s_as_truth[:,target_class].argsort()[index_to_investigate]\n",
    "print(\"Index-{} least confident Class {} prediction is on row {} of predictions_with_1s_as_truth\".format(index_to_investigate, target_class, idx_least_confident_prediction_for_class_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probabilities:\n",
      " [  6.27088451e-07   3.96269005e-04   8.70094092e-06   4.94674663e-10\n",
      "   6.77184449e-05   9.98399714e-01   1.11846526e-04   2.25250852e-07\n",
      "   1.01389939e-03   9.99006775e-07]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEICAYAAACwDehOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG81JREFUeJzt3XmYHWWZ9/HvjySYQAIIiQJJSECCEHZsNhlZBHwJOIRX\nQeEVlUWioyyKIxe4IMMyIgqCikIUDAISkVEnahB4FaKAgYRlwASZiWFJE5aOBMIqQe7543m6KE7O\n6XM6dPXJ8vtcV19dVc9TVXctp+6qp+rUUURgZmYGsEa7AzAzsxWHk4KZmRWcFMzMrOCkYGZmBScF\nMzMrOCmYmVnBSQGQdIakq5Zz3KMk3dpD+fWSPl6vrqTnJW3Ww7hzJO29PHH1MM13SrpH0nOSTuzL\nafcVSWMlhaSB7Y4Fer/d+nC+t0j6xHKMF5I2ryKmBvMbL2l2f81vRSRpiqSz2x0HgKQTJZ27vOOv\ntElB0sOSXsof0Ccl/UjS0HbHVSsiJkTEFQ3KhkbEfKi/U0XE1hFxSx+HdApwS0QMi4hv9/G0Vwvl\n7dbIipbYKnYW8M12B/FmLW8SbgdJkyU9KOk1SUfVFE8GjpT0tuWZ9kqbFLJ/joihwE7AzsCXayso\nWdmXsy+NAeYsz4irwgFuVViGFYmkjYB9gF+2O5bVzH8Bnwburi2IiJeB64GPLc+EV4mDZUQ8RloJ\n20CR8c+RdBvwIrCZpI0lTZP0tKR5ko6rmcxgST/NzSp3S9q+u0DSqZL+msvmSvq/NeNK0nckPSvp\nL5L2LRU0PPvovsyXNAn4CHBKvvL5VS5/WNJ+uXuNUhx/k3StpPVz2WBJV+Xhz0iaJentdeb3e9IH\n+Lt5PltIWlfSjyV1SXpE0pe7k2huNrlN0rckPQ2cUWeaDePK5T+T9EReN3+QtHWpbIik8/N8n5V0\nq6Qhpcl/RNKjkhZJ+lK9dZinM0XSJZJuyttohqQxNev5M5L+B/ifPGzLXP/pfMb1oVL9DfK+skTS\nncA76m23Jsvwh1z9mbyud8/1j5H0gKTFkm6oiXP/vP88K+m7gHpY5gGSvljaL++SNLpOvYOUmguX\nSFog6YxSWcP9Jm/7+XnaD0n6SINQ9gfuzgciJB3dvf/m/nmSri31L5C0Q7NtULMM+0i6v9T///N2\n6e6/VdIhubvhZzUv062SvpnX/0OSJuSyc4D38Ppn47sNYvknSbfn9bVAy56lI+mtkn6t9JlanLtH\n1cSxzLpVOhbMyNt/kaSfNljnRMTFEfE74OUGVW4BDmo0fo8iYqX8Ax4G9svdo0lnv2fl/luAR4Gt\ngYHAIGAG8D1gMLAD0AXsm+ufASwFDs11/xV4CBiUyw8DNiYl0Q8DLwAb5bKjgFeBz+VxPww8C6xf\niuUTpbq3lpYhgM1z9xTg7B6W8bPATGAU8BbgUuCaXPZJ4FfAWsAA4F3AOg3WWxFP7v8x8J/AMGAs\n8N/AsTXLdkJej0PqTK9hXLn8mDzttwAXAveWyi7O8YzMcb871xub180PgCHA9sDfga0aLNMU4Dlg\nzzz+RXXW803A+nl6awMLgKPzcu0ELAK2zvWnAtfmetsAj/Ww3Zotw8DSeIcA84Ct8ny/DNyey4YD\nS3h9H/xcXvefaLDMXwDuB95JSh7bAxvUiW9vYFvSvrsd8CRwSE/7TV7uJcA7c72NutdNnTi+AVxc\n6t8MeCbPbyPgEeCxUtniXNbjNqiZx2DgpbyOBgJPAAtJ+9WQXNa97M0+q0uB4/Ly/kuejup9NurE\nsQlpPzsib6MNgB1qP795+Afzeh0G/Az4ZS5ruG6Ba4Av5dgHA//UwnHwVuCoOsN3Ap5ermNrlQfu\nKv9IB8zn8w74COmAP6S0cc8s1R0N/AMYVhr2NWBK7j4DmFkqWwN4HHhPg3nfC0ws7WjFjpWH3Ql8\ntHZH480lhQfISay0My3NH5JjgNuB7VpYb+V4BpAOtuNL5Z8k3XPojvfRJtNrGFeduuvlZV43r+OX\ngO3r1Bub642qWaeHN4hhCjC11D80b+/RpfX83lL5h4E/1kzjUuCreZ0sBbYslf17ve3W4jKUk8L1\n5IRb2s9eJDXpfaxmHxTQSeOk8GD3PlinrNiv6pRdCHwrd9fdb0gHrmdIB7ZlTgRq6v4AOLdm2ALS\nQelwUvv2ncCWpAQwrdk2aDCfPwIfAHYDbiQl7QNIV7739RBf7Wd1XqlsrbyuNqz9bDSY1mnAL3rY\nB89uULYDsLjZuiWdoE2mtN83+6NxUhgH/KPV6ZT/Vvbmo0MiYr2IGBMRn46Il0plC0rdG5Oy5nOl\nYY+Qzu6WqR8Rr5E+kBsDSPqYpHvzJeMzpLPH4aVxH4u8JUrT3vhNLdmyxgC/KMXwAOnA93bgSuAG\nYKqkhZLOkzSohWkOB9bM8XZruF56G1du4jg3X84vISW57vkOJ50N/bWHaT9R6n6RdLBvpLz9ngee\n5o3boLwcY4Bdu2POcX8E2BAYQUq05frl9VPWyjKUjQEuKs3zadLBf2SOtbwMQc/rfnQr85W0q6Sb\nc1PGs8CneH3frbvfRMQLpIP2p4DHJf1G0pYNZrGYdDZcNoN0hbJn7r4F2Cv/zch1etoG9bQyzVY+\nq8U+FREv5s5WH1BpdZ2vJelSpSbFJaSmxPUkDWiybk8h7Q93Kj15eEyLcdUzjNRi0Wsre1LoSfkg\nvRBYX1J5592E1CzQrWiPVWpTHwUsVGrz/QFwPOkSdT3gz7yxvXekpHL/JnmeyxtvPQuACTkJdv8N\njojHImJpRPxbRIwnNV+8n9ZuMi0inRWPqYm9vF6WOy7g/wETgf1IVwdj8zjK836Zmvb6N6G8/YaS\nmorK26C8HAuAGTUxD42IfyE1K75anh5pndTT0zLUW28LgE/WzHdIRNxOujItL4NqYqg3rVbW3U+A\naaSrpnWBS8j7bk/7TUTcEBH7k678/kL6DNRzH7BFzbDuA/h7cvcMlj2A97QN6qlNCstMs8XPak9a\n2ddbWeefJzXr7RoR6+SY4fX1XnfdRsQTEXFcRGxMumL/npb/0eKtSDeje21VTgqFiFhAukz+Wr65\nth1wLHB1qdq7JH1A6emUz5KaVWaSLveCdLBA0tHkG9olbwNOlDRI0mGkDTK9l2E+SWpzbeQS4Jy8\n4yNphKSJuXsfSdtKGkBqr1xKOlvvUUT8g3QZfo6kYXnaJwO9+c5Gw7hIZyt/B/5GulT/99K8XwMu\nBy5QeghggKTdJb2lF/MuOzDfBFyT9IjkHXm71/NrYAtJH83bbJCknSVtldfJz4Ez8hnfeODj9SbS\nZBm6gNd44za9BDhN+Wa70k3+w3LZb4CtS/vgiTQ+awb4IXCWpHFKtpO0QZ16w0hXyS9L2oWUqMnz\nr7vfSHq7pIMlrU3afs/TeH+6CdhJ0uDSsBmkZp0hEdFJavo5gNTWfk+u03AbNJjP7aQD7S7AnREx\nh3y1wes39Vv5rPak2WfwamA/SR+SNFDpgYQd6tQbRmpWfEbpoYuvdhf0tG4lHabXb0gvzstSd71L\nWjOvcwGD8nGtfDzfi9Rc2WurRVLIjiCdqS4EfkFqu7ypVP6fpMu6xcBHgQ/kM6m5wPnAn0g7zbbA\nbTXTvoPUhrcIOAc4NCL+1sv4LgPG58veeo/3XUQ647tR0nOkhLVrLtsQuI70wX6A9KFs9cB+Aulm\n3HxS++RPSAe6VvUU14/JNxqBubms7F9JN0tnkZpSvs7y75M/IX34nibdMG30tAy5GfF9pDbvhaQm\nha+TbhBDOtMcmodPAX7Uw3zrLkNumjgHuC1v090i4he5fGpuVvgzMCHHtIh0k/RcUhIdx7L7WdkF\npIR+I2m7X0a66Vrr08CZeducnsfp1mi/WYN0trswL9NeeTrLiIgngd+Trgi7h/036WD3x9y/hLR/\n3ZaTbivboHY+L5Aev5wTEa/kwX8CHomIp3KdVj6rPbkIOFTpiaFlvsMTEY8CB5LWzdOk+xXb19Yj\n3bcZQjoezAR+Wyrrad3uDNwh6XnSZ+qkiHioQaw3khLPu0n3IV4iX5HkZHEgUPf7Uc1033U3W2lJ\nmgJ0RsQy31Ox6uWrqSuAXcIHlLaTdAKpufCU5RnfX+Qxszcln6Hv3O44LImI77yZ8Ven5iMzM2vC\nzUdmZlbwlYKZmRVWunsKw4cPj7Fjx7Y7DDOzlcpdd921KCJGNKu30iWFsWPHMnv2av3qdjOzXpPU\n6Jv5b+DmIzMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs0JlSUHS5ZKekvTnBuWS9G2ln+u7T9JOVcVi\nZmatqfJKYQrpdbmNTCC9CXIcMAn4foWxmJlZCypLChHxB9KrYRuZCPw4kpmkXybaqKp4zMysuXbe\nUxjJG39usJM3/gxkQdIkSbMlze7q6uqX4MzMVkft/EZzvZ/Iq/t2voiYTPohCTo6OvwGP1spjD31\nN5VO/+FzD6p0+rZ6aueVQidv/A3aUfT+d43NzKwPtTMpTAM+lp9C2g14NiIeb2M8ZmarvcqajyRd\nA+wNDJfUSfr93EEAEXEJ6YftDwTmAS8CR1cVi5mZtaaypBARRzQpD+AzVc3fzMx6z99oNjOzgpOC\nmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4KTgpmZFZwUzMys4KRgZmYF\nJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHM\nzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs0Kl\nSUHSAZIelDRP0ql1yjeRdLOkeyTdJ+nAKuMxM7OeVZYUJA0ALgYmAOOBIySNr6n2ZeDaiNgROBz4\nXlXxmJlZc1VeKewCzIuI+RHxCjAVmFhTJ4B1cve6wMIK4zEzsyaqTAojgQWl/s48rOwM4EhJncB0\n4IR6E5I0SdJsSbO7urqqiNXMzKg2KajOsKjpPwKYEhGjgAOBKyUtE1NETI6IjojoGDFiRAWhmpkZ\nVJsUOoHRpf5RLNs8dCxwLUBE/AkYDAyvMCYzM+tBlUlhFjBO0qaS1iTdSJ5WU+dRYF8ASVuRkoLb\nh8zM2qSypBARrwLHAzcAD5CeMpoj6UxJB+dqnweOk/RfwDXAURFR28RkZmb9ZGCVE4+I6aQbyOVh\np5e65wJ7VBmDmZm1zt9oNjOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZw\nUjAzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzM\nrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVmgpKUg6T9I6kgZJ+p2kRZKOrDo4MzPrX61eKbwv\nIpYA7wc6gS2AL1QWlZmZtUWrSWFQ/n8gcE1EPF1RPGZm1kYDW6z3K0l/AV4CPi1pBPBydWGZmVk7\ntHSlEBGnArsDHRGxFHgRmFhlYGZm1v9avdG8FvAZ4Pt50MZAR1VBmZlZe7R6T+FHwCvAu3N/J3B2\nJRGZmVnbtJoU3hER5wFLASLiJUCVRWVmZm3RalJ4RdIQIAAkvQP4e7ORJB0g6UFJ8ySd2qDOhyTN\nlTRH0k9ajtzMzPpcq08ffRX4LTBa0tXAHsBRPY0gaQBwMbA/qblplqRpETG3VGcccBqwR0QslvS2\n3i+CmZn1lZaSQkTcJOluYDdSs9FJEbGoyWi7APMiYj6ApKmkJ5bmluocB1wcEYvzfJ7qZfxmZtaH\nemw+krRl/r8TMAZ4HFgIbJKH9WQksKDU35mHlW0BbCHpNkkzJR3QII5JkmZLmt3V1dVktmZmtrya\nXSmcDEwCzq9TFsB7exi33o3oqDP/ccDewCjgj5K2iYhn3jBSxGRgMkBHR0ftNMzMrI/0mBQiYlLu\nnBARb/gGs6TBTabdCYwu9Y8iXWXU1pmZvxD3kKQHSUliVrPAzcys77X69NHtLQ4rmwWMk7SppDWB\nw4FpNXV+CewDIGk4qTlpfosxmZlZH+vxSkHShqT7AEMk7cjrTULrAGv1NG5EvCrpeOAGYABweUTM\nkXQmMDsipuWy90maC/wD+EJE/O1NLZGZmS23ZvcU/g/p0dNRwAWl4c8BX2w28YiYDkyvGXZ6qTtI\n9y1Obi1cMzOrUrN7ClcAV0j6YET8Rz/FZGZmbdKs+ejIiLgKGCtpmbP5iLigzmhmZraSatZ8tHb+\nP7TqQMzMrP2aNR9dmv//W/+EY2Zm7dSs+ejbPZVHxIl9G46ZmbVTs+aju/olCjMzWyG08vSRmZmt\nJpo1H10YEZ+V9CuWfW8REXFwZZGZmVm/a9Z8dGX+/82qAzEzs/Zr1nx0V/4/I7+/aEvSFcODEfFK\nP8RnZmb9qKUf2ZF0EHAJ8FfS+482lfTJiLi+yuDMzKx/tfpznOcD+0TEPCh+o/k3gJOCmdkqpNVX\nZz/VnRCy+YB/OtPMbBXT7OmjD+TOOZKmA9eS7ikchn8Ix8xsldOs+eifS91PAnvl7i7grZVEZGZm\nbdPs6aOj+ysQMzNrv1afPhoMHAtsDRS/zRwRx1QUl5mZtUGrN5qvBDYk/RLbDNIvsT1XVVBmZtYe\nrSaFzSPiK8AL+X1IBwHbVheWmZm1Q6tJYWn+/4ykbYB1gbGVRGRmZm3T6pfXJkt6K/AVYBrpl9i+\nUllUZmbWFi0lhYj4Ye6cAWxWXThmZtZOLTUfSdpA0nck3S3pLkkXStqg6uDMzKx/tXpPYSrptRYf\nBA4FFgE/rSooMzNrj1bvKawfEWeV+s+WdEgVAZmZWfu0eqVws6TDJa2R/z5EekuqmZmtQpq9EO85\n0gvwBJwMXJWL1gCeB75aaXRmZtavmr37aFh/BWJmZu3X6j0FJB0M7Jl7b4mIX1cTkpmZtUurj6Se\nC5wEzM1/J+VhZma2Cmn1SuFAYIeIeA1A0hXAPcCpVQVmZmb9r9WnjwDWK3Wv29eBmJlZ+7V6pfA1\n4B5JN5OeRNoTOK2yqMzMrC2aXilIEnArsBvw8/y3e0RMbWHcAyQ9KGmepIZNTZIOlRSSOnoRu5mZ\n9bGmVwoREZJ+GRHvIr0htSWSBgAXA/sDncAsSdMiYm5NvWHAicAdvYrczMz6XKv3FGZK2rmX094F\nmBcR8yPiFdL7kybWqXcWcB7wci+nb2ZmfazVpLAPKTH8VdJ9ku6XdF+TcUYCC0r9nXlYQdKOwOhm\n33mQNEnSbEmzu7q6WgzZzMx6q9UbzROWY9qqMyyKQmkN4FvAUc0mFBGTgckAHR0d0aS6mZktp2bv\nPhoMfArYHLgfuCwiXm1x2p3A6FL/KGBhqX8YsA1wS7qXzYbANEkHR8TsFudhZmZ9qFnz0RVABykh\nTADO78W0ZwHjJG0qaU3gcEo3qiPi2YgYHhFjI2IsMBNwQjAza6NmzUfjI2JbAEmXAXe2OuGIeFXS\n8cANwADg8oiYI+lMYHZEtPwkk5mZ9Y9mSWFpd0c+yPdq4hExHZheM+z0BnX37tXEzcyszzVLCttL\nWpK7BQzJ/SJ9hWGdSqMzM7N+1ez3FAb0VyBmZtZ+vXkhnpmZreKcFMzMrOCkYGZmBScFMzMrOCmY\nmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZw\nUjAzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzM\nrOCkYGZmBScFMzMrOCmYmVnBScHMzAqVJgVJB0h6UNI8SafWKT9Z0lxJ90n6naQxVcZjZmY9qywp\nSBoAXAxMAMYDR0gaX1PtHqAjIrYDrgPOqyoeMzNrrsorhV2AeRExPyJeAaYCE8sVIuLmiHgx984E\nRlUYj5mZNVFlUhgJLCj1d+ZhjRwLXF+vQNIkSbMlze7q6urDEM3MrKzKpKA6w6JuRelIoAP4Rr3y\niJgcER0R0TFixIg+DNHMzMoGVjjtTmB0qX8UsLC2kqT9gC8Be0XE3yuMx8zMmqjySmEWME7SppLW\nBA4HppUrSNoRuBQ4OCKeqjAWMzNrQWVJISJeBY4HbgAeAK6NiDmSzpR0cK72DWAo8DNJ90qa1mBy\nZmbWD6psPiIipgPTa4adXurer8r5m5lZ7/gbzWZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMz\nKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4K\nZmZWcFIwM7OCk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkV\nnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlaoNClIOkDSg5LmSTq1TvlbJP00l98haWyV\n8ZiZWc8qSwqSBgAXAxOA8cARksbXVDsWWBwRmwPfAr5eVTxmZtZclVcKuwDzImJ+RLwCTAUm1tSZ\nCFyRu68D9pWkCmMyM7MeDKxw2iOBBaX+TmDXRnUi4lVJzwIbAIvKlSRNAibl3uclPVhJxPUNr41n\nNeHlXsGpb6+rV5rl7mOr03KPaaVSlUmh3hl/LEcdImIyMLkvguotSbMjoqMd824nL/fqxctt3aps\nPuoERpf6RwELG9WRNBBYF3i6wpjMzKwHVSaFWcA4SZtKWhM4HJhWU2ca8PHcfSjw+4hY5krBzMz6\nR2XNR/kewfHADcAA4PKImCPpTGB2REwDLgOulDSPdIVweFXxvAltabZaAXi5Vy9ebgNAPjE3M7Nu\n/kazmZkVnBTMzKzgpNBAs1d0rIokjZZ0s6QHJM2RdFK7Y+pPkgZIukfSr9sdS3+StJ6k6yT9JW/7\n3dsdU3+Q9Lm8n/9Z0jWSBrc7phWBk0IdLb6iY1X0KvD5iNgK2A34zGqy3N1OAh5odxBtcBHw24jY\nEtie1WAdSBoJnAh0RMQ2pIdhVsQHXfqdk0J9rbyiY5UTEY9HxN25+znSwWFke6PqH5JGAQcBP2x3\nLP1J0jrAnqQnAYmIVyLimfZG1W8GAkPyd6TWYtnvUa2WnBTqq/eKjtXi4Ngtv7F2R+CO9kbSby4E\nTgFea3cg/WwzoAv4UW46+6GktdsdVNUi4jHgm8CjwOPAsxFxY3ujWjE4KdTX0us3VlWShgL/AXw2\nIpa0O56qSXo/8FRE3NXuWNpgILAT8P2I2BF4AVjl76FJeivp6n9TYGNgbUlHtjeqFYOTQn2tvKJj\nlSRpECkhXB0RP293PP1kD+BgSQ+TmgrfK+mq9obUbzqBzojoviK8jpQkVnX7AQ9FRFdELAV+Dry7\nzTGtEJwU6mvlFR2rnPza8suAByLignbH018i4rSIGBURY0nb+vcRsVqcNUbEE8ACSe/Mg/YF5rYx\npP7yKLCbpLXyfr8vq8EN9lZU+ZbUlVajV3S0Oaz+sAfwUeB+SffmYV+MiOltjMmqdwJwdT4Bmg8c\n3eZ4KhcRd0i6Drib9NTdPfiVF4Bfc2FmZiVuPjIzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZ\nWcFJwczMCv8LxOtqIpfgmE0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f902ecacc50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for this one example we can get all 10 class probabilities\n",
    "probabilities_per_class = predictions_probabilities_with_1s_as_truth[idx_least_confident_prediction_for_class_1]\n",
    "print(\"Probabilities:\\n\", probabilities_per_class)\n",
    "_ = plt.bar(np.arange(probabilities_per_class.shape[0]), probabilities_per_class)\n",
    "plt.ylabel(\"Probabilities\")\n",
    "plt.title(\"Probabilities for each predicted class (we want class {})\".format(target_class));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We'd *incorrectly* predicted class 5 for this example with probability 0.9983997138505155\n"
     ]
    }
   ],
   "source": [
    "# the *most* confident class has the highest probability\n",
    "idx_most_confident_class_for_this_example = probabilities_per_class.argsort()[-1]\n",
    "print(\"We'd *incorrectly* predicted class {} for this example with probability {}\".format(idx_most_confident_class_for_this_example, probabilities_per_class[idx_most_confident_class_for_this_example]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show image for this mis-predicted example\n",
    "\n",
    "We can show the image and class (1) for this example, the cell above shows the mis-predicted label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y (target label) =  1\n",
      "Image for X[420] = \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAECCAYAAADesWqHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC0dJREFUeJzt3d+rZfV5x/H3p+OYiT/CQLRBHYkV2gEJ7YwMtmFAWieJ\n2gR70wuFBBpaphdtUFoISW9K/oGQXpRAUFMhakiMA0VaJ9JEJNCazOiYqGNCFIPTiRnTEPwROqPm\n6cXZUyYy7VnHnO86P573Czaz9znr7Oc5c/js79p7r72eVBWSevmNtW5A0vwMvtSQwZcaMvhSQwZf\nasjgSw2ti+AnuSHJ95P8MMmnBte6M8mJJE+OrHNGvcuTfDPJ0SRPJbl1cL1tSb6d5IlFvc+MrLeo\nuSXJ40keGF1rUe/5JN9LciTJocG1tie5L8kzi7/h+wfW2rn4nU5fXk5y25BiVbWmF2AL8CxwJXAu\n8ARw1cB61wJXA0/O9PtdAly9uH4h8IPBv1+ACxbXtwKPAn8w+Hf8G+Ae4IGZ/k+fBy6aqdZdwF8s\nrp8LbJ+p7hbgReC9I+5/Paz41wA/rKrnquoU8GXgT0YVq6pHgJ+Nuv+z1PtxVT22uP4KcBS4bGC9\nqqpXFze3Li7DjtJKsgP4MHD7qBprJcm7WFoo7gCoqlNV9fOZyu8Dnq2qH4248/UQ/MuAF864fYyB\nwVhLSa4AdrO0Co+ssyXJEeAE8FBVjaz3OeCTwC8H1nirAr6e5HCS/QPrXAm8BHxx8VTm9iTnD6x3\nppuBe0fd+XoIfs7ytU13HHGSC4CvAbdV1csja1XVm1W1C9gBXJPkfSPqJPkIcKKqDo+4///H3qq6\nGrgR+Ksk1w6qcw5LTws/X1W7gdeAoa9BASQ5F7gJ+OqoGush+MeAy8+4vQM4vka9DJFkK0uhv7uq\n7p+r7mK39GHghkEl9gI3JXmepado1yX50qBa/6uqji/+PQEcYOnp4gjHgGNn7DHdx9IDwWg3Ao9V\n1U9GFVgPwf8O8NtJfmvxSHcz8M9r3NOqSRKWniMerarPzlDv4iTbF9ffCXwAeGZErar6dFXtqKor\nWPq7faOqPjqi1mlJzk9y4enrwIeAIe/QVNWLwAtJdi6+tA94ekStt7iFgbv5sLQrs6aq6o0kfw0c\nZOmVzDur6qlR9ZLcC/whcFGSY8DfV9Udo+qxtCp+DPje4nk3wN9V1b8MqncJcFeSLSw9sH+lqmZ5\nm20m7wEOLD2ecg5wT1U9OLDeJ4C7F4vSc8DHB9YiyXnAB4G/HFpn8daBpEbWw66+pJkZfKkhgy81\nZPClhgy+1NC6Cv7gwy/XrJb1rLfe6q2r4ANz/ufO+oe0nvXWU731FnxJMxhyAM+5eUdtY+UfYnqd\nk2zlHavez1rX+nXr/c7v/mLFP/PSf73Jxe/e8rbqvR2/Tr0ffPe8Ff/MRvr7zVnvv3mNU3XybB98\n+xVDDtndxvn8fvaNuOuWDh48svxGG9j1l+5a6xY2jUfr3yZt566+1JDBlxoy+FJDBl9qyOBLDRl8\nqSGDLzVk8KWGJgV/zhFXksZbNviLkzb+I0un/L0KuCXJVaMbkzTOlBV/1hFXksabEvw2I66kLqZ8\nSGfSiKvFiQP2A2xj5Z+2kjSfKSv+pBFXVfWFqtpTVXvm/PiipJWbEvxNPeJK6mjZXf25R1xJGm/S\niTgWc95GzXqTNDOP3JMaMvhSQwZfasjgSw0ZfKkhgy81ZPClhgy+1NCQSTqb3cHjm3uyjTY/V3yp\nIYMvNWTwpYYMvtSQwZcaMvhSQwZfasjgSw0ZfKkhgy81NGWE1p1JTiR5co6GJI03ZcX/J+CGwX1I\nmtGywa+qR4CfzdCLpJn4HF9qaNU+luvsPGnjWLUV39l50sbhrr7U0JS38+4F/h3YmeRYkj8f35ak\nkaYMzbxljkYkzcddfakhgy81ZPClhgy+1JDBlxoy+FJDBl9qyOBLDW2K2XnOsltd11+6a61b0GCu\n+FJDBl9qyOBLDRl8qSGDLzVk8KWGDL7UkMGXGjL4UkMGX2poysk2L0/yzSRHkzyV5NY5GpM0zpRj\n9d8A/raqHktyIXA4yUNV9fTg3iQNMmV23o+r6rHF9VeAo8BloxuTNM6KnuMnuQLYDTw6ohlJ85j8\nsdwkFwBfA26rqpfP8n1n50kbxKQVP8lWlkJ/d1Xdf7ZtnJ0nbRxTXtUPcAdwtKo+O74lSaNNWfH3\nAh8DrktyZHH548F9SRpoyuy8bwGZoRdJM/HIPakhgy81ZPClhgy+1JDBlxoy+FJDBl9qyOBLDRl8\nqSGDLzVk8KWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYamnGV3W5JvJ3liMTvvM3M0Jmmc\nKQM1TgLXVdWri/PrfyvJv1bVfwzuTdIgU86yW8Cri5tbF5ca2ZSksaZO0tmS5AhwAnioqpydJ21g\nk4JfVW9W1S5gB3BNkve9dZsk+5McSnLodU6udp+SVtGKXtWvqp8DDwM3nOV7zs6TNogpr+pfnGT7\n4vo7gQ8Az4xuTNI4U17VvwS4K8kWlh4ovlJVD4xtS9JIU17V/y6we4ZeJM3EI/ekhgy+1JDBlxoy\n+FJDBl9qyOBLDRl8qSGDLzU05ci9de/6S3fNWu/g8SOz1pNWmyu+1JDBlxoy+FJDBl9qyOBLDRl8\nqSGDLzVk8KWGDL7UkMGXGpoc/MVQjceTeKJNaYNbyYp/K3B0VCOS5jN1hNYO4MPA7WPbkTSHqSv+\n54BPAr8c2IukmUyZpPMR4ERVHV5mO2fnSRvElBV/L3BTkueBLwPXJfnSWzdydp60cSwb/Kr6dFXt\nqKorgJuBb1TVR4d3JmkY38eXGlrRqbeq6mGWxmRL2sBc8aWGDL7UkMGXGjL4UkMGX2rI4EsNGXyp\nIYMvNbQpZufNzVl92uhc8aWGDL7UkMGXGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtTQpEN2F6fW\nfgV4E3ijqvaMbErSWCs5Vv+PquqnwzqRNBt39aWGpga/gK8nOZxk/8iGJI03dVd/b1UdT/KbwENJ\nnqmqR87cYPGAsB9gG+etcpuSVtOkFb+qji/+PQEcAK45yzbOzpM2iCnTcs9PcuHp68CHgCdHNyZp\nnCm7+u8BDiQ5vf09VfXg0K4kDbVs8KvqOeD3ZuhF0kx8O09qyOBLDRl8qSGDLzVk8KWGDL7UkMGX\nGjL4UkMGX2rI4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQwZfasjgSw0ZfKmhScFPsj3JfUmeSXI0\nyftHNyZpnKkDNf4BeLCq/jTJueDEDGkjWzb4Sd4FXAv8GUBVnQJOjW1L0khTdvWvBF4Cvpjk8SS3\nLwZr/Iok+5McSnLodU6ueqOSVs+U4J8DXA18vqp2A68Bn3rrRo7QkjaOKcE/BhyrqkcXt+9j6YFA\n0ga1bPCr6kXghSQ7F1/aBzw9tCtJQ019Vf8TwN2LV/SfAz4+riVJo00KflUdAfYM7kXSTDxyT2rI\n4EsNGXypIYMvNWTwpYYMvtSQwZcaMvhSQ1OP3FMjB48fmbXe9ZfumrWeXPGllgy+1JDBlxoy+FJD\nBl9qyOBLDRl8qSGDLzVk8KWGlg1+kp1JjpxxeTnJbXM0J2mMZQ/ZrarvA7sAkmwB/hM4MLgvSQOt\ndFd/H/BsVf1oRDOS5rHS4N8M3DuiEUnzmRz8xTn1bwK++n9839l50gaxkhX/RuCxqvrJ2b7p7Dxp\n41hJ8G/B3XxpU5gU/CTnAR8E7h/bjqQ5TB2h9Qvg3YN7kTQTj9yTGjL4UkMGX2rI4EsNGXypIYMv\nNWTwpYYMvtSQwZcaSlWt/p0mLwFv5zP7FwE/XeV21kMt61lvrnrvraqLl9toSPDfriSHqmrPZqtl\nPeutt3ru6ksNGXypofUW/C9s0lrWs966qreunuNLmsd6W/ElzcDgSw0ZfKkhgy81ZPClhv4Hqy6F\n+xDrI9gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f902e93e208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idx_for_truth_for_this_example = np.arange(true_labels_1s_indices.shape[0])[true_labels_1s_indices][idx_least_confident_prediction_for_class_1]\n",
    "show_example(idx_for_truth_for_this_example, X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y (target label) =  1\n",
      "Image for X[420] = \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAECCAYAAADesWqHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADANJREFUeJzt3fuLnOUZxvHr6iZrDmpDahQxYrSVQJDWSIiVgNZEJR6I\nhRZqQKHSklKsKBVE2x+K/4DaH4pFoqmgRjSaUsR6wCOCjU1i1JyURBTjIRtJNSZqYuLdH+aNXcO2\n+246z7Ozub8fGDKz+2bue7O55nln5p33dkQIQC7fGu0GANRH8IGECD6QEMEHEiL4QEIEH0ioJ4Jv\ne6HtN2xvsX1T4Vp32x6wvb5knUH1Trb9rO1NtjfYvq5wvQm2X7b9alPvlpL1mpp9tl+x/WjpWk29\nt22/bnud7dWFa02xvcL25uZ3eE7BWjObn+ngZZft64sUi4hRvUjqk7RV0mmS+iW9KmlWwXrnSjpL\n0vpKP9+Jks5qrh8j6c3CP58lHd1cHy9plaQfFv4ZfyvpfkmPVvo3fVvScZVq3SPpl831fklTKtXt\nk/ShpFNK3H8vrPhzJW2JiLciYp+kByRdXqpYRLwgaWep+x+i3gcRsba5/qmkTZJOKlgvImJ3c3N8\ncyl2lJbt6ZIulbS0VI3RYvtYdRaKuyQpIvZFxMeVyi+QtDUi3ilx570Q/JMkvTvo9jYVDMZosj1D\n0mx1VuGSdfpsr5M0IOmpiChZ73ZJN0r6qmCNQ4WkJ22vsb2kYJ3TJO2QtKx5KrPU9uSC9Qa7QtLy\nUnfeC8H3EF874o4jtn20pIclXR8Ru0rWiogDEXGmpOmS5to+o0Qd25dJGoiINSXu/3+YFxFnSbpY\n0jW2zy1UZ5w6TwvviIjZkvZIKvoalCTZ7pe0SNJDpWr0QvC3STp50O3pkt4fpV6KsD1endDfFxGP\n1Krb7JY+J2lhoRLzJC2y/bY6T9Hm2763UK2vRcT7zZ8Dklaq83SxhG2Stg3aY1qhzgNBaRdLWhsR\n20sV6IXg/1PS6bZPbR7prpD0t1HuqWtsW53niJsi4tYK9abZntJcnyjpAkmbS9SKiJsjYnpEzFDn\n9/ZMRFxZotZBtifbPubgdUkXSSryDk1EfCjpXdszmy8tkLSxRK1DLFbB3XypsyszqiJiv+3fSHpC\nnVcy746IDaXq2V4u6UeSjrO9TdIfIuKuUvXUWRWvkvR687xbkn4XEY8VqneipHts96nzwP5gRFR5\nm62SEySt7Dyeapyk+yPi8YL1rpV0X7MovSXp6oK1ZHuSpAsl/aponeatAwCJ9MKuPoDKCD6QEMEH\nEiL4QEIEH0iop4Jf+PDLUatFPer1Wr2eCr6kmv+4VX+R1KNeL9XrteADqKDIATz9PiomaOQfYvpS\nezVeR3W9n9Gu9f/WO/6ML0b8dz7ZuV/fnnp4B2a+9/mUEf+d/bs+07hjJx1WvfFbR/7zjaXfX816\nX2iP9sXeoT749g1FDtmdoMk62wtK3HVKv/7rlqr1fv9asdMhDGn6T4odoZ3Oqni61Xbs6gMJEXwg\nIYIPJETwgYQIPpAQwQcSIvhAQgQfSKhV8GuOuAJQ3rDBb07a+Cd1Tvk7S9Ji27NKNwagnDYrftUR\nVwDKaxP8NCOugCzafEin1Yir5sQBSyRpgg7vU1oA6miz4rcacRURd0bEnIiYU/PjiwBGrk3wj+gR\nV0BGw+7q1x5xBaC8VifiaOa8lZr1BqAyjtwDEiL4QEIEH0iI4AMJEXwgIYIPJETwgYQIPpBQkUk6\ntX113uyq9a5Z+lDVej+evLtqvY2nr65a73lNrFoPrPhASgQfSIjgAwkRfCAhgg8kRPCBhAg+kBDB\nBxIi+EBCBB9IqM0IrbttD9heX6MhAOW1WfH/Imlh4T4AVDRs8CPiBUk7K/QCoBKe4wMJde1juczO\nA8aOrq34zM4Dxg529YGE2rydt1zSS5Jm2t5m+xfl2wJQUpuhmYtrNAKgHnb1gYQIPpAQwQcSIvhA\nQgQfSIjgAwkRfCAhgg8kVGR2niccpb7vzSxx10N6YvmyarUyWHnb/Kr1puqlqvXAig+kRPCBhAg+\nkBDBBxIi+EBCBB9IiOADCRF8ICGCDyRE8IGE2pxs82Tbz9reZHuD7etqNAagnDbH6u+XdENErLV9\njKQ1tp+KiI2FewNQSJvZeR9ExNrm+qeSNkk6qXRjAMoZ0XN82zMkzZa0qkQzAOpoHXzbR0t6WNL1\nEbFriO8vsb3a9up9Bz7rZo8AuqxV8G2PVyf090XEI0NtM3h2Xn8fQzOBXtbmVX1LukvSpoi4tXxL\nAEprs+LPk3SVpPm21zWXSwr3BaCgNrPzXpTkCr0AqIQj94CECD6QEMEHEiL4QEIEH0iI4AMJEXwg\nIYIPJFRkdp72H5A++leRux7K+Rsur1ZLkq6b8XTVerP6t1etN+2xrVXrHahaDRIrPpASwQcSIvhA\nQgQfSIjgAwkRfCAhgg8kRPCBhAg+kBDBBxJqc5bdCbZftv1qMzvvlhqNASinzbH6eyXNj4jdzfn1\nX7T994j4R+HeABTS5iy7IWl3c3N8c4mSTQEoq+0knT7b6yQNSHoqIpidB4xhrYIfEQci4kxJ0yXN\ntX3Godt8Y3beV593u08AXTSiV/Uj4mNJz0laOMT3/jM771sTu9QegBLavKo/zfaU5vpESRdI2ly6\nMQDltHlV/0RJ99juU+eB4sGIeLRsWwBKavOq/muSZlfoBUAlHLkHJETwgYQIPpAQwQcSIvhAQgQf\nSIjgAwkRfCChIrPzYv9+Hdg+UOKuh9R/YbVSkqQbbruyar2tP/tz1Xo48rHiAwkRfCAhgg8kRPCB\nhAg+kBDBBxIi+EBCBB9IiOADCRF8IKHWwW+GarximxNtAmPcSFb86yRtKtUIgHrajtCaLulSSUvL\ntgOghrYr/u2SbpT0VcFeAFTSZpLOZZIGImLNMNt9PTvvS+3tWoMAuq/Nij9P0iLbb0t6QNJ82/ce\nutHg2XnjdVSX2wTQTcMGPyJujojpETFD0hWSnomIumeiANBVvI8PJDSiU29FxHPqjMkGMIax4gMJ\nEXwgIYIPJETwgYQIPpAQwQcSIvhAQgQfSKjI7LwjXf8nPF5ibON/MJAQwQcSIvhAQgQfSIjgAwkR\nfCAhgg8kRPCBhAg+kBDBBxJqdchuc2rtTyUdkLQ/IuaUbApAWSM5Vv/8iPioWCcAqmFXH0iobfBD\n0pO219heUrIhAOW13dWfFxHv2z5e0lO2N0fEC4M3aB4QlkjSBE3qcpsAuqnVih8R7zd/DkhaKWnu\nENswOw8YI9pMy51s+5iD1yVdJGl96cYAlNNmV/8ESSttH9z+/oh4vGhXAIoaNvgR8ZakH1ToBUAl\nvJ0HJETwgYQIPpAQwQcSIvhAQgQfSIjgAwkRfCAhZucdhlPv2FK13ptX76lab8cl361ab+qygar1\nwIoPpETwgYQIPpAQwQcSIvhAQgQfSIjgAwkRfCAhgg8kRPCBhFoF3/YU2ytsb7a9yfY5pRsDUE7b\nY/X/KOnxiPip7X6JiRnAWDZs8G0fK+lcST+XpIjYJ2lf2bYAlNRmV/80STskLbP9iu2lzWCNb7C9\nxPZq26u/1N6uNwqge9oEf5yksyTdERGzJe2RdNOhGzFCCxg72gR/m6RtEbGqub1CnQcCAGPUsMGP\niA8lvWt7ZvOlBZI2Fu0KQFFtX9W/VtJ9zSv6b0m6ulxLAEprFfyIWCdpTuFeAFTCkXtAQgQfSIjg\nAwkRfCAhgg8kRPCBhAg+kBDBBxJyRHT9To/11DjbC7p+v1md99rno91CUc9/f+Jot3DEWBVPa1fs\n9HDbseIDCRF8ICGCDyRE8IGECD6QEMEHEiL4QEIEH0iI4AMJDRt82zNtrxt02WX7+hrNAShj2HPu\nRcQbks6UJNt9kt6TtLJwXwAKGumu/gJJWyPinRLNAKhjpMG/QtLyEo0AqKd18Jtz6i+S9NB/+T6z\n84AxYiQr/sWS1kbE9qG+yew8YOwYSfAXi9184IjQKvi2J0m6UNIjZdsBUEPbEVqfSfpO4V4AVMKR\ne0BCBB9IiOADCRF8ICGCDyRE8IGECD6QEMEHEiL4QEJFZufZ3iHpcD6zf5ykj7rcTi/Uoh71atU7\nJSKmDbdRkeAfLturI2LOkVaLetTrtXrs6gMJEXwgoV4L/p1HaC3qUa+n6vXUc3wAdfTaig+gAoIP\nJETwgYQIPpAQwQcS+jdkHLSJdQTtuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9030f8abe0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_example(idx_for_truth_for_this_example, X_test_greyscale, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "* Now we can see which image is mis-classified, you should look at lots more examples\n",
    "* Understand why the misclassification might occur\n",
    "* Might other features help?\n",
    "* Might our greyscale cut-off process be sub-optimal?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Confirm Notebook ran to completion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed running this Notebook at: 2017-10-04 11:56:22.805217\n"
     ]
    }
   ],
   "source": [
    "print(\"Completed running this Notebook at:\", datetime.datetime.now())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}