02_fig_in_synthetic_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Feature-Inversion-in-Synthetic-Data\" data-toc-modified-id=\"Feature-Inversion-in-Synthetic-Data-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Feature Inversion in Synthetic Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Train-Models-on-Synthetic-Data\" data-toc-modified-id=\"Train-Models-on-Synthetic-Data-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Train Models on Synthetic Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Dataset\" data-toc-modified-id=\"Dataset-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span>Dataset</a></span></li><li><span><a href=\"#Train-Standard-Model\" data-toc-modified-id=\"Train-Standard-Model-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span>Train Standard Model</a></span></li><li><span><a href=\"#Train-Adversarially-Robust-Model\" data-toc-modified-id=\"Train-Adversarially-Robust-Model-1.1.3\"><span class=\"toc-item-num\">1.1.3&nbsp;&nbsp;</span>Train Adversarially Robust Model</a></span></li><li><span><a href=\"#Evaluating-Standard-and-Robust-Models\" data-toc-modified-id=\"Evaluating-Standard-and-Robust-Models-1.1.4\"><span class=\"toc-item-num\">1.1.4&nbsp;&nbsp;</span>Evaluating Standard and Robust Models</a></span></li></ul></li><li><span><a href=\"#Evaluating-Input-Gradients\" data-toc-modified-id=\"Evaluating-Input-Gradients-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Evaluating Input Gradients</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Inversion in Synthetic Data\n",
    "In this notebook, we show that standard and robust fully-connected networks (FCNs) trained on a simple synthetic data distribution exhibit FIG (feature inversion in gradeints) and RF-FIG (robustness fixes FIG). The synthetic dataset comprises data points where the label $y=\\pm 1$ with probability $0.5$ and $x=y \\cdot e_j$ where $j$ is chosen uniformly at random from $\\{1,...,d\\}$. \n",
    "\n",
    "See Section 5 of the [paper](https://arxiv.org/abs/2102.12781) for more information about (i) the dataset and (ii) a theoretical justification that justifies our counter-intuitive empirical findings on FIG and RF-FIG. For more information about the evaluation framework, see Section 3 of our [paper](https://arxiv.org/abs/2102.12781). Finally, to know more about obtaining attribution-masked data and retraining models on masked datasets, check out the README files and scripts in this repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T11:47:30.350536Z",
     "start_time": "2021-02-24T11:47:30.315077Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import json \n",
    "import random\n",
    "import time\n",
    "from collections import defaultdict\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import torch.nn.functional as F\n",
    "from torch import nn, optim\n",
    "from robustness.model_utils import make_and_restore_model\n",
    "from robustness.datasets import CIFAR\n",
    "\n",
    "sys.path.append('scripts/')\n",
    "import utils\n",
    "import data_utils as du\n",
    "import gpu_utils as gu\n",
    "import ptb_utils as au\n",
    "import misc_utils as mu\n",
    "import synth_models as sm\n",
    "from synthetic_data import SemiRandomSignalCoordinate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Models on Synthetic Data\n",
    "In this section, we present the code for the synthetic data (see ```get_dataset()```) and train standard and adversarially robust models on the dataset. Then, we evaluate the adversarial robustness of the standard and adversarially trained models on test data. In particular, we train a one-hidden-layer ReLU FCN with width $20000$ on a $10$-dimensional synthetic dataset with learning rate $0.1$ for $1500$ epochs, until the models obtain small standard and/or adversarial test loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:26:30.892722Z",
     "start_time": "2021-02-24T13:26:30.862206Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_dataset(): \n",
    "    args = {\n",
    "        'input_dim': 10,\n",
    "        'num_train': 10_000,\n",
    "        'num_test': 1_000,\n",
    "        'batch_size': 250\n",
    "    }\n",
    "    \n",
    "    sd_tr = SemiRandomSignalCoordinate(args['input_dim'], 1, 1./args['input_dim'])\n",
    "    sd_te = SemiRandomSignalCoordinate(args['input_dim'], 1, 1./args['input_dim'])\n",
    "\n",
    "    tr_dl = sd_tr.get_dataloader(args['num_train'], args['batch_size'])\n",
    "    te_dl = sd_te.get_dataloader(args['num_test'], args['batch_size'])\n",
    "\n",
    "    return {\n",
    "        'obj': (sd_tr, sd_te),\n",
    "        'loaders': (tr_dl, te_dl)\n",
    "    }\n",
    "\n",
    "def get_model(device):\n",
    "    args = {\n",
    "        'input_dim': 10,\n",
    "        'num_classes': 2,\n",
    "        'width': 25_000, \n",
    "        'depth': 1, \n",
    "        'activation': nn.ReLU,\n",
    "        'lr': 0.1,\n",
    "        'weight_decay': 0.0,\n",
    "        'momentum': 0.0,\n",
    "        'decay_gap': 50,\n",
    "        'decay_factor': 0.75\n",
    "    }\n",
    "    \n",
    "    model = sm.get_fcn(args['input_dim'], args['width'], \n",
    "                       args['num_classes'], activation=args['activation'])\n",
    "    model = model.to(device)\n",
    "    \n",
    "    opt = optim.SGD(model.parameters(), lr=args['lr'], \n",
    "                    weight_decay=args['weight_decay'], \n",
    "                    momentum=args['momentum'])\n",
    "    \n",
    "    def lr_scheduler(epoch):\n",
    "        pow = epoch//args['decay_gap']\n",
    "        return max(0.01, args['decay_factor']**pow)\n",
    "\n",
    "    sched = optim.lr_scheduler.LambdaLR(opt, lr_scheduler)\n",
    "\n",
    "    return model, opt, sched\n",
    "\n",
    "def linf_pgd_adversarial_training(train_data, test_data, epsilon, data_bounds, device):\n",
    "    # setup attack\n",
    "    attack = au.Linf_PGD_Attack(epsilon, lr=0.25*epsilon, num_iter=8, loss_type='untargeted', \n",
    "                                rand_eps=0.0, num_classes=2, bounds=data_bounds, device=device)\n",
    "\n",
    "    # setup model\n",
    "    model, opt, sched = get_model(device)\n",
    "    \n",
    "    # train for 500 epochs\n",
    "    stats = utils.pgd_adv_fit_model(model, opt, train_data, test_data, attack, \n",
    "                                    sch=sched, min_loss=0.0, max_epochs=500, \n",
    "                                    device=device, epoch_gap=10)\n",
    "    \n",
    "    return stats, model.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:26:33.719151Z",
     "start_time": "2021-02-24T13:26:32.251647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEdCAYAAADpZhdxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dX4xcWX4X8O8hmQiEkqnxjJSIoOy4mtXvBchstYcHIgGZqd4NIihottsOIi+IdbfnKUKaddsP8ABSvO2shPI00+0gXoLAbntIRJDY7fYGkOCBdfdM4OmnxeXdKIsSad0u7wqBdu0+PJzf7bp9+96qe+v+qarb38+oNFX31q1zq1z9u6fOv5/z3oOIiNrpz8z6BIiIqD4M8kRELcYgT0TUYgzyREQtxiBPRNRiDPJENRCR7qzPYVbO83ufRz8+6xMgWiQWwDYAdABsq+phynN6ALoABg2f3rzoiUhnzGezAeAxgB1VHTZ+dueMW/Rx8iLSAXAHQPSHdQjgEYAtVW30j0xEtgCsq+prNZZxHcCKqq6UfJ0egIOUXUMA9wBs8g/wNPvMHgJYU9X9Mc/bVtWN5s5s/kz6DESkD2AXwHLTf6fnzULX5K1W9RihxrSNEKCWAFy3p7TxD20JQL/C19sEcN/ud+y1bwK4LCLvptXGzrGbAB5NCPBbALaaO6W5tSUiW6q6mbZTVfdFZIDw/Tv1d2oVt8sI3/W7AC7AfhmN++wp3UIHeYTAPlDVpfhGEbmF8KWohdWm79dZA8kqw2pHVV68hokyDkVkB6HG+hDAVL9KmviMZlB2B6EiMU6PNVNAVQf2y2ecI4QAnnRJVXdE5BmAu3ZB6CJcPBnkC1r0jtdLSPlHV9VhXTVQq2Vsod6LSO1ljGPNNGsAOhYwC5nl+c+47FWEJggKdkVkvehBsaD+KPZ33AfwzUrP7pxY9CB/hBDoqWJWG91HaKKgfFYQ+jMouIdQWZhGH8Be7PEaWIufyqI312wi1Bb2EDpaz3wJrI30OoDX4h2JVsPYjrZHnaYA3kWoCV5CaOu/GtUmRGQXwKq9xJ6IAKG541SThv1MTX2N2HP69pweQmfxptVgxpaR1rmbKA8IHc+bFfya2QPQF5Fu1AQR+9ncR2i+2AewEds/6fzHHm/PWcXp2niyjGk/u6izeU1Vo36IKvXTOqvtPfcRLgK34v8uIrIN4EBVd6o4AStrFaFZqQMAqnq7wtfO/T7s72raStgKgFuxx11VPRSR1Zr+7VproWvy9o+9Abvqi4gXkd3EON1t+3/yZ+MGgP3EH2UH4ef2NsIFpIvTP7+vInz5ouOXACwnXnfSa0RBbA8heK0hBOU9O+88ZcRfq4/RKJmrVuYQwJWsYwqIAm/884z6A6LzvIDTo3Qmnf/Y4+0z2EXoDF5G+HwGsIBV8rMbAtjB9EMbLyD8eixqw4LfAGf7U9ZLnM8pVnHZUtXbqrpjwX1p0nEFTPM+xn1eJ/+uKZJDMPft3561+YIWvSYP+9LtWLBbQ+iVfywiK6q6bx1AhwhfytvASbttNF43aSP6RSAiSwCu25jfodVMoi/yYEwHW+Zr2P47AG7HRh7ctxrPhqpu5iwjso1wsYoPqaykVohRJ+PJH2JytISIrCF83quqen/SZzTpeIR/FyDUFIcINfV4zW3qz84eT9VpbQG0i4zmB/tOpdXi+wgjRIBQw96O7Yve66NpzilRTvRr7mJiWyXDYEu8j2Hiux+3BeBARPrJX+HJIcLnfUhqGQsf5CP2JdkXkU2EmuE2RrWYWwjNOlGzw7odkxYM41/Ux1OeTuZr2B9EByHwJzs1C9USrfYaTc6pQ1SDz2z2sYto/LmFpBwf/bE/FJG7CBewqLmsss8uLwtumwi1+Itj5g6kTn6KXeyjf6v4BauPcDHKDMR28XiGMHFo3L/zFsL37nLsgjPIGsJozX57eYcklngfA4zmryRfcyAiywi/xIZIadak8loT5COx9vVtEemp6qGq3rcv0QbCH+wVnP6Snjq+inMYszsKZss4W8sqGqii16pryN4KcFIDPmE/m69Y+YWD+7jj7d9vBSFobdnzo18qVX52ucQqD+sItc7ljH/fkzbwDKsADhOf5QryNT8MMLlG3seoOWWirOCfQ9H3kTns1C4YBwh9KlX9+qSE1gV5kxb07gFYtzH0PYRgPwsn51bBeOp4m3mlgT7WgXc7sX3PyotqjkD6zNms1514vAXWZauRriNMrLmOWCBpeiy6jdveQBhtlPbdOcL4C94VnA2EURPjuHKjCX6Z7HMCmllGoej76CL7AryJ8AsgM8CLyF6y6YaKWeiOV+vRT7OJMKoi/tNvC6FWsW77pu3Aib6w42ptmeychkiZFRn7Y81VhgW6aNZg1msVZsfu2nneSmzvw2pe9l7GdbadOocCxwM4me9wG+Gn/ttVfnZTygzkFozTJvZEuoiN8461Y5fuSJzQ3NOPD0Swx/1p5j+You/jQtEmrth5rqPa2d3n0sLW5O2P+rJ9Ee5j9MW7glBTP1WziHXA3kSJsczWnAAAG/b/K6padCzwGkI7ZDQKp4PQlDRA+MldpIyN2GtFU8BX7DXz1IA6sSDQQRiGGbVBvxv/A7XzGiLUrIEQcM9cYLLOP8/x9u+5aZ9L1J7bw6ijb+rPzt7nJjIWFqvAuJp0ct9NTGiPL+g2Es0m1p/QTbSnH9lQxG3EfqXZ39MBwuiccU0nRd/HVL8uYufMJSJKWtiavH2pLmI0TPGm3Y4QFj1Ka3OPgkLWL4C8biPUMLYwxZc4ao6wc9lDGDGSrJHnKiPxWrux18jbGbuF0Dn8GPZHjhAoLmYEwqsIF4Bde27WDM+s8590/D2Ei/aG7dtAGE2zA5T+7KJfcnXNht2X7Kn8VwFcEZGo07iLCocDRm3s9vrrVsZRYtz6wAJ8D9nfqUkVg9zvw8q5m7aPmrPwq1AWYV/KDU2sdUOUh/UnDLN+VVlQ28gz3E9EPEJlpPHRJFFTjaZMkhKR9SKdoOPeh/1ayFwNdtLnac95pjWu6noeLGxNfkobKF+LJ0plge7MrwQR2Ys3O1iQPZzhcMG8o3pOmeJ9XGi6g3xejfmFBxFZTfaTpG2bVuuDvIh07QOLpslzqBZNa9wMzciWnF2U6wKsciGjpBnvVn96uV3KqHlPGqWV+33IaNmQcS5MKK8VrG/kTsa+HnDSDDkUkV7atjLlL2zHawFRUhEgrFnCRBg0rU0ATyRlhmZEwxo6W3J6luctAKsi8ro9Z2bNhRYwsmbY9ic01eR6H9aJ2xk3gs3mSnRxen2aVrLvRNYw0isYLcQ2QOhLej1l29S/+lof5K3GwjY9Ks2C9mvWsbmNjFE6trzCOuxXY8YggEbF2to3kJHUZFJbfIH30U9r77fz6CEEtqcYM4PYntdHGP1VaHbugung9DyC1zO2Ta31QZ6oank6JudwBueK1SYP6g6W4y4GdlGcWCuNPa+SFTTPs3kP8udn6A9RjVQVGC3D3NbBB67sC/zoe4PcMecv/8Lf3sDZ1W2BsM5Q3ot8fBJdB+EXDjK2TaXRIO+cexPhJ1jUlvcUwL73/tOsYx79xb9X/4kBuPTHv4sf/4mfbaSsFz/8biPv69If/26jnx/LKlcO0Mz3vY2fX1RWaccvcz/VAvlUv9hifTZ3McoDEZ9zkLZtKo2NrnHOfQDgGsKKevt2ewLgmnOu9Z0vRLQA/HH+W07WyXzJ/h95CJw0S0UjcIa2oOKZbWXeUpM1+be992mJLB4455gyjYhm7zh/8M7L+ijuJ7Ytx+6f+TVQZZ9Ok0H+idXY9zDqOY7WWWn9WFkimn++QA19UTQW5L33N5xzn8NoHCgQ2uTvee8/aeo8iIgy1VCTn7VGO14tmDOgE9F8Yk2eiKjFXv5o1mdQuXlfhXKuT46I5krpcfI/HPz33DHnJ7p/rXR5TZj7mjzHyU+vzeOh21YWx8lXU1ZZ7HglImozdrwSEbUYa/JERC3Wwo5XBnkiogiba4iIWozNNURELcaaPBFRe3mff6nhRcHJUETUFqUnJ/2/T38/d8z5s2/9MidDJTnn3kJYDP9RPFGIc+497/3Hace0ddIGJ72wrGQ5QDsnQ/3oe80sMvvKG93yL/LyRfnXmDNNJg25BeAmwtX2mnPubmz3zabOg4go0/HL/LcF0WRNvue9/4Ldv+Oc6znnPvTev48KfmYREZXWwtE1jdXkATy35hoAgPf+EMAN59xHAF5t8DyIiNIdH+e/LYjGgrz3/jKAty2Zd7Ttuff+GqZMhktEVKkacrzOWtNJQ+5kbP/NJs+DiCjVi+o7Xi2B9xBAT1VvJ/b1ABxglAJ1X1U3RGRLVTdFZL1svtcmm2uIiOaa9y9z3/KwIA5V3QcwjB7HXFBVp6pLANYAbNn2dRF5jAryX3MyFBFRpPq29isA9uz+ACHH9WG004J/pKuq9+3+WmLf1OY+yFeRCOA8l9XG98SyFqccoKLx602pvq29A+Ao9vj1tCeJSB9APKj3RARIaeIpau6DfNsmvTRZVhvfU1vLanoyVNuyoAEVXbgK1ORFZB3AesqunSna0VfiNfcosIvIioj0y9Tq5z7IExE1pkBN3gL5pGA+BHDB7ncAPM143klbvXXUwppungIo9VOIHa9ERJGXL/Lf8rmLUZDuwppkRKQTPUFEuggXg8gAo6abJQCPyrwlBnkiokjFk6FU9RA4aXMfRo8BPEw89ShxzGWr0T+OHTMVNtcQEUVqmMma1j6vqsux+wMAG5OOmRaDPBFRZIFmsubFIE9EFFmgNWnyYtIQImqL0qvZ/t/fu5075vy5X7m+EKvnzn1Nvm3joZssq43vqa1lNT1OvslEHgs1Tp5JQ6rlnPtgluUTEZ3SwqWGm07/93WMmmAcgGXn3AoAxBKKEBHNxgIF77yabq7ZRpj9ddd7/33n3D0AVxs+ByKidPPdRzmVpteTf+Ccuwhgyzl3EDb5502eAxFRJtbky/PePwHwvnPuiwCeNV0+EVGmFna8zmx0jff+AYAHsyqfiOgM1uSJiFqshW3ynAxFRG1RfjLUv7qefzLUP7zNyVBVaGtyA06GYlnJcoDmJkO17fOLyiqNzTVERC3GBcqIiNrLv3g561OoHIM8EVGENXkiohY7bt9YDwZ5IqJICztemeOViChSwyqUIrIqIn0RuZ6xf8v+v573mCIY5ImIIt7nv+UgIj0AUNV9AMPoccK6iDwGMChwTG5z31zz4offbaysSsbZzllZbXxPLGtxymm6rNKqH11zBcCe3R8A6AM4TDxnzQJ6kWNyayzIO+d+ynv//djjdwD0AAy89x9nHdfWSRttnPTS1mxDnAy1OGWVVv3omg6Ao9jj11Oe0xMRAOip6u2cx+TWZE3+DsIVCs65ryFcqZ4AWHLOfei9f7/BcyEiOqvA6BprQ19P2bWjqjt5X8cCO0RkRUT6uU8gpyaDfHydh+fe+6+e7AjJQ4iIZsoX6FC1QD4pmA8REiUBoYb+NL5TRFbtte7bvu6kY4pqsuP1lnPukeV1PXLO3XLOveOc+wqsw4GIaKaOff5bPncRAjfs//sAICId2zaItgFYAvAo65hpNRbkvfefeO8vITTRDBFq9l0A2977G02dBxFRppcv899yUNVDALBmmGH0GMDD2P7LVqN/rKqHY46ZyiwyQzFZCBHNpxomQ6W1z6vq8oT9udv0J5n7IZRERI1p4bIGTBpCRG1ROonH//knl3PHnD//z+8xaUgV2jqel+Ohy5XVtmQyHCdfTVmltbAmP/dBnoioKUWGUC4KBnkiosgLBnkiovZi0hAiohZjmzwRUXt5BnkiohZjkCciarEWjq4pNBlKRD4AMFDVj+3xXQCrAB4DuKyqn1Z8fu27rBJRXUpPTvrBtV/KHXN+8qP/2MrJUBt2g4h8GSHAfx7ANYT14t+u9OzAyVCLUA7LqqYcoJ2ToZqcuFbWnK8AMJWiQX4Jo2WBVwDcV9WHIvIMwDcnHWzZoOC9/4Zz7k2Ei8TYzFBERI1hmzyGAF4VkVcR8g6u2nZv+zI55z5EyFO45Jxbsc37AFaccyvMDEVEM8cgj68A+ATAM8Ta5hGacB5OOPaC9/4OADjnvuW9/6xtf2jpAImIZurcD6FU1dsicgjgVZzOVrKHydnEnXPuSwjprB44527ZcT0Az4ucBxFRLV6c8yAvIu/Fau9xQwAXEbI+pfLeX3bOfRHAgff+E+fc5xASe/8v7/3lIudBRFSHc1+TB7AL4Mcy9m1hwugaywoV3f8EoemHiGg+tDDIF83xmjUutIPQ7EJEtLiOC9xyEpFVEemLyPWM/et224pt24r2TfdGRnJNhhKRY4yfmOQAHKhq1ePk23dZJaK6lJ6c9Gztb+WOOa/t/qeJ5YlID0BXVe9bwH4UT8xtyboHqjoQkV0A26q6b8PSjwBsqOp++qvnk7e5ZgXhA/w6gDWcHS45UNXM9vgy2jbppcmy2vie2loWJ0OVV8VkqCI19JyuIAwwAcIcoz5OD1Lp2m3H9ndt+1rZ4B7JFeRV9SEAiMi+qj6Y9HwiokXkqx9d00GokUdej+9U1Z3Ywx6Au9F9EQGAnqreLnMChdrkVfXzZQojIppn/jj/rUrWrLMXNeWo6m2ryb9uTTpTK7wKpYi8ifCTopPclzG8kohoMRTrUF0HkNYxuhOroQ8BXLD7HQBPM16uH9XYRWQVAFT1vj2/m3FMLkXHyX8ZYdZrWoeDR/bwSiKiuVekhm6BfGfC0+4CuGT3u7BJpCLSUdWh3V+PBfg+Qtt8tEbYEoDt/Gd1VtGa/E0AvwnglqpylioRtUvFzTCqeigilyx4D2Mjax4CWLbtWyKyiVDjX7Nj1kXkCMDj+GicaRQN8kcAPmKAJ6I2On5R/WsmOlejbcv2/30Ar+U5ZlpFk4Z8GYBX1a9WdQITcJw8EeVVepz8n/7i38wdc376D/5zK5OGvA3giyJyE8AjJMbLq+qVqk6MiKhxfiHidiHT5HiNj5Mv9IlYopA+QmcCEHqO9733mWkD2zbppcmymp6I0rbPr8my2jwZqsmyyqp6aOQ8KLrU8NSrRTrnPgDwBkIGqWgmVwfANefcM+/9zWlfm4ioCv6YNfky3vbepzXnPHDO3WvwPIiIUp27mryIfA3Arqr+tj2+O+75E9rkn8QShUTTfC8grIszyDyKiKghxy/PX03+9cTjqT8B7/0NSxTSj73uUwD3bG15IqKZOnfNNap6KfG4VAanrEQhzrk3vfffLvPaRERlFRhRvjCmWbvmpxCyQEWL5uwB2FTVH5Q4j00A75c4noiotDbW5ItOhnoVIY/rAKMRMn2E/K7LqvrtzIKc+whhLfoBQrNPVLADcNF7n2waAjgZiojyKx2hv/3WSu6Y8+anewtxRShak98BcE9Vr8U3WkaTjwD8UtaB3vtrzjl4768l99kFIFVbkxtwPDTLSpYDcJx82bLKYnNNGAmTlss1GjUzyWbaxrTAT0TUtOOXRdNez7+i7+gIoWkm6TWczn6SynvPhc2IaG7NKmlInYoG+R0AOyLymWiDiLyF0FRzv8oTIyJq2rF3uW+LouiyBrdFZAnAE8smDoSlCe6rKpclIKKF5hcoeOdVeAilqm6ISDSE0gN4pKqczEREC6+NQyinWrtGVQeYnPaKiGihnMdlDc4QkQ8AbGCUXPYQwG+o6r+r8sSIiJq2SG3teRWdDPURgKsIa8pHQyYvA3gHwLqq/suKz6+Fo1aJqCalI/T/vPh3c8ecv/Lk3+cqT0RWERIs9aKE3ZP2TzqmiKI1+csAbic6We+IyDaAGwDGBnnn3FsImcsfxROFOOfe895/nHZMkxMpfvS9ZhbDfOWNLie9sKwz5QCcDFW2rLKqngwlIj0g5HIVka6I9OKJudP2R/uyjilqmnHyacsNfwVnV6w8xZYZvolwtb3mnIu/DkfmENHM1TCE8gpGaVIHGK35NW7/pGMKKVqT37YCk+n63kV68I/ree+/YPfvOOd6zrkPvffvo4KfWUREZdUwhLKD0xNFk5XhtP2TjilkmmUN3hWRlcT2PoB9SzJyQlW/EHv43Dn3VtRM470/dM7dsHVrXi164kREVXvJIZQY4nQi70jatlO895edc1edc8No7Xhb5uCac+7LBc+DiKhyRWryIrIOYD1l146qRkPMhwgZ8IBQQ3+aeG7W/nHHFNJYIm8A8N7fydi1W+Z1iYiqUGQIpQXySfOF7iIMNgHCsPN9ABCRjqoOs/ZnbJvKvCy5lro6JRFRk3yBWx7RqBgR6QMYxkbJPMzaP+aYqUw143UaiaQhp3YhrGzJzFBENFN1TIaKNd3Ety1P2F/ZigKFJkOVLsy5j7KShmSsKc/JUESUV+kI/V9/ZjV3zPmFP7m/EL20jdXkTeGkIW2dtNHGSS9tm0wGcDLUopVV1ssWjuZuNMgzaQgRzbPjFrYdNF2TJyKaW8esyRMRtZdnkAdE5D2EtRU6yX2JGa5ERAtlgVK35lZonLyIfBkhl6tDWOLgCYDndp+IaKG9hMt9WxRFJ0PdANC3ma8DVb1m9y/j7Ph3IqKFclzgtiiKJg05BnBRVb8jIo8A/CNV/UMRuYiQ67XUamkpWtjXTUQ1KV29/g8//fdzx5y/86f/ZiGq80Xb5A8BLAH4DoB7AG6LyAZqXJagreN5OR56ccpqYvz/K2+EbJr8XpQrq6wWLkJZuLlmEyHIw1JS/SUAjxFWYrtVtHDn3AdFjyEiqssxXO7boii6CuVD2MI69nhJRD6nqp/kOd4593WMmmAcgGXn3AoAxBKKEBHNRBvbhwsFeRF5T1VP5WJV1U9E5B27/40JL7GNsE7yXe/9951z9xASgxMRzdwLtzg19LyKtsnvAvixlO0OIc/r2+MO9t4/cM5dBLDlnDsIm7jUARHNhzbW5Iu2yWdd5joAehn7TvHeP7G8rs/sRkQ0F9o4hDJXTd6GTnoAXkRepjzFATgoUrD3/gEsbaBz7s0oJSAR0ay0cXRN3uaaFYRA/nWExB/DxP6Bqj4pcR6bYNIQIpqxRRo1k1fRyVBfV9XPT1XQhMxQ3vu0iVRtbCIjonqUjtC/8xd+LXfM+bX//TsLcUUoOoRyqgAPhMQgzrnUBCF2AUjV1kkbnPTCspLlAM1Nhvrxn/jZ2ssBgBc//O5CTYZ6sRBhu5imV6EsnBmKiKgps2g6EJFVhCbwnk0yTe5ft7tLqrpp27ZUdVNE1iflg210FUoOlySieXbs8t+qICI9AFDVfQDD6HFsfx/AvgXyrj0GgHUReYwcC0NyFUoiIjODIZRXMBrIMgDQT+zvxrYN7DEArKnqkl0cxioa5F9DWKsGCFedn7f7BwiBnohoYc0gyHcAHMUenxqAoqo7seaYHoBH0X0R6YvI9UkFzP0qlERETXlZoBnG2srXU3btTGonL8qacfZU9RA4WSASIrIiIv1xNfqiQT5ahfIbqhoF+McIbfQTryhERPOsSA3dAvnEYB7rOI0bRO3wCOt5AaFW/zTjZfqxwL5q5d+353czjgHQ8CqURETzrI7RNRNq9XcBXLL7XQD7ACAiHVUd2v31WIDvI7TNR32gSwgLP2YqNBnKCnlTVb9d6KDpcTIUEeVVeszLb/1c/slQv/5H1UyGspr+AEA3uiCIyIGqLltQ30Vot7+A0OG6b8cc2TFnhl3G5QryInILwCpO/ywYIlyFbqjq9ye9hnPup7z33489fgehI2Hgvf844zDftkkvTZbVxvfU1rKangzVtsxaAPDKG93SQfdfFAjy/7iiIF+3saNrROQdEXkKYAOhmeYagM/b/38bwK8CeCYiv5ijrDvRHefc1xAC/BMAS865D6c7fSKi6pyrVSgtOfd9APdUNTkjNWqX3xSRLQD7ItJV1e+MKSt+1Xvuvf/qyY6QPISIaKaKjK5ZFONq8tsIQ3bGLjlg02w/xoTGfwC3nHOPLK/rkXPulnPuHefcV8CJVEQ0B85VTR6hx/ednK9zC8A3xz3Be/8JgEvOuS8CeAOhZt8FsO29L7NMMRFRJdo40mNckH9VVT/N8yKqeigiuQqMJwuJMGkIEc2D4xaG+aLLGtSFM2aJaOba2FyTOYTS0vx1VPUHk15ERF4FcKSqaUm+Q0Gnk4Y4jH4ZMWkIEVWhdLfpP/vMP8gdc/7pd/71QnTTjmuucQiLkFVSEJOGNF9Wm5NDtK2sNo+TZ9KQ2RoX5DdqKI9JQ4hobrWxTT4zyKvqnax902LSECKaZ+0L8VOk/yMiaqtF6lDNi0GeiMicq+YaIqLz5uWsT6AGDPJERIY1eSKiFmtfiJ8iaUjD5vrkiGiulB7l/utv/mrumPNb3/63CzGqfu5r8m2dtMFJLywrWQ7QzslQTU7IK8u3sF7ZaJC3bFDw3n/DOfcmQrapcZmhiIgawyGUJVj2p0OETFArtnkfwIpzbsV7/35T50JElOYla/KlXPDe3wEA59y3vPefte0PLR0gEdFMzWJ0jYisIuTM7qUl5RaRLVXdFJH1WKLvscfENbnUsHPOfckyQz2IZYb6AACXOyCimWt6qWER6QGAqu4jLAjZS3nauog8hmXQy3nMicaCvPf+MoBnAB56728AuIeQFHxo+4iIZsoX+K8iVxBq5EAI4v2U56yp6pIF9bzHnGi049WyQkX3PwHwCcDMUEQ0H4rU0EVkHcB6yq6dqFklhw6Ao9jjtLwaPVvyPWqayXPMiXkZQrkJgB2vRDRTRTpeLZDnDeZTi9rcRWRFRMbW2tM0Obomnhnq1C4AF5ER5KtIBJBXG8tq43tiWYtTDlDN+PWmHNcwOdRq/EmDqE0dwAXb1gHwNHHsKgCo6n3b1510TFJjQX7azFDMbDT/5bCsasoB2jkZapEyQ9UxtmZC081dAJfsfhdhWDlEpKOqQ4RKcVQxXgKwDeBR2jFZmk7kzcxQRDS3juFz36qgqocAYM0ww+gxgIex/ZetRv9YVQ/HHJOq6Y5XDpUkork1i2UN0mr6qro8YX/uvoB56XglIpo5LmtARNRiL1sY5hnkiYhM+0I8gzwR0Yk5z68xFSYNIaK2KJ3E41d+7pdzx5zf+6PfZ9IQIqJFwuaakkg1f0wAAAusSURBVCxRSB9hUD8QZmrte+8/zTqmrZM22jjp5UffS05mrscrb3Rb+W8FcDJU2bLKYmaoEmxJ4TcAfBOjGVodANecc8+89zebOhciojQvffvq8k3W5N/23l9J2f7AOXevwfMgIkrVvhDfbJB/4py7BWAPo2UyLwBYwdlFy4iIGsfmmhK89zecc59DaJOP1j9+CuCerS1PRDRTs0j/V7emR9d4hGUy9+Kdrc6597z3Hzd8LkREp8z5kPKpNLYKpXPuKwCiztVrzrm7sd3sdCWimXuJ49y3RdHYZCjn3Ne891+IPe4BuOq9f98598h7fynlsPZdVomoLqUnJ/2Nn303d8z5L999yMlQCc+dc29FzTTe+0Pn3A1LGPJq1kFtHc/L8dDlympbMhmOk6+mrLLaWKtsrLnGe38ZwNs2ISra9twShtSeJ5GIaJKmk4Y0oemkIXcytv9mk+dBRJRmkYJ3Xly7hojIcHQNEVGLzWJ0jYisikhfRK6n7OuJiBeRx3bbtu1b9v/1Sa/PIE9EZLz3uW9VEJEeAKjqPoBh9Djmgqo6VV0CsAZgy7avi8hj5FgtgM01RERmBm3yVxCWegFCwO4DOIx2WvCPdFX1vt1fS+zLxCBPRGSK1NCtqSStuWRHVfOOGOxgtJYXMFryJVlWH6PVewGgJyIA0FPV2+MKYGYoImqL0pOTfv5n/nrumPOHf/LfSpdnbezbqnpogXxFVTdTnreVtR3A3rhafdNJQ94CcAnAo7xr17R10gYnvbCsZDkAGpnk1dQEL2ARJ0NVX6/M6BwdRO3wCKvxAqFW/zTjZU7a6kVkFQCs6eYpgO648ptMGnIL4WT2EdaueS22vvxNAFygjIhmqo6kIROabu4iVHyBUXyEiHRUdWj3uwgXg8gAow7XJQDb48pvcnRNz3t/xXt/x2a5bjnnPrR9C7EGBBG127H3uW9VUNVD4KTNfRg9BvAw8dSjxDGXrUb/OHZMqrlfu4aIqCmzSBqSVtNX1eXY/QGAjUnHZOHaNUREpumafBOangz1CEDfOmDjHjd8HkREZ/gC/y2KppOG3EBof2fSECKaOy/9ce7bomiyTf5zsaQhd5xzPefch97798GOVyKaA36BgndeTWaGugfgNxLj419FWIvhXe/9Z1MOW5zfREQ0a6Uri595/a/mjjnfefo/FqJy2lhN3nt/2Tl31Tk39N5/27Y9R2i6+XLWcW2dtPGj701cV6i0V97otvbza1tZzAxVTVllzfkKAFNh0hAiIsOkIURELcaaPBFRiy3SqJm8GOSJiAxr8kRELcY2eSKiFmtjTZ5JQ4ioLUqPW7/wk5/NHXOOfvAtjpOvQtvGrgPNjV9v83jotpXFcfLVlFXWnFd6p9L0AmWnOOc+mGX5RERxXLumJOfc1zFqgnEAlp1zKwAQW9eGiGgmFmkJ4byabq7ZRshneNd7/31bz+Zqw+dARJRqkZYQzqvpZQ0eOOcuIqT+Owib/PMmz4GIKMusavIi0stK42dp/oYAeqp6O2tblsbb5L33T2x54Wd2IyKaC9773LeqWH7X1HW9RKQHAKq6D2AoIr20beNev9Eg75x7yzn3Jcv1+sBS/8E5916T50FElObYH+e+VcWC9VHG7isINXYAGADoZ2zL1GRmqFsIGaCYGYqI5tIsavITdHD6AvB6xrZMTbbJ96bJDFXF2Ne8Xnmj21hZTb2vJj8/lrU4ZbXxPVWhfd2uzQb559ZM8ykAeO8PnXM3nHMfAXg145iFmFFGRO3w4offzR1zRGQdwHrKrh1V3Uk8L2lgzTSTDBFGJAKhBv/U7qdtSzX3maGIiOaRBfKdnM8rREQ6qjoEcBfAJdvcBRBdGNK2pWq049V7fycK8IntlWSGcs5tOed2nXPXq3i9HOVtN1TOrnNuzzm31UBZ21ZeI++tbawZctdunVmfT1Vi76nW70Xsb7j27/q8sOGQl+z/kYcAEA2rtBE4Q1U9TNs2toAiHQ3zfAPQA7Bl97cAdGssq4MwsetZA+9rHaE/AwB2o/s1fobdJsqKlbkKYK/G1+8COLD3swugU/P7Oaj7M4t9L/bsdgDges3/Ruuxcvs1ldOP/Q1v1/k3fJ5uM127pmJ9hC88AHwTE4YVleG9H3rvNxCGL9Vt33sfXan3MPqZVjnv/aH3PnpPXdT8/pxzXYzaFuu0771fs9tw8tOn45xbRfh5XTvv/Y73fsV7vwLgEYD7NRY3QKjYAMBSjeWsYPQ3fIAa/4bPkzYF+dcxGlY0RL1fxsbEgi4ALCP8QdfGmhseIyw9UVtANJsA7tVcBgCsNtQE9TaApah5reayAADOuT6AYeJ7UimrZLxu3wt47/N0GE7jKUYXE6Alf8Oz1qYg32rOuR6AC7FafS3s9ZcBrFhNuxZW6z1o4EJyBGDTe78G4IIFxToNray9BsoCwnvbrLMA++7Be79kj+v6Xuwj1OZh/x87aoTyaVOQf4oCw4oWiXXgbVjwqJ0F3l2Etti6rCBcSHYBXKqrs9ya1qKmjG8iNEPV5TFGzQ0DhD6O2jTYsXsFo/e1h5q+F1EFxn5xDdBMc2jrtSnIJ2sBdf2knIUt6wOoldWuI8uo8Y/Mex9dtK4CeOS9H7vI0rQStem3UW9zV/w7WHufBkKbda2/7MxjjC5YK6j/e7GB0OlaZz/DudGaIB+rBewi/GSu9ctvtY2uDTmss1njOoC+c+7AbnXWrvejIZQA0JI/skGsjXxQ5/ciahe3z2+pgc+vixCAa+W93wHwtn2Gnbrel3OuY0OFdwHcqqOM82jec7wSEVEJranJExHRWQzyREQtxiBPRNRiDPJERC3WdCJvWiAich3ABsIojiHC0LlbqjqTUTci4gGs5FyilYjAIE8pRKSDsApeF2Eo2z7CRLM1hLHmczG0UkQOAByp6srEJxOdUwzylOYOQoBfVtX4xJd5q0FzOWSiCRjk6RQR6SJMW19LBPi5M00yhklsje49VWVWMmoFdrxS0gZCIoK5aJIhonJYk6ekHgqs7xLrnL2A0Jxz1dKW5XqOiOwirBS5i7D08J6q3rZ+gTsIvyoOkdI0Y8dCVdfijxEWIovKu6Wqt2PH9BGSyvQQOpNP9ovIHmwNc+vkBcIvmvu2bQujnJ47qroZe383ERbGGwDYjpdJNEusyVNSNJJmIhHZRgimGwgLmh0BeGIBOvdzEAJrlLkpaoLZte1rdsu7tni0ts+KvdaWNUHF39+mvd6a7V8FAOvAXbP7zm5RgI9W5YzOZ1VE1mMXjXftNZtKJkOUC2vylBTPApTJAuc6gKVY2/2GiFxCCHobeZ5j27rx59hxfZweLrlpNeZJhrGa+S0A1xFbETLRjj8QkfsIS+lGzVNnLnCxforlWH7NTYTa+y07ZmC/Thjgaa4wyFPSIUZNEuNESYSTQW0fo9p0nucAwGHiOT0AmHI8/ElTk6oORQRIXLTsYnHFtk/Mdo9RGroDe724fYRfJ89EZB/W3DTFeRPVgs01lHQXQCeROb5uVdZ+xzY1ichjhKacq6q6hFHz0MTXjTXhxG9De501K3sr1jdANHMM8nSKNUfcB3An0Zad9AjhYpDMftTHqGac5zlpoiaRk4QfiTb8qdh5dAFsRM0uyJdIfB/p7+OEqt63DuAV1JtRi6gQNtdQmqsIHZ8H1q4dzXhdAdBX1WVVPbT27F0R2UBosriJEETfBcIFY9Jz0qjqwJo+tmPHbZV9U3Y+QGjfj0bYrOL0BSfqF1iF/SpQ1X0R2Ym9jyi13wbCqJ8VjEb/sOOV5gpr8nSGNUGsIHQqXgFwgBD0ewgXgOh5awi1/m2EZRA6CJ2TwyLPybCGECx3EQJ8VbNbNwFcRnhPZ9JEWt/AYazcaHsU0LcRsjHdtPuHCBfAh/aaXTt3ornAzFBERC3GmjwRUYsxyBMRtRiDPBFRizHIExG1GIM8EVGLMcgTEbUYgzwRUYsxyBMRtRiDPBFRizHIExG12P8H0mqG6CElOzgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize subsample of dataset\n",
    "dataset = get_dataset()\n",
    "bounds = dataset['obj'][0].bounds\n",
    "train_data, test_data = dataset['loaders']\n",
    "\n",
    "fig, ax = plt.subplots(1,1,figsize=(6,4))\n",
    "ax = sns.heatmap(dataset['obj'][1].X[:25], linewidth=0.1)\n",
    "utils.update_ax(ax, r'Synthetic Dataset: $\\{(y\\cdot e_j, y)\\}^{n}_1$', 'Coordinates', 'Data points', \n",
    "                legend_loc=False, despine=False, title_fs=20, label_fs=18)\n",
    "\n",
    "fig.savefig('plots/synthetic_data.pdf', dpi=fig.dpi, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Standard Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:27:40.234760Z",
     "start_time": "2021-02-24T13:26:34.515205Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min loss: 0.0\n",
      "Epoch\tl-tr\ta-tr\ta-te\ts-te\ttime\n",
      "0   0.8625   0.9212   1.0000   1.0000   0.1469\n",
      "10   0.0000   1.0000   1.0000   1.0000   1.3617\n",
      "20   0.0000   1.0000   1.0000   1.0000   1.2811\n",
      "30   0.0000   1.0000   1.0000   1.0000   1.3431\n",
      "40   0.0000   1.0000   1.0000   1.0000   1.2692\n",
      "Epoch 49, LR : 0.1 -> 0.07500000000000001\n",
      "50   0.0000   1.0000   1.0000   1.0000   1.2779\n",
      "60   0.0000   1.0000   1.0000   1.0000   1.3465\n",
      "70   0.0000   1.0000   1.0000   1.0000   1.2857\n",
      "80   0.0000   1.0000   1.0000   1.0000   1.3662\n",
      "90   0.0000   1.0000   1.0000   1.0000   1.2961\n",
      "Epoch 99, LR : 0.07500000000000001 -> 0.05625\n",
      "100   0.0000   1.0000   1.0000   1.0000   1.2805\n",
      "110   0.0000   1.0000   1.0000   1.0000   1.3496\n",
      "120   0.0000   1.0000   1.0000   1.0000   1.2786\n",
      "130   0.0000   1.0000   1.0000   1.0000   1.3545\n",
      "140   0.0000   1.0000   1.0000   1.0000   1.2796\n",
      "Epoch 149, LR : 0.05625 -> 0.0421875\n",
      "150   0.0000   1.0000   1.0000   1.0000   1.3505\n",
      "160   0.0000   1.0000   1.0000   1.0000   1.2799\n",
      "170   0.0000   1.0000   1.0000   1.0000   1.3080\n",
      "180   0.0000   1.0000   1.0000   1.0000   1.3637\n",
      "190   0.0000   1.0000   1.0000   1.0000   1.2848\n",
      "Epoch 199, LR : 0.0421875 -> 0.031640625\n",
      "200   0.0000   1.0000   1.0000   1.0000   1.3533\n",
      "210   0.0000   1.0000   1.0000   1.0000   1.2900\n",
      "220   0.0000   1.0000   1.0000   1.0000   1.2914\n",
      "230   0.0000   1.0000   1.0000   1.0000   1.3463\n",
      "240   0.0000   1.0000   1.0000   1.0000   1.2923\n",
      "Epoch 249, LR : 0.031640625 -> 0.02373046875\n",
      "250   0.0000   1.0000   1.0000   1.0000   1.3320\n",
      "260   0.0000   1.0000   1.0000   1.0000   1.2930\n",
      "270   0.0000   1.0000   1.0000   1.0000   1.2698\n",
      "280   0.0000   1.0000   1.0000   1.0000   1.3658\n",
      "290   0.0000   1.0000   1.0000   1.0000   1.2609\n",
      "Epoch 299, LR : 0.02373046875 -> 0.0177978515625\n",
      "300   0.0000   1.0000   1.0000   1.0000   1.3345\n",
      "310   0.0000   1.0000   1.0000   1.0000   1.2796\n",
      "320   0.0000   1.0000   1.0000   1.0000   1.2973\n",
      "330   0.0000   1.0000   1.0000   1.0000   1.3687\n",
      "340   0.0000   1.0000   1.0000   1.0000   1.2701\n",
      "Epoch 349, LR : 0.0177978515625 -> 0.013348388671875001\n",
      "350   0.0000   1.0000   1.0000   1.0000   1.3530\n",
      "360   0.0000   1.0000   1.0000   1.0000   1.3389\n",
      "370   0.0000   1.0000   1.0000   1.0000   1.3493\n",
      "380   0.0000   1.0000   1.0000   1.0000   1.2777\n",
      "390   0.0000   1.0000   1.0000   1.0000   1.2694\n",
      "Epoch 399, LR : 0.013348388671875001 -> 0.01001129150390625\n",
      "400   0.0000   1.0000   1.0000   1.0000   1.3565\n",
      "410   0.0000   1.0000   1.0000   1.0000   1.2719\n",
      "420   0.0000   1.0000   1.0000   1.0000   1.3477\n",
      "430   0.0000   1.0000   1.0000   1.0000   1.2812\n",
      "440   0.0000   1.0000   1.0000   1.0000   1.2889\n",
      "Epoch 449, LR : 0.01001129150390625 -> 0.007508468627929688\n",
      "450   0.0000   1.0000   1.0000   1.0000   1.3417\n",
      "460   0.0000   1.0000   1.0000   1.0000   1.2838\n",
      "470   0.0000   1.0000   1.0000   1.0000   1.3531\n",
      "480   0.0000   1.0000   1.0000   1.0000   1.2967\n",
      "490   0.0000   1.0000   1.0000   1.0000   1.3114\n",
      "Epoch 499, LR : 0.007508468627929688 -> 0.005631351470947266\n"
     ]
    }
   ],
   "source": [
    "device = gu.get_device(0) # change 0 to None if cpu\n",
    "epsilon = 0.0 # standard training\n",
    "\n",
    "std_stats, std_model = linf_pgd_adversarial_training(train_data, test_data, epsilon, bounds, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Adversarially Robust Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:33:25.510779Z",
     "start_time": "2021-02-24T13:27:40.236284Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Min loss: 0.0\n",
      "Epoch\tl-tr\ta-tr\ta-te\ts-te\ttime\n",
      "0   37.1659   0.0157   0.0000   0.4790   0.7836\n",
      "10   14.3692   0.0020   0.0000   0.5520   6.9158\n",
      "20   5.1011   0.0013   0.0000   0.8940   7.0665\n",
      "30   2.2670   0.0124   0.0410   1.0000   6.9280\n",
      "40   1.4748   0.0529   0.0410   1.0000   6.9262\n",
      "Epoch 49, LR : 0.1 -> 0.07500000000000001\n",
      "50   0.7085   0.5650   0.7090   1.0000   6.9216\n",
      "60   0.8215   0.2833   0.1350   1.0000   6.9013\n",
      "70   0.7996   0.3048   0.5890   1.0000   6.9371\n",
      "80   0.8328   0.2765   0.2730   1.0000   6.8229\n",
      "90   0.7538   0.4643   0.7600   1.0000   6.9513\n",
      "Epoch 99, LR : 0.07500000000000001 -> 0.05625\n",
      "100   0.6460   0.8207   0.8330   1.0000   6.8618\n",
      "110   0.6430   0.7924   0.6740   1.0000   6.8561\n",
      "120   0.6531   0.7570   0.8640   1.0000   7.0416\n",
      "130   0.6563   0.7062   0.6910   1.0000   6.8761\n",
      "140   0.6347   0.7744   0.8180   1.0000   7.0031\n",
      "Epoch 149, LR : 0.05625 -> 0.0421875\n",
      "150   0.5867   0.9655   0.9070   1.0000   6.8941\n",
      "160   0.5776   0.9770   1.0000   1.0000   6.8731\n",
      "170   0.5618   0.9846   0.9040   1.0000   6.8993\n",
      "180   0.5500   0.9899   1.0000   1.0000   6.9345\n",
      "190   0.5394   0.9953   1.0000   1.0000   6.9470\n",
      "Epoch 199, LR : 0.0421875 -> 0.031640625\n",
      "200   0.5164   0.9993   1.0000   1.0000   6.8579\n",
      "210   0.5073   0.9989   1.0000   1.0000   6.8541\n",
      "220   0.4962   1.0000   1.0000   1.0000   6.9988\n",
      "230   0.4838   1.0000   1.0000   1.0000   6.8642\n",
      "240   0.4716   1.0000   1.0000   1.0000   6.8932\n",
      "Epoch 249, LR : 0.031640625 -> 0.02373046875\n",
      "250   0.4558   1.0000   1.0000   1.0000   6.8538\n",
      "260   0.4483   1.0000   1.0000   1.0000   6.9114\n",
      "270   0.4384   1.0000   1.0000   1.0000   6.9079\n",
      "280   0.4312   1.0000   1.0000   1.0000   6.8747\n",
      "290   0.4206   1.0000   1.0000   1.0000   6.9838\n",
      "Epoch 299, LR : 0.02373046875 -> 0.0177978515625\n",
      "300   0.4098   1.0000   1.0000   1.0000   6.8401\n",
      "310   0.4027   1.0000   1.0000   1.0000   6.9059\n",
      "320   0.3957   1.0000   1.0000   1.0000   6.8551\n",
      "330   0.3886   1.0000   1.0000   1.0000   6.9055\n",
      "340   0.3826   1.0000   1.0000   1.0000   6.9693\n",
      "Epoch 349, LR : 0.0177978515625 -> 0.013348388671875001\n",
      "350   0.3742   1.0000   1.0000   1.0000   6.8387\n",
      "360   0.3702   1.0000   1.0000   1.0000   6.8874\n",
      "370   0.3644   1.0000   1.0000   1.0000   6.8273\n",
      "380   0.3600   1.0000   1.0000   1.0000   6.8544\n",
      "390   0.3540   1.0000   1.0000   1.0000   6.9896\n",
      "Epoch 399, LR : 0.013348388671875001 -> 0.01001129150390625\n",
      "400   0.3481   1.0000   1.0000   1.0000   6.8794\n",
      "410   0.3458   1.0000   1.0000   1.0000   6.9521\n",
      "420   0.3431   1.0000   1.0000   1.0000   6.8243\n",
      "430   0.3388   1.0000   1.0000   1.0000   6.8032\n",
      "440   0.3344   1.0000   1.0000   1.0000   6.9219\n",
      "Epoch 449, LR : 0.01001129150390625 -> 0.007508468627929688\n",
      "450   0.3301   1.0000   1.0000   1.0000   6.8550\n",
      "460   0.3284   1.0000   1.0000   1.0000   6.9182\n",
      "470   0.3259   1.0000   1.0000   1.0000   6.8633\n",
      "480   0.3232   1.0000   1.0000   1.0000   6.9905\n",
      "490   0.3203   1.0000   1.0000   1.0000   6.9240\n",
      "Epoch 499, LR : 0.007508468627929688 -> 0.005631351470947266\n"
     ]
    }
   ],
   "source": [
    "device = gu.get_device(0) # change 0 to None if cpu\n",
    "epsilon = 0.45 \n",
    "\n",
    "linf_stats, linf_model = linf_pgd_adversarial_training(train_data, test_data, epsilon, bounds, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Standard and Robust Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:33:52.813127Z",
     "start_time": "2021-02-24T13:33:50.857303Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmUAAAEfCAYAAAAA3rEdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3gU5fnw8e8kBAiHTYAAKgmSBBhDgCIYD8EzQfHcUjA92LdqC7G1P7EVPL1IW6W1QHyr1FpDbLW/nowg9dAKlqC2SvCASJUlDkKCLCjHkCyHBEKY94+ZXXY3u5vdZHdnd3N/rmuu3Z2ZnblnZnf23ud55hlF13WEEEIIIYS1UqwOQAghhBBCSFImhBBCCBEXJCkTQgghhIgDkpQJIYQQQsQBScqEEEIIIeKAJGVCCCGEEHGgh9UBiPCpqjoR+LCD2QZomtYYwXUuAmZrmjYgUssMcb33AlM1TZsay/X6xLAdyAMmaZq20ao4ujtVVWcAy+ngsx3k+7ERqNI0bXEn1r0IuFfTNCXc94rIUVU1E6gEJmJ8JzcCG4BFmqbVxTiWqJ8TI3X+C/KdaAReAO6L5O+F6DxJyhLbMowfqXaS6AuWD5RYtXLzZJZnvizF+BEQieE+YIX5PBM4D7hPVdUyjAQ7Wb4j3YKqqnnAdqAOqMBIKPKBe81ZyiwKLZoiff7z/U6UAA8AN6uqOkX+dFpPkrLE9qGmadVWBxEJ5j/CFb7/djVNK8Pak60rEasDZmOc1ERiaPT5PG1UVfUFoB5YRBz/iAf6PnRzFUCdpmn5niNVVX2U03+cIi4WxyKG5z9/34llwFpz6FSpn3xeI0falAnLmVUSi4jiibULZgBVwBog0yw5EwnKLB2rBm62OpZA4vz7YKXzMI6dF03TGqNVwhOLY2H18Ta/EzMxzm/3djS/L6vjTzaSlAkRgEfV5QqMdhcQx6UrImQTgQargxBha8BIzESEmSVc1RhVmcJCUn2ZxFRVrcBoiKr4jL8X45/NAE3TGs22Gosw2hdkYnw5y4IVRftr+Kyq6myMKgZ3Q+yOlq2q6nKM0iiANaqqglHEPsBjPbN9XwNTzOWeh1G1OMv337IZz30YidUyc3QJMDDExrllGNUlrlg3YpSwtEvMzATOFQ8YjY/vc8UUbHoY+9K9L8xjW6JpWn6oxy+UGPBpRO8vjkBCiSPM43cvxo9Eg7ms7cHW30FsrjZlrn/0ZT7TXfumBKO6ukLTtGV+luO5DxvM7av2mB7qsZyBd+lCtRnTIkL4PhDa/isx55lobtN9rlgDrV/TtLpg0/zt2xi6D1iuquoajIb97UrNQv0ch7IfOzo3eSx7YqBleMzj91iEe/7zsz7wOdd0wRqgRFXVPI9zXlfP32H/tnR3UlKW2CpUVdX9DItc08F9AvZUCmz0OGm5fqBmAVOBgXR8dWeoOlq2a7xr3nxgUgfLzMS4wKGC00mX1wUP5jZXYJwQZmKcFM4zn08JMfabOd0oFoxqzEzzBOu5rhKPbZplxtSIsZ87nB4uVVU/xNiWCnNUh8cvhBhcy5rts7oyoDrERvGhfo5COX6uPw4bzHk+NF+Hw/39AA5h/OhMxPgBcydc5mflQ4wf1KkYx7nCTHx9VXrEDcYPUVjVNub8yzE+W5MwPpN1GPsllO9DqJ//NRg/gjMx9uMaVVXzgq2/g9gspWnaCox9UoKxLbqqqst99n84n+OO9mPUj0WI6/BcVkTPJT5ciZLn/uzq+Tuavy1JSUrKEttijB8QX3UAZgmI6wu7AtwlBhPxKCnQNM2r8bqqqjOB7aqqzjBPhJ3W0bLNf62uk0FdGP+gyjz+9ecD96qqmulx0n0Ao+HpMnMegOWh/ps0T36ZeO/fFZxO8jz/pVdgnPA9L1tfFsb0cGQCDZ7LCvH4BY3BLCHZiPG5WGwup91nJZgwP0ehHD+veD3aroTK8/tRhvFD7a/ErxJY7BF/tbkv1qiqutyzREbTNPcPjqqqrtK7+wivWtvVLvFRM5aNeCT/IX4fOtp/vtu0QlXV88w4Pwi0fo8/cH5js5r5fV5mfj9nYvxx2q6q6lRN06o78TkOuB/DODd1+lhomnZfmOe/SJ5LfLnidSfgXT1/R/O3JVlJUpbYtoeQZCzj9CXjcPof5At+5gXcP9AQhYabEVz2Bo/n/qq1MvHTbsjnZBnMTIxiePf+NWOvw6MK0/y32646zGN9Qad3UtBl+e7jMGJ4FKN6yFV9MdtcXqdO+h0c64DHz6yeyeR0qYdLuF1YuL8fqqreh7E9N+PxIxYg+casWnI1gPZ7hbO5fdWE387Jtby1qqpWYfzIhlv1FMr+u1dt33C7AeM4B1p/2LGZJZGRsEwzrjbskJkEVZvH9UOMz4rrqsxwPscdnUdC0ZVjEbIonUs8ub6nAY93V8/f0fxtSRaSlCW/CowTguufSSl+qqPMf8ilGF+WiH5horHsEBKraoy+dyowSg7L8K6y7cjNGNU5fn9wVFUtMX8YXNsT6B9uR9PD5u/fdAf7OKQYNE1bYSYiZRilP+4S1lCFeqw7OA4DQ4k3HOY/+sUYJW2eP87B9k0dHX9e6wgzKTNjmWrGsgiMUjctjA5CO9h/rpgn0T6RbQi2/k7G1lFzg1CF3W+cR/uwClVVJ2qatjGcz3EY54OgMQSZHPRYhLmqiJ9LfEyF9ueXrp6/o/nbkowkKUtyHqU7rhPTRIx//26q0XDW1SDT9a8vIvX+0Vx2B7ZjnPRc69pIiG3JPEpPZtL+X2MeRvuQMozEz7MdRqAf9mDTuyyEfRxODC8As1Wj76eJhNEvWwSPtevHamDQucJkVhXNVlV1kUe1iue+8XesNxBcp46rmdBPMqvWZgOLVFW9V+vE3Qb8cMcTqDos2PrDjS0CDcy7yt82dvpzHGEdHotOLCvi5xKzFG4GZpWvx/gufactPP8nLGno3z1UADNc7UU86/LNE28JZuNn8wTb2S+81z/mMJbt+hGOZGPiUow2GwpGO6JwenAvw6i6XKFpWp3PUI3x4+3al3UY29TupG9WlQadHiSGkEofQtnHYcawCOM4zMbYByF1ThzJz5H5XldJh6d8P7OH61HMNj/m6w3+1mV+V1yNuP3y2OaO9lHAY2m2XVqM8ZkqMkd36fvgsf/atb/zPd4B1t/hNCuo/i+8ALOxu09y2KnPsR+xOBYhraML55KgzPcuN+N81Gd8p8/fEf5t6TakpCyxTTXr5/2p9vhn5mqg/gA+xfhm8X8jxr9hML6Yofyr3A7uq+SqMb58Xlc8hbpscz6AMvOxVNO0mb7zhanOXG+VGScY1ZehnJzb/WP0UYFRXeKqEi7DbBSO0TZpIEZVQKb52NH0DvdlIGEcv45icC3P1VD6AYK0O+xCHKF61FxWBcYPxkS820Z2iqZpi1VVfQCj8fVMM+5ZGG2QBmLsG9c/+xW+nxdz/92Hsd8q8fkhI4RjqZ7uqsVVtZ5nbl+FGWMkvg8zOX28K8x4y4A61bh61+/6O4rNKuYP/M1mfCs4fbFCKX5K/zv7OfYV7WOB8ccxnHWE9D0OwnWFLXjcesxczhTPP65dPX9H4ZzQLUhJWWJzdfvgb3D3PG8mZxvNcf6u1pyF8aVcjvFjFLB0wMML5jJd8w/CfwPUUJe9GOMHbBGR+TfVgLG9izyGNaqqbleDdGGgnr76zN9+cnGd5EvhdFUQp0tWXCeeslCmE/q+DKTDfRxCDJ5cPxzh/hB35nPkl1lCsxijbV8Fxj6J1An9UYyS44nmulZg/KC5ujMow/h37++H0XV3hw8xPmO+JbChHMsXMBKLMo/1LfZpiN6l74PP8V6DkUC6SlmCrT+U2GLO3Me5nO524gFO92E3SfN/JV9nP8e+onkswlpHmN9jfxZh/HHYzuluZqqB3ADV0F09f0fsnNBdKLoeqQtnhIgPZpuwNRglIZ5dgZxnjl8RgZK4pGWW8pRpPvcYFCKRyOdYJCIpKRPJKA+8286Z7WOqMf4VWt4RZpwrw+LqKiEiQD7HIuFImzKRjF7AaMfg6r+oGqMqcypGEXvIXQ90F2aV7kSMBt2et6USImHI51gkOknKRNIxG5jmYrRhcDVabsRIzibFwSX88cjVcB3MBvBWBiNEJ8nnWCQ0aVMmhBBCCBEHErqkTFXVHkA2sEvTtJNWxyOEEEII0VmJ3tA/G6hfu3ZtK6BHa9ixY0fUli2DHJdkGuSYxOcgxyX+hqQ5JpdfbgxWx5FYxyWgRE/KYqK5udnqEIQfclzijxyT+CTHJf7IMYlPVh+XhK6+FEIIIYSF5s+3OoKkIkmZEEIIITqnpMTqCJKKVF8KIYQQonM2bTIGERGWlJQpirJG1/WgHXgqipKHcW9H1z0bl+m6Ln3OCCGEEPHi7ruNx7fesjSMZBHTpExRlBKMXpZDKe9cruv6JPN9GzA6BIzp/Qo3vFpB1kdPMqptFztSszlw7o8474Zw7hMdP1zbkt22i10Jvi18vBzefoxzDmyFrNFwyT0wPkFvZWluC4m+LXJM4pMcl/iTbMdk90ZobYbfXpjY2xInYpqU6bpeDaAoyqJg8ymKMhFo8Hhfo5nQxcyGVys478N7zYBgxKmdjPjwXjZAwiUzybQtfLwcVn4fAAVgf637dcKdDDy2BUjcbZFjEp/kuMSfZDwmrceM14m8LXEkXhv652HcFsdTg6Ioebqu1/nOnJuby8mT3n3HdvVOBVkfPWk8uWI+XPgDePd38OZCRn34c97bsc4936n0QVz0/V+7X7/71CyUtuN+l9ln/FcZd9l0ALZu/DeH1v0+4Pq/8r2n6N2nn7HMv/wcpWG73/mUs87l/K//GICDe3ex7YUH281zzsFqv9syeuPDQG37hfbNgis9rqhZdT+cbPEfaMENMHKK8Xz3h7DxTwG3iWmPQlq68bzmSTi4zf98Z50Lk75rPD+yH978xelp9r/73Rbefgx69ITtb/pfZjxuU6BteeuXp09qW16O/216+zH/2/HaXPjc/K5cdCdkjYr/bdq6uuNtcYn3bQr0+fK3LfG+TcG+93mXe58j4nmbXn/Q/3a8NtdIbgKd9+Jxm1zf+8yzwXYWXHHj6WMiSVmnxWtSNjDA+MxQF1Bb6yfZCMOotl3GX5kLfwC9+hmPby4kg6NccPBl93wOzqC2drb79di9r9JP8d/PSfWWLHoMKQBgl/09pnosx9fGLT8ivW9/APrU/4vxJzf7ne/doweprZ0GQMOXdUwOskzfbbHpR+DDZ9vNdqJfNtvP/Lr79egP/0jqyWN+F7n3ZF8aWs8CwLZzHcP8LM9FO/sWTqX1BWD4phfpu8//LSib9jn4os/5AKQd3slIf8v02Rb9wFYa/vs6g7S/+F1mXG+T77Y07OBT8/M7JAG26ZwDW41//T7bQUuj+/P1ef/zODb0ZNxvk47S4ba4JMo2hbItCbNNfr7327ds9H+OiMNtOufovoCfr6ZNr3R83oujbRrt+t7bzoKUVK9j8mkXf3+t1NLS0uX8oSMFBQUBp8VrUtZA+wQsUKJGfX092dnZEQ1gR2o2I07tNP7JXPgDePcpAPYzgLoxd7rnS+2TwXkeO/j98fejt7X6XWbuOZPJN+ftn9bGe72UgOs/d9x4evbqDcCmC+7kvUO7/c7X94yR7gPcdMZg3jvQvs+YvC2/ZTCH2m3LAWUAWdc+1G7+nr0zvD801y6CANs0NLuIoWea8w7pCQP7Btwmdcx4ozQLQJkHh7/0O1/GwFwy8s1lNp8Jyv87PfHfi+HInnbbomSNZtDFt8LISX6XGZfbFGhbBuWdjrVfAmxT1mij6sJnO+h3BlxmVJufrV5pnLzjfJuUd34NTY6g2+IS99sU4PPlb1vifpuCfO9HjrvA+xwRz9uUkQ1Nu/wek4zL7wx83ovHbXrb/N6XjofCr3kdk2BJR7yrra21NH5LbkiuKMohXdcHBJk+EVjkeYWmv/eoqjoC4zZLEU/KvNpheY6ftDjh2mEl07a0a1viMv2ZxCsyT5ZtSZbtANmWeJUs25Is2wHJtS0eYpSUBSyRiZuSMrMLjAZd1xt1Xd+oKMpAn2nVsYznvBvK2ABJccWi77Zs18/irz1n8NPrZnf43rjj+rK//Rj6ga0oiXz1kse2JPQVZXJM4pMcl/iTjMfkuQXQtBNyesB10p6sq2JaUmaWgJUAi4DFwBqPKzKXm6+X+cwbsJ+yaJaUebK6ODOSjp9s47LFb7HH2cLTt0xi2tgzrA6p05LpuCQLOSbxSY5L/EmaY3L55bBvC9x83EjKivyUniUQq0vKYtqjv67rG3VdX6zruqLr+n2uhMycNtOVkPnMW20+SsexEdCrRyp3XJYHwNK1n3X5KlUhhBDdXJ8s49H+krVxJAG5zVI39I3zhzO4fy+2fOlkbe0+q8MRQgiRyPoMBFs2DCkA+aPfJZKUdUO901Ipu9QsLXtDSsuEEEJ0QUoP+PFmuHYJKIF7FRAdk6Ssm/r2BWeT1a8n2p7DbN9/xOpwhBBCJDJJxiIibq6+FLGV3jOVpd88l5GD+zHE1tvqcIQQQiSixx8//bz5EHz6Txh9DfQdZF1MCUxKyrqx4vwsSciEEEJ03oQJxgCwsgxevhM+fdXamBKYJGWCk22neOPTvdK2TAghRHiqq40BjPtnglyF2QWSlHVzuq4z4+n13P7cBtbXHbQ6HCGEEIlk4UJjADjnOqPRf/1/4Kj8nnSGJGXdnKIoTDlnCGD0WyaEEEJ0Sp+BkHsZ6G3w6T+sjiYhSVIm+O7kEfTv3YN36xp4T0rLhBBCdFbhV43HLVKF2RmSlAlsvdO4fXIuYPRbJoQQQnSKeh0oqVD3bzjWYHU0CUeSMgHA7ZNz6derB+u2HWTDDvkiCSGE6IS+gyDvcsi5AI7IHWPCJUmZACCjTxq3Fo8AYOkb26wNRgghRGKoqDAGT9+qgttXwZBzrIkpgUnnscLtexfn8rp9D1PHDEXXdRTpoVkIIUQwqtp+XGpa7ONIEpKUCbcBfXvyrx9fKsmYEEKI0LxqdhR7ww3e43UdvvjIeD5sYmxjSmBSfSm8SEImhBAiZI89Zgy+/vs3qLwC3no09jElMEnKRDtNza2Uv67xk6pNVocihBAiEY2cCkoKbH8TmhutjiZhSFIm2jlx8hSVb9ex8qPd2L9osjocIYQQiabfYBhxMZxqBe01q6NJGJKUiXYG9+/Fty84G4An5UpMIYQQnTHmJuNR7oUZMknKhF9ll+XRs0cKqzbv4dM9TqvDEUIIkWgKbjSrMN+QKswQSVIm/Bpq6803i3IA+I2UlgkhhPDnT38yBn/6DYGzJxtVmFtXxzauBCVJmQjojsvz6ZmawmuffMlnew9bHY4QQoh4k5NjDIGMuQn6DIITR2IXUwKTpEwEdGZGOjPPy0bX4Z+ffGl1OEIIIeJNVZUxBHLud+CerVD0/djFlMCk81gR1J1XjOS6cWdyUf4gq0MRQggRb373O+OxtNT/9LTesYslCUhJmQjqrMx0ikdmSaeyQgghOq+lCer/Y3UUcU+SMhGyrXsPs7ux2eowhBBCJJLmRigfDX+eAS1yNX8wkpSJkPxp/Q6u+vV/eKJ6q9WhCCGESCTpmXDWudB2HLa+bnU0cU2SMhGSi0cNJkWBlRt342g4ZnU4QgghEsmYrxqPW6Qj2WAkKRMhyc3qy1cnDOPkKZ2n3tpudThCCCHiwYoVxtCRMTcaj5+tgePSxVIgkpSJkN155UgUBVZ86JC2ZUIIISAryxg6YjsLci6UKswOSFImQpY/uB83jD+L1jadp6W0TAghxHPPGUMoCqUKsyMxT8oURclTFOVeRVFKzMfMIPNOVBRltqIoM8x582IZq2jvR2ZpWdUHDvY0tVgdjhBCCCuFk5S57oXZ4gRdj2ZUCcuKzmOX67o+CUBRlA1AJTAzwLwluq4vdr1QFKUCKIt+iCKQ0UP7c+NXzqJXjxSk6zIhhBAhyxhm9O7fb7DVkcStmCZliqJMBBpcr3Vdb1QUpSTIW8oURVmm67rcXj6OPF46QTqTFUIIET5JyIKKdfVlHuCbYDUEqZZcBNSbVZizgfv8zZSbm4uiKF6DiB7Zv0IIITpN1+HL/8IJ6V7JV6yrLwcGGO+3XZmu68vMNmeuKstq2id1ftXW1oYfXQAtLS0RXV6yqNl5lBftjcy/fCgD0mNfEy7HJf7IMYlPclziT7Ick+HHjMRqZxjbMqzm/2JzrGVX8S84nDMlWqF1SiyOS0FBQcBpsf4lbaB9AhYoUUNRlHvNNmWLzZKyNUC+73z19fVkZ2dHNFBPtbW1QXdid/XY+x+wZd9x/r2nBw9cG/v9I8cl/sgxiU9yXOJP0hyTt94CoKBPn9Dfc+gKcKwlu2kDXPWj6MTVSVYfl7CqL1VVvbKL66vzN1LX9Y2+48y2Zhs95lkGrDDbpYk4cNeUUQD87/rPOXjkuMXRCCGEiLk+fYwhHGNuMh63vi5VmD7CbVNWrarqQVVVf6eq6hXhrsxMvtwlY2ZbsmrP1x5dZDQA7RIwfwmcsMb47EyuUAfT3NrG79+ptzocIYQQsfbUU8YQjszhMGwStB6DbWuiE1eCCjcpGwA8gFGFuNZM0B5VVXVCGMuY5eqnDJgBzPKYtgi4GdzJV52rkb+iKPcCVWHGK6Lsf8zSsj/W7KDx2AmLoxFCCBFTL7xgDOFy3QvTLh3JegqrTZmmaU3AMmCZqqqZGP2LTQWWq6oKsAJ4WtO0zwMtw0y2XKVd1T7TZvq8DuGGWsJKE4cP4JJRWbz92QH+8E49P7lKtTokIYQQ8W7MTbDmIaMKs7UZ0tKtjigudLpLDE3TGoGDGNWMijlMAupVVX1dVdUREYlQxL05ZmnZy//9grZT0kuzEEKIDgw4G846F3r0hP2a1dHEjbCvvjQb+5dhVD0qwHJgpqZpH5nTMzGqIf8FjI5cqCJenTdiIE98YwIlBUNJTZE+zIQQQoTg5v+F/mdCaprVkcSNsJIyVVW3YXQAuxG4Q9O0St95NE1rVFW1Au+2YiLJ3TRhmNUhCCGESCSZw62OIO6EW1JWDfxK07QdHcy3HaOtmehmjh4/ibb3MBOHD7A6FCGEENFm9lPWJccPGzcpz5A/9+G2KfsQPzcEV1V1g6qq33O91jStSdO0tV0NTiSWL5uauWTxm9z+3AccOX7S6nCEEELEO/tLsDgfqn9mdSRxIdyk7F7gAz/jfwXc0fVwRCI7w9ab3Ky+NB5r5U/rA16AK4QQIlmUlxtDZ505HtqOg7YKWlsiF1eCCjcpy8ejl30P2/HT0avoXhRFcV+JWfl2HcdOSGmZEEIktX/8wxg6a2AenDEeThyG7W9ELq4EFW5SVgec62d8HiBdugsuGZXFhJxMGo6e4C/v7rQ6HCGEEPGu0OxIdot0JBtuUrYMeMbzFktmFxnLgKcjGZhITJ6lZRX/qaP5RJvFEQkhhIhrrt79tVVwsnvfRzmspEzTtMXAMxi3WGpTVbUNWANUaprWhUplkUwuVwczblgGB44c52/vS2mZEEKIIAblw9BxcNzZ7asww+7RX9O0+zBuKl6KcZ/KgZqm3R/pwETiUhSFu6aMYsyZNvIG97U6HCGEENGSnm4MXVV4k/H4xUddX1YCC7tHf3DfYknuSykCKikYQknBEBRFevgXQoiktWpVZJYz6TYYX9rtO5TtzG2WpmOUkmX6TtM07epIBCUSnyRjQgghQtY3y+oI4kJY1Zeqqs7DKCFTMHrsrweakN77RQCOhmPct+Jjnpe2ZUIIkXweecQYIkXXwfll5JaXYMJtU3Y/UKJp2s1AnaZpd5jPb8boLkMILx/vaqJqg4PfvLGNEydPWR2OEEKISFq71hgiobkRnjwPnroATp6IzDITTLhJ2QCMjmIBGlVV/Yr5/EOMxEwIL9eMPYNRQ/qxu7GZlRt3WR2OEEKIeJWeCak9oaUJ6v9tdTSWCDcp24jRqz/AC8BiVVVHAPdFMiiRPFJSFH505UgAfvvWNlrbpLRMCCFEAGPMqzC7aUey4SZl92EmZWafZSMxSs5mA49GNjSRLK4ffxZ5WX1xNDTz0ke7rQ5HCCFEvHJ1JPvpP6Gt1dpYLBBu57FrNU2r9HidD5ynaVqKdB4rAkn1LC17cxsnpbRMCCGSw6BBxhApQ86BwedA86FuWYUZVpcYqqrOBZZpmuZ0jdM0rXv39CZCcuNXzmLp2s/YcfAYH+w4xEX5EfwSd3NOp5N9+/bR2mr9v8rW1lZqa2utDkP4iJfjkpaWxpAhQ7DZbFaHIiLlxRcjv8wxX4V//wrsL8HIksgvP46F20/ZVcAh4PdRiEUksR6pKTw6fTwD+/ZEPaO/1eEkDafTyd69exk2bBjp6emW9w/X3NxMeiR69xYRFQ/HRdd1mpub2b3baMIgiZkIqNBMyna8Y3SR0Y36vQy3TdkijMb9Z0cjGJHcLsofJAlZhO3bt49hw4bRp08fyxMyIYJRFIU+ffowbNgw9u3bZ3U4IlIeeMAYImlIAfyfl+HO97pVQgbhl5TNxCgpq1NVdSM+fZNpmlYaqcBE8tJ1nQ2fH2LS8AGkpHSvL1yktba2Wl4CIkQ40tPT46KqXUTI+vXRWW7e5dFZbpwLt6RsIEa3GC9i9Oav+AxCdOjuqk3MfHo9q+17rA4lKUgJmUgk8nkVYTlxFNpOWh1FzIRVUmb23i9ElxSNGMjLm75g6drPmFZ4hpSWiaRkt9ux2Wzk5ORYHQorVqzgO9/5jtVhCBGe1/8vfPB7+ObfIP8Kq6OJiXBLyoTospnnZXOGrTef7jnMv7bstTocYQGn08ldd91FUVERRUVF3HXXXTidzo7fGIH1FhUVRX0ZTqeT8vLyuEjIAHbt2sXq1autDkOI8PToDSebYcvLVkcSM+F2iTE92HRN01Z2LRzRHfTqkcoPLs/np6/YWbr2M64uHCpVGt3MrbfeykUXXcTChQtpamri+eefx+FwUFhYaHVoETF//nxmzZpldRhud999N5deeinTpk2zOhSRbLKzo7fswq/C2+VQ+ypcWw6p4TaDTzzhbuGKAON18zG1C7GIbqS0KIffvrmNLV86WVu7j5IxQ60OScSIw+HA4eCabpMAACAASURBVHCwcqXxH85mszFv3jyveRYsWEBhYSGlpYl37ZDT6WTLli0UFxfHdL0Oh4PVq1dTWFiI3W6ntLTUq9uJa665hqqqqoTcpyKO/fnP0Vv20LEwMB8atsPn6yDvsuitK06E26N/iu+A0fj/RWBSVCIUSal3Wipllxm3UV36xmfout7BO0Q0vbxpN1f9+t/kP/gaV/3637y8KXq3w8rIyMDpdMakutIKq1atinlCBjBnzhxmzZpFcXExpaWlzJ8/32v6tGnTpApTJBZFMUrLoNvcC7PLbco0TWvEuCfmoq6HI7qTb50/nNysvlw6ajCtbZKUWeXlTbuZ8/wmtu49Qtspna17jzDn+U1RS8xsNhvFxcVMmTKFyspKHA6H1/S77rqLqqoqysvLKSkpoaamBoAlS5a426BVVlZ6vaeoqIjVq1dTUlJCUVGR+z0ANTU1FBUVUVJSQkVFhdf7/C2zqKgIu93O9OnTqaqq6nAZvtatWxfzali73U5GRob7tc1mY71PVwVjx4712i9CRMTddxtDtLhuUF77Kpxqi9564kQkK2hDuheCoih5wAyMrjUmAst0XW8MMv8Mz9e6rgeqQhUJJr1nKtU/uYxUufoy4kbc/8+A0375tXF864LhAPz1vZ08+PdPALhn6mhuvziXP7xTz2NrtjLn+U3cNGGY+33X/+ZtNu/2X7q141fXhRXfs88+y5IlS1i2bBnl5eVcffXVLF26FIClS5f6rb4cN24c8+bNw+l0MmXKFIqLi72Sn5qaGqqrq6mqqqKyspLi4mKcTie33XYbTzzxBNOmTXMnWcGWCUap08MPPxzSMnzt2rXLbwP/qqoqnE4nNpvNbxWi0+nscNmB3utwOOjf37tj5oyMDBwOhzsWV1WmKwYhImLTpugu/4zxMCAXDtXDFx9B9nnRXZ/Fwm3o7++MkQnkAQETKx/LdV2fBKAoygagEqNT2nYURbkXqNN1fYWiKJnAWgK3axMJSBKy+HH7xbn07dWD2y/O5bE1W6O+vnnz5jFv3jzsdjtz5syhsrIyaON4VyN1m83GRRddxObNm72SMtf04uJiysvLgdNViZ7TOlom4K4GDGUZvpxOp1epFRglcq5tDsRms3X64oCmpqaAsfiuo6mpSZIykTgUBW56EmxnwcA8q6OJunBLyvz9gjZhJEvBy/QBRVEmAg2u17quNyqKEqyE7QFd1we45iVAu7Xc3FxOnvTuXE7aKCWO4yfbeGHDLt6vb2DpNybIlZgREGrJ1bcuGM5zNfVs3XuEP7xTz+0X5/L7d+oBGD20n9e8//ifSyIeJ0BhYSGzZs1i9erVQZMSu91ORUUFu3btwuFwMG7cOK/p/kqnnE5n0G4pAi3TM/HqaBm+/CU9zzzzDHPnzqWqqoqxY8dGvHozIyODw4cPt4vD33zJ2pZPJLERF1sdQczEuvNYfyVqDYqi5Om67nXLJjNZqzOrLxsxqjpX+M4XSG1tbRdDPa2lpSWiyxPejrWeYvGqnRw+forJQ3W+cmZotw2S42LcZqm5ublLy5g1eTjzVm7hsTVbvUrIZk0eHvayXTedDld6ejptbW3u9548eZITJ064X+/atYvvfve7LFu2jIKCAh555BGvbdd1nZaWFpqbm2lpaXHH0bt3b3bs2OGez3NaoGXquk7Pnj3d7wm2DH9sNhv79+8nKyvLHXtBQQG33HKLex5/73U6nbz44otB91P//v2ZMWNGu/FDhgzx2n+ufZKXl0dzc7M7XofD4bVtVmltbe32391kOX8NP3YMgJ2x2pa2VkhNi9riY3FcCgoKAk4Lt/ry+0C+pmkP+Iz/AHha07Tfd7CIgQHGZ/oZlwdMdLUhM6s6PwTyfWesr68nO4p9pdTW1gbdiaLryg70pPxfW3lp23G+ceXEkN4jx8XYB1299+XM83Pp2bMnv31zG9v3HyV/cF/uvGKkV3uyUDU3N3cYT01NDeXl5cydO5exY8ficDhYunQpc+fOdb93wIAB7Nmzh9bWVpqamti/fz+ZmZlMnDgRp9PJ+++/T25urnt+RVHo3bs36enp9O7dG0VRSE9P56abbmLhwoV89NFHFBcX89JLL7mnBVqma7pr2cGW4c/w4cNpaWlxTx81ahQpKSkd7pf09HR++MMfhrW/XSZOnMiRI0fc63A4HBQXF7tfex6XUaNGdWodkZSWlibf3WQ5f02YAARPNCJi+5uw+n7IvRSuXRK11Vh9XMKtvrzPHHwtMsd3lJQ10D4BC5So1eFxw3OzqjPPX6maSHz/p3gEy/5Tx7t1DbxXd5AL8gZZHVK3ctOEYZ1KwjrD1WVDeXk5drudnJwcZs2a5dWx6eTJk5kzZw6vv/66u8G9zWajqKiIsWPHhlydaLPZeOKJJ5gzZw4ZGRlejeRDXWawZfgzZswY7Ha7VxXo3LlzqaysdFdbRqPLjEceecS9DrvdzsKFC72mezb6FyJili2LzXp6Z8D+T6G5EaYtgpTkvCGREk7bK1VVTwF5mqbt8Bl/LrBB07SgnceabcoW6bo+1WPcIVe7MZ9584A1uq7ne84LTHIlZaqqjgDq165dKyVlSeDx6q08Xv0Zk0cO4i/fv7DD+eW4xN8+CKWkLNnV1NRQWVnJs88+a3Uobs3NzbzyyivY7XYefvhhq8OJu8+tFWQfhEnX4Ynx0LgTblsFZ0enL8AYHZeADafDTTXrgHP9jM8D6jt6s67rG/EoGTMTr2rP1+ZVlpiJV6PHtEyMKzGllCxJ3VacS/9ePVi37SAbdjR0/AYh4lBxcTEOhyPuGtSvXr1aevMXkTd7tjFEm6LAGLMjWXvydiQbblK2DHhGVVX37dpVVb3SHP90iMuYpSjKvWZD/hmA5+VWiwDPiwlmKoqySFGU2cADBOg6QySHjD5p3Dp5BABVHziCzyxEHJs1a1aHnczGkqvhcrLcW1TEka1bjSEWXElZ7Stw6lRs1hlj4V59uVhV1UHAWlVVPes9l2iaVh7KMszSso3my2qfaTN9Xtfhvw2bSFK3T85lxKC+3DThLKtDEaLTSktLueuuu7Db7XGRCD3++OM8+eSTVochRNcMmwgZw6FpJ+x6H4Z33Mwl0YTdUk7TtPuAARglWjcDAzVNuz/SgYnuaUDfnnx9UjY9UpOzEafoPpYuXRoXCRlARUWFdBgrEp+iwJgbjee1r1obS5R0pUuMFz3Gh9olhhAh23e4heOtp8gZ2MfqUIQQQsSDSbcajfzzp1gdSVSEWxxxH/CBn/GLgDu6Ho4QhjVb9nLJojd5+B9brA5FCCFEIBMmuPsqi4msUXDOdZDWO3brjKFw+ynL53R7ME/bMXrcFyIiJuRkoihGcmb/oonCszI6fpMQQojYevxx69Z96lTS9VcW0y4xhAjV4P69+PYFZwPw5BvbLI5GCCFE3GhuhOW3wlMXJt1VmFZ0iSFESMouzaNnjxRWbd6Dtudwx28QIgHZ7XYcjuTpAqaqqsrqEEQs3XKLMcRSLxs43ocDGuz+MLbrjrKwkjJN0xYDz2B0idGmqmobsAaoDLVLDCFCNcTWm2+dPxyA37zxmcXRiHh32223UVRURElJiXsIJ0EoKiqKeYevTqeT8vLypLr90c6dO1m9erXVYYhY2bXLGGIpJQUKzKswtyRXR7Jd6RKjFOkSQ0RZ2WV59ExN4Z+ffMm2fVJalmwcDgc1NTURKymaO3cu1dXVVFdXs3LlSlavXk1lZWVElh0N8+fPZ9asWR3PmEDmzZvHQw89ZHUYItkVmh3JbnnZuAVTkuhUCzlN05o0TVuhadqLmqY1qao6QVXVRyMdnBBnZqRzc1E2l44anEzfO2GqrKx035Yo0mw2Gw8//DDLYnXDZA8LFizosJTO6XSyZcuWqNycPBCHw0FlZaX7/pyhlgwuWLDA6/WSJUtQVZWioiJuu+22dsfvmmuukWpMEV3Z50P/s6DJAbv9XX+YmMK9+tJNVdUJGKVlszFKznSMWyEJEVE/u6FQOpONto+Xw9uPwYGtkDUaLrkHxkf3rmZVVVVMmzYNp9NJRkZ0rq6Nt/tPelq1alVMEzKAOXPmsHLlSgDGjh3L/PnzWbp0adD32O12Vq1a5XUj83HjxqFpWsD3TJs2jcrKSrnXpoielBSjI9n3noYtf4fsSVZHFBFh/dK5SsRUVf0M+BCj37K1wExN01KjEaAQkpBF2cfLYeX3YX8t6G3G48rvG+OjqKqqiuLiYlatWhWVnu8dDgcPPfQQsz1uluxwOJg+fTpFRUXcdddd7ZK2qqoqSkpKKCoq8moXVVRU5C4NcjgcFBUVAUbS52rLVlRURFVVFXfddRdVVVWUl5dTUlJCTU2N3/jWrVsX0x7/7Xa7V/Jrs9lYv3590Pd0NqkdO3ZswO0WSeaii4zBCmOSrwqzw5IyjxKxGRj9lAGswOgGY4CmaU3RC0+I0zY5GnnyjW3Mv66AEVl9rQ4nvv0sSMnT9Y/DebcZzzc8C/+423h+xXy48Afw7u/gzYVGYuZZWlZxKXz53wDrC+80YLfbyc7Oxul0RrSR+4IFCygvL8fpdGKz2Zg7d65Xac306dN54oknKC4upqqqiltvvdVdcuRSXV1NTU0Nt912Gx988EHQ2xOtWrWKMWPG8OyzzwJGwlZaWsqCBQsoLCwMWlK0a9cuv9teVVXljt/f+51OZ4dVg/7e63A46N+/v9e4jIwMHA5HwGNQU1PDtGnT2o13OBysXr0am83GunXrKCsr89pPrueu7RBJ7FELWy7lXABTHwH1WuMWTEkgYFJmthFzJWKNwAvAHZqmrTWnN2BUWQoRE39593Oqa/cysG8ai2d8xepwks+FP4Be/YzHNxdGdVWbN29m8uTJAX/0O+vhhx+mtLQUp9PJlClTuOaaa9zTqqqquOiii9xVhqWlpVRWVnrdNNwVS3FxMcXFxSHFt379evcywkkw/VXbLlmyBDAaywdis9k6dXFAU5P/xDlQaVhNTU3A6tXS0lJ3spWRkcGcOXPcialnnE1NTZKUiehJSYHJd1kdRUQFKym7DzgEzNA0bWWQ+YSIiTuvGMmLG3excuNu/ufKUVaHE99CLbk67zZ4r8Kosnz3d2ZJ2VPGtMEF3vOW/Sdi4dntdvr37x80+egKm83GNddcQ0VFhXsdO3fubJc05eTk4HA4/FYjuqYFU1pays6dO7n11lsB3KVwofCXsDzzzDPMnTuXqqoqxo4dG9HqzYyMDA4f9r6COViilpGRETCh8hxfWFjot6oyIyMjrtv0iQj5+teNxxdfDD6fCEmwpOxmc1ihqup2jCrL5zVNc9VfSCmZiKkRWX356oRhrPxoN9f/5h0Ot7Qycsg+7rxiJDdNGGZ1eInrknuMqso3F3qXkF1yT9RW6XA4eOKJJ/yOr6mp8UpIXJ2rhlsSNXfuXKZMmeKuWhs+fDjr1q1rt75Ay3Q4HCElWPPmzWPevHmsXr2a8vLydtWhgfgmLa5t7KgUrLPVl4G201/i50qyXEmpa53FxcU4nU4eeuihDrfT4XBIKVl3cPCg1RHA+5VGU4yv/hbO8nfTocQRMCnTNG0FRiKGqqozMBK0j1RVPYRRlZmJJGYixs45sz98BE3NrQBs3XuEOc9vApDErLNc7cZidPWlq5TM9wfb6XS6q8yWLFnCuHHjyMnJIScnh2nTplFTUxO09MaXq7SsvLychx9+2P3ctY7KykpsNptXUlJTU0NpaSl2u53Nmze7q+Q8E6jnn3/ea/6MjAwKCwu94urfvz87d+7E6XTS1NTkNyHKycnxKqkKNeHsbPVlYWGh1/ocDgcXeTTQdjgc7v3rr8rWleQ5nU6vhK+mpoarr77a7zqTqVNcEcf2fwr77EaD/wRPykK6rM3sk+xmTdNSgDJgEKAAO1RV/Z3nbZeEiKYVHxo9R98zdTT2n1/NPVNHA/DbN+X+mF0yfibc+S78tMF4jGJ3GIWFhXzjG9+gpqYGu93uLpXZvHkzpaWl5OTksHTpUmpqavjkk0/cSVNxcTGbN28Oa12uqkBXqc1zzz1HeXk5RUVF1NTU8Nxzz3nNb7fbmT59OnPmzPGaVlpayvTp05k+fTqZmZle73nooYcoKipiwYIFzJ07F4DJkyfzwgsvMH369IBVoGPGjMFut7eL19WPWDSuXnzkkUfcy1+9ejULF54uGV2yZAmrVq3ymt/pdLo7362srHTvx5ycHKqqqqiqqmLdunVey4HgJZBCRJzrKkz7Swl/Faaid2EDzBK02UAJoMe6WwxVVUcA9WvXriU7Oztq66mtraWgoKDjGUXU5T/4Gm2ndOw/v5q+vXpw9PhJCn/6OqkpCtt/ea3V4cVcvH02m5ubSU9PD2lez5IZaN+wfMmSJaxfv54nnniCnJwcnE5nwPZficjVgatvA/loCOe4REJVVRV2u92rbzNP8fa5tULS7IPLLzce33rLuhhOtcFj58DRfUbb1zM7fyFYjI5LwEtFO915LLSr4vx6V5YlRCjyB/dl694j/OGdem6/OJffv1PvHi8Si29JSnFxsbtvsKamJsrKyigrK2P+/PlkZmaSk5OTVLckKi4uZsGCBUnZbcTq1avdpYYiyU2ZYnUEkJIKBTfAht8bpWVdSMqs1qWkzJOmaXLphYi6O68YyZznN/HYmq08tmar13iR+Py1Zeqox/lENmvWLK8rRJOBq0o2WUo0RQfi5T6nhV81krItL8GUBQnbb5l0lS4Syk0ThvHENyYwemg/UhQYNaQf3zw/h6sLz7A6NCHCVlpaisPhaNe2LJGVl5f7vbJWiKgaXgx9sqChDvZ8YnU0nRaxkjIhYuWmCcO4acIwamtreWpTC39730H2gD5SWiYSUrKVBMaijZyII64Omn0uEom51B5Q8jPonQFZiduPpZSUiYT2jSKjXdKTb2zjy6Zmi6MRQohuprnZGOLBxO8YNylPi91FLZEmSZlIaJNHZnHN2DNobm3j0dc+tTocIYQQotMkKRMJ78FrC+jVI4VX/vsFH+xosDocIYQQVvliE7z4fXjncasj6ZSwkjJVVacHGD9FVdUrIxOSEOHJGdiHOy7LB+CnL9tpO5XYnQcKIYTopOYG+GQ5bPpLQnYkG25J2fIg0xZ1JRAhuuKOy/IZlpnOli+d/OPjL6wORwghuofrrzeGeDHiUkgfaNwybl+t1dGELdyrLwN1/JEJTOxiLEJ0WnrPVH52YyF7nS1cP/4sq8MRQojuId46CU7tAQXXw8b/NfosGzrG6ojCElJSpqrqKYybj+uqqrb5mUUBPgxlWYqi5AEzgI0YidwyXdcbQ3hfha7rZaGsQ3RPU8cMtToEIYQQVhtzk5GU2V+CKx60OpqwhFp9ORW4GiP5uhm4ymfI1zStKMRlLdd1fbGu69XAMqCyozcoijLRXK8QIdnT1MLnB49aHYawiMPhcN94u7KyEqfTGdL7FixY4PV6yZIlqKpKUVERt912W8CbiwvRbV1++en7X8aL3MsgfQAc0GBfYl2VH1JJmaZpawFUVa3uyu2UzOTKfXmcruuNiqKUdPCezM6uT3RP73x2gNl/2sD47Az+NutClAS93YbovDlz5rBy5UoAxo4dy/z58zvspNVut7Nq1Sqvm2iPGzcOTdOiGqsQIsJS0+Cc6+CjPxtVmEPutzqikIXb0H+N5xWYqqpWqarapqrqVlVVJ4Tw/jzAt6qywazSDKRE1/WNYcYpurFxwzLo1SOFd+saeO2TPVaHI2LMbreTkZHhfm2z2Vi/fn3Q94RakiaESBDnfgcufxDGzrA6krCE29B/NlAGoKrqPIy2YVcBd2BUQ3ZUhTkwwHi/pWFmKVp1R0Hl5uZy8uRJr3F6Al4KKyIjo08a864+hwf//gm/+OcWrjxnCOk9U60OS/ioqqrC6XRis9koLS31muZ0Oqmqqgr6fn/vA6Pqsn///l7jMjIycDgc5OTk+F1WTU2N35uhOxwOVq9ejc1mY926dZSVlWGz2TraNCGE1YZfaAwJJtykLB+oM59PBVZomrZWVdVDwAchvL+B9gmY30TNrLZsCOUiAH9qayN3KWxLS0tElyciI9hxGd9PJ39gT7Y3tPDIi+/ynQmB/g8kttbWVpr93OKk59VXtxvXNn06bWVlcOwYPb/2tfbTb7mFtu98Bw4coOe3v91++qxZtM2YgbJrF2nf+57XtBOvvw4Yf4b8xePr8ceNjh3vvvtugHbvSUtL45ZbbulwOf7WtX//ftra2rym6brO/v37ycrKajf/u+++y7nnnktzc3O7+G+88UZ3Eta7d29+9KMfUVFR0WFc8SbU4xIrra2t3f6cmiy/K8OPHQNgZxJsC8TmuBQUFAScFm5S1ghkqKqaAZRglJSBcWVmKMlTnb+RAaonS8B9tSZApqIos4FqXde9llNfX092dnYIq++c2traoDtRWKOj47Ko7xnMeHo9K+xOfnD1ueQM7BPD6GKjtraW9HQ/93lLad8yIbVnT0hPNzpUDDY9PT349N692013xdDc3Ow/Hh/PPfccc+fO5ZVXXmHs2LEUFhZ2+J5QDR48mGPHjnnF4XQ66dWrV7vYnE4nQ4YMYehQ48pdRVG85vF8PnHiRN57772Qti/ehHpcYiUtLa3bn1OT5nfl1luB4ImGZY4fgfcrYK8dZvwhpLdYfVzCTcp+BXwEHALqNE1baY4vA9Z29GZd1zcqiuIusjATrmqf1w26rjfqur7C872KoqDr+rIw4xXd2HkjBnLThLN4edMX/PK1Wn53yySrQ4qdt94KPK1Pn+DTs7KCT8/JCT69Aw6Hg8LCQmbNmhVwnq5UXwaqovSX+NXU1Lhj8lxvcXExTqeThx56yH3BgBDCjx/+0OoIAkvtCeuegJYmuPwByBpldUQdCisp0zRtsaqqG4EMvNt6rcHodywUsxRFuZfT/ZR5npkXmctyJ19mNeZs8/m9wArfkjIhAnngmgIajp7gzitGWh2KMAVKmjzZbLagSVswhYWFNDU1uV87HA4uuugir9cZGRnYbDa/7chciZ7T6fRK+mpqarjaT7WwEN2aWX1JnzisiejRE9Tr4L9/Nfosu2ye1RF1KNySMjRNa9fwPpxuMsyqSlcCV+0zbaaf+RuBxeYgRFjOyOjNn753gdVhCB9z586lsrLSXXpVXFwc0eU/8sgj7uXb7XYWLlzonrZkyRImT57slXB5lsxVVlYybdo0cnJyyMnJcY/fuXOn13KEEMC11xqPXSg9j6oxNxlJ2ZaXky8pC3RDcheP6kwh4tIXjc2clRk/bWu6q+Li4ognYp4KCwsDJnz++itzlcz5ls5FM0YhRAzkXwG9bLD3Ezi4HQblWx1RUOGWlK0IMN7V/4T0OyDiUvOJNn701428v6OBN+deTla/XlaHJIQQItp69AL1Wvj4ebD/HS6Ns3t1+gir81hN01J8B4wuLV4EulErapFoeqel0KbrHG45Sfnr0kO7EEJ0G4VfNR63vGRtHCEIt0f/djRNawTuw2ikL0RcUhSFh64fQ1qqQtUGBx/v6lT3d0IIIRJN/pUw7ma4+CdGl0BxrMtJmYeg97AUwmr5g/tx++RcdB1+9oqdU6fi+8sphBBx79Zb3X2Vxa0eveDrlTB2OsT5vZDDbejvr+OgTPzf01KIuPOjK0ey8qPdbNzZyEubdjN9YvQ6HRZCiKQX7wlZggm3pEzxMzRhdBwrJWUi7vXvncb9084B4NFVn3Lk+MkO3hH/5D6vIpHI5zXJHDhgDIlg21r4+x3QUG91JAGF23nszdEKRIhY+dq5w6ja4ODc4ZnEd0F2x9LS0mhubqZPPHbcKIQfzc3NpKWlWR2GiJQZ5t0W47WfMk+b/gqbV8Dgc+Diu62Oxq9ItikTIiGkpCj8bdaFPHBNAX17hd1/clwZMmQIu3fv5tixY1ICIeKaruscO3aM3bt3M2TIEKvDEd1RAlyFGfYvkqqq38e42tJ1o/CNwC81Tft7JAMTIppSU06XkbW0ttE7LTG72LPZbAB88cUXtLa2WhwNtLa2SilIHIqX45KWlsbQoUPdn1shYmpkCfTsB198BId2wIARVkfUTrgN/X8F3IvRiexioAG4ClihqupsTdN+H/kQhYie1Zu/5OevbuEXXxvLlecMtTqcTrHZbHHzI1dbW0tBQYHVYQgfclyEANLSYfTVsPlF47ZLk+dYHVE74VZfzgbu1TTtZk3TKjVNe1HTtDKgFLg/8uEJEV27DjXzZVMLD7+6heMn26wORwghRDSNMasw7fFZhRluUtaAn1staZq2AhgUkYiEiKHvFo9g5JB+7Dh4jD+8s8PqcIQQIrH84AfGkChGlkBaH/hiIxz63Opo2gk3KVsBTPEdqarqlcCaiEQkRAylpabw0xvGAPCbNz5jr7PF4oiEECKBlJYaQ6Lo2QfOnwWX3gs9elsdTTsB25QF6Ch2ADBPVVXfrjFKMBr8C5FwLhk1mKsLh/K6fS+PvlbL49841+qQhBAiMTgcxmNOjrVxhGPqw1ZHEFCwkjJ/HcU2Ytx8vMlneBGI397YhOjA/OvG0LNHCi9t+oINOxqsDkcIIRLDd75jDCIiApaUSUexojvJGdiHOy7N46VNX3Ci7ZTV4QghhIimpt1GR7IZ2TD261ZH49aZfspswCJO31ZpDXCfpmmHIxmYELH2wytG8sMrRiZsn2VCCCFC5HgP1iyA7PPjKikLq6G/qqoZwA6gCKPK8kXgfGCHqqojIh2cELHUOy3VKyGTHvKFECJJjb4aeqTDrvehaZfV0biFW1K2DHhB07Q7PEeqqroceBqYFqnAhLBK47ET/HrNVnqkpvDQ9WOsDkcIIUSk9ewLo6ZC7Suw5RW46IdWRwSEn5RNBSb6Gf8o0iWGSBJfNrXwp3c/R1EUbj4vB/WM/laHJIQQ8emee6yOoPMKv2omZS/FTVLWmc5jc/2MH2BOEyLhSsY0SAAAF2pJREFUFZxp45YLz6btlM7PX7VLNaYQQgRyww3GkIhGXW30VeZ4D5xfWB0NEH5StgxYpqrq2a4RqqpOwKi6bNfTvxCJ6idTR5PZJ42a7QdZvXmP1eEIIUR80jRjSES9+hk9/INRhRkHwkrKNE1bDLwB1KuqelBV1YPAh8BGTdMeiEaAQlghs09P5l6lArDwn7U0n5D7YgohRDtlZcaQqMbfDOdcD0POsToSIPySMswbkI/EuAH5/cB5mqYl0D0WhAjNN88fTsGZNnY3NlPxn+1WhyOEECLSxtwE3/gL5F1udSRAJ/opA9A0rQ6o8xynquoITdN2RCIoIeJBaorCz28s5OaK9TgamtF1HUVRrA5LCCFEkgqalJkdxVZiXHG5HSjTNO1zP/M8CMwDpNdNkVTOzx3Iv358KaOHyhWYQgiRlNpaoe4t2PMJZF1jaSgBqy89OorNx+gkdjBQ59lJrKqqc4FDwGzgjvZLESLxSUImhBBJrPUY/LUU1v6cc14oht9eCB8vtySUYCVli4Bqj3tg3q+qagWwSFXVNcBiIBPjFktLohynEJb78PMG/vzuTpbMGE+P1LCbYwohRPKZP9/qCLpu679ANy7mUvRTsL8WVn7fmDZ+ZkxDCfbLMgX4pc+4XwEzgQqM7jEGSEImuoOTbaf4yQv/5e8f7eYv7+20OhwhhIgPJSXGkMjefsx4vGI+PLDbePQcH0PBkrI8TdM2eY7QNK3efDpJ07T7NU1ril5oQsSPHqkpzL/OuOXSY//SaDh6wuKIhBAiDmzaZAyJ7MBW4/HCHxh9l134A+/xMdTZOpjGzq5QUZQ8RVHuVRSlxHzMDDLvREVRZpvzLVcUJa+z6xWiq0oKhnDp6ME4W06y5PUE7SxRCCEi6e67jSGRZY02Ht/9HRw/Au8+5T0+hoImZaqqBmrhnNGFdS7XdX2xruvVGFWglf5mMpO183RdX6br+mKMKlO5v6awjKIoLLh+DD1SFJ7/YCebd0tBsRBCJLxLzPt3vrkQHh0Gb/7Ce3wMBWvorwCNqqr6m7bR33hN04J2iaEoykQ87pGp63qjoiiBKqPzgPswEjeADUCeoiiZuq53uqROiK4YOaQftxaP4Jl36vnpK3ZW3HGR9F0mhBCJzNWY/+3H0A9sRckabSRkMW7kD8GTsmjcNyGP9lWfDYqi5Om67tUZra7rGxVFmeQx6jyg0V9Clpuby8mTJ73GyU2kRbTcVTKKlzbtZuvew9QfOEre4H5WhySEEKIrxs+E8TP5tLaWgoICy8JQYpm8KIoyG5iq6/pMj3HbgZm6rm/s4L3LgSpd1903Pjf7TKuvq6trl5Rt2bIlYnG3tLTQu3fviC1PRIaVx2Xz3hayM9LI7C39JXuS70p8kuMSf5LlmAz/7ncB2PnHP1ocSWTE4rgUFBQErF7p1G2WuqABo28zTwM7epOZzHklZJ7q6+vJzs6OQHj+1VqcOQv/rDwu8nHwT74r8UmOS/xJmmPy+OMAybEtWH9cYp2U1fkbGayUzGxzVmdeGCBEXDlx8hR/rNlByZih5Gb1tTocIYSIreJiqyNIKjHtltxMvtwlY2YXF9Werz27yHBdGOBKyBRFmRHDcIXo0P9bs5VfvFbLwn9ErrpcCCESRk2NMYiIsOJeMbNc/ZQBM4BZHtMWATeDO2FbC3yoKIquKIpuThcibtx+8Qj69erB2k/38ean+6wORwghYuvBB41BRESsqy9dpWWu6spqn2kzPZ7XAQNiGJoQYRvSvzdzpoziF6/V8vA/tjB5ZBY9e8h9MYUQQoRPfj2E6KLvFo8gb3Bf6g8c5dl19R2/QQghhPBDkjIhuqhnjxQWXG/cF3Pp2s/Y52yxOCIhhBCJSJIyISLgcnUIJQVDOXqijb++v9PqcIQQQiSgmLcpEyJZPXR9AdPGnsH0c4dZHYoQQsSG2U+ZiAxJyoSIkLMH9eXsQdJXmRCiG5kwweoIkopUXwoRBY6GY7zz2QGrwxBCiOiqrjYGERFSUiZEhGl7DnPDk+/Qv1cP3ph7ORnpaVaHJIQQ0bFwofFYUmJtHElCSsqEiLDRQ/vxlewMDh49wRPVn1kdjhBCiAQhSZkQEaYoCj+7sZAUBf64fgef7T1sdUhCCCESgCRlQkRB4VkZfPP84bSd0vnZq3Z0Xbc6JCGEEHFOkjIhouSeq1Qy0tNYt+0gr9v3Wh2OEEKIOCdJmRBRMrBvT+65ajQAi1Z/yqlTUlomhEgyFRXGICJCrr4UIoq+df5wtD2HubV4BCkpitXhCCFEZKmq1REkFUnKhIiiHqkp/OJr46wOQwghouPVV43HG26wNo4kIdWXQsSIruts3HnI6jCEECJyHnvMGERESFImRAzous6s/93A9KdqWL/9oNXhCCGEiEOSlAkRA4qiMD47E4Cfv2rnZNspiyMSQggRbyQpEyJGZl+aR/aAdD7dc5i/vb/T6nCEEELEGUnKhIiR3mmpzL9uDADl/9rKoaMnLI5ICCFEPJGkTIgYurpwKBePzKKpuZXH1mhWhyOEEF3zpz8Zg4gIScqEiCFFUfjpDWNITVH463s72fKF0+qQhBCi83JyjEFEhPRTJkSMjRran7JL80hLTSE3q6/V4QghROdVVRmPpaXWxpEkJCkTwgL3TjsHgJc37ea3b25j+/6j5A/uy51XjOSmCcMsjk4IIUL0u98Zj5KURYQkZUJY5OVNu5nz/Cb36617j7hfS2ImhBDdj7QpE8Iiv31zGwD3TB2N/edXc89U4+blC/9ZyzufHWBPUwu6LjcxF0KI7kKSMiEssn3/UQBuvziXvr16cPvFuQDsP3ycW37/Hhc+upbt+4+4539L28e/7Huo239EOp8VQogkJNWXQlgkf3Bftu49wh/eqef2i3P5/Tv1AGSk92D00P7sOHiMswedvhDgt29u44Mdxr0ze6amMCKrDyOH9GPk4H5cmD+I4vwsS7ZDCCFEZEhSJoRF7rxiJHOe38Rja7by2Jqt7vEP3zTWb5uy4vws0nv2YPu+I+xubGbr3iNs3WuUpDU1t7qTss/2HuYXr9UycnA/I2kzh8w+PWOzYUKI7mPFCqsjSCqSlAlhEVfiFerVlz8225wBHD1+krr9R9m2/zDb9h2haMRA97QtXzp5S9vPW9p+r/dn9etJ/uB+PPXtiQzq1wsAZ0sr/Xv1QFGUSG+eEKI7yJIS+kiKeVKmKEoeMAPYCEwElum63tjVeYVIRDdNGNapKy379urBuOwMxmVntJt2Uf4gnr5lItv2HWH7/qPm4xEOHDmBs7mRjPQ097y3/uF9tD2HyTerQfM9StbOHtiHHqnS7FQIEcRzzxmPt95qZRRJw4qSsuW6rk8CUBRlA1AJzIzAvEIIYEj/3kwbe6bXuFOndL50tvBFY7NXonXoWCtHT7Tx8a4mPt7V5PWe2yfnsuAG416dXzY1815dAyOH9CNvcF/69PQ+dbj6W9u27wgjh+xL6P7WkqnvODku8SfZjsmIhY/T3HqKBYdyE35b4uG4xDQpUxRlItDgeq3reqOiKCVdnVcIEVxKisKwzHSGZaZ7jX9z7uU0HD3Btn1H3CVqrucjh/Rzz/du3UF+XPVf9+thmemMHNKP/MH9OHL8JC9scLinJXJ/a8nUd5xsS/xJlu2A09vy/Ik2IDm2xcXKbVFi2Q+SoigzgFJd12d6jNsOTNV1vS7ceVVVHQHU19XVcfLkSa91RXK7amtrKSgoiNjyRGTIcYkuXdfdbc3e+ewAf373c7btP8KOA0c5eer090sBdIz+1m6/OJc/vFPPY2u2kqLgVVUK8P1L8rjzipEArNt2gP/f3v3FxnHUcQD/TpykUVIRJwVVkfJALlKHN9DZfamoiPAZIQRv5xSJ58aoQqoKyFaeKiEh40rlj6pG2CkPvCC19jORsA2hjXhpfOLNTIivFQ00DfRypWli7NjDw8w4e+fdu93z7t7c3fcjWfadd/dmdm52fzszO/uD31UiP//3Lz6LU8dNEPnDN/+KP6k7ocs9c/bzeP17RQBA7bNNjL16NXKbr57/Mr7+pScBAJffruLS1ZsN///kwRZ29N68HD54AMcOD4Vu09c8ReXlqScfxx9e+hq++cu38dF/N3oiTzNX1kLzcmhI4O8//dbuer7n6WdX/oYbH91rWVdOHD2MP/74nPd5ct+va1d+glPHj+DSy29E1nvf89SurmQgchBv3t2XJyPeH97nsnusra3FSlAcGxsbqW6P0sFyyc8TAF4cPQrgKB7uaNz+dAv/+GQLH3yyid9WzDQdwfnWXl26gR1tukeDPvjXR1hbM+9V/3l/z/+Dbty4ifrj5hD14X/uRi57++P67vegvrHdcpvV9z/AKV2zaYneZnNeNh/uYPNh+NxwvuepOS8379zD2toaPv70Pu7e3+6JPLlrgOa8bG3rhmOA73m6eedeaD6CdWV7e7tn8gQAp44fwdAB0bLe90qeoupK2lo1JnSjpWxSaz0eeO8ugDGtdSXpsq6lbGVlBadPn84s3WyR8RPLxQ/f+MWfG67+f3PtPfx86QbOfuEYFr7/TMOyRw4d2B2PtvlwB/f+9zBskwDM1fbQAXNB+enGFra2w49VB4cEPnfEXJnv7GjUH0QfxI89NoTHDpoWrweb23iw1XhimPj1X7D+789i5cX3PEXlxV391+9vYifi8O9bnr792juR5bLyo3O76/mep++8dq1tXREAThx7NH2Nr3ly3y/XUvb6y29E1hXf89SurmQgsqUs76CsCGC2OdDSWp/oZFkGZYON5eKH5vEYzq+++5WeH1viMC/d1S956Zd8AI/ycmTLdENuHDoCoLfz0izDvPjRfam1rgghdrsl7ZQXy02va1rrertlicgPwfnW3A0CvXoXVtK543zGcvFPP5fJUz1aJoBf5ZJrSxmw2wJWQsjcY0KIBQBLWuv5dssC+bSUuYHOfDC0X1gu/mGZ+Inl4p++KpNLl8zvF17objpSkGO5+NF9mTYGZYOL5eIflomfWC7+6asyOXfO/L56tZupSIUPQRmn6yYiIiLyAIMyIiIiIg8wKCMiIiLyQDeefZmmIQC4fft2Zh9w8KDZRbdu3crsMyg5lot/WCZ+Yrn4p6/KxD1Npw/ykle5jI2NfRHALaXUnknYen2g/1cBvNPtdBARERElcEYp9X7zm73eUvYugGcBfAgg/JkNRERERH4JbY7r6ZYyIiIion7Bgf5EREREHmBQRt4SQhSEEFNCiJL9PRxjnaU80kZE1E4nxzC73lzWaSM/sfvSss/WLCPikU6dLkudE0Ksaq1H7N/DAC5rrScili0BKACY01pHzpZM+5ewrhQBjAIYBvA0gGmtdTWvtA6KDsrkJEyZjAOYZZlkI8kxLLBOEcCK1vpEHmkcNAnryiyAKQB1ANcBTGZdV3p9oH+aFgKV5zqAywCiKk+SZakD9sBUc6+11nUbeIXSWi/b9WZzSN6gi/X9tyeh0cCzbEsAlgCczTGtgyLJMWkFwBlbp04CmIMJzihFSY9hdp1YLWm0L0nqyrt5X+Sz+xLhlQfmQej7Wpb2pQBzdRJUs1c51CUJv/8FANOB19cBFHjiSVcHx6QzgZaBGkyrGaWvk2NYSWtdyTBNA60Xzt8MyowklYfBQj6iThQ8oXdX7O+/PbmMBN4aBVBnV3/qEh2Tmvb/JBoDZ0pPomOYbUVbzi45hOTn74IQomzHBM7mcUHJ7ksjSeVhsJCPGvbuU17Rd1+i739IAPB86imixMekwLiaJdf1T6mLfQyzJ/saL1gyl7Su7I43E0LUACwg465+tpQZSQIABgv5CB1Myab9ruvo+y+EuADgTa31YiapGmyJy0RrXdVavwKgzjuWM5PkGFbCo1aZMoBhIcQF9sCkLlFdCQbJttwy7+pkS5mRpPIwWMiB1rpiByED2L2yX256zSvL/CX+/ttumSpbZDITu0xcC5kNyADgLQBzQogC78BMV5JjWPPFihAC7gYZSlWSulKEuVt2JGSVzLClDLsF0rLyuL7kdstSqp53c/zAdLUEu75mAZx3L4QQRSHEFMwV5my7u5yoM0nqin1dhDnxuLtjyzkmdyAkLJMCgCcCqxdgxvkxIMtG7GMYYLox7XEMdj22lKUoYV2pwtyZ7P5XApB5Sz/nKbPsyaOEkLlLhBALMGMv5tstS9Tv4tYVe8BbRWN3QVVrzSkxUpbw+FXGoxPTOIAZtvTToEhYV9z8l4CZymcm63M9gzIiIiIiD7D7koiIiMgDDMqIiIiIPMCgjIiIiMgDDMqIiIiIPMB5yogGlJTyAgK3fFtVAItKKe8evSOlXAAApVTUw4N7Qov9vgxgWim177u7pJQawLhSitP1EPUQtpQR0QmllFBKCZhHIZWllKvdTlSepJSrUspcZ7YP7PMTMPt9FMB7Uko+so1oQDEoIxpwwZYZ27IyAqBoW3QGxRzMc+1yp5SqK6WWlVJu5vCL3UgHEXUfgzIiamCDtGXs48G7UsqS7ULzSlS6lFLzSikfHmuzDDOhpTd8LUuifsSgjIjC1LH3wb2UPTfTOBENIA70J6IwJQAz7oWUchaA686cdzcC2MH3NZiuv2kASzAtbCX7f9fCMqGUWgwbrG+XmVBKLUZtUynlHqA9bNPiHhU0H7wpQUpZgnmmYBEmsJxx69oxY0nSNQUz1uskTAvW866r1y0P4N3AMjOBdCYSSPf1pvzE2V/DAC7bfVLB3psIgtsqwezbRbt8wY5rCy3jVvusk3wSUWsMyohol5SyAHNSrwWCmQWYIMcFBnNSyvVAd18J5sHK0wDeUkq9IqUsA1hwJ/wONGyz6f0FmMCvDGC2KS0Fu07V/r0kpawqpRaVUuNx0yWlnLOfNWm3NQ0zCP9MYAxeGSYoG7fLzUopF5VSsR7uHdIlOK+UmoyzbpMFmJsEJmCCsj3bsPkpwIwXLMMEgCMweWtVxrH3GRHtH4MyogEnpbxr/3TdlYswJ2wXpJUBjCilKva9aZjB6MFA6GxTMLLfaR3CtgkAy4EA7BUp5dMwAdM8YMaGBZatSikXATxn8xQrXTbPF5o+f1JKOQoTzLigpx4IXGcATNl0xwrKYB5w7BTtZ6zDtETF6sK0aS2hcfqLadvKF3QepqWvCrPfLgIYVUpVYpRxpg9gJqJHGJQR0RhsIBEyR1bJ/l6VUkatX4nbOpRA3G0uwQQUu2xA8hxMkFmA6XpMogQTcO0JCJs+67r7QylVt/sn9ji8pu1XASzaFq0VmGky4ijabbXLYy3ktXsvThkTUQ4YlBFRtc2EpXWlVKsgIe2ALMk2G4IN29JUhWkVqgS67bKQRQvSHIALUspi3NaymBYBXJRSVmACOTdOzmlXxkSUA959SUStLMMMrk9zmobd1qQUJkodh71b0aaxAGAyENCc7GCb1xGe5xKSt7ol5VqtgkFpq/3l8l5qsQxgArEagHWYLtixQCCeRRkTUQfYUkZEkZRSVSnlPIAFKaUb9F6ECXxazWPmBpCXYVuUbBdbDWZiWtd6NZswSSUpZUkptWy3fQF2cLptGQPMmCp3B2YZjYFUVLqCea7YsWguzzWY8VUFmK7e1Nn9MQkzLi34qKWW+8uWzzLMwHyX1rB9WrDv7/lfjDJuu8+IKB1sKSOiluwdgXP2Zx0mQAmddiGwThWmFWcBjYHAHEzr1br9e2bv2i1VYAbE37XbnWyanmEaZlD7KkwrWnPAFZWu5vRPwHT5uTFewzAD4VPrspRSavdj0+sG7Aen1YizvyZgAieXp7CycVNlLNmfVSnlugv2WpVx3H1GRPsntOZEzURE/cp2bS4hcDepDcbcHGgjLVYnohyxpYyIqL8V0XQ3q/17DnxqA5FXGJQREfW3RZhxaVPuJgDbejaNR/O3EZEH2H1JRNTnbHflLMy4tWHYMWaePISdiCwGZUREREQeYPclERERkQcYlBERERF5gEEZERERkQcYlBERERF5gEEZERERkQcYlBERERF54P8zYHE71NaGCQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "device = gu.get_device(0)\n",
    "accs = {}\n",
    "epsilons = np.linspace(0, 0.5, 11)\n",
    "\n",
    "models = {\n",
    "    r'Standard': std_model, \n",
    "    r'Robust': linf_model\n",
    "}\n",
    "\n",
    "labels = {\n",
    "    'Standard': r'Standard ($\\epsilon=0$)',\n",
    "    'Robust': r'$\\ell_{\\infty}$ Robust ($\\epsilon=0.45$)'\n",
    "}\n",
    "\n",
    "pgd_attack = lambda eps: au.Linf_PGD_Attack(eps, 0.25*eps, 8, 'untargeted', \n",
    "                                            rand_eps=0., num_classes=2, \n",
    "                                            bounds=bounds, device=device)\n",
    "\n",
    "for model_name, model in models.items():\n",
    "    model = model.to(device)\n",
    "    accs[model_name] = mu.evaluate_robustness([model], test_data, pgd_attack, \n",
    "                                              epsilons, device, print_info=False)[0]\n",
    "    model = model.cpu()\n",
    "    del model\n",
    "    \n",
    "fig, ax = plt.subplots(1,1,figsize=(10,4))\n",
    "for model_name, model_accs in accs.items():\n",
    "    x, y = map(np.array, zip(*sorted(model_accs.items())))\n",
    "    ax.plot(x, y, marker='o', ms=5, mfc='w', mew=2, lw=2, ls='--', label=labels[model_name])\n",
    "ax.axvline(0.45, color='red', ls='--', label=r'$\\epsilon=0.45$')\n",
    "utils.update_ax(ax, r'Evaluating Accuracy and Robustness | Synthetic Data', \n",
    "                r'Perturbation Budget', r'Robust Accuracy', ticks_fs=14, label_fs=18, title_fs=20)\n",
    "ax.grid()\n",
    "ax.legend(loc='best', ncol=1, fontsize=14, frameon=True, fancybox=True)\n",
    "\n",
    "ax.set_ylim(-0.05,1.05)\n",
    "\n",
    "fig.savefig('plots/synthetic_data_robustness.pdf', dpi=fig.dpi, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown above, adversarially training FCNs with $\\ell_{\\infty}$ perturbation budget of $\\epsilon=0.45$ confers adversarial robustness at test time that is significantly more than that of standard FCNs ($\\epsilon=0$). Next, we evaluate the input gradients of these two models to illustrate FIG and RF-FIG on the synthetic dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating Input Gradients\n",
    "In this section, we evaluate whether input gradients of standard and adversarially \n",
    "robust models that are trained on synthetic data highlight instance-specific\n",
    "features that are relevant and _necessary_ for prediction in this dataset.\n",
    "We first evaluate whether top-$k$ and bottom-$k$ gradient attributions higlight\n",
    "the \"signal\" coordinate $j$ for the data point $(y \\cdot e_j, y)$. We then provide \n",
    "visualizations of the input gradients of standard and robust models, which\n",
    "further substantiate FIG and RF-FIG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:33:58.021882Z",
     "start_time": "2021-02-24T13:33:57.971177Z"
    }
   },
   "outputs": [],
   "source": [
    "# compute loss gradients w.r.t. input \n",
    "test_dataset = dataset['obj'][1]\n",
    "device = gu.get_device(0)\n",
    "\n",
    "grads = {}\n",
    "for model_name, model in models.items():\n",
    "    model = model.to(device)\n",
    "    grads[model_name] = test_dataset.get_input_gradients(model, device)\n",
    "    model = model.cpu()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:33:59.485979Z",
     "start_time": "2021-02-24T13:33:59.258301Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pr[bottom-most gradient attribution higlights signal]\n",
      "Standard Model: 0.562\n",
      "Robust Model: 0.000\n"
     ]
    }
   ],
   "source": [
    "# FIG: bottom-most input gradient coordinate highlights signal coordinate of data point in standard models\n",
    "print (\"Pr[bottom-most gradient attribution higlights signal]\")\n",
    "\n",
    "for model_name, G in grads.items():\n",
    "    tracker = []\n",
    "    for x, g in zip(test_dataset.X, G):\n",
    "        signal_coord = np.argsort(np.abs(x))[-1]\n",
    "        grad_coord = np.argsort(np.abs(g))[0]\n",
    "        tracker.append(float(signal_coord==grad_coord))\n",
    "    print (\"{} Model: {:.3f}\".format(model_name, np.mean(tracker)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:34:00.824125Z",
     "start_time": "2021-02-24T13:34:00.591379Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pr[top-most gradient attribution higlights signal]\n",
      "Standard Model: 0.041\n",
      "Robust Model: 1.000\n"
     ]
    }
   ],
   "source": [
    "# RG-FIG: top-most input gradient coordinate highlights signal coordinate of data point in robust models\n",
    "print (\"Pr[top-most gradient attribution higlights signal]\")\n",
    "\n",
    "for model_name, G in grads.items():\n",
    "    tracker = []\n",
    "    for x, g in zip(test_dataset.X, G):\n",
    "        signal_coord = np.argsort(np.abs(x))[-1]\n",
    "        grad_coord = np.argsort(np.abs(g))[-1]\n",
    "        tracker.append(float(signal_coord==grad_coord))\n",
    "    print (\"{} Model: {:.3f}\".format(model_name, np.mean(tracker)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, the attribution quality, $AQ(\\mathcal{A}_{\\mathcal{G}}, 1)$ (i.e., at level $k=1$) \n",
    "of standard and robust FCNs is $-0.521$ and $1.000$ respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-24T13:47:24.168583Z",
     "start_time": "2021-02-24T13:47:22.855685Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABBAAAAF8CAYAAAB2a0h2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdf3CdV33v+4/ThsQJJoqdqRMoJJZpv7lwMwmy057+gPySAqRNKESyyQ3QdkqklGYODBOs+DDlnkuH48jNYeBM2ollmB4guYltBUrcBojlxAk97Wlji2QycFktlgMUEndqRyEkDj9q3T+e9ViPtvfW3vvZz9rPj/1+zWgk7R/fvbR/fLWe9az1Xcvm5+cFAAAAAACwlFPybgAAAAAAACg+BhAAAAAAAEBTDCAAAAAAAICmGEAAAAAAAABNMYAAAAAAAACaYgABAAAAAAA0xQACAAAAAABo6hfzbgAA4GRmNi9p1jm3Nu+2hGJmg5LGJA1K6pM067+2OeemutSGUUnbJA0556a78Zi9wMyGFb226/1Fs5KmJW1xzs3l1jCdeN/tkTTSrfdZKD5P1DMnaaekCefcbBebBACoOGYgAAC6zsx2KTqIG1R0oDMuaUZSv6RdZjaRY9tGzew5P7iANpnZNkm7FL2WOyVNSjoqaZOkAzk2LTeB31NzkoYSXyOKnvMNkg7yPgYAZIkZCACArvKDB8OSpp1zQ3WuH1Z0UJSnPv+FNvjXblTSpHNurOa6AUUDRr0q1HvqaJ3ZM1N+IGePpG1mdrTssy0AAMXAAAIAoC3xGU3n3GTK+zYcPPBxcz3Q8X9X239bGp08lwW1UZJqBw/8ZTOKZpkEl8fzutRjdvM9lXjMWTNbJ+mQpAlJqT9XFXyfAgBSYgkDAKBd44rWt6e9b/J7r+vkuSyi/rwb4OXxvBbutfT1JrZI6u9wKUPh/jYAQD4YQAAAdIWZ9Ss6wJz2Z6NRPdOSZGab8m4ITohnDTAAAADoGEsYAKAkktXjFR2Ij/nvM5JuSh6UJ247pqiA3WZ/26OSxpPLBJK3TU5Rrt2hwK+pHk1cH1eAb7gcocaA/97W4EGifUOJv2Ug3qHCH6xu9PHj6vPjtdX+/Rr8CUXr8Gck7WjyeCM1z1Nf4v4rFR0s35R8nFZfo1aeSz/gEj9e31J/21J8Qcph345p1dnlop33VhNb/N81YWZDit4/0/Xa61+3CdXZDSFxXfzey+x5rfM48Xun7t/a7HVv8bVsuPODf30GfRtmfTvGs9o9wTk3Z2ZzWvj8JR97yc9Oq89nq59BAED5MQMBAMpnQtHB9ISis4sDkvY2uO02RQdbOxQd+MS7HJx0MNGCbf5x5xQd6MRV31tdjnCp/34wxWNL0d9xQNHfOyUt2gpy2n/foqj6/KJq//5g/ICiA7VJ/7dcqug5bMofRB5SdDA15R9nsPZxEpq9Rq08l/HfGk8fn9TCQWxLbTazg4p2P5jyMeYUvf7bUrZ7Sf6AcZ2ig+BBRbsxPGdmB/xrlRQPVm2uE2pM0lyd4oBZPK+1sbY1iNXq6576c2FmBxS9PrP+9lO+HVnvQjLrH+/EEpMWPztN/7ZWP4MAgGpgBgIAlFDN2T9JGjWzgTpnihdVw0/MKogPntp5zPgs71HVP7hLzR+o1T5e7dnLYUlbnXPjidtMS1pbE2tW0UHycOJsb3zAPJRo92TtGdYlbFc0C2Bd4nmYlnSg5nGS7W/4GjV7Lv0AT5+iM93J2O3UjtiuaMBobeJs9mR8dt/MdtV7Ddt4b9XlH2udP1gd1sKZ6T1mdmKWiz8zPiVp2Mz64zb6v71f0tYG8VM/ryn+1qave9rPhZ95MKCTZyWM1/s8dOho7QWtfHZa+dva+AwCACqAGQgAUD61HfL4jH69M9OLzgL6g7dZ5bOd3uP++9o61+2S9Fziq95Z75nk4MES4oPlZEG/QUXTrmsPgFqdDTEsaSp5EJ3YVaDe2vJ2XqN64r9hc5rZIv4AdFjRANKiqfDOua2KzijXey47bXfycWadc1udc+sUzUqYU7SlYPJ12eK/J5/Djf57vVkSmbWvxVjtvu7t2KToPV1v8Cnrqf/xc37SQEKNep+dNLKKAwAoGGYgAED51B70tnuwMa3oTGt/7cFlYPFBWL3Bi3EtHLjtaXD/RjUL4jXxca2A+KBlVeL65OO3JXEAP5xYA55U7zns6DXyZ+e3KjrIPODXsE+r9bXx6/33RtPIZxO3Ser0vVWXc27GzMYVDQoMy88u8JfPKJoFEg9ojCo6sM78eW0nVsrXvSWJ92Rms3ia6Jc0W6cuyJKfnVZlFQcAUHwMIAAAusLvSz8jaaB2SnxNAchGB4X1CvENK5q9MKPo4HRW0QHMrizb7m1Vg0GMEJxz436JxZiig+5hRQezazMa+Ml6mnwz8cFy7QyUbYpmJgwreo37VH/2QV66+rpnLbF9Y23xxkw+O13+DAIAcsYAAgD0nkHpxFr1pYQ4wLxJ0VnxtmswqP706+06uSL8oin/fuBCqlOFvhX+LLkk9XV7+0n/Go0rWhc/oOi5G1PzWgj7/fchLRQrTIp3HciUmfUtMf0+Pit90rIaXw8gLtYXL7XJVcjXvdP3ZJsmFNUvqH3PNP3stCirOACAEqAGAgBU27rkL76AXr/qH1TWDhhsrHMbKTrIS7W22R+ITUkaNLMszlDWG+SoNzV/2j9m7YFNq4MYk/LLPmqvqLO7QDvqPpd+B4VFlycOYpsO7PiD+GlFMxYW/c2JHRi2nHTHzh3w77FFfE2G+HF31rnfpKLK/fEuGZ1K/R6t0c7r3u5jblX0njzp/dPheyqO0ed3eehTNHBXq9XPjrT039ZOHABAyTEDAQCqbdTMVioqYDik6ACttoBePBNhzFdPXylpRI0PGPYrWoYQHxAOOufqFUasyzk34gcPhs3sOUUHlPEBSrz7QKvigYFdimonrFVUN6DWuKIz3wfMbFLR+veNav2Ab1zRc3fQ3/+AosGZDYqej7Rr2Rs9l4OKqthPK5opcEQLAzqtDryMqP7fPKCouGKI6vizinZ4mFD0nMwqeo7jA+KRBjMUtih63QZU/2C3XR29RxPaed3beky/RGVQ0e4UU4o+o2t97KOqX2y0npU1Aw59ij7rG/zP4w1e61Y/O83+tnbiAABKjhkIAFBt44oO4sYUnRWckrQmeRDnp8lvVXSgt93fdpcaT5OP1zlv8DHb3rPeOTei6CBnv48THzzOKjrIXLfE3ZNGFB1gD/t29PvLah9vRtGBX1ywb8z/HS2dhXfOzfkDpq2KDii3aeFs+UmP14a6z6U/4BtTNJgzqoXneKTVbQITbZ70bU3G6HQHgUaPOaTodZ1S9FqMauF9t7bRoIV/P84oKvSXxXKBjt+jcbvaeN3bfkz/Po8/exM+9k7VzBxqok/RgXv8tcu3YVrR9pN1t8NUi58db6m/rZ04AICSWzY/X6+wMACgzOIzm5LGirCeHFiKX+LwnKKz5Y0OeAEAQM6YgQAAAPK22X9nsAsAgAKjBgIAAOg6X5hwTFF9h02Sti6xgwMAACgABhAAAEAejiqqkdAnaarONoMAAKBgqIEAAAAAAACaogYCAAAAAABoigEEAAAAAADQFAMIAAAAAACgKQYQAAAAAABAUwwgAAAAAACAphhAAAAAAAAATTGAAAAAAAAAmmIAAQAAAAAANMUAAgAAAAAAaIoBBAAAAAAA0BQDCAAAAAAAoCkGEAAAAAAAQFMMIAAAAJnZsJkN592OTlXl7wBQbeSq9vB8Fccv5t0AAADaYWajksYkDUiak7Rf0i7n3GSuDavDzPZI6nfOrS3yY/lO2S5JwdvZBbOSDpjZkHNuOu/GAEVjZv2SBpxzU116vPk6F8e5eyLt5zSRt9Y552Y6aGLXVSzndszMdknaIWlG0lHn3Fydm5HbC4IBhIIwsz5Jz0mads4N5d2eLDT4hyFFCWCbc25rN9sDoPz8QfKgok7GVkl9/vdtZqYiDiIUnf//s0vSmHNuNu/2LMW3dVTR/xEpGjBZ9L/EOTdjZuOK/qazu9xEoNDMbFDSkKQtZjbaxZw5I2k88Xu/pBFJe8xsyjk30qV25K5qObfNeKOSZusMAAz7r/h2c5LGk+9PcntxsIQBoc0o+kcVf434yybM7IBPTADQlO94DCrqVKxzzo0758b8GfcxRWez0L7tijp0ZRh82SVpyjk35c+eTvlBpUXiDq6Zbet2A4Gi8n2ubT53zqm7Z7+POuemE1+T/oTZVknDZrapi23JW+Vybivi91+Dq8cVDQqsk7TWOXd2veeH3F4MDCAgtNp/GPEoczz9eG/O7QNQHiOS5uqd/fCd0UVTWM1skz/bhgZ8h25YjTt1hZE4c3XijJ3/ea7ButhJRWfOAEQm5D/r/rN/JN/mSM65cUXLGcbybks70v5/qXjObWaDote6LufcnHNupoVZGeT2nDGAgKZCdML9qOK4pIFOCqJwgAD0lH5JR9u4/WZFgw5obIP/XoYzYSOSDtS5/HHVP/iID5QouoWe5w9cN2jhsz4oqSjryPcryu9lkvb/S5VzbkO+r57V+43cnjMGENCKIJ1wfxZxTtGIeFocIAC9Y1pSP52GTI0oOsPU8KxQgQxqYR1u0qyk9bUXxmfKJG0M3C6gDDZL2p/4rF9aoMKD/YqWt/aCyubcJvqyqvdAbs8fRRSRt52SRs2sv+iFZADkblzRtMVdZjalaC3vSWc0/NrIeHrjqJ+GKUW1E7YmikKNKeq41i3s6u+3TdGazI2JmNP1Cn75MywTiZhb6v0RrTx+TYXuAX/bWefcWDuP1YJBSU0rsfuq7YOKngvF7chBvRkoc4qKadazX1G7gV43Kl/E0OePVOvYs+ZzXb+iOlnJy/sU5bhBSSsVDSCPN+grrjSz+Lb9ij73I8mD9EQ+X1TB36/nX++cOztx2UDisaUox04558ab/X9p4U+ues49SavFOmtODlwqacsSAy3k9hwxgFBw7XRgEx3OIUUdzo3++7RSJNIMkmQr4qlRA/KjnC12rjM5QABQHs65OTNbpyjPDSsqvCVFnbGbEjlum7/NHkX5L57lFHc8NyjKDVOSDio6SJ8ws1V+PW6tXYrOjm1RlF+HzWwieVvf4Yw75FsVrS3erERuS2jn8Sf83zodx2/zsRpKFLF9vMntJnzs8VY6gSG0UnDXzPrqdDZnJA02uA7oCT5n9Ck6aSNFfb96uS50O5Kf4/WK8umooj5qsi/ar4X+YTw4ulHSwQZb+O1RlE+3KTrDP6ioxta6lE3dqyiXjil63tZq4WC12f+Xhnok59beZkCtFTgeUuJYxcymFb0HGhX6JLfniAGE8mjagU2IE+kO//uglv4QNpI6SbbhxLYwicta6VxnfYAAoAT8lNu1vlOyUQtbPw2a2Zq4CJMk+cGFk7aL8h2yRZ0y32HdpMXbjMVmEwO2W/0WtcM1t41z0NrEGbKtZnawzt/QzuMPK+pUJc9YtfxYTaz03xt2vvze3P3OubY64vH92mzPtiU6yysbXF57m9q/JS4S10tTpIFaG+WnzfuD07QzljoxqGi78qSZ5Jn/hLjA4JrEweFWMzvgr6vtzyZPbE3GJ5nMbLDeLLWl+NkZfT7mSfdt9v+liSA5N0C+ldLn3FrrWxkEcTVb2Pv36swSxzrk9hwxgFAerXRgY2OJD+vWeN/0dhNph0myVfHUqFWJx23auQ5wgACgRHwOmJE0nphRtV3pa6I8rmhgtt5yqtpq2TNKdNb8mZpBRVNca+/b6lTPRo8/mRw8yOixYvHfULcwpZ/VNqyaqcWtqLfEIyNLdWobTbVtdj+g6gYkzfitEvfkdMZ2Rov7X7sUFdJelPMSU/fH67RzXNIeMxuuGVStPYCMl7uNqP3CfXFbxsxstk6e7USQnBsw30rt59wTWl26sIRZNT7WIbfniAGE8liyA1ujNtmNKToLnyaRhhYXYWl25mypzn2rsogBoGCcc5NmNqaoo9GSxAyGeM3sUgfetZ3Yo1qcf+Ofl5yWmvLxa9cpt/1YS4gfs9HBxGb/fcw/v1K0NW8ea3HjjmrD16nBQVE7u3YAVRUv5ZwLdDKoFUdrlimMK+rbTmjxwG+c4+qdVY6nwl+qJeoI+LPXyVgtc87Nmlm8TeCwmc0pOiG11Hr8VvVCzpV0YiCo0772QTV+DcntOWIAoTyadWAb8slwrtXbd1k8DW3R+qg2O/d1ZREDQGlMq87ZrHoSMxam/fdZRR3Yruwr3ebj5znYOaBopkPuO934A4I51T/b1K/m02iBXrZS0We5MFsHJgd+a/J2X833pHbONqfOCc65Mb8MIl4mt0mJgoYBVSnnDkpaZ2a1f0ufopmDI4pmw0z55XfjNbNKUGAMIPSOoh48jyoaET8x0pxF5z7vAwQA2WtSLKlfOrG9U1K9zs82RUsDTpzV8QOOacWP2WqdmU4ev93HaiVWo/8Pc0o5gBFoTe5+1W9rnxrProtff85WoSf5M8F9WqiLVSQ3KarRtU0L0/bj/mC9WQZxnmxWhLBP0d/cytr4uoMSNcvkNimqozXgFm992e70+SA5N1C+ldLlXEknlhKfxPfPJ+rMhKn3d69V49eQ3J4jBhB6gP/nIXWQSJe4PDWfRPoULbFIardz3Y0DBAD5O2RmJ1WlTqwbre1snlQTYInK0pembVRc7ElR0a6JeBDDP9ainRE6ffx2HqsFzdaQTivl4HOgM2jxLkO1hnTyMr9YsynDQGX5/t+wWtshYFAL/bEdtWeD/UH0KkXF6yazqKPgnJuxaEve4bhOl581OyNpk5ltqxkUnlB00qk219cePMdLAZKDJidNyU/kzbmay1bWPG7cf07mynZrzsT3qY2TlCrnBpyxkCbnprGtZmAmNqyFosG1yO05YgChmuKtG2Pxh7ztROqlSZJL8h3+bYoKIE4mLm+3c92VAwQAhTAraZuvIr7f/75eCwfON9Xcfr+iArITWpihMJI4AJeis19DaqN+QgPxmbQDfv3sEZ08OFo7AJD28Vt6rGZ8R11qPJth3D/GomJmedWRiac8Jx8/HhheYurrWn89dW/Qk1y0tfWlShy0xsVY48+NHzzo9/lxQNJ2Mxtzzg35204oUdTQnwDKajnEuBYOFOMlAiNayHE7FfX1hhXl8XoHtBNmdtTfLp5tOlVzUBrngM0+761U/eJ8g5J2WbSNYHz/eLZssm9d9//LUn9olXOumT2nqF5DK7Pjavvp07UFF/2A1aJjhBrk9hwxgFBNE2YWjxKPKUpsaROplCJJJqz0/5hicfKPR8QXrSdL0bnu5gECgBw559b5wceNigYNBhXlka0NtnmaUDTAMKooV8RnMkYUnVkZ9feflrRV0TrXtG2bMbMh/xibfMwRRTtD1HaWOnr8Nh+rmTktTAuufZxZM1un6H9KvCd7n7I7cEjjKkX/t+LCu2trt/+qsV751pEAcpM4uBpX9DneL18Xyi1seyj/+6S/z4yitesH/dT4o6rZEcEfWNbuhJC6jWa2VdGMg2Hn3JS/bI2iHDeoqI+6X9JQgwPGcS3UvDqqOv8TkrMdFOXKOG/G/yfi20352gxjWlj2ul8nD9I2+v/STFVz7n4tvT3lNi08z9vNbKOi99Wsf23k+/JS9DcfJLcX17L5+fm82wCdGA1+TtJ08gOTWMs/VFO9do+ivVXPTlw2rKhTOqLogHmDv2qyXufa/2MYVvSBn1a0L/BEnbiDPq7kk2QrVXwt2mqynllF05W21rvST7nbpYWzitO+jZu0eN/zhm1rJwYA9CrfqRt1zi3Luy0h+P9DjQaYgJ7h+0UDkmZq+0D1BgP87Q9I2llvF4CsBhB6TdVzbreQ2/PFAEKFJAYQhlo5wAfKrE4xo+R18cDYQKOBKgAnpqMeUAX/byT+JzJoHFC7ubjVy9A9SwwgbFN0Vn+kzvUMIKRQ5ZzbLeT2k3U7D5+SZeMBoBv8zJPtDa6L1+dNS5ozimcCDfkOx5xS1FAogTFFa2jpYAbSbi5u9bLutB5JyfpR/udxPyN2RFFNgMHE9fHSKbSp4jm3W8jtCXnkYQYQAJSOT3CNtu7ZqIV1eLOKzp4AaOwmRVXQi7rdb9v82dNkVXkEkCIXt3oZusjPJNhgZpv8OvTN8bIFf906RdsZHvDLX6ez2IWhh1Uu53YLuf1keeRhiigCqJo+LU6kq/JqCFAGvmjYtBpv2VVGuxQVD+YsaX7q5eJWL0OXLVHtPj5rXpXckLuK5txuIbe3J0geZgZChfjKtcv4UAEA2jQiqd9PTS41fwa1L+De6ADQqcrk3G4htxdHt2cgULERKI/UFYJ/9u+zqT7r/+dvvT25bVLS5FJnR2rMaWG/6z5F25liMXIxFnHOxT9OqPXtyAop8bdU5X3e1VycUR6WGufiVi+ruqq8P5FClXJut+Sc26vWJ+4oD3d9CcMvvuI1QeL+/Kc/0M/+PVwtjVPP6Q/Wdilqf8jnhrZXL3432t5tPiGm2u/YzPr8mswdWthruF8UejpJ6Fz51Jprg8S+6NBuvfzkg0FiS9LpF1+jFz/xvmDxz/zo5zV70dVBYvc/9ZD2rQ53Uubyw7u0/5d/L1j89f/61/rxh68LEvuVn3xA97z6PUFiS9KNP7xbz11/eZDYZ9+/L0jcpXSSh6WWcnGrl1UefeL6yt53ou3dj0+feLHQeZglDACyd/w/0n21yG81s95/j+2VTqzVjKvSzjXa1gYAKi9gHpbaz8WtXtb5Hw4ABVGwPnEWeZgiigCyN388aHhfFXqq5rJ1iZ9Tnz0DgMooYC5u9TIAqIQK5mEGEABk73jYZAkAaAG5GADyVcE8vGx+vqt1KCgYA5RH6oIxP/3hN1N91l/x6jemfky0hVwMlEdXczF5uGvIw0B50CdOoIhiiygYk0/8Mrc9dPxCF4yp4GhrlVBEsT6KKDZGEcXGCl1EkVxcaPSJ6yt734m2dz8+feLuYgkDgOwFXu8FAGgBuRgA8lXBPMwAAoDstVnJGwAQALkYAPJVwTzMAAKA7FVwtBUASodcDAD5qmAeZgABQPYquN4LAEqHXAwA+apgHmYXBgCNpK7++pOD/zvVZ/20tf+psBVnK4ZcDJRHV3MxebhryMNAedAnTqjULgxlrRwaOj5tr2b8blRZTq2Co61V8tPZfwoW+xX9v6YXPxpmN4AzP7Er+Hv+2EN/GSz+8qs/oOffe1WQ2Gd9Ya8OXTwUJLYkrXlyjw5fcVmw+KsfeVQ/+ebeILFPe+NV+vHm64PElqRXbrk/aJ7vCLm40Og7dT8+ba9mfPrE3XVK3g0AAAAAAADFRw0EANmrYMEYACgdcjEA5KuCeZgBBADZq+CWNQBQOuRiAMhXBfMwRRQBNJK+YMz/90i6gjH/xxWFLRhTMeRioDy6movJw11DHgbKgz5xAkUUeyA+bc83fqiiLqee0x+87alVsGBMlYQuNPTE+dcFiX3Jdx/Qy/9wb5DYknT6b9ygF25+W7D4K+76qqbOuzFI7OFn7tHXzx0OEluS3vzslO567XuCxb/5+3cHLb55+/nh2n7bd+/WD3/ziiCxX/33j3QWgFxcaPSduh+ftucbnz5xNbCEAUD2KrjeCwBKh1wMAPmqYB5mAAFA9io42goApUMuBoB8VTAPM4AAIHPz89UrGAMAZUMuBoB8VTEPM4AAIHsVnK4FAKVDLgaAfFUwD7MLA4BGUld/fXnmgVSf9dMHritsxdmKIRcD5dHVXEwe7hryMFAe9IkT2IWhB+LT9nzj92TF2QqOtlbJT2f/KVjsV/T/muY2hqlI37fjkeA7SLz0yZuCxT/jw9s1d0Og5+beR/TsWy4PEluSzn1snw5fHi7+6n37dOxLtweJvfydtwXPlS9+4n1BYp/50c93FoBcXGj0nbofn7bnG58+cTWwhAFA9o5Xb70XAJQOuRgA8lXBPMwAAoDsVXC0FQBKh1wMAPmqYB5mAAFA9iq4ZQ0AlA65GADyVcE8TBFFAI2kLxjzD/emKxjzGzcUtmBMxZCLgfLoai4mD3cNeRgoD/rECV2fgVDm4hlljU/bqxm/G21PrYKjrVXy8te/ECz26W9+b6mLKP7oj4aCxX/VZ/fo6UvCxL/giT2aOu/GILElafiZe/TUmmuDxb/o0G69cMs1QWKvuPNB7Vm9MUhsSRo6vCPoe74j5OJCo0/c/fi0vZrx6RN3F0sYAGSvgskSAEqHXAwA+apgHmYAAQAAAG0zs2FJc5IGnHNb61y/SdKspJXOuUkzG5B0wF8mSdPOuTEzm3DOjZvZqHNusmt/AACUXB55+JSM/wYA0Pz8f6T6AgBkJ2Qe9p1QOeemJc3FvyeuH/TXT0laa2b9ijqwy5xzayWNSJrwNx81s4Na6NACQCWE7BPnlYe7PgPh1HP6g8XuaH1KxePT9mrGD9321Co4XatKTn/ze4PG73jd9hJC/g+RojoFIV3wRLj4w8/cEyy2FNUpCGnFnQ8Giz10eEew2FLY93xHwubijZLiN/SspEFJM4nrhyQ97n8+KGmw5qxWv+/UStKI7wD3FPrE+cSn7dWM36N94lzycNcHEMpcPKOs8X/+0x8ELzzG69r9+IUuGFPBPW+rJHQ+OHLtZUFir9r9qF5+/P4gsSXp9Euv17HPbw4Wf/n7tgTNB995w1uDxJak13/ra8FeVyl6bX/2b/8SJPapv/QreuFD4QpArvjUbr30qbEgsc/40LbOAoTNxX2SjiZ+X1Vz/RFJKxO3PXG9PyuW7KgOmJnUYApuVZW5f1DW+PSJqxm/h/vEueRhaiAAyB4zEAAgfylysZmNShqtc9Vkm/UJpiTFIyurFHVkY0PJM11xZ9XMhsxssBdnIwCoqJR94oxycZA8zAACgOwxAwEA8pciF/uOaSud0zktPrOV7JjKOTdrZjsSa3KTp31PrNP1BcDiNbpHJIVdpwQA3ZSyT9xiLs4lD1NEEUD2jh9P9wUAyE7YPLxDC53MfvmpsGbW578PSFrvnJuR1Bevs/VFvOYScWa1MI12raT9af9cACicsH3iXPIwAwgAsjd/PN0XACA7AfOw75DG62jn4t8l7U1cf9Sf2UuwSC0AACAASURBVKot5nC0Js4Gf7uDiTgAUH4B+8R55eFl8/PzLTUwI119MAAdWZb2jse+8j9SfdaXv/0/p35MtIVcDJRHV3MxebhryMNAedAnTmAXhh6I3422h6poG7KarcTr2ix+aixHKLSXnwy3Xd7pF1+jl/7iliCxz/iTO4NXz37p0zcHi3/GB+/S188dDhL7zc9O6R9f/a4gsSXp13/4RT21JtxOBhcd2q0Xt/x+kNhnbv6cvvG6dwSJLUlv+t6Xg77nO0IuLrQy9w/KGp8+cWO8rkvHT62CeZgiigCyx3IEAMgfuRgA8lXBPMwAAoDsVXC0FQBKh1wMAPmqYB5mAAFA9io42goApUMuBoB8VTAPU0QRQCPpC8Z86fZ0BWPeeVthC8ZUDLkYKI+u5mLycNeQh4HyoE+cQBHFHohP26sZP2ShHikq1pNaBUdbq+Rnh12w2KeuNh2+/PIgsVfv2xf8Pf/CLdcEi7/izge1+9wbgsS+9tl79dz1lweJLUln379Pj7/mncHiX/qDL+n5378qSOyzPrdX//M17wkSW5L+4Ad369DFQ0Fir3lyT2cByMWFRt+p+/FpezXj0yfuLpYwAMheBdd7AUDpkIsBIF8VzMMMIADIXgWTJQCUDrkYAPJVwTzMAAKA7HW3tgoAoB5yMQDkq4J5+JS8GwAAAAAAAIqPXRgANJK+4uy9/3e6irM3/D+FrThbMeRioDy6movJw11DHgbKgz5xArsw9EB82l7N+IWuOFvB9V5Vwi4M9bELQ2PswtBYoXdhIBcXGn2n7sen7dWMT5+4u6iBACB7gbesMbNhSXOSBpxzW5e4vt85Nxm0MQBQVBXcPgwASqWCeZgaCACyd/x4uq8WmNmAJDnnpiXNxb/XXD/rr5+tvR4AekagPAwAaFHAPnFeGEAAkL35+XRfrdmoaHaBJM1KGqxzmwn/vd85N9PhXwMA5RQuDwMAWhG2T5wLiigCaCR9wZi/2pSuYMwfbm36mGa2TdI259yMmQ1KGnLOjde5zaCkcefcVJq2VBy5GCiPrubiVvIwMkEeBsqjkH3ivFBEsUDxQxX/OPWc/tI+N1V4Xcv83KSWcuqVmY1KGq1z1WSrtQzMrE/RDIVtkrab2YxzLlxlnRIKXWgoZBHFn3xzb5DYknTaG6/Si3+6IVj8M/9spx5eHSb+lYd36qur3x0ktiS97fB9evqSMIUCJemCJ/boxY+Faf+ZH79PU+fdGCS2JA0/c4+OvvOyILFXfunRzgIUfBpsrytz/4A+cX1l7/eVNX4V+8RFRhFFANlLWTDGDxI0GyiYk7TS/9wn6UjN9aOStjjn5sxsRtKwpJMKLQJA5VWweBcAlEoF8zA1EABkbv74fKqvFu2QFO+n0y9pWjox82CRuNBiFn8TAJRNwDwMAGhB4D5xLpiBACB7Aadr+doH6339g7lEkcS9ktY557aa2SYzm5W0km0cAfSsCk6dBYBSqWAeZgABQPYCT9eqNyjgnFuX+JklCwBQwamzAFAqFczD7MIAoJHU1V9f+otbUn3Wz/iTOwtbcbZiyMVAeXQ1F5OHu4Y8DJQHfeKESu3CELqyOJVJux+/zG0PHZ+Ks0grdK58as21QWJfdGi3Xn7ywSCxJen0i6/Ri594X7D4Z37085q96Oogsfufekj7Vo8EiS1Jlx/epf2//HvB4q//17/Wjz98XZDYr/zkA7rn1e8JEluSbvzh3Xru+suDxD77/n2dBSAXFxp94vrK3nei7d2PT5+4u1jCACB7FUyWAFA65GIAyFcF8zADCACy192lUQCAesjFAJCvCuZhBhAAZK+Co60AUDqBc7GZDSvaKnegXvFaM9skadGOOGY24ZwbN7PRxGVLxgGA0qpgHqaIIoBG0heM+eRN6QrGfHh7YQvGVAy5GCiPrubiVvOwmQ1I6nfOTZnZqKT9iW115bfaHfBb605I2uacmzWz5yQdlTTmnJtuFqfCyMNAeRSyT5xXHqaIYosoGJNP/DK3PXT8YheMoV9UZGXNld3I88ceuCNY/OXX3apn33J5kNjnPrZPR669LEhsSVq1+1G9cMs1weKvuPNB/eTbjwaJfdqFl2nuxiuDxJakvnse1rGdHw8Se/mGj3UWIGwu3ihpj/95VtKgpGSHc0jS4/7ng/76SUkjzrnpNuJUVplzJX2n7scvc9tDx+/hPnEueZglDACyV8E9bwGgdMLm4j5FZ7Biq2quPyJpZeK28fUDZiYtTJNtFgcAyquCeZgBBADZYwYCAOQvRS7201dH61w1Ga+VbdGUpDH/8ypFHVnFa2vNbMhPrwWA6krZJ84oFwfJwwwgAMjcPEUUASB3aXKx75i20jmd0+IzW0dq4sya2Q6/tlaSZn2RLjnnpvzt+5vFAYAyS9snbjEX55KHuz6A0NEakiZOPac/WGwpbNtDx6ft1Ywfuu2pMQOh0MqcK0O3ffl1twaNf+5j+4LFXrU7TA2B2Io7Hwwa/7QLw9Vw6Lvn4WCxpQxqFYQSNhfvkLTe/9wvaVqSzKzPOTfnO6zrnXOTZjbmi3MNKFpfK0lrJW2TtL9enF5Q5lxJ3ymf+GVue+j4PdonziUPV6qIYlkLf4SOX+ZiOmV+3kPHL3TBGGogFFrofBCqmN+q3Y/q5cfvDxJbkk6/9Hod+/zmYPGXv29L0HzwnTe8NUhsSXr9t74WvEjjz/7tX4LEPvWXfkUvfOjaILElacWnduulT401v2EKZ3xoW2cBAuZi59yMma3301/nEhW790pa56/v92e7tiXuM2pmRyUdjO/TIE7llbl/UNb49ImrGb9X+8R55WGWMADIHjMQACB/gXNxvXW4zrl1iZ+nWrxPO7UVAKA8KpiHGUAAkD1qIABA/sjFAJCvCuZhBhAAZI8ZCACQP3IxAOSrgnmYAQQA2aMGAgDkj1wMAPmqYB5eNj/f1VGR6g3BANW1LO0dX/zoSKrP+pmf2JX6MdEWcjFQHl3NxeThriEPA+VBnziBXRh6IH7Z217WarlS+V/XtNLueYvuOLY3XL2y5VeN6huve0eQ2G/63peD54PDV4TbaWD1I4/qW2t/J0jsNxz8W/39edcHiS1Jv/nM/cHzzUt3fTBI7DNu/rQOXTwUJLYkrXlyj779q9cEiX3hP3e2dSa5uNjK3D8oa/yyt50+cfdjx/HTqmIeZgkDgOxVcL0XAJQOuRgA8lXBPMwAAoDsVTBZAkDpkIsBIF8VzMMMIADIXgULxgBA6ZCLASBfFczDFFEE0Ejq4i0//vB1qT7rr/zkA4UtGFMx5GKgPLqai8nDXUMeBsqDPnECRRR7ID5tzzd+qII33Sh2g2oKXYTpifOvCxL7ku8+oJf/4d4gsSXp9N+4QS/c/LZg8Vfc9VVNnXdjkNjDz9yjr587HCS2JL352Snd9dr3BIt/8/fv1osfHQkS+8xP7NLt54dr+23fvVs//M0rgsR+9d8/EiQuioG+U/fj0/Z849MnrgaWMADI3HwF13sBQNmQiwEgX1XMwwwgAMheBZNlkZjZrZJmnXNf9L/vkDQs6aCkDc65J/JsH4CCIBcHQx4G0JIK5uFT8m4AgAo6fjzdF1o1JmlOkszsI4o6rVdLelLS9hzbBaBIyMMhkYcBNFfBPjEzEABkr4KjrQWzVlK8kHBI0pRzbq+ZPSfp8fyaBaBQyMUhkYcBNFfBPMwuDAAaSV399YWb35bqs77irq8WtuJskZjZUUlXSHpa0nOShp1zXzSzN0mads6tahKCXAyUR1dzMXm4NeRhoKfQJ05gF4YeiE/bqxk/ZDVbKapom1aXByZ70e2SvqGo03piDa6iKbV7m935p7P/FKxhr+j/taDV9EO/54899JfB4i+/+gN6/r1XBYl91hf26tDFQ0FiS9KaJ/fo8BWXBYu/+pFH9ZNvNn3rpnLaG6/SjzdfHyS2JL1yy/1B83wnyMVBdZSHJfrEecSn7dWMT5+4u1jCACB7FZyuVSTOua1mNiPpLEnTiav2SJrJp1UACodcHAx5GEBLKpiHGUAAkL0KJssiMbN3Jc52Jc1JWiPpUJebBKCIyMXBkIcBtKSCeZgBBACZq+KetwWzS9IvNLhuQtKlXWwLgIIiFwdFHgbQVBXzcNcHEDpdz5dX7LLHp+3VjN/JmqygKpgsC6ZRYZ0+SQPN7vyK/l/LtjU1zvzErmCxQ7/nl1/9gaDxz/pCmHX+UlSnIKTVjzwaNP5pbwxTH0KK6hSEFPr/SGrk4pA6ysMSfae84tP2asanT9w9FFEsUPxQxT9OPae/tM9NFV7XMj83qRV7+9rSMrPjiip3z5vZf9S5yTJJB5rFCV1o6PDllweJvXrfvmCF9qToAPbFP90QLP6Zf7ZTD68OE//Kwzv11dXvDhJbkt52+D49fUm4Io0XPLFHL34sTPvP/Ph9mjrvxiCxJWn4mXt09J1hCkyu/FKHgzbk4sxllYcl+sRLxadP3N3YZY9Pn7i7WMIAIHNVnK5VEEOKOqcPSRpRtNY2adY5x7pbAJLIxYGQhwG0rIp5mAEEANmrYLIsAufcXkkys2nnXNg52QDKj1ycOfIwgLZUMA8zgAAgexWcrlUkzrmr824DgBIgFwdDHgbQkgrm4bYHEMzsVkXTs77of98haVjSQUkbnHNPZNtEAGVTxelaRWNmF0jqV1Swa5EGW4sB6DHk4rDIwwCaqWIeXjY/394fZWb/ImnMOfewmX1E0u2SrpZ0s6QLnHNLbVtTvWcQqK5GFaabeu76y1N91s++f1/qx+wlidxb7/mad8412lrsxG2ybxWAQLqai9vJw2Y2rKgGwIBzbmud6zdJmpW00jk36S8b9Vevdc6N+8smnHPjZjYa367oyMNATylsnziPPJxmCcNa3wgpKiQz5Zzba2bPSXq82Z3LXH2zrPFpezXjh6xSLBV4OxxI0mZJfy5pi3Pu+Xbv/LPDLvsWeaeutqC7MIR+z79wyzXB4q+480HtPveGILGvffZePXf95UFiS9LZ9+/T4695Z7D4l/7gS3r+98Ns43jW5/bqf77mPUFiS9If/OBuHbo4zA4Vobfm7ISZDUiSc27azPrNbMA5N5O4ftBfP2VmE2bWr+hs/bRzbtbMdpnZoHNuWtKo7wSP5fG3pNRRHpboE+cRn7ZXM36v9onzysNpBhDmJJ1lZmdJGlS0fEGKRlJrK9EC6EFVnK5VMEcl3ZW20wqgNwTOxRslxSMcs4r6hDOJ64e0cGLpoL9eijqvk/4+ca98xHdgy4Q8DKCpKubhNAMIt0v6hqTnlKiFoGi0ItwG3QDKo4IFYwpmm6LB2zvybgiAAgubi/sUHUTHVtVcf0TSysRtV8VTZb0BSTvin81MajAFt6DIwwCaq2AebnsAwTm31cxmJJ0lKTlKsUeLRzwA9Kh5BhBCu1TS9Wa2WdJ+1cz+cs5tzKVVAAolTS72a2NH61w12WZ9giktTIVdpagjGz/GgKQ98VTbuLNqZkOJ6bRFRx4G0FTaPnFGuThIHk6zC8O7GlSWnZO0RtKhpe7/85/+oN2HbFnI2GWPT9urGb+oa7KYgdAVyf3H2yruc+pqy7gpi63ety9Y7NDv+RV3Phg0/rXP3hss9tn37wsWW4rqFIR01ufCTWL8gx/cHSy2VOBaBSlyse+YttI5ndPiM1tHklf69bU74jW6WqifJUmDic7qsL/9lI9R0H9sdaXOwxJ9p7zi0/Zqxq9an7jFXJxLHk6zhGGXpEaVZScUjcg2FKrAxann9Je28Efo+LS9mvG70fa0mIEQlnNuQyf3f/nrX8iqKSc5/c3v1dzGK4LE7tvxSPAiST/6ozDF8CTpVZ/do6cvCRP/gif2aOq8G4PElqThZ+7RU2uuDRb/okO7gxWwXHHng9qzOtzJ4KHDO4K+5zsROBfvkLTe/9wvPyvVzPqcc3O+w7reOTdpZmO+Yypf4TvutA4q6tDGH+y1ipYGFF6neViiT5xHfNpezfg93CfOJQ+fkqKhjUZY+xStowDQ646n/GqRmQ2b2aDfmqbe9QP+NsP1rq8CM3u/me03s58nLvuamYUrlQ+gXALm4Xjaq+98ziUqf+9NXH/U5+FtidtOmNlBv3tXfLsN/nYHkxXEi448DKCpgH3ivPJwyzMQzOy4op0W5s3sP+rcZJmkA63GA1BdIUdbm21Z440558bMbFOD60vNzG6XdL2k2xTNCottl/RfJIWdaw6gFELPBqu3Dtc5ty7x81TNddOSzm4lTtGRhwG0oop5uJ0lDEOKBgkekjSik7dsnHXOLVn/AEBvCJwsl9yyJh49lRYKwlTQqKRh59zDZpbcH+iAmAkGwGM5WVDkYQBNVTEPtzyA4JzbK0lmNu2cu7/Z7RsJWeCizIU/Qsen7dWMH7rtaQVOls22rLlUOjFTYbDCgwhxhzW5rKxfTQrZSlGdgpA6Xbe9lNBFkl712bDF8C54Ilz84WfuCRZbiuoUhBSygOXQ4R3Nb9SBkO/5TlSx41owqfOwRJ84r/i0vZrxe7RPnIs02zhe3dEDlrh4Rlnj//ynPwheeIzXtfvxi1wwRvNtF6OWlOn2YUecczO+TsJw7fStCtguadLMRuQ7sGb2Jkl3+a8lhc4HR669LEjsVbsf1cuPpx6/bur0S6/Xsc9vDhZ/+fu2BM0H33nDW4PElqTXf+trwV5XKXptf/Zv/xIk9qm/9Ct64UPhCkCu+NRuvfSpseY3TOGMD3VYTzBlLkZLOsrDEn3iPOLTJ65m/Cr2iYsszS4MMrMLFI2w9tVe12CLRwA9JO1oaxZb1ihavhDPUJhVNCOhUgMIzrlxM+uXX7phZkcUPReTzrk7cm0cgMKo4pmvoiAPA2hFFfNw2wMIZvYRSber/m4M82q8xSOAHjF/POho65Jb1vjfhxPXPx6yMXlxzo34zutV/qJp6tAASAqci3seeRhAM1XMw2lmIGyW9OeStjjnns+4PQAqIORoq1+asL7BljXrnHOzZjYXb+FYweULJzjnkvv2AsAiVTzzVTTkYQBLqWIeTjOAcFTSXQweAGhkPvB6rxa2rCndlmBLMbOvSdrlnPuM/33JinDOuY1daRiAQgudi3sJeRhAGlXMw2kGELYpmh6can1Xmatvljl+6MrlvK75xKfibM+o3Wmio/9GofPBqt2PBot9+qXXB4stRYUOQwr5mX39t74WLLYU9nWVomKHoaz4VNgdJDoudhgIuThTmeZhqdz9gzLHp09czfj0ibsnzQDCpZKuN7PNkvYrKmh2QrMR1zJX3yxr/G60PVRF25DVbCVe12bx06rieq88OefW1/y+oZN4Lz8Zbru80y++Ri/9xS1BYp/xJ3cGr5790qdvDhb/jA/epa+fO9z8him8+dkp/eOr3xUktiT9+g+/qKfWhNvJ4KJDu/Xilt8PEvvMzZ/TN173jiCxJelN3/ty0Pd8J8jF2ck6D0v0ifOIT5+4MV7XpeOnVcU8nGoXBknJfbSq96wAAAAAAIBF2h5AyGLEFUC1zc/n3YLqMLMr272Pc+7hEG0BUC7k4myQhwGkVcU8nHYGAgA0VMXpWjmarnNZ/O+o9omOL2c7XQDk4uyQhwGkUsU8vGy+ybBIxlVnKzgGA1RW6oz39CVDqT7rFzyxp3pZNgAzu0nSWufcbYnL+iTtVLRLzhebhCAXA+XR1VxMHm4NeRjoKfSJE1qZgZBp1dkyF88oa3zavnT80EXZyvzcpFXF6VoFMyppJHmBc27OzMYl3S5pyY5r6Pf88384GCT2WX81rZf/aVeQ2JJ0+q+NaO6GK4LF77v3EX3x3P8rSOx3Pfv/av8v/16Q2JK0/l//Wg+ufnew+Nccvk/H/uaTQWIv/90P6ztveGuQ2FK0+8Uzvx3mfXPe3z3S0f3JxUF1lIcl+sR5xKftS8enT9w4flpVzMNNBxBCVJ0FUG1VnK5VMOskvUnS0zWXr5S0/qRbA+hJ5OKgyMMAmqpiHk5VA8HMXiVpQlJ8mmmPpHHn3AtZNQxAec3PVy9ZFsz9kj5jZnPOuUekE0W+tinaXhcAyMVhkYcBNFXFPNz2AIKZnSXpkKRZLWznOCjpaTNb55x7OrvmASij+eN5t6Dy3i9pStJeM0sW8pqWxCwxAJLIxYGRhwE0VcU8nGYGwqSknc65m5MXmtkuSXdJelsWDQNQXscrONpaJM655yUNmVm/pAFFxbhmnHOH8m0ZgCIhF4dDHgbQiirm4aa7MNQys6OSBmpnGpjZgKQ9zrnaootJFSwjAVRW6oznLnx7qs+6ffsr1cuyxUQuBsqjq7mYPNw15GGgPOgTJ6SZgXBU0hqdXDTmbH/d0g9Y4uqbVCbtfvwytz10/EJXnK1gwZiiMbN3Sdooqa/2OufckiXnQ+eyx84daX7DFN7y7K7g7/mXPnlTsPhnfHi73IVvDxLbvv0V7Vm91C7KnRk6vEPPvuXyYPHPfWyfjn3mw0FiL3//JzV70dVBYktS/1MPae7GK4PE7rvn4Y7uTy4Oq5M8LNEnboQ+cfdjlz0+feLuOiXFfSYlTZrZ+fEFZnaJouULU1k1DEB5zc+n+0JrzOwjivLtMklDiurSPO9/BgBJ5OGQyMMAWlHFPnHbMxCcc1vNbK2kQ2b2nL+4T9KUc25zpq0DUEpVHG0tmNskDTrnHjaz78Q1acxsWNJV+TYNQFGQi4MiDwNoqop5OM0MBDnnxiS9XlHyvE3SeudcuLmTAErl+PyyVF9o2dmSDvqf58zsYv/zAVH9G4BHHg6KPAygqSr2iVsqomhm71c0JatP0kPOuf+e8vEKPiEDQELq7PXUmmtTfdYvOrS72BmzIMxsv6RN/szXJkVnu8YUDeiONClmK5GLgTLpai4mD7eGPAz0FPrECU2XMPgEOaBoX9s5SR81szFFsw5+1PYDBiyeUdaCLlL5C4vQ9u7HL3TBGLpFoY1LWivpYb+sbEzRmbBlkjY1u3NZc2U38vyxB+4IFn/5dbcGK0R47mP7dOTay4LElqRVux/VC7dcEyz+ijsf1E++/WiQ2KddeFmwIodSVOjw2M6PB4m9fMPHOro/uTiojvKwRJ+4kbL3nWh79+PTJ+6uJQcQzOx2RbMOzvb73caX71FUTPHdYZsHoIyKPvWq7JxzeyXtTfy+1sze5Jz7Ro7NAlAwoXOxX+8/p2h77611rt8kaVbSSufcZKP7NItTRORhAK2oYh5uVgPhekmjycEDb0xSmH26AJTe/PyyVF9ojZndamavSl5GpxVArZB52MwGJMk5N62oBsBAzfWD/vopSWvNrL/efZrFKSryMIBWhOwT55WHmw0g9DvnTtqE2Dk3K2lZbeIEAHTF1WIQF0C+Nio6WyVFZ7cGa64f8pdL0dT+wQb3aRanqMjDAPKWSx5OtQuDN9f8JgB6URX3vC2YCUlbzez8vBsCoLgC5+E+SUcTv9cWDTwiaWXitmsb3KdZnKIiDwNoKnCfOJc83EoRxRXOuRfqXDUvaY2ZHUpe2KywYidFKJo59Zz+YLGlsG0PHZ+2VzN+6LanRQ2E4EYkPSdp1sxmtDC6LElqtq1umXNl6LYvv+7WoPHPfWxfsNirdocpQhhbceeDQeOfdmG4IpB995w0mTJTnRY7DCVNLjazUUmjda6ajNfPtmhK0ZJXKeqMHlHUSa2KjvKwVO5cSd8pn/hlbnvo+FXrE2eUi4Pk4WYDCMsUrYNodN1Mnct/YckHLHH1zdDxQ1XMpVpuNeMXu+IsAwiBrVSUf+Mc3NYT/tPvP5l5g2KveO3Fev69VwWJfdYX9pa+sviPb31HkNivvOPLeuL864LElqRLvvuAnvntK4LFP+/vHgm6k8HzfxhuVvxZfzWtl+54f5DYZ9z6mY7unyYX+45pK53TOS0+s3WkJs6sme1IrKWdVdSBrXefhnEKrKM8LNEnXio+feLuxi57/Cr2iVvMxbnk4WYDCGNNrgeAkzADISzn3Ia82wCg+ALn4h2S1vuf+xVt9y0z63POxUW41jvnJs1szDk3ZWaz9e7T4LJCIw8DaEUV8/CSAwjOue3t/x0Aeh3lDLqjXiHbZsvIAPSOkLnYOTdjZut9le8551x8Jn6vpHX++n6/Ndi2pe7TIE4pkIcBLKWKebhpDQQAaBczELJjZtdLut059yt1rp7Twv+mZZLmzWzYOfelrjUQQGGFzsX11uE659Ylfp5q8T7t1FboOvIwgLSqmIeXzXe39DknJoHySJ3x/te5w6k+67/17BQjDzXM7CFJB51zf1znuuOKCnnNKnq9Nkt6lXPurU3CkouB8uhqLiYPn4w8DPQ8+sQJXZ+BUObiGWWNH7IYjRS2IE2Zn/fQ8YtcMOZ4hu2A1ku6fYnrDzjnnpYkM7tPLRQ/C50PfvAbVwaJ/Zp/eFjHHumsqNxSll/xfv3728PtBHDOVx7V/zp3OEjs33p2Sn/52vcEiS1JH/j+3Xpo9buDxb/68H36118P87755X98WH+7+oYgsSXpdw7fq2//6jVBYl/4z53tfEEuzkzmeViiT5xHfPrE1YxPn7i7WMIAIHPz6QdqcbLavXmTxmuum1W1tkkD0AFycWbIwwBSqWIeZgABQOaOMzEzS7OSrpL0RO0Vzrk/r7loUDV7kQPoXeTizJCHAaRSxTx8St4NAFA9x7Us1RfqmpT0X8zs/KVuZGZrJN0mX2UXAMjDmSEPA0ilin1iBhAAZG5ey1J94WTOua2KqnzPmNkf1buNmb1f0n5JR51zd3SzfQCKizycDfIwgLSq2CdOtQuDmb1L0kbVWePVpOpsBSdxAJWVOnvtWb0x1Wd96PCOYmfMnJjZWZI+I+l6Sc8p6qTOSer3X2dLmpJ0k3Pu+RZCkouB8uhqLiYP10ceBnoafeKEtmsgmNlHJE0oSpJDiqZ1rZQ0LGlP0wcscfXNssan7fnGD1XtN2SlX6mzirNFHzktG98ZHTGzAUljrVbxsAAAIABJREFUktZIWqdone0uSRPOuUOtxgtdgfqJ868LEvuS7z6gl//h3iCxJen037hBL9z8tmDxV9z1VU2dd2OQ2MPP3KOvB9rhQZLe/OyU7gq4y8PN379bL350JEjsMz+xS7efH67tt333bv3wN68IEvvVf/9IR/cnF2cn6zws0SfOIz5tzzc+feJqSFNE8TZJg865h83sO865myXJzIYVFZgB0OOquGVNETjnZhR1XAGgKXJx9sjDANpRxTycpgbC2ZIO+p/nzOxi//MBSRsyaRWAUjue8gsAkB3yMADkq4p94jQDCDOS1vqfd0raamYXKNoHFwAAAAAAVFDbRRTN7CpJ/c657f73g5IuUFRcYlOTyrMUjAHKI/Wirb9dfUOqz/rvHL63egvFiolcDJRHV3MxebhryMNAedAnTmi7BoJzbq+kvYnf15rZm5xz32jpASks0jB+mQuL8Lp2P3432p7W8cKmPEjhiygevvzyILFX79unn3xzb/MbpnTaG6/Si38abiXemX+2Uw+vDhP/ysM79dXV7w4SW5Ledvg+PX3JULD4FzyxRy9+LEz7z/z4fcGKV0pRAcuj77wsSOyVX3q0o/uTi4utzP0D+sT1lb3fV9b49Im7q+0lDH4Lx0Wcc98wsyvN7MpsmgWgzI5rWaovtKZeHvaXX0UeBhAjD4dDHgbQiir2idPUQNjV4PJlirZ3BNDj5lN+oWWN8rBEHgbgkYeDIg8DaKqKfeI02zg2GhLpkzTQQVsAVETRq8dWAHkYQFPk4qDIwwCaqmIebrmIopkd19IDIsskHXDOXbrEbYo+oAJgQer5U1Pn3Zjqsz78zD3FnrOVs4zysJrEAFAsXc3F5OGlkYeBnkSfOKGdGQhDip68hySNSJqruX7WOXeo6QOWuHhGWeOXve2hC76V+bkpasGY0L0iMxtWlIMGnHNbl7jdpqWuL6FM8nDoz9S/vz1MQblzvvKoju38eJDYkrR8w8d07LO3hov/R3cEK3T4tsP36S9f+54gsSXpA9+/W/tWjwSLf/nhXfrxre8IEvuVd3xZdwV8bm7+/t368UfeGST2K//8Sx3dnyPUIDLJwxJ94jzil73t9Im7HzuOn1YV83DLAwh+9wWZ2bRz7v5wTQJQdiGna5nZgCQ556bNrN/MBpxzM3VuNyip2RmgUiEPA2hHFafO5o08DKAdVczDabZxvDpEQwBUR+AtazZK2uN/npU0KOmkAYQqIw8DaEUVtw8rCvIwgFZUMQ+nKaIYb12zUVGhmEWcc2/ttFEAyi3w9jN9ko4mfl9VewM/K2HazMZCNiQvjbYPiznnvtittgAorqJvBVZm5GEArahiHm57AMHMPqJoe5opRevAJiWtlDSshbOCAHpYAdZ7rcy7AYFNNbg8fup/oVsNAVBcBcjFVUYeBtBUFfNwmhkIt0kadM49bGbfcc7dLJ0oanZVszt3UoQiz9hlj1/mtp96Tn+w2FK5n5vQbU8r7XQtMxuVNFrnqknn3KT/eU4LAwR9ko7UxBhwzk2na0E5OOdOqb3MzPoUDej+t2b3D/2ZOucrjwaLvXzDx4LFlqJChyG97fB9wWJ/4Pt3B4stRYUOQ3rlHV8OFvvmwM9Np8UOQ6ni1Nmi6DQPS+XuH5Q5fpnbTp84n9idqGIeTjOAcLakg/7nOTO72Dn3pKQDkrZJ+uOl7hyqemiZK4eGjk/bqxm/yBVn0xaM8YMEk01utkPSev9zv6RpKeq4OefmJPWbWfwftmGRxapxzs2Z2bikuyQtuZTs5a9/IVg7Tn/zezW38Yogsft2PBK8AvWP/mgoWPxXfXaPnr4kTPwLntijqfNuDBJbkoafuUdPrbk2WPyLDu3WC7dcEyT2ijsf1J7VG4PElqShwzuCvuc7UcXiXUXWTh6W6BPnEZ+2VzN+FfvERXbS6GkLZiSt9T/vlLTVzC6QNJ5VowCU23zKr1bEgwF+l4W5xODAXn/9lHNuStEshZPqtPSAwbwbAKAYQuVhNEUeBiApbJ84L2lmIIwrGkB42Dm31RcpO6hoT9xNWTYOQDmFnq6VWM6QvGxdnds0m81QSma2o87FfYpmZNTuSQ6gR4XOxX756pykAefc1iWu73fOTfpteA8o2kFHkqadc2NmNuGcGzez0Xr5vYjIwwBaUcU83PYMBOfcXufc9sTvayWtd86d4pwLu3gUQCkcT/mFli2r8/W8olkYnPkCIClsHvadUPmaM3Px7zXXz/rrZ/3vK51zy3zfcURRUW5JGjWzg1ro0JYBeRhAUyH7xHnl4bTbOF7gnHs6/t05941W7xuy+EeZC3+Ejk/bqxm/qAVjEJZzbkMn9z/9ze/Nqil1dbpueymhC0i96rNhNxO64Ilw8YefuSdYbCmqUxDSijsfDBZ76HC9k8XZCfmeL7CNWth9a1bRQXNtvZkJRTt29dcpbtvvl5tJ0kjZit92mocl+sR5xaft1Yzfo33iXPJwywMIZrZF0VaN/f53KZoOsUPSbc65H7X0gCUunlHW+D//6Q+CFx7jde1+fArGIK3Q+eDItZcFib1q96N6+fH7g8SWpNMvvV7HPr85WPzl79sSNB985w1Na7al9vpvfS3Y6ypFr+3P/u1fgsQ+9Zd+RS98KFwByBWf2q2XPjUWJPYZH9rW0f0D5+I+SUcTv69KXumcmzGzWX9Ga1GdLF/DJtlRHfD9yrpTcKuqzP2DssanT1zN+D3cJ84lDzcdQDCzKyXtUjQ1a6ekrYpGOPolvV7STZLGzGzQOdeTQ/AAFpuv4JY1RWNmt0oakx/UVTTi/N+cc8XcTw5A16XJxS1up9tKnD5FJ5q2SdpuZjPOufjIbSh5pivurJrZkO9PlmI2AnkYQDNp+8RZ5OJQeXjJAQQzWyNpStJO59zNNVfv9d/HzWxC0rSZ9TvnvtvKHwSgupiBEJaZ3aVo8PZ+RYO6krRB0pQvfvPZ3BoHoDDS5OI2CtDOKdrtRorOgh2puX5U0ha/teGMolmscb46sU7XF/iSn0Z7RAsH44VGHgbQisBbm+eSh5sVUdwmaU+dwYNFnHPjkr7obw+gx1FEMbgNkrY65zY457b7ryFJn5F0W85tA1AQgfPwDi10Mvvlp8L6M16LxAW+/PW1uxTMamEa7VpJ+9trRm7IwwCaCtwnziUPNxtAWC9pS5PbxLYoKtAAoMdVcc/bgjmq6J9GrdtVs/4NQO8KmYedczPSiXW0c/Hv8jNU/XTYUTMbrrMt2NGaOBv8GbCDiThFRx4G0FTIPnFeeXjZ/HzjJprZfzjnfqHFv6GV23OMAJRH6koGn37de1J91j/4vbupntACM/uIpPnarXPN7P2S1jnn/rhJCHIxUB5dzcXk4daQh4GeQp84IdU2jh09IFVV6yp7ZVLa3v34PVxxFtFsr6vMrHbW16CiejRfS17onFtUnr+subIbef7YA3c0v2FKy6+7Vc++5fIgsc99bF/wXRJeuOWaYPFX3PmgfvLtR4PEPu3CyzR345VBYktS3z0P69jOjweJvXzDxzq6P7k4qI7ysESfuJGy951oe/fj0yfurlZ2YVjhnHuhhdudlU2TAJRdFZNlwcwpKtxVK9wehwBKh1wcFHkYQFNVzMPNBhCWSZrze0ICQEuYlxmWc25D3m0AUHzk4nDIwwBaUcU83GwAYawrrQBQKccLu2oLAHoHuRgA8lXFPLxkEcUAqjgIA1RV6pR3+/npCsbc9t3iFowpGjO7QNGWPfW26vlik7uTi4Hy6GouJg+3jjwM9Az6xAmVKqJY1sIfoeN3o+2hivWUuVBP6PhFLhhDrygsX/37dtX/hzYvacndc0IX1/rOG06qFZaJ13/ra3r5f9fbNS0bp/+njTryOwELEf7to/rXXw9TzO+X//Hh4PngH1/9rmDxf/2HX9QLH3h7kNgr/vIreuzckSCxJektz+7Sv781zPvmnK91VliSXBxOp3lYok+cR3z6xI3xui4dP60q5uGuDyAAqL7jlUyXhbJZ0p9L2uKcez7vxgAoJnJxUORhAE1VMQ8zgAAgc1WsOFswRyXdRacVwFLIxUGRhwE0VcU8fEreDQBQPfMpv9CybZKG824EgGIjDwdFHgbQVBX7xMxAAJC5Ko62Fsylkq43s82S9ivaj/wE59zGXFoFoFDIxUGRhwE0VcU8zC4MABpJXf31v55/Y6rP+n/97j2FrThbJGa2c6nrW9ifnFwMlEdXczF5uDXkYaCn0CdOYBeGHohPxdnGeF2Xjp9WFfe8LZIWOqZLYheG+tiFoTF2YWisyLswkIvD6TQPS/SJ84hPn7gxXtel46dVxTzMEgYAmatixVkAKBtyMQDkq4p5mAEEAJmrXqoEgPIhFwNAvqqYhxlAAJC5KhaMyYuZHVeb/3+cc78QqDkASoRcnA3yMIC0qpiHKaIIoJHUq7bGL7gh1Wd94ul7K7hSrDNmNqr2O67bm9yEXAyUR1dzMXn4ZORhoOfRJ06giGIPxKdgTGO8rkvHT4teUXacc5NZx6SIYn0UUWyMIoqNFbmIIrk4GyHysESfOI/49Ikb43VdOn5aVczDLGEAkLkqTtcCgLIhFwNAvqqYhxlAAJC5KlacBYCyIRcDQL6qmIcZQACQueqlSgAoH3IxAOSrinmYIooAGkldvOWDF7w71Wf900/fV9iCMRVDLgbKo6u5mDzcNeRhoDzoEydQRLGN+KELj5X5uaHt3Y9f7IIx9IuK7Cf//HfBYp/2q7+t5997VZDYZ31hb/A8HKqQnxQV83tqzbVBYl90aHfwIod3vvY9weLf8v279dz1lweJffb9+7Rvdbgiipcf3qUX/vPvBom94n/8TUf3JxcXW5n7B/SJux+/zG0PHZ8+cXexhAFA5qpYMAYAyoZcDAD5qmIeZgABQOaqWDAGAMqGXAwA+apiHmYAAUDmqpcqAaB8yMUAkK8q5uFT8m4AgOo5rvlUXwCA7ITOw2Y2bGaDZrapyfWjicsm/PfRVuMAQFmF7hPnkYfZhQFAI6mrv950wUiqz/r2p3cVtuJsxZCLgfLoai5uNQ+b2YCkfufclO+E7nfOzdRcL+fcjJkNSjrqf35O0lFJY8656WZxKow8DJRHIfvEeeXhSu3CUNaKsFL5K5PS9u7Hp+Is0iprruxGnj/2wB3B4i+/7lY9+5bLg8Q+97F9OnLtZUFiS9Kq3Y/qhVuuCRZ/xZ0P6ifffjRI7NMuvExzN14ZJLYk9d3zsI7t/HiQ2Ms3fKyj+wfOxRsl7fE/z0oalFTb4ZyQNKSoYzrtLxtJ/NxqnEoqc66k79T9+GVue+j4PdwnziUPs4QBAAAA7epTdAYrtip5pT97NWtmB2tuN1AzTXbJOACAhnLJwxRRBJC5Km5ZAwBlkyYX++mro3WumnTOTbYRp0/SnKRtkrab2YxzbtY5t9VfP+Sn1AJAZaXtE2eRi0PlYQYQAGSOJQwAkL80udh3TFvpnM5JWul/7pN0pOb6UUlbnHNzZjYjadjMZv1jTPnb97cQBwBKK22fuMVcnEse7voAQidrSJo59Zz+YLGlsG0PHZ+2VzN+6LanxQyEYitzrgzd9uXX3Ro0/rmP7QsWe9XuMDUEYivufDBo/NMuDFfDoe+eh4PFljqvVRBK4Fy8Q9J6/3O/pGkpOuPlnJtL3tAX6epXtLY2Xpy/VtFZsf314vSCMudK+k75xC9z20PH79E+cS55uFJFFMta+CN0/LK3nUJA3Y8dx0/reHd3d0Gbju1teRZy25ZfNapvvO4dQWK/6XtfDp4PDl8R7iB29SOP6ltrfydI7Dcc/Fv9/XnXB4ktSb/5zP3B881Ld30wSOwzbv60Dl08FCS2JK15co++/athCkxe+M+dDdqEzMW+kvd6P/11LlGxe6+kdc65rWa2yZ/tWhlPuTWzUTM7KulgfJ8GcSqvzP2DssYve9vpE3c/dhw/rSrmYZYwAMgcwwcAkL/QubjeOlzn3LrEz1tbvE+4UU0AyFEV8zADCAAyd5whBADIHbkYAPJVxTzMAAKAzFFEEQDyRy4GgHxVMQ8zgAAgcxRRBID8kYsBIF9VzMPL5rtb7Kx6QzBAdS1Le8eR89+R6rO+67tfTv2YaAu5GCiPruZi8nDXkIeB8qBPnMAuDG3EL2vl07JXnC1r20PHL3LF2SpO16qS0LnssXNHgsR+y7O7wu8E8MmbgsU/48Pb5S58e5DY9u2vaM/qjUFiS9LQ4R169i2XB4t/7mP7dOwzHw4Se/n7P6nZi64OEluS+p96SHM3XhkkdqfbT5KLi63M/QP6xN2PX+a2h45Pn7i7WMIAIHOhp2uZ2bCkOUkD9arLmtmo/3Gtc248cHMAoJCqOHUWAMqkinn4lLwbAKB65ufnU321wswGJMk5Ny1pLv49cf2gpGm/HU2//x0Aek6oPAwAaE3IPnFeGEAAkLnjmk/11aKNimYfSNKspNoBgv7EZbP+dwDoOQHzMACgBYH7xLmgiCKARlIXb7n2db+b6rO++3t/0/QxzWybpG3OuRk/u2Co0TIFM9sjadw5N5OmPRVGLgbKo6u5uJU8jEyQh4HyKGSfOC8UUeyB+LR96fgUAmocP620BWN87YLROldN+iUJ7cQakLSHwYOT/fSH3wwW+xWvfqOe//2rgsQ+63N7g39efzT21mDxX7Xta3rxE+8LEvvMj35eT625NkhsSbro0G4989tXBIt/3t89EjRX/v/t3W1sZFd9x/HfRQ2wLFBniboJIYLMVj1RIwryZltVQNgkdngSLxB2tig8VGrWlhCv8iKrCDUvUqGV90VeISTvUqk8rGDjRbxATSC2kyWlQnRjCxqlyqnqCSgN2a2a9fCwpAQpty/umd2b2RmPPXP+cx/m+5GubM/Df+5c3/n56Picc58/YHNOStINZ1d18cG7TWrvfuDkUM+v4+JddVLl9kFV67PvW9enTdy7/qDqmMMsogggukGHXoVOgn4dBS1Je8L3E5Je6vG4qW4LLALAuCj7MFgAqLs65jBrIAComlO6vK5BQ9KKJDnnJtoPcM7NtTsPWEQRAAAAiIMOBADRWa44256SEDoGWrkpCqu52xeccxvOuU2L9wcAVVC3lb8BoGrqeBUGpjAAiM76mrfd1kPw3u8PX1ckXW28CwBQenW8/jgAVEkdc5irMADoZeDVX++84cMDfdYfe/77pV1xtmbIYqA6RprF5PDIkMNAddAmzuEqDGNQn32vZ/1RrJY7qDouGFMnrzT/zaz26xt/qYtfnDWpvftLS+bn/MuPfcWs/q47P69ffcboChXfWNVz75k2qS1JN/5sWedv+6BZ/b1P/FC/f2bVpPYbbr5Dv73/kya1JenNR79jmvPDIIvLjbbT6Ouz7/WsT5t4tJjCACC6ss/dAoBxQBYDQLHqmMN0IACIro69rQBQNWQxABSrjjlMBwKA6NIahiUAVA1ZDADFqmMOs4gigF4GXrzl1uvvGOiz/uQLq6VdMKZmyGKgOkaaxeTwyJDDQHXQJs5hEcUxqM++b13fetGVKh+bQdEqKrdXfvmMWe3Xv/1m/epzRgsFfm3V/PP66/kPmdV/6+IPdPFLnzWpvfuLX9fTN37cpLYkvfu57+nF999mVv+6Hz1hmpXPH7A5JyXphrOruvjg3Sa1dz9wcqjnk8XlVuX2QVXrs+9b16dN3Lv+oOqYw0xhABBdHed7AUDVkMUAUKw65jAdCACiq2NYAkDVkMUAUKw65jAdCACiq+MlawCgashiAChWHXOYDgQA0dWxtxUAqoYsBoBi1TGHuQoDgF4GXv31wNtvHeizfvaXT5Z2xdmaIYuB6hhpFu8kh51zM5Jakia998e2uL/hvT8ebpsLd+/z3h8Jty1474845+bajxsD5DBQHaVtExeRwyMfgWC1uqflyp4Sq7YWVb/K+25dv9QrztZwuFad/N+/fMOs9hs/8Bm1Dtms1j9x6gn7qzD83bRZ/bf+47J+/l6b+u/66bJOX2dzJQBJmnnxpPlVHn7zhY+a1H7Llx/R8t5DJrUlafr8KdNzfhiWWeycm5Qk7/2Kc67hnJv03q933N/03q8756bCz3skrXjvm865JefclPd+RdJcaOTOm+1wCdEmHn199r2e9ce1TVxUDr/O6P0AGGOvKh1oAwDEY5zDh5T9V0uSmpKmujxmIXxthEZtI/e4ZvhZkma99/tCIxYAasO4TVxIDtOBAAAAgJ2akHQh9/Pb8neGhmrTObfRfpz3/nhuaOykpKfa34f/jt1nvM8AUCeF5DCLKAKIjikMAFC8QbI4zI2d63LX8Z2sT+Ccm1D2n7FFSSecc+ve+2a4b1LScnuobXvernNuOjecFgAqb9A2cYwstsrhkXcgXHVNo/+DBjTM/JS612ff61nfet8HxXSEcnvjBz5jWn/YedtbsfwbImXrFFh610/t6s+8eNKstpStU2DpLV9+xKz29PlTZrUl23N+GINkcWiYbqdx2lI2l1bK/gv2Usf9c5KOeu9bzrl1STOS2gt8TeUaqzPhdU+HGrYf8hKhTVxMffa9nvXr1ibeZhYXksMj70Co8uIZ1ot3VfnYsO+jr1/qBWPoQCg16yx78tpZk9q3nlsyP+d/99Bhs/pvuveE/E0fMantnn3UfKHAc7ceNKt/7ZNn9PJX7zWpveueh9R8950mtSWp8fRjat19u0ntiZOPD/V84yw+JemW8H1D0oqU/cfLe9/KP7C9wFe4fy7XaJ1SNge3HUr7lP2nbCxUuX1Am3j09au879b1x7hNXEgOswYCgOheTdOBNgBAPJY53B72GhqfrdzK36vh/mMKq3q3LwsWHrvgnNtwzm3m6twV/gO2kV9BHACqzrJNXFQOswYCgOgYgQAAxbPO4m7zcL33+3PfH+u4b0XS1dupAwB1UMccpgMBQHSMJgCA4pHFAFCsOuZwMuLV0ut3BIH6SgZ94k1/cmCgz/qz/3N24NfEjpDFQHWMNIvJ4ZEhh4HqoE2cwyKKY1Cffd+6PgsB9a4/qDr2ttbJK798xqz2699+s371uTtMav/x11bNP6+/nv+QWf23Lv5AF7/0WZPau7/4dT1948dNakvZFRhefP9tZvWv+9ETpln5/AGbc1KSbji7qosP3m1Se/cDw11Zgywutyq3D6pan33fuj5t4t71B1XHHGYKA4DoWAMBAIpHFgNAseqYw3QgAIiujr2tAFA1ZDEAFKuOOUwHAoDo6tjbCgBVQxYDQLHqmMN0IACILk1fLXoXAGDskcUAUKw65jBXYQDQy8Crv77zbX8x0Gf9Fy/9e2lXnK0ZshiojpFmMTk8MuQwUB20iXO4CsMY1Gff61l/FKvlDmrEHZPYoT+c92a1r9rrdP7gQZPae8+cMT/nf/OFj5rVf8uXH9H3rv2USe2Pn/uWNj950KS2JF39nTM6e/0nzOofeOG7plfv+KfrP21SW5L+9oVv6rn3TJvUvvFny0M9nywuN9pOo6/PvtezPm3i0WIKA4DoXuUfKwBQOLIYAIpVxxymAwFAdHXsbQWAqiGLAaBYdczh1xW9AwAAAAAAoPxYRBFALwMv3nLdxJ8P9Fl/sfUfpV0wpmbIYqA6RprF5PDIkMNAddAmzmERxR3Ut16co8rHhn0fff1R7Pug6njN2zqxzrInr501qX3ruSXzc/53Dx02q/+me0/I3/QRk9ru2Ue1vPeQSW1Jmj5/SuduPWhW/9onz+jlr95rUnvXPQ+p+e47TWpLUuPpx9S6+3aT2hMnHx/q+WRxuVW5fUCbePT1q7zv1vVpE48WayAAiK6O870AoGrIYgAoVh1zmA4EANHVccVZAKgashgAilXHHKYDAUB0dextBYCqIYsBoFh1zGE6EABE92oNwxIAqoYsBoBi1TGHuQoDgF4GXv316jf/6UCf9c3f/ldpV5ytGbIYqI6RZjE5PDLkMFAdtIlzuApDiepbrWhruZqtVP1VVatav8wrztZxvledWK+eff7gQZPae8+c0e+fWTWpLUlvuPkOXfz7u8zq7/6Hh/X4Xpv6t59/WN/f+zcmtSXpw+e/rZ+/d9qs/rt+uqyLD9js/+4Hv63T191tUluSZl48qQuf+KBJ7T3f/eFQzyeLy63K7QPaxN1Vvd1X1fq0iUeLKQwAoqvjfC8AqBqyGACKVcccpgMBQHR1nO8FAFVDFgNAseqYw3QgAIgureFwLQCoGrIYAIpVxxxmEUUAvQy8eMuuXe8c6LP+8su/KO2CMTVDFgPVMdIsJodHhhwGqoM2cQ6LKI5BfcvFaCTbBWmqfNyt65d5wZg6zveqE+s8eOGvbzepff2PH9fLT3zVpLYk7brtHv3vR2wWw5Okax79of712hmT2u87d1pfueHTJrUl6fPPf1OPGS7SeOf5b+u//8rmvHnHTx7XP+/9lEltSfrY+W/p2T/7qEntm/7zkaGeTxaXW5XbB1WtT5u4nvVpE48WUxgARGc9XMs5NyOpJWnSe39sp/cDwDgoURY3vPfHez2HzAZQV3XM4dfFfQsAkPW2DrJth3NuUpK89yuSWu2ft3s/AIwLqxyWtp3FzXB/0zk32e05ZDaAOitBmzh6DtOBACA6y7CUdEhZD6kkNSVN7fB+ABgLlh0I2l7WLoSvDe/9eo/nkNkAaqvgNrFkkMN0IAComglJF3I/v22H9wMAhrdl1oaGatM5t5F7XLfnkNkAMJhCcnjUayAkwyxC0Y9l7arXv+qahlltyXbfq3zcretb7/ug/vDKCwOtHOucm5M01+Wu4+15WxjeVdc0TFf2vf7Hj5vV3nXbPWa1pWyhQ0vvO3farPbnn/+mWW0pW+jQ0jt+YnfefOz8t8xqS8MvdmhlkCyOlcPOuQll/9FalHTCObe+032pOdrEBdWnTVzP+rSJu9YwyWEWUQRQGiEQ+4ViS9Ke8P2EpJd2eD8AoIdt5rDUP2vnJB313rdCo3VkVFy1AAAKbklEQVRmi+eQ2QCQE6lNbJLDTGEAUDWnJLX/fdCQtCJd6mXteT8AIKp+WXxJe2GuHs8hswFgMIXkcCk7EJIkmUySpNuQjRi1G0mSrCVJshm+Rh3HFOovJUmyETYWAwrCsVkOx2Wt6P3ZiSRJJnK/17XYv9ckSRZztYc+J3t9hsJ72Ay/h0qudB3mc8k5NyWp1f5Z0mqf+7FDVllMDherqllsncPhNcjibdpGFh+TNOecm3HOzXnvj3d7Dpm9NdrE9VTVHJZoE5dJYTk86MqQVpukSUkbkuaM6m9IaoTv75O0KWkicv2p3HtJY9bP1d2UdF/kumthfzdzW5R9V9abtZk7NjGP+UzHPre3pcjHZq7jvTQi1V6StJB7LxsRzo8rPkPhdRbD91OSNmOeP2z12iyzuA45nKsdNYstczjUr2wWW+ZwqEkWs5Vqs8zhUL/yWWyRw6EubeKtjw1t4jHeCt+Bjl/moqTlcCJaNFqn2idL7ra1mK/V+QEK4TMZ+X0shxPfIiyj7muu9qVAMKo/0bEtSZqJWHuz47bFGOdNCN6047ZLf3AHqNfzM9TlddZiHSO2em2WWVyXHA51o2exZQ6H+pXMYsscbp8zZDFbmTbaxNt+DdrEV9anTZySw5ZbqRZRTNN0XpKshvKkabqSJEmz4+ampH0RX+NS/faQnjRNow3HC0NwliUdiFXTWpIkE8p6EaMd505pmravXaokSWbCbXbLnWc6z6VBTHapsx5u3/E80F6foXAudr7OU8rOI+vjhIqxzOI65HCoSxZ3UUAWx8hhiSxGydAm7o8c7o42cYYctlPKNRAs5cMsaA9tiSbMtVlQ1uM3HbHuhKQjaZoei1Wzi4UwH2gj4nyg9vylydxcu/si1X6NcIxOSDocq2YI4ofb+xyOy0aaplYLPV1Q/D8s7cu45LV0+XcDjEyVczjUts5iixyWKpzFBeSwRBaj5qqcxbSJ+6NN3BM5PKSx60DIy/XKxb6+fEPZydlU3JPxhKSFiPW6aaVperWkeUlrIXyG1T4GB9I03S/pDmWhbLFgyf2SHs73vsYQejEPJUmSKhvmFKv+uq48R27R5UupxNKtHpfKQuEqmMOSfRZb5LBU8Sw2zGGJLMaYq2AW0ybujzZxd+TwkMa2AyGs6rmgyP+ZkrJhQmmazodgWGiH8jDCcJtJg2DPm03TdFbKhrYpC/u7ItRtB8vRUHtd2XVND0Wo3WlO2ZynqMLwp1NpmiaS9ktajLHqbOj9byZJshheZ0bdh3AN64KyP+BAaVQth6WRZLFVDksVz2KrHJbIYoy3qmUxbeJto03cHTk8pLHsQAg9iEuSprsM34ptRXFCYVbSnjDUaVPZ/KmFJEmiDTXrcixaijNsqKmsFzffQxl9qJDhXOcpZQsBHcvVP66sRzqGaUm3hEBuKJt/FXUIobLj3dnjuk/xQxnYlormsGScxYY5LFU4i0eQwxJZjDFU0SymTdwHbeItkcNDKtUiiiO0Kulw7KBMkmSiyzChCWVDcoYShgtd+nAmSbIsadl47ldDET60aZo2k+yasfnj01D8D+qkskVQLFzocluU/Q/n4f72z0mSbLZ7vWMJiyV1/g5uUegBBwpQuRyWCsniKDks1SKLzXJYIosxtiqXxbSJt4U2ce/XIIeHNHYjEELIHIndIxc0kiRZbs9jCr10UzIYPhRbWORmOQxja69sG3Mu3DGFuWrh+Ewp/gfVpPcwDF3b0z4m4RjdpUi/1/y8tyRJlmQXYMd1+Xcwo+y6w6w2i5Ejh7sbQQ5LFc1i6xwONclijBWyuDvaxL3RJoYkFX4dyfym7Be5qew6sZthi3YNVmUf0LTLthj5NdbCvq9pwGuXbuN1lhX/mrdzynpXN0P9RuT6i6H2Rszfa8cxMbmurrLe4WWL32vuvN8Y9nfa7zOkbJiiye+XrT6bZRbXKYfDa0XNYuscDq9RySy2zOFQnyxmK81Gm3hHr0ObuPsxoU1MDptsSTiAAAAAAAAAPY3dFAYAAAAAALBzdCAAAAAAAIC+6EAAAAAAAAB90YEAAAAAAAD6ogMBAAAAAAD0RQcCAAAAAADo64+K3gFsn3PuPknzyq6/2pLUlHTUe3+6oP1JJU1771eKeH0AGDVyGACKRxYDxaEDoQKccxOSVpWF5FFJK5L2SJqVdEBSIWHZyTm3JumC93666H0BgJjIYQAoHlkMFI8OhGo4oSwo93vvm7nby9bLuVj0DgCAEXIYAIpHFgMFowOh5JxzDUkzkmY7grJ0vPfHY9d0zk1JWvbeJ7FrA8B2kMPkMIDikcVkMcqBRRTLb15Sq6g5XQAAchgASoAsBkqAEQjlNynpqe0+OLeozB5lw7kOe+9b232Mc25J0gVJS5KOKOvpPBbmnJ1Q1vO7ri5Ds8Jz5b2fzf8s6Wzu9Y5674/lnjMlaSG8z1b+fufcsqSp8H0anjLb/sPhnFuQNBduP+69P5J7f/dLmlC2qM5i/jUBYIfIYZHDAApHFossRvEYgVB+7dVl+3LOLSoLpXlJ+5WF3nMh6Lb9GGUBtRS29hCspXD7bNj2bXP/Z8LX6VBrIQxBy7+/I6HebLh/RpLCwjOz4fskbO2gXAq12/sz45yby4XvHaHmvLLABIBBkcMihwEUjiwWWYziMQKh/JrKeg23FAJoTtK+3LyweefcLcrCY347jwm3NfKPCc+b0msvT3Mk9Gr208r1nh6VdF+o35SumCPWdM6dlnRIl1fRveIPRW4O3H7v/Xq47YiyHtaj4TnN0INMUAIYFjnc/b2SwwBGiSzu/l7JYowUHQjlt67LQ5K2MqUsmDrDYUWXezy38xhJWu94zKQkDXht20tDzbz3Leec1BH+IXQPhdsb6r+S7lT4uhbq5a0o60HedM6tKAw3G2C/AaCNHL4SOQxg1MjiK5HFGDmmMJTfKUkT7SFMIxKzh3LLoWbOuQ1lQ7kOe+/36fLwsL51c0O48lsr1JkNr72Qm3cGAIMgh3vUJYcBjBBZ3KMuWYxRogOh5MJwpNOSTnTMk+r0lLJQney4fUqXey+385hu2kOi2r2c6pgfNpCwHw1J8+1hV8oWlelnRd3fxyXe+9Nh4ZppvbYnGQB2hBzuihwGMFJkcVdkMUaOKQzVcFjZgi1rYc7UirJQmZY05b3f771fD3Ollpxz88qGLN2vLIzukLLg7feYbrz3zTD0aTH3vIVh31TYHymbO9ZedXZGrw3u9pyzGYWeW+/9inPueO59NMNz55WthDutyyvismAMgBjIYXIYQPHIYrIYBWMEQgWEIUjTyhZDOSRpTVl4TioL0vbjZpX1zC5KWlU2f2p//pI123lMD7PKQmdJWVBeccmaAR2RdJey9zStjl7fMO9sPfe67dvbwbgoaUNZ6C+Gx+5R9t7WlP0hmI20rwDGFDlMDgMoHllMFqN4SZqm/R8FAAAAAADGGiMQAAAAAABAX3QgAAAAAACAvuhAAAAAAAAAfdGBAAAAAAAA+qIDAQAAAAAA9EUHAgAAAAAA6IsOBAAAAAAA0BcdCAAAAAAAoC86EAAAAAAAQF//D1Wl6o7yRdhZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1296x360 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualizing input gradients of standard and robust models\n",
    "fig, axs = plt.subplots(1,3,figsize=(18, 5))\n",
    "xticks = np.arange(1,11)\n",
    "\n",
    "# plot dataset\n",
    "sns.heatmap(np.abs(test_dataset.X[:50]), ax=axs[0], vmin=0, \n",
    "            vmax=1, linewidth=0.01, yticklabels=False, xticklabels=xticks)\n",
    "axs[0].tick_params(which='both', axis='both', length=3)\n",
    "axs[0].set_xticklabels(xticks, fontsize=13)\n",
    "\n",
    "utils.update_ax(axs[0], title=r'Input Data', xlabel=r'Coordinates', ylabel=r'Data Points', \n",
    "                ticks_fs=15, label_fs=18, title_fs=22, despine=False, legend_loc=False)\n",
    "\n",
    "for ax, (model_name, G) in zip(axs[1:], grads.items()):\n",
    "    # normalize gradients to improve visualization\n",
    "    G = np.abs(G[:50])\n",
    "    G /= G.max(axis=1).reshape(-1, 1)\n",
    "\n",
    "    # plot gradients\n",
    "    ax = sns.heatmap(G, ax=ax, vmax=1, vmin=0.8, linewidth=0.01, yticklabels=False, xticklabels=xticks)\n",
    "    utils.update_ax(ax, title=labels[model_name], xlabel=r'Coordinates', ylabel=r'Input Gradients', \n",
    "                    ticks_fs=15, label_fs=18, title_fs=22, despine=False, legend_loc=False)\n",
    "\n",
    "    ax.tick_params(which='both', axis='both', length=3)\n",
    "    ax.set_xticklabels(xticks, fontsize=15)\n",
    "    \n",
    "sup = fig.suptitle('Input Gradients on Synthetic Data', y=1.04, fontsize=23)\n",
    "fig.savefig('plots/feature_inversion_synthetic_data.pdf', dpi=fig.dpi, bbox_inches='tight', bbox_extra_artists=[sup])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three subplots above visualize the input data points, the input gradients of standard fully-connected network (FCN), and the input gradients of adversarially trained FCN respectively:\n",
    "- The first subplot illustrates data points sampled from the synthetic distribution; the columns and rows denote input coordinates and individual data points that are sampled from the synthetic data distribution described in Section 5 of the paper.\n",
    "- Ths second subplot substantiates FIG, as the minimum magnitude coordinate (darker in color) of standard input gradients often, but not always, highlights the relevant signal coordinate $j$ of the input data points. \n",
    "- The third subplot substantiates RF-FIG, as the maximum magnitude coordinate of robust input gradients consistently highlights the relevant signal coordinates. \n",
    "- As discussed in the appendix, the mean absolute cosine similarity between input and input gradients of standard and robust models are 0.29 and 1.00, indicating that unlike standard models, robust models’ input gradients are directionally (and visually) aligned with input data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "246px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}