Benchmarks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Toxic Comment Classification Benchmarks\n",
    "Collection of Deep Learning Text Classification Models and Benchmarks; All gpu models contained within the repo were trained with 4 v100's using the AWS p3.8xlarge instance and the AWS deep learning AMI. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image # display images\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import os\n",
    "\n",
    "# ignore warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# import model utils\n",
    "from models.utils import RocAucEvaluation\n",
    "from models.utils import tok_embed\n",
    "\n",
    "# sklearn\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. [Competition Overview](#Competition-Overview)\n",
    "    1. [Description](#Description)\n",
    "    2. [Evaluation](#Evaluation)\n",
    "    3. [Data-Overview](#Data-Overview)\n",
    "2. [Hardware Used](#Hardware-Used)\n",
    "3. [Load Data](#Load-Data)\n",
    "4. [Embeddings](#Embeddings)\n",
    "5. [Models](#Models)\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Competition Overview\n",
    "Discussing things you care about can be difficult. The threat of abuse and harassment online means that many people stop expressing themselves and give up on seeking different opinions. Platforms struggle to effectively facilitate conversations, leading many communities to limit or completely shut down user comments. The Kaggle Toxic Comment Classification Challenge sponsored by the Conversation AI team sets out to discover and apply machine learning to identify toxic comments. This (potentially) will allow platforms to identify toxic comments and to successfully fascilitate discussions at scale.\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "### Description\n",
    "The Conversation AI team, a research initiative founded by Jigsaw and Google (both a part of Alphabet) are working on tools to help improve online conversation. One area of focus is the study of negative online behaviors, like toxic comments (i.e. comments that are rude, disrespectful or otherwise likely to make someone leave a discussion). So far they’ve built a range of publicly available models served through the Perspective API, including toxicity. But the current models still make errors, and they don’t allow users to select which types of toxicity they’re interested in finding (e.g. some platforms may be fine with profanity, but not with other types of toxic content).\n",
    "<br>\n",
    "<br>\n",
    "In this competition, you’re challenged to build a multi-headed model that’s capable of detecting different types of of toxicity like threats, obscenity, insults, and identity-based hate better than Perspective’s current models. You’ll be using a dataset of comments from Wikipedia’s talk page edits. Improvements to the current model will hopefully help online discussion become more productive and respectful.\n",
    "<br>\n",
    "<br>\n",
    "***Disclaimer: the dataset for this competition contains text that may be considered profane, vulgar, or offensive.***\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "### Evalution\n",
    "We randomly select 10% of the training data as the development set. The evaluation metric is the mean column-wise ROC AUC. In other words, the score is the average of the individual AUCs of each predicted column.\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "### Data Overview\n",
    "You are provided with a large number of Wikipedia comments which have been labeled by human raters for toxic behavior. The types of toxicity are:\n",
    "- toxic\n",
    "- severe_toxic\n",
    "- obscene\n",
    "- threat\n",
    "- insult\n",
    "- identity_hate\n",
    "\n",
    "We must create a model which predicts a probability of each type of toxicity for each comment.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "File descriptions:\n",
    "- train.csv - the training set, contains comments with their binary labels\n",
    "- test.csv - the test set, you must predict the toxicity probabilities for these comments. To deter hand labeling, the test set contains some comments which are not included in scoring.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "***Source: Toxic Comment Classification Challenge, https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/***\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "## Hardware Used\n",
    "These models were trained on the amazon p3.8xlarge instance type and a deep learning AMI\n",
    "\n",
    "### p3.8xlarge\n",
    "\n",
    "| Instance Size | GPUs - Tesla V100 | GPU Peer to Peer | GPU Memory (GB) | vCPUs | Memory (GB) | Network Bandwidth | EBS Bandwidth | On-Demand Price/hr* | 1-yr Reserved Instance Effective Hourly* | 3-yr Reserved Instance Effective Hourly* |\n",
    "|---------------|-------------------|------------------|-----------------|-------|-------------|-------------------|---------------|---------------------|------------------------------------------|------------------------------------------|\n",
    "| p3.8xlarge    | 4                 | NVLink           | 64              | 32    | 244         | 10 Gbps           |  7 Gbps        | 12.24 USD             | 7.96 USD                                  | 9.87 USD                               |\n",
    "\n",
    "https://aws.amazon.com/ec2/instance-types/p3/\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GLOBE PATHS\n",
    "TRAIN_FILE = '../data/train.csv'\n",
    "TEST_FILE = '../data/test.csv'\n",
    "EMBEDDING_FILE = '../data/glove.42B.300d.txt'\n",
    "\n",
    "\n",
    "# load\n",
    "train = pd.read_csv(TRAIN_FILE)\n",
    "test = pd.read_csv(TEST_FILE)\n",
    "\n",
    "\n",
    "# fill and store X's an y's\n",
    "X_train = train[\"comment_text\"].fillna(\"fillna\").values\n",
    "y_train = train[[\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]].values\n",
    "X_test = test[\"comment_text\"].fillna(\"fillna\").values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "The word embeddings are pre-trained on much larger unannotated corpora to achieve better generalization given limited amount of training data (Turian et al., 2010). In particular, our experiments utilize the GloVe embeddings trained by Pennington et al. (2014) on 6 billion tokens of Wikipedia 2014 and Gigaword 5. Words not present in the set of pre-trained words are initialized by zeros. The dimension of word embeddings is 300.\n",
    "\n",
    "More info on GloVe Embeddings: https://nlp.stanford.edu/projects/glove/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokenizing...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "getting coefs: 1917494it [02:23, 13334.83it/s]\n",
      "creating embedding matrix: 100%|██████████| 394787/394787 [00:00<00:00, 455687.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1917494 word vectors.\n"
     ]
    }
   ],
   "source": [
    "# embedding parameters\n",
    "max_features = 300000\n",
    "maxlen = 200\n",
    "embed_size = 300\n",
    "embedding_path='../data/glove.42B.300d.txt'\n",
    "\n",
    "\n",
    "\n",
    "# get tokenized/padded and embedding matrix\n",
    "x_train, x_test, embedding_matrix = tok_embed(X_train, X_test, embedding_path=embedding_path, \n",
    "                                             max_features = max_features, \n",
    "                                             maxlen = maxlen, \n",
    "                                             embed_size = embed_size, \n",
    "                                             verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "### Common Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# common hyperparameters\n",
    "gpus = 4\n",
    "\n",
    "# get test/train and ROC AUC callback\n",
    "[X_tra, X_val, y_tra, y_val] = train_test_split(x_train, y_train, test_size=.1, random_state=233)\n",
    "RocAuc = RocAucEvaluation(validation_data=(X_val, y_val), interval=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BLSTM 2DCNN\n",
    "Bidirectional LSTM with 2 Dimensional Convolution and 2D Max Pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 143613 samples, validate on 15958 samples\n",
      "Epoch 1/3\n",
      "143613/143613 [==============================] - 518s 4ms/step - loss: 0.0584 - acc: 0.9804 - val_loss: 0.0524 - val_acc: 0.9818\n",
      "\n",
      " ROC-AUC - epoch: 1 - score: 0.983916 \n",
      "\n",
      "Epoch 2/3\n",
      "143613/143613 [==============================] - 515s 4ms/step - loss: 0.0457 - acc: 0.9837 - val_loss: 0.0498 - val_acc: 0.9818\n",
      "\n",
      " ROC-AUC - epoch: 2 - score: 0.986655 \n",
      "\n",
      "Epoch 3/3\n",
      "143613/143613 [==============================] - 514s 4ms/step - loss: 0.0397 - acc: 0.9858 - val_loss: 0.0490 - val_acc: 0.9837\n",
      "\n",
      " ROC-AUC - epoch: 3 - score: 0.985306 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# import model\n",
    "from models.gpu_model import blstm_2dcnn\n",
    "\n",
    "# model\n",
    "blstm_2dcnn_model = blstm_2dcnn(maxlen, max_features, embed_size, embedding_matrix,\n",
    "                                embedding_dropout = .5,\n",
    "                                blstm_units = 300,\n",
    "                                blstm_dropout = .2,\n",
    "                                cnn_filters = 100,\n",
    "                                cnn_kernel_size = (5,5),\n",
    "                                max_pool_size = (5,5),\n",
    "                                dense_dropout = .4,\n",
    "                                l2_reg = .00001,\n",
    "                                gpus = gpus)\n",
    "\n",
    "# fit\n",
    "hist = blstm_2dcnn_model.fit(X_tra, y_tra, batch_size=32, epochs=3, validation_data=(X_val, y_val),\n",
    "                 callbacks=[RocAuc], verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "159571/159571 [==============================] - 561s 4ms/step - loss: 0.0586 - acc: 0.9801\n",
      "Epoch 2/2\n",
      "159571/159571 [==============================] - 560s 4ms/step - loss: 0.0455 - acc: 0.9839\n",
      "153164/153164 [==============================] - 32s 207us/step\n"
     ]
    }
   ],
   "source": [
    "# model\n",
    "blstm_2dcnn_model = blstm_2dcnn(maxlen, max_features, embed_size, embedding_matrix,\n",
    "                                embedding_dropout = .5,\n",
    "                                blstm_units = 300,\n",
    "                                blstm_dropout = .2,\n",
    "                                cnn_filters = 100,\n",
    "                                cnn_kernel_size = (5,5),\n",
    "                                max_pool_size = (5,5),\n",
    "                                dense_dropout = .4,\n",
    "                                l2_reg = .00001,\n",
    "                                gpus = gpus)\n",
    "\n",
    "# fit on all data and save weights\n",
    "hist = blstm_2dcnn_model.fit(x_train, y_train, batch_size=32, epochs=2, verbose=1)\n",
    "\n",
    "# predict\n",
    "y_pred = blstm_2dcnn_model.predict(x_test, batch_size=256, verbose=1)\n",
    "\n",
    "# store\n",
    "submission = pd.read_csv('../data/sample_submission.csv')\n",
    "submission[[\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]] = y_pred\n",
    "submission.to_csv('submissions/blstm_2dcnn_model_submission.csv.gz', compression='gzip', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8wAAABnCAIAAABaeVyJAAAAA3NCSVQICAjb4U/gAAAAGXRFWHRTb2Z0d2FyZQBnbm9tZS1zY3JlZW5zaG907wO/PgAAIABJREFUeJzt3W1sHFWaL/AHu4lP7GCf2ASXHRIfX/CkAhGpHVhSM5mbFKyGaVbLnUZiRV/NSPRqpKVZ6S5mpdWab/6GRystzayutllpLo00q220SFMoo00zaKCSOzAVNkAFeUhlAtdlvIkrMTHHJjanTdt7P3TbbrfbLwkd54X/T6PB7j7n1FN12p2nTz9VddN0foZWFp492y26VmkAAAAAAPDNMRQMa52dazar24BQAAAAAAC+UZBkAwAAAADUGJJsAAAAAIAai6zZ4nMpNyAOAAAAAIAbxtpJ9lbONyAOAAAAAIBrn5QT62mGchEAAAAAgBpDkg0AAAAAUGNIsgEAAAAAagxJNgAAAABAjSHJBgAAAACoMSTZAAAAAAA1hiQbAAAAAKDGkGQDAAAAANQYkmwAAAAAgBpDkg0AAAAAUGNIsgEAAAAAagxJNgAAAABAjSHJBgAAAACoMSTZAAAAAAA1hiQbAAAAAKDGkGQDAAAAANQYkmwAAAAAgBqLXO0AAAAAAOAqURPZd4LMp9Ph7BoNWQMz9Z19RpuG5HF9cJwAAAAAvpEmzsZfPvlKfp2tp4+Njqf9O7y40JE/rgMOEgAAAMA3UN7+9alX8rTv3nuyB7aJNVPCqYn0v3tPBZ/E39vm7WvaiACvc6jJBgAAAPjmKVzMjsxR447MejJsImpqSf5pz0GiEyc/D694cDeCDVzJLnwe/9n7r8zyw8/cG42s+6krQX2eOvyH1McXh4kaGrfE9+mp+1p4ZaNZ99Bvv+MXDsYOOD03X/mYamhdkQfHf9f91vSu+03/4Nf9MKrO+/rLZ4a37RpN3K5d5hhf2Zmjj45Fnnj8e5md9ZfQr3Ah/jPvldnmw//rj6PsMrcNANct6buuF0iuGaalL3sbn28UuK4XSOLCME1RbKX8nO2GamlDplkxUzo5b9kTIhq3tOIGHdcLJSsfC5aq/aREBSOi0Hc8XyrGhW4aovIdX3qOK7lpGZiUS1CYC2eJmhr5+lOvhs1aPVFeSaLL/Rf/G+T6KReZ+E/zn08F+reDR7Z+rWyqMJX6l/efGaf2jlufbCk4vnz5rfdD9t3cnoZaRQqXgbWw9qm6Sz+Xok5ribQXNrPr54UMALWh/Exfnx0Uf8mm7Wj/QNJYlnq5qb4BZ2HRLS2i/f1Jg5P0c9n5vot0EdW9bNaRFY8z04hbWpgb6Eu7C8+lRTw1EF+W7H2zXZlJiYrQ7u/LeIqIMVKKSFQOLN1MKh0YmmUYV2C3AC7PdZObyHEZ1GIcdX54YJyotcf98U5BFHb/R8fhyde9C3JPJz7+Xj03Rx/dfznfPUW2pn5yMFXzcADgmhdkBuyARDw1EOdBpr/PzqXS0UyvKG+j3HTKCZmRHOiLCuVn+/uyuVTGyvTqWixlxxbaSTeVHHDIiBlcWBm7d/EJpz+Z8pgZM0jmUmlXMrM31Wdp0k33DeSyKdtMxZds8RvuCk2KdAYyntKi/QNJg6sg29ebzaWz0XRCEJEMXM9z7UywoXsKsB4bn2TPhb5vvXnmSL6uq6M99ciuWEtleUB40o+/OXpkeo7qNx3c05N9SJNH3t79riIi8t/f7NMPHzuQbfpEf/lM2NGTvm20/8TFYarbe2dP9sHN9mF/YERN1LMfHtiTva+l2hJDxNrBud4miIiIN0UaiPIFUkREX3nHP0q+89mxfN0usaN3Sa/Z4PTp5Jujr0/OEW3ap3elv7/TYKWii70HdseHhgZG1ARt+sH992QPtnD6yv7F0UdH2dMPbw+PDb8yXqCG5r975J6B7irr5aVB7t8VHR1KjczkadMPD9yT3jHRe/iTV8bnGhqb+x8x+nbeTESkJtJv+AP+xWGi9tbW3gfv6isNuGrkJ08nj46+PjlH9ewHe7rTD3auVXqVd44MJt+Tp2aJGrY8+eCe1J4mVhiLPv/h6w3tv/urPWZkoURk9+eJztJBni04R97re08O06Z9Pd2Zh2/X55utY5qKhyvy5I++l+6sVxf+s/fQJ5mxQp7qdu3YkY7daTEiNZE+/FH/x9PniFpaWwce3pPsvLmy0GiF47PyNC1V1r2lmScfvGugZ3PVFyQN/kfH4cmuvff5D7UwIvXpCe2Vz9jd3w7+dOFrllnPG0wc/exEnrp2dPU1jT7lF3742AF7h4z/zHtlyWWSGl/6y+8kWlafEQCoENiuJGYkYoIR6fG4kRvwXDugJQld4HqKmBmPCkbE9FjCsPs9LxeQXt6KpGc7irRYwqx4Uwgd21Mk4nGdlOv6RFqpbISb8bieS/muF8YFvjSfd6UmJQgkEdOLpSBMmKaWDWRY+k4htDNpV1YUmQBcEzb+xMfJ3renjL1dT4vI8Ojoo6+OBBXPT3wa+9WZI9T6/Pd3Pbdj7siJ38cH89ru7ufvbiQiau14/oFdva2lUuP86CfpQvvAw3c83jh34uNT5i/+4HV0pR/o2DurXnvro8xElc2zzm9l4/emjWIhcj53XOaJ9u5u0Yjk6Y+stz47Vtjy5P07ojTaGxQWeslPT1r2mdfzzX/3QM9zgo75p61DYwtfGp44OlTc7j6aef3dwcyFhX7qhTcviHt7Xrq3uSU/+dNDn3gFWsmJ94aD7jsyD7TvpZnXjh43D42Je3e9dG8zTU8+W+o4lc4ef8q/yPWufzrQoU+MP/vq8b7zs7R65EMnrV+dcRra/+nPdv9TD71+4qT1689XfzeSJwej70rZueOl79/xRMPFFw9/mL6waoei8aG+kYbkvTseb5o55p+KvjmxsJVLm6bCF/2vnnpxbFPigV0v3bslHBmOHb4gadZ9w3vqY6Xv7XnpgQ59fPypXy4/mCsen9IRXnGair5Mv7rY3ZiSP7XfH7gwW/0F2b39INHw6XNBgYhmvUE5QZHYnuaFD3Xy9KD1xmcnCs1P3t8Vqz/X68+Unog0JvZ3PXd/13P373habCIiamgUqFQCuFTSDySRZmjFvzomDI1IBX5loQcRkVLz70aKFJEMw6WtAjvrE+mxmKjo52WzATEzHl1Io9ViT6WIKKy6wW+qKzYpwtQ5KS/nSiqWxYfEdEsUW+q96Ww2m4rhsw5cezZ+JZv1PXZvXxtRYRu9ePyF8TP2BdFbtoynJj73idp33p402tierdbItGqu422dyT1j/b+fptb25H1tjEidJyKixo70Q8KIkDk1+srRaer8VuZgG6Ov+MejD48oZ2I2uWyZvMysd8yLB3PUvD19bxNRPnfsswmqe/zRb6e7bybqED93nxkvtsznjp4bpsgTMWNgZz0Zt4S/+MjJfx4UthVXPVru1DMH2xjNakPnHgiUM/lVb1tpGz/4/p6B3Q1UaPIGj7+Q/8LPk7HCIW/p1jP72hhtY4PnHh0jsX/PwJ4GKmx2Bt9/efqLoED6+eGBMaLWO+xHhCCKN09pv5pMH5H9f75ltcjfPjdM7LlHdiXb6qm7wTntveIH7kNbxcqHJRiZzlOdsbcrsbsh3t0SPz+nrefcyIb2bPwuM0JkbPL/+ZMT/hn/wRad1jlNZeMUptxJoua2XuN2PdJudsugvpHRjHe+QNSY2Lcz0UKxznY3H9EiRGV5tjq70vEpTUa1abq5rPvQwCjRth77kZ2CKNqQjx1T3vmCaqrygqSmtmQHHRkdy018S2+5mB0qUEN7onPhxbbwWjLS3TcTbdd//s5TpRnZHN13Z5SIpsYSPx8h2vR07C4LNZ0Al6qU7fL5vx5W/KFiQVNYJnMcL5N2kjFBgZ3xyvrOj+RmcyExK25VLGNLJ+so0mJxk1FxHZV5npPKmL1RTXrZbEBESilFhD/hois3KXpvf1L2pQcSTmmM2EBf5dcOANeeDU+y67eYxYwq0hhtoxemZ9ypWSrLsVhLm0GfHfG9zac37W1tNLq39+1f+eIeTVuKp8qxhjoi0lo2MSKiOlZfR0RqtXsXzfrH37eOXsw33PrLuG5GiArKHSeixthtxc0xsyNC4wUiooJyxomIRW+rJyKKbE0l9hdHCYiISGstvZUU/6MWM7+IaCke4TpWT0RzqywhLx0kIppLHfl8Rzn+xTBRe8fW4sd13tYiaPLU+OdhIbJW5OrZ/+M8u7gpFeRJrBhIvehubjnx2eu/+i17kxltW6w9oo8tSWera20tXZq+qcVooBP5qaBApST7kqaJ3RJrpSPjI7ufH+1qZUZHe3J/KyMyOzbR+PRf/PNv+pob9daW+L47okv7rXh8qK3aES6fpsXuXZ0txe7C+COvePrMRPUXpGU00+hkZujL5M5z9jS169sXPz5VvpY2mR0RGp8r29qXmUODL+dp3/3GwM7r69o1ANeI4l9z5XtqRbrLjGR/POzPOqk+h4gJy+ChJ1l5dhbmsq4ikYhXnp4X2LZPpMfn17e51dvr9w/k7IFem4gZlsEcjxhDhr3oSk2K8lIDaU9xIxozNek5tmsPpESqr/JzEcA15irUZC/+WCBavgTQcnvuiUjq+KhzXvnj8uUxmZtsCB7ZWtMYZv3jnvnW5ETDrf/6xJ7YfIpffGNg88uRZYHNx7zWHUevlrUib3zusbuiizUJEdFEq3zFyXv2eLGR1OAFb3zaG/ns2Mi4X/9du2etIJZl4Zf7T09T74/u4++M2Gen/PGLr/3+4msjMyd/8i3jT+9/67ahzNAX/vj0kWD0SPAF/eW+ZA2vhV8o+/9yK7wgte7tB2nyyOCYOzU2TJEn9jaX72/FjFCkruyVP+sd8f5iZK5lxy774C34BxrgcjCNEZEKJREnIpKhIiKuVWZdTI8PZGMyDCVxocl0wiGm8cVWftYOiBmlC/Qtkm4mFxKzysu0uZlM2QkZhJJrgvn9CYe4hj/hRVdqUqSTcUISyYH+qEZEUUvvTw64mWxgJcVG7BfA5drwmuzZabdYg1uYzk0QETOXVnTIs/+ZPqn0/X+US3wneHL3PqJz579YuO6EWnM9dR0R+Mc98y050XjrL5/YG1/YemST0UREM85kMZUuBJPzG4s0ms1EpJwLs0REhYneF9/i/3vQrUEwl4C33tJOdG50ong05IWJgKjltq3aapEzs5mIZoL6LUZni9G5KRwZy52dXrUme9YfHMmOb0o+cq/zk/8ePNbaQnPeyIwqLsYXvir2VRemKi8GMvmZX9zsxISXJ2psWdeV7ZebupA5Maa6e+wf7/P/2ny+mWhywi98aR8f8SIdqT//Y/fJ731wd4Ro2plY8qFnxeOz6s7KiangwpeyQPy2W9qJhs9/Ufz4EQ6+p/3DEcv7csUXZFNbchvR2MjAoKKGtmRnPRW+DC5MBVOz868llRsvRjjjnZ1Z2KQcOhl7d5oa2+3YZV9WHOAbjwtDIwq9Ur1v6HkhkVh29eTQzdk5V3JNCI2Xmuvmwl+edLKOJG5VOeMxWzrjcWFAFTi2nfMUF0JwRqEbKGK6Ka7cPl53rtSkSKmIiC2k4ZxzKq/qBrhWbfxKtup/9QNpbKVPR16YpoYdXbGWJcuHLH+h/93P1MfTqX2cPh32iNo7btGIGNusEZ0KPood2hbf1xO/3M0Hgx+ab8kJoq6WueyvP8gWN9q2M/VgW/TeLfTGxfS/f2I9uI2NBr0jC50aYvtv7bU/e9H22L429unIC5Nz7Xd26JHV1oNrjnV29W0bfWbsdOzQTPK2mezbk3lq7N/PGdWvHPnmUuSHPuT72vjoyLO+ahF3J+9b9n1eGXV25NkTM5mzM/09dZ4nJ6jO6mYsUmc20WuT431vftp7m8q+LfMV3fLn4q9Geu+sc98bOUG0977temS1rawsnz06/Hr95/6D2438WHqSqHmrHplzTgw/M37Gmbgj3jKdHSoQNUeXfjxb+fisrDDZ+/L7L+cjT/7oe+nbunpbR58dOxU9NJ1onsm+J8/NNvbt2Mwmq78giRqs+7bQ4Yuvz9eKhJ7X/cY0bds1mri9OCMvHhoU+9rYyJmBsYWDO5b85blhoq62utw7fo6IiEX3CQu3pwW4NFo0pmfTfqY/RVHh5zIBMSNmaUQUpBK9jmRWKtsrVJDLZAPHTyaimnQymYC4FV+4XUlgZz0iPV55FT7lZ7PL1reZ8uyMo5wwmTBZmMvkJIl4TN+g3b0+XKFJEabBszk/k86xmKFRYKcdRcwqnfkoXTsXqOIpqIGTyfpcRGMo2IZrwoYn2Q23po25gbc/OTVbt2tHVzrWWbGYx7rvyh34KHls9KnDo0Sb9um70g9tZUR0W9fA3RcSv59+3R8l/Y74ZV7y7Kvg5HhxJX14dHx44eHJrf3UJgzj8PiHSW/k0VfO7Nqxve/ui0/9vrQAyXv2uA+fSh4598JRSbTp4N270g+1cdrQJJuoqTf+bTrsD/jDT/nU3tr6/Pfv6r2tnojWE/lP3xoniuy7syf9sLZq5PXGg8ZL+Y/6/U/+58dEDY2PH9DTPTcT3Zx8uMs5NPz6idOJRt67f7v/xplTZd0advT0NZ3pe2t6gjYd3Ktn911u2tjUmXnsYvLXZ1544yRRXVdHx78+3K1H6vXH7vYPnc68e+o1opZm/nexPYmWiuqOFY/PukSa+uIG/fsfUv7IM0Qtza3PPXRXbxtR2wovSCKte/tBOnWEItGltSK0MCPvffbsG+N77+we0NVf+HOMiPJf+rNERMMjoz8tfRbaRHt2WE2Xcp9LACDi0b7+MJWynXSaiDQj0VulRFfEe5N+f9pJ9ztERJqZ7OtdWJwOHE9xYSUquykvFzBNj1asb+vJvrjsz9qpfpuImLD6+nGN7ApXaFJEcqBXDSz0ICasZG9vqWA7yGWyXqmhdOwskaHHTNySBq4FN03nZ1Z5Ojx7tlt0bVg0ADcGefaCm6/Td2wVESL6Kpc9+vDIpqef+G5q/Uk/AKyLklIxvup3ViRlKBXTtNVbrY8MA0m8NmPdsK7QpMgwkIrh4NdOYSz2/Ievtd4x9BMh1tlFjcX+8RK73IiGgmGts3PNZtfNHR8vU2F2acVC/VW//7YqLD19MlJ/td8sZisq3VkEieDXNBue9h9+V7V0dPTrW2j83MAIUXN7ohUHFqDm1srliIj48pPvLhvXBEoR1nKFJgXHvtYiW6Lb6LWxkYHTHeme9dyy4Sv32P97jairY/XznaDkxl7Jnkq96D4zWfbAVf/spc5a/3jySNkDC3cNvFrU0Af81fGyAuu6xx8/kN2JdPDrKUzZb/p9g/LULBFF9oqOgYfuiK521XYAAICNps76xr+cOUXU3rxFXyvNllMXT0wTNdz6y5/sjX2zTyVa50r2jZ1kAwAAAMCK5NlPe389lB0rVF7PoIq6vWI71owISTYAAAAAQM2tM8ne8OtkAwAAAADc6JBkAwAAAADUGJJsAAAAAIAau5aSbOnlsplMxqm8X/e1LXQytr9hN3eVbjbrXvotcIJjv2O/CCv7XfjY+NkHToGIZv03/+9Nf/+b4v+0F9/rO/3lCiN95fzbEePY1CVHAAAAAPBNcg0l2aHnSxFNJCxcfPFqqGsXxpd/+yf/9bcHnP2R7KEP0xOXM4oa+kDLnJ3P5qdSP3+79+zsqj0AAAAAbkDXUJJNpBjHdeavupv1PXf2Nk1nz67jYj6VZoPzanG9XE27U3M1jAwAAADgerGh9z9UgZNzA6mIaboVNbXyZ3K2ExIF2YwnonFLk57jeIEkpumWZWpM+bYdmnFLI6LQyeSkGY/pjKj88dJQoes4fqiIcd2MmoIRUeVoRETSdxw3kMS4MCxL50SkAtdx/VAR03TTMgUjkm7WIUNIzwsVcd0qDagCN+f6UjHN0Be2LH3H8QKpiAuzNOLq+z4fAheGkF5oJKKCiCj0cq4XSmJcmJa1/P5WKvRyOS9UjAszagkmPTsnS8dg+fGo9JV77MPE2/IUsceNlpXKXBQRUR1RPvdrLzk4LalO9PTYj3SK8kaFqezhwd7T0yqyKfrdezL3NQZHjpvvTufppPYPQ4n/cWf4+uBreaJ/OZrt6PF+fLs2EfYeOp05P6MatiQfvCe1e/OKMQIAAABc5zZyJTtwHCmiiUQibpLveuUVwkxE41FB3Iwn4pZGoZPzmZlIJBJRLXTcgIhpgoW+JCKSQUgk/VARkQoDpenlGaX03YBZiUQiYWmh70uiKqMRSTfnkRFPJBIxQ7mOJ4mU7zihFp1v5wSlDFQGITfjiUTMJN/zJREpP+eEIlrsHMyXkAeuJ/VoIpGIG+R7FYXlVfZdujlX6bFEIhEV0pdlLX1ejFZXjuMtK8BWQcDMeCKRsDTp5HxFXNdZ6IfF5/yK41HZeehU7Fhd/xPWf/31PdELF04VljfJe94fUlNbEp03q0/9xOmmzF89IJ/cpQ+dHlhS+DHrv/1h8kKH89cPyCe61Tsf9p8n/eC+4M+2tGzbHf7N/vSd7faT9z3duOnpHx0If3y7Rl+mf3nS2XFP+Dd/Ej661Tn8YeayylEAAAAArgsbmWQLKx4zOBExrnElVz5ZMPQCphdTRS4EC4OQiAuNZCCJZBAywxTKD4lIBpJXLPUqRUpKRaSZsajBq48mvYCEIRgRMRGNWTonFfohN4rrz1w3eOiFxQiZZghORFzTmJJEpAJfakZxTzTDKKW0ikiRlFIRE1ascil5+b7LICRR3B4TpjG/E4EfsNKYXDc0GQSVWTYTZjFKzdC5DEIiJkpZtgrXyLFnPe8Cu/O/xdrqKXJLfP+2svsMzZ0LvM1//5ub/v630feo97F7Ei3EOnUvsdtiRE1bY81z/kRZ7Udh2vZnrP0deoSoZVvytpnc0Kpnf06MZcYb++5rYUS8c0dvy3R65DLKUQAAAACuDxtcLuI6fiCVUopIrNxMKpKenfHmH9AUEXGhKTdUggISluChF4SK+yETJivvzA1Ld9xc1lXENMOKGrzaaEoppi3045wRSUnEWekxxhjR/MeAJeMTkVLF5+dbFn9kwjQDx7WzDhEXVmWdx7J9V1JR+SjzY1cMrpalrmxhXMYYKVLFLNvzQyl8pRmrnTc6F07N8R2bShE3NPLIQqZb1y6M4M/bluxsYSb39mB66GKQn5P5OXPJUAVvqvCafZTV1xHN0SyJ1sIqWbbKT8vIJq2h+FtEayJ5oUDUsEqsAAAAANevDUyyQ8f2yIrGBSfp2bmVr0PHGCNeqrkuowmu/CAg0kzOmM5dP/AlE0ZFDsw0IxoziEj6tu36eqzaaCFjSqpS/qzCQHLBWTHFJUalXJezUnHysviKDYsWf+S6FdOJSIWO7XhSWItZdpV9Z4xooevCIGx+cFZ6uOyzwHxbWd6rmJEzoTPP86SsnmPPqqk5FqlnVMeb6uT4jKLNjEjNKlmlXGSxl/vm+32T3bkfG0bTTObnbmbJsxHRFHni4e9ldtYvCW6FsVhDIytMhPniK64QTpHWtqEf8AAAAAA20saViygliWkaJ1KB569SLEKk6ZoqllOTChx7vjpa03noesX6EKYJFni+0irPCwxyGXuxjpmx6qNpukZBsR5EBa7jhYqY0Hnolcq+fU9p+rL0dn5QTSy0DL356mvp2tlipIyoYvm72r5zoVHoB8XKcm+hJlvTNVmq6Ja+J/nyKFSwsG1faaXzK5nQWRhILsSycGfD82Op0zP6nVsY1Rv6FukHuSmiwlT27bHh6ntYNCcn51hri95UL4eGU+NzKl9WLhJpjHdHcu+MBgWiwsTAL44lP/2KiChSp6Ym568bXsciBX98hoiopS3RMp06PqGI1NmR1NSWZDeWsQEAAOCGtXGriUyYuudkMy7jwjAN6bqejBm8elPLkk4uk1HFa3+IUnmDJjiFTGilX5jyNL1yAGFagZPLeIoYF5YlGFG10TTL0nNOdv4xgxORHrVkrthVM6z5rVbBDcuUjl3cFV0wn4iIm6aec+2MU+peHlj1fS+F4DIuDMEpnN/3aOg4xWh1M7psB4nrQrrZTKiYpi9GyTTBKNTF0raFC/Gfea/MRvbpPRljMxHx3Xdlhj5MvvibRMOWxL72vSOr3FbmZuvADv3V43wwIjq3DzzAk2+fdvfsmn+23njwnv5Dg+bPTiuKGHv07M6biYh3ikTkw+/845kf/NmB3O7GhLHFOvzOTUe2f/Cknozt8g95/N051rgl+cg98aaVtwwAAABwnbtpOj+zytPh2bPdomuVBnDZZOCFTNc1RiSDXM7Toit85liX0LUdteq1+wAAAADgaxsKhrXOzjWboS72quGceU7xHumMC9O67Axb+XbWlVxYUWTYAAAAANcErGQDAAAAAKzXOleyr6nbqgMAAAAA3AiQZAMAAAAA1BiSbAAAAACAGkOSDQAAAABQY0iyAQAAAABqDEk2AAAAAECNIckGAAAAAKgxJNkAAAAAADWGJBsAAAAAoMaQZAMAAAAA1Ni1l2SHTiaTC6s8oXw7Y3vyaw6vfDuT/dqjXCtqc0yWWXEOAAAAAGA9rr0k+1qnAjuTcS41A728XhspzGUydnC1owAAAAC4ISDJvlSMG6ap8w3pVRPKt9eTPXPDNI2rEiAAAADADSdytQO4/nBxOcny5fWqASXDdZWTME3Xr3QsAAAAAN8QG5tkS99xvEAqIsZ1M2oKtvi4G0giLswlmZ4KXMf1Q0VMM0xtlVHdQBIRF5ZlCU5EYS6TY9GEVewTOpkcLfxGpGTgOE4giXGhW1ZxATd0Mg5ZJvddr7hBy9LJd5z536JGsXuQyzjzQ1fbdPUHy3uVNWFcGJalVw1AtyxTY8t2d+VjIgPXcX2piLgwLUvnJN2s7SsicjIZYnosbvKVZiF0MjllJaJilaNrzocKAAAAAKvZyHIR6TluqFnxRCIRM5jvOEHx8dCxXalZsXg8ZvLAC8o65ByfjGg8Ho/q0qt+gp90c64U0XgiEbN46Di+WjMQ5XuBZsbj8ZjJQ89eLJVWgReQEY3FoiYPvVwu55MRjcdiJg89Z/nmq2567XhUkLNdJaLxRCJuCenauUBF/xUtAAAEWklEQVQtPFUWgF9lk6sck9DJOaGw4olEPKYr13YCIm7GEzGdEbfiiUTc5KvMQnXSc1ypReOJRDyqSTd3w5wyCgAAAHAlbWSSzfVoPFZcm+W6zqlYx6ACNyBhmoIzxjXDWqwLlr4vuWnpGmOMC8sUVUeVUjFNaIyICysej+rL136XEWWjahT4C1k2101D45xruqERkW4aGmec64ZGKlyWLVfd9JrxSN8LmWEaGiNimmEJFnoLY5cFIEjJ5Xn9SsdEBV5AulUcleuGoMAPyjouRFF9FlaipCQuiqGasXgMVdsAAAAA67Ch5SJMBa7jB6EsppRMIyJSoSIuFnI3xjmRIiJSMlSML9ZLcI1TlXxQE5py7WyoaVzThNDF2kk213jZz4wCqai4HcYXejMiWvLb8hXyqpteMx4VyvIImGDkBopERQBVd2PlY6JCSSq0M37ZbipVbZiqs7ASLgQ5uYytCc650NdzdAEAAABgA5Ns5du2S7oVtQRnKrCzbunxrzcs06NxLSzyHM8rFR5XbHtptlk9+6zNptcVzxWgiLgRK184Z6zKPq40CytgwkrEwiAMwzAMHNsTZZXtAAAAALCCDSwXkYEslYUQkVooQ2YaKy/FWCyRYIyTknLhmRXqGmQQKKYJ3TCtaNTgKiyOVb70LCsqPcpHkqEizi8v46666RXiWcQ0TmoxAhUoYutdIF75mDDBSErF5im5tNZkvssKs7DiLoZBuLA3FqcwQFE2AAAAwJo2MMlmnFEYhIqUDFwnVKSUJCImdK58xwsVkQo9x1/I4jRdMOm6gSx2caveykWFnmMXr36hwiCQpYyZcU6h50ulZOi5fkVmGHpOaVTHDZluXN7abNVNrxBPGa7rXHrFq4Co0HMCEua6qzBWPCZMMwQFrhsoIpKBk8t587k4YySDUEqlaMVZWGkXpefknGKoMvAl8SqXOwEAAACAChtYLsINSw+dXNYnpulmNBo6OdeVwuR61JKOm8t6xDTdNIR0StmhZkVNx3HtrENcmKbBc/6yUVmxt51xiIhrRrR4VTtumEbouHbWZZpumiJ0lkRiaIGTdSQxLqzo5VZzVN909Xgq+kWV69pZl4oBXEoBxorHhAkrZjqOm804REwzolYpc+e6IQLHsQMS0YSlrTQL1XdRWFHDKYZavKggirIBAAAA1nbTdH5mlafDs2e7RdeGRQMAAAAAcC0bCoa1zs41m+G26gAAAAAANYYkGwAAAACgxpBkAwAAAADUGJJsAAAAAIAaQ5INAAAAAFBjSLIBAAAAAGoMSTYAAAAAQI0hyQYAAAAAqDEk2QAAAAAANYYkGwAAAACgxpBkAwAAAADUGJJsAAAAAIAaQ5INAAAAAFBjSLIBAAAAAGoMSTYAAAAAQI0hyQYAAAAAqDEk2QAAAAAANRZZs8XnUm5AHAAAAAAAN4y1k+ytnG9AHAAAAAAA1z4pJ9bTDOUiAAAAAAA1hiQbAAAAAKDG/j8QGg1w8VZQ4QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "image/png": {
       "width": 1000
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image('benchmark_assets/blstm_2dcnn.png', width=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BGRU 2DCNN\n",
    "Bidirectional GRU with 2 Dimensional Convolution and 2D Max Pooling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 143613 samples, validate on 15958 samples\n",
      "Epoch 1/3\n",
      "143613/143613 [==============================] - 506s 4ms/step - loss: 0.0622 - acc: 0.9798 - val_loss: 0.0580 - val_acc: 0.9810\n",
      "\n",
      " ROC-AUC - epoch: 1 - score: 0.982623 \n",
      "\n",
      "Epoch 2/3\n",
      "143613/143613 [==============================] - 502s 3ms/step - loss: 0.0509 - acc: 0.9835 - val_loss: 0.0529 - val_acc: 0.9835\n",
      "\n",
      " ROC-AUC - epoch: 2 - score: 0.986074 \n",
      "\n",
      "Epoch 3/3\n",
      "143613/143613 [==============================] - 502s 3ms/step - loss: 0.0453 - acc: 0.9855 - val_loss: 0.0626 - val_acc: 0.9810\n",
      "\n",
      " ROC-AUC - epoch: 3 - score: 0.986195 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# import model\n",
    "from models.gpu_model import bgru_2dcnn\n",
    "\n",
    "# model\n",
    "bgru_2dcnn_model = bgru_2dcnn(maxlen, max_features, embed_size, embedding_matrix,\n",
    "                                embedding_dropout = .5,\n",
    "                                bgru_units = 300,\n",
    "                                bgru_dropout = .2,\n",
    "                                cnn_filters = 100,\n",
    "                                cnn_kernel_size = (5,5),\n",
    "                                max_pool_size = (5,5),\n",
    "                                dense_dropout = .4,\n",
    "                                l2_reg = .00001,\n",
    "                                gpus = gpus)\n",
    "\n",
    "# fit\n",
    "hist = bgru_2dcnn_model.fit(X_tra, y_tra, batch_size=32, epochs=3, validation_data=(X_val, y_val),\n",
    "                 callbacks=[RocAuc], verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "159571/159571 [==============================] - 549s 3ms/step - loss: 0.0611 - acc: 0.9801\n",
      "Epoch 2/3\n",
      "159571/159571 [==============================] - 548s 3ms/step - loss: 0.0516 - acc: 0.9833\n",
      "Epoch 3/3\n",
      "159571/159571 [==============================] - 548s 3ms/step - loss: 0.0492 - acc: 0.9850\n",
      "153164/153164 [==============================] - 31s 204us/step\n"
     ]
    }
   ],
   "source": [
    "# model\n",
    "bgru_2dcnn_model = bgru_2dcnn(maxlen, max_features, embed_size, embedding_matrix,\n",
    "                                embedding_dropout = .5,\n",
    "                                bgru_units = 300,\n",
    "                                bgru_dropout = .2,\n",
    "                                cnn_filters = 100,\n",
    "                                cnn_kernel_size = (5,5),\n",
    "                                max_pool_size = (5,5),\n",
    "                                dense_dropout = .4,\n",
    "                                l2_reg = .00001,\n",
    "                                gpus = gpus)\n",
    "\n",
    "# fit on all data and save weights\n",
    "hist = bgru_2dcnn_model.fit(x_train, y_train, batch_size=32, epochs=3, verbose=1)\n",
    "\n",
    "# predict\n",
    "y_pred = bgru_2dcnn_model.predict(x_test, batch_size=256, verbose=1)\n",
    "\n",
    "# store\n",
    "submission = pd.read_csv('../data/sample_submission.csv')\n",
    "submission[[\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]] = y_pred\n",
    "submission.to_csv('submissions/bgru_2dcnn_model_submission.csv.gz', compression='gzip', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8QAAABmCAIAAACC8s/YAAAAA3NCSVQICAjb4U/gAAAAGXRFWHRTb2Z0d2FyZQBnbm9tZS1zY3JlZW5zaG907wO/PgAAIABJREFUeJzt3X9sHGWaJ/Cv7SZ+sTPuNzGQspkklYVsCohIzcJOageOFKyGaUbLbSOxok8zEj030tFz0u2YlVbj+c//4dVKO82sTtustLuNNKNrdCPRiNWmmdGQSm5hK2wClShHKhM4l+MlriTEvA6xeR3a9v1RbbvdXd1ut3+GPB+hl7i76q236rWTpx8/9VbL5NQNVPEvXtyj7q5+nRBCCCGEkFvQkDes9PZWv966/kMhhBBCCCHkq4GCaUIIIYQQQpoUqfXGZ0Ks5zgIIYQQQgi56dQMprdxvp7jIIQQQgghZNMSYjz0dSrzIIQQQgghpEkUTBNCCCGEENIkCqYJIYQQQghpEgXThBBCCCGENImCaUIIIYQQQppEwTQhhBBCCCFNomCaEEIIIYSQJlEwTQghhBBCSJMomCaEEEIIIaRJFEwTQgghhBDSJAqmCSGEEEIIaRIF04QQQgghhDSJgmlCCCGEEEKaRME0IYQQQgghTaJgmhBCCCGEkCZRME0IIYQQQkiTKJgmhBBCCCGkSZGNHgAhhBBCCFlfcjz3rpe9MOlPL7Eha2eGtqtf71YoZqyBLgwhhBBCyK1k/GLi1bOvTTW49eTx0bGMe4+TUDUKG8PQVSGEEEIIuXVM5X997rUpHHzowdxjd6pLRoIT45l/cX7kfZw4eadzsHM9BnizoZppQgghhJBbRvF6bmQGHTuzjUTSADqjqe/uPQScOvuZv+aDuymtfWa6+Fni5++/Ns0Pv/hQbN3y4PKz9OHfpT+6Pgy0d2xNHNTSD0d59VbnT96eF1HtD/ynt7H1GtqqaGTk8rKrvfrJ8J37RpNfV1Z4vFWYxC/z2WPPXIk8/9yj2V1tyzn01cTPndemuw7/jz+M3VyTRAhZK8K1bccTXNENU6v+u720kWfbjifAVd0w1GAr6Rbyti8Xb8gUMx5ThZ23vLJ3GFfNmDHXuXAt2/EF46puGir9XRRi9SfFEFbBqXpDjSVMBQCk59iO60tW3hlpRHHGnwY6O3jj/6C33660AVNSACuNKL6KvoplHsWJ9C/ff3EMO3rueCFatFzx6pH3ffatwv72jR7ZLY1F2Y6J1uXfvtCqRCM7irezr+K3KiFk2aSb7e/Pe8EXuUw+NjCY0iujW2Gn+wet+SRaRo0NDKR0DuEWcnP7LtDUWIy7+WzOLX+VmVoQTM93xgAJZPVUeiBGAUW5tZkUzcnlLFHxOjP0hKnAyw/0Z535QDujxgcHkxp9yiEb4ysYocjLw4NjwPa99vd3qYC/5997Dl97y7kq9vfSB9eNc1vsmUea+fVQZFv6h4fSqz4cQsjNycsO5j2oifRggnvZgf58IZ2JZfvU8m2knUlbPtNTg/0xVbq5gf5cIZ01s32aEk/n4/PbCTudGrSgx3UOSAkosXQ6VZF3FnY6bfnc7E/3GVzYg6lBO5tzY33aupzuzWGNJkU1s/m+hTesgVTaYUZcB0Qhk3WkGh8cSGpc2Om+QSufseLpGP0rTzbEugXTM77rmm9/cnSqdXfPjvTT++LRNgCYuNL/ppseuTHV1vHCEz3Obz4+3nXP0AuqctnVXv3EVx8o7BpNHZ+MPWukztv3vScPfvtRW28Hvsz/4tgzo5Eff+/RdG91zUDE3Mm51q0CAHhnpB2YKiL4BCsufJQ8PPLGtZno9jsG98+U7TXtnT+fenv0rWszwJaD2u7Mt3fpDN6Jf9tzZPLAY/clhoYGR+Q4tnznmw/mDkU56r1VIai48Hv2Zu4aHTh1fRitB+7dm3vi9vxhd3BEjrexP31sf+7hKAOAKevoh32nxk5NVRao1Bv52fOpY6NvXZtBG/vO/j2ZJ3qXrILyz7qJt0ePTs6gbcuh/XtzTyoKprLZf/3BFfbSf32kv3uutAPbj/z5N8zak9jwqZWm7IXvPZrpbYMczxz+cOCjyUtAdPv2waf2p3pvA6aso2dSJ8W5aaB96wtP7E/v72SVFSbh16fRuZDjmd+4g+71YSDaxVNP3D+49/bQq4Ez/95z+NruAw+7T0YZIC+cUl77lD3wB953F+pq5mekveuOwYeKA0cEf+Bh97sdheyxZ64sOuyh+GPW3tuWmBJCyNK8vC3A9GRcZYCWSOiFQcfOe1gUuHm2I8GMRExlANPiST0/4DgFD1r5VhBO3pJQ4kmDA56QAFNYVTrVLTgSajJpcADc6M9mhWQUs5Vbs0kp51t5R0JNJDQAEqpp6mZcCyYlpjHLlr4AaGLIhli3GxCv9b0zoR/Y/WM1Mjw6+syvRjwAxYl07vRfjdxQ1Z0/e+hrzrGPjy/eZ2rk7OPHBLo6tOUUaLDe388lHsrowQ2nU4UTYgo4cF9UATD+H4nXht+41vqdA7v7e24MHLs2v5e4cNbMf/LWVNdPHt/7korj7nnzzSvzv146dWzI6dmdebznIG689d6Z7NWFw9V5q8LU6MeZ4o7Bp+55rmPm1EfnjF/8LtjxwLR848iH2XEA087bJx5/b8zb3vOzx3cn2PVXj5yIn5hA/ZEPnTX/+ROrfcff/cl9f7cXb506a/76M1lrEIHxC/F//uQotv/s2/te2jlz9NT/TZxpZIGcsEls9NTKTdu/cX70kdQO7P2nx3u0sbEfvf6xU4Q4eyb2nhC9O//p2/c83379lcOnM5UXs/b1AbD0XHyR+dWJH7nXubb77x7r0SfEX+XfH7w6HXo1lD13HwKGz1/yigCmnTNiHJH4/q6Ff2jLZmRg5430ETF3lq3aw7tf+ubul765+6UHuqIAsEXvpDt9CVkNwvUEoOhK8KPIVF0BpOdW1gIAgJRzfxFKSED4/uKtvHzOBbR4XA26lgB8OzPQ19fXNzCYtYNfpQnbBbiqwrMLuVyuYLmSc6omKLeGk7Kwn5PLeWBGIqivUWKpvr6EzgFAeoW8I6EYZsU+hKy62dnZ0HbdMtOs/9mH+ruB4p145cTLY5/kr6qpqeH0GLD9nsKfqSqQ2gX1V5cule803fGz5x/uu6sNgBvebX3TznEn4c2g6+7MQ50APHf4LWD3AT3/ZJRhOhZ59xunbgAApgrHLg0j8nxcH9zVBv1r/i8+tKY+84p3Bh9zo/dq2UPdDNPK0KXHPWld+7Kvu5RorPNWpY6ezJOqHoExMfrasUn0/n72UDfDl/yj0adGpDU+nWoXg44EujLP3J/oBPZGnL//+OjxYUe/n9cZ+TuXhsFeenpfqrsNe9qt885rrmc/uc2ofV3k+GcusGPX11N6N9u/zRyZlF2twEztPQJhk9jgqUXL+7nhXC4CHcmDu5JRxHt32FMRJTLtjUxOoVU/sDt5X3tiTzRxeUapWIFH1r4+DcyFvDg0OArcuTf/9C4ViLVPxY9L53JRdoZdjc7uVA+Ojl4pjP++Fr2eGyqifUey7Ncgld9L7N1vnAxmpE3bf28/AHyRzY2MAwe++eBgyO9PCCHLJ4PQaz6aLSWSxeL8gWoazLKcbMZKxVV4+axTtu9cT3au4IOZCbN0F5yQgFcocFNT4Vp23nH8/ky/7gsJwMuk+jwwSAkgFxtMp6g8d97aTco8YeUsCSWeMBZfdi/X15fzAGb0DSbVVTofQmpqaWmpbmdnZ9crmG7bagThVKQj1o2XJ2/YE9PJa58PAzt6tgU3crCe7RoWB9Pb707c1XQUMu2eeN88dn2q/Y7XE5oRATDtXbgBtOp7tzIAaNP2drQHIWlRWmMAWCw4XGRbOvlI0IsHAFC2l/52CP4niwuHqfNWpc6twe13rL0VgBLdwgCglbW1ApDTwMRnzjTQta2UVe+M6u04Nfm5OzWtLDFy+dN/tH66cCTpTaFOMM2i3To+Peo6t5/fcmB7h77n7v5HbgOWSk6HTWKKNXZqi2wxerZgbPIHf//b/q4ObXs0cfCeGNrYnq7oqU/f+ud/ZW8zvXuruV/tZ0D59ax5fUonW38uxNjnw8Du3mjw/abq33B0AMB46NWAqXdh9Fp26IvUrkv5SezQ7tYXflyqvpf2bG0/OVZ2tGn3qPODkZn2nfvyh6L0ry4hqyT4Yar83VvFjxjTUwMJfyBnpfstgKmmzn1HLKrN8As5W0JNJuZuk1MT/QMxrurBohDxTKqvYOcdEbwtfZ7K5GMKhDWYStuFjJWg8twFazYpc7x83gW0RGW+GooRT8J17YKd7s/wdPVNj4SswOzsbBArl7ehr69fzfTCH4vA3F3RAMpWaaj6VXhba7M/GNPuCcc4cm28/Y7/9fz+eHRuDNMzQHmf80ecG95SD9XcIEuOvOOlZ++PLRTDRNROYKH8oUr064XnI+kTo9Zl6Y6JV6+IwrV27+mOyoNWD2Pe3CQ2pU3/7jeP3DWUHfrcHZs86o0e9T7HfzuY2rvfiY+kz1x1xiadkU+Pj4y5bd/K723yGCGKZW258KuxTdlz9yFcO3rmij1xZRiR5w+U1XiEzMiiqyGGzsXem0T7jnz86+rqnQEhtzqmMAAL1bHClwC4UhnYMi0xmIsL3xfgqiIySQtM4Qtbubm8B6aX1lgDAKbq+sL+qqGygi88wQzGAKnFgi25GTcytuW7PiiYnrN2kxIQdrbgg5lVZdQAU824aiKm9SfThSx9xCGrq1Yeurpdr1LO6Uk7KCktThbGATAj2sa7WBQYvvh58GseOTZWp5ajlGucKgJAUdqVZbiLDuaecIwjYrzjjtefP5CIzue2W/ldW4AZ93Lw63j4V2UpGRvpMLoASOvqNAAUx/teOcL/5xm7Tpp5LXRGtTbg2rgbxMET484U0BHV2uuMnBldAG54bVv13qjeu8UfuVK4OFm/Zlpc/I/MWak98o1C8o+8F+47CFy6/LlfCgeL/tQ0AExMuBUfLcImsanz/CJ/YsSJ9KT/7A/tFx794IEIMGmNf+meGcmNbUk9/ZD1w//kPbs9ihln5MaiE6l5feoca1qMT3hXvxBF8Lu+tgMYvlz6fvPPnFT+5qjpfFHraqCzO3UncGVk8IxEe3eqtw340r864Y1PSbQqd20BZpwLpQF6FyYXviUnLiZfHx0G+8mz99Ha2ISsJq7qCuA7pXpc33F8QNUrF3727UK+YAuuqKrCS5trxnyMJqycJcDLozMv25dI9GW9uS18XwJM4VB0BYCYq1mQQkiAqqbLrdmklPazcqU7D+c79HN98Xi8Lz+3QhQrRQhL3C1ESD216qEr2qpIGljHmmk58KsPhL4NF0ZenkT7zt3xKFjnrkT7p69cOZ/49Uxy+2T+5OIaj8X49o52yFPHf5fu3MEuDGcma27pnTltHBHjwO7oTO7XH+QAAKx7V/qJbn3/jn0nR06982F6++8ZxSsD78730h5/5I6+/Kev5B12sJtdGHn52syOe3u0CMLuoFgzbPvAfvbGKZF8/cMBbYtzavgUcOiR3XqkDTVHfntp5G+e5ge7+ejIT10ZVR9IPVz3OFNXB977VH40mT7IcWHYAXb0fE1BRL0rgivF7Nsf6zpznKFTlfuFTCIuN3GeM/6p4RfHPrHG70lEJ3NDRaArFm2V7shPT93IXrwxsLfVccQ4Ws09jOF6A9enVI0Tonit79X3X52KvPC9RzN37e7bPvrTK+dib04mu27kTopL0x39O29n10KvBoB28+GtOHz9rfkaj8tD5qsj58APv/hQbP+OfSdHzr13OsV2GvJq+uT83+JfZF8/+8Y00LVVuuf7XQCtmrYn2UureRCyckosruUybnYgjZjqFrIemB43FQBeOtlnCWamc32q9ArZnGe5qWRMEVY264GbpdvVAHj5nANoiYRa1q+uIm/l02klYXJh57IeoMUMDm7E9WzayaazLKYxr5BxANWke93KrdWkAIB0c7mqfLVi6ErO87KDaZYwuHRLs2JUpLQJWY5GMtA12nULptvvyOgzg+98fG66dd/O3Zl4rwIgsi2d2CvfHHr11Hm7oyt1aM/Bwx8frzEivkfLak7KHXvx8LWD2p7BnR//aCR0wy+9s2NBjnB4dGx4/uVr2wbQze+6p/BUMfn26Iv596Nddwx8a7t9pFTnyvfut586lzp66eVjAthy6IF9mSe7OdY3mEab/uQfHI582OeMvjiK9vatzz+uZfTbAaCBkf/VkTEgcvDevZmnFF5dv1aG7bm/8NiHqeOjPzo8Cmw5qO3LPLmNAcah+398+cOXR0d+cJk998ie58fOv1qemw+bxKYSAZ2pZx9w3zyffe/cG0C0i/8kvj8ZbcMT+j9NfTjgfvxfPgLaO557TMvsvW1xVUbt69OISGd/Qse//C7tjrwIRLu2v/Tk/X3dQHf41QCg7Ln7EM4dRSS2qMYDQGlGEm+PvnLsXP7OHYNP8NRvBCJAsegE33/XPn35ZGnbA107KZgmZFXwWP+An07nrUwGgKIn+/or71YD1ERfyh3IWJkBCwAUI9XfN5/X9CxHctVMLtqN6anBpBjIWplBK/g6PtAX4wC42dfvDQ7m82kHAJhq9vVXRny3ujWaFADSKXhM0WIV+Wo1OdAnBjPW/HRp8f7+OMXSpHHLiZUr89DVbcvk1I3qY/gXL+5Rd6/5qRQn7JFJ2d5l9rYDwNWPtH8cPtfzwGffryq1ImSTkeOfWZeL/K7tQa2L7xzv+c31A49/y3m44eCeENI8KYRkS5RbCOELyRRlGUUZUvi+ZLy8lHeuM98T4Mvq7JazRpNSpzPfE+Bh00XqKV6J/+z0G9vvGfqhqja4i7wS/9tl7nJTaTCe9oYv7OjpqbgfERv8BMTi5ODrp9+YZs99c2esUxZOjZxD658+tG15PxTF6cXJ0baNf+70phvSdMW6FixCK7Wt2PgnyfylSx38Jw/fqcpr2ZPXga2pvRRJE7I+lorZAIBX3wS3dLeKWqMzRaWIbQlrNCl1OqNJaUpka+xOvHFlZPB8T2ZvIw/y+NI+/v/eAHbPLb9202kiD42wDHSpxUKLlhbMzm5oZhoQF7zU2yP5KzemUPnAv8ZMpF+xX7xW9sLGf2zadEOSQx/wX42VrXvX+txzj+V2UTy9QtOuczb1zpWjkzMAdmy/o//bWt+u5TxeiBBCCNkI8qKr//KTc8COrq1LPhdPTFw/NQm03/H6Dw/EO5fY+KbQbI3H7JA3rPT2Vne4wcE0IYQQQghZZ+Lihb5fD+WuFBt4AHLrAfXuwSfviTW5hNd6a6oSehZoCW9bWjBbaoeGLyg9PeWvlDqhYJoQQgghhHwlNVfjEapWZnq91pkmhBBCCCFkVTW3MnT9emi0tIS2oY9xAQXThBBCCCHkJrX89aGDvRbaRbcU1o21a4XsFEwTQgghhJCbxnLy0Euvy9F4BrplLvqusAmCaeEUctls1vKX3nQT8a1s3l23R5cKO5ezl//8GO/4v7Ff+JX7Xf1I//kHVhHAtPv2/2n5698G/ymvnOw//0WNnr60/vdR/fjEskdACCGEELKqlpWNxvLz0PVLR6ptfDDtO65QY8mkeZMuXniTa92h6l/85R/P/uVj1iOR3JunM+PN9CKHPlCyF+ei9on0P7zTd3F6FUdJCCGEkFvTqlVFB92tOA9dbeODaUAyenjRxrtN239vX+dk7mIDi+RUmvYuy4X8t5y0J2ZWcWSEEEIIuWWtpCp61fPQoZutx6P5pGcVbE9IMEUzY4ZS/k4hb/mAl8s6aixhKsKxLMcTYIpmmobCpJvP+0bCVAD4VrYgjERcY0D566WufNuyXF+Ccc2IGSoDUNkbAAjXsmxPgHFVN02NA5CebdmuL8EUzTANlQHCzlnQVeE4vgTXzFKH0rMLtiskU3Rt/sjCtSzHExJcNUo91j/3uSFwVVeF4+vJmAoAvlOwHV+AcdUwzeqnOknfKRQcXzKuGjFTZcLJF0TpGlRfj0pf2sdPJ98R58Ce06O1ylMkALQCU4VfO6kzkwKt6t69+ad71fKNihO5w2f6zk/KyJbYtx7MPtzhHT1hvDc5hbPK3wwl//O9/ltn3pgCfnks17PX+f7XlXG/783z2cs3ZPvW1BMPpu+jZwQSQgghpJ5lrhJdNw89u2it6OYWxZsXmrpeh8y0Z1lCjSWTyYQB13bKK3iZGkvEVHAjkUyYCnyr4DIjmUwmY4pv2R7AFJX5rgAA4fmAcH0JQPqeVLTyyFG4tsfMZDKZNBXfdQUQ0hsg7IIDPZFMJuO6tC1HANK1LF+JzW1neaVIU3g+NxLJZNyA67gCgHQLlq/Ggp29uRJvz3aEFksmkwkdrlNR+B1y7sIu2FKLJ5PJmCpcUbaly4PRatKynKoCael5zEgkk0lTEVbBleCaxnzXD95zK65H5c5D5+LHWweeN2f//MHY1avnitWbTDnO79ITW5O9t8kLbvJ8Z/a/Py5e2KcNnR9cVLAx7b5zOnW1x/rzx8Xze+S7pwcuQzt00PuTrdE77/P/4pHMvTvyLzz8444tP/7eY/73v67gi8zrZ62dD/p/8cf+M9usw6ezTZWREEIIIeTWsZxsdLD9QrvWeeiytjTYdQimVTMR1zkAxhUuRe2b9nzHY1oQEnJVZb7nA1xVIDwBCM9nuqFK1wcgPMErUrdSQgohAcWIx3Qe3ptwPKi6ygAwNRY3NQ7puz7Xg3wy13TuO34wQqboKgfAFYVJAUB6rlD04EwUXS+FrhKQEEJIMNWMV6aGq89deD7U4HhMNfS5k/Bcj5X65JquCM+rjKaZagSjVHSNC88HmFqKpqW/RCw97ThX2b2/F+9uQ+RriUfuLHsez8wlz7n9r3/b8tf/GjuJvmcfTEbBejUneZ/JgM5t8a4Zd7ysZqM4mXdvmI/0aBEgemfqrhuFobp3YY5fyY519D8cZQDv3dkXncyMNFFGQgghhJCvrGVVRWP1VudY0lK7lGpI1qfMw7ZcT0gpJaDW3kxICCefdeZeUCQArirS9qUKD6qpct/xfMldn6kGK9+Z66Zm2YWcLcEU3YzpPKw3KSVT5vfjnAFCAJyVXmOMAXPh/qL+AUgZvD+3ZfBHphqGZ9n5nAVw1aysz6g6dykkynuZ67uic1kVorL5fhljkJBBNO24vlBdqej17t+c8Sdm+M4tpRG3d/DIfETbukPVvT/rXnSyxRuFd85khq57UzNiasZY1FXRmSi+kT/G2lqBGUxD3V6sE03LqUkR2aK0B19FlE6Iq0Wgvc5YCSGEEHJLaaIqer52YyVV0RUaqScBWqrbtQ+mfSvvwIwlVA7h5Au113djjIGXaqLLKCqXrucBisEZ07jteq5gql4R6zJFj8V1AMLN521Xi4f15jMmhSzFydL3BFc5C0JZMJRiWs5KxcNV4ws2DCz8kWtmXAMgfStvOUI1F6LpkHNnDJjfdb4TNtc5K71cFvPPbSvK9woib6ZqzHEcIcJj6Wk5McMibQytvLNVjN2QuJ0BclqKkDKPhb3st9/vv7an8H1d77yR/Qc7u+jdiNoZef6pR7O72hYNrkZfrL2DFcf9KSACoOhPQOlej89vhBBCCNnM1r8qeskhNZiHBloq2jUv85BSgCkKB6TnuHWKPABFU2RQ7gzpWfm56mVF477tBHUdTFGZ57hSqbw/zytk8wt1xoyF96ZoCrygjkN6tuX4EkzVuO+UyrJdRypaVRg716mizm/pO3PV0cLO54KRMqAinR127lxV4LteUPntzNdMK5oiShXXwnUErx6F9OaP7UqldJ8jUzXme4KratVwp/3LV9Lnb2j3bmVo07WtwvUKE0BxIvfOleHwMwzMiGszbHtU62wTQ8PpsRk5VVbmEelI7IkU3h31ikBxfPAXx1MXvgSASKucuDa37nYrixTdsRsAEO1ORifTJ8YlIC+OpCe2pvZQWpoQQgi51a1/VXSFBuuhy9vZ2dnQds3ThEw1NMfKZW3GVd3QhW07Iq7z8E1NU1iFbFYGa22opbIEReXwmaqUvmDSUbTKDlTD9KxC1pFgXDVNlQFhvSmmqRWs3NxrOgegxUxRCHZVdHPuqCG4bhrCygenoqnMBQBuGFrBzmet0u7lAws/99IQbMZVXeXw58495ltWMFrNiFWdILimCjuX9SVTtIVRMkVl8DV18bbFq4mfO69NRw5qe7P67QD4ffdnh06nXvltsn1r8uCOAyN1Hr9ym/nYTu1XJ/iZiNp79+DjPPXOeXv/vrl32/QnHhx484zx8/MSEX2/ltt1GwDeqyYjp//obz/5zp88VrivI6lvNQ+/23L07g9e0FLxfe6bDn9vhnVsTT39YKKz9pEJIYQQ8tXVYDYatfPQ1dnotcxDV2agW1pKbeURJ6duVB/Av3hxj7q7+nWycsJzfKZpCgOEVyg4SqzGZ4uG+HbeknXXxCOEEEII2Xwaj63X/ljV1SKojqeHPE/pvbv6dSpgXW+cM8cKng3OuGqYTUfS0s3nbMFVM0aRNCGEEEI2o5Vnoxupil5yGM3locOOEtJSZpoQQgghhKyt5dxxuNbZ6Iby0A1mpmdnN8XjxAkhhBBCyFfEkmtFh6/RgYV21VeJrhhgdTtfDz3f1tqyom1poWCaEEIIIYSsnuWu0bHGq3OUBlXd1lqdI2hD9wodIwXThBBCCCGkSSt6cmHQxWo8s3BZeeigbS4PXdGuxzrThBBCCCHkq6rBFaOxqmtFb2AeGqisSKFgmhBCCCGELMPqZqNXmIdevPGa56Grzocy04QQQgghZDlWNxu9FlXRa5eHLm9nZ2cpM00IIYQQQpa2qbLRFUOrbptYnSMk7q9Zi1J6s6WlBXQDIiGEEEIIWdKmyUaXhlPe1s9Dh2aj6+ehQyo85vLQs7OzaEF5S8E0IYQQQggJ0Vg2Otiyqg26qJuNXlKDVdErzEMvdU9k6c0gD13dUjBNCCGEEEJCNJaNDrZcaBvPRldYu6ropuuhq/PQwSmXt5smmPatbLbgh7wh3Xw274gVdi/dfDa34l42i9W5JlVqzgEhhBBCbhEbmI1uuiq61jbz7SrmoeceSL7QbppgerPyNWYWAAAJWklEQVSTXj6btZYbaTa313ryC9ls3tvoURBCCCFkc9gE2ejSQMrbzZOH3sSZ6c2Ocd0wNL4ue60K6eYbiZK5bhj6hgyQEEIIIZtDk9loLLSrmo1eaVX0muahq7uJ1D9DMo+rzQTFze21CqTwGyoDYYqmrfVYCCGEELKZNbhSx+wsWlpKGWegZga4kWx0aM/LzEYvaufHVtkG3dXMQ1e2dU9/NvQQ6xJMC9eyHE9IgHHNiBkqW3jd9gTAVWNRRCc927JdX4IpuqHU6dX2BACumqapcgB+IVtgsaQZ7ONb2QLmvwKk8CzL8gQYVzXTDBKyvpW1YBrctZ3ggKapwbWsua9ierC7V8hac12HHTr8xfK9yjZhXNVNUwsdgGaahsKqTrf2NRGebdmukABXDdPUOISdy7sSgJXNgmnxhMFrzYJvZQvSTMbUOlfXmBsqIYQQQr5SlgwfERYzV8etoTvWUiMbXRkfV/XWaDZ6UT/VkXVYvrlWG3r6Fe06lHkIx7J9xUwkk8m4zlzL8oLXfStvC8WMJxJxg3uOV7ZDwXKhxxKJREwTTviNdsIu2EKNJZLJuMl9y3LlkgORruMpRiKRiBvcd/ILpczSczzosXg8ZnDfKRQKLvRYIh43uO9Y1YcPPfTS45FeIW9LNZZIJhOmKux8wZPzb5UNwA05ZJ1r4lsFy1fNRDKZiGvSzlsewI1EMq4xcDORTCYMXmcWwgnHsoUSSySTiZgi7MJX5tZNQgghhJRZu3WjK6ywNrpiy8bvgFxhPXRVZ2UXYa5dh2Caa7FEPMi1ck3jCOoPpGd7UA1D5YxxRTcX6naF6wpumJrCGOOqaaihvQohmaIqDOCqmUjEtOpcbhW1rFcFnjsfTXPN0BXOuaLpCgDN0BXOONd0BdKviopDD73keITr+Ew3dIUBTNFNlfnOfN9lA1AhRXX8XuuaSM/xoJlBr1zTVXiuV7bj/CjCZ6EWKQS4GgzViCfiVFVNCCGEfFUsVRsdno1e/9roWtugToAbNu6V1EOHthXB/XqUeTDp2Zbr+SIIHZkCANKX4Op8jMY4ByQASOFLxhfqHLjCERL3Kaoi7XzOVxSuKKqqqUsH01zhZX9m8IREcBzG5/dmABZ9VZ3xDj30kuORvigfAVMZbE9CrRhA6GnUvibSF5B+PuuWnaaUYd2EzkItXFVhFbJ5ReWcq1ojV5cQQgghN4UGaqMXFQevJBu9ktroVa+KXuqUWyo6rx/Qz7drH0xLN5+3oZkxU+VMevmcXXp9Zd0yLZZQ/IBjOU6pMLji2IujyvAoc3UO3dB41oAEuB4vT4QzFnKOtWahBqaaybjv+b7v+56Vd9SyynNCCCGE3Gyaq41eyEZvgtroVa+KDj/lygx0+VDDQvn1qJkWniiVcwCQ82XCTGHlJRQLpQ2McUgh5t+pUY8gPE8yRdV0w4zFdC79oK/yVLKoqNAo70n4Epw3F1mHHrrGeBYwhUMujEB6EqzRhG/ta8JUBiEkmyPF4hqRuV1qzELNU/Q9f/5sTA7fo6JpQggh5Oa15HodWHFtdO319UpDCNql1ugA6tdGz51PedtcVXRVN2Un3nB5ybqsM804g+/5ElJ4tuVLSCkAMFXj0rUcXwLSdyx3PlpTNJUJ2/ZEsIsd+sgT6TtWPlhtQvqeJ0qRMeMcvuMKKYXv2G5FBOg7VqlXy/aZpjeXaw09dI3xlOGaxoUTrLohfcfyoBoNV0/UvCZM0VV4tu1JAMKzCgVnLuZmDMLzhZASNWeh1ikKxypYwVCF5wrwkOVFCCGEELKZ1Y5u16Q2uvYGNauil1UbvT5V0Q3d5ri4XfsyD66bmm8Vci6YohmxmG8VbFuoBtdiprDsQs4BUzRDV4VVigIVM2ZYlp3PWeCqYei84Fb1yoK981kLAFf0WLBaHNcN3bfsfM5mimYYqm8tGomueFbOEmBcNWPNVmGEHzp8PBX7xaRt53M2ggEsp3Ci5jVhqhk3LMvOZS2AKXrMLEXoXNNVz7LyHtRY0lRqzUL4KapmTLeCoQaL9VHRNCGEEHKTaWz16BXVRq+wKrqiNnqzVUWH5qGrh9cyOXWj+ur7Fy/uUXevZP4IIYQQQsiGWDKgrKxdXtm60Y0dtNF+luqzRsAd1oaWaNdql+zNGx7e0dO7ETXThBBCCCFkHS2ZjcYqrxtdOmx5u8Kq6Ipqk1Wpil6VPHR1S8E0IYQQQshNr05tdHiFdLDb8muj5/Zb/RWjV1gVHb5rjXroBquiQ7PdFS0F04QQQgghN73Gn2W4abPRjazRsaarczSSh6740IF1eQIiIYQQQghZfRuSjZ7bbO2y0aWX1391jkby0NU5cwqmCSGEEEJuSstePXql2eiQnPQaZKPXqSq6uTw0gsfElLUUTBNCCCGE3GTq5KTRWOTaVDa6dPDydlNlo9ckD112RdHSUtHOgmqmCSGEEEJuNqv7LMMKN0s2ep3y0GHZ6Pm2hYJpQgghhJCbwlIV0sE2ZW2w2/IrpDdVNrpypxp56CWz0auVh65uKZgmhBBCCLkJrOnq0Rubjd6Qqujm8tDl7ewsKDNNCCGEELJ5rdt6HRubjW6uKnpZ2ejVykOXty0toMw0IYQQQsjm1fh6HZSNRt1sdNjHimby0NUtBdOEEEIIIZvOciukb5Fs9JLp4qWz0eUjW34eurwNUDBNCCGEELLprKRCepmrR5cOWN6uPBs9O7ui5xeuMA9dryp6ZXno8jZAwTQhhBBCyCayWut11FEnGx20S+akl8xGr8+K0dUxdJ1s9LKqomvloatRME0IIYQQsonUzkkH7wZbhceBy1o9eslsdEVOeqNqo5vLRqPZ1Tnq56GrUTBNCCGEELLB6lRIYznZ6DWtkN4ktdFrkYeuzkY3joJpQgghhJANtownGtbNRq/6eh1rl42uFbIvKxu9dlXRjYvUeuMzIZbTDyGEEEIIWbYa9xe2lFVizJUrVIWTVRs3eaAV9LB03UUQw1fWXYS1K+qtaruGDlnWokYRdlngHq5mML2N8yWvKSGEEEIIWRUrCXOX6m15oW1INBm2aZ3IfoUnsrox+rIi9TrtZ5+Nz1/b8kQ+lXkQQgghhKyrZawhHezQbIX0/AHL2w2sjd7Aqujyeug6aeZGVEwBBdOEEEIIIetqGWtIN18hXTpUebu81aPnxlrermJt9MZWRTcu9JNPefv/AYhshw/D/sl4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "image/png": {
       "width": 1000
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image('benchmark_assets/bgru_2dcnn.png', width=1000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:kc]",
   "language": "python",
   "name": "conda-env-kc-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}