t81_558_class_03_2_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bLEEW13uCtiJ"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oX6K_-JCtiL"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FzlMdsudCtiL"
   },
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb)\n",
    "* **Part 3.2: Introduction to Tensorflow and Keras** [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CpVJpj2DCtiM"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wU1AMzx8CtiM",
    "outputId": "9783a6d5-c6b8-433a-b832-1f83ea7d8007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Zg-zNx0CtiN"
   },
   "source": [
    "# Part 3.2: Introduction to Tensorflow and Keras\n",
    "\n",
    "TensorFlow [[Cite:GoogleTensorFlow]](https://research.google/pubs/pub45381/) is an open-source software library for machine learning in various kinds of perceptual and language understanding tasks. It is currently used for research and production by different teams in many commercial Google products, such as speech recognition, Gmail, Google Photos, and search, many of which had previously used its predecessor DistBelief. TensorFlow was originally developed by the Google Brain team for Google's research and production purposes and later released under the Apache 2.0 open source license on November 9, 2015.\n",
    "\n",
    "* [TensorFlow Homepage](https://www.tensorflow.org/)\n",
    "* [TensorFlow GitHib](https://github.com/tensorflow/tensorflow)\n",
    "* [TensorFlow Google Groups Support](https://groups.google.com/forum/#!forum/tensorflow)\n",
    "* [TensorFlow Google Groups Developer Discussion](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss)\n",
    "* [TensorFlow FAQ](https://www.tensorflow.org/resources/faq)\n",
    "\n",
    "## Why TensorFlow\n",
    "\n",
    "* Supported by Google\n",
    "* Works well on Windows, Linux, and Mac\n",
    "* Excellent GPU support\n",
    "* Python is an easy to learn programming language\n",
    "* Python is extremely popular in the data science community\n",
    "\n",
    "## Deep Learning Tools\n",
    "TensorFlow is not the only game in town. The biggest competitor to TensorFlow/Keras is PyTorch. Listed below are some of the deep learning toolkits actively being supported:\n",
    "\n",
    "* **[TensorFlow](https://www.tensorflow.org/)** - Google's deep learning API.  The focus of this class, along with Keras.\n",
    "* **[Keras](https://keras.io/)** - Acts as a higher-level to Tensorflow.\n",
    "* **[PyTorch](https://pytorch.org/)** - PyTorch is an open-source machine learning library based on the Torch library, used for computer vision and natural language applications processing. Facebook's AI Research lab primarily develops PyTorch. \n",
    "\n",
    "Other deep learning tools:\n",
    "\n",
    "* **[Deeplearning4J](http://deeplearning4j.org/)** - Java-based. Supports all major platforms. GPU support in Java!\n",
    "* **[H2O](http://www.h2o.ai/)** - Java-based.  \n",
    "\n",
    "In my opinion, the two primary Python libraries for deep learning are PyTorch and Keras. Generally, PyTorch requires more lines of code to perform the deep learning applications presented in this course. This trait of PyTorch gives Keras an easier learning curve than PyTorch. However, if you are creating entirely new neural network structures in a research setting, PyTorch can make for easier access to some of the low-level internals of deep learning.\n",
    "\n",
    "## Using TensorFlow Directly\n",
    "\n",
    "Most of the time in the course, we will communicate with TensorFlow using Keras [[Cite:franccois2017deep]](https://www.manning.com/books/deep-learning-with-python), which allows you to specify the number of hidden layers and create the neural network. TensorFlow is a low-level mathematics API, similar to [Numpy](http://www.numpy.org/). However, unlike Numpy, TensorFlow is built for deep learning. TensorFlow compiles these compute graphs into highly efficient C++/[CUDA](https://en.wikipedia.org/wiki/CUDA) code.\n",
    "\n",
    "## TensorFlow Linear Algebra Examples\n",
    "\n",
    "TensorFlow is a library for linear algebra. Keras is a higher-level abstraction for neural networks that you build upon TensorFlow. In this section, I will demonstrate some basic linear algebra that directly employs TensorFlow and does not use Keras. First, we will see how to multiply a row and column matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bkh9pHqPCtiO",
    "outputId": "c2b05474-0644-4c5c-c7fc-fb94e1e6a69d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[12.]], shape=(1, 1), dtype=float32)\n",
      "12.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Create a Constant op that produces a 1x2 matrix.  The op is\n",
    "# added as a node to the default graph.\n",
    "#\n",
    "# The value returned by the constructor represents the output\n",
    "# of the Constant op.\n",
    "matrix1 = tf.constant([[3., 3.]])\n",
    "\n",
    "# Create another Constant that produces a 2x1 matrix.\n",
    "matrix2 = tf.constant([[2.],[2.]])\n",
    "\n",
    "# Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.\n",
    "# The returned value, 'product', represents the result of the matrix\n",
    "# multiplication.\n",
    "product = tf.matmul(matrix1, matrix2)\n",
    "\n",
    "print(product)\n",
    "print(float(product))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pMrXqwskCtiP"
   },
   "source": [
    "This example multiplied two TensorFlow constant tensors.  Next, we will see how to subtract a constant from a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qKyFCaoTCtiP",
    "outputId": "3c75a8cf-8d1c-48df-fb71-f603053cf616"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([-2. -1.], shape=(2,), dtype=float32)\n",
      "[-2. -1.]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "x = tf.Variable([1.0, 2.0])\n",
    "a = tf.constant([3.0, 3.0])\n",
    "\n",
    "# Add an op to subtract 'a' from 'x'.  Run it and print the result\n",
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())\n",
    "# ==> [-2. -1.]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "epQ9LrM6CtiP"
   },
   "source": [
    "Of course, variables are only useful if their values can be changed.  The program can accomplish this change in value by calling the assign function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vtnW5aU-CtiP",
    "outputId": "77d3ae14-dfac-4877-dd7f-d30ae9c9751d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2,) dtype=float32, numpy=array([4., 6.], dtype=float32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign([4.0, 6.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zHwJP8MjCtiQ"
   },
   "source": [
    "The program can now perform the subtraction with this new value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0e26Fe-GCtiQ",
    "outputId": "979cf4e5-4d6d-45e2-c7a7-c12e6b169baf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([1. 3.], shape=(2,), dtype=float32)\n",
      "[1. 3.]\n"
     ]
    }
   ],
   "source": [
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rq61YFBYCtiQ"
   },
   "source": [
    "In the next section, we will see a TensorFlow example that has nothing to do with neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iMVIS9pBCtiQ"
   },
   "source": [
    "## TensorFlow Mandelbrot Set Example\n",
    "\n",
    "Next, we examine another example where we use TensorFlow directly. To demonstrate that TensorFlow is mathematical and does not only provide neural networks, we will also first use it for a non-machine learning rendering task. The code presented here can render a [Mandelbrot set](https://en.wikipedia.org/wiki/Mandelbrot_set). Note, I based this code on a Mandelbrot [example](https://chromium.googlesource.com/external/github.com/tensorflow/tensorflow/+/r0.10/tensorflow/g3doc/tutorials/mandelbrot/index.md) that I originally found with TensorFlow 1.0. I've updated the code slightly to comply with current versions of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 537
    },
    "id": "v4ll1OLwCtiQ",
    "outputId": "e0f9479e-7bd7-4c4f-f8dc-b658a72ee747"
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAIIAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDyuiiivROUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiilAJOAKAEoxnpUixf3j+VSABegp2Jc0RrET14p4RV7U6ighybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAIQCMEVG0X938qlopDTaK5BBwRSVYIDdRUTRkcjkUrGikmMooooKCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACjGelPWMnrwKlVQo4osS5JEaxH+LipAABgClopmbbYUUUUxBRRRQAUUUUAFFFFABRRRQAUUoBJwKeEA6800rickhgBPQU4R+pqSiqUUZubG7Fpdo9BTgpNLsPtVqD7EOfmM2j0FIUWnkEdaSk0NSZGYz25phBHWp6CAetS4lKb6kFFPZO4/KmVDVjRNPYKKKKBhRRRQAUUUUAMaMNyODURBU4NWKQgMMGlYpSsV6KcyFfpTaRqncKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKkWLu35UCbSGKpbpUyoF9zTsY6UU7Gbk2FFFFMkKKKKACiiigAooooAKKKKACiigDJwKACnKmeT0pypjk9afVKPczlPsAAHSiiirMwp6r3NIoyafWkI31IlLoRzS+SudhYdzkYH1qQFWGQRilYBgQRkGsq4jFvMiRsyRyHaQOle/h8LRrU7Ws1+JjFcztfU01Kuu5CGHqKQqD7UseAgA6CnEV5+Kw6hNqOwRmRFSKSpaQqDXA6fY2U+5HTWUN9aeQRSVm10ZafVEJBBwaSpiARg1GylfpWbjY2jK42iiikUFFFFABRRRQAEAjBqF028jpU1FIaditRT3TbyOlMpGydwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoooxnpQAU5ULfT1p6x45b8qkp2Ic+wiqF6UtFFMzCiiigAooooAKKKKACiiigAooooAKKKeqZ5NCVxNpDVUsalVQopelFaJWMpSbCiiimSFAGTRT1GOe9VGN2JuwoGBSjrRSiumEbsxkxay9Rj3Oj5fjrjoPc9q1Kq3FtHMdzKCcY5r2sFs1e1+vYinNQmmylY3jea/muyqBwsi7SPqK1gciufmh2XKxoM91QnAzWnb3QeJS3ykjoa66mHdRckneS38/M2r01pOGzL1RySLGVBB+ZtowKpy6kkblCCeM5zgVCss11dqUUIYgThj+HI7VzRwMOZxm/lfUzjSna70RqEVGRj6VIm/wAtfMxvwN23pn2oIyMV4demlJoqEiKgjIwaCMHFFchsRMmOR0ptT1Gydx+VQ49jSM+jGUUUVJoFFFFABRRRQAVC6beR0qaggEYNIaditRTnXafam0jbcKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW+lTKoXpS0UzNybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUoBPSnKnc/lUgGBgVSiQ522GqgHuadRRV2M27hRRRQIKKAM9KkC4+tVGLYm7CKvc06iitkraIybuAFOoorrpU3sjKUgqNuhpxaq886xjlgK9jC0XfQhJt6FK+jDgnJB5H15qiJN4LOWCk4YL379T+P6VLc3DSKAFPzjK85yM4/pTJkRLdHAwH6JnJVu+fwx2Fb4qtBSgoPXun0vb+ux6tGLjG0iW3tfPbc6qF/ugYrWtreOFQEXGKoWVwpAXOSOuB0rUQ9K0xUfZw5YbHDiJzbs9h9Np1Ia+brx1IgxjDIplS0xhjntXDOPU3i+g2iiisyxrJu571ERg4NT0jKGFS43LjK25DRSlSp5pKg1CiiigAooooARgGGDUBBBwasUyRdwyOopMqLsQ0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigApyoW6dPWnLF3b8qlp2Ic+w1UC9OvrTqKKZmFFFFABRRRQAUUUUAFFFFABRRRQAUU4IT9KeEA+tNRbJckhioT7CpAoXpS0VaVjNybCiiimSFFFKFJppXASnBfWnBQKWtFDuQ5dgAx0ooorRLsQFOFAGKK6KdPUzlICcVVurjyYiwP/ANao9QEzKPLfag+9gc1ktKSxDozliBmRto7f/X7969ulRVOHPZvTp/XQ1o0Oe0mzQjviw2yYDgc81UvJ9zZVuen4f5xTZbeRpEKeWF2hQynIJHXn1ppWG3hDXeVxwVUc5zkZ+o3fgKcsfThRXNG769Leep1wowjLmQkk6WkEU7xgyZzgALnnOf1I4B/SqsGpmWaRJtxSRQirnOPTJP8Ah19KbeX9tcWyQIsiqhJHHfHXGee/0B71WsJIobgSysRtzgBc54P5H096+Uq4xvEQUZrl0v8Arfbodah7rbWpr/LFcqke7KNg5OR79h05rZjlBUc1kCOG7lM6XICOGI45BHbA/OmC6LfIHIB43Hj/AD+dfW4OtRq0HzO1tXf9O/4nFXouo1Y3TcRqyqzcscCpqw4sNe7Zc5HTcAMkHqBjgcf55rbByK5cVSi6anBbnFUp+zaQlHWnEU2vFnGzKTuRkY+lJUvWmFcdK55QtqjVS7jaKKKgoCAetRMu0+1S0UmrlRlYgopzLjkdKbWbVjVO4UUUUDCiiigCGRcHI6GmVYIyMGoCNpIpM1i76CUUUUigooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW6dPWpVQL06+tOopmbk2FFFFMkKKKKACiiigAooooAKKKKACinBCfpTwgH1pqLZLkkMCE/SnhAPc06irUUjNybCiiimSFFFFABQBnpShSfpUgGOlXGDZLlYaFx1p1FLitox7GbfcSilxS4rRU31J5kJiloorWMLbEN3CiikJrrpU38yGyN+RWNqG0Y5GeuK2JDgVj+ar3ZSRRIHwFH93J56jrj0r1vaeyoNtXvp/XY6sJF81+xWv8Az5IYp4nWOMkFmGRg9CemcA46Z7deAMm5uZbuQSSnLAYq7qFxGqNbwjYeFkGAQ3f7319u3WsyvhsyqL2rhFvpdX0uuny/O569KOl/uCiiivNNixZ3RtJ/MC5yMH1x/L862LW7h1B9ph8sJ0bOQOn5cKeua5+tzTN0+nsglCsHIJLc7cDP4YHTPY+lenltV+0VOT01e39evyMKySXN1LFtFJPIZCwJPGTzjGOQfWtyJdkYUdAMVk2IaOUoG3KADnHXPpWsp6V9pVhBUUqa0/y02/yPHxUm52Y+kIpaK8WpT6GMWNopSKSuWUWjVO4hUGmFSKkorNwTKUmiKipCoNMKkVk4NGikmJUTLjkdKlo61DVy4uxBRSsu00lZmydwooooAKZIuRn0p9FA07MrUU512tim1JsFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABTlQt06etOWPu35VLTsQ59hqoF6dfWnUUUzMKKKKACiiigAooooAKKKKACinBCfapAoFNRbJc0iMIT7U8IB9adRVpJGbk2FFFFMkKKKKACiilAJoASnhfWlC4pa1jC25m5dgoxRTq6KcOZmcpWDFFZl3eTQTPGxXBGVK8Ef5/wA+lC6hlFUDMpAwp4yT0r1qeDi7rm1Q3RqNKS6mizqgyzAD1JpQQayJbl7rEYVgQRu25OORnBFSafNJsIkYEDgd/wBe9axwsHLkjq7Xv0CVCUYczevY1KTIqPcKa8gUZqo4V3sc9myUtTS1Zn26RkZwvyrwSeBnOMZ/WhL7LFXI3DuMgH8DXRSo03LlT1/yN/q00rsuyNxj86yLzbuDEcZ6VYkuSUyFyuM5B7c9fyP1qhNN5hrtXs3CUeZdVudFCnKLuRaoPOt4ZIgSsa7XIPT0Pr3+lZNbkT+cotpfmj2kKOmCTnnHUZArMvLOS0cblYIxO0sACcYz396/P81oP2rqLXv5dF8n/wAA9Sm+X3WVqKKK8g1CtmwQRWnmECObja7rj5c54HftyR/KqVnp01yUdlZLdicy44GP84q88wZEiVAsacAde3rXt5PheaspTVl/Wv8Al5+RhUfMrR6bmjYZWNQSSBzjPStNWGKwoLkIBnir8VydgcqQh/i7dcc+lfdVoQaUU0r7f8A8mtSk22aIalyDVI3SquefwqvDqGPnfcY2cjd2XivPq4WMfidr6GMaM5JtI1qKjjlWRdwOR/KpK86ph3FkXtowxSYpaKx9g29h8wmBRioLvYYTuAwOearabdtKFhIUbE/P6V0Sy9ez5lvr+BpFScXJdC4y9xTalIqNhg14tSFmaxlcaRkYqEjBxU9Mde4rGSNoO2hHRRRUGoUUUUAMkXK59KhqzUDrtbFJmkH0G0UUUiwooooAKKKKACiiigAooooAKKKekeeT0oBuw1VLHiplQL9fWnAADAop2MnK4UUUUyQooooAKKKKACiiigAopQpPSpAgHuaaTZLkkMCE/SnqoX606irSSM3JsKKKKZIUUUUAFFFFABRQAT0p4UDrVKLYm0hoUn6VIBjpRRWsYpGbdwoopQKtK7sS3YBS0UV20Y66GUmUb941hPmdDxVC3VyUMgUwryjE/dPUDI/D/PFX7uEyZIbjaVx25rFuF2TfOF652gYH0r169OpOj7qVl167f16ndhOVx5b6ly7ibCiBAgTDOfulTz6/09B7VXgacHZGvzDHB46/5/Wm6vLcx7HjdfLxkP8ALlu2c9ScdcVVfV3zPsXO9Qqs3Xjufr/XvXjxzNYSUoOVr2e1/u/4NkdUYOUFpf8ArqbEd8CSPmAJO3PcUy6ufkIBGfT1qrLMlzAs4neV+h3YGPwH4f5xTHCR2vnzM43Z24GcYI7d85r24ZjRdHnlv5drb/dqYrDpNSsWPPjtbAzKZFyTwOC46Dntzjv68VHuW9tmmgEgEZw24/KBjrnA9PwGKrX8ll5KxKzPjLIyjBII43ZHr+n14zY55YlZY3KhiCceoOQfY18rWzOdGr7r038767/h8ux0xp395b+ZrSXIgkiaJvMDhd4yFTJyRkjHHse2emBTJ9QgjRI7di0LcuhHIwfoOT9T/KsiiuL+1Kyvy9bfpf77a3+Rr7FXu/6/4boaR1iTyyscMatnhiAcLjpjH4/XtTJNXupgRL5cgOcBl4HQ9O/TvmqFFcv1uvr72+ny+4fsodhzOrKAI1UjuM8/mackgQf6pCw6M2T+mcfpUdFYqck7/wCX+Rduhbi1K6hi8pHATB+XaO4xn/PpUzaruLf6NF8zZ5HIX0B/rWdRW6xddO/MyeSPY24tQtbrMM2YlyBGxGSPqeOOg5p1vdrdvIAobZHhVbpgc9RyBwPb6dDhU+OWSI5jkZD6qcV1wzaun735f0vnvt2IdGPQ29kuws7iMxuoyWA69859qk85bxjtZvKhUjzNuQxOM/Qc/hx61jS3089uIZZGcBs5J6/X1q3pUcfk3DtKPM2jYnmbecnn8MD/ACa9CWaSxVeOl1b/AD0/4bV9DN0+VXf9bG3p8mxPL3KyrjDKODxWhvrBhMtuPmBDDHyYOef6+3t9M2vtoKjB69hX1cI0q0bxle255tag3O66mgbmIMV8wZHUZ6UomVhlWBHrmsZDvdjNGBCcyEsSCcDtz1xk/TNE8rpM0sAYwuxCtnIY5yT/AJ9K5fa0IVXTnpbrdW7j+qX0TLOoS5XYSQh6kf4VJpZKq4cAMG5A7e3/AOuqEq3PmAFMlfmyBkcdalkBt5h5TuA+Cq9W98k9Dn61c6tGVblg73VtO9/+H62NXS/dchucGmsMrVawhmTfJM+5nx+GKtHrXiY6jGnPlickdHZO5FRQRg0V5R0ELDBxSVJIOM1HWbVmbxd0FFFFIYUyRcjPpT6DyMUhp2ZWopSMEikpGwUUUUAFFFFABRRRQAUUoBY4FTIgUe9AnKwiR45PWn0UVRk3cKKKKBBRRRQAUUUUAFFFKqlqAbsJ1p6x+v5U8KF6UtWo9zJz7B0oooqiAopQpPal2e9NRbFdDaKfsHrRtFVyMXMhlFSbR6UtP2YucjCk04IO9OoqlBITkwoooqyQoopQKpRb2E3YAKWiiuiELaIzbuFITxQTio3kC9TivRw9Bt6EbjZOh+lZTCN7mVZHVOmN/Tr/AJ/Wr7zKR1rJvJFZuvNetLD81Fweh14ZNSKWryTMUBSRLcnKDJ2njsPWsyt+KA6hbtAYxkYPnEZPHQZ7elY9zHBGxEMxfJyBjgL259fw/wAK+BzLDyjUdXpt8/JdvS/metTkl7nUsafepaRSeapkRuAm7oeOf0/lUE95JM8pG1FkOWVRwen+A/yar0VxvFVORU1ol+O/X57bF8kb3CiiiucsKKKKACiiigAooooAKKKKACiiigApVYo6sMZByMjNJRQBet74CSR7lpGLdPLCjnuc9u3Tr3q2imVwbY5XG9SWGQB6+nSsarVldNBJsZyIZPlcHkAeuPUV62BzGpSfs3Kye79XrcylBJXQt3eyyyyJnEZblAQRnGCf/r0ltf3FqzGOQ/MMc81Vorg+s1ebn5ncvkja1jcspZktmeZ1Jb54yArHIB6n8enUfznht7gx5TDc7treuOv1qK2gZrBYc7PLPzBv4m5OeCcDHfvitexQhG5JG4gZxkduxr7PLORUH7S7ku/yf5+X3nnYipyK8S1CpWMbic+9OPWnU09a4sXPmlc8+G9xjjmm09xxTK8yaszpjsIRkYqGp6icYY1nI1g+g2iiioNAooooAilHINR1O4yh9uagqWaxegUUUUFBRRRQAUoBY4FABY4FTqoUYFBMpWBVCjApaKKoyCiiigAooooAKKKKACigAnpUqoF5700rickhqp3P5VJRRVpWMW2woopQM/SqSuIACaeFApabNJ5MRkKlgOTjriuqjQdSXLHcylMdRSqwZQQetLitp4acNGZ86G0UuKMVn7Nj5kJRS4FLij2TDmQ2inUU/ZeYuYTFGKWirUIoXMwooorRRb0RNwpCcClprGuujS1sS2ULnUBFKYwhJAyeO3tVVnkuHy7IqEbSM5Iwc9Mj0H50/UARMj4GF7nt71BLI8VnOYCwkQHIVSAgB6evTOT7CuvFTdKLjsmtNN/Lfy8n2PQowjyqUVqQK0/lk7DtxngY49cenHWkMssVk08aK2xsPlVPB/X+f+FX+1i14JWjHlY27eCQMY6496df3qqXgiiEZ2gHD525wSOnXt1x145rx8RnMa1Fx5vh8tb200/z+7Y7fZtPbcgvLwTKiRqqqEAbA4J4J689c96pUUV8xUqzqu82dMYpKyCiiisxhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAOjCFwJCQp7jt6Vr/ZbCAb4/MuCSdpY4VTn26n9Kxqu2NysbeW6ocjCu+SF5z0APv2713YKrShNKpG+u/X07fgZzi3qbMCi4dXdVyBj1J+tbEQCgADgViwxXED4bAUZ+btxn8unf1rWt5N8YJBB7g9Qa+9cqM6VqO3X/gnjYpSve+hZpDQDmlrw69N6nPFjSMioqlqNvvGvNqLqdMGJUcg6GpKa/3axexrHciooorM2CiiigAquwwxFWKilHzA+tJlwepHRRRSNAooooAnRdo96dRRVGG4UUUUAFFFFABRRRQAUqqWNKq7j7VKAB0pqNyJSsIqhRS0UVoZBRRRQAoGTUgGBikUYHvS1tCNjOTuKOtJIiuhVgCD1BpRTJpPLiLbS2Ow616WCg3NW3MJ6vQyLjfaz7IpWCsMMCc7Rnr+ta8R+Qc9qw53a5uAfLbaDkrg5I55/wA+tasEqmMYIxivcnD2kZRXRr8jevF8kW9y3RUYf3oEqsSAwOOtedLCPsct2SUE4pu6qt1cGFNwGeQDjrVUsJzMaTk7InjuY5JHRTyhwalrAjbOoMwcHnkqTj8+/wCVbSOCOtaywylDnitDStS9m1ZktFN3UbvauZ4fyMbjqKbu9qCSatUmFxSajY8YpScVTvJmjQbcZyM5OK7KFK2pUIuTsiG8mQAqepHSs63w0xTa53jHyPtI5zU6zx+SzTc5yJDtzgcgYP8AnqOtVDcWv2aXyGmLKck4Xp0HUepA/I9sVzZhmEFD2TjbS+tk7d7eh61GnyJopXtvb20zJFMZcccdjxnP6/561KdJI0rl3OWNNr4WtKEptwVl0/rU7oppahRRRWRQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAalkBJbySMS7sdoDnGMcnHPPGO3cfjq2d3HHGFLAY9awNOkEd6gIJWTMZwOfm4qydRhiZhbpJjGBITgnpnI6Y6/pX1OV5rToUHGpv3f3+bOStR9o+VnSi6QpkMMetRWt6z3HluDtc/uztxnuay1zdEtDIrRu2WCjPl59R7Zq7aSrHeODCVYsEGFxgY4/QZr23WpV0o07aq79ey/z8jhlh1CL6s1jUb9RUhpj9K+drqzZjTYykb7p+lLRXKbEFFFFZHQFFFFABTJBlPpT6QjIIoGtGV6KKKk2CiiigCzRRRVGAUUUUAFFFFABTlTPJ6Uqp3P5VJVKPczlPogoooqzMKKKKACnKvc0KueTT60hHqyJS6IKKKUCt4x5mZt2FpGGRS0HpXoYfR3RjIyNRtmdwyqCo64OCaqCSSI7Q4UHOCewz36fXpW44zWXeRqUPY4zXvU4OonKMmm15fLc66Fa6UJK6IjdSxxAt3xggHGPXJoguWglJlG3cAaZaTMJz8v7tVJKqTwPbn/635Upnsrtpg8gikV+WbaN3pjPP+cn286WYSpVFGr0uui6d9v+Bc6nTjqraM0Uu1fjoR1yKqXsm9ODgg5B71TkDwPnejBv4k+7nuP8+opDHMyBjjDDIy4yfw/A13RxmH5Fz6NrVfmRHDcsuaJND++aZo4SHA3ZB4B/oO/4VZhuZIwhk6MOo5HUjH6VVeRNPgS48tiz9MnOPYj0OOv19KbaTi8sfKWPfOg/vHPUc49BnA988da8qlmUYVeS+nZ9v62XkzSpTUltoaL33zBE27iCck4HAzUlrdGVTuPIOMjHP5ZrKgEscx3KuzgPuwVIz1+nTmp3/wBFcSRj903Tacgfj3PBr0IYqnVrOH2ej8zGeGio2W5sB/xpS1Zkd78yqwKluRkY49atCbI+9XQqMZax1Rxyoyi9SZmwMmqF467CG/Gpnk9Dk1QllQTHzMEY4JGRnP6961f7mDna9jWhTvIYk7PEY58+RsI3FfujGOP1/wDr1hShFkZUO5Rxn1961725u7dVBaOSFl5HKhvwzk+vHrz04xa/P82rqpU219OnS2uzPXoxsrhRRRXkmwUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFACo7RuroSGU5BHY1YFjcOu+KJ2j3YBIAPqMjPHBH51WrVsTFPaiLzER05OVI4J6579QOcdPxrrwVGNeqqU3ZMzqS5VdFyGERmJbSMYAUuxADMccnnJH6itWztnjbzZWG8jGFAAx26Co7FUEQIAGQDitAdK+1q4eGEhyw3tY8avXcnygaa33acaa33a8Grq2RDoR0UUVym5C33j9aSlbhjSVkzdbBRRRQMKKKKAK7DDEUlPlGH+tMqTZbBRRRQMs0UUVRgFFFOVCfpRa4m7DQCelSqmOT1pQABxS1ajYzlO+wUUUVRAUUAE9KeE9aai2JtIYBnpTwuOtOAxRWsYJbkOVwoopQK1jFyIbsAFLRRXVCk+hk5BSMe1BOKguJTHGWAyRXo4eg20kTq3Yc7AVk3jAttLFRjGew/D8/89XvfjcQRgj3z+oqnNI0gDlSIycbuv+TXqr2dOldysn1X6HbQoyUtUPjJNmyW5USNwVPVsg9/w6f41z5JYknqeTXR2kq2QeSSfMSnDKoJHPGTxxWJezRTTOY4wMtkvk89f/rdeeM96+Iza0knzbdO/n+mu/Q9Kk/eaS/rsaFtfQJZRi5+dsHBB3N1Ixjt1B544PrVSXVbmQLtbYy45XA6Yx/n6elUaK4J4+tKKinayS030t136GipRTuyzPf3NwhSWXcpOcbQBn/P9fWmRXUsCYibyznJZeGPTjPpxUNFc/t6vNz8zv3vqVyRta2heTVJkikTJDOSxZTjJOcn689sdKsWFxbvbG2ncq5ffvI9jxn9fxxWTRW0MZVi43d0tPk3fp+ZLpx6aHTTiG4VntnBkg+VxjB6kc9sdfoP0Z5rwv5coAYDkdcVjWd/NZMTEcA9QO/Bxn16/wD6q0zKt5arct8smNp245xj8+v+R0+oyrN1ZQfq1287v+m2csqDWj2Jnn+XqPoKjt/3s/Rml6oq8ZP17VW3VLFIscMshLIRx5gG7bn24/D3r2sfioewai9/6/roKNPlTKOpz3Ul0UuWyydAOmDyD+tUqdJI0sjSOcsxLE+pNNr82rVHUm5vqdsI8sUgooorMoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAq/YTiK3nVpxGpKgAk8568AdOOfoOvFUKtWlotyGZ5hEqsASVJ6g+n0/WujCSqRrJ0leXQidram1b3D/ACxw5bcNyFgFyPxrWsJHltEeT73esyGLMawwRyMg48wvyw9DwDj29q17aLyIFj3FiO5r7arVqzo/vlZ2XTr1PHxPJ03JD1pj9KcetNftXhVXuZw6DKKKK5jYif75ptOk+9Taye5vHYKKKKBhRRRQBFKOhqOppfu/jUNSzWOwUUUUFFmlClulOVM8npUnStFE5ZTtsNVAPc06iirsZt3CinBSaUKB71Sg2S5JDQCacFA606itFBIhybCiijFWlfYkKMUuKWtY0+5Dl2DFFFITgV10qV2Ztlae9WCULIMKRw3v6VKkodQwPBGao6lEXUN1VeqjvVeC5MI2O2R1U56ivbhh4OytutP1NlRUqalHc1i3pVeZgVI7YqB7olgqYZm4AzjNVTeB1zXTSoqMrdQhRluQSMY5Nh2lC2TuUHr15xmn3l59ijzDFmKQ4BI+Xp7r7tjn147VGixzOoyxkJHyEYDc9M544qDUr7dbxwJDLEMdHJ6eoPcHkV87nNWEXJx06bddOnn8vzPThG9kynf3Ru7pmGNgOF4xkdP6VVoor5CrUlVm5yer/r8DsjFRVkFFFFZjCiiigAooooAKntrya0LGIgbhg8f16jt+VQUVUJyhLmg7MTSaszdnhZ4Y7pFISQDg9jj+VV7svFpwQyMBI2THg4Ppz07fpUmiXSZNtOWKn7gLHA/DPr6D1o8Q4S4hiX7qpnrx1r2p4mpPDOotrWfk3pb9fmc6l76pNefyRjUUUV4Z0hRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABV+waLyXUyFZC2QoB5wOO/uaoVu6YvlmFVSHzNufMxknJPQjHrg/SvTyilUqYleztda6/IyrS5YXNbTpN4dGwHQ4IBzgfX860Kr2lstuhA6sdx+tWD0r6XG1XJ+89TwpuMptx2G0x+tPpjferxamxrHcbRRRWJoRyfe/CmU+T734Uys3ubx2CiiikMKKKKAGyfcNQVYb7p+lV6TNIbBRRRSLNCinBc08LjoK6owbPOckhgT1p4AHSlxRiuiNF9EZOYlLgDqaWqt7IFt2PUrhgPoc13YbB+0kkyOZt2RawKKghuEkAwwPGetTZFaVMFKDtYlya3FopCwFQpco8jxg/MhwacMHOS0FdvYnopu8e1IX960jhX2FdjicU0tVS8uTDCWQjIIqFb0MmScEdcGvQo4T7y1Sk1zE1w4wc1lxFWeVC21uNn1z1Hv/nBonmaWXajctgDnvT4LfZumnT5IzkPuADc+/b06VOPrQjT9h9q6/q/3/cehRp+zjdjI7UoyyPJ8ik7iuVIx6ZHrTRbwu0gjmBKjhW+Ug8cH+X1rLnvXe4aSFpEBHUsc9c59uf8APWq3mPvL723k5LZ5z6181Wzm07JuVtL3tp5ad9dfQ7I0pPVux0EMEtuJZWUBoxlTjPJ447HrWDcSCa4eQdGOe/8AUn+daEOo79PktnC8RnJc53HtjPQjjv249Dl1wZjXVWMHGV76v8rNd1+uhVNSu3JBRRRXlmwUUUUAFFFFABRRRQAUUUUAOjdo3DKSCPQkfyq3qQfzYndifMjDjPoc4/lVKppgRHASPvR5Bz1+Zh/SumnWkqE6V9HZ/c/+CS17yZDRRRXMUFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVJFE00gUcDPLdhVRi5NRitQJ7CAyXCuYiyLk5x8uccAk8YzXSaXaiOCNipXAIC5Pr/AJ/IVV0+wXamVIjHIVjnJP8An07CtxBgV9pgsAsFR5qi99/gePjMTze5EdSGlpD1rkryvc44ISo2+8akqM9TXDU2N4biUUUVkaEcn3vwplPk+9+FMrN7m8dgooopDCiiigBG+6fpVerNVqTNIBRRRSLL2+87W9v/AN/z/wDEUnmX2f8Aj3gx/wBdj/8AE1KH9aeDXq0q0Uv6/wAjypRfb+vvK5kvv+feD/v8f/iaYZb/AHgeRDjH/PU//E1czS5rsp4mC/pf5Ge3Rf18yiZL/wD54Q/9/T/8TWTeyXReTzI4w2zgeYfQ+3NdJVO+gV4HIUbjwCfXtXbSnGtF01Jpv0/yX5mlGqoTV4owtOlvS7tsDOSdxc7T/KtQTajzmCLrx+8P+FXILRIiWA+Y9T61Y2irp8tCmqTm5W66f5MdbERnK6ijGE2pFW3QR9T1bH9KrRvcCbdGimU53jeePT+H+proSgNV47NY5pJO7nP0raNWDt77VvTX/wAlCNeKT91GS8+obk3RqDngBzyfypZJ9R8klolHuGPH6VueWPQUhQdxTjVi38b/AA/+RF9Yjp7iObeS6dlEygIcZ+Y9PypsoIn2lI1TJztkJ7nvj+lbN9bF4SEXJyKgWwUR/Moz/KqnhnW2qyWndfdt8zqp4mCje3yM+2G4kssZcfcDSEZP/fPNQ3sl8086+XlAuXAXcq5XJ5I45yfr9KszRCCYEAHaQQD3/wA4qeKVbtXhnEe1jhARluuTyOf5ZryMbg6kW488rX+L5PR6barbudHPf30v68jmqKmuLcwysisJAoyWXt9ahr46pTlSk4TVmjsTTV0FFaFvpzNZvcsoZfLYjGflIz16enr/AIHPq6tCdJRc1bmV16CjNSbt0CiiisSgooooAKKKKACiiigAooooAKM0VLKcxwcKMIenf5m61SV02IioooqRhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUATQLbtnzpHU84AXjp6/wD1q1rNEG0AAKeRjo3Xn371h1Nb3DQOOTsJBZQeuP68mvWynMKeDrc04Jrv1RlVpucbJnaQ42jFWB0rH0++D7ULh89GXoT/ADrXU5FfX4lxqRVSDumeBVhKErMdTT1p1IeteFV2HESoz1NSVGeprkqbG8BKKKKyLI5PvfhTKfJ978KZWb3N47BRRRSGFFFFABVarNVqTNIBRRRSLNCgHFRq/Y/nUlbJnC1bceH9adUVAJFaqb6kOPYlzQwVxhhkZzzTA/rTgwNb067i7xZm4D6KbRWjrN7kcg6im0uaarCcBaKTNLW8arJcRCAaYV/OpKQjiu2jXZDVijcRgg8VlxIgklkcZKAMo/H/AD/ga0dRl8tQo4LfxdhVOCEzfPyqqfl5/X6/59K9SrF16SpJ76+i8/U78O+WHM9hiXMbuN4KFiSZGPK8k9gD1/maYDaRyyyKpdz825uNze2PxP6Vca08t1eNV3KcgHpmqpsyq4PWud5XGUrKWnfRu+35G8a0HqOSdrqCaExjDAbFUZ5HbHfgVz80flTOmc7TjOR/QkVtxbIZUfe3mgjAIwoOe5z0xVPUrOSNEnaaOXdwWUBeO3ux6+/TNfM5vhJRTS15W/W3V/f6/Lr005RUtNL/AJmbRRRXzp0hRRRQAUUUUAFFFFABRRRQAoG4gDGTxycVavyPMiQIF2RBDgY3YzzVrRLZJZ2lkcAJ2zyfXv8Ah+PtT/EKqLuJ0I2smcD69f8APpXoQpQjg5Sk/ebVvRP/AD/IwdS9VRsY9FFFeebhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAF3T5m85YCVKMSQrLnLY49xkgV0unXYliRWdWcjPBHIrja6DTfM2QyIhEe3hQ3A5wep7kV9JkeKnJvDSa5d9b/AIHFjKUZQudHSGoredJ0yhBwcH2NSnpXZiqfK2eRG6dmJUbfeNSVG33q82psbw3EooorE0I5PvfhTKfJ978KZWb3N47BRRRSGFFFFACNwpPtVerDfdP0qvSZpAKKKKRZZpyuV+lNoqjBq5MGDdKWoKer9j+dWpdzNw7ElFAORkUVRmKGIp4YGo6KpTaE4pktFRhiKcGB9q0U0yHFodS5pKK0Ta2IaHUU2lBrop1dSJRKdzZG5lG9sRqOAOpNTRwiNAozwMZqeivUjjG1YlylZR6IiK1XmQBT6VcIqvcRs8TBeprtw1e8rNig9TDkjaWTKIxTPLAcY+tPv7a5mhEVs5MSdRz6DPO3/e7/AIVNJYMWYsepzgdP1qs4liQR7tqZ529T1/xPtWGPwVXFJz022T1S338/I9WnUi7KL2Mm8tWtLh4yG25+UkdR/jVeuhigh1FnidXDsRhsgnA7ZI449P16VjXdt9nkYK6sueBuBI9Pr9enIr4vG4J0m5w+H8n2/wAvI7IVLvlluV6KKK881CiiigAooooAKkiglm3eWhYKMsew+pp0FrLcMFjQknOODzwT/SteKL7DaqrEx3BBbK9cEjHP4f57d+DwM68lzJpem/p/VjOc7aLcdJO0NpFZoQdgG4gY59OOO9VL5zNYpnfuRuem3/PWnEFjkkk+9SoPMs5oWkfYfmKJyTjknnj/APVX0WKyx06N2ulvw/S3n8zJSilp3/4cxKKVlZGKsCrA4IIwQaSvjjpCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACr+nrGqmfzHWZW2rtIGMj8/WqFXLG6hgV0mWQq7KfkxnADev1FdWClThXjKr8JE78uh02moV8x24LtnaOn+etaNYsNw0cayxTxvAxwpbOfoemOhrWgmWeFZEPBr7HE+yqR5qOy/X7zw60JKXM+o6mP1p5601+1eHUWjHDcZRRRWBqRyfe/CmU6T71NrN7m8dgooopDCiiigBsn3DUFTS/c/GoaTNYbBRRRSKLNFFFUYBRRRQAoJHSpFYN9aiopp2JcUyeimK+eD1p9WncyasFFFFMQoJFPDA1HRVRk0JxTJaKardjTq2TT2M2rCg0tNpw5rohO+jM5IKawp1Ield1CbuZNELpn+lZN4oD5Ynb1wOp/p3/AM99hzWXeMoQ554r3aF5wavY3w7akV42e3smkhLMSc43fKDg9f04+lc8xZnJcksTkk9c1v2iP57oGCqylWYMCOf0P0+tJ/Z1nA0z3JLlnwq4IK/XBHXn6j07fI5lhHWmlB7N97Pz9ejt+B60ZqDd92VLXTY7qySRmEZ5G5eSTnuPoOg9RVOWyuIFUvGwz2we/T+v5GtSdzK4VY2QJxtJyQfT/P8AWlF1KqqNo3KMBsnP8/8AOBWksl9pCLUem662trbbX+rjU5LW/wAjGe3miz5kMiY67lIx/nI/OkEUhjEgUlCduR68f4it64RtQtUhSVS4OSB1Y8449Mnk/wCFR6fC1laG5V4xKwIALc9Rxjp7/Q15ay29Xljfl76J37W/4b8CvatLXcyhYz+U0jLtC5yCDnjOfp071bsLGB7f7VOzeXu2bcd8H8+349qu20k0k5RE4dtzBcjHqc/160+WNZGEMKqEBySvIJ5Oc9e9ejQyde2jBL3lvfVeT+XZ9e5EqrWj/AeUi05JBblt85BIzjGGPGByO4/ycRMj3EpldQGPXb0NTraOzo0jZKjGenHarYhAHINfT4HBww0feWupxTrJbO77ma8GB0xTbY+VOW3BXH3VYfK31/Q1ovEMfL+VUWhzccMVIwVwcEnPb3/wroxqjOhK/QqlU5tGY+oCX7a7TQ+Uzc7cY/H8aq1saj9pujHH5JCgAgyOMjPPXOOR69wfSsevzXG0vZ1Xa9n5deq6Xt3/AAR6NJ3igooorkNAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACrdnBFNDMZFY7SuCrYI68dMc/0+tVK19PAgtPMczEScBAwCkZ5/wDQRnpXbl9FVsRGDV0Z1XaOhbt45FIaLLqE2YcY3DGOcGtfTw4s08zO73qGxC+Uo4JwKvjpX2+LpU6MOSC6WPFr1nJ8rQHrTH6U801vu185V6ih0I6KKK5jYif75ptOf7xptZPc3WwUUUUDCiiigCOXoBUVPl+/+FMqWbR2CiiigZZoooqjAKKKKACiiigAp6v2NMooTsJq5PRUSvjg9KlrRO5i1YKKKKYgp6tng0yinF2YmrktApAcilroT6oya6DqQ8CgUyZXaIhG2sehxnFelhLSmkzGS6Gff3TRMFGAG43HtVBt0pztZlGRwfvc/hjj2/CnTiSO5CysGLHAYgHj6EVqW8CrGMDtXuJX5uZtRWmjt6v/AC+Z280aME0rszAlwYQE3IFGAAR069R3/wA9qbHA91MfMGCAMnue9bvlr6fpSJbpGWKrgtyayi6FNppf8Ht936mf1t2ehRjsQmSRnPsBVe8iEcZOK2dvtVS8t2kQbMBgwOcciuiliU/dj8jOnWbmnJmXAGtnm+eMuF27e+e4/Q9Papo4XnSNGG1VHKgdwTj9DUcaqt8VC7Ru6Fs/l61tJGAK5qUKcUqsr81/TX0OnEVnDbqZr2JU7kUEYIKHgN6frzUtnamNSWXGTwMdB+ZrQ2+1Lt+lN4mKk5LdnI68nHlZGEx0GKCpqTaaTGKhV7vcxuyFlDfWqF5EpUk1qEZ+tUb2JnQbQDk8g+ldlGqrG1GXvIz4/PeF0meQW+wlj/s9c/z+vNYMoQSN5Zyh5Ht7fhXQiEzWxSSSQKAd46hAAenP9fXrVBrC3W2laKZnYnbgxE7R19Pbr6Z+lfH5th3Od4JJat73T7f11/H2KU7Xv+BlUUrKyMVYEEdiKSvnWmnZnUFFFFIAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAACTgDJNW1vniTy40jKBshmjGSMAc+nQdKTT0L30RDFQh3lsdAOf6VK+m/MTDOrp2yPmPrx+f5V2YejXa56O9/63Ik43szWido/La2/eRuAS2D8pI6HHAPPua1LW6MvySIySYztKnp61hwrHZPtjhOVbEjH5icdcccVoWkMhu3PmbSrDICgcY4HHsa+woe2qQUK3a/8Awf6+Xc83EU4OLl+Jqmmt92nGmP0rxq2jZyQ6DKKKK5Dchb7x+tJRRWR0BRRRQAUUUE4BNAEDnLmm0UVJuFFFFAFhWDDNLVdWKnIqcEMMimZSjYWiiimSFFFFABRRRQAU5W2n2ptFANXJgQRkUtQgkdKlVg1WncxlGwtFFFUSKDg1J1qKnK2PpVwlbRkyVyQdaGOF5pKZcRmeFowQN3BJGcCvRwcoqaUnZGElcx7x1nuP3QLqMF8cjH+c/ma2IVHljHTFOSJEUAKOlP6V6dfF07NQ3fX00CpPnSilogoozRmvPdZt7kcoUhGaWinGtJMXKVorNIppJRyX9e1WQMUUVrLEyluN3k7thRRRWXtWKwUUUVcaqe4mhpFMdcipaYwruoVGmIyry3B3HOCetZ8G1ZSWeNUUZO/ODWjfs3mLGBkN1GcE+w96qzKYrSd4ciRhzwMfe4wfTqfwrfH1E6TjGN5eS23t/SPUw7airvcyb97WWdntywHX5uh6cD0x/T86dXhpVx9rEDYx1MgBIxjPHr3+tP1CwaIvMhiKgAsI84B4Bx2xk9OvPpXwdajXm5VZxt3O+M4q0UzOooorjNQooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooq3ZWhuHLYDKvJUOAx+nB9fxq6cHOXKhN2V2WNP2CCXG3zk/vYXAPBA5ye3GP586lnaB13hmG4Y4OKijfz5MG3QKc8BR745/Lp+Va1rF5cQBxnvgYr73LsM8LQcKkd+631/q55mKrWWm5H9iTy9uMj3qOztJFut5yqR8LwBuHTB79u9aQFLRiMWmrNarY4Y1Z2a7iGo36in1Gxyxr56rK5pBCUh+6fpS01/u1zvY2W5FRRRWZuFFFFABTJDhPrT6ilPIFIqKuyOiiikahRRRQAUqsVORSUUAWAQwyKWq6sVORU6sGGRTMpRsLRRRTJCiiigAooooAKOlFFAEivng0+oKcrkdeRVKXczlDsS0UgIIyKWrMxQ2PpTwQelR0VcZtEuKZLRUYY07f7Vp7RMjlY6ik3CjcPWndBZi0UZzRTEFGaKKd2gFzRmkoqlUkieVDqKbSg1pGaZLi0LSEZFLRXTSqcr1IaMy7sZJZvMV+2AD/DVdC9u+xkKgDJZD0BOOBkD0/GtojNQSwJIMMufavT/d1k1LRvrr/VjeniWkoy2MENOQdw+YDbll5A/yfrSGKd7ExwxFnc43bDwvQgH/AD1/LXa2RVwoAHtWXcxiNiQcE8HBratl9OpR5aej/S1vl+XkdlPEc7sZd3YvbbD1UoGOOcdM8jjrkfhVSt+ArBbNPPHHJB02sAxz/Tmse5FtuP2cvwcYI4I9c9fwx/hXw2NwkKLvGXy6/Ly9bHdCbbs/vIKKKK841CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooqe0tzcThcMUHzSFeoXuaunCVSShFXbE2krsijCFx5hIXqcdfpWx/oUihLW4Ubc4SUbc89ie3PANYtAODkda6MLi3h5XST1+f3kyjfW509p+6ZVbGSMjB7f0/GtiPkcVz8MqtaieVRIZOhHG3qNp6ntweta9jJuQgEkA8HGM/TnpX3NLFSxdG9tuvf8Ar17nj4qlb3i7SGlpD1rxsQ2mzmgIeBUVPc8UyuCo9bHTBaBTJD0FPqJzlqxlsaQWo2iiioNgooooAKrscsTUznCmoKTNILqFFFFIsKKKKACiiigApQSDkGkooAnVw31p1VqkSTs3507mco9iWiiimQFFFFABRRRQAUUUUAKCQcinq4PXio6KadhOKZPRUSsV+lSBgatO5k4tC0UUUyQooooAKXJHekooAcHNODA1HRVKbRLiiWiowxFPDA1qppkuLQtFFFUSKDS02lBrWE+jIkuotIRS0V3UZ6mTRBIOD9KymWN7mXzlZ1GMKCe5x/OtC6mEeVwSdpb8BWLcszzFWBU5wc+vT6V7E2vq8lzWf9fodmEg2ypqxlWVY2ZCq8AqOTgDqcc/0rNrc1gzGGOJIWNvgFGwePf6ketUG0yceaU+ZYwGBxgkHp+PP/66+HxuHq1KzcE2rL5abLul5f8AD+pSmuRXEsrL7asiqyoyfNubPT0qGa2mgdkdD8vUjkdu/wCI/Otby47a0EEbSbycsHUqefbPsP0pHEUto0bMUckliFyXyR9M+vNdn9kylQU2rNLddX2+92uL2rvdbf1qYlFat9ZWscSyxyMiHKqWXJfA+g6nj9fWssIzKzBSQvLEDp9a8WtQlSlyyNozUlcSiiisSgooooAKKKKACiiigAooooAKVVZ2CqCWJwAByTTzC6wiVlwhOAT3/wA/1q9pi2rRT+aqecigpvYgHk5/HpXRTw8pTUJaX11JlKyuQW9kZWkWVmhKYJ3Rkjn19KvRuLUGOIB49uDnOH68n061PGXmbeFxJn7wAxnOc/4+vHpzYFntAK5DKcg19fgMjUIt1NH08+39dTkniEnaRz9zbSQzuuzgc/KCQB1qOG3lnLCJC20ZOB0rfVfMLQzlDEcgksBhsfe9zg/16ikl8yGdobZvlQkgr976E9e1eVLI/wB+6cXe3T8d+nTu+5oq+livYrJNZFTAyrH93arEO2OvXrwfb29bdveyqgWNNzY/DgVHJLOJcrGqljwoUHJ9fr6VJKyPIBD8yqQdyjKOSM5OOh/P8K9zCQrYRfV5a3Xrptey+7btv0wmozV2jbifegJ/lilPWqtjdGdWRoyjp1Bq0etcWYQ5KljzVFxk0xjnnFNpSck0lePJ3Z0LRAeBmoOtSSHAx61HWcmbQWlwoooqSwooooAilPQfjUdKxyxNJUmyVkFFFFAwooooAKKKKACiiigAooooAcrlfcelTKwYcVXoBIORRclxuWaKjWQHg8GpKozasFFFFAgooooAKKKKACiiigB4cjrzTwQelQ0dKakS4Jk9FRiT1p4IPQ1aaZk4tC0UUUxBRRRQAUUUUAODetPqKlBIrSM7bkuPYkopAQaWtTMcKKaKdXVRnqjKSKN/EskR3MVx3BxWdbhZCEeKPy+gY4G5uwz/AJ/pVq9hnmlc7WWNRj72d3vgf5/lSLYyeXG+cumCqtwB07fhXtVKXtqfLZade+my/wCDsdlGap01eW/4EV7I0eHAWTf95iR8p5HGOnT1PSq8UtxuaQDeWAB3dCP8/wCTVh4XtcSNtC7txAAIXkdM1JpyMyZMe0dRjpj2qMPhqcZyhVWujT/rqauoo0uaOq2GfZXkbMshfbkLu9M1HdW2EyBz9eK2AmBTZIg6ketejCtCK5I6I41iHzXZiS2kU2nldw3Bjtk2YAxzyfyHXvTY410+2eJWSR3YMRtPA7EHAIP8u3eriW1xCriPjdzkcHOfUc4xSLaMxJk3AZ4Vmzj+n+RXkxypTq3n5631tr+d/uO11463d0Zk1nGZYYoUUMdu5XBGSOMZODnPp79MZpk+mIPL+zuzr/G56DJwOMD+Z/StVoHjQrHhVYFWx/F1x/P/ABqnIjxoU3NsJyVzxms/7AjJS5kvL5W+69tfXuaQr3as/wCv62KLaTdiNnEe/a2MLknpkH6Y/Gozp12CQYHUjPDcE/T1/CtKEMpEkjFY1UldxwGx2GeOv5VR1DUpbs7PMJjXODjbkccEZ6cV85isHQw9+Zvys9381t/wx0KU27KxTaNlUElcH0YH+VOSF5B8gBJ6LuGT9B1NR0V5acb7fj/wDXUmW0uHXcsLlcE5xxwM/wBKsHSbobjtUhW2k54+v06/lUljqr26pbyn/R8ndtX5uff2qeSFlIckMrYIZTkcjIr18HgaGKmowb+emvy6ef6mTnJXvoNh0mOPdLdSExKRtKDh/Uc4PoeO2aksrR7WSVlbJK/LzgdxyT9R09e3QyxwNKo3EkDgc9KupbM6Kj4IUbc4ySM5xnsOK+iWQ06TTjb73pp+Pp/wTkniLXuyhJOZoXW5UururAHOABnPf6U6KBLJ3UE+VIMrgDcpGMgn/OPzrSNoMZHDDkEdRVeOzllxHI7bUY9euMdj6Vricvp+0U6SXb779unl5kQxEGnfRD7CMMpYKVTjaCcnp/jmtDZxjiiCBYYwi9BU2K2qYnlsk9jz6k+eTaKZsYWLEr945Izxmnrbog+Ufj3qzRiso413sJzm92Y1/FtAJyqeoGTmpdMBkEjuBuLc49e/+RVu7RTAwbGMdziqumQSJiVs7XXI+bpXTOpde1v0aOhT5qDXU0QiqSwAyeppGOBTiajY5OK+axFZzd2RTiNoopjtgY9a4m7HQld2GMctmkoorI3WgUUUUAFMkbC49afUDtuakyoq7G0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKeshXg8imUUA1csBgwyKWq4JByDUqyA8Hg07mbjbYfRRRTICiiigAooooAKKKKACjpRRQA8SetPDA9KhopqTIcEyeiow5HXmnhgelWmmQ4tC0UUUyQooooAAcU8NmmUVUZNCauS0oNRhvWn1tGfVGTj3HdaMD0ptLmuqGIa6mbgNkiSRdrDIyDSqiqMAUuaWuhYqTVrkuL2DA9KTApaKqNcnlGFaaV9KlpCO9dlKu+4tirIgx0+orKvAAQuQMnGa2pBkGsYQGS7OSMR4Zg3IIzz9PpXfPEclByfXT0udmF1d30Kepu1vaRQq2VlXJzk4GQePxGelY9aep27Y89XZ0ODtXlUHTrnjnOOKzK+AzOTliHfayt6d/n9/fU9elblCiiivPNQrbsz51gokzIykBIwNpx0yvr0649c9BWJW1pflQWTTvGGYybd2TnHH+ev8q9HK5zhXvHt/l/SMa3wmhYkOox1rTVRisyxYtIAoYRgAKD29a1lHSvva1ZuCk1Y8XEaTsKFpQoFLRXk1K7WhgkFFBOKbmuGdXuWojqTIpKKy9qy+UHVHGHUMM5wRQMKoVQAAMADtQSB1phYnpSniJcvK2NQFZuwplFFcjdzZKwhOBk1ETk5pXbceOlNrKTubxjYKKKKRQUUUUANdtq+5qCnO25vYU2pZrFWQUUUUFBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAPWQrweRUoYMMiq9KCQcg0XJcbliimLIDweDT6ozasFFFFAgooooAKKKKACiiigAooooAeHI6808MDUNFNSZLgmT0VGHI6808MD0q00zNxaFooopkhSgkUlFAEgYGlqKnBiOtaxn3Icew+ikBBpa0IFzS02itI1GtyXHsOooorrpT1sZNEb8CsXUFBx1J+taGoTvCoARtp6sO1ZLK8jHc6tgjIf5fT/H1r241I06Lcu3a/odeFpu6mRag32a2SEwZRsCRgefUjOPUcdeh68isiWKSFtsilWxnBroJZzFLHiIbQA64bJ5x3/Dp7UyVIr+MG5by5DzvA64OB9Byc/QGvlsdgZ1kqvNpsu3muu7v5fej0ac3Hdepz9FaF1pqWsSym4yjZ24UZPTpz6n9D1qCygjuZvLdmBIOMD2JJP09O9eI8NUVRU2tXtquuxupprmGW1u91MI0xnqfp9Oprbs7WWzdhcTh0YY2A5JGFGPyPb0HpRElnZSmLy33KrAyDgkn8+MdKZ5LcFizRryAfT8+OnrX0OW5VNx9rHddddfJHPUqc2j0RPZzmIldhwPmOPQ//rrbiYMgYdCMisaIB78tLknrhlIIyeh49+v/ANattQAoA6V7dao3QTm9f+B17s8zFKPMrC0hNKabXiVJ/eZRQUUUwv6VztpbmqVxxIFNLE9KbRWTm2WopBRRRUFBUbvngUO+eBTKiUjSMerCiiipNAooooAKZI21eOpp54Gars25iaTKirsSiiikahRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAU9ZCvB5FMooBq5YDBhkUtVwSDkGpVkB4PBp3M3G2w+iiimQFFFFABRRRQAUUUUAFFFFABRRRQA9Xx15p4IIyKhoBI6VSkQ4J7E9FMV+x/On1SdzNprcKKKKYgp4f1plFNSa2E0mS0VGCRTwQa2jJMhxsLTqbRWsJ8pm1cVlDDBFU7y1EkR2rz7DmrtFerh8VKGlzNNxd0Y8Vi333UBiOg7VVu4BGcKOTzgV0BWqtxbrJzzn1FetRrRlDkil5XN4Yl895GPcWz3VnFDnY2eA3Qc7ecd857HjPpVG0sZBcszPt8obwwzhu/Xr3H/wBatCaB4NrrtDJ0Kjrz1pbiTfAqeZlgBuYNw/PT14xnn3rwMTlajWhOej9fOy+7fT0Z6UKl17uzGsFe6V0bdvOcYPTvzgVsRwrtGRmqNlbDhjn1wa1kFe9Lmw9Pkvc87E1E2kuhAbKJ2RiuNhyMVaopDXjYrEyn8XQwV5biGiimMe1eVKVtWbJX0Bmz9KbRRWDdzVKwUUUhIAyaQC1Ez54HShmLfSm1DkaxjbVhRRRUlhRRRQAUUU122r70AtRkjZ+UfjUdFFSbJWQUUUUDCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHrIV4PIqUEEZFV6UEg5FFyXG5YopiyBuDwafVGbVgooooEFFFFABRRRQAUUUUAFFFFABTlYr9KbRQDVyZWDUtQVIr9jVqXcylDsPoooqiAo6UUUAPVs/WnVFUinI961hK+jM5RtqOBpabSiumnLoZSXUWo26VJUEsqpkEjPXFexhG5NWMra6GffSKoIJ5OeKoBMLskJUZ3BsZA7f5+lT3Ewe6G0huMbW6Hn/ACfwq9BaZi/e/Mx+9716VZOcuVuyj163/wCAehGaowTfUq2tyYm2SjHvnitaGRZFDKwIPcVny6WHclSoXH3cf4U2FZbO6CFQ5cHG3j36dqid6qcZWfn/AMAxqRp1FeD17GvTaELGNS67WIGVznB9KOlfNV371jKCEY4FR0pOTmkrhlK7OiKsFFFMZ8cDrUt2KSuOZgtREknmk60Vm3c1jGwUUUUigooooAKKKKAEJwMmoGbcc0533HjpTKlmsY2CiiigoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAp6yFeDyKZRQDVywCCMilquCQcg1KsgPB4NO5m42H0UUUyAooooAKKKKACiiigAooooAKKKKAHKxX6VICCMioaUEg5FNSsTKNyaimqwb606tDJqwUoODSUUCJQcigdaYp7U+uiEr6mMlbQdWVqTRh0ViNx7FcnH8/5Vqk4FZN7IklxGAcqh3OR2Fe/gablCS6CofxLkNla+bIyTBmUDIJXYD9AOnb8q2woApsYAUYpxNZYmuqcVShsiak3Vldi1FJCJGjbO0o2en6U/NISBXAsZOGsQUBxNRs2fpSFiaSuCpUcmbRjYKKCcDJqJn3cDpWTdjVRuKz9hTKKKzbuapWCiiigYUUUUAFFFFABUUj54HSh3zwOlR0maRj1YUUUUiwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHK5X3HpUysG6VXoouS4plmioll/vfnUgIIyDTM2mhaKKKYgooooAKKKKACiiigAooooAKer9j+dMooTsJpMnoqEMV6VIHB+taKVzJxaHU9Wz9aZRVxlYhq4s6PKm1XUA9QRyfx7U5IIo02KihfQCkDmjf7V6CzCagoLSxk6b2HIqxoFXOB6mgsBTCxNJXJVryqS5nuUoWHFiabRRWLbe5olYKQsFHNNZ+w/OoycnJqHIuML7isxY0lFFQahRRRQAUUUUAFFFBIAyaAConfPA6Ujvu4HSmUrmkY9WFFFFIsKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAClBK9DSUUASrKP4uKkqtSqxXoadyHDsWKKjWUHrxUmc9KCGmgooopiCiiigAooooAKKKKACiiigBwcjvThIO/FR0U02iXFMm3Ke9LUFFPmJ9mT9KQso71DRRzD9mSGT0FMLE9aSik22UopBRRRSGFFFFABRRRQAUUZx1qJpey/nSGk2PZwvXr6VEzFj7elNopGiikFFFFBQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFKCV6GiigCRZf7w/KnghuhoopkSirXFooopmYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUxpAOnNFFJlxSZEzFjzSUUUjQKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//9k=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries for simulation\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# Imports for visualization\n",
    "import PIL.Image\n",
    "from io import BytesIO\n",
    "from IPython.display import Image, display\n",
    "\n",
    "def DisplayFractal(a, fmt='jpeg'):\n",
    "  \"\"\"Display an array of iteration counts as a\n",
    "     colorful picture of a fractal.\"\"\"\n",
    "  a_cyclic = (6.28*a/20.0).reshape(list(a.shape)+[1])\n",
    "  img = np.concatenate([10+20*np.cos(a_cyclic),\n",
    "                        30+50*np.sin(a_cyclic),\n",
    "                        155-80*np.cos(a_cyclic)], 2)\n",
    "  img[a==a.max()] = 0\n",
    "  a = img\n",
    "  a = np.uint8(np.clip(a, 0, 255))\n",
    "  f = BytesIO()\n",
    "  PIL.Image.fromarray(a).save(f, fmt)\n",
    "  display(Image(data=f.getvalue()))\n",
    "\n",
    "# Use NumPy to create a 2D array of complex numbers\n",
    "\n",
    "Y, X = np.mgrid[-1.3:1.3:0.005, -2:1:0.005]\n",
    "Z = X+1j*Y\n",
    "\n",
    "xs = tf.constant(Z.astype(np.complex64))\n",
    "zs = tf.Variable(xs)\n",
    "ns = tf.Variable(tf.zeros_like(xs, tf.float32))\n",
    "\n",
    "\n",
    "\n",
    "# Operation to update the zs and the iteration count.\n",
    "#\n",
    "# Note: We keep computing zs after they diverge! This\n",
    "#       is very wasteful! There are better, if a little\n",
    "#       less simple, ways to do this.\n",
    "#\n",
    "for i in range(200):\n",
    "    # Compute the new values of z: z^2 + x\n",
    "    zs_ = zs*zs + xs\n",
    "\n",
    "    # Have we diverged with this new value?\n",
    "    not_diverged = tf.abs(zs_) < 4\n",
    "\n",
    "    zs.assign(zs_),\n",
    "    ns.assign_add(tf.cast(not_diverged, tf.float32))\n",
    "    \n",
    "DisplayFractal(ns.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZeWypKgGCtiR"
   },
   "source": [
    "Mandlebrot rendering programs are both simple and infinitely complex at the same time. This view shows the entire Mandlebrot universe simultaneously, as a view completely zoomed out. However, if you zoom in on any non-black portion of the plot, you will find infinite hidden complexity. \n",
    "\n",
    "## Introduction to Keras\n",
    "\n",
    "[Keras](https://keras.io/) is a layer on top of Tensorflow that makes it much easier to create neural networks. Rather than define the graphs, as you see above, you set the individual layers of the network with a much more high-level API. Unless you are researching entirely new structures of deep neural networks, it is unlikely that you need to program TensorFlow directly.  \n",
    "\n",
    "**For this class, we will usually use TensorFlow through Keras, rather than direct TensorFlow**\n",
    "\n",
    "## Simple TensorFlow Regression: MPG\n",
    "\n",
    "This example shows how to encode the MPG dataset for regression. This dataset is slightly more complicated than Iris because:\n",
    "\n",
    "* Input has both numeric and categorical\n",
    "* Input has missing values\n",
    "\n",
    "This example uses functions defined above in this notepad, the \"[helpful functions](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/jeffs_helpful.ipynb)\". These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "To encode categorical values that are part of the feature vector, use the functions from above if the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bmb3cFrUCtiR",
    "outputId": "6aee69f2-aa06-411d-a7be-4362e4e35f7b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "13/13 - 1s - loss: 1456707.3750\n",
      "Epoch 2/100\n",
      "13/13 - 0s - loss: 672845.3750\n",
      "Epoch 3/100\n",
      "13/13 - 0s - loss: 233912.3594\n",
      "Epoch 4/100\n",
      "13/13 - 0s - loss: 55254.3047\n",
      "Epoch 5/100\n",
      "13/13 - 0s - loss: 6793.9985\n",
      "Epoch 6/100\n",
      "13/13 - 0s - loss: 973.5875\n",
      "Epoch 7/100\n",
      "13/13 - 0s - loss: 1006.3837\n",
      "Epoch 8/100\n",
      "13/13 - 0s - loss: 819.4775\n",
      "Epoch 9/100\n",
      "13/13 - 0s - loss: 617.4404\n",
      "Epoch 10/100\n",
      "13/13 - 0s - loss: 573.3674\n",
      "Epoch 11/100\n",
      "13/13 - 0s - loss: 563.9692\n",
      "Epoch 12/100\n",
      "13/13 - 0s - loss: 555.4481\n",
      "Epoch 13/100\n",
      "13/13 - 0s - loss: 548.3217\n",
      "Epoch 14/100\n",
      "13/13 - 0s - loss: 542.3599\n",
      "Epoch 15/100\n",
      "13/13 - 0s - loss: 535.7351\n",
      "Epoch 16/100\n",
      "13/13 - 0s - loss: 531.8860\n",
      "Epoch 17/100\n",
      "13/13 - 0s - loss: 526.1758\n",
      "Epoch 18/100\n",
      "13/13 - 0s - loss: 520.2935\n",
      "Epoch 19/100\n",
      "13/13 - 0s - loss: 515.1531\n",
      "Epoch 20/100\n",
      "13/13 - 0s - loss: 510.9998\n",
      "Epoch 21/100\n",
      "13/13 - 0s - loss: 506.2865\n",
      "Epoch 22/100\n",
      "13/13 - 0s - loss: 500.7303\n",
      "Epoch 23/100\n",
      "13/13 - 0s - loss: 495.5735\n",
      "Epoch 24/100\n",
      "13/13 - 0s - loss: 491.1421\n",
      "Epoch 25/100\n",
      "13/13 - 0s - loss: 486.0746\n",
      "Epoch 26/100\n",
      "13/13 - 0s - loss: 480.8650\n",
      "Epoch 27/100\n",
      "13/13 - 0s - loss: 478.1591\n",
      "Epoch 28/100\n",
      "13/13 - 0s - loss: 472.8083\n",
      "Epoch 29/100\n",
      "13/13 - 0s - loss: 466.3623\n",
      "Epoch 30/100\n",
      "13/13 - 0s - loss: 460.8217\n",
      "Epoch 31/100\n",
      "13/13 - 0s - loss: 455.6329\n",
      "Epoch 32/100\n",
      "13/13 - 0s - loss: 450.3487\n",
      "Epoch 33/100\n",
      "13/13 - 0s - loss: 445.4745\n",
      "Epoch 34/100\n",
      "13/13 - 0s - loss: 440.2110\n",
      "Epoch 35/100\n",
      "13/13 - 0s - loss: 435.0027\n",
      "Epoch 36/100\n",
      "13/13 - 0s - loss: 429.8761\n",
      "Epoch 37/100\n",
      "13/13 - 0s - loss: 424.3570\n",
      "Epoch 38/100\n",
      "13/13 - 0s - loss: 418.7068\n",
      "Epoch 39/100\n",
      "13/13 - 0s - loss: 413.6172\n",
      "Epoch 40/100\n",
      "13/13 - 0s - loss: 408.2991\n",
      "Epoch 41/100\n",
      "13/13 - 0s - loss: 403.3253\n",
      "Epoch 42/100\n",
      "13/13 - 0s - loss: 397.9519\n",
      "Epoch 43/100\n",
      "13/13 - 0s - loss: 392.1912\n",
      "Epoch 44/100\n",
      "13/13 - 0s - loss: 386.9039\n",
      "Epoch 45/100\n",
      "13/13 - 0s - loss: 382.2827\n",
      "Epoch 46/100\n",
      "13/13 - 0s - loss: 376.5594\n",
      "Epoch 47/100\n",
      "13/13 - 0s - loss: 371.0152\n",
      "Epoch 48/100\n",
      "13/13 - 0s - loss: 367.0930\n",
      "Epoch 49/100\n",
      "13/13 - 0s - loss: 360.7953\n",
      "Epoch 50/100\n",
      "13/13 - 0s - loss: 355.2081\n",
      "Epoch 51/100\n",
      "13/13 - 0s - loss: 351.5594\n",
      "Epoch 52/100\n",
      "13/13 - 0s - loss: 343.8980\n",
      "Epoch 53/100\n",
      "13/13 - 0s - loss: 341.4225\n",
      "Epoch 54/100\n",
      "13/13 - 0s - loss: 334.4555\n",
      "Epoch 55/100\n",
      "13/13 - 0s - loss: 328.9076\n",
      "Epoch 56/100\n",
      "13/13 - 0s - loss: 323.1662\n",
      "Epoch 57/100\n",
      "13/13 - 0s - loss: 318.5339\n",
      "Epoch 58/100\n",
      "13/13 - 0s - loss: 313.3682\n",
      "Epoch 59/100\n",
      "13/13 - 0s - loss: 308.1586\n",
      "Epoch 60/100\n",
      "13/13 - 0s - loss: 302.7293\n",
      "Epoch 61/100\n",
      "13/13 - 0s - loss: 297.7411\n",
      "Epoch 62/100\n",
      "13/13 - 0s - loss: 293.1888\n",
      "Epoch 63/100\n",
      "13/13 - 0s - loss: 289.2863\n",
      "Epoch 64/100\n",
      "13/13 - 0s - loss: 282.9069\n",
      "Epoch 65/100\n",
      "13/13 - 0s - loss: 277.5980\n",
      "Epoch 66/100\n",
      "13/13 - 0s - loss: 274.5194\n",
      "Epoch 67/100\n",
      "13/13 - 0s - loss: 269.3029\n",
      "Epoch 68/100\n",
      "13/13 - 0s - loss: 264.0596\n",
      "Epoch 69/100\n",
      "13/13 - 0s - loss: 258.5668\n",
      "Epoch 70/100\n",
      "13/13 - 0s - loss: 254.6998\n",
      "Epoch 71/100\n",
      "13/13 - 0s - loss: 249.3832\n",
      "Epoch 72/100\n",
      "13/13 - 0s - loss: 246.2544\n",
      "Epoch 73/100\n",
      "13/13 - 0s - loss: 240.0666\n",
      "Epoch 74/100\n",
      "13/13 - 0s - loss: 235.0115\n",
      "Epoch 75/100\n",
      "13/13 - 0s - loss: 230.9614\n",
      "Epoch 76/100\n",
      "13/13 - 0s - loss: 226.7763\n",
      "Epoch 77/100\n",
      "13/13 - 0s - loss: 222.6867\n",
      "Epoch 78/100\n",
      "13/13 - 0s - loss: 217.8021\n",
      "Epoch 79/100\n",
      "13/13 - 0s - loss: 213.8861\n",
      "Epoch 80/100\n",
      "13/13 - 0s - loss: 209.1690\n",
      "Epoch 81/100\n",
      "13/13 - 0s - loss: 205.4605\n",
      "Epoch 82/100\n",
      "13/13 - 0s - loss: 200.9314\n",
      "Epoch 83/100\n",
      "13/13 - 0s - loss: 196.6704\n",
      "Epoch 84/100\n",
      "13/13 - 0s - loss: 192.4684\n",
      "Epoch 85/100\n",
      "13/13 - 0s - loss: 188.9522\n",
      "Epoch 86/100\n",
      "13/13 - 0s - loss: 185.1580\n",
      "Epoch 87/100\n",
      "13/13 - 0s - loss: 181.2294\n",
      "Epoch 88/100\n",
      "13/13 - 0s - loss: 176.8582\n",
      "Epoch 89/100\n",
      "13/13 - 0s - loss: 172.9136\n",
      "Epoch 90/100\n",
      "13/13 - 0s - loss: 169.4359\n",
      "Epoch 91/100\n",
      "13/13 - 0s - loss: 166.0690\n",
      "Epoch 92/100\n",
      "13/13 - 0s - loss: 162.4823\n",
      "Epoch 93/100\n",
      "13/13 - 0s - loss: 159.2503\n",
      "Epoch 94/100\n",
      "13/13 - 0s - loss: 154.9118\n",
      "Epoch 95/100\n",
      "13/13 - 0s - loss: 152.5691\n",
      "Epoch 96/100\n",
      "13/13 - 0s - loss: 148.5987\n",
      "Epoch 97/100\n",
      "13/13 - 0s - loss: 145.1593\n",
      "Epoch 98/100\n",
      "13/13 - 0s - loss: 142.3996\n",
      "Epoch 99/100\n",
      "13/13 - 0s - loss: 139.3435\n",
      "Epoch 100/100\n",
      "13/13 - 0s - loss: 135.2217\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f27b1545150>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "82O2rvqiCtiR"
   },
   "source": [
    "## Introduction to Neural Network Hyperparameters\n",
    "\n",
    "If you look at the above code, you will see that the neural network contains four layers. The first layer is the input layer because it contains the **input_dim** parameter that the programmer sets to be the number of inputs the dataset has. The network needs one input neuron for every column in the data set (including dummy variables).  \n",
    "\n",
    "There are also several hidden layers, with 25 and 10 neurons each. You might be wondering how the programmer chose these numbers. Selecting a hidden neuron structure is one of the most common questions about neural networks. Unfortunately, there is no right answer. These are hyperparameters. They are settings that can affect neural network performance, yet there are no clearly defined means of setting them.\n",
    "\n",
    "In general, more hidden neurons mean more capability to fit complex problems. However, too many neurons can lead to overfitting and lengthy training times. Too few can lead to underfitting the problem and will sacrifice accuracy. Also, how many layers you have is another hyperparameter. In general, more layers allow the neural network to perform more of its feature engineering and data preprocessing. But this also comes at the expense of training times and the risk of overfitting. In general, you will see that neuron counts start larger near the input layer and tend to shrink towards the output layer in a triangular fashion. \n",
    "\n",
    "Some techniques use machine learning to optimize these values. These will be discussed in [Module 8.3](t81_558_class_08_3_keras_hyperparameters.ipynb).\n",
    "\n",
    "## Controlling the Amount of Output\n",
    "\n",
    "The program produces one line of output for each training epoch. You can eliminate this output by setting the verbose setting of the fit command:\n",
    "\n",
    "* **verbose=0** - No progress output (use with Jupyter if you do not want output).\n",
    "* **verbose=1** - Display progress bar, does not work well with Jupyter.\n",
    "* **verbose=2** - Summary progress output (use with Jupyter if you want to know the loss at each epoch).\n",
    "\n",
    "## Regression Prediction\n",
    "\n",
    "Next, we will perform actual predictions. The program assigns these predictions to the **pred** variable. These are all MPG predictions from the neural network. Notice that this is a 2D array? You can always see the dimensions of what Keras returns by printing out **pred.shape**. Neural networks can return multiple values, so the result is always an array. Here the neural network only returns one value per prediction (there are 398 cars, so 398 predictions). However, a 2D range is needed because the neural network has the potential of returning more than one value.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HbErLyX_CtiR",
    "outputId": "d3ee0abe-e5b6-4575-8f7f-666a868e0e3f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (398, 1)\n",
      "[[22.539425]\n",
      " [27.995203]\n",
      " [25.851433]\n",
      " [25.711117]\n",
      " [23.701847]\n",
      " [31.893755]\n",
      " [35.556503]\n",
      " [34.45243 ]\n",
      " [36.27014 ]\n",
      " [31.358776]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(f\"Shape: {pred.shape}\")\n",
    "print(pred[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jk_lWnScCtiR"
   },
   "source": [
    "We would like to see how good these predictions are.  We know the correct MPG for each car so we can measure how close the neural network was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vrY0Vs9oCtiR",
    "outputId": "3be93e76-a814-4349-8907-ef9125938a7e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 11.552907365195134\n"
     ]
    }
   ],
   "source": [
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i3yqZwhnCtiS"
   },
   "source": [
    "The number printed above is the average number of predictions above or below the expected output. We can also print out the first ten cars with predictions and actual MPG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Tl7hv8NnCtiS",
    "outputId": "742d6cac-12d6-43a9-d442-64cd8e3e4523"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Car name: chevrolet chevelle malibu, MPG: 18.0, predicted MPG: [22.539425]\n",
      "2. Car name: buick skylark 320, MPG: 15.0, predicted MPG: [27.995203]\n",
      "3. Car name: plymouth satellite, MPG: 18.0, predicted MPG: [25.851433]\n",
      "4. Car name: amc rebel sst, MPG: 16.0, predicted MPG: [25.711117]\n",
      "5. Car name: ford torino, MPG: 17.0, predicted MPG: [23.701847]\n",
      "6. Car name: ford galaxie 500, MPG: 15.0, predicted MPG: [31.893755]\n",
      "7. Car name: chevrolet impala, MPG: 14.0, predicted MPG: [35.556503]\n",
      "8. Car name: plymouth fury iii, MPG: 14.0, predicted MPG: [34.45243]\n",
      "9. Car name: pontiac catalina, MPG: 14.0, predicted MPG: [36.27014]\n",
      "10. Car name: amc ambassador dpl, MPG: 15.0, predicted MPG: [31.358776]\n"
     ]
    }
   ],
   "source": [
    "# Sample predictions\n",
    "for i in range(10):\n",
    "    print(f\"{i+1}. Car name: {cars[i]}, MPG: {y[i]}, \" \n",
    "          + f\"predicted MPG: {pred[i]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U_dpgVtfCtiS"
   },
   "source": [
    "## Simple TensorFlow Classification: Iris\n",
    "\n",
    "Classification is how a neural network attempts to classify the input into one or more classes.  The simplest way of evaluating a classification network is to track the percentage of training set items classified incorrectly.  We typically score human results in this manner.  For example, you might have taken multiple-choice exams in school in which you had to shade in a bubble for choices A, B, C, or D.  If you chose the wrong letter on a 10-question exam, you would earn a 90%.  In the same way, we can grade computers; however, most classification algorithms do not merely choose A, B, C, or D.  Computers typically report a classification as their percent confidence in each class.  Figure 3.EXAM shows how a computer and a human might respond to question number 1 on an exam.\n",
    "\n",
    "**Figure 3.EXAM: Classification Neural Network Output**\n",
    "![Classification Neural Network Output](images/class-multi-choice.png \"Classification Neural Network Output\")\n",
    "\n",
    "As you can see, the human test taker marked the first question as \"B.\" However, the computer test taker had an 80% (0.8) confidence in \"B\" and was also somewhat sure with 10% (0.1) on \"A.\" The computer then distributed the remaining points to the other two.  In the simplest sense, the machine would get 80% of the score for this question if the correct answer were \"B.\" The computer would get only 5% (0.05) of the points if the correct answer were \"D.\" \n",
    "\n",
    "We just saw a straightforward example of how to perform the Iris classification using TensorFlow.  The iris.csv file is used rather than using the built-in data that many Google examples require.  \n",
    "\n",
    "**Make sure that you always run previous code blocks.  If you run the code block below, without the code block above, you will get errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aLp65T9JCtiS",
    "outputId": "cc56ca71-b7cf-41ff-afbe-76cbb465dcca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "5/5 - 0s - loss: 1.2338\n",
      "Epoch 2/100\n",
      "5/5 - 0s - loss: 0.9754\n",
      "Epoch 3/100\n",
      "5/5 - 0s - loss: 0.9132\n",
      "Epoch 4/100\n",
      "5/5 - 0s - loss: 0.8415\n",
      "Epoch 5/100\n",
      "5/5 - 0s - loss: 0.7983\n",
      "Epoch 6/100\n",
      "5/5 - 0s - loss: 0.7550\n",
      "Epoch 7/100\n",
      "5/5 - 0s - loss: 0.7182\n",
      "Epoch 8/100\n",
      "5/5 - 0s - loss: 0.6823\n",
      "Epoch 9/100\n",
      "5/5 - 0s - loss: 0.6457\n",
      "Epoch 10/100\n",
      "5/5 - 0s - loss: 0.6140\n",
      "Epoch 11/100\n",
      "5/5 - 0s - loss: 0.5891\n",
      "Epoch 12/100\n",
      "5/5 - 0s - loss: 0.5572\n",
      "Epoch 13/100\n",
      "5/5 - 0s - loss: 0.5335\n",
      "Epoch 14/100\n",
      "5/5 - 0s - loss: 0.5083\n",
      "Epoch 15/100\n",
      "5/5 - 0s - loss: 0.4858\n",
      "Epoch 16/100\n",
      "5/5 - 0s - loss: 0.4658\n",
      "Epoch 17/100\n",
      "5/5 - 0s - loss: 0.4442\n",
      "Epoch 18/100\n",
      "5/5 - 0s - loss: 0.4237\n",
      "Epoch 19/100\n",
      "5/5 - 0s - loss: 0.4071\n",
      "Epoch 20/100\n",
      "5/5 - 0s - loss: 0.3884\n",
      "Epoch 21/100\n",
      "5/5 - 0s - loss: 0.3738\n",
      "Epoch 22/100\n",
      "5/5 - 0s - loss: 0.3598\n",
      "Epoch 23/100\n",
      "5/5 - 0s - loss: 0.3479\n",
      "Epoch 24/100\n",
      "5/5 - 0s - loss: 0.3361\n",
      "Epoch 25/100\n",
      "5/5 - 0s - loss: 0.3204\n",
      "Epoch 26/100\n",
      "5/5 - 0s - loss: 0.3102\n",
      "Epoch 27/100\n",
      "5/5 - 0s - loss: 0.2999\n",
      "Epoch 28/100\n",
      "5/5 - 0s - loss: 0.2881\n",
      "Epoch 29/100\n",
      "5/5 - 0s - loss: 0.2833\n",
      "Epoch 30/100\n",
      "5/5 - 0s - loss: 0.2747\n",
      "Epoch 31/100\n",
      "5/5 - 0s - loss: 0.2630\n",
      "Epoch 32/100\n",
      "5/5 - 0s - loss: 0.2533\n",
      "Epoch 33/100\n",
      "5/5 - 0s - loss: 0.2457\n",
      "Epoch 34/100\n",
      "5/5 - 0s - loss: 0.2399\n",
      "Epoch 35/100\n",
      "5/5 - 0s - loss: 0.2328\n",
      "Epoch 36/100\n",
      "5/5 - 0s - loss: 0.2301\n",
      "Epoch 37/100\n",
      "5/5 - 0s - loss: 0.2241\n",
      "Epoch 38/100\n",
      "5/5 - 0s - loss: 0.2138\n",
      "Epoch 39/100\n",
      "5/5 - 0s - loss: 0.2093\n",
      "Epoch 40/100\n",
      "5/5 - 0s - loss: 0.2020\n",
      "Epoch 41/100\n",
      "5/5 - 0s - loss: 0.1973\n",
      "Epoch 42/100\n",
      "5/5 - 0s - loss: 0.1912\n",
      "Epoch 43/100\n",
      "5/5 - 0s - loss: 0.1863\n",
      "Epoch 44/100\n",
      "5/5 - 0s - loss: 0.1815\n",
      "Epoch 45/100\n",
      "5/5 - 0s - loss: 0.1769\n",
      "Epoch 46/100\n",
      "5/5 - 0s - loss: 0.1740\n",
      "Epoch 47/100\n",
      "5/5 - 0s - loss: 0.1700\n",
      "Epoch 48/100\n",
      "5/5 - 0s - loss: 0.1647\n",
      "Epoch 49/100\n",
      "5/5 - 0s - loss: 0.1622\n",
      "Epoch 50/100\n",
      "5/5 - 0s - loss: 0.1580\n",
      "Epoch 51/100\n",
      "5/5 - 0s - loss: 0.1552\n",
      "Epoch 52/100\n",
      "5/5 - 0s - loss: 0.1499\n",
      "Epoch 53/100\n",
      "5/5 - 0s - loss: 0.1494\n",
      "Epoch 54/100\n",
      "5/5 - 0s - loss: 0.1453\n",
      "Epoch 55/100\n",
      "5/5 - 0s - loss: 0.1414\n",
      "Epoch 56/100\n",
      "5/5 - 0s - loss: 0.1415\n",
      "Epoch 57/100\n",
      "5/5 - 0s - loss: 0.1360\n",
      "Epoch 58/100\n",
      "5/5 - 0s - loss: 0.1332\n",
      "Epoch 59/100\n",
      "5/5 - 0s - loss: 0.1305\n",
      "Epoch 60/100\n",
      "5/5 - 0s - loss: 0.1274\n",
      "Epoch 61/100\n",
      "5/5 - 0s - loss: 0.1252\n",
      "Epoch 62/100\n",
      "5/5 - 0s - loss: 0.1230\n",
      "Epoch 63/100\n",
      "5/5 - 0s - loss: 0.1226\n",
      "Epoch 64/100\n",
      "5/5 - 0s - loss: 0.1201\n",
      "Epoch 65/100\n",
      "5/5 - 0s - loss: 0.1177\n",
      "Epoch 66/100\n",
      "5/5 - 0s - loss: 0.1180\n",
      "Epoch 67/100\n",
      "5/5 - 0s - loss: 0.1158\n",
      "Epoch 68/100\n",
      "5/5 - 0s - loss: 0.1135\n",
      "Epoch 69/100\n",
      "5/5 - 0s - loss: 0.1121\n",
      "Epoch 70/100\n",
      "5/5 - 0s - loss: 0.1213\n",
      "Epoch 71/100\n",
      "5/5 - 0s - loss: 0.1095\n",
      "Epoch 72/100\n",
      "5/5 - 0s - loss: 0.1092\n",
      "Epoch 73/100\n",
      "5/5 - 0s - loss: 0.1059\n",
      "Epoch 74/100\n",
      "5/5 - 0s - loss: 0.1060\n",
      "Epoch 75/100\n",
      "5/5 - 0s - loss: 0.1033\n",
      "Epoch 76/100\n",
      "5/5 - 0s - loss: 0.1030\n",
      "Epoch 77/100\n",
      "5/5 - 0s - loss: 0.1097\n",
      "Epoch 78/100\n",
      "5/5 - 0s - loss: 0.1020\n",
      "Epoch 79/100\n",
      "5/5 - 0s - loss: 0.1021\n",
      "Epoch 80/100\n",
      "5/5 - 0s - loss: 0.0962\n",
      "Epoch 81/100\n",
      "5/5 - 0s - loss: 0.0995\n",
      "Epoch 82/100\n",
      "5/5 - 0s - loss: 0.0985\n",
      "Epoch 83/100\n",
      "5/5 - 0s - loss: 0.0965\n",
      "Epoch 84/100\n",
      "5/5 - 0s - loss: 0.0994\n",
      "Epoch 85/100\n",
      "5/5 - 0s - loss: 0.0949\n",
      "Epoch 86/100\n",
      "5/5 - 0s - loss: 0.0949\n",
      "Epoch 87/100\n",
      "5/5 - 0s - loss: 0.0913\n",
      "Epoch 88/100\n",
      "5/5 - 0s - loss: 0.0932\n",
      "Epoch 89/100\n",
      "5/5 - 0s - loss: 0.0915\n",
      "Epoch 90/100\n",
      "5/5 - 0s - loss: 0.0924\n",
      "Epoch 91/100\n",
      "5/5 - 0s - loss: 0.0901\n",
      "Epoch 92/100\n",
      "5/5 - 0s - loss: 0.0877\n",
      "Epoch 93/100\n",
      "5/5 - 0s - loss: 0.0869\n",
      "Epoch 94/100\n",
      "5/5 - 0s - loss: 0.0883\n",
      "Epoch 95/100\n",
      "5/5 - 0s - loss: 0.0869\n",
      "Epoch 96/100\n",
      "5/5 - 0s - loss: 0.0857\n",
      "Epoch 97/100\n",
      "5/5 - 0s - loss: 0.0853\n",
      "Epoch 98/100\n",
      "5/5 - 0s - loss: 0.0843\n",
      "Epoch 99/100\n",
      "5/5 - 0s - loss: 0.0851\n",
      "Epoch 100/100\n",
      "5/5 - 0s - loss: 0.0880\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f27afbe1210>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QPM30WdBCtiS",
    "outputId": "a02c3d4e-239f-44b2-bb1f-f8904114ac29"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Print out number of species found:\n",
    "\n",
    "print(species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dM8-xyDxCtiS"
   },
   "source": [
    "Now that you have a neural network trained, we would like to be able to use it. The following code makes use of our neural network. Exactly like before, we will generate predictions. Notice that three values come back for each of the 150 iris flowers. There were three types of iris (Iris-setosa, Iris-versicolor, and Iris-virginica).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3YzlVpw-CtiS",
    "outputId": "a253eb73-5a42-4276-87f7-c287bdc52edb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (150, 3)\n",
      "[[9.9768412e-01 2.3087766e-03 7.1474560e-06]\n",
      " [9.9349666e-01 6.4763017e-03 2.6995105e-05]\n",
      " [9.9618298e-01 3.7991456e-03 1.7790366e-05]\n",
      " [9.9207532e-01 7.8882594e-03 3.6453897e-05]\n",
      " [9.9791318e-01 2.0800228e-03 6.7602941e-06]\n",
      " [9.9684995e-01 3.1442614e-03 5.8112000e-06]\n",
      " [9.9547136e-01 4.5086881e-03 1.9946103e-05]\n",
      " [9.9625921e-01 3.7288493e-03 1.2040506e-05]\n",
      " [9.9011189e-01 9.8296851e-03 5.8434536e-05]\n",
      " [9.9447203e-01 5.5067884e-03 2.1272421e-05]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(f\"Shape: {pred.shape}\")\n",
    "print(pred[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JCay-JrvCtiS"
   },
   "source": [
    "If you would like to turn of scientific notation, the following line can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "nXZne7ZICtiS"
   },
   "outputs": [],
   "source": [
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nOvMqI7QCtiS"
   },
   "source": [
    "Now we see these values rounded up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F9PSDexjCtiS",
    "outputId": "cce7841f-4c30-45d6-f80a-ee886986844b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print(y[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qDC7hxqECtiS"
   },
   "source": [
    "Usually, the program considers the column with the highest prediction to be the prediction of the neural network.  It is easy to convert the predictions to the expected iris species.  The argmax function finds the index of the maximum prediction for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "367mbx_PCtiT",
    "outputId": "ea639c6e-7ba5-404e-afb9-8921e0891dd1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1\n",
      " 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n",
      "Expected: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "print(f\"Predictions: {predict_classes}\")\n",
    "print(f\"Expected: {expected_classes}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lrcy0Q4xCtiT"
   },
   "source": [
    "Of course, it is straightforward to turn these indexes back into iris species. We use the species list that we created earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NTORTvygCtiT",
    "outputId": "93b065ef-9c84-4103-992c-a77fe7be8519"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(species[predict_classes[1:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ljez1ZRACtiT"
   },
   "source": [
    "Accuracy might be a more easily understood error metric.  It is essentially a test score.  For all of the iris predictions, what percent were correct?  The downside is it does not consider how confident the neural network was in each prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zth2S2OcCtiT",
    "outputId": "245de0c3-22c8-4337-c631-a2967a8d44d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9733333333333334\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jY07aiLICtiT"
   },
   "source": [
    "The code below performs two ad hoc predictions.  The first prediction is a single iris flower, and the second predicts two iris flowers.  Notice that the **argmax** in the second prediction requires **axis=1**?  Since we have a 2D array now, we must specify which axis to take the **argmax** over.  The value **axis=1** specifies we want the max column index for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FZUSWVGnCtiT",
    "outputId": "3a38bf92-837c-469e-a672-8c7f53894650"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00065001 0.17222181 0.8271282 ]]\n",
      "Predict that [[5. 3. 4. 2.]] is: Iris-virginica\n"
     ]
    }
   ],
   "source": [
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred)\n",
    "print(f\"Predict that {sample_flower} is: {species[pred]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ENSZaaRICtiT"
   },
   "source": [
    "You can also predict two sample flowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fdSSkkmwCtiT",
    "outputId": "2be33dec-832b-445f-dc82-35181e7932a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00065001 0.17222157 0.8271284 ]\n",
      " [0.9887937  0.01117751 0.00002886]]\n",
      "Predict that these two flowers [[5.  3.  4.  2. ]\n",
      " [5.2 3.5 1.5 0.8]] \n",
      "are: Index(['Iris-virginica', 'Iris-setosa'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0],[5.2,3.5,1.5,0.8]],\\\n",
    "        dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(f\"Predict that these two flowers {sample_flower} \")\n",
    "print(f\"are: {species[pred]}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_03_2_keras.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}