Reinforcement-Learning/demo/search_maze_by_deep_q_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Setting: Maze Solving\n",
    "\n",
    "This notebook demonstrates a simple maze solving algorithm based on Deep Q-Learning. According to the Reinforcement learning theory, the Q-Learning algorithm allow the agent to search the goal in 2D maze map. The agent can search and learn by the reward values in each point.\n",
    "\n",
    "In relation to reinforcement learning theory, the state of agent is 2D position coordinates and the action is to dicide the direction of movement. Within the wall, the agent is movable in a cross direction and can advance by one point at a time. After moving into a new position, the agent can obtain a reward. On greedy searching, this extrinsically motivated agent performs in order to obtain some reward as high as possible. Each reward value is plot in map.\n",
    "\n",
    "The map data that agent will observe is as follow.\n",
    "\n",
    "$$\\boldsymbol{M}_{reward} =\n",
    "\\begin{pmatrix}\n",
    "r_{1,1} & \\cdots & r_{1,j} & \\cdots & r_{1,m} \\\\\n",
    "\\vdots & \\ddots &        &        & \\vdots \\\\\n",
    "r_{i,1} &        & r_{i,j} & \\cdots & r_{i,m} \\\\\n",
    "\\vdots &        &        & \\ddots & \\vdots \\\\\n",
    "r_{n,1} & \\cdots & r_{n,j} & \\cdots & r_{n,m}\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "where $(i, j)$ is tuple of 2D position coordinates in the maze map. Then $r_{i,j}$ is reward value that the agent can get by arriving the point: $(i, j)$. This notebook defines the reward function as follows\n",
    "\n",
    "$$r_{i, j} = 1.0 - \\sqrt{(i - G_i)^2 + (j - G_j)^2} - \\mathcal{P}_{repeating}$$\n",
    "\n",
    "where $(G_i, G_j)$ is the tuple of goal point and $\\mathcal{P}_{repeating}$ is a penalty point in the case that the agent will revisit points that the agent have already reached. If the agent reaches the goal, the reward is $1.0$. This will be maximum reward.\n",
    "\n",
    "The state of agent is $s_{i,j}$. In this case, the form of agent's state transition matrix is designed to correspond with $\\boldsymbol{M}_{reward}$ as follow.\n",
    "\n",
    "$$\\boldsymbol{M}_{state} =\n",
    "\\begin{pmatrix}\n",
    "s_{1,1} & \\cdots & s_{1,j} & \\cdots & s_{1,m} \\\\\n",
    "\\vdots & \\ddots &        &        & \\vdots \\\\\n",
    "s_{i,1} &        & s_{i,j} & \\cdots & s_{i,m} \\\\\n",
    "\\vdots &        &        & \\ddots & \\vdots \\\\\n",
    "s_{n,1} & \\cdots & s_{n,j} & \\cdots & s_{n,m}\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "\n",
    "The possible actions of agent can be limited by $s_{i,j}$. Agent can select adjacent elements in $\\boldsymbol{M}_{state}$.\n",
    "\n",
    "So in this problem setting, the maze map data can be as generated below in detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem solution: Q-Learning algorithm.\n",
    "\n",
    "This notebook considers problem setteing in which an agent interacts with an environment $\\mathcal{E}$, in this case the Maze solving, in a sequence of actions, observations and rewards. At each time-step the agent selects an action at from the set of legal game actions, \n",
    "\n",
    "$$\\boldsymbol{M}_{action} =\n",
    "\\begin{pmatrix}\n",
    "a_{1,1} & \\cdots & a_{1,j} & \\cdots & a_{1,m} \\\\\n",
    "\\vdots & \\ddots &        &        & \\vdots \\\\\n",
    "a_{i,1} &        & a_{i,j} & \\cdots & a_{i,m} \\\\\n",
    "\\vdots &        &        & \\ddots & \\vdots \\\\\n",
    "a_{n,1} & \\cdots & a_{n,j} & \\cdots & a_{n,m}\n",
    "\\end{pmatrix}\n",
    "$$.\n",
    "\n",
    "$\\boldsymbol{M}_{state}$, $\\boldsymbol{M}_{action}$, and $\\boldsymbol{M}_{reward}$ can be assembled as a matrix of rank-3 of 3 channels and considered as a feature points which will be observed by the agent. The state/action-value function is as follows\n",
    "\n",
    "$$Q(s, a) = Q(\\boldsymbol{M}_{state}, \\boldsymbol{M}_{action}, \\boldsymbol{M}_{reward})$$\n",
    "\n",
    "The goal of the agent is to interact with the Maze map by selecting actions in a way that maximises future rewards. We can make the standard assumption that future rewards are discounted by a factor of $\\gamma$ per time-step, and define the future discounted return at time $t$ as $R_t = \\sum_{t'=t}^{T}\\gamma^{t'-t}r_{t'}$\n",
    ", where $T$ is the time-step at which the agent will reach the goal. This notebook defines the optimal state/action-value function $Q^{\\ast}(s, a) = Q^{\\ast}(\\boldsymbol{M}_{state}, \\boldsymbol{M}_{action}, \\boldsymbol{M}_{reward})$ as the maximum expected return achievable by following any strategy, after seeing some state $s$ and then taking some action $a$, $Q^{\\ast}(s, a) = \\max \\pi \\mathbb{E}[R_t \\mid s_t = s, a_t = a, \\pi ]$, where $\\pi$ is a policy mapping sequences to actions (or distributions over actions).\n",
    "\n",
    "The optimal state/action-value function obeys an important identity known as the Bellman equation. This is based on the following *intuition*: if the optimal value $Q^{\\ast}(s', a')$ of the sequence $s'$ at the next time-step was known for all possible actions $a'$ , then the optimal strategy is to select the action $a'$ maximising the expected value of $r + \\gamma Q^{\\ast}(s', a')$, \n",
    "\n",
    "$$Q^{\\ast}(s', a') = \\mathbb{E}_{s' \\sim \\mathcal{E}}[r + \\gamma \\max_{a'} Q^{\\ast}(s', a')\\mid s, a]$$\n",
    "\n",
    "The basic idea behind many reinforcement learning algorithms is to estimate the state/action-value function, by using the Bellman equation as an iterative update,\n",
    "\n",
    "$$Q_{i+1}(s, a) = \\mathbb{E}[r + \\gamma \\max_{a'}Q_i(s', a') \\mid s, a]$$.\n",
    "\n",
    "Such *value iteration algorithms* converge to the optimal state/action-value function, $Q_i \\rightarrow Q^{\\ast}$ as $i \\rightarrow \\infty$. In practice, this basic approach is totally impractical, because the state/action-value function is estimated separately for each sequence, without any **generalisation**. Instead, it is common to use a **function approximator** to estimate the state/action-value function,\n",
    "\n",
    "$$Q(s, a; \\theta) \\approx Q^{\\ast}(s, a)$$\n",
    "\n",
    "# Problem re-setting: Generalisation, or a function approximation.\n",
    "\n",
    "The Reinforcement learning theory presents several issues from a perspective of deep learning theory. Firstly, deep learning applications have required large amounts of handlabelled training data. Reinforcement learning algorithms, on the other hand, must be able to learn from a scalar reward signal that is frequently sparse, noisy and delayed.\n",
    "\n",
    "The difference between the two theories is not only the type of data but also the timing to be observed. The delay between taking actions and receiving rewards, which can be thousands of timesteps long, seems particularly daunting when compared to the direct association between inputs and targets found in supervised learning. \n",
    "\n",
    "Another issue is that most deep learning algorithms assume the data samples to be independent, while in reinforcement learning one typically encounters sequences of highly correlated states. Furthermore, in Reinforcement learning the data distribution changes as the algorithm learns new behaviours, which can be problematic for deep learning methods that assume a fixed underlying distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem solution: Deep Q-Network.\n",
    "\n",
    "In this problem setting, the function of nerual network or deep learning is a function approximation with weights $\\theta$ as a Q-Network. A Q-Network can be trained by minimising a loss functions $L_i(\\theta_i)$ that changes at each iteration $i$,\n",
    "\n",
    "$$L_i(\\theta_i) = \\mathbb{E}_{s, a \\sim \\rho(\\cdot)}[y_i - Q(s, a, \\theta_i)]^2$$\n",
    "\n",
    "where $y_i = \\mathbb{E}_{s' \\sim \\mathcal{E}}[r + \\gamma \\max_{a'} Q(s', a', \\theta_{i-1}) \\mid s, a]$ is the target for iteration $i$ and $\\rho(\\cdot)$ is a so-called behaviour distribution. This is probability distribution over states and actions. The parameters from the previous iteration $\\theta_{i−1}$ are held fixed when optimising the loss function $L_i(\\theta_i)$. Differentiating the loss function\n",
    "with respect to the weights we arrive at the following gradient,\n",
    "\n",
    "$$\\nabla_{\\theta_i}L_i(\\theta_i) = \\mathbb{E}_{s, a \\sim \\rho(\\cdot);s' \\sim \\mathcal{E}}\\left[\\left(r + \\gamma \\max_{a'} Q(s', a'; \\theta_{i-1}) - Q(s, a; \\theta_i)\\right)\\nabla_{\\theta_i}Q(s, a; \\theta)\\right]$$.\n",
    "\n",
    "Because of computational complexity with high rank matrix: $Q(s, a) = Q(\\boldsymbol{M}_{state}, \\boldsymbol{M}_{action}, \\boldsymbol{M}_{reward})$, this notebook adopts a **Deep convolutional neural networks(Deep CNNs)** and implements with library: [accel-brain-base](https://github.com/accel-brain/accel-brain-code/tree/master/Accel-Brain-Base) which is Python library for building the CNNs.\n",
    "\n",
    "To simplify expression, the output from the Deep CNNs as a function approximator is an affine transformation is as follows\n",
    "\n",
    "$$y'_i = F_{activate}\\left((\\boldsymbol{M}_{state}, \\boldsymbol{M}_{action}, \\boldsymbol{M}_{reward}) \\ast W_{\\theta_{i}} + \\vec{b}\\right)$$\n",
    "\n",
    "where $\\ast$ is a convolution operator. $W_{\\theta_{i}}$ is weight matrix in CNNs and $\\vec{b}$ is a bias vector. And $F_{activate}$ is an activation function such as Logistic function, Tanh function, or ReLu function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules and instantiate objects.\n",
    "\n",
    "Import Python modules from [accel-brain-base](https://github.com/accel-brain/accel-brain-code/tree/master/Accel-Brain-Base) and [pyqlearning](https://github.com/accel-brain/accel-brain-code/tree/master/Reinforcement-Learning) and instantiate objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:41.746757Z",
     "start_time": "2020-07-11T08:05:32.106064Z"
    }
   },
   "outputs": [],
   "source": [
    "from pyqlearning.samplabledata.policysampler._mxnet.maze_policy import MazePolicy\n",
    "from accelbrainbase.computableloss._mxnet.l2_norm_loss import L2NormLoss\n",
    "from accelbrainbase.controllablemodel._mxnet.dqlcontroller.dqn_controller import DQNController\n",
    "from accelbrainbase.observabledata._mxnet.functionapproximator.function_approximator import FunctionApproximator\n",
    "from accelbrainbase.noiseabledata._mxnet.gauss_noise import GaussNoise\n",
    "from accelbrainbase.observabledata._mxnet.neural_networks import NeuralNetworks\n",
    "from accelbrainbase.observabledata._mxnet.convolutional_neural_networks import ConvolutionalNeuralNetworks\n",
    "from accelbrainbase.observabledata._mxnet.convolutionalneuralnetworks.mobilenet_v2 import MobileNetV2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:41.989206Z",
     "start_time": "2020-07-11T08:05:41.747754Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import mxnet as mx\n",
    "import mxnet.ndarray as nd\n",
    "import mxnet as mx\n",
    "import mxnet.ndarray as nd\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from mxnet.gluon.nn import Conv2D\n",
    "from mxnet.gluon.nn import BatchNorm\n",
    "from mxnet import MXNetError\n",
    "from logging import getLogger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:42.165723Z",
     "start_time": "2020-07-11T08:05:41.990156Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from logging import getLogger, StreamHandler, NullHandler, DEBUG, ERROR\n",
    "\n",
    "logger = getLogger(\"accelbrainbase\")\n",
    "handler = StreamHandler()\n",
    "handler.setLevel(DEBUG)\n",
    "logger.setLevel(DEBUG)\n",
    "logger.addHandler(handler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:42.871473Z",
     "start_time": "2020-07-11T08:05:42.167683Z"
    }
   },
   "outputs": [],
   "source": [
    "policy_sampler = MazePolicy(\n",
    "    batch_size=25,\n",
    "    map_size=(20, 20), \n",
    "    moving_max_dist=3,\n",
    "    memory_num=8,\n",
    "    possible_n=5,\n",
    "    repeating_penalty=0.5,\n",
    "    ctx=mx.gpu(),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup hyperparameters.\n",
    "\n",
    "In relation to our problem setting, the function of `DQNController` is to solve Maze problem with Deep Q-Network. This object can be delegated `FunctionApproximator` and received hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:42.891162Z",
     "start_time": "2020-07-11T08:05:42.872470Z"
    }
   },
   "outputs": [],
   "source": [
    "computable_loss = L2NormLoss()\n",
    "ctx = mx.gpu()\n",
    "\n",
    "output_nn = NeuralNetworks(\n",
    "    # is-a `ComputableLoss` or `mxnet.gluon.loss`.\n",
    "    computable_loss=computable_loss,\n",
    "    # `list` of int` of the number of units in hidden/output layers.\n",
    "    units_list=[1],\n",
    "    # `list` of act_type` in `mxnet.ndarray.Activation` or `mxnet.symbol.Activation` in input gate.\n",
    "    activation_list=[\"sigmoid\"],\n",
    "    # `list` of `float` of dropout rate.\n",
    "    dropout_rate_list=[0.0],\n",
    "    # `list` of `mxnet.gluon.nn.BatchNorm`.\n",
    "    hidden_batch_norm_list=[None],\n",
    "    # `bool` for using bias or not in output layer(last hidden layer).\n",
    "    output_no_bias_flag=True,\n",
    "    # `bool` for using bias or not in all layer.\n",
    "    all_no_bias_flag=True,\n",
    "    # Call `mxnet.gluon.HybridBlock.hybridize()` or not.\n",
    "    hybridize_flag=True,\n",
    "    # `mx.gpu()` or `mx.cpu()`.\n",
    "    ctx=ctx,\n",
    ")\n",
    "\n",
    "cnn = MobileNetV2(\n",
    "    # is-a `ComputableLoss` or `mxnet.gluon.loss`.\n",
    "    computable_loss=computable_loss,\n",
    "    # is-a `mxnet.initializer.Initializer` for parameters of model. If `None`, it is drawing from the Xavier distribution.\n",
    "    initializer=None,\n",
    "    # `int` of the number of filters in input lauer.\n",
    "    input_filter_n=32,\n",
    "    # `tuple` or `int` of kernel size in input layer.\n",
    "    input_kernel_size=(3, 3),\n",
    "    # `tuple` or `int` of strides in input layer.\n",
    "    input_strides=(1, 1),\n",
    "    # `tuple` or `int` of zero-padding in input layer.\n",
    "    input_padding=(1, 1),\n",
    "    # `list` of information of bottleneck layers whose `dict` means ...\n",
    "    # - `filter_rate`: `float` of filter expfilter.\n",
    "    # - `filter_n`: `int` of the number of filters.\n",
    "    # - `block_n`: `int` of the number of blocks.\n",
    "    # - `stride`: `int` or `tuple` of strides.\n",
    "    bottleneck_dict_list=[\n",
    "        {\n",
    "            \"filter_rate\": 1,\n",
    "            \"filter_n\": 32,\n",
    "            \"block_n\": 1,\n",
    "            \"stride\": 1\n",
    "        },\n",
    "        {\n",
    "            \"filter_rate\": 1,\n",
    "            \"filter_n\": 32,\n",
    "            \"block_n\": 2,\n",
    "            \"stride\": 1\n",
    "        },\n",
    "    ],\n",
    "    # `int` of the number of filters in hidden layers.\n",
    "    hidden_filter_n=64,\n",
    "    # `tuple` or `int` of pooling size in hidden layer.\n",
    "    # If `None`, the pooling layer will not attatched in hidden layer.\n",
    "    pool_size=None,\n",
    "    # is-a `NeuralNetworks` or `mxnet.gluon.block.hybridblock.HybridBlock`.\n",
    "    output_nn=output_nn,\n",
    "    # `str` of name of optimizer.\n",
    "    optimizer_name=\"sgd\",\n",
    "    # Call `mxnet.gluon.HybridBlock.hybridize()` or not.\n",
    "    hybridize_flag=True,\n",
    "    # `mx.gpu()` or `mx.cpu()`.\n",
    "    ctx=ctx,\n",
    ")\n",
    "\n",
    "function_approximator = FunctionApproximator(\n",
    "    model=cnn, \n",
    "    initializer=None,\n",
    "    hybridize_flag=True,\n",
    "    scale=1.0, \n",
    "    ctx=ctx, \n",
    ")\n",
    "\n",
    "DQN = DQNController(\n",
    "    function_approximator=function_approximator,\n",
    "    policy_sampler=policy_sampler,\n",
    "    computable_loss=L2NormLoss(),\n",
    "    optimizer_name=\"SGD\",\n",
    "    learning_rate=1e-06,\n",
    "    hybridize_flag=True,\n",
    "    scale=1.0,\n",
    "    ctx=ctx,\n",
    "    initializer=None,\n",
    "    recursive_learning_flag=False,\n",
    ")\n",
    "DQN.alpha_value = 0.3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize map.\n",
    "\n",
    "Before executing `DQNController.lean()`, `MazePolicy` created a map at random and has `np.ndarray` of the data as `MazePolicy.map_arr`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:05:43.057754Z",
     "start_time": "2020-07-11T08:05:42.892159Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAI/CAYAAABnDp3VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAcYklEQVR4nO3df6zsd13n8dd7ueBGbKRIwNpWUJeQoFmQvam6qKmL1tIQqoZ125i1CuaKK4kku4moCZy42UTXqImLkVTpgoZFdtVqsxahURM0EaQ0BcoWbCG4XNtto2VbCCZu9b1/nLmbs6dz3vf2/Jg5Z+7jkZycme/3O/P9fL/nO2ee9ztz5lZ3BwCA5f7RugcAAHCciSUAgIFYAgAYiCUAgIFYAgAYiCUAgMGpdQ9gmaryeQYAwEp1dy2b7swSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADA4US1V1bVV9oqrur6o3LJn/RVX1rsX8D1TV8w6yPgCAVdt3LFXVU5L8cpKXJ3lhkhur6oW7FntNks929z9J8otJfna/6wMAWIeDnFm6Ksn93f2p7v67JL+Z5Ppdy1yf5O2Ly7+V5GVVtfSjxAEAjqODxNLlST6z4/rZxbSly3T340keTfJlB1gnAMBKHeQ/0l12hmj3f4B7IctsL1h1JsmZA4wHAODQHeTM0tkkV+64fkWSB/ZapqpOJfnSJI8su7Puvrm7T3f36QOMCQDgUB0klj6Y5PlV9VVV9bQkNyS5bdcytyW5aXH5VUn+qLuXnlkCADiO9v0yXHc/XlWvS/KeJE9Jckt3f6yqfjrJnd19W5K3JvmNqro/22eUbjiMQQMArEodxxM9VXX8BgUAbLTuXvoX+z7BGwBgIJYAAAZiCQBgIJYAAAZiCQBgIJYAAAZiCQBgIJYAAAYH+Y90T7yrr756o9YDm2Zra2uj1gPsz7ofo84sAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMqrvXPYYnqKrjN6gTYGtra2PWs0nbYj3Hfz2bxs/n+PKzOd66u5ZNd2YJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGCw71iqqiur6o+r6t6q+lhV/diSZa6uqker6u7F1xsPNlwAgNU6dYDbPp7k33b3XVV1SZIPVdUd3f0/di33J939igOsBwBgbfZ9Zqm7H+zuuxaXP5fk3iSXH9bAAACOg0N5z1JVPS/J1yf5wJLZ31RVH66qd1fV1x7G+gAAVuUgL8MlSarqS5L8dpLXd/dju2bfleS53f35qrouye8mef4e93MmyZmDjufJ2Nra2qj1wKbx2DneNunns0nbskoXy/Pogc4sVdVTsx1K7+ju39k9v7sf6+7PLy7fnuSpVfWsZffV3Td39+nuPn2QMQEAHKaD/DVcJXlrknu7+xf2WObLF8ulqq5arO9v9rtOAIBVO8jLcC9N8q+TfLSq7l5M+8kkX5kk3f2WJK9K8iNV9XiSv01yQ3f3AdYJALBS+46l7v7TJHWeZd6c5M37XQcAwLr5BG8AgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgMGpdQ8Altna2lr3EDgGVnUcbNp6Nol9xnHgzBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMTq17ABeDra2tdQ/hxLHPYP88fnAMHC5nlgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABgeOpar6dFV9tKrurqo7l8yvqvqlqrq/qj5SVS856DoBAFblsP67k2/r7r/eY97Lkzx/8fUNSX5l8R0A4Nhbxctw1yf59d72/iTPqKrLVrBeAIADO4xY6iTvraoPVdWZJfMvT/KZHdfPLqYBABx7h/Ey3Eu7+4GqenaSO6rq4939vh3za8lteveERWgtiy0AgLU58Jml7n5g8f3hJLcmuWrXImeTXLnj+hVJHlhyPzd39+nuPn3QMQEAHJYDxVJVPb2qLjl3Ock1Se7ZtdhtSb5/8Vdx35jk0e5+8CDrBQBYlYO+DPecJLdW1bn7+i/d/QdV9dok6e63JLk9yXVJ7k/yhSQ/eMB1AgCszIFiqbs/leRFS6a/ZcflTvKjB1kPAMC6+ARvAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAIDBYfzfcJzH1tbWRq1nFTZpWzj+HG/7Y789eZu2zzZte/bizBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwEAsAQAMxBIAwODUugcAy2xtba17CFxEVnW8bdpxvWnbs0n8bA6XM0sAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAIN9x1JVvaCq7t7x9VhVvX7XMldX1aM7lnnjwYcMALA6p/Z7w+7+RJIXJ0lVPSXJXyW5dcmif9Ldr9jvegAA1umwXoZ7WZJPdvdfHtL9AQAcC4cVSzckeece876pqj5cVe+uqq89pPUBAKzEvl+GO6eqnpbklUl+Ysnsu5I8t7s/X1XXJfndJM/f437OJDlz0PE8GVtbW6tc3ZHbtO1ZhVXts01bD8eb4+D48rM5mQ7jzNLLk9zV3Q/tntHdj3X35xeXb0/y1Kp61rI76e6bu/t0d58+hDEBAByKw4ilG7PHS3BV9eVVVYvLVy3W9zeHsE4AgJU40MtwVfXFSb4jyQ/vmPbaJOnutyR5VZIfqarHk/xtkhu6uw+yTgCAVTpQLHX3F5J82a5pb9lx+c1J3nyQdQAArJNP8AYAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGJxa9wDWaWtra91DgBPJY2d/Nm2/rWJ7Nm2fcTI5swQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMKjuXvcYnqCqjt+gWKmtra2NWg/Hm+MNSJLurmXTnVkCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAwQXFUlXdUlUPV9U9O6Y9s6ruqKr7Ft8v3eO2Ny2Wua+qbjqsgQMArMKFnll6W5Jrd017Q5I/7O7nJ/nDxfX/T1U9M8mbknxDkquSvGmvqAIAOI4uKJa6+31JHtk1+fokb19cfnuS71py0+9Mckd3P9Ldn01yR54YXQAAx9ZB3rP0nO5+MEkW35+9ZJnLk3xmx/Wzi2kAACfCqSO+/2X/Id3S/yS3qs4kOXO0wwEAeHIOcmbpoaq6LEkW3x9esszZJFfuuH5FkgeW3Vl339zdp7v79AHGBABwqA4SS7clOffXbTcl+b0ly7wnyTVVdenijd3XLKYBAJwIF/rRAe9M8mdJXlBVZ6vqNUl+Jsl3VNV9Sb5jcT1Vdbqqfi1JuvuRJP8+yQcXXz+9mAYAcCJc0HuWuvvGPWa9bMmydyb5oR3Xb0lyy75GBwCwZj7BGwBgIJYAAAZiCQBgIJYAAAZiCQBgIJYAAAZiCQBgIJYAAAbVvfT/tV2rqlrJoLa2tlaxGgCOuVU9H3je2Z9V7bfurmXTnVkCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABiIJQCAgVgCABicWvcA1mlra2vdQzhUq9qeTdtv4LHDpnFMHy5nlgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABueNpaq6paoerqp7dkz7uar6eFV9pKpurapn7HHbT1fVR6vq7qq68zAHDgCwChdyZultSa7dNe2OJF/X3f80yV8k+Ynh9t/W3S/u7tP7GyIAwPqcN5a6+31JHtk17b3d/fji6vuTXHEEYwMAWLvDeM/Sq5O8e495neS9VfWhqjpzCOsCAFipUwe5cVX9VJLHk7xjj0Ve2t0PVNWzk9xRVR9fnKladl9nkqw0qLa2tla5uo2xSfttVduySfuM/XMcwMm07zNLVXVTklck+b7u7mXLdPcDi+8PJ7k1yVV73V9339zdp723CQA4TvYVS1V1bZIfT/LK7v7CHss8vaouOXc5yTVJ7lm2LADAcXUhHx3wziR/luQFVXW2ql6T5M1JLsn2S2t3V9VbFst+RVXdvrjpc5L8aVV9OMmfJ/n97v6DI9kKAIAjct73LHX3jUsmv3WPZR9Ict3i8qeSvOhAowMAWDOf4A0AMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMDi17gFcDLa2ttY9BC4SjjVglS6W3znOLAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADE6tewAXg62trXUP4VCtYns2bZ9tmlX9fDZtPcD+rPsx6swSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMDgvLFUVbdU1cNVdc+OaVtV9VdVdffi67o9bnttVX2iqu6vqjcc5sABAFbhQs4svS3JtUum/2J3v3jxdfvumVX1lCS/nOTlSV6Y5MaqeuFBBgsAsGrnjaXufl+SR/Zx31club+7P9Xdf5fkN5Ncv4/7AQBYm4O8Z+l1VfWRxct0ly6Zf3mSz+y4fnYxDQDgxNhvLP1Kkq9J8uIkDyb5+SXL1JJpvdcdVtWZqrqzqu7c55gAAA7dvmKpux/q7r/v7n9I8qvZfsltt7NJrtxx/YokDwz3eXN3n+7u0/sZEwDAUdhXLFXVZTuufneSe5Ys9sEkz6+qr6qqpyW5Iclt+1kfAMC6nDrfAlX1ziRXJ3lWVZ1N8qYkV1fVi7P9stqnk/zwYtmvSPJr3X1ddz9eVa9L8p4kT0lyS3d/7Ei2AgDgiJw3lrr7xiWT37rHsg8kuW7H9duTPOFjBQAATgqf4A0AMBBLAAADsQQAMBBLAAADsQQAMBBLAAADsQQAMBBLAACD834o5Sbb2tpa9xBYM8cAq+R4Y1XHwKatZ92cWQIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGIglAICBWAIAGJxa9wDWaWtra91DYM0cA8fbpv18Nm17NomfDRNnlgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABmIJAGAglgAABqfOt0BV3ZLkFUke7u6vW0x7V5IXLBZ5RpL/3d0vXnLbTyf5XJK/T/J4d58+pHEDAKzEeWMpyduSvDnJr5+b0N3/6tzlqvr5JI8Ot/+27v7r/Q4QAGCdzhtL3f2+qnresnlVVUm+N8m/ONxhAQAcDwd9z9K3JHmou+/bY34neW9VfaiqzhxwXQAAK3chL8NNbkzyzmH+S7v7gap6dpI7qurj3f2+ZQsuYmojg2pra2vdQzhxVrXPrIdNtEnHm2Oa42DfZ5aq6lSS70nyrr2W6e4HFt8fTnJrkquGZW/u7tPeBA4AHCcHeRnu25N8vLvPLptZVU+vqkvOXU5yTZJ7DrA+AICVO28sVdU7k/xZkhdU1dmqes1i1g3Z9RJcVX1FVd2+uPqcJH9aVR9O8udJfr+7/+Dwhg4AcPQu5K/hbtxj+g8smfZAkusWlz+V5EUHHB8AwFr5BG8AgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgIFYAgAYiCUAgMGpdQ/gYrC1tbXuIRyqTdueTbJpP5tVbc+m7TdYlTe96U0rWc+6H6POLAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADMQSAMBALAEADKq71z2GJ6iq4zcoAGCjdXctm+7MEgDAQCwBAAzEEgDAQCwBAAzEEgDAQCwBAAzEEgDAQCwBAAzOG0tVdWVV/XFV3VtVH6uqH1tMf2ZV3VFV9y2+X7rH7W9aLHNfVd102BsAAHCUzvsJ3lV1WZLLuvuuqrokyYeSfFeSH0jySHf/TFW9Icml3f3ju277zCR3JjmdpBe3/Wfd/dnzrNMneAMAK7XvT/Du7ge7+67F5c8luTfJ5UmuT/L2xWJvz3ZA7fadSe7o7kcWgXRHkmuf/PABANbjSb1nqaqel+Trk3wgyXO6+8FkO6iSPHvJTS5P8pkd188upgEAnAinLnTBqvqSJL+d5PXd/VjV0jNVT7jZkmlLX2KrqjNJzlzoeAAAVuGCzixV1VOzHUrv6O7fWUx+aPF+pnPva3p4yU3PJrlyx/UrkjywbB3dfXN3n+7u0xc6eACAo3Yhfw1XSd6a5N7u/oUds25Lcu6v225K8ntLbv6eJNdU1aWLv5a7ZjENAOBEuJC/hvvmJH+S5KNJ/mEx+Sez/b6l/5rkK5P8zyT/srsfqarTSV7b3T+0uP2rF8snyX/o7v983kH5azgAYMX2+mu488bSOoglAGDV9v3RAQAAFzOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwEEsAAAOxBAAwuOD/SHfF/jrJXz7J2zxrcbuL1cW+/Yl9kNgHF/v2J/ZBYh8k9sF+tv+5e804lp/gvR9VdefF/J/wXuzbn9gHiX1wsW9/Yh8k9kFiHxz29nsZDgBgIJYAAAabFEs3r3sAa3axb39iHyT2wcW+/Yl9kNgHiX1wqNu/Me9ZAgA4Cpt0ZgkA4NCdqFiqqmur6hNVdX9VvWHJ/C+qqnct5n+gqp63+lEenaq6sqr+uKruraqPVdWPLVnm6qp6tKruXny9cR1jPUpV9emq+uhi++5cMr+q6pcWx8FHquol6xjnUaiqF+z42d5dVY9V1et3LbNxx0BV3VJVD1fVPTumPbOq7qiq+xbfL93jtjctlrmvqm5a3agP1x774Oeq6uOL4/zWqnrGHrcdHzMnxR77YKuq/mrH8X7dHrcdnz9Ogj22/107tv3TVXX3HrfdlGNg6fPgkf8+6O4T8ZXkKUk+meSrkzwtyYeTvHDXMv8myVsWl29I8q51j/uQ98FlSV6yuHxJkr9Ysg+uTvLf1z3WI94Pn07yrGH+dUnenaSSfGOSD6x7zEe0H56S5H8lee6mHwNJvjXJS5Lcs2Paf0zyhsXlNyT52SW3e2aSTy2+X7q4fOm6t+cQ98E1SU4tLv/ssn2wmDc+Zk7K1x77YCvJvzvP7c77/HESvpZt/675P5/kjRt+DCx9Hjzq3wcn6czSVUnu7+5PdfffJfnNJNfvWub6JG9fXP6tJC+rqlrhGI9Udz/Y3XctLn8uyb1JLl/vqI6l65P8em97f5JnVNVl6x7UEXhZkk9295P9ANcTp7vfl+SRXZN3Pt7fnuS7ltz0O5Pc0d2PdPdnk9yR5NojG+gRWrYPuvu93f344ur7k1yx8oGt0B7HwYW4kOePY2/a/sVz3fcmeedKB7Viw/Pgkf4+OEmxdHmSz+y4fjZPDIX/t8ziF8ijSb5sJaNbscVLjF+f5ANLZn9TVX24qt5dVV+70oGtRid5b1V9qKrOLJl/IcfKJrghe/9i3PRjIEme090PJtu/QJM8e8kyF8uxkCSvzvYZ1WXO95g56V63eCnylj1efrkYjoNvSfJQd9+3x/yNOwZ2PQ8e6e+DkxRLy84Q7f5TvgtZ5sSrqi9J8ttJXt/dj+2afVe2X5Z5UZL/lOR3Vz2+FXhpd78kycuT/GhVfeuu+Rt/HFTV05K8Msl/WzL7YjgGLtTGHwtJUlU/leTxJO/YY5HzPWZOsl9J8jVJXpzkwWy/FLXbxXAc3Jj5rNJGHQPneR7c82ZLpl3QcXCSYulskit3XL8iyQN7LVNVp5J8afZ3yvbYqqqnZvsAeUd3/87u+d39WHd/fnH59iRPrapnrXiYR6q7H1h8fzjJrdk+xb7ThRwrJ93Lk9zV3Q/tnnExHAMLD517eXXx/eEly2z8sbB4k+orknxfL96YsdsFPGZOrO5+qLv/vrv/IcmvZvm2bfRxsHi++54k79prmU06BvZ4HjzS3wcnKZY+mOT5VfVVi39V35Dktl3L3Jbk3LvbX5Xkj/b65XESLV6TfmuSe7v7F/ZY5svPvU+rqq7K9s/4b1Y3yqNVVU+vqkvOXc72G1zv2bXYbUm+v7Z9Y5JHz52e3SB7/ity04+BHXY+3m9K8ntLlnlPkmuq6tLFyzPXLKZthKq6NsmPJ3lld39hj2Uu5DFzYu16P+J3Z/m2Xcjzx0n27Uk+3t1nl83cpGNgeB482t8H635n+5N8F/x12X7n+yeT/NRi2k9n+xdFkvzjbL8scX+SP0/y1ese8yFv/zdn+5ThR5Lcvfi6Lslrk7x2sczrknws23/t8f4k/3zd4z7kffDVi2378GI7zx0HO/dBJfnlxXHy0SSn1z3uQ94HX5zt+PnSHdM2+hjIdhg+mOT/ZPtfh6/J9vsR/zDJfYvvz1wsezrJr+247asXvxPuT/KD696WQ94H92f7PRjnfh+c+2vgr0hy++Ly0sfMSfzaYx/8xuJx/pFsP2FetnsfLK4/4fnjpH0t2/7F9Lede/zvWHZTj4G9ngeP9PeBT/AGABicpJfhAABWTiwBAAzEEgDAQCwBAAzEEgDAQCwBAAzEEgDAQCwBAAz+L2/V9jSogZ8NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt.figure(figsize=(10, 10))\n",
    "plt.imshow(DQN.policy_sampler.map_arr, cmap=\"gray\")\n",
    "plt.tick_params(labelbottom=\"off\",bottom=\"off\")\n",
    "plt.tick_params(labelleft=\"off\",left=\"off\")\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Black squares represent a wall.\n",
    "- Light gray squares represent passages.\n",
    "- A dark gray square represents a start point.\n",
    "- A white squeare represents a goal point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schedule the learning.\n",
    "\n",
    "Determine the number of learning and the number of searches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:08:25.516917Z",
     "start_time": "2020-07-11T08:05:43.058714Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 1\n",
      "Reward value(mean): 0.72168726\n",
      "Predicted Q-value(mean): 0.3228165\n",
      "Real Q-value(mean): 0.30044883\n",
      "Loss of Q-Value(mean): 3.3106699e-07\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 2\n",
      "Reward value(mean): 0.72737074\n",
      "Predicted Q-value(mean): 0.35185394\n",
      "Real Q-value(mean): 0.29695767\n",
      "Loss of Q-Value(mean): 1.4602917e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 3\n",
      "Reward value(mean): 0.7274797\n",
      "Predicted Q-value(mean): 0.35046464\n",
      "Real Q-value(mean): 0.2918044\n",
      "Loss of Q-Value(mean): 1.7120048e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 4\n",
      "Reward value(mean): 0.7276944\n",
      "Predicted Q-value(mean): 0.35259783\n",
      "Real Q-value(mean): 0.28627583\n",
      "Loss of Q-Value(mean): 2.7223687e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 5\n",
      "Reward value(mean): 0.7276595\n",
      "Predicted Q-value(mean): 0.34113201\n",
      "Real Q-value(mean): 0.281028\n",
      "Loss of Q-Value(mean): 1.6719949e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 6\n",
      "Reward value(mean): 0.7277541\n",
      "Predicted Q-value(mean): 0.35182557\n",
      "Real Q-value(mean): 0.27787262\n",
      "Loss of Q-Value(mean): 4.370446e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 7\n",
      "Reward value(mean): 0.7262966\n",
      "Predicted Q-value(mean): 0.3500482\n",
      "Real Q-value(mean): 0.2741395\n",
      "Loss of Q-Value(mean): 3.782849e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 8\n",
      "Reward value(mean): 0.7283848\n",
      "Predicted Q-value(mean): 0.34707275\n",
      "Real Q-value(mean): 0.27429387\n",
      "Loss of Q-Value(mean): 3.7202968e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 9\n",
      "Reward value(mean): 0.7279176\n",
      "Predicted Q-value(mean): 0.32964247\n",
      "Real Q-value(mean): 0.2725196\n",
      "Loss of Q-Value(mean): 2.1405202e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 10\n",
      "Reward value(mean): 0.72811174\n",
      "Predicted Q-value(mean): 0.34951118\n",
      "Real Q-value(mean): 0.2720947\n",
      "Loss of Q-Value(mean): 4.445664e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 11\n",
      "Reward value(mean): 0.72758377\n",
      "Predicted Q-value(mean): 0.34265956\n",
      "Real Q-value(mean): 0.27125221\n",
      "Loss of Q-Value(mean): 3.0017318e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 12\n",
      "Reward value(mean): 0.72698724\n",
      "Predicted Q-value(mean): 0.3472695\n",
      "Real Q-value(mean): 0.2702089\n",
      "Loss of Q-Value(mean): 4.1073552e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 13\n",
      "Reward value(mean): 0.7295966\n",
      "Predicted Q-value(mean): 0.3314108\n",
      "Real Q-value(mean): 0.2717233\n",
      "Loss of Q-Value(mean): 2.1839653e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 14\n",
      "Reward value(mean): 0.7271761\n",
      "Predicted Q-value(mean): 0.34743622\n",
      "Real Q-value(mean): 0.2705942\n",
      "Loss of Q-Value(mean): 4.3175944e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 15\n",
      "Reward value(mean): 0.72744757\n",
      "Predicted Q-value(mean): 0.3538975\n",
      "Real Q-value(mean): 0.27021283\n",
      "Loss of Q-Value(mean): 5.7007648e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 16\n",
      "Reward value(mean): 0.7278209\n",
      "Predicted Q-value(mean): 0.35173085\n",
      "Real Q-value(mean): 0.27038968\n",
      "Loss of Q-Value(mean): 5.098e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 17\n",
      "Reward value(mean): 0.72667146\n",
      "Predicted Q-value(mean): 0.35117686\n",
      "Real Q-value(mean): 0.27055752\n",
      "Loss of Q-Value(mean): 5.0723615e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 18\n",
      "Reward value(mean): 0.72804224\n",
      "Predicted Q-value(mean): 0.32869083\n",
      "Real Q-value(mean): 0.27104533\n",
      "Loss of Q-Value(mean): 1.6627971e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 19\n",
      "Reward value(mean): 0.7276897\n",
      "Predicted Q-value(mean): 0.34694722\n",
      "Real Q-value(mean): 0.27091828\n",
      "Loss of Q-Value(mean): 4.4570475e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 20\n",
      "Reward value(mean): 0.72685724\n",
      "Predicted Q-value(mean): 0.3239829\n",
      "Real Q-value(mean): 0.26959965\n",
      "Loss of Q-Value(mean): 1.20493505e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 21\n",
      "Reward value(mean): 0.72762686\n",
      "Predicted Q-value(mean): 0.3477357\n",
      "Real Q-value(mean): 0.27059415\n",
      "Loss of Q-Value(mean): 4.3552645e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 22\n",
      "Reward value(mean): 0.7266849\n",
      "Predicted Q-value(mean): 0.33239377\n",
      "Real Q-value(mean): 0.27047595\n",
      "Loss of Q-Value(mean): 2.192057e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 23\n",
      "Reward value(mean): 0.72783685\n",
      "Predicted Q-value(mean): 0.350147\n",
      "Real Q-value(mean): 0.2699975\n",
      "Loss of Q-Value(mean): 4.7356833e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 24\n",
      "Reward value(mean): 0.72722936\n",
      "Predicted Q-value(mean): 0.3452536\n",
      "Real Q-value(mean): 0.2701682\n",
      "Loss of Q-Value(mean): 3.9241328e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 25\n",
      "Reward value(mean): 0.7273581\n",
      "Predicted Q-value(mean): 0.3457255\n",
      "Real Q-value(mean): 0.27032402\n",
      "Loss of Q-Value(mean): 3.884753e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 26\n",
      "Reward value(mean): 0.7266953\n",
      "Predicted Q-value(mean): 0.34857708\n",
      "Real Q-value(mean): 0.27075183\n",
      "Loss of Q-Value(mean): 4.635687e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 27\n",
      "Reward value(mean): 0.72650385\n",
      "Predicted Q-value(mean): 0.3515291\n",
      "Real Q-value(mean): 0.26977512\n",
      "Loss of Q-Value(mean): 5.2391606e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 28\n",
      "Reward value(mean): 0.72657233\n",
      "Predicted Q-value(mean): 0.35462204\n",
      "Real Q-value(mean): 0.27065215\n",
      "Loss of Q-Value(mean): 5.634439e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 29\n",
      "Reward value(mean): 0.7269027\n",
      "Predicted Q-value(mean): 0.3446158\n",
      "Real Q-value(mean): 0.26968783\n",
      "Loss of Q-Value(mean): 4.1779727e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 30\n",
      "Reward value(mean): 0.7269568\n",
      "Predicted Q-value(mean): 0.34584954\n",
      "Real Q-value(mean): 0.27003828\n",
      "Loss of Q-Value(mean): 4.0158684e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 31\n",
      "Reward value(mean): 0.72724724\n",
      "Predicted Q-value(mean): 0.34492978\n",
      "Real Q-value(mean): 0.2700177\n",
      "Loss of Q-Value(mean): 3.8352784e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 32\n",
      "Reward value(mean): 0.7277472\n",
      "Predicted Q-value(mean): 0.35000825\n",
      "Real Q-value(mean): 0.2710946\n",
      "Loss of Q-Value(mean): 4.8118665e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 33\n",
      "Reward value(mean): 0.7262417\n",
      "Predicted Q-value(mean): 0.34631774\n",
      "Real Q-value(mean): 0.26962247\n",
      "Loss of Q-Value(mean): 4.318737e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 34\n",
      "Reward value(mean): 0.7271491\n",
      "Predicted Q-value(mean): 0.3469808\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Real Q-value(mean): 0.27100727\n",
      "Loss of Q-Value(mean): 4.1797568e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 35\n",
      "Reward value(mean): 0.72679764\n",
      "Predicted Q-value(mean): 0.34692773\n",
      "Real Q-value(mean): 0.26990673\n",
      "Loss of Q-Value(mean): 4.2021813e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 36\n",
      "Reward value(mean): 0.72756445\n",
      "Predicted Q-value(mean): 0.35093272\n",
      "Real Q-value(mean): 0.270275\n",
      "Loss of Q-Value(mean): 4.9644666e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 37\n",
      "Reward value(mean): 0.7267267\n",
      "Predicted Q-value(mean): 0.32993987\n",
      "Real Q-value(mean): 0.27075517\n",
      "Loss of Q-Value(mean): 1.8748635e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 38\n",
      "Reward value(mean): 0.7280676\n",
      "Predicted Q-value(mean): 0.34462652\n",
      "Real Q-value(mean): 0.27075002\n",
      "Loss of Q-Value(mean): 3.421545e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 39\n",
      "Reward value(mean): 0.72863805\n",
      "Predicted Q-value(mean): 0.35062137\n",
      "Real Q-value(mean): 0.270786\n",
      "Loss of Q-Value(mean): 4.6319605e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 40\n",
      "Reward value(mean): 0.727491\n",
      "Predicted Q-value(mean): 0.34867066\n",
      "Real Q-value(mean): 0.2705778\n",
      "Loss of Q-Value(mean): 4.453325e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 41\n",
      "Reward value(mean): 0.7274571\n",
      "Predicted Q-value(mean): 0.3485008\n",
      "Real Q-value(mean): 0.27116242\n",
      "Loss of Q-Value(mean): 4.0714796e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 42\n",
      "Reward value(mean): 0.7282559\n",
      "Predicted Q-value(mean): 0.34942165\n",
      "Real Q-value(mean): 0.27049258\n",
      "Loss of Q-Value(mean): 4.6654823e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 43\n",
      "Reward value(mean): 0.7278075\n",
      "Predicted Q-value(mean): 0.34639192\n",
      "Real Q-value(mean): 0.2703017\n",
      "Loss of Q-Value(mean): 4.1738233e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 44\n",
      "Reward value(mean): 0.7258529\n",
      "Predicted Q-value(mean): 0.35129148\n",
      "Real Q-value(mean): 0.26997778\n",
      "Loss of Q-Value(mean): 5.172071e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 45\n",
      "Reward value(mean): 0.72729546\n",
      "Predicted Q-value(mean): 0.33154443\n",
      "Real Q-value(mean): 0.27069396\n",
      "Loss of Q-Value(mean): 2.0102423e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 46\n",
      "Reward value(mean): 0.7278584\n",
      "Predicted Q-value(mean): 0.33384654\n",
      "Real Q-value(mean): 0.27115723\n",
      "Loss of Q-Value(mean): 2.7870112e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 47\n",
      "Reward value(mean): 0.72638464\n",
      "Predicted Q-value(mean): 0.34827885\n",
      "Real Q-value(mean): 0.27039552\n",
      "Loss of Q-Value(mean): 4.4127806e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 48\n",
      "Reward value(mean): 0.72711724\n",
      "Predicted Q-value(mean): 0.3282828\n",
      "Real Q-value(mean): 0.27018467\n",
      "Loss of Q-Value(mean): 1.5127023e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 49\n",
      "Reward value(mean): 0.7283792\n",
      "Predicted Q-value(mean): 0.3524722\n",
      "Real Q-value(mean): 0.27060544\n",
      "Loss of Q-Value(mean): 5.176096e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 50\n",
      "Reward value(mean): 0.7278372\n",
      "Predicted Q-value(mean): 0.35211495\n",
      "Real Q-value(mean): 0.27032712\n",
      "Loss of Q-Value(mean): 5.6956884e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 51\n",
      "Reward value(mean): 0.7270784\n",
      "Predicted Q-value(mean): 0.35146415\n",
      "Real Q-value(mean): 0.26972544\n",
      "Loss of Q-Value(mean): 5.238012e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 52\n",
      "Reward value(mean): 0.72579306\n",
      "Predicted Q-value(mean): 0.3518839\n",
      "Real Q-value(mean): 0.26960987\n",
      "Loss of Q-Value(mean): 5.6320685e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 53\n",
      "Reward value(mean): 0.72714525\n",
      "Predicted Q-value(mean): 0.3464441\n",
      "Real Q-value(mean): 0.2701884\n",
      "Loss of Q-Value(mean): 4.5304416e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 54\n",
      "Reward value(mean): 0.7271706\n",
      "Predicted Q-value(mean): 0.34758812\n",
      "Real Q-value(mean): 0.2697311\n",
      "Loss of Q-Value(mean): 4.756638e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 55\n",
      "Reward value(mean): 0.72690344\n",
      "Predicted Q-value(mean): 0.34872746\n",
      "Real Q-value(mean): 0.2704088\n",
      "Loss of Q-Value(mean): 4.3414122e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 56\n",
      "Reward value(mean): 0.72723114\n",
      "Predicted Q-value(mean): 0.344664\n",
      "Real Q-value(mean): 0.2703429\n",
      "Loss of Q-Value(mean): 3.857141e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 57\n",
      "Reward value(mean): 0.7265987\n",
      "Predicted Q-value(mean): 0.3469059\n",
      "Real Q-value(mean): 0.2708804\n",
      "Loss of Q-Value(mean): 3.905326e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 58\n",
      "Reward value(mean): 0.72749734\n",
      "Predicted Q-value(mean): 0.34949982\n",
      "Real Q-value(mean): 0.27027923\n",
      "Loss of Q-Value(mean): 4.916775e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 59\n",
      "Reward value(mean): 0.72646546\n",
      "Predicted Q-value(mean): 0.3462891\n",
      "Real Q-value(mean): 0.26990235\n",
      "Loss of Q-Value(mean): 4.308445e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 60\n",
      "Reward value(mean): 0.7265467\n",
      "Predicted Q-value(mean): 0.34501866\n",
      "Real Q-value(mean): 0.27046975\n",
      "Loss of Q-Value(mean): 3.620773e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 61\n",
      "Reward value(mean): 0.7266111\n",
      "Predicted Q-value(mean): 0.324264\n",
      "Real Q-value(mean): 0.2708165\n",
      "Loss of Q-Value(mean): 1.350527e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 62\n",
      "Reward value(mean): 0.72786146\n",
      "Predicted Q-value(mean): 0.3283206\n",
      "Real Q-value(mean): 0.27034843\n",
      "Loss of Q-Value(mean): 2.3748167e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 63\n",
      "Reward value(mean): 0.7264657\n",
      "Predicted Q-value(mean): 0.34368062\n",
      "Real Q-value(mean): 0.26962557\n",
      "Loss of Q-Value(mean): 3.4016197e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 64\n",
      "Reward value(mean): 0.72557694\n",
      "Predicted Q-value(mean): 0.34876153\n",
      "Real Q-value(mean): 0.2699081\n",
      "Loss of Q-Value(mean): 4.7449637e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 65\n",
      "Reward value(mean): 0.7277516\n",
      "Predicted Q-value(mean): 0.34927455\n",
      "Real Q-value(mean): 0.27065513\n",
      "Loss of Q-Value(mean): 4.4470416e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 66\n",
      "Reward value(mean): 0.72665673\n",
      "Predicted Q-value(mean): 0.32820675\n",
      "Real Q-value(mean): 0.27071503\n",
      "Loss of Q-Value(mean): 1.7098751e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 67\n",
      "Reward value(mean): 0.72640556\n",
      "Predicted Q-value(mean): 0.34530422\n",
      "Real Q-value(mean): 0.2695932\n",
      "Loss of Q-Value(mean): 3.7835824e-05\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iter: 68\n",
      "Reward value(mean): 0.7268579\n",
      "Predicted Q-value(mean): 0.34763545\n",
      "Real Q-value(mean): 0.27010384\n",
      "Loss of Q-Value(mean): 4.7256493e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 69\n",
      "Reward value(mean): 0.7274717\n",
      "Predicted Q-value(mean): 0.34939992\n",
      "Real Q-value(mean): 0.27032122\n",
      "Loss of Q-Value(mean): 4.2302752e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 70\n",
      "Reward value(mean): 0.7275699\n",
      "Predicted Q-value(mean): 0.34575057\n",
      "Real Q-value(mean): 0.27063686\n",
      "Loss of Q-Value(mean): 3.9785336e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 71\n",
      "Reward value(mean): 0.72802806\n",
      "Predicted Q-value(mean): 0.34853604\n",
      "Real Q-value(mean): 0.27046788\n",
      "Loss of Q-Value(mean): 4.513056e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 72\n",
      "Reward value(mean): 0.72717994\n",
      "Predicted Q-value(mean): 0.33103985\n",
      "Real Q-value(mean): 0.27043262\n",
      "Loss of Q-Value(mean): 1.810317e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 73\n",
      "Reward value(mean): 0.7273686\n",
      "Predicted Q-value(mean): 0.3477022\n",
      "Real Q-value(mean): 0.2708875\n",
      "Loss of Q-Value(mean): 4.538932e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 74\n",
      "Reward value(mean): 0.72841734\n",
      "Predicted Q-value(mean): 0.3562375\n",
      "Real Q-value(mean): 0.27138647\n",
      "Loss of Q-Value(mean): 6.1572144e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 75\n",
      "Reward value(mean): 0.7280574\n",
      "Predicted Q-value(mean): 0.34683678\n",
      "Real Q-value(mean): 0.27123743\n",
      "Loss of Q-Value(mean): 3.9655464e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 76\n",
      "Reward value(mean): 0.7268767\n",
      "Predicted Q-value(mean): 0.34620154\n",
      "Real Q-value(mean): 0.27018592\n",
      "Loss of Q-Value(mean): 4.194353e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 77\n",
      "Reward value(mean): 0.72817445\n",
      "Predicted Q-value(mean): 0.33407554\n",
      "Real Q-value(mean): 0.27135885\n",
      "Loss of Q-Value(mean): 2.0714328e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 78\n",
      "Reward value(mean): 0.7276999\n",
      "Predicted Q-value(mean): 0.34836125\n",
      "Real Q-value(mean): 0.27049565\n",
      "Loss of Q-Value(mean): 4.4544166e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 79\n",
      "Reward value(mean): 0.7270957\n",
      "Predicted Q-value(mean): 0.35089797\n",
      "Real Q-value(mean): 0.27065757\n",
      "Loss of Q-Value(mean): 4.829337e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 80\n",
      "Reward value(mean): 0.7264259\n",
      "Predicted Q-value(mean): 0.34896615\n",
      "Real Q-value(mean): 0.27033263\n",
      "Loss of Q-Value(mean): 4.8078477e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 81\n",
      "Reward value(mean): 0.72761315\n",
      "Predicted Q-value(mean): 0.34319773\n",
      "Real Q-value(mean): 0.27112633\n",
      "Loss of Q-Value(mean): 3.2363903e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 82\n",
      "Reward value(mean): 0.72746885\n",
      "Predicted Q-value(mean): 0.3491772\n",
      "Real Q-value(mean): 0.2697926\n",
      "Loss of Q-Value(mean): 4.6828492e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 83\n",
      "Reward value(mean): 0.7270718\n",
      "Predicted Q-value(mean): 0.33195767\n",
      "Real Q-value(mean): 0.2704697\n",
      "Loss of Q-Value(mean): 2.0025715e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 84\n",
      "Reward value(mean): 0.7275728\n",
      "Predicted Q-value(mean): 0.3287259\n",
      "Real Q-value(mean): 0.2708867\n",
      "Loss of Q-Value(mean): 1.9716363e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 85\n",
      "Reward value(mean): 0.72686887\n",
      "Predicted Q-value(mean): 0.32740158\n",
      "Real Q-value(mean): 0.27033558\n",
      "Loss of Q-Value(mean): 1.6640795e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 86\n",
      "Reward value(mean): 0.72660613\n",
      "Predicted Q-value(mean): 0.3217147\n",
      "Real Q-value(mean): 0.26962668\n",
      "Loss of Q-Value(mean): 1.3181737e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 87\n",
      "Reward value(mean): 0.72742593\n",
      "Predicted Q-value(mean): 0.3304747\n",
      "Real Q-value(mean): 0.2697171\n",
      "Loss of Q-Value(mean): 1.8276305e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 88\n",
      "Reward value(mean): 0.7277395\n",
      "Predicted Q-value(mean): 0.35007367\n",
      "Real Q-value(mean): 0.27117926\n",
      "Loss of Q-Value(mean): 4.7461475e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 89\n",
      "Reward value(mean): 0.72665566\n",
      "Predicted Q-value(mean): 0.35194573\n",
      "Real Q-value(mean): 0.27028432\n",
      "Loss of Q-Value(mean): 5.1371117e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 90\n",
      "Reward value(mean): 0.7269672\n",
      "Predicted Q-value(mean): 0.348124\n",
      "Real Q-value(mean): 0.2704197\n",
      "Loss of Q-Value(mean): 4.173433e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 91\n",
      "Reward value(mean): 0.7271521\n",
      "Predicted Q-value(mean): 0.35324407\n",
      "Real Q-value(mean): 0.27048996\n",
      "Loss of Q-Value(mean): 5.266298e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 92\n",
      "Reward value(mean): 0.72758937\n",
      "Predicted Q-value(mean): 0.34423915\n",
      "Real Q-value(mean): 0.2706886\n",
      "Loss of Q-Value(mean): 3.8129794e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 93\n",
      "Reward value(mean): 0.7271548\n",
      "Predicted Q-value(mean): 0.33362767\n",
      "Real Q-value(mean): 0.27130812\n",
      "Loss of Q-Value(mean): 2.6364049e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 94\n",
      "Reward value(mean): 0.72780234\n",
      "Predicted Q-value(mean): 0.3435525\n",
      "Real Q-value(mean): 0.26969922\n",
      "Loss of Q-Value(mean): 3.6121677e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 95\n",
      "Reward value(mean): 0.72785795\n",
      "Predicted Q-value(mean): 0.3478775\n",
      "Real Q-value(mean): 0.27081573\n",
      "Loss of Q-Value(mean): 4.293824e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 96\n",
      "Reward value(mean): 0.727211\n",
      "Predicted Q-value(mean): 0.328074\n",
      "Real Q-value(mean): 0.27053958\n",
      "Loss of Q-Value(mean): 1.7477307e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 97\n",
      "Reward value(mean): 0.7270864\n",
      "Predicted Q-value(mean): 0.34597006\n",
      "Real Q-value(mean): 0.26935112\n",
      "Loss of Q-Value(mean): 4.4119755e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 98\n",
      "Reward value(mean): 0.727853\n",
      "Predicted Q-value(mean): 0.34628356\n",
      "Real Q-value(mean): 0.26968673\n",
      "Loss of Q-Value(mean): 4.2521093e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 99\n",
      "Reward value(mean): 0.7265214\n",
      "Predicted Q-value(mean): 0.3278292\n",
      "Real Q-value(mean): 0.2701181\n",
      "Loss of Q-Value(mean): 1.91413e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 100\n",
      "Reward value(mean): 0.7266696\n",
      "Predicted Q-value(mean): 0.3271879\n",
      "Real Q-value(mean): 0.27019042\n",
      "Loss of Q-Value(mean): 1.563892e-05\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Iter: 200\n",
      "Reward value(mean): 0.72676104\n",
      "Predicted Q-value(mean): 0.34901822\n",
      "Real Q-value(mean): 0.27011144\n",
      "Loss of Q-Value(mean): 4.8732203e-05\n",
      "----------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iter: 300\n",
      "Reward value(mean): 0.7280766\n",
      "Predicted Q-value(mean): 0.34774455\n",
      "Real Q-value(mean): 0.27125397\n",
      "Loss of Q-Value(mean): 3.864362e-05\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keyboard Interrupt.\n"
     ]
    }
   ],
   "source": [
    "# Execute learning.\n",
    "DQN.learn(\n",
    "    # The number of searching.\n",
    "    iter_n=500,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize learning logs.\n",
    "\n",
    "Check the difference between inferneced Q-Values and real observed Q-Values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:08:25.709989Z",
     "start_time": "2020-07-11T08:08:25.533871Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABI0AAAI/CAYAAAD6GilwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXhU9dnG8ftAgLDvS0iALKAStoAgViwiUDesuyhSl1esK0q1Wqmt1bpbd6ut+1I3xH0DaVUs2loVBRdETQgJJJFVVtmT8/7xeJwsk2SSzMw5k3w/18V1klnO/ICBzNzzPM/PcV1XAAAAAAAAQHnN/F4AAAAAAAAAgofQCAAAAAAAAFUQGgEAAAAAAKAKQiMAAAAAAABUQWgEAAAAAACAKgiNAAAAAAAAUEWS3wuoi27durnp6el+LwMAAAAAAKDR+OSTT9a5rtu98uUJFRqlp6dr4cKFfi8DAAAAAACg0XAcpzDc5bSnAQAAAAAAoApCIwAAAAAAAFRBaAQAAAAAAIAqEmqmEQAAAAAAiL3du3erqKhIO3bs8HspiKLk5GSlpaWpRYsWEd2e0AgAAAAAAFRQVFSk9u3bKz09XY7j+L0cRIHrulq/fr2KioqUkZER0X1oTwMAAAAAABXs2LFDXbt2JTBqRBzHUdeuXetUPUZoBAAAAAAAqiAwanzq+ndKaAQAAAAAAAKnXbt2vj32c889p4EDB+rggw+uct2SJUs0fvx47bXXXsrKytJVV12lsrKyKre7+uqr9fvf/77CZYsXL9bAgQNrfOxx48Zp4cKFDfsNRAmhEQAAAAAAQDkPP/yw/va3v2n+/PkVLt++fbuOOuoozZw5U99++62++OILffTRR7rrrruqnGPKlCl69tlnK1w2a9YsnXLKKTFdezQRGgEAAAAAgIRQWFioCRMmaOjQoZowYYJWrFghySqDBg8erGHDhmns2LGSrCJov/32U05OjoYOHarc3Nwq53vmmWc0ZMgQDR48WJdffrkk6ZprrtH777+vc889V5dddlmF2z/99NMaM2aMDjnkEElSmzZtdM899+iWW26pcu69995bnTp10ocffvjTZbNnz9bJJ58sSTrvvPM0cuRIDRo0SFdddVXY32/5aqvnn39eZ5xxhiRp7dq1Ov744zVq1CiNGjVK//nPfyRJ//73v5WTk6OcnBwNHz5cW7Zsqf0PtQbsngYAAAAAABLC9OnTddppp+n000/XI488oosuukgvv/yyrrnmGs2bN0+pqanauHGjJOm+++7TjBkzNHXqVO3atUulpaUVzlVSUqLLL79cn3zyiTp37qxDDjlEL7/8sv70pz/pnXfe0a233qqRI0dWuM+SJUu07777VrgsKytL27dv18aNG9WpU6cK102ZMkWzZs3S6NGj9b///U9du3bVgAEDJEnXX3+9unTpotLSUk2YMEGff/65hg4dGtGfw4wZM3TxxRfrwAMP1IoVK3TooYdq6dKluvXWW3XvvfdqzJgx2rp1q5KTk+v051sZoREAAAAAAKjeb34jLV4c3XPm5Eh33lnnu33wwQd68cUXJUmnnnqqfve730mSxowZozPOOEOTJ0/WcccdJ0n62c9+puuvv15FRUU67rjjfgprPB9//LHGjRun7t27S5KmTp2qBQsW6Jhjjqn28V3XDTtM2nXdsLc/+eSTdcABB+i2227TrFmzNGXKlJ+umz17th544AHt2bNH3333nb766quIQ6O33npLX3311U/fb968WVu2bNGYMWN0ySWXaOrUqTruuOOUlpYW0fmqQ3saAAAAAABISF6Ac9999+m6667TypUrlZOTo/Xr1+uUU07Rq6++qtatW+vQQw/VO++8U+G+1QU9NRk0aFCVIdX5+fnq1q2bOnXqpEMPPVQ5OTk666yzJEl9+vRRenq6/v3vf+uFF17Q5MmTJUnLly/Xrbfeqrfffluff/65Jk2apB07dlT7+5NU4fqysjJ98MEHWrx4sRYvXqzi4mK1b99eM2fO1EMPPaTt27dr//3319dff13n32N5VBoBAAAAAIDq1aMiKFYOOOAAzZo1S6eeeqqeeuopHXjggZKkZcuWafTo0Ro9erRee+01rVy5Ups2bVJmZqYuuugi5efn6/PPP9f48eN/Otfo0aM1Y8YMrVu3Tp07d9YzzzyjCy+8sMbHnzp1qm644Qa99dZbmjhxorZv366LLrpIf/7znyVJ8+bNq3KfKVOm6OKLL1ZWVtZPlT+bN29W27Zt1bFjR61evVpz587VuHHjqty3Z8+eWrp0qfbee2+99NJLat++vSTpkEMO0T333PPTzKXFixcrJydHy5Yt05AhQzRkyBB98MEH+vrrr7XPPvvU/Q/6R1QaAQAAAACAwNm2bZvS0tJ++nX77bfr7rvv1qOPPqqhQ4fqiSee+GnXsssuu+yngdZjx47VsGHD9Oyzz2rw4MHKycnR119/rdNOO63C+VNSUnTjjTfq4IMP1rBhwzRixAgdffTRNa6pdevWevXVV3X99ddrr732Urdu3TRmzBhNnTq12vuceOKJWrJkyU8DsCVp2LBhGj58uAYNGqQzzzxTY8aMCXvfm266SUceeaTGjx+vlJSUny6/++67tXDhQg0dOlTZ2dm67777JEl33nnnTwPBW7durcMPP7zmP+RaOPUpx/LLyJEj3cplYAAAAAAAILqWLl2qgQMH+r2MwHv55Zd1ySWXaP78+erXr5/fy4lIuL9bx3E+cV13ZOXbUmkEAAAAAABQD8ccc4zy8/MTJjCqK0IjAAAAAAAAVEFoBAAAAAAAgCoIjQAAAAAAQBWJNAMZkanr3ymhEQAAAAAAqCA5OVnr168nOGpEXNfV+vXrlZycHPF9kmK4HgAAAAAAkIDS0tJUVFSktWvX+r0URFFycrLS0tIivj2hEQAAAIKptFSaMkWaMUMaM8bv1QBAk9KiRQtlZGT4vQz4jPY0AAAABFNJifTcc9Jrr/m9EgAAmiRCIwAAAARTUZEdV6zwdx0AADRRhEYAAAAIJkIjAAB8RWgEAAAal9Wrpd/9TvrhB79XgoYiNAIAwFeERgAAoHF5/XXpllukG27weyVoKC80Ki6W9uzxdy0AADRBhEYAAKBxKSy04223Sfn5/q4FDVNcbMeyMhuKDQAA4orQCAAANC6FhVKXLlJSknTZZX6vBg1RVGR/j1IoDAQAAHFDaAQAABqXggIpO1u64grpxReld97xe0Wor6IiKSfHvmauEQAAcUdoBAAAGpfCQqlfP+mSS6SMDGnGjKrzcAoKpAsvlJYu9WWJiEBZmbWn/exn9j2hEQAAcUdoBAAAGo89e6w6pV8/KTnZ5hp9+aV0//12fWmpdOed0qBB0j33SEcfLW3e7O+aEd6aNfb3uffeUteuhEYAAPiA0AgAADQeJSUWDPXrZ98fc4w0frx05ZXSggXSmDHSxRdL48ZJTz9tg7LPPltyXV+XjTC8ndPS0qS+fQmNAADwQZLfCwAAAIgab1iyFxo5jnTXXdKwYdJBB0nduklPPSVNmWLXrVghzZwpjR0rnX++f+tGVV5olJpqf595ef6uBwCAJohKIwAA0HhUDo0kafBg6eabpXPOsRlGp5xigZFku6tNmmTVRwsXxn+9qF5xsR2pNAIAwDdUGgEAgMbDC4369q14+aWXhr99s2bS449LI0ZIJ54offqp1LlzbNeIyBQVSUlJUo8e9ve5ebO0aZPUsaPfKwMAoMmg0ggAADQehYVS9+5SmzaR36drV2n2bKtsOfPM2K0NdVNUZK1pzZqFQkCqjQAAiCtCIwAA0HgUFlZsTYvU6NHSn/8svfyytGhR9NeFuisqstY0idAIAACfEBoBAIDGo76hkSSde67UqpX08MPRXRPqJ1xo5LUfAgCAuCA0AgAAjYPrWiVKfUOjzp2lE06QnnxS2r49umtD3bhuqD1Nknr2lFq0oNIIAIA4IzQCAACNw9q1Fvakp9f/HNOm2bDlF16I2rJQDxs2SDt2hCqNmjWT+vQhNAIAIM4IjQAAQOPgtS7Vt9JIkg46SMrKokXNb0VFdvRCI8la1AiNAACIK0IjAADQOEQjNGrWzKqN3n1Xys2NyrJQD4RGAAAEAqERAABoHKIRGknS6adLzZtLjzzS8DWhfqoLjYqLpT17/FkTAABNEKERAABoHAoLpQ4dpE6dGnae3r2lI46QHnuMgMIvRUVW9dWrV+iyfv2ksjKppMS/dQEA0MQQGgEAgMahsLDhVUaes86SVq2S5syJzvlQN0VFoR3TPH372pEWNQAA4obQCAAANA7RDI2OOEJKSZEeeig650PdFBdXbE2TCI0AAPABoREAAGgcohkaJSVJZ5whvfGGBRiIr6KiqqFRnz52JDQCACBuCI0AAEDi27RJ2rgxeqGRJJ15ps3Quf/+6J0TkQkXGrVtK3XtGhp4DgAAYo7QCAAAJL5o7ZxWXv/+0gknSDfcIC1YEL3zomabN9uvyqGRZC1qVBoBABA3hEYAACDxxSI0kmymUVaWdOKJoW3gEVteOyChEQAAviM0AgAAiS9WoVHHjtLLL0vbtknHHy/t2BHd86MqL5xLTa16HaERAABxRWgEAAASX2Gh1KqV1KNH9M89cKD0j39IH30kTZ8uuW70HwMhtVUabd5sM6wAAEDMERoBAIDEV1hogUKzGL20OfZY6Y9/lB5+WHrggdg8BkxtlUYS1UYAAMQJoREAAEh8hYXRb02r7OqrpSOOkC68UJo3L7aP1ZQVFUnduknJyVWvIzQCACCuCI0AAEDii0do1Ly59NRT0qBB0tFHExzFSlFR+NY0KfR3TGgEAEBcEBoBAIDEtmOHtHp17EMjSerUSXrrLZtzRHAUGzWFRj17Si1aEBoBABAnhEYAACCxeQFCPEIjSera1YKj7GyCo1goKgo/z0iymVV9+hAaAQAQJ4RGAAAgsRUW2jE9PX6PSXAUGzt2SOvXV19pJNlcI0IjAADigtAIAAAkDtetuuW9FxrFq9LI06WLBUdZWdKMGfF97MaquNiOtYVG3t85AACIKUIjAACQOG69VerdW3rzzdBlhYU2pLq6lqZY6tLFKo2WLZP27In/4zc2RUV2rC00Ki6Wdu+Oz5oAAGjCIgqNHMc5zHGcbxzHyXMcZ2aY61s5jvPsj9d/6DhO+o+XT3UcZ3G5X2WO4+T8eN27P57Tu65HNH9jAACgEXriCWnVKunww6WZMy04KCy0wCgpyZ81ZWZaYOQFHojctm0Vv48kNNpvP6msTHryyditCwAASIogNHIcp7mkeyUdLilb0hTHcbIr3WyapA2u6/aXdIekmyXJdd2nXNfNcV03R9Kpkgpc111c7n5Tvetd110Thd8PAABorFaulL74Qrr2Wumcc6Sbb5bGjZMWL45/a1p5WVl2zM/3bw2J6N13pXbtpFGjpLvvltasCYVGNVWNHXmkNHq0dOWVVUMnAAAQVZFUGu0nKc913XzXdXdJmiXp6Eq3OVrS4z9+/bykCY7jOJVuM0XSMw1ZLAAAaMLmzrXj8cdL990nPfOMhUhffOFvaJSZacdly/xbQyK6/Xapc2eptNRmQvXuLd1xh9Sxo9S+ffX3cxzpllusRe3OO+O3XgAAmqBIQqNUSSvLfV/042Vhb+O67h5JmyR1rXSbk1Q1NHr0x9a0K8OETAAAACFvvGE7pO2zj31/8snSJ59IkyZJxx3n37rS0qw1jkqjyC1fLr3+unT++dKnn0pffildeqnUooVVEdXm5z+3WVI33WQVSgAAICYiCY3ChTluXW7jOM5oSdtc1/2y3PVTXdcdIunnP/46NeyDO87ZjuMsdBxn4dq1ayNYLgAAaHR27rSdyo44wipNPAMGWPhw7LH+ra15cwuzCI0i9/e/S82aWZuhJA0aZAHQihXSvHmRneOmm6w97dprY7dOAACauEhCoyJJfcp9nyappLrbOI6TJKmjpO/LXX+yKlUZua5b/ONxi6SnZW1wVbiu+4DruiNd1x3ZvXv3CJYLAAAanQULLCA44gi/VxJeVhahUaS2bZMeesiCvsoDr+tSeL7PPtKvf22tit9+G901AgAASZGFRh9LGuA4TobjOC1lAdCrlW7zqqTTf/z6BEnvuK7rSpLjOM0knSibhaQfL0tyHKfbj1+3kHSkpC8FAAAQzpw5UqtW0sEH+72S8DIzmWkUqVmzpA0bpOnTG36uq6+258UVVzT8XAAAoIpaQ6MfZxRNlzRP0lJJs13XXeI4zjWO4xz1480eltTVcZw8SZdImlnuFGMlFbmuW/7jt1aS5jmO87mkxZKKJT3Y4N8NAABonObMscCoTRu/VxJeZqYFIRs2+L2SYHNd6a9/lQYPlsaObfj5evaUfvc76YUXpP/+t+HnAwAAFSRFciPXdedImlPpsj+V+3qHrJoo3H3flbR/pct+kLRvHdfaOLiutGiRNGKE3ysBACAx5OVZ+1E0KlNixdtBbfly2xEM4X3wgbR4sbWURWsPlN/+VrrnHumuu6QDDojOOQEAgKTI2tMQTU88IY0cKf3+99Lu3X6vBgCA4Js7146HH+7vOmqSlWVH5hrV7N57pY4dpalTo3fOtm2lYcOkwsLonRMAAEgiNIq/E06QzjrLdvwYO1YqKPB7RQAABNucOdJee0n9+/u9kuplZNiRuUbVW7VKeu456YwzpHbtonvuXr3s/AAAIKoIjeKtTRvpgQekZ5+VvvpKysmxF1AAAKCqbduk+fODu2uap0MHqVs3Ko1q8uCDVmV9/vnRP3dKivTddzYGIJEUFUkrVvi9CgAAqkVo5JfJk62nf5997Otrr/V7RQAABM/8+dLOndKkSX6vpHaZmYRG1Vm3zgZgH3qoVY1FW0qKtGtX4gwid13pkUekvfe214EAAAQUoZGfMjKk996z3WAef9zv1QAAEDxz5tjMmp//3O+V1C4ri/a06kyfLm3cKP3lL7E5f69edkyEFrVNm6RTTpGmTZPKyqRvvvF7RQAAVIvQyG8tWkijR1tpcmmp36sBACA4XNdCo4kTpVat/F5N7TIz7ec5G11U9Nxz1pZ/1VXS0KGxeYyUFDt+911szh8tH30kDR9ufybXX29/Jhs3Sps3+70yAADCIjQKgsxMe4FZXOz3SgAACI5ly2zDiMMO83slkcnMtA+AVq70eyXBsWaNzTAaOVK6/PLYPU4ihEaffSaNGWPPkQULpCuuCA1QZ+c3AEBAERoFgfeCgTkIAACEeK1egwf7u45IZWbakZ/nxnUtMNq82drwk5Ji91iJ0J72739Le/bYaIIDDrDL+vWzI8OwAQABRWgUBF5otHy5v+sAACBIvOoL74110GVl2ZG5RubZZ6UXXrDNPrKzY/tYHTpIrVsHu9IoN1dq317q0yd0mffcptIIABBQhEZB0Lev1KwZoREAAOWtWCE1bx5qPQq63r2lli2pNJKs4ueCC6T995d++9vYP57j2PMk6KFR//62Vk/PnvacITQCAAQUoVEQtGghpaURGgEAUF5hof18jGVbUzQ1by6lpxMaSdIdd0hbtkiPPWZ/LvHQq1ew29Nyc6UBAype1qyZVR4RGgEAAorQKCgyM3mRCQBAeStWWDVuIuHnuXnjDWnsWGnvveP3mEGuNNq1y4a6Vw6NJGtRIzQCAAQUoVFQZGRQaQQAQHmFhYkzz8iTlWUzjVzX75X4Z+VKacmS+O96F+TQqKBAKiurPjRiEDYAIKAIjYIiI8Ne6Gzf7vdKAADwX2mpVFSUmJVGmzZJGzb4vRL/vPmmHQ8/PL6Pm5Jif/ZBfC2Vm2vH6kKj776zaiQAAAKG0CgovB3UKE8GAEAqKbHgKNEqjTIz7diUW9TmzrU5PbHeMa2yXr3sGMS5RrWFRq5rFVoAAAQMoVFQ8CITAIAQr10n0SqNsrLsuGyZv+vwy65d0ltvWZVR+V3C4sHbZS+ILWq5uVLHjlK3blWv857jfHAIAAggQqOg8CqNmGsEAEDoDXSiVRp5P8+b6odA//2v7ZoW79Y0KRQaBbXSaMCA8EGa9xwnNAIABBChUVD06iUlJxMaAQAghSqN+vTxdx111a6d1KNH0w2N3nxTSkqSxo+P/2N77WlBrTTq3z/8dX36WJhEaAQACCBCo6BwHCk9ndAIAADJ3kB36WIhTKLJzGzcodFnn1lFUThz50oHHih16BDfNUlS9+5Ss2bBC4127rQQNNw8I0lq2dKqpNhBDQAQQIRGQdLYX2QCABCpFSsSrzXNk5XVuGcaXXqpdOihVQc3FxdLn3/uT2uaJDVvLvXsGbz2tOXLpbKy6kMjyZ7rVBoBAAKI0ChIMjKoNAIAQLLQKNGGYHsyMy1QaaxbqK9cKW3dKl14YcXL33zTjn6FRpK1qAWt0qimndM8ffsSGgEAAonQKEgyMqRNm6QNG/xeCQAA/nFdewOdqJVGmZlWWdJY242Ki6WuXaVXXpFefjl0+ZtvSqmp0uDB/q0tJSUxQ6N+/SyMKyuLz5oAAIgQoVGQsIMaAAD2AcqWLYldaSQ1zpbzLVusyuiSS6ShQ6Xp0+2yPXukf/1LOuyw8DuExUtKSvDa03Jzpc6dLWirTr9+VpkWtLUDAJo8QqMgacwvMgEAiJTXppOolUZZWXb0Kkwak+JiO/brJ91/v1RSIv3xj9IHH1jY52drmmTtaatXS6Wl/q6jvNzcmquMpNBzvbFWpwEAEhahUZBQaQQAQOiNc6JWGvXubW1a8+f7vZLoKymxY2qqtP/+0nnnSX/9q3TjjVJSkjRxor/rS0mxFq+1a/1dR3l1CY2YawQACBhCoyDp2NHKlwmNAABNWaJXGjmONGmSNG9e4xuG7YVGvXvb8YYbrLpn7lzpgAPstYyfUlLsGJQ2rx07bFZRbaGRF5ASGgEAAobQKGjYQQ0A0NStWCG1aiV17+73SurvyCNt9s+CBX6vJLq89jQvNOrYUbr7bvv6iCP8WVN5XmgUlGHY+fk22L1//5pv16GD1KkToREAIHAIjYImI4OZRgCApq2w0CovmiXwy5QJE6TkZOn11/1eSXSVlFjA0a5d6LLjj7eqqunT/VuXp1cvOwYlNIpk5zRPv36ERgCAwEngV2ONVGamVFDAlqsAgKZrxYrEnWfkadNGGj9eeu01qzRpLIqLbZ5ReY4jHXKI1LatP2sqL2jtaXUNjRiEDQAIGEKjoMnIsPkHQfmEDACAeCssTNx5RuUdeaRVD3/zjd8riZ6SklBrWhAlJ1ubV1BeR+XmSl272szK2lBpBAAIIEKjoGEHNQBAU7Zzp73hT/RKI8mGYUuNq0UtXKVR0PTqFazQKJIqI8me85s3Sxs3xnZNAADUAaFR0HihEXONAABNUVGRHRtDpVHfvtKwYdWHRuvW2bDsRFFWZmFMkCuNJGtRC0p7Wl5e5KGR95yn2ggAECCERkHTr5/NBqDSCADQFHkzXRpDpZFkLWrvvy9t2FDx8rVrpUGDpOOO82dd9bFunbR7d/ArjVJSglFptH27tHIloREAIKERGgVNcrJ9gkdoBABIVH/7mzRihHTHHdL339ftvt4b5sZQaSRZaFRaaruLlTd9urRmjfSvf0lLlviztroqKbFj0CuNvPY0vweQL1tmR0IjAEACIzQKooyM+odGfr9AAgDg1VelL76QLrnEAoZTT5Xeey+yn1FepVFaWmzXGC+jRkndu1dsUXvuOWn2bPvzadXKQrZEUFxsx0SoNNq+Xdqyxd91eDun9e8f2e179LDnAzuoAQAChNAoiDIy6jfT6KWX7IXSG29Ef00AgGB6/XXp5JOD9aFBbq50/PHSZ59J06ZZiDR2rHT++bWvs7DQKkVatYrPWmOteXPpiCOkuXOlPXusuuj886WRI6Wbb7a/u8cflzZt8nultUuUSqOUFDv63aLmhUaRVho5jrVlUmkEAAgQQqMgysy0T/N27oz8Pv/9r3TKKTYj4aSTpE8+id36AADBUFpq1SrPPhuc0GHXLqmgwN4oDx0q3XuvhQ2/+Y10333SDTfUfP8VKxpPa5rnyCOtTe9//5MuuMB2yHrsMSkpydrUfvhB+sc//F5l7YqLLdjo1cvvldTMW18QQqPu3aWOHSO/T79+hEYAgEAhNAqijAz7JDbS8uTcXOmoo6yUf/FiqWtXe4FKeTMANG4vvRSqZvB2HfNbQYHtslW+JadtW+n226Vf/Ur64x+tsqY6hYWNZwi255BDLCC66CLp+eelP//ZhmBLVnG0334WrgWpWiyckhJroWrRwu+V1MyrNPJ7B7Xc3MirjDyERgCAgCE0CqKMDDtGMtdo7Vrp8MPtk7+5c6UhQ6Q5c6yX/4gjpI0bY7tWAIA/XNfam7w2rqCERnl5dqz8ZtlxpIcfliZMkM46ywZAV+Z9YNLYKo06dJAOOkhatMgCoksvrXj99OnSN99Ib7/tz/oiVVwc/HlGUnDa0/Ly6hcarV4t7dgRmzUBAFBHhEZB5H06++mnNd9u2zbpl7+0F3GvvRa636BB0osvSt9+azMldu2yX4sWSY88Il1+ef1mJgEAguOdd6SFC0MBRFBCo5qG/7ZsKb3wgpSdbT+fFi+ueP2aNdaa3dgqjSRp8mSpTRvp0Uet6qi8E0+0NqZ77/VnbZEqKQn+PCNJ6tTJwlQ/Q6Pt2+31WaRDsD1eYLpyZfTXBABAPRAaBVHv3tKYMRbwlJVVf7tp06SPPpKeflraf/+K140fLz30kL2pyMqS2rWz7Y+nTZP+8hfpwQdj+3sAAMTWzTfb7JbLL7cqnqCERnl5VlnTvXv46zt2tIrYTp2sIvbbb0PXeW3Vja3SSJJ+/WurIMnOrnpdcrJVX736arBbkxKl0sibu+Rne1pBgR0zM+t2Py8wDfLzAADQpBAaBdW559qntfPnh79+0SJp1izpyiulY48Nf5vTTrNtfIcOlX77W7v9N99I++wjLV0au7UDAGLrk0+svevii6X27aWePYMTGuXmWnWF41R/m9RU6c03bTexsWOlJUvscu+NcmOsNHIc+wCnOueea8f77ovPeupq1y5riU+ESiPJWtT8rDTyKrrrGhqlp1e8PwAAPiM0CqoTTpC6dJHuvz/89X/5i71RuMnz2eIAACAASURBVPjims9z3nnSG29IN95ou6rttZd9ykloBACJ6y9/sWqec86x79PSghMaRTrHJTtb+ve/pWbNpHHjrFWtMVca1aZvX9vU4qGHgjnPxgtgEqHSSErc0KhvX6vSe++96K8JAIB6IDQKquRk6YwzbGecyuXVy5ZJs2fbp5KdOtX93NnZ9qJ+586oLBUAEEd5ebYD1/nnh7byDkpotHu3teVEOsdl4EBpwQKb9XPwwfYhR7t29fvZ1hjMmCGtW2fVV1995fdqKiopsWOiVBr53Z6Wn2+7BlbXplmdZs1sWPzbbwd/Nz0AQJNAaBRkZ59tpfuPPlrx8ttusyGav/lN/c6bnW2zkrxhpQCAxHHrrbbl+YwZocuCEhoVFEilpXXbMap/fwuOunSxOXz9+tXc2taYjRtnHwrl50vDh9vcqj17/F6VKS62YyJVGq1fb211fsjPtyqj+jyXJ0ywKimqwgEAAUBoFGR7722fvD7wgL0Il2xnmUcftXlF9f20b+BAOwbtU0wAQM02bZIee0w6/XSrpPCkpdl1W7b4tjRJNe+cVpN+/Sw4GjhQysmJ/roSyYkn2oynI4+UZs6UDjxQ+vprv1eVeJVGKSl29KvaaNmyuremeSZOtONbb0VvPQAA1BOhUdCde659cvvPf9r3d99tbWWXXVb/c+69t33yRWgEAInlm2/sZ8CkSRUvT0uzo1cN4pe8PDvWpdLIk5oqff659Pjj0V1TIurZ01oQn3nGgrhRo6R58/xdU0mJ1LKl1LWrv+uIlBeq+hEauW6o0qg+0tPtvm+/HdVlAQBQH4RGQXfMMVKPHjYQe8sW6d57bbe0vfaq/zlbt5YyMih7BoBEU92g6KCERrm5tklDXee4eJKSpObNo7umROU40sknS198YZVbkyZZlZlfioutyihRWge9wGbhwvg/9urV0vbtUlZW/c8xcaL07rvBaU8EADRZhEZB17KldOaZ0muvSVddJW3cKF1+ecPPm51NpREAJBpvS/rqQiO/5xp5O6clSrCQCHr3tl3mxo+X/u//pOuu82dAcklJ4swzkkKtjg8+GP8/r/runFbehAnS5s3+hF4AAJRDaJQIfv1re8Fzxx02JHO//Rp+zuxs6dtv+QQLABJJYaHUoUPV3cW8N/N+h0a5uXWfZ4Tadeggvf66zTO88kprXY/3z2+v0ihROI5tKLJ4cfyDl2iERuPH25G5RgAAnxEaJYLMTOmQQ+zraFQZSfYJ3K5doRc2AIDgKyysWmUkScnJNmvGz9Bo926bwVefeUaoXcuW1p52xRW2QcY998T38ROt0kiSTjlFatPG/rziKT/fQqtw/1Yj1a2bVUox1wgA4DNCo0Rx44326eKhh0bnfNnZdqRFDQASR3WhkWQtan6GRgUFttMnlUax4zjS9dfboOwlS+L3uFu22K9EqjSSpI4dbS7UM89Yq1e85OdbwJac3LDzTJwo/fe/0rZt0VkXAAD1QGiUKIYPl665JnpzIvbZx44MwwaAxBGE0OjVV+2NbGUN2TkNdZOaGt+h5yUldky00EiSzjlH+uEHC47ipSE7p5U3YYJVhb//fsPPBQBAPREaNVUdOtgbDCqNACAxbN5smyH4HRqdf769Ea8sN9eOVBrFXrxDI++xEq09TZJGjZKGDYtvi9qyZdEJjX7+c6lFC+YaAQB8RWjUlGVnU2kEAImiup3TPGlp0rp10o4dsVvDtm0WIHz5pfT55xWvy8uT2reXevSI3ePDUGkUOW8g9qefSp98EvvH277d/ryiERq1bSv97GfMNQIA+IrQqCkbONBCo7Iyv1cCAKhNJKGRFNswYdmy0NdPPVXxOm/ntGi1UaN6qanS+vU1B4SbNtkcomjwnlOJGBpJ0tSpUuvW0v33x/6xCgrsmJUVnfNNnCgtWmR/3wAA+IDQqCnLzrZPjVes8HslAIDaeKFR377hr/dCo1i2qHlzi/r2tRkx5T90yMtjnlG8eG1iXgVQOJMnSwceaLvaNVRJibW1t2vX8HP5wRuI/fTT0QvSquPtShuNSiPJ5hq5rjR/fnTOBwBAHREaNWXeDmq0qAFA8BUW2rbrPXuGvz4eoZE3t+j3v5dWrpTee8++371bWr6c0ChevNCopqoyr4Xwzjsb/njFxYk5z6i8s8+Oz0DsaIdGo0ZZ2ydzjQAAPiE0asoGDrQjw7ABIPgKC63Cp1k1P7q9N/WxrjTq1k069VSbt+K1qBUWSqWlDMGOl9paEXfulL77TkpKkq6+uuEVxSUlidua5hk9WhoyJPYDsfPz7d9G9+7ROV+LFtJBBzHXCADgG0KjpqxrVxtYSmgEAMG3YkX184wkq0bo2DH2odGAAfam+Nhjpeees4DCq0Ci0ig+agsIV660lqYrr7TvL7qoYY/XGCqNHEeaNs2GYX/xReweJz/fqoyiOdvroIPs3966ddE7JwAAESI0auq8YdgAgGArLKw5NJKsAiXW7WleNdHUqdLGjdKcOaFZR1QaxUeHDhbcVVdp5M2/GjtWuuoq6ZVX7Fd9lJVZ1VKiVxpJ0imnWPXV44/H7jGWLYtea5pn0CA78noNAOADQqOmLjvbKo1c1++VAACq47UbRRIaxWr3tO3bLZDyqokmTrRq1aeesjCpfXv7HrHnOFb5U1to1K+fdPHFFjpceKG0dWvdH2vdOptZleiVRpK1jE2aJD35pLRnT/TP77qhSqNo8mZQUhkOAPABoVFTl51t2/KuWuX3SgAA1Vm50o5+VhotW2ZHr5ooKUk66STp9delhQvt8mi25KBmNYVGBQU2+yotzWbi3HefPYeuuabuj+Pt0NYYKo0k6fTTpdWrpXnzon/u1astXI12aNSnj9SmDZVGAABfEBo1dQzDBoDgK185UpO0NPsQIBrbrFcWrgVt6lSrgvrgA+YZxVttlUapqRYYSdKBB9o8n9tvlxYtqtvjeI/RWEKjSZNspmMsWtSivXOap1kzaZ99eK0GAPAFoVFTR8kzAARfXUIj17VWtmgLFxrtt5+UlVX1csReaqpVAZWVVb2uoEBKT6942c03Sz17SpMnW4VxpLyd1/r2re9Kg6VlS5tt9Mor0oYN0T23Fxp5/yaiKTubSiMAgC8IjZq6Xr1stx1eiABAcBUWWuuXt9V6dbzrY9GilptrFRqdO4cucxyrNpKoNIq31FSrKAu3o1a4oeldu0rPPistXy6deWbkswwLC61iqVevhq85KE4/Xdq1S5o1K7rnzc+3fxO1hbv1MXCg/bvevDn65wYAoAaERk2d44SGYQMAgqmw0NqDvHaj6sQyNMrLCx8MTZtmFUcHHRT9x0T1vL/ryi1qe/bY33+44OLAA63i6MUXpTvvjOxxVqywmTrNGtFLxhEjpMGDG9aiFm6oeH6+hXnJyfU/b3W8yvCvv47+uQEAqEEjegWAeqPkGQCCLVzlSDixrjQK14LWt6/04YdSRkb0HxPV83YzqxwaFRdLpaVV29M8l1wiHXOM9LvfSf/5T+2PE+lzL5E4jlUbffhh/UKYr76yndiuu67i5bHYOc3jzaDk9RoAIM4IjWAvRNaskdav93slAIBwIn3j3rGj1LZt9EOj7dtt9y3mFgWHFxpV/ruubf6V40iPPmrXT55sP/9r0hhDI8naKps3r1+10SOPSDt2SH/6k/TGG6HLly2LXWiUlWWVhlSGAwDijNAIoTcB3nbKAIDgKC21wCaSN+6OY2FCtEOj5cvtyNyi4OjZ01rGKlcaFRTYsabnS6dO0vPP24dFZ5xR/e127bJh240xNEpJkQ49VHriCfs3Fqndu+0+hx8u5eRY+JSba8FqSUnsQqOkJGmvvag0AgDEHaERQrt8eLt+AACCY9Uqm1MT6Rv3tLToh0a5uXak0ig4kpJsOHXl0MirNKptt7OcHGtVmzvXAo9wiopsYHZjDI0ka1ErLpb+9a/I7/Pmm1addd55NhsqKUk69ljpyy/t+liFRhIzKAEAviA0QugFDpVGABA8tbUbVRaL0Cgvz45UGgVLamr4SqNevSIbxjxkiB2r+/lf1+deojnqKKs4mjpVeu+9yO7z2GNSjx7SYYfZ3KhZs6z65+ST7fpYhkYDB1rVX3UhHwAAMUBoBKlNG3vRRGgEAMFTn9CopKRuLTe1ycuTunSROneO3jnRcOFCo8LC6odgV+aFgF4lWWWNPTRKTpYWLJC6dZMmTpSefLLm269bJ732mvSrX4V2Mpw4UbrpplC1tle9HQvZ2VJZmfTtt7F7DAAAKiE0gsnMJDQCgCCqT2hUWiqtXh29NVS3cxr8VV1oFOlzJZLQyHGkPn3qv8ag699f+uAD6YADpFNPla6+2lrywnn6aZtpVHkO1KWXSlOm2Adw3bvHbq3soAYA8AGhEUxWFqERAARRYaHUtavtihaJtDQ7RrNFLS+P1rQgSkuTNm6Utm2z78vKpBUrIg+NOna0kKOm0CglRWrZMjrrDaouXaR582zG0Z//bOHRrl1Vb/fYY9K++4ba+jyOY1VK33xjX8fKXnvZ8HPmGgEA4ojQCCYryz6tpE8eAIKlrluee6FR5QqU+tqxw4IIKo2CJzXVjt7f9apVFnZE2p4mWRhYU2jUWFvTKmvZUnr0Uem666SnnrIZReWDo88+kxYtqn63uWbNpPbtY7vG5GSrDKfSCAAQR4RGMF4PvretMgAgGAoLa98Jq7xoVxotX27tOlQaBU/l0KigwI51CXoIjUIcR/rDH6S77pJeekk66aRQcPTYYxYsTZni6xLZQQ0AEG+ERjBeaESLGgAEh+vW/Y17t2725nbFiuiswds5jUqj4KkcGnnzr+paaVRSIv3wQ8XLy8qklSubVmjkuegi6e67pZdftuDohx+s/eyoo6xV1E8DB1rIt3t37B5j/nyrrAIAQIRG8BAaAUDwfP+9vWGtyxt3x5FGjbJWm+pa1MrKpGnTpEmT7OuaeFUohEbB44VGXlVZfXY78yrIKv/891rdmmJoJEkXXhgKjkaNsp3TqmtNi6fsbAuMYvV67fnnbUe4iy+OzfkBAAmH0AimWzfrxfe2jAUA+K++W54/9JDNIjrlFGnPnqrXX3ed9Mgj0pw5tW8znpcnde7sf4UFqmrXTurQoWJ7Wl2GpkuhMLByi5r33KtLa2Rjc+GF0l//ajOEevWSDj3U7xXFdge1OXPs/wzJ5jdVt4scAKBJITSCcRx2UAOAoKlvaLTPPtLf/iYtWCBde23F6156SbrqKtshar/9pJkzpa1bqz9Xbi5VRkGWmlqxPa0urWlSqNKoutCoqVYaeaZPt38zTzwhJSX5vRr7ty1Ff67R/PnS8cdLQ4dKN91ku/J5M7IAAE0aoRFCCI0AIFi8uUT1eeN+2mm2hfi119obQkn64otQWPTAAzbw97vvpBtvrP48eXkMwQ6y8qFRQUHdnyvt20s9exIa1eSYY6xlKwjat5f69IlupdEHH0i//KXtzPbmm9K4cXb5okXRewwAQMIiNEJIZqbtklNa6vdKAACSvXFv06b+rWH33CPttZc0daq9yTz6aGtneukl2757//2lX/1Kuu228Ltn7txpwRWVRsGVlmahUX2GpnvC7aC2YoW1JcZ6G3nU3cCBkVUarV9fe7XQJ59IRxxh7XdvvWXjCoYMkZo3lz79NCrLBQAkNkIjhGRl2dDL6ganAgDiywsBHKd+92/XTpo92wZq5+TY/+8vvij17h26zU032RvEyy6rev/ly21QNpVGwZWaatViq1dL27fXvT1NCh8a1TeAQuxlZ0tff13zEPtXX5X23tsCptmzw99m/nzp4IOljh2lt9+WUlLs8uRkux+VRgAARRgaOY5zmOM43ziOk+c4zsww17dyHOfZH6//0HGc9B8vn+o4zuJyv8ocx8mpdN9XHcf5Mhq/GTQQO6gBQLBE44370KG2C9SePdaStv/+Fa9PTZV+/3vphRekd9+teF1enh2pNAqu1FSrEP74Y/u+vpVGq1ZJW7aELiM0Cq6BAy0g9NpXy9u2TTr/fKsq7NNHGjFCOukk6ZprKg62fvFF6bDD7Db/+U/Vv+vhwwmNAACSIgiNHMdpLuleSYdLypY0xXGc7Eo3myZpg+u6/SXdIelmSXJd9ynXdXNc182RdKqkAtd1F5c793GSapi+ibgiNAKAYCkoqF/lSGVnn23VRqefHv763/7W3jTOmCF99JF0yy3SpEm2k5LjUGkUZKmpdvzPf+xY30ojKRQSNqTVDbGX/ePL8Motap99Jo0cKf397/Zv+n//k955x+aYXXWVtanu2GG7K554orTvvtJ774WeQ+WNGGEVbKtWxf73AwAItEi2gdhPUp7ruvmS5DjOLElHSyr/k+poSVf/+PXzku5xHMdx3Qp7dU6R9Iz3jeM47SRdIulsSdXUzSKu+vSxnUEIjQDAf1u22EySaIRGkrWgVKd1awuKJk+WRo+2y/bZx95k/vKX9Z+phNirHBrVt9JIsha14cNt56wtWwiNgmrgQDvOnSutXGmzhz79VFq82GYS/fOf0i9+Ebr944/bfa64wkLhZcukww+XnntOats2/GMMH27HRYvstgCAJiuS0ChV0spy3xdJGl3dbVzX3eM4ziZJXSWtK3ebk2ThkudaSbdJ2lbHNSNWkpLszUl+vt8rAQB4u1dFKzSqzQknSPfdJ3XqJB10kA3GRfB5odHHH9uQ806d6n4Or/3Qm2vEzmnB1rWr/fu85x77vnNnqwy69FLpkkuk7t0r3t5xrAV1772t6mjqVOnRR6UWLap/jJwfp0kQGgFAkxdJaBRu+qZbl9s4jjNa0jbXdb/88fscSf1d173Ym39U7YM7ztmyaiT17ds3guWiQbKyqDQCgCDwdj2KV2jkONI558TnsRA9PXrYhz47d1ooUB9t29pwdEKjxPHiizbYft997f+ISIblH3ec7ZSWnFz7bTt2tF114zHX6D//kUaNklq2jP1jAQDqLJJB2EWS+pT7Pk1SSXW3cRwnSVJHSd+Xu/5klWtNk/QzSfs6jlMg6X1JezmO8264B3dd9wHXdUe6rjuye+VPThB9hEYAEAzxDo2QmJo1C+2G15CQZ8CA0EwjQqPg+9nPrDowI6NuuytGEhh5RoyIfWj02mvSgQdaqxwAIJAiCY0+ljTAcZwMx3FaygKgVyvd5lVJ3nTNEyS9480zchynmaQTJc3ybuy67t9d1+3tum66pAMlfeu67riG/EYQJVlZNsvg++9rvy0AIHYKCuwNXo8efq8EQee1qDUkYOzfv2KlUevWNh8HTdfw4fZB4qZNsTn/nj3S5Zfb1998E5vHAAA0WK2hkeu6eyRNlzRP0lJJs13XXeI4zjWO4xz1480eltTVcZw82XDrmeVOMVZSkTdIGwGXmWlHqo0AwF/e7lV1qSJA0+SFRg2tNFqzRtq8mecejDcMe/Himm9XX489Ji1datVyzNMEgMCKZKaRXNedI2lOpcv+VO7rHbJqonD3fVfS/jWcu0DS4EjWgTjIyrLjsmXWXw4A8EdBAa1piEy0QiPJqo0KCyXmSGLECDsuWmTD8aNp2zbpqquk/fe3isrly6N7fgBA1ETSnoamhEojAAgGQiNEKhrtaZVDI+YZoWdPKSUlNnON7rxTKimRbrnFPrCk0ggAAovQCBW1bWvbuBIaAYB/tm6V1q0jNEJkxo+XxoyRBg6s/zm8SuPPP5fWriU0ghk+XPr00+iec9066eabpaOOsiHYmZnSqlVWfQQACBxCI1TFDmoA4C9v9ypCI0Ri332l99+3D37qq00bKS1Nevtt+57QCJKFRkuXStu3R++c111nwfhNN9n3GRl29HaMBAAECqERqqJMGAD85b154o074mnAAGnhQvua5x4km2tUWip9+WV0zpefL/3tb9K0aaHKOG80Aq89ASCQCI1QVVaWVFws7djh90oAoGnyQiMqjRBPAwZIZWX2NaERpNAOauXnGrmudMUV0rhx0hdfRH4u15UuuURKSpKuvjp0OaERAAQaoRGqysqyH+zsZAEA/igslFq1skG0QLx4w7CbN5d69/Z3LQiG9HSpU6fQXKPSUuncc6Ubb5Q+/thaI2+4Qdqzp/Zz3XKL9Mor0rXXVnx+detmrZW87gSAQCI0QlXsoAYA/ioosEqPZvyYRhx5oVFamlWDAI5j1UaLFlkwdPrp0gMPWKVRYaF07LHSH/4gHXCAzT6qzrx50syZ0kknWbVR5cfIzKTSCAACilejqMrbQYXQCAD8UVBAaxrizwuNaE1DecOH2656kydLTz0lXX+9/erWTXr2WfuVn2+3u/baqkOzly2TTj5ZGjJEevhhC4kqIzQCgMAiNEJV3btL7doRGgGAXwiN4IfMTHtDT2iE8oYPtzmXL70k3XmnVRmVN3mytGSJdNRR0p/+ZAOun3/eRh1s3Sodc4w9r156qfod/jIyrD3NdWP/+wEA1AmhEapyHHZQAwC//PCDtHYtoRHiLzlZuuwyaepUv1eCIBk3zl4XPvigNGNG+Nv07CnNni29847UoYN04onS+PFWYfTVV1aN5I0/CCczM/R/HwAgUGhYR3jp6VQaAYAfCgvtSLUH/HDzzX6vAEGTlibl5UV224MPtqHZDz4o/fGP0vffS3/5i/SLX9R8v4wMO+bnSz16NGy9AICootII4aWnW3sEZcIAEF8FBXak0ghAIkpKks47T8rNlebOlS69tPb7eFVI7KAGAIFDaITw+vWzPvTvv/d7JQDQtBAaAWgMunSRDjss/ODryrz/7xiNAACBQ2iE8Lwf3t6bFwBAfBQWSi1bSr16+b0SAIiPNm3s/zxCIwAIHEIjhOeFRt5sDQBAfBQUWLVnM35EA2hCMjNpTwOAAOIVKcKj0ggA/FFQQGsagKYnM5NKIwAIIEIjhNepk9S+PaERAMSbV2kEAE1JRoa0cqW0e7ffKwEAlENohPAcxz7ppj0NAOJn2zZpzRoqjQA0PZmZUlmZtGKF3ysBAJRDaITqpadTaQQA8eQF9YRGAJqazEw70qIGAIFCaITq9etHaAQA8eT9n0toBKCpyciwI6ERAAQKoRGql54ubd4sbdzo90oAoGkgNALQVPXuLbVsyQ5qABAwhEaoHjuoAUB8FRZKLVpIKSl+rwQA4qt5c3vtSaURAAQKoRGqR2gEAPFVUCD17Ss148czgCYoI4PQCAAChlelqJ635TOhEQBE3/33SwsWVLysoIDWNABNV2Ym7WkAEDCERqhe165S27ah3XwAANGxaJF07rnS+PHSgw+GLic0AtCUZWRI33/PPE0ACBBCI1TPcezNC5VGABBdt94qtWsnTZggnX22dOml0tat0urVhEYAmq7MTDtSbQQAgUFohJr160doBADRVFgoPfushUVvvCFNny7ddpv0i1/Y9YRGAJoqQiMACBxCI9QsPZ32NACIpjvvtErO3/xGSkqS/vpX6e67pY8+susJjQA0VRkZdmQYNlC7Bx6wfzMZGRa4ZmVJe+8tvfmm3ytDI5Pk9wIQcOnp0oYN0qZNUseOfq8GABLbhg02w+ikk6Q+fUKXX3ih1L+/9Pjj0vDh/q0PAPzUqZPUuTOhERCJxx6Tdu+2+Yiua5fNni3985/SYYf5ujQ0LoRGqJm3g1phoTR0qL9rAYBEd//90g8/SJddVvW6ww+3XwDQlLGDGlC73bulTz+Vzj9fuv320OX/+5/03Xf+rQuNEu1pqJnXJkGLGgA0zM6d0l132eyiYcP8Xg0ABFNGBpVGQG0+/9xeV4weXfHylBSppMSfNaHRIjRCzbzQiGHYANAwTz0lrVplO6UBAMLLzLTXnWVlfq8ECC5vDuJ++1W8vHdvKo0QdYRGqFn37lLr1oRGANAQZWXSrbdam6+3SxoAoKr0dGnXLgvZAYT30Uf2Pq3y5hlUGiEGCI1QM8exuUaERgBQP65rwyqXLrUqI8fxe0UAEFzem2DmGgHV+/BDqzKq/Jqid2+bnbhliz/rQqNEaITapacz0wgA6mrTJumvf5UGDpSmTbPjySf7vSoACDZGIwA127RJ+vrrqvOMJKs0kmhRQ1QRGqF26en84AaASO3ZI118sZSaKl10kdSxo/SPf9guJy1a+L06AAi2WIZGW7aEZsEAieqTT6yKufI8IykUGtGihigiNELt+vWT1q+Xtm71eyUAEHxz5kh33ikdeaS9OfnwQ+nUU6XkZL9XBgDB17q11LNnbEKju++WDjhA+v776J8biJcPP7TjqFFVr+vd245UGjXcpk32QSAIjRAB7xMfWtQAoHavvCJ16GDVReFe0AEAaharKvevvpJKS6XFi6N/7ljatk169lmrLgE++kgaMEDq0qXqdVQaRceePVL//tKIEdLHH/u9Gt8RGqF29JYDQGRKS6XXXpOOOEJq2dLv1QBAYopVaJSba8dFi6J/7liaNctm4i1d6vdK4DfXtUqjcPOMJGuJb92aSqOGWrVKWrfOgub997eNTLZt83tVviE0Qu369bMjoREA1OzDD6W1a6Wjj/Z7JQCQuLxNWMrKondO103c0MjbSW7FCn/XAf8VF1sgFG6ekWS7qaWkEBo1VHGxHf/xD+nXv5Zuu00aMkSaP9/fdfmE0Ai169lTatWK9jQAqM0rr0hJSdLhh/u9EgBIXOnp0u7d0X3j+/330saN9vWnn0bvvPHghUVFRf6uA/7zBrlXFxpJFhrRntYw3r+1QYOk++6zsMhxpIkTpfff93dtPiA0Qu2aNbNqIyqNAKBmr7wijRtn5eEAgPqJxWgEr8poxAjpm28Sq9XEC4286gc0XR9+aDux5uRUf5vevak0aigvNEpNteO4cVah2K+f9H//l1j/f0QBoREiE6vecgBoLL75xn7RmgYADRPL0OjEE63t7fPPo3fuWPOq/ak0wkcfWWDUqlX1t/Gz0ujJJ6V77615aPt330n/+1/81lQfxcX2BVo8WwAAIABJREFUZ9y1a+iy9u2lRx6R8vKkP/zBv7X5gNAIkaHSCEAiW79eWrjQWhIWL7Y3C9FuuX3lFTsedVR0zwsATU0s5mnm5lr1/PHH2/eJMteotDQUFhEaNW2lpfZaproh2J6UFGnLFumHH+KzLs/LL0unnSZNny6deaa0a1fV27z3njRsmDR2rLRzZ3zXVxdFRVJamrWklTdunHTBBdJddzWpNjVCI0QmPd2GuzaxUjwAjcQvfiGNGiXtu680fLi9YElPl/7+9+g9xquv2rn79o3eOQGgKWrd2mZqRjs06tvXttHu0sW/uUabN9ft9qtX23wnifa0pm7pUmnr1prnGUnWnibFt0Vt0SJp6lR7rXXlldJjj9l8R2+OmGSzgcaPlzZtiv7MsmgrKgq1plV20032GrIJtakRGiEyXpkww7ABJJrSUmnJEmnyZKsGeukl6YUXpEMPlWbMkP7738jPtX699MQTds7y1qyx89CaBgDREe3RCHl50oABVjkwfHj8K4127rSfOR071q1CwXvtnZ5OpVFT5w3BjqTSSIpfi9p331mVdZcuVm10zTXS449bVdEBB0jffiudc4503nn2Id4//mH3C3IIWlxslUbhtGvX5NrUCI0QGa9MmNAIQKIpKbES6QkT7EXNMcdIxx0nzZpl/7cdf3zkn3addZaVXp97bsV+/ddft+8JjQAgOqIZGrmuVRoNGGDfDx8uffFFqIIn1vLzpQMPlO6+276vS5WTNwT7gAOkDRvi33KE4PjwQ6lTJ6uWq0k8K422b7fXPhs2SK+9FgqsTjtN+uc/bQ377CM98IA0c6bdZtAgu01QQ1DXrTk0kppcmxqhESLTq5cdV6/2dx0AUFfLltkxM7Pi5Z06WdXR5s3SCSeE770v75VX7BO0kSOlhx6SLrssFBy98oq1PQwbFv31A0BTlJ5uH1aWlTX8XOvWWUtM+dBo1y5r94m1F1+0Hdtyc63KtU2buoVh5UMjKdjVGYitjz6y1rRmtbyFj1elUVmZdMYZNmfpqaeq7ug2bpxVYU+caB/U3Xij1Lx5qO0rqM/l9eutMrC69jSP16Z26aU1D/5uBAiNEJmePe24Zo2/6wCAusrPt2Pl0EiSBg+WHn3UXtRcfHH159i6VbrwQmnIELvtBRdIt90m3XCD9bP/619WxVR5YCIAoH4yMqwSKBpvfL2d07wKjREj7BjruUY33WTVrAMGWDvcccfVvYKqsNA+5Bg40L4P6httxNa2bVYdV9s8I0nq3Nl2/op1pdGcOdLs2RYGVVdpPXCgVRyddFLosk6dLDwNaqWRt66aKo0ka1N74QX7QLGRv/5L8nsBSBBt29pQQkIjAIkmP98+2apuQPXkydLHH0u33mpvJKZNq3qbq66SVq60T8patLAWg02bpD/+Ufrkk1B5NgAgOrx5mgUFtb95q01enh29SqMBA+xN66JFVikRKw89JB10kDRvXmiL9LqGRitW2M8v788gqG+0EVtvvGHzFPffv/bbOo5VG8U6NPrySztecEHd7uc4VsUT1AA00tBIsqrFJoBKI0TGcaQePQiNACSeZctsdlFSDZ+T3HijzTw66yzpt7+tuA3s4sXWs3722aH2gGbNrELp6KOtxa1jR3tjAACIjvKhUUPl5tr/2xkZ9n3z5tZOHMth2K5rbz732y8UGEn1D428VhlCo6bnhx/stcmQIdIhh0R2n5SU2LenLV8udetmFTd1FeTQyFtXbe1pTQihESLXowczjQAknvx8KSur5tskJdlwxgsukG6/3V7kL1lin+qdc47Utau1GVS+z6xZ0sknS7/5jVUgAQCiw6sOjVZolJ4utWwZumz4cPtQIBozk8JZt84+gOjTp+Ll6ek2NHjTpsjOU1hoH3y0bWttR0F9o43YueYaq3b++98jf63Ru3fsK42WLw/f+h+JtLTgBqBFRRYsezN9QWiEOqDSCEAiWrYsshc1rVtL99xj4dF339nA68mTbfDkHXfYi/XKkpOlZ56Rrr466ssGgCatdWt70xat0MhrTfMMHy5t2RKaexdt1bW4eBVUkexIvHmztHFjKEBLTQ3uG23ExpIl9mHWmWdKY8ZEfr94VRp51Xt1lZpq64tVaNsQRUX259e8ud8rCQxCI0SO0AhAotm0yXbBqMsnYUceacMmx4+3XW8mTpSmTIndGgEA4dW1lSsc1w0fGsV6GPbKlXasLjSK5PflnaNfv9C5CI2aDteVzj9f6tBBuvnmut03JcVeA23fHpu1lZZa8Fnf0CgtzQbdr1sX3XVFQ3ExrWmVEBohcl5o1Mi3FATQiHifINfWnlZZz57S669b1dEzzzT6XTEAIJCiERqtXWsVRd7OaZ5Bg6zNOFZzjbxwJ1x7mhTZ78urRvIqjRpDaLRhg81pQu2efFJasMDa47t1q9t9e/e2Y6xa1IqLLfSpb3takGd0FRU1fPh+I0NohMj17Gn/OUTagw0AfvNCo/q8qHEcqzqq6ws1AEB0pKdbwFBaWv9z5ObasXKlUatWFhzFMjRq0cI+dC2vWzfbuS2S0MgLV8qHRmvWSLt2RXWpcTVzpnTwwX6vIvg2bJAuvdR2Swu3q2ttUlLsGKsWteXL7diQ9jQpmDO6CI2qIDRC5LwferSoAUgUDQmNAAD+Sk+3DywbUi1RXWgk2VyjTz+NTRX9ypX2xrhZpbdbjhN5BVVhoQVPXgCQmmprjfWA41jKy7OfzWvX+r2SYPvTn6x16+9/r/ocikSsK42811cNaU+TgldptHmzVSbSnlYBoREi54VG7KAGIFEsW2Y7n3Xs6PdKAAB1VZdWrurk5tpAW+9c5Q0fbuFFLKoxaqpWiDQ0WrHCzuGFBkF9o10XXojx2Wf+riPoXnjBNuPIyanf/eNRadSsWagKrq569rR/l0GrNPLWQ6VRBYRGiByVRgASTX5+3ecZAQCCIVqhUUZG+K3KvWHYsWhRW7kyOqFR+Tfl3vmC9ka7LlatsuPixf6uI8h27LBwLTu7/ufo2tWe87GqNFq+3OZ1hft3FQlvS/ugPZer2/WwiSM0QuQIjQAkmmXLaE0DgETlBSYNCY3y8qoOwfYMG2btYtEOjVzX3nxWHoLtSU+3mTW1zQktLAztnCYFe3hwJHbssN+3RGhUE28Aen1bvyR7XqekxLY9rSHrk4I52N0LsWhPq4DQCJHzhsESGgFIBHv22AsvQiMASEytW1s1Qn1DI9e1SqNw84wkqX17G4b9wgtSWVm9l1nF+vXSzp01VxpJoXAgnD177A1s+UqjTp1siHbQ3mhHyqsykgiNauI938O1VNZFSkps29MaGhqlplZfabR0qVWKx/u57j2eNxMKkgiNUBctWkhduhAaAUgMK1fajju0pwFA4srIqH9otHq1tHVr9aGRJF1+uc3Xee65yM+7Zo00eLD0/PPhr1+50o41VRpJNf++SkosyCofGjlOMKszIuVVvYwYIX39tVUeoaqG7kzm6d07NpVG/8/efYfHUZ7rH79HltxxwbasYsmWbOGK6eUkoYQOoYPp4ISekBwgIZDkgM8BEkIxIRQTEkJCCb0Fk9AMBEJJCITQjDGqtlbYsnEvuEia3x+P57cqu9KW2Z1d7fdzXb5G2p3deWVL1sw9z/u8X31l75vsTbnuvpdfftmqmf797+SOEa9QSBo1SurfP73HzXCERohPYSGhEYDsUFtrWyqNACB7xdr/J5LuVk7znHqqBUBXXWUrtfXEdaXzz5fmz5deeinyPj31RYklNPKqkNpPT/PeM9P6wMTKCzAOO8xu6syfH+x4MlVDg9S3b7iZdaJSVWnkfd/6UWm0bp396eyjj2y7eHFyx4hXUxNT0yIgNEJ8Ro8mNAKQHbzlYKk0AoDsNW6cXTi2tsb/2poa23YXGvXpI/3iFxYw3Xtvz+95773SM8/YRf2CBZH38SqNooVGI0faNLPuQiPvYrnz6lSlpdlfaXT44bZlilpk9fUWFuYlealeXGw9pPyu6PKrEsoLZyKFoF5o5P0spUt3qx7mMEIjxKew0Ep9ASDT1dbaST3z0gEge40bZxVAiVRMVFdL+fldq3U6O+oo6b/+S7r6apt6E01Dg3TxxdJ++0lnnhk9NAqF7LijR0d+3nF6rqCKFhqNGROeupZtliyxIGTvvaXBgwmNoqmvT76fkRQ+/2nfS8oPXmjkx/Q0qWsI2toqffKJfZzuSiNCo4gIjRAfpqcByBbeyh59+gQ9EgBAomKZyhVNdbX9HsjP734/x5Guu84qHu68M/I+bW3St79tH993nzXQXrFC+vLLrvuGQlZF0V2lSE+h0aJF4Yqk9saMsSbZ2Xg+vnSpBWn5+bZyHaFRZA0NyVfxSOHpbX5PUaursyb10ULRWEWrNKqrkzZutI/TGRpt2mQ/z0xP64LQCPEpLLQyxy1bgh4JAHSvro5+RgCQ7ZINjbqbmtbe/vtLhxwi/fKX0tq1XZ+/5Rbp9del226zyqXJk+3xSNVGjY3Rm2B7Yqk06lxlJEWvzsgGS5aEg4ydd7YG5NlYMZVK69dLy5f7Gxr53Qzbq4RynOTexwtnOn8ve1PTpk1L7/Q0L1yj0qgLQiPEp7DQtpHuqgBApnBdm55GaAQA2c0LTrwpMbFyXetpFGtoJFm10YoV0s03h99jxQrphRekn/1MOvZYaeZMe6670CiWKS7jxtmN2DVrIj+/eHHkaXXRLrSzQefQaN26+P9dezuvAbqf09P8rjSqr/fn/GrAAFuZu3Ol0UcfWZXeYYfZ2GNpUO8HbxyERl0QGiE+XmiUjSWxAHKHdyJOE2wAyG79+1vFwZw50uefx/66W2+VNmyQdtwx9tfstpt04onS7NnSHnvYBe3Ikda4efhw6be/DVdXlJXZ1LHOoZHrxh4aSeGQoPN7LFrUuyuNdtrJtkxR68ivJtOSfe/m5/tbaeS64en/figtjRwa7bCDNHGiVaIlG3q9+660cGHP+3k/U0xP64LQCPEhNAKQDWprbUulEQBkvyeesIvVgw6KrcfJrbdKl14qnXCCdNZZ8R3ruuusimjECOn0021a2rPPSh9/HD4PlqwSYtKkrqHRihXWGyWW6WlS5Clqq1fbNKVIodGoUVJBQeQVpzJZa6tdPxQV2efTptnfIaFRR973gx+VRnl59vftZ2i0apVN3/QzNIo0PW369PDPULJT1E46Sfra16T587vfzxsHlUZd9NAVDujEa3jGCmoAMlldnW2pNAKA7DdxovTSS9Z36KCDpDfeiN6E97bbpEsukY4/Xnr4YQtY4lFVJb33Xmz7Tp5sY2nPu8CNtdIoUmjkBWORpqfl5UW+0M50y5ZZ1YhXaTRggIVuhEYd1dfb3037gDIZxcX+Tk/zsxJKsp+T//wn/Pm6dXYOd/bZ4dA0mWbYq1eHf8YOOUR6663ogVxTk7TddvYHHVBphPjkSqXRZ59ZGXJNTdAjAZAILzTy66QGABCsnXeWnnvOLuwOOcQqHjq7/Xbp4oul446THnkk/sAoXpMn2wXt+vXhx7wwp6dKI29ltEihkTdlLVKlkZSdoZFX7eKFRlK4GTbC/Goy7Sku9rfSyDu/8quSu7TUihG8RZY++cS27SuNkgmNvKbaN94offWVdPDBtopfJLFMK81RhEaIz5AhUt++vT80eucdS6a5+wFkp9pauws9aFDQIwEA+OVrX5P+/Ge7uffNb0rf+5503nnWnPq446T//m9rVp2OwEgKN8Nu3y8l1ikujhN9BTXvIjlaaDRmTPZNT/Mu1DuHRo2NNqWvvTVrpOeftymJuaahwd8bXiUlmV9pJIWDLS/kmT5dGjzYbuInMz3Ne7/TTrPQ+YsvpEMPteu8zgiNoiI0Qnwcx6qNento5PVD8Xu1AQDpUVfH1DQA6I0OPlh69FFblvzxx+1C8PXXrV/JOefYc337pmcskVZQa2y05sOxTC+KFhotWiT16xf9PcaMsQvcbApVolUaSR2rjVxXOvNM6YgjLDjKNV6lkV+Kiy2U8yp5klVfbw3ihwzx5/28ptNeCPrRR/beXmBaXp58pdH221t4tvfeFjovWCAdeaS0cWPHfZuaaIIdBaER4peO0OiPf7STgpaW1B4nGq/0ktAIyE61tTTBBoDe6thj7QJv+XLbNjTYymq//336AiNJmjDBAqL2oVEoZBeeffr0/PruKo3Ky6NPUSottak2kaboZSovNPIaYUuRV1C7/35rPN63r3TlldkVjCVr9Wr742elkVc5k0zw0l59vb/nV15I41XoeU2wve99P0Kj9u938MHSQw9Jb78tffe74e+vlhb7HqXSKCJCI8Qv1aHRli32S+Lll221jCAQGgHZa8sWu9NLpREAIJUKCiw46lxpFOuF57hxFvysWdPx8cWLIzfB9njvn019jZYssalG/fqFHysstAoQLzRqbLSeVPvsI/32t9Yg+amnghlvEPxcOc0zZYptP/3Un/erq0tNqNXUZAGOF/J4ysoSn57W1marHnrhpOfEE6VZsyyg/N3v7LHmZlvhj9AoIkIjxC/VodFDD1lYM2SIdNNNwdxhIDQCsldDg/2/QaURACDVJk/uWmnUUxNsjxcOeI2vJfv91dAQvZ+R1PFCO1ssWdJxappn550tNHJd6dxzpa1bbcbBmWfa6mqzZtnFfC7wQiM/QxkvNPIaTCejrc2+V/0c3/DhUv/+9r28eLG0dm3H0Ki83ILVdevif++6OpuC1v79PLNmSYcdZn3Q3n03/LPE9LSICI0Qv9GjLY1NRZjjutLs2fbDPXu29P770quv+n+c7mzYYF+flF2/jAEYL/Sl0ggAkGqTJ9tqu1u32nlsPM10vdCo/RS1K6+0ptH77BP9ddlaaRQpNNppJwvdbr9deuklO/8fP96m911zjVXIPPxw+scbBL+bTEt2E37sWH9Coy++sGpuP2/KOU54NcD2TbA9XniaSLVRpPfz5OVJf/qTfU+ecEK42o1Ko4gIjRC/wkJp8+bEEt+evPCCNTK87DK7wzB6tFUbpZN3wTl6NJVGQDbyezlYAACimTzZ+qHU1FjD4U2bEg+N7r5buu668Ipw0RQV2cV2tNBo3ToLYGbNkq69NtavJLW6qzRqaZEuvVQ66CDpwgvDz51wgoVK//d/Fsr1dvX10nbbWfWNn6ZOteurZHnnV36GWlJ4NUAv5Jk2LfycV7WXSF+jDz+0cMirtupsxAjpySetWODSS8NjQReERoift5JDKqao3XST/bCecoqVKl58sfTiix1XVUg17z/Eb3zDSiTXr0/fsQEkr7ZWGjCgY7NNAABSYdIk2y5YEA5xYp2eNnKkNHCghUYvvmiNeQ89VJozJ3oTbMl6KRUVWfXF4sXWB/TOO6Xvf1/abTdp2DB7n2uvteDo/feT+hKT5rpWPRUtNJJsefV77un4deflST//uf1ev/fetAw1UA0NFsh092+fiGnTpM8+Sz54S0UllNSx0qiy0oIzT7KVRlVV9jMWzW67SXfcYdPY+vWzIAldEBohfqkKjf79b+lvf5MuucR+GUp2t2HwYCtVTZf2oZEUXu0BQHaoq7OTDr9PugAA6Kx9aORd2MZareA4Vm30yivWnHfaNOnxx8Pnwd0ZM8Z6/4wdaytCXXSRBStDh0r/8z9WabR4sd2E/f3vE/nK/LN6tc1SiBQaTZggHX64jTFSH6dvfcuWSr/mGqvi6s3q6/1tgu2ZNs2mldXUJPc+9fX2Pdtdv61EjBljszs+/LDrVLKSEgsPE6k06txUO5pzz7XAdb/9OHeMgtAI8UtVaHTTTTbv9rzzwo8NH26fP/xwxyaBqVRXZ79wd9zRPmeKGpBdamuZmgYASI/Bg62yKJFKI8lCgo8+snPPv/ylY5VFd2bNkn74Q+muu6z/Zyhk09JefdUCloMPtnHMmCE9+KD17AyKdwM2UgVwXp703HM2zkgcx6qNQqHwSle9kdcA3e8qHik83SvZvkZ1dRbwtF8Bzw+lpRZqLVzYNeTJz7fgKN7QaN06G2/nldMicRzrqfXii/EdI4cQGiF+qQiNGhrszsoFF1hw1N4ll9gP869/7d/xuuNVKXjd82mGDWQP17WfYZpgAwDSxVtBLRSyi1zvXDkWU6daUPTcc/H1UznySOnmm+3c+ZvftPPWSFUS555r7RaeeCL29/abFxpFqjSKxYEHSrvuKj31lH9jyjQrVlhLjFRUGk2aZOFcsn2N6utTE2q1X7EsUmVQeXn809O8gCyWSiP0iNAI8Rs1yraJhkZz50o33mjbzz+35ne33GKrJFx8cdf9y8utx9Hdd9uSi6nmVSmUlNjnVBoB2WP5crubSqURACBdJk+2njGLF9v5Y58+sb/22mvtZkeqLm732UfaYQc7jw5KsqGRZM2MvZ46vZHXDD0VocyAAXYzLdlKo1SFRu3D0mihUbyVRt2tnIa4ERohfv36WYM9b1n6WG3YIJ19tnTMMdIVV9h24kRrTjZnjnTaaR2T5vZ+/GN7fap/4bW12X+IXhO2QYMIjYBsUltrW0IjAEC6TJ5sjXTffju+qWmSnVePHJmacUlWfXTuudJbb1k1VBD8CI0qK62Sa8sWf8aUabxALBWVRpJNUUsmNFq3zmZfVFX5NyaPd/03cGDk8zev0qitLfb3/Ogjm73id/+lHBVTaOQ4zmGO4yx0HKfGcZyfRHi+n+M4j257/h3HccZte/x0x3E+aPenzXGcnbc994LjOB86jjPfcZy7HMeJI5JH4AoL46s0+vhjaY89rEHflVdKX34p/fOf9vmPfiSdfLL0v/8b/fXTp9sdhjffTHbk3fviC/tl5DXRLSkhNAKyidfInulpAIB0mTzZtl7Pl0xz1lk2bS6ohthLl1ogEGu/pkgqKiw0SKQhcjZIR2hUXZ14M3EvcJw61b8xeYqKbPrctGmRq/TKyuz6bPny2N/Ta4JNY2tf9BgabQtz5kg6XNIUSac6jjOl027nSFrluu4ESbdIukGSXNd90HXdnV3X3VnSmZIaXNf9YNtrTnJddydJ0ySNkhSl+xkyUqyhketa07o997SpZfPmWRnuiBHSXntJM2dKv/ylNejrqdxxp53CpYap0vmCk9AIyC61teHVaAAASAcvNJLirzRKh9GjrcL//vttFbN0W7LEqoySuYD3rhN66xS1hgZbAGjo0NS8/7RpFrp99llir/f6IaUiNMrPt75LX/965Oe9aqFYA0PXjX3lNMQklkqjPSXVuK5b57ruFkmPSDqm0z7HSLpv28dPSDrQcbr8r3CqpIe9T1zXXbvtw3xJfSW5cY4dQYo1NPrDH6xB3777Sh98YI3sEjV9uq2gtmZN4u/Rk85TW0pLaYQNZJO6Ovu57d8/6JEAAHLFqFF2Q1TKzEojyaaoffml9Mwz6T/2kiWRV06Lh3du3ltDo1T1C/J4K6gl2gx7/nybSpmq6f//+IcVEkQSb2i0aJE1fyc08k0soVGppPbtykPbHou4j+u6LZLWSBrRaZ+T1S40kiTHcV6UtEzSOlnYhGwRS2i0ZYst+bnnntLzz9tdjmR4P/gff5zc+3Snrs7KI73/nLxKI5dME8gK3uqHAACkk1dtlKmh0cEH2/ltEFPUvEqjZJSUSAUF4VkBvU1DQ2pDo6oq+/tLtK/R/PlWDRRPk/d4DBlioVQkXvVerCuoeTNTdtop+XFBUmyhUaQ6ws5X0N3u4zjOXpI2uq7b4bvUdd1DJRVL6ifpgIgHd5zzHcd5z3Gc95bHM48RqVVYaEtDtrRE3+eBBywR/t//tSAmWV5olMopanV19gu1oMA+Lymxub+rV6fumAD8U1tLPyMAQPp5oVEmTk+T7GL/7LOtVUS6q3X8CI369JHGju2dlUaua6FRKqfWFxTYAkSJhkaffpqaqWmx2H5764kVa6WRd63oVVchabFcyYcktf/fb4ykzk1e/v8+juPkSxoqaWW7509Rpyojj+u6myTNVdcpb97zv3Ndd3fXdXcf5S31juAVFtp/cF9+Gfn5rVulX/xC2n136fDD/TlmaanN9U11aNS+SqGkxLb0NQIy31df2c8qlUYAgHTbaSe7STp2bNAjie7ss62v0Cmn2GI0qWz54Nm40aYKJRsaSfb7vTeGRs3NdpM6lZVGUuIrqK1bZ4FNUKGR49hN/XhCo/HjpcGDUzuuHBJLaPSupCrHcSocx+krC4DmdtpnrqSZ2z4+UdKrrmvzeRzHyZM1uX7E29lxnMGO4xRv+zhf0hGSEuzKhUB4U82iTVF76CH7T33WLP+61juOVRulOjRqX6VAaARkD+9EktAIAJBu555rKwMn244hlcrKpDvvtJu+3/mOjfWEE6zPUXetGNrabFpbrNOD2luyxLZ+hEYVFb1zelqqV07zTJtmFU3r18f3uk8/tW1QoZFk37vxTE+jn5GvegyNtvUo+r6kFyUtkPSY67rzHce5xnGco7ftdo+kEY7j1Ej6oaSftHuLfSWFXNdt/xM+SNJcx3E+kvShrK/RXUl/NUifwkLbRgqNWlutymjnnaUjj/T3uNOnW0+jtjZ/31eyFH3Zso4XnKXb2nfRDBvIfJ1XPwQAIF369ZP22CPoUfTswgulmhoLuC64QHrrLenYY6Vbbon+mhtukM47T9p///hvpC5dalu/QqMVK+ycvTfxQqN0VBpJ4RAoVqlcOS1WsVYabdwoVVcTGvkspkYzrus+57ruDq7rjndd9xfbHpvluu7cbR9vcl13huu6E1zX3bN9QOS67muu6+7d6f2aXdfdw3Xd6a7rTnVd9wfbwilki+5Co0cftR9WP6uMPNOnWzre0BD7a+69N/o0uvYiVSl4v+CoNAIynxcaUWkEAEB0jiPttZd0661SKCQdd5x0+eXSa6913ff116Urr7QVkJcts4basZxXe/ysNOqNK6itWSP96lfWsycdlUZS/FPU5s+3VWlTHWpm1P+1AAAgAElEQVR1p7zcAsjNm7vfb/58Ky4gNPKVD92JkZOihUatrdLPfy7tuKN0TMQ2VcmJtxl2ba2V315xRc/7RrrgHDDA+igRGgGZr7ZW2m47aeTIoEcCAEB2yM+3G6wTJkgnn2whkqe52fofTZggPf209Oyzdr582GGx90PyQqOiouTH6oUWvWWK2rp19nf50Ud2033gwNQer6LCrm3iDY0+/TS1K6fFwmsw39Psj/fesy0rp/mK0AiJGTbMfsl0Do2efFJasEC66ip/VkzrbOpUuzsSa2j02bZWWfff33NJY7QqhZISQiMgG3iN7P2ucAQAoDcbMkR66ilpwwZpxgyr5mhtlU47zVYQfuIJuymz//728YcfSkcdZVOBJOuHtGKFXbB3XnF4yRK7ZvDjho4XGvWGSqMNG6QjjpDefdcCI79bekSSlydNmZJYpVGQU9MkqzSSur+e27RJuvFGK14IsiqqF8oPegDIUnl50qhRHUOj2loLi6ZMsaZ6qTBokN3tiDc0cl3pppuk22+Pvm9dnYVh22/f8XFCIyA71NaGlzwGAACxmzJF+uMfpZNOki691M7zX31V+sMf7CLc861vSQ8+KJ16qrTnnraUe12drZAmSbvuKv3jH1Lfvvb5kiXWcNuPm8nbb28BV7aHRhs3Wuj29tvSww/b9MB0mTZNmjcv9v3XrrUG1NkQGt16q7Uwefnl1BQv5DD+NpG4wkIrW21pCae6S5faD2wqf1DjWUFt4UJpxAhp5kxb9aG5Ofq+tbWRe6GUltIIG8h0bW12Ekk/IwAAEjNjhvTjH0u/+Y10zTXSt79tbR46O+kkm9LWt6/dXJ050xppX3+99P77dhPZs2SJP/2MJKskzvYV1Nra7Ob6a6/ZTIiTTkrv8adNs5vhK1fGtn8mrJwmSWPG2DbaCmrNzbYQ01FHWf8t+IpKIySusNDKG/fYQ/rgA1t54Y47wiuOpcr06eES2kGDut934UJp4kTpJz+xX27eL7RI6uoiN00rKbFfeG1tpNZAplq61MqSCY0AAEjcddfZ+f2yZdKcOdH3O/NM+9NZXZ1V9x96qHTAAXYOPXasf+OrqJA+/9y/90u3f/1LeuEF6eabpdNPT//xvWbY8+dL++zT8/7eymlTpqRuTLEYMMCq36JVGl11lfTVV9Ls2ekdV47gChiJGz3aSgCbm62X0dNPpz4wkizYcd3Ylov0QqOqKkvy77xTWrWq636trfa1RLrgLCmx55cvT3roAFKktta248cHOw4AALJZfr70179K77yTWGPmX/1K2mEHC5RWrLCbOn5VGkl2rl5fb9cC2eiZZ6yhdKQKrnTwKoZi7Wv06afBr5zmKS+PHBp9+KF0zz3S979v33vwHaEREnfhhdKsWfafyfHHp++43rzqnqaorVljgdbEifb5T39qqxTccUfXfb/4QtqyJXpo5O0DIDNFa2QPAADi4ziJr5Q1aJD16Vm+XDr7bNv6sXKap6LCKkq6azmRyebOlfbbz1ZnDsKYMdYX6uOPY9t//nzrFxnkymmeiROtz9Zll1kgKVl4+MMfWl/aWbOCHV8vRmiExH3969LVV9sPaTpVVNgvpJ5Co4ULbeuFRtOn2zzXX/9aWr++477eBWekKgVCIyDz1dba9FE/S+ABAED8dtnFprnNnWsX9X5WGmXzCmo1NXaz/eijgxuD40jf+IZV5tx7b8/7Z8LKaZ6bb7YV/X71K7tmu/56W3nu1VftmjSoIC4HEBoh++TlWbVRvKGRJP3sZ9b47be/7bivN7Wlu0ojmmEDmauuzsqWCwqCHgkAAPjhD6WDDrKPCY3M3Lm2DTI0kqQHHrB+Rt/5jv07tbRE3m/NGikUypzQqKjIVvj78EMb/09/aqv4TZ4sXXBB0KPr1QiNkJ28FdS6m8+8cKGVUravHtp7b2vK98tf2nKhra32eF2d7VtW1vV9iooslafSCMhc0VY/BAAA6ZeXZ6uDXXCBtO++/r3vuHG2zcYV1J55xm58B90faPvtrRn3D35giwQdeaS0enXX/bz+sUE3we5sxx2lZ5+V/v536bjjpLvv5qZhihEaITtNn24VQ90FOQsX2n/Kfft2fPy226xh9xln2Ps88YSVi0arUigosJXiCI2AzFVXRxNsAAAySXGxdNdd/k4bGjjQbuhmW6XRl19Kb74pHXNM0CMx+fl2TfS739n0rr326rqcvRcaZUqlUWf77GMran/960GPpNcjNEJ2mj7dtt1NUfNWTuts6lTpP/+RHnvMKpVmzLD5sN1VKZSUEBoBmWr9elsamEojAAB6v4qK7AuNnntOamvLnNDIc9550iuv2HXOGWeEZ2FI1s9owIDgK6MQOEIjZKeeVlBra5OqqyOHRpKVzM6YYSsH/OlP0k47SYcfHv14hEZA5uqukT0AAOhdKiuzb3raM8/Y9cSuuwY9kq722UeaM8eme914Y/hxb+W0PCKDXMd3ALLTsGE2nSxaaLR4sbRpU/TQyNOnj3T66dIHH0g/+lH0/UpKaIQNZCrvxJFKIwAAer+KCptKtXVr0COJzaZN0osvWgPsTA1gzjxTOvlkW7b+vffssUxaOQ2BytDvWiAGXjPsSCKtnJaM0lKb/pItv5yAXOKtfkilEQAAvV9Fhc0q6NyDJ1O9+qq0YUPwq6Z1x3Gk3/zG+lCddprdLG9qyrwm2AgEoRGy1/Tp0mefSZs3d33O79CopMS2S5f6834A/FNXZ002hw0LeiQAACDVvMribJmiNneuNHiwreCcyYYPlx54wBYIOu44e4xKI4jQCNlsp52klhabWtbZwoXSkCHS6NH+HMsLjehrBGSeujqmpgEAkCu8xszZ0Ay7rc1Co8MOk/r1C3o0PdtvP+mKK6R337XPCY0gQiNks4MOsuUin36663PeymmO48+xCI2AzFVby9Q0AAByxZgxdg2QDaHRe+9JS5Zk9tS0zq6+WtptN2m77aRx44IeDTIAoRGy1/bbSwceKD3+uOS6HZ/zQiO/eKERzbCBzNLaKjU0UGkEAECu6NNHGjs2O6anzZ1r4/3Wt4IeSez69pVeeMF6MWVq426kFd8FyG4zZtgvjP/8J/zYhg1SKCRNmuTfcUaNsjsaVBoBmeWLL6xBvVeqDgAAer+KisyvNGprkx5+WNp/f7vZnU1GjpR23z3oUSBDEBohux17rKX3jz8efuzzz23rZ6VRXp6tJkBoBGSWhgbbUj4NAEDuyIbQ6O9/t5vb3/lO0CMBkkJohOw2YoStRNB+iprfK6d5SkoIjYBMQ2gEAEDuqayUli+X1q8PeiTR3XOPNHSodPzxQY8ESAqhEbLfjBnWCNdbRW3hQmuAPWGCv8cpKaGnEZBpFi2ybXl5sOMAAADpk+krqK1ZIz3xhHTqqdKAAUGPBkgKoRGy33HH2RS1J56wzxcutOZ4fv8HXVJiqx8AyBwNDVJRkdS/f9AjAQAA6eLdHP7734MdRzSPPCJt2iSdfXbQIwGSRmiE7DdypPTNb4anqPm9cpqnuFhatUr66iv/3xuABb/z58f3mkWLmJoGAECu2WUXad99pZ/9TFq8OOjRdPWHP0jTptFMGr0CoRF6hxkzpOpq6cMPrRF2KkKjkhLbLl3q/3sDue6DD6STTpKuvz6+1zU0WGUhAADIHXl50h//KLW2SuecE+5t2pONG+0mVaz7J+KTT6R//cvG5TipOw6QJoRG6B2OO85+edx2mzXES1WlkUQzbMBvriv96Ee29XoUxaKtze4uUmkEAEDuqayUZs+WXn5Zuuuu2F7zP/9jN5s/+ih14/rjH6WCAun001N3DCCNCI3QO4waJe2/v/TAA/Z5KiuN6GsE+Osvf5FefVUaMiS+EvOlS6UtW6g0AgAgV11wgXTwwdKPf2wL43Snvl6aM8c+XrAgNePZssWuR44+2q5PgF6A0Ai9x4wZUkuLfUylEZBZNmyI/PjWrdJll0mTJkkXXiiFQuGf4540NNiWSiMAAHKT49jS9n36SN/5jlUhR3PVVbaf41gP1FT4y1+k5ctpgI1ehdAIvcfxx9sUtUGDpNJS/99/xAgrNaXSCIhPXZ3dbTv99K6N5O+6y/qQzZ4tjR9vvQli/RnzprJRaQQAQO4qK7MWFW+8Id16a+R9/vMf6cEHpUsukcrLUxca/eEPNjvhkENS8/5AAPKDHgDgm8JCK09dvz41Tefy8mxpbyqNgPg88IAtO/vww9aw/umnLdhdtUr6v/+TDjpIOuII6cUXbf9Fi+wEsCdepRGhEQAAue2ss6SnnpIuv9xCm5NP7vj8T34ibb+9dMUV0vvv+xMaffyxLeSxdq20bp20Zo30/PN2jHwus9F78N2M3uWRR6xSIVVKSqg0AuLhuhYaHXCA9IMfSGecIe2xh/TnP0uPPmrB0c03W9DrhT+x9jVatMgqmAYNSt34AQBA5nMc6U9/ko48Ujr1VLuJfM459tzLL0svvWTnG8OGWRuLt9+2c5REbzS3tEj77WfnMZ78fGtncd55yX89QAZhehp6l2HDbBpZqhQXU2kExOOdd6wx5RlnSMccYydp/fpJ++4r3X67ndBNn277lpfbNtYV1BoaqDICAABmu+2s0ufQQ6Vzz7Wpam1tVvkzdqx00UW238SJFiolc07/7rsWGP3mN1Jzs7RxozXBDoWkigp/vh4gQxAaAfGg0giIzwMPSP37W88xSdpxRzvR2ntvafBg6dprw/sOGmShbzyVRjTBBgAAnoEDrZr5uOOsf9HRR9t0tGuvtZtWUnjBnGSmqM2bZ1VKM2ZYi4wBA1LTHgPIAIRGQDyKi6WVK60/C4DubdliU9COPVYaMiT8+MiR0t/+ZqFPUVHH15SXx1Zp5LpUGgEAgK769ZMee0w680zpr3+VdtrJFuPw+BUa7bZbamc4ABmC0AiIR0mJbZcuDXYcQDZ44QVpxQqbmtaZ41gZeWdjx8ZWabRsmYW3VBoBAIDO8vOle++V7rjDVk3La3fZW1pqFUmJhkZr10r/+IctwAPkAEIjIB7FxbalrxHQsz/9yRpVx7PsrFdp5Lrd7+dVI1FpBAAAIsnLsz5GU6d2fXyHHRIPjV57zRbeITRCjiA0AuLhVRoRGgHdW71amjtXOuUUqaAg9teNHWvNKVev7n6/hgbbUmkEAADiNXFi4qHRvHlWqfS1r/k7JiBDERoB8fAqjWiGjWy3fr39SZUnn5Q2b7Z+AvGIdQU1Ko0AAECiJk60G1CJ9CmdN0/ab79wY22glyM0AuIxcqTNkabSCNmsuVmaNEk68cTUHeOBB6z0e/fd43udFwL11NeooUEaPrxjg20AAIBYTJxoU+FrauJ73eLFVqHE1DTkEEIjIB55ebbaE5VGyFYtLTZlrKlJevHF2FYqi9fixdLrr1uVUbzLz8ZTaUSVEQAASESiK6jNm2dbQiPkEEIjIF4lJVQaIXv97GfWwPGaa+zzBx7w/xgPPmjb9svbxqqwUOrfP7ZKI/oZAQCAROywg20TCY2Ki7s21wZ6MUIjIF7FxVQaITs9+aR0003Sd78rXXWVzce///6eVyqL1/PPS3vsIVVUxP9axwmvoBaN6xIaAQCAxG23nd0Ijic0amuTXnlFOuig+CupgSxGaATEi0ojZKPPPpO+8x1pr72kW26xx2bOlKqrpXfe8fdYtbXJ3YErL+++0mjlSmnDBqanAQCAxMW7gtoHH0hffikdckjqxgRkIEIjIF7FxXbRunlz0CMBYrN+vXT88bbKx+OPh1f7OOEEacAA6b77/DvWpk0WqlZWJv4eY8d2X2nU0GBbKo0AAECidtjBQqNYK65fesm2Bx2UujEBGYjQCIhXSYltmaKGbPHAA9KCBdJDD0llZeHHhwyRjjtOeuSRyCHoH/4gHXWU9OabsR/LC3QSmZrmKS+Xli6NHsx6gRKVRgAAIFETJ0qrV0vLl8e2/7x50o472qI4QA4hNALiRWiEbLNggTR4cOQ7YzNn2gnTX/7S8fF//lO64ALrT7TPPtIRR0jvv9/zserrbZtMaOSFQY2NkZ+n0ggAACQrnhXUNm60m2ismoYcRGgExKu42Lb0NUK2qK6WqqoiN2088ED7nm4/RW3lSunkk60qadEi6frrLUTabTdpxgyrAoqmrs62yUxPKy+3bbS+RosWWQPLYcMSPwYAAMht8YRGb7whbdlCaIScRGgExItKI2QbLzSKpE8f6YwzrKJo2TJbGWTmTAuGHntMKi2VrrjCKohmzZKeecZCpGjq66X+/ZMr3fYqjaL1NfJWTmPlEgAAkKhx46S+fWMLjd58086Z9t035cMCMg2hERCvkSOl/HwqjZAdtm61kCVaaCRJZ50ltbRYb6PZs22q2s03S7vvHt5n6FDp6qulXXaR5s+P/l719TY1LZlAZ8wYe320SqOGBvoZAQCA5PTpI02YEFtotHix3TgeODD14wIyTH7QAwCyTl6eVVFQaYRsUF8vtbZ2HxpNmybtuqt00032fT1jhnTRRZH3nTRJeuWV6O9VV5dcPyPJ7voVF0evNFq0iDt9AAAgeRMndn8zzNPUZNXXQA6i0ghIRHExlUbIDtXVtu0uNJJsSlooZKXad98dvVJo0iQ7cVq3rutzrmuhUTL9jDzl5ZErjVavltasodIIAAAkb+JEO3fZurX7/QiNkMMIjYBElJRQaYTsUFNj255CozPPtD9PPWVT0aKZNMm2kUq5V62S1q5NvtJIslAoUqWR9xgrpwEAgGRNnGhT9L2FPKJparLp80AOIjQCEkGlEbJFdbU0ZIj14urO8OHS/fdL06d3v9/kybb97LOuz9XX29avSqPGRmvM3V5Dg20JjQAAQLK8FdQ+/zz6PmvXWoU1lUbIUYRGQCJKSqQVK6TNm4MeCdA9b+U0v1YaGz/eGsEvWND1OS808qvSaPNmW9GtPa/SiOlpAAAgWV5o1F0z7KYm2xIaIUcRGgGJKC627dKlwY4D6IkXGvmloMCCo0iVRl5ptx+hUXm5bTv3NWposJVLeqqcAgAA6Mn229s5BaEREBWhEZCIkhLbMkUNmWzLFqvM8TM0kqyvUbTpaSNG2HS4ZHmVRO37GrW1SW+8YdPf/KqcAgAAuW3ixO5Do1DItvQ0Qo4iNAIS4VUa0QwbmayuzoKWVIRG1dXWOLLz8fyoMpLClUbtQ6O775bee0+67DJ/jgEAADB+fPeNsL1KI++mMZBjCI2ARFBphGxQXW1bv0OjyZNtadrOJ1j19f40wZakYcOsYsmbnrZkiXTFFdIBB0hnneXPMQAAACor7Zx+06bIzzc12TS2AQPSOy4gQxAaAYkYNUrq04dKI2S2VIVGkybZtv0UtdZW6zfkV6WRZNVGXqXRxRfbydxddzE1DQAA+KeiQnLdjtXN7TU10c8IOY3QCEhEXp5UVESlETJbdbU0fLj1GfKTt9JI+9Doiy+s+sivSiPJ+hotXiz99a/S449LV17pfwAGAABym3fu4q0C21koRD8j5DRCIyBRxcVUGiGz+b1ymmfYMAtNFywIP+adaPldaVRXJ33ve9KUKdLll/v33gAAAFL43CVaXyMqjZDj8oMeAJC1Skqi35EAMkFNjfSNb6TmvSdP7lhp5J1o+V1ptHat/XnjDalvX//eGwAAQLIbwf36RT6v37pVWraM0Ag5jUojIFFUGiGTbdpkU7tSNZ1r0iQLjVzXPq+vt2mb3qpnfvDe64ILUhd+AQCA3JaXJ40bF7nSaMkSO9chNEIOIzQCElVSIn35pbRlS9AjAbqqq7OTnFSGRqtX290373hjxkgFBf4d47DDpB//WLrhBv/eEwAAoLPKysiVRqGQbelphBxGaAQkqrjYtkuXBjsOIJJUrZzm8VZQ8/oa1df7OzVNsibeN94oDR3q7/sCAAC0V1ERudKoqcm2VBohhxEaAYkqKbEtK6ghE3mh0YQJqXl/LzTy+hrV1/vbBBsAACBdKiulNWukVas6Pk5oBBAaAQnzKo3oa4RMVF0tjRhh1TqpMGaMNGiQhUZffWXhqd+VRgAAAOngncN0rjYKhaxJ9vbbp39MQIYgNAISRaURMll1deqmpknWNHLiRAuNFi2yx6g0AgAA2cg7h+nc16ipyW6UOU76xwRkCEIjIFGjRtmFM5VGyESpDo0km6K2YEH4rhyVRgAAIBt5oVHnSqOmJqamIecRGgGJ6tNHKiykETYyz8aNVk6djtBo8WLpk0/scyqNAABANho61KagRao0IjRCjiM0ApJRVERohMxTW2vbdIRGkvTCC9KAAdLo0ak9HgAAQKpUVnasNHJdQiNAhEZAcoqLCY2QeWpqbJvq0GjyZNu++aZVGTHfHwAAZKuKio6VRitWSJs3W08jIIcRGgHJKCqipxEyT3W1bVMdGk2YYH29tm5lahoAAMhulZVSQ4PU2mqfNzXZlkoj5DhCIyAZRUVSc7PU1hb0SICw6mrrtzVkSGqP079/OCyiCTYAAMhmFRV2I8xbGZnQCJBEaAQkp7jY7kasWBH0SICwdKyc5vH6GlFpBAAAspl3A8zraxQK2ZbQCDmO0AhIRlGRbZmihkySztDI62tEpREAAMhm3g0wr69RU5P1aywuDm5MQAYgNAKS4YVGNMNGpti0ycqqx49Pz/GmTrVtukIqAACAVCgvt16NXqVRU5OtDFtQEOy4gIDlBz0AIKt5dx4IjZApmpttm667YqedJo0aJU2Zkp7jAQAApELfvlJZWcdKI6amAVQaAUmh0giZxguNRo9Oz/H69pW+9a30HAsAACCVKio69jQiNAIIjYCkDB4sDRpETyNkjnSHRgAAAL1FZWXHSqMxY4IdD5ABCI2AZBUVUWmEzEFoBAAAkJiKCrsZvGKFtGoVlUaACI2A5BUXExohc3ihUWFhsOMAAADINt5qsG+9ZVtCI4DQCEhaURHT05A5mpuloUOl/v2DHgkAAEB2qaiw7Ztv2pbQCIgtNHIc5zDHcRY6jlPjOM5PIjzfz3GcR7c9/47jOOO2PX664zgftPvT5jjOzo7jDHQc56+O43zmOM58x3Gu9/fLAtKI6WnIJM3NTE0DAABIhFdp9MYbtqWnEdBzaOQ4Th9JcyQdLmmKpFMdx+m8tvI5kla5rjtB0i2SbpAk13UfdF13Z9d1d5Z0pqQG13U/2Paa2a7rTpK0i6SvO45zuC9fEZBuxcXSmjXSV18FPRKA0AgAACBRhYXSwIHSv/9tn1NpBMRUabSnpBrXdetc190i6RFJx3Ta5xhJ9237+AlJBzqO43Ta51RJD0uS67obXdf927aPt0h6XxIxLrJTUZFtvV4yQJAIjQAAABLjODZFbetWabvt7A+Q42IJjUolNbb7PLTtsYj7uK7bImmNpBGd9jlZ20Kj9hzHGSbpKEmvxDZkIMN4oRF9jZAJCI0AAAAS5/U1osoIkBRbaNS5YkiS3Hj2cRxnL0kbXdf9pMOLHCdfFiTd5rpuXcSDO875juO85zjOe8uXL49huECaFRfblr5GCNqWLbY8LKERAABAYry+RvQzAiTFFhqFJJW1+3yMpC+i7bMtCBoqaWW7509RhCojSb+TVO267q+jHdx13d+5rru767q7jxo1KobhAmnmVRoRGiFoy5bZltAIAAAgMV5oRKURICm20OhdSVWO41Q4jtNXFgDN7bTPXEkzt318oqRXXdd1JclxnDxJM2S9kP4/x3F+LguXLkl8+EAGGDXK5j8zPQ1B8/pqERoBAAAkhulpQAc9hkbbehR9X9KLkhZIesx13fmO41zjOM7R23a7R9IIx3FqJP1Q0k/avcW+kkLtp585jjNG0v/IVmN733GcDxzHOdeXrwhIt/x8C46oNELQCI0AAACSw/Q0oIP8WHZyXfc5Sc91emxWu483yaqJIr32NUl7d3ospMh9kIDsVFxMaITgERoBAAAkZ+pU6fbbpVNOCXokQEaIKTQC0IOiIkIjBI/QCAAAIDmOI33/+0GPAsgYsfQ0AtCToiJ6GiF4zc3S4MHSwIFBjwQAAABAL0BoBPihuNgu2Nvagh4JcllzM1VGAAAAAHxDaAT4oahI2rpVWrUq6JEglxEaAQAAAPARoRHgh6Ii2zJFDUEiNAIAAADgI0IjwA/FxbalGTaCRGgEAAAAwEeERoAfvEojQiMEpaVFWrGC0AgAAACAbwiNAD8QGiFoy5dLrktoBAAAAMA3hEaAH7bbThowgJ5GCE5zs20JjQAAAAD4hNAI8IPjWF8jKo0QFEIjAAAAAD4jNAL8UlREaITgEBoBAAAA8BmhEeCXoiKmpyE4hEYAAAAAfEZoBPiF6WkIUnOz9dUaPDjokQAAAADoJQiNAL8UFUmrVkmbNwc9EuSi5marMnKcoEcCAAAAoJcgNAL8UlRkW2+aEJBOXmgEAAAAAD4hNAL8UlxsW/oaIQiERgAAAAB8RmgE+MWrNKKvEYJAaAQAAADAZ4RGgF8IjRCU1lZp+XJCIwAAAAC+IjQC/FJYaE2ImZ6GdFuxQmprIzQCAAAA4CtCI8AvBQXSyJFUGiH9vObrhEYAAAAAfERoBPipqIjQCOlHaAQAAAAgBQiNAD8RGiEIhEYAAAAAUoDQCPBTcTE9jZB+hEYAAAAAUoDQCPCTV2nkukGPBLmkuVnq21caOjTokQAAAADoRQiNAD+VlkpbtkgLFgQ9EuSS5marMnKcoEcCAAAAoBchNAL8dNJJ0pAh0qWXUm2E9PFCIwAAAADwEaER4KeiIunaa6WXXpKeeiro0SBXEBoBAAAASAFCI8Bv3/ueNH26dMkl0oYNQY8GuYDQCAAAAEAKEBoBfsvPl+bMkUIh6ec/D3o06O3a2qRlywiNAAAAAPiO0AhIhW98Q5o5U7r5Zumzz4IeDXqzVauklhZCIwAAAAC+IzQCUuWGG6SBA6Uf/ICm2Eid5mbbEhoBAAAA8BmhEZAqo0dLv/iF9PLL0t+1WuIAAB7uSURBVOOPBz0a9FaERgAAAABShNAISKULL5QmTJDuuSfokaC3IjQCAAAAkCKERkAq9ekj7bqrVFcX9Eiia2mxRsqNjUyjy0aERgAAAABShNAISLXKSqmhQWptDXokYeefbxVQw4dLBQUWOJSXS+PHS//939K8edLmzak7/t/+Jt15Z+reP5c0N9uKfcOHBz0SAAAAAL1MftADAHq9ykqr5gmFpLFjgx6NtGSJdPfd0n/9l3T44dKIEdLIkVZl9NJL9tztt0uDB0uXXy5ddZW/x29tlc4+24K0sjLpqKP8ff9c09wsFRZKedwDAAAAAOAvQiMg1SorbVtXlxmh0csv23bOHGmXXTo+94MfSBs3Sq++Kl15pXTvvf6HRs8/b4HRsGFW8fTJJxZcITGhkFRaGvQoAAAAAPRC3JoGUq19aJQJ5s2zyqKddor8/MCB0pFHSgcfLDU1+d/n6I47pJISG8eXX9p0OCQuFLKKLQAAAADwGaERkGplZdYQOxNCI9e1SqMDD+x5OlNpqfU1WrnSv+NXV0svvmiryu2+u1UxPfSQ9NRT/h0j1zQ2SmPGBD0KAAAAAL0QoRGQavn5Ni0tE0KjTz+1nkYHH9zzvt6Up6Ym/45/553WePu88+zzn/5U2m03C5GWL/fvOLli7Vpp3ToqjQAAAACkBKERkA6VlZkRGs2bZ9sgQqMNG6Q//lE68USpqMgeKyiwvklr1kjf/a7/U+F6u8ZG21JpBAAAACAFCI2AdMiU0Ojll6WqKqm8vOd9/Q6NHnzQwqGLLur4+LRp0tVXS08+KT37rD/HyhWhkG2pNAIAAACQAoRGQDpUVlrT57VrgxvDli3Sa6/FVmUkScXFtvUjNHJda4C9887S177W9fnLLpP695feeCP5Y+USKo0AAAAApBChEZAO3gpq9fXBjeGf/7QpYrGGRn37SoWF/oRGb74pffyxVRk5Ttfn8/Mt+PBCEMQmFLK/z5KSoEcCAAAAoBciNALSwQuNgpyi9vLLtmLa/vvH/prSUn9CozvukIYNk047Lfo+5eXS4sXJHyuXNDZaf6iCgqBHAgAAAKAXIjQC0iETQqN586Q997TwJlZ+hEZLlkhPPSWdfbY0cGD0/crKqDSKVyhEPyMAAAAAKUNoBKTD8OEW1gQVGq1eLf3rX9JBB8X3utJS6Ysvkjv2P/4htbRIp5zS/X5lZXaslpbkjpdLGhvpZwQAAAAgZQiNgHQJcgW1116T2tpi72fkKS2Vli+XNm9O/Ng1NbbdYYfu9ysrszEuWZL4sXINlUYAAAAAUojQCEiXIEOjefOkQYOkvfeO73WlpbZNJsipqZFGjpSGDu1+Py/8oK9RbNaskdato9IIAAAAQMoQGgHpUlkpNTRIra3pP/a8edJ++9mKaPHwQqNk+hrV1koTJvS8X3m5belrFJtQyLZUGgEAAABIEUIjIF0qK6UtW5LvERSvRYuk6ur4p6ZJ/oRGNTWxhUZe+EFoFBvv74lKIwAAAAApQmgEpEuyK6hVV0szZ0rLlsX3umeftW0ioVFJiW0TDY02bbJwI5bQaMgQ+0NoFBuv0ojQCAAAAECKEBoB6ZJsaHTrrdL990szZkhbt8b2mvvuky69VNp1V2nKlPiPOXy41L9/4qFRfb3kurGFRpJVGxEaxaaxUXKccLAHAAAAAD4jNALSpbxcystLLDRqbZWeeEKqqJD+/nfpkku63991pVmzpG9/23oZvfKKBQzxchybopZoaOStnBZPaEQj7NiEQlJRkVRQEPRIAAAAAPRS+UEPAMgZBQUWHCUSGr35ptTcLD36qPTuu9Ls2dLOO0vnndd1382bpbPPlh56yLZ33ZVcsJBMaFRba9vx42Pbv7xc+ve/EztWrmlspAk2AAAAgJSi0ghIp8rKxEKjxx+XBgyQvvUt6frrpUMOkS66SHrrrfA+mzdLc+dKBxxggdF110m//33ylSjJVhoNHSqNGBHb/mVl0vLl1gsJ3QuF6GcEAAAAIKUIjYB0SiQ08qamHXGENGiQ1KeP9Mgj0tix0gknWPXRzJlSYaF0zDHSZ59JDz8s/fSniU1J68wLjVw3/td6K6fFOg6vcsZr8ozIXJdKIwAAAAApR2gEpFNlpa1+tn597K/xpqaddFL4seHDpT//WdqwQTrlFOmZZ6Tjj5eef15autQe80tpqVUxrVwZ/2u90ChWXghCM+zurV1r30NUGgEAAABIIXoaAenkraBWXy/tuGNsr2k/Na29qVOl11+3kOjAA6V+/fwdq6e01LZNTbFPM5NshbeGhvgCLC80ohl297xQjUojAAAAAClEpRGQTl5oFOsUtdZW6cknw1PTOtt1V3suVYGR1DE0isfixTb+WJtgS+HKGSqNuudN36PSCAAAAEAKERoB6RRvaPTmm1ZJNGNG6sbUk0RDo5oa28YzPW3AAGnUKEKjnlBpBAAAACANCI2AdNp+e2nIkNhDo2hT09KpuNi26QiNJAtCCI26FwpZc3Hv3wYAAAAAUoDQCEgnx7Fqo/r6nvdtPzVt8ODUjy2avn1tZbZEQqOBA6WiovheR2jUs8ZGC4wKCoIeCQAAAIBejNAISLfKytgqjd56K/ipaZ7S0sRCowkTLCiLR1kZjbB7EgrRzwgAAABAyhEaAenmVRq1tXW/32OPSf37Bzs1zVNSIn3xRXyvqa2Nrwm2p7zclpRfuzb+1+aKxkb6GQEAAABIufygBwDknMpKadMmqyIqKbFwpLHRlqf//PPwn3/8wwKjIKemeUpLpX/9K/b9W1stNDryyPiP5YUhjY3S1Knxv763c137uzn00KBHAgAAAKCXIzQC0s1bQe3rX5dWruxaUTN8uDRxok1Lu+yy9I8vktJSaflyafNmqV+/nvdvapK2bIm/CbZEaNSTNWukDRuoNAIAAACQcoRGQLrtvbd0+OE29ayszHrTlJXZtKyJE6URI4IeYVelpbZdskQaN67n/RNdOU3qGBqhq1DItvQ0AgAAAJBihEZAug0dKj33XNCjiI8XGjU1pT40KimR8vJohh2NF6ZRaQQAAAAgxWiEDaBn7UOjWNTW2jS2RKph8vNtOXkqjSKj0ggAAABAmhAaAehZvKFRTY1UUWEVQ4koLyc0iqax0f5ei4uDHgkAAACAXo7QCEDPhg+3HkzxhEaJTE3zlJVFDo0++0xatCjx9+0NQiGpqEgqKAh6JAAAAAB6OUIjAD1zHKs2iiU0cl3/QiPXDT+2aZO0//7SRRcl/r69QWMj/YwAAAAApAWhEYDYxBoaLV0qbdyYfGi0aZP05Zfhx+67T2pulj7/PPH37Q1CIfoZAQAAAEgLQiMAsYk1NKqttW2yoZEUnqLW2irNnm0fNzTY57nIdak0AgAAAJA2hEYAYuOFRu2njEVSU2Pb8eMTP1Z5uW290Ojpp+19DzlE2ro1vIJYrlmzRtqwgUojAAAAAGlBaAQgNqWl0ubN0sqV3e9XUyP16SONHZv4sdpXGrmudOONVrl02WX2eF1d4u+dzbwQjUojAAAAAGlAaAQgNiUltu1pilpNjTRuXHKre40aJfXtayHJ669L775rgVFVlT2fq6GRt3KcV4kFAAAAAClEaAQgNqWlto0WGm3dKj3/vPT228n1M5KkvDybgrV4sXTDDVJhoXTWWfZYfn7uhkbe1L9k/34BAAAAIAb5QQ8AQJbw+uhccon02GPS9On2Jz9fevRR6fHHbbWzYcOk009P/nhlZVZltGSJ9ItfSAMG2ONjx4abbeeamhpp6FBpxIigRwIAAAAgBxAaAYhNebl09dXSG29YRdG994afGzBAOvpo6bTTpEMPlfr18+d4r78uDRokffe74ccrK3O30qi21hqMO07QIwEAAACQAwiNAMTGcaRZs8KfL1smffyxtHatdPDB0uDB/h7Pa/Z8/vnS8OHhx8ePt6qmXFRTI+26a9CjAAAAAJAjCI0AJKawUDrwwNS9/667SkOGSJde2vHxykppxQpbfn7o0NQdP9O0tEgNDdJJJwU9EgAAAAA5gkbYADLTCSdYj6TOy8tXVtq2vj79YwrS4sUWHI0fH/RIAAAAAOSImEIjx3EOcxxnoeM4NY7j/CTC8/0cx3l02/PvOI4zbtvjpzuO80G7P22O4+y87blfOI7T6DjOej+/IAC9SEFB18e80CjXmmGzchoAAACANOsxNHIcp4+kOZIOlzRF0qmO40zptNs5kla5rjtB0i2SbpAk13UfdF13Z9d1d5Z0pqQG13U/2PaaZyXt6c+XASBneKFRrjXDJjQCAAAAkGaxVBrtKanGdd0613W3SHpE0jGd9jlG0n3bPn5C0oGO02V5n1MlPex94rruP13XXZLYsAHkLG/J+VwLjWprbZW64uKgRwIAAAAgR8QSGpVKamz3eWjbYxH3cV23RdIaSSM67XOy2oVGAJCwysrcC41qaqyfUZc8HgAAAABSI5bQKNIVihvPPo7j7CVpo+u6n8QxNu+15zuO857jOO8tX7483pcD6I1yNTRiahoAAACANIolNApJar980RhJX0Tbx3GcfElDJa1s9/wpSrDKyHXd37muu7vruruPGjUqkbcA0NtUVtry8y0tQY8kPdraLCRj5TQAAAAAaRRLaPSupCrHcSocx+krC4DmdtpnrqSZ2z4+UdKrruu6kuQ4Tp6kGbJeSACQvMpKC4xCoaBHkh5ffCFt2kSlEQAAAIC06jE02taj6PuSXpS0QNJjruvOdxznGsdxjt622z2SRjiOUyPph5J+0u4t9pUUcl23w1wSx3FudBwnJGmg4zghx3H+L/kvB0BO8CpucmWKGiunAQAAAAhAfiw7ua77nKTnOj02q93Hm2TVRJFe+5qkvSM8frmky+MYKwCYykrb1tVJBxwQ7FjSobbWtkxPAwAAAJBGsUxPA4DMMmaMlJ+fW5VGBQVSWVnP+wIAAACATwiNAGSfPn2kcePCFTi9XU2NVFFhQRkAAAAApAmhEYDsVFmZO5VGtbVMTQMAAACQdoRGALLT+PG5ERq5rlUa0QQbAAAAQJoRGgHITpWV0sqV0urVQY8ktZYvl9atIzQCAAAAkHaERgCyk7eCWn19sONItZoa2xIaAQAAAEgzQiMA2ckLjXp7M2zv66OnEQAAAIA0IzQCkJ280Ki39zWqqZHy8my1OAAAAABII0IjANlpyBBp5MjcCI3Ky6V+/YIeCQAAAIAcQ2gEIHtVVvb+0Ki2lqlpAAAAAAJBaAQge+VCaFRTQxNsAAAAAIEgNAKQvSorpUWLpJaWoEeSGqtXSytWEBoBAAAACAShEYDsVVlpgVFjY9AjSQ1WTgMAAAAQIEIjANnLC1P+/e9gx5EqNTW2pdIIAAAAQAAIjQBkrz33lHbYQbrwwt7Z28gLjSorgx0HAAAAgJxEaAQgew0cKP3lL1Jbm3TkkdKaNV33aWiQjjtOeuyxtA8vabW1UnGxNGhQ0CMBAAAAkIMIjQBkt6oq6amnpOpq6aSTOjbF/vOfpV12se0551iAlE1YOQ0AAABAgAiNAGS//feX7rpLeukl6ZJLpM2bpYsvtgqjCROkV1+1/c49V3LdQIcaF0IjAAAAAAEiNALQO5xzjnTZZdKcOdLEidJtt1lw9Oab0je/Kc2eLb3yivTb3wY90ths2CAtWcLKaQAAAAACQ2gEoPe4/nqrLlqzRnr6aenXv5b69bPnzj9fOuggC5bq64MdZyy8xt5UGgEAAAAICKERgN6jTx/pySetQufYYzs+5zjS738v5eVZVVJbWzBjjJW3chqhEQAAAICAEBoB6F0cR+rfP/JzY8dKN98s/e1v1gMpk3mhEdPTAAAAAASE0AhAbjn3XOmQQ6TLL5dWrgx6NNHV1kojRkjDhgU9EgAAAAA5itAIQG5xHOtrtGGD9NFHQY8mOlZOAwAAABAwQiMAuaeqyrbV1cGOozs1NUxNAwAAABAoQiMAuaesTOrbN9w3KNNs3iw1NlJpBAAAACBQhEYAck+fPlJlZeZWGjU02OpuhEYAAAAAAkRoBCA3VVVlbmjEymkAAAAAMgChEYDcVFVl4UxbW9Aj6aq21rZUGgEAAAAIEKERgNxUVSVt2iQ1NQU9kq5qaqTttpNGjQp6JAAAAAByGKERgNyUySuo1dRYlZHjBD0SAAAAADmM0AhAbvKmfmXiCmq1tfQzAgAAABA4QiMAuamsTOrXL75KozlzLGxy3dSNq6VFqq+nnxEAAACAwBEaAchNeXlWzRNPaPTqq1YFtGJF6sbV2Cht3UpoBAAAACBwhEYAcldVVXyh0aef2raxMTXjkcIrpzE9DQAAAEDACI0A5K6qKgtp2tp63nfz5nDAlMrQyOuxRKURAAAAgIARGgHIXRMmWBgUSwj0+edSa6t9nOrQqH9/qaQkdccAAAAAgBgQGgHIXVVVto1lBbX588MfL16cmvFIVvlUWWk9lwAAAAAgQFyVAMhdXmgUS1+j+fMtyCktTX2lEVPTAAAAAGQAQiMAuau01KaCxRoaTZhgf1IVGrW1WaURoREAAACADEBoBCB35eVZQBNraDR1qlRenrrQaMkS6auvWDkNAAAAQEYgNAKQ22IJjTZvtmljU6dKZWVSKBRuiu2n2trwmAAAAAAgYIRGAHJbVZVUV9d9CLRwoU0d80Kj1lZp6VL/x+I15CY0AgAAAJABCI0A5LaqKmnLlu6nnHkrp3mhkZSaKWo1NVJ+vk2BAwAAAICAERoByG2xrKA2f77Up4+0ww7hQGfxYv/HUlsrjRtnwREAAAAABIzQCEBuizU0mjBB6tcv9ZVGTE0DAAAAkCEIjQDktpISacCAnkOjqVPt46FDpcGD/Q+NXNdCI1ZOAwAAAJAhCI0A5DbH6X4FtU2bbNqYFxo5jlUb+R0arVghrV1LpREAAACAjEFoBABVVdFDo/Yrp3nKy/3vacTKaQAAAAAyDKERAFRVSfX1UktL1+far5zmSUWlkRcaMT0NAAAAQIYgNAKAqipp69bI1UPz59tqZjvsEH6srExqbpY2b/bn+K2t0u9+Jw0ZIlVW+vOeAAAAAJAkQiMA6G4Ftfnz7fm+fcOPeSuohUL+HP+666Q33pDuvNNWaAMAAACADEBoBABeH6FooVH7qWmS9TSS/Jmi9vbb0tVXS2ecIZ1+evLvBwAAAAA+ITQCgOJiadCgrqHRV1/ZymlTpnR83Ks0SjY0WrPGgqLycmnOnOTeCwAAAAB8lh/0AAAgcI5j1UbvvmsrpeVty9MXLpRct2ul0Zgxtk0mNHJd6cL/1979x9pZ13cAf3/6AzLHjzksRuktNliT1X8617AmKGEubkUTCwkGyIJEjWykGJbwTzcTt+wvMNlIjI7ERQI0zGK2FZsIuulMCAYddWlWfkhWCx13EKE6GQadVr/743luernn3vb29J57bu99vZKTc873POfp5/zxyXP7zvf7ff6kO8ejj3b7GQEAACwhZhoBJMlHP5o89ljyyU8eH5vtzmlJ8oY3JBdcMPvG2fO1e3eyZ0+3NG3btuHPAwAAMCJmGgEkySc+kTz1VHL77d0dzD7+8eN3TpvaKHu6DRuGn2n04x8nO3cml1+e7Np1enUDAACMiNAIIOmWqH32s8mRI8nNNycXX9yFRu94x+vvnDZlYiJ59tnh/q2vfCX5yU+SO+5IVq8+vboBAABGxPI0gClr1iQPPNAtR7vmmu7OZjOXpk2ZmBh+edrevd3m25deOnytAAAAIyY0ApjuvPO6mUDnnpscPXri0OiVV5JXXz218//0p8nDDyc7dhzfcBsAAGAJ8j8WgJnWr++Co40bk/e+d/ZjNmzonk91X6Ovfz157bXk6qtPr0YAAIARs6cRwGy2bEkOH57784mJ7vn555PNm+d/3r17k/PPT6644rTKAwAAGDUzjQCGMRUancq+RseOJfv2JR/4wOybawMAACwhQiOAYbz1rd0d105ledq3vpX88IeWpgEAAGcEoRHAMNau7YKjUwmNHnwwOfvsZPv20dUFAACwQIRGAMOamJj/8rTWuv2M3ve+5JxzRlsXAADAAhAaAQxrYmL+M40OHEiOHEmuumq0NQEAACwQoRHAsKZCo9ZOfuyDDyarViUf/ODo6wIAAFgAQiOAYW3YkPzsZ93m1iezd2/y7ncn69aNvi4AAIAFIDQCGNbERPd8sn2Nvv/95OBBS9MAAIAzitAIYFhTodGJ9jVqLbn//u610AgAADiDrBl3AQBnrBOFRpOTye7dyX33Jd/7XnLZZcnGjYtbHwAAwGkQGgEM68ILk7POSu68M3nooWTNmmTt2m6Po0ce6WYZvec9yW23JddeO+5qAQAATonQCGBYq1Ylt96aPPpocvRocuxY8otfdMHRpz6V3HBDcskl464SAABgKEIjgNPx6U+PuwIAAICRsBE2AAAAAAOERgAAAAAMEBoBAAAAMEBoBAAAAMAAoREAAAAAA4RGAAAAAAwQGgEAAAAwQGgEAAAAwAChEQAAAAADhEYAAAAADJhXaFRV26vqmao6VFW7Zvn87Kp6oP/8O1X1tn78j6rqwLTHr6pqS//Z71TVwf47n6mqWsgfBgAAAMDwThoaVdXqJJ9LcmWSzUmur6rNMw77WJL/aa29PcmdSe5Iktba/a21La21LUluSPJca+1A/527ktyUZFP/2L4AvwcAAACABTCfmUaXJjnUWjvcWvt5kj1Jdsw4ZkeSe/vX/5Dk92eZOXR9ki8mSVW9Jcl5rbXHWmstyX1JrhryNwAAAACwwOYTGl2U5Plp7yf7sVmPaa0dS/JKkgtmHHNt+tCoP37yJOcEAAAAYEzmExrNttdQO5Vjqup3k7zWWnviFM459d2bqmp/Ve1/+eWX51EuAAAAAKdrPqHRZJKJae/XJ3lhrmOqak2S85P8aNrn1+X4LKOp49ef5JxJktba51trW1trW9etWzePcgEAAAA4XfMJjR5PsqmqNlbVWekCoH0zjtmX5Mb+9TVJ/rXfqyhVtSrJh9LthZQkaa29mOTVqtrW73304SRfPq1fAgAAAMCCWXOyA1prx6rqliRfS7I6yd2ttSer6q+S7G+t7UvyhSS7q+pQuhlG1007xeVJJltrh2ec+uYk9yT5tSQP9w8AAAAAloDqJwSdEbZu3dr2798/7jIAAAAAlo2q+m5rbevM8fksTwMAAABghREaAQAAADBAaAQAAADAAKERAAAAAAPOqI2wq+rlJEfGXccCeFOSo+MuAhigN2Fp0puw9OhLWJr0JsO6uLW2bubgGRUaLRdVtX+2XcmB8dKbsDTpTVh69CUsTXqThWZ5GgAAAAADhEYAAAAADBAajcfnx10AMCu9CUuT3oSlR1/C0qQ3WVD2NAIAAABggJlGAAAAAAwQGi2yqtpeVc9U1aGq2jXuemAlq6rnqupgVR2oqv392G9W1b9U1X/2z28cd52wnFXV3VX1UlU9MW1s1j6szmf6a+h/VNW7xlc5LG9z9OZfVtV/99fNA1X1/mmf/Vnfm89U1R+Op2pY3qpqoqq+WVVPV9WTVXVrP+66ycgIjRZRVa1O8rkkVybZnOT6qto83qpgxfu91tqWabcm3ZXkG621TUm+0b8HRueeJNtnjM3Vh1cm2dQ/bkpy1yLVCCvRPRnszSS5s79ubmmtPZQk/d+z1yV5Z/+dv+3/7gUW1rEkt7XWfivJtiQ7+/5z3WRkhEaL69Ikh1prh1trP0+yJ8mOMdcEvN6OJPf2r+9NctUYa4Flr7X2SJIfzRieqw93JLmvdb6d5Deq6i2LUymsLHP05lx2JNnTWvu/1tqzSQ6l+7sXWECttRdba//ev341ydNJLorrJiMkNFpcFyV5ftr7yX4MGI+W5J+r6rtVdVM/9ubW2otJd2FOcuHYqoOVa64+dB2F8bulX+Zy97Ql3HoTFllVvS3Jbyf5Tlw3GSGh0eKqWcbcvg7G57LW2rvSTd3dWVWXj7sg4IRcR2G87kpySZItSV5M8tf9uN6ERVRV5yT5xyR/2lr73xMdOsuY3uSUCI0W12SSiWnv1yd5YUy1wIrXWnuhf34pyd50U+l/MDVtt39+aXwVwoo1Vx+6jsIYtdZ+0Fr7ZWvtV0n+LseXoOlNWCRVtTZdYHR/a+2f+mHXTUZGaLS4Hk+yqao2VtVZ6TYM3DfmmmBFqqpfr6pzp14n+YMkT6TryRv7w25M8uXxVAgr2lx9uC/Jh/u7wWxL8srUdHxg9GbshXJ1uutm0vXmdVV1dlVtTLfp7r8tdn2w3FVVJflCkqdba38z7SPXTUZmzbgLWElaa8eq6pYkX0uyOsndrbUnx1wWrFRvTrK3u/ZmTZK/b619taoeT/KlqvpYkv9K8qEx1gjLXlV9MckVSd5UVZNJ/iLJ7Zm9Dx9K8v50m+y+luQji14wrBBz9OYVVbUl3fKW55L8cZK01p6sqi8leSrd3Z12ttZ+OY66YZm7LMkNSQ5W1YF+7M/juskIVWuWNAIAAADwepanAQAAADBAaAQAAADAAKERAAAAAAOERgAAAAAMEBoBAAAAMEBoBAAAAMAAoREAAAAAA4RGAAAAAAz4fzfpuo7i197NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def moving_avg(arr, window=50):\n",
    "    return np.convolve(arr, np.ones(window) / window, mode='same')[window:-window]\n",
    "\n",
    "plt.figure(figsize=(20, 10))\n",
    "plt.plot(moving_avg(np.abs(DQN.q_logs_arr[:, 0] - DQN.q_logs_arr[:, 1])), label=\"Loss of Q-Values\", color=\"red\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference the routes.\n",
    "\n",
    "After learning, `DQN` can provide a method `inference` for infernecing the routes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:09:18.472805Z",
     "start_time": "2020-07-11T08:08:25.710980Z"
    }
   },
   "outputs": [],
   "source": [
    "# Reset Epsilon greedy rate.\n",
    "DQN.epsilon_greedy_rate = 0.9\n",
    "DQN.policy_sampler.inferencing_mode = True\n",
    "DQN.policy_sampler.reset_agent_pos()\n",
    "\n",
    "# Inference the routes.\n",
    "state_arr_list, q_value_arr_list = DQN.inference(\n",
    "    iter_n=500\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:09:18.548483Z",
     "start_time": "2020-07-11T08:09:18.473681Z"
    }
   },
   "outputs": [],
   "source": [
    "result_df_list = []\n",
    "for i in range(len(state_arr_list)):\n",
    "    arr_tuple = np.where(state_arr_list[i][0][0].asnumpy() == 1)\n",
    "\n",
    "    x = arr_tuple[0][0]\n",
    "    y = arr_tuple[1][0]\n",
    "    \n",
    "    q_value = q_value_arr_list[i].asnumpy()[0][0]\n",
    "    result_df_list.append((x, y, q_value))\n",
    "result_df = pd.DataFrame(result_df_list, columns=[\"x\", \"y\", \"q_value\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-11T08:09:21.956218Z",
     "start_time": "2020-07-11T08:09:18.560453Z"
    }
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from PIL import ImageFont\n",
    "from PIL import ImageDraw\n",
    "\n",
    "font = ImageFont.truetype(\"<path to your font>\", 16)\n",
    "\n",
    "img_list = []\n",
    "\n",
    "for i in range(result_df.shape[0]):\n",
    "    x = result_df.x.values[i]\n",
    "    y = result_df.y.values[i]\n",
    "    q = result_df.q_value.values[i]\n",
    "\n",
    "    now_map_arr = DQN.policy_sampler.map_arr.copy()\n",
    "    now_map_arr[x, y] = 2\n",
    "    now_map_arr = 255 * (now_map_arr - now_map_arr.min()) / (now_map_arr.max() - now_map_arr.min())\n",
    "    img = Image.fromarray(np.uint8(now_map_arr))\n",
    "    img = img.resize((400, 400))\n",
    "\n",
    "    bg_img = Image.new(\"RGB\", (420, 420), (0, 0, 0))\n",
    "    bg_img.paste(img, (10, 10))\n",
    "    draw = ImageDraw.Draw(bg_img)\n",
    "    draw.multiline_text((5, 5), 'Inferenced Q-Value: ' + str(q), fill=(255, 255, 255), font=font)\n",
    "    img_list.append(bg_img)\n",
    "\n",
    "img_list[0].save(\n",
    "    'img/DQN_agent_demo.gif',\n",
    "    save_all=True,\n",
    "    append_images=img_list[1:], \n",
    "    optimize=False, \n",
    "    duration=40, \n",
    "    loop=0\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='img/DQN_agent_demo_default.gif?1'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- Gupta, J. K., Egorov, M., & Kochenderfer, M. (2017, May). Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (pp. 66-83). Springer, Cham.\n",
    "- Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}