examples/interactive_bluefog_helloworld.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run BlueFog with Jupyter Notebook\n",
    "\n",
    "The **goal** of this ipython notebook is to show you how you can run BlueFog code in ipython notebook environment.\n",
    "\n",
    "As you know, the BlueFog framework is aimed to run decentralized algorithm over a cluster of processes or machines.\n",
    "Meanwhile, jupyter notebook is an execellent interactive frontend talking with single IPython process. The contradiction between a single frontend process and a cluster of backend processescan be completely circumvented, fortunately. The main idea is using the jupyter notebook as the driver to inform how the cluster will execute the BlueFog code. On one hand, the execution of BlueFog still runs over multiple processes so that you can enjoy the scalability and parallelism brought by BlueFog. On the other hand, you (the programmer) can interactively execute with BlueFog and monitor and control the code in the single place.\n",
    "\n",
    "## Prerequisites: \n",
    "To begin with, make sure that you have installed OpenMPI and BlueFog in your system first.\n",
    "Besides these, you also need to install [ipyparallel](https://ipyparallel.readthedocs.io/en/latest/index.html) package for managering the parallel computation.\n",
    "Usually, you can install it through `pip install jupyter ipyparallel`.\n",
    "\n",
    "## Steps to start Bluefog with Jupyter Notebook\n",
    "\n",
    "- **Step 1: Start jupyter notebook**\n",
    "\n",
    "```shell\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "- **Step 2A: Start the `ibfrun` (interactive bluefog run) in a single machine**\n",
    "\n",
    "```shell\n",
    "ibfrun start -np 4\n",
    "```\n",
    "where `np` means the number of process. In this example command, it will start 4 processes. You are free to choose any number between 1 and the number of CPU processes in you machine. \n",
    "\n",
    "- **Step 2B: Start the `ibfrun` in multiple machines**\n",
    "\n",
    "```shell\n",
    "ibfrun start -np 16 -H machine-a:4,machine-b:4,machine-c:4,machine-d:4\n",
    "```\n",
    "where `machine-a` to `machine-d` are the machine name or the address of the machine and `:4` after the machine name means on that machine launch 4 processes. Before you run this command, please make sure that all the machines is able to be login through `ssh` **without** typing the password.\n",
    "\n",
    "That's it!\n",
    "\n",
    "**Other remarks**\n",
    "- Please make sure the jupyter notebook command and the ibfrun command are under the same machine.\n",
    "- If you want to communicate over GPUs through NCCL, the number of process in each machine should not exceed the number GPUs instead of CPUs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:23.697595Z",
     "start_time": "2021-01-16T06:50:22.766611Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connect the Jupyter Notebook to the workers\n",
    "\n",
    "The first step is to let the jupyter notebook connect to the workers (cluster). To achieve that, import the ipypyarallel library since we rely on it to manage the workers, then connect it through `ipp.Client(profile=\"bluefog\")` as shown in the following cell.\n",
    "\n",
    "`rc` (remote client) class is the key that you will use to send the code from this notebook (driver) to the workers. Remote client is called as such because it talks with the workers, which may not be on the local host.\n",
    "To check if the notebook has connected to the worker succesfully, you can look the ID of rc through `rc.ids`.\n",
    "Depending on how many processes you start, the length of rc will be the same number as the number of processes you start the `ibfrun`, i.e., the number after `-np`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:24.460204Z",
     "start_time": "2021-01-16T06:50:24.258809Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipyparallel as ipp\n",
    "rc = ipp.Client(profile=\"bluefog\")\n",
    "rc.ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you saw the `ids` is an empty list, this means ipyparallel cannot connect it with the workers. You can stop the `ibfrun` by `ctrl+c`. Kill all remainning `ipcontroller` or `mpi` processes through `ps` if necessary. Then, try to rerun the `ibfrun` again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Parallel execution through `px`\n",
    "\n",
    "`px` -- parallel execution is a [magic command](https://ipython.readthedocs.io/en/stable/interactive/magics.html) in IPython environment similar as `%matplotlib` or `%%timeit`. If you are not familar with magic command, you can think it as a special command in IPython that will change the behavior of IPython interpreting the following python code. \n",
    "\n",
    "Same as other magic commands, `%px` is for one line and `%%px` is for whole cell. For example, \n",
    "`%px print(\"Hello world\")` will put `print(\"Hello world\")` to run under the `px` environment. And, if you put the `%%px` at the top of one cell, the rest of code in that cell will be run under the `px` environment.\n",
    "\n",
    "The code under the `px` env will be executed in each worker in parallel instead of running in this notebook.\n",
    "It is important to remember that each worker is one seperated IPython environment. The following is a hello-world example to work with bluefog in jupyter notebook, which is already sufficient for most  usage cases. You can find more advanced usage of `px` in the [document page](https://ipyparallel.readthedocs.io/en/latest/magics.html).\n",
    "\n",
    "\n",
    "All BlueFog code should start with `bf.init()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:27.337955Z",
     "start_time": "2021-01-16T06:50:25.579567Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] Hello, I am 0 among 4 processes\n",
      "[stdout:1] Hello, I am 1 among 4 processes\n",
      "[stdout:2] Hello, I am 3 among 4 processes\n",
      "[stdout:3] Hello, I am 2 among 4 processes\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "import bluefog.torch as bf\n",
    "\n",
    "bf.init()\n",
    "message = f\"Hello, I am {bf.rank()} among {bf.size()} processes\"\n",
    "print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see `Hello, I am {x} among {np} processes` repeated `np` time where `x` ranges from 0 to np-1. This `x` is called rank. Note the code is the same while the outputs from the cluster of processes are different. This is the typical usage of BlueFog code, which belongs to the Single Instruction, Mutliple Data [SIMD](https://en.wikipedia.org/wiki/SIMD) pattern.\n",
    "\n",
    "You can also use `%px` for one line. A typical usage of `%px` is mixing the control flow code of notebook and the execution code in the workers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:27.504324Z",
     "start_time": "2021-01-16T06:50:27.343746Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "==========================================\n",
      "[stdout:0] Hello, I am 0 among 4 processes\n",
      "[stdout:1] Hello, I am 1 among 4 processes\n",
      "[stdout:2] Hello, I am 3 among 4 processes\n",
      "[stdout:3] Hello, I am 2 among 4 processes\n",
      "==========================================\n",
      "[stdout:0] Hello, I am 0 among 4 processes\n",
      "[stdout:1] Hello, I am 1 among 4 processes\n",
      "[stdout:2] Hello, I am 3 among 4 processes\n",
      "[stdout:3] Hello, I am 2 among 4 processes\n",
      "==========================================\n"
     ]
    }
   ],
   "source": [
    "print(\"=\" * 42)\n",
    "for _ in range(2):\n",
    "    %px print(message)\n",
    "    print(\"=\" * 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because each worker is a IPython environment, it will remember the context of execution. Hence, the `message` variable is still the same `message` variable created in previous cell and `bf.init()` just needs to be called once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Moving the data between notebook and workers\n",
    "\n",
    "Only executing the BlueFog code in multiple workers through notebook is not interesting enough. Because all running results only remain in the workers. The notebook does not know those results by default. You can try to run `print(message)` in the cell without `px`, you will encounter the `NameError`, i.e., that variable is not defined under the notebook.\n",
    "\n",
    "Fortunately, we can easily extract or pass the data or objects between notebook and workers.\n",
    "The key componenet is [Direct View class](https://ipyparallel.readthedocs.io/en/latest/direct.html#) through `dview=rc[:]`. Using `dview.push()` and `dview.pull()`, they transmit the data between workers and notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:33.853374Z",
     "start_time": "2021-01-16T06:50:33.808900Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "import torch\n",
    "x = torch.FloatTensor([bf.rank()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now each worker creates a tensor `x` containing the 1x1 value equal to its corresponding rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:34.988406Z",
     "start_time": "2021-01-16T06:50:34.889322Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dview = rc[:] # A DirectView of all engines\n",
    "dview.block=True\n",
    "x_at_worker0 = dview.pull('x', targets=0)\n",
    "x_at_worker0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the code above did is that notebook pulls the value `x` of worker 0 and stores that as `x_at_worker0` in the notebook. Note the worker 0 may not be the worker that has rank 0. Hence, the value of `x_at_worker0` may not be zero. You also pull the results from all workers or part of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:35.972698Z",
     "start_time": "2021-01-16T06:50:35.947314Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([1.]), tensor([3.])]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs_at_1_and_2 = dview.pull('x', targets=[1, 2])\n",
    "xs_at_1_and_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:36.279305Z",
     "start_time": "2021-01-16T06:50:36.249560Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([0.]), tensor([1.]), tensor([3.]), tensor([2.])]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs_all = dview.pull('x')\n",
    "xs_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also push the data from the notebook to all workers through `dview.push(arg)`, where the `arg` is a dictionary that map from the variable name to variable value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:37.155838Z",
     "start_time": "2021-01-16T06:50:37.138090Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None, None, None]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Push the data into all workers\n",
    "dview.push({'seed': 12345})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:37.645339Z",
     "start_time": "2021-01-16T06:50:37.612124Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] I received seed as value:  12345\n",
      "[stdout:1] I received seed as value:  12345\n",
      "[stdout:2] I received seed as value:  12345\n",
      "[stdout:3] I received seed as value:  12345\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "print(\"I received seed as value: \", seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Working with multiple GPUs\n",
    "\n",
    "If you have multiple GPUs, it is typical that each worker is pinned to one device.\n",
    "Then, you can let the tensor wihin each worker set on different GPUs as shown in following code.\n",
    "\n",
    "*Note: If you want to pin multiple workers in one device, you cannot use NCCL as communication backend.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] tensor([0.], device='cuda:0')\n",
      "[stdout:1] tensor([1.], device='cuda:1')\n",
      "[stdout:2] tensor([3.], device='cuda:3')\n",
      "[stdout:3] tensor([2.], device='cuda:2')\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.set_device(bf.local_rank())\n",
    "    x = torch.FloatTensor([bf.rank()]).cuda()\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A simple average-consensus example\n",
    "\n",
    "Now, with the `px` magic command and the direct-view class, we are good to run a simple average-consensus example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:38.737388Z",
     "start_time": "2021-01-16T06:50:38.713095Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "\n",
    "# It is important to give different random seeds to different workers.\n",
    "torch.random.manual_seed(seed * bf.rank() + bf.rank())\n",
    "\n",
    "data_size = 10\n",
    "max_iters = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:43.089433Z",
     "start_time": "2021-01-16T06:50:39.579791Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "import bluefog.common.topology_util as topology_util\n",
    "\n",
    "# Set exponential-2 graph as topology, which is default actaully.\n",
    "assert bf.set_topology(topology_util.ExponentialTwoGraph(bf.size()))\n",
    "\n",
    "# Create a random data to run consensus\n",
    "x = torch.randn(data_size, dtype=torch.double)\n",
    "\n",
    "# Use allreduce to find the truth of average for reference\n",
    "x_bar = bf.allreduce(x, average=True)\n",
    "\n",
    "mse = []\n",
    "for ite in range(max_iters):\n",
    "    # The following code is equivalent to:\n",
    "    #  x_i = \\sum_{j\\in Neighbor(i)} a_ij x_j\n",
    "    # where a_ij is entry of the combination matrix \n",
    "    # that definied through the set_topology command.\n",
    "    x = bf.neighbor_allreduce(x, name='x')\n",
    "    mse.append(torch.norm(x-x_bar, p=2) / torch.norm(x_bar, p=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:43.158410Z",
     "start_time": "2021-01-16T06:50:43.091291Z"
    }
   },
   "outputs": [],
   "source": [
    "mse_0 = dview.pull('mse', block=True, targets=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:43.805803Z",
     "start_time": "2021-01-16T06:50:43.172042Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fd3130a40d0>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmRElEQVR4nO3deXyV5Zn/8c+VhBDCErYk7DsJBBDQiGsVlLRQabGdOhVbdVpaKtaOte10dKrdf12ddn62tEiVQW0F0aqliFLcQAtFghs7hLAlAgk7AbJf80eONAkJS3KSJznn+369nhc8d55zzvVwk/N99tvcHRERiU4xQRcgIiLBUQiIiEQxhYCISBRTCIiIRDGFgIhIFIsLuoAL0b17dx8wYEDQZYiItCpr16494O7Jdf2sVYXAgAEDyM7ODroMEZFWxcx21fczHQ4SEYliCgERkSimEBARiWIKARGRKKYQEBGJYgoBEZEophAQEYliURECRSXlfH/RBo4XlwVdiohIixIVIZC98xB//Mcubpq9ir1HTwVdjohIixEVITA+PYW5/3YpeYdP8alZK9n4wbGgSxIRaRECDQEzG2Rmj5rZM039WdekJfP0HVcA8K8Pr2LTXgWBiEiDQ8DM5ppZgZmtr9U+ycy2mFmOmd17tvdw91x3n97QGi7U8J6deO6rVxIfF8P3F21AQ2uKSLRrzJ7APGBS9QYziwVmAZOBDGCamWWY2SgzW1xrSmnEZzdYz6R23JOVxuodh1i6YX8QJYiItBgNDgF3XwEcqtU8DsgJbeGXAguAqe6+zt2n1JoKGlF3o0y7tC9pqR346YubKCmvCKoMEZHAhfucQG9gT7X5vFBbncysm5nNBsaa2X31LDPDzLLNLLuwsDAsRcbFxnD/DRnsOniSx1buDMt7ioi0RoGeGHb3g+5+h7sPdvef1rPMHHfPdPfM5OQ6x0RokGvSkpmQnsxvXsnhYFFJ2N5XRKQ1CXcI5AN9q833CbW1SN+5IYMTpeXMWZEbdCkiIoEIdwisAYaa2UAziwduBhaF+TPCZkhKB6Zc1Is/rd7N0VO6m1hEok9jLhGdD6wC0s0sz8ymu3s5cBewFNgELHT3DeEptWncce1gikrK+eM/6h19TUQkYjV4jGF3n1ZP+xJgSYMramYZvToxPj2ZuW/u4ItXDaRdfGzQJYmINJuoeGzEudw5fggHT5Ty9No9515YRCSCKASASwd04ZL+XXh4eS5lFZVBlyMi0mwUAoCZcef4weQfOcWSdXuDLkdEpNkoBEKuG5ZCr6QEFr+vEBCR6KEQCDEzJmak8sa2Qk6V6lESIhIdFALVZGWkUlxWyZs5B4IuRUSkWSgEqrlsYDc6to1j2cZ9QZciItIsFALVxMfFMH5YCq9sKqCiUmMNiEjkUwjUkpWRysETpbyz+3DQpYiINDmFQC3j05NpE2ss26gBZ0Qk8ikEaumU0IbLB3VTCIhIVFAI1CErI5XcAyfIKSgKuhQRkSalEKjDxOGpADz6Zi6VOkEsIhFMIVCHXp3b8YWrBjD/rT3c/dS7GodYRCJWgx8lHem+OyWD1E4J/OzFzew/Wsyc2y6hc2J80GWJiISV9gTqYWbcce1gHpo2lnf3HOEbC98LuiQRkbALNATMbLiZzTazZ8xsZpC11OeTo3vxrY+l8ermAlZsLQy6HBGRsGrM8JJzzazAzNbXap9kZlvMLMfM7j3be7j7Jne/A/hX4KqG1tLUbr9yAP27JfLjFzZSrvEGRCSCNGZPYB4wqXqDmcUCs4DJQAYwzcwyzGyUmS2uNaWEXvNJ4AVa8JCUbeNiuW/yMLbuL2LBGo0+JiKRo8Eh4O4rgEO1mscBOe6e6+6lwAJgqruvc/cptaaC0PsscvfJwOfq+hwzm2Fm2WaWXVgY3OGYj43owWUDu/KrZVs5VlwWWB0iIuEU7nMCvYHqm8p5obY6mdl4M3vIzB6mnj0Bd5/j7pnunpmcnBzeai+AmfHAlAwOnyxl1ms5gdUhIhJOgV4i6u6vA68HWcOFGNk7iamje/HEql3MvHawLhkVkVYv3HsC+UDfavN9Qm0RY+b4IZwsreCxlbuCLkVEpNHCHQJrgKFmNtDM4oGbgUVh/oxApffoyMThKcxbuYOTpeVBlyMi0iiNuUR0PrAKSDezPDOb7u7lwF3AUmATsNDdN4Sn1JZj5vjBHD5ZxoK3dKWQiLRuDT4n4O7T6mlfQgu+3DMcLunflXEDu/LIG7l8/vL+xMfpxmsRaZ307dVAM8cP5oOjxfzl3Yg65SEiUUYh0EDj05IZ1qMjj6/SCWIRab0UAg1kZkwd05t1+Uf54MipoMsREWkQhUAjZGVUDT7z8iYNRSkirZNCoBGGpHRgUPf2Go9YRFothUAjZWWk8o/cg3qekIi0SgqBRsrKSKWswnl9i8YaEJHWRyHQSGP7daFb+3gdEhKRVkkh0EixMcb1w1N4fXMBpeUacEZEWheFQBhkZfTgeEk5q3ccDLoUEZELohAIg6uHdCehTYwOCYlIq6MQCIN28bFcNyyFp9bs4cV1e4MuR0TkvCkEwuTHN45iRK9O3Pnk2zzyRi7uHnRJIiLnpBAIk67t43nyy5czaUQPfvzCJn764uagSxIROSeFQBgltIll1i0Xc8tl/ZizIpc1Ow8FXZKIyFkpBMIsJsa4/4bh9OiUwI8Wb6SyUoeFRKTlCjQEzGy8mb1hZrPNbHyQtYRTYnwc356Uzvt5R3le4w2ISAvWmOEl55pZgZmtr9U+ycy2mFmOmd17jrdxoAhIAPIaWktLdOOY3ozuk8QvXtqisYhFpMVqzJ7APGBS9QYziwVmAZOBDGCamWWY2SgzW1xrSgHecPfJwH8CP2hELS1OTIzxwJQM9h0rZs6K3KDLERGpU4NDwN1XALXPfI4Dctw9191LgQXAVHdf5+5Tak0F7v7hcxYOA23r+hwzm2Fm2WaWXVjYuh7SljmgKzdc1JOHl+dy6ERp0OWIiJwh3OcEegN7qs3nhdrqZGafNrOHgSeA39a1jLvPcfdMd89MTk4Oa7HN4Z6JQykur2Deyp1BlyIicoZATwy7+7Pu/hV3/6y7vx5kLU1lSEpHPpqRymMrd1JUonMDItKyhDsE8oG+1eb7hNqi2h3XDuboqTIWvLU76FJERGoIdwisAYaa2UAziwduBhaF+TNanbH9unDFoG784Y1cSsorgi5HROS0xlwiOh9YBaSbWZ6ZTXf3cuAuYCmwCVjo7hvCU2rrdueEwew/VsLz70T9jpGItCBxDX2hu0+rp30JsKTBFUWoq4d0Z2TvTsxenstNl/QlJsaCLklERI+NaC5mxhevGsiOAyd4N+9I0OWIiAAKgWZ1/bBUYmNMg8+ISIuhEGhGSYltuGxgV4WAiLQYCoFmlpWRSk5BETsOnAi6FBERhUBzy8pIBWDZxn0BVyIiohBodn26JDK8ZycdEhKRFkEhEICsjFTW7jrMwaKSoEsRkSinEAjARzNSqXR4ZXNB0KWISJRTCARgRK9O9EpK0CEhEQmcQiAAZsbEjFTe2FZIwbHioMsRkSimEAjIZy/tS4wZn/rdSrbtPx50OSISpRQCARnRK4mnZlxBaUUln/79SlZtPxh0SSIShRQCARrVJ4nn7rySHp0SuG3uatbnHw26JBGJMgqBgPXpksjTd1xBp4Q2/HDxRtw96JJEJIooBFqAzonx3JOVxls7DrF0g+4kFpHmoxBoIW6+tC9pqR34yZLNGn1MRJpNoCFgZh8xs9lm9oiZrQyylqDFxcZw/w0Z7D50ksdW7gy6HBGJEo0ZXnKumRWY2fpa7ZPMbIuZ5ZjZvWd7D3d/w93vABYDjzW0lkhxTVoyE9KT+c0rOXqkhIg0i8bsCcwDJlVvMLNYYBYwGcgApplZhpmNMrPFtaaUai+9BXiyEbVEjO/cMJyi0nLm/n1H0KWISBRocAi4+wrgUK3mcUCOu+e6eymwAJjq7uvcfUqtqQDAzPoBR929zjumzGyGmWWbWXZhYWFDy201hqR0ZPLIHjy+ahfHi8uCLkdEIly4zwn0BvZUm88LtZ3NdOB/6/uhu89x90x3z0xOTg5DiS3fzGuHcLy4nD+t3h10KSIS4QK/Osjdv+fuUX1SuLZRfZL4yNDuPPrmDorLdKWQiDSdcIdAPtC32nyfUJtcoJnXDqbweAl/fjsv6FJEJIKFOwTWAEPNbKCZxQM3A4vC/BlR4YrB3RjdJ4mHl+dSXlEZdDkiEqEac4nofGAVkG5meWY23d3LgbuApcAmYKG7bwhPqdHFzJg5fgi7D53k5U0afEZEmkZcQ1/o7tPqaV8CLGlwRXLaxOEpdElsw0vr9zJpZI+gyxGRCBT4iWGpX1xsDNcNS+XVzQWU6ZCQiDQBhUALl5WRyrHictbsqH1LhohI4ykEWrhr0rrTNi6Gv2k8YhFpAgqBFi4xPo6rh3Rn2cb9GmtARMJOIdAKTMxIJf/IKTbt1VjEIhJeCoFW4PrhKZjBMh0SEpEwUwi0AikdExjTtzPLNmnUMREJL4VAK5GVkcr6/GN8cORU0KWISARRCLQSH82oulnsZy9uprRc9wyISHgoBFqJISkd+GZWGove+4AvzHuLYxprQETCQCHQinzt+qE8eNNoVuce4qbfr2Lf0eKgSxKRVk4h0Mp85pI+PPbFceQdPsm9z74fdDki0sopBFqhq4Z05+6JQ3l9SyHLt0b+kJsi0nQUAq3U7VcOoH+3RH68eKPGGxCRBlMItFJt42K5b/JwthUUMX/NnnO/QESkDoGGgJllmNlCM/u9mX0myFpao4+NSOWygV359bKtHD2lq4VE5MI1ZmSxuWZWYGbra7VPMrMtZpZjZvee420mA79x95nAbQ2tJVqZGQ9MyeDwyVJmL98edDki0go1Zk9gHjCpeoOZxQKzqPpyzwCmhbb2R5nZ4lpTCvAEcLOZ/RLo1ohaotbI3kncMKonT6zapb0BEblgDQ4Bd18B1B7pZByQ4+657l4KLACmuvs6d59SayoITV8F7gUONHgtotzM8YMpKinnj//YFXQpItLKhPucQG+g+lnKvFBbncxsgJnNAR4HflnPMjPMLNvMsgsLdTlkXUb0SuLatGTmvrmD4rKKoMsRkVYk0BPD7r7T3We4++fc/c16lpnj7pnunpmcnNzcJbYad44fzMETpSzM1pVCInL+wh0C+UDfavN9Qm3SxMYN7MrF/Trz8PJcDUovIuct3CGwBhhqZgPNLB64GVgU5s+QOpgZM8cPIf/IKf763gdBlyMirURjLhGdD6wC0s0sz8ymu3s5cBewFNgELHT3DeEpVc7l+mEpDEpuz1O6eUxEzlNcQ1/o7tPqaV8CLGlwRdJgMTHGx0f25Hev53D4RCld2scHXZKItHB6bESEycpIpdLh1c0FQZciIq2AQiDCjOqdRGqnthqUXkTOi0IgwsTEGBOHp7JiW6HuGRCRc1IIRKCsjFROllawcrtuwhaRs1MIRKArBnejQ9s4HRISkXNSCESgtnGxXJuWzMubCqis9KDLEZEWTCEQobIyUik8XsK7eUeCLkVEWjCFQISakJ5CbIyxdP2+oEsRkRZMIRChkhLbMHF4Cn94I5cnVu0MuhwRaaEUAhHs158dw3XDUnjgLxv4yZJNOj8gImdQCESwxPg4Hr41k1sv78+cFbl85/l1QZckIi2MQiDCxcYYP5w6gi9/ZCDz39rDmp21B4MTkWimEIgCZsY9WWn06JTAD/+6UYeFROQ0hUCUSIyP49uT0lmXf5Tn3tE4PyJSRSEQRW4c05uL+iTxi6WbOVlaHnQ5ItICKASiSEyM8cCUDPYfK2H28tygyxGRFqDZQsDMBpnZo2b2zNnapGldOqArHx/Vg0feyOXIydKgyxGRgJ1XCJjZXDMrMLP1tdonmdkWM8sxs3vP9h7unuvu08/VJk3v368fysnSCh5buSvoUkQkYOe7JzAPmFS9wcxigVnAZCADmGZmGWY2yswW15pSwlq1NMqwHp24flgK81bu0LkBkSh3XiHg7iuA2heYjwNyQlvzpcACYKq7r3P3KbWmBo91aGYzzCzbzLILCwsb+jZSy8zxgzl8sowFb2lQepFo1phzAr2B6t8geaG2OplZNzObDYw1s/vqa6vN3ee4e6a7ZyYnJzeiXKkuc0BXxg3oyiNv5FJaXhl0OSISkGY7MezuB939Dncf7O4/ra9Nms/MCYP54Ggxi977IOhSRCQgjQmBfKBvtfk+oTZpJcanJTO8ZyceXr4dd91FLBKNGhMCa4ChZjbQzOKBm4FF4SlLmoOZ8fnL+7GtoIhtBUVBlyMiATjfS0TnA6uAdDPLM7Pp7l4O3AUsBTYBC919Q9OVKk0ha3gqgMYjFolSceezkLtPq6d9CbAkrBVJs0rplMCYvp3528b9fHXCkKDLEZFmpsdGCFkZqby35wj7jxUHXYqINDOFgJCVUXVI6OVNOiQkEm0UAsLQlA7075ao8wIiUUghIJgZWcNTWZlzkKISPUZCJJooBASoOiRUWlHJ8i16NIdINFEICACX9O9Cl8Q2LNu4L+hSRKQZKQQEgLjYGK4blsormwvILdSNYyLRQiEgp824ZhBtYmP49O9Xkr2z9kNjRSQSKQTktPQeHXnuzivpkhjPLY+s5oX39wZdkog0MYWA1NC/W3v+PPNKLuqdxNfmv836/KNBlyQiTUghIGfo2j6euV+4lC6J8fxw8UY9YVQkgikEpE6dEtpwT1Yab+04xEvrdcWQSKRSCEi9br60L+mpHfnpi5spKa8IuhwRaQIKAalXXGwM908Zzu5DJ5n3951BlyMiTUAhIGf1kaHJXDcshd++msOhE6VBlyMiYaYQkHO6b/IwjpeUM+/vO4IuRUTCrNlCwMwGmdmjZvZMtbbhZjbbzJ4xs5nNVYtcmKGpHfloRiqPrdqlB8yJRJjzHV5yrpkVmNn6Wu2TzGyLmeWY2b1new93z3X36bXaNrn7HcC/AlddaPHSfGaOH8zRU2XMX7076FJEJIzOd09gHjCpeoOZxQKzgMlABjDNzDLMbJSZLa41pdT3xmb2SeAFNExliza2XxeuGNSNR97M1ZVCIhHkvELA3VcAtR8mMw7ICW3hlwILgKnuvs7dp9SaCs7y3ovcfTLwubp+bmYzzCzbzLILC/WY4yDdOWEw+4+V8Nzb+UGXIiJh0phzAr2BPdXm80JtdTKzbmY2GxhrZveF2sab2UNm9jD17Am4+xx3z3T3zOTk5EaUK4119ZDujOqdxMMrcqmo1F3EIpEgrrk+yN0PAnfUansdeL25apDGMTPuuHYwX33ybVZsK2RCer1H+USklWjMnkA+0LfafJ9Qm0SwiRkptI+P1XjEIhGiMSGwBhhqZgPNLB64GVgUnrKkpWobF8u16cm8vHE/lTokJNLqne8lovOBVUC6meWZ2XR3LwfuApYCm4CF7r6h6UqVliIrI5WC4yW8r8dMi7R653VOwN2n1dO+BF3aGXUmpKcQG2Ms27iPMX07B12OiDSCHhshF6xzYjzjBnTVeQGRCKAQkAbJykhl6/4idh08EXQpItIICgFpkKyMVADtDYi0cgoBaZC+XRMZ1qMjf1MIiLRqCgFpsKyMVLJ3HtI4AyKtmEJAGmzSyB5UOnz9qXf1iGmRVkohIA02olcSP/v0KP6ec4DPPryKgmPFQZckIhdIISCNcvO4fjxyeyY7DpzgU79byZ5DJ4MuSUQugEJAGm1CegpPzbiCY6fKuP/59bjrcRIirYVCQMJiVJ8k7p44lOVbC3l1c73DR4hIC6MQkLC5/coBDE5uz48Wb9ToYyKthEJAwqZNbAzf+8QIdh48ydw3dwZdjoicB4WAhNU1aclMHJ7Kb17dxn5dLSTS4ikEJOwemDKc8grnf17eFnQpInIOCgEJu/7d2vPZS/vydPYe8g7rklGRlkwhIE1i5vjBxJgx67XtQZciImfRbCFgZoPM7FEze6Za23gze8PMZpvZ+OaqRZper87ttDcg0gqc7/CSc82swMzW12qfZGZbzCzHzO4923u4e667T6/dDBQBCUDehRQuLZ/2BkRavvPdE5gHTKreYGaxwCxgMpABTDOzDDMbZWaLa00p9bzvG+4+GfhP4AcNWwVpqbQ3INLynVcIuPsK4FCt5nFATmgLvxRYAEx193XuPqXWVOctpO5eGfrrYaBtXcuY2Qwzyzaz7MLCwvNaKWk57pxQtTfw6Js7gi5FROrQmHMCvYE91ebzQm11MrNuZjYbGGtm94XaPm1mDwNPAL+t63XuPsfdM909Mzk5uRHlShB6JrXj+uEp/PW9DyivqDz3C0SkWcU11we5+0HgjlptzwLPNlcNEowbx/bmxfX7eDPnAOPT6zsyKCJBaMyeQD7Qt9p8n1CbSA3j05NJateG59/Rfw+RlqYxIbAGGGpmA80sHrgZWBSesiSStI2L5YaLerJ0w35OaAQykRblfC8RnQ+sAtLNLM/Mprt7OXAXsBTYBCx09w1NV6q0Zp8a25tTZRUs08D0Ii3KeZ0TcPdp9bQvAZaEtSKJSJf060KfLu147p18bhxb7/UDItLM9NgIaRYxMcbUMb14Y1shhcdLzvh5WUWlxiAQCUCzXR0kcuOY3sx6bTu/eGkz/3bVAIb36ERhUQlPrNrFk2/tpqi4nNF9kxg3sCuTR/ZkZO+ket/reHEZK7YeoLyy6rLTpHZtuDYtGTOrc3l3Z8W2Axw5WQpAXEwM16R1p2NCm3o/Y9PeY2zdf/z0/Og+nRnQvX29yx8sKuHv2w+eHl4ztVMClw/qVu/y5RWVvLalkJOlVedJ2sbFMj49mYQ2sfW+5p3dh9ldbRznywZ2o0dSQr3Lf3DkFGt2/vMWnwHd2jO6b+d6lz9VWsHrWwooDV3O2z4+jgnDUoiNqf/fdVXuwdPBHmPGVUO607V9fL2fkVtYxLr8o6fnh/XoRHqPjvUuf/RUGW9sK6SisurftUtiPB8Z2r3evq6sdJZvK+TYqTKgapyLa9KS6dC2/q+7DR8cJaeg6PT82L5d6Nctsd7lC4+XsCr3n33dq3M7Lh3Qtd7lS8sreX1LAafKqjZ02sbFMmFYMm3j6u/rtbsO17jJsn+39ow5S981lEJAms3Q1I58cnQvnl6bx9Nr8+jWPp6jp8qocOf6YakMSm7P6h2HmL08l9nLc/neJzK49fL+Z/yyby8s4suPZZN74ESN9ikX9eSXnxlNu/iav1inSiv41jPv8cL7e2u0D0puzyO3ZTIouUONdnfniX/s4gd/3Xj6iwcgoU0M/33TGG64qOcZ6/beniPMeCKb/cdq7uXcfkV/7p+SQZvYmjvdR06W8tUn3+bvOQdrtI/u25k5t15CaqeaX+yVlc7/vLKNh16p+Xjuzolt+N0tF3PlkO5n1LR8ayF3Pfk2x4trnoz/ZlYad1035Ix/1w+OnGLGE9mszz9Wo/3atGQemjaWpHY1A7O0vJLv/3UDT67eXaO9Z1ICf7gts84Q/8u7+Xz7mfcpKf/nPSNxMcaPbhzJtHH9zlh+2/7jfOnxbHYdrHnH+Y1jevGzf7nojMA8UVLONxe+x0sb9tVoH5rSgUduz6R/t5oh7u7M/ftO/t8LG6nW1STGx/Lrz47hYyN6nFHT2l2H+coTazlQVLOvv3T1QO6dPIy4Wn19sKiEO//0Nqt31Lzf9uJ+nZl96yWkdDyzrx/82xZ+93rNx61MG9e3SULAWtOg4JmZmZ6dnR10GdJIBceKeTPnAG/mHKBrYjy3XtG/xi/n0ZNlfGPhu7yyuYBp4/ryg0+OJD6u6hfrtS0F/Pv8d2gTG8ODN110+nVLN+zjl0u3MKJXJ+bcmkmvzu0AyD9yihmPZ7Nx7zH+42Ppp3+pdx08wbeefp+yikp+e8vFXJtWdSNiaXkl31u0nvlv7WHi8BS+PWkYsTFGcVkF3/3LBtbuOszXrhvCPRPTiAltHT//Tj7f/vP7pHRsy4M3jSa5Y9XN7/NX7+aRN3dw5eBuzLrlYrqEto637j/Olx/PZu+RYr7/yRFcNqhqC3J9/lHue3YdHdrGMee2zNO/8CdKyvnGwndZumE/N13Sh69cOxgzOHaqjG8/8z65B07w3SkZ3HZFVWC6O4++uYOfLNlEWmpHfv4vF9EhIQ53mPVaDs+9k88NF/Xkl5+5iMT4qu3AtbsO8ZUn3qa4rIKf/8tFDOtZtWW+MucAP/jrRvp1TeQPt2cyOBSYB4tKmPnHt3lr5yHuuHYwN2X2AWD/sWK+tfA9Dp0s5ZefGc0nRvcCoKLS+eXSLcxevp1xA7vy/U+MoG2bGMornJ8s2cTyrYXcdkV/HqgWmK9s2s/dC94loU0sD950EX27Vm2Zv/D+Xn61bCuj+yTx8K2Zp/eE9hw6yZcfz2br/uPcO3kY1w9PBSC38AT/8cx7uMPvPncxV4UCs6S8gu88t55n1ubxsRGpfOuj6cTEGKdKK/jO8+t5b88RvpGVxteqBebC7D3c/9x6enZO4MGbRtO1fTzu8Md/7GLeyp1ck5bMb24eS1JiVWBu2nuMLz2WzYGiEn5040gu6d8FqNpo+K/n1tElMZ45t2Yyqk9VYB4vLuPrCz78v9+P6VcP5MOs7pgQd0ZgnC8zW+vumXX+TCEgLVFFpfPfoa2hHp0SQl9izo4DJ0jv0Yk/3HYJfbrU3F1/dfN+7p7/Lhj0CG1J7ztaNbrZQ9PGMmFYzRvVqn9pDErugAHHi8vZd6yYr04YzDez0k9/0UPVl8Z3n9/AU9l76NOlHe3axFLpzvbCE1w2sCu///wlZxwG+fPaPO57dh3t28bSvUNVOOQdPkWHhDhmf/6S018KH9q87xhffjyb/UdL6B86HHH4ZBmHTpRw/w0ZfOGqATW24I8Xl3HPU+/y8qYCBnZvT1yMUVpRya6DJ5k8sgcP3jSa9tUOg7g7c1bk8rOXNtO9Q1s6h7budx48Qe/O7Xjk9kyGpNQ8NPPWjkPM/ONaTpVV0DsUroVFJZwqreAXn7mIqWNqnugvPF7CzD+uJXvXYQYntyfGjJOlFeQfOcXnLuvH9z4x4nSof9jXP39pM3NW5NIzKYEObeNwqvb4RvZK4uFbLzkd6h/624Z93PPUu8TG2Om9pr1Hi4kx+O0tF3NNWs2nC+w+eJIvPb6G7YUnGNi9PQYcKy5j/7ES7r5+KHdfP7RGXxeXVfBfz67j2Xfy6du1HQlxsVS4k1t4gquHdOe3t4ylc2LNvl7w1m4e+Mt6Oia0oVvo/8GewydJateGObdmnnEYbn3+UWY8ns2BE6X07/phX5dy+GQZ3/9EBp+vYy+4oRQC0mq9uG4vi9ftrXreLNCrcwL3ZKWd3oKtLafgOL97bTvFoZPMCXGx3DlhCENSOtS5/MnScn69bCv5R04BYBifGN2TSSPPPOQDVV+iC7P3sHzrP59jNTSlI3ddN+SMQz4femf3Yeat3ElZ6Dh7x7Zt+HrWUHomtatz+UMnSvnVsi0cOlF1/iLGjGnj+p3egq2tstJ55M1c3t1z5HTbxf268MWrBtb4YqtuxdZCFmbvoTL0+9+9Q1u+mZV+egu2trzDJ3nolW0Uhe7zaBMbw5euHnR6C7a20vJKfvvqNnIK/3mcfUJ6Cjdl9q1zeYDF73/AknX/PGTXt2siX78+7YzDex/asu84s5dvP31BQbs2cdx13RAG1nPepqikqq/3Hv1nX08d04uP1nHIB6r6+k+rd7Ny+4HTbcN7dGLm+MFnHPL50Npdh3hs5a4a56rumZhGSqe6t+APFJXwq2VbT5+rio2J4XOX9TvruaSGUAiIiESxs4WALhEVEYliCgERkSimEBARiWIKARGRKKYQEBGJYgoBEZEophAQEYliCgERkSjWqm4WM7NCYFcDX94dOHDOpSJPNK53NK4zROd6R+M6w4Wvd393T67rB60qBBrDzLLru2MukkXjekfjOkN0rnc0rjOEd711OEhEJIopBEREolg0hcCcoAsISDSudzSuM0TnekfjOkMY1ztqzgmIiMiZomlPQEREalEIiIhEsagIATObZGZbzCzHzO4Nup6mYGZ9zew1M9toZhvM7O5Qe1czW2Zm20J/djnXe7VGZhZrZu+Y2eLQ/EAzWx3q86fMLP5c79GamFlnM3vGzDab2SYzuyIa+trM7gn9/15vZvPNLCES+9rM5ppZgZmtr9ZWZ/9alYdC6/++mV18IZ8V8SFgZrHALGAykAFMM7OMYKtqEuXAN909A7gc+GpoPe8FXnH3ocAroflIdDewqdr8z4Ffu/sQ4DAwPZCqms7/B15y92HAaKrWPaL72sx6A/8OZLr7SCAWuJnI7Ot5wKRabfX172RgaGiaAfz+Qj4o4kMAGAfkuHuuu5cCC4CpAdcUdu6+193fDv39OFVfCr2pWtfHQos9BtwYSIFNyMz6ADcAj4TmDbgOeCa0SEStt5klAdcAjwK4e6m7HyEK+hqIA9qZWRyQCOwlAvva3VcAh2o119e/U4HHvco/gM5mVvcg2XWIhhDoDeypNp8XaotYZjYAGAusBlLd/cPRu/cBqUHV1YT+B/g2UBma7wYccffy0Hyk9flAoBD439AhsEfMrD0R3tfung88COym6sv/KLCWyO7r6urr30Z9x0VDCEQVM+sA/Bn4ursfq/4zr7oeOKKuCTazKUCBu68NupZmFAdcDPze3ccCJ6h16CdC+7oLVVu9A4FeQHvOPGQSFcLZv9EQAvlA32rzfUJtEcfM2lAVAH9y92dDzfs/3DUM/VkQVH1N5Crgk2a2k6pDfddRdby8c+iQAURen+cBee6+OjT/DFWhEOl9PRHY4e6F7l4GPEtV/0dyX1dXX/826jsuGkJgDTA0dAVBPFUnkhYFXFPYhY6DPwpscvdfVfvRIuD20N9vB/7S3LU1JXe/z937uPsAqvr2VXf/HPAa8JnQYhG13u6+D9hjZumhpuuBjUR4X1N1GOhyM0sM/X//cL0jtq9rqa9/FwG3ha4Suhw4Wu2w0bm5e8RPwMeBrcB24DtB19NE63g1VbuH7wPvhqaPU3V8/BVgG/Ay0DXoWpvw32A8sDj090HAW0AO8DTQNuj6wryuY4DsUH8/D3SJhr4GfgBsBtYDTwBtI7GvgflUnfcoo2rPb3p9/QsYVVdAbgfWUXX11Hl/lh4bISISxaLhcJCIiNRDISAiEsUUAiIiUUwhICISxRQCIiJRTCEgIhLFFAIiIlHs/wAnaTezsBozDQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogy(mse_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also plot the multiple figures from multiple works by adding `--targets 1:3` after `px`. In this case, worker 1 and worker 2 will plot the figures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:50:44.710530Z",
     "start_time": "2021-01-16T06:50:43.807694Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[output:1]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmYUlEQVR4nO3deXxV9Z3/8dcnO2EnJGEJS9gJIgop7oobglK1reOIVjtTWlxqFzu/trYdZ2o703XaTm1RpNVabQu1VCtSLChKYRSV4AZhDWELSxLCEiBk//z+yAVDSBCSm5zk3vfz8bgPON+ce+/n8CX3fb7nnHu+5u6IiEh0igm6ABERCY5CQEQkiikERESimEJARCSKKQRERKJYXNAFnI3evXv74MGDgy5DRKRDWb169T53T23sZx0qBAYPHkxOTk7QZYiIdChmtr2pn+lwkIhIFFMIiIhEMYWAiEgUUwiIiEQxhYCISBRTCIiIRDGFgIhIFIuKECgsLec7C3I5VlkTdCkiIu1KVITAjv1lPPXGNmb/Y0vQpYiItCtREQIfG9yLaef2ZfY/trDr4LGgyxERaTeiIgQAvnn9aAB++NKGgCsREWk/oiYE+vfoxD1XDOXF93fz9tb9QZcjItIuRE0IANxzxVD6dk/i4RdzqanV3MoiIlEVAp0SYnlw6ihyd5ey4P1dQZcjIhK4QEPAzIaY2RNmNr+t3vPj5/ZjZHpXHn1tC7UaDYhIlGt2CJjZk2ZWZGZrG7RPMbONZpZnZg+e7jXcPd/dZzS3huaIiTHuu3Iom4uOsGRdYVu+tYhIu9OSkcBTwJT6DWYWC8wCpgJZwHQzyzKzsWa2sMEjrQXv3SI3jO3LoJRkHl2Wh7tGAyISvZodAu6+HGh4mc1EIC+0h18JzANucvc17j6twaPoTN7HzGaaWY6Z5RQXFze33JPExcZw7xVD+aDgECs27wvLa4qIdEThPifQH9hZb7kg1NYoM0sxs9nA+Wb2zcbWcfc57p7t7tmpqY1Okdksnxjfnz7dkvjVa3lhe00RkY4m0BPD7l7i7ve4+1B3/0FbvndiXCwzLx/C21v3s3q7vjcgItEp3CGwCxhQbzkj1NYu3TZxAJ3iY3nhvd1BlyIiEohwh8AqYLiZZZpZAnAbsCDM7xE2yQlxXD6iN0tyC3W5qIhEpZZcIjoXWAmMNLMCM5vh7tXA/cBiYD3wrLvnhqfU1nHdmD7sLS1nza5DQZciItLm4pr7RHef3kT7ImBRsytqY1eNSiM2xlicu5dxA3oEXY6ISJuKqttGNKZHcgIXZPbSF8dEJCpFfQhA3SGhvKIjbCk+EnQpIiJtSiEAXJuVDsCSXI0GRCS6KASAfj06MbZ/d5as2xt0KSIibUohEHLdmHTe3XGQwtLyoEsREWkzCoGQyWP6APCrV/P0nQERiRoKgZDhaV2466JBPPPmdr44913Kq2qCLklEpNU1+3sCkcbMePjGMQzomcz3X1rP7kPH+M1d2aR0SQy6NBGRVqORQD1mxucvH8Jjd0xg3e5Svj7/g6BLEhFpVQqBRkw5pw8PXDuCpRuK+Mem8MxhICLSHikEmvCvlwxmUEoy31u4jqqa2qDLERFpFQqBJiTGxfLt60eTV3SEP7y5PehyRERahULgNK7NSufSYb352cub2H+0MuhyRETCTiFwGmbGQ9OyOFpZwyNLNwddjohI2CkEPsLIPl355Pn9mfv2DooPVwRdjohIWAUaAmY22sxmm9l8M7s3yFpO595JQ6msqeXJ17cGXYqISFi1ZGaxJ82syMzWNmifYmYbzSzPzB483Wu4+3p3vwe4FbikubW0tiGpXbh+bF+eWbmdQ2VVQZcjIhI2LRkJPAVMqd9gZrHALGAqkAVMN7MsMxtrZgsbPNJCz7kR+BvtfDayL0waxpGKap5euS3oUkREwqbZIeDuy4H9DZonAnnunu/ulcA84CZ3X+Pu0xo8ikKvs8DdpwJ3NPY+ZjbTzHLMLKe4OLgvbmX168bVo9J48vWtHK2oDqwOEZFwCvc5gf7AznrLBaG2RpnZJDN7xMwep4mRgLvPcfdsd89OTU0Nb7Vn6b4rh3GgrIq5b+8ItA4RkXAJ9AZy7r4MWBZkDWdjwqCeTBjUk/mrC/jcZUOCLkdEpMXCPRLYBQyot5wRaosY14/ty4a9h9lecjToUkREWizcIbAKGG5mmWaWANwGLAjzewRqsuYjFpEI0pJLROcCK4GRZlZgZjPcvRq4H1gMrAeedffc8JTaPgzolUxW326aj1hEIkKzzwm4+/Qm2hfRzi/3bKnJY9L5xdLNFB+uILWrJp0RkY5Lt41ohuvG9MEdlq7XISER6dgUAs0wqk9XBvTqxOJcHRISkY5NIdAMZsbkrD68nlfCEX1xTEQ6MIVAM103pg+VNbX8Y6OmnxSRjksh0EwTBvWkV+cE5q3aoeknRaTDUgg0U2yMcd+koazYvI/PPrWK0nLdXVREOh6FQAt87rIh/PiWc1m5pYRPPfoGO/eXBV2SiMhZUQi00K3ZA3j6sxMpLC3n00+8RXlVTdAliYicMYVAGFw8rDez7hjP9pIyzT4mIh2KQiBMLhueyjWj0/nVq3kUlZYHXY6IyBlRCITRv98wmqqaWn68eGPQpYiInBGFQBgN7t2Zz16ayfzVBby/82DQ5YiIfCSFQJjdf+UwendJ5HsL1+HuQZcjInJaCoEw65oUz1euGU7O9gO8saUk6HJERE5LIdAKbpmQQVrXRH71al7QpYiInFagIRCaaH6Fmc02s0lB1hJOSfGxzLx8CCvzS1i9/UDQ5YiINKklM4s9aWZFZra2QfsUM9toZnlm9uBHvIwDR4AkoKC5tbRH0ycOpEdyPI++ptGAiLRfLRkJPAVMqd9gZrHALGAqkAVMN7MsMxtrZgsbPNKAFe4+FfgG8HALaml3OifG8dlLMlm6oYh1u0uDLkdEpFHNDgF3Xw7sb9A8Echz93x3rwTmATe5+xp3n9bgUeTux2+/eQBodJ5GM5tpZjlmllNc3LFu2/yZiwbTJTGOWcs0GhCR9inc5wT6AzvrLReE2hplZp80s8eBZ4BfNbaOu89x92x3z05NTQ1rsa2te3I8t18wkJfW7KHkSEXQ5YiInCLQE8Pu/py73+3u/+zuy4KspbXcOK4ftQ5LNxQFXYqIyCnCHQK7gAH1ljNCbVFrTL9u9OuexJJcTUovIu1PuENgFTDczDLNLAG4DVgQ5vfoUMyMyWP6sGJzMWWVmo9YRNqXllwiOhdYCYw0swIzm+Hu1cD9wGJgPfCsu+eGp9SOa/KYdCqqa1m+qWOd2BaRyBfX3Ce6+/Qm2hcBi5pdUQSaOLgXPZLjWZJbyJRz+gZdjojICbptRBuIi43h6lHpvLK+UJPSi0i7ohBoI5PHpFNaXs3bWxt+tUJEJDgKgTZy+fBUkuJjWJK7N+hSREROUAi0kU4JsVw+PJXFuYUcq9Rk9CLSPigE2tD0iQMpPFzObXNWUnRY8xCLSPAUAm3oylFpzLkzm02FR/jErDfYuPdw0CWJSJRTCLSxa7PSefbui6iqqeWWx95g5/6yoEsSkSimEAjA2IzuzL/nYqpqa/nhSxuCLkdEophCICADU5K594ph/G3NHt7M11zEIhIMhUCAZl4+hP49OvHdF9dRU+tBlyMiUUghEKBOCbE8OHUU6/aU8mzOzo9+gohImCkEAjbt3L58bHBPfrJ4I6XlVUGXIyJRRiEQMDPjoWlZ7D9ayTMrtwddjohEGYVAO3BuRg8mjUzlif/bqjkHRKRNKQTaifuvHMb+o5XMe1vnBkSk7SgE2onswb2YmNmLOcvzqajWvYVEpG0EGgJmdpmZzTaz35jZG0HW0h7cf+Uw9paW8/w7UT0ts4i0oZZML/mkmRWZ2doG7VPMbKOZ5ZnZg6d7DXdf4e73AAuB3zW3lkhx2fDenJvRncf+sYVqTT4jIm2gJSOBp4Ap9RvMLBaYBUwFsoDpZpZlZmPNbGGDR1q9p94O/LEFtUQEM+O+SUPZXlLGaxs1H7GItL6WzDG83MwGN2ieCOS5ez6Amc0DbnL3HwDTGnsdMxsIHHL3Rm+paWYzgZkAAwcObG65HcbVo9PplhTH4ty9XJuVHnQ5IhLhwn1OoD9Q//KWglDb6cwAftvUD919jrtnu3t2ampqGEps3+JjY7h6dDpL1xfqkJCItLrArw5y9/9096g/KVzf5Kx0DpRVsWrbgaBLEZEIF+4Q2AUMqLecEWqTs3D5iFQS4mJYsk7zEYtI6wp3CKwChptZppklALcBC8L8HhGvc2Iclw/vzZLcQtx1d1ERaT0tuUR0LrASGGlmBWY2w92rgfuBxcB64Fl3zw1PqdFlclYfdh08xro9pUGXIiIRrCVXB01von0RsKjZFQkAV49OI8ZgcW4hY/p1D7ocEYlQgZ8YlsaldEkke1AvluTqvICItB6FQDs2eUw6G/YeZtu+o0GXIiIRSiHQjk05pw8JsTF87ukcdu4vC7ocEYlACoF2LKNnMk/PmEjx4QpunvU6q7frewMiEl4KgXbuwiEpPHffxXRJimP6r99k5ZaSoEsSkQiiEOgAhqZ24fn7LqFf9yS+/dc1VOl2EiISJgqBDqJX5wQempZFfvFRntZcxCISJgqBDuSqUWlcPiKV/31lEyVHKoIuR0QigEKgAzEzHrphNGWVNfzs5U1BlyMiEUAh0MEMT+/KnRcOYu7bO1i3W7eUEJGWUQh0QA9cM4KuSfH8/BWNBkSkZRQCHVD35Hj+9ZLBvLyukA17NRoQkeZTCHRQ/3LxYDonxPLYsi1BlyIiHZhCoIPqkZzApy8cxIvv79a9hUSk2RQCHdiMSzOJi43h8eUaDYhI8wQaAmaWZWbPmtljZnZLkLV0RGndkvjn7AHMX13AnkPHgi5HRDqglsws9qSZFZnZ2gbtU8xso5nlmdmDH/EyU4Ffuvu9wF3NrSWazbx8CO7w1Bvbgi5FRDqglowEngKm1G8ws1hgFnUf7lnA9NDe/lgzW9jgkQY8A9xmZj8BUlpQS9Qa0CuZS4f35qU1ezUfsYictWaHgLsvB/Y3aJ4I5Ll7vrtXAvOAm9x9jbtPa/AoCj2+ADwI7GvsfcxsppnlmFlOcXFxc8uNaJOz+rBjfxkbCw8HXYqIdDDhPifQH9hZb7kg1NYoMxtsZnOAp4GfNLaOu89x92x3z05NTQ1rsZHimqw0zGDx2sKgSxGRDibQE8Puvs3dZ7r7He7+f0HW0pGldU1i/MCeLFmn+YhF5OyEOwR2AQPqLWeE2qSVXTcmndzdpRQc0DSUInLmwh0Cq4DhZpZpZgnAbcCCML+HNOLarD4ALMnVISEROXMtuUR0LrASGGlmBWY2w92rgfuBxcB64Fl3zw1PqXI6mb07MyK9iw4JichZiWvuE919ehPti4BFza5Imu26MX2Y9Voe+49W0qtzQtDliEgHoNtGRJDJWX2odVi6XoeEROTMKAQiyDn9u5HRsxM/e3kT6/foFtMi8tEUAhHEzHj8zgnUuvNPs1eybGNR0CWJSDunEIgwY/p1569fuISBvZKZ8bsc/rK6IOiSRKQdUwhEoL7dO/Hney7iY4N78tALayksLQ+6JBFppxQCEapzYhw//tQ4qmucH/19Q9DliEg7pRCIYANTkplxWSbPvbOLd3ccCLocEWmHFAIR7gtXDiO1ayIPv7iO2lrdalpETqYQiHBdEuP4xpRRvLfzIH99T7dxEpGTKQSiwCfP78+5Gd35n8UbqayuDbocEWlHFAJRICbGeOCaEew+VK7RgIicRCEQJSaNTGVMv248tmwLNTo3ICIhCoEoYWZ84cphbN13lJfW7gm6HBFpJxQCUeS6MX0YktqZWa9t0aT0IgIoBKJKbIxx36RhrN9Tymu6r5CI0IYhYGZDzOwJM5t/ujZpXTed14/+PTrx2LItQZciIu3AGYWAmT1pZkVmtrZB+xQz22hmeWb24Olew93z3X3GR7VJ64qPjeH2CwayatsB9hw6FnQ5IhKwMx0JPAVMqd9gZrHALGAqkAVMN7MsMxtrZgsbPNLCWrW0yHVj6uYjfnmdJp8RiXZnFALuvhzY36B5IpAX2puvBOYBN7n7Gnef1uChA9DtyLC0LgxJ7axJ6UWkRecE+gM76y0XhNoaZWYpZjYbON/MvtlUWyPPm2lmOWaWU1xc3IJypb7rxvThzfwSDpVVBV2KiASozU4Mu3uJu9/j7kPd/QdNtTXyvDnunu3u2ampqW1VbsSbnJVOda3z6kaNBkSiWUtCYBcwoN5yRqhNOoBxGT1I75bI4rUKAZFo1pIQWAUMN7NMM0sAbgMWhKcsaW0xMca1Wen8Y1Mx5VU1QZcjIgE500tE5wIrgZFmVmBmM9y9GrgfWAysB55199zWK1XCbXJWH45V1fB/m/cFXYqIBCTuTFZy9+lNtC8CFoW1ImkzFw5JoWtSHItz93JNVnrQ5YhIAHTbiCiWEBfD1aPSeGntXnK2NbwCWESigUIgyv3b5JGkdk3k9l+/xQuaa0Ak6igEotyAXsk8d+/FnDegB1+e9x6zXssLuiQRaUMKAaFn5wSe+dxEbhzXj58s3sjKLSVBlyQibUQhIAAkxsXy41vOpX+PTjz8Yq5mHxOJEgoBOSEpPpZvXT+aDXsPM2/VjqDLEZE2oBCQk1w/tg8TM3vx0yWbOHRM9xUSiXQKATmJmfEf07I4UFbJI0s3B12OiLQyhYCc4pz+3bntYwP43Rvb2FFSFnQ5ItKKFALSqC9fPYIYMx5frmkoRSKZQkAa1ad7Ep+akMGfcwooLC0PuhwRaSUKAWnSvVcMpbq2lt+syA+6FBFpJQoBadLAlGRuHNePP7y1gwNHK4MuR0RagUJATuu+K4dRVlnDb9/YFnQpItIKFAJyWiPSuzI5K52nXt/KkYrqoMsRkTBTCMhH+txlQygtr+bVDUVBlyIiYdZmIWBmQ8zsCTObX69ttJnNNrP5ZnZvW9UiZ2fCoJ707pLIkty9QZciImF2ptNLPmlmRWa2tkH7FDPbaGZ5Zvbg6V7D3fPdfUaDtvXufg9wK3DJ2RYvbSM2xrg2K41lG4upqNZ8xCKR5ExHAk8BU+o3mFksMAuYCmQB080sy8zGmtnCBo+0pl7YzG4E/oamqWzXJmf14UhFNW/oNtMiEeWMQsDdlwMN5x+cCOSF9vArgXnATe6+xt2nNXg0eTDZ3Re4+1TgjsZ+bmYzzSzHzHKKi4vPbKsk7C4elkLnhFgdEhKJMC05J9Af2FlvuSDU1igzSzGz2cD5ZvbNUNskM3vEzB6niZGAu89x92x3z05NTW1BudISiXGxTBqVxsvrCjXXgEgEiWurN3L3EuCeBm3LgGVtVYO0zOSsdP72wR7e23mACYN6BV2OiIRBS0YCu4AB9ZYzQm0Soa4clUZ8rLE4tzDoUkQkTFoSAquA4WaWaWYJwG3AgvCUJe1Rt6R4Lhram8W5e3HXISGRSHCml4jOBVYCI82swMxmuHs1cD+wGFgPPOvuua1XqrQH141JZ3tJGUvWaTQgEgnO9Oqg6e7e193j3T3D3Z8ItS9y9xHuPtTd/7t1S5X24OPj+jGmXzfu/f1qfqf7CYl0eLpthJyVbknxPHv3RVw1Ko3/XJDLdxbk6mohkQ5MISBnrXNiHI/fmc1nL8nkqTe28dvXtwZdkog0k0JAmiU2xviPj2dxxYhUfrF0M/uOVARdkog0g0JAWuShaaMpq6zhp0s2BV2KiDSDQkBaZFhaV+66aBB/WrWDdbtLgy5HRM6SQkBa7CtXj6B7p3i+uzBX3x8Q6WAUAtJi3ZPj+eq1I3gzfz8v6/sDIh2KQkDCYvrEgQxKSWbWsi0aDYh0IAoBCYu42BhmXj6E93ceZGW+5hwQ6SgUAhI2nxqfQe8uiTy2bEvQpYjIGVIISNgkxccy49JMVmzex5qCQ0GXIyJnQCEgYfXpCwfSNSmO2f/QaECkI1AISFh1TYrnzgsHsWjtHrbuOxp0OSLyERQCEnb/ekkmcTHGvLd3BF2KiHwEhYCEXWrXRC4ckqLvDIh0AG0WAmY2xMyeMLP59dommdkKM5ttZpPaqhZpfddmpZO/7yhbio8EXYqInMaZziz2pJkVmdnaBu1TzGyjmeWZ2YOnew13z3f3GQ2bgSNAElBwNoVL+3bN6HQAjQZE2rkzHQk8BUyp32BmscAsYCqQBUw3sywzG2tmCxs80pp43RXuPhX4BvBw8zZB2qN+PToxpl83XlEIiLRrcWeykrsvN7PBDZonAnnung9gZvOAm9z9B8C0M3zd2tBfDwCJja1jZjOBmQADBw48k5eVduLarPQTcw307tJo94pIwFpyTqA/sLPeckGorVFmlmJms4HzzeybobZPmtnjwDPArxp7nrvPcfdsd89OTU1tQbnS1q4ZnY47vLqhKOhSRKQJZzQSCAd3LwHuadD2HPBcW9UgbWtMv270657Ey+sKuTV7QNDliEgjWjIS2AXU/83OCLWJAGBmXJOVzorNxZRX1QRdjog0oiUhsAoYbmaZZpYA3AYsCE9ZEimuzUqnvKqWFZv3BV2KiDTiTC8RnQusBEaaWYGZzXD3auB+YDGwHnjW3XNbr1TpiC7ITKF3lwQe+NN7/Pb1rVTX1H70k0SkzVhHmgAkOzvbc3Jygi5DztK2fUd56IW1rNi8jzH9uvHTW8cxqk+3oMsSiRpmttrdsxv7mW4bIa1ucO/OPP3ZiTx6x3iKDlfwmSffZt+RiqDLEhHa8OogiW5mxvVj+zI4pTOfePR1vjT3XZ6ZcQGxMXbKursPHiN3d+mJ5UEpyYxI79rka1dU17BySwlVNXWj2s6JsVyYmUJMI6993Hs7D1J8uC6IYgwmZvaia1J8k+sXHChj/Z7DJ5ZHpndlYEpyk+sfrajmra0lHD/61atzPBMG9WpyfXfnra37OVxeDUBcrHHRkBSS4mObfE5e0ZGT7tR6bkZ30rslNbn+wbJKcrYd4PjYv2/3JM7p373J9WtqnZVbSjgWOqmfFB/DRUNSiIttet8xd/chdh8sB8CACYN60rNzQpPrF5WW8369uScyeyczLK3pvi6vqmFlfgnVob7ukhjHhUN6YdZ0X7+z4wAlRyoBiI2BiZkpdEls+qNv5/4yNuz9sK9H9enKgF5N9/WRimreyi+hNvQPm9IlgfEDeza5vrvzZv5+jlTU9XV8rHHR0BQS45ru65VbSujfo9Np/881l0JA2lRWv2587+Zz+Pr8D/jZyxv52nWjTvysqLScWa/l8ce3d5z4QIe6D+lvXT+aGZdmnvLLXlhazj2/X827Ow6e1H7dmHR+eut5p/yyV9XU8v1F6/nt69tOah+Uksyv78puNGwWrdnD//vz+5RVfniFU2JcDD/81Fg+cX7GKevnFx/h80/nsKX45Ftp337BQL7z8TEkxJ38IXqssoZv/OUDFry/+6T2c/p34/E7s+nfo9NJ7e7O79/czsMvrqO69sN/p25JcTwy/XwmjTz1C/rv7zzI3c+sZm9p+UntX7xqGA9cM+KUwDxYVskX5757ygn9S4al8Mvp4+nV4IO9ttb5xdLN/GLp5pPa07slMvvTEzi/kQ/F5ZuK+eLcdzl0rOpEW1yM8dC0LO66aNApfb374DHu+f1qPmgwYdEN5/blJ7ecS3LCyX1dWV3Ldxfm8vs3T76b7ZDUzsy5M5thaV1OqemF93bxjb98QHnVh+eukuJj+Mkt4/j4uH6nrJ9XdJjPP736lNum33XRIB6alkV8g8A8WlHN1+a/z6I1e09qHzegB7M/PZ6+3U/t69++vo3/XrSeKef0Ydbt40+poaV0TkAC8eBfPmDeqp3cddEgqmqckiMVLN9cTHWNc+vHBvBPEzKIj43BHWa9lsffc/dy83n9+OGnzj2xd7x6+wHu+f1qjlZU872bzmFkn7oP8De27ONHf9/IkN6d+fVd2Qzu3RmAkiMVfOGP7/Bm/n4+e0kmnxxf993GwtJyHnxuDUcrqvnZreOYck5foG5P+KdLNvLosi2MH9iDb9+QRWJcDFU1tfzo7xt4M38/My7N5JtTR53YO351QyFfnvse8XExfP8TY8noWfdL/bc1e3hs2RYmDOrJY3eMJy20x75zfxl3P7Oa9XtL+bdrR5z4AN9SfIR/f34t8XExzLp9PBcNTQHqRj3/8ddc/pSzk6tGpfGVa4YTY8axqhr+44VcNuwt5WvXjeTeK4ae+BCdv7qAbz2/hrSuiXz/E2Pp1TkBd3jmzW08m1PA1aPS+Plt59EtNBLasLeUmU+vZu+hch6aNvrEB/h7Ow/y3YXrSOuayJw7s8nqV3de53B5FQ/86X1eWV/Ip8Zn8C8XD8YMDpRV8q3n11B4qIL/uvkcbv1Y3RXl7s6vV+Tzw5c2MCK9Kw/fOIbOiXHUuvPI0jxeWV/ILRMy+K+bzznR129v3c99f1hNeVUt/3XzOSc+wJdvLuZ/Fm9kRHpX5tyZfWJPufhwBff9YTWrth3g7suHnPgA333wGN96fg3lVbX87z+fxzVZdfe4qq6p5ceLNzJneT4TB/fiwetHkRAbQ2VNLT9YtL7uda4YwtevG3Vi9Lokdy9fffZ9kuJj+cEnx9K3e12fvvDeLn69YisTM3vx6B3jT3xbfnvJUWY+vZrNRYf52nWjuGx4bwA2Fx3m359fS6eEOB779Hg+NrhuxFheVcO3nl/Dc+/sYnJWOj+9ddxpR6unc7pzAgoBCUR5VQ2fefJtcrYfoGdyAr06xzMuowf3XzWMQSmdT1rX3Zn1Wh4/fXkTvbsk0qNT3S/CtpKj9OvRiTl3Zp8IgONez9vH/X98h8rqWvqF9qSLj1RwrLKm0T34wtJy7n5mNe/tPMiwtC4YUFZZw66Dx5g+cSDfuTHrpOF6VU0t//239Tz1xjb6dU+ic2IcTt2Hd1bfbjx+5wQyep48dP/bB3UjivhYO3HYZs+hcszgkennc2WDPfgtxUeY+XQO20rKGBIKstLyKgpLKxrdgz9WWcPX//IBL76/m4G9kkmMi6HGnfzio1w8NIVf3X7yHnz9EUX3TvEnfrbzQBndO8U3ugd/fERxoKySgaFDJPuPVnLwWBUP3TCaz1w8+KQ9+ANHK/nSvLoRxZDenYmNMSqqa9mxv4wbxvblJ/908h58/RFFWtdEuof6euu+owzslcycu07dgz8+oqip9RMfxEWHK6ioruHHt4zjxgZ78LsPHuPuZ1azZtehU/q6sT34+iOK/j06kZwQi1N3OG5cRndm3znhlD344yOKxLhY0romnnjf+LgYfjn9fC4bfvLdDzYXHmbmM6vZub+MzFBfHzpWRdHhCh64ZgRfvGrYaQ9vfhSFgLRLx//vne54bn3LNhbx59UFJ56X2iWRr147ku7Jje8d7dxfxiNLN3O08vix1xg+d+kQxmY0fhy8orqGXy7NI3/fh7e/vmpUOrdMOPWQz3EvvLeLxbkfDu0H9ErmK1ePoFNC48d31+8pZc7yfCqq6w4tdYqP4/6rhp34xW/ocHkVP3t5E4Wlx4+zGzed14/JY/o0uv7xD/aV+SUn2kb36ca9k4Y2eSx/1bb9PL1yOzW1dYdAuneK54FrRpwYrTRUfLiCn7+yiYNlx4+zx/DpCwZywZCURtevrqnl8eX55O7+8DDOBZkpjR7yOW7p+kKee3fXib5O65rEVyePODFaaWh7yVF++WoeZaG+ToiN4fOXD2FMv8b7uryqhl8s3cz2kg8P40zO6sPN5zd55xuee6eAV9Z/eEPEwSmd+dLVw5s8b7N21yF+syKfytCJoeSEOL501fAmj+sfOlbFz1/eRNHhUF+bccv4DK4c1dT9N8+cQkBEJIrpElEREWmUQkBEJIopBEREophCQEQkiikERESimEJARCSKKQRERKKYQkBEJIp1qC+LmVkxsL2ZT+8NROP0VtG43dG4zRCd2x2N2wxnv92D3D21sR90qBBoCTPLaeobc5EsGrc7GrcZonO7o3GbIbzbrcNBIiJRTCEgIhLFoikE5gRdQECicbujcZshOrc7GrcZwrjdUXNOQEREThVNIwEREWlAISAiEsWiIgTMbIqZbTSzPDN7MOh6WoOZDTCz18xsnZnlmtmXQ+29zOxlM9sc+vPUGb8jgJnFmtm7ZrYwtJxpZm+F+vxPZpbwUa/RkZhZDzObb2YbzGy9mV0UDX1tZg+E/n+vNbO5ZpYUiX1tZk+aWZGZra3X1mj/Wp1HQtv/gZmd1Wz0ER8CZhYLzAKmAlnAdDPLCraqVlEN/Ju7ZwEXAl8IbeeDwFJ3Hw4sDS1Hoi8D6+st/wj4ubsPAw4AMwKpqvX8Avi7u48CxlG37RHd12bWH/gSkO3u5wCxwG1EZl8/BUxp0NZU/04FhoceM4HHzuaNIj4EgIlAnrvnu3slMA+4KeCaws7d97j7O6G/H6buQ6E/ddv6u9BqvwNuDqTAVmRmGcANwG9CywZcBcwPrRJR221m3YHLgScA3L3S3Q8SBX0NxAGdzCwOSAb2EIF97e7Lgf0Nmpvq35uAp73Om0APM+t7pu8VDSHQH9hZb7kg1BaxzGwwcD7wFpDu7ntCP9oLpAdVVyv6X+DrQG1oOQU46O7VoeVI6/NMoBj4begQ2G/MrDMR3tfuvgv4H2AHdR/+h4DVRHZf19dU/7boMy4aQiCqmFkX4C/AV9y9tP7PvO564Ii6JtjMpgFF7r466FraUBwwHnjM3c8HjtLg0E+E9nVP6vZ6M4F+QGdOPWQSFcLZv9EQAruAAfWWM0JtEcfM4qkLgD+4+3Oh5sLjQ8PQn0VB1ddKLgFuNLNt1B3qu4q64+U9QocMIPL6vAAocPe3QsvzqQuFSO/ra4Ct7l7s7lXAc9T1fyT3dX1N9W+LPuOiIQRWAcNDVxAkUHciaUHANYVd6Dj4E8B6d/9ZvR8tAD4T+vtngBfaurbW5O7fdPcMdx9MXd++6u53AK8Bt4RWi6jtdve9wE4zGxlquhpYR4T3NXWHgS40s+TQ//fj2x2xfd1AU/27ALgrdJXQhcCheoeNPpq7R/wDuB7YBGwBvh10Pa20jZdSNzz8AHgv9LieuuPjS4HNwCtAr6BrbcV/g0nAwtDfhwBvA3nAn4HEoOsL87aeB+SE+vuvQM9o6GvgYWADsBZ4BkiMxL4G5lJ33qOKupHfjKb6FzDqroDcAqyh7uqpM34v3TZCRCSKRcPhIBERaYJCQEQkiikERESimEJARCSKKQRERKKYQkBEJIopBEREotj/By0SDytyp30jAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "engine": 1,
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[output:2]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmr0lEQVR4nO3deXxU9b3/8ddnsrIjkBBIQLawJIKyiHvFpSxKpbUb1GoX7o/iUts+2ltpe1uv3tvrvW1ve12oSNVatYWqtRVXcMECFZVQUHYIexBIwhIIELJ9fn9kCCEkiMkkJ5N5Px+PecD5zmTmc+abnPec7zlzvubuiIhIbAoFXYCIiARHISAiEsMUAiIiMUwhICISwxQCIiIxLD7oAj6Jbt26eZ8+fYIuQ0QkqixfvrzQ3VPqui+qQqBPnz7k5OQEXYaISFQxs+313afhIBGRGKYQEBGJYQoBEZEYphAQEYlhCgERkRimEBARiWEKARGRGBYTIbB93xF+/vJaio6VBV2KiEiLEhMhUFhcyu8Wb2Xh+vygSxERaVFiIgSG9+pMaockFqzdE3QpIiItSkyEQChkfDqrO29vKKCkrCLockREWoyYCAGAcdlpHC2tYMmmwqBLERFpMWImBC7u15UOyfHMX6MhIRGRE2ImBBLjQ1w9OJU31u2lvKIy6HJERFqEQEPAzPqZ2WNm9lxzvN647DQOHC0jZ/uB5ng5EZEWr8EhYGaPm1m+ma2u1T7ezDaYWa6ZzTjTc7j7Fnef2tAaPqkrB6aQGB/SkJCISFhj9gSeAMbXbDCzOGAmMAHIAqaYWZaZDTWzl2rdUhvx2g3SLimeKwZ0Y8Gavbh7c7+8iEiL0+AQcPdFwP5azaOB3PAn/FJgLjDJ3Ve5+8Rat7P65paZTTOzHDPLKSgoaGi51cZlp7Hr4DEW6SwhEZGIHxNIB3bWWM4Lt9XJzLqa2SxguJn9qK7HuPtsdx/l7qNSUuqcIvMTGT80jb7d2vH/nszhhZW7Gv18IiLRLNADw+6+z92nu3t/d7+vOV6zY3ICz996KRdkdOY7c1fywJubNDQkIjEr0iGwC+hVYzkj3NainNMukaf+ZTSfG57Or1/fyBPvbAu6JBGRQEQ6BJYBmWbW18wSgcnAvAi/RkQkxcfx6y+dzxWZ3fj16xvZV3w86JJERJpdY04RnQMsBQaZWZ6ZTXX3cuAOYD6wDnjG3ddEptTIMzN+NjGLo6UV/O/rG4MuR0Sk2cU39AfdfUo97a8ArzS4omaW2b0Dt1xyLk+8s42bLupNds9OQZckItJsYuayEWfy3WsG0rlNAve+uFYHiUUkpigEgE5tE/jBuEG8t3U/r63Wt4lFJHYoBMImX9ib/intuF+njIpIDFEIhMWFjNvGDGD9nsO8uU7TUIpIbFAI1HDDBT3JOKcNDy3M1d6AiMQEhUANCXEhpl/Zn5U7D/LO5n1BlyMi0uQUArV8YWQGqR2SeOit3KBLERFpcgqBWpIT4pj2qX4s3bKP5dtrXyRVRKR1UQjU4SsX9aZjcjx/fG9H0KWIiDQphUAd2ibGc+2Q7ry5Lp8yzUcsIq2YQqAeY7PTKDpWxrKtGhISkdZLIVCPKwemkJyg+YhFpHVTCNSjTWIcV2SmsGCt5iMWkdZLIXAG47LT2F1Uwod5RUGXIiLSJBQCZ3DtkFTiQsaCtRoSEpHWKdAQMLMhZjbLzJ4zs1uDrKUundsmclHfLsxfszfoUkREmkRjZhZ73MzyzWx1rfbxZrbBzHLNbMaZnsPd17n7dOBLwGUNraUpjc3qTm5+MZsLioMuRUQk4hqzJ/AEML5mg5nFATOBCUAWMMXMssxsqJm9VOuWGv6ZG4CXaaGzkY3NTgPgueV5AVciIhJ5jZlecpGZ9anVPBrIdfctAGY2F5jk7vcBE+t5nnnAPDN7GfhT7fvNbBowDaB3794NLbfBenZuw4Tz0nj47c2EDL7/6UGEQtbsdYiINIVIHxNIB3bWWM4Lt9XJzMaY2QNm9gj17Am4+2x3H+Xuo1JSUiJb7Vl6YMpwpozuxcyFm/n23BWUlFUEUoeISKQ1eE8gEtz9beDtIGs4GwlxIf7rc0Pp260d9726nooKZ9bNI4MuS0Sk0SIdAruAXjWWM8JtUc/MmPap/pRVOL+cv4HFmwq4IjOYPRMRkUiJ9HDQMiDTzPqaWSIwGZgX4dcI1L9c0Zdzu7bl3hfX6uJyIhL1GnOK6BxgKTDIzPLMbKq7lwN3APOBdcAz7r4mMqW2DEnxcfzb9Vlsyi/mj+9uD7ocEZFGaczZQVPqaX+FFnq6Z6RcOySVKzK78evXN3LDBel0aZcYdEkiIg2iy0Y0gJnxs4lZHCmt4Devbwy6HBGRBlMINFBm9w5MGd2Luct2sLvoWNDliIg0iEKgEaZf2R93mL1oS9CliIg0iEKgETLOactnh6cz5/0dFBYfD7ocEZFPTCHQSLeO6c/x8koeX7I16FJERD4xhUAj9U9pz3VDe/DU0u0UHSsLuhwRkU9EIRABt43pz+Hj5Tz5zragSxER+UQUAhGQ3bMTYwal8NS726ms1HzEIhI9FAIRMumCnuQfPs6KnQeDLkVE5KwpBCLk6kHdidd8xCISZRQCEdKpbQKX9O/KgjV7cdeQkIhEB4VABI3N6s7WwiNsytd8xCISHRQCEfTprKr5iBes0ZCQiEQHhUAEpXVK5oJenZm/Zm/QpYiInBWFQISNy05j1a4idh3UReVEpOULNATCE80vNrNZZjYmyFoiZWx2dwBe15CQiESBxsws9riZ5ZvZ6lrt481sg5nlmtmMj3kaB4qBZCCvobW0JP1T2jMgtT0vfribCn1xTERauMbsCTwBjK/ZYGZxwExgApAFTDGzLDMbamYv1bqlAovdfQJwF3BPI2ppUb56UW+Wbz/At55azpHj5UGXIyJSr8ZML7nIzPrUah4N5Lr7FgAzmwtMcvf7gIlneLoDQFJDa2lpvn5ZX8yMe15cw5ceWcpjX7uQtE7JQZclInKaSB8TSAd21ljOC7fVycxuNLNHgKeAh+p5zDQzyzGznIKCgogW25S+dmkfHvvahWwrPMLnH36HYu0RiEgLFOiBYXd/3t2/5e5fdve363nMbHcf5e6jUlJSmrnCxrlqcCq//8Zodh08xm8X5gZdjojIaSIdAruAXjWWM8JtMWt03y7cODydRxdvZce+o0GXIyJyikiHwDIg08z6mlkiMBmYF+HXiDp3TRhMfJzxny+vDboUEZFTNOYU0TnAUmCQmeWZ2VR3LwfuAOYD64Bn3H1NZEqNXt07JnP7VQNYsHYvSzYVBl2OiEg1i6YrXo4aNcpzcnKCLqNBSsoqGPubRSQnhHj1O58iLmRBlyQiMcLMlrv7qLru02UjmklyQhwzJgxm495iXl61O+hyREQAhUCzGp+dRmZqe367MFfTUIpIi6AQaEahkHHbVf1Zv+cwb67PD7ocERGFQHP7zLCe9O7Slofe2qQZyEQkcAqBZhYfF2L6lf35IK+IJbk6U0hEgqUQCMDnR6aT1jGZh97St4hFJFgKgQAkxcfxL1f05b2t+1m3+1DQ5YhIDFMIBGTSBemYwXxNPiMiAVIIBCSlQxIje5/DAs1HLCIBUggEaFx2Gmt3H2Lnfl1YTkSCoRAI0In5iDUkJCJBUQgE6Nyu7Ric1oEFazUkJCLBUAgEbGxWd3K27Wdf8fGgSxGRGKQQCNjY7DQqHd5Yp70BEWl+CoGAZffsSHrnNjpLSEQCoRAImJkxNrs7i3MLOXi0NOhyRCTGBBoCZnaFmc0ys0fN7J0gawnSjcMzqKx0vvzIu+Qd0OmiItJ8GjO95ONmlm9mq2u1jzezDWaWa2YzzvQc7r7Y3acDLwF/aGgt0W5oRif+8M3RfFR0jM/OfIeVOw8GXZKIxIjG7Ak8AYyv2WBmccBMYAKQBUwxsywzG2pmL9W6pdb40a8Af2pELVHvsgHd+Ottl9ImMcSXH1nKqryioEsSkRjQ4BBw90XA/lrNo4Fcd9/i7qXAXGCSu69y94m1bvkAZtYbKHL3w3W9jplNM7McM8spKChoaLlRYUBqB/5622V0SE7gZ/NWa74BEWlykT4mkA7srLGcF247k6nA7+u7091nu/sodx+VkpISgRJbtm7tk/jh+EGs2HGQv63cFXQ5ItLKBX52kLvf7e4xe1C4Ll8YkcGwjE7896vrOXK8POhyRKQVi3QI7AJ61VjOCLfJJxAKGXd/Jpu9h47z27c18YyINJ1Ih8AyINPM+ppZIjAZmBfh14gJI889h89e0JPfLd6qq4yKSJNpzCmic4ClwCAzyzOzqe5eDtwBzAfWAc+4+5rIlBp7ZkwYAqBpKEWkycQ39AfdfUo97a8ArzS4IqmW1imZyRf2Ys77O/jOtZn07Nwm6JJEpJUJ/MCwnNm3ruyPO8xetCXoUkSkFVIItHDpndvwueHpzHl/BwWHdblpEYkshUAUuHVMf8oqKnlsydagSxGRVkYhEAX6pbTnuqE9ePrd7RQdLQu6HBFpRRQCUeL2qwZQfLycZ3J2fvyDRUTOkkIgSgzp0ZGsHh15TZPSi0gEKQSiyLjsNP654wD5h0uCLkVEWgmFQBQZd1533OGNtflBlyIirYRCIIoM6t6Bc7u2Zb6GhEQkQhQCUcTMGJvVnXc2F3K4RGcJiUjjKQSizLjsNMoqnIUbWvcEOyLSPBQCUWZ473Po1j5JQ0IiEhEKgSgTFzI+nZXK2+vzOV5eEXQ5IhLlFAJRaGx2GkdKK1iyqTDoUkQkyikEotCl/buS2iGJHzz7Acu27Q+6HBGJYoGGgJllmdkzZvawmX0hyFqiSVJ8HM986xLOaZvITb97j7+t0AyeItIwjZlZ7HEzyzez1bXax5vZBjPLNbMZH/M0E4AH3f1W4JaG1hKL+nRrx/O3XcqIczvz3T+v5I/vbQ+6JBGJQo3ZE3gCGF+zwczigJlUbdyzgCnhT/tDzeylWrdU4Clgspn9EujaiFpiUue2iTz5zYu4fEA37ntlveYbEJFPrMEh4O6LgNoD0qOBXHff4u6lwFxgkruvcveJtW754dvtwAygzqOcZjbNzHLMLKegQOfG15YYH+LeSdkcL6/gl/PXB12OiESZSB8TSAdqXus4L9xWJzPrY2azgSeBX9b1GHef7e6j3H1USkpKRIttLfqltOcbl/Xl2eV5fJh3MOhyRCSKBHpg2N23ufs0d7/J3ZcEWUu0u+PqAXRtl8g9L67F3YMuR0SiRKRDYBfQq8ZyRrhNmljH5AT+ddwglm8/wAsrPwq6HBGJEpEOgWVAppn1NbNEYDIwL8KvIfX44sheZPXoyG/e2Eh5RWXQ5YhIFGjMKaJzgKXAIDPLM7Op7l4O3AHMB9YBz7j7msiUKh8nFDLuvGYA2/cd5eVVu4MuR0SiQHxDf9Ddp9TT/grwSoMrkkYZm5VGZmp7Zi7M5TPDehIKWdAliUgLpstGtDKhkHH7VQPYuLeY19ftDbocEWnhFAKt0MRhPejdpS0zF+bqTCEROSOFQCsUHxfi1jH9+TCviMW60qiInIFCoJW6cUQ6aR2TeXTJ1qBLEZEWTCHQSiXFxzFpeE/eyS2k6KjmIxaRuikEWrFx2WmUVzoLN+QHXYqItFAKgVbsgozOpHbQfMQiUj+FQCsWChljs7vz9oYCSso0H7GInE4h0MqNzUrjWJnmIxaRuikEWrmL+3WlQ3K8hoREpE4KgVYuMT7ENYNTeWPdXl1UTkROoxCIAWOz0zhwtIyc7QeCLkVEWhiFQAy4cmAKifEhXlutISEROZVCIAa0S4rn00O68/S72/nzsh1BlyMiLUiDLyUt0eW+zw/lUEkZd/1lFVsLj/LDcYN0mWkRab49ATPrZ2aPmdlzZ2qTptExOYHff/1CbrqoN7P+vpkfPPtB0CWJSAtwViFgZo+bWb6Zra7VPt7MNphZrpnNONNzuPsWd5/6cW3SdOLjQvznZ8/jjqsG8PyKXSxcr8tJiMS6s90TeAIYX7PBzOKAmcAEIAuYYmZZZjbUzF6qdUuNaNXSYGbGnddk0q9bO/7jpbWUluu0UZFYdlYh4O6LgP21mkcDueFP86XAXGCSu69y94m1bg3+yGlm08wsx8xyCgoKGvo0UkNifIifTsxiS+ERnly6LehyRCRAjTkmkA7srLGcF26rk5l1NbNZwHAz+1F9bbW5+2x3H+Xuo1JSUhpRrtR01eBUxgxK4f43NlFw+HjQ5YhIQJrtwLC773P36e7e393vq69Nms+/XZ/FsbIKfjV/Q9CliEhAGhMCu4BeNZYzwm0SJQaktueWS/rw7PKdbC08EnQ5IhKAxoTAMiDTzPqaWSIwGZgXmbKkuUwf04/4uBAPv50bdCkiEoCzPUV0DrAUGGRmeWY21d3LgTuA+cA64Bl3X9N0pUpTSO2QzOQLe/H8P3ex6+CxoMsRkWZ2tmcHTXH3Hu6e4O4Z7v5YuP0Vdx8YHtP/edOWKk3lW1f2B2D23zcHXImINDddO0hI79yGG0ekM3fZTp0pJBJjFAICwK1jBlBWUcmjS7YEXYqINCOFgADQt1s7rh/Wk6eXbudoaXnQ5YhIM1EISLXJF/biSGkFizZqPmKRWKEQkGqj+3ahU5sEFmg+YpGYoRCQaglxIa4Zksqb6/Mp03zEIjFBISCnGJedRtGxMt7fWvt6gSLSGikE5BSfykwhOSHEfA0JicQEhYCcok1iHJ/KTGHBmr1UVnrQ5YhIE1MIyGnGZaex51AJq3YVBV2KiDQxhYCc5pohqcSFTENCIjFAISCn6dw2kYv6dlEIiMQAhYDU6fphPdhccIR/n7eGCh0bEGm14oMuQFqmyRf2ZkvBER5bspUd+4/ywJThtE/Sr4tIa6M9AalTXMj46cQs/uOz5/H3jQV8cdZSDpeUBV2WiERYs4WAmfUzs8fM7LkabUPMbJaZPWdmtzZXLXL2br74XB69ZRTr9xziwbc0+5hIa3O2M4s9bmb5Zra6Vvt4M9tgZrlmNuNMz+HuW9x9aq22de4+HfgScNknLV6ax1WDU/niyAx+/4+tbCkoDrocEYmgs90TeAIYX7PBzOKAmcAEIAuYYmZZZjbUzF6qdUut74nN7AbgZeCVBq2BNIsfjBtEUnwcP395XdCliEgEne30kouA2heTGQ3khj/hlwJzgUnuvsrdJ9a65Z/huee5+wTgprruN7NpZpZjZjkFBQVnt1YScakdkvn21QN4c30+b2+otztFJMo05phAOrCzxnJeuK1OZtbVzGYBw83sR+G2MWb2gJk9Qj17Au4+291HufuolJSURpQrjfX1y/rQp2tb/uOltbrKqEgr0Wzn/Ln7PmB6rba3gbebqwZpnKT4OH5yfRb/78kc/rI8j8mjewddkog0UmP2BHYBvWosZ4TbpBW7dkgqQ9M7MevvmynX3oBI1GtMCCwDMs2sr5klApOBeZEpS1oqM+P2q/qzbd9RXl61O+hyRKSRzvYU0TnAUmCQmeWZ2VR3LwfuAOYD64Bn3H1N05UqLcXYrDQyU9vz24WbdblpkSh3tmcHTXH3Hu6e4O4Z7v5YuP0Vdx/o7v3d/edNW6q0FKGQcdtV/dmw9zBvrNsbdDki0gi6bIQ0yGeG9aR3l7bMfHsz7tobEIlWCgFpkPi4ENOv7M8HOw/ynuYjFolaCgFpsBtHpNMmIY6XP9QBYpFopRCQBktOiOPKgSksWLtHB4hFopRCQBplbHZ39h46zgd5B4MuRUQaQCEgjXLN4O7Eh4z5a3SWkEg0UghIo3Rqm8DF/bqyYM0enSUkEoUUAtJo47K7s6XwCLn5mmtAJNooBKTRPp2VBsD8NXsCrkREPimFgDRaWqdkLujVmQVrdVxAJNooBCQixmZ358O8Ij46eCzoUkTkE1AISESMz64aErrrLx9yqKQs4GpE5GwpBCQi+qW05xefH8bSzfv4wsPvsHP/0aBLEpGzoBCQiPnShb148puj2VNUwud++w/W7zkUdEki8jEUAhJRlw7oxvO3XQbAT/66Wt8dEGnhmi0EzKyfmT1mZs/VaBtjZovNbJaZjWmuWqRpDUhtz/fHDmL59gM6bVSkhTvbmcUeN7N8M1tdq328mW0ws1wzm3Gm53D3Le4+tXYzUAwkA3mfpHBp2b44MoPM1Pb8z2sbKNNcxCIt1tnuCTwBjK/ZYGZxwExgApAFTDGzLDMbamYv1bql1vO8i919AnAXcE/DVkFaovi4EDMmDGZr4RHmvL8j6HJEpB7xZ/Mgd19kZn1qNY8Gct19C4CZzQUmuft9wMSzfN4THxEPAElnVbFEjasHp3Jxvy7c/8YmPjc8nQ7JCUGXJCK1NOaYQDqws8ZyXritTmbW1cxmAcPN7EfhthvN7BHgKeChen5umpnlmFlOQUFBI8qV5mZm/Pi6Iew7Usqji7cGXY6I1OGs9gQiwd33AdNrtT0PPP8xPzcbmA0watQonWoSZYZldObaIak89e52bh3Tn+SEuKBLEpEaGrMnsAvoVWM5I9wmcopvXt6X/UdKeWGlfj1EWprGhMAyINPM+ppZIjAZmBeZsqQ1uaRfVwandeCxJVv1vQGRFuZsTxGdAywFBplZnplNdfdy4A5gPrAOeMbd1zRdqRKtzIypl/dl495i/pG7L+hyRKSGswoBd5/i7j3cPcHdM9z9sXD7K+4+0N37u/vPm7ZUiWY3XNCTbu2TeGzJlqBLEZEadNkIaRZJ8XHcfPG5LNxQwOYCzUAm0lIoBKTZ3HRxbxLjQ3zld+/yw+c+4MUPPmLHvqOUlusbxSJBabZTREW6tU9i9s0j+fOynby2eg/P5FRdKcQMUtonMS47jbs/k0V83OmfTdydp9/dzjM5eVSGDy53a5/EvZOyObdruzpfb8WOA9z36nqOHC8HICEuxK1j+jMuPPdBbfuPlHL3vDVsqbGnctWgVL57bWadNVVWOg8tzD3l+ki9u7TlnknZpHZIrvM1Fm7I58E3N3E8HHxtE+P4wdhBXNSva52PzztwlLtfWMOeQyVA1Xv12QvSmXp5X8zstMcfL6/gl69tYOmWk8deBqd15N9vyKr3y3p/W7GL3/9jK+WVVe9rpzYJ/HRiFkN6dKzz8Rv2HObel9Zw8GjVvBHxIeOWS/rw+ZEZdT6++Hg59764hjUfnbyq7IV9ujBjwuA6Txl2d37/j238dcWu6r5O7ZDEvZPOo1eXtnW+xrJt+/nFa+s5WloBQGJ8iG9fPYCrB3ev8/GFxce5+4U1bNt3pLrt2iHdufOaTOJCp7+vFZXOA29u4o11J2fP69O1HfdOyqZr+7q/5/rG2r3MfDu3+kNOu8R47powiJHndqnz8Tv3H+VnL6wm//BxAEJmfH5EOl+7tE+dfR0pFk1na4waNcpzcnKCLkMioLyiklW7itiUX8xHB4+xaW8xL6/azXVD07h/8nASamx0S8oq+PFfV/H8P3cxLKMTqR2q/uiWbTuAu/PAlOGMGXTqlUn+vGwHP/3bGrq1TySrZ9XGbGvhETYXHOHOqwfw3WsHEqrxx77moyKmPbmcguLjXDGgG2ZVG693t+zn8gHdeHDKcM5pl1j9+KJjZXx37goWbihgdJ8udGwTjzv8Y3Mhndsk8vBXRzC89znVj3d3fvv2Zn61YAN9urajf0pVcK3fc5g9RSX8dGIWt1xy7il/7O9sLuSOP62grLySi/pVbTgKi0tZufMgE4f14BdfGEbbxJOf4/IPlTD96eX8c8dBLh/QjeSEEBWVzqJNhZzbtS2zbx7FgNT2p/TBf72ynsf/sZXBaR3IOKcNAB/kFVFcUs4vvziMicN6nvK+vrpqN99/9gPaJsZzQa9OAOw6WMK63Yf4+qV9+Mn1Q07puy0FxUx7ajlbC49wRWY34kPG8fJKFm8q5PxenXnkqyNJ63QyMI+VVjDj+Q95YeVHnN+rMyntq97z97buJy5kPDRlBJdndjvlfX36vR3cM28N3TsmM6RHh6rXLTzC1sIjfO/agdxx1YBT+vrDvINMf2o5+46Ucnm4rw+VlPP+1v1cOTCFByYPp1Pbk4F58Ggpd85dyaKNBVzUtwsdkqv6ekluIV3bJfLIzaMYmtGp+vGVlc6Db+Xymzc20i+lHf26VfX1ut2HyT9cwj03nMdXLup9yvu6eFMB356zgspKZ3Tfqr4uOHycD/KK+NzwdO67cWijvmNjZsvdfVSd9ykEpKV4dPEW/vPldVw7pDszbxpOcUk5G/Ye5r5X1rNqVxHfu3Yg37765B/0jn1HmfZUDhv2Hub2MQMYmFa1AVi6uZA57+/kisyqjXfntlUbkpKyCn76t9U8uzyPawanMml41Rfc8w+V8KsFGzinbSKzvjqS83t1rq7pmZyd/NtfV5PaMYnvXjuQxPgQ5RWVPPhWLjv3H+XuG7L56kW9qzfeaz86xLSncsg/dJzvfXog6eEN66urdvPq6j3ccH5P/ufzw2iTWPUHfaikjO/NXcmb6/O5cUR6dZhtKSjmwbdy6dutHbNvHkm/lKqNt7vzyKIt/OK19Qzs3oHpV/YnFDJKSiv41YINFB8v51dfPJ/rhvaoXod3t+zj9j/+k+PllfzruEGc0y4Rd2fO+zt4d8t+vnFZH3583cmNd/6hEm774z/J2X6Ab1zWpzrMPtx5kEeXbGV4787M+upIunes2niXV1Ty36+u59ElWxndtws3hd+Pg0dL+eX8DSTEhXjoK8O5tP/Jjfdrq3fz/Wc+oE1iPD8YO5C2SfFV6/b3Lazbc4gfjB3EbWP6V7+v2wqPMO2pHHLzi7nj6szqMFu8sYBnl+dx1aAU/m/ycDq1Saju6x89v4q/rtjF2KzuTDy/Ksx2HzzG/76+kZT2STxy80jOSz+58f7Tezu4e95qenZuw51XZ5IQH6KsvJL739zEnqIS7p2UzeTRJzfeq/KKmP70cgqLj/P9sQNJ61TV1y9+8BGvr93LjSPS+a/Pndx4Fx0t4865K/j7xgK+NCqDyzNTAMjde5iHFuaSmdqB2beMrN6zrax0Zi7M5ddvbCSrR0ceuXkkGefUvSf0cRQCEjWeXLqNn72whjYJcRwrq9q175AUz2++fAHXZp2+a3+0tJy7/rKKFz/46JT2b13Zjx+OG3zarv2JYaV7XlxbPfwBMLpvF2Z+ZQQpHU7ftV+5s+qT44khGagainr4qyO4sM/pu/YHjpRy59wVLN5UWN0WMvjxdUPqHMaprHTuf3MT97+56ZT2cdnd+d8vXUD7pNNHbRdtrPrkWHTs5FSeJz7tDwqHYU0fHTzG9KeX82FeUXVbUnyI+24cyo0jTh/GKS2v5N6X1vD0u6de/G/K6F78+w3ZJMWf/qn0hZW7uOsvH1JSdvIYT3bP+jdem/Yert5LOKFjcjz3TxnOVYNOv+Zk8fFy/vXZD3h19amXJ//21QP4Xq09Ozg5rPTzV9ZRUaOvL+3flYe+MoIuNfbsTli+/QC3Pr28ekgGqoaiZt08khE19uxO2Fd8nDv+tOKU4be4kPFv1w/h63UM41RUOr9+fQMzF24+pf36oVV7du3q6Ou31u/lO3NWMmZwKg9OGX7a/WdDISBR5eUPd7Mkt5D+Ke0Y2L0DwzI6VX+ar4u7k3fgWPU4e7ukOHqEP5XVp7D4ePWYdsiqxndrb0RqOlZawa6Dx6qXe3RKrvMP9oTKSmf7/qPVG59ObRLqDJia9h4q4XDJieMXRu8ubc84Fny4pIy9h05urHp1aVPnxvmE8opKtu8/yok/+a7tEk8Z4qrLroPHOBYeZ09OCH3sJ9GDR0spLC4Fqo5fnNulbZ3HU044Xl7Bzv0n39fUjkl0PMOFBt2dnfuPURq+PHn7pPhThpPqUnD4eHVYxoWMc7u0PWNfHy0t56ODJwO/Z+fkU4bdaqvd153bJtCtnuMEJ+wpKqE4fKwqMS5Ery5tztjXWwqK6dIu8Yx/B2eiEBARiWFnCgGdIioiEsMUAiIiMUwhICISwxQCIiIxTCEgIhLDFAIiIjFMISAiEsMUAiIiMSyqvixmZgXA9gb+eDeg8GMf1frE4nrH4jpDbK53LK4zfPL1PtfdU+q6I6pCoDHMLKe+b8y1ZrG43rG4zhCb6x2L6wyRXW8NB4mIxDCFgIhIDIulEJgddAEBicX1jsV1hthc71hcZ4jgesfMMQERETldLO0JiIhILQoBEZEYFhMhYGbjzWyDmeWa2Yyg62kKZtbLzBaa2VozW2Nm3wm3dzGz181sU/jf0+fIawXMLM7MVpjZS+Hlvmb2XrjP/2xmDZuSqYUys85m9pyZrTezdWZ2SSz0tZl9L/z7vdrM5phZcmvsazN73MzyzWx1jbY6+9eqPBBe/w/NbMQnea1WHwJmFgfMBCYAWcAUM8sKtqomUQ58392zgIuB28PrOQN4090zgTfDy63Rd4B1NZb/B/iNuw8ADgBTA6mq6dwPvObug4HzqVr3Vt3XZpYO3AmMcvfzgDhgMq2zr58Axtdqq69/JwCZ4ds04OFP8kKtPgSA0UCuu29x91JgLjAp4Joizt13u/s/w/8/TNVGIZ2qdf1D+GF/AD4bSIFNyMwygOuBR8PLBlwNPBd+SKtabzPrBHwKeAzA3Uvd/SAx0NdAPNDGzOKBtsBuWmFfu/siYH+t5vr6dxLwpFd5F+hsZj3O9rViIQTSgZ01lvPCba2WmfUBhgPvAd3dfXf4rj1A96DqakL/B/wQqAwvdwUOunt5eLm19XlfoAD4fXgI7FEza0cr72t33wX8CthB1ca/CFhO6+7rmurr30Zt42IhBGKKmbUH/gJ8190P1bzPq84HblXnBJvZRCDf3ZcHXUszigdGAA+7+3DgCLWGflppX59D1afevkBPoB2nD5nEhEj2byyEwC6gV43ljHBbq2NmCVQFwB/d/flw894Tu4bhf/ODqq+JXAbcYGbbqBrqu5qq8fLO4SEDaH19ngfkuft74eXnqAqF1t7X1wJb3b3A3cuA56nq/9bc1zXV17+N2sbFQggsAzLDZxAkUnUgaV7ANUVceBz8MWCdu/+6xl3zgK+F//814IXmrq0pufuP3D3D3ftQ1bdvuftNwELgC+GHtar1dvc9wE4zGxRuugZYSyvva6qGgS42s7bh3/cT691q+7qW+vp3HnBL+Cyhi4GiGsNGH8/dW/0NuA7YCGwGfhJ0PU20jpdTtXv4IbAyfLuOqvHxN4FNwBtAl6BrbcL3YAzwUvj//YD3gVzgWSAp6PoivK4XADnh/v4bcE4s9DVwD7AeWA08BSS1xr4G5lB13KOMqj2/qfX1L2BUnQG5GVhF1dlTZ/1aumyEiEgMi4XhIBERqYdCQEQkhikERERimEJARCSGKQRERGKYQkBEJIYpBEREYtj/B++TM2uIQIyMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "engine": 2,
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%px  --targets 1:3\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.semilogy(mse)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interrupt the hanged worker process\n",
    "\n",
    "It is quite common in the code dealing with the multiple machine communication that some unexpected bugs or deadlock cause the worker process(es) hang forever. Recall the worker processes are inside a seperated environment from the notebook. Hence, the 'interrupt the kernel' button, i.e. black square box, in the toolbar of the notebook interrupts the code of execution in notebook but the worker processes are still running."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `sleep()` function to simulate that situation. If you run the following code and click the interrupt button afterwards, you will be able to execute other cells but the workers still continue to sleep, which will block the running of other requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:36:53.518983Z",
     "start_time": "2021-01-16T06:36:50.581376Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "import time\n",
    "time.sleep(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:46:17.603676Z",
     "start_time": "2021-01-16T06:46:17.545780Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "\n",
    "# You will not see the output immediately, even if you stop the previous cell.\n",
    "# It is because the workers are still sleeping. The interruption in notebook\n",
    "# does not interrupt the workers.\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, BlueFog provides a utility function `interrupt_hanged_processes()`. After running this function, the previous `sleep()` function will be interrupted so that you should be able to execute other functions afterwards. Meanwhile, the context of IPython remains.\n",
    "\n",
    "*Note: `interrupt_hanged_processes()` is supported under the localhost mode only.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-17T07:30:31.620123Z",
     "start_time": "2021-01-17T07:30:30.737987Z"
    }
   },
   "outputs": [],
   "source": [
    "from bluefog.run.interactive_run import interrupt_hanged_processes\n",
    "interrupt_hanged_processes(profile=\"bluefog\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-16T06:46:41.734611Z",
     "start_time": "2021-01-16T06:46:41.677432Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "\n",
    "# Now you should see the output immediately.\n",
    "print(\"Done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What happens after you run `ibfrun`\n",
    "\n",
    "If you are curious about what happened after running the `ibfrun` command, the following figure should give you a good overview.\n",
    "\n",
    "<img src=\"./images/ibfrun.png\" alt=\"ibfrun\" width=\"780\"/>\n",
    "\n",
    "When you start the jupyter notebook, you will create a IPython client, which will keep recieving and executing the command from the notebook. `ibfrun` will create a controller and a bunch of workers (They are actually IPython Kernels) under the MPI environment. The controller knows how to communicate with the IPython client and the workers. When you execute the code under `px` environment, the following procedure happens:\n",
    "\n",
    "1. The notebook sends the user code back to the IPython client.\n",
    "2. The IPython client notices this code is needed to be run under `px` so it forwards the code to the controller.\n",
    "3. The controller forwards the code to the worker according to `--targets`. By default, it forwards to all workers.\n",
    "4. The workers accept the command and execute it in parallel. If there is data exchange among the processes, the communication will be performed through the MPI or NCCL.\n",
    "5. After all workers finish executing the code, the jupyter notebook will be notified that the execution is done.\n",
    "\n",
    "\n",
    "As for the `push` and `pull` commands in direct view, their execution procedures are similar as the `px` data flow.\n",
    "\n",
    "Last, notice that `ibfrun` is just a thin wrapper for convenience over the command provided by the `ipyparallel`. If you have some speical requirement or you have problem of executing `ibfrun`, you can use the `ipcontroller` or `ipengin` command directly. Check [ipyparallel document](https://ipyparallel.readthedocs.io/en/latest/process.html) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Suspend BlueFog\n",
    "Bluefog launches a background process, which may consume CPUs a lot.\n",
    "even if you don't use it actively. You can suspend it through `bf.suspend()`.\n",
    "Note after it is suspended, you are no longer able to run BlueFog communication functions.\n",
    "You can continue the usage through `bf.resume()` again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-02T01:58:05.610701Z",
     "start_time": "2021-01-02T01:58:05.579153Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "bf.suspend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-01-02T01:58:06.154380Z",
     "start_time": "2021-01-02T01:58:06.130947Z"
    }
   },
   "outputs": [],
   "source": [
    "%%px\n",
    "bf.resume()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}