pytorch_ipynb/gan/wgan-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.2.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Runs on CPU or GPU (if available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Zoo -- Wasserstein Generative Adversarial Networks (GAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementation of a very simple/rudimentary Wasserstein GAN using just fully connected layers.\n",
    "\n",
    "The Wasserstein GAN is based on the paper\n",
    "\n",
    "- Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875. (https://arxiv.org/abs/1701.07875)\n",
    "\n",
    "The main differences to a regular GAN are annotated in the code. In short, the main differences are \n",
    "\n",
    "1. Not using a sigmoid activation function and just using a linear output layer for the critic (i.e., discriminator).\n",
    "2. Using label -1 instead of 1 for the real images; using label 1 instead of 0 for fake images.\n",
    "3. Using Wasserstein distance (loss) for training both the critic and the generator.\n",
    "4. After each weight update, clip the weights to be in range [-0.1, 0.1].\n",
    "5. Train the critic 5 times for each generator training update.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Settings and Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch dimensions: torch.Size([128, 1, 28, 28])\n",
      "Image label dimensions: torch.Size([128])\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Device\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Hyperparameters\n",
    "random_seed = 0\n",
    "generator_learning_rate = 0.0005\n",
    "discriminator_learning_rate = 0.0005\n",
    "NUM_EPOCHS = 100\n",
    "BATCH_SIZE = 128\n",
    "LATENT_DIM = 50\n",
    "IMG_SHAPE = (1, 28, 28)\n",
    "IMG_SIZE = 1\n",
    "for x in IMG_SHAPE:\n",
    "    IMG_SIZE *= x\n",
    "\n",
    "## WGAN-specific settings\n",
    "num_iter_critic = 5\n",
    "weight_clip_value = 0.01\n",
    "\n",
    "\n",
    "\n",
    "##########################\n",
    "### MNIST DATASET\n",
    "##########################\n",
    "\n",
    "# Note transforms.ToTensor() scales input images\n",
    "# to 0-1 range\n",
    "train_dataset = datasets.MNIST(root='data', \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='data', \n",
    "                              train=False, \n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          shuffle=True)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE, \n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.shape)\n",
    "    print('Image label dimensions:', labels.shape)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "\n",
    "def wasserstein_loss(y_true, y_pred):\n",
    "    return torch.mean(y_true * y_pred)\n",
    "\n",
    "\n",
    "class GAN(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(GAN, self).__init__()\n",
    "        \n",
    "        \n",
    "        self.generator = nn.Sequential(\n",
    "            nn.Linear(LATENT_DIM, 128),\n",
    "            nn.LeakyReLU(inplace=True),\n",
    "            #nn.Dropout(p=0.5),\n",
    "            nn.Linear(128, IMG_SIZE),\n",
    "            nn.Tanh()\n",
    "        )\n",
    "        \n",
    "        self.discriminator = nn.Sequential(\n",
    "            nn.Linear(IMG_SIZE, 128),\n",
    "            nn.LeakyReLU(inplace=True),\n",
    "            #nn.Dropout(p=0.5),\n",
    "            nn.Linear(128, 1),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "            \n",
    "    def generator_forward(self, z):\n",
    "        img = self.generator(z)\n",
    "        return img\n",
    "    \n",
    "    def discriminator_forward(self, img):\n",
    "        pred = model.discriminator(img)\n",
    "        return pred.view(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(random_seed)\n",
    "\n",
    "model = GAN()\n",
    "model = model.to(device)\n",
    "\n",
    "optim_gener = torch.optim.Adam(model.generator.parameters(), lr=generator_learning_rate)\n",
    "optim_discr = torch.optim.Adam(model.discriminator.parameters(), lr=discriminator_learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/100 | Batch 000/469 | Gen/Dis Loss: -0.5135/-0.0640\n",
      "Epoch: 001/100 | Batch 100/469 | Gen/Dis Loss: -0.6194/-0.1880\n",
      "Epoch: 001/100 | Batch 200/469 | Gen/Dis Loss: -0.5824/-0.1102\n",
      "Epoch: 001/100 | Batch 300/469 | Gen/Dis Loss: -0.5471/-0.0933\n",
      "Epoch: 001/100 | Batch 400/469 | Gen/Dis Loss: -0.5313/-0.0952\n",
      "Time elapsed: 0.31 min\n",
      "Epoch: 002/100 | Batch 000/469 | Gen/Dis Loss: -0.5421/-0.0747\n",
      "Epoch: 002/100 | Batch 100/469 | Gen/Dis Loss: -0.5374/-0.0782\n",
      "Epoch: 002/100 | Batch 200/469 | Gen/Dis Loss: -0.5264/-0.0668\n",
      "Epoch: 002/100 | Batch 300/469 | Gen/Dis Loss: -0.5445/-0.0666\n",
      "Epoch: 002/100 | Batch 400/469 | Gen/Dis Loss: -0.5525/-0.0531\n",
      "Time elapsed: 0.67 min\n",
      "Epoch: 003/100 | Batch 000/469 | Gen/Dis Loss: -0.5433/-0.0530\n",
      "Epoch: 003/100 | Batch 100/469 | Gen/Dis Loss: -0.5621/-0.0404\n",
      "Epoch: 003/100 | Batch 200/469 | Gen/Dis Loss: -0.5358/-0.0523\n",
      "Epoch: 003/100 | Batch 300/469 | Gen/Dis Loss: -0.5589/-0.0481\n",
      "Epoch: 003/100 | Batch 400/469 | Gen/Dis Loss: -0.5147/-0.0620\n",
      "Time elapsed: 0.99 min\n",
      "Epoch: 004/100 | Batch 000/469 | Gen/Dis Loss: -0.5347/-0.0536\n",
      "Epoch: 004/100 | Batch 100/469 | Gen/Dis Loss: -0.5992/-0.0476\n",
      "Epoch: 004/100 | Batch 200/469 | Gen/Dis Loss: -0.5231/-0.0504\n",
      "Epoch: 004/100 | Batch 300/469 | Gen/Dis Loss: -0.5679/-0.0426\n",
      "Epoch: 004/100 | Batch 400/469 | Gen/Dis Loss: -0.5395/-0.0475\n",
      "Time elapsed: 1.33 min\n",
      "Epoch: 005/100 | Batch 000/469 | Gen/Dis Loss: -0.4806/-0.0564\n",
      "Epoch: 005/100 | Batch 100/469 | Gen/Dis Loss: -0.6035/-0.0468\n",
      "Epoch: 005/100 | Batch 200/469 | Gen/Dis Loss: -0.5137/-0.0450\n",
      "Epoch: 005/100 | Batch 300/469 | Gen/Dis Loss: -0.5013/-0.0563\n",
      "Epoch: 005/100 | Batch 400/469 | Gen/Dis Loss: -0.5868/-0.0287\n",
      "Time elapsed: 1.66 min\n",
      "Epoch: 006/100 | Batch 000/469 | Gen/Dis Loss: -0.4917/-0.0369\n",
      "Epoch: 006/100 | Batch 100/469 | Gen/Dis Loss: -0.5030/-0.0502\n",
      "Epoch: 006/100 | Batch 200/469 | Gen/Dis Loss: -0.4951/-0.0464\n",
      "Epoch: 006/100 | Batch 300/469 | Gen/Dis Loss: -0.6160/-0.0365\n",
      "Epoch: 006/100 | Batch 400/469 | Gen/Dis Loss: -0.5193/-0.0601\n",
      "Time elapsed: 1.97 min\n",
      "Epoch: 007/100 | Batch 000/469 | Gen/Dis Loss: -0.5331/-0.0579\n",
      "Epoch: 007/100 | Batch 100/469 | Gen/Dis Loss: -0.5478/-0.0290\n",
      "Epoch: 007/100 | Batch 200/469 | Gen/Dis Loss: -0.4940/-0.0487\n",
      "Epoch: 007/100 | Batch 300/469 | Gen/Dis Loss: -0.5937/-0.0365\n",
      "Epoch: 007/100 | Batch 400/469 | Gen/Dis Loss: -0.5475/-0.0380\n",
      "Time elapsed: 2.30 min\n",
      "Epoch: 008/100 | Batch 000/469 | Gen/Dis Loss: -0.5870/-0.0421\n",
      "Epoch: 008/100 | Batch 100/469 | Gen/Dis Loss: -0.5566/-0.0393\n",
      "Epoch: 008/100 | Batch 200/469 | Gen/Dis Loss: -0.4578/-0.0559\n",
      "Epoch: 008/100 | Batch 300/469 | Gen/Dis Loss: -0.5766/-0.0405\n",
      "Epoch: 008/100 | Batch 400/469 | Gen/Dis Loss: -0.5155/-0.0310\n",
      "Time elapsed: 2.62 min\n",
      "Epoch: 009/100 | Batch 000/469 | Gen/Dis Loss: -0.4966/-0.0448\n",
      "Epoch: 009/100 | Batch 100/469 | Gen/Dis Loss: -0.4337/-0.0660\n",
      "Epoch: 009/100 | Batch 200/469 | Gen/Dis Loss: -0.5877/-0.0370\n",
      "Epoch: 009/100 | Batch 300/469 | Gen/Dis Loss: -0.5682/-0.0198\n",
      "Epoch: 009/100 | Batch 400/469 | Gen/Dis Loss: -0.5051/-0.0430\n",
      "Time elapsed: 2.95 min\n",
      "Epoch: 010/100 | Batch 000/469 | Gen/Dis Loss: -0.4873/-0.0286\n",
      "Epoch: 010/100 | Batch 100/469 | Gen/Dis Loss: -0.4787/-0.0412\n",
      "Epoch: 010/100 | Batch 200/469 | Gen/Dis Loss: -0.6193/-0.0297\n",
      "Epoch: 010/100 | Batch 300/469 | Gen/Dis Loss: -0.5719/-0.0421\n",
      "Epoch: 010/100 | Batch 400/469 | Gen/Dis Loss: -0.5004/-0.0257\n",
      "Time elapsed: 3.26 min\n",
      "Epoch: 011/100 | Batch 000/469 | Gen/Dis Loss: -0.5055/-0.0461\n",
      "Epoch: 011/100 | Batch 100/469 | Gen/Dis Loss: -0.4339/-0.0311\n",
      "Epoch: 011/100 | Batch 200/469 | Gen/Dis Loss: -0.6052/-0.0417\n",
      "Epoch: 011/100 | Batch 300/469 | Gen/Dis Loss: -0.5660/-0.0375\n",
      "Epoch: 011/100 | Batch 400/469 | Gen/Dis Loss: -0.5332/-0.0258\n",
      "Time elapsed: 3.58 min\n",
      "Epoch: 012/100 | Batch 000/469 | Gen/Dis Loss: -0.4895/-0.0338\n",
      "Epoch: 012/100 | Batch 100/469 | Gen/Dis Loss: -0.4584/-0.0317\n",
      "Epoch: 012/100 | Batch 200/469 | Gen/Dis Loss: -0.5424/-0.0308\n",
      "Epoch: 012/100 | Batch 300/469 | Gen/Dis Loss: -0.5662/-0.0438\n",
      "Epoch: 012/100 | Batch 400/469 | Gen/Dis Loss: -0.4723/-0.0480\n",
      "Time elapsed: 3.90 min\n",
      "Epoch: 013/100 | Batch 000/469 | Gen/Dis Loss: -0.5315/-0.0265\n",
      "Epoch: 013/100 | Batch 100/469 | Gen/Dis Loss: -0.4478/-0.0395\n",
      "Epoch: 013/100 | Batch 200/469 | Gen/Dis Loss: -0.6145/-0.0308\n",
      "Epoch: 013/100 | Batch 300/469 | Gen/Dis Loss: -0.5426/-0.0317\n",
      "Epoch: 013/100 | Batch 400/469 | Gen/Dis Loss: -0.5044/-0.0273\n",
      "Time elapsed: 4.24 min\n",
      "Epoch: 014/100 | Batch 000/469 | Gen/Dis Loss: -0.5857/-0.0308\n",
      "Epoch: 014/100 | Batch 100/469 | Gen/Dis Loss: -0.4632/-0.0271\n",
      "Epoch: 014/100 | Batch 200/469 | Gen/Dis Loss: -0.4915/-0.0430\n",
      "Epoch: 014/100 | Batch 300/469 | Gen/Dis Loss: -0.4571/-0.0239\n",
      "Epoch: 014/100 | Batch 400/469 | Gen/Dis Loss: -0.4950/-0.0555\n",
      "Time elapsed: 4.56 min\n",
      "Epoch: 015/100 | Batch 000/469 | Gen/Dis Loss: -0.5115/-0.0419\n",
      "Epoch: 015/100 | Batch 100/469 | Gen/Dis Loss: -0.5577/-0.0200\n",
      "Epoch: 015/100 | Batch 200/469 | Gen/Dis Loss: -0.5806/-0.0407\n",
      "Epoch: 015/100 | Batch 300/469 | Gen/Dis Loss: -0.5311/-0.0247\n",
      "Epoch: 015/100 | Batch 400/469 | Gen/Dis Loss: -0.4802/-0.0566\n",
      "Time elapsed: 4.92 min\n",
      "Epoch: 016/100 | Batch 000/469 | Gen/Dis Loss: -0.4536/-0.0400\n",
      "Epoch: 016/100 | Batch 100/469 | Gen/Dis Loss: -0.5354/-0.0286\n",
      "Epoch: 016/100 | Batch 200/469 | Gen/Dis Loss: -0.5230/-0.0277\n",
      "Epoch: 016/100 | Batch 300/469 | Gen/Dis Loss: -0.4963/-0.0256\n",
      "Epoch: 016/100 | Batch 400/469 | Gen/Dis Loss: -0.5028/-0.0228\n",
      "Time elapsed: 5.27 min\n",
      "Epoch: 017/100 | Batch 000/469 | Gen/Dis Loss: -0.5002/-0.0224\n",
      "Epoch: 017/100 | Batch 100/469 | Gen/Dis Loss: -0.4974/-0.0233\n",
      "Epoch: 017/100 | Batch 200/469 | Gen/Dis Loss: -0.4967/-0.0251\n",
      "Epoch: 017/100 | Batch 300/469 | Gen/Dis Loss: -0.5007/-0.0202\n",
      "Epoch: 017/100 | Batch 400/469 | Gen/Dis Loss: -0.5016/-0.0198\n",
      "Time elapsed: 5.59 min\n",
      "Epoch: 018/100 | Batch 000/469 | Gen/Dis Loss: -0.5041/-0.0187\n",
      "Epoch: 018/100 | Batch 100/469 | Gen/Dis Loss: -0.4997/-0.0150\n",
      "Epoch: 018/100 | Batch 200/469 | Gen/Dis Loss: -0.5153/-0.0248\n",
      "Epoch: 018/100 | Batch 300/469 | Gen/Dis Loss: -0.4866/-0.0284\n",
      "Epoch: 018/100 | Batch 400/469 | Gen/Dis Loss: -0.4979/-0.0143\n",
      "Time elapsed: 5.92 min\n",
      "Epoch: 019/100 | Batch 000/469 | Gen/Dis Loss: -0.5012/-0.0175\n",
      "Epoch: 019/100 | Batch 100/469 | Gen/Dis Loss: -0.4956/-0.0285\n",
      "Epoch: 019/100 | Batch 200/469 | Gen/Dis Loss: -0.4957/-0.0239\n",
      "Epoch: 019/100 | Batch 300/469 | Gen/Dis Loss: -0.4967/-0.0165\n",
      "Epoch: 019/100 | Batch 400/469 | Gen/Dis Loss: -0.4951/-0.0361\n",
      "Time elapsed: 6.28 min\n",
      "Epoch: 020/100 | Batch 000/469 | Gen/Dis Loss: -0.4956/-0.0271\n",
      "Epoch: 020/100 | Batch 100/469 | Gen/Dis Loss: -0.4974/-0.0174\n",
      "Epoch: 020/100 | Batch 200/469 | Gen/Dis Loss: -0.5101/-0.0281\n",
      "Epoch: 020/100 | Batch 300/469 | Gen/Dis Loss: -0.5421/-0.0134\n",
      "Epoch: 020/100 | Batch 400/469 | Gen/Dis Loss: -0.5162/-0.0199\n",
      "Time elapsed: 6.60 min\n",
      "Epoch: 021/100 | Batch 000/469 | Gen/Dis Loss: -0.5339/-0.0183\n",
      "Epoch: 021/100 | Batch 100/469 | Gen/Dis Loss: -0.5280/-0.0215\n",
      "Epoch: 021/100 | Batch 200/469 | Gen/Dis Loss: -0.5358/-0.0175\n",
      "Epoch: 021/100 | Batch 300/469 | Gen/Dis Loss: -0.5243/-0.0169\n",
      "Epoch: 021/100 | Batch 400/469 | Gen/Dis Loss: -0.5205/-0.0170\n",
      "Time elapsed: 6.94 min\n",
      "Epoch: 022/100 | Batch 000/469 | Gen/Dis Loss: -0.5277/-0.0142\n",
      "Epoch: 022/100 | Batch 100/469 | Gen/Dis Loss: -0.5203/-0.0218\n",
      "Epoch: 022/100 | Batch 200/469 | Gen/Dis Loss: -0.5025/-0.0174\n",
      "Epoch: 022/100 | Batch 300/469 | Gen/Dis Loss: -0.4963/-0.0200\n",
      "Epoch: 022/100 | Batch 400/469 | Gen/Dis Loss: -0.5015/-0.0262\n",
      "Time elapsed: 7.28 min\n",
      "Epoch: 023/100 | Batch 000/469 | Gen/Dis Loss: -0.4411/-0.0624\n",
      "Epoch: 023/100 | Batch 100/469 | Gen/Dis Loss: -0.3992/-0.0383\n",
      "Epoch: 023/100 | Batch 200/469 | Gen/Dis Loss: -0.4401/-0.0334\n",
      "Epoch: 023/100 | Batch 300/469 | Gen/Dis Loss: -0.5975/-0.0422\n",
      "Epoch: 023/100 | Batch 400/469 | Gen/Dis Loss: -0.5350/-0.0105\n",
      "Time elapsed: 7.61 min\n",
      "Epoch: 024/100 | Batch 000/469 | Gen/Dis Loss: -0.6127/-0.0073\n",
      "Epoch: 024/100 | Batch 100/469 | Gen/Dis Loss: -0.5962/-0.0066\n",
      "Epoch: 024/100 | Batch 200/469 | Gen/Dis Loss: -0.5505/-0.0109\n",
      "Epoch: 024/100 | Batch 300/469 | Gen/Dis Loss: -0.5428/-0.0151\n",
      "Epoch: 024/100 | Batch 400/469 | Gen/Dis Loss: -0.5220/-0.0129\n",
      "Time elapsed: 7.94 min\n",
      "Epoch: 025/100 | Batch 000/469 | Gen/Dis Loss: -0.5298/-0.0157\n",
      "Epoch: 025/100 | Batch 100/469 | Gen/Dis Loss: -0.5273/-0.0156\n",
      "Epoch: 025/100 | Batch 200/469 | Gen/Dis Loss: -0.5280/-0.0135\n",
      "Epoch: 025/100 | Batch 300/469 | Gen/Dis Loss: -0.5153/-0.0122\n",
      "Epoch: 025/100 | Batch 400/469 | Gen/Dis Loss: -0.5199/-0.0169\n",
      "Time elapsed: 8.28 min\n",
      "Epoch: 026/100 | Batch 000/469 | Gen/Dis Loss: -0.5183/-0.0158\n",
      "Epoch: 026/100 | Batch 100/469 | Gen/Dis Loss: -0.5154/-0.0142\n",
      "Epoch: 026/100 | Batch 200/469 | Gen/Dis Loss: -0.5109/-0.0104\n",
      "Epoch: 026/100 | Batch 300/469 | Gen/Dis Loss: -0.4933/-0.0196\n",
      "Epoch: 026/100 | Batch 400/469 | Gen/Dis Loss: -0.4977/-0.0119\n",
      "Time elapsed: 8.62 min\n",
      "Epoch: 027/100 | Batch 000/469 | Gen/Dis Loss: -0.4141/-0.0329\n",
      "Epoch: 027/100 | Batch 100/469 | Gen/Dis Loss: -0.4162/-0.0315\n",
      "Epoch: 027/100 | Batch 200/469 | Gen/Dis Loss: -0.4167/-0.0266\n",
      "Epoch: 027/100 | Batch 300/469 | Gen/Dis Loss: -0.4595/-0.0283\n",
      "Epoch: 027/100 | Batch 400/469 | Gen/Dis Loss: -0.5303/-0.0371\n",
      "Time elapsed: 8.96 min\n",
      "Epoch: 028/100 | Batch 000/469 | Gen/Dis Loss: -0.5231/-0.0361\n",
      "Epoch: 028/100 | Batch 100/469 | Gen/Dis Loss: -0.4554/-0.0312\n",
      "Epoch: 028/100 | Batch 200/469 | Gen/Dis Loss: -0.7106/-0.0115\n",
      "Epoch: 028/100 | Batch 300/469 | Gen/Dis Loss: -0.5543/-0.0177\n",
      "Epoch: 028/100 | Batch 400/469 | Gen/Dis Loss: -0.6160/-0.0154\n",
      "Time elapsed: 9.30 min\n",
      "Epoch: 029/100 | Batch 000/469 | Gen/Dis Loss: -0.6364/-0.0123\n",
      "Epoch: 029/100 | Batch 100/469 | Gen/Dis Loss: -0.5711/-0.0187\n",
      "Epoch: 029/100 | Batch 200/469 | Gen/Dis Loss: -0.4897/-0.0145\n",
      "Epoch: 029/100 | Batch 300/469 | Gen/Dis Loss: -0.5124/-0.0121\n",
      "Epoch: 029/100 | Batch 400/469 | Gen/Dis Loss: -0.5241/-0.0105\n",
      "Time elapsed: 9.63 min\n",
      "Epoch: 030/100 | Batch 000/469 | Gen/Dis Loss: -0.5315/-0.0108\n",
      "Epoch: 030/100 | Batch 100/469 | Gen/Dis Loss: -0.5224/-0.0134\n",
      "Epoch: 030/100 | Batch 200/469 | Gen/Dis Loss: -0.5097/-0.0133\n",
      "Epoch: 030/100 | Batch 300/469 | Gen/Dis Loss: -0.5200/-0.0090\n",
      "Epoch: 030/100 | Batch 400/469 | Gen/Dis Loss: -0.5248/-0.0105\n",
      "Time elapsed: 9.98 min\n",
      "Epoch: 031/100 | Batch 000/469 | Gen/Dis Loss: -0.5087/-0.0090\n",
      "Epoch: 031/100 | Batch 100/469 | Gen/Dis Loss: -0.5033/-0.0110\n",
      "Epoch: 031/100 | Batch 200/469 | Gen/Dis Loss: -0.5164/-0.0088\n",
      "Epoch: 031/100 | Batch 300/469 | Gen/Dis Loss: -0.4947/-0.0122\n",
      "Epoch: 031/100 | Batch 400/469 | Gen/Dis Loss: -0.5088/-0.0133\n",
      "Time elapsed: 10.32 min\n",
      "Epoch: 032/100 | Batch 000/469 | Gen/Dis Loss: -0.5059/-0.0182\n",
      "Epoch: 032/100 | Batch 100/469 | Gen/Dis Loss: -0.5062/-0.0120\n",
      "Epoch: 032/100 | Batch 200/469 | Gen/Dis Loss: -0.4882/-0.0181\n",
      "Epoch: 032/100 | Batch 300/469 | Gen/Dis Loss: -0.5167/-0.0123\n",
      "Epoch: 032/100 | Batch 400/469 | Gen/Dis Loss: -0.4355/-0.0403\n",
      "Time elapsed: 10.65 min\n",
      "Epoch: 033/100 | Batch 000/469 | Gen/Dis Loss: -0.4432/-0.0723\n",
      "Epoch: 033/100 | Batch 100/469 | Gen/Dis Loss: -0.5519/-0.0591\n",
      "Epoch: 033/100 | Batch 200/469 | Gen/Dis Loss: -0.4846/-0.0375\n",
      "Epoch: 033/100 | Batch 300/469 | Gen/Dis Loss: -0.4178/-0.0216\n",
      "Epoch: 033/100 | Batch 400/469 | Gen/Dis Loss: -0.4613/-0.0146\n",
      "Time elapsed: 11.00 min\n",
      "Epoch: 034/100 | Batch 000/469 | Gen/Dis Loss: -0.4342/-0.0706\n",
      "Epoch: 034/100 | Batch 100/469 | Gen/Dis Loss: -0.4615/-0.0238\n",
      "Epoch: 034/100 | Batch 200/469 | Gen/Dis Loss: -0.3540/-0.0359\n",
      "Epoch: 034/100 | Batch 300/469 | Gen/Dis Loss: -0.5712/-0.0543\n",
      "Epoch: 034/100 | Batch 400/469 | Gen/Dis Loss: -0.6412/-0.0186\n",
      "Time elapsed: 11.35 min\n",
      "Epoch: 035/100 | Batch 000/469 | Gen/Dis Loss: -0.6714/-0.0031\n",
      "Epoch: 035/100 | Batch 100/469 | Gen/Dis Loss: -0.6070/-0.0054\n",
      "Epoch: 035/100 | Batch 200/469 | Gen/Dis Loss: -0.5227/-0.0107\n",
      "Epoch: 035/100 | Batch 300/469 | Gen/Dis Loss: -0.5110/-0.0098\n",
      "Epoch: 035/100 | Batch 400/469 | Gen/Dis Loss: -0.5154/-0.0102\n",
      "Time elapsed: 11.67 min\n",
      "Epoch: 036/100 | Batch 000/469 | Gen/Dis Loss: -0.5224/-0.0131\n",
      "Epoch: 036/100 | Batch 100/469 | Gen/Dis Loss: -0.5149/-0.0125\n",
      "Epoch: 036/100 | Batch 200/469 | Gen/Dis Loss: -0.5129/-0.0117\n",
      "Epoch: 036/100 | Batch 300/469 | Gen/Dis Loss: -0.5269/-0.0085\n",
      "Epoch: 036/100 | Batch 400/469 | Gen/Dis Loss: -0.5165/-0.0101\n",
      "Time elapsed: 12.00 min\n",
      "Epoch: 037/100 | Batch 000/469 | Gen/Dis Loss: -0.5115/-0.0081\n",
      "Epoch: 037/100 | Batch 100/469 | Gen/Dis Loss: -0.5068/-0.0089\n",
      "Epoch: 037/100 | Batch 200/469 | Gen/Dis Loss: -0.5203/-0.0099\n",
      "Epoch: 037/100 | Batch 300/469 | Gen/Dis Loss: -0.6883/-0.0153\n",
      "Epoch: 037/100 | Batch 400/469 | Gen/Dis Loss: -0.5024/-0.0118\n",
      "Time elapsed: 12.34 min\n",
      "Epoch: 038/100 | Batch 000/469 | Gen/Dis Loss: -0.5234/-0.0365\n",
      "Epoch: 038/100 | Batch 100/469 | Gen/Dis Loss: -0.4579/-0.0261\n",
      "Epoch: 038/100 | Batch 200/469 | Gen/Dis Loss: -0.5098/-0.0212\n",
      "Epoch: 038/100 | Batch 300/469 | Gen/Dis Loss: -0.4729/-0.0302\n",
      "Epoch: 038/100 | Batch 400/469 | Gen/Dis Loss: -0.4838/-0.0219\n",
      "Time elapsed: 12.67 min\n",
      "Epoch: 039/100 | Batch 000/469 | Gen/Dis Loss: -0.5327/-0.0189\n",
      "Epoch: 039/100 | Batch 100/469 | Gen/Dis Loss: -0.4758/-0.0155\n",
      "Epoch: 039/100 | Batch 200/469 | Gen/Dis Loss: -0.4985/-0.0180\n",
      "Epoch: 039/100 | Batch 300/469 | Gen/Dis Loss: -0.4737/-0.0193\n",
      "Epoch: 039/100 | Batch 400/469 | Gen/Dis Loss: -0.4705/-0.0269\n",
      "Time elapsed: 12.99 min\n",
      "Epoch: 040/100 | Batch 000/469 | Gen/Dis Loss: -0.4603/-0.0263\n",
      "Epoch: 040/100 | Batch 100/469 | Gen/Dis Loss: -0.4751/-0.0199\n",
      "Epoch: 040/100 | Batch 200/469 | Gen/Dis Loss: -0.4686/-0.0207\n",
      "Epoch: 040/100 | Batch 300/469 | Gen/Dis Loss: -0.5360/-0.0169\n",
      "Epoch: 040/100 | Batch 400/469 | Gen/Dis Loss: -0.6049/-0.0596\n",
      "Time elapsed: 13.33 min\n",
      "Epoch: 041/100 | Batch 000/469 | Gen/Dis Loss: -0.6496/-0.0281\n",
      "Epoch: 041/100 | Batch 100/469 | Gen/Dis Loss: -0.5531/-0.0407\n",
      "Epoch: 041/100 | Batch 200/469 | Gen/Dis Loss: -0.6386/0.0135\n",
      "Epoch: 041/100 | Batch 300/469 | Gen/Dis Loss: -0.6813/-0.0239\n",
      "Epoch: 041/100 | Batch 400/469 | Gen/Dis Loss: -0.5506/-0.0002\n",
      "Time elapsed: 13.66 min\n",
      "Epoch: 042/100 | Batch 000/469 | Gen/Dis Loss: -0.5998/-0.0058\n",
      "Epoch: 042/100 | Batch 100/469 | Gen/Dis Loss: -0.4141/-0.0027\n",
      "Epoch: 042/100 | Batch 200/469 | Gen/Dis Loss: -0.4707/-0.0013\n",
      "Epoch: 042/100 | Batch 300/469 | Gen/Dis Loss: -0.4913/-0.0009\n",
      "Epoch: 042/100 | Batch 400/469 | Gen/Dis Loss: -0.4970/-0.0010\n",
      "Time elapsed: 13.99 min\n",
      "Epoch: 043/100 | Batch 000/469 | Gen/Dis Loss: -0.4962/-0.0006\n",
      "Epoch: 043/100 | Batch 100/469 | Gen/Dis Loss: -0.5052/-0.0012\n",
      "Epoch: 043/100 | Batch 200/469 | Gen/Dis Loss: -0.5064/-0.0010\n",
      "Epoch: 043/100 | Batch 300/469 | Gen/Dis Loss: -0.5079/-0.0012\n",
      "Epoch: 043/100 | Batch 400/469 | Gen/Dis Loss: -0.4998/-0.0010\n",
      "Time elapsed: 14.33 min\n",
      "Epoch: 044/100 | Batch 000/469 | Gen/Dis Loss: -0.5121/-0.0015\n",
      "Epoch: 044/100 | Batch 100/469 | Gen/Dis Loss: -0.5056/-0.0010\n",
      "Epoch: 044/100 | Batch 200/469 | Gen/Dis Loss: -0.5047/-0.0012\n",
      "Epoch: 044/100 | Batch 300/469 | Gen/Dis Loss: -0.5081/-0.0010\n",
      "Epoch: 044/100 | Batch 400/469 | Gen/Dis Loss: -0.5133/-0.0007\n",
      "Time elapsed: 14.68 min\n",
      "Epoch: 045/100 | Batch 000/469 | Gen/Dis Loss: -0.5155/-0.0010\n",
      "Epoch: 045/100 | Batch 100/469 | Gen/Dis Loss: -0.5078/-0.0011\n",
      "Epoch: 045/100 | Batch 200/469 | Gen/Dis Loss: -0.5059/-0.0011\n",
      "Epoch: 045/100 | Batch 300/469 | Gen/Dis Loss: -0.5207/-0.0011\n",
      "Epoch: 045/100 | Batch 400/469 | Gen/Dis Loss: -0.5002/-0.0003\n",
      "Time elapsed: 15.01 min\n",
      "Epoch: 046/100 | Batch 000/469 | Gen/Dis Loss: -0.5206/-0.0010\n",
      "Epoch: 046/100 | Batch 100/469 | Gen/Dis Loss: -0.5157/-0.0005\n",
      "Epoch: 046/100 | Batch 200/469 | Gen/Dis Loss: -0.5434/0.0001\n",
      "Epoch: 046/100 | Batch 300/469 | Gen/Dis Loss: -0.5366/-0.0013\n",
      "Epoch: 046/100 | Batch 400/469 | Gen/Dis Loss: -0.5257/-0.0014\n",
      "Time elapsed: 15.34 min\n",
      "Epoch: 047/100 | Batch 000/469 | Gen/Dis Loss: -0.5006/-0.0001\n",
      "Epoch: 047/100 | Batch 100/469 | Gen/Dis Loss: -0.5025/-0.0004\n",
      "Epoch: 047/100 | Batch 200/469 | Gen/Dis Loss: -0.5043/-0.0003\n",
      "Epoch: 047/100 | Batch 300/469 | Gen/Dis Loss: -0.5245/-0.0033\n",
      "Epoch: 047/100 | Batch 400/469 | Gen/Dis Loss: -0.4847/-0.0107\n",
      "Time elapsed: 15.67 min\n",
      "Epoch: 048/100 | Batch 000/469 | Gen/Dis Loss: -0.5049/0.0003\n",
      "Epoch: 048/100 | Batch 100/469 | Gen/Dis Loss: -0.5225/-0.0009\n",
      "Epoch: 048/100 | Batch 200/469 | Gen/Dis Loss: -0.5005/-0.0018\n",
      "Epoch: 048/100 | Batch 300/469 | Gen/Dis Loss: -0.5225/-0.0006\n",
      "Epoch: 048/100 | Batch 400/469 | Gen/Dis Loss: -0.5155/-0.0025\n",
      "Time elapsed: 15.98 min\n",
      "Epoch: 049/100 | Batch 000/469 | Gen/Dis Loss: -0.5135/-0.0034\n",
      "Epoch: 049/100 | Batch 100/469 | Gen/Dis Loss: -0.5281/-0.0018\n",
      "Epoch: 049/100 | Batch 200/469 | Gen/Dis Loss: -0.5114/-0.0005\n",
      "Epoch: 049/100 | Batch 300/469 | Gen/Dis Loss: -0.5071/-0.0013\n",
      "Epoch: 049/100 | Batch 400/469 | Gen/Dis Loss: -0.5111/-0.0011\n",
      "Time elapsed: 16.33 min\n",
      "Epoch: 050/100 | Batch 000/469 | Gen/Dis Loss: -0.5187/-0.0012\n",
      "Epoch: 050/100 | Batch 100/469 | Gen/Dis Loss: -0.4168/-0.0256\n",
      "Epoch: 050/100 | Batch 200/469 | Gen/Dis Loss: -0.4484/-0.0350\n",
      "Epoch: 050/100 | Batch 300/469 | Gen/Dis Loss: -0.5676/-0.0066\n",
      "Epoch: 050/100 | Batch 400/469 | Gen/Dis Loss: -0.5349/-0.0024\n",
      "Time elapsed: 16.66 min\n",
      "Epoch: 051/100 | Batch 000/469 | Gen/Dis Loss: -0.5686/-0.0007\n",
      "Epoch: 051/100 | Batch 100/469 | Gen/Dis Loss: -0.5810/0.0022\n",
      "Epoch: 051/100 | Batch 200/469 | Gen/Dis Loss: -0.4438/-0.0046\n",
      "Epoch: 051/100 | Batch 300/469 | Gen/Dis Loss: -0.4393/-0.0023\n",
      "Epoch: 051/100 | Batch 400/469 | Gen/Dis Loss: -0.5755/0.0053\n",
      "Time elapsed: 16.99 min\n",
      "Epoch: 052/100 | Batch 000/469 | Gen/Dis Loss: -0.4925/0.0103\n",
      "Epoch: 052/100 | Batch 100/469 | Gen/Dis Loss: -0.5260/-0.0100\n",
      "Epoch: 052/100 | Batch 200/469 | Gen/Dis Loss: -0.5333/0.0018\n",
      "Epoch: 052/100 | Batch 300/469 | Gen/Dis Loss: -0.5024/-0.0213\n",
      "Epoch: 052/100 | Batch 400/469 | Gen/Dis Loss: -0.5430/-0.0025\n",
      "Time elapsed: 17.34 min\n",
      "Epoch: 053/100 | Batch 000/469 | Gen/Dis Loss: -0.5829/-0.0004\n",
      "Epoch: 053/100 | Batch 100/469 | Gen/Dis Loss: -0.5723/-0.0023\n",
      "Epoch: 053/100 | Batch 200/469 | Gen/Dis Loss: -0.5355/-0.0050\n",
      "Epoch: 053/100 | Batch 300/469 | Gen/Dis Loss: -0.5303/-0.0001\n",
      "Epoch: 053/100 | Batch 400/469 | Gen/Dis Loss: -0.5371/-0.0007\n",
      "Time elapsed: 17.66 min\n",
      "Epoch: 054/100 | Batch 000/469 | Gen/Dis Loss: -0.5082/-0.0001\n",
      "Epoch: 054/100 | Batch 100/469 | Gen/Dis Loss: -0.5390/-0.0015\n",
      "Epoch: 054/100 | Batch 200/469 | Gen/Dis Loss: -0.4982/-0.0002\n",
      "Epoch: 054/100 | Batch 300/469 | Gen/Dis Loss: -0.5167/-0.0011\n",
      "Epoch: 054/100 | Batch 400/469 | Gen/Dis Loss: -0.5119/-0.0010\n",
      "Time elapsed: 17.99 min\n",
      "Epoch: 055/100 | Batch 000/469 | Gen/Dis Loss: -0.4969/-0.0014\n",
      "Epoch: 055/100 | Batch 100/469 | Gen/Dis Loss: -0.5009/-0.0034\n",
      "Epoch: 055/100 | Batch 200/469 | Gen/Dis Loss: -0.5014/-0.0006\n",
      "Epoch: 055/100 | Batch 300/469 | Gen/Dis Loss: -0.4971/-0.0019\n",
      "Epoch: 055/100 | Batch 400/469 | Gen/Dis Loss: -0.5323/0.0006\n",
      "Time elapsed: 18.33 min\n",
      "Epoch: 056/100 | Batch 000/469 | Gen/Dis Loss: -0.5050/-0.0004\n",
      "Epoch: 056/100 | Batch 100/469 | Gen/Dis Loss: -0.5021/0.0003\n",
      "Epoch: 056/100 | Batch 200/469 | Gen/Dis Loss: -0.5068/-0.0003\n",
      "Epoch: 056/100 | Batch 300/469 | Gen/Dis Loss: -0.4968/-0.0002\n",
      "Epoch: 056/100 | Batch 400/469 | Gen/Dis Loss: -0.5033/-0.0003\n",
      "Time elapsed: 18.65 min\n",
      "Epoch: 057/100 | Batch 000/469 | Gen/Dis Loss: -0.4989/0.0002\n",
      "Epoch: 057/100 | Batch 100/469 | Gen/Dis Loss: -0.5037/-0.0011\n",
      "Epoch: 057/100 | Batch 200/469 | Gen/Dis Loss: -0.5484/-0.0086\n",
      "Epoch: 057/100 | Batch 300/469 | Gen/Dis Loss: -0.5550/-0.0085\n",
      "Epoch: 057/100 | Batch 400/469 | Gen/Dis Loss: -0.4872/-0.0482\n",
      "Time elapsed: 18.98 min\n",
      "Epoch: 058/100 | Batch 000/469 | Gen/Dis Loss: -0.5460/-0.0154\n",
      "Epoch: 058/100 | Batch 100/469 | Gen/Dis Loss: -0.5258/-0.0130\n",
      "Epoch: 058/100 | Batch 200/469 | Gen/Dis Loss: -0.5730/-0.0151\n",
      "Epoch: 058/100 | Batch 300/469 | Gen/Dis Loss: -0.5880/-0.0066\n",
      "Epoch: 058/100 | Batch 400/469 | Gen/Dis Loss: -0.5276/-0.0017\n",
      "Time elapsed: 19.32 min\n",
      "Epoch: 059/100 | Batch 000/469 | Gen/Dis Loss: -0.5009/-0.0001\n",
      "Epoch: 059/100 | Batch 100/469 | Gen/Dis Loss: -0.5275/-0.0017\n",
      "Epoch: 059/100 | Batch 200/469 | Gen/Dis Loss: -0.5311/-0.0007\n",
      "Epoch: 059/100 | Batch 300/469 | Gen/Dis Loss: -0.5059/-0.0051\n",
      "Epoch: 059/100 | Batch 400/469 | Gen/Dis Loss: -0.4749/-0.0049\n",
      "Time elapsed: 19.67 min\n",
      "Epoch: 060/100 | Batch 000/469 | Gen/Dis Loss: -0.4892/-0.0026\n",
      "Epoch: 060/100 | Batch 100/469 | Gen/Dis Loss: -0.4848/-0.0038\n",
      "Epoch: 060/100 | Batch 200/469 | Gen/Dis Loss: -0.5323/-0.0019\n",
      "Epoch: 060/100 | Batch 300/469 | Gen/Dis Loss: -0.5086/-0.0020\n",
      "Epoch: 060/100 | Batch 400/469 | Gen/Dis Loss: -0.4910/-0.0010\n",
      "Time elapsed: 20.01 min\n",
      "Epoch: 061/100 | Batch 000/469 | Gen/Dis Loss: -0.4976/-0.0008\n",
      "Epoch: 061/100 | Batch 100/469 | Gen/Dis Loss: -0.5041/-0.0012\n",
      "Epoch: 061/100 | Batch 200/469 | Gen/Dis Loss: -0.4993/-0.0012\n",
      "Epoch: 061/100 | Batch 300/469 | Gen/Dis Loss: -0.4994/-0.0012\n",
      "Epoch: 061/100 | Batch 400/469 | Gen/Dis Loss: -0.5033/-0.0015\n",
      "Time elapsed: 20.35 min\n",
      "Epoch: 062/100 | Batch 000/469 | Gen/Dis Loss: -0.5482/-0.0118\n",
      "Epoch: 062/100 | Batch 100/469 | Gen/Dis Loss: -0.4645/-0.0004\n",
      "Epoch: 062/100 | Batch 200/469 | Gen/Dis Loss: -0.5043/-0.0009\n",
      "Epoch: 062/100 | Batch 300/469 | Gen/Dis Loss: -0.5003/-0.0042\n",
      "Epoch: 062/100 | Batch 400/469 | Gen/Dis Loss: -0.4903/-0.0025\n",
      "Time elapsed: 20.67 min\n",
      "Epoch: 063/100 | Batch 000/469 | Gen/Dis Loss: -0.5466/-0.0051\n",
      "Epoch: 063/100 | Batch 100/469 | Gen/Dis Loss: -0.5092/-0.0039\n",
      "Epoch: 063/100 | Batch 200/469 | Gen/Dis Loss: -0.5159/-0.0006\n",
      "Epoch: 063/100 | Batch 300/469 | Gen/Dis Loss: -0.5257/-0.0003\n",
      "Epoch: 063/100 | Batch 400/469 | Gen/Dis Loss: -0.5294/-0.0015\n",
      "Time elapsed: 21.00 min\n",
      "Epoch: 064/100 | Batch 000/469 | Gen/Dis Loss: -0.5153/-0.0013\n",
      "Epoch: 064/100 | Batch 100/469 | Gen/Dis Loss: -0.5105/0.0000\n",
      "Epoch: 064/100 | Batch 200/469 | Gen/Dis Loss: -0.4765/-0.0003\n",
      "Epoch: 064/100 | Batch 300/469 | Gen/Dis Loss: -0.5048/-0.0014\n",
      "Epoch: 064/100 | Batch 400/469 | Gen/Dis Loss: -0.4918/-0.0005\n",
      "Time elapsed: 21.32 min\n",
      "Epoch: 065/100 | Batch 000/469 | Gen/Dis Loss: -0.5068/0.0001\n",
      "Epoch: 065/100 | Batch 100/469 | Gen/Dis Loss: -0.5119/-0.0008\n",
      "Epoch: 065/100 | Batch 200/469 | Gen/Dis Loss: -0.4892/-0.0004\n",
      "Epoch: 065/100 | Batch 300/469 | Gen/Dis Loss: -0.4936/-0.0001\n",
      "Epoch: 065/100 | Batch 400/469 | Gen/Dis Loss: -0.4926/-0.0001\n",
      "Time elapsed: 21.66 min\n",
      "Epoch: 066/100 | Batch 000/469 | Gen/Dis Loss: -0.4988/0.0000\n",
      "Epoch: 066/100 | Batch 100/469 | Gen/Dis Loss: -0.5103/-0.0004\n",
      "Epoch: 066/100 | Batch 200/469 | Gen/Dis Loss: -0.4922/-0.0004\n",
      "Epoch: 066/100 | Batch 300/469 | Gen/Dis Loss: -0.4989/-0.0002\n",
      "Epoch: 066/100 | Batch 400/469 | Gen/Dis Loss: -0.5048/-0.0019\n",
      "Time elapsed: 21.99 min\n",
      "Epoch: 067/100 | Batch 000/469 | Gen/Dis Loss: -0.5015/-0.0024\n",
      "Epoch: 067/100 | Batch 100/469 | Gen/Dis Loss: -0.5198/-0.0024\n",
      "Epoch: 067/100 | Batch 200/469 | Gen/Dis Loss: -0.4921/-0.0017\n",
      "Epoch: 067/100 | Batch 300/469 | Gen/Dis Loss: -0.5008/-0.0028\n",
      "Epoch: 067/100 | Batch 400/469 | Gen/Dis Loss: -0.5131/-0.0021\n",
      "Time elapsed: 22.32 min\n",
      "Epoch: 068/100 | Batch 000/469 | Gen/Dis Loss: -0.4900/-0.0063\n",
      "Epoch: 068/100 | Batch 100/469 | Gen/Dis Loss: -0.4959/-0.0015\n",
      "Epoch: 068/100 | Batch 200/469 | Gen/Dis Loss: -0.5173/-0.0020\n",
      "Epoch: 068/100 | Batch 300/469 | Gen/Dis Loss: -0.5096/-0.0038\n",
      "Epoch: 068/100 | Batch 400/469 | Gen/Dis Loss: -0.5546/-0.0015\n",
      "Time elapsed: 22.66 min\n",
      "Epoch: 069/100 | Batch 000/469 | Gen/Dis Loss: -0.5353/-0.0017\n",
      "Epoch: 069/100 | Batch 100/469 | Gen/Dis Loss: -0.5418/0.0001\n",
      "Epoch: 069/100 | Batch 200/469 | Gen/Dis Loss: -0.4909/-0.0024\n",
      "Epoch: 069/100 | Batch 300/469 | Gen/Dis Loss: -0.4976/-0.0024\n",
      "Epoch: 069/100 | Batch 400/469 | Gen/Dis Loss: -0.5184/-0.0018\n",
      "Time elapsed: 23.00 min\n",
      "Epoch: 070/100 | Batch 000/469 | Gen/Dis Loss: -0.5407/-0.0057\n",
      "Epoch: 070/100 | Batch 100/469 | Gen/Dis Loss: -0.2587/-0.0069\n",
      "Epoch: 070/100 | Batch 200/469 | Gen/Dis Loss: -0.4635/-0.0057\n",
      "Epoch: 070/100 | Batch 300/469 | Gen/Dis Loss: -0.4772/-0.0102\n",
      "Epoch: 070/100 | Batch 400/469 | Gen/Dis Loss: -0.4399/-0.0302\n",
      "Time elapsed: 23.34 min\n",
      "Epoch: 071/100 | Batch 000/469 | Gen/Dis Loss: -0.4708/-0.0118\n",
      "Epoch: 071/100 | Batch 100/469 | Gen/Dis Loss: -0.5367/-0.0193\n",
      "Epoch: 071/100 | Batch 200/469 | Gen/Dis Loss: -0.4875/-0.0226\n",
      "Epoch: 071/100 | Batch 300/469 | Gen/Dis Loss: -0.4844/-0.0030\n",
      "Epoch: 071/100 | Batch 400/469 | Gen/Dis Loss: -0.4174/-0.0123\n",
      "Time elapsed: 23.70 min\n",
      "Epoch: 072/100 | Batch 000/469 | Gen/Dis Loss: -0.4664/-0.0101\n",
      "Epoch: 072/100 | Batch 100/469 | Gen/Dis Loss: -0.4391/0.0001\n",
      "Epoch: 072/100 | Batch 200/469 | Gen/Dis Loss: -0.5560/-0.0022\n",
      "Epoch: 072/100 | Batch 300/469 | Gen/Dis Loss: -0.5898/-0.0008\n",
      "Epoch: 072/100 | Batch 400/469 | Gen/Dis Loss: -0.5763/-0.0010\n",
      "Time elapsed: 24.02 min\n",
      "Epoch: 073/100 | Batch 000/469 | Gen/Dis Loss: -0.5196/-0.0027\n",
      "Epoch: 073/100 | Batch 100/469 | Gen/Dis Loss: -0.5437/-0.0004\n",
      "Epoch: 073/100 | Batch 200/469 | Gen/Dis Loss: -0.5042/-0.0007\n",
      "Epoch: 073/100 | Batch 300/469 | Gen/Dis Loss: -0.4372/-0.0003\n",
      "Epoch: 073/100 | Batch 400/469 | Gen/Dis Loss: -0.3764/-0.0199\n",
      "Time elapsed: 24.33 min\n",
      "Epoch: 074/100 | Batch 000/469 | Gen/Dis Loss: -0.4134/-0.0347\n",
      "Epoch: 074/100 | Batch 100/469 | Gen/Dis Loss: -0.6345/-0.0140\n",
      "Epoch: 074/100 | Batch 200/469 | Gen/Dis Loss: -0.4699/-0.0140\n",
      "Epoch: 074/100 | Batch 300/469 | Gen/Dis Loss: -0.4119/-0.0144\n",
      "Epoch: 074/100 | Batch 400/469 | Gen/Dis Loss: -0.4991/-0.0149\n",
      "Time elapsed: 24.66 min\n",
      "Epoch: 075/100 | Batch 000/469 | Gen/Dis Loss: -0.3295/-0.0345\n",
      "Epoch: 075/100 | Batch 100/469 | Gen/Dis Loss: -0.5274/-0.0075\n",
      "Epoch: 075/100 | Batch 200/469 | Gen/Dis Loss: -0.5596/-0.0007\n",
      "Epoch: 075/100 | Batch 300/469 | Gen/Dis Loss: -0.5620/-0.0026\n",
      "Epoch: 075/100 | Batch 400/469 | Gen/Dis Loss: -0.5708/-0.0008\n",
      "Time elapsed: 25.00 min\n",
      "Epoch: 076/100 | Batch 000/469 | Gen/Dis Loss: -0.5527/-0.0021\n",
      "Epoch: 076/100 | Batch 100/469 | Gen/Dis Loss: -0.5448/-0.0029\n",
      "Epoch: 076/100 | Batch 200/469 | Gen/Dis Loss: -0.5523/-0.0016\n",
      "Epoch: 076/100 | Batch 300/469 | Gen/Dis Loss: -0.5130/-0.0018\n",
      "Epoch: 076/100 | Batch 400/469 | Gen/Dis Loss: -0.4828/-0.0032\n",
      "Time elapsed: 25.35 min\n",
      "Epoch: 077/100 | Batch 000/469 | Gen/Dis Loss: -0.5601/-0.0026\n",
      "Epoch: 077/100 | Batch 100/469 | Gen/Dis Loss: -0.5800/-0.0029\n",
      "Epoch: 077/100 | Batch 200/469 | Gen/Dis Loss: -0.5020/-0.0055\n",
      "Epoch: 077/100 | Batch 300/469 | Gen/Dis Loss: -0.5133/-0.0024\n",
      "Epoch: 077/100 | Batch 400/469 | Gen/Dis Loss: -0.5388/-0.0032\n",
      "Time elapsed: 25.66 min\n",
      "Epoch: 078/100 | Batch 000/469 | Gen/Dis Loss: -0.5453/-0.0017\n",
      "Epoch: 078/100 | Batch 100/469 | Gen/Dis Loss: -0.5146/-0.0041\n",
      "Epoch: 078/100 | Batch 200/469 | Gen/Dis Loss: -0.5264/-0.0020\n",
      "Epoch: 078/100 | Batch 300/469 | Gen/Dis Loss: -0.5192/-0.0036\n",
      "Epoch: 078/100 | Batch 400/469 | Gen/Dis Loss: -0.5178/-0.0012\n",
      "Time elapsed: 25.99 min\n",
      "Epoch: 079/100 | Batch 000/469 | Gen/Dis Loss: -0.5092/-0.0012\n",
      "Epoch: 079/100 | Batch 100/469 | Gen/Dis Loss: -0.5094/-0.0024\n",
      "Epoch: 079/100 | Batch 200/469 | Gen/Dis Loss: -0.5215/-0.0021\n",
      "Epoch: 079/100 | Batch 300/469 | Gen/Dis Loss: -0.5094/-0.0026\n",
      "Epoch: 079/100 | Batch 400/469 | Gen/Dis Loss: -0.5103/-0.0022\n",
      "Time elapsed: 26.32 min\n",
      "Epoch: 080/100 | Batch 000/469 | Gen/Dis Loss: -0.5110/-0.0023\n",
      "Epoch: 080/100 | Batch 100/469 | Gen/Dis Loss: -0.5115/-0.0024\n",
      "Epoch: 080/100 | Batch 200/469 | Gen/Dis Loss: -0.5097/-0.0020\n",
      "Epoch: 080/100 | Batch 300/469 | Gen/Dis Loss: -0.5151/-0.0162\n",
      "Epoch: 080/100 | Batch 400/469 | Gen/Dis Loss: -0.5094/-0.0035\n",
      "Time elapsed: 26.65 min\n",
      "Epoch: 081/100 | Batch 000/469 | Gen/Dis Loss: -0.4744/-0.0154\n",
      "Epoch: 081/100 | Batch 100/469 | Gen/Dis Loss: -0.5015/-0.0085\n",
      "Epoch: 081/100 | Batch 200/469 | Gen/Dis Loss: -0.4256/-0.0220\n",
      "Epoch: 081/100 | Batch 300/469 | Gen/Dis Loss: -0.5114/-0.0264\n",
      "Epoch: 081/100 | Batch 400/469 | Gen/Dis Loss: -0.5249/-0.0219\n",
      "Time elapsed: 26.97 min\n",
      "Epoch: 082/100 | Batch 000/469 | Gen/Dis Loss: -0.5402/-0.0034\n",
      "Epoch: 082/100 | Batch 100/469 | Gen/Dis Loss: -0.4999/-0.0112\n",
      "Epoch: 082/100 | Batch 200/469 | Gen/Dis Loss: -0.5099/-0.0076\n",
      "Epoch: 082/100 | Batch 300/469 | Gen/Dis Loss: -0.5023/-0.0015\n",
      "Epoch: 082/100 | Batch 400/469 | Gen/Dis Loss: -0.5537/-0.0003\n",
      "Time elapsed: 27.33 min\n",
      "Epoch: 083/100 | Batch 000/469 | Gen/Dis Loss: -0.5181/-0.0019\n",
      "Epoch: 083/100 | Batch 100/469 | Gen/Dis Loss: -0.5098/-0.0013\n",
      "Epoch: 083/100 | Batch 200/469 | Gen/Dis Loss: -0.5034/-0.0004\n",
      "Epoch: 083/100 | Batch 300/469 | Gen/Dis Loss: -0.5203/-0.0005\n",
      "Epoch: 083/100 | Batch 400/469 | Gen/Dis Loss: -0.5169/-0.0004\n",
      "Time elapsed: 27.68 min\n",
      "Epoch: 084/100 | Batch 000/469 | Gen/Dis Loss: -0.5269/-0.0023\n",
      "Epoch: 084/100 | Batch 100/469 | Gen/Dis Loss: -0.5413/0.0009\n",
      "Epoch: 084/100 | Batch 200/469 | Gen/Dis Loss: -0.5025/-0.0005\n",
      "Epoch: 084/100 | Batch 300/469 | Gen/Dis Loss: -0.4953/-0.0007\n",
      "Epoch: 084/100 | Batch 400/469 | Gen/Dis Loss: -0.4954/-0.0002\n",
      "Time elapsed: 28.01 min\n",
      "Epoch: 085/100 | Batch 000/469 | Gen/Dis Loss: -0.5085/-0.0006\n",
      "Epoch: 085/100 | Batch 100/469 | Gen/Dis Loss: -0.5094/-0.0003\n",
      "Epoch: 085/100 | Batch 200/469 | Gen/Dis Loss: -0.4905/-0.0010\n",
      "Epoch: 085/100 | Batch 300/469 | Gen/Dis Loss: -0.4985/-0.0005\n",
      "Epoch: 085/100 | Batch 400/469 | Gen/Dis Loss: -0.5243/-0.0019\n",
      "Time elapsed: 28.32 min\n",
      "Epoch: 086/100 | Batch 000/469 | Gen/Dis Loss: -0.5194/-0.0017\n",
      "Epoch: 086/100 | Batch 100/469 | Gen/Dis Loss: -0.4573/-0.0016\n",
      "Epoch: 086/100 | Batch 200/469 | Gen/Dis Loss: -0.4788/-0.0000\n",
      "Epoch: 086/100 | Batch 300/469 | Gen/Dis Loss: -0.4678/-0.0021\n",
      "Epoch: 086/100 | Batch 400/469 | Gen/Dis Loss: -0.4891/-0.0018\n",
      "Time elapsed: 28.65 min\n",
      "Epoch: 087/100 | Batch 000/469 | Gen/Dis Loss: -0.5471/-0.0294\n",
      "Epoch: 087/100 | Batch 100/469 | Gen/Dis Loss: -0.5285/-0.0102\n",
      "Epoch: 087/100 | Batch 200/469 | Gen/Dis Loss: -0.5309/-0.0333\n",
      "Epoch: 087/100 | Batch 300/469 | Gen/Dis Loss: -0.6362/-0.0185\n",
      "Epoch: 087/100 | Batch 400/469 | Gen/Dis Loss: -0.4112/-0.0210\n",
      "Time elapsed: 28.99 min\n",
      "Epoch: 088/100 | Batch 000/469 | Gen/Dis Loss: -0.4944/-0.0390\n",
      "Epoch: 088/100 | Batch 100/469 | Gen/Dis Loss: -0.4935/-0.0400\n",
      "Epoch: 088/100 | Batch 200/469 | Gen/Dis Loss: -0.5029/-0.0067\n",
      "Epoch: 088/100 | Batch 300/469 | Gen/Dis Loss: -0.5156/-0.0054\n",
      "Epoch: 088/100 | Batch 400/469 | Gen/Dis Loss: -0.5058/-0.0001\n",
      "Time elapsed: 29.32 min\n",
      "Epoch: 089/100 | Batch 000/469 | Gen/Dis Loss: -0.4798/-0.0009\n",
      "Epoch: 089/100 | Batch 100/469 | Gen/Dis Loss: -0.5277/0.0001\n",
      "Epoch: 089/100 | Batch 200/469 | Gen/Dis Loss: -0.5040/0.0000\n",
      "Epoch: 089/100 | Batch 300/469 | Gen/Dis Loss: -0.5422/0.0008\n",
      "Epoch: 089/100 | Batch 400/469 | Gen/Dis Loss: -0.5238/0.0003\n",
      "Time elapsed: 29.64 min\n",
      "Epoch: 090/100 | Batch 000/469 | Gen/Dis Loss: -0.5062/-0.0008\n",
      "Epoch: 090/100 | Batch 100/469 | Gen/Dis Loss: -0.5131/-0.0002\n",
      "Epoch: 090/100 | Batch 200/469 | Gen/Dis Loss: -0.5049/-0.0001\n",
      "Epoch: 090/100 | Batch 300/469 | Gen/Dis Loss: -0.4949/0.0002\n",
      "Epoch: 090/100 | Batch 400/469 | Gen/Dis Loss: -0.4944/0.0001\n",
      "Time elapsed: 29.98 min\n",
      "Epoch: 091/100 | Batch 000/469 | Gen/Dis Loss: -0.5045/-0.0002\n",
      "Epoch: 091/100 | Batch 100/469 | Gen/Dis Loss: -0.5035/-0.0003\n",
      "Epoch: 091/100 | Batch 200/469 | Gen/Dis Loss: -0.4985/-0.0002\n",
      "Epoch: 091/100 | Batch 300/469 | Gen/Dis Loss: -0.5009/-0.0007\n",
      "Epoch: 091/100 | Batch 400/469 | Gen/Dis Loss: -0.5033/-0.0012\n",
      "Time elapsed: 30.33 min\n",
      "Epoch: 092/100 | Batch 000/469 | Gen/Dis Loss: -0.5053/-0.0002\n",
      "Epoch: 092/100 | Batch 100/469 | Gen/Dis Loss: -0.5040/-0.0006\n",
      "Epoch: 092/100 | Batch 200/469 | Gen/Dis Loss: -0.5090/-0.0003\n",
      "Epoch: 092/100 | Batch 300/469 | Gen/Dis Loss: -0.4823/-0.0001\n",
      "Epoch: 092/100 | Batch 400/469 | Gen/Dis Loss: -0.4990/-0.0004\n",
      "Time elapsed: 30.66 min\n",
      "Epoch: 093/100 | Batch 000/469 | Gen/Dis Loss: -0.5077/-0.0012\n",
      "Epoch: 093/100 | Batch 100/469 | Gen/Dis Loss: -0.5071/-0.0007\n",
      "Epoch: 093/100 | Batch 200/469 | Gen/Dis Loss: -0.4967/-0.0016\n",
      "Epoch: 093/100 | Batch 300/469 | Gen/Dis Loss: -0.5844/-0.0025\n",
      "Epoch: 093/100 | Batch 400/469 | Gen/Dis Loss: -0.4284/0.0000\n",
      "Time elapsed: 30.98 min\n",
      "Epoch: 094/100 | Batch 000/469 | Gen/Dis Loss: -0.5161/-0.0128\n",
      "Epoch: 094/100 | Batch 100/469 | Gen/Dis Loss: -0.5243/-0.0205\n",
      "Epoch: 094/100 | Batch 200/469 | Gen/Dis Loss: -0.4952/-0.0063\n",
      "Epoch: 094/100 | Batch 300/469 | Gen/Dis Loss: -0.7724/-0.0133\n",
      "Epoch: 094/100 | Batch 400/469 | Gen/Dis Loss: -0.6496/0.0011\n",
      "Time elapsed: 31.30 min\n",
      "Epoch: 095/100 | Batch 000/469 | Gen/Dis Loss: -0.5300/-0.0065\n",
      "Epoch: 095/100 | Batch 100/469 | Gen/Dis Loss: -0.4371/-0.0022\n",
      "Epoch: 095/100 | Batch 200/469 | Gen/Dis Loss: -0.5217/-0.0012\n",
      "Epoch: 095/100 | Batch 300/469 | Gen/Dis Loss: -0.5196/-0.0022\n",
      "Epoch: 095/100 | Batch 400/469 | Gen/Dis Loss: -0.4919/-0.0045\n",
      "Time elapsed: 31.63 min\n",
      "Epoch: 096/100 | Batch 000/469 | Gen/Dis Loss: -0.5167/-0.0010\n",
      "Epoch: 096/100 | Batch 100/469 | Gen/Dis Loss: -0.5098/-0.0008\n",
      "Epoch: 096/100 | Batch 200/469 | Gen/Dis Loss: -0.4873/-0.0005\n",
      "Epoch: 096/100 | Batch 300/469 | Gen/Dis Loss: -0.4967/-0.0003\n",
      "Epoch: 096/100 | Batch 400/469 | Gen/Dis Loss: -0.4824/-0.0007\n",
      "Time elapsed: 31.97 min\n",
      "Epoch: 097/100 | Batch 000/469 | Gen/Dis Loss: -0.4844/-0.0011\n",
      "Epoch: 097/100 | Batch 100/469 | Gen/Dis Loss: -0.4870/-0.0010\n",
      "Epoch: 097/100 | Batch 200/469 | Gen/Dis Loss: -0.4856/-0.0008\n",
      "Epoch: 097/100 | Batch 300/469 | Gen/Dis Loss: -0.4727/-0.0015\n",
      "Epoch: 097/100 | Batch 400/469 | Gen/Dis Loss: -0.4953/-0.0007\n",
      "Time elapsed: 32.30 min\n",
      "Epoch: 098/100 | Batch 000/469 | Gen/Dis Loss: -0.5012/-0.0007\n",
      "Epoch: 098/100 | Batch 100/469 | Gen/Dis Loss: -0.5168/-0.0021\n",
      "Epoch: 098/100 | Batch 200/469 | Gen/Dis Loss: -0.5064/-0.0004\n",
      "Epoch: 098/100 | Batch 300/469 | Gen/Dis Loss: -0.5039/-0.0007\n",
      "Epoch: 098/100 | Batch 400/469 | Gen/Dis Loss: -0.5418/-0.0339\n",
      "Time elapsed: 32.61 min\n",
      "Epoch: 099/100 | Batch 000/469 | Gen/Dis Loss: -0.5468/-0.0073\n",
      "Epoch: 099/100 | Batch 100/469 | Gen/Dis Loss: -0.4540/-0.0241\n",
      "Epoch: 099/100 | Batch 200/469 | Gen/Dis Loss: -0.5279/-0.0195\n",
      "Epoch: 099/100 | Batch 300/469 | Gen/Dis Loss: -0.4611/-0.0213\n",
      "Epoch: 099/100 | Batch 400/469 | Gen/Dis Loss: -0.4722/-0.0154\n",
      "Time elapsed: 32.94 min\n",
      "Epoch: 100/100 | Batch 000/469 | Gen/Dis Loss: -0.4558/-0.0172\n",
      "Epoch: 100/100 | Batch 100/469 | Gen/Dis Loss: -0.5864/-0.0064\n",
      "Epoch: 100/100 | Batch 200/469 | Gen/Dis Loss: -0.4810/-0.0077\n",
      "Epoch: 100/100 | Batch 300/469 | Gen/Dis Loss: -0.4921/-0.0127\n",
      "Epoch: 100/100 | Batch 400/469 | Gen/Dis Loss: -0.4773/-0.0143\n",
      "Time elapsed: 33.30 min\n",
      "Total Training Time: 33.30 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()    \n",
    "\n",
    "discr_costs = []\n",
    "gener_costs = []\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    model = model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "\n",
    "        \n",
    "        \n",
    "        features = (features - 0.5)*2.\n",
    "        features = features.view(-1, IMG_SIZE).to(device) \n",
    "        targets = targets.to(device)\n",
    "\n",
    "        # Regular GAN:\n",
    "        # valid = torch.ones(targets.size(0)).float().to(device)\n",
    "        # fake = torch.zeros(targets.size(0)).float().to(device)\n",
    "        \n",
    "        # WGAN:\n",
    "        valid = -(torch.ones(targets.size(0)).float()).to(device)\n",
    "        fake = torch.ones(targets.size(0)).float().to(device)\n",
    "        \n",
    "\n",
    "        ### FORWARD AND BACK PROP\n",
    "        \n",
    "        \n",
    "        # --------------------------\n",
    "        # Train Generator\n",
    "        # --------------------------\n",
    "        \n",
    "        # Make new images\n",
    "        z = torch.zeros((targets.size(0), LATENT_DIM)).uniform_(-1.0, 1.0).to(device)\n",
    "        generated_features = model.generator_forward(z)\n",
    "        \n",
    "        # Loss for fooling the discriminator\n",
    "        discr_pred = model.discriminator_forward(generated_features)\n",
    "        \n",
    "        \n",
    "        # Regular GAN:\n",
    "        # gener_loss = F.binary_cross_entropy_with_logits(discr_pred, valid)\n",
    "        \n",
    "        # WGAN:\n",
    "        gener_loss = wasserstein_loss(valid, discr_pred)\n",
    "        \n",
    "        optim_gener.zero_grad()\n",
    "        gener_loss.backward()\n",
    "        optim_gener.step()\n",
    "        \n",
    "        # --------------------------\n",
    "        # Train Discriminator\n",
    "        # --------------------------        \n",
    "\n",
    "        \n",
    "        # WGAN: 5 loops for discriminator\n",
    "        for _ in range(num_iter_critic):\n",
    "        \n",
    "            discr_pred_real = model.discriminator_forward(features.view(-1, IMG_SIZE))\n",
    "            # Regular GAN:\n",
    "            # real_loss = F.binary_cross_entropy_with_logits(discr_pred_real, valid)\n",
    "            # WGAN:\n",
    "            real_loss = wasserstein_loss(valid, discr_pred_real)\n",
    "\n",
    "            discr_pred_fake = model.discriminator_forward(generated_features.detach())\n",
    "\n",
    "            # Regular GAN:\n",
    "            # fake_loss = F.binary_cross_entropy_with_logits(discr_pred_fake, fake)\n",
    "            # WGAN:\n",
    "            fake_loss = wasserstein_loss(fake, discr_pred_fake)\n",
    "\n",
    "            # Regular GAN:\n",
    "            discr_loss = (real_loss + fake_loss)\n",
    "            # WGAN:\n",
    "            #discr_loss = -(real_loss - fake_loss)\n",
    "\n",
    "            optim_discr.zero_grad()\n",
    "            discr_loss.backward()\n",
    "            optim_discr.step()        \n",
    "\n",
    "            # WGAN:\n",
    "            for p in model.discriminator.parameters():\n",
    "                p.data.clamp_(-weight_clip_value, weight_clip_value)\n",
    "\n",
    "        \n",
    "        discr_costs.append(discr_loss.item())\n",
    "        gener_costs.append(gener_loss.item())\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "        ### LOGGING\n",
    "        if not batch_idx % 100:\n",
    "            print ('Epoch: %03d/%03d | Batch %03d/%03d | Gen/Dis Loss: %.4f/%.4f' \n",
    "                   %(epoch+1, NUM_EPOCHS, batch_idx, \n",
    "                     len(train_loader), gener_loss, discr_loss))\n",
    "\n",
    "    print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))\n",
    "    \n",
    "print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEzCAYAAAAvu0StAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOydd3gUVffHvycJSSD0hN5C7xAgIF2qgKigrwiKil0UVOS1xBcLdsTy41UR5EVAxQ4CCoIC0gUkQIDQBEKUTkggEAIh5f7+mNlkdzM7OztlZzc5n+fZZ3fqvTM7c8+955x7DgkhwDAMwzCeCLG7AgzDMExgw4KCYRiGUYUFBcMwDKMKCwqGYRhGFRYUDMMwjCphdlfAbEJCQkTZsmXtrgbDMExQkZ2dLYQQioOHEicoypYti8uXL9tdDYZhmKCCiK542saqJ4ZhGEYVFhQMwzCMKiwoGIZhGFVYUDAMwzCqsKBgGIZhVGFBwTAMw6jCgoJhGIZRhQUFwzCBwaldwPHtdteCUaDETbhjGE3kXgFCwoDQMnbXhHHwaW/pe3KmvfVgisEjCqZ0cfWi9P1mTeB/ff1bthDApdP+LTMYCcRkansXAWf3210L22BBwQQmu38AJlcCMk8AaQeBvGvGz7nzK2BKPem8AHB6j/FzauHaZeCDVsCiR4H3mwMnWL1SyMkkYFpb4Hxq0bpl/za3jMXjgG/u1H986kbgh/uAT7oCF0+aVi3TuHAMyM6wtAhbBQURDSaig0R0mIgSFLZHENF38vatRBTr/1qWcubdVNSwaiH3KvDTE8Dr1aSGWQ8FBcCPD0m/N/4fML0L8E4s8P0YaZteDv1afN3pZP3n08rh1cDFE8Du76Tlc4eArZ8CfynUx1eEAPYskO75yZ3Gz2c2edeA9CPAkd+Vt8+6HrjwD/Df9kXrEj8zVub5v4v+16RvgKT5wMFfgEtn9J3vi2FFv39/E8hK01+3ggJgzmDpmXirrvRurX9Xss/4wp4FwB8fScdPawNMbai/Thogu3JmE1EogL8ADARwHMA2AHcKIfY57fM4gHZCiLFENArArUKIkWrnjYqKEhwU0AQupwNZZ4AZ3aRlrXrj7fOAn59yXdf2DuBf/9Ne9qb/AitfVt7WfChw59fqx18+B0RUAMIiXNd/PwbYt7j4/sNnAHF3aa+fr3gTtLXigNiewOaPgWdTgKhoqaddsTZQtgoAAkI9mBPdz/3yeamhjagIVKoLNOgOELnuc3Y/kLJOOne7O4pvz7sGiHygjMYozPl5rvW7li0JxciKwIIHiu9/57dS2fW7er437s/b3sVAi5s83weXY+VzPr4V+OQ6121hkcAj64DqLbyfB5BGnTN7Fl8/IVn6f0JCtZ3HwW8vSg28Eo+uB2q1V96WlyP9b7XaA1/dDhxepXD8BqBcVel/1wERZQshohS32SgougGYLIQYJC+/AABCiLed9vlV3mczEYUBOA2gmlCpNAsKk3B/ge/8Fmg+xPtxiXOBpROKr39iBxDdWF/ZxbZ7EVqO41/OcH2Rvx0NHFiqrQ5M8NBsCFCrHZC6CWg+WGqMSzM6nQHUBIWdqqc6AI45LR+X1ynuI4TIA5AJINr9RET0CBElElFiXl6eRdU1idyrwLp3zdG5A8CHHYC36wNLnzamlvHGN6MkXag30g4qr/+oo7Zr1jKsP3tAsl1kngB+fAT4pBuwdgpwbBuweXrRfnt+KPqdl8NCoqTy13Jg3TvA3xtZSFiEne6xpLDOfaSgZR8IIWYBmAVIIwrjVbOQzR8Ba94A8nOAfk4PdUE+cGQNULVhUc/7zF5JBz2zh7Q8eAoQGi6pKmb3A7o/AWSkSNsS5wBdHwdimhadM+MokH4YiO0FUAgQFi6tv3RaMqoCQJ8XgB7yCODaZaAgF6hQU7nu09q4Lg95F1j+rPT7nkVApXrA1hmer/2Naur3Rivu6gQAOLsPWPu267pFj0ousOWqSsbxkkD/l4G//1BWPTD2UbEO0PpWoMvD0shm9auS6raEwKons7iaCUxtJDXmvzwDXDcWGPKO6z4ndri6ZDqGiNkZrsaoyZmS0PhyuG91cFcPuatwXkwzr7Euabx0Tt+ciosngQ9aej6HVkeAuLuBOh2B+Ackm8Hh1VKnoVI9qbPgEPIOvh4l9aS90fkhYNDbRccr1eeuH4BmNxRff2af1PFwXJPj2Lt/BKq3BDa8D0RWkr4dlK+hrYG84wvg+3vV9/nPKSDvanFD7aC3pfcrxINCxNs9v3UW0PZ24LWq8v5OqprkH4EF9wM3vAlElC9ubytWlo9qnp+eAHZ8obwtphkwfpuHcnxwKLFA9WTniGIbgKZE1BDACQCjALhbFH8CMAbAZgC3A/hdTUjYyqGVQEGeJCQAYOtMV0HhLiQAyRvkl2eKe4TMHSoNo33lm1HSd1R14PLZ4ttZSChjZIJXxdpFv41M3hs+3XW5SX/1/fv+R1lQvHJBEjT5udInvJzncyT8A1Co1CAqUaOV63LD3sDR9UV1GyoLiP4vSx2lMuWAnydIXkbeaDXM+z7h5YrX3+hkvM4PAe1lf5jHt0gjTmda3yo5QjTuLwkib4LCV1oO8ywoxv3p+bhu4yVnhyqxrq7EfsI2QSGEyCOi8QB+BRAKYI4QYi8RvQYgUQjxE4DPAHxJRIcBZEASJoHFoZWSF4ISp5OB3Gzgs4GSysidjzoqH6dHSDijJCRKEsM+AUQB0PGeonW+9LicGTzFeH0eWSupg/wJeehNOzyYQst4F1yRPt6ze5ZI913tXErKYnfqdpG+q7UA0g5oL//ZFG37PbIWmNVHeVv/V4p+V29ZfDsR0HSg9jr5StMBnre5e585M+hN6fPFsNIlKABACPELgF/c1r3s9PsqgBH+rpdXNrwPrH5NenA9CQmgyLYAAPkmGa/tokZb4MFfgXC3kenZA8o2A3dGfA60llVp2RmSTSHWye3Ql4a+w2jt+3rDvUeph9odpI8/Uap3GZXRgyllhsCr/8t1Y4HkRUCuivr3X7Ol70FvAvP/5b3cf30GHE+U3Ia1ULsDMPA1ZRfryIrazqGF+AfNO5dWopsAKWs9b6/e2pJieWa2Fs7uB768Ffh7s9SgrX5NWv9uI3vr5Y3qrbzvo5V7FxcXEoA2f/TRC4uEBCAZl2PdfNOfTNJWjzAPvv2tb9V2vDt2xHq6fa7r8h1f+n4OJUGR8I+++phJzbbApJPAvUuKbxv0tnStVRpIy3U7ez6P8//Z9nZgiI8jvx4mq4zceWQdcNMH1pahRLkYz9smZwKPWzOy5aCAWvjlWSB1g+fZpYFC/5eBXm7hD66cl2Y1A0DNdsDp3cWPezFNajBP7pRGS0pupGq91Q73ADtVGju14baDqhpnlt7/i/L6mz+U4vH4SmRl348xwh1fAq1uAdrcZuw8Zd3qXaNNYAU4bNSn+Lpuj7suq6m+bnjTeB1ePg+8VsX4eZSoHWfNeb1RrVnxdUPfBzreZ2mxPKLQQuoGu2ugjR5PF19XtorU05icCYzdIBnzHLyULn3CwiX9aJ2OnnX2akbR1ireWaO8zKL2hUfWSnVUwpNKIbaX+jndRzZWMmSqJCTMIMqtZ9ltnG/H1/JDQzc5U5r0CAC9n/Xt2EruU6p0EBIChHsw1NvJXd8DZau6rkvQMEcJAFordDBqd9Q2Y90APKLwRqCPIpzx5C7ozND3gW2yjtish0ttol/NtuaUAfhuB6hUD7hvKbD8eckLTYmICsbrpZXrHrXu3L7awLTMsjeDkFB7w4ZP3CdN9DyzB4gKEK+/ZoOA+5e72va02k7cDd63zvLceTIRHlGo8ddvkm2ipNH7WWCkp4B9OryPPXnCAEDl+r6fzwwGvi65PwKSm/LkTKCiQgwc93hQZhPiJ3VQ8o++7d/Kxzk6VtH7Oamxc8bbKNAXIisB5asBjfvp67RYdZ/cR4S+4Gz/aaPBGcAEWFB4Qgjg68BzuDKFfi8CLW9S3qZnmooZnkNm02188fkBE/f6vx6d/eQZoyasldAaFM9q+k2S5jUMmQr0e8nu2iggvw8Ol16ziIoBBkzWd2yjPsCzR6RRicUqJwcsKDwRbCOJSmb13HUIisb9pFAgduPs5aVFDecPBr4mOQLco8PQ7gtaBfywT4CmCrOw7ea6R4G68XbXojiO+9rRaRb5BJNC0/eUbYqN+vh+bFSMFBnYTwTI2xQAXM2UQgockmPopKyxry6edLoOo7SDek46TrN6rkrqmcoN1I8JCQH6FEsn4js3vmfs+Ec3SBOqtLqJXv+8sfK0EBYBTDolCVNL0SgoOowGRgdo3KsIWU9fxcvz5lfk+xpRoej9q1zPvNMn/COFUAlwWFA4yM8F9i0Bzh+V4umbRUgY8B8dWbEchre4u6Xv+5YV32f0AunBnXTGPL9xpaGsVsNn/W7Gylby6PCF0DCg10R1t8t75HwUj22WwmCUFOIV8j4EG3U6SrazIe/aXZMi2suZ8ayaUBlZqXgcrwCEBYUSb9VSXv/0PmnorpXezwEvp0sT1SbK+XYdbonuxuSB8iS+W+SkJs8eloTA8OnSt7sbZ+X6RZ4SZSLVp/8bRWsCmwdWuAoLX2cKq6mLHjNpIlHjvtL9dI9jFMwMmy5NSisJtLxJ3RXb37QYKj0vATXK8T8BaIUMUBwqnw6jgSXyxKEHV0khwd2jW4ZXAOLulAx1DirW9qxSeild6g1rHRWM2yZ5cviDnhOLT+JTo/WtwD+bpd/OIc+1oObzXsOa0ARBzTOHpU5IIDWsTImERxTuXMsqvu6lc67Lj6wFGvSUjG/l3CbO3DYbeOEYcKMPw2dfPReqNZNTZFpECyePqAGveI4uqkTnh4COY+QFH0c5oWWKp4Jse4fk4cEUp3w1FhKMX2BB4Y57PttBbxUPjVC7A3D/siJ1z/3LgYhK0mS2diO0q4Fie0nhLwKNchqDrykRElokKPSow9zj/4RFGPM5ZxjGMKx6cic73XVZS2iEBt2BF3QEZLsvQFNzOgzpvoZdKMThgaPHbmKhrYVhGF3wiKIQhQZqwh7/VyMgkBv6UJ2zlh2+53pGFI48C1Vku48311yGYSyHRxRqBGJAMX+it3PvsLn4mhgHKDKAD3pTyr7WREPkWYZhLIUFhRruhmpGG7XipFhLce6ZbTXQ+SEpZHYDg3MyGIYxDRYUnhgw2e4a2IcjkB2F6jueCOjxpP5jWUgwTEDBgsITHe71vk9JpceTQM5FKa0lwzClHhYUntCan7ckEh4FDH7b7lowDBMgsNeTAytDYDAMwwQxLCgYhmEYVVhQMAzDMKqwoFCiaiO7a8AwDBMwsKBQoslAu2vAMAwTMLCgUOLPT+2uAcMwTMDAgoJhGIZRhQWFEpbnN2YYhgkeWFAo0XOi3TVgGIYJGFhQKBGmM7w2wzC289rP+zDy0812V6NEwSE8lAjh28IwwcqcTUftrkKJg0cUSrinPmUYhinFsKBw4BzrKTTcvnowDMMEGCwolGDVE2MSBQUCefkFdleDYQxhi6AgoqpEtJKIDsnfVRT2iSOizUS0l4h2E9FIv1WQVU+MSdw75080mbTc7mowjCHsGlEkAFgthGgKYLW87E42gHuFEK0BDAYwjYgq+6V2rHpiTGLj4XN2V4FhDGOXoBgG4HP59+cAhrvvIIT4SwhxSP59EsBZANX8UrsQHlEwDMM4sEtQ1BBCnAIA+bu62s5E1AVAOIAjHrY/QkSJRJSYl5dnvHahbKNgGIZxYFmLSESrANRU2DTJx/PUAvAlgDFCCEWroBBiFoBZABAVFSV8rKqjpKKfrHpiGIYpxDJBIYQY4GkbEZ0holpCiFOyIDjrYb+KAJYBeFEIscWiqhaHVU8MwzCF2KV6+gnAGPn3GABL3HcgonAAiwB8IYT4wY91A0JC/VocwzBMIGOXoJgCYCARHQIwUF4GEcUT0Wx5nzsA9AZwHxElyZ84v9TOefIdwzBMKccWq60QIh1Af4X1iQAekn/PBzDfz1VjGIZh3OCZ2UxAc/biVRzLyLa7GgxTqmE/UAesbgpIury1GgCQOmWozTVhmNILjygYxg/sP3XR7iowjG5YUDCMHziXlWN3FRgNnLhwBbEJy5B8ItPuqgQULCiYgOX3A2fsroJpCJ3TQBn/8vsBaUrXN3/+Y3NNAgsWFEzA8sC8RLurwOggPSsHX2/lhrYkwcZshvEDpWlA8cQ3O/HHkXR0aVgVTaqXt7s6uihN/5cWeERRCHs9MYwZpGddAwDk6kjYdOsnm3D37K1mV0kz3Aoow4KCMUzapRzEJizDn0cz7K5KwCIsMFJ8t+0fnLl41fTzGsXhaa7nknf+c4FzeCiQk5ePzOxc28pnQcEYJjFVEhBzNh41/ZyMMmmXcvD8wj24f+42u6viESErcM5evIqV+4LLMSHQnA/unr0V7V/7zbbyWVAwpiFM1Owu2H7ctHMFAma3O/kF0hnTLwee2y3JQwpHYztq1hY8/EViYZ0DmUCdd7st9byt5bOgYAxjRNVQaihF98a9rU1Nv2xLPRjzYEHBBCQ5eb4bQksjgSicS0bHIagrbzosKBwE6pizlLJo5wm7q2AqZqrlgMB+XAsFhds1W2HQNxtivydFWFAwJiDrpG2uRWkkEO+5o7ENArlQDDWBfuD0RVs9j+yEBQVjGLNUDUfSsnD37K24ci3feKUCjC0p5npxBXK/t2hE4Yq/5cbek+bGaxo8bQNum7HJ1HMGCywoGMOY1Wi9sXQfNh4+h80pJc+Pftb6FLur4Dccz4ND1WTXwGLohxt9Psab6ulIWuk0zLOgYJggJiDVO6SsigzIujKaYEHBmAi3BH4jkHVPbhSOMHQ+H8knMpGSlmVehTTAQs0VFhSFBNGb5wdu+mgDvticqmlfMskFh99NHwjgm1WkepK/DZ7vpo82ot/76wyeRRuB7E1mJywoGEWST1zEy0v2+nQM98KYggKBpGMX5CV391j/14cxBxYUjGHM6oRxZ84HAvRm/W9DkdH+wc8T8d22fwK1qqqwUHOFBQVjGkbfLX43g5+j54q8gi5k5+L5hXtsrI3vaBFqJdF92xssKBjDmK3X5dmxvhA84rWk9NJzC0pfeBkWFEwx9HqYOPzmv992DPtPXdRd/sWrpXP2qy8EqjBV6jSUEPlQqmFB4YDdHQpZnHTSp/0dty5PDiP93MLdGPLfDbrLf+rbJN3HMnbj+T3y5B6beSUXsQnLMH/L36pnztORMc8KSsrIyBdYUDCmseFQyZtRbSWZV3KRlZNn6Bzn5LSjgYy7u6wzy/ecwtqDZwEAX25WFxT8fNlHmN0VYEoWViWnSbuUg2oVIiw5t120f/U3hBCQ8vZQn491HgAfy8hGvarlTKyZfnxVPT321Q6n/dSfHbMj8Ooui0cUTKAwfc1hxCYss6dwH8fWzvryV3/2be6FVrb/HbipUdMu5WD38Qved5Q5c/EqMi5LIwEz5Or4r3d43ykAMOwV54cGmjXQyvCIIkB599eDhb/XHjyLBtFRaBgTha0p6YguH44m1StYVvby5NO+HeD0cn3hRX2gRnaOZ7fDw2f9G8LBF4b8dz3OZV1D6hRtI4Pr3lptavlBkGEUgJQ7u3y18h63/3VG/T/W24g/Nn87mlQvj3/f0FzfCRgeUVjBsOmb8H3isWLr5206iuPnswEAmdm5SHXyOc/NL8CZi1cVz3ff3G3o+95aAMDIWVsw4IP15lfaiUM2Nco5KsbKywHsu67VTnAh+5olo8TL14zZOfzFsOnGQnTrHVEsTz6Nj34/bFpZ/lSBqXEsIxvP/LALuX4w8rOgKMS8MeeuYxfw3ILdLuvSs3Iw+ed9uPezPwFIvdA+cuMPAI98kYjr3lqNq7naGsTbZ/yB3w+cMa3ODqyyMWhC5e0sCRoB58loZpKSdhn7Tup3RzYTpf/Jse7SVVeB5o8Gzle0uB0Himvy8wt3Y8H24/jzqPVqWRYUJvB3+mV8sPIvCCHw31WHFPfJlxvBi1fzUFAgcDLTdfSw5mAaAGk2qzN/nbmkeL7Ev8/jyW/MdyM9dFa5PDXMem2uqAjJkqA7tlIEHzgdIILCB2P2hkNpPp27NLqlBgosKEzggXnb8OHqQzh+/gr+b9VfyjvJD/mF7Gto9J9fCldvS3XtDWRecRUUD32e6LFcIiD7Wh5WJJ/SV3EFQmxskdV01IHSiwsU+G5Yi5pMChTVk4PRs7daXgYLChO4misNoVPTvasW8txUOyNmbnaJHfPAvG0u25172e4TjkJDCC8t3oux83cg+YQ5aR9DArQFKhEjCgvbl0DpbXsT6C1fWoGCQLa+B9Fz5s93whZBQURViWglER2Sv6uo7FuRiE4Q0cf+rKMe7pHtD0qcuHDF47ZzWTke90t32pbspocOISo0jpsV9kJPbgmz8lGUfAK4gfQTV3LzAztWkoa/yE6hfP6ysuOEp/VmoUlQEFFjIoqQf/choieJqLKBchMArBZCNAWwWl72xOsA/JO1RCO/7j2Nhi8sw2V5Vq3Q8OTcNuMPw+W+uNg1EmcIOamKTHp49TT5BR6u/5KJMZusEkVXc/NxIds/s5u3pJhndGThzDg7B9z0ke/5wX1B64hiIYB8ImoC4DMADQF8baDcYQA+l39/DmC40k5E1AlADQC/GSjLdD747S8IAfyTIfXmtbTRarJE7Z13HqUfOeuq2iIihIQU388I7jYKtZGQg5xc5R7iT7t8ixmlikUN47CPNyHutZWWnNudb7f9Y9m5T3twrQ4E3J99vfam1QfO+nzMIQ/OIB7RULUMkzoWh89m4ccdx306xvlW7j5epG4+men9PTWCVkFRIITIA3ArgGlCiKcB1DJQbg0hxCkAkL+ru+9ARCEA3gfwrLeTEdEjRJRIRIl5eTp9yn1oiA7L0VUd7n1Gh6Jae4fuPfcQKnrpzDKwuQuKHlN+97leDsx0tbWq/3zQ14bEAGYa5N3PlKPRrdpqrBzofPOn74J24P8VzTk6a5Iw7W9SWtaB/7cOE7/f5dMxnrQXVqvDtAqKXCK6E8AYAEvldWXUDiCiVUSUrPAZprHMxwH8IoQoPnPNDSHELCFEvBAiPizM+snmjgZwW+p5HMvINtyb2/73eU37uTfIaZdyCl9Msx4UPS96oBhSA51A0BblFwi8s+KAi13MTLRcol1eQ+55t7empCMxVVkd6P5Ma1Ev+4qeU9r1qmkVFPcD6AbgTSHEUSJqCGC+2gFCiAFCiDYKnyUAzhBRLQCQv5XGlN0AjCeiVADvAbiXiKZorK9fCCVgzFzPBmytPPnNTpdlT14h7j1050VPvXpfCdHh9hRo7oLBhhmGSK3/wMbD5zBj7RH850f7Ms9NWpQMADh/2b95R9wj9Y6ctQW3z9zsss7T07/jH9fO3NQVB/wyYVAIERDh1TUJCiHEPiHEk0KIb2QPpQpCCCON9k+QRieQv5colDlaCFFfCBEL4BkAXwgh1Izefic0hFTTIh7RmQDomocHQ0l+OFRFZjXVSi+Ktwe1duWyiutz801UPQVAb9wqnvx2p/edvKD19jg6ITl51jQ+WtSoC7ZLevmEH3d72TNwyHN7lj9ZewQLt/tmX3CQm1+ApbuL7HcLtx9Hm1d+VXzP5m/5G00mLS9ctmv0rtXraa3sploVwC4Ac4noAwPlTgEwkIgOARgoL4OI4olotoHz+pWQEFKdoDb8Y32xbXxpFNf9Jc1uNWtorHQ9S3erT+grE6L8GH25OdWEGkmUhAl32R46FQ6nCCNo/vfNdZIzhJkdCbPRMkpWiuemhZYvrcD4r4s6B//+YReycvIUc5Ms2nnCZTnUpolOWlVPlYQQFwHcBmCuEKITgAF6CxVCpAsh+gshmsrfGfL6RCHEQwr7zxNCjNdbnlWEemnRL+lMSpN11ffjzOppKD2HOXnqhlJPt8FTw1haSbukbBf4O924oNBKURKhwG2k7cQXt+Md/2gPLe+M+6RbBx+sLB7Vwb3jpiYoElMzkG1RgEitgiJMtiXcgSJjdgnDd0kdEkKa3Ed9ZeiHvvtEm+ZgpHAbAkBFiqRj2gz+DrakpBez6ZzKvIJjHnrvszek6K6bHehVxVk1/+Lw2Uu6Va2lCU8RogHlEP3FRvgq7/ntMzfj2R+sUedpFRSvAfgVwBEhxDYiagRAOfpdCee0UzA/byMKAMi4fA0pPr5AeryozOohKql49BrKzeyz+tLrXrr7JEbN2oL4N1znR3R7+3f0mrpG8Zg3lu03VL9AI79AKDpFqKUlNcKAD9ab5jZakvH1XXJvYs5cugohBL714Cr82z4fc8loRJMvqRDiBwA/OC2nAPiXJTWyHc+Nf/a1PNzxaZGXhBZ9YcfXV/pFr2inIsHTs29mY5TiQ4jurfIM6PPZ2r1qujaq6nOdApnG//kF8Q2qYMFj3V3WF03kt+aJKSlOBz/uOIGzF3Mw/6HrTD2vr7Y2pQmwJy9cQYIHrzWr7D5ajdl1iWgREZ0lojNEtJCI6lpSI7tRedJHzdriYnjckpKu6ZT+yPFg1ohCqQEJFm326v1nEJuwDCv2uvaqrubmY+a6I6rH+iOmvxbSs3IKvYLU0NLgJCrMzymcoGnRn5pqUc4NO9h4+Fzh76smeYn52mfcetS1jSEUz+vhD7TOTpsLKWTHCHn5bnndQCsqZSvkWXY6T5kHgG+36fN6sAK2TQLT5Fwgzkbj5XtO4bGviueUzssvKMwBAgROOtFxX+/AlpQMXNewKupVLWf6+R39oMRU32w+Wkk6ps/AG8gcOnMJY+YYny8FwGdTqPtzGRpCps2Z8gWtNopqQoi5Qog8+TMPQDUL62UjwTl2PpUZmLF+vv3zH8QmLPNL4L09CqHWlYQEAHy6PgUPf+E514ddHEmTeuS9pq7B/lPakxF5GjG5O1s48pt4mqtjFDvzmZiBe+1X7TvjEgbE+PmN3Z/Y6CiTauIbWgXFOSK6m4hC5c/dALTpXYKNIH3Q1bLDGcZLD8aTvpuoyJPj+Hlrg5b5ipUB+ozgPBpa/5f2DHDOumln4TBqluvMY0ufE9jn528VD3npTPg6gurnlP5YD7n5BfjHj+7UDrQKigcguQhJqSsAACAASURBVMaeBnAKwO2QwnqUIBwvWnA+6O/+ehB/OOlUA4FAVocdy1AXXFk5eVi1z/yc5L7w+R+pHrcpCecNh9Jw6MwlbHWynXm7TneOZWRrCk2x/e8M/J1+Gev/SsOvTjahQB9RZPrg4KCFx+Zv92l/vXOrHLy0JBnvK8y3sBqtXk//ALjFeR0RTQAwzYpK2YKjVQvwB12NpXtOoXuTGLur4YQI2tv53IJd+GXPaax5pg8axnge7n/qxUhuhHNZ15CTl4+0SzmoW8W7vcKROGvCgKaazn81Nx+RZUILlzOv5KLX1DW4s0s9vH1bO8VjhBA4fv4K/jVjs+L2QB9Q/GfRHkwf3dHjdl+fVzWV75aUdFQuVwYtalb07aSQHGAyFGKAbTpsjyLHSIa7iabVIiBwCIrgzQ6r9Iw7en1GMGNgEEijC0899Y9/P1TooeaYt3HZSw/w7eUHTK2bM9fyC3DP7D/R8501+OOI9tHinuPa0uK2eGkF/nIKs+6Y1bv2oOvzknH5Gpq/uByJqRl48PNEj3NRAH1BJf3JgdMXMWfjURcvwdz8AsOB93LzC4qNxEbN2oLB0zboOl+vd35H5zdXGaqTmRhpFQP7ifCR/HxJd2tRrDS/oDTsv/7dtbjXzWMjNmEZRn66GWcvXS0mRJQadG+N/JIk5QRF57KKekSeemppl3K8hggxm1d+2qu4/r3f/sIve6S4Vo576Q/XZjX+lMNg3/W/rZobs8s+hHG4wclQ6zC0nsq8ihXJReqkbakZyMkrwNQVB/G7l+RBVque9p28iMNn9c8AP5J2Ga8t3YeGL/xSuK7j6yvR4fWVyM0vwDwVdZ8avaeuQeuXf1XclnklF4OnrXcRymosSTqBkwHmnGJEUARQH9E4WfnhyBMheDP/bruropttbrH1nRu5MXP+RNKxC5i+5jAAYOvRDHR5c3UxITJAx+zazzYe9bjN0ctyTtX44LxtiE1YhszsXHR+cxXGfrkdsQnLfC7XmXST8is4co87esb5KlJyl59dQc/Khu41B8/iP4s8hwl3j7G15uBZTcLYeTAwdv72wuxrYfKGPz3kbnDGjACHatz44QYM+MDcGeCXrubh0tU8fL31n2Iu8Fo5lXnVoydZ+1d/w4HTl1yEsieEEHjq2yRddbASVRsFEV2CskAgAMqxpYOV0FA0yZmPCpFheMXuuujkwOlLWLzzBJbuPolb4upg3qaiBnzdX2mFkWaVEEJg0uJkw8Y2dzKvuBoPn1uwqzClpaNRWXPQmGoMMCdUNyCNjkZf1wDX5KHlNQ9DzLz8Agyb7hod+HJOHqIijCXOCgshj0Hjuk/5HZ+M7ojHPbj8OnBv7O6fuw0AkDplqOL+3ycewx3x9Ypl+5v4/S4s231KVwrSYMQMwb/mwFk0iC6nO9Cj80gnkFB9qoUQFfxVEcYcJnwn9UZW7fft5c7JK8DXW313GX196T7V7e5t3veJRbOOb/7YvITwB06Zk9K0fEQYZq47UjiH4e3lB7BkXA+s+ysNrWtXREz5CADqM3Vz8vIRERbqcbsa3obp3oQEANSvWk6xZ+9Qq7nz3ILdkqA4XfweBqqQyMrJQ3mDQtmdH91CemvFuTNx/7xtZlUnoAhey60fWJF8Ck3+8wtOXrjil2xWdvHTrpNo8dIKj9vzCgSu5RUUU22lnrusqnYCXLP1GVUvAZKnTkpaFhIW7nbKWS6QbkKWOEAaeU1xMlDvOnYB1/IKMGbOn4h/YxViE5YhL78AbV4pro9u/cqvaPXyCjR/cQW2/52hK6yKGTYRT+ofb0KmWoUIw2VrYUQn49F/nG0oALBo53E8/pVvrqpmcd1bgWN0tgrrE0wHC8LtG8DY+dKL9eay/WhXt5L/6+Qn3FOxuvP60n04lpGNeX+kYvlTvdCyVkX8kHgMzy7wHtLYzHADF7KvIe61ooiw3247hjeGt8GLi5MNnbd/i+qFPWelhrrZi8tdltXCwDvsA57cR71RrUKEx7wVVjJp0R58pWNEqQct6rndxy+gec0KHkdmBGDT4XOYs/Goy6jHU1RVK/El+KQe/jsqzna7BY8oZNSiaSafzLTUDTIYcHhsZFy+hi0p6ZqEBGDuS6QUxtqokACkHA0LH+umeX93Xb6Z2JVQyF9CwkGiF8P4LR9vQvMXV3jMHw8Ao2dvLaYa8xRVNRjZ8kJ/TBjQFLe0r213VVhQOHA8jwJSPgNnNYk/M5CZwcd3dVBcb8TF3eH2uPt4JkbN2qL/RAYwS73kTmhI8ZzI/saRVtMqb9xGMVH49pGuPh/Xto75I+lr+QU4p9FLzZMDxjIP9paSQq+mMahZKRITBjQDESEizN6mmgWFjENFQoBLPttgI3XKUNzUrjY6x1Yptu2T0Z1wX/dYQ+d/Z0XJG1kRCPWji898jikf7rc6PLdgN/48mqE4G9cMnh/SAl0bRVtybl/5eus/+HGHNsPx4iTl/bzN5/CVF4e2xDcPd0WrWr7PoraCaSPjXJafHdTcpppIsKCQiY6SGoWRnevZXBNz+N+98cXWDW5TE6/c3ErX+bQY8ze/0E/Xue3m4tVc1KpUFqlThrp8El/0bxR956RYZjOodU3dx/Zqan5YmN80xtHyNJnTbB7q1QjdGkfj03s6GTrP1NvbYcptbXH4zSGGzhNd3tWxoKXNAowFhYxjeBcaWjImnFcup9wb1pszeauGxD61KgXe1Jon+zfF430aq+4TFur5NUidMhRP9dcWO8mdOpXNvR+NYqJQIdK//idEwCejO+KHsa42nB/GdkO4yn0LVtwnK3piaNtaxdb9OqE37oivh1Fd6is+U50aVMGg1jV01auHzTHcSt4/bQAi4Gc/9WCsoEn18i7LM+/u5JORtiRyc7taGONF3ebNdvP0wGY+lTmwldQYvHRTS5+O80aBEH4LKTKub5FwrRBZBp1jXVPFdo6tioE6Gz2zeX14G0y9vR12vjQQ2yYNMHQurTP8wxVsBs1rKk87+/7RbtjwXF8sfKw7hsfV8XruR3o3Ulz/8k36tAFmwILCiRCigIux4guhbqOFwW1qolODqkh6eSAOvD7YplppZ9fLN5h+zujyEahRMVJ1HzMdjT6+qwOeH9wCHetXRs+m1TD3/s6mnTs1PRu1Kqlfi1kMbKVBVRUgQXz6t6iOO+LroUpUuOG5IGqjS2d8GZd38SFb4b7XBiFhcAvFbQ/0bOj1eL0jFm+woHAi2JVOniJ3Vi4X7hJOeuz16qoYrYzoVBd3d62P2zrU0aQSUQrXfeStGwEA4/s2QaVyZXTVY/lTvTxuqxpljkG6hYfeojs3tauNJtXL48fHe5g+c7hO5bL46qGumKESJttslJ6oRY93B2DuHBm99GgSjdomqvi0Pi/uKtymbqN5vZQLD9MdgfeT0R3x6T3FbZNmwBPunNCrvw8UtD5fCUNaeEyd6Qu9mlXzycd7zTN9cP7yNURFhLkM3T3FIPLEm7e2QffGMfgp6SRuaF3DL4a+T+/phPlb/sb/NhTNRH92UHO8++vBwmWzBLASj/ZuhId7N0JM+QgMUdCP+5MO9SWPuhHxdbHcbYb03V3rY/4W/83J+PKB60w9X5Pq5fHZmHh0alAFyScu4u7Ptiru595UeFIX+ROjccbU4BGFE0EuJ0xJQ+mLjlfPRKAqUeGK+l1faBgThYYxUXhqQFNThISWfnGD6ChMGuqqI368T+NCe0TlcmWQMERZZaCHbrIr60d3dsCqib3xwo0tC+NM+Yovuu0hbWqii5s9whP9WtRwEfJVypVBn2bVfa6fEczIf7Hy6d4uy/1b1kDlcurPqXupt8RpexesbGOsHOHxiMKJIJcTpuQC8Fe8HyOUCRBvGyLCW7e2xcp9Zzw+O1U8eJ954xsdk+OU8HW0NuNuyT105z/nfS5r58s3aE6aFEg0raGsVqysogolAube1xn3z9uGPyf1Vww1Mq5vY3Rr5O6tZF0rY+Ws/sB44wKEQM/O5Q291W+kkupTifb1KuOzMdboQu1Az20bfV19AEWjOE+vaFy9yvoqpZMO9U0uT0PnY/LNrdCmjjSya1u3Ej69pxP2v2a988SPsq3EKpp5ECCANEmzb4vqSJ0yFNUrKDsYPDuoBXpaMAfFE02qWRfsmwWFE8EtJvRFHn1hSAssGtcDANC9saTu2PBcX4/739ahDj6/vzP6t7TPNVKP4bCmF88nX3nz1rYAgDLyvJuWOvIie8LI/IuJPrryOtO7WbXiKzX0Uu/r0RBLnyhyKBjUuibKhusLs66VN4a3Qcf6xaMP+IraqEENvYN3K1VPStEFzIJVT04EuzFbz1PYunYlVCpbBuuf7VuodlJz5fvALbSAHXiaTKiG2q3xJR91vaplcSzjSuFyhcgyUuiH2uYJipUTe3vfyQPOc2lm3q0+y/jGtjXxy54iY/T9BsO7uHN/j1jM3ZTqcfttHetoDuXhjK/qNDW0zGtQwgp7wK8T9P/vVsMjCieCXPNUmLLSFxxRc+tHl7O8Fxio5PoQEHD1xD7F5qR0axyNSmX19UyVKBeuv/9WzcngPbiN+lyI/xsZh6HtJA+qlLduRN8WCoZoA52nV25ujY4qqjAtNrVyBp7J5U/1UkwPEFM+HPENpNHIo9fr81a6kmt+fhr3CbOBBI8onDiXVRSQrXvjaPxxJN3G2viO+4Q7NcLDQnAtr8CyaKWBxlP9m5oSgtqox5bVaJ0wBgARYaGYfldHTL/LwgqpoKVTPr5fE0xdcdD7jk78OqE3jqRloWWtiujZJKZYatgfH+thWE2jt1MZrH3RwH7qbeTebrF2V8FnfOn8OXTZdSqbq7uPtVBPaoRRXerbVvYHd7T3a3nTRsbhm4fN8Zoyk+cG+x4BtWKk60itRkXvXnnNa1bAjfJck3/f4Fpm9QoRpujyzfAwDCZYUHhgcJuaGOJl6B5o+DI4eLR3I2x8vi+aVDfXU6KUDFB8wt8qheEd6qBbY2Mhxc36H53tfr50vvo2r4ap/2qHO7vUd5nn0Liab/fSjLlFSugVFGp20EAWPbaonoioKoDvAMQCSAVwhxCimOM2EdUHMBtAPUjP7o1CiFR/1fPjuzoiN79ANZ90IHHi/BXvO8kQEepWMb/370mdYEWo6mCBAroJUMdozR2+/X2bK3hUqTD3/i6Fv5vWqICjb9+I6WsO4454Y2kAzDJCs+rJPyQAWC2EaApgtbysxBcA3hVCtATQBYC52Uq8EBpCiCwTiif7NfFnsboNo1b1nsygbpXAC0HuL0qZlkKR8f2aGGokiQjj+zVFdYNuzmbZ5Erbf2qXoBgG4HP59+cAhrvvQEStAIQJIVYCgBAiSwhhS07Snk196w0ZZVOCvgRAASwnELx9qdKJFZN8A0Etad6IQq/qyZTi/Y5dgqKGEOIUAMjfSgFimgG4QEQ/EtFOInqXiBR95YjoESJKJKLEvLw80ysb36AKnuzXBH/obMB9pXxEmOZopc4EQhJ2YbA5GNDSv7GC/IEvjcqDGkJJ+xOzGjYh9E0INZs3h7c15TxBP+fKRywTFES0ioiSFT7DNJ4iDEAvAM8A6AygEYD7lHYUQswSQsQLIeLDwoybXdzzTYeEECbe0By1K5f1W5Jzd28NLVgdRVXJJ92dAg/u5VrfKy0x97XiLbOdA6s9tXxpU8JKSIZFB84NaqWyZfDBHe1x13X2eaA55o0YxQo5YUSMWh2jzbJWTwgxQAjRRuGzBMAZIqoFAPK3ku3hOICdQogUIUQegMUA/BKIX63n89QA17SY/VpUx/MeEo040DM68NRgdGrgOWyB1b2ct27V3xuL0Rjnv3tj84zenRtqi4KqNyqrEdrWqYRvHu5aLFKr1lGhUTWj95Am5vT+He+S49m8rWNdVJcbNaOjT63Eq7wz/kbtFTVkwzFwrBbsUj39BGCM/HsMgCUK+2wDUIWIHAaCfgD2+aFuLkl+3HGf1NY5tioe7qXeC14xoXdh2Ggz+FfHuorrK1qcT9mIend8P315pw1hv6YDQHHVU7UKEfj5iZ7o1jga34/thln3SKE2aleKROva3kdtAFDHoHOA1kbaaAPkyNPQtEaRW+ug1pLbub884czMMujAiobZSFBSqx91uwTFFAADiegQgIHyMogonohmA4AQIh+S2mk1Ee2B9N/8zx+Ve3aQZ7WPezTQm9vXUp0N+/4IabKVt96tL5nYoiKUBVmkiSE4lALTaWlcPIU69teMZufeo1bDpdXqZm/nd6gMgz16sRI3tq2F1ClDXSbOtaxVEUfeuhHD4+q45Oa2igqR5oVXcaA7KGCQOnXYIiiEEOlCiP5CiKbyd4a8PlEI8ZDTfiuFEO2EEG2FEPcJIa55Pqt5qD1Y1zWKxu2dinr03uYi/Eve96n+6j1qrbHkCdZ4pLgzqnNxf3Utjb3dnfgFj3XHi0NbAlAPbuhMoLy8vti/jCYI0voMWfV/hoYQiAjPDmrh91DsZqD7mZEP83fUWqPwzGwFGldTz8+glPvZmRcUMp2pzXHo27wa2tTRpnIAPPfszXyGHu7dqFjD1UJDKG1v98YfPNizITa/0E81n4A/cR8tuv9PdauUxVP9m2LOfdpVJFpylKsRAA5IhQSjA5HROrevq0842nWrWFAooMco3NopzHRbHxp9QBqKz3ZLBKRWA0+hEMw0ZkeWCcUTOiYa3ubBfuJPiAi1KhVXnTWwKQ5VTPkIbHmhf+Gye4Y+IsLTA5uhQbR2IeuvxtUfxbiPKL41KbuflRiesa7zOLtiTLGg0IGSmkhLI7Rt0gC8MbyN4jalVIpKvD68jceestmPUF4gdTtl1MJWe8NTQ1xJpxrAF2pWisQIWQ15b7cGhs8XKOoyM7jeLWFSVxMdPwINx7+mN22ps6Dwlm/ETFhQ6OCW9sWTndR26sG296BzrVYhAnd3Ld5IaO0lPDuouepcCbM7G/6whfiKEcOkkvYvpnw43r29nYEaaae6HPlUr37aXNT/XH/+98E4ec22OjsV68/niAWFDpTCFD/rFEI5KiIM7etWwkiVAGa3tK9dOL/CkVe3XHho4YQgpQfR2WagJDDM7mWambXNLNzVNr6gJJBn3N1JV8Y8PTzRryleGNLCo3uzLxhtp7QKgmBsxAMZo/fT2T3f+UzT77J2ihknLtLJxIHNkH0tv3DZoTpyGBmXjO/p8VhHKse8/ALsPpFZmPt3n5eE9Pc4qSz6taiG/acuumwvDe+0nix+Duz2Po0sE4pHrzfHHdQuHbkV+ONvmWfyXArdsZ7kb70jNmenGOdTdNE4uVQvLCh08qSCu+u2SQMQUUZ7jzcsNERzgvjnB7fQbMcoyeTmG0lBWfzlDkT1miYM9gq86ciD9bZ4ok/zwIghZrQzx15PJYBqFSKKZeTSSxm3EB7uywNa1jClHDUCsRFdfUB/pHmlEYVeo2Kwo/WqnW+ZVZMmg3EkbFj1p1MUO5frz9vGIwonYsqHu+TNtpNujaLx7KDmOJKWhR93nCi2vYPCSMTKF65fC209skBueJXUBYFbW3XsaFs3PtcXaVk5NpQceNjWs7dJf8ojCidW/7uPi7+7nRARxvVtgspltRtafQkD4itKM7Wtwjm6aLSJ19RWIfptAMs1VfxlzHamekXtsah8IRhdfY2G8ND73Cl1dsqqxKYzCxYUTlQqWwY1KxnLoGU2T/Zvgts61MGdXbyHZlaaZGYMe1pR5yi1ZnpePaZgSPZXBFOzMdq4NvWSxztYBai/0GvMdgRH1Ov5plRqmzrWeyeyoAhwKpcLxwcj4xAVUVxLuPrf12PhY91tqFVwojhsL6UN4iQ5HpY3/GE/CEYbhV5qVIxE6pShhTHgfMUud2UWFEFM42rlVfNTGMWsXqUR18Txfa3NVx6sckJve+Fwo7QjB0eJwq7gfDaVy8ZsxiO1nUKN926mP294XQO5E4zmXfBGsKpY9LYXXzzQBftOXfQaWdcRyl4p3LyVlDMxVL6V2GVXcVZ5+XN0wYKC8Uj7epWx7MmeaFmzYsDnSmhULQopaZe97te9cTT+OJLuhxpZSwudaW8jy4RqmrvTunYlzBjd0VAHQSvOT9aKp3pbXp4Z2BbBw6lch7typbLWh/JgQcGo4quXi9kddK29ph8e7YYjGgSF+1yAYDVmD2xVA6smXo8BH6yzrIwhbc3JL62Vro2qKobHCURsc491eh/a162EV25uheFxxWPPmV6u5SUwfkFPXm5/EetD+Gy9RJeP0BXGIFhVTwDQxIvnUrDgEN5WZKIzypcPdvGa6tifxFQosi0REe7v0RBVLHSLd8AjihLAD2O7oUm1wG001FLF+ht3wVDRD8N2Rp1ODapg0o0tXTJHBgq9mlZDr6bV8L8NR13W26V6qmgwYZVeAucNZnTTObaqX3oVdmD1+xiMaTidMZKfI1AgIjzcu1FQPcOBYMz2a7m2lMqUWAaaHIPKfJuHySe0mcf6FLkPf3RnBxtrUrqw6zmyK0sjCwrGVKpEhdv2MGvByjAndnNz+9p2V6HUYFd/g6PHMiUGMw3EZr8YTasHrtGfCSJsHppOHNjMr+WxoGBMJ9KHnBz+xkjiI4YJFPxt1A7cN5oJWjrUsy6siFGCdd6EJwI5rHtJxuzuhlW5PswisGvHBCVmjspLmvGZKRmY6X1ULjwUO14aaNr5rIAFBVOqCMbcB2rweMIezJATKyb0AiCF4CivEB06kGBBwZgOjwL8z8BW1qfGZYow4xHXkzZ5UOuaAIDrGkWbUAPtBLYYY0o9JW0EYBVsqgg+9Pxl3ZvEIHXKUNPr4g0WFAxTisnNzcXx48dx9epVu6sS0PzvFtcAiZXKXsH+/fsNnTOvoAD/u6UWwkLI8Ll8ITIyEnXr1kWZMtpHNCwomIAmwmRvkOubV8Obv/jvpfQXetV9x48fR4UKFRAbG2tb9rRgIPf4BZflmhUjUb2isbTJ1/LyIU5fQnhoiO6w8b4ihEB6ejqOHz+Ohg21BztkGwUT0FSJCsc3D3c1zdjXrEbJmnBnVOV09epVREdHs5DwlSC9XUSE6Ohon0eQLCiYgKdb42gMbiMZ8UZfV9/m2gQmRtotFhKlCz3/NwsKxnTy8s23rEbJKTJLSg4G82ArdrDimItRLjzwLQAsKBjTqe+Uj5lHAIw3zpw5g7vuuguNGjVCp06d0K1bNyxatMi2+qxduxZ//PGH4XPcdNNNqvuEhYagafXyhnLK+wtbBAURVSWilUR0SP5WjPlARFOJaC8R7SeiD4nHyEHBY30aF/7m/i6jhhACw4cPR+/evZGSkoLt27fj22+/xfHjxy0tNy8vz+M2LYLCvSFSO58aZcPDAj4fPWCf11MCgNVCiClElCAvP++8AxF1B9ADQDt51UYA1wNY68d6Mjpwzmin19gaUz4cvZtVK7Y+8F8pezCjC/Xqz3ux7+RF4ydyolXtinjl5tYet//+++8IDw/H2LFjC9c1aNAATzzxBAAgPz8fCQkJWLt2LXJycjBu3Dg8+uijWLt2LSZPnoyYmBgkJyejU6dOmD9/PogI27dvx8SJE5GVlYWYmBjMmzcPtWrVQp8+fdC9e3ds2rQJt9xyC5o1a4Y33ngD165dQ3R0NL766itcuXIFM2fORGhoKObPn4+PPvoI9evXx8Oj78X59HOoEh2D197/GHWrNMd9992HqlWrYufOnejYsSPef/99xWvMyMjAAw88gJSUFJQrVw6zZs1Cu3btsG7dOjz11FMAJLvB+vXrkZWVhZEjR+LixYvIy8vDjBkz0KtXLxP/EX3YJSiGAegj//4cUuP/vNs+AkAkgHBI7UMZAGf8Uz3GLPQGrUt80TX2DY9MSiZ79+5Fx44dPW7/7LPPUKlSJWzbtg05OTno0aMHbrjhBgDAzp07sXfvXtSuXRs9evTApk2bcN111+GJJ57AkiVLUK1aNXz33XeYNGkS5syZAwC4cOEC1q1bBwA4f/48tmzZAiLC7NmzMXXqVLz//vsYO3Ysypcvj2eeeQYAcPPNN+Pmf43CLSPuxIqF3+CdlxOwaPFiAMBff/2FVatWITQ01OM1vPLKK+jQoQMWL16M33//Hffeey+SkpLw3nvvYfr06ejRoweysrIQGRmJWbNmYdCgQZg0aRLy8/ORnZ1tyn02il2CooYQ4hQACCFOEVF19x2EEJuJaA2AU5AExcdCCEUHeCJ6BMAjABAeXnIT0zCMO2bOyFbr+fuLcePGYePGjQgPD8e2bdvw22+/Yffu3ViwYAEAIDMzE4cOHUJ4eDi6dOmCunWlPNtxcXFITU1F5cqVkZycjIEDpY5Gfn4+atUqmiw3cuTIwt/Hjx/HyJEjcerUKVy7ds3jvILNmzdj8oeSoLn1jjvx9qsvFm4bMWKEqpAAgI0bN2LhwoUAgH79+iE9PR2ZmZno0aMHJk6ciNGjR+O2225D3bp10blzZzzwwAPIzc3F8OHDERcX5+sttATLbBREtIqIkhU+wzQe3wRASwB1AdQB0I+IeivtK4SYJYSIF0LEh4UFvgdBacKshqypPP+hfgBnz7OTYA110rp1a+zYsaNwefr06Vi9ejXS0tIASCPSjz76CElJSUhKSsLRo0cLRxQRERGFx4WGhiIvLw9CCLRu3bpw/z179uC3334r3C8qKqrw9xNPPIHx48djz549+PTTTzXNLYgoEwIiKsxr4nw+TyiNqokICQkJmD17Nq5cuYKuXbviwIED6N27N9avX486dergnnvuwRdffOH1/P7AMkEhhBgghGij8FkC4AwR1QIA+fuswiluBbBFCJElhMgCsBxAV6vqy1iDWfkf7r6uPhaP64F+LTj4XUmiX79+uHr1KmbMmFG4zlndMmjQIMyYMQO5ubkAJFXP5cuXPZ6vefPmSEtLw+bNmwFIIUr27t2ruG9mZibq1KkDAPj8888L11eoUAGXLl0qXO7evTtW/CSNaMvErwAAFeZJREFUCFb9vBA9evRAlA8TQHv37o2vvvoKgGQoj4mJQcWKFXHkyBG0bdsWzz//POLj43HgwAH8/fffqF69Oh5++GE8+OCDLkLUTuxyj/0JwBj59xgASxT2+QfA9UQURkRlIBmyS17shRKOWSMKIkJcvcrmnIwJGIgIixcvxrp169CwYUN06dIFY8aMwTvvvAMAeOihh9CqVSt07NgRbdq0waOPPqrqYRQeHo4FCxbg+eefR/v27REXF+fRg2ny5MkYMWIEevXqhZiYmML1N998MxYtWoS4uDhs2LABH374IZZ8/zVuH9gDX82fj+kff+TTNU6ePBmJiYlo164dEhISCoXStGnT0KZNG7Rv3x5ly5bFkCFDsHbtWsTFxaFDhw5YuHBhobHbbsiODFlEFA3gewD1IQmEEUKIDCKKBzBWCPEQEYUC+ARAb0i2zBVCiInezh0VFSXUehyMf4hNWAYAuL1TXbw3or3NtXHFUTcAtkTiNJNf9pzC41/twODWNTHznk4+H79//360bNnSgpqVLHbLsZ7a1S0ZnRWl/52IsoUQiro0WxT6Qoh0AP0V1icCeEj+nQ/gUT9XjTGZAo5/7Rd4hpG1lAsPQ/Y1fXMlSgJs+WWsheWEpbAc9g+NYqJKdaeHBQVjLQHc0+3TvPiEvmCFRxTWEhJCCAnkh9liONYTU2r58M4OdleBYYICFhRMqUVPzmKGKY2woGAs5ekBzeyuAsMwBmFBwVhKvao8k9pKzJrQaCehoaGIi4tD69at0b59e3zwwQcoKCgAACQmJuLJJ580XMbMmTN9nuXcvXt33eXNmzcPJ0+e1H08IM2/eO+99wydwyzYmM0wJYBgDeEBAGXLlkVSUhIA4OzZs7jrrruQmZmJV199FfHx8YiPjzd0/ry8PJfotFoxkpNi3rx5aNOmDWrXrq35mPz8fK9xo+yCBQXDBDGmemwuTwBO7zHxhABqtgWGTNG8e/Xq1TFr1ix07twZkydPxrp16/Dee+9h6dKlimG5K1SogKlTp+LLL79ESEgIhgwZgilTphQLKX7p0qXCiLB9+vRBhw4dsH37dqSlpeGLL77A22+/jT179mDkyJF44403AADly5dHVlaWakjz1157DT///DOuXLmC7t2749NPP8XChQuRmJiI0aNHo2zZsti8eTP++OMPPPPMM8jLy0Pnzp0xY8YMREREIDY2Fg888AB+++03jB8/HqNGjVK8L0lJSRg7diyys7PRuHFjzJkzB1WqVMGHH36ImTNnIiwsDK1atcK3337r8T4ZgVVPDFMSCN4BRTEaNWqEgoICnD3rGgLOEZY7KSkJGzZsQNmyZbF8+XIsXrwYW7duxa5du/Dcc88V7u8IKf7vf/+7WBnh4eFYv349xo4di2HDhmH69OlITk7GvHnzkJ6eXmz/nTt3Ytq0adi3bx9SUlKwadMmAMD48eOxbds2JCcn48qVK1i6dCluv/12xMfH46uvvkJSUhKICPfddx++++477NmzpzDPhIPIyEhs3LjRo5AAgHvvvRfvvPMOdu/ejbZt2+LVV18FAEyZMgU7d+7E7t27MXPmTI/3ySg8omCYICYiTOrrVfAhSJ1HfOj5W41SaCGlsNyrVq3C/fffj3LlJFtY1apVC/d3Dinuzi233AIAaNu2LVq3bl0YirxRo0Y4duwYoqOjXfZXCmnes2dPrFmzBlOnTkV2djYyMjLQunVr3HzzzS7HHjx4EA0bNkSzZpJjx5gxYzB9+nRMmDDBaz0BKXjhhQsXcP311xceP2LECABAu3btMHr0aAwfPhzDhw/3eJ+MwiMKhgliBrSsgecHt8CkoSUnXlNKSgpCQ0NRvbprmhqlsNxCCHjKkKwWAtwRojwkJMQlXHlISIhi0EGlkOZXr17F448/jgULFmDPnj14+OGHFUOVe4unpyVUuSeWLVuGcePGYfv27ejUqRPy8vIU75NRWFAwTBATEkJ4rE9jVCghc0LS0tIwduxYjB8/vpgAUArLfcMNN2DOnDmFockzMjL8VleHUIiJiUFWVlZhciXANVR5ixYtkJqaisOHDwMAvvzyy8LRgRYqVaqEKlWqYMOGDS7HFxQU4NixY+jbty+mTp2KCxcuICsrS/E+GYVVT4wlvDCkBeJjq3rfkSn1XLlyBXFxccjNzUVYWBjuueceTJxYPFD0tGnTsGbNGoSGhqJVq1YYMmQIIiIikJSUhPj4eISHh+PGG2/EW2+95Zd6V65cGQ8//DDatm2L2NhYdO7cuXDbfffdh7FjxxYas+fOnYsRI0YUGrN99cL6/PPPC43ZjRo1wty5c5Gfn4+7774bmZmZEELg6aefRuXKlfHSSy8Vu09GsSXMuJVwmHHGG0uSTiA6KgI9m8Z437mEw2HGSydBEWacYexkWFwdu6vAMEEF2ygYhmEYVVhQMEwpp6Spnxl19PzfLCgYphQTGRmJ9PR0FhalBCEE0tPTERkZ6dNxbKNgmFJM3bp1cfz4caSlpdldFcZPREZG+jwJj72eGIZhGFWvJ1Y9MQzDMKqwoGAYhmFUYUHBMAzDqFLibBREdA3AbgOniAFwzqTqBCul/R6U9usH+B4Ape8etBNChCttKHGCwihElCiEMJZSK8gp7fegtF8/wPcA4HvgDKueGIZhGFVYUDAMwzCqsKAoziy7KxAAlPZ7UNqvH+B7APA9KIRtFAzDMIwqPKJgGIZhVGFBwTAMw6jCgkKGiAYT0UEiOkxECXbXxyhENIeIzhJRstO6qkS0kogOyd9V5PVERB/K176biDo6HTNG3v8QEY1xWt+JiPbIx3xInjLc2wgR1SOiNUS0n4j2EtFT8vpScR+IKJKI/iSiXfL1vyqvb0hEW+Vr+Y6IwuX1EfLyYXl7rNO5XpDXHySiQU7rg+K9IaJQItpJREvl5VJ3DwwhhCj1HwChAI4AaAQgHMAuAK3srpfBa+oNoCOAZKd1UwEkyL8TALwj/74RwHIABKArgK3y+qoAUuTvKvLvKvK2PwF0k49ZDmCI3descA9qAego/64A4C8ArUrLfZDrVF7+XQbAVvm6vgcwSl4/E8Bj8u/HAcyUf48C8J38u5X8TkQAaCi/K6HB9N4AmAjgawBL5eVSdw+MfHhEIdEFwGEhRIoQ4hqAbwEMs7lOhhBCrAeQ4bZ6GIDP5d+fAxjutP4LIbEFQGUiqgVgEICVQogMIcR5ACsBDJa3VRRCbBbSW/SF07kCBiHEKSHEDvn3JQD7AdRBKbkP8nVkyYtl5I8A0A/AAnm9+/U77ssCAP3lEdIwAN8KIXKEEEcBHIb0zgTFe0NEdQEMBTBbXiaUsntgFBYUEnUAHHNaPi6vK2nUEEKcAqRGFEB1eb2n61dbf1xhfcAiqxA6QOpVl5r7IKtckgCchSTgjgC4IITIk3dxrnPhdcrbMwFEw/f7EmhMA/AcgAJ5ORql7x4YggWFhJJeuTT5DXu6fl/XByREVB7AQgAThBAX1XZVWBfU90EIkS+EiANQF1Lvt6XSbvJ3ibt+IroJwFkhxHbn1Qq7lth7YAYsKCSOA6jntFwXwEmb6mIlZ2R1CeTvs/J6T9evtr6uwvqAg4jKQBISXwkhfpRXl7r7IIS4AGAtJBtFZSJyZLd0rnPhdcrbK0FSX/p6XwKJHgBuIaJUSGqhfpBGGKXpHhjHbiNJIHwgpYRNgWSkchikWttdLxOuKxauxux34WrEnSr/HgpXI+6f8vqqAI5CMuBWkX9Xlbdtk/d1GHFvtPt6Fa6fINkNprmtLxX3AUA1AJXl32UBbABwE4Af4GrIfVz+PQ6uhtzv5d+t4WrITYFkxA2q9wZAHxQZs0vlPdB97+yuQKB8IHm8/AVJhzvJ7vqYcD3fADgFIBdSr+dBSLrW1QAOyd+Oxo4ATJevfQ+AeKfzPADJcHcYwP1O6+MBJMvHfAx5ln8gfQD0hKQG2A0gSf7cWFruA4B2AHbK158M4GV5fSNI3lqH5QYzQl4fKS8flrc3cjrXJPkaD8LJsyuY3hs3QVEq74HeD4fwYBiGYVRhGwXDMAyjCgsKhmEYRhUWFAzDMIwqLCgYhmEYVVhQMAzDMKqwoGAYN4goS/6OJaK7TD73f9yW/zDz/AxjBSwoGMYzsQB8EhREFOplFxdBIYTo7mOdGMbvsKBgGM9MAdCLiJKI6Gk5wN67RLRNzlfxKAAQUR8578XXkCbqgYgWE9F2OQ/EI/K6KQDKyuf7Sl7nGL2QfO5kOb/FSKdzryWiBUR0gIi+cuS8IKIpRLRPrst7fr87TKkhzPsuDFNqSQDwjBDiJgCQG/xMIURnIooAsImIfpP37QKgjZBCUAPAA0KIDCIqC2AbES0UQiQQ0XghBelz5zYAcQDaA4iRj1kvb+sAKYTESQCbAPQgon0AbgXQQgghiKiy6VfPMDI8omAY7dwA4F45bPdWSKFAmsrb/nQSEgDwJBHtArAFUtC4plCnJ4BvhBTt9QyAdQA6O537uBCiAFIYklgAFwFcBTCbiG4DkG346hjGAywoGEY7BOAJIUSc/GkohHCMKC4X7kTUB8AAAN2EEO0hxVuK1HBuT+Q4/c4HECakXAldIEXGHQ5ghU9XwjA+wIKCYTxzCVIKVQe/AnhMDl0OImpGRFEKx1UCcF4IkU1ELSBFl3WQ6zjejfUARsp2kGqQUtn+6alico6NSkKIXwBMgKS2YhhLYBsFw3hmN4A8WYU0D8B/Ial9dsgG5TQopz5dAWAsEe2GFGl0i9O2WQB2E9EOIcRop/WLIOXe3gUp4u1zQojTsqBRogKAJUQUCWk08rS+S2QY73D0WIZhGEYVVj0xDMMwqrCgYBiGYVRhQcEwDMOowoKCYRiGUYUFBcMwDKMKCwqGYRhGFRYUDMMwjCosKBiGYRhVWFAwDMMwqrCgYBiGYVRhQcEwDMOowoKCYRiGUYUFBcMwDKMKCwqGYRhGlRKXj4KIVkDKOayXGADnTKpOMJRrZ9mlrVw7y+ZrLh1lGyn3nBBisNIGzkfhBhElCiHiS0u5dpZd2sq1s2y+5tJRtlXlsuqJYRiGUYUFBcMwDKMKC4rizCpl5dpZdmkr186y+ZpLR9mWlMs2CoZhGEYVHlEwDMMwqrCgYBiGYVRhQSFDRIOJ6CARHSaiBIvLmkNEZ4ko2WldVSJaSUSH5O8qFpRbj4jWENF+ItpLRE/5o2wiiiSiP4lol1zuq/L6hkS0VS73OyIKN7NctzqEEtFOIlrqr7KJKJWI9hBREhElyuss/5/lcioT0QIiOiD/39388D83l6/V8blIRBP8eM1Py89XMhF9Iz93/vifn5LL3EtEE+R1llyzL20HSXwot2m7iaij3nJZUEBqRABMBzAEQCsAdxJRKwuLnAfAfWJLAoDVQoimAFbLy2aTB+DfQoiWALoCGCdfp9Vl5wDoJ4RoDyAOwGAi6grgHQD/J5d7HsCDJpfrzFMA9jst+6vsvkKIOCffdn/8zwDwXwArhBAtALSHdO2Wli2EOChfaxyATgCyASyyulwAIKI6AJ4EEC+EaAMgFMAoWPw/E1EbAA8D6ALpPt9ERE1h3TXPg/a2YwiApvLnEQAzdJcqhCj1HwDdAPzqtPwCgBcsLjMWQLLT8kEAteTftQAc9MN1LwEw0J9lAygHYAeA6yDNIA1T+g9MLrOu/AL1A7AUAPmjbACpAGLc1ll+rwFUBHAUsrOKHc8YgBsAbPLjNdcBcAxAVUgRJ5YCGGT1/wxgBIDZTssvAXjOymvW2nYA+BTAnUr7+frhEYWE4yFzcFxe509qCCFOAYD8Xd3KwogoFkAHAFv9Ubas+kkCcBbASgBHAFwQQuTJu1h5z6dBenkL5OVoP5UtAPxGRNuJ6BF5nT/+50YA0gDMldVts4koyk9lOxgF4Bv5t+XlCiFOAHgPwD8ATgHIBLAd1v/PyQB6E1E0EZUDcCOAevDvvfZUlmntGgsKCVJYV2L9homoPICFACYIIS76o0whRL6QVBJ1IQ3TWyrtZna5RHQTgLNCiO3Oq/1RNoAeQoiOkFQA44iotwVlKBEGoCOAGUKIDgAuwzoVVzFkO8D/t3d3IVZVYRjH/09mw2jl2CeF1RRFF4GUhYhFBHZTgUEFIkIS3uSNXUVFEATddFMheVPYRSUJhYl0IclkURTa55SpVJTUYH4VFgMhg7xdvOvkaZrZjXn20vD5wWGvWXPY79kf57x7r733WouB1yvGnA3cDVwJXArMJNf7eD3dzhGxi2ze2gJsBobJJt5TQc/2cyeKNEIeBXTMAfZW/gz7JV0CUKYH2ggiaTqZJNZFxIaasQEi4jDwLnmNZEBSp2PKttb5zcBiSXuA9WTz03M1YkfE3jI9QLbVz6fOuh4BRiJiW/n7DTJx1NrOdwCfRcT+8neNuLcDP0TEwYgYAzYAC6mznddGxLyIuBX4FfiWit+phlg9+11zokgfA9eUOyTOIk+bN1X+DJuA5aW8nLx+0FOSBKwFdkXEM7ViS7pQ0kAp95Nf6l3AVuC+tuICRMRjETEnIgbJ7fpORCxrO7akmZLO6ZTJNvsdVNjOEbEP+EnStaVqEbCzRuxiKceanagU90dggaQZZT/vLHPr+5iki8r0cuAectlrrWsaYm0C7i93Py0Afus0UR23Xl9U+r++yLbFb8i288dbjvUa2Y46Rmb9FWS7+RB5NDIEnNdC3FvIU88vgS/K6862YwNzgc9L3B3AE6X+KmA78B3ZTNHX8nq/DXirRuwy/+Hy+rqzT9XYziXO9cAnZZ1vBGZX2sdmAL8As7rqai3zk8Duso+9AvTV2MeA98mkNAwsanOZj+e3g2x6WlN+074i7wj7T3HdhYeZmTVy05OZmTVyojAzs0ZOFGZm1siJwszMGjlRmJlZIycKsymSdHRc76g9e9pZ0mB3j6Bmp5Iz//0tZlb8EdkNidlpxWcUZidIOe7E08oxN7ZLurrUXyFpqIwFMFSe3EXSxZLeVI7PMSxpYZnVNEkvlnEN3i5PsSNplaSdZT7rT9Ji2mnMicJs6vrHNT0t6frf7xExH3ie7EuKUn45IuYC64DVpX418F7k+BzzyCe3IccNWBMR1wGHgXtL/aPADWU+D7a1cGaT8ZPZZlMkaTQizp6gfg85MNP3pdPFfRFxvqRDZP//Y6X+54i4QNJBYE5EHOmaxyCwJXLwGSQ9AkyPiKckbQZGya44NkbEaMuLavY3PqMw642YpDzZeyZypKt8lGPXEO8i++y5Efi0qzdUsyqcKMx6Y0nX9KNS/pDssRZgGfBBKQ8BK+GvAZ3OnWymks4ALouIreTgSwPAP85qzNrkIxOzqesvo/R1bI6Izi2yfZK2kQdfS0vdKuAlSQ+TI849UOofAl6QtII8c1hJ9gg6kWnAq5Jmkb2BPhs5podZNb5GYXaCyjWKmyLi0Mn+LGZtcNOTmZk18hmFmZk18hmFmZk1cqIwM7NGThRmZtbIicLMzBo5UZiZWaM/AYKFh4Jw1yrRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax1 = plt.subplot(1, 1, 1)\n",
    "ax1.plot(range(len(gener_costs)), gener_costs, label='Generator loss')\n",
    "ax1.plot(range(len(discr_costs)), discr_costs, label='Discriminator loss')\n",
    "ax1.set_xlabel('Iterations')\n",
    "ax1.set_ylabel('Loss')\n",
    "ax1.legend()\n",
    "\n",
    "###################\n",
    "# Set scond x-axis\n",
    "ax2 = ax1.twiny()\n",
    "newlabel = list(range(NUM_EPOCHS+1))\n",
    "iter_per_epoch = len(train_loader)\n",
    "newpos = [e*iter_per_epoch for e in newlabel]\n",
    "\n",
    "ax2.set_xticklabels(newlabel[::10])\n",
    "ax2.set_xticks(newpos[::10])\n",
    "\n",
    "ax2.xaxis.set_ticks_position('bottom')\n",
    "ax2.xaxis.set_label_position('bottom')\n",
    "ax2.spines['bottom'].set_position(('outward', 45))\n",
    "ax2.set_xlabel('Epochs')\n",
    "ax2.set_xlim(ax1.get_xlim())\n",
    "###################\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEUAAACoCAYAAAAGjlD3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dfYxV9b3v8e8XGBhmhhmehgGG4UEFLRVBnSJRsFVLRdtUTWt6bHpqG3MxaZsck/5xaJv0nNz0Wv84p+c26c1pOCli1dsT2+qtubX2GnwAtDEOPiCCWsqTwAgMKDM8DI+/+wfbe1nr+x1mMftprb3er6SB9WEx+7fp/sza83Pv79YQggAAAAAAAOTNsGovAAAAAAAAoBrYFAEAAAAAALnEpggAAAAAAMglNkUAAAAAAEAusSkCAAAAAAByiU0RAAAAAACQS0VtiqjqMlV9T1W3quqKUi0KQHHoJpBOdBNIJ7oJpBPdRCVoCGFof1F1uIi8LyJLRWS3iLwmIveEEDYP9HcmTpwYZs6cOaTbQ+3bsGFDTwihtdrryDq6iVKjm6VBN1FqdLM06CZKjW6WBt1EKe3YsUN6enrU+7MRRXzdhSKyNYSwTUREVf9TRO4QkQEfpDNnzpSurq4ibhK1TFV3VnsNNYJuoqToZsnQTZQU3SwZuomSopslQzdRMp2dnQP+WTFvn2kXkQ/OO95dyCJUdbmqdqlq14EDB4q4OQAJ0U0gnegmkE50E0gnuomKKGZTxHvpiXkvTghhZQihM4TQ2drKq8iACqCbQDrRTSCd6CaQTnQTFVHMpshuEek473iaiOwtbjkASoBuAulEN4F0optAOtFNVEQxmyKvichsVZ2lqiNF5O9E5OnSLAtAEegmkE50E0gnugmkE91ERQx50GoI4bSqfk9E/iwiw0VkVQjhnZKtDMCQ0E0gnegmkE50E0gnuolKKebTZySE8IyIPFOitQAoEboJpBPdBNKJbgLpRDdRCcW8fQYAAAAAACCz2BQBAAAAAAC5VNTbZ1B6IZhPmZJTp04Nmnl/r76+3mTDhtl9MC8DAKAUzp49Gzk+c+aMOaevr89kp0+fNtmxY8dMdvz4cZN518QJEyZEjsePH2/OGTEi2dMiVe9TIgEAQBbx0zAAAAAAAMglNkUAAAAAAEAusSkCAAAAAAByiU0RAAAAAACQSwxarSJvEJw3gM4bLHfixIkhfX1v+Ko3MI4hcgCAC/GuMb29vSaLX8O8c+LDWEVEuru7TeYNHveum83NzSZra2uLHHuDXIcPH24yhpEDpeN1Pf6ck+egQOl41+qkmXdNrFVc6QEAAAAAQC6xKQIAAAAAAHKJTREAAAAAAJBLzBSpIu89k957t5qamhJlSfDeaADAxfLmAHizrUaNGjXoed5sq3379pls5MiRJvNmbHkzSmbNmjXo2ryvzywDoLx4HgqUjzdj6/jx4ybz5nN518T4ddP7ObVWrpt8ZwIAAAAAALnEpggAAAAAAMglNkUAAAAAAEAuFTVTRFV3iEifiJwRkdMhhM5SLApAcegmkE50E0gnugmkE91EJZRi0OpNIYSeEnydmhZCMJk3DKe/vz/ReQ0NDZHjESPs/5XebXqZN0Av6SAsBmalGt0E0qkmuukNVfWuV83NzZHjpNem9evXm2z79u0ma2xsNJk3DK6jo8NkQExNdBOoQXQzxrtuHj161GR79uwx2YYNG0x28uRJk33pS1+KHLe2tl7MEjOFn2gBAAAAAEAuFbspEkTk/6jqBlVdXooFASgJugmkE90E0oluAulEN1F2xb595oYQwl5VnSQiz6nquyGEteefUHjwLhcRmT59epE3ByAhugmkE90E0oluAulEN1F2Rb1SJISwt/DrfhF5SkQWOuesDCF0hhA6a/l9SECa0E0gnegmkE50E0gnuolKGPIrRVS1UUSGhRD6Cr//goj815KtLMO8wTf79u0z2be+9S2TTZkyxWSTJ0822d133x05njt3rjnHG77qDVU9deqUybxhO6NHjzZZfJhdXV2dOQeVRTeL5/XE89RTT5msszM6FH3ixInmHG8wJGpfVrrpXcNUNdF53vDtI0eORI5ffPFFc85bb71lsoMHD5rslVdeMZk33PWqq64y2Ycffhg59q633jXMu+9JsyS8f7Ohfi0MTVa6mTVeNz3xDvD4xyfo5sX5+OOPTRb/mVHEH746e/ZskzU1NQ36tWpFMW+faRORpwrfuEaIyP8MITxbklUBKAbdBNKJbgLpRDeBdKKbqIghb4qEELaJyPwSrgVACdBNIJ3oJpBOdBNIJ7qJSuEjeQEAAAAAQC6xKQIAAAAAAHKp2I/kzT1vIOnGjRtNdtttt5nMGyLnDZdqaGgwWUtLS+R4/Pjx5hwv8wZIHj9+PNHa+vv7TRYf8OoNd2VgFtLMG4LsDTceqiuvvNJkGzZsMNnIkSNLdptAUt4gRG+AatKsu7vbZJs3b44ce4NWe3p6TLZu3TqTHTt2zGTeNebhhx822aFDhyLHixcvNud4n1rg/Rt5A1nr6+sTrW3MmDGRY6/78SHmQJp43V+zZo3JnnzySZN5Hbv//vsjx+PGjTPneMP+gTzzri/ecH/vZz+vw++//77J5syZEzk+ffq0OadWPmSDV4oAAAAAAIBcYlMEAAAAAADkEpsiAAAAAAAgl9gUAQAAAAAAucSg1Yt06tSpyPHy5cvNOY8++qjJvCE3Hm/wzYkTJ0w2ffr0yLE3uPH666832e7du0323nvvmayjo8Nkw4bZPbSjR49GjhmEhbTwuuQNNPSGRhWjsbExcuwNcvUGNwKlFu+Adx2KX9NE7Pd1Ef865J23a9cuk8WviS+//LI5Z//+/Ylu0xssN23aNJMdOXLEZDt27Igce0Nb29raTPaXv/zFZFdccYXJvGvuhAkTTDZq1KjIca0MqUNt8r5HLFu2zGSbNm0ymTcwOD64UUSks7Mzcrx06dKLWSKAgpdeeslk3vPQvr4+k3kflhH/GXH+/PlFrC7deKUIAAAAAADIJTZFAAAAAABALrEpAgAAAAAAcolNEQAAAAAAkEsMWr0Abxjigw8+GDkuZqiqxxtUOm/ePJNde+21g/69PXv2mOz48eOJbtMbNjdmzBiTtbS0RI69IXhANUycONFkxQxV/exnP2uyW265xWTf/e53I8d79+4153gDX4FSS/L9+OTJkybzhhR3d3ebzBvU9uc//9lk77zzTuT48OHD5hxvrQ0NDSaLDzIWEbn88stN1t7ebrL4gPJt27aZc7zB497Q1nXr1pmstbXVZN51Mz603BtiDlSD1/1Vq1aZzOt+f3+/yZqbm0121113mSw+aNUb0AogyuvhE088YbLe3l6TeV33BoNfc801keNa/jmPKzEAAAAAAMglNkUAAAAAAEAusSkCAAAAAAByadBNEVVdpar7VXXTedl4VX1OVf9a+HVceZcJII5uAulEN4F0optAOtFNVFuSQaurReQXIvLr87IVIrImhPCQqq4oHP9j6ZdXXTt27DDZz3/+88hxMUNVvYFx8QGqIiKPPfaYycaNi35f8Aaojhhh/+89cOCAybyBlEkHrTIMq6pWS0676YkPjPzoo48S/b2xY8eazBugPHv2bJPFBzeK2GGuM2bMMOfU8qAqiEhKu+k97rzrhDe8bdSoUSbbtWuXyU6dOmWy+GDhKVOmJFqH97XiQ99E/A4vXbrUZEePHo0cb9myxZyzdetWk3nXeW+4a11dncm8a2Q8Y9BqRa2WFHYzLbznkpdcconJrrzySpN5gxs/9alPmWzZsmUmiw/857llLq0Wujkgr1/79+83mTfs3PuQAe/5gPdzntf/WjXolTiEsFZEDsXiO0TkkcLvHxGRO0u8LgCDoJtAOtFNIJ3oJpBOdBPVNtT/PNEWQugWESn8OmmgE1V1uap2qWqX9yoFACVFN4F0optAOtFNIJ3oJiqm7K/ZDCGsDCF0hhA6W1tby31zABKim0A60U0gnegmkE50E8VKMlPEs09Vp4QQulV1iojYNzVlzJkzZ0z229/+1mTx9yQnFX+/pIjIzTffbLJvf/vbJps6darJjhw5Ejk+ePCgOefVV181mfd+0c985jMm895X5r3nG6lTc91MKt4J7z3J3nuZV61aZTJvzo73nkyvE9/85jcjxw8++KA5p6mpyWSoeVXvZtKZIvX19SaLzwW5mK8Xn8fT09NjzvGuc94T2zlz5gz69UVEOjo6TLZ27drIsXc997L4vCIRf85Ic3Ozybz74M0eQVVVvZtp4T02lyxZYjJvLtDevXtN5s0eic/EExE5duxY5Ji5WyigmwXec9Curi6TJZ2n580omTx5ssny1MWhvlLkaRG5t/D7e0XkD6VZDoAi0U0gnegmkE50E0gnuomKSfKRvL8Rkb+IyOWqultV7xORh0Rkqar+VUSWFo4BVBDdBNKJbgLpRDeBdKKbqLZB3w8RQrhngD+6pcRrAXAR6CaQTnQTSCe6CaQT3US1lX3QKgAAAAAAQBoxObOgt7fXZGvWrBn073kDaLwBjzNmzDDZ/PnzTXbdddeZrL+/32Q7duyIHL/44ovmHG94rJd5Q+TyNFgHteHEiROR46VLl5pzfvzjH5vMG6oa/1oi/pCr+OBGETvk6t133zXneEMggXJL+n3dG6rqDV8dNsz+dxVvsOhll10WOd65c6c5p62tzWSXXnqpybwBj97f9a6b8Y9pfOutt8w5hw8fNpnHGxbb0NCQKPP+3YA08AYle5k38HjUqFEm855zxoeii/jD/QH8f319fSbzhhZ7nUtq/347xzZP16v83FMAAAAAAIDzsCkCAAAAAAByiU0RAAAAAACQS2yKAAAAAACAXGLQasGePXtM5g2ciQ+qq6urM+d4Q2muvfZaky1ZssRk3d3dJjt06JDJfvGLXwy61k9/+tMm8wa+Njc3m4xBq8iajRs3Ro69YVM/+tGPTOYNWt2+fbvJurq6THb27FmTxYdKdnZ22sUCKeZd1zxNTU0mW7x4scni18R58+aZc7wueYNWvYHH27ZtM5k3zPHxxx+PHHuD61paWhKtzfu+cfTo0UR/N/69yRvODlRDCMFk3mP42LFjJvOukbt27TLZnXfeabLGxsbIMc9BgSjvuvz++++bzLtGeryOvfHGGxe/sBrCK0UAAAAAAEAusSkCAAAAAAByiU0RAAAAAACQS2yKAAAAAACAXMrloFVvaJQ3qM0bQBofdDNy5Ehzjjc0bdKkSSZ7/fXXTeYNTD116pTJtm7dGjlubW015/T09Jjso48+Mtntt99uMoZcIc28IaoPPPBA5Njr9MmTJ8u2pk/Euz5u3Liy3yYwVN71yrvmeJ3zrhNtbW2Dnjdq1KhEX8s7z1vbli1bTPbMM8+YLD7Msb293ZzjPT/o7e01mffvdvjwYZN5w1xHjIg+9fIG6HENRiXEhzJ6PX/77bdN9pOf/MRka9euNZn32P7Tn/5ksvgQ5IaGBrtYIMe86+HmzZtN5nXOu256g9K9n2nzhFeKAAAAAACAXGJTBAAAAAAA5BKbIgAAAAAAIJcG3RRR1VWqul9VN52X/bOq7lHVNwv/s0MpAJQV3QTSiW4C6UQ3gXSim6i2JINWV4vIL0Tk17H830II/1LyFVWAN0ht2DC7PzRhwgSTTZ06NXLsDZVbvHixyRYtWmQybxDkSy+9ZLJ9+/aZLD4ca86cOeacWbNmmWzTpk0m8wbrMOQtE1ZLjXUzqRMnTpjs4MGDkWNvsFQlxIcgI5dWS0a66X2v966H3nUzfh0S8bs5fvz4yLE3MM67DnlDH72hqn/84x9N9uabb5rssssuixwvWLDAnONd072h6PFhqSL+v9vRo0dNFh/i7v3beoNcURKrJSPdrIR4x7xh/I8++qjJ1qxZY7IjR44kuk1v4Pn27dsjx94QZNS81UI3B+RdI1esWGGyxx57zGTetfqWW24xWd5/9hv0lSIhhLUicqgCawFwEegmkE50E0gnugmkE91EtRUzU+R7qrqx8HKnAT9zUlWXq2qXqnYdOHCgiJsDkBDdBNKJbgLpRDeBdKKbqIihbor8u4hcKiILRKRbRP51oBNDCCtDCJ0hhM7W1tYh3hyAhOgmkE50E0gnugmkE91ExQxpUySEsC+EcCaEcFZE/kNEFpZ2WQCGgm4C6UQ3gXSim0A60U1UUpJBq4aqTgkhdBcO7xIRO70zY2bOnGmy+fPnm+ySSy6JHN9+ux2EPGnSJJNt2LDBZJs3bzbZ8ePHTeYNpRo9enTk2BvKduiQfWve9OnTTeYNjEM21WI3PXV1dSabN29e5Hj9+vXmHK9LpfbKK6+U/TaQPVnvZtJhoPGBxx5vmGl/f7/JvGGpK1euNNnLL79sspaWFpPV19dHjq+//voLrvMT3oBybwisN0DWy+JCCInWgfLIejdLyRtQ/sYbb5jMGyBczG14Q48Bunlh3nVu2bJlJlu3bp3JvvKVr5RlTVk26E/DqvobEfmciExU1d0i8k8i8jlVXSAiQUR2iMj9ZVwjAAfdBNKJbgLpRDeBdKKbqLZBN0VCCPc48a/KsBYAF4FuAulEN4F0optAOtFNVFsxnz4DAAAAAACQWWyKAAAAAACAXMrlhE1vOFx7e7vJbrzxRpP19PREjq+++mpzjjdE6oYbbjDZ888/b7IPPvjAZJ6GhoZBzzl8+LDJFi9ebLKxY8cmuk0gLbwOf/nLX44cewMOn332WZOVesihN8wYyBJviKjXk507d5psx44dJrv00ksjx9dcc405Z9euXSZ74YUXTOYNjOvt7TVZY2Ojybzbjfvb3/5mMm8I7OWXX26yadOmmaypqclkqho5ZtAqqiV+LfWeW3ofOuANN076OG5ubjaZ11cAF+Z96MDUqVNN5v2c5/1dr8Px61Ut45UiAAAAAAAgl9gUAQAAAAAAucSmCAAAAAAAyKVczhTx3h9VX19vMu/9wR0dHZHjEydOmHO8mSLbtm0zWdL3Lo8ePdpkc+bMuehzRETuuOMOk3nzGYA08zr89a9/PXLc19dnznnttddMFp8TdDG8+SH0CVl3+vRpk3300Ucm82Zgede/PXv2RI5HjLBPPd58802T/e53v0u0Dm8Gincf4rPDvPdUe7O4Zs+ebbIFCxaYbNasWSbzZhvF7z/fM1Atw4ZF/9uo102vE8X46U9/arI8zS0AhsKb9+E9f926davJZsyYYbL4z7Mi9vtB3uT73gMAAAAAgNxiUwQAAAAAAOQSmyIAAAAAACCX2BQBAAAAAAC5lMtBq96wGm+4lDdwJj7kbeTIkeYcb1jq888/b7KNGzeazBsYd/LkSZNdccUVkeMrr7zSnLNs2TKTjRkzxmRA1nhD2eJdXLRokTnnlVdeMdmzzz5rMu97hOeXv/xlovOArPOuTTNnzjRZV1eXyQ4ePBg5fvfdd805W7ZsMVl3d7fJvAGqHu/63dLSEjn2hp3HzxERmTt3rsmuuuoqkzU3N5vMW2/8+xdDJlEt8ceeN6DcG9yY5GuJ+IOGb7311oSrA/AJ71rifXiA9wEAY8eOTfT18o5XigAAAAAAgFxiUwQAAAAAAOQSmyIAAAAAACCXBt0UUdUOVX1BVbeo6juq+g+FfLyqPqeqfy38Oq78ywXwCboJpBPdBNKJbgLpRDdRbUkGrZ4Wke+HEF5X1TEiskFVnxORb4nImhDCQ6q6QkRWiMg/lm+p5eUNkWtsbDTZnj17IscffvihOWf9+vUme+KJJ0x27Ngxk3kDHtvb2032xS9+MXLsDX2bPHmyyRjoVlNy0c2kmpqaIsfeUOFNmzYN+evX1dWZ7Lbbbhvy10NNy3Q3veGIU6ZMMVlvb6/JOjs7TRa/JnoDVHfu3Gkyb8i4xxuU3tHRYbL4ALo5c+aYc7zr5vjx403m/Rt5hg8fbrL4erkuV1Smu1lq8cdefX29Ocd7/HsfMnD27FmT3XTTTSbzOgYI3YyI/zx4+PBhc87HH39sMm8wsnfeV7/61UFvUyRf16dBXykSQugOIbxe+H2fiGwRkXYRuUNEHimc9oiI3FmuRQKw6CaQTnQTSCe6CaQT3US1XdRMEVWdKSJXi8irItIWQugWOfdAFpFJA/yd5arapapdBw4cKG61AFx0E0gnugmkE90E0oluohoSb4qoapOI/F5EHggh2NfMDiCEsDKE0BlC6GxtbR3KGgFcAN0E0oluAulEN4F0opuolkSbIqpaJ+ceoI+HEJ4sxPtUdUrhz6eIyP7yLBHAQOgmkE50E0gnugmkE91ENQ06aFXPTVj5lYhsCSH87Lw/elpE7hWRhwq//qEsK6wQb1Db6NGjTRYf3vbGG2+Yc7xBq4cOHUq0jrFjx5rs/vvvN1l8YNykSfbVZHkajpNHeelmUsOGRfd4vUHGU6dONdm+fftMdsMNN5hszZo1RawOeZKlbnrXCS9raWkxmTeo1Dsv3rGenp4hr827Vl9yySUm+853vmOy+LBY7/uBN0Ay/r3lYnj3oZivh+JkqZvV0NDQYLIbb7zRZN4Q5CVLlpjsBz/4QWkWhppHN6PiHwCyfft2c87DDz9ssvgHgoj4z4e7urpMNnfuXJN5w5drVZJPn7lBRP5eRN5W1TcL2Q/l3IPzCVW9T0R2icjd5VkigAHQTSCd6CaQTnQTSCe6iaoadFMkhLBeRAZ6ycEtpV0OgKToJpBOdBNIJ7oJpBPdRLXxGk4AAAAAAJBLbIoAAAAAAIBcSjJTpOaEEEx29uxZk9XV1ZksPiCtra3NnLNw4UKTffzxxyY7evSoya677jqTfe1rXzNZ/HaHDx9uzgHyJD7Q0BsO99BDD5nM6/7NN99cuoUBGeMNM/Wuh94w8vb2dpPFB7Ju2rTJnDNjxgyTeR+rOGbMGJPddNNNJvv85z9vsokTJ0aOvftUDO+5BUNVkSXeoFVv2L+XNTU1mYznpsDQxIcZ79y505zT19dnMm8wanzIuIj/s2qpr4lZw9UaAAAAAADkEpsiAAAAAAAgl9gUAQAAAAAAucSmCAAAAAAAyKWaH7TqDT47ffq0yQ4ePGgybxBqT09P5HjkyJHmHG8Q3Pz5803mDbO79dZbTeYNmxs1apTJgDyLD0zt7e0158QHV4n4AxmBPPOGg8YHGYuI9Pf3myxJ76644opEX98b3Ohd+77xjW+YzLtuetfcUvLuA5Al3uDxxsZGk3kDVHn8A0OT5ANApk6das657777THbkyBGTdXZ2mmz27Nkmy/tgZF4pAgAAAAAAcolNEQAAAAAAkEtsigAAAAAAgFxiUwQAAAAAAORSzQ9aTaquri5RVl9fHzkePXq0OWfhwoUmu/rqq012/Phxk40dO9Zk3mA5BloBUfFOeEMav/CFL1RqOUAmeNcSb9iid7368MMPTXbgwAGTvffee4N+fW9oeUdHh8nmzZtnsgkTJpgs7wPjgCS8AY9x5R5QDOSddx1uaGiIHC9atMicc/3115dtTXnEK0UAAAAAAEAusSkCAAAAAAByiU0RAAAAAACQS4O+UVBVO0Tk1yIyWUTOisjKEMLPVfWfReS/iMgnbyD+YQjhmXItdKi892l5s0K8WR7e+5S99zjH9ff3m8x7P7Y3K8TLvPdfxzPvPqG2Zb2bpRbvOp1AtWS9m8OG2f9e0tLSYrLJkyebLD53S0Tk2LFjkeOdO3eacw4fPmyy3bt3m+y6664zmTePxLtuxu8Xs7nyJ+vdLIbXiTNnzkSOvRkjXr+AUstzNz3edRjllWR60mkR+X4I4XVVHSMiG1T1ucKf/VsI4V/KtzwAF0A3gXSim0A60U0gnegmqmrQTZEQQreIdBd+36eqW0SkvdwLA3BhdBNIJ7oJpBPdBNKJbqLaLuq1Oao6U0SuFpFXC9H3VHWjqq5S1XED/J3lqtqlql3eR/UBKB7dBNKJbgLpRDeBdKKbqIbEmyKq2iQivxeRB0IIvSLy7yJyqYgskHM7e//q/b0QwsoQQmcIobO1tbUESwZwProJpBPdBNKJbgLpRDdRLUlmioiq1sm5B+jjIYQnRURCCPvO+/P/EJH/XZYVloE3XG3EiET/FIkGs40ePTpR5g208rIkA+OQT7XWTaBWZLmb3nXOG7Y4bdo0k02cONFk8Seoc+fONecsWbLEZPEhkCIis2bNMpk33JUhqhhIlrtZDO95I88lkSZ57SbSYdDvhnrumcWvRGRLCOFn5+VTzjvtLhHZVPrlARgI3QTSiW4C6UQ3gXSim6i2JC+PuEFE/l5E3lbVNwvZD0XkHlVdICJBRHaIyP1lWSGAgdBNIJ3oJpBOdBNIJ7qJqkry6TPrRcR7HWrNf0Y0kGZ0E0gnugmkE90E0oluotp4MyEAAAAAAMilZNNFcVGSDnhLet7w4cOLWQ4AACVXV1eXKBszZkzk2BvQCgAAUC28UgQAAAAAAOQSmyIAAAAAACCX2BQBAAAAAAC5xKYIAAAAAADIJQ0hVO7GVA+IyE4RmSgiPRW74fLI+n1I4/pnhBBaq72IPKKbqZLG9dPNKqGbqZLG9dPNKqGbqZLG9dPNKqmhbmZ9/SLpuw8D9rKimyL/70ZVu0IInRW/4RLK+n3I+vpRHrXwuMj6fcj6+lEetfC4yPp9yPr6UR618LjI+n3I+vpRHll/XGR9/SLZug+8fQYAAAAAAOQSmyIAAAAAACCXqrUpsrJKt1tKWb8PWV8/yqMWHhdZvw9ZXz/KoxYeF1m/D1lfP8qjFh4XWb8PWV8/yiPrj4usr18kQ/ehKjNFAAAAAAAAqo23zwAAAAAAgFxiUwQAAAAAAORSxTdFVHWZqr6nqltVdUWlb38oVHWVqu5X1U3nZeNV9TlV/Wvh13HVXOOFqGqHqr6gqltU9R1V/YdCnpn7gPKjm5VHN5EE3aw8uokkstbNrPdShG5icFnrpUj2u1kLvazopoiqDheR/yEit4nIXBG5R1XnVnINQ7RaRJbFshUisiaEMFtE1hSO0+q0iHw/hPApEVkkIt8t/Ltn6T6gjOhm1dBNXBDdrBq6iQvKaDdXS7Z7KUI3cQEZ7aVI9ruZ+V5W+pUiC0VkawhhWwjhpIj8p4jcUeE1XKm3SoYAAAI0SURBVLQQwloRORSL7xCRRwq/f0RE7qzooi5CCKE7hPB64fd9IrJFRNolQ/cBZUc3q4BuIgG6WQV0EwlkrptZ76UI3cSgMtdLkex3sxZ6WelNkXYR+eC8492FLIvaQgjdIuceCCIyqcrrSURVZ4rI1SLyqmT0PqAs6GaV0U0MgG5WGd3EAGqlm5l9TNNNOGqllyIZfUxntZeV3hRRJ+MzgStEVZtE5Pci8kAIobfa60Gq0M0qopu4ALpZRXQTF0A3q4huYgD0soqy3MtKb4rsFpGO846nicjeCq+hVPap6hQRkcKv+6u8ngtS1To59yB9PITwZCHO1H1AWdHNKqGbGATdrBK6iUHUSjcz95imm7iAWumlSMYe01nvZaU3RV4TkdmqOktVR4rI34nI0xVeQ6k8LSL3Fn5/r4j8oYpruSBVVRH5lYhsCSH87Lw/ysx9QNnRzSqgm0iAblYB3UQCtdLNTD2m6SYGUSu9FMnQY7oWeqkhVPYVRap6u4j8dxEZLiKrQgj/raILGAJV/Y2IfE5EJorIPhH5JxH5XyLyhIhMF5FdInJ3CCE+ICcVVHWxiKwTkbdF5Gwh/qGce69XJu4Dyo9uVh7dRBJ0s/LoJpLIWjez3ksRuonBZa2XItnvZi30suKbIgAAAAAAAGlQ6bfPAAAAAAAApAKbIgAAAAAAIJfYFAEAAAAAALnEpggAAAAAAMglNkUAAAAAAEAusSkCAAAAAAByiU0RAAAAAACQS/8XN9o4AfRXodMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x180 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "##########################\n",
    "### VISUALIZATION\n",
    "##########################\n",
    "\n",
    "\n",
    "model.eval()\n",
    "# Make new images\n",
    "z = torch.zeros((5, LATENT_DIM)).uniform_(-1.0, 1.0).to(device)\n",
    "generated_features = model.generator_forward(z)\n",
    "imgs = generated_features.view(-1, 28, 28)\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=5, figsize=(20, 2.5))\n",
    "\n",
    "\n",
    "for i, ax in enumerate(axes):\n",
    "    axes[i].imshow(imgs[i].to(torch.device('cpu')).detach(), cmap='binary')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}