01. Vision_Transformer_basic_tutorial-MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vision Transformer basic tutorial - ViT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/mulkong/ViT_tutorial-pytorch/blob/main/01.%20Vision_Transformer_basic_tutorial-MNIST.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -Uqq einops\n",
    "!pip install -Uqq ipywidgets\n",
    "!pip install -Uqq torchsummary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed Dec 29 11:00:36 2021       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla V100-SXM2...  On   | 00000000:00:05.0 Off |                  Off |\n",
      "| N/A   50C    P0    74W / 300W |  13330MiB / 32510MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   1  Tesla V100-SXM2...  On   | 00000000:00:06.0 Off |                  Off |\n",
      "| N/A   42C    P0    67W / 300W |   9024MiB / 32510MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   2  Tesla V100-SXM2...  On   | 00000000:00:07.0 Off |                  Off |\n",
      "| N/A   34C    P0    41W / 300W |      2MiB / 32510MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "gpu_info = !nvidia-smi\n",
    "gpu_info = '\\n'.join(gpu_info)\n",
    "\n",
    "if gpu_info.find('failed') >= 0:\n",
    "    print('Not connected to a GPU')\n",
    "else:\n",
    "    print(gpu_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch import Tensor\n",
    "import torchvision\n",
    "from einops import rearrange, reduce, repeat\n",
    "from einops.layers.torch import Rearrange, Reduce\n",
    "from torchsummary import summary\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Vision Transformer Architecture - pytorch\n",
    "* * * \n",
    "\n",
    "## 특징\n",
    "- NLP 분야에서 사용된 Transformerd를 응용하여 Vision task에서 사용할 수 있도록 고안된 첫 논문 -> ViT\n",
    "- Transformer만 사용해서 image classificaiton task에 적용\n",
    "- 논문에서 Architecture의 hyper-parameter에 따라 여러가지 모델들을 제안하고 있음.\n",
    "<img src=\"imgs/vit table1.png\"> \n",
    "\n",
    "## 학습 과정\n",
    "\n",
    "<img src=\"imgs/ViT Architecture gif.gif\">\n",
    "\n",
    "\n",
    "1. 이미지 입력 \n",
    "    - 논문에서는 224x224x3 이미지\n",
    "2. patch로 자름\n",
    "    - 논문에서는 16 x 16로 자름.\n",
    "3. 각각의 patch들을 1D-vector로 풀어냄\n",
    "4. 1D vector로 만들어진 patch들을 Linear Projection을 통해 각 patch의 embedding vector로 표현.\n",
    "    - 논문상에서는 768차원의 patch embedding vector로 표현.\n",
    "5. 각 patch embedding에 classification token, position embedding 추가.\n",
    "6. Transformer Encoder\n",
    "7. Transformer output\n",
    "8. classification 수행\n",
    "\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Linear Projection of Flattened Patches\n",
    "## 1-1. Patch Embedding\n",
    "- 1d vector로 만들어진 patch들을 Linear Projection(nn.Linear or nn.Conv2D 사용)을 통해 표현된 patch embedding vector에 cls_token, position embedding 추가함\n",
    "\n",
    "## 1-2. classification token(=cls_token)\n",
    "- classification을 위해 사용되는 token이며 **학습을 통해 결정되는 parameter**\n",
    "\n",
    "## 1-3. position embedding\n",
    "- patch의 위치 정보를 가지고 있는 embedding\n",
    "- position embedding을 사용 안하면?\n",
    "    - 이미지 입력 -> 일정한 크기의 patch로 자름 -> 자른 patch를 sequence로 생각하여 transformer Encoder로 입력 -> 각 patch가 어떤 위치에서 왔는지 위치 정보가 손실 -> 학습 안됨\n",
    "\n",
    "#### 따라서, Transformer Encoder에 입력되는 parameter는 (classification token, patch embedding) + position embedding이다.\n",
    "<img src=\"imgs/figure_vit_1.png\"> </br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PatchEmbedding(nn.Module):\n",
    "    def __init__(self, in_channels: int=3, patch_size:int=16, emb_size:int=768, img_size:int=224):\n",
    "        super().__init__()\n",
    "        self.patch_size = patch_size\n",
    "        self.projection = nn.Sequential(\n",
    "            nn.Conv2d(in_channels, emb_size, kernel_size=patch_size, stride=patch_size),\n",
    "            Rearrange('b e (h) (w) -> b (h w) e')\n",
    "        )\n",
    "        # 학습에 의헤 결정되는 parameter\n",
    "        self.cls_token = nn.Parameter(torch.randn(1,1, emb_size))\n",
    "        self.positions = nn.Parameter(torch.randn((img_size // patch_size) **2 + 1, emb_size))\n",
    "        \n",
    "    def forward(self, x:Tensor) -> Tensor:\n",
    "        b, _, _, _ = x.shape\n",
    "        x = self.projection(x)\n",
    "        cls_tokens = repeat(self.cls_token, '() n e -> b n e', b=b)\n",
    "        # prepend the cls token to the input\n",
    "        x = torch.cat([cls_tokens, x], dim=1)\n",
    "        # add position embedding\n",
    "        x += self.positions \n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Transformer Encoder\n",
    "\n",
    "## 2-1. NLP Transformer Encoder vs Vision Transformer Encoder\n",
    "- Transformer는 `layer를 깊게 쌓으면 학습이 어렵다`라는 단점이 존재한다.\n",
    "    - 이런 단점을 극복하여 학습이 이루어지게 하기 위해서는 `layer normalization의 위치가 중요하다`라는 것이 후속 연구들을 통해 증명이 되었다.\n",
    "\n",
    "- ViT는 NLP 분야에서 보여진 `layer를 깊게 쌓으면 학습이 어렵다`라는 문제와 이를 극복하기 위해 `layer normalization의 위치가 중요하다`라는 연구를 받아드려 Vision Transformer Encoder에 적용시켰다.\n",
    "    - 기존 Vanilla Transformer : Multi-head attention -> normalization\n",
    "    - ViT: normalization -> Multi-head attention\n",
    "\n",
    "<center>\n",
    "    <img src=\"imgs/MLP Transformer Encoder.png\", height=\"250px\", width=\"250px\"/>\n",
    "    <img src=\"imgs/Transformer Encoder.png\", height=\"250px\", width=\"250px\">\n",
    "</center>\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 2-2. 입력 형태\n",
    "\n",
    "위 그림에서 보면 `input Embedding`블럭을 거친 후 position encoding과 더해지는 부분이 있다. 그 부분을 코드로 작성하면 아래와 같다.\n",
    "\n",
    "\n",
    "``` python\n",
    "transformer_input = torch.cat((model.cls_token, patches), dim=1) + pos_embed\n",
    "```\n",
    "\n",
    "그리고 본 튜토리얼에서 사용된 `PatchEmbedding class`에서는 다음과 forward 함수에서 같이 사용되었다.\n",
    "```python\n",
    "x = torch.cat([cls_tokens, x], dim=1)\n",
    "x += self.positions \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-3. Multi-head attention architecture\n",
    "- ViT base model을 기준으로 말을 하면 self-attention을 12번 수행했다고 나와있다. ViT에서는 아래 그림 중 `Scaled Dot-Product Attention` 그림을 보면 된다.\n",
    "\n",
    "<center>\n",
    "    <img src=\"imgs/figure_vit_2.png\", height=\"100%\", width=\"100%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2-3-1. Self-Attention\n",
    "- 순서 1: layer normalization 거친 후 얻어진 $z$\n",
    "\n",
    "- 순서 2: weight metrix $W_Q$,  $W_K$,  $W_V$에 곱해져서 Query, key, Value가 계산된다.\n",
    "    \n",
    "    ⭐️ 이떄, weight metrix의 학습을 통해 `attention`이 학습된다.\n",
    "    \n",
    "- 순서 3: Query, Key 값들의 `dot product[torch.einsum('bhqd, bhkd -> bhqk', queries, keys)]`를 통해 `Similarity 계산`\n",
    "    - 각 이미지 패치들이 다른 모든 패치들과 얼마나 유사한지 파악을 하는데 도움을 줍니다.\n",
    "    - NLP와 동일하게 안정적인 gradient를 얻기 위해 $Q /dot K^T$행렬에 key vector dimension의 제곱근값으로 나눈다.\n",
    "    \n",
    "- 순서 4: 위에서 계산된 값은 비정규화된 형태이다. 이를 Softmax를 취해 0 ~ 1의 attention score계산하여 정규화 작업을 진행한다.\n",
    "\n",
    "- 순서 5: 계산된 attention score에 `Value`를 곱해 최종적인 output을 얻는다.\n",
    "\n",
    "- ⭐️ `self-attention`은 query, key vector의 내적 연산 후 key 차원의 제곱근으로 나누기 때문에 `scaled dot proudct attention` 이라고도 한다.\n",
    "<center>\n",
    "    <img src=\"imgs/Multi-Head attention.png\", height=\"100%\", width=\"100%\"/>\n",
    "</center>\n",
    "\n",
    "### 2-3-2. Multi-head self-attention\n",
    "\n",
    "#### 개념\n",
    "`attention 결과의 정확도를 높이기` 위해 `single head attention`말고 `multi-head attention`을 사용한 후 결과값을 더한다.\n",
    "\n",
    "#### 순서\n",
    "- 순서 1: layer normalization 거친 후 얻어진 768 차원의 $z$ \n",
    "- 순서 2: Multi-head self-attention 통과\n",
    "- 순서 3: 64차원으로 감소\n",
    "- 순서 4: self-attention을 12번 수행하니까 768차원으로 다시 증가\n",
    "- 순서 5: 결국 차원 수가 입력일때와 동일하게 유지됨.\n",
    "\n",
    "<center>\n",
    "    <img src=\"imgs/figure_vit_3.png\", height=\"100%\", width=\"100%\"/>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, emb_size:int=768, num_heads:int=8, dropout:float=0.):\n",
    "        super().__init__()\n",
    "        self.emb_size = emb_size\n",
    "        self.num_heads = num_heads\n",
    "        self.qkv = nn.Linear(emb_size, emb_size * 3)\n",
    "        self.att_drop = nn.Dropout(dropout)\n",
    "        self.projection = nn.Linear(emb_size, emb_size)\n",
    "        \n",
    "    def forward(self, x : Tensor, mask: Tensor = None) -> Tensor:\n",
    "        qkv = rearrange(self.qkv(x), \"b n (h d qkv) -> (qkv) b h n d\", h=self.num_heads, qkv=3)\n",
    "        queries, keys, values = qkv[0], qkv[1], qkv[2]\n",
    "        # dot product: MatMul(Query, Keys)\n",
    "        energy = torch.einsum('bhqd, bhkd -> bhqk', queries, keys) # batch, num_heads, query_len, key_len\n",
    "        if mask is not None:\n",
    "            fill_value = torch.finfo(torch.float32).min\n",
    "            energy.mask_fill(~mask, fill_value)\n",
    "            \n",
    "        # Scale\n",
    "        scaling = self.emb_size ** (1/2)\n",
    "        att = F.softmax(energy, dim=-1) / scaling\n",
    "        att = self.att_drop(att)\n",
    "        \n",
    "        # dot product: MatMul(attention, values)\n",
    "        out = torch.einsum('bhal, bhlv -> bhav ', att, values)\n",
    "        \n",
    "        out = rearrange(out, \"b h n d -> b n (h d)\")\n",
    "        out = self.projection(out)\n",
    "        return out      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-4. MLP\n",
    "MLP는 1개의 hidden layer로 구성되 FCN(Fully-connected latyer)이다.\n",
    "<center>\n",
    "    <img src=\"imgs/figure_vit_4.png\", height=\"100%\", width=\"100%\"/>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(nn.Sequential):\n",
    "    def __init__(self, emb_size:int, expansion:int=4, drop_p:float=0.):\n",
    "        super().__init__(\n",
    "            nn.Linear(emb_size, expansion * emb_size), # (768, 3072)\n",
    "            nn.GELU(), # activation function\n",
    "            nn.Dropout(drop_p),\n",
    "            nn.Linear(expansion * emb_size, emb_size) # (3072, 768)\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-5. Residual layer\n",
    "residual addition을 계산하는 class를 만들어 줍니다.\n",
    "\n",
    "자세한 내용은 residual network paper를 읽어보시거나 코드 구현체를 참고하시기 바랍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualAdd(nn.Module):\n",
    "    def __init__(self, fn):\n",
    "        super().__init__()\n",
    "        self.fn = fn\n",
    "\n",
    "    def forward(self, x, **kwargs):\n",
    "        res = x\n",
    "        x = self.fn(x, **kwargs)\n",
    "        x += res\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-6. Transformer Encoder Block\n",
    "이제 아래 그림과 같이 위에서 구성해준 class를 사용해서 Transformer Encoder class를 구현해보겠습니다.\n",
    "<center>\n",
    "    <img src=\"imgs/Transformer Encoder.png\", height=\"250px\", width=\"250px\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerEncoderBlock(nn.Sequential):\n",
    "    def __init__(self,\n",
    "                 emb_size: int = 768,\n",
    "                 drop_p: float = 0.,\n",
    "                 forward_expansion: int = 4,\n",
    "                 forward_drop_p: float = 0.,\n",
    "                 ** kwargs):\n",
    "        super().__init__(\n",
    "            ResidualAdd(nn.Sequential(\n",
    "                nn.LayerNorm(emb_size),\n",
    "                MultiHeadAttention(emb_size, **kwargs),\n",
    "                nn.Dropout(drop_p)\n",
    "            )),\n",
    "            ResidualAdd(nn.Sequential(\n",
    "                nn.LayerNorm(emb_size),\n",
    "                MLP(\n",
    "                    emb_size, expansion=forward_expansion, drop_p=forward_drop_p),\n",
    "                nn.Dropout(drop_p)\n",
    "            )\n",
    "            ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-7. Transformer Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerEncoder(nn.Sequential):\n",
    "    def __init__(self, depth: int = 12, **kwargs):\n",
    "        super().__init__(*[TransformerEncoderBlock(**kwargs) for _ in range(depth)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막 계층인 MLP Head(classifictaion head)는 분류 작업을 할 클래스의 확률을 보여주는 fully connected layer 입니다.\n",
    "\n",
    "확률을 보여주기 전에 전체 시퀀스(sequence)에 대해 reduction을 적용합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassificationHead(nn.Sequential):\n",
    "    def __init__(self, emb_size: int = 768, n_classes: int = 1000):\n",
    "        super().__init__(\n",
    "            Reduce('b n e -> b e', reduction='mean'),\n",
    "            nn.LayerNorm(emb_size), \n",
    "            nn.Linear(emb_size, n_classes))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. ViT\n",
    "PatchEmbedding, TransformerEncoder, ClassificationHead 클래스를 이용하여 최종적인 ViT Architecture를 구현합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1          [-1, 768, 14, 14]         590,592\n",
      "         Rearrange-2             [-1, 196, 768]               0\n",
      "    PatchEmbedding-3             [-1, 197, 768]               0\n",
      "         LayerNorm-4             [-1, 197, 768]           1,536\n",
      "            Linear-5            [-1, 197, 2304]       1,771,776\n",
      "           Dropout-6          [-1, 8, 197, 197]               0\n",
      "            Linear-7             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-8             [-1, 197, 768]               0\n",
      "           Dropout-9             [-1, 197, 768]               0\n",
      "      ResidualAdd-10             [-1, 197, 768]               0\n",
      "        LayerNorm-11             [-1, 197, 768]           1,536\n",
      "           Linear-12            [-1, 197, 3072]       2,362,368\n",
      "             GELU-13            [-1, 197, 3072]               0\n",
      "          Dropout-14            [-1, 197, 3072]               0\n",
      "           Linear-15             [-1, 197, 768]       2,360,064\n",
      "          Dropout-16             [-1, 197, 768]               0\n",
      "      ResidualAdd-17             [-1, 197, 768]               0\n",
      "        LayerNorm-18             [-1, 197, 768]           1,536\n",
      "           Linear-19            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-20          [-1, 8, 197, 197]               0\n",
      "           Linear-21             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-22             [-1, 197, 768]               0\n",
      "          Dropout-23             [-1, 197, 768]               0\n",
      "      ResidualAdd-24             [-1, 197, 768]               0\n",
      "        LayerNorm-25             [-1, 197, 768]           1,536\n",
      "           Linear-26            [-1, 197, 3072]       2,362,368\n",
      "             GELU-27            [-1, 197, 3072]               0\n",
      "          Dropout-28            [-1, 197, 3072]               0\n",
      "           Linear-29             [-1, 197, 768]       2,360,064\n",
      "          Dropout-30             [-1, 197, 768]               0\n",
      "      ResidualAdd-31             [-1, 197, 768]               0\n",
      "        LayerNorm-32             [-1, 197, 768]           1,536\n",
      "           Linear-33            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-34          [-1, 8, 197, 197]               0\n",
      "           Linear-35             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-36             [-1, 197, 768]               0\n",
      "          Dropout-37             [-1, 197, 768]               0\n",
      "      ResidualAdd-38             [-1, 197, 768]               0\n",
      "        LayerNorm-39             [-1, 197, 768]           1,536\n",
      "           Linear-40            [-1, 197, 3072]       2,362,368\n",
      "             GELU-41            [-1, 197, 3072]               0\n",
      "          Dropout-42            [-1, 197, 3072]               0\n",
      "           Linear-43             [-1, 197, 768]       2,360,064\n",
      "          Dropout-44             [-1, 197, 768]               0\n",
      "      ResidualAdd-45             [-1, 197, 768]               0\n",
      "        LayerNorm-46             [-1, 197, 768]           1,536\n",
      "           Linear-47            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-48          [-1, 8, 197, 197]               0\n",
      "           Linear-49             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-50             [-1, 197, 768]               0\n",
      "          Dropout-51             [-1, 197, 768]               0\n",
      "      ResidualAdd-52             [-1, 197, 768]               0\n",
      "        LayerNorm-53             [-1, 197, 768]           1,536\n",
      "           Linear-54            [-1, 197, 3072]       2,362,368\n",
      "             GELU-55            [-1, 197, 3072]               0\n",
      "          Dropout-56            [-1, 197, 3072]               0\n",
      "           Linear-57             [-1, 197, 768]       2,360,064\n",
      "          Dropout-58             [-1, 197, 768]               0\n",
      "      ResidualAdd-59             [-1, 197, 768]               0\n",
      "        LayerNorm-60             [-1, 197, 768]           1,536\n",
      "           Linear-61            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-62          [-1, 8, 197, 197]               0\n",
      "           Linear-63             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-64             [-1, 197, 768]               0\n",
      "          Dropout-65             [-1, 197, 768]               0\n",
      "      ResidualAdd-66             [-1, 197, 768]               0\n",
      "        LayerNorm-67             [-1, 197, 768]           1,536\n",
      "           Linear-68            [-1, 197, 3072]       2,362,368\n",
      "             GELU-69            [-1, 197, 3072]               0\n",
      "          Dropout-70            [-1, 197, 3072]               0\n",
      "           Linear-71             [-1, 197, 768]       2,360,064\n",
      "          Dropout-72             [-1, 197, 768]               0\n",
      "      ResidualAdd-73             [-1, 197, 768]               0\n",
      "        LayerNorm-74             [-1, 197, 768]           1,536\n",
      "           Linear-75            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-76          [-1, 8, 197, 197]               0\n",
      "           Linear-77             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-78             [-1, 197, 768]               0\n",
      "          Dropout-79             [-1, 197, 768]               0\n",
      "      ResidualAdd-80             [-1, 197, 768]               0\n",
      "        LayerNorm-81             [-1, 197, 768]           1,536\n",
      "           Linear-82            [-1, 197, 3072]       2,362,368\n",
      "             GELU-83            [-1, 197, 3072]               0\n",
      "          Dropout-84            [-1, 197, 3072]               0\n",
      "           Linear-85             [-1, 197, 768]       2,360,064\n",
      "          Dropout-86             [-1, 197, 768]               0\n",
      "      ResidualAdd-87             [-1, 197, 768]               0\n",
      "        LayerNorm-88             [-1, 197, 768]           1,536\n",
      "           Linear-89            [-1, 197, 2304]       1,771,776\n",
      "          Dropout-90          [-1, 8, 197, 197]               0\n",
      "           Linear-91             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-92             [-1, 197, 768]               0\n",
      "          Dropout-93             [-1, 197, 768]               0\n",
      "      ResidualAdd-94             [-1, 197, 768]               0\n",
      "        LayerNorm-95             [-1, 197, 768]           1,536\n",
      "           Linear-96            [-1, 197, 3072]       2,362,368\n",
      "             GELU-97            [-1, 197, 3072]               0\n",
      "          Dropout-98            [-1, 197, 3072]               0\n",
      "           Linear-99             [-1, 197, 768]       2,360,064\n",
      "         Dropout-100             [-1, 197, 768]               0\n",
      "     ResidualAdd-101             [-1, 197, 768]               0\n",
      "       LayerNorm-102             [-1, 197, 768]           1,536\n",
      "          Linear-103            [-1, 197, 2304]       1,771,776\n",
      "         Dropout-104          [-1, 8, 197, 197]               0\n",
      "          Linear-105             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-106             [-1, 197, 768]               0\n",
      "         Dropout-107             [-1, 197, 768]               0\n",
      "     ResidualAdd-108             [-1, 197, 768]               0\n",
      "       LayerNorm-109             [-1, 197, 768]           1,536\n",
      "          Linear-110            [-1, 197, 3072]       2,362,368\n",
      "            GELU-111            [-1, 197, 3072]               0\n",
      "         Dropout-112            [-1, 197, 3072]               0\n",
      "          Linear-113             [-1, 197, 768]       2,360,064\n",
      "         Dropout-114             [-1, 197, 768]               0\n",
      "     ResidualAdd-115             [-1, 197, 768]               0\n",
      "       LayerNorm-116             [-1, 197, 768]           1,536\n",
      "          Linear-117            [-1, 197, 2304]       1,771,776\n",
      "         Dropout-118          [-1, 8, 197, 197]               0\n",
      "          Linear-119             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-120             [-1, 197, 768]               0\n",
      "         Dropout-121             [-1, 197, 768]               0\n",
      "     ResidualAdd-122             [-1, 197, 768]               0\n",
      "       LayerNorm-123             [-1, 197, 768]           1,536\n",
      "          Linear-124            [-1, 197, 3072]       2,362,368\n",
      "            GELU-125            [-1, 197, 3072]               0\n",
      "         Dropout-126            [-1, 197, 3072]               0\n",
      "          Linear-127             [-1, 197, 768]       2,360,064\n",
      "         Dropout-128             [-1, 197, 768]               0\n",
      "     ResidualAdd-129             [-1, 197, 768]               0\n",
      "       LayerNorm-130             [-1, 197, 768]           1,536\n",
      "          Linear-131            [-1, 197, 2304]       1,771,776\n",
      "         Dropout-132          [-1, 8, 197, 197]               0\n",
      "          Linear-133             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-134             [-1, 197, 768]               0\n",
      "         Dropout-135             [-1, 197, 768]               0\n",
      "     ResidualAdd-136             [-1, 197, 768]               0\n",
      "       LayerNorm-137             [-1, 197, 768]           1,536\n",
      "          Linear-138            [-1, 197, 3072]       2,362,368\n",
      "            GELU-139            [-1, 197, 3072]               0\n",
      "         Dropout-140            [-1, 197, 3072]               0\n",
      "          Linear-141             [-1, 197, 768]       2,360,064\n",
      "         Dropout-142             [-1, 197, 768]               0\n",
      "     ResidualAdd-143             [-1, 197, 768]               0\n",
      "       LayerNorm-144             [-1, 197, 768]           1,536\n",
      "          Linear-145            [-1, 197, 2304]       1,771,776\n",
      "         Dropout-146          [-1, 8, 197, 197]               0\n",
      "          Linear-147             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-148             [-1, 197, 768]               0\n",
      "         Dropout-149             [-1, 197, 768]               0\n",
      "     ResidualAdd-150             [-1, 197, 768]               0\n",
      "       LayerNorm-151             [-1, 197, 768]           1,536\n",
      "          Linear-152            [-1, 197, 3072]       2,362,368\n",
      "            GELU-153            [-1, 197, 3072]               0\n",
      "         Dropout-154            [-1, 197, 3072]               0\n",
      "          Linear-155             [-1, 197, 768]       2,360,064\n",
      "         Dropout-156             [-1, 197, 768]               0\n",
      "     ResidualAdd-157             [-1, 197, 768]               0\n",
      "       LayerNorm-158             [-1, 197, 768]           1,536\n",
      "          Linear-159            [-1, 197, 2304]       1,771,776\n",
      "         Dropout-160          [-1, 8, 197, 197]               0\n",
      "          Linear-161             [-1, 197, 768]         590,592\n",
      "MultiHeadAttention-162             [-1, 197, 768]               0\n",
      "         Dropout-163             [-1, 197, 768]               0\n",
      "     ResidualAdd-164             [-1, 197, 768]               0\n",
      "       LayerNorm-165             [-1, 197, 768]           1,536\n",
      "          Linear-166            [-1, 197, 3072]       2,362,368\n",
      "            GELU-167            [-1, 197, 3072]               0\n",
      "         Dropout-168            [-1, 197, 3072]               0\n",
      "          Linear-169             [-1, 197, 768]       2,360,064\n",
      "         Dropout-170             [-1, 197, 768]               0\n",
      "     ResidualAdd-171             [-1, 197, 768]               0\n",
      "          Reduce-172                  [-1, 768]               0\n",
      "       LayerNorm-173                  [-1, 768]           1,536\n",
      "          Linear-174                 [-1, 1000]         769,000\n",
      "================================================================\n",
      "Total params: 86,415,592\n",
      "Trainable params: 86,415,592\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.57\n",
      "Forward/backward pass size (MB): 364.33\n",
      "Params size (MB): 329.65\n",
      "Estimated Total Size (MB): 694.56\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "class ViT(nn.Sequential):\n",
    "    def __init__(self,     \n",
    "                in_channels: int = 3,\n",
    "                patch_size: int = 16,\n",
    "                emb_size: int = 768,\n",
    "                img_size: int = 224,\n",
    "                depth: int = 12,\n",
    "                n_classes: int = 1000,\n",
    "                **kwargs):\n",
    "        super().__init__(\n",
    "            PatchEmbedding(in_channels, patch_size, emb_size, img_size),\n",
    "            TransformerEncoder(depth, emb_size=emb_size, **kwargs),\n",
    "            ClassificationHead(emb_size, n_classes)\n",
    "        )\n",
    "        \n",
    "summary(ViT(), (3, 224, 224), device='cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. ViT를 이용해서 MNIST 분류 작업\n",
    "- MNIST는 (28x28x1)이므로 ViT 논문에 나와있는 이미지와는 다르다. 따라서 28x28에 맞게끔 parameter를 수정해야 합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "BATCH_SIZE_TRAIN = 1024\n",
    "BATCH_SIZE_TEST = 1000\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_mnist = torchvision.transforms.Compose([\n",
    "    torchvision.transforms.ToTensor(),\n",
    "    torchvision.transforms.Normalize((0.1307,), (0.3081,))\n",
    "])\n",
    "\n",
    "train_dataset = torchvision.datasets.MNIST('/data/mnist', train=True, download=True,\n",
    "                                       transform=transform_mnist)\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE_TRAIN, shuffle=True)\n",
    "\n",
    "test_dataset = torchvision.datasets.MNIST('/data/mnist', train=False, download=True,\n",
    "                                      transform=transform_mnist)\n",
    "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=BATCH_SIZE_TEST, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1\n",
      "[    0/60000 (  0%)]  Loss: 2.3608\n",
      "\n",
      "Average test loss: 2.2755  Accuracy: 1141/10000 (11.41%)\n",
      "\n",
      "Epoch: 2\n",
      "[    0/60000 (  0%)]  Loss: 2.2721\n",
      "\n",
      "Average test loss: 0.9613  Accuracy: 6751/10000 (67.51%)\n",
      "\n",
      "Epoch: 3\n",
      "[    0/60000 (  0%)]  Loss: 0.9662\n",
      "\n",
      "Average test loss: 0.5506  Accuracy: 8211/10000 (82.11%)\n",
      "\n",
      "Epoch: 4\n",
      "[    0/60000 (  0%)]  Loss: 0.5761\n",
      "\n",
      "Average test loss: 0.3705  Accuracy: 8878/10000 (88.78%)\n",
      "\n",
      "Epoch: 5\n",
      "[    0/60000 (  0%)]  Loss: 0.3964\n",
      "\n",
      "Average test loss: 0.2672  Accuracy: 9151/10000 (91.51%)\n",
      "\n",
      "Epoch: 6\n",
      "[    0/60000 (  0%)]  Loss: 0.2644\n",
      "\n",
      "Average test loss: 0.2323  Accuracy: 9291/10000 (92.91%)\n",
      "\n",
      "Epoch: 7\n",
      "[    0/60000 (  0%)]  Loss: 0.2442\n",
      "\n",
      "Average test loss: 0.2008  Accuracy: 9372/10000 (93.72%)\n",
      "\n",
      "Epoch: 8\n",
      "[    0/60000 (  0%)]  Loss: 0.1976\n",
      "\n",
      "Average test loss: 0.2015  Accuracy: 9360/10000 (93.60%)\n",
      "\n",
      "Epoch: 9\n",
      "[    0/60000 (  0%)]  Loss: 0.1794\n",
      "\n",
      "Average test loss: 0.1689  Accuracy: 9449/10000 (94.49%)\n",
      "\n",
      "Epoch: 10\n",
      "[    0/60000 (  0%)]  Loss: 0.1933\n",
      "\n",
      "Average test loss: 0.1711  Accuracy: 9460/10000 (94.60%)\n",
      "\n",
      "Execution time: 387.51 seconds\n"
     ]
    }
   ],
   "source": [
    "def train_epoch(model, optimizer, data_loader, loss_history):\n",
    "    total_samples = len(data_loader.dataset)\n",
    "    model.train()\n",
    "\n",
    "    for i, (data, target) in enumerate(data_loader):\n",
    "        data = data.to(device)\n",
    "        target = target.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        output = F.log_softmax(model(data), dim=1)\n",
    "        loss = F.nll_loss(output, target)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if i % 100 == 0:\n",
    "            print('[' +  '{:5}'.format(i * len(data)) + '/' + '{:5}'.format(total_samples) +\n",
    "                  ' (' + '{:3.0f}'.format(100 * i / len(data_loader)) + '%)]  Loss: ' +\n",
    "                  '{:6.4f}'.format(loss.item()))\n",
    "            loss_history.append(loss.item())\n",
    "\n",
    "def evaluate(model, data_loader, loss_history):\n",
    "    model.eval()\n",
    "    \n",
    "    total_samples = len(data_loader.dataset)\n",
    "    correct_samples = 0\n",
    "    total_loss = 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for data, target in data_loader:\n",
    "            data = data.to(device)\n",
    "            target = target.to(device)\n",
    "            output = F.log_softmax(model(data), dim=1)\n",
    "            loss = F.nll_loss(output, target, reduction='sum')\n",
    "            _, pred = torch.max(output, dim=1)\n",
    "            \n",
    "            total_loss += loss.item()\n",
    "            correct_samples += pred.eq(target).sum()\n",
    "\n",
    "    avg_loss = total_loss / total_samples\n",
    "    loss_history.append(avg_loss)\n",
    "    print('\\nAverage test loss: ' + '{:.4f}'.format(avg_loss) +\n",
    "          '  Accuracy:' + '{:5}'.format(correct_samples) + '/' +\n",
    "          '{:5}'.format(total_samples) + ' (' +\n",
    "          '{:4.2f}'.format(100.0 * correct_samples / total_samples) + '%)\\n')\n",
    "    \n",
    "    \n",
    "N_EPOCHS = 10\n",
    "\n",
    "start_time = time.time()\n",
    "# MNIST에 맞게 Hyper-parameter 수정.\n",
    "model = ViT(in_channels=1, patch_size=7, emb_size=128, img_size=28, depth=64, n_classes=10).to(device)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.003)\n",
    "\n",
    "train_loss_history, test_loss_history = [], []\n",
    "for epoch in range(1, N_EPOCHS + 1):\n",
    "    print('Epoch:', epoch)\n",
    "    train_epoch(model, optimizer, train_loader, train_loss_history)\n",
    "    evaluate(model, test_loader, test_loss_history)\n",
    "\n",
    "print('Execution time:', '{:5.2f}'.format(time.time() - start_time), 'seconds')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvBklEQVR4nO3deXyU5b3//9dnZpJM9j2QECRAQgDZCYgCkohUlFasW6uIS4+1Vtva2lrtt7XL+ban9qdf9VgXjlW0dcFarVqFo1RlU0Q22RSEsIcAWSAh+zbX7497WE0gQCbXJPN5Ph7zSGbmnns+mYfOm/u+Pvd1iTEGpZRSoctluwCllFJ2aRAopVSI0yBQSqkQp0GglFIhToNAKaVCnMd2AacrJSXFZGVl2S5DKaW6lFWrVpUZY1Jbe67LBUFWVhYrV660XYZSSnUpIrKzref01JBSSoU4DQKllApxGgRKKRXiutwYgVKq+2pqaqKoqIj6+nrbpXRZXq+XzMxMwsLC2v0aDQKlVNAoKioiNjaWrKwsRMR2OV2OMYby8nKKioro27dvu1+np4aUUkGjvr6e5ORkDYEzJCIkJyef9hGVBoFSKqhoCJydM/n8QiYItpfV8P/mf8nizaVUNzTbLkcppYJGyIwRrN9TyRMLCvEZcAkMzogjr08SY/smkZeVSFqs13aJSinLKioqePnll7njjjtO+7WXXXYZL7/8MgkJCe3a/re//S0xMTH87Gc/O+336mghEwSXD8/gooFpfLbrICu2H2DFjoO8smIXzy/dAUBWchRjspKcW98kspKj9BBVqRBTUVHBk08+2WoQNDc34/G0/ZU5b968QJYWUCETBAAxER4m5qQyMceZbqOx2cfnxZWs2OEEw/sb9/OPVUUApMREMCYrkbysJMZmJTEoPRaPO2TOpCkVku677z62bt3KiBEjmDJlCtOmTeP+++8nMTGRTZs2sXnzZq644gp2795NfX09d911F7fddhtwdPqb6upqLr30UiZMmMDSpUvp1asXb731FpGRkW2+75o1a7j99tupra2lf//+zJ49m8TERB577DFmzZqFx+Nh8ODBvPLKKyxatIi77roLcMYDFi9eTGxs7Fn93SEVBCcK97gYeU4iI89J5LYLweczbCurZvn2g/5wOMD/btgHQHS4m1F9Esnrk8SYvomM7J1IZLjb8l+gVPf1u7c/54viQx26z8EZcfzmG+e2+fwDDzzAhg0bWLNmDQALFy5k9erVbNiw4Ug75uzZs0lKSqKuro4xY8Zw1VVXkZycfNx+tmzZwpw5c/jLX/7Ctddey+uvv84NN9zQ5vveeOON/PnPf2bSpEn8+te/5ne/+x2PPvooDzzwANu3byciIoKKigoAHnroIZ544gnGjx9PdXU1Xu/Zn9YO6SA4kcslZKfFkp0Wy/XnnQPA3so6Vuw4fDrpAI9+sBljwOMShvSKd8YY+iQyJiuJxOhwy3+BUqqjjR079rie/Mcee4w33ngDgN27d7Nly5avBEHfvn0ZMWIEAKNHj2bHjh1t7r+yspKKigomTZoEwE033cQ111wDwLBhw5gxYwZXXHEFV1xxBQDjx4/n7rvvZsaMGVx55ZVkZmae9d+oQXAK6fGRXD48ksuHZwBQWdvE6l0HWb7jACt3HOD5j3fw9OJtAGSnxfjHGZxgyEyM1HEGpc7Qyf7l3pmio6OP/L5w4ULef/99PvnkE6KiosjPz2+1Zz8iIuLI7263m7q6ujN677lz57J48WLefvtt/vCHP7B+/Xruu+8+pk2bxrx58xg/fjzvvfceAwcOPKP9H6ZBcJrio8IoGJhGwcA0AOqbWlhXVHnkVNI7a4uZs3wXAOnxXv8YgzPWkNsjFpdLg0GpYBUbG0tVVVWbz1dWVpKYmEhUVBSbNm1i2bJlZ/2e8fHxJCYmsmTJEiZOnMgLL7zApEmT8Pl87N69m4KCAiZMmMArr7xCdXU15eXlDB06lKFDh7JixQo2bdqkQdBuW/4N8+6BW9+H6JQO2603zM3Yvk4bKkCLz/DlvipW7jzA8u0HWL69nLfXFgMQ6/WQ1yeRK0b2YvqIXh1Wg1KqYyQnJzN+/HiGDBnCpZdeyrRp0457furUqcyaNYtBgwaRm5vLuHHjOuR9//rXvx4ZLO7Xrx/PPfccLS0t3HDDDVRWVmKM4Uc/+hEJCQncf//9LFiwAJfLxbnnnsull1561u8vxpgO+DM6T15enjmjhWn2rIa/FMA3n4bh3+r4wtpgjKHoYN2RI4aPCsvYfaCO5f9nMmlxeu2CUsfauHEjgwYNsl1Gl9fa5ygiq4wxea1tHzr9kOkjIDoVCv/dqW8rIvROiuLKUZn88cphzLphNACLNpd2ah1KKdWW0AkClwv6T4bCD8DXYq2MwelxpMVGsPBLDQKlVHAInSAAyJkCdQeg+DNrJYgIBblpLN5SSnOLz1odSil1WGgFQf+LQFzOwLFF+bmpVNU3s3pXhdU6lFIKQi0IopKg1+hOHyc40ficFDwuYeGXJVbrUEopCLUgAMie4nQQ1ZRZKyHOG8boPoks0HECpVQQCL0gyLkYMLD1Q6tlFAxMY+PeQ+yr1LVZlQoWh2cfPVOPPvootbW1rT6Xn5/PGbW+d4LQC4L0kRCVEhTjBACLNuvpIaWCRSCDIJiFXhC4XJA9GbZ+AD57XTu5PWJJj/eyYJOeHlIqWBw7DfU999wDwIMPPsiYMWMYNmwYv/nNbwCoqalh2rRpDB8+nCFDhvD3v/+dxx57jOLiYgoKCigoKDjp+8yZM4ehQ4cyZMgQ7r33XgBaWlq4+eabGTJkCEOHDuWRRx4BnEnuBg8ezLBhw/j2t78dkL87dKaYOFb2FFj3d6eNNHO0lRJEhPzcVN5eu5emFh9hutaBUsf73/tg3/qO3WfPoXDpA20+feI01PPnz2fLli0sX74cYwyXX345ixcvprS0lIyMDObOnQs4cxDFx8fz8MMPs2DBAlJS2p7Gpri4mHvvvZdVq1aRmJjI1772Nd5880169+7Nnj172LBhA8CRaadbm4q6o4Xmt0//iwCx3j2Un5tGdUMzK3cctFqHUqp18+fPZ/78+YwcOZJRo0axadMmtmzZwtChQ/n3v//Nvffey5IlS4iPj2/3PlesWEF+fj6pqal4PB5mzJjB4sWL6devH9u2beOHP/wh7777LnFxccDRqahffPHFk66QdjZC84ggOtlpI93yb8i/z1oZ47NTCHMLCzeXcH7/5FO/QKlQcpJ/uXcWYwy/+MUv+N73vveV51avXs28efP41a9+xeTJk/n1r399Vu+VmJjI2rVree+995g1axavvvoqs2fPbnUq6o4OhNA8IgDnKuM9q6Cm3FoJMREexmQlsVDHCZQKCidOQ33JJZcwe/ZsqqurAdizZw8lJSUUFxcTFRXFDTfcwD333MPq1atbfX1rxo4dy6JFiygrK6OlpYU5c+YwadIkysrK8Pl8XHXVVfz+979n9erVx01F/ac//YnKysojtXSk0DwiAGecYOEfnTbSYddYKyM/N5X/mreJ4oo6MhLaXtNUKRV4J05D/eCDD7Jx40bOP/98AGJiYnjxxRcpLCzknnvuweVyERYWxlNPPQXAbbfdxtSpU8nIyGDBggWtvkd6ejoPPPAABQUFGGOYNm0a06dPZ+3atdxyyy34/E0sf/zjH9ucirqjhc401Cfy+eChHKeD6Mqnz35/Z2jL/iqmPLKY//rm0CPLYyoVqnQa6o6h01C31+E20sL3rbaRZqfF0CshUqebUEpZE7pBAM7podpy67OR5uem8nFhGY3NOhupUqrzhXYQBFEbaU1jCyt3HLBah1LBoKudrg42Z/L5hXYQHNtGatEF/ZMJd7tYoKeHVIjzer2Ul5drGJwhYwzl5eV4vae3DG7odg0dljMFFj7gtJFG2+nlj47wMLZvEgu/LOWX0069vVLdVWZmJkVFRZSWakv1mfJ6vWRmZp7WazQIgqiN9PdzN1J0sJbMxChrdShlU1hYGH379rVdRsgJ2KkhEektIgtE5AsR+VxE7mplGxGRx0SkUETWicioQNXTpoyREJUcFOMEgK5lrJTqdIEcI2gGfmqMGQyMA+4UkcEnbHMpkOO/3QY8FcB6Wnfcovb2unb6p0bTO0nbSJVSnS9gQWCM2WuMWe3/vQrYCPQ6YbPpwN+MYxmQICLpgaqpTTlToLYM9lpuIx2QxseF5TQ0t1irQykVejqla0hEsoCRwKcnPNUL2H3M/SK+GhaIyG0islJEVgZkEKn/ZEBgy/sdv+/TUDAwlbqmFpZv1zZSpVTnCXgQiEgM8DrwY2PMoTPZhzHmaWNMnjEmLzU1tWMLBH8b6Sjr4wTn90sh3OPScQKlVKcKaBCISBhOCLxkjPlnK5vsAXofcz/T/1jny54CRSuh1t6/xiPD3Yzrl6zXEyilOlUgu4YEeBbYaIx5uI3N/gXc6O8eGgdUGmP2Bqqmk8qZQjAsap8/IJVtpTXsKu96654qpbqmQB4RjAdmAheJyBr/7TIRuV1EbvdvMw/YBhQCfwHuCGA9J5cxEiKTrF9lXDDQ30aqi9orpTpJwC4oM8Z8BMgptjHAnYGq4bS43MfPRuqyM/tG35Ro+iRHsfDLUm48P8tKDUqp0BLacw2dKPtwG+kaq2UU5KaxdGsZ9U3aRqqUCjwNgmNl+9tIC+22kU7KTaW+ycen2kaqlOoEGgTHik5xxgosjxOc3y+ZCI+LBZt0nEApFXgaBCfKmQJ77LaResPcnN8/mUWb9XoCpVTgaRCcKOdrYHzW20gLctPYXlbDjrIaq3Uopbo/DYITBUkbaX6ucwW1TkKnlAo0DYITndhGakmf5Gj6pUSzQKebUEoFmAZBa4KkjXRSbirLtpVT16htpEqpwNEgaE2QtJEW5KbR0Oxj2bZyq3Uopbo3DYLWBEkb6di+SUSGuXWcQCkVUBoEbQmSNtIL+iez4MtSnNk4lFKq42kQtCV7SlC0kebnprLrQC3btY1UKRUgGgRt6TXKaSO1PE5weFF77R5SSgWKBkFbXG7of5H1NtLeSVH0T43WcQKlVMBoEJxMzhSoKYV9a62WUZCbxqfbDlDb2Gy1DqVU96RBcDL9Jzs/LS9qn5+bRmOLj0+2ahupUqrjaRCcTEyq00ZqeVH7MX0TiQp361rGSqmA0CA4lewpULTCahtphMfNBf1TWLBJ20iVUh1Pg+BUcvxtpNsWWC2jYGAqeyrq2FpabbUOpVT3o0FwKr1GQ2RiUIwTACzYpG2kSqmOpUFwKkHSRtorIZIBPWJYuFnHCZRSHUuDoD2yp0BNCexbZ7WMgtw0lm8/QHWDtpEqpTqOBkF7ZPvbSC13D03KTaWpxbC0sMxqHUqp7kWDoD1i0iB9hPVxgrw+ScREeHS6CaVUh9IgaK+cKVC03GobabjHxfjsZBZ9WaJtpEqpDqNB0F6HF7W33Uaam0ZxZT2b92sbqVKqY2gQtFeQtJFO0kXtlVIdTIOgvYKkjTQ9PpKBPWN1ugmlVIfRIDgdQdJGmp+bxsodB6mqb7Jah1Kqe9AgOB1B0kZakJtKs8/wsbaRKqU6gAbB6QiSNtJRfRKJjfCwUNtIlVIdQIPgdB1uI607aK2EMLeLiQNSWKiL2iulOoAGwek6sqi93TbS/AFp7DtUz6Z9VVbrUEp1fRoEpyszD7wJ1he1P9xGqt1DSqmzpUFwuoKkjbRHnJfB6XE6TqCUOmsaBGciZwpU74f9662WUTAwlVU7D1JZp22kSqkzF7AgEJHZIlIiIhvaeD5fRCpFZI3/9utA1dLhsi92fm6x20aan5tGi7aRKqXOUiCPCJ4Hpp5imyXGmBH+238GsJaOFZMG6cOtjxOM7J1AnNfDgk06TqCUOnMBCwJjzGLA3lSdgZY9BXYvh7oKayV43C4mDkhl4WZtI1VKnTnbYwTni8haEflfETm3rY1E5DYRWSkiK0tLg2RwNGcKmJagmI20tKqBz4sPWa1DKdV12QyC1UAfY8xw4M/Am21taIx52hiTZ4zJS01N7az6Tq5XHnjjrV9lPGmA83ks2hwkAamU6nKsBYEx5pAxptr/+zwgTERSbNVz2tyeo22kFk/LpMZGMLRXvI4TKKXOmLUgEJGeIiL+38f6aym3Vc8ZyZ4C1ftgn9020vzcVFbvOkhlrbaRKqVOXyDbR+cAnwC5IlIkIv8hIreLyO3+Ta4GNojIWuAx4Numq414HmkjnW+1jPzcNHwGFm/R00NKqdPnCdSOjTHXneL5x4HHA/X+nSK2B/Qc5pweuvBn1soY0TuBhKgwFn5ZyjeGZ1irQynVNdnuGur6cr5mvY3U7RIuzEll0eYSfL6udVCllLJPg+BsBUkbaX5uKmXVjdpGqpQ6bRoEZytI2kgvHJCKiM5GqpQ6fRoEZytI2khTYiIY1iuehRoESqnTpEHQEYKmjTSNz3ZXcLCm0WodSqmuRYOgIxxuI7W8qH1+bipG20iVUqdJg6AjHG4jtTxOMCwzgaTocF2sRil1WjQIOkrOFNj9aRC0kaawaHOptpEqpdpNg6CjZB9uI11otYyCgWkcqGlk3Z5Kq3UopbqOdgWBiNwlInHieFZEVovI1wJdXJeSOcZpI7U8TjAxx2kj1e4hpVR7tfeI4DvGmEPA14BEYCbwQMCq6orcHuhXAIUfWG0jTYoOZ0TvBBboOIFSqp3aGwTi/3kZ8IIx5vNjHlOH5UyBqr2wv9VlmjtN/oA01hVVUF7dYLUOpVTX0N4gWCUi83GC4D0RiQV8gSuriwqSRe0LBmobqVKq/dobBP8B3AeMMcbUAmHALQGrqquK7Qk9h1pf1H5IRjwpMeEs2KRBoJQ6tfYGwfnAl8aYChG5AfgVoG0prcmeAruWQb29j8flEi4ckMriLaW0aBupUuoU2hsETwG1IjIc+CmwFfhbwKrqynKCo400PzeNitom1uyusFqHUir4tTcImv2rh00HHjfGPAHEBq6sLixzLETEWx8nuDAnBZfAIm0jVUqdQnuDoEpEfoHTNjpXRFw44wTqRG4P9M+3PhtpQlQ4I89J1DZSpdQptTcIvgU04FxPsA/IBB4MWFVdXXZwtJEW5Kayfk8lpVXaRqqUalu7gsD/5f8SEC8iXwfqjTE6RtCWIGkjzc9NA2DRZj0qUEq1rb1TTFwLLAeuAa4FPhWRqwNZWJcWlx4UbaSD0+NIjY3Q6SaUUiflaed2v8S5hqAEQERSgfeB1wJVWJeXPQU+/m+njdQbb6UEl0uYNCCV+Z/vo7nFh8etcwwqpb6qvd8MrsMh4Fd+Gq8NTUHSRlqQm8ah+mZtI1VKtam9X+bvish7InKziNwMzAXmBa6sbiBI2kgn5KTgdokuaq+UalN7B4vvAZ4GhvlvTxtj7g1kYV3ekTZSu7ORxkeGMfqcRF21TCnVpnaf3jHGvG6Mudt/eyOQRXUb2VOgqhj2f261jPyBqXxefIiSQ/VW61BKBaeTBoGIVInIoVZuVSJyqLOK7LKCZVH7AU4b6UJtI1VKteKkQWCMiTXGxLVyizXGxHVWkV1WXDr0GGp9UftB6bH0iNM2UqVU67TzJ9ByLobdy6De3gGUiJA/II0lW8poatFlJJRSx9MgCLTsKeBrtt9GOjCVqvpmVu88aLUOpVTw0SAItN5jISLO+jjB+OwUPC7RcQKl1FdoEASaOwz65TvjBBbbSGO9YeRlJbJgk44TKKWOp0HQGXL8baQlX1gtIz83jU37qthXqW2kSqmjNAg6Q5DMRlrgn41Uu4eUUsfSIOgMcRnQY4j12UgH9IghPd6rVxkrpY6jQdBZsi+GXZ/YbyPNTeOjwjIam7WNVCnlCFgQiMhsESkRkVaX6RLHYyJSKCLrRGRUoGoJCjn+NtLti6yWkZ+bSnVDM6u0jVQp5RfII4Lngaknef5SIMd/uw14KoC12Nf7PKeNdMt8q2WMz04hzC06TqCUOiJgQWCMWQwcOMkm04G/GccyIEFE0gNVj3XuMOg3yXobaUyEhzFZSTpOoJQ6wuYYQS9g9zH3i/yPfYWI3CYiK0VkZWlpF/4Cyw6ONtKC3DS+3F9FcUWd1TqUUsGhSwwWG2OeNsbkGWPyUlNTbZdz5nKmOD+tL2rvfIZ6VKCUArtBsAfofcz9TP9j3VeQtJFmp8XQKyGSueuLMRZPUymlgoPNIPgXcKO/e2gcUGmM2Wuxns4xYCrs/Bg2vmOtBBHhpgv68HFhOb97+wsNA6VCnCdQOxaROUA+kCIiRcBvgDAAY8wsnDWPLwMKgVrglkDVElQm3u3MRPrad2DmG5A13koZ353Yj5JDDTzz0Xa8YW7unZqLiFipRSllV8CCwBhz3SmeN8CdgXr/oBUeDTP+AbOnwpxvwy3zoOfQTi9DRPjltEHUNbUwa9FWosLd/GhyTqfXoZSyr0sMFnc7UUkw858QEQsvXAkHtlspQ0T4v9OHcNWoTB7+92aeXrzVSh1KKbs0CGyJz3RODfma4IVvQrWdC7xcLuH/u3oYXx+Wzn/N28QLn+ywUodSyh4NAptSc2HGa1C9H168EuorrZThdgmPfGsEUwb34P63PufVlbtP/SKlVLehQWBbZh5c+wKUbIRXZkCTnbUCwtwuHr9+JBNzUrj39XW8taZ7d/IqpY7SIAgGORfDFbNgxxL4563ga7FSRoTHzdMz8xiblcTdr67l3Q37rNShlOpcGgTBYtg1MPVPsPFtmHu3tfmIIsPdPHvzGIZlxvPDOat1cjqlQoAGQTAZdztM/Cmseh4W/MFaGTERHp6/ZSwDesTyvRdWsXRrmbValFKBp0EQbC66H0bdCIsfhGWzrJURHxnGC/9xHn2So7j1rytZueNkE8kqpboyDYJgIwLTHoGBX4d374X1r1krJSk6nBdvPY8ecV5ueW4F64oqrNWilAocDYJg5PbAVc9CnwnwxvesTlKXFuvlpVvPIz4qjJnPLmfjXntLbSqlAkODIFiFeeG6lyF1EPz9Rihaaa2UjIRIXr51HJFhbmY++ymFJdXWalFKdTwNgmDmjYcbXoeYVHjpGij90lop5yRH8dJ3zwNgxjPL2FleY60WpVTH0iAIdrE9nKkoXB5nXqLKImul9E+N4cVbz6Oh2cf1f/lUVzhTqpvQIOgKkvo5RwYNh5wwqLXXwTOwZxwvfOc8DtU1cf1fllFyyM6V0EqpjqNB0FWkD4Pr5sDBHfDytdBo79TM0Mx4nv/OGEqqGpjxzKeUVzdYq0UpdfY0CLqSrAlw9bOwZxW8eiO0NFkrZXSfJJ65KY9dB2qZ+exyKmvt1aKUOjsaBF3NoG/A1x91WkrfvAN8PmulXNA/hf+ZOZotJVXc9NxyqhuardWilDpzGgRd0eibYPKvYf2rMP+X1uYlAsjPTePx60exfk8l33l+BXWNdibMU0qdOQ2CrmrC3XDe92HZk/DRI1ZLueTcnjzyrRGs2HGA215YSX2ThoFSXYkGQVclApf8Fwy9Bj74Haz6q9VyLh+ewZ+uGsaSLWX84OXVNLXYO2WllDo9GgRdmcsF05+E7IvhnR/DxneslnNtXm/+7/RzeX9jCT9+ZQ3NGgZKdQkaBF2dJxyu/RtkjILXvgM7PrJazszzs/jlZYOYu34vP39tHT6fvfELpVT7aBB0B+HRMOMfkJgFc66DfeutlvPdC/tx95QB/POzPfzyzQ0Yi4PZSqlT0yDoLqKSYOY/ISLWufr4wHar5fzwomy+n9+fOct38Z/vfKFhoFQQ0yDoTuIznXmJfE3wwjehar+1UkSEn1+Sy80XZPHcxzt4aL69CfOUUienQdDdpObCjNegej+8dBXUV1orRUT4zTcGc93Y3jyxYCuPf7jFWi1KqbZpEHRHmXnwrRegZCPMuR6a7E0MJyL8/oqhfHNkLx6av5lnlmyzVotSqnUaBN1V9sVwxSzY+RG8/h/gs3eRl9slPHj1MC4b2pPfz93IC8t2WqtFKfVVGgTd2bBrYOqfYNM78M5PrE5F4XG7ePRbI5k8MI3739zAa6vsrauglDqeBkF3N+52mPgzWP1X+PD3VksJ97h4YsYoJmSn8PPX1vL22mKr9SilHBoEoeCiX8Gom2DJQ7BsltVSvGFunr5xNHl9kvjJ39cw//N9VutRSmkQhAYRmPYwDPw6vHsvrPuH1XKiwj08e3Me5/aK5wcvf8Zrq4po0SuQlbJGgyBUuD1w1bPQZwK8eTtsed9qObHeMP52y1gGpcfys3+s5eKHF/Hqit00Nuv8REp1Ng2CUBLmhetehrRB8OpM2PWp1XLio8L45x3jeXLGKKLC3fz89XXkP7iA5z/erlNZK9WJpKtd+p+Xl2dWrlxpu4yurboEnv0aVOyEc78JE38KPc61WpIxhoWbS3niw0JW7jxISkw435nQl5nj+hDrDbNam1LdgYisMsbktfqcBkGIqimDpY/BimehsRpyL3O6izJHWy3LGMPy7Qd4fEEhS7aUEef1cPMFWdw8vi9J0eFWa1OqK7MWBCIyFfhvwA08Y4x54ITnbwYeBPb4H3rcGPPMyfapQdDBag/A8qdh2VNQXwH98p1AyJrgDDJbtHZ3BU8uLOS9z/cTFe7m+rHn8N0L+9Ejzmu1LqW6IitBICJuYDMwBSgCVgDXGWO+OGabm4E8Y8wP2rtfDYIAaaiClbNh6eNQUwK9z3MCIWeK9UDYvL+KpxZu5V9ri3GLcHVeJt+f1J/eSVFW61KqK7EVBOcDvzXGXOK//wsAY8wfj9nmZjQIgktTHXz2Inz831C5G3oOc8YQBl3urIhm0a7yWp5atJXXVxXRYgzTh2fw/fz+5PSItVqXUl2BrSC4GphqjLnVf38mcN6xX/r+IPgjUIpz9PATY8zuVvZ1G3AbwDnnnDN6506dqybgWppg3d9hycNwYCukDIAJd8PQq8Ftd/B2X2U9f1myjZc/3UV9cwuXDO7JnQXZDM2Mt1qXUsEsmIMgGag2xjSIyPeAbxljLjrZfvWIoJP5WuCLN51A2L8BEs6B8T+GETOcdlSLDtQ08tzH23l+6Q6q6pu5cEAqPyjIZmzfJKt1KRWMgvbU0Anbu4EDxpiT/rNOg8ASY2Dzu7D4IdizEmJ6wgU/hLxbnKUyLTpU38SLy3by7JLtlNc0MiYrkTsLspk0IBWxPL6hVLCwFQQenNM9k3G6glYA1xtjPj9mm3RjzF7/798E7jXGjDvZfjUILDMGti9yAmHHEohMgnF3wNjvQmSC1dLqGlt4ZcUunl68jb2V9QzpFced+dlccm5PXC4NBBXabLaPXgY8itM+OtsY8wcR+U9gpTHmXyLyR+ByoBk4AHzfGLPpZPvUIAgiu5c7gbDlPYiIgzG3wvl3QnSK1bIam3288VkRTy3cyo7yWrLTYrgjvz/fGJ5BmFsvplehSS8oU4G1dx0s+X/wxVvg8cLom53TRvG9rJbV4jPMXb+XJxcUsmlfFZmJkdw+qT9Xj87EG+a2WptSnU2DQHWOsi3w0SNOtxECI66HCT+GpH5Wy/L5DB9uKuHxBYWs2V1BWmwE353Yj+vPO4foCI/V2pTqLBoEqnMd3Olch/DZi+BrgiFXw8S7ncnuLDLGsHRrOU8sKGTp1nISosK45YK+3HxBFvFROp+R6t40CJQdVftg6Z9h5XPQVOOsh3DhzyBjpO3KWL3rIE98WMgHm0qIifBww7g+zDjvHL1aWXVbGgTKrtoDzlxGy/8H6iuh/2QnEPpcYLsyvig+xJMLC5m7fi/GQF6fRKaPyOCyoekkx0TYLk+pDqNBoIJD/SFY8Qx88gTUlsE5F8CFP3WCwXK//+4DtfxrbTFvrdnD5v3VuF3CxJwUpo/IYMrgnsToWILq4jQIVHBprIXPXnDGEQ7tca5W7n+REwh9L7R+PcKmfYd4a00x/1pTzJ6KOrxhLqYM7sn04RlcOCCVcI+2oKquR4NABafmRlj/D9g0F7YvhsYqEDdk5h0Nhl6jwGWn1dPnM6zadZC31uxh7rq9HKxtIj4yjMuGpjN9RAZjs5L0QjXVZWgQqODX0gRFK6DwA9j6IRR/BhjwxjtrJBwOhoTeVspravHx0ZYy3lqzh/lf7Ke2sYX0eC/fGJ7B5cMzODcjTqezUEFNg0B1PTXlsH0hFH7oBENVsfN4cg5kT3aCIWuClXmOahubeX9jCf9as4eFX5bS7DP0T41m+oheTB+RQZ9ku3MvKdUaDQLVtRkDpZucQCj8AHZ+DM314A53FtA5HAw9hnb6mgkHaxr53w37eHPNHpZvPwDAiN4JTB+RwbRh6aTF6mpqKjhoEKjupakedi11gmHrAmd6bIDoVOhX4ARDvwKI7dGpZRVX1PH22mLeWlPMF3sP4RIYn53C5cMzuGRIT+K8etGaskeDQHVvVfv8oeC/1ZY7j/cYCv39wdB7XKeun7Blf5W/HbWYXQdqCfe4uHhQGpcP70V+bqrOdaQ6nQaBCh0+H+xbB1s/cI4Wdi1zprnwRDpjCv0vcoIhZUCnXLtgjGHN7greWlPMO+uKKatuJNbr4dIhPZk+ohfj+iXj1s4j1Qk0CFToaqiGHR/5jxY+gPJC5/G4TOdoof9FTldSVOBXNWtu8fHJtnLe/KyY9z7fR3VDM6mxEXxjWAbTR2QwLDNeO49UwGgQKHXYwZ1HQ2HbYmioBMSZ/yhlAMSkQUwP/y3t6P3IxA49gqhvauHDTSW8tWYPCzaV0tjiIys5imnD0slIiCQmwkOs10NMRBjREW5iI8KI8XqIifDoBW3qjGgQKNWalmYoXu10Im1fDJW7oboEWhq+uq0r7GgwRKe1HRgxaRAec1qhUVnXxHsb9vHW2j0s3VrOqf6XDPe4iInwHL15PcRGeIg+5vfDj8ec8Pux4RId7tEL4kKIBoFS7WWMMzFedQlU73duNaX+3495rLoUakrA+L66j7Co9gVGdNpXBrDrGls4VN9EVX0zNQ3NVDc0U1Xv/Kyub3J+NrRQ3dBEdf3R52sam4+739DcSl2tODYooiOOhkh8ZBi9EiPplRBJZmIkmUlR9Izz6nhGF3ayINCZtJQ6logz11FkAqQOOPm2vhZnZtWTBUb5Vti5FOoOtL6PiPjjwiEyJo3I6BR6RKdCVIrTEpuY4iz/GRHX7iONxmbfkSA5cqtvpsr/s6bh6O/VDccETH0TJVX1HKxtorTq+CMjj0tIT/CSmRBFr0R/QCRG+X9G0jPOi0eXAu2SNAiUOlMuN8SkOjeGnHzb5sajQXEkMI4NjVLYu9a531jV+j7c4f5wSHYC4nBQHL5/5LEUwqNTCI+KITE6/Iz/vPqmFoor6ig6WMeeijqKDtZSdNC5v2RLKfsPHR8UbpfQM877lYDolRhJ78QoesZ7dc3oIKVBoFRn8IQ7azi3Zx3npnpnmu6aUmeqjZrS1u+XFzr3m2raeE/vkWBwbqkQdTg0Uo4LDqJTvjJdhzfMTb/UGPqlxrS6+4bmFvZW1PvDofbIzz0VdSzdWsa+Q/XHjXe4BNLjnWDITDjxiMIJCh0It0ODQKlgE+aF+Ezn1h6Nte0LjtIvnd+b69t436ijwRCZAAjg/yY/8o1+9H4EkIUh67jnAS/Q0+DrYWhsbqGh2UdDczONzT4am1poKPPRuK+Fppaj4xilGEqBcLcQ4XER7nYR7nHhDougyZuMLyoFE52GxKTgiU3DE9cDb3wPvAk9CY9OsL6eRaAZY46M+wTiYkQNAqW6uvAoCD/HWdfhVIyBxhp/UBy+lX71fn3l0dcc+ZKV9t33P+YS8IZ58IYBcsJqbyL4DDS2GOqbWqhv9tHQ5KO+2Uddk4+DjT4aaluIoIYUikmWQyRI60c+DSaMAxJHpSRQ6U6k2p1AbVgideHJNHqT/UGSii8qFVd0ElGRkUSFu4mJ8BAV7nE6qCI8RPt/jwr3tGtQ/PCXc31TC3VNLdQ3+ahrbKG+ucX5m054rK7RCcW6xpbjXnN427pjXnP8Yz7qm1swBu7I78/Ppw48ZW2nS4NAqVAiAhExzi0xy2opLpyDh7Ym/mhq8XGwtpHahhaKGprZUldHY1UpLVX7MdVlSE0p7tpSPPXlhDeU4204QHrzAaKbdxDXUEEYza3u94CJodzEU04cZSaeL00c5SaOcuIpM85j1Z4E6sOTITyGaP8cUQ3HfFnXNTlf6mfSdOkEpJvIMDfeMDfeMNeR+1HhHpKi3Xg9QmSYEBXmwusRosKcI6WRfeJP/w3bQYNAKRWUwtwuZ/bW2MOPxAM9gaGnfvHhNuDDRzg1JbRUldBUVUJEVQnpVSWk15bhqi0hrH4jYY2VX91HCzTWh1PVmECjy4tLDC4MrjBwhRkE/30BwYcL56ec+NM424rxAQb89zE+aDLQePhxn/8U20nSxfwYcn53Gp9i+2gQKKW6n2PbgFOyAXD7b61qbjw6rlJd6g+PUsJrSkmuKYWmOhCXs19xvvK/el9aeb6t10gb+zjmfmv7zRwTkI9Lg0AppTzhEJfh3EKQ9moppVSI0yBQSqkQp0GglFIhToNAKaVCnAaBUkqFOA0CpZQKcRoESikV4jQIlFIqxHW5FcpEpBTYeYYvTwHKOrCcrk4/j+Pp53GUfhbH6w6fRx9jTGprT3S5IDgbIrKyraXaQpF+HsfTz+Mo/SyO190/Dz01pJRSIU6DQCmlQlyoBcHTtgsIMvp5HE8/j6P0szhet/48QmqMQCml1FeF2hGBUkqpE2gQKKVUiAuZIBCRqSLypYgUish9tuuxSUR6i8gCEflCRD4Xkbts12SbiLhF5DMRecd2LbaJSIKIvCYim0Rko4icb7smW0TkJ/7/RzaIyBwRaWuJ5S4tJIJARNzAE8ClwGDgOhEZbLcqq5qBnxpjBgPjgDtD/PMAuAvYaLuIIPHfwLvGmIHAcEL0cxGRXsCPgDxjzBCclS6/bbeqwAiJIADGAoXGmG3GmEbgFWC65ZqsMcbsNcas9v9ehfM/ei+7VdkjIpnANOAZ27XYJiLxwIXAswDGmEZjTIXVouzyAJEi4gGigGLL9QREqARBL2D3MfeLCOEvvmOJSBYwEvjUcik2PQr8HPBZriMY9AVKgef8p8qeEZFo20XZYIzZAzwE7AL2ApXGmPl2qwqMUAkC1QoRiQFeB35sjDlkux4bROTrQIkxZpXtWoKEBxgFPGWMGQnUACE5piYiiThnDvoCGUC0iNxgt6rACJUg2AP0PuZ+pv+xkCUiYTgh8JIx5p+267FoPHC5iOzAOWV4kYi8aLckq4qAImPM4SPE13CCIRRdDGw3xpQaY5qAfwIXWK4pIEIlCFYAOSLSV0TCcQZ8/mW5JmtERHDOAW80xjxsux6bjDG/MMZkGmOycP67+NAY0y3/1dcexph9wG4RyfU/NBn4wmJJNu0CxolIlP//mcl004Fzj+0COoMxpllEfgC8hzPyP9sY87nlsmwaD8wE1ovIGv9j/8cYM89eSSqI/BB4yf+Ppm3ALZbrscIY86mIvAasxum0+4xuOtWETjGhlFIhLlRODSmllGqDBoFSSoU4DQKllApxGgRKKRXiNAiUUirEaRAoFWAikq+zmqpgpkGglFIhToNAKT8RuUFElovIGhH5H/8aBdUi8oh/TvoPRCTVv+0IEVkmIutE5A3/vDSISLaIvC8ia0VktYj09+8+5pg5/l/yX6mKiDzgXxdinYg8ZOlPVyFOg0ApQEQGAd8CxhtjRgAtwAwgGlhpjDkXWAT8xv+SvwH3GmOGAeuPefwl4AljzHCceWn2+h8fCfwYZz2MfsB4EUkGvgmc69/P7wP5NyrVFg0CpRyTgdHACv+0G5NxvrB9wN/927wITPDP2Z9gjFnkf/yvwIUiEgv0Msa8AWCMqTfG1Pq3WW6MKTLG+IA1QBZQCdQDz4rIlcDhbZXqVBoESjkE+KsxZoT/lmuM+W0r253pnCwNx/zeAniMMc04iya9BnwdePcM963UWdEgUMrxAXC1iKQBiEiSiPTB+X/kav821wMfGWMqgYMiMtH/+ExgkX+1tyIRucK/jwgRiWrrDf3rQcT7J/v7Cc6ykEp1upCYfVSpUzHGfCEivwLmi4gLaALuxFmYZaz/uRKccQSAm4BZ/i/6Y2fonAn8j4j8p38f15zkbWOBt/wLogtwdwf/WUq1i84+qtRJiEi1MSbGdh1KBZKeGlJKqRCnRwRKKRXi9IhAKaVCnAaBUkqFOA0CpZQKcRoESikV4jQIlFIqxP3/wxrVjhEs+LYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_loss_history, label='train loss')\n",
    "plt.plot(test_loss_history, label='test loss')\n",
    "plt.xlabel('epochs')\n",
    "plt.ylabel('loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "- https://github.com/FrancescoSaverioZuppichini/ViT#transformer-1\n",
    "- https://arxiv.org/abs/2010.11929\n",
    "- https://www.youtube.com/watch?v=bgsYOGhpxDc&t=580s\n",
    "- https://blog.promedius.ai/transformer/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}