7-7,DIN网络.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1009a864",
   "metadata": {},
   "source": [
    "\n",
    "# 7-7，DIN网络"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32bf51aa",
   "metadata": {},
   "source": [
    "阿里妈妈在CTR预估领域有3篇比较有名的文章。\n",
    "\n",
    "2017年的深度兴趣网络, DIN(DeepInterestNetwork)。 \n",
    "\n",
    "2018年的深度兴趣演化网络, DIEN(DeepInterestEvolutionNetWork)。\n",
    "\n",
    "2019年的深度会话兴趣网络, DSIN(DeepSessionInterestNetWork)。\n",
    "\n",
    "这3篇文章的主要思想和相互关系用一句话分别概括如下：\n",
    "\n",
    "第1篇DIN说，用户的行为日志中只有一部分和当前候选广告有关。可以利用Attention机制从用户行为日志中建模出和当前候选广告相关的用户兴趣表示。我们试过涨点了嘻嘻嘻。\n",
    "\n",
    "第2篇DIEN说，用户最近的行为可能比较远的行为更加重要。可以用循环神经网络GRU建模用户兴趣随时间的演化。我们试过也涨点了嘿嘿嘿。\n",
    "\n",
    "第3篇DSIN说，用户在同一次会话中的行为高度相关，在不同会话间的行为则相对独立。可以把用户行为日志按照时间间隔分割成会话并用SelfAttention机制建模它们之间的相互作用。我们试过又涨点了哈哈哈。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61ec286f",
   "metadata": {},
   "source": [
    "参考材料：\n",
    "\n",
    "* DIN论文： https://arxiv.org/pdf/1706.06978.pdf\n",
    "\n",
    "* 推荐系统中的注意力机制： https://zhuanlan.zhihu.com/p/51623339\n",
    "\n",
    "* 阿里经典兴趣网络： https://zhuanlan.zhihu.com/p/429433768\n",
    "\n",
    "* 从DIN到DIEN看阿里CTR算法的进化脉络： https://zhuanlan.zhihu.com/p/78365283\n",
    "\n",
    "* DIN+DIEN，机器学习唯一指定涨点技Attention： https://zhuanlan.zhihu.com/p/431131396\n",
    "\n",
    "* Attention机制简单总结： https://zhuanlan.zhihu.com/p/46313756\n",
    "\n",
    "* 代码实现参考： https://github.com/GitHub-HongweiZhang/prediction-flow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8945860f",
   "metadata": {},
   "source": [
    "本篇文章我们主要介绍DIN，下一篇文章我们介绍DIEN。 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda4c256",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "295bc52b",
   "metadata": {},
   "source": [
    "## 〇，Attention原理概述\n",
    "\n",
    "众所周知，Attention机制在深度学习领域是非常通用的涨分技巧。其主要作用是提升模型的自适应能力。\n",
    "\n",
    "Attention机制的一些常用功能和典型范例总结如下：\n",
    "\n",
    "* 1,动态特征选择，根据样本不同动态地赋予特征以不同的权重，典型范例如SENet中的SEAttention，DIN中的Attention.\n",
    "\n",
    "* 2,动态特征交互，动态地构建特征之间的交互强弱关系，提取高阶特征。典型范例如Transformer中的的Attention。\n",
    "\n",
    "* 3,动态模块集成，类似多模型融合集成，但是不同子模块的权重是动态的。典型范例如MOE中的门控注意力机制。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c35b4aca",
   "metadata": {},
   "source": [
    "在许多Attention机制的应用场景中，输入分成Query(Q)和Key(K)。Query是当前关注项的Embedding向量，Key是待和当前关注项进行匹配的Embedding向量。\n",
    "\n",
    "例如在广告CTR领域，Query就是当前待预估的广告，Key就是用户历史上点击过的广告，通过Attention机制建立当前待预估的广告和用户历史上点击过的广告的相关性强弱。\n",
    "\n",
    "又比如在NLP翻译领域，Query就是当前正在解码的译文单词词向量，Key就是原文单词序列的词向量，通过Attention机制可以建立译文单词和原文单词的对应关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eedb3786",
   "metadata": {},
   "source": [
    "Attention机制的核心实现是计算注意力权重，一些的常用实现形式如下：\n",
    "\n",
    "$$attention = f(Q,K)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51810e56",
   "metadata": {},
   "source": [
    "* 1,多层感知机方法\n",
    "\n",
    "先将Query和Key进行拼接，然后接一个多层感知机。\n",
    "\n",
    "这种方法不需要Query和Key的向量长度相等，Query和Key之间的交互方式是通过学习获得的。\n",
    "\n",
    "$$f(Q,K) = mlp([Q;K])$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7901a46b",
   "metadata": {},
   "source": [
    "* 2,Bilinear方法\n",
    "\n",
    "通过一个权重矩阵直接建立Query和Key的关系映射，计算速度较快，但是需要Query和Key的向量长度相同。\n",
    "$$f(Q,K) = QWK^T$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "809dbd90",
   "metadata": {},
   "source": [
    "* 3,Scaled-Dot Product\n",
    "\n",
    "这种方式直接求Query和Key的内积相似度，没有需要学习的参数，计算速度极快，需要Query和Key的向量长度相同。考虑到随着向量维度的增加，最后得到的权重也会增加，对其进行scaling。\n",
    "\n",
    "$$f(Q,K)=softmax(\\frac{QK^T}{\\sqrt{d_k}})$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "591230c9",
   "metadata": {},
   "source": [
    "## 一，DIN原理解析"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c2ae013",
   "metadata": {},
   "source": [
    "阿里的展示广告系统主要用到了如下4类特征\n",
    "\n",
    "* (1) 用户画像特征。\n",
    "* (2) 用户行为特征，即用户点击过的商品。\n",
    "* (3) 待曝光的广告特征，广告其实也是商品。\n",
    "* (4) 上下文特征。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h3eet3webnj20jj09rdgq.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "633bda3e",
   "metadata": {},
   "source": [
    "DIN、DIEN和DSIN主要聚焦在对用户行为日志的建模。\n",
    "\n",
    "用户行为日志反应的是用户的兴趣，如何从行为日志中建模出一个好的用户兴趣的表示？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19fcaf5b",
   "metadata": {},
   "source": [
    "最基础的建模方法是 Embedding+SumPooling. 把用户的过去所有点击行为做Embedding, 然后求和。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e601de0",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h3eezc7meij20d908x0t5.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b250a1",
   "metadata": {},
   "source": [
    "这个SumPooling的实现不要太简单。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "129dfbd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class SumPooling(nn.Module):\n",
    "    def __init__(self, dim):\n",
    "        super(SumPooling, self).__init__()\n",
    "        self.dim = dim\n",
    "\n",
    "    def forward(self, x):\n",
    "        return torch.sum(x, self.dim)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70958acf",
   "metadata": {},
   "source": [
    "这种建模方式存在着一个巨大的缺陷，那就是用户的兴趣表示是确定的，和候选广告无关。\n",
    "\n",
    "不管来个啥候选广告，用户过去的所有行为日志全部一把梭哈丢进去求和。\n",
    "\n",
    "很显然，如果我们如果建模出和候选广告相关的用户兴趣表示，效果应该会好很多。\n",
    "\n",
    "那么，如何做到这一点呢？我们可以用候选广告来和用户历史行为日志求相关性，用相关性对历史行为日志做加权。\n",
    "\n",
    "这是很自然的，我们主要聚焦(Attention)用户历史行为日志中那些和候选广告相关的部分。\n",
    "\n",
    "于是，duang的一下，DIN模型的模型架构就出来了。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h3ef9wpx4uj20mb0c0dh0.jpg) \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bece470",
   "metadata": {},
   "source": [
    "这里注意力机制比较值得玩味，它是一种mlp形式的注意力结构，但在输入端不是简单地拼接了$Q$和$K$，而是将$Q,K,Q-K,Q*K$都一起打包拼接了，这样模型更加容易学习Q和K之间的相似性关系。\n",
    "\n",
    "此外，这里用mask技巧将keys中填充的的部分的注意力赋值为0，以及维度变换等一些实现上的细节，也是很值得揣摩的。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4079669c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn \n",
    "\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, input_size, hidden_layers,\n",
    "                 dropout=0.0, batchnorm=True):\n",
    "        super(MLP, self).__init__()\n",
    "        modules = OrderedDict()\n",
    "        previous_size = input_size\n",
    "        for index, hidden_layer in enumerate(hidden_layers):\n",
    "            modules[f\"dense{index}\"] = nn.Linear(previous_size, hidden_layer)\n",
    "            if batchnorm:\n",
    "                modules[f\"batchnorm{index}\"] = nn.BatchNorm1d(hidden_layer)\n",
    "            modules[f\"activation{index}\"] = nn.PReLU() \n",
    "            if dropout:\n",
    "                modules[f\"dropout{index}\"] = nn.Dropout(dropout)\n",
    "            previous_size = hidden_layer\n",
    "        self.mlp = nn.Sequential(modules)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.mlp(x)\n",
    "    \n",
    "    \n",
    "class Attention(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            input_size,\n",
    "            hidden_layers,\n",
    "            dropout=0.0,\n",
    "            batchnorm=True,\n",
    "            return_scores=False):\n",
    "        \n",
    "        super().__init__()\n",
    "        self.return_scores = return_scores\n",
    "        \n",
    "        self.mlp = MLP(\n",
    "            input_size=input_size * 4,\n",
    "            hidden_layers=hidden_layers,\n",
    "            dropout=dropout,\n",
    "            batchnorm=batchnorm,\n",
    "            activation=activation)\n",
    "        self.fc = nn.Linear(hidden_layers[-1], 1)\n",
    "\n",
    "    def forward(self, query, keys, keys_length):\n",
    "        \"\"\"\n",
    "        Parameters\n",
    "        ----------\n",
    "        query: 2D tensor, [Batch, Hidden]\n",
    "        keys: 3D tensor, [Batch, Time, Hidden]\n",
    "        keys_length: 1D tensor, [Batch]\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        outputs: 2D tensor, [Batch, Hidden]\n",
    "        \"\"\"\n",
    "        \n",
    "        batch_size, max_length, dim = keys.size()\n",
    "\n",
    "        query = query.unsqueeze(1).expand(-1, max_length, -1)\n",
    "\n",
    "        din_all = torch.cat(\n",
    "            [query, keys, query - keys, query * keys], dim=-1)\n",
    "\n",
    "        din_all = din_all.view(batch_size * max_length, -1)\n",
    "\n",
    "        outputs = self.mlp(din_all)\n",
    "\n",
    "        outputs = self.fc(outputs).view(batch_size, max_length)  # [B, T]\n",
    "\n",
    "        # Scale\n",
    "        outputs = outputs / (dim ** 0.5)\n",
    "\n",
    "        # Mask\n",
    "        mask = (torch.arange(max_length, device=keys_length.device).repeat(\n",
    "            batch_size, 1) < keys_length.view(-1, 1))\n",
    "        outputs[~mask] = -np.inf\n",
    "\n",
    "        # Activation\n",
    "        outputs = torch.sigmoid(outputs)  # [B, T]\n",
    "\n",
    "        if not self.return_scores:\n",
    "            # Weighted sum\n",
    "            outputs = torch.matmul(\n",
    "                outputs.unsqueeze(1), keys).squeeze()  # [B, H]\n",
    "            \n",
    "        return outputs \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ad4fb2",
   "metadata": {},
   "source": [
    "我们期待的效果是这样的，和候选广告(query)越相关的用户历史浏览记录(keys)，其注意力权重值越高😋。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03e9fc73",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h3efhfac81j20i706vaaj.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0858c415",
   "metadata": {},
   "source": [
    "除了用Attention机制从用户行为日志中建模出和当前候选广告相关的用户兴趣表示这个主要创新外，DIN这篇文章还有一些其他的微创新。\n",
    "\n",
    "* 引入转折点可以学习的Dice激活函数代替PReLU激活函数\n",
    "* 介绍一种Mini-batch Aware 的L2正则化方法\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85db494f",
   "metadata": {},
   "source": [
    "## 二，DIN的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19a6c9f",
   "metadata": {},
   "source": [
    "下面是一个DIN模型的完整pytorch实现。\n",
    "\n",
    "这里的AttentionGroup类比较特别，是为了建立候选广告属性和历史广告属性的pair关系。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d4512b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F \n",
    "from collections import OrderedDict\n",
    "\n",
    "class MaxPooling(nn.Module):\n",
    "    def __init__(self, dim):\n",
    "        super(MaxPooling, self).__init__()\n",
    "        self.dim = dim\n",
    "\n",
    "    def forward(self, x):\n",
    "        return torch.max(x, self.dim)[0]\n",
    "\n",
    "\n",
    "class SumPooling(nn.Module):\n",
    "    def __init__(self, dim):\n",
    "        super(SumPooling, self).__init__()\n",
    "        self.dim = dim\n",
    "\n",
    "    def forward(self, x):\n",
    "        return torch.sum(x, self.dim)\n",
    "\n",
    "class Dice(nn.Module):\n",
    "    \"\"\"\n",
    "    The Data Adaptive Activation Function in DIN, a generalization of PReLu.\n",
    "    \"\"\"\n",
    "    def __init__(self, emb_size, dim=2, epsilon=1e-8):\n",
    "        super(Dice, self).__init__()\n",
    "        assert dim == 2 or dim == 3\n",
    "\n",
    "        self.bn = nn.BatchNorm1d(emb_size, eps=epsilon)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "        self.dim = dim\n",
    "        \n",
    "        # wrap alpha in nn.Parameter to make it trainable\n",
    "        self.alpha = nn.Parameter(torch.zeros((emb_size,))) if self.dim == 2 else nn.Parameter(\n",
    "            torch.zeros((emb_size, 1)))\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        assert x.dim() == self.dim\n",
    "        if self.dim == 2:\n",
    "            x_p = self.sigmoid(self.bn(x))\n",
    "            out = self.alpha * (1 - x_p) * x + x_p * x\n",
    "        else:\n",
    "            x = torch.transpose(x, 1, 2)\n",
    "            x_p = self.sigmoid(self.bn(x))\n",
    "            out = self.alpha * (1 - x_p) * x + x_p * x\n",
    "            out = torch.transpose(out, 1, 2)\n",
    "        return out\n",
    "\n",
    "\n",
    "class Identity(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "    def forward(self, x):\n",
    "        return x\n",
    "    \n",
    "def get_activation_layer(name, hidden_size=None, dice_dim=2):\n",
    "    name = name.lower()\n",
    "    name_dict = {x.lower():x for x in dir(nn) if '__' not in x and 'Z'>=x[0]>='A'}\n",
    "    if name==\"linear\":\n",
    "        return Identity()\n",
    "    elif name==\"dice\":\n",
    "        assert dice_dim\n",
    "        return Dice(hidden_size, dice_dim)\n",
    "    else:\n",
    "        assert name in name_dict, f'activation type {name} not supported!'\n",
    "        return getattr(nn,name_dict[name])()\n",
    "    \n",
    "def init_weights(model):\n",
    "    if isinstance(model, nn.Linear):\n",
    "        if model.weight is not None:\n",
    "            nn.init.kaiming_uniform_(model.weight.data)\n",
    "        if model.bias is not None:\n",
    "            nn.init.normal_(model.bias.data)\n",
    "    elif isinstance(model, (nn.BatchNorm1d,nn.BatchNorm2d,nn.BatchNorm3d)):\n",
    "        if model.weight is not None:\n",
    "            nn.init.normal_(model.weight.data, mean=1, std=0.02)\n",
    "        if model.bias is not None:\n",
    "            nn.init.constant_(model.bias.data, 0)\n",
    "    else:\n",
    "        pass\n",
    "\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, input_size, hidden_layers,\n",
    "                 dropout=0.0, batchnorm=True, activation='relu'):\n",
    "        super(MLP, self).__init__()\n",
    "        modules = OrderedDict()\n",
    "        previous_size = input_size\n",
    "        for index, hidden_layer in enumerate(hidden_layers):\n",
    "            modules[f\"dense{index}\"] = nn.Linear(previous_size, hidden_layer)\n",
    "            if batchnorm:\n",
    "                modules[f\"batchnorm{index}\"] = nn.BatchNorm1d(hidden_layer)\n",
    "            if activation:\n",
    "                modules[f\"activation{index}\"] = get_activation_layer(activation,hidden_layer,2)\n",
    "            if dropout:\n",
    "                modules[f\"dropout{index}\"] = nn.Dropout(dropout)\n",
    "            previous_size = hidden_layer\n",
    "        self.mlp = nn.Sequential(modules)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.mlp(x)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d005ea71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "class Attention(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            input_size,\n",
    "            hidden_layers,\n",
    "            dropout=0.0,\n",
    "            batchnorm=True,\n",
    "            activation='prelu',\n",
    "            return_scores=False):\n",
    "        \n",
    "        super().__init__()\n",
    "        self.return_scores = return_scores\n",
    "        \n",
    "        self.mlp = MLP(\n",
    "            input_size=input_size * 4,\n",
    "            hidden_layers=hidden_layers,\n",
    "            dropout=dropout,\n",
    "            batchnorm=batchnorm,\n",
    "            activation=activation)\n",
    "        self.fc = nn.Linear(hidden_layers[-1], 1)\n",
    "\n",
    "    def forward(self, query, keys, keys_length):\n",
    "        \"\"\"\n",
    "        Parameters\n",
    "        ----------\n",
    "        query: 2D tensor, [Batch, Hidden]\n",
    "        keys: 3D tensor, [Batch, Time, Hidden]\n",
    "        keys_length: 1D tensor, [Batch]\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        outputs: 2D tensor, [Batch, Hidden]\n",
    "        \"\"\"\n",
    "        \n",
    "        batch_size, max_length, dim = keys.size()\n",
    "\n",
    "        query = query.unsqueeze(1).expand(-1, max_length, -1)\n",
    "\n",
    "        din_all = torch.cat(\n",
    "            [query, keys, query - keys, query * keys], dim=-1)\n",
    "\n",
    "        din_all = din_all.view(batch_size * max_length, -1)\n",
    "\n",
    "        outputs = self.mlp(din_all)\n",
    "\n",
    "        outputs = self.fc(outputs).view(batch_size, max_length)  # [B, T]\n",
    "\n",
    "        # Scale\n",
    "        outputs = outputs / (dim ** 0.5)\n",
    "\n",
    "        # Mask\n",
    "        mask = (torch.arange(max_length, device=keys_length.device).repeat(\n",
    "            batch_size, 1) < keys_length.view(-1, 1))\n",
    "        outputs[~mask] = -np.inf\n",
    "\n",
    "        # Activation\n",
    "        outputs = torch.sigmoid(outputs)  # [B, T]\n",
    "\n",
    "        if not self.return_scores:\n",
    "            # Weighted sum\n",
    "            outputs = torch.matmul(\n",
    "                outputs.unsqueeze(1), keys).squeeze()  # [B, H]\n",
    "            \n",
    "        return outputs \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2873e610",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionGroup(object):\n",
    "    def __init__(self, name, pairs,\n",
    "                 hidden_layers, activation='dice', att_dropout=0.0):\n",
    "        self.name = name\n",
    "        self.pairs = pairs\n",
    "        self.hidden_layers = hidden_layers\n",
    "        self.activation = activation\n",
    "        self.att_dropout = att_dropout\n",
    "\n",
    "        self.related_feature_names = set()\n",
    "        for pair in pairs:\n",
    "            self.related_feature_names.add(pair['ad'])\n",
    "            self.related_feature_names.add(pair['pos_hist'])\n",
    "\n",
    "    def is_attention_feature(self, feature_name):\n",
    "        if feature_name in self.related_feature_names:\n",
    "            return True\n",
    "        return False\n",
    "\n",
    "    @property\n",
    "    def pairs_count(self):\n",
    "        return len(self.pairs)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "219af657",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DIN(nn.Module):\n",
    "    def __init__(self, num_features,cat_features,seq_features, \n",
    "                 cat_nums,embedding_size, attention_groups,\n",
    "                 mlp_hidden_layers, mlp_activation='prelu', mlp_dropout=0.0,\n",
    "                 d_out = 1\n",
    "                 ):\n",
    "        super().__init__()\n",
    "        self.num_features = num_features\n",
    "        self.cat_features = cat_features\n",
    "        self.seq_features = seq_features\n",
    "        self.cat_nums = cat_nums \n",
    "        self.embedding_size = embedding_size\n",
    "        \n",
    "        self.attention_groups = attention_groups\n",
    "        \n",
    "        self.mlp_hidden_layers = mlp_hidden_layers\n",
    "        self.mlp_activation = mlp_activation\n",
    "        self.mlp_dropout = mlp_dropout\n",
    "        \n",
    "        self.d_out = d_out\n",
    "        \n",
    "        #embedding\n",
    "        self.embeddings = OrderedDict()\n",
    "        for feature in self.cat_features+self.seq_features:\n",
    "            self.embeddings[feature] = nn.Embedding(\n",
    "                self.cat_nums[feature], self.embedding_size, padding_idx=0)\n",
    "            self.add_module(f\"embedding:{feature}\",self.embeddings[feature])\n",
    "\n",
    "        self.sequence_poolings = OrderedDict()\n",
    "        self.attention_poolings = OrderedDict()\n",
    "        total_embedding_sizes = 0\n",
    "        for feature in self.cat_features:\n",
    "            total_embedding_sizes += self.embedding_size\n",
    "        for feature in self.seq_features:\n",
    "            total_embedding_sizes += self.embedding_size\n",
    "        \n",
    "        #sequence_pooling\n",
    "        for feature in self.seq_features:\n",
    "            if not self.is_attention_feature(feature):\n",
    "                self.sequence_poolings[feature] = MaxPooling(1)\n",
    "                self.add_module(f\"pooling:{feature}\",self.sequence_poolings[feature])\n",
    "\n",
    "        #attention_pooling\n",
    "        for attention_group in self.attention_groups:\n",
    "            self.attention_poolings[attention_group.name] = (\n",
    "                self.create_attention_fn(attention_group))\n",
    "            self.add_module(f\"attention_pooling:{attention_group.name}\",\n",
    "                self.attention_poolings[attention_group.name])\n",
    "\n",
    "        total_input_size = total_embedding_sizes+len(self.num_features)\n",
    "        \n",
    "        self.mlp = MLP(\n",
    "            total_input_size,\n",
    "            mlp_hidden_layers,\n",
    "            dropout=mlp_dropout, batchnorm=True, activation=mlp_activation)\n",
    "        \n",
    "        self.final_layer = nn.Linear(mlp_hidden_layers[-1], self.d_out)\n",
    "        self.apply(init_weights)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        final_layer_inputs = list()\n",
    "\n",
    "        number_inputs = list()\n",
    "        for feature in self.num_features:\n",
    "            number_inputs.append(x[feature].view(-1, 1))\n",
    "\n",
    "        embeddings = OrderedDict()\n",
    "        for feature in self.cat_features:\n",
    "            embeddings[feature] = self.embeddings[feature](x[feature])\n",
    "\n",
    "        for feature in self.seq_features:\n",
    "            if not self.is_attention_feature(feature):\n",
    "                embeddings[feature] = self.sequence_poolings[feature](\n",
    "                    self.embeddings[feature](x[feature]))\n",
    "                \n",
    "        for attention_group in self.attention_groups:\n",
    "            query = torch.cat(\n",
    "                [embeddings[pair['ad']]\n",
    "                 for pair in attention_group.pairs],\n",
    "                dim=-1)\n",
    "            keys = torch.cat(\n",
    "                [self.embeddings[pair['pos_hist']](\n",
    "                    x[pair['pos_hist']]) for pair in attention_group.pairs],\n",
    "                dim=-1)\n",
    "            #hist_length = torch.sum(hist>0,axis=1)\n",
    "            keys_length = torch.min(torch.cat(\n",
    "                [torch.sum(x[pair['pos_hist']]>0,axis=1).view(-1, 1)\n",
    "                 for pair in attention_group.pairs],\n",
    "                dim=-1), dim=-1)[0]\n",
    "            \n",
    "            embeddings[attention_group.name] = self.attention_poolings[\n",
    "                attention_group.name](query, keys, keys_length)\n",
    "\n",
    "        emb_concat = torch.cat(number_inputs + [\n",
    "            emb for emb in embeddings.values()], dim=-1)\n",
    "\n",
    "        final_layer_inputs = self.mlp(emb_concat)\n",
    "        output = self.final_layer(final_layer_inputs)\n",
    "        if  self.d_out==1:\n",
    "            output = output.squeeze() \n",
    "\n",
    "        return output\n",
    "\n",
    "    def create_attention_fn(self, attention_group):\n",
    "        return Attention(\n",
    "            attention_group.pairs_count * self.embedding_size,\n",
    "            hidden_layers=attention_group.hidden_layers,\n",
    "            dropout=attention_group.att_dropout,\n",
    "            activation=attention_group.activation)\n",
    "    \n",
    "    def is_attention_feature(self, feature):\n",
    "        for group in self.attention_groups:\n",
    "            if group.is_attention_feature(feature):\n",
    "                return True\n",
    "        return False\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f072a67",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6d5f3d99",
   "metadata": {},
   "source": [
    "## 三，Movielens数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e751916e",
   "metadata": {},
   "source": [
    "下面是一个基于Movielens评价数据集的完整范例，根据用户过去对一些电影的评价结果，来预测用户对候选电影是否会给好评。\n",
    "\n",
    "这个数据集不大，用CPU就能跑。😁\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee45395",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d46756cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.preprocessing import QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline, FeatureUnion \n",
    "from sklearn.impute import SimpleImputer \n",
    "from collections import Counter\n",
    "\n",
    "#类别特征预处理\n",
    "class CategoryEncoder(BaseEstimator, TransformerMixin):\n",
    "    \n",
    "    def __init__(self, min_cnt=5, word2idx=None, idx2word=None):\n",
    "        super().__init__() \n",
    "        self.min_cnt = min_cnt\n",
    "        self.word2idx = word2idx if word2idx else dict()\n",
    "        self.idx2word = idx2word if idx2word else dict()\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        if not self.word2idx:\n",
    "            counter = Counter(np.asarray(x).ravel())\n",
    "\n",
    "            selected_terms = sorted(\n",
    "                list(filter(lambda x: counter[x] >= self.min_cnt, counter)))\n",
    "\n",
    "            self.word2idx = dict(\n",
    "                zip(selected_terms, range(1, len(selected_terms) + 1)))\n",
    "            self.word2idx['__PAD__'] = 0\n",
    "            if '__UNKNOWN__' not in self.word2idx:\n",
    "                self.word2idx['__UNKNOWN__'] = len(self.word2idx)\n",
    "\n",
    "        if not self.idx2word:\n",
    "            self.idx2word = {\n",
    "                index: word for word, index in self.word2idx.items()}\n",
    "\n",
    "        return self\n",
    "\n",
    "    def transform(self, x):\n",
    "        transformed_x = list()\n",
    "        for term in np.asarray(x).ravel():\n",
    "            try:\n",
    "                transformed_x.append(self.word2idx[term])\n",
    "            except KeyError:\n",
    "                transformed_x.append(self.word2idx['__UNKNOWN__'])\n",
    "\n",
    "        return np.asarray(transformed_x, dtype=np.int64)\n",
    "\n",
    "    def dimension(self):\n",
    "        return len(self.word2idx)\n",
    "\n",
    "#序列特征预处理（类别序列） \n",
    "class SequenceEncoder(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, sep=' ', min_cnt=5, max_len=None,\n",
    "                 word2idx=None, idx2word=None):\n",
    "        super().__init__() \n",
    "        self.sep = sep\n",
    "        self.min_cnt = min_cnt\n",
    "        self.max_len = max_len\n",
    "\n",
    "        self.word2idx = word2idx if word2idx else dict()\n",
    "        self.idx2word = idx2word if idx2word else dict()\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        if not self.word2idx:\n",
    "            counter = Counter()\n",
    "\n",
    "            max_len = 0\n",
    "            for sequence in np.array(x).ravel():\n",
    "                words = sequence.split(self.sep)\n",
    "                counter.update(words)\n",
    "                max_len = max(max_len, len(words))\n",
    "\n",
    "            if self.max_len is None:\n",
    "                self.max_len = max_len\n",
    "\n",
    "            # drop rare words\n",
    "            words = sorted(\n",
    "                list(filter(lambda x: counter[x] >= self.min_cnt, counter)))\n",
    "\n",
    "            self.word2idx = dict(zip(words, range(1, len(words) + 1)))\n",
    "            self.word2idx['__PAD__'] = 0\n",
    "            if '__UNKNOWN__' not in self.word2idx:\n",
    "                self.word2idx['__UNKNOWN__'] = len(self.word2idx)\n",
    "\n",
    "        if not self.idx2word:\n",
    "            self.idx2word = {\n",
    "                index: word for word, index in self.word2idx.items()}\n",
    "\n",
    "        if not self.max_len:\n",
    "            max_len = 0\n",
    "            for sequence in np.array(x).ravel():\n",
    "                words = sequence.split(self.sep)\n",
    "                max_len = max(max_len, len(words))\n",
    "            self.max_len = max_len\n",
    "\n",
    "        return self\n",
    "\n",
    "    def transform(self, x):\n",
    "        transformed_x = list()\n",
    "\n",
    "        for sequence in np.asarray(x).ravel():\n",
    "            words = list()\n",
    "            for word in sequence.split(self.sep):\n",
    "                try:\n",
    "                    words.append(self.word2idx[word])\n",
    "                except KeyError:\n",
    "                    words.append(self.word2idx['__UNKNOWN__'])\n",
    "\n",
    "            transformed_x.append(\n",
    "                np.asarray(words[0:self.max_len], dtype=np.int64))\n",
    "\n",
    "        return np.asarray(transformed_x, dtype=object)\n",
    "    \n",
    "    def dimension(self):\n",
    "        return len(self.word2idx)\n",
    "\n",
    "    def max_length(self):\n",
    "        return self.max_len\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a37690",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "15291fb8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocess number features...\n",
      "preprocess category features...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 353.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "preprocess sequence features...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 63.96it/s]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "from tqdm import tqdm \n",
    "\n",
    "dftrain = pd.read_csv(\"./eat_pytorch_datasets/ml_1m/train.csv\")\n",
    "dfval = pd.read_csv(\"./eat_pytorch_datasets/ml_1m/test.csv\")\n",
    "\n",
    "for col in [\"movieId\",\"histHighRatedMovieIds\",\"negHistMovieIds\",\"genres\"]:\n",
    "    dftrain[col] = dftrain[col].astype(str)\n",
    "    dfval[col] = dfval[col].astype(str)\n",
    "\n",
    "num_features = ['age']\n",
    "cat_features = ['gender', 'movieId', 'occupation', 'zipCode']\n",
    "seq_features = ['genres', 'histHighRatedMovieIds', 'negHistMovieIds']\n",
    "\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "encoders = {}\n",
    "\n",
    "print(\"preprocess number features...\")\n",
    "dftrain[num_features] = num_pipe.fit_transform(dftrain[num_features]).astype(np.float32)\n",
    "dfval[num_features] = num_pipe.transform(dfval[num_features]).astype(np.float32)\n",
    "\n",
    "print(\"preprocess category features...\")\n",
    "for col in tqdm(cat_features):\n",
    "    encoders[col] = CategoryEncoder(min_cnt=5)\n",
    "    dftrain[col]  = encoders[col].fit_transform(dftrain[col])\n",
    "    dfval[col] =  encoders[col].transform(dfval[col])\n",
    "    \n",
    "print(\"preprocess sequence features...\")\n",
    "for col in tqdm(seq_features):\n",
    "    encoders[col] = SequenceEncoder(sep=\"|\",min_cnt=5)\n",
    "    dftrain[col]  = encoders[col].fit_transform(dftrain[col])\n",
    "    dfval[col] =  encoders[col].transform(dfval[col])\n",
    "    \n",
    "from collections import OrderedDict\n",
    "from itertools import chain\n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "class Df2Dataset(Dataset):\n",
    "    def __init__(self, dfdata, num_features, cat_features,\n",
    "                 seq_features, encoders, label_col=\"label\"):\n",
    "        self.dfdata = dfdata\n",
    "        self.num_features = num_features\n",
    "        self.cat_features = cat_features \n",
    "        self.seq_features = seq_features\n",
    "        self.encoders = encoders\n",
    "        self.label_col = label_col\n",
    "        self.size = len(self.dfdata)\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.size\n",
    "\n",
    "    @staticmethod\n",
    "    def pad_sequence(sequence,max_length):\n",
    "        #zero is special index for padding\n",
    "        padded_seq = np.zeros(max_length, np.int32)\n",
    "        padded_seq[0: sequence.shape[0]] = sequence\n",
    "        return padded_seq\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        record = OrderedDict()\n",
    "        for col in self.num_features:\n",
    "            record[col] = self.dfdata[col].iloc[idx].astype(np.float32)\n",
    "            \n",
    "        for col in self.cat_features:\n",
    "            record[col] = self.dfdata[col].iloc[idx].astype(np.int64)\n",
    "            \n",
    "        for col in self.seq_features:\n",
    "            seq = self.dfdata[col].iloc[idx]\n",
    "            max_length = self.encoders[col].max_length()\n",
    "            record[col] = Df2Dataset.pad_sequence(seq,max_length)\n",
    "\n",
    "        if self.label_col is not None:\n",
    "            record['label'] = self.dfdata[self.label_col].iloc[idx].astype(np.float32)\n",
    "        return record\n",
    "\n",
    "    def get_num_batches(self, batch_size):\n",
    "        return np.ceil(self.size / batch_size)\n",
    "    \n",
    "ds_train = Df2Dataset(dftrain, num_features, cat_features, seq_features, encoders)\n",
    "ds_val = Df2Dataset(dfval,num_features, cat_features, seq_features, encoders)\n",
    "dl_train = DataLoader(ds_train, batch_size=128,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size=128,shuffle=False)\n",
    "\n",
    "cat_nums = {k:v.dimension() for k,v in encoders.items()} \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d172c2de",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f6af8c8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "59a8bb82",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "d1944dc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_net():\n",
    "    din_attention_groups = [\n",
    "        AttentionGroup(\n",
    "            name='group1',\n",
    "            pairs=[{'ad': 'movieId', 'pos_hist': 'histHighRatedMovieIds'}],\n",
    "            activation='dice',\n",
    "            hidden_layers=[16, 8], att_dropout=0.1)\n",
    "    ]\n",
    "\n",
    "    net = DIN(num_features=num_features,\n",
    "               cat_features=cat_features,\n",
    "               seq_features=seq_features,\n",
    "               cat_nums = cat_nums,\n",
    "               embedding_size=16,\n",
    "               attention_groups=din_attention_groups,\n",
    "               mlp_hidden_layers=[32,16],\n",
    "               mlp_activation=\"prelu\",\n",
    "               mlp_dropout=0.25,\n",
    "               d_out=1\n",
    "               )\n",
    "    return net \n",
    "\n",
    "net = create_net() \n",
    "\n",
    "out = net.forward(batch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52fea343",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a8656b3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------\n",
      "Layer (type)                            Output Shape              Param #\n",
      "==========================================================================\n",
      "Embedding-1                                 [-1, 16]                   64\n",
      "Embedding-2                                 [-1, 16]                4,480\n",
      "Embedding-3                                 [-1, 16]                  368\n",
      "Embedding-4                                 [-1, 16]                1,984\n",
      "Embedding-5                              [-1, 6, 16]                  320\n",
      "MaxPooling-6                                [-1, 16]                    0\n",
      "Embedding-7                             [-1, 10, 16]               61,888\n",
      "MaxPooling-8                                [-1, 16]                    0\n",
      "Embedding-9                             [-1, 10, 16]               28,656\n",
      "Linear-10                                   [-1, 16]                1,040\n",
      "BatchNorm1d-11                              [-1, 16]                   32\n",
      "BatchNorm1d-12                              [-1, 16]                   32\n",
      "Sigmoid-13                                  [-1, 16]                    0\n",
      "Dropout-14                                  [-1, 16]                    0\n",
      "Linear-15                                    [-1, 8]                  136\n",
      "BatchNorm1d-16                               [-1, 8]                   16\n",
      "BatchNorm1d-17                               [-1, 8]                   16\n",
      "Sigmoid-18                                   [-1, 8]                    0\n",
      "Dropout-19                                   [-1, 8]                    0\n",
      "Linear-20                                    [-1, 1]                    9\n",
      "Linear-21                                   [-1, 32]                3,648\n",
      "BatchNorm1d-22                              [-1, 32]                   64\n",
      "PReLU-23                                    [-1, 32]                    1\n",
      "Dropout-24                                  [-1, 32]                    0\n",
      "Linear-25                                   [-1, 16]                  528\n",
      "BatchNorm1d-26                              [-1, 16]                   32\n",
      "PReLU-27                                    [-1, 16]                    1\n",
      "Dropout-28                                  [-1, 16]                    0\n",
      "Linear-29                                    [-1, 1]                   17\n",
      "==========================================================================\n",
      "Total params: 103,332\n",
      "Trainable params: 103,332\n",
      "Non-trainable params: 0\n",
      "--------------------------------------------------------------------------\n",
      "Input size (MB): 0.000801\n",
      "Forward/backward pass size (MB): 0.006302\n",
      "Params size (MB): 0.394180\n",
      "Estimated Total Size (MB): 0.401283\n",
      "--------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.summary import summary \n",
    "summary(net,input_data=batch);\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "983cca10",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf63bab5-6c77-4641-bbac-80603cbc7bda",
   "metadata": {},
   "source": [
    "我们使用梦中情炉torchkeras来实现最优雅的训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "e05c3365-fd6b-49c9-9b11-1cc500ed49ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator\n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            #loss\n",
    "            preds = self.net(batch)\n",
    "            labels = batch['label']\n",
    "            loss = self.loss_fn(preds,labels)\n",
    "\n",
    "        #backward()\n",
    "        if self.stage==\"train\" and self.optimizer is not None:\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        all_preds = self.accelerator.gather(preds)\n",
    "        all_labels = self.accelerator.gather(labels)\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {self.stage+\"_\"+name:metric_fn(all_preds, all_labels).item() \n",
    "                        for name,metric_fn in self.metrics_dict.items()}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "\n",
    "KerasModel.StepRunner = StepRunner \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f9b2977",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "78f8b329",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras.metrics import AUC\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer,\n",
    "                  )    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "597c3158",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< 🐌 cpu is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAGJCAYAAADIVkprAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABnmUlEQVR4nO3dd3xTVf8H8E+atuke0EEXtFgEZD6sioqAoCj+sFJQpkwHMmSIAsp01QcekC3yCE5kWIqooAhIEXnYS9mr0FK6GG3pbpPz++Oa0LRJm6ZJbsfn/Xrl1eTec3NP0rT3mzO+RyGEECAiIiKyIDu5K0BERES1DwMMIiIisjgGGERERGRxDDCIiIjI4hhgEBERkcUxwCAiIiKLY4BBREREFscAg4iIiCyOAQYRERFZHAMMspq5c+dCoVDg1q1bclfFZq5duwaFQoEvv/xS7qqQFc2fPx/NmjWDRqORuyo2Ux3/nletWoWGDRuioKBA7qqQAQwwqNb56KOP8MMPP8hdjVotIyMDr776Knx9feHq6oru3bvj+PHjJh+v0Wjw6aefom3btnB2dkb9+vXxxBNP4NSpU0aPWbduHRQKBdzc3MrsUygURm9PPvmkXtkPP/wQzz33HPz9/aFQKDB37lyT6w0AWVlZ+Pe//41p06bBzq7if6H5+fkoKiqq1Dlqo9jYWAwYMACNGzeGi4sLmjZtijfffBMZGRllyk6ePBnt2rVDvXr14OLigubNm2Pu3LnIzs7WKzdixAgUFhbis88+s9GroMqwl7sCRJb20UcfoX///nj++eflrkqtpNFo8Oyzz+LUqVN466234OPjg5UrV6Jbt244duwYmjRpUuFzjBo1CuvWrcOwYcMwfvx45OTk4MSJE0hLSzNYPjs7G2+//TZcXV0N7v/mm2/KbDt69CiWLFmCp556Sm/7zJkz0aBBA/zrX//Cjh07THjF+tauXYvi4mIMGjTIaJkjR45g6dKl+O2335CWlgaFQoGgoCD07dsXb7zxBsLDwyt93pru1VdfRWBgIIYOHYqGDRvi77//xvLly7F9+3YcP34czs7OurJHjhxBly5dMHLkSDg5OeHEiRP4+OOPsWvXLvzxxx+6wM7JyQnDhw/HokWLMGHCBCgUCrleHhkiiKxkzpw5AoBIT0+36XldXV3F8OHDbXpOrfj4eAFAfPHFF7Kc3xY2btwoAIjvv/9ety0tLU14eXmJQYMGmXx8bGysyeecNm2aaNq0qRgyZIhwdXU16ZjRo0cLhUIhEhMT9bbHx8cLIYRIT08XAMScOXNMrocQQrRu3VoMHTrU4L6ioiIxbtw4oVAoRJcuXcR//vMf8dNPP4nNmzeLjz76SLRt21Y4OTmJ5cuXV+qc1UFV/5737NlTZttXX30lAIj//ve/FR7/n//8RwAQBw4c0Nt+9OhRAUDs3r3brHqR9TDAIKvR/kM6d+6ceOGFF4S7u7uoV6+eeOONN0ReXl6Z8t98841o166dcHJyEt7e3mLAgAEiISFBr8zFixdFVFSU8Pf3FyqVSgQFBYkBAwaIjIwMIYQQAMrcjAUbKSkpQqlUirlz55bZd/78eQFALFu2TAghxO3bt8Wbb74pWrZsKVxdXYW7u7t4+umnxcmTJ/WOMyfAKCgoELNmzRLt2rUTHh4ewsXFRTz22GPi999/1yu3Z88eAaDMP2pj59S+7z4+PsLJyUk8+OCD4p133jG5Xsa88MILwt/fX6jVar3tr776qnBxcRH5+fnlHh8RESE6deokhBBCrVaL7OzscstfvHhRODo6im3btonhw4ebFGDk5+cLLy8v0a1bN6NlzAkwrl69KgCIL7/80uD+oUOHCm9vb/HLL78YfY4vv/xSODk5iU8//bTMvhs3boiRI0cKPz8/4ejoKB566CGxZs0avTLaz8GGDRvEjBkzhL+/v3BxcRF9+vQp8/cihBCbNm3S/V3Vr19fDBkyRNy4caNMuYo+L9q/50uXLonhw4cLT09P4eHhIUaMGCFycnKMvt7yZGVlCQBiypQpFZaNiYkRAAy+t9r/K1S9sIuErO7FF19EaGgooqOjcfDgQSxduhR3797F119/rSvz4YcfYtasWXjxxRfx8ssvIz09HcuWLcPjjz+OEydOwMvLC4WFhejVqxcKCgowYcIENGjQAElJSfj555+RkZEBT09PfPPNN3j55ZfRqVMnvPrqqwCABx54wGC9/P390bVrV2zatAlz5szR27dx40YolUq88MILAICrV6/ihx9+wAsvvICwsDCkpqbis88+Q9euXXH27FkEBgaa/f5kZWXh888/x6BBg/DKK6/g3r17WLNmDXr16oXDhw+jbdu2lX7Ov/76C126dIGDgwNeffVVhIaG4sqVK/jpp5/w4YcfAgCKioqQmZlp0vPVq1dP1yx94sQJtGvXrsz4g06dOmH16tW4ePEiWrVqZfS1Hj58GGPHjsU777yDZcuWITs7G2FhYfj444/x4osvljlm0qRJ6N69O3r37o1NmzaZVN/t27cjIyMDQ4YMMam8qf73v/8BANq1a1dm3zfffIMtW7bg0KFDaNGiBQBACIGcnBzduJFbt27hpZdego+PD1544QU888wzaNSoEQAgNTUVDz/8MBQKBcaPHw9fX1/88ssvGD16NLKysjBp0iS983344YdQKBSYNm0a0tLSsHjxYvTs2RMnT57UdTd8+eWXGDlyJDp27Ijo6GikpqZiyZIl2L9/v+7vCjDt86L14osvIiwsDNHR0Th+/Dg+//xz+Pn54d///nel38+UlBQAgI+PT5l9xcXFyMjIQGFhIU6fPo2ZM2fC3d0dnTp1KlO2Xbt22L9/f6XPT1Ymd4RDtZf2G89zzz2nt33s2LECgDh16pQQQohr164JpVIpPvzwQ71yf//9t7C3t9dtP3HiRJmmeUMq00Xy2WefCQDi77//1tv+0EMPiSeeeEL3OD8/v8w39vj4eKFSqcR7772ntw2VbMEoLi4WBQUFetvu3r0r/P39xahRo3TbKtOC8fjjjwt3d3dx/fp1vbIajabM85ly03YrCCG9vyXrpbVt2zYBQPz6669GX+vx48cFAFG/fn3h7+8vVq5cKdatWyc6deokFApFmW+nP//8s7C3txdnzpwRQgiTWzD69esnVCqVuHv3rtEy5rRgzJw5UwAQ9+7d09uu0WhEWFiYWLx4sW7b1q1bRWBgoAAgGjZsKHbs2KH3Xvbt21evhWD06NEiICBA3Lp1S++5Bw4cKDw9PUVubq4Q4v7vLSgoSGRlZenKbdq0SQAQS5YsEUIIUVhYKPz8/ETLli31Wgx//vlnAUDMnj1bt82Uz4v277n0775v376ifv36Fb95BowePVoolUpx8eLFMvsOHDig9xls2rSpwW4WIaTWM2dnZ7PqQNbDWSRkdePGjdN7PGHCBADSt0xAGl2u0Wjw4osv4tatW7pbgwYN0KRJE+zZswcA4OnpCQDYsWMHcnNzLVK3qKgo2NvbY+PGjbptp0+fxtmzZzFgwADdNpVKpfvGrlarcfv2bbi5uaFp06aVmj1hiFKphKOjIwBpAOWdO3dQXFyMDh06mPXc6enp+OOPPzBq1Cg0bNhQb1/JQXBt2rTBzp07Tbo1aNBAd1xeXh5UKlWZ8zo5Oen2G6OdBXD79m1s3boVr7/+OgYPHozdu3ejfv36+OCDD3RlCwsLMXnyZIwZMwYPPfSQya8/KysL27ZtQ+/evXXf0C3l9u3bsLe3LzOT5dixY0hLS8Po0aMBAElJSRg0aBA6deqEzZs3Y/LkyRg1apTeMc8//zzi4uIASC0dmzdvRp8+fSCE0Ps76NWrFzIzM8t8FoYNGwZ3d3fd4/79+yMgIED3d3X06FGkpaVh7Nixut8NADz77LNo1qwZtm3bBsD0z4vWmDFj9B536dIFt2/fRlZWVoXvX0nfffcd1qxZgzfffNPgwOCHHnoIO3fuxA8//KAb4Ft6FomWt7c38vLyLPZ/gSyDXSRkdaX/eTzwwAOws7PDtWvXAACXLl2CEMLo7AMHBwcAQFhYGKZMmYJFixZh3bp16NKlC5577jkMHTpUF3xUlo+PD3r06IFNmzbh/fffByB1j9jb2yMqKkpXTqPRYMmSJVi5ciXi4+OhVqt1++rXr2/WuUv66quvsHDhQpw/f15vSmNYWFiln+vq1asAgJYtW5ZbztvbGz179qz08zs7OxvMO5Cfn6/bX96xgPS6IiIidNvd3NzQp08ffPvttyguLoa9vT0++eQT3Lp1C/PmzatU/TZv3oz8/HyLd4+U59ixY+jQoYMu8Fi3bh2CgoIQExMDpVIJAPDy8sLIkSN1x/j7+yM9PR2AdJHPyMjA6tWrsXr1aoPnKD3DpvTfi0KhQHh4uO7v6vr16wCApk2blnmuZs2a4c8//wRg+udFq3QQ4u3tDQC4e/cuPDw8THqOffv2YfTo0ejVq1eZLhgtDw8P3eczMjIS3333HSIjI3H8+HG0adNGr6wQAoDhgIjkwwCDbK70PwGNRgOFQoFffvlF98+4pJLfFhcuXIgRI0Zg69at+O233/DGG2/oxnYEBwebVZ+BAwdi5MiROHnyJNq2bYtNmzahR48eev3CH330EWbNmoVRo0bh/fff141JmDRpUpWTLX377bcYMWIEnn/+ebz11lvw8/ODUqlEdHQ0rly5oitn7J9nyWCnMgoLC3Hnzh2Tyvr6+up+NwEBAUhOTi5TRrutvPEo2n3+/v5l9vn5+aGoqAg5OTkAgA8++ABjx45FVlaW7ttxdnY2hBC4du0aXFxc4OfnV+Z51q1bB09PT/zf//2fSa+tMurXr4/i4mLcu3dPr/Xg9u3beq/72rVr+Ne//qX3eS49diAxMVEXnGo/Q0OHDsXw4cMNnrt169YWex1VYehvFLh/ka/IqVOn8Nxzz6Fly5aIiYmBvb1pl6GoqCi89NJL2LBhQ5kA4+7du3BxcSk3uCXbY4BBVnfp0iW9b+KXL1+GRqNBaGgoAKlFQwiBsLAwPPjggxU+X6tWrdCqVSvMnDkT//vf//Doo49i1apVuub1yn6Lef755/Haa6/pukkuXryIGTNm6JWJiYlB9+7dsWbNGr3tGRkZBgeoVUZMTAwaN26M2NhYvbqXHniq/aZYOjGR9puqVuPGjQFIXT3l+d///ofu3bubVMf4+Hjd76tt27bYt28fNBqN3kDPQ4cOwcXFpdzfYWBgoG5wbmk3b96Ek5MT3N3dkZCQgOzsbMyfPx/z588vUzYsLAyRkZFlEqolJydjz549GDFihMFunKpq1qwZAOn9KHnB9/Dw0Bsw26BBAxw+fFjvWG1LASBdjNesWaP7hu7r6wt3d3eo1WqTW5UuXbqk91gIgcuXL+vqpR08euHCBTzxxBN6ZS9cuKDbb+rnxRKuXLmCp59+Gn5+fti+fbvBpGnGFBQUQKPRGByYHB8fj+bNm1uyqmQBHINBVrdixQq9x8uWLQMAPPPMMwCkbyZKpRLz5s0r8y1ICIHbt28DkPrWi4uL9fa3atUKdnZ2ek32rq6uBrMDGuPl5YVevXph06ZN2LBhAxwdHcsk6VIqlWXq9v333xu8UFaW9hthyec/dOgQDhw4oFeuUaNGUCqV+OOPP/S2r1y5Uu+xr68vHn/8caxduxYJCQl6+0qew9wxGP3790dqaipiY2N1227duoXvv/8effr00buwX7lyRa8VBgAGDBiAxMRE7Ny5U+/4rVu34oknnoCdnR38/PywZcuWMrfu3bvDyckJW7ZsKRMEAsCGDRug0Wis1j3SuXNnANL4hpKaN2+OI0eO6FoiIiMjceLECcyePRtXr17Fvn378NZbbwGQZuH069cPN27cwMSJEwFIn4F+/fph8+bNBi/02q6Ukr7++mvcu3dP9zgmJgbJycm6v6sOHTrAz88Pq1at0vv7+OWXX3Du3Dk8++yzAEz/vFRVSkoKnnrqKdjZ2WHHjh3w9fU1WC4jI8Ng5tPPP/8cgPS6Sjt+/DgeeeQRi9WVLESGgaVUR2hHnbdq1Ur06dNHrFixQgwdOlQAEIMHD9YrGx0dLQCIRx55RMyfP198+umn4u233xZNmjQRCxYsEEIIsWXLFhEUFCQmTZokVq5cKZYuXSo6duwoHBwc9JLv9O7dW7i6uoqFCxeK9evXi4MHD1ZY12+//VYAEO7u7qJPnz5l9s+ePVsAECNGjBCrV68WEyZMEPXq1RONGzcWXbt21ZUzZxbJ2rVrdbNtPvvsMzF9+nTh5eUlWrRoIRo1aqRXduDAgcLe3l5MmTJFrFixQjzzzDOiffv2Zc558uRJ4ebmJurXry9mzJghVq9eLd555x3Rpk0bk+tlTHFxsXj44YeFm5ubmDdvnlixYoVo0aKFcHd3F+fPn9cr26hRozKvISUlRQQEBAh3d3cxZ84csWjRIvHggw8KZ2fnMnlFSqtoFkn79u1FYGBgmRk/JX399dfi/fffFzNmzBAARPfu3cX7778v3n//fXHt2rUKX3/Lli3LJBTLz88Xnp6eYsuWLbptH330kbCzsxMAhL29vViyZIluRsRTTz0lrl69qvccKSkpolGjRsLFxUVMnDhRfPbZZyI6Olq88MILwtvbW1dOO4ukVatWonXr1uKTTz4R06dPF05OTiI8PFwvJ8UXX3whAIiIiAixePFiMWPGDOHi4iJCQ0P1ZtiY8nkxlmhLe46SM40MadOmjQAg3n77bfHNN9/o3X777TdduS1btoiQkBAxefJksXLlSrF48WLRr18/oVAoRIcOHcrMuNIm2tq1a1e55yfbY4BBVqP9h3T27FnRv39/4e7uLry9vcX48eMNJtravHmzeOyxx4Srq6twdXUVzZo1E+PGjRMXLlwQQkhJjkaNGiUeeOAB4eTkJOrVqye6d+9e5h/L+fPnxeOPPy6cnZ3LTbRVUlZWlq78t99+W2Z/fn6+ePPNN0VAQIBwdnYWjz76qDhw4IDo2rVrlQMMjUYjPvroI9GoUSOhUqnEv/71L/Hzzz+L4cOHl7k4p6eni379+gkXFxfh7e0tXnvtNXH69GmD5zx9+rTo27ev8PLyEk5OTqJp06Zi1qxZJterPHfu3BGjR48W9evXFy4uLqJr167iyJEjZcoZCjCEEOLKlSuib9++wsPDQzg7O4snnnhCHD58uMLzlhdgaJOjVZS0qWvXrkan4xqbBlnSokWLhJubm27aqNacOXNE48aNxZ07d3TbkpKSxB9//CFSUlKEEEL8+eefIi0tzehzp6aminHjxomQkBDh4OAgGjRoIHr06CFWr16tK6MNMNavXy9mzJgh/Pz8hLOzs3j22WfLTDMVQsqc+q9//UuoVCpRr149o4m2Kvq8VDXAMPaeA9D7G7p8+bIYNmyYaNy4sXB2dhZOTk6iRYsWYs6cOQaTsk2bNk00bNhQb0otVQ8KISzYBkZEVMtlZmaicePGmD9/vm5aKiDNonn00UehVCqxdetWBAQEGDw+JiYGffv2NTpYsiJxcXHo3r07vv/+e/Tv39+s56gtCgoKEBoaiunTp+u6m6j64BgMIqJK8PT0xNtvv40FCxbozSBycnLC9u3boVAo0LRpU0ybNg1//PEHrl+/jvPnz+Prr79G586dMXz48CrnTiHJF198AQcHhzK5Oah6YAsGkZWYMg3U09OTU+tqmcLCQixfvhzLly9HfHy8bruTkxP69u2LefPmmbTirDFswaCagtNUiazElGmgX3zxBUaMGGGbCpFNODo6YsqUKZgyZQquXbuGpKQkODk5oXnz5nBxcZG7ekQ2wxYMIiu5e/cujh07Vm6ZFi1aGO2rJyKqyRhgEBERkcVxkCcRERFZnKxjMP744w8sWLAAx44dQ3JyMrZs2VImg2JpcXFxmDJlCs6cOYOQkBDMnDmzUn3YGo0GN2/ehLu7OxfGISIiqgQhBO7du4fAwEC9pQIMkTXAyMnJQZs2bTBq1Ci9lSuNiY+Px7PPPosxY8Zg3bp12L17N15++WUEBASgV69eJp3z5s2bCAkJqWrViYiI6qzExMQKF5isNmMwFApFhS0Y06ZNw7Zt2/Ry9Q8cOBAZGRn49ddfTTpPZmYmvLy8kJiYaPLSwkRERCStCRUSEoKMjAx4enqWW7ZGTVM9cOBAmZUGe/XqhUmTJhk9pqCgQG+hH+3iQB4eHgwwiIiIzGDKEIMaNcgzJSUF/v7+etv8/f2RlZWFvLw8g8dER0fD09NTd2P3CBERkfXVqADDHDNmzEBmZqbulpiYKHeViIiIar0a1UXSoEEDpKam6m1LTU2Fh4eH0XTLKpUKKpXKFtUjIiKif9SoAKNz587Yvn273radO3eic+fOFj2PEALFxcVQq9UWfV6yDaVSCXt7e05DJiKSkawBRnZ2Ni5fvqx7HB8fj5MnT6JevXpo2LAhZsyYgaSkJHz99dcAgDFjxmD58uV4++23MWrUKPz+++/YtGkTtm3bZrE6FRYWIjk5Gbm5uRZ7TrI9FxcXBAQEwNHRUe6qEBHVSbIGGEePHtVbDGrKlCkAgOHDh+PLL79EcnIyEhISdPvDwsKwbds2TJ48GUuWLEFwcDA+//xzk3NgVESj0SA+Ph5KpRKBgYFwdHTkt+AaRgiBwsJCpKenIz4+Hk2aNKkwGQwREVletcmDYStZWVnw9PREZmZmmWmq+fn5iI+PR6NGjbjqYQ2Xm5uL69evIywsDE5OTnJXh4jIptRqYN8+IDkZCAgAunQBlMqqP29519DSatQYDFvhN96aj79DIqotKhssxMYCEycCN27c3xYcDCxZApiQNNti+F+YiIiomoqNBUJDge7dgcGDpZ+hodJ2Y+X799cPLgAgKUnabuw4a2CAQUREVA1VNlhQq6WWC0MDH7TbJk2SytkCu0isxFr9X7YQGhqKSZMmlZuCnYiIrKeiYEGhAF5/HcjJAa5fB+LjgRMnygYjpY9LTJSuTd26Wa3qOgwwrECO/q9u3bqhbdu2WLx4cZWf68iRI3B1da16pYiIyCz79lUcLKSlAcOGVf65k5PNr1dlMMCwMG2TVumoU9ukFRNj20E2WkIIqNVq2NtX/Cv39fW1QY2IiMgYU4OA5s2Bhx+WxmXk5wPR0RUfExBQpaqZjGMwTJSTY/yWny+VMaX/a+JE/f4vY89ZGSNGjMDevXuxZMkSKBQKKBQKfPnll1AoFPjll1/Qvn17qFQq/Pnnn7hy5QoiIyPh7+8PNzc3dOzYEbt27dJ7vtDQUL2WEIVCgc8//xx9+/aFi4sLmjRpgh9//NGkuqnVaowePRphYWFwdnZG06ZNsWTJEr0y3bp1K9Md8/zzz2PEiBG6xwUFBZg2bRpCQkKgUqkQHh6ONWvWVOp9IiKqCbKygN9+M63sypXA2rXA7NnA++9LreXG0jcpFEBIiNRlbwsMMEzk5mb81q+fVMaUJq0bN6RyWqGhhp+zMpYsWYLOnTvjlVdeQXJyMpKTk3Wrxk6fPh0ff/wxzp07h9atWyM7Oxu9e/fG7t27ceLECTz99NPo06ePXkIzQ+bNm4cXX3wRf/31F3r37o0hQ4bgzp07FdZNo9EgODgY33//Pc6ePYvZs2fjnXfewaZNmyr1GocNG4b169dj6dKlOHfuHD777DO4VfaNIiKSmVoNxMUB69dLP0t+4SwoAObPB8LCgC+/LP95DAULSqXUFa/dX7o8ACxebMPxgKKOyczMFABEZmZmmX15eXni7NmzIi8vr8w+KTwwfOvdWyrz3Xfll9Pevvvu/vP6+BguU1ldu3YVEydO1D3es2ePACB++OGHCo9t0aKFWLZsme5xo0aNxCeffFLitUPMnDlT9zg7O1sAEL/88kvlKyqEGDdunOjXr5/RugshRGRkpBg+fLgQQogLFy4IAGLnzp0mn6O83yURkRw2bxYiOFj/f31wsLRdCCEKC4Vo3Fja3rSpEFOmCKFQSLeSx2i3aY8z5TwhIcbLV0Z519DSOAbDRNnZxvdpo0FT+7VKlrt2zewqmaRDhw56j7OzszF37lxs27YNycnJKC4uRl5eXoUtGK1bt9bdd3V1hYeHB9LS0kyqw4oVK7B27VokJCQgLy8PhYWFaNu2rcmv4eTJk1AqlejatavJxxARVSfGxufduKE/Pm/RIiAzU8p5YW8PPPqo4UkDixcbH88XFQVERso/k5EBholMmVTRpYv0i09KMjwOQ6GQ9pds0rL2ZI3Ss0GmTp2KnTt34j//+Q/Cw8Ph7OyM/v37o7CwsNzncXBw0HusUCig0WgqPP+GDRswdepULFy4EJ07d4a7uzsWLFiAQ4cO6crY2dlBlHrDioqKdPednZ0rPA8RUXVV3vg8QNo+aZIUFERG6u8zN1hQKm0zFbU8DDAsSNv/1b+/FEyU/DBZu//L0dHRpOXl9+/fjxEjRqBv374ApBaNa1ZsRtm/fz8eeeQRjB07VrftypUremV8fX2RXGLItFqtxunTp3UL4bVq1QoajQZ79+5Fz549rVZXIiJrqGh8HlB+forqECyYg4M8LSwqSmrqCgrS3x4cbN0pqqGhoTh06BCuXbuGW7duGW1daNKkCWJjY3Hy5EmcOnUKgwcPNqklwlxNmjTB0aNHsWPHDly8eBGzZs3CkSNH9Mo88cQT2LZtG7Zt24bz58/j9ddfR0ZGht5rGz58OEaNGoUffvgB8fHxiIuLq/RAUSIiOdy8aVo5W+WnsBUGGFYQFSWNrdizB/juO+lnfLx1819MnToVSqUSDz30EHx9fY2OqVi0aBG8vb3xyCOPoE+fPujVqxfatWtntXq99tpriIqKwoABAxAREYHbt2/rtWYAwKhRozB8+HAMGzYMXbt2RePGjXWtF1qffvop+vfvj7Fjx6JZs2Z45ZVXkFPZ+bxERDIIDDStnK3yU9gKl2svQbtcO5f4rvn4uyQiazO2JERGBrBiBTBihNSarVZLQYaxcfHa8Xnx8dV/SQku105ERGRFhpaECAyUsmru2iUly7p9W5oVolQCn34qjc8DbDs+T07sIqEqGTNmDNzc3AzexowZI3f1iIgsztgqpzdvSvuysoCHHgI6d76/T67xeXJiF0kJbFavvLS0NGRlZRnc5+HhAT8/PxvXSMLfJRFZg1otZWAub1ZI/fpSt0mp2f264+XOT1EV7CIhm/Hz85MtiCAisjVTppzevg3s31+7ppyag10kREREJqqrU07NwQCDiIjIBBkZ0uqlpqhtU07NwQCDiIioAseOAe3aSV0f5bH1kujVGQMMIiIiI4QAVq0CHnlEylMRFiYtqa5QVJMl0asxBhhEREQGZGcDQ4cCr78OFBYCzz0ntWS89Vbdm3JqDs4isRK1ENiXkYHkwkIEODqii5cXlKXD3WokNDQUkyZNwqRJk+SuChGRTRmbOvrSS8APP0j3o6OBqVPvt1JUlyXRqzMGGFYQm56OiZcv40ZBgW5bsEqFJeHhiPL1lbFmRERUkqGMnMHB0srY770HnD4NrF1reExFXZpyag52kVhYbHo6+p85oxdcAEBSQQH6nzmD2PR0mWpGREQlGcvImZQkbb90CTh3jgM2zcUAowJCCOSo1SbdsoqL8calSzCUGlW7beLly8gqLjbp+UxNsrp69WoEBgaWWXY9MjISo0aNwpUrVxAZGQl/f3+4ubmhY8eO2LVrl9nvyaJFi9CqVSu4uroiJCQEY8eORXZ2tm7/3Llz0bZtW71jFi9ejNDQUL1ta9euRYsWLaBSqRAQEIDx48ebXSciospQq6WWC0P/ZrXbJk0qO5CTTMcukgrkajRw27fPIs8lANwoKIDnn3+aVD67Sxe4mtCh98ILL2DChAnYs2cPevToAQC4c+cOfv31V2zfvh3Z2dno3bs3PvzwQ6hUKnz99dfo06cPLly4gIYNG1b6ddjZ2WHp0qUICwvD1atXMXbsWLz99ttYaeoEcUjLr0+ZMgUff/wxnnnmGWRmZmJ/RfO/iIgspKKMnEIAiYlSOXaDmIcBRi3g7e2NZ555Bt99950uwIiJiYGPjw+6d+8OOzs7tGnTRlf+/fffx5YtW/Djjz+a1WpQciBoaGgoPvjgA4wZM6ZSAcYHH3yAN998ExMnTtRt69ixY6XrQkRkDlMzbTIjp/kYYFTAxc4O2SZ2wP2RkYHef/9dYbntrVrhcS8vk85tqiFDhuCVV17BypUroVKpsG7dOgwcOBB2dnbIzs7G3LlzsW3bNiQnJ6O4uBh5eXlISEgw+flL2rVrF6Kjo3H+/HlkZWWhuLgY+fn5yM3NhYuLS4XHp6Wl4ebNm7pgiIjI1kwdb8+MnObjGIwKKBQKuCqVJt2eqlcPwSoVjHXZKQCEqFR4ql49k55PUYnOvz59+kAIgW3btiExMRH79u3DkCFDAABTp07Fli1b8NFHH2Hfvn04efIkWrVqhcLCwkq/H9euXcP//d//oXXr1ti8eTOOHTuGFStWAIDu+ezs7MqMHykqKtLdd3Z2rvR5iYgsKSmp/P3MyFl1DDAsSKlQYEl4OACUCTK0jxeHh1slH4aTkxOioqKwbt06rF+/Hk2bNkW7du0AAPv378eIESPQt29ftGrVCg0aNMC1a9fMOs+xY8eg0WiwcOFCPPzww3jwwQdxs9TqP76+vkhJSdELMk6ePKm77+7ujtDQUOzevdusOhARVdWwYcCTT0r3mZHTOhhgWFiUry9iWrRAkEqltz1YpUJMixZWzYMxZMgQbNu2DWvXrtW1XgBAkyZNEBsbi5MnT+LUqVMYPHhwmRknpgoPD0dRURGWLVuGq1ev4ptvvsGqVav0ynTr1g3p6emYP38+rly5ghUrVuCXX37RKzN37lwsXLgQS5cuxaVLl3D8+HEsW7bMrDoREZli504pOycgBRG//QZs3syMnFYj6pjMzEwBQGRmZpbZl5eXJ86ePSvy8vKqfJ5ijUbsuXNHfJeSIvbcuSOKNZoqP2dF1Gq1CAgIEADElStXdNvj4+NF9+7dhbOzswgJCRHLly8XXbt2FRMnTtSVadSokfjkk09MOs+iRYtEQECAcHZ2Fr169RJff/21ACDu3r2rK/Ppp5+KkJAQ4erqKoYNGyY+/PBD0ahRI73nWbVqlWjatKlwcHAQAQEBYsKECVV49fos+bskoppNoxHio4+EUCiE6NdPCLVaf39xsRB79gjx3XfSz+JiOWpZM5R3DS1NIYSJyRZqiaysLHh6eiIzMxMeHh56+/Lz8xEfH4+wsDA4OTnJVEOyBP4uieqm0mm/O3QAXn0VWL9e2j92rJSl055THMxS3jW0NL7FRERUKxhK++3gABQVSQHFsmXAmDHy1a+u4RgM0rNu3Tq4ubkZvLVo0ULu6hERGWQs7bd2AtusWQwubI0tGKTnueeeQ0REhMF9Dg4ONq4NEVHFykv7DUgDOj//HHj3Xc4KsSUGGKTH3d0d7u7ucleDiMhkTPtdPbGLxIA6Nu61VuLvkKjuYNrv6okBRgnaLoDc3FyZa0JVpf0dsluHqPYzNZ03037bFrtISlAqlfDy8kJaWhoAwMXFpVLpukl+Qgjk5uYiLS0NXl5eULLDlajW69IF8PYG7t41vF+hkJJnMe23bTHAKKVBgwYAoAsyqGby8vLS/S6JqHa7eBHIyTG8j2m/5cMAoxSFQoGAgAD4+fnpLdBFNYeDgwNbLojqiLw8YOBAoLAQaN0auHNHf8BncLAUXDDtt+0xwDBCqVTyIkVEVM1NnQr89Rfg5wfs2CEtw14yk2eXLmy5kAsDDCIiqpG2bAFWrpTuf/01oO0V5VTU6oGzSIiIqEbavl36OXUq0KuXvHWhstiCQURENdLq1UDPnkDfvnLXhAxhgEFERDWSQgEMGCB3LcgYdpEQEVGNsXcvMGgQkJEhd02oIrIHGCtWrEBoaCicnJwQERGBw4cPGy1bVFSE9957Dw888ACcnJzQpk0b/PrrrzasLRERyeX2bWDIEGDDBuCjj+SuDVVE1gBj48aNmDJlCubMmYPjx4+jTZs26NWrl9EkVzNnzsRnn32GZcuW4ezZsxgzZgz69u2LEydO2LjmRERkS0IAI0cCSUlA06bAnDly14gqohAyrgoVERGBjh07Yvny5QAAjUaDkJAQTJgwAdOnTy9TPjAwEO+++y7GjRun29avXz84Ozvj22+/NXiOgoICFBQU6B5nZWUhJCQEmZmZ8PDwsPArIiIiS1Gr7+e0OHAAWLYMcHQEDh0C2raVu3Z1U1ZWFjw9PU26hsrWglFYWIhjx46hZ8+e9ytjZ4eePXviwIEDBo8pKCiAk5OT3jZnZ2f8+eefRs8THR0NT09P3S0kJMQyL4CIiKwmNhYIDQW6dwcGD5aCCwB46SUGFzWFbAHGrVu3oFar4e/vr7fd398fKSkpBo/p1asXFi1ahEuXLkGj0WDnzp2IjY1Fcjlr8M6YMQOZmZm6W2JiokVfBxERWVZsLNC/v37Kb621a6X9VP3JPsizMpYsWYImTZqgWbNmcHR0xPjx4zFy5EjY2Rl/GSqVCh4eHno3IiKqntRqYOJEacyFMZMmSeWoepMtwPDx8YFSqURqaqre9tTUVKOrYPr6+uKHH35ATk4Orl+/jvPnz8PNzQ2NGze2RZWJiMjK9u0z3HKhJQSQmCiVo+pNtgDD0dER7du3x+7du3XbNBoNdu/ejc6dO5d7rJOTE4KCglBcXIzNmzcjMjLS2tUlIiIb+Ptv08qV0zNO1YSsmTynTJmC4cOHo0OHDujUqRMWL16MnJwcjBw5EgAwbNgwBAUFITo6GgBw6NAhJCUloW3btkhKSsLcuXOh0Wjw9ttvy/kyiIioinJzgQULTM9vERBg3fpQ1ckaYAwYMADp6emYPXs2UlJS0LZtW/z666+6gZ8JCQl64yvy8/Mxc+ZMXL16FW5ubujduze++eYbeHl5yfQKiIjIFCWnnJZcRl0I4PvvgbfeAhISpLKOjkBRkeFxGAoFEBwsHU/Vm6x5MORQmTm8RERUdbGx0sDNkmMrgoOBGTOATZuk9N8AEBIC/Oc/UuDxwgvStpJXKIVC+hkTA0RF2abupK8y11AudkZERFajnXJa+qtsUhIwfjygUgHOzsC0aVIrhouLtD8mxnBQsngxg4uagi0YRERkFWq1lCzL2KwQhQKoXx84fBgICzN8vKFuFZIPWzCIiEh2pkw5vXULuH7dcIChVALdulmtemRlNSrRFhER1RymTiXllNPaiQEGERFZhalTSTnltHZigEFERFbRsaM05dQYhUKaOcIpp7UTAwwiIrI4tRoYNgwoLJQea6eYamkfL17MgZu1FQMMIiKyODs7oHVraRrq3LlAUJD+/uBg5rOo7ThNlYiIrCY+XpohwimntQOnqRIRkSx++gl44gnA1VV6rJ1+yimndQ+7SIiIyCI2bQIiI4GePYGcHLlrQ3JjgEFERFW2axcwdKiUPKt9+/spv6nuYoBBRERVcvQo0LevtALqCy8AS5aUnTVCdQ/HYBARkclKD9Zs0ADo3RvIzgZ69AC++YaDN0nCAIOIiExiaNl1pVIKOtq1A7ZskaalEgEMMIiIyATGll1Xq6Wf48cD7u62rxdVXxyDQURE5VKrpZYLY1mTFApgzpz7wQYRwACDiIgqYMqy64mJUjkiLQYYRERULi67TuZggEFEROXisutkDgYYRERUri5dpMXJjOGy62QIAwwiIiqXUgk8+6zhfVx2nYxhgEFEROW6eFFKoAUAnp76+7jsOhnDPBhERGRUcTHw0ktAbi7QvTuwYwewfz+XXaeKMcAgIiKj1GrgsceAS5eAr74CHBy47DqZhl0kRERklEoFLFwoBRghIXLXhmoSBhhERFRGfr5+Zs769eWrC9VMDDCIiKiMqVOBxx8HrlyRuyZUU3EMBhER6fn1V2DFCun+1avAAw/IWx+qmdiCQUREOrdvA6NGSfffeAN48kl560M1FwMMIiICIC1a9tpr0hTU5s2Bjz+Wu0ZUkzHAICIiAMC33wKbNwP29tJ9Z2e5a0Q1GcdgEBHVUWq1tMR6crKU8nvcOGn7vHlAu3by1o1qPgYYRER1UGwsMHEicOPG/W0ODkDTpsDbb8tXL6o92EVCRFTHxMYC/fvrBxeAlBb84kXgxx/lqRfVLgwwiIjqELVaarkQouw+7bZJk/STbBGZgwEGEVEtoFYDcXHA+vXST2MBwr59ZVsuShICSEyUyhFVBcdgEBHVcIbGUwQHA0uW3F9GvaAA2L7d9KmnycmWryfVLQwwiIhqMO14itJdHklJ0vaYGKBvX6BlS+DyZdOfNyDAsvWkuoddJERENZSp4yk0GikjZ1AQMGUK4O8vTUs1RKGQVk3t0sVq1aY6ggEGEVENVZnxFB9/DFy/Li29vnKltL90kKF9vHgxoFRapcpUhzDAICKqoUwdJ5GcDHh43A8aoqKkrpOgIP1ywcHSdu24DaKq4BgMIqIaytRxEobKRUUBkZH3M3kGBEjdImy5IEthgEFEVEN16QIEBgI3bxrer1BIrRLGxlMolUC3blarHtVx7CIhIqqhlEpp0KYhHE9BcmOAQURUg735JrB6tTQzpCSOpyC5KYQwNMGp9srKyoKnpycyMzPh4eEhd3WIiMxSVCQtTqZVcmVUjqcga6nMNZRjMIiIapiEBKB7d2DBgvstFBxPQdUNu0iIiGqQ/HygXz/g6lUgOpqLklH1xQCDiKgGmTABOHoUqFdPGmPBbhCqrmQPMFasWIHQ0FA4OTkhIiIChw8fLrf84sWL0bRpUzg7OyMkJASTJ09Gfn6+jWpLRCSfzz+XbgqFtGpqo0Zy14jIOFkDjI0bN2LKlCmYM2cOjh8/jjZt2qBXr15IS0szWP67777D9OnTMWfOHJw7dw5r1qzBxo0b8c4779i45kREtnX0KDB+vHT/gw+Ap56Stz5EFZF1FklERAQ6duyI5cuXAwA0Gg1CQkIwYcIETJ8+vUz58ePH49y5c9i9e7du25tvvolDhw7hzz//NOmcnEVCRDXN7dtAu3bS4M7ISGkFVTvZ25+pLqrMNVS2j2hhYSGOHTuGnj173q+MnR169uyJAwcOGDzmkUcewbFjx3TdKFevXsX27dvRu3dvo+cpKChAVlaW3o2IqDpTq4G4OKkbJC4OcHWVllxv0gT46isGF1QzyDZN9datW1Cr1fAvlR3G398f58+fN3jM4MGDcevWLTz22GMQQqC4uBhjxowpt4skOjoa8+bNs2jdiYisJTZWWoK95CqpwcHAkiXAe+9Ji5YR1QQ1Kg6Oi4vDRx99hJUrV+L48eOIjY3Ftm3b8P777xs9ZsaMGcjMzNTdEhMTbVhjIiLTxcYC/fuXXYI9KUnavmuXPPUiModsLRg+Pj5QKpVITU3V256amooGDRoYPGbWrFl46aWX8PLLLwMAWrVqhZycHLz66qt49913YWeg3VClUkGlUln+BRARWZBaLbVcGBoVJ4Q0c2TSJGkMBqemUk0gWwuGo6Mj2rdvrzdgU6PRYPfu3ejcubPBY3Jzc8sEEcp//tLqWMZzIqpl9u0r23JRkhBAYqJUjqgmkDVV+JQpUzB8+HB06NABnTp1wuLFi5GTk4ORI0cCAIYNG4agoCBER0cDAPr06YNFixbhX//6FyIiInD58mXMmjULffr00QUaREQ1UXKyZcsRyU3WAGPAgAFIT0/H7NmzkZKSgrZt2+LXX3/VDfxMSEjQa7GYOXMmFAoFZs6ciaSkJPj6+qJPnz748MMP5XoJREQWERBg2XJEcuNqqkRE1cCNG0BYGFBcbHi/QiHNJomP5xgMkk+NyINBRET3ZWQATk7SfYVCf5/28eLFDC6o5mCAQURUDbRsCfzxB7ByJRAUpL8vOFha2Ey7NDtRTcAuEiIimVy5AqSkAI8+qr9drZZmiyQnS2MuunRhywVVD5W5hso6yJOIqK66cAF44gkgKwvYvRvo1On+PqUS6NZNtqoRWQS7SIiIbOzMGaBrV+DmTaBhQ+lGVNuwBYOIyIpKd3d4eAC9egG3bgFt2gA7dwK+vnLXksjyGGAQEVmJoYXLFAopK2f79sBvvwH16slXPyJrYoBBRGQF2oXLSg+j1z6eOJHBBdVuHINBRGRh5S1cBkitGO++K5Ujqq0YYBARWRgXLiNigEFEZHEXL5pWjguXUW1m1hiMzMxMqNVq1CvVgXjnzh3Y29szgRUR1TqmJL9KSgIWLpSycZqCC5dRbWZWC8bAgQOxYcOGMts3bdqEgQMHVrlSRETVSWwsEBoKdO8ODB4s/QwNlbYDwNWrwGuvAY0bA598AhQUAA4Oxp9PoQBCQqQghai2MivAOHToELp3715me7du3XDo0KEqV4qIqLrQzgYpPaYiKUnaHhsLbNsGrF4NFBYCjz0G/PILsH69FEhw4TKqq8zqIikoKECxgTWFi4qKkJeXV+VKERFVB+XNBhFCChYmTZIyc+7fD4wbp98qERNTNg9GcLAUXHDhMqrtzAowOnXqhNWrV2PZsmV621etWoX27dtbpGJERHIzdTbIsWOAgV5jREUBkZFcuIzqJrMCjA8++AA9e/bEqVOn0KNHDwDA7t27ceTIEfz2228WrSARkVxMneVRXjkuXEZ1lVljMB599FEcOHAAISEh2LRpE3766SeEh4fjr7/+QheOWiKiWsLUWR6cDUJUlkIIY7nmaqfKrGVPRHWbWi3N9jDWQqFQSGMq4uPZ7UF1Q2WuoWZ1kSQkJJS7vyHXHiaiWkCpBJo1MxxgcDYIUfnMCjBCQ0OhKD33qgQ1E+wTUS3x889Anz7AuXP6gQZngxCVz6wA48SJE3qPi4qKcOLECSxatAgffvihRSpGRCSXggJApZLuu7gAu3eblsmTiO4zK8Bo06ZNmW0dOnRAYGAgFixYgCiG9ERUQ927Bzz1FNCzJ/Dee/e7QjgbhKhyLLrYWdOmTXHkyBFLPiURkc3k5gLPPQccPAisWAHcvCl3jYhqLrNaMLKysvQeCyGQnJyMuXPnokmTJhapGBGRLRUUSOMp4uIAd3dgxw4gKEjuWhHVXGYFGF5eXmUGeQohEBISYnARNCKi6qT0eIqHHwYGDpSCChcXYPt2oGNHuWtJVLOZFWDs2bNH77GdnR18fX0RHh4Oe3uznpKIyCZiY8uuD+LsDOTlSQM7f/xRWrCMiKrGrGiga9euAICzZ88iISEBhYWFuHv3Li5evAgAeO655yxXQyIiC9GujFo6vaB2jcY33wT+Wf2AiKrIrEyeV69eRVRUFP766y8oFApon0LbbVKd82AwkydR3aRWA6Gh5S9eFhLCrJxE5anMNdSsWSQTJ05EaGgo0tLS4OLigtOnT+OPP/5Ahw4dEBcXZ85TEhFZVUUrowLSyqj79tmmPkS1nVldJAcOHMDvv/8OHx8f2NnZQalU4rHHHkN0dDTeeOONMom4iIjkZomVUYnIdGa1YKjVari7uwMAfHx8cPOfyeKNGjXChQsXLFc7IiILqV/ftHJcGZXIMsxqwWjZsiVOnTqFsLAwREREYP78+XB0dMTq1avRuHFjS9eRiKhKDh+WZo6UR7syapcutqkTUW1nVgvGzJkzodFoAADvvfce4uPj0aVLF2zfvh1Lly61aAWJiMxVXAy88w7QuTNw/jzg5SVtL71WI1dGJbI8s1owevXqpbsfHh6O8+fP486dO/D29i53lVUiIksrbxEypRL46y9AowGGDAGWLpUydZbOg8GVUYksz6xpqjUZp6kS1R6GkmYFBQH//rcUUADSeiKHDgF9+94vw5VRicxTmWsoAwwiqpGMJc3S2ryZLRJElmb1PBhERHJSq6WWi/K+Hk2YIJUjInkwwCCiGseUpFk3bzJpFpGcGGAQUY3DpFlE1R+XPiWiaqO8wZfFxcD168ADD5ieDItJs4jkwwCDiKoFQzNCgoOBefOkgOOzzwAHB+DSJSnwCA4GkpIMj8Ng0iwi+THAICLZGZsRcuMGMHr0/cc+PsCVK0CTJsCSJdIxCoX+cUyaRVQ9cAwGEcnKlBkhjo7AV19JAUeTJtK2qCggJkbKe1FScLC0nVNUieTFFgwikpUpM0IKC4GGDQGVSn97VBQQGcmkWUTVEQMMIpJVVWeEKJVAt24Wqw4RWQi7SIhIVhcvmlaOM0KIaha2YBCRLIqLgXffBebPL78cZ4QQ1UxswSAim7t1C3jmmfvBxXPPSYEEl1Enqj0YYBCRTR0/DnToAOzaBbi6Ahs3Alu3ckYIUW3DLhIisgpDWTkPHgR69gTy84HwcGDLFqBlS6k8Z4QQ1S7VogVjxYoVCA0NhZOTEyIiInD48GGjZbt16waFQlHm9uyzz9qwxkRUnthYIDQU6N4dGDxY+hkaKk1Hbd0a+L//A44cuR9caGlnhAwaJP1kcEFUc8keYGzcuBFTpkzBnDlzcPz4cbRp0wa9evVCWlqawfKxsbFITk7W3U6fPg2lUokXXnjBxjUnIkO0WTlL57ZISpICh3HjpC4RLy9ZqkdENqIQorz8edYXERGBjh07Yvny5QAAjUaDkJAQTJgwAdOnT6/w+MWLF2P27NlITk6Gq6trheWzsrLg6emJzMxMeHh4VLn+RHSfWn2/pcIQ7YyQ+Hi2ThDVRJW5hsraglFYWIhjx46hZ8+eum12dnbo2bMnDhw4YNJzrFmzBgMHDjQaXBQUFCArK0vvRkTWUVFWTiGAxESpHBHVbrIGGLdu3YJarYa/v7/edn9/f6SkpFR4/OHDh3H69Gm8/PLLRstER0fD09NTdwsJCalyvYnIsKpm5SSi2kP2MRhVsWbNGrRq1QqdOnUyWmbGjBnIzMzU3RITE21YQ6K6IycH+OEH08oyKydR7SfrNFUfHx8olUqkpqbqbU9NTUWDBg3KPTYnJwcbNmzAe++9V245lUoFVekVkojI4u7eBbZvL78Ms3IS1R2ytmA4Ojqiffv22L17t26bRqPB7t270blz53KP/f7771FQUIChQ4dau5pEtYZaDcTFAevXSz/V6qo9X27u/fvBwcCqVcA77zArJxFVgy6SKVOm4L///S+++uornDt3Dq+//jpycnIwcuRIAMCwYcMwY8aMMsetWbMGzz//POrXr2/rKhPVSMZyU8TGln+coaBECOD774HGjYEdO+6XHTIE+PBDZuUkomqQyXPAgAFIT0/H7NmzkZKSgrZt2+LXX3/VDfxMSEiAnZ1+HHThwgX8+eef+O233+SoMlGNo81NUXpSelKStN3YhT82Fpg4UX9mSECAFCwcOSI9XrIE6NVL/zhm5SQi2fNg2BrzYFBdY25uCmNBiZZSKa2G+s47AIc5EdUNlbmGyt6CQUTWZWpuimefBRo2BBwcpOBh/XrjwQUA+PoCs2ezVYKIDGOAQVTLmZpzouRYClOkpEjBS7dula4SEdUBDDCIajlTc0688orUglFUBJw8Cfz4Y8XHMGEWERnDAIOolmvbVhojUVBgeL92DMann97v7oiLMy3AYMIsIjJG9mmqRGQ96enAU0+VH1wAZXNTdOkiBR2lc1mUPC4khAmziMg4BhhEtVRCghQAHDkC1K8P/PvfUtBQkrHcFEqlNP0UYMIsIjIPp6kS1ULnzwM9e0p5Lho2lAZwNmsmTVmtTG4KQ3kwQkKk4IIJs4jqnspcQxlgENVCqanAY48Bjo5ScFG65aIyKhuUEFHtxTwYRHVAeRd+f39g1y7AzU3qHqkKpZJTUYmo8hhgENVAhrou6tWT1hhZtkx63KiRPHUjIgI4yJOoxtGm8C6dnfPOHWD5cmD+fHnqRURUEgMMomrA1GXU1Wqp5aK8kVPLl1d9GXYioqpigEEks8oso17RuiKAtK7Ivn3WqCkRkekYYBDJyFh3h3YZ9c2bgcuXgQ0bpFYJU1NzM4U3EcmNgzyJZFJed4d22wsv3L/fqpXpqbmZwpuI5MYWDCKZmNLdIYS0fHpEBHDvHlN4E1HNwQCDSCYVBRdaa9YABw8CDz/MFN5UgZMngWeekX4SyYwBBpGNaTTAxo3AjBmmlQ8J0X8cFSWtHxIUpL/d2LoiVIds3gz8+qvhEcJENsZU4UQWZizDphDA9u3Au+8Cp05JZe3spO2G/gq1y6jHxxtukWAKbyqjbVvpw9W2LXDihNy1oVqIqcKJZGIow2ZwMPDaa9IXy/37pW0eHsDUqUDjxsBLL0nBRMkgw5TuDqbwJj2pqfcj15MngbQ0wM9P1ipR3cYAg8hCtFNOS7dG3LgBzJol3XdyAiZMAKZNu79GiLOz4aCEK5ZSpezYUfbxSy/JUxciMMAgsghTMmy6uQFnzxoeUxEZye4OqhrNtm2AUgk7tRoapRLYtg12DDBIRgwwiCzAlCmn2dnAlStlAwyA3R1kgqQkqRvEgN/v3EHHn3+G+z854u3UamT9/DOO7tqFJ+rVM/x8/v5lRwoTWRADDCILOHvWtHLMsElmGzYM+P13g7ueAKApNW/ZLTcXTzz5pPHn69ED6p07sS8jA8mFhQhwdEQXLy8ojSVZqSK1EDY7F1UPDDCIquDWLeDjj4GlS00rzwybZLYxY4Djx4GMDIO77Ur1z5V+rMfLC4cGDUL/gwdxo6BAtzlYpcKS8HBE+fpaosY6senpmHj5sk3ORdUHp6kSGVHRNNATJ4DHH5e6PgDA0REoLDT8XBVNOaXqr1p8A09LkwKNLVvKTj2qgFAooBAC6NsX2z74AH3S0lD6aO2riWnRwmIX/tj0dPQ/c8Ym5yLrq8w1lAEG1QmVzRlhbLrp4sVAv37S4+JioEULafDmBx8AubnS2iGA4SmnTIJVc1XlG7hVApNNm4DXXoPm3j3Y/TPuojzFdnbIcnHBpKlTcbp3b5zPy0OeRmOwrALSa4t/+OEq11MtBEJLtZJY61xkGwwwysEAo+4xFiwsWWL4gm9suqnWhg3AgAHS/aQkIDDwfhBh6FwhIZxyak3Wblmoyjdwa3UNaITAj2fPwvPll9Ht4EGU92oFgD0PP4whb7+NFG9vk8+xp00bdKtEeUPi7t5Fd21uDiufi2yjMtdQpgqnWq2i5dBLZ1Q2Zbrp669L5QBpEH7Ja1lUFHDtGrBnD/Ddd9LP+HgGF9YSm56O0IMH0f3UKQw+dw7dT51C6MGDiE1Pt8jzq4XAxMuXywQXAHTbJl2+DLWBD4w2MCn97T2poAD9z5wpt45qIRB39y7Wp6Yi7u5d3fMLIbD11i386+hR9E1Px59NmqDYrvx/42qlEt169sT1yEic69gRbwYHl1te65i278/Euhkqt/PuXZPOlWysb9EMptaPrI8tGFRrqdVAaKjx6aMKhZTocNUqoGFDoF07IC4O6N694ufes4fTSq2hMq0RtujbN/Ub+JjAQLR3c4OnvT287O3hplQi6swZpBi5cJbXNWCw1cPRESMaNMCvd+/i6L17AAAPpRIXXnsN/ufOVdiCoSiROtzU1wQAj3h4YFRAAF709YW7vX2FLTJCCBy+dw/rU1OxMT3d6OsvzVItGDVhMGm1GMtTBewiKQcDjLrD1GABAMaOBVasANavBwYPrrj8d98BgwZVqXpUSmUuDrbq21+fmorB586ZfXxFVjZpgv6+vvBxcIBCoTAaNJXkameHicHBmOrgAO+GDfX2aRQK2Amh+6knJQXw99e9d0kFBUbP42RnhwKNRrffxc4OndzdEZeZWaas9t2N8vHBiexsXM3P1+3zVipRKARyjIz3AIB69vZIfeQR2FfQElMROQaTVjZYqAkBUEW4FgkRTM85ERZ2P9+QqdNIOd3UsoxdHLTdCTEtWuApb2+cyc3F39nZ2H7njtHgApC+tScWFGBfRobZ34yLNBoc/6e1oCI9vbzgpFQis7gYGcXFuFlQgNvFxRUeN/bSJYy9dAmudnYIdXLClfz8coMLd6USFzp1QoBKBXz1ld4+oVRC7eaGM6NHo9maNVBkZ0NRcgDojh3AsGFQKhRYEh6O/mfOQAHonU97aVzXvDk6e3jgm9RUrE1OxoW8PIPBBUocv/nWLQBSAPS8jw8G+fvjSW9v/Hz7NvqfOaNXtqQ7xcV46fx5rHrwQXjam3dJqqgrSwGpKyvSx8dgAGBOq0JlgwVTPuPGgoya2urBFgyqtTZuBAYOrLhcye4ObbdKUpJ5K5zSfab+U6yoNQIAlAAqnitR1nuhoZjZqBEUpc5bXt2KNRqsS0vDe9eu6X0bN8RYS4mp3RA+9va4XVxcblBRmq47YcAAaWqSdjnevn2l/j4/v7LTWRUKaYrThg265zF0gQxRqbC41AVSCIGVSUkYf/lyhXWb1agRpjVsCNdSfxzGzvW4pyc2pKVBDSDMyQnrH3oIEWb8X67KYFJzWhUq21qiFgKNDhxAkqW6zGRs9WAXSTkYYNR+N25Ii4lt3Aj4+Ej/aysTLGgHhgK2m25qq28o5pzHmt/uNELg29RUDD9/3qT6+zs4oLWbG7yUSnz/zzfmirR2dcVrgYEY4u8Pz3LGEXzywAMoEALzrl3Dpbw8AICfgwN616uHr/5J0W3o276hb54VdUOUvKAUC4Hr+flYm5yMfycmVvh6vmveHIPq15dWy8vKAry8gM8+A158sWzhf6azIiNDWsL3zh29D7upv1tTu4q+a94cg/z9De4zdq6DmZkYdO4cruXnw16hwPuhoXi7YUPYKRQm1+/f169jenx8hfXr5+ODsUFBeNjDAy5KpVndKqYExPXt7fFGUBCu5ufjSn4+zubk4I4JLVr9fXzQq149NHd1RXMXF8RlZFS7HCIMMMrBAKPmKywWWPlnBq7cLcQD3o4Y+5gXHO0VyM0F/vMfKbNmXp4UEIwcCXzxhXRcZYIFW043tdU3FHO/qVn6293k4GAoFQocuXcPx+7dwz0T8jgAwKomTfDaP31ZpowjcLGzQ7FGg8ISjx/28MDvRjJhluTj4IC3Q0IwNigIrv9ciEz5tl+S9n0ATAtMKvUt3N5eyvIWFna/1cIYbWvGtWvA3r2Au3uF5yjN2tNNM4uL8dqFC9j4z8yant7eGOjri7nXrxv97CXm52N9Whq+TU3F3zk5lTqfvUKBdq6uOJuXh2wjnz9tEHi+UydkFBfjVlERbhUVYW9GBt67fr3Sr9EcdgCMjV6RK4cIA4xyMMCwDlt9A3/7p3QsKrwMdf37/3SUt1V45mo4Ti3zhfYL4GOPSXku2rWTgoU3Jgkk1csA6hcCtx0RfNcLSz5RlBssVDY5l+64ajYTwtzzmP3t7sAB3KjEtENHhQKFJvwbKn3xMuUC3s3LC9+kpuKzmzdxLje3wnMoAHwQGoo3goPhVmo8gKVacowFJpVp9VAqFNIHtDL9dJUtX5W6mUEIgS9SUjDh0iXklpMETABo4eKCs7m5uro4AFDa2SG/nOO87O3xlLc39mdlldv6YCmPe3riqXr18ICTEzKLizHm0qUKj+nn44NstRpnc3ORaGIdjQV11vqfzACjHAwwLM9W38Df/ikdC9ykC4revDzNP4/ntEDD675YsEDqbtYlv0pPx8RLl3GjsOS0PxWWNJF3zQVbzYQw5zymHFPfwQFzGzXCzcJCJBYUILGgABdzcnCzqKjCOj1Xvz4ifXzQwd0dTZ2dEX74sFkXr8qMI1h24wYmXrlSYd0snfTJnIATML07xlZsVbfT2dlod+wYiky4ND3u6Ymh/v7o7+uLPf90J1RUPyEEEgoKMD8hAStv3jSpTkpIn3cfBwfYKxT4y4QWk5KfI3MCtC+SkzHqwoUKzzOqQQPMf+AB1Hdw0G2z5v9kBhjlYIBRMVt+Azf1XIXFAi5bD0Jdr0A/uNDSAIrbKtx9+mF4ut8vUJX6mTMFzdRzFWs0iElPxyAT+rWr+g3F1ObtRz084Glvj2IhkFZYiJOVbHaujNJ99VW5eNlyHIEtmNMdYyu2qJupn9f1zZtjYKnfU2XqZ+p5fmzZEs/Wrw+7UsF3ZQNia3WZAYCDQoFn6tXDUH9/FAuBIefOWa1VlNNUyWyV/QZelalhlTnXyj8z9LpFyrADhG8BXj52GX2aucPFzg4qOzuMuXjRrPpV9huAKRkfXzp3DosSE5FQUICkggKjfaul7c/KKnPRNKV+RRoNjmdn4zMT5+vuz8oysUb3tXNzQ2cPDwSrVAhxcsKtwkJMMqGVIMDRUe9xlK8vYlq0MPiaKrp4KRUKk1ocSp+zquWsJcrXF5E+PtVyWqIt6mZqVk9Df2uVqV8XLy8Eq1QVBgq9SwQXAEya5rs4PLzMOSv7GTelfp5KJUKdnHAyJwc/3r6NH2/fLlMnLVP+J1saWzBIx9Rv4EUaDU5mZ+PrlBQsN6GJ8Ulvbzzl7Y3Wbm5o7eoKf0dHbLl1q9xzfdGsGRqqVDiZnY0jd7PxY0IGctys0286LzQUUT4+CHd2hlMlRpZnFxfjSn4+LuXmYsedO/g8JaVS57UHUPG4comnUonHvbzQzcsLGiHw9tWrBusnAAz198etoiL8mZlpdACbIVOCg9HS1RX2CgUu5ebi/YSECo8p3bpS1b56a47lscU4Aqo6W65fUpWWM3Nbc6zRZXYmJwfrUlOxJjkZaSZ0UVblvWMXSTkYYBhmytQrd6USbVxdcSw72+hKjKbwsbfHPbUaBRphuLtDG2qboRU8EOitRK5GgxsFBYivII9BSQoADVUqpBQWoqCcPwuVQgFve3ukmPCHXNqEoCAM9vNDIycn+Dg4oPGhQxVmVHQAcM/M99vb3h6Pe3pib0YGMtVqky+sVbkYcxwBVYWtA8GqdPvYYnB7Zeq3LjUVQ63cDcgAoxwMMAyrTH8fIF24HnR2xiETMh2O8vfHPY0Gf2Vn41JensldA473HFB4whO47IbAAlckD7gE4VVoeIk+DaC8q0Ju5MNwtFdU6jU1dXZGcmEhsirxbV+rvr09mri4wE2pxC4TFnYyZybEc/Xr42R2NvZkZGBzerpJ7/m4wEC8EhiIVq6usCuRgrq88xibRVKZY0oeW5fHEVDV2DoQrO6ZMi093ootGFbCAMMwUwfAvRoQgMnBwXjQxQUCgP/vB3FbUWD0ol9fqJD6xP1vGrlqNRYmJGL29WsVnmtKbnOkrPPHmDHStNNpP/8zi0RA/3z/zCJ5K7sF5vfRz55n6jchOwDpRUVYmZSEeSbMcX8/NBTjgoLg/c/I7ap+4zf1gleVgYrm5nKozt/uzFWd60YSBoKVZ4vWHw7ypEq5VViI3+7cMansID8/NHN1BfDPkuXLw4E3zkgXeQMX/YL/hOOYuwKdOkmb791SYsdCT6B/xecKcXHEwnX3H8/v4wv81KJsHoy7KkxxDNcLLoDKD8byc3RENy8vkwKMxzw9dcGFOecqqTID06oyUNGcAXpVGdRn6uBLOVTnupGkOg92ra6q8n/IGtiCUYNYOs3z5dxcfHLjBr5ISalwTIWhyFe3WmmXdGD8ZcCvxPiNVBWwIhzY54sBA+4vgXDtGhD2gADWHwR8jLd8IF2FbzUPY8ggw1NWDWXyNMaqyY6qcC5zcKAiEVXEmv+H2IJRC1kyzfO4wEAcuXcPW27d0l2k2rm5oaunJz65kWSwC0Io7ke+RUVAenqJ1Ur3+QL7fYBWGbpMmfjbC9BIF7h/GjwASMsnvDREgW+WhwPzjLd8YEU4gt4zfIF0tFdgUjfTv31W5ptQVb8BWPtbV3X7hkJE1U91af1hC0YNYMk0z6X1rlcPU0NC0M3LC1u2KNBvSTowrlRrRJrUGvHOY764eRPYuhVo3x54991/WjAqUHK1UuD+iqU3Ghs4V6oKWBmOkHhfWVcsre79v9W9fkRUO3GQZzlqWoBhyvRRD6USbwQFQaFQQEDKErni5s1yF5FytbPD/9q1Q2s3N+k82ov+DQB2wmhrhFZgIHDxItCsmXlLm2tXLBV2AmhZ4lynvaDQKKyyYmllVfeBgNW9fkRU+7CLpBbZl5FR4cI8WWo1PjAhKVJJORoN7pTI47BvX4mVQzUK4JThLoi+faVVRh97TAoaliyRAgWFwvBqpYsXG26FiIqSVjKdOFGBGyXOZa0VS81R3QcCVvf6EVHdxgCjmjtgYvrmJ7288KCLCxT/ZGHcYUJOhpIpeU3MJo0XXgC6dr3/+H6goL+0eXBwxYFCVBQQGWneiqVERFS9McCoppILCjAzPh5rTUw//U6jRrpvs3F375oUYJScyhgQYFq9DJWrSqCgVOqPzyAiotrB0CRBm1qxYgVCQ0Ph5OSEiIgIHD58uNzyGRkZGDduHAICAqBSqfDggw9i+/btNqqt5aiFQNzdu1ifmoq4u3eh/qd/IU+txgfXrqHJoUO64MLZzs5o5mwFpMF9Xby8dNu0i+RU6pguQImHZY9RSN0XXboY3q8NFAYNkn6yFYKIqG6TtQVj48aNmDJlClatWoWIiAgsXrwYvXr1woULF+Dn51emfGFhIZ588kn4+fkhJiYGQUFBuH79OrzKuzJWQwanjzo6op+vL2Jv3ULiP9sf9vDAJw88gJuFhZWalmjOVMbPPwcyMgzXt6LxFERERKXJOoskIiICHTt2xPLlywEAGo0GISEhmDBhAqZPn16m/KpVq7BgwQKcP38eDiWyKFaG3LNITJk+2lClwr8bN8YAPz8o/rm6WzPN81dfASNGSPefew44flx/PEV1GnhJRETyqRHTVAsLC+Hi4oKYmBg8//zzuu3Dhw9HRkYGtm7dWuaY3r17o169enBxccHWrVvh6+uLwYMHY9q0aVAa+WpdUFCAghIX2KysLISEhMgSYJgy5dRTqcSNzp3hZl+2ccnSmTwBYONGYPBgQKMBxo8Hli6V7nPgJRERlVYjpqneunULarUa/qUWZPL398f58+cNHnP16lX8/vvvGDJkCLZv347Lly9j7NixKCoqwpw5cwweEx0djXnz5lm8/uYwZcppplqNo/fuGZx+aM60xPKO2bIFGDJECiheflmacqpQcOAlERFVneyDPCtDo9HAz88Pq1evRvv27TFgwAC8++67WLVqldFjZsyYgczMTN0tMTHRhjXWV3JaqCXKVYUQwDffSAm2hg4FVq0C7GrUp4GIiKoz2VowfHx8oFQqkZqaqrc9NTUVDRo0MHhMQEAAHBwc9LpDmjdvjpSUFBQWFsLRwAqSKpUKKpXKspU3Q7FGgz3GRlGWYuqKmVWhUEgLkK1cKXWNsAuEiIgsSbbvrI6Ojmjfvj12796t26bRaLB792507tzZ4DGPPvooLl++DE2JlT8vXryIgIAAg8FFdXE+JwePnTiB/1aQzcrQ9NGqUqulVU/Xr5d+Xrp0P+OmoyMwaRJgYLgHERFRlcjaKD5lyhT897//xVdffYVz587h9ddfR05ODkaOHAkAGDZsGGbMmKEr//rrr+POnTuYOHEiLl68iG3btuGjjz7CuHHj5HoJ5VILgQUJCWh79CgO3bsHT6US4wIDoQDK5KiwxkqYsbHS+iLdu0sDObt3B5o2BZ59Vhp3QUREZC2yfncdMGAA0tPTMXv2bKSkpKBt27b49ddfdQM/ExISYFdiYEBISAh27NiByZMno3Xr1ggKCsLEiRMxbdo0uV6C0VkaF3JzMfL8eV2q76fr1cN/H3wQwU5OeMLb2+Ay6pZcCVO3mFipOUJCAL/8AmzaBAwcaJFTERERlcHVVKvAWMKsHt7e2JiejnyNBh5KJT4JD8fIBg10OS0A666EqbcyqgHlrXJKRERkTI2YplrTGUuYdaOwEF/9M3D1KW9vfN60KUKcnMocb82VMPVWRjVACCAxUSrH6ahERGQNDDDMoBYCEy9fLjcbp7e9Pba1agV7GeZ+mroyqqnliIiIKouZD8xgSsKsu8XF+DMz00Y10leVlVGJiIgsgQGGGapTwixDunSRxlgYG9JR0cqoREREVcUAwwymJsKyRcIsQ5RKKe03UDbI4MqoRERkCwwwzNDFywvBKlWZXBZa1kiYZarbt6UVUVu3BmJigKAg/f3BwdJ2roxKRETWxEGeZlAqFFgSHo7+Z85AAegN9rRGwixTqdXAoEHAzp3SAM7Dh4HISK6MSkREtscAw0xRvr6IadHC6gmzKmPmTCm4cHEB1q7lyqhERCQfBhhVEOXri0gfH6slzKqMzZuBjz+W7q9ZA7RqZfMqEBER6TDAqCJrJswy1blzwIgR0v0pU5gCnIiI5MdBnjVcVhbQty+QnS11hfz733LXiIiIiAFGjZebC3h7S7NFNm7k0utERFQ98HIkE7XaMrM7GjQA4uKAhATAz8/i1SQiIjILAwwZxMYCEyfqL0gWHCwlxyovP0XJoMTFBfi//5OCEpUKaNLE+vUmIiIyFbtIbCw2Fujfv+xqp0lJ0vbYWOPHhYYC3bsDgwcDzz8vdY3ExFi7xkRERJXHAMOG1Gqp5UIYWIZVu23SJKlcScaCknv3gBdfNB6UEBERyYUBhg3t21c2SChJCCAxUQomtNRq4OWXDQclWoaCEiIiIjkxwLCh5GTTysXF3b+/bx9w967xstqgZN++KlWNiIjIohhg2FBAgGnlRo26f9/UoMTUckRERLbAAMOGunSRZosYyySuUAAhIcD8+fe3mRqUmFqOiIjIFhhg2JBaDbzzjnS/dJChfbx4sX4+DFODki5dLF5dIiIiszHAsKH33wemTgXGjpUyb5YUHCxNOS2dB0OplPJjAKYHJURERHJjgGEjR44A0dFSau9u3YBr14A9e4DvvpN+xscbT7IVFSUFH6YGJURERHJTCFHeBMjaJysrC56ensjMzISHh4dNzpmXB7RvL616OnAgsH69ec9jqfTiRERE5qjMNZSpwm1g1iwpuGjQAFi+3PznUSql1g8iIqLqjl0kVrZvH7BokXT/v/8F6teXtz5ERES2wADDirKzgREjpGRYo0ZJi5MRERHVBQwwrMjRERgyBGjcGPjkE7lrQ0REZDsMMKzI0RF47z3g9GnARuNJiYiIqgUGGFaQkwMUFd1/7OwsX12IiIjkwADDCsaPBx5+GDhzRu6aEBERyYPTVC3sxx+BL7+UsmxmZMhdGyIiInkwwKiiksmvXFyAV1+Vtk+dCjz6qLx1IyIikgsDjCqIjQUmTgRu3NDfHhwsDe4kIiKqqzgGw0yxsUD//mWDC0Datn277etERERUXTDAMINaLbVcGFvFRaEAJk2SyhEREdVFDDDMsG+f4ZYLLSGAxESpHBERUV3EAMMMycmWLUdERFTbMMAwQ0CAZcsRERHVNgwwzNClizRTRKEwvF+hAEJCpHJERER1EQMMMyiVwJIl0v3SQYb28eLFUjkiIqK6iAGGmaKigJgYIChIf3twsLQ9KkqeehEREVUHTLRVBVFRQGTk/UyeAQFStwhbLoiIqK5jgFFFSiXQrZvctSAiIqpe2EVCREREFscAg4iIiCyOAQYRERFZHAMMIiIisjgGGERERGRxDDCIiIjI4urcNFXxzxrrWVlZMteEiIioZtFeO7XX0vLUuQDj3r17AICQkBCZa0JERFQz3bt3D56enuWWUQhTwpBaRKPR4ObNm3B3d4fCwGplWVlZCAkJQWJiIjw8PGSoYfXA90HC90HC90HC90HC9+G+uvZeCCFw7949BAYGws6u/FEWda4Fw87ODsHBwRWW8/DwqBMflorwfZDwfZDwfZDwfZDwfbivLr0XFbVcaHGQJxEREVkcAwwiIiKyOAYYpahUKsyZMwcqlUruqsiK74OE74OE74OE74OE78N9fC+Mq3ODPImIiMj62IJBREREFscAg4iIiCyOAQYRERFZHAMMIiIisjgGGKWsWLECoaGhcHJyQkREBA4fPix3lWxq7ty5UCgUerdmzZrJXS2r++OPP9CnTx8EBgZCoVDghx9+0NsvhMDs2bMREBAAZ2dn9OzZE5cuXZKnslZU0fswYsSIMp+Pp59+Wp7KWlF0dDQ6duwId3d3+Pn54fnnn8eFCxf0yuTn52PcuHGoX78+3Nzc0K9fP6SmpspUY+sw5X3o1q1bmc/EmDFjZKqxdXz66ado3bq1LplW586d8csvv+j214XPgjkYYJSwceNGTJkyBXPmzMHx48fRpk0b9OrVC2lpaXJXzaZatGiB5ORk3e3PP/+Uu0pWl5OTgzZt2mDFihUG98+fPx9Lly7FqlWrcOjQIbi6uqJXr17Iz8+3cU2tq6L3AQCefvppvc/H+vXrbVhD29i7dy/GjRuHgwcPYufOnSgqKsJTTz2FnJwcXZnJkyfjp59+wvfff4+9e/fi5s2biIqKkrHWlmfK+wAAr7zyit5nYv78+TLV2DqCg4Px8ccf49ixYzh69CieeOIJREZG4syZMwDqxmfBLIJ0OnXqJMaNG6d7rFarRWBgoIiOjpaxVrY1Z84c0aZNG7mrISsAYsuWLbrHGo1GNGjQQCxYsEC3LSMjQ6hUKrF+/XoZamgbpd8HIYQYPny4iIyMlKU+ckpLSxMAxN69e4UQ0u/fwcFBfP/997oy586dEwDEgQMH5Kqm1ZV+H4QQomvXrmLixInyVUom3t7e4vPPP6+znwVTsAXjH4WFhTh27Bh69uyp22ZnZ4eePXviwIEDMtbM9i5duoTAwEA0btwYQ4YMQUJCgtxVklV8fDxSUlL0Phuenp6IiIioc58NAIiLi4Ofnx+aNm2K119/Hbdv35a7SlaXmZkJAKhXrx4A4NixYygqKtL7TDRr1gwNGzas1Z+J0u+D1rp16+Dj44OWLVtixowZyM3NlaN6NqFWq7Fhwwbk5OSgc+fOdfazYIo6t9iZMbdu3YJarYa/v7/edn9/f5w/f16mWtleREQEvvzySzRt2hTJycmYN28eunTpgtOnT8Pd3V3u6skiJSUFAAx+NrT76oqnn34aUVFRCAsLw5UrV/DOO+/gmWeewYEDB6BUKuWunlVoNBpMmjQJjz76KFq2bAlA+kw4OjrCy8tLr2xt/kwYeh8AYPDgwWjUqBECAwPx119/Ydq0abhw4QJiY2NlrK3l/f333+jcuTPy8/Ph5uaGLVu24KGHHsLJkyfr3GfBVAwwSM8zzzyju9+6dWtERESgUaNG2LRpE0aPHi1jzag6GDhwoO5+q1at0Lp1azzwwAOIi4tDjx49ZKyZ9YwbNw6nT5+uE2ORymPsfXj11Vd191u1aoWAgAD06NEDV65cwQMPPGDralpN06ZNcfLkSWRmZiImJgbDhw/H3r175a5WtcYukn/4+PhAqVSWGfmbmpqKBg0ayFQr+Xl5eeHBBx/E5cuX5a6KbLS/f342ymrcuDF8fHxq7edj/Pjx+Pnnn7Fnzx4EBwfrtjdo0ACFhYXIyMjQK19bPxPG3gdDIiIiAKDWfSYcHR0RHh6O9u3bIzo6Gm3atMGSJUvq3GehMhhg/MPR0RHt27fH7t27dds0Gg12796Nzp07y1gzeWVnZ+PKlSsICAiQuyqyCQsLQ4MGDfQ+G1lZWTh06FCd/mwAwI0bN3D79u1a9/kQQmD8+PHYsmULfv/9d4SFhentb9++PRwcHPQ+ExcuXEBCQkKt+kxU9D4YcvLkSQCodZ+J0jQaDQoKCurMZ8Esco8yrU42bNggVCqV+PLLL8XZs2fFq6++Kry8vERKSorcVbOZN998U8TFxYn4+Hixf/9+0bNnT+Hj4yPS0tLkrppV3bt3T5w4cUKcOHFCABCLFi0SJ06cENevXxdCCPHxxx8LLy8vsXXrVvHXX3+JyMhIERYWJvLy8mSuuWWV9z7cu3dPTJ06VRw4cEDEx8eLXbt2iXbt2okmTZqI/Px8uatuUa+//rrw9PQUcXFxIjk5WXfLzc3VlRkzZoxo2LCh+P3338XRo0dF586dRefOnWWsteVV9D5cvnxZvPfee+Lo0aMiPj5ebN26VTRu3Fg8/vjjMtfcsqZPny727t0r4uPjxV9//SWmT58uFAqF+O2334QQdeOzYA4GGKUsW7ZMNGzYUDg6OopOnTqJgwcPyl0lmxowYIAICAgQjo6OIigoSAwYMEBcvnxZ7mpZ3Z49ewSAMrfhw4cLIaSpqrNmzRL+/v5CpVKJHj16iAsXLshbaSso733Izc0VTz31lPD19RUODg6iUaNG4pVXXqmVAbih9wCA+OKLL3Rl8vLyxNixY4W3t7dwcXERffv2FcnJyfJV2goqeh8SEhLE448/LurVqydUKpUIDw8Xb731lsjMzJS34hY2atQo0ahRI+Ho6Ch8fX1Fjx49dMGFEHXjs2AOLtdOREREFscxGERERGRxDDCIiIjI4hhgEBERkcUxwCAiIiKLY4BBREREFscAg4iIiCyOAQYRERFZHAMMIiIisjgGGERU48XFxUGhUJRZcIqI5MMAg4iIiCyOAQYRERFZHAMMIqoyjUaD6OhohIWFwdnZGW3atEFMTAyA+90X27ZtQ+vWreHk5ISHH34Yp0+f1nuOzZs3o0WLFlCpVAgNDcXChQv19hcUFGDatGkICQmBSqVCeHg41qxZo1fm2LFj6NChA1xcXPDII4/gwoUL1n3hRGQUAwwiqrLo6Gh8/fXXWLVqFc6cOYPJkydj6NCh2Lt3r67MW2+9hYULF+LIkSPw9fVFnz59UFRUBEAKDF588UUMHDgQf//9N+bOnYtZs2bhyy+/1B0/bNgwrF+/HkuXLsW5c+fw2Wefwc3NTa8e7777LhYuXIijR4/C3t4eo0aNssnrJyID5F7OlYhqtvz8fOHi4iL+97//6W0fPXq0GDRokG4J+A0bNuj23b59Wzg7O4uNGzcKIYQYPHiwePLJJ/WOf+utt8RDDz0khBDiwoULAoDYuXOnwTpoz7Fr1y7dtm3btgkAIi8vzyKvk4gqhy0YRFQlly9fRm5uLp588km4ubnpbl9//TWuXLmiK9e5c2fd/Xr16qFp06Y4d+4cAODcuXN49NFH9Z730UcfxaVLl6BWq3Hy5EkolUp07dq13Lq0bt1adz8gIAAAkJaWVuXXSESVZy93BYioZsvOzgYAbNu2DUFBQXr7VCqVXpBhLmdnZ5PKOTg46O4rFAoA0vgQIrI9tmAQUZU89NBDUKlUSEhIQHh4uN4tJCREV+7gwYO6+3fv3sXFixfRvHlzAEDz5s2xf/9+vefdv38/HnzwQSiVSrRq1QoajUZvTAcRVW9swSCiKnF3d8fUqVMxefJkaDQaPPbYY8jMzMT+/fvh4eGBRo0aAQDee+891K9fH/7+/nj33Xfh4+OD559/HgDw5ptvomPHjnj//fcxYMAAHDhwAMuXL8fKlSsBAKGhoRg+fDhGjRqFpUuXok2bNrh+/TrS0tLw4osvyvXSiag8cg8CIaKaT6PRiMWLF4umTZsKBwcH4evrK3r16iX27t2rG4D5008/iRYtWghHR0fRqVMncerUKb3niImJEQ899JBwcHAQDRs2FAsWLNDbn5eXJyZPniwCAgKEo6OjCA8PF2vXrhVC3B/keffuXV35EydOCAAiPj7e2i+fiAxQCCGEzDEOEdVicXFx6N69O+7evQsvLy+5q0NENsIxGERERGRxDDCIiIjI4thFQkRERBbHFgwiIiKyOAYYREREZHEMMIiIiMjiGGAQERGRxTHAICIiIotjgEFEREQWxwCDiIiILI4BBhEREVnc/wM7xjql9+OG5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='33' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      33.00% [33/100] [00:31<01:03]\n",
       "      <br>\n",
       "      ████████████████████100.00% [10/10] [val_loss=1.3538, val_auc=0.6238]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_auc without improvement in 10 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "dfhistory = model.fit(train_data = dl_train,\n",
    "    val_data = dl_val,\n",
    "    epochs=100,\n",
    "    ckpt_path='checkpoint',\n",
    "    patience=10,\n",
    "    monitor='val_auc',\n",
    "    mode='max',\n",
    "    plot=True,\n",
    "    cpu=True\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4970e005",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "a69e3841",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████| 10/10 [00:00<00:00, 61.64it/s, val_auc=0.647, val_loss=0.686]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'val_loss': 0.6855947375297546, 'val_auc': 0.6471151113510132}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(dl_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "117522ea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8547262e",
   "metadata": {},
   "source": [
    "### 4，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "d1a3c8bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6471136914166838\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "model.eval()\n",
    "dl_val = model.accelerator.prepare(dl_val)\n",
    "with torch.no_grad():\n",
    "    result = torch.cat([model.forward(t) for t in dl_val])\n",
    "\n",
    "preds = F.sigmoid(result)\n",
    "labels = torch.cat([x['label'] for x in dl_val])\n",
    "\n",
    "val_auc = roc_auc_score(labels.numpy(),preds.numpy())\n",
    "print(val_auc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee699dff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "55f2ad02",
   "metadata": {},
   "source": [
    "### 5, 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "5df6b615",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.save(model.net.state_dict(),\"best_din.pt\")\n",
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(\"best_din.pt\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "6297c73d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6471136914166838\n"
     ]
    }
   ],
   "source": [
    "net_clone.eval()\n",
    "labels = torch.tensor([x[\"label\"] for x in ds_val])\n",
    "preds = torch.cat([net_clone(x).data for x in dl_val]) \n",
    "val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())\n",
    "print(val_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b109369",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d2fee274",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}