3.12. 权重衰减.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "3.12. 权重衰减.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/chongzicbo/Dive-into-Deep-Learning-tf.keras/blob/master/3.12.%20%E6%9D%83%E9%87%8D%E8%A1%B0%E5%87%8F.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8fcjNARAht4z",
        "colab_type": "text"
      },
      "source": [
        "##3.12. 权重衰减\n",
        "&emsp;&emsp;上一节中我们观察了过拟合现象，即模型的训练误差远小于它在测试集上的误差。虽然增大训练数据集可能会减轻过拟合，但是获取额外的训练数据往往代价高昂。本节介绍应对过拟合问题的常用方法：权重衰减（weight decay）。\n",
        "\n",
        "###3.12.1. 方法\n",
        "&emsp;&emsp;权重衰减等价于 $L_2$ 范数正则化（regularization）。正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。我们先描述 $L_2$ 范数正则化，再解释它为何又称权重衰减。\n",
        "\n",
        "&emsp;&emsp;$L_2$ 范数正则化在模型原损失函数基础上添加 $L_2$ 范数惩罚项，从而得到训练所需要最小化的函数。 $L_2$ 范数惩罚项指的是模型权重参数每个元素的平方和与一个正的常数的乘积。以“线性回归”一节中的线性回归损失函数\n",
        "$$\n",
        "\\ell(w_1, w_2, b) = \\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right)^2\n",
        "$$\n",
        "\n",
        "为例，其中 $w_1,w_2$ 是权重参数， $b$ 是偏差参数，样本 $i$ 的输入为 $x^{(i)}_1,x^{(i)}_2$ ，标签为 $y^{(i)}$ ，样本数为 n 。将权重参数用向量 $w=[w1,w2]$ 表示，带有 $L_2$ 范数惩罚项的新损失函数为\n",
        "\n",
        "$$\n",
        "\\ell(w_1, w_2, b) + \\frac{\\lambda}{2n} \\|\\boldsymbol{w}\\|^2,\n",
        "$$\n",
        "\n",
        "&emsp;&emsp;其中超参数 $\\lambda>0$ 。当权重参数均为0时，惩罚项最小。当 $\\lambda$ 较大时，惩罚项在损失函数中的比重较大，这通常会使学到的权重参数的元素较接近0。当 $\\lambda$ 设为0时，惩罚项完全不起作用。上式中 $L_2$ 范数平方 $||w||^2$ 展开后得到 $w^2_1+w^2_2$ 。有了 $L_2$ 范数惩罚项后，在小批量随机梯度下降中，我们将“线性回归”一节中权重 $w_1$ 和 $w_2$ 的迭代方式更改为\n",
        "\n",
        "$$\n",
        "w_1\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_1 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_1^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\\n",
        "w_2 \\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_2 -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_2^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right).\n",
        "$$\n",
        "\n",
        "&emsp;&emsp;可见， $L_2$ 范数正则化令权重 $w_1$ 和 $w_2$ 先自乘小于1的数，再减去不含惩罚项的梯度。因此， $L_2$ 范数正则化又叫权重衰减。权重衰减通过惩罚绝对值较大的模型参数为需要学习的模型增加了限制，这可能对过拟合有效。实际场景中，我们有时也在惩罚项中添加偏差元素的平方和。\n",
        "\n",
        "###3.12.2. 高维线性回归实验\n",
        "&emsp;&emsp;下面，我们以高维线性回归为例来引入一个过拟合问题，并使用权重衰减来应对过拟合。设数据样本特征的维度为 $p$ 。对于训练数据集和测试数据集中特征为 $x_1,x_2,…,x_p$ 的任一样本，我们使用如下的线性函数来生成该样本的标签：\n",
        "$$\n",
        "y = 0.05 + \\sum_{i = 1}^p 0.01x_i +  \\epsilon,\n",
        "$$\n",
        "&emsp;&emsp;其中噪声项 ϵ 服从均值为0、标准差为0.01的正态分布。为了较容易地观察过拟合，我们考虑高维线性回归问题，如设维度 $p$=200 ；同时，我们特意把训练数据集的样本数设低，如20。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wzs_gHIJh-RX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 64
        },
        "outputId": "9891113b-5612-4fb5-b494-3df1e93ff67c"
      },
      "source": [
        "%matplotlib inline\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from IPython import display\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f1k7qI-7Q-9Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tf.enable_eager_execution() #启用动态图计算"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4NPYK0dXRC0z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "n_train,n_test,num_inputs=20,100,200\n",
        "true_w,true_b=tf.ones((num_inputs,1))*0.01,0.05\n",
        "features=tf.random.normal(shape=(n_train+n_test,num_inputs))\n",
        "labels=tf.matmul(features,true_w)+true_b\n",
        "labels+=tf.random.normal(shape=labels.shape,stddev=0.01)\n",
        "train_features,test_features=features[:n_train,:],features[n_train:,:]\n",
        "train_labels,test_labels=labels[:n_train],labels[n_train:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OeyJV_3_R9ez",
        "colab_type": "text"
      },
      "source": [
        "###3.12.3. 从零开始实现\n",
        "&emsp;&emsp;下面先介绍从零开始实现权重衰减的方法。我们通过在目标函数后添加 $L_2$ 范数惩罚项来实现权重衰减。\n",
        "####3.12.3.1. 初始化模型参数\n",
        "&emsp;&emsp;首先，定义随机初始化模型参数的函数。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xv-XJLqeF2w-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def init_params():\n",
        "  w=tf.Variable(tf.random.normal(shape=(num_inputs,1),stddev=1),trainable=True)\n",
        "  b=tf.Variable(tf.zeros(shape=(1,)),trainable=True)\n",
        "  return [w,b]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zNTe9oeHvF6M",
        "colab_type": "text"
      },
      "source": [
        "####3.12.3.2. 定义 L2 范数惩罚项\n",
        "&emsp;&emsp;下面定义 $L_2$ 范数惩罚项。这里只惩罚模型的权重参数。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vd_eCBsmtnJy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def l2_penalty(w):\n",
        "  return tf.reduce_sum((w**2))/2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3q1HRe18tw8T",
        "colab_type": "text"
      },
      "source": [
        "####3.12.3.3. 定义训练和测试\n",
        "&emsp;&emsp;下面定义如何在训练数据集和测试数据集上分别训练和测试模型。与前面几节中不同的是，这里在计算最终的损失函数时添加了 $L_2$ 范数惩罚项。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UDTR4QJxuGuL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def linreg(X,w,b):\n",
        "  return tf.matmul(X,w)+b \n",
        "\n",
        "def squared_loss(y_hat,y):\n",
        "  return (y_hat-tf.reshape(y,shape=y_hat.shape))**2/2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D-P_uox9IEAx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def use_svg_display():\n",
        "  #用矢量图显示\n",
        "  display.set_matplotlib_formats('svg')\n",
        "def set_figsize(figsize=(3.5,2.5)):\n",
        "  use_svg_display()\n",
        "  #设置图的尺寸\n",
        "  plt.rcParams['figure.figsize']=figsize\n",
        "\n",
        "def semilogy(x_vals,y_vals,x_label,y_label,x2_vals=None,y2_vals=None,legend=None,figsize=(3.5,2.5)):\n",
        "  set_figsize(figsize)\n",
        "  plt.xlabel(x_label)\n",
        "  plt.ylabel(y_label)\n",
        "  plt.semilogy(x_vals,y_vals)\n",
        "  if x2_vals and y2_vals:\n",
        "    plt.semilogy(x2_vals,y2_vals,linestyle=':')\n",
        "    plt.legend(legend)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "No_Wn7YCt7Yi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size,num_epochs,lr=1,100,0.003\n",
        "buffer_size=1000\n",
        "net,loss=linreg,squared_loss\n",
        "train_iter=tf.data.Dataset.from_tensor_slices((train_features,train_labels)).shuffle(buffer_size=buffer_size).batch(batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G58kRHV9u2pn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sgd(params,loss,t,lr,batch_size):\n",
        "  for param in params:\n",
        "    dl_dp=t.gradient(loss,param) #求梯度\n",
        "    param.assign_sub(lr*dl_dp/batch_size) #更新梯度\n",
        "def fit(lambd):\n",
        "  w,b=init_params()\n",
        "  train_ls,test_ls=[],[]\n",
        "  for _ in range(num_epochs):\n",
        "    for X,y in train_iter:\n",
        "      with tf.GradientTape(persistent=True) as t:\n",
        "        l=loss(net(X,w,b),y)+lambd*l2_penalty(w)\n",
        "      sgd([w,b],l,t,lr,batch_size)\n",
        "    train_ls.append(loss(net(train_features,w,b),train_labels).numpy().mean())\n",
        "    test_ls.append(loss(net(test_features,w,b),test_labels).numpy().mean())\n",
        "  semilogy(range(1,num_epochs+1),train_ls,'epochs','loss',range(1,num_epochs+1),test_ls,['train','test'])\n",
        "  print('L2 norm of w:',tf.norm(w).numpy())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1zHMaUfBI8bp",
        "colab_type": "text"
      },
      "source": [
        "####3.12.3.4. 观察过拟合\n",
        "&emsp;&emsp;接下来，让我们训练并测试高维线性回归模型。当lambd设为0时，我们没有使用权重衰减。结果训练误差远小于测试集上的误差。这是典型的过拟合现象。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fX73bddtIcrK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "d91ae0b3-4657-424e-db12-ba3ff18c0df6"
      },
      "source": [
        "fit(lambd=0)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "L2 norm of w: 14.709459\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 258.944602 180.65625\" width=\"258.944602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 258.944602 180.65625 \nL 258.944602 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 55.778125 143.1 \nL 251.078125 143.1 \nL 251.078125 7.2 \nL 55.778125 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m14b18c4b0e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(59.680759 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 20 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(92.367278 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(128.235046 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <defs>\n       <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n      </defs>\n      <g transform=\"translate(164.102815 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 80 -->\n      <defs>\n       <path d=\"M 31.78125 34.625 \nQ 24.75 34.625 20.71875 30.859375 \nQ 16.703125 27.09375 16.703125 20.515625 \nQ 16.703125 13.921875 20.71875 10.15625 \nQ 24.75 6.390625 31.78125 6.390625 \nQ 38.8125 6.390625 42.859375 10.171875 \nQ 46.921875 13.96875 46.921875 20.515625 \nQ 46.921875 27.09375 42.890625 30.859375 \nQ 38.875 34.625 31.78125 34.625 \nz\nM 21.921875 38.8125 \nQ 15.578125 40.375 12.03125 44.71875 \nQ 8.5 49.078125 8.5 55.328125 \nQ 8.5 64.0625 14.71875 69.140625 \nQ 20.953125 74.21875 31.78125 74.21875 \nQ 42.671875 74.21875 48.875 69.140625 \nQ 55.078125 64.0625 55.078125 55.328125 \nQ 55.078125 49.078125 51.53125 44.71875 \nQ 48 40.375 41.703125 38.8125 \nQ 48.828125 37.15625 52.796875 32.3125 \nQ 56.78125 27.484375 56.78125 20.515625 \nQ 56.78125 9.90625 50.3125 4.234375 \nQ 43.84375 -1.421875 31.78125 -1.421875 \nQ 19.734375 -1.421875 13.25 4.234375 \nQ 6.78125 9.90625 6.78125 20.515625 \nQ 6.78125 27.484375 10.78125 32.3125 \nQ 14.796875 37.15625 21.921875 38.8125 \nz\nM 18.3125 54.390625 \nQ 18.3125 48.734375 21.84375 45.5625 \nQ 25.390625 42.390625 31.78125 42.390625 \nQ 38.140625 42.390625 41.71875 45.5625 \nQ 45.3125 48.734375 45.3125 54.390625 \nQ 45.3125 60.0625 41.71875 63.234375 \nQ 38.140625 66.40625 31.78125 66.40625 \nQ 25.390625 66.40625 21.84375 63.234375 \nQ 18.3125 60.0625 18.3125 54.390625 \nz\n\" id=\"DejaVuSans-56\"/>\n      </defs>\n      <g transform=\"translate(199.970584 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-56\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#m14b18c4b0e\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 100 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(232.657102 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epochs -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(135.595313 171.376563)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m5308cf7883\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5308cf7883\" y=\"121.797101\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-11}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n      </defs>\n      <g transform=\"translate(20.878125 125.59632)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(231.391602 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5308cf7883\" y=\"96.950672\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-8}}$ -->\n      <g transform=\"translate(25.278125 100.749891)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-56\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5308cf7883\" y=\"72.104244\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-5}}$ -->\n      <defs>\n       <path d=\"M 10.796875 72.90625 \nL 49.515625 72.90625 \nL 49.515625 64.59375 \nL 19.828125 64.59375 \nL 19.828125 46.734375 \nQ 21.96875 47.46875 24.109375 47.828125 \nQ 26.265625 48.1875 28.421875 48.1875 \nQ 40.625 48.1875 47.75 41.5 \nQ 54.890625 34.8125 54.890625 23.390625 \nQ 54.890625 11.625 47.5625 5.09375 \nQ 40.234375 -1.421875 26.90625 -1.421875 \nQ 22.3125 -1.421875 17.546875 -0.640625 \nQ 12.796875 0.140625 7.71875 1.703125 \nL 7.71875 11.625 \nQ 12.109375 9.234375 16.796875 8.0625 \nQ 21.484375 6.890625 26.703125 6.890625 \nQ 35.15625 6.890625 40.078125 11.328125 \nQ 45.015625 15.765625 45.015625 23.390625 \nQ 45.015625 31 40.078125 35.4375 \nQ 35.15625 39.890625 26.703125 39.890625 \nQ 22.75 39.890625 18.8125 39.015625 \nQ 14.890625 38.140625 10.796875 36.28125 \nz\n\" id=\"DejaVuSans-53\"/>\n      </defs>\n      <g transform=\"translate(25.278125 75.903462)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-53\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5308cf7883\" y=\"47.257815\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g transform=\"translate(25.278125 51.057034)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5308cf7883\" y=\"22.411386\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g transform=\"translate(31.178125 26.210605)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g transform=\"translate(14.798438 84.807812)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pf090bb0ecc)\" d=\"M 64.655398 21.634097 \nL 66.448786 27.797862 \nL 68.242175 33.02193 \nL 70.035563 37.79195 \nL 71.828951 42.301286 \nL 73.62234 46.623169 \nL 75.415728 50.794932 \nL 77.209117 54.842713 \nL 79.002505 58.787158 \nL 80.795894 62.644535 \nL 82.589282 66.42689 \nL 84.38267 70.144392 \nL 86.176059 73.804992 \nL 87.969447 77.417142 \nL 89.762836 80.985707 \nL 91.556224 84.514902 \nL 93.349613 88.001773 \nL 95.143001 91.457099 \nL 96.936389 94.887976 \nL 98.729778 98.255929 \nL 100.523166 101.609826 \nL 102.316555 104.993118 \nL 104.109943 108.217975 \nL 105.903332 111.609841 \nL 107.69672 114.811294 \nL 109.490108 117.866008 \nL 111.283497 120.014671 \nL 113.076885 121.039625 \nL 114.870274 122.790127 \nL 116.663662 123.772036 \nL 118.457051 125.397798 \nL 120.250439 126.102948 \nL 122.043827 126.326428 \nL 123.837216 127.435443 \nL 125.630604 128.679243 \nL 127.423993 128.898487 \nL 129.217381 129.076832 \nL 131.01077 129.34003 \nL 132.804158 129.695315 \nL 134.597546 130.719524 \nL 136.390935 131.198292 \nL 138.184323 132.036817 \nL 139.977712 132.113113 \nL 141.7711 131.832495 \nL 143.564489 131.946313 \nL 145.357877 131.019315 \nL 147.151265 131.348305 \nL 148.944654 131.751516 \nL 150.738042 132.124615 \nL 152.531431 132.55403 \nL 154.324819 132.933824 \nL 156.118208 133.776277 \nL 157.911596 134.430598 \nL 159.704985 134.257311 \nL 161.498373 134.010306 \nL 163.291761 132.826683 \nL 165.08515 134.526109 \nL 166.878538 133.87357 \nL 168.671927 134.483264 \nL 170.465315 134.284204 \nL 172.258704 135.261673 \nL 174.052092 134.428992 \nL 175.84548 133.873111 \nL 177.638869 134.554752 \nL 179.432257 134.115892 \nL 181.225646 134.951794 \nL 183.019034 135.210007 \nL 184.812423 135.782506 \nL 186.605811 136.210467 \nL 188.399199 134.933039 \nL 190.192588 135.144512 \nL 191.985976 134.224227 \nL 193.779365 135.218914 \nL 195.572753 135.035954 \nL 197.366142 135.274346 \nL 199.15953 135.920322 \nL 200.952918 136.178983 \nL 202.746307 136.000623 \nL 204.539695 135.256953 \nL 206.333084 135.765293 \nL 208.126472 135.817674 \nL 209.919861 135.609231 \nL 211.713249 135.998553 \nL 213.506637 134.858683 \nL 215.300026 135.747872 \nL 217.093414 135.476273 \nL 218.886803 135.888155 \nL 220.680191 136.002592 \nL 222.47358 136.191741 \nL 224.266968 136.250325 \nL 226.060356 136.922727 \nL 227.853745 136.777917 \nL 229.647133 136.527167 \nL 231.440522 136.336845 \nL 233.23391 136.105188 \nL 235.027299 135.7173 \nL 236.820687 135.466618 \nL 238.614075 135.751156 \nL 240.407464 135.937583 \nL 242.200852 136.912837 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#pf090bb0ecc)\" d=\"M 64.655398 13.377273 \nL 66.448786 13.446165 \nL 68.242175 13.466927 \nL 70.035563 13.474607 \nL 71.828951 13.477835 \nL 73.62234 13.47931 \nL 75.415728 13.480016 \nL 77.209117 13.480361 \nL 79.002505 13.480531 \nL 80.795894 13.480616 \nL 82.589282 13.480657 \nL 84.38267 13.480678 \nL 86.176059 13.480688 \nL 87.969447 13.480693 \nL 89.762836 13.480696 \nL 91.556224 13.480697 \nL 93.349613 13.480698 \nL 95.143001 13.480699 \nL 96.936389 13.480699 \nL 98.729778 13.480699 \nL 100.523166 13.480699 \nL 102.316555 13.480699 \nL 104.109943 13.480699 \nL 105.903332 13.480699 \nL 107.69672 13.480699 \nL 109.490108 13.480699 \nL 111.283497 13.480699 \nL 113.076885 13.480699 \nL 114.870274 13.480699 \nL 116.663662 13.480699 \nL 118.457051 13.480699 \nL 120.250439 13.480699 \nL 122.043827 13.480699 \nL 123.837216 13.480699 \nL 125.630604 13.480699 \nL 127.423993 13.480699 \nL 129.217381 13.480699 \nL 131.01077 13.480699 \nL 132.804158 13.480699 \nL 134.597546 13.480699 \nL 136.390935 13.480699 \nL 138.184323 13.480699 \nL 139.977712 13.480699 \nL 141.7711 13.480699 \nL 143.564489 13.480699 \nL 145.357877 13.480699 \nL 147.151265 13.480699 \nL 148.944654 13.480699 \nL 150.738042 13.480699 \nL 152.531431 13.480699 \nL 154.324819 13.480699 \nL 156.118208 13.480699 \nL 157.911596 13.480699 \nL 159.704985 13.480699 \nL 161.498373 13.480699 \nL 163.291761 13.480699 \nL 165.08515 13.480699 \nL 166.878538 13.480699 \nL 168.671927 13.480699 \nL 170.465315 13.480699 \nL 172.258704 13.480699 \nL 174.052092 13.480699 \nL 175.84548 13.480699 \nL 177.638869 13.480699 \nL 179.432257 13.480699 \nL 181.225646 13.480699 \nL 183.019034 13.480699 \nL 184.812423 13.480699 \nL 186.605811 13.480699 \nL 188.399199 13.480699 \nL 190.192588 13.480699 \nL 191.985976 13.480699 \nL 193.779365 13.480699 \nL 195.572753 13.480699 \nL 197.366142 13.480699 \nL 199.15953 13.480699 \nL 200.952918 13.480699 \nL 202.746307 13.480699 \nL 204.539695 13.480699 \nL 206.333084 13.480699 \nL 208.126472 13.480699 \nL 209.919861 13.480699 \nL 211.713249 13.480699 \nL 213.506637 13.480699 \nL 215.300026 13.480699 \nL 217.093414 13.480699 \nL 218.886803 13.480699 \nL 220.680191 13.480699 \nL 222.47358 13.480699 \nL 224.266968 13.480699 \nL 226.060356 13.480699 \nL 227.853745 13.480699 \nL 229.647133 13.480699 \nL 231.440522 13.480699 \nL 233.23391 13.480699 \nL 235.027299 13.480699 \nL 236.820687 13.480699 \nL 238.614075 13.480699 \nL 240.407464 13.480699 \nL 242.200852 13.480699 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 55.778125 143.1 \nL 55.778125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 251.078125 143.1 \nL 251.078125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 55.778125 143.1 \nL 251.078125 143.1 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 55.778125 7.2 \nL 251.078125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 188.803125 91.328125 \nL 244.078125 91.328125 \nQ 246.078125 91.328125 246.078125 89.328125 \nL 246.078125 60.971875 \nQ 246.078125 58.971875 244.078125 58.971875 \nL 188.803125 58.971875 \nQ 186.803125 58.971875 186.803125 60.971875 \nL 186.803125 89.328125 \nQ 186.803125 91.328125 188.803125 91.328125 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_14\">\n     <path d=\"M 190.803125 67.070312 \nL 210.803125 67.070312 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_15\"/>\n    <g id=\"text_14\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g transform=\"translate(218.803125 70.570312)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 190.803125 81.748437 \nL 210.803125 81.748437 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_17\"/>\n    <g id=\"text_15\">\n     <!-- test -->\n     <g transform=\"translate(218.803125 85.248437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pf090bb0ecc\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fl4s5Ln4IkTW",
        "colab_type": "text"
      },
      "source": [
        "####3.12.3.5. 使用权重衰减\n",
        "&emsp;&emsp;下面我们使用权重衰减。可以看出，训练误差虽然有所提高，但测试集上的误差有所下降。过拟合现象得到一定程度的缓解。另外，权重参数的 L2 范数比不使用权重衰减时的更小，此时的权重参数更接近0。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U_Gw5kPyJFY2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "9a43321b-f815-4602-cc0b-49c4cbe9c8b1"
      },
      "source": [
        "fit(lambd=3)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "L2 norm of w: 0.03571673\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 254.544602 180.65625\" width=\"254.544602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 254.544602 180.65625 \nL 254.544602 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 51.378125 143.1 \nL 246.678125 143.1 \nL 246.678125 7.2 \nL 51.378125 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m446f46f908\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(55.280759 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 20 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(87.967278 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(123.835046 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <defs>\n       <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n      </defs>\n      <g transform=\"translate(159.702815 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 80 -->\n      <defs>\n       <path d=\"M 31.78125 34.625 \nQ 24.75 34.625 20.71875 30.859375 \nQ 16.703125 27.09375 16.703125 20.515625 \nQ 16.703125 13.921875 20.71875 10.15625 \nQ 24.75 6.390625 31.78125 6.390625 \nQ 38.8125 6.390625 42.859375 10.171875 \nQ 46.921875 13.96875 46.921875 20.515625 \nQ 46.921875 27.09375 42.890625 30.859375 \nQ 38.875 34.625 31.78125 34.625 \nz\nM 21.921875 38.8125 \nQ 15.578125 40.375 12.03125 44.71875 \nQ 8.5 49.078125 8.5 55.328125 \nQ 8.5 64.0625 14.71875 69.140625 \nQ 20.953125 74.21875 31.78125 74.21875 \nQ 42.671875 74.21875 48.875 69.140625 \nQ 55.078125 64.0625 55.078125 55.328125 \nQ 55.078125 49.078125 51.53125 44.71875 \nQ 48 40.375 41.703125 38.8125 \nQ 48.828125 37.15625 52.796875 32.3125 \nQ 56.78125 27.484375 56.78125 20.515625 \nQ 56.78125 9.90625 50.3125 4.234375 \nQ 43.84375 -1.421875 31.78125 -1.421875 \nQ 19.734375 -1.421875 13.25 4.234375 \nQ 6.78125 9.90625 6.78125 20.515625 \nQ 6.78125 27.484375 10.78125 32.3125 \nQ 14.796875 37.15625 21.921875 38.8125 \nz\nM 18.3125 54.390625 \nQ 18.3125 48.734375 21.84375 45.5625 \nQ 25.390625 42.390625 31.78125 42.390625 \nQ 38.140625 42.390625 41.71875 45.5625 \nQ 45.3125 48.734375 45.3125 54.390625 \nQ 45.3125 60.0625 41.71875 63.234375 \nQ 38.140625 66.40625 31.78125 66.40625 \nQ 25.390625 66.40625 21.84375 63.234375 \nQ 18.3125 60.0625 18.3125 54.390625 \nz\n\" id=\"DejaVuSans-56\"/>\n      </defs>\n      <g transform=\"translate(195.570584 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-56\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#m446f46f908\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 100 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(228.257102 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epochs -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m6ea93e8d2e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m6ea93e8d2e\" y=\"127.06734\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-3}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g transform=\"translate(20.878125 130.866559)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m6ea93e8d2e\" y=\"103.005202\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <g transform=\"translate(20.878125 106.804421)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m6ea93e8d2e\" y=\"78.943064\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g transform=\"translate(20.878125 82.742283)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m6ea93e8d2e\" y=\"54.880926\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g transform=\"translate(26.778125 58.680145)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#m6ea93e8d2e\" y=\"30.818788\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- $\\mathdefault{10^{1}}$ -->\n      <g transform=\"translate(26.778125 34.618007)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <defs>\n       <path d=\"M 0 0 \nL -2 0 \n\" id=\"mad04c610ee\" style=\"stroke:#000000;stroke-width:0.6;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"139.648921\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_13\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"136.642628\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_14\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"134.310766\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_15\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"132.405495\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_16\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"130.794613\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_17\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"129.399202\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_18\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"128.168363\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_19\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"119.823915\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"115.586783\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_21\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"112.58049\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"110.248628\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_23\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"108.343358\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_24\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"106.732475\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_19\">\n     <g id=\"line2d_25\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"105.337064\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_20\">\n     <g id=\"line2d_26\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"104.106225\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_21\">\n     <g id=\"line2d_27\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"95.761777\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_22\">\n     <g id=\"line2d_28\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"91.524645\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_23\">\n     <g id=\"line2d_29\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"88.518352\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_24\">\n     <g id=\"line2d_30\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"86.18649\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_25\">\n     <g id=\"line2d_31\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"84.28122\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_26\">\n     <g id=\"line2d_32\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"82.670337\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_27\">\n     <g id=\"line2d_33\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"81.274926\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_28\">\n     <g id=\"line2d_34\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"80.044087\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_29\">\n     <g id=\"line2d_35\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"71.699639\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_30\">\n     <g id=\"line2d_36\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"67.462507\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_31\">\n     <g id=\"line2d_37\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"64.456214\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_32\">\n     <g id=\"line2d_38\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"62.124352\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_33\">\n     <g id=\"line2d_39\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"60.219082\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_34\">\n     <g id=\"line2d_40\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"58.608199\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_35\">\n     <g id=\"line2d_41\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"57.212788\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_36\">\n     <g id=\"line2d_42\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"55.981949\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_37\">\n     <g id=\"line2d_43\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"47.637501\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_38\">\n     <g id=\"line2d_44\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"43.400369\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_39\">\n     <g id=\"line2d_45\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"40.394076\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_40\">\n     <g id=\"line2d_46\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"38.062214\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_41\">\n     <g id=\"line2d_47\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"36.156944\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_42\">\n     <g id=\"line2d_48\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"34.546061\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_43\">\n     <g id=\"line2d_49\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"33.15065\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_44\">\n     <g id=\"line2d_50\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"31.919811\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_45\">\n     <g id=\"line2d_51\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"23.575363\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_46\">\n     <g id=\"line2d_52\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"19.338231\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_47\">\n     <g id=\"line2d_53\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"16.331938\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_48\">\n     <g id=\"line2d_54\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"14.000076\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_49\">\n     <g id=\"line2d_55\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"12.094806\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_50\">\n     <g id=\"line2d_56\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"10.483923\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_51\">\n     <g id=\"line2d_57\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"9.088512\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_52\">\n     <g id=\"line2d_58\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#mad04c610ee\" y=\"7.857673\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_59\">\n    <path clip-path=\"url(#pae297adab8)\" d=\"M 60.255398 29.440831 \nL 62.048786 51.143221 \nL 63.842175 70.774205 \nL 65.635563 88.568994 \nL 67.428951 103.282542 \nL 69.22234 112.76067 \nL 71.015728 117.183644 \nL 72.809117 119.022148 \nL 74.602505 119.9027 \nL 76.395894 120.429438 \nL 78.189282 120.810457 \nL 79.98267 121.125733 \nL 81.776059 121.409691 \nL 83.569447 121.678199 \nL 85.362836 121.938686 \nL 87.156224 122.194722 \nL 88.949613 122.44798 \nL 90.743001 122.699208 \nL 92.536389 122.948753 \nL 94.329778 123.196821 \nL 96.123166 123.443417 \nL 97.916555 123.688571 \nL 99.709943 123.932257 \nL 101.503332 124.174482 \nL 103.29672 124.415195 \nL 105.090108 124.654408 \nL 106.883497 124.892068 \nL 108.676885 125.128156 \nL 110.470274 125.362645 \nL 112.263662 125.595538 \nL 114.057051 125.826772 \nL 115.850439 126.056347 \nL 117.643827 126.284235 \nL 119.437216 126.510404 \nL 121.230604 126.734853 \nL 123.023993 126.957533 \nL 124.817381 127.178433 \nL 126.61077 127.397516 \nL 128.404158 127.614782 \nL 130.197546 127.830182 \nL 131.990935 128.043717 \nL 133.784323 128.255362 \nL 135.577712 128.465106 \nL 137.3711 128.672899 \nL 139.164489 128.878744 \nL 140.957877 129.082615 \nL 142.751265 129.284492 \nL 144.544654 129.484353 \nL 146.338042 129.682205 \nL 148.131431 129.877994 \nL 149.924819 130.071728 \nL 151.718208 130.263388 \nL 153.511596 130.452958 \nL 155.304985 130.640421 \nL 157.098373 130.825759 \nL 158.891761 131.008971 \nL 160.68515 131.190033 \nL 162.478538 131.368942 \nL 164.271927 131.545689 \nL 166.065315 131.720259 \nL 167.858704 131.892649 \nL 169.652092 132.062845 \nL 171.44548 132.230847 \nL 173.238869 132.39665 \nL 175.032257 132.560236 \nL 176.825646 132.72161 \nL 178.619034 132.880781 \nL 180.412423 133.03772 \nL 182.205811 133.192443 \nL 183.999199 133.34495 \nL 185.792588 133.495236 \nL 187.585976 133.643313 \nL 189.379365 133.789165 \nL 191.172753 133.932806 \nL 192.966142 134.074245 \nL 194.75953 134.213478 \nL 196.552918 134.350521 \nL 198.346307 134.485372 \nL 200.139695 134.618044 \nL 201.933084 134.748533 \nL 203.726472 134.87687 \nL 205.519861 135.003055 \nL 207.313249 135.127097 \nL 209.106637 135.249015 \nL 210.900026 135.368811 \nL 212.693414 135.4865 \nL 214.486803 135.602111 \nL 216.280191 135.715641 \nL 218.07358 135.827117 \nL 219.866968 135.936556 \nL 221.660356 136.043967 \nL 223.453745 136.149376 \nL 225.247133 136.252792 \nL 227.040522 136.354236 \nL 228.83391 136.453746 \nL 230.627299 136.551314 \nL 232.420687 136.64698 \nL 234.214075 136.740758 \nL 236.007464 136.832668 \nL 237.800852 136.922727 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_60\">\n    <path clip-path=\"url(#pae297adab8)\" d=\"M 60.255398 13.377273 \nL 62.048786 17.261571 \nL 63.842175 20.996548 \nL 65.635563 24.714895 \nL 67.428951 28.433813 \nL 69.22234 32.151249 \nL 71.015728 35.861937 \nL 72.809117 39.560089 \nL 74.602505 43.23944 \nL 76.395894 46.892729 \nL 78.189282 50.511078 \nL 79.98267 54.083315 \nL 81.776059 57.59531 \nL 83.569447 61.029347 \nL 85.362836 64.363663 \nL 87.156224 67.572383 \nL 88.949613 70.626102 \nL 90.743001 73.493446 \nL 92.536389 76.143783 \nL 94.329778 78.550843 \nL 96.123166 80.696614 \nL 97.916555 82.57443 \nL 99.709943 84.19016 \nL 101.503332 85.561134 \nL 103.29672 86.713108 \nL 105.090108 87.676418 \nL 106.883497 88.48228 \nL 108.676885 89.16006 \nL 110.470274 89.735642 \nL 112.263662 90.230796 \nL 114.057051 90.663197 \nL 115.850439 91.046846 \nL 117.643827 91.392644 \nL 119.437216 91.708956 \nL 121.230604 92.00217 \nL 123.023993 92.277128 \nL 124.817381 92.537499 \nL 126.61077 92.786057 \nL 128.404158 93.024914 \nL 130.197546 93.25566 \nL 131.990935 93.479526 \nL 133.784323 93.697445 \nL 135.577712 93.910137 \nL 137.3711 94.11816 \nL 139.164489 94.32195 \nL 140.957877 94.52184 \nL 142.751265 94.718099 \nL 144.544654 94.91094 \nL 146.338042 95.100528 \nL 148.131431 95.28699 \nL 149.924819 95.470439 \nL 151.718208 95.650954 \nL 153.511596 95.828609 \nL 155.304985 96.003454 \nL 157.098373 96.175536 \nL 158.891761 96.344894 \nL 160.68515 96.511555 \nL 162.478538 96.675544 \nL 164.271927 96.836886 \nL 166.065315 96.995594 \nL 167.858704 97.15169 \nL 169.652092 97.305183 \nL 171.44548 97.456089 \nL 173.238869 97.604418 \nL 175.032257 97.750184 \nL 176.825646 97.893398 \nL 178.619034 98.034073 \nL 180.412423 98.172216 \nL 182.205811 98.307846 \nL 183.999199 98.440974 \nL 185.792588 98.571608 \nL 187.585976 98.699772 \nL 189.379365 98.825472 \nL 191.172753 98.948728 \nL 192.966142 99.069557 \nL 194.75953 99.187974 \nL 196.552918 99.304001 \nL 198.346307 99.417651 \nL 200.139695 99.528949 \nL 201.933084 99.637914 \nL 203.726472 99.744567 \nL 205.519861 99.848932 \nL 207.313249 99.951031 \nL 209.106637 100.050885 \nL 210.900026 100.148526 \nL 212.693414 100.243973 \nL 214.486803 100.337252 \nL 216.280191 100.428393 \nL 218.07358 100.51742 \nL 219.866968 100.604362 \nL 221.660356 100.689249 \nL 223.453745 100.772105 \nL 225.247133 100.852963 \nL 227.040522 100.931851 \nL 228.83391 101.008797 \nL 230.627299 101.083835 \nL 232.420687 101.156991 \nL 234.214075 101.228297 \nL 236.007464 101.297785 \nL 237.800852 101.365487 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 51.378125 143.1 \nL 51.378125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 246.678125 143.1 \nL 246.678125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 51.378125 143.1 \nL 246.678125 143.1 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 51.378125 7.2 \nL 246.678125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 184.403125 44.55625 \nL 239.678125 44.55625 \nQ 241.678125 44.55625 241.678125 42.55625 \nL 241.678125 14.2 \nQ 241.678125 12.2 239.678125 12.2 \nL 184.403125 12.2 \nQ 182.403125 12.2 182.403125 14.2 \nL 182.403125 42.55625 \nQ 182.403125 44.55625 184.403125 44.55625 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_61\">\n     <path d=\"M 186.403125 20.298437 \nL 206.403125 20.298437 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_62\"/>\n    <g id=\"text_14\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_63\">\n     <path d=\"M 186.403125 34.976562 \nL 206.403125 34.976562 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_64\"/>\n    <g id=\"text_15\">\n     <!-- test -->\n     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pae297adab8\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4vNnce1JJSn",
        "colab_type": "text"
      },
      "source": [
        "###3.12.4. 简洁实现\n",
        "&emsp;&emsp;这里我们直接在Sequential实例中添加Dense网络层时通过kernel_regularizer参数来指定使用L2正则还是L1正则，并且可以指定衰减参数。同样，偏差衰减也可以通过bias_regularizer参数进行指定。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fjjfPF66J1sA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fit_and_plot(wd):\n",
        "  test_iter=tf.data.Dataset.from_tensor_slices((test_features,test_labels)).batch(batch_size)\n",
        "  net=tf.keras.Sequential()\n",
        "  net.add(tf.keras.layers.Dense(1,kernel_regularizer=keras.regularizers.l2(wd)))\n",
        "  optimizer=tf.keras.optimizers.SGD(learning_rate=lr)\n",
        "  loss=keras.losses.MeanSquaredError()\n",
        "  net.compile(optimizer, loss=loss)\n",
        "  history=net.fit_generator(train_iter,validation_data=test_iter,epochs=num_epochs,verbose=0)\n",
        "  train_loss=history.history['loss']\n",
        "  val_loss=history.history['val_loss']\n",
        "  semilogy(range(1,num_epochs+1),train_loss,'epochs','loss',range(1,num_epochs+1),val_loss,['train','test'])\n",
        "  return net,history #history包含有训练损失和验证损失    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3DXKGTtOypj",
        "colab_type": "text"
      },
      "source": [
        "与从零开始实现权重衰减的实验现象类似，使用权重衰减可以在一定程度上缓解过拟合问题。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HnIBDnjjLhk_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "9e78f53e-9f55-4df5-ad39-2134783d0a4c"
      },
      "source": [
        "fit_and_plot(wd=0)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tensorflow.python.keras.engine.sequential.Sequential at 0x7f27081e5668>,\n",
              " <tensorflow.python.keras.callbacks.History at 0x7f27081e5be0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 258.944602 180.65625\" width=\"258.944602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 258.944602 180.65625 \nL 258.944602 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 55.778125 143.1 \nL 251.078125 143.1 \nL 251.078125 7.2 \nL 55.778125 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m1c54aab05a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"62.862009\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(59.680759 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.729778\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 20 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(92.367278 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"134.597546\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(128.235046 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.465315\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <defs>\n       <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n      </defs>\n      <g transform=\"translate(164.102815 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"206.333084\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 80 -->\n      <defs>\n       <path d=\"M 31.78125 34.625 \nQ 24.75 34.625 20.71875 30.859375 \nQ 16.703125 27.09375 16.703125 20.515625 \nQ 16.703125 13.921875 20.71875 10.15625 \nQ 24.75 6.390625 31.78125 6.390625 \nQ 38.8125 6.390625 42.859375 10.171875 \nQ 46.921875 13.96875 46.921875 20.515625 \nQ 46.921875 27.09375 42.890625 30.859375 \nQ 38.875 34.625 31.78125 34.625 \nz\nM 21.921875 38.8125 \nQ 15.578125 40.375 12.03125 44.71875 \nQ 8.5 49.078125 8.5 55.328125 \nQ 8.5 64.0625 14.71875 69.140625 \nQ 20.953125 74.21875 31.78125 74.21875 \nQ 42.671875 74.21875 48.875 69.140625 \nQ 55.078125 64.0625 55.078125 55.328125 \nQ 55.078125 49.078125 51.53125 44.71875 \nQ 48 40.375 41.703125 38.8125 \nQ 48.828125 37.15625 52.796875 32.3125 \nQ 56.78125 27.484375 56.78125 20.515625 \nQ 56.78125 9.90625 50.3125 4.234375 \nQ 43.84375 -1.421875 31.78125 -1.421875 \nQ 19.734375 -1.421875 13.25 4.234375 \nQ 6.78125 9.90625 6.78125 20.515625 \nQ 6.78125 27.484375 10.78125 32.3125 \nQ 14.796875 37.15625 21.921875 38.8125 \nz\nM 18.3125 54.390625 \nQ 18.3125 48.734375 21.84375 45.5625 \nQ 25.390625 42.390625 31.78125 42.390625 \nQ 38.140625 42.390625 41.71875 45.5625 \nQ 45.3125 48.734375 45.3125 54.390625 \nQ 45.3125 60.0625 41.71875 63.234375 \nQ 38.140625 66.40625 31.78125 66.40625 \nQ 25.390625 66.40625 21.84375 63.234375 \nQ 18.3125 60.0625 18.3125 54.390625 \nz\n\" id=\"DejaVuSans-56\"/>\n      </defs>\n      <g transform=\"translate(199.970584 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-56\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"242.200852\" xlink:href=\"#m1c54aab05a\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 100 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(232.657102 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epochs -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(135.595313 171.376563)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"m5f3fec80fb\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5f3fec80fb\" y=\"122.732894\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-13}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n       <path d=\"M 40.578125 39.3125 \nQ 47.65625 37.796875 51.625 33 \nQ 55.609375 28.21875 55.609375 21.1875 \nQ 55.609375 10.40625 48.1875 4.484375 \nQ 40.765625 -1.421875 27.09375 -1.421875 \nQ 22.515625 -1.421875 17.65625 -0.515625 \nQ 12.796875 0.390625 7.625 2.203125 \nL 7.625 11.71875 \nQ 11.71875 9.328125 16.59375 8.109375 \nQ 21.484375 6.890625 26.8125 6.890625 \nQ 36.078125 6.890625 40.9375 10.546875 \nQ 45.796875 14.203125 45.796875 21.1875 \nQ 45.796875 27.640625 41.28125 31.265625 \nQ 36.765625 34.90625 28.71875 34.90625 \nL 20.21875 34.90625 \nL 20.21875 43.015625 \nL 29.109375 43.015625 \nQ 36.375 43.015625 40.234375 45.921875 \nQ 44.09375 48.828125 44.09375 54.296875 \nQ 44.09375 59.90625 40.109375 62.90625 \nQ 36.140625 65.921875 28.71875 65.921875 \nQ 24.65625 65.921875 20.015625 65.03125 \nQ 15.375 64.15625 9.8125 62.3125 \nL 9.8125 71.09375 \nQ 15.4375 72.65625 20.34375 73.4375 \nQ 25.25 74.21875 29.59375 74.21875 \nQ 40.828125 74.21875 47.359375 69.109375 \nQ 53.90625 64.015625 53.90625 55.328125 \nQ 53.90625 49.265625 50.4375 45.09375 \nQ 46.96875 40.921875 40.578125 39.3125 \nz\n\" id=\"DejaVuSans-51\"/>\n      </defs>\n      <g transform=\"translate(20.878125 126.532113)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(231.391602 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-51\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5f3fec80fb\" y=\"98.371204\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-10}}$ -->\n      <g transform=\"translate(20.878125 102.170423)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(231.391602 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5f3fec80fb\" y=\"74.009515\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{-7}}$ -->\n      <defs>\n       <path d=\"M 8.203125 72.90625 \nL 55.078125 72.90625 \nL 55.078125 68.703125 \nL 28.609375 0 \nL 18.3125 0 \nL 43.21875 64.59375 \nL 8.203125 64.59375 \nz\n\" id=\"DejaVuSans-55\"/>\n      </defs>\n      <g transform=\"translate(25.278125 77.808734)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-55\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5f3fec80fb\" y=\"49.647825\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- $\\mathdefault{10^{-4}}$ -->\n      <g transform=\"translate(25.278125 53.447044)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-52\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"55.778125\" xlink:href=\"#m5f3fec80fb\" y=\"25.286136\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g transform=\"translate(25.278125 29.085355)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_13\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g transform=\"translate(14.798438 84.807812)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_12\">\n    <path clip-path=\"url(#pfa41dcddf5)\" d=\"M 64.655398 13.377273 \nL 66.448786 22.954757 \nL 68.242175 31.332143 \nL 70.035563 39.336683 \nL 71.828951 47.057697 \nL 73.62234 56.879686 \nL 75.415728 66.528081 \nL 77.209117 75.240361 \nL 79.002505 83.392095 \nL 80.795894 92.205957 \nL 82.589282 101.927864 \nL 84.38267 111.659099 \nL 86.176059 120.031531 \nL 87.969447 126.970032 \nL 89.762836 129.44421 \nL 91.556224 129.56834 \nL 93.349613 130.26557 \nL 95.143001 131.825931 \nL 96.936389 131.370167 \nL 98.729778 132.30117 \nL 100.523166 133.637355 \nL 102.316555 131.974591 \nL 104.109943 134.575403 \nL 105.903332 135.690724 \nL 107.69672 133.905842 \nL 109.490108 134.570485 \nL 111.283497 135.404594 \nL 113.076885 135.827417 \nL 114.870274 135.736354 \nL 116.663662 134.713775 \nL 118.457051 133.508004 \nL 120.250439 134.546695 \nL 122.043827 135.023885 \nL 123.837216 135.193991 \nL 125.630604 136.361186 \nL 127.423993 134.022598 \nL 129.217381 134.712678 \nL 131.01077 133.795081 \nL 132.804158 134.809008 \nL 134.597546 132.848418 \nL 136.390935 134.647138 \nL 138.184323 132.813051 \nL 139.977712 133.569046 \nL 141.7711 132.303661 \nL 143.564489 133.338884 \nL 145.357877 132.33807 \nL 147.151265 133.012564 \nL 148.944654 134.157532 \nL 150.738042 134.155035 \nL 152.531431 134.835778 \nL 154.324819 132.932238 \nL 156.118208 133.273759 \nL 157.911596 133.618277 \nL 159.704985 135.571234 \nL 161.498373 135.665734 \nL 163.291761 134.233919 \nL 165.08515 136.028575 \nL 166.878538 134.891471 \nL 168.671927 131.957816 \nL 170.465315 133.415466 \nL 172.258704 133.126205 \nL 174.052092 134.099004 \nL 175.84548 134.426214 \nL 177.638869 133.175613 \nL 179.432257 135.008748 \nL 181.225646 136.922727 \nL 183.019034 134.424024 \nL 184.812423 133.64086 \nL 186.605811 133.086918 \nL 188.399199 132.442399 \nL 190.192588 135.370517 \nL 191.985976 133.172424 \nL 193.779365 133.913555 \nL 195.572753 133.477745 \nL 197.366142 134.760514 \nL 199.15953 135.351191 \nL 200.952918 133.926253 \nL 202.746307 136.110734 \nL 204.539695 133.060395 \nL 206.333084 131.736555 \nL 208.126472 132.285525 \nL 209.919861 131.780761 \nL 211.713249 130.953692 \nL 213.506637 133.731833 \nL 215.300026 132.819455 \nL 217.093414 133.236048 \nL 218.886803 132.428411 \nL 220.680191 133.318642 \nL 222.47358 136.27078 \nL 224.266968 135.62053 \nL 226.060356 133.34967 \nL 227.853745 133.025339 \nL 229.647133 133.732664 \nL 231.440522 131.084294 \nL 233.23391 131.892406 \nL 235.027299 134.866186 \nL 236.820687 134.75071 \nL 238.614075 134.634961 \nL 240.407464 132.439521 \nL 242.200852 133.185673 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_13\">\n    <path clip-path=\"url(#pfa41dcddf5)\" d=\"M 64.655398 16.137165 \nL 66.448786 16.1166 \nL 68.242175 16.151024 \nL 70.035563 16.14376 \nL 71.828951 16.146428 \nL 73.62234 16.145374 \nL 75.415728 16.145745 \nL 77.209117 16.145639 \nL 79.002505 16.145666 \nL 80.795894 16.14566 \nL 82.589282 16.145661 \nL 84.38267 16.145661 \nL 86.176059 16.145661 \nL 87.969447 16.145661 \nL 89.762836 16.145661 \nL 91.556224 16.145661 \nL 93.349613 16.145661 \nL 95.143001 16.145661 \nL 96.936389 16.145661 \nL 98.729778 16.145661 \nL 100.523166 16.145661 \nL 102.316555 16.145661 \nL 104.109943 16.145661 \nL 105.903332 16.145661 \nL 107.69672 16.145661 \nL 109.490108 16.145661 \nL 111.283497 16.145661 \nL 113.076885 16.145661 \nL 114.870274 16.145661 \nL 116.663662 16.145661 \nL 118.457051 16.145661 \nL 120.250439 16.145661 \nL 122.043827 16.145661 \nL 123.837216 16.145661 \nL 125.630604 16.145661 \nL 127.423993 16.145661 \nL 129.217381 16.145661 \nL 131.01077 16.145661 \nL 132.804158 16.145661 \nL 134.597546 16.145661 \nL 136.390935 16.145661 \nL 138.184323 16.145661 \nL 139.977712 16.145661 \nL 141.7711 16.145661 \nL 143.564489 16.145661 \nL 145.357877 16.145661 \nL 147.151265 16.145661 \nL 148.944654 16.145661 \nL 150.738042 16.145661 \nL 152.531431 16.145661 \nL 154.324819 16.145661 \nL 156.118208 16.14566 \nL 157.911596 16.14566 \nL 159.704985 16.14566 \nL 161.498373 16.14566 \nL 163.291761 16.145661 \nL 165.08515 16.14566 \nL 166.878538 16.145661 \nL 168.671927 16.145661 \nL 170.465315 16.145661 \nL 172.258704 16.145661 \nL 174.052092 16.145661 \nL 175.84548 16.145661 \nL 177.638869 16.145661 \nL 179.432257 16.145661 \nL 181.225646 16.145661 \nL 183.019034 16.145661 \nL 184.812423 16.145661 \nL 186.605811 16.145661 \nL 188.399199 16.145661 \nL 190.192588 16.145661 \nL 191.985976 16.145661 \nL 193.779365 16.145661 \nL 195.572753 16.145661 \nL 197.366142 16.145661 \nL 199.15953 16.145661 \nL 200.952918 16.145661 \nL 202.746307 16.145661 \nL 204.539695 16.145661 \nL 206.333084 16.145661 \nL 208.126472 16.145661 \nL 209.919861 16.145661 \nL 211.713249 16.145661 \nL 213.506637 16.145661 \nL 215.300026 16.145661 \nL 217.093414 16.145661 \nL 218.886803 16.145661 \nL 220.680191 16.145661 \nL 222.47358 16.145661 \nL 224.266968 16.145661 \nL 226.060356 16.145661 \nL 227.853745 16.145661 \nL 229.647133 16.145661 \nL 231.440522 16.145661 \nL 233.23391 16.145661 \nL 235.027299 16.145661 \nL 236.820687 16.145661 \nL 238.614075 16.145661 \nL 240.407464 16.145661 \nL 242.200852 16.145661 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 55.778125 143.1 \nL 55.778125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 251.078125 143.1 \nL 251.078125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 55.778125 143.1 \nL 251.078125 143.1 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 55.778125 7.2 \nL 251.078125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 188.803125 91.328125 \nL 244.078125 91.328125 \nQ 246.078125 91.328125 246.078125 89.328125 \nL 246.078125 60.971875 \nQ 246.078125 58.971875 244.078125 58.971875 \nL 188.803125 58.971875 \nQ 186.803125 58.971875 186.803125 60.971875 \nL 186.803125 89.328125 \nQ 186.803125 91.328125 188.803125 91.328125 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_14\">\n     <path d=\"M 190.803125 67.070312 \nL 210.803125 67.070312 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_15\"/>\n    <g id=\"text_14\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g transform=\"translate(218.803125 70.570312)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_16\">\n     <path d=\"M 190.803125 81.748437 \nL 210.803125 81.748437 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_17\"/>\n    <g id=\"text_15\">\n     <!-- test -->\n     <g transform=\"translate(218.803125 85.248437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pfa41dcddf5\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"55.778125\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qhdGLX93O5Oy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "091b2bf6-70e5-488d-8789-9b779fbfd8ca"
      },
      "source": [
        "fit_and_plot(wd=3)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tensorflow.python.keras.engine.sequential.Sequential at 0x7f270817e198>,\n",
              " <tensorflow.python.keras.callbacks.History at 0x7f270817eeb8>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 252x180 with 1 Axes>"
            ],
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Created with matplotlib (https://matplotlib.org/) -->\n<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 254.544602 180.65625\" width=\"254.544602pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n <defs>\n  <style type=\"text/css\">\n*{stroke-linecap:butt;stroke-linejoin:round;}\n  </style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 254.544602 180.65625 \nL 254.544602 0 \nL 0 0 \nz\n\" style=\"fill:none;\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 51.378125 143.1 \nL 246.678125 143.1 \nL 246.678125 7.2 \nL 51.378125 7.2 \nz\n\" style=\"fill:#ffffff;\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <defs>\n       <path d=\"M 0 0 \nL 0 3.5 \n\" id=\"m80ede9e441\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"58.462009\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- 0 -->\n      <defs>\n       <path d=\"M 31.78125 66.40625 \nQ 24.171875 66.40625 20.328125 58.90625 \nQ 16.5 51.421875 16.5 36.375 \nQ 16.5 21.390625 20.328125 13.890625 \nQ 24.171875 6.390625 31.78125 6.390625 \nQ 39.453125 6.390625 43.28125 13.890625 \nQ 47.125 21.390625 47.125 36.375 \nQ 47.125 51.421875 43.28125 58.90625 \nQ 39.453125 66.40625 31.78125 66.40625 \nz\nM 31.78125 74.21875 \nQ 44.046875 74.21875 50.515625 64.515625 \nQ 56.984375 54.828125 56.984375 36.375 \nQ 56.984375 17.96875 50.515625 8.265625 \nQ 44.046875 -1.421875 31.78125 -1.421875 \nQ 19.53125 -1.421875 13.0625 8.265625 \nQ 6.59375 17.96875 6.59375 36.375 \nQ 6.59375 54.828125 13.0625 64.515625 \nQ 19.53125 74.21875 31.78125 74.21875 \nz\n\" id=\"DejaVuSans-48\"/>\n      </defs>\n      <g transform=\"translate(55.280759 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_2\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"94.329778\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- 20 -->\n      <defs>\n       <path d=\"M 19.1875 8.296875 \nL 53.609375 8.296875 \nL 53.609375 0 \nL 7.328125 0 \nL 7.328125 8.296875 \nQ 12.9375 14.109375 22.625 23.890625 \nQ 32.328125 33.6875 34.8125 36.53125 \nQ 39.546875 41.84375 41.421875 45.53125 \nQ 43.3125 49.21875 43.3125 52.78125 \nQ 43.3125 58.59375 39.234375 62.25 \nQ 35.15625 65.921875 28.609375 65.921875 \nQ 23.96875 65.921875 18.8125 64.3125 \nQ 13.671875 62.703125 7.8125 59.421875 \nL 7.8125 69.390625 \nQ 13.765625 71.78125 18.9375 73 \nQ 24.125 74.21875 28.421875 74.21875 \nQ 39.75 74.21875 46.484375 68.546875 \nQ 53.21875 62.890625 53.21875 53.421875 \nQ 53.21875 48.921875 51.53125 44.890625 \nQ 49.859375 40.875 45.40625 35.40625 \nQ 44.1875 33.984375 37.640625 27.21875 \nQ 31.109375 20.453125 19.1875 8.296875 \nz\n\" id=\"DejaVuSans-50\"/>\n      </defs>\n      <g transform=\"translate(87.967278 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-50\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_3\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"130.197546\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- 40 -->\n      <defs>\n       <path d=\"M 37.796875 64.3125 \nL 12.890625 25.390625 \nL 37.796875 25.390625 \nz\nM 35.203125 72.90625 \nL 47.609375 72.90625 \nL 47.609375 25.390625 \nL 58.015625 25.390625 \nL 58.015625 17.1875 \nL 47.609375 17.1875 \nL 47.609375 0 \nL 37.796875 0 \nL 37.796875 17.1875 \nL 4.890625 17.1875 \nL 4.890625 26.703125 \nz\n\" id=\"DejaVuSans-52\"/>\n      </defs>\n      <g transform=\"translate(123.835046 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-52\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_4\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.065315\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- 60 -->\n      <defs>\n       <path d=\"M 33.015625 40.375 \nQ 26.375 40.375 22.484375 35.828125 \nQ 18.609375 31.296875 18.609375 23.390625 \nQ 18.609375 15.53125 22.484375 10.953125 \nQ 26.375 6.390625 33.015625 6.390625 \nQ 39.65625 6.390625 43.53125 10.953125 \nQ 47.40625 15.53125 47.40625 23.390625 \nQ 47.40625 31.296875 43.53125 35.828125 \nQ 39.65625 40.375 33.015625 40.375 \nz\nM 52.59375 71.296875 \nL 52.59375 62.3125 \nQ 48.875 64.0625 45.09375 64.984375 \nQ 41.3125 65.921875 37.59375 65.921875 \nQ 27.828125 65.921875 22.671875 59.328125 \nQ 17.53125 52.734375 16.796875 39.40625 \nQ 19.671875 43.65625 24.015625 45.921875 \nQ 28.375 48.1875 33.59375 48.1875 \nQ 44.578125 48.1875 50.953125 41.515625 \nQ 57.328125 34.859375 57.328125 23.390625 \nQ 57.328125 12.15625 50.6875 5.359375 \nQ 44.046875 -1.421875 33.015625 -1.421875 \nQ 20.359375 -1.421875 13.671875 8.265625 \nQ 6.984375 17.96875 6.984375 36.375 \nQ 6.984375 53.65625 15.1875 63.9375 \nQ 23.390625 74.21875 37.203125 74.21875 \nQ 40.921875 74.21875 44.703125 73.484375 \nQ 48.484375 72.75 52.59375 71.296875 \nz\n\" id=\"DejaVuSans-54\"/>\n      </defs>\n      <g transform=\"translate(159.702815 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-54\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_5\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"201.933084\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 80 -->\n      <defs>\n       <path d=\"M 31.78125 34.625 \nQ 24.75 34.625 20.71875 30.859375 \nQ 16.703125 27.09375 16.703125 20.515625 \nQ 16.703125 13.921875 20.71875 10.15625 \nQ 24.75 6.390625 31.78125 6.390625 \nQ 38.8125 6.390625 42.859375 10.171875 \nQ 46.921875 13.96875 46.921875 20.515625 \nQ 46.921875 27.09375 42.890625 30.859375 \nQ 38.875 34.625 31.78125 34.625 \nz\nM 21.921875 38.8125 \nQ 15.578125 40.375 12.03125 44.71875 \nQ 8.5 49.078125 8.5 55.328125 \nQ 8.5 64.0625 14.71875 69.140625 \nQ 20.953125 74.21875 31.78125 74.21875 \nQ 42.671875 74.21875 48.875 69.140625 \nQ 55.078125 64.0625 55.078125 55.328125 \nQ 55.078125 49.078125 51.53125 44.71875 \nQ 48 40.375 41.703125 38.8125 \nQ 48.828125 37.15625 52.796875 32.3125 \nQ 56.78125 27.484375 56.78125 20.515625 \nQ 56.78125 9.90625 50.3125 4.234375 \nQ 43.84375 -1.421875 31.78125 -1.421875 \nQ 19.734375 -1.421875 13.25 4.234375 \nQ 6.78125 9.90625 6.78125 20.515625 \nQ 6.78125 27.484375 10.78125 32.3125 \nQ 14.796875 37.15625 21.921875 38.8125 \nz\nM 18.3125 54.390625 \nQ 18.3125 48.734375 21.84375 45.5625 \nQ 25.390625 42.390625 31.78125 42.390625 \nQ 38.140625 42.390625 41.71875 45.5625 \nQ 45.3125 48.734375 45.3125 54.390625 \nQ 45.3125 60.0625 41.71875 63.234375 \nQ 38.140625 66.40625 31.78125 66.40625 \nQ 25.390625 66.40625 21.84375 63.234375 \nQ 18.3125 60.0625 18.3125 54.390625 \nz\n\" id=\"DejaVuSans-56\"/>\n      </defs>\n      <g transform=\"translate(195.570584 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-56\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_6\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"237.800852\" xlink:href=\"#m80ede9e441\" y=\"143.1\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 100 -->\n      <defs>\n       <path d=\"M 12.40625 8.296875 \nL 28.515625 8.296875 \nL 28.515625 63.921875 \nL 10.984375 60.40625 \nL 10.984375 69.390625 \nL 28.421875 72.90625 \nL 38.28125 72.90625 \nL 38.28125 8.296875 \nL 54.390625 8.296875 \nL 54.390625 0 \nL 12.40625 0 \nz\n\" id=\"DejaVuSans-49\"/>\n      </defs>\n      <g transform=\"translate(228.257102 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-49\"/>\n       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_7\">\n     <!-- epochs -->\n     <defs>\n      <path d=\"M 56.203125 29.59375 \nL 56.203125 25.203125 \nL 14.890625 25.203125 \nQ 15.484375 15.921875 20.484375 11.0625 \nQ 25.484375 6.203125 34.421875 6.203125 \nQ 39.59375 6.203125 44.453125 7.46875 \nQ 49.3125 8.734375 54.109375 11.28125 \nL 54.109375 2.78125 \nQ 49.265625 0.734375 44.1875 -0.34375 \nQ 39.109375 -1.421875 33.890625 -1.421875 \nQ 20.796875 -1.421875 13.15625 6.1875 \nQ 5.515625 13.8125 5.515625 26.8125 \nQ 5.515625 40.234375 12.765625 48.109375 \nQ 20.015625 56 32.328125 56 \nQ 43.359375 56 49.78125 48.890625 \nQ 56.203125 41.796875 56.203125 29.59375 \nz\nM 47.21875 32.234375 \nQ 47.125 39.59375 43.09375 43.984375 \nQ 39.0625 48.390625 32.421875 48.390625 \nQ 24.90625 48.390625 20.390625 44.140625 \nQ 15.875 39.890625 15.1875 32.171875 \nz\n\" id=\"DejaVuSans-101\"/>\n      <path d=\"M 18.109375 8.203125 \nL 18.109375 -20.796875 \nL 9.078125 -20.796875 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.390625 \nQ 20.953125 51.265625 25.265625 53.625 \nQ 29.59375 56 35.59375 56 \nQ 45.5625 56 51.78125 48.09375 \nQ 58.015625 40.1875 58.015625 27.296875 \nQ 58.015625 14.40625 51.78125 6.484375 \nQ 45.5625 -1.421875 35.59375 -1.421875 \nQ 29.59375 -1.421875 25.265625 0.953125 \nQ 20.953125 3.328125 18.109375 8.203125 \nz\nM 48.6875 27.296875 \nQ 48.6875 37.203125 44.609375 42.84375 \nQ 40.53125 48.484375 33.40625 48.484375 \nQ 26.265625 48.484375 22.1875 42.84375 \nQ 18.109375 37.203125 18.109375 27.296875 \nQ 18.109375 17.390625 22.1875 11.75 \nQ 26.265625 6.109375 33.40625 6.109375 \nQ 40.53125 6.109375 44.609375 11.75 \nQ 48.6875 17.390625 48.6875 27.296875 \nz\n\" id=\"DejaVuSans-112\"/>\n      <path d=\"M 30.609375 48.390625 \nQ 23.390625 48.390625 19.1875 42.75 \nQ 14.984375 37.109375 14.984375 27.296875 \nQ 14.984375 17.484375 19.15625 11.84375 \nQ 23.34375 6.203125 30.609375 6.203125 \nQ 37.796875 6.203125 41.984375 11.859375 \nQ 46.1875 17.53125 46.1875 27.296875 \nQ 46.1875 37.015625 41.984375 42.703125 \nQ 37.796875 48.390625 30.609375 48.390625 \nz\nM 30.609375 56 \nQ 42.328125 56 49.015625 48.375 \nQ 55.71875 40.765625 55.71875 27.296875 \nQ 55.71875 13.875 49.015625 6.21875 \nQ 42.328125 -1.421875 30.609375 -1.421875 \nQ 18.84375 -1.421875 12.171875 6.21875 \nQ 5.515625 13.875 5.515625 27.296875 \nQ 5.515625 40.765625 12.171875 48.375 \nQ 18.84375 56 30.609375 56 \nz\n\" id=\"DejaVuSans-111\"/>\n      <path d=\"M 48.78125 52.59375 \nL 48.78125 44.1875 \nQ 44.96875 46.296875 41.140625 47.34375 \nQ 37.3125 48.390625 33.40625 48.390625 \nQ 24.65625 48.390625 19.8125 42.84375 \nQ 14.984375 37.3125 14.984375 27.296875 \nQ 14.984375 17.28125 19.8125 11.734375 \nQ 24.65625 6.203125 33.40625 6.203125 \nQ 37.3125 6.203125 41.140625 7.25 \nQ 44.96875 8.296875 48.78125 10.40625 \nL 48.78125 2.09375 \nQ 45.015625 0.34375 40.984375 -0.53125 \nQ 36.96875 -1.421875 32.421875 -1.421875 \nQ 20.0625 -1.421875 12.78125 6.34375 \nQ 5.515625 14.109375 5.515625 27.296875 \nQ 5.515625 40.671875 12.859375 48.328125 \nQ 20.21875 56 33.015625 56 \nQ 37.15625 56 41.109375 55.140625 \nQ 45.0625 54.296875 48.78125 52.59375 \nz\n\" id=\"DejaVuSans-99\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 75.984375 \nL 18.109375 75.984375 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-104\"/>\n      <path d=\"M 44.28125 53.078125 \nL 44.28125 44.578125 \nQ 40.484375 46.53125 36.375 47.5 \nQ 32.28125 48.484375 27.875 48.484375 \nQ 21.1875 48.484375 17.84375 46.4375 \nQ 14.5 44.390625 14.5 40.28125 \nQ 14.5 37.15625 16.890625 35.375 \nQ 19.28125 33.59375 26.515625 31.984375 \nL 29.59375 31.296875 \nQ 39.15625 29.25 43.1875 25.515625 \nQ 47.21875 21.78125 47.21875 15.09375 \nQ 47.21875 7.46875 41.1875 3.015625 \nQ 35.15625 -1.421875 24.609375 -1.421875 \nQ 20.21875 -1.421875 15.453125 -0.5625 \nQ 10.6875 0.296875 5.421875 2 \nL 5.421875 11.28125 \nQ 10.40625 8.6875 15.234375 7.390625 \nQ 20.0625 6.109375 24.8125 6.109375 \nQ 31.15625 6.109375 34.5625 8.28125 \nQ 37.984375 10.453125 37.984375 14.40625 \nQ 37.984375 18.0625 35.515625 20.015625 \nQ 33.0625 21.96875 24.703125 23.78125 \nL 21.578125 24.515625 \nQ 13.234375 26.265625 9.515625 29.90625 \nQ 5.8125 33.546875 5.8125 39.890625 \nQ 5.8125 47.609375 11.28125 51.796875 \nQ 16.75 56 26.8125 56 \nQ 31.78125 56 36.171875 55.265625 \nQ 40.578125 54.546875 44.28125 53.078125 \nz\n\" id=\"DejaVuSans-115\"/>\n     </defs>\n     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n      <use x=\"304.541016\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_7\">\n      <defs>\n       <path d=\"M 0 0 \nL -3.5 0 \n\" id=\"me5d8b52a22\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#me5d8b52a22\" y=\"121.392264\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- $\\mathdefault{10^{-2}}$ -->\n      <defs>\n       <path d=\"M 10.59375 35.5 \nL 73.1875 35.5 \nL 73.1875 27.203125 \nL 10.59375 27.203125 \nz\n\" id=\"DejaVuSans-8722\"/>\n      </defs>\n      <g transform=\"translate(20.878125 125.191483)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-50\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_8\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#me5d8b52a22\" y=\"81.978491\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- $\\mathdefault{10^{-1}}$ -->\n      <g transform=\"translate(20.878125 85.77771)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.684375)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.684375)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-8722\"/>\n       <use transform=\"translate(186.855469 38.965625)scale(0.7)\" xlink:href=\"#DejaVuSans-49\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_9\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.378125\" xlink:href=\"#me5d8b52a22\" y=\"42.564718\"/>\n      </g>\n     </g>\n     <g id=\"text_10\">\n      <!-- $\\mathdefault{10^{0}}$ -->\n      <g transform=\"translate(26.778125 46.363937)scale(0.1 -0.1)\">\n       <use transform=\"translate(0 0.765625)\" xlink:href=\"#DejaVuSans-49\"/>\n       <use transform=\"translate(63.623047 0.765625)\" xlink:href=\"#DejaVuSans-48\"/>\n       <use transform=\"translate(128.203125 39.046875)scale(0.7)\" xlink:href=\"#DejaVuSans-48\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_10\">\n      <defs>\n       <path d=\"M 0 0 \nL -2 0 \n\" id=\"m1c705b2404\" style=\"stroke:#000000;stroke-width:0.6;\"/>\n      </defs>\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"142.000889\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_11\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"137.076582\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_6\">\n     <g id=\"line2d_12\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"133.256992\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_7\">\n     <g id=\"line2d_13\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"130.136161\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_8\">\n     <g id=\"line2d_14\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"127.497535\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_9\">\n     <g id=\"line2d_15\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"125.211854\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_10\">\n     <g id=\"line2d_16\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"123.19574\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_11\">\n     <g id=\"line2d_17\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"109.527536\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_12\">\n     <g id=\"line2d_18\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"102.587115\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_13\">\n     <g id=\"line2d_19\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"97.662808\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_14\">\n     <g id=\"line2d_20\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"93.843219\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_15\">\n     <g id=\"line2d_21\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"90.722387\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_16\">\n     <g id=\"line2d_22\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"88.083762\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_17\">\n     <g id=\"line2d_23\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"85.79808\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_18\">\n     <g id=\"line2d_24\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"83.781966\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_19\">\n     <g id=\"line2d_25\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"70.113763\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_20\">\n     <g id=\"line2d_26\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"63.173342\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_21\">\n     <g id=\"line2d_27\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"58.249035\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_22\">\n     <g id=\"line2d_28\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"54.429446\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_23\">\n     <g id=\"line2d_29\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"51.308614\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_24\">\n     <g id=\"line2d_30\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"48.669989\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_25\">\n     <g id=\"line2d_31\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"46.384307\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_26\">\n     <g id=\"line2d_32\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"44.368193\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_27\">\n     <g id=\"line2d_33\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"30.69999\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_28\">\n     <g id=\"line2d_34\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"23.759569\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_29\">\n     <g id=\"line2d_35\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"18.835262\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_30\">\n     <g id=\"line2d_36\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"15.015673\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_31\">\n     <g id=\"line2d_37\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"11.894841\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_32\">\n     <g id=\"line2d_38\">\n      <g>\n       <use style=\"stroke:#000000;stroke-width:0.6;\" x=\"51.378125\" xlink:href=\"#m1c705b2404\" y=\"9.256215\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_11\">\n     <!-- loss -->\n     <defs>\n      <path d=\"M 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 0 \nL 9.421875 0 \nz\n\" id=\"DejaVuSans-108\"/>\n     </defs>\n     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-108\"/>\n      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-111\"/>\n      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-115\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_39\">\n    <path clip-path=\"url(#pe5bb3dccd5)\" d=\"M 60.255398 13.377273 \nL 62.048786 30.180471 \nL 63.842175 43.409994 \nL 65.635563 55.851686 \nL 67.428951 68.066877 \nL 69.22234 80.050452 \nL 71.015728 91.607571 \nL 72.809117 102.378359 \nL 74.602505 111.855803 \nL 76.395894 119.495922 \nL 78.189282 125.012482 \nL 79.98267 128.585239 \nL 81.776059 130.723639 \nL 83.569447 131.963693 \nL 85.362836 132.696838 \nL 87.156224 133.159188 \nL 88.949613 133.479499 \nL 90.743001 133.724453 \nL 92.536389 133.92778 \nL 94.329778 134.106395 \nL 96.123166 134.268815 \nL 97.916555 134.419402 \nL 99.709943 134.560467 \nL 101.503332 134.693309 \nL 103.29672 134.818738 \nL 105.090108 134.937303 \nL 106.883497 135.049434 \nL 108.676885 135.155489 \nL 110.470274 135.255789 \nL 112.263662 135.35063 \nL 114.057051 135.440288 \nL 115.850439 135.525033 \nL 117.643827 135.605114 \nL 119.437216 135.680771 \nL 121.230604 135.752234 \nL 123.023993 135.819726 \nL 124.817381 135.883455 \nL 126.61077 135.943624 \nL 128.404158 136.00042 \nL 130.197546 136.054026 \nL 131.990935 136.104616 \nL 133.784323 136.152357 \nL 135.577712 136.197401 \nL 137.3711 136.239898 \nL 139.164489 136.279991 \nL 140.957877 136.317809 \nL 142.751265 136.353484 \nL 144.544654 136.387132 \nL 146.338042 136.418873 \nL 148.131431 136.448807 \nL 149.924819 136.477039 \nL 151.718208 136.503667 \nL 153.511596 136.528781 \nL 155.304985 136.552464 \nL 157.098373 136.574804 \nL 158.891761 136.59587 \nL 160.68515 136.615745 \nL 162.478538 136.634485 \nL 164.271927 136.652166 \nL 166.065315 136.668843 \nL 167.858704 136.684573 \nL 169.652092 136.699416 \nL 171.44548 136.713416 \nL 173.238869 136.726626 \nL 175.032257 136.739089 \nL 176.825646 136.750852 \nL 178.619034 136.76195 \nL 180.412423 136.772428 \nL 182.205811 136.782312 \nL 183.999199 136.791649 \nL 185.792588 136.800459 \nL 187.585976 136.808777 \nL 189.379365 136.816636 \nL 191.172753 136.824051 \nL 192.966142 136.831058 \nL 194.75953 136.837676 \nL 196.552918 136.843928 \nL 198.346307 136.849835 \nL 200.139695 136.855422 \nL 201.933084 136.860694 \nL 203.726472 136.865681 \nL 205.519861 136.870393 \nL 207.313249 136.874855 \nL 209.106637 136.879068 \nL 210.900026 136.883056 \nL 212.693414 136.886827 \nL 214.486803 136.890397 \nL 216.280191 136.893771 \nL 218.07358 136.896967 \nL 219.866968 136.899994 \nL 221.660356 136.902857 \nL 223.453745 136.905571 \nL 225.247133 136.908138 \nL 227.040522 136.910573 \nL 228.83391 136.912877 \nL 230.627299 136.915063 \nL 232.420687 136.917136 \nL 234.214075 136.919099 \nL 236.007464 136.92096 \nL 237.800852 136.922727 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"line2d_40\">\n    <path clip-path=\"url(#pe5bb3dccd5)\" d=\"M 60.255398 20.987413 \nL 62.048786 33.330142 \nL 63.842175 45.567731 \nL 65.635563 57.490165 \nL 67.428951 68.855732 \nL 69.22234 79.27994 \nL 71.015728 88.179536 \nL 72.809117 95.042243 \nL 74.602505 99.734794 \nL 76.395894 102.609409 \nL 78.189282 104.244412 \nL 79.98267 105.149651 \nL 81.776059 105.661693 \nL 83.569447 105.971224 \nL 85.362836 106.177922 \nL 87.156224 106.331757 \nL 88.949613 106.457234 \nL 90.743001 106.566239 \nL 92.536389 106.664532 \nL 94.329778 106.754928 \nL 96.123166 106.838849 \nL 97.916555 106.91707 \nL 99.709943 106.990066 \nL 101.503332 107.058176 \nL 103.29672 107.121681 \nL 105.090108 107.180833 \nL 106.883497 107.235872 \nL 108.676885 107.287027 \nL 110.470274 107.334519 \nL 112.263662 107.378563 \nL 114.057051 107.419365 \nL 115.850439 107.457122 \nL 117.643827 107.492023 \nL 119.437216 107.524247 \nL 121.230604 107.553962 \nL 123.023993 107.581332 \nL 124.817381 107.606508 \nL 126.61077 107.629635 \nL 128.404158 107.650848 \nL 130.197546 107.670275 \nL 131.990935 107.688038 \nL 133.784323 107.704249 \nL 135.577712 107.719014 \nL 137.3711 107.732435 \nL 139.164489 107.744604 \nL 140.957877 107.75561 \nL 142.751265 107.765536 \nL 144.544654 107.774459 \nL 146.338042 107.782452 \nL 148.131431 107.789583 \nL 149.924819 107.795915 \nL 151.718208 107.801507 \nL 153.511596 107.806416 \nL 155.304985 107.810694 \nL 157.098373 107.814389 \nL 158.891761 107.817545 \nL 160.68515 107.820207 \nL 162.478538 107.822413 \nL 164.271927 107.8242 \nL 166.065315 107.825603 \nL 167.858704 107.826652 \nL 169.652092 107.827379 \nL 171.44548 107.82781 \nL 173.238869 107.827973 \nL 175.032257 107.82789 \nL 176.825646 107.827584 \nL 178.619034 107.827077 \nL 180.412423 107.826386 \nL 182.205811 107.825529 \nL 183.999199 107.824526 \nL 185.792588 107.823387 \nL 187.585976 107.82213 \nL 189.379365 107.820768 \nL 191.172753 107.819312 \nL 192.966142 107.817773 \nL 194.75953 107.816162 \nL 196.552918 107.814489 \nL 198.346307 107.812763 \nL 200.139695 107.810991 \nL 201.933084 107.809181 \nL 203.726472 107.80734 \nL 205.519861 107.805474 \nL 207.313249 107.803589 \nL 209.106637 107.801691 \nL 210.900026 107.799785 \nL 212.693414 107.797873 \nL 214.486803 107.795962 \nL 216.280191 107.794054 \nL 218.07358 107.792153 \nL 219.866968 107.790261 \nL 221.660356 107.788382 \nL 223.453745 107.786518 \nL 225.247133 107.784671 \nL 227.040522 107.782843 \nL 228.83391 107.781036 \nL 230.627299 107.779251 \nL 232.420687 107.777491 \nL 234.214075 107.775755 \nL 236.007464 107.774045 \nL 237.800852 107.772362 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 51.378125 143.1 \nL 51.378125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 246.678125 143.1 \nL 246.678125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 51.378125 143.1 \nL 246.678125 143.1 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 51.378125 7.2 \nL 246.678125 7.2 \n\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n   </g>\n   <g id=\"legend_1\">\n    <g id=\"patch_7\">\n     <path d=\"M 184.403125 44.55625 \nL 239.678125 44.55625 \nQ 241.678125 44.55625 241.678125 42.55625 \nL 241.678125 14.2 \nQ 241.678125 12.2 239.678125 12.2 \nL 184.403125 12.2 \nQ 182.403125 12.2 182.403125 14.2 \nL 182.403125 42.55625 \nQ 182.403125 44.55625 184.403125 44.55625 \nz\n\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n    </g>\n    <g id=\"line2d_41\">\n     <path d=\"M 186.403125 20.298437 \nL 206.403125 20.298437 \n\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_42\"/>\n    <g id=\"text_12\">\n     <!-- train -->\n     <defs>\n      <path d=\"M 18.3125 70.21875 \nL 18.3125 54.6875 \nL 36.8125 54.6875 \nL 36.8125 47.703125 \nL 18.3125 47.703125 \nL 18.3125 18.015625 \nQ 18.3125 11.328125 20.140625 9.421875 \nQ 21.96875 7.515625 27.59375 7.515625 \nL 36.8125 7.515625 \nL 36.8125 0 \nL 27.59375 0 \nQ 17.1875 0 13.234375 3.875 \nQ 9.28125 7.765625 9.28125 18.015625 \nL 9.28125 47.703125 \nL 2.6875 47.703125 \nL 2.6875 54.6875 \nL 9.28125 54.6875 \nL 9.28125 70.21875 \nz\n\" id=\"DejaVuSans-116\"/>\n      <path d=\"M 41.109375 46.296875 \nQ 39.59375 47.171875 37.8125 47.578125 \nQ 36.03125 48 33.890625 48 \nQ 26.265625 48 22.1875 43.046875 \nQ 18.109375 38.09375 18.109375 28.8125 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 20.953125 51.171875 25.484375 53.578125 \nQ 30.03125 56 36.53125 56 \nQ 37.453125 56 38.578125 55.875 \nQ 39.703125 55.765625 41.0625 55.515625 \nz\n\" id=\"DejaVuSans-114\"/>\n      <path d=\"M 34.28125 27.484375 \nQ 23.390625 27.484375 19.1875 25 \nQ 14.984375 22.515625 14.984375 16.5 \nQ 14.984375 11.71875 18.140625 8.90625 \nQ 21.296875 6.109375 26.703125 6.109375 \nQ 34.1875 6.109375 38.703125 11.40625 \nQ 43.21875 16.703125 43.21875 25.484375 \nL 43.21875 27.484375 \nz\nM 52.203125 31.203125 \nL 52.203125 0 \nL 43.21875 0 \nL 43.21875 8.296875 \nQ 40.140625 3.328125 35.546875 0.953125 \nQ 30.953125 -1.421875 24.3125 -1.421875 \nQ 15.921875 -1.421875 10.953125 3.296875 \nQ 6 8.015625 6 15.921875 \nQ 6 25.140625 12.171875 29.828125 \nQ 18.359375 34.515625 30.609375 34.515625 \nL 43.21875 34.515625 \nL 43.21875 35.40625 \nQ 43.21875 41.609375 39.140625 45 \nQ 35.0625 48.390625 27.6875 48.390625 \nQ 23 48.390625 18.546875 47.265625 \nQ 14.109375 46.140625 10.015625 43.890625 \nL 10.015625 52.203125 \nQ 14.9375 54.109375 19.578125 55.046875 \nQ 24.21875 56 28.609375 56 \nQ 40.484375 56 46.34375 49.84375 \nQ 52.203125 43.703125 52.203125 31.203125 \nz\n\" id=\"DejaVuSans-97\"/>\n      <path d=\"M 9.421875 54.6875 \nL 18.40625 54.6875 \nL 18.40625 0 \nL 9.421875 0 \nz\nM 9.421875 75.984375 \nL 18.40625 75.984375 \nL 18.40625 64.59375 \nL 9.421875 64.59375 \nz\n\" id=\"DejaVuSans-105\"/>\n      <path d=\"M 54.890625 33.015625 \nL 54.890625 0 \nL 45.90625 0 \nL 45.90625 32.71875 \nQ 45.90625 40.484375 42.875 44.328125 \nQ 39.84375 48.1875 33.796875 48.1875 \nQ 26.515625 48.1875 22.3125 43.546875 \nQ 18.109375 38.921875 18.109375 30.90625 \nL 18.109375 0 \nL 9.078125 0 \nL 9.078125 54.6875 \nL 18.109375 54.6875 \nL 18.109375 46.1875 \nQ 21.34375 51.125 25.703125 53.5625 \nQ 30.078125 56 35.796875 56 \nQ 45.21875 56 50.046875 50.171875 \nQ 54.890625 44.34375 54.890625 33.015625 \nz\n\" id=\"DejaVuSans-110\"/>\n     </defs>\n     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n     </g>\n    </g>\n    <g id=\"line2d_43\">\n     <path d=\"M 186.403125 34.976562 \nL 206.403125 34.976562 \n\" style=\"fill:none;stroke:#ff7f0e;stroke-dasharray:1.5,2.475;stroke-dashoffset:0;stroke-width:1.5;\"/>\n    </g>\n    <g id=\"line2d_44\"/>\n    <g id=\"text_13\">\n     <!-- test -->\n     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\n      <use xlink:href=\"#DejaVuSans-116\"/>\n      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\n      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\n      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\n     </g>\n    </g>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pe5bb3dccd5\">\n   <rect height=\"135.9\" width=\"195.3\" x=\"51.378125\" y=\"7.2\"/>\n  </clipPath>\n </defs>\n</svg>\n"
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ehz85749Lyq_",
        "colab_type": "text"
      },
      "source": [
        "###3.12.5. 小结\n",
        "* 正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。\n",
        "* 权重衰减等价于 $L_2$ 范数正则化，通常会使学到的权重参数的元素较接近0。\n",
        "* 权重衰减可以在添加神经网络层(如Dense)时通过kernel_regularizer超参数来指定。\n",
        "* 可以定义不同的优化器Optimizer实例对不同的模型参数使用不同的迭代方法。"
      ]
    }
  ]
}