nn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nn.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dair-ai/pytorch_notebooks/blob/master/nn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ee4B4v5tAp1C",
        "colab_type": "text"
      },
      "source": [
        "## A Simple Neural Network from Scratch with PyTorch and Google Colab"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4cEhtf_Ap1E",
        "colab_type": "text"
      },
      "source": [
        "In this tutorial we will implement a simple neural network from scratch using PyTorch. The idea of the tutorial is to teach you the basics of PyTorch and how it can be used to implement a neural network from scratch. I will go over some of the basic functionalities and concepts available in PyTorch that will allow you to build your own neural networks. \n",
        "\n",
        "This tutorial assumes you have prior knowledge of how a neural network works. Don’t worry! Even if you are not so sure, you will be okay. For advanced PyTorch users, this tutorial may still serve as a refresher. This tutorial is heavily inspired by this [Neural Network implementation](https://repl.it/talk/announcements/Build-a-Neural-Network-in-Python/5457) coded purely using Numpy. In fact, I tried re-implementing the code using PyTorch instead and added my own intuitions and explanations. Thanks to [Samay](https://repl.it/@shamdasani) for his phenomenal work, I hope this inspires many others as it did with me.\n",
        "\n",
        "Since we are working on Google Colab, we will need to install the PyTorch library. You can do this by using the following command:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SpBiWQF2BrJK",
        "colab_type": "code",
        "outputId": "858d4853-ca0d-4d5b-f61c-10481b46f309",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        }
      },
      "source": [
        "!pip3 install torch torchvision"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n",
            "\u001b[K    100% |████████████████████████████████| 591.8MB 26kB/s \n",
            "tcmalloc: large alloc 1073750016 bytes == 0x61f82000 @  0x7f400bb202a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n",
            "\u001b[?25hCollecting torchvision\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 23.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n",
            "Collecting pillow>=4.1.1 (from torchvision)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/92/e3/217dfd0834a51418c602c96b110059c477260c7fee898542b100913947cf/Pillow-5.4.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 6.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n",
            "Installing collected packages: torch, pillow, torchvision\n",
            "  Found existing installation: Pillow 4.0.0\n",
            "    Uninstalling Pillow-4.0.0:\n",
            "      Successfully uninstalled Pillow-4.0.0\n",
            "Successfully installed pillow-5.4.0 torch-1.0.0 torchvision-0.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MP9ewMSlC7JU",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "The `torch` module provides all the necessary **tensor** operators you will need to implement your first neural network from scratch in PyTorch. That's right! In PyTorch everything is a Tensor, so this is the first thing you will need to get used to."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bKmXKSQnAp1G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1EWBBl1nAp1M",
        "colab_type": "text"
      },
      "source": [
        "## Data\n",
        "Let's start by creating some sample data using the `torch.tensor` command. In Numpy, this could be done with `np.array`. Both functions serve the same purpose, but in PyTorch everything is a Tensor as opposed to a vector or matrix. We define types in PyTorch using the `dtype=torch.xxx` command. \n",
        "\n",
        "In the data below, `X` represents the amount of hours studied and how much time students spent sleeping, whereas `y` represent grades. The variable `xPredicted` is a single input for which we want to predict a grade using the parameters learned by the neural network. Remember, the neural network wants to learn a mapping between `X` and `y`, so it will try to take a guess from what it has learned from the training data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fsAVbHnjAp1P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = torch.tensor(([2, 9], [1, 5], [3, 6]), dtype=torch.float) # 3 X 2 tensor\n",
        "y = torch.tensor(([92], [100], [89]), dtype=torch.float) # 3 X 1 tensor\n",
        "xPredicted = torch.tensor(([4, 8]), dtype=torch.float) # 1 X 2 tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RC0ru9kCAp1U",
        "colab_type": "text"
      },
      "source": [
        "You can check the size of the tensors we have just created with the `size` command. This is equivalent to the `shape` command used in tools such as Numpy and Tensorflow. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sfC-B1BEAp1W",
        "colab_type": "code",
        "outputId": "d2ec7994-41ad-41fa-a69c-c0b7123ef7cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "print(X.size())\n",
        "print(y.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([3, 2])\n",
            "torch.Size([3, 1])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zrND9MS9Ap1f",
        "colab_type": "text"
      },
      "source": [
        "## Scaling\n",
        "\n",
        "Below we are performing some scaling on the sample data. Notice that the `max` function returns both a tensor and the corresponding indices. So we use `_` to capture the indices which we won't use here because we are only interested in the max values to conduct the scaling. Perfect! Our data is now in a very nice format our neural network will appreciate later on. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hlBvtfAmAp1i",
        "colab_type": "code",
        "outputId": "23e1d24b-fa29-4173-f884-f44bc8a48cea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# scale units\n",
        "X_max, _ = torch.max(X, 0)\n",
        "xPredicted_max, _ = torch.max(xPredicted, 0)\n",
        "\n",
        "X = torch.div(X, X_max)\n",
        "xPredicted = torch.div(xPredicted, xPredicted_max)\n",
        "y = y / 100  # max test score is 100\n",
        "print(xPredicted)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([0.5000, 1.0000])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R1kTs5S5Ap1m",
        "colab_type": "text"
      },
      "source": [
        "Notice that there are two functions `max` and `div` that I didn't discuss above. They do exactly what they imply: `max` finds the maximum value in a vector... I mean tensor; and `div` is basically a nice little function to divide two tensors. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xRvMSpEFAp1n",
        "colab_type": "text"
      },
      "source": [
        "## Model (Computation Graph)\n",
        "Once the data has been processed and it is in the proper format, all you need to do now is to define your model. Here is where things begin to change a little as compared to how you would build your neural networks using, say, something like Keras or Tensorflow. However, you will realize quickly as you go along that PyTorch doesn't differ much from other deep learning tools. At the end of the day we are constructing a computation graph, which is used to dictate how data should flow and what type of operations are performed on this information. \n",
        "\n",
        "For illustration purposes, we are building the following neural network or computation graph:\n",
        "\n",
        "\n",
        "![alt text](https://drive.google.com/uc?export=view&id=1l-sKpcCJCEUJV1BlAqcVAvLXLpYCInV6)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7pDC5SfAp1p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Neural_Network(nn.Module):\n",
        "    def __init__(self, ):\n",
        "        super(Neural_Network, self).__init__()\n",
        "        # parameters\n",
        "        # TODO: parameters can be parameterized instead of declaring them here\n",
        "        self.inputSize = 2\n",
        "        self.outputSize = 1\n",
        "        self.hiddenSize = 3\n",
        "        \n",
        "        # weights\n",
        "        self.W1 = torch.randn(self.inputSize, self.hiddenSize) # 3 X 2 tensor\n",
        "        self.W2 = torch.randn(self.hiddenSize, self.outputSize) # 3 X 1 tensor\n",
        "        \n",
        "    def forward(self, X):\n",
        "        self.z = torch.matmul(X, self.W1) # 3 X 3 \".dot\" does not broadcast in PyTorch\n",
        "        self.z2 = self.sigmoid(self.z) # activation function\n",
        "        self.z3 = torch.matmul(self.z2, self.W2)\n",
        "        o = self.sigmoid(self.z3) # final activation function\n",
        "        return o\n",
        "        \n",
        "    def sigmoid(self, s):\n",
        "        return 1 / (1 + torch.exp(-s))\n",
        "    \n",
        "    def sigmoidPrime(self, s):\n",
        "        # derivative of sigmoid\n",
        "        return s * (1 - s)\n",
        "    \n",
        "    def backward(self, X, y, o):\n",
        "        self.o_error = y - o # error in output\n",
        "        self.o_delta = self.o_error * self.sigmoidPrime(o) # derivative of sig to error\n",
        "        self.z2_error = torch.matmul(self.o_delta, torch.t(self.W2))\n",
        "        self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)\n",
        "        self.W1 += torch.matmul(torch.t(X), self.z2_delta)\n",
        "        self.W2 += torch.matmul(torch.t(self.z2), self.o_delta)\n",
        "        \n",
        "    def train(self, X, y):\n",
        "        # forward + backward pass for training\n",
        "        o = self.forward(X)\n",
        "        self.backward(X, y, o)\n",
        "        \n",
        "    def saveWeights(self, model):\n",
        "        # we will use the PyTorch internal storage functions\n",
        "        torch.save(model, \"NN\")\n",
        "        # you can reload model with all the weights and so forth with:\n",
        "        # torch.load(\"NN\")\n",
        "        \n",
        "    def predict(self):\n",
        "        print (\"Predicted data based on trained weights: \")\n",
        "        print (\"Input (scaled): \\n\" + str(xPredicted))\n",
        "        print (\"Output: \\n\" + str(self.forward(xPredicted)))\n",
        "        "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qm5gimnyAp1s",
        "colab_type": "text"
      },
      "source": [
        "For the purpose of this tutorial, we are not going to be talking math stuff, that's for another day. I just want you to get a gist of what it takes to build a neural network from scratch using PyTorch. Let's break down the model which was declared via the class above. \n",
        "\n",
        "## Class Header\n",
        "First, we defined our model via a class because that is the recommended way to build the computation graph. The class header contains the name of the class `Neural Network` and the parameter `nn.Module` which basically indicates that we are defining our own neural network. \n",
        "\n",
        "```python\n",
        "class Neural_Network(nn.Module):\n",
        "```\n",
        "\n",
        "## Initialization\n",
        "The next step is to define the initializations ( `def __init__(self,)`) that will be performed upon creating an instance of the customized neural network. You can declare the parameters of your model here, but typically, you would declare the structure of your network in this section -- the size of the hidden layers and so forth. Since we are building the neural network from scratch, we explicitly declared the size of the weights matrices: one that stores the parameters from the input to hidden layer; and one that stores the parameter from the hidden to output layer. Both weight matrices are initialized with values randomly chosen from a normal distribution via `torch.randn(...)`. Note that we are not using bias just to keep things as simple as possible.  \n",
        "\n",
        "```python\n",
        "def __init__(self, ):\n",
        "    super(Neural_Network, self).__init__()\n",
        "    # parameters\n",
        "    # TODO: parameters can be parameterized instead of declaring them here\n",
        "    self.inputSize = 2\n",
        "    self.outputSize = 1\n",
        "    self.hiddenSize = 3\n",
        "\n",
        "    # weights\n",
        "    self.W1 = torch.randn(self.inputSize, self.hiddenSize) # 3 X 2 tensor\n",
        "    self.W2 = torch.randn(self.hiddenSize, self.outputSize) # 3 X 1 tensor\n",
        "```\n",
        "\n",
        "## The Forward Function\n",
        "The `forward` function is where all the magic happens (see below). This is where the data enters and is fed into the computation graph (i.e., the neural network structure we have built). Since we are building a simple neural network with one hidden layer, our forward function looks very simple:\n",
        "\n",
        "```python\n",
        "def forward(self, X):\n",
        "    self.z = torch.matmul(X, self.W1) \n",
        "    self.z2 = self.sigmoid(self.z) # activation function\n",
        "    self.z3 = torch.matmul(self.z2, self.W2)\n",
        "    o = self.sigmoid(self.z3) # final activation function\n",
        "    return o\n",
        "```\n",
        "\n",
        "The `forward` function above takes the input `X`and then performs a matrix multiplication (`torch.matmul(...)`) with the first weight matrix `self.W1`. Then the result is applied an activation function, `sigmoid`. The resulting matrix of the activation is then multiplied with the second weight matrix `self.W2`. Then another activation if performed, which renders the output of the neural network or computation graph. The process I described above is simply what's known as a `feedforward pass`. In order for the weights to optimize when training, we need a backpropagation algorithm. \n",
        "\n",
        "## The Backward Function\n",
        "The `backward` function contains the backpropagation algorithm, where the goal is to essentially minimize the loss with respect to our weights. In other words, the weights need to be updated in such  a way that the loss decreases while the neural network is training (well, that is what we hope for). All this magic is possible with the gradient descent algorithm which is declared in the `backward` function. Take a minute or two to inspect what is happening in the code below:\n",
        "\n",
        "```python\n",
        "def backward(self, X, y, o):\n",
        "    self.o_error = y - o # error in output\n",
        "    self.o_delta = self.o_error * self.sigmoidPrime(o) \n",
        "    self.z2_error = torch.matmul(self.o_delta, torch.t(self.W2))\n",
        "    self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)\n",
        "    self.W1 += torch.matmul(torch.t(X), self.z2_delta)\n",
        "    self.W2 += torch.matmul(torch.t(self.z2), self.o_delta)\n",
        "```\n",
        "\n",
        "Notice that we are performing a lot of matrix multiplications along with the transpose operations via the `torch.matmul(...)` and `torch.t(...)` operations, respectively. The rest is simply gradient descent -- there is nothing to it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9t26Dr5zAp1u",
        "colab_type": "text"
      },
      "source": [
        "## Training\n",
        "All that is left now is to train the neural network. First we create an instance of the computation graph we have just built:\n",
        "\n",
        "```python\n",
        "NN = Neural_Network()\n",
        "```\n",
        "\n",
        "Then we train the model for `1000` rounds. Notice that in PyTorch `NN(X)` automatically calls the `forward` function so there is no need to explicitly call `NN.forward(X)`. \n",
        "\n",
        "After we have obtained the predicted output for ever round of training, we compute the loss, with the following code:\n",
        "\n",
        "```python\n",
        "torch.mean((y - NN(X))**2).detach().item()\n",
        "```\n",
        "\n",
        "The next step is to start the training (foward + backward) via `NN.train(X, y)`. After we have trained the neural network, we can store the model and output the predicted value of the single instance we declared in the beginning, `xPredicted`.  \n",
        "\n",
        "Let's train!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9sTddOpLAp1w",
        "colab_type": "code",
        "outputId": "a02d2b93-34da-4068-f1f2-843f1e30abf8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17156
        }
      },
      "source": [
        "NN = Neural_Network()\n",
        "for i in range(1000):  # trains the NN 1,000 times\n",
        "    print (\"#\" + str(i) + \" Loss: \" + str(torch.mean((y - NN(X))**2).detach().item()))  # mean sum squared loss\n",
        "    NN.train(X, y)\n",
        "NN.saveWeights(NN)\n",
        "NN.predict()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "#0 Loss: 0.28770461678504944\n",
            "#1 Loss: 0.19437099993228912\n",
            "#2 Loss: 0.129642054438591\n",
            "#3 Loss: 0.08898762613534927\n",
            "#4 Loss: 0.0638350322842598\n",
            "#5 Loss: 0.04783045873045921\n",
            "#6 Loss: 0.037219222635030746\n",
            "#7 Loss: 0.029889358207583427\n",
            "#8 Loss: 0.024637090042233467\n",
            "#9 Loss: 0.020752854645252228\n",
            "#10 Loss: 0.01780204102396965\n",
            "#11 Loss: 0.015508432872593403\n",
            "#12 Loss: 0.013690348714590073\n",
            "#13 Loss: 0.012224685400724411\n",
            "#14 Loss: 0.011025689542293549\n",
            "#15 Loss: 0.0100322300568223\n",
            "#16 Loss: 0.009199750609695911\n",
            "#17 Loss: 0.008495191112160683\n",
            "#18 Loss: 0.007893583737313747\n",
            "#19 Loss: 0.007375772576779127\n",
            "#20 Loss: 0.006926907692104578\n",
            "#21 Loss: 0.006535270716995001\n",
            "#22 Loss: 0.006191555876284838\n",
            "#23 Loss: 0.005888286512345076\n",
            "#24 Loss: 0.005619380157440901\n",
            "#25 Loss: 0.0053798723965883255\n",
            "#26 Loss: 0.005165652371942997\n",
            "#27 Loss: 0.004973314236849546\n",
            "#28 Loss: 0.0048000202514231205\n",
            "#29 Loss: 0.004643348511308432\n",
            "#30 Loss: 0.00450127711519599\n",
            "#31 Loss: 0.004372074268758297\n",
            "#32 Loss: 0.004254247527569532\n",
            "#33 Loss: 0.004146536346524954\n",
            "#34 Loss: 0.004047831054776907\n",
            "#35 Loss: 0.003957169130444527\n",
            "#36 Loss: 0.0038737261202186346\n",
            "#37 Loss: 0.0037967758253216743\n",
            "#38 Loss: 0.0037256714422255754\n",
            "#39 Loss: 0.0036598537117242813\n",
            "#40 Loss: 0.003598827635869384\n",
            "#41 Loss: 0.0035421468783169985\n",
            "#42 Loss: 0.0034894247073680162\n",
            "#43 Loss: 0.003440307453274727\n",
            "#44 Loss: 0.0033944963943213224\n",
            "#45 Loss: 0.003351695602759719\n",
            "#46 Loss: 0.003311669686809182\n",
            "#47 Loss: 0.003274182789027691\n",
            "#48 Loss: 0.0032390293199568987\n",
            "#49 Loss: 0.0032060390803962946\n",
            "#50 Loss: 0.0031750358175486326\n",
            "#51 Loss: 0.0031458677258342505\n",
            "#52 Loss: 0.003118406282737851\n",
            "#53 Loss: 0.0030925225000828505\n",
            "#54 Loss: 0.0030680971685796976\n",
            "#55 Loss: 0.0030450366903096437\n",
            "#56 Loss: 0.003023233264684677\n",
            "#57 Loss: 0.0030026088934391737\n",
            "#58 Loss: 0.002983089303597808\n",
            "#59 Loss: 0.0029645822942256927\n",
            "#60 Loss: 0.00294703827239573\n",
            "#61 Loss: 0.0029303862247616053\n",
            "#62 Loss: 0.002914572134613991\n",
            "#63 Loss: 0.0028995368629693985\n",
            "#64 Loss: 0.0028852447867393494\n",
            "#65 Loss: 0.002871639095246792\n",
            "#66 Loss: 0.002858673455193639\n",
            "#67 Loss: 0.0028463276103138924\n",
            "#68 Loss: 0.0028345445170998573\n",
            "#69 Loss: 0.0028233081102371216\n",
            "#70 Loss: 0.0028125671669840813\n",
            "#71 Loss: 0.002802313072606921\n",
            "#72 Loss: 0.0027925113681703806\n",
            "#73 Loss: 0.002783131552860141\n",
            "#74 Loss: 0.0027741591911762953\n",
            "#75 Loss: 0.00276556215249002\n",
            "#76 Loss: 0.0027573201805353165\n",
            "#77 Loss: 0.002749415347352624\n",
            "#78 Loss: 0.002741842297837138\n",
            "#79 Loss: 0.0027345670387148857\n",
            "#80 Loss: 0.00272758468054235\n",
            "#81 Loss: 0.0027208721730858088\n",
            "#82 Loss: 0.002714422531425953\n",
            "#83 Loss: 0.002708215033635497\n",
            "#84 Loss: 0.0027022461872547865\n",
            "#85 Loss: 0.0026964957360178232\n",
            "#86 Loss: 0.002690958557650447\n",
            "#87 Loss: 0.0026856244076043367\n",
            "#88 Loss: 0.002680474892258644\n",
            "#89 Loss: 0.002675510011613369\n",
            "#90 Loss: 0.002670713933184743\n",
            "#91 Loss: 0.0026660896837711334\n",
            "#92 Loss: 0.0026616165414452553\n",
            "#93 Loss: 0.0026572979986667633\n",
            "#94 Loss: 0.0026531198527663946\n",
            "#95 Loss: 0.002649075584486127\n",
            "#96 Loss: 0.002645164029672742\n",
            "#97 Loss: 0.0026413705199956894\n",
            "#98 Loss: 0.0026377029716968536\n",
            "#99 Loss: 0.002634142292663455\n",
            "#100 Loss: 0.00263069081120193\n",
            "#101 Loss: 0.0026273438706994057\n",
            "#102 Loss: 0.0026240937877446413\n",
            "#103 Loss: 0.0026209382340312004\n",
            "#104 Loss: 0.002617868361994624\n",
            "#105 Loss: 0.002614888595417142\n",
            "#106 Loss: 0.0026119956746697426\n",
            "#107 Loss: 0.002609172137454152\n",
            "#108 Loss: 0.0026064326521009207\n",
            "#109 Loss: 0.002603760687634349\n",
            "#110 Loss: 0.00260116346180439\n",
            "#111 Loss: 0.002598624676465988\n",
            "#112 Loss: 0.0025961531791836023\n",
            "#113 Loss: 0.0025937433820217848\n",
            "#114 Loss: 0.0025913880672305822\n",
            "#115 Loss: 0.0025890925899147987\n",
            "#116 Loss: 0.002586849732324481\n",
            "#117 Loss: 0.002584656234830618\n",
            "#118 Loss: 0.0025825174525380135\n",
            "#119 Loss: 0.0025804194156080484\n",
            "#120 Loss: 0.0025783723685890436\n",
            "#121 Loss: 0.002576368162408471\n",
            "#122 Loss: 0.002574402838945389\n",
            "#123 Loss: 0.002572478959336877\n",
            "#124 Loss: 0.0025705902371555567\n",
            "#125 Loss: 0.0025687431916594505\n",
            "#126 Loss: 0.002566935494542122\n",
            "#127 Loss: 0.0025651559699326754\n",
            "#128 Loss: 0.002563410671427846\n",
            "#129 Loss: 0.0025617002975195646\n",
            "#130 Loss: 0.0025600148364901543\n",
            "#131 Loss: 0.0025583638343960047\n",
            "#132 Loss: 0.002556734485551715\n",
            "#133 Loss: 0.002555140992626548\n",
            "#134 Loss: 0.0025535663589835167\n",
            "#135 Loss: 0.0025520166382193565\n",
            "#136 Loss: 0.002550497418269515\n",
            "#137 Loss: 0.0025489996187388897\n",
            "#138 Loss: 0.002547516720369458\n",
            "#139 Loss: 0.0025460589677095413\n",
            "#140 Loss: 0.0025446258950978518\n",
            "#141 Loss: 0.0025432079564779997\n",
            "#142 Loss: 0.0025418128352612257\n",
            "#143 Loss: 0.0025404333136975765\n",
            "#144 Loss: 0.0025390759110450745\n",
            "#145 Loss: 0.002537728287279606\n",
            "#146 Loss: 0.0025364060420542955\n",
            "#147 Loss: 0.0025350917130708694\n",
            "#148 Loss: 0.002533797873184085\n",
            "#149 Loss: 0.002532513812184334\n",
            "#150 Loss: 0.0025312507059425116\n",
            "#151 Loss: 0.0025300011038780212\n",
            "#152 Loss: 0.0025287508033216\n",
            "#153 Loss: 0.0025275293737649918\n",
            "#154 Loss: 0.002526313764974475\n",
            "#155 Loss: 0.00252510909922421\n",
            "#156 Loss: 0.0025239146780222654\n",
            "#157 Loss: 0.0025227360893040895\n",
            "#158 Loss: 0.002521563321352005\n",
            "#159 Loss: 0.002520401030778885\n",
            "#160 Loss: 0.002519249450415373\n",
            "#161 Loss: 0.0025181034579873085\n",
            "#162 Loss: 0.0025169753935188055\n",
            "#163 Loss: 0.0025158498901873827\n",
            "#164 Loss: 0.0025147362612187862\n",
            "#165 Loss: 0.002513629151508212\n",
            "#166 Loss: 0.002512530190870166\n",
            "#167 Loss: 0.0025114361196756363\n",
            "#168 Loss: 0.0025103483349084854\n",
            "#169 Loss: 0.0025092700961977243\n",
            "#170 Loss: 0.0025081969797611237\n",
            "#171 Loss: 0.0025071338750422\n",
            "#172 Loss: 0.0025060747284442186\n",
            "#173 Loss: 0.0025050221011042595\n",
            "#174 Loss: 0.002503973664715886\n",
            "#175 Loss: 0.002502931747585535\n",
            "#176 Loss: 0.002501895884051919\n",
            "#177 Loss: 0.0025008656084537506\n",
            "#178 Loss: 0.00249984092079103\n",
            "#179 Loss: 0.002498818328604102\n",
            "#180 Loss: 0.002497798763215542\n",
            "#181 Loss: 0.0024967871140688658\n",
            "#182 Loss: 0.00249578058719635\n",
            "#183 Loss: 0.0024947759229689837\n",
            "#184 Loss: 0.0024937766138464212\n",
            "#185 Loss: 0.002492778468877077\n",
            "#186 Loss: 0.0024917826522141695\n",
            "#187 Loss: 0.0024907945189625025\n",
            "#188 Loss: 0.002489812206476927\n",
            "#189 Loss: 0.002488828031346202\n",
            "#190 Loss: 0.0024878503754734993\n",
            "#191 Loss: 0.0024868694599717855\n",
            "#192 Loss: 0.002485897159203887\n",
            "#193 Loss: 0.002484926488250494\n",
            "#194 Loss: 0.0024839574471116066\n",
            "#195 Loss: 0.0024829902686178684\n",
            "#196 Loss: 0.002482031239196658\n",
            "#197 Loss: 0.0024810675531625748\n",
            "#198 Loss: 0.002480114810168743\n",
            "#199 Loss: 0.00247915368527174\n",
            "#200 Loss: 0.0024782009422779083\n",
            "#201 Loss: 0.002477245405316353\n",
            "#202 Loss: 0.0024762984830886126\n",
            "#203 Loss: 0.002475348999723792\n",
            "#204 Loss: 0.002474398585036397\n",
            "#205 Loss: 0.0024734551552683115\n",
            "#206 Loss: 0.002472516382113099\n",
            "#207 Loss: 0.002471569227054715\n",
            "#208 Loss: 0.002470628125593066\n",
            "#209 Loss: 0.0024696916807442904\n",
            "#210 Loss: 0.002468749647960067\n",
            "#211 Loss: 0.0024678176268935204\n",
            "#212 Loss: 0.0024668758269399405\n",
            "#213 Loss: 0.002465949160978198\n",
            "#214 Loss: 0.0024650150444358587\n",
            "#215 Loss: 0.00246407906524837\n",
            "#216 Loss: 0.002463151467964053\n",
            "#217 Loss: 0.002462216652929783\n",
            "#218 Loss: 0.0024612878914922476\n",
            "#219 Loss: 0.002460360061377287\n",
            "#220 Loss: 0.0024594322312623262\n",
            "#221 Loss: 0.0024585050996392965\n",
            "#222 Loss: 0.002457576571032405\n",
            "#223 Loss: 0.0024566520005464554\n",
            "#224 Loss: 0.002455727430060506\n",
            "#225 Loss: 0.002454800298437476\n",
            "#226 Loss: 0.002453884808346629\n",
            "#227 Loss: 0.0024529551155865192\n",
            "#228 Loss: 0.002452034503221512\n",
            "#229 Loss: 0.002451109467074275\n",
            "#230 Loss: 0.0024501883890479803\n",
            "#231 Loss: 0.002449269639328122\n",
            "#232 Loss: 0.0024483499582856894\n",
            "#233 Loss: 0.002447424689307809\n",
            "#234 Loss: 0.0024465022142976522\n",
            "#235 Loss: 0.0024455797392874956\n",
            "#236 Loss: 0.0024446637835353613\n",
            "#237 Loss: 0.002443745033815503\n",
            "#238 Loss: 0.0024428225588053465\n",
            "#239 Loss: 0.0024419049732387066\n",
            "#240 Loss: 0.002440983895212412\n",
            "#241 Loss: 0.0024400672409683466\n",
            "#242 Loss: 0.002439146162942052\n",
            "#243 Loss: 0.0024382262490689754\n",
            "#244 Loss: 0.002437308896332979\n",
            "#245 Loss: 0.0024363857228308916\n",
            "#246 Loss: 0.002435472561046481\n",
            "#247 Loss: 0.0024345542769879103\n",
            "#248 Loss: 0.0024336313363164663\n",
            "#249 Loss: 0.00243271142244339\n",
            "#250 Loss: 0.00243179383687675\n",
            "#251 Loss: 0.0024308778811246157\n",
            "#252 Loss: 0.0024299558717757463\n",
            "#253 Loss: 0.0024290340952575207\n",
            "#254 Loss: 0.002428111620247364\n",
            "#255 Loss: 0.002427193336188793\n",
            "#256 Loss: 0.002426273887977004\n",
            "#257 Loss: 0.002425355603918433\n",
            "#258 Loss: 0.002424436155706644\n",
            "#259 Loss: 0.002423514612019062\n",
            "#260 Loss: 0.002422596327960491\n",
            "#261 Loss: 0.0024216733872890472\n",
            "#262 Loss: 0.0024207504466176033\n",
            "#263 Loss: 0.002419829135760665\n",
            "#264 Loss: 0.0024189057294279337\n",
            "#265 Loss: 0.0024179841857403517\n",
            "#266 Loss: 0.002417063107714057\n",
            "#267 Loss: 0.0024161438923329115\n",
            "#268 Loss: 0.0024152155965566635\n",
            "#269 Loss: 0.0024142952170222998\n",
            "#270 Loss: 0.0024133676197379827\n",
            "#271 Loss: 0.002412450732663274\n",
            "#272 Loss: 0.002411528956145048\n",
            "#273 Loss: 0.0024105983320623636\n",
            "#274 Loss: 0.0024096802808344364\n",
            "#275 Loss: 0.0024087547790259123\n",
            "#276 Loss: 0.0024078262504190207\n",
            "#277 Loss: 0.0024068995844572783\n",
            "#278 Loss: 0.0024059752468019724\n",
            "#279 Loss: 0.002405051840469241\n",
            "#280 Loss: 0.002404116792604327\n",
            "#281 Loss: 0.0024031943175941706\n",
            "#282 Loss: 0.0024022667203098536\n",
            "#283 Loss: 0.002401341451331973\n",
            "#284 Loss: 0.002400410594418645\n",
            "#285 Loss: 0.0023994811344891787\n",
            "#286 Loss: 0.0023985551670193672\n",
            "#287 Loss: 0.0023976238444447517\n",
            "#288 Loss: 0.0023966955486685038\n",
            "#289 Loss: 0.0023957621306180954\n",
            "#290 Loss: 0.002394832205027342\n",
            "#291 Loss: 0.0023939006496220827\n",
            "#292 Loss: 0.002392966765910387\n",
            "#293 Loss: 0.00239203916862607\n",
            "#294 Loss: 0.002391106216236949\n",
            "#295 Loss: 0.0023901707027107477\n",
            "#296 Loss: 0.002389240777119994\n",
            "#297 Loss: 0.0023883050307631493\n",
            "#298 Loss: 0.0023873704485595226\n",
            "#299 Loss: 0.0023864342365413904\n",
            "#300 Loss: 0.0023854991886764765\n",
            "#301 Loss: 0.0023845701944082975\n",
            "#302 Loss: 0.0023836297914385796\n",
            "#303 Loss: 0.0023826900869607925\n",
            "#304 Loss: 0.0023817545734345913\n",
            "#305 Loss: 0.002380818361416459\n",
            "#306 Loss: 0.0023798795882612467\n",
            "#307 Loss: 0.0023789377883076668\n",
            "#308 Loss: 0.0023780011106282473\n",
            "#309 Loss: 0.0023770590778440237\n",
            "#310 Loss: 0.0023761214688420296\n",
            "#311 Loss: 0.0023751859553158283\n",
            "#312 Loss: 0.0023742406629025936\n",
            "#313 Loss: 0.002373295836150646\n",
            "#314 Loss: 0.0023723554331809282\n",
            "#315 Loss: 0.002371413866057992\n",
            "#316 Loss: 0.0023704750929027796\n",
            "#317 Loss: 0.002369531663134694\n",
            "#318 Loss: 0.0023685868363827467\n",
            "#319 Loss: 0.002367644337937236\n",
            "#320 Loss: 0.002366698579862714\n",
            "#321 Loss: 0.0023657495621591806\n",
            "#322 Loss: 0.0023648033384233713\n",
            "#323 Loss: 0.002363859675824642\n",
            "#324 Loss: 0.0023629090283066034\n",
            "#325 Loss: 0.0023619639687240124\n",
            "#326 Loss: 0.0023610175121575594\n",
            "#327 Loss: 0.002360069891437888\n",
            "#328 Loss: 0.002359122270718217\n",
            "#329 Loss: 0.0023581702262163162\n",
            "#330 Loss: 0.0023572223726660013\n",
            "#331 Loss: 0.002356275450438261\n",
            "#332 Loss: 0.0023553166538476944\n",
            "#333 Loss: 0.0023543667048215866\n",
            "#334 Loss: 0.0023534176871180534\n",
            "#335 Loss: 0.002352464245632291\n",
            "#336 Loss: 0.0023515131324529648\n",
            "#337 Loss: 0.0023505568969994783\n",
            "#338 Loss: 0.0023496015928685665\n",
            "#339 Loss: 0.002348652807995677\n",
            "#340 Loss: 0.002347696339711547\n",
            "#341 Loss: 0.0023467380087822676\n",
            "#342 Loss: 0.0023457861971110106\n",
            "#343 Loss: 0.0023448301944881678\n",
            "#344 Loss: 0.0023438704665750265\n",
            "#345 Loss: 0.002342912135645747\n",
            "#346 Loss: 0.002341957064345479\n",
            "#347 Loss: 0.0023409996647387743\n",
            "#348 Loss: 0.0023400387726724148\n",
            "#349 Loss: 0.002339078113436699\n",
            "#350 Loss: 0.002338117454200983\n",
            "#351 Loss: 0.0023371621500700712\n",
            "#352 Loss: 0.0023361986968666315\n",
            "#353 Loss: 0.00233523640781641\n",
            "#354 Loss: 0.0023342801723629236\n",
            "#355 Loss: 0.002333313226699829\n",
            "#356 Loss: 0.002332353265956044\n",
            "#357 Loss: 0.002331388648599386\n",
            "#358 Loss: 0.0023304217029362917\n",
            "#359 Loss: 0.0023294605780392885\n",
            "#360 Loss: 0.002328496426343918\n",
            "#361 Loss: 0.002327530412003398\n",
            "#362 Loss: 0.0023265639320015907\n",
            "#363 Loss: 0.0023255993146449327\n",
            "#364 Loss: 0.0023246288765221834\n",
            "#365 Loss: 0.0023236607667058706\n",
            "#366 Loss: 0.002322700573131442\n",
            "#367 Loss: 0.0023217289708554745\n",
            "#368 Loss: 0.0023207550402730703\n",
            "#369 Loss: 0.002319787396118045\n",
            "#370 Loss: 0.002318824175745249\n",
            "#371 Loss: 0.0023178488481789827\n",
            "#372 Loss: 0.002316881902515888\n",
            "#373 Loss: 0.0023159075062721968\n",
            "#374 Loss: 0.002314941259101033\n",
            "#375 Loss: 0.0023139675613492727\n",
            "#376 Loss: 0.0023129950277507305\n",
            "#377 Loss: 0.0023120215628296137\n",
            "#378 Loss: 0.002311046002432704\n",
            "#379 Loss: 0.002310073934495449\n",
            "#380 Loss: 0.002309101400896907\n",
            "#381 Loss: 0.0023081284016370773\n",
            "#382 Loss: 0.00230714725330472\n",
            "#383 Loss: 0.00230617169290781\n",
            "#384 Loss: 0.0023051972966641188\n",
            "#385 Loss: 0.002304219640791416\n",
            "#386 Loss: 0.0023032415192574263\n",
            "#387 Loss: 0.002302265027537942\n",
            "#388 Loss: 0.0023012871388345957\n",
            "#389 Loss: 0.002300310181453824\n",
            "#390 Loss: 0.002299328101798892\n",
            "#391 Loss: 0.0022983483504503965\n",
            "#392 Loss: 0.0022973709274083376\n",
            "#393 Loss: 0.002296391176059842\n",
            "#394 Loss: 0.002295407932251692\n",
            "#395 Loss: 0.00229442841373384\n",
            "#396 Loss: 0.002293441677466035\n",
            "#397 Loss: 0.0022924619261175394\n",
            "#398 Loss: 0.0022914784494787455\n",
            "#399 Loss: 0.0022904963698238134\n",
            "#400 Loss: 0.0022895135916769505\n",
            "#401 Loss: 0.0022885303478688\n",
            "#402 Loss: 0.0022875459399074316\n",
            "#403 Loss: 0.0022865592036396265\n",
            "#404 Loss: 0.0022855724673718214\n",
            "#405 Loss: 0.0022845915518701077\n",
            "#406 Loss: 0.002283601788803935\n",
            "#407 Loss: 0.002282612957060337\n",
            "#408 Loss: 0.002281626919284463\n",
            "#409 Loss: 0.0022806443739682436\n",
            "#410 Loss: 0.0022796487901359797\n",
            "#411 Loss: 0.0022786634508520365\n",
            "#412 Loss: 0.0022776739206165075\n",
            "#413 Loss: 0.0022766822949051857\n",
            "#414 Loss: 0.0022756929975003004\n",
            "#415 Loss: 0.0022747062612324953\n",
            "#416 Loss: 0.00227371440269053\n",
            "#417 Loss: 0.0022727230098098516\n",
            "#418 Loss: 0.002271731849759817\n",
            "#419 Loss: 0.0022707392927259207\n",
            "#420 Loss: 0.002269746968522668\n",
            "#421 Loss: 0.002268751384690404\n",
            "#422 Loss: 0.002267759060487151\n",
            "#423 Loss: 0.0022667646408081055\n",
            "#424 Loss: 0.0022657769732177258\n",
            "#425 Loss: 0.002264777896925807\n",
            "#426 Loss: 0.002263784408569336\n",
            "#427 Loss: 0.0022627897560596466\n",
            "#428 Loss: 0.0022617937065660954\n",
            "#429 Loss: 0.002260798355564475\n",
            "#430 Loss: 0.0022597969509661198\n",
            "#431 Loss: 0.002258802531287074\n",
            "#432 Loss: 0.0022578088100999594\n",
            "#433 Loss: 0.0022568099666386843\n",
            "#434 Loss: 0.002255811123177409\n",
            "#435 Loss: 0.0022548120468854904\n",
            "#436 Loss: 0.0022538129705935717\n",
            "#437 Loss: 0.0022528113331645727\n",
            "#438 Loss: 0.002251812256872654\n",
            "#439 Loss: 0.00225081411190331\n",
            "#440 Loss: 0.0022498099133372307\n",
            "#441 Loss: 0.002248812699690461\n",
            "#442 Loss: 0.002247813157737255\n",
            "#443 Loss: 0.0022468070965260267\n",
            "#444 Loss: 0.002245804527774453\n",
            "#445 Loss: 0.0022448061499744654\n",
            "#446 Loss: 0.002243800787255168\n",
            "#447 Loss: 0.0022427986841648817\n",
            "#448 Loss: 0.0022417923901230097\n",
            "#449 Loss: 0.0022407902870327234\n",
            "#450 Loss: 0.0022397860884666443\n",
            "#451 Loss: 0.002238777931779623\n",
            "#452 Loss: 0.002237774431705475\n",
            "#453 Loss: 0.00223676860332489\n",
            "#454 Loss: 0.0022357627749443054\n",
            "#455 Loss: 0.002234755316749215\n",
            "#456 Loss: 0.0022337529808282852\n",
            "#457 Loss: 0.0022327450569719076\n",
            "#458 Loss: 0.0022317382972687483\n",
            "#459 Loss: 0.002230728277936578\n",
            "#460 Loss: 0.0022297168616205454\n",
            "#461 Loss: 0.0022287091705948114\n",
            "#462 Loss: 0.002227703807875514\n",
            "#463 Loss: 0.002226694021373987\n",
            "#464 Loss: 0.002225684467703104\n",
            "#465 Loss: 0.0022246765438467264\n",
            "#466 Loss: 0.0022236653603613377\n",
            "#467 Loss: 0.0022226530127227306\n",
            "#468 Loss: 0.002221642527729273\n",
            "#469 Loss: 0.0022206297144293785\n",
            "#470 Loss: 0.0022196185309439898\n",
            "#471 Loss: 0.0022186103742569685\n",
            "#472 Loss: 0.0022175933700054884\n",
            "#473 Loss: 0.0022165849804878235\n",
            "#474 Loss: 0.0022155700717121363\n",
            "#475 Loss: 0.0022145553957670927\n",
            "#476 Loss: 0.0022135439794510603\n",
            "#477 Loss: 0.0022125281393527985\n",
            "#478 Loss: 0.002211514627560973\n",
            "#479 Loss: 0.002210496924817562\n",
            "#480 Loss: 0.0022094829473644495\n",
            "#481 Loss: 0.0022084659431129694\n",
            "#482 Loss: 0.0022074568551033735\n",
            "#483 Loss: 0.002206437522545457\n",
            "#484 Loss: 0.0022054200526326895\n",
            "#485 Loss: 0.0022044044453650713\n",
            "#486 Loss: 0.0022033853456377983\n",
            "#487 Loss: 0.0022023695055395365\n",
            "#488 Loss: 0.002201352035626769\n",
            "#489 Loss: 0.0022003341000527143\n",
            "#490 Loss: 0.002199317794293165\n",
            "#491 Loss: 0.0021982965990900993\n",
            "#492 Loss: 0.0021972774993628263\n",
            "#493 Loss: 0.00219626072794199\n",
            "#494 Loss: 0.0021952392999082804\n",
            "#495 Loss: 0.002194217639043927\n",
            "#496 Loss: 0.002193200634792447\n",
            "#497 Loss: 0.002192180836573243\n",
            "#498 Loss: 0.0021911589428782463\n",
            "#499 Loss: 0.0021901384461671114\n",
            "#500 Loss: 0.002189117018133402\n",
            "#501 Loss: 0.0021880920976400375\n",
            "#502 Loss: 0.0021870729979127645\n",
            "#503 Loss: 0.0021860499400645494\n",
            "#504 Loss: 0.0021850315388292074\n",
            "#505 Loss: 0.002184005454182625\n",
            "#506 Loss: 0.0021829840261489153\n",
            "#507 Loss: 0.002181959105655551\n",
            "#508 Loss: 0.0021809397730976343\n",
            "#509 Loss: 0.002179911592975259\n",
            "#510 Loss: 0.002178889000788331\n",
            "#511 Loss: 0.0021778629161417484\n",
            "#512 Loss: 0.002176836598664522\n",
            "#513 Loss: 0.002175812376663089\n",
            "#514 Loss: 0.0021747888531535864\n",
            "#515 Loss: 0.0021737609058618546\n",
            "#516 Loss: 0.002172738080844283\n",
            "#517 Loss: 0.002171711064875126\n",
            "#518 Loss: 0.0021706840489059687\n",
            "#519 Loss: 0.0021696598269045353\n",
            "#520 Loss: 0.0021686323452740908\n",
            "#521 Loss: 0.0021676046308130026\n",
            "#522 Loss: 0.0021665773820132017\n",
            "#523 Loss: 0.0021655478049069643\n",
            "#524 Loss: 0.0021645205561071634\n",
            "#525 Loss: 0.002163497731089592\n",
            "#526 Loss: 0.002162465127184987\n",
            "#527 Loss: 0.0021614336874336004\n",
            "#528 Loss: 0.0021604085341095924\n",
            "#529 Loss: 0.0021593787241727114\n",
            "#530 Loss: 0.0021583528723567724\n",
            "#531 Loss: 0.0021573195699602365\n",
            "#532 Loss: 0.0021562918554991484\n",
            "#533 Loss: 0.0021552571561187506\n",
            "#534 Loss: 0.0021542287431657314\n",
            "#535 Loss: 0.0021532000973820686\n",
            "#536 Loss: 0.0021521716844290495\n",
            "#537 Loss: 0.0021511383820325136\n",
            "#538 Loss: 0.0021501071751117706\n",
            "#539 Loss: 0.0021490773651748896\n",
            "#540 Loss: 0.0021480440627783537\n",
            "#541 Loss: 0.002147009363397956\n",
            "#542 Loss: 0.0021459797862917185\n",
            "#543 Loss: 0.002144948346540332\n",
            "#544 Loss: 0.002143915044143796\n",
            "#545 Loss: 0.0021428829059004784\n",
            "#546 Loss: 0.002141848672181368\n",
            "#547 Loss: 0.0021408156026154757\n",
            "#548 Loss: 0.002139780670404434\n",
            "#549 Loss: 0.0021387485321611166\n",
            "#550 Loss: 0.002137715695425868\n",
            "#551 Loss: 0.0021366847213357687\n",
            "#552 Loss: 0.0021356476936489344\n",
            "#553 Loss: 0.0021346136927604675\n",
            "#554 Loss: 0.0021335785277187824\n",
            "#555 Loss: 0.002132538938894868\n",
            "#556 Loss: 0.002131509128957987\n",
            "#557 Loss: 0.002130476525053382\n",
            "#558 Loss: 0.0021294394973665476\n",
            "#559 Loss: 0.002128403866663575\n",
            "#560 Loss: 0.002127366838976741\n",
            "#561 Loss: 0.0021263323724269867\n",
            "#562 Loss: 0.002125295577570796\n",
            "#563 Loss: 0.002124261111021042\n",
            "#564 Loss: 0.0021232208237051964\n",
            "#565 Loss: 0.002122187288478017\n",
            "#566 Loss: 0.0021211470011621714\n",
            "#567 Loss: 0.002120112767443061\n",
            "#568 Loss: 0.002119072712957859\n",
            "#569 Loss: 0.0021180338226258755\n",
            "#570 Loss: 0.00211700308136642\n",
            "#571 Loss: 0.0021159613970667124\n",
            "#572 Loss: 0.0021149280946701765\n",
            "#573 Loss: 0.0021138915326446295\n",
            "#574 Loss: 0.0021128482185304165\n",
            "#575 Loss: 0.0021118095610290766\n",
            "#576 Loss: 0.0021107716020196676\n",
            "#577 Loss: 0.002109734108671546\n",
            "#578 Loss: 0.0021087005734443665\n",
            "#579 Loss: 0.002107657492160797\n",
            "#580 Loss: 0.00210661836899817\n",
            "#581 Loss: 0.002105577616021037\n",
            "#582 Loss: 0.0021045382600277662\n",
            "#583 Loss: 0.002103500533849001\n",
            "#584 Loss: 0.0021024595480412245\n",
            "#585 Loss: 0.0021014243829995394\n",
            "#586 Loss: 0.002100378042086959\n",
            "#587 Loss: 0.002099341945722699\n",
            "#588 Loss: 0.00209829886443913\n",
            "#589 Loss: 0.0020972639322280884\n",
            "#590 Loss: 0.0020962206181138754\n",
            "#591 Loss: 0.002095181494951248\n",
            "#592 Loss: 0.0020941428374499083\n",
            "#593 Loss: 0.002093098359182477\n",
            "#594 Loss: 0.002092057839035988\n",
            "#595 Loss: 0.0020910180173814297\n",
            "#596 Loss: 0.002089978661388159\n",
            "#597 Loss: 0.0020889334846287966\n",
            "#598 Loss: 0.0020878936629742384\n",
            "#599 Loss: 0.0020868529099971056\n",
            "#600 Loss: 0.002085815416648984\n",
            "#601 Loss: 0.0020847702398896217\n",
            "#602 Loss: 0.0020837283227592707\n",
            "#603 Loss: 0.0020826871041208506\n",
            "#604 Loss: 0.0020816465839743614\n",
            "#605 Loss: 0.002080598147585988\n",
            "#606 Loss: 0.002079556928947568\n",
            "#607 Loss: 0.0020785192027688026\n",
            "#608 Loss: 0.0020774772856384516\n",
            "#609 Loss: 0.002076430944725871\n",
            "#610 Loss: 0.00207538646645844\n",
            "#611 Loss: 0.00207435037009418\n",
            "#612 Loss: 0.002073307754471898\n",
            "#613 Loss: 0.002072261879220605\n",
            "#614 Loss: 0.0020712194964289665\n",
            "#615 Loss: 0.0020701782777905464\n",
            "#616 Loss: 0.0020691361278295517\n",
            "#617 Loss: 0.0020680923480540514\n",
            "#618 Loss: 0.0020670518279075623\n",
            "#619 Loss: 0.0020660050213336945\n",
            "#620 Loss: 0.0020649584475904703\n",
            "#621 Loss: 0.0020639190915971994\n",
            "#622 Loss: 0.002062877407297492\n",
            "#623 Loss: 0.0020618324633687735\n",
            "#624 Loss: 0.0020607870537787676\n",
            "#625 Loss: 0.00205974536947906\n",
            "#626 Loss: 0.0020587043836712837\n",
            "#627 Loss: 0.0020576564129441977\n",
            "#628 Loss: 0.0020566147286444902\n",
            "#629 Loss: 0.002055570250377059\n",
            "#630 Loss: 0.0020545274019241333\n",
            "#631 Loss: 0.0020534859504550695\n",
            "#632 Loss: 0.002052436349913478\n",
            "#633 Loss: 0.0020513960625976324\n",
            "#634 Loss: 0.0020503487903624773\n",
            "#635 Loss: 0.0020493092015385628\n",
            "#636 Loss: 0.002048263093456626\n",
            "#637 Loss: 0.002047223038971424\n",
            "#638 Loss: 0.002046172507107258\n",
            "#639 Loss: 0.002045132452622056\n",
            "#640 Loss: 0.002044085180386901\n",
            "#641 Loss: 0.002043043961748481\n",
            "#642 Loss: 0.0020420013461261988\n",
            "#643 Loss: 0.0020409554708749056\n",
            "#644 Loss: 0.002039908664301038\n",
            "#645 Loss: 0.002038867911323905\n",
            "#646 Loss: 0.0020378208719193935\n",
            "#647 Loss: 0.0020367794204503298\n",
            "#648 Loss: 0.0020357321482151747\n",
            "#649 Loss: 0.002034691860899329\n",
            "#650 Loss: 0.002033643191680312\n",
            "#651 Loss: 0.002032601274549961\n",
            "#652 Loss: 0.002031555864959955\n",
            "#653 Loss: 0.0020305109210312366\n",
            "#654 Loss: 0.002029466675594449\n",
            "#655 Loss: 0.002028421498835087\n",
            "#656 Loss: 0.002027378184720874\n",
            "#657 Loss: 0.0020263351034373045\n",
            "#658 Loss: 0.0020252885296940804\n",
            "#659 Loss: 0.0020242466125637293\n",
            "#660 Loss: 0.002023200271651149\n",
            "#661 Loss: 0.002022160217165947\n",
            "#662 Loss: 0.0020211131777614355\n",
            "#663 Loss: 0.0020200731232762337\n",
            "#664 Loss: 0.0020190232899039984\n",
            "#665 Loss: 0.0020179767161607742\n",
            "#666 Loss: 0.002016937592998147\n",
            "#667 Loss: 0.002015892183408141\n",
            "#668 Loss: 0.0020148518960922956\n",
            "#669 Loss: 0.0020138081163167953\n",
            "#670 Loss: 0.0020127587486058474\n",
            "#671 Loss: 0.0020117172971367836\n",
            "#672 Loss: 0.0020106742158532143\n",
            "#673 Loss: 0.002009629737585783\n",
            "#674 Loss: 0.002008582465350628\n",
            "#675 Loss: 0.0020075414795428514\n",
            "#676 Loss: 0.002006495138630271\n",
            "#677 Loss: 0.0020054553169757128\n",
            "#678 Loss: 0.002004409907385707\n",
            "#679 Loss: 0.002003363100811839\n",
            "#680 Loss: 0.002002324676141143\n",
            "#681 Loss: 0.002001277869567275\n",
            "#682 Loss: 0.002000238513574004\n",
            "#683 Loss: 0.001999191241338849\n",
            "#684 Loss: 0.0019981495570391417\n",
            "#685 Loss: 0.0019971048459410667\n",
            "#686 Loss: 0.0019960617646574974\n",
            "#687 Loss: 0.0019950189162045717\n",
            "#688 Loss: 0.0019939783960580826\n",
            "#689 Loss: 0.001992932753637433\n",
            "#690 Loss: 0.001991888275370002\n",
            "#691 Loss: 0.0019908458925783634\n",
            "#692 Loss: 0.001989804906770587\n",
            "#693 Loss: 0.0019887599628418684\n",
            "#694 Loss: 0.0019877159502357244\n",
            "#695 Loss: 0.001986677525565028\n",
            "#696 Loss: 0.0019856367725878954\n",
            "#697 Loss: 0.001984592527151108\n",
            "#698 Loss: 0.001983546419069171\n",
            "#699 Loss: 0.001982505898922682\n",
            "#700 Loss: 0.0019814646802842617\n",
            "#701 Loss: 0.0019804220646619797\n",
            "#702 Loss: 0.0019793810788542032\n",
            "#703 Loss: 0.0019783375319093466\n",
            "#704 Loss: 0.0019772977102547884\n",
            "#705 Loss: 0.0019762550946325064\n",
            "#706 Loss: 0.0019752129446715117\n",
            "#707 Loss: 0.001974171493202448\n",
            "#708 Loss: 0.001973131438717246\n",
            "#709 Loss: 0.001972092781215906\n",
            "#710 Loss: 0.0019710464403033257\n",
            "#711 Loss: 0.0019700077828019857\n",
            "#712 Loss: 0.001968963770195842\n",
            "#713 Loss: 0.0019679246470332146\n",
            "#714 Loss: 0.0019668852910399437\n",
            "#715 Loss: 0.001965844538062811\n",
            "#716 Loss: 0.001964807277545333\n",
            "#717 Loss: 0.0019637665245682\n",
            "#718 Loss: 0.0019627264700829983\n",
            "#719 Loss: 0.00196168408729136\n",
            "#720 Loss: 0.00196064286865294\n",
            "#721 Loss: 0.0019596030469983816\n",
            "#722 Loss: 0.001958560897037387\n",
            "#723 Loss: 0.001957525731995702\n",
            "#724 Loss: 0.001956489635631442\n",
            "#725 Loss: 0.001955445623025298\n",
            "#726 Loss: 0.001954407896846533\n",
            "#727 Loss: 0.0019533671438694\n",
            "#728 Loss: 0.0019523290684446692\n",
            "#729 Loss: 0.0019512904109433293\n",
            "#730 Loss: 0.0019502503564581275\n",
            "#731 Loss: 0.0019492128631100059\n",
            "#732 Loss: 0.0019481779308989644\n",
            "#733 Loss: 0.0019471339182928205\n",
            "#734 Loss: 0.0019461024785414338\n",
            "#735 Loss: 0.0019450596300885081\n",
            "#736 Loss: 0.0019440216710790992\n",
            "#737 Loss: 0.001942987204529345\n",
            "#738 Loss: 0.0019419504096731544\n",
            "#739 Loss: 0.0019409122178331017\n",
            "#740 Loss: 0.0019398737931624055\n",
            "#741 Loss: 0.0019388411892578006\n",
            "#742 Loss: 0.001937802298925817\n",
            "#743 Loss: 0.001936764339916408\n",
            "#744 Loss: 0.001935729756951332\n",
            "#745 Loss: 0.0019346913322806358\n",
            "#746 Loss: 0.0019336584955453873\n",
            "#747 Loss: 0.0019326211186125875\n",
            "#748 Loss: 0.001931585487909615\n",
            "#749 Loss: 0.0019305492751300335\n",
            "#750 Loss: 0.0019295121310278773\n",
            "#751 Loss: 0.0019284767331555486\n",
            "#752 Loss: 0.0019274475052952766\n",
            "#753 Loss: 0.0019264090806245804\n",
            "#754 Loss: 0.0019253772916272283\n",
            "#755 Loss: 0.0019243452697992325\n",
            "#756 Loss: 0.0019233074272051454\n",
            "#757 Loss: 0.0019222754053771496\n",
            "#758 Loss: 0.0019212419865652919\n",
            "#759 Loss: 0.0019202110124751925\n",
            "#760 Loss: 0.0019191773608326912\n",
            "#761 Loss: 0.0019181432435289025\n",
            "#762 Loss: 0.0019171085441485047\n",
            "#763 Loss: 0.0019160775700584054\n",
            "#764 Loss: 0.0019150450825691223\n",
            "#765 Loss: 0.0019140088697895408\n",
            "#766 Loss: 0.0019129784777760506\n",
            "#767 Loss: 0.0019119485514238477\n",
            "#768 Loss: 0.0019109140848740935\n",
            "#769 Loss: 0.0019098850898444653\n",
            "#770 Loss: 0.001908852718770504\n",
            "#771 Loss: 0.001907822792418301\n",
            "#772 Loss: 0.0019067925168201327\n",
            "#773 Loss: 0.0019057630561292171\n",
            "#774 Loss: 0.0019047335954383016\n",
            "#775 Loss: 0.0019037051824852824\n",
            "#776 Loss: 0.0019026693189516664\n",
            "#777 Loss: 0.0019016433507204056\n",
            "#778 Loss: 0.0019006148213520646\n",
            "#779 Loss: 0.0018995892023667693\n",
            "#780 Loss: 0.0018985569477081299\n",
            "#781 Loss: 0.0018975288840010762\n",
            "#782 Loss: 0.0018965002382174134\n",
            "#783 Loss: 0.0018954715924337506\n",
            "#784 Loss: 0.0018944436451420188\n",
            "#785 Loss: 0.0018934140680357814\n",
            "#786 Loss: 0.0018923920579254627\n",
            "#787 Loss: 0.0018913644598796964\n",
            "#788 Loss: 0.001890333485789597\n",
            "#789 Loss: 0.0018893079832196236\n",
            "#790 Loss: 0.0018882853910326958\n",
            "#791 Loss: 0.001887254766188562\n",
            "#792 Loss: 0.0018862345023080707\n",
            "#793 Loss: 0.0018852058565244079\n",
            "#794 Loss: 0.00188418326433748\n",
            "#795 Loss: 0.0018831556662917137\n",
            "#796 Loss: 0.0018821310950443149\n",
            "#797 Loss: 0.0018811067566275597\n",
            "#798 Loss: 0.001880083349533379\n",
            "#799 Loss: 0.001879060291685164\n",
            "#800 Loss: 0.0018780353711917996\n",
            "#801 Loss: 0.0018770135939121246\n",
            "#802 Loss: 0.0018759918166324496\n",
            "#803 Loss: 0.0018749730661511421\n",
            "#804 Loss: 0.0018739477964118123\n",
            "#805 Loss: 0.0018729200819507241\n",
            "#806 Loss: 0.0018719009822234511\n",
            "#807 Loss: 0.001870879321359098\n",
            "#808 Loss: 0.001869861502200365\n",
            "#809 Loss: 0.0018688408890739083\n",
            "#810 Loss: 0.001867820625193417\n",
            "#811 Loss: 0.0018667984986677766\n",
            "#812 Loss: 0.0018657720647752285\n",
            "#813 Loss: 0.001864760648459196\n",
            "#814 Loss: 0.0018637363100424409\n",
            "#815 Loss: 0.0018627209356054664\n",
            "#816 Loss: 0.0018617023015394807\n",
            "#817 Loss: 0.0018606797093525529\n",
            "#818 Loss: 0.0018596658483147621\n",
            "#819 Loss: 0.0018586452351883054\n",
            "#820 Loss: 0.0018576303264126182\n",
            "#821 Loss: 0.001856614020653069\n",
            "#822 Loss: 0.001855594920925796\n",
            "#823 Loss: 0.0018545795464888215\n",
            "#824 Loss: 0.001853560097515583\n",
            "#825 Loss: 0.0018525446066632867\n",
            "#826 Loss: 0.0018515288829803467\n",
            "#827 Loss: 0.00185050955042243\n",
            "#828 Loss: 0.0018494967371225357\n",
            "#829 Loss: 0.0018484825268387794\n",
            "#830 Loss: 0.001847467734478414\n",
            "#831 Loss: 0.0018464555032551289\n",
            "#832 Loss: 0.0018454398959875107\n",
            "#833 Loss: 0.0018444285960868\n",
            "#834 Loss: 0.0018434150842949748\n",
            "#835 Loss: 0.0018424022709950805\n",
            "#836 Loss: 0.001841390854679048\n",
            "#837 Loss: 0.0018403776921331882\n",
            "#838 Loss: 0.0018393672071397305\n",
            "#839 Loss: 0.00183835718780756\n",
            "#840 Loss: 0.0018373435596004128\n",
            "#841 Loss: 0.001836334471590817\n",
            "#842 Loss: 0.0018353263149037957\n",
            "#843 Loss: 0.0018343138508498669\n",
            "#844 Loss: 0.0018333062762394547\n",
            "#845 Loss: 0.001832296489737928\n",
            "#846 Loss: 0.0018312829779461026\n",
            "#847 Loss: 0.0018302792450413108\n",
            "#848 Loss: 0.0018292715540155768\n",
            "#849 Loss: 0.0018282626988366246\n",
            "#850 Loss: 0.0018272522138431668\n",
            "#851 Loss: 0.001826247200369835\n",
            "#852 Loss: 0.0018252409063279629\n",
            "#853 Loss: 0.001824233098886907\n",
            "#854 Loss: 0.0018232259899377823\n",
            "#855 Loss: 0.001822225865907967\n",
            "#856 Loss: 0.0018212157301604748\n",
            "#857 Loss: 0.0018202122300863266\n",
            "#858 Loss: 0.0018192125717177987\n",
            "#859 Loss: 0.0018182039493694901\n",
            "#860 Loss: 0.0018171994015574455\n",
            "#861 Loss: 0.0018161969492211938\n",
            "#862 Loss: 0.00181519181933254\n",
            "#863 Loss: 0.0018141911132261157\n",
            "#864 Loss: 0.001813187263906002\n",
            "#865 Loss: 0.0018121921457350254\n",
            "#866 Loss: 0.0018111892277374864\n",
            "#867 Loss: 0.0018101868918165565\n",
            "#868 Loss: 0.0018091824604198337\n",
            "#869 Loss: 0.001808184664696455\n",
            "#870 Loss: 0.0018071848899126053\n",
            "#871 Loss: 0.0018061831360682845\n",
            "#872 Loss: 0.0018051863880828023\n",
            "#873 Loss: 0.0018041870789602399\n",
            "#874 Loss: 0.0018031877698376775\n",
            "#875 Loss: 0.0018021933501586318\n",
            "#876 Loss: 0.0018011946231126785\n",
            "#877 Loss: 0.0018001968273892999\n",
            "#878 Loss: 0.0017991961212828755\n",
            "#879 Loss: 0.001798199606128037\n",
            "#880 Loss: 0.0017972056521102786\n",
            "#881 Loss: 0.0017962086712941527\n",
            "#882 Loss: 0.0017952205380424857\n",
            "#883 Loss: 0.0017942209960892797\n",
            "#884 Loss: 0.001793228555470705\n",
            "#885 Loss: 0.0017922349506989121\n",
            "#886 Loss: 0.001791241578757763\n",
            "#887 Loss: 0.001790247275494039\n",
            "#888 Loss: 0.0017892572795972228\n",
            "#889 Loss: 0.0017882628599181771\n",
            "#890 Loss: 0.0017872735625132918\n",
            "#891 Loss: 0.0017862803069874644\n",
            "#892 Loss: 0.001785286352969706\n",
            "#893 Loss: 0.0017842984525486827\n",
            "#894 Loss: 0.0017833089223131537\n",
            "#895 Loss: 0.0017823184607550502\n",
            "#896 Loss: 0.0017813298618420959\n",
            "#897 Loss: 0.0017803410300984979\n",
            "#898 Loss: 0.0017793524311855435\n",
            "#899 Loss: 0.0017783649964258075\n",
            "#900 Loss: 0.001777378492988646\n",
            "#901 Loss: 0.0017763897776603699\n",
            "#902 Loss: 0.0017754010623320937\n",
            "#903 Loss: 0.001774418051354587\n",
            "#904 Loss: 0.0017734314315021038\n",
            "#905 Loss: 0.0017724483041092753\n",
            "#906 Loss: 0.0017714608693495393\n",
            "#907 Loss: 0.0017704787896946073\n",
            "#908 Loss: 0.0017694927519187331\n",
            "#909 Loss: 0.0017685088096186519\n",
            "#910 Loss: 0.0017675244016572833\n",
            "#911 Loss: 0.001766547211445868\n",
            "#912 Loss: 0.001765563734807074\n",
            "#913 Loss: 0.001764580956660211\n",
            "#914 Loss: 0.0017636003904044628\n",
            "#915 Loss: 0.0017626197077333927\n",
            "#916 Loss: 0.0017616351833567023\n",
            "#917 Loss: 0.0017606564797461033\n",
            "#918 Loss: 0.0017596777761355042\n",
            "#919 Loss: 0.0017587020993232727\n",
            "#920 Loss: 0.001757721765898168\n",
            "#921 Loss: 0.0017567459726706147\n",
            "#922 Loss: 0.0017557647079229355\n",
            "#923 Loss: 0.0017547908937558532\n",
            "#924 Loss: 0.0017538117244839668\n",
            "#925 Loss: 0.0017528367461636662\n",
            "#926 Loss: 0.0017518624663352966\n",
            "#927 Loss: 0.0017508859746158123\n",
            "#928 Loss: 0.0017499076202511787\n",
            "#929 Loss: 0.0017489390447735786\n",
            "#930 Loss: 0.0017479656962677836\n",
            "#931 Loss: 0.0017469911836087704\n",
            "#932 Loss: 0.0017460188828408718\n",
            "#933 Loss: 0.0017450453015044332\n",
            "#934 Loss: 0.00174407206941396\n",
            "#935 Loss: 0.0017430986044928432\n",
            "#936 Loss: 0.0017421283992007375\n",
            "#937 Loss: 0.001741158775985241\n",
            "#938 Loss: 0.0017401917139068246\n",
            "#939 Loss: 0.0017392206937074661\n",
            "#940 Loss: 0.0017382544465363026\n",
            "#941 Loss: 0.0017372820293530822\n",
            "#942 Loss: 0.001736316829919815\n",
            "#943 Loss: 0.0017353454604744911\n",
            "#944 Loss: 0.0017343764193356037\n",
            "#945 Loss: 0.0017334137810394168\n",
            "#946 Loss: 0.0017324457876384258\n",
            "#947 Loss: 0.0017314818687736988\n",
            "#948 Loss: 0.001730515738017857\n",
            "#949 Loss: 0.0017295492580160499\n",
            "#950 Loss: 0.0017285882495343685\n",
            "#951 Loss: 0.0017276207217946649\n",
            "#952 Loss: 0.0017266602953895926\n",
            "#953 Loss: 0.0017256977735087276\n",
            "#954 Loss: 0.0017247359501197934\n",
            "#955 Loss: 0.0017237764550372958\n",
            "#956 Loss: 0.0017228134674951434\n",
            "#957 Loss: 0.0017218533903360367\n",
            "#958 Loss: 0.0017208936624228954\n",
            "#959 Loss: 0.001719936146400869\n",
            "#960 Loss: 0.001718974090181291\n",
            "#961 Loss: 0.0017180143622681499\n",
            "#962 Loss: 0.001717058359645307\n",
            "#963 Loss: 0.0017161048017442226\n",
            "#964 Loss: 0.001715144026093185\n",
            "#965 Loss: 0.0017141870921477675\n",
            "#966 Loss: 0.0017132310895249248\n",
            "#967 Loss: 0.0017122785793617368\n",
            "#968 Loss: 0.0017113216454163194\n",
            "#969 Loss: 0.001710368786007166\n",
            "#970 Loss: 0.0017094146460294724\n",
            "#971 Loss: 0.001708458294160664\n",
            "#972 Loss: 0.0017075081123039126\n",
            "#973 Loss: 0.0017065554857254028\n",
            "#974 Loss: 0.0017056027427315712\n",
            "#975 Loss: 0.0017046512803062797\n",
            "#976 Loss: 0.0017037037760019302\n",
            "#977 Loss: 0.0017027502181008458\n",
            "#978 Loss: 0.0017018018988892436\n",
            "#979 Loss: 0.001700854511000216\n",
            "#980 Loss: 0.0016999054932966828\n",
            "#981 Loss: 0.001698957639746368\n",
            "#982 Loss: 0.0016980115324258804\n",
            "#983 Loss: 0.0016970612341538072\n",
            "#984 Loss: 0.0016961172223091125\n",
            "#985 Loss: 0.0016951701836660504\n",
            "#986 Loss: 0.001694221398793161\n",
            "#987 Loss: 0.0016932813450694084\n",
            "#988 Loss: 0.0016923333751037717\n",
            "#989 Loss: 0.0016913922736421227\n",
            "#990 Loss: 0.0016904502408578992\n",
            "#991 Loss: 0.0016895070439204574\n",
            "#992 Loss: 0.0016885654767975211\n",
            "#993 Loss: 0.001687621814198792\n",
            "#994 Loss: 0.0016866797814145684\n",
            "#995 Loss: 0.001685741706751287\n",
            "#996 Loss: 0.0016847997903823853\n",
            "#997 Loss: 0.0016838625306263566\n",
            "#998 Loss: 0.0016829235246405005\n",
            "#999 Loss: 0.0016819849843159318\n",
            "Predicted data based on trained weights: \n",
            "Input (scaled): \n",
            "tensor([0.5000, 1.0000])\n",
            "Output: \n",
            "tensor([0.9505])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/torch/serialization.py:241: UserWarning: Couldn't retrieve source code for container of type Neural_Network. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9nBzkgdbjcA",
        "colab_type": "text"
      },
      "source": [
        "The loss keeps decreasing, which means that the neural network is learning something. That's it. Congratulations! You have just learned how to create and train a neural network from scratch using PyTorch. There are so many things you can do with the shallow network we have just implemented. You can add more hidden layers or try to incorporate the bias terms for practice. I would love to see what you will build from here. Reach me out on [Twitter](https://twitter.com/omarsar0) if you have any further questions or leave your comments here. Until next time!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zcms4BCySKXj",
        "colab_type": "text"
      },
      "source": [
        "## References:\n",
        "- [PyTorch nn. Modules](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules)\n",
        "- [Build a Neural Network with Numpy](https://enlight.nyc/neural-network)\n"
      ]
    }
  ]
}