class_11_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "approximate-classic",
   "metadata": {},
   "source": [
    "# PyTorch\n",
    "\n",
    "Here's your best friend when working with PyTorch: https://pytorch.org/docs/stable/index.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spread-happiness",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "from mpl_toolkits.mplot3d import Axes3D, art3d\n",
    "from matplotlib.patches import Circle, Ellipse\n",
    "import seaborn as sns\n",
    "import matplotlib.ticker as ticker\n",
    "from IPython.display import Markdown, display, HTML\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# Fix the dying kernel problem (only a problem in some installations - you can remove it, if it works without it)\n",
    "import os\n",
    "os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hearing-street",
   "metadata": {},
   "source": [
    "## PyTorch basic operations tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regular-niger",
   "metadata": {},
   "source": [
    "**Task 1.** Calculate the sigmoid (logistic) function on every element of the following array [0.3, 1.2, -1.4, 0.2, -0.1, 0.1, 0.8, -0.25] and print the last 5 elements. Use only tensor operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "agreed-single",
   "metadata": {},
   "outputs": [],
   "source": [
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "another-catch",
   "metadata": {},
   "source": [
    "**Task 2.** Calculate the dot product of the following two vectors:<br/>\n",
    "$x = [3, 1, 4, 2, 6, 1, 4, 8]$<br/>\n",
    "$y = [5, 2, 3, 12, 2, 4, 17, 9]$<br/>\n",
    "a) by using element-wise mutliplication and torch.sum,<br/>\n",
    "b) by using torch.dot,<br/>\n",
    "b) by using torch.matmul and transposition (x.T)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "forbidden-journalism",
   "metadata": {},
   "outputs": [],
   "source": [
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acute-amber",
   "metadata": {},
   "source": [
    "**Task 3.** Calculate the following expression<br/>\n",
    "$$\\frac{1}{1 + e^{-x_0 \\theta_0 - \\ldots - x_9 \\theta_9 - \\theta_{10}}}$$\n",
    "for<br/>\n",
    "$x = [1.2, 2.3, 3.4, -0.7, 4.2, 2.7, -0.5, 1.4, -3.3, 0.2]$<br/>\n",
    "$\\theta = [1.7, 0.33, -2.12, -1.73, 2.9, -5.8, -0.9, 12.11, 3.43, -0.5, -1.65]$<br/>\n",
    "and print the result. Use only tensor operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "falling-holder",
   "metadata": {},
   "outputs": [],
   "source": [
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "latter-vector",
   "metadata": {},
   "source": [
    "# Tensor gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manual-serial",
   "metadata": {},
   "source": [
    "**Task 4.** Calculate the derivative $f'(w)$ using PyTorch and backward propagation (the backward method of the Tensor class) for the following functions and points:\n",
    "  - $f(w) = w^3 + w^2$ and $w = 2.0$,\n",
    "  - $f(w) = \\text{sin}(w)$ and $w = \\pi$,\n",
    "  - $f(w) = \\ln(w * e^{3w})$ and $w = 1.0$.\n",
    "  \n",
    "Print the values of those derivatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "copyrighted-perry",
   "metadata": {},
   "outputs": [],
   "source": [
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frequent-sarah",
   "metadata": {},
   "source": [
    "**Task 5.** Calculate the derivative $\\frac{\\partial f}{\\partial w_1}(w_1, w_2, w_3)$ using PyTorch and backward propagation (the backward method of the Tensor class) for the following functions and points:\n",
    "  - $f(w_1, w_2) = w_1^3 + w_1^2 + w_2$ and $(w_1, w_2) = (2.0, 3.0)$,\n",
    "  - $f(w_1, w_2, w_3) = \\text{sin}(w_1) * w_2 + w_1^2 * w_3$ and $(w_1, w_2) = (\\pi, 2.0, 4.0)$,\n",
    "  - $f(w_1, w_2, w_3) = e^{w_1^2 + w_2^2 + w_3^2} + w_1^2 + w_2^2 + w_3^2$ and $(w_1, w_2, w_3) = (0.5, 0.67, 0.55)$.\n",
    "  \n",
    "Print the values of those derivatives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dietary-columbia",
   "metadata": {},
   "outputs": [],
   "source": [
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "innovative-provider",
   "metadata": {},
   "source": [
    "# Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broadband-margin",
   "metadata": {},
   "source": [
    "**Task 6.** Write the calculate_se_array(base_w, x, y_target, weight_idx, weight_start, weight_end) which generates a numpy array of squared errors for the following sigmoid function $\\text{sigmoid}(w * x)$ and target value $y_target$, where $x$ is a vector of dimension 3 and $w$ is a vector of weights of the same dimension. The function should generate a set of weight vectors w which have the same value as base_w on all coordinates other than weight_idx, and should have 100 values on the weight_idx coordinate ranging from weight_start to weight_end. Example for weight_idx =1: \n",
    "\n",
    "w = [[base_w[0], weight_start, base_w[2]], [base_w[0], weight_start + step, base_w[2]], ..., [base_w[0], weight_end - step, base_w[2]], [base_w[0], weight_end, base_w[2]]]\n",
    "\n",
    "Then the function should calculate and return an array of squared errors between the value of the sigmoid on x with respect to all those weight vectors and the target value y_target. This will allow you to see how the error of a single neuron depends on every weight separately.\n",
    "\n",
    "The function should work even if base_w and x are tensors (a safe way to do is tensor.cpu().detach().numpy()).\n",
    "\n",
    "You're encouraged to experiment with different neural nets to see how the error depends on all its parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cloudy-growing",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_se_array(base_w, x, y_target, weight_idx, weight_start, weight_end):\n",
    "    ########################\n",
    "    # Write your code here #\n",
    "    ########################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "signed-belarus",
   "metadata": {},
   "source": [
    "In the next cell you can test the method and in the cell after that you can see the error plots for the training you saw in the lecture. The red dot indicates the current weight value and the error it gives. The red dot should arrive at the lowest point at the end of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "representative-sauce",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([2., -1., 3.], requires_grad=False)\n",
    "w = torch.tensor([-0.65, -0.4, -1.], requires_grad=True)\n",
    "y_target = 0.65\n",
    "\n",
    "base_w = w\n",
    "weight_idx = 0 \n",
    "weight_start = -5\n",
    "weight_end = 5\n",
    "\n",
    "print(calculate_se_array(base_w, x, y_target, weight_idx, weight_start, weight_end))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "supposed-sellers",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "x = torch.tensor([2., -1., 3.], requires_grad=False)\n",
    "w = torch.tensor([-0.65, -0.4, -1.], requires_grad=True)\n",
    "y_target = 0.65\n",
    "\n",
    "optimizer = optim.SGD([w], lr=0.1)\n",
    "\n",
    "losses = []\n",
    "n_epochs = 100\n",
    "for epoch in range(n_epochs):\n",
    "\n",
    "    optimizer.zero_grad()\n",
    "    y = torch.sigmoid(torch.sum(x * w))\n",
    "    loss = torch.pow(y - y_target, 2)\n",
    "    loss.backward()\n",
    "    losses.append(loss.item())\n",
    "    optimizer.step()\n",
    "\n",
    "    if epoch > 0 and epoch % 10 == 0:\n",
    "        print(\"Epoch: {}\".format(epoch))\n",
    "        print(w.cpu().detach().numpy())\n",
    "        print(loss.item())\n",
    "        for i in range(3):    \n",
    "            w_range = np.linspace(-5, 5, 100)\n",
    "            errors = calculate_se_array(w, x, y_target, i, -5, 5)\n",
    "            sns.lineplot(x=w_range, y=errors).set_title('Training loss')\n",
    "            plt.scatter([w[i].item()], [loss.item()], color='red')\n",
    "            plt.xlabel(\"weight {}\".format(i))\n",
    "            plt.ylabel(\"squared loss\")\n",
    "            plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "short-border",
   "metadata": {},
   "source": [
    "**Task 7*.** Train a neural network with:\n",
    "  - two input neurons, \n",
    "  - four hidden neurons with no bias and sigmoid activation in the first hidden layer,\n",
    "  - four hidden neurons with no bias and with sigmoid activation in the second hidden layer,\n",
    "  - one output neuron with no bias and without sigmoid activation \n",
    "  \n",
    "to get a good approximation of $f(x) = x_1 * x_2 + 1$ on the following dataset $D = \\{(1.0, 1.0), (0.0, 0.0), (2.0, -1.0), (-1.0, 0.5), (-0.5, -2.0)\\}$, i.e. the network should satisfy:\n",
    "  - $\\text{net}(1.0, 1.0) \\sim 2.0$,\n",
    "  - $\\text{net}(0.0, 0.0) \\sim 1.0$,\n",
    "  - $\\text{net}(2.0, -1.0) \\sim -1.0$,\n",
    "  - $\\text{net}(-1.0, 0.5) \\sim 0.5$,\n",
    "  - $\\text{net}(-0.5, -2.0) \\sim 2.0$.\n",
    "  \n",
    "Use seed=6789 to initialize the network.\n",
    "\n",
    "After training print all weights and separately print $w_{1, 2}^{(1)}$ (the weight from the second input to the first hidden neuron in the first hidden layer) and $w_{1, 3}^{(3)}$ (the weight from the third hidden neuron in the second hidden layer to the output unit).\n",
    "\n",
    "Print the values of the network on the training points and verify that these values are closer to the real values of the $f$ function than $\\epsilon = 0.1$, i.e. $|\\text{net}(x) - f(x)| < \\epsilon$ for $x \\in D$.\n",
    "\n",
    "Because this network is only tested on the training set, it will certainly overfit if trained long enough. Train for 1000 epochs and then calculate\n",
    "  - $\\text{net}(2.0, 2.0)$,\n",
    "  - $\\text{net}(-1.0, -1.0)$,\n",
    "  - $\\text{net}(3.0, -3.0)$.\n",
    "  \n",
    "How far are these values from real values of the function $f$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "documentary-petersburg",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TwoLayerModel(nn.Module):\n",
    "    def __init__(self, seed):\n",
    "        super().__init__()\n",
    "\n",
    "        self.seed = torch.manual_seed(seed)\n",
    "\n",
    "        ########################\n",
    "        # Write your code here #\n",
    "        ########################\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        ########################\n",
    "        # Write your code here #\n",
    "        ########################\n",
    "\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "extraordinary-combat",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the data\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "# Initialize the neural network and optimizer\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "# Perform the training for 1000 epochs\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "print(\"All weights - layer by layer starting from left to right\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "    \n",
    "print()\n",
    "print(\"w_{{1, 2}}^{{(1)}} (rounded to 4 decimal points)\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "print()\n",
    "print(\"w_{{1, 3}}^{{(2)}} (rounded to 4 decimal points)\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "print()\n",
    "print(\"Values on the training data (verify the difference is below epsilon)\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "print()\n",
    "print(\"NN value on x=[2.0, 2.0] vs real value\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "\n",
    "print()\n",
    "print(\"NN value on x=[-1.0, -1.0] vs real value\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n",
    "\n",
    "print()\n",
    "print(\"NN value on x=[3.0, -3.0] vs real value\")\n",
    "\n",
    "########################\n",
    "# Write your code here #\n",
    "########################\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}