01-pytorch-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to PyTorch\n",
    "\n",
    "PyTorch is a platform for deep learning in Python or C++. In this lecture we will focus in the **Python** landscape. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensors\n",
    "\n",
    "Tensors are elementary units of PyTorch. They are very similar to numpy arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(0)\n",
    "\n",
    "import torch\n",
    "torch.manual_seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1.0, 2.0, 3.0])\n",
    "y = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = y ** 2\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadly speaking, a tensor is like a numpy array that can carry gradient information from the chain of operations applied on top of it. There are other flavors that make them different, but this is the key distinction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating tensors "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# directly from data\n",
    "data = [[0, 1], [1, 0]]\n",
    "x_data = torch.tensor(data)\n",
    "x_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from a numpy array\n",
    "x_numpy = np.array([[1, 2], [3, 4]])\n",
    "x_torch = torch.from_numpy(x_numpy)\n",
    "x_torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert it back to a numpy array\n",
    "x_numpy = x_torch.numpy()\n",
    "x_numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with constant data\n",
    "x = torch.ones(2, 3)  # 2 rows and 3 columns\n",
    "print(x)\n",
    "y = torch.zeros(3, 2) # 3 rows and 2 columns\n",
    "print(y)\n",
    "z = torch.full((3, 1), -5)  # 3 row and 1 columns (aka column vector)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with random data\n",
    "x = torch.rand(2, 3)  # uniform distribution U(0, 1)\n",
    "print(x)\n",
    "y = torch.randn(2, 3)  # standard gaussian N(0, 1)\n",
    "print(y)\n",
    "z = torch.randint(0, 10, size=(2, 3))  # random integers [0, 10)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# other initializations\n",
    "print(torch.arange(5))  # from 0 (inclusive) to 5 (exclusive)\n",
    "print(torch.arange(2, 8))  # from 2 to 8\n",
    "print(torch.arange(2, 8, 2))  # from 2 to 8, with stepsize=2\n",
    "\n",
    "print(torch.linspace(0, 1, 6))  # returns 6 linear spaced numbers from 0 to 1 (inclusive)\n",
    "print(torch.linspace(-1, 1, 8))  # returns 8 linear spaced numbers form -1 to 1 \n",
    "\n",
    "print(torch.eye(3))  # identity matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the full set of creation ops [here](https://pytorch.org/docs/stable/torch.html#creation-ops)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand(3, 4, requires_grad=True)\n",
    "print(x.device)\n",
    "print(x.shape)\n",
    "print(x.dtype)\n",
    "print(x)\n",
    "print(x.data)\n",
    "print(x[0, 0])\n",
    "print(x[0, 0].item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensor data types:\n",
    "\n",
    "<table class=\"docutils colwidths-auto align-default\">\n",
    "<thead>\n",
    "<tr class=\"row-odd\"><th class=\"head\"><p>Data type</p></th>\n",
    "<th class=\"head\"><p>dtype</p></th>\n",
    "<th class=\"head\"><p>Legacy Constructors</p></th>\n",
    "</tr>\n",
    "</thead>\n",
    "<tbody>\n",
    "<tr class=\"row-even\"><td><p>32-bit floating point</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.float32</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.float</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.FloatTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>64-bit floating point</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.float64</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.double</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.DoubleTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>64-bit complex</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.complex64</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.cfloat</span></code></p></td>\n",
    "<td></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>128-bit complex</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.complex128</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.cdouble</span></code></p></td>\n",
    "<td></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>16-bit floating point <a class=\"footnote-reference brackets\" href=\"#id3\" id=\"id1\">1</a></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.float16</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.half</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.HalfTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>16-bit floating point <a class=\"footnote-reference brackets\" href=\"#id4\" id=\"id2\">2</a></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.bfloat16</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.BFloat16Tensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>8-bit integer (unsigned)</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.uint8</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.ByteTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>8-bit integer (signed)</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.int8</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.CharTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>16-bit integer (signed)</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.int16</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.short</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.ShortTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>32-bit integer (signed)</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.int32</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.int</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.IntTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-even\"><td><p>64-bit integer (signed)</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.int64</span></code> or <code class=\"docutils literal notranslate\"><span class=\"pre\">torch.long</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.LongTensor</span></code></p></td>\n",
    "</tr>\n",
    "<tr class=\"row-odd\"><td><p>Boolean</p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.bool</span></code></p></td>\n",
    "<td><p><code class=\"docutils literal notranslate\"><span class=\"pre\">torch.*.BoolTensor</span></code></p></td>\n",
    "</tr>\n",
    "</tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Casting tensors accoding to regular Python rules:\n",
    "```\n",
    "complex > floating > integral > boolean\n",
    "```\n",
    "\n",
    "Also, be careful with casts to the same dtypes to avoid underflow/overflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "float_tensor = torch.randn(2, 2, dtype=torch.float)\n",
    "int_tensor = torch.ones(1, dtype=torch.int)\n",
    "long_tensor = torch.ones(1, dtype=torch.long)\n",
    "uint_tensor = torch.ones(1, dtype=torch.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "long_tensor_big_number = long_tensor * 2**33\n",
    "long_tensor_big_number, long_tensor_big_number.int()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "float_tensor, float_tensor.long()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the full list of attributes [here](https://pytorch.org/docs/stable/tensor_attributes.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scalar\n",
    "x = torch.tensor(2)\n",
    "print(x)\n",
    "print(x.shape)\n",
    "print(x.item())  # access the (single) element inside the tensor\n",
    "print('')\n",
    "\n",
    "# vector\n",
    "x = torch.rand(4)\n",
    "print(x)\n",
    "print(x.shape)\n",
    "print('')\n",
    "\n",
    "# matrix\n",
    "x = torch.rand(4, 3)\n",
    "print(x)\n",
    "print(x.shape)\n",
    "print('')\n",
    "\n",
    "# n-dimensional array\n",
    "x = torch.rand(3, 4, 3)  # e.g., image with width=3, height=4, and channels=3\n",
    "print(x)\n",
    "print(x.shape)\n",
    "print('')\n",
    "\n",
    "from matplotlib import pyplot as plt; plt.imshow(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = torch.arange(8)\n",
    "v2 = torch.arange(10, 18)\n",
    "\n",
    "print(\"v1: %s\" % v1)\n",
    "print(\"v2: %s\" % v2)\n",
    "print(\"Dot product: %d\" % v1.dot(v2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can also change a value inside the array manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2[1] = 25\n",
    "print(v2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Accessing values:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual tensor positions are scalars, or 0-dimension tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(v1[0])\n",
    "print(v1[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.item()` returns a Python number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = v1[0].item()\n",
    "print(number)\n",
    "print(isinstance(number, int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Numpy-style indexing:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = torch.randn(3, 4, 3)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "m[0,1,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m[:, 1, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m[0, :, -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m[:, :, -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m[..., -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Elementwise operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 + v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 * v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some caveats when working with integer values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 / v2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = v1.float()\n",
    "y = v2.float()\n",
    "x / y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Operations with constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregating tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x ** 2).sum().sqrt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.mean(), x.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.min(), x.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.norm(p=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Joining tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.cat([x, y])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z = torch.stack([x, y])\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.vstack([z, x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1 = torch.rand(5, 4)\n",
    "m2 = torch.rand(4, 5)\n",
    "\n",
    "print(\"m1: %s\\n\" % m1)\n",
    "print(\"m2: %s\\n\" % m2)\n",
    "print(m1.dot(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops... that can be misleading if you are used to numpy. In PyTorch, `dot` is reserved for vectors only.\n",
    "For matrices, call `mm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(m1.mm(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the now-default-python operator for matrix multiplication `@`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(m1 @ m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if I have batched data? It's better to use `.bmm()` (this is a common source of error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1 = torch.rand(2, 5, 4)\n",
    "m2 = torch.rand(2, 4, 5)\n",
    "\n",
    "print(m1.bmm(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`@` will work as `.bmm()`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(m1 @ m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if I have even more dimensions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1 = torch.rand(2, 3, 5, 4)\n",
    "m2 = torch.rand(2, 3, 4, 5)\n",
    "\n",
    "print(m1.bmm(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.bmm` works only with 3d tensors. For higher dimensionalities, we can use the more general `matmul`. In fact, the `@` operator is a shorthand for `matmul` (which is implemented in the magic method `__matmul__` )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(m1.matmul(m2).shape)\n",
    "print(m1.matmul(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anoter option is to use the powerful `einsum` function. Let's say our input have the following representation:\n",
    "- `b` = batch size \n",
    "- `c` = channels\n",
    "- `i` = `m1` timesteps\n",
    "- `j` = `m2` timesteps\n",
    "- `d` = hidden size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "torch.einsum('bcid,bcdj->bcij', m1, m2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See more about `einsum` here: https://pytorch.org/docs/master/generated/torch.einsum.html#torch.einsum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting\n",
    "\n",
    "Broadcasting means doing some arithmetic operation with tensors of different ranks, as if the smaller one were expanded, or broadcast, to match the larger.\n",
    "\n",
    "Let's experiment with a matrix (rank 2 tensor) and a vector (rank 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = torch.rand(5, 4)\n",
    "v = torch.arange(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"m:\", m)\n",
    "print(\"v:\", v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m_plus_v = m + v\n",
    "print(\"m + v:\\n\", m_plus_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Proof check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"m[0] = %s\\n\" % m[0])\n",
    "print(\"v = %s\\n\" % v)\n",
    "\n",
    "row_sum = m[0] + v\n",
    "print(\"m[0] + v = %s\\n\" % row_sum)\n",
    "print(\"(m + v)[0] = %s\" % m_plus_v[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also reshape tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view(2, 2)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view(4, 1)\n",
    "v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that shape `[4, 1]` is not broadcastable to match `[5, 4]`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m + v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... but `[1, 4]` is!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view(1, 4)\n",
    "m + v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Squeezing and Unsqueezing\n",
    "\n",
    "Broadcasting is one of the most important concepts for manipulating n-dimensional arrays. PyTorch offers some ways of expanding the rank of a tensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = torch.rand(4).view(1, 4, 1)\n",
    "print(v)\n",
    "print(v.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.squeeze().shape  # \"compress\" all single-dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.squeeze(0).shape  # \"compress\" only the (0-indexed) single-dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.unsqueeze(1).shape  # \"add\" a new dimension BEFORE the (1-indexed) dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using numpy notation (better since it explicitily says where a new dimension is being created)\n",
    "v[:, None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.unsqueeze(1).unsqueeze(-1).unsqueeze(1).shape  # what unsqueeze(1).unsqueeze(1) does?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v[:, None, None, ..., None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also use .view(dims) as long te specified dims are valid\n",
    "v.view(1, 1, 1, 4, 1, 1).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General Broadcast Semantics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two tensors are “broadcastable” if the following rules hold:\n",
    "\n",
    "- Each tensor has at least one dimension.\n",
    "\n",
    "- When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand(5,7,3)\n",
    "y = torch.rand(5,7,3)\n",
    "z = x + y\n",
    "# same shapes are always broadcastable (i.e. the above rules always hold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.rand((0,))\n",
    "y = torch.rand(2,2)\n",
    "print(x.shape)\n",
    "z = x + y\n",
    "# x and y are not broadcastable, because x does not have at least 1 dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# can line up trailing dimensions\n",
    "x = torch.empty(5,3,4,1)\n",
    "y = torch.empty(  3,1,1)\n",
    "z = x + y\n",
    "# x and y are broadcastable.\n",
    "# 1st trailing dimension: both have size 1\n",
    "# 2nd trailing dimension: y has size 1\n",
    "# 3rd trailing dimension: x size == y size\n",
    "# 4th trailing dimension: y dimension doesn't exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# but:\n",
    "x = torch.empty(5,2,4,1)\n",
    "y = torch.empty(  3,1,1)\n",
    "z = x + y\n",
    "# x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always take care of tensor shapes! It is a good practice to debug how some expression is evaluated before inserting adding it to your codebase. \n",
    "\n",
    "<!-- In other words, **you can use pytorch's dynamic graph creation ability to debug your model by printing tensor shapes!** -->\n",
    "\n",
    "See more here: https://pytorch.org/docs/master/notes/broadcasting.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful Functions\n",
    "\n",
    "Pytorch (and other libraries) have many functions that operate on tensors. Let's try some of them and plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a vector x with values from -10 to 10, and intervals of 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.arange(-10, 10, 0.1, dtype=torch.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.sin()\n",
    "plt.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.tanh()\n",
    "plt.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.exp()\n",
    "plt.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = torch.log(x)\n",
    "pl.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# But what about GPUs?\n",
    "How do I use A GPU?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "my_device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a GPU you should get something like: \n",
    "`device(type='cuda', index=0)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can initialize a tensor in a specfic device\n",
    "torch.ones(5, device=my_device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can move data to the GPU by doing .to(device)\n",
    "data = torch.eye(3)  # data is on the cpu \n",
    "data.to(my_device)  # data is moved to my_device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the computation happens on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = data + data\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can get a tensor's device via the .device attribute\n",
    "res.device\n",
    "z = torch.arange(10)\n",
    "z = z.to(res.device)\n",
    "print(z.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic differentiation with `autograd`\n",
    "\n",
    "Central to all neural networks in PyTorch is the `autograd` package. \n",
    "\n",
    "We can say that it is the _true_ power behind PyTorch. The autograd package provides automatic differentiation for all operations on Tensors. It is a **define-by-run** framework, which means that your backprop is defined by how your code is run, and that **every single iteration can be different**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.Tensor` is the central class of the package. If you set its attribute `.requires_grad` as `True`, it starts to track all operations applied on it. When you finish your computation you can call `.backward()` and have all the gradients computed automatically. The gradient for this tensor will be accumulated into the `.grad` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting requires_grad in directly via tensor's constructor\n",
    "x = torch.tensor(2., requires_grad=True)\n",
    "\n",
    "# or by setting .requires_grad attribute\n",
    "# you can do this at any moment to track operations on x\n",
    "x.requires_grad = True  \n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x.requires_grad)\n",
    "print(x.grad)  # no gradient yet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's perform a simple operation on x\n",
    "y = x ** 2\n",
    "\n",
    "print(\"Grad of x:\", x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you want to compute the derivatives, you can call .backward() on a Tensor\n",
    "y.backward()\n",
    "print(\"Grad of y with respect to x:\", x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To stop a tensor from tracking history, you can call `.detach()` to detach it from the computation history, and to prevent future computation from being tracked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2., requires_grad=True)\n",
    "print(x)\n",
    "\n",
    "y = x ** 2\n",
    "print(y)\n",
    "\n",
    "c = y.detach()  # c will be treated as a constant! c has the same contents as y but requires_grad=False\n",
    "print(c)\n",
    "\n",
    "z = c * y.exp()  \n",
    "print(z)\n",
    "\n",
    "z.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prevent tracking history (and using memory), you can also wrap the code block in with `torch.no_grad()`: This can be particularly helpful when evaluating a model because the model may have trainable parameters with `requires_grad=True`, but for which we don’t need the gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(2.)\n",
    "x.requires_grad = True\n",
    "print('x:', x)\n",
    "\n",
    "y = x ** 2\n",
    "print('y:', y)\n",
    "\n",
    "with torch.no_grad():\n",
    "    y = 2 * y\n",
    "    print('x:', x)  # Try to think why x.requires_grad is True\n",
    "    print('y:', y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There’s one more class which is very important for autograd implementation - a `Function`.\n",
    "\n",
    "`Tensor` and `Function` are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each tensor has a `.grad_fn` attribute that references a `Function` that has created the `Tensor` (except for `Tensor`s created by the user - their `grad_fn` is `None`).\n",
    "\n",
    "Let's go back and see the `grad_fn` in our previous example:\n",
    "```\n",
    "input -> x -> Pow(2) -> y -> Exp() -> Mul(constant) -> output\n",
    "```\n",
    "\n",
    "We can create a `Function` and manually define its gradient (this is particularly useful for originally non-differentiable operations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Exp(torch.autograd.Function):\n",
    "    @staticmethod\n",
    "    def forward(ctx, i):\n",
    "        result = i.exp()\n",
    "        ctx.save_for_backward(result)\n",
    "        return result\n",
    "    \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        result, = ctx.saved_tensors\n",
    "        return grad_output * result\n",
    "\n",
    "# Use it by calling the apply method:\n",
    "x = torch.arange(4)\n",
    "output = Exp.apply(x)\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you still don't believe autograd works, here's something that I think will change your mind --- we're going to compute the derivative of an unnecessarily complicated function:\n",
    "\n",
    "$$ y(x) = \\sum_{x_i} e^{0.001 x_i^2} + \\sin(x_i^3) \\cdot \\log(x_i)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def complicated_func(X):\n",
    "    return torch.sum(torch.exp(0.001 * X ** 2) + torch.sin(X ** 3) * torch.log(X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.arange(1, 10, 0.1, dtype=torch.float, requires_grad=True)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = complicated_func(x)\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concepts not covered in this lecture\n",
    "\n",
    "PyTorch's `autograd` is a very powerfull tool. For instance, it can calculate the Jacobian and Hessian of any given function! Here is a list of more advanced things that you can accomplish with `autograd`:\n",
    "\n",
    "- Vector-Jacobian products for non-scalar outputs (e.g., when `y` is a vector)\n",
    "- Compute Jacobian and Hessian\n",
    "- Retain the computation graph (useful for inspecting gradients inside a model)\n",
    "- Sparse gradients\n",
    "- Register and remove hooks (useful for saving gradients)\n",
    "- How to set up user-designed `Function`s properly\n",
    "- Numerical gradient checking\n",
    "\n",
    "\n",
    "More info: https://pytorch.org/docs/stable/autograd.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The interaction of `autograd` with `nn.Module`s and `nn.Parameters`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next notebook we will see how to build a linear regression model using PyTorch's `nn.Module`. You will see that you don't need to worry about gradients when using `nn.Module` and `nn.Parameter`. This is because they automatically keep track of gradients for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# w.x + b\n",
    "lin = torch.nn.Linear(2, 1, bias=True)  # nn.Linear is a nn.Module\n",
    "lin.weight  # lin.weight is a nn.Parameter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(lin.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <b>Exercise:</b> Derive the gradient \n",
    "    <br><br>\n",
    "    $$\n",
    "    \\dfrac{\\partial \\big[\\sum_{x_i} e^{0.001 x_i^2} + \\sin(x_i^3) \\cdot \\log(x_i)\\big]}{\\partial x}\n",
    "    $$\n",
    "    <br>\n",
    "    and make a function that computes it. Check that it gives the same output as `x.grad` in our previous example.\n",
    "</center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}