cs231n-deep-learning-for-computer-vision/2023/assignments/a2/additional-notes/007_convolution_backward.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# stride = 1, pad = 0\n",
    "\n",
    "## Forward\n",
    "\n",
    "The forward pass of layer $l$: $Z^{[l]} = A^{[l]}*W^{[l]} + \\mathbf{b}^{[l]}$, when ignore $\\mathbf{b}$ for clearity.\n",
    "\n",
    "$$\n",
    "Z^{[l]} = A^{[l]}*W^{[l]} \\tag{F1}\n",
    "$$\n",
    "\n",
    "or expanded as:\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " a_{00} & a_{01} & a_{02} & a_{03} & a_{04} \\\\\n",
    " a_{10} & a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
    " a_{20} & a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
    " a_{30} & a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
    " a_{40} & a_{41} & a_{42} & a_{43} & a_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " w_{00} & w_{01} & w_{02} \\\\\n",
    " w_{10} & w_{11} & w_{12} \\\\\n",
    " w_{20} & w_{21} & w_{22} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " z_{00} & z_{01} & z_{02} \\\\\n",
    " z_{10} & z_{11} & z_{12} \\\\\n",
    " z_{20} & z_{21} & z_{22} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{cases}\n",
    "    z_{00} = a_{00}w_{00} + a_{01}w_{01} + a_{02}w_{02} + a_{10}w_{10} + a_{11}w_{11} + a_{12}w_{12} + a_{20}w_{20} + a_{21}w_{21} + a_{22}w_{22} \\\\\n",
    "    z_{01} = a_{01}w_{00} + a_{02}w_{01} + a_{03}w_{02} + a_{11}w_{10} + a_{12}w_{11} + a_{13}w_{12} + a_{21}w_{20} + a_{22}w_{21} + a_{23}w_{22} \\\\\n",
    "    z_{02} = a_{02}w_{00} + a_{03}w_{01} + a_{04}w_{02} + a_{12}w_{10} + a_{13}w_{11} + a_{14}w_{12} + a_{22}w_{20} + a_{23}w_{21} + a_{24}w_{22} \\\\\n",
    "    z_{10} = a_{10}w_{00} + a_{11}w_{01} + a_{12}w_{02} + a_{20}w_{10} + a_{21}w_{11} + a_{22}w_{12} + a_{30}w_{20} + a_{31}w_{21} + a_{32}w_{22} \\\\\n",
    "    z_{11} = a_{11}w_{00} + a_{12}w_{01} + a_{13}w_{02} + a_{21}w_{10} + a_{22}w_{11} + a_{23}w_{12} + a_{31}w_{20} + a_{32}w_{21} + a_{33}w_{22} \\\\\n",
    "    z_{12} = a_{12}w_{00} + a_{13}w_{01} + a_{14}w_{02} + a_{22}w_{10} + a_{23}w_{11} + a_{24}w_{12} + a_{32}w_{20} + a_{33}w_{21} + a_{34}w_{22} \\\\\n",
    "    z_{20} = a_{20}w_{00} + a_{21}w_{01} + a_{22}w_{02} + a_{30}w_{10} + a_{31}w_{11} + a_{32}w_{12} + a_{40}w_{20} + a_{41}w_{21} + a_{42}w_{22} \\\\\n",
    "    z_{21} = a_{21}w_{00} + a_{22}w_{01} + a_{23}w_{02} + a_{31}w_{10} + a_{32}w_{11} + a_{33}w_{12} + a_{41}w_{20} + a_{42}w_{21} + a_{43}w_{22} \\\\\n",
    "    z_{22} = a_{22}w_{00} + a_{23}w_{01} + a_{24}w_{02} + a_{32}w_{10} + a_{33}w_{11} + a_{34}w_{12} + a_{42}w_{20} + a_{43}w_{21} + a_{44}w_{22} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "## Backward\n",
    "\n",
    "### The weights gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & \\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & \\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & \\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{cases}\n",
    "    \\delta{w_{00}} = a_{00}\\delta{z_{00}} + a_{01}\\delta{z_{01}} + a_{02}\\delta{z_{02}} + a_{10}\\delta{z_{10}} + a_{11}\\delta{z_{11}} + a_{12}\\delta{z_{12}} + a_{20}\\delta{z_{20}} + a_{21}\\delta{z_{21}} + a_{22}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{01}} = a_{01}\\delta{z_{00}} + a_{02}\\delta{z_{01}} + a_{03}\\delta{z_{02}} + a_{11}\\delta{z_{10}} + a_{12}\\delta{z_{11}} + a_{13}\\delta{z_{12}} + a_{21}\\delta{z_{20}} + a_{22}\\delta{z_{21}} + a_{23}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{02}} = a_{02}\\delta{z_{00}} + a_{03}\\delta{z_{01}} + a_{04}\\delta{z_{02}} + a_{12}\\delta{z_{10}} + a_{13}\\delta{z_{11}} + a_{14}\\delta{z_{12}} + a_{22}\\delta{z_{20}} + a_{23}\\delta{z_{21}} + a_{24}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{10}} = a_{10}\\delta{z_{00}} + a_{11}\\delta{z_{01}} + a_{12}\\delta{z_{02}} + a_{20}\\delta{z_{10}} + a_{21}\\delta{z_{11}} + a_{22}\\delta{z_{12}} + a_{30}\\delta{z_{20}} + a_{31}\\delta{z_{21}} + a_{32}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{11}} = a_{11}\\delta{z_{00}} + a_{12}\\delta{z_{01}} + a_{13}\\delta{z_{02}} + a_{21}\\delta{z_{10}} + a_{22}\\delta{z_{11}} + a_{23}\\delta{z_{12}} + a_{31}\\delta{z_{20}} + a_{32}\\delta{z_{21}} + a_{33}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{12}} = a_{12}\\delta{z_{00}} + a_{13}\\delta{z_{01}} + a_{14}\\delta{z_{02}} + a_{22}\\delta{z_{10}} + a_{23}\\delta{z_{11}} + a_{24}\\delta{z_{12}} + a_{32}\\delta{z_{20}} + a_{33}\\delta{z_{21}} + a_{34}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{20}} = a_{20}\\delta{z_{00}} + a_{21}\\delta{z_{01}} + a_{22}\\delta{z_{02}} + a_{30}\\delta{z_{10}} + a_{31}\\delta{z_{11}} + a_{32}\\delta{z_{12}} + a_{40}\\delta{z_{20}} + a_{41}\\delta{z_{21}} + a_{42}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{21}} = a_{21}\\delta{z_{00}} + a_{22}\\delta{z_{01}} + a_{23}\\delta{z_{02}} + a_{31}\\delta{z_{10}} + a_{32}\\delta{z_{11}} + a_{33}\\delta{z_{12}} + a_{41}\\delta{z_{20}} + a_{42}\\delta{z_{21}} + a_{43}\\delta{z_{22}} \\\\\n",
    "    \\delta{w_{22}} = a_{22}\\delta{z_{00}} + a_{23}\\delta{z_{01}} + a_{24}\\delta{z_{02}} + a_{32}\\delta{z_{10}} + a_{33}\\delta{z_{11}} + a_{34}\\delta{z_{12}} + a_{42}\\delta{z_{20}} + a_{43}\\delta{z_{21}} + a_{44}\\delta{z_{22}} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & z\\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & z\\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & z\\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " a_{00} & a_{01} & a_{02} & a_{03} & a_{04} \\\\\n",
    " a_{10} & a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
    " a_{20} & a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
    " a_{30} & a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
    " a_{40} & a_{41} & a_{42} & a_{43} & a_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " \\delta{z_{00}} & \\delta{z_{01}} & \\delta{z_{02}} \\\\\n",
    " \\delta{z_{10}} & \\delta{z_{11}} & \\delta{z_{12}} \\\\\n",
    " \\delta{z_{20}} & \\delta{z_{21}} & \\delta{z_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}} = A^{[l]} * \\delta{Z^{[l]}} \\tag{B1}\n",
    "$$\n",
    "\n",
    "### The activations gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{cases}\n",
    "    \\delta{a_{00}} = w_{00}\\partial{z_{00}} \\\\\n",
    "    \\delta{a_{01}} = w_{01}\\partial{z_{00}} + w_{00}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{02}} = w_{02}\\partial{z_{00}} + w_{01}\\partial{z_{01}} + w_{00}\\partial{z_{02}} \\\\\n",
    "    \\delta{a_{03}} = w_{02}\\partial{z_{01}} + w_{01}\\partial{z_{02}} \\\\\n",
    "    \\delta{a_{04}} = w_{02}\\partial{z_{02}} \\\\\n",
    "    \\delta{a_{10}} = w_{10}\\partial{z_{00}} + w_{00}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{11}} = w_{11}\\partial{z_{00}} + w_{10}\\partial{z_{01}} + w_{01}\\partial{z_{10}} + w_{00}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{12}} = w_{12}\\partial{z_{00}} + w_{11}\\partial{z_{01}} + w_{10}\\partial{z_{02}} + w_{02}\\partial{z_{10}} + w_{01}\\partial{z_{11}} + w_{00}\\partial{z_{12}} \\\\\n",
    "    \\delta{a_{13}} = w_{12}\\partial{z_{01}} + w_{11}\\partial{z_{02}} + w_{02}\\partial{z_{11}} + w_{01}\\partial{z_{12}} \\\\\n",
    "    \\delta{a_{14}} = w_{12}\\partial{z_{02}} + w_{02}\\partial{z_{12}} \\\\\n",
    "    \\delta{a_{20}} = w_{20}\\partial{z_{00}} + w_{10}\\partial{z_{10}} + w_{00}\\partial{z_{20}} \\\\\n",
    "    \\delta{a_{21}} = w_{21}\\partial{z_{00}} + w_{20}\\partial{z_{01}} + w_{11}\\partial{z_{10}} + w_{10}\\partial{z_{11}} + w_{01}\\partial{z_{20}} + w_{00}\\partial{z_{21}} \\\\\n",
    "    \\delta{a_{22}} = w_{22}\\partial{z_{00}} + w_{21}\\partial{z_{01}} + w_{20}\\partial{z_{02}} + w_{12}\\partial{z_{10}} + w_{11}\\partial{z_{11}} + w_{10}\\partial{z_{12}} + w_{02}\\partial{z_{20}} + w_{01}\\partial{z_{21}} + w_{00}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{23}} = w_{22}\\partial{z_{01}} + w_{21}\\partial{z_{02}} + w_{12}\\partial{z_{11}} + w_{11}\\partial{z_{12}} + w_{02}\\partial{z_{21}} + w_{01}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{24}} = w_{22}\\partial{z_{02}} + w_{12}\\partial{z_{12}} + w_{02}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{30}} = w_{20}\\partial{z_{10}} + w_{10}\\partial{z_{20}} \\\\\n",
    "    \\delta{a_{31}} = w_{21}\\partial{z_{10}} + w_{20}\\partial{z_{11}} + w_{11}\\partial{z_{20}} + w_{10}\\partial{z_{21}} \\\\\n",
    "    \\delta{a_{32}} = w_{22}\\partial{z_{10}} + w_{21}\\partial{z_{11}} + w_{20}\\partial{z_{12}} + w_{12}\\partial{z_{20}} + w_{11}\\partial{z_{21}} + w_{10}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{33}} = w_{22}\\partial{z_{11}} + w_{21}\\partial{z_{12}} + w_{12}\\partial{z_{21}} + w_{11}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{34}} = w_{22}\\partial{z_{12}} + w_{12}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{40}} = w_{20}\\partial{z_{20}} \\\\\n",
    "    \\delta{a_{41}} = w_{21}\\partial{z_{20}} + w_{20}\\partial{z_{21}} \\\\\n",
    "    \\delta{a_{42}} = w_{22}\\partial{z_{20}} + w_{21}\\partial{z_{21}} + w_{20}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{43}} = w_{22}\\partial{z_{21}} + w_{21}\\partial{z_{22}} \\\\\n",
    "    \\delta{a_{44}} = w_{22}\\partial{z_{22}} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & \\delta{z_{00}} & \\delta{z_{01}} & \\delta{z_{02}} & 0              & 0               \\\\\n",
    "    0              & 0              & \\delta{z_{10}} & \\delta{z_{11}} & \\delta{z_{12}} & 0              & 0               \\\\\n",
    "    0              & 0              & \\delta{z_{20}} & \\delta{z_{21}} & \\delta{z_{22}} & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " w_{22} & w_{21} & w_{20} \\\\\n",
    " w_{12} & w_{11} & w_{10} \\\\\n",
    " w_{02} & w_{01} & w_{00} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}} = PAD(\\delta{Z^{[l]}}) * ROT180(W^{[l]}) \\tag{B2}\n",
    "$$"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b54cbabe928d24ba"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# stride = 1, pad = 1\n",
    "\n",
    "## Forward\n",
    "\n",
    "The forward pass of layer $l$: $Z^{[l]} = A^{[l]}*W^{[l]} + \\mathbf{b}^{[l]}$, when ignore $\\mathbf{b}$ for clearity.\n",
    "\n",
    "$$\n",
    "Z^{[l]} = A^{[l]}*W^{[l]} \\tag{F1}\n",
    "$$\n",
    "\n",
    "or expanded as:\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " 0      & 0      & 0      & 0      & 0      & 0      & 0      \\\\\n",
    " 0      & a_{00} & a_{01} & a_{02} & a_{03} & a_{04} & 0      \\\\\n",
    " 0      & a_{10} & a_{11} & a_{12} & a_{13} & a_{14} & 0      \\\\\n",
    " 0      & a_{20} & a_{21} & a_{22} & a_{23} & a_{24} & 0      \\\\\n",
    " 0      & a_{30} & a_{31} & a_{32} & a_{33} & a_{34} & 0      \\\\\n",
    " 0      & a_{40} & a_{41} & a_{42} & a_{43} & a_{44} & 0      \\\\\n",
    " 0      & 0      & 0      & 0      & 0      & 0      & 0      \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " w_{00} & w_{01} & w_{02} \\\\\n",
    " w_{10} & w_{11} & w_{12} \\\\\n",
    " w_{20} & w_{21} & w_{22} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " z_{00} & z_{01} & z_{02} & z_{03} & z_{04} \\\\\n",
    " z_{10} & z_{11} & z_{12} & z_{13} & z_{14} \\\\\n",
    " z_{20} & z_{21} & z_{22} & z_{23} & z_{24} \\\\\n",
    " z_{30} & z_{31} & z_{32} & z_{33} & z_{34} \\\\\n",
    " z_{40} & z_{41} & z_{42} & z_{43} & z_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "    \\begin{cases}\n",
    "    z_{00} = \\ \\  0w_{00} + \\ \\  0w_{01} + \\ \\  0w_{02} + \\ \\  0w_{10} + a_{00}w_{11} + a_{01}w_{12} + \\ \\  0w_{20} + a_{10}w_{21} + a_{11}w_{22} \\\\\n",
    "    z_{01} = \\ \\  0w_{00} + \\ \\  0w_{01} + \\ \\  0w_{02} + a_{00}w_{10} + a_{01}w_{11} + a_{02}w_{12} + a_{10}w_{20} + a_{11}w_{21} + a_{12}w_{22} \\\\\n",
    "    z_{02} = \\ \\  0w_{00} + \\ \\  0w_{01} + \\ \\  0w_{02} + a_{01}w_{10} + a_{02}w_{11} + a_{03}w_{12} + a_{11}w_{20} + a_{12}w_{21} + a_{13}w_{22} \\\\\n",
    "    z_{03} = \\ \\  0w_{00} + \\ \\  0w_{01} + \\ \\  0w_{02} + a_{02}w_{10} + a_{03}w_{11} + a_{04}w_{12} + a_{12}w_{20} + a_{13}w_{21} + a_{14}w_{22} \\\\\n",
    "    z_{04} = \\ \\  0w_{00} + \\ \\  0w_{01} + \\ \\  0w_{02} + a_{03}w_{10} + a_{04}w_{11} + \\ \\  0w_{12} + a_{13}w_{20} + a_{14}w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{10} = \\ \\  0w_{00} + a_{00}w_{01} + a_{01}w_{02} + \\ \\  0w_{10} + a_{10}w_{11} + a_{11}w_{12} + \\ \\  0w_{20} + a_{20}w_{21} + a_{21}w_{22} \\\\\n",
    "    z_{11} = a_{00}w_{00} + a_{01}w_{01} + a_{02}w_{02} + a_{10}w_{10} + a_{11}w_{11} + a_{12}w_{12} + a_{20}w_{20} + a_{21}w_{21} + a_{22}w_{22} \\\\\n",
    "    z_{12} = a_{01}w_{00} + a_{02}w_{01} + a_{03}w_{02} + a_{11}w_{10} + a_{12}w_{11} + a_{13}w_{12} + a_{21}w_{20} + a_{22}w_{21} + a_{23}w_{22} \\\\\n",
    "    z_{13} = a_{02}w_{00} + a_{03}w_{01} + a_{04}w_{02} + a_{12}w_{10} + a_{13}w_{11} + a_{14}w_{12} + a_{22}w_{20} + a_{23}w_{21} + a_{24}w_{22} \\\\\n",
    "    z_{14} = a_{03}w_{00} + a_{04}w_{01} + \\ \\  0w_{02} + a_{13}w_{10} + a_{14}w_{11} + \\ \\  0w_{12} + a_{23}w_{20} + a_{24}w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{20} = \\ \\  0w_{00} + a_{10}w_{01} + a_{11}w_{02} + \\ \\  0w_{10} + a_{20}w_{11} + a_{21}w_{12} + \\ \\  0w_{20} + a_{30}w_{21} + a_{31}w_{22} \\\\\n",
    "    z_{21} = a_{10}w_{00} + a_{11}w_{01} + a_{12}w_{02} + a_{20}w_{10} + a_{21}w_{11} + a_{22}w_{12} + a_{30}w_{20} + a_{31}w_{21} + a_{32}w_{22} \\\\\n",
    "    z_{22} = a_{11}w_{00} + a_{12}w_{01} + a_{13}w_{02} + a_{21}w_{10} + a_{22}w_{11} + a_{23}w_{12} + a_{31}w_{20} + a_{32}w_{21} + a_{33}w_{22} \\\\\n",
    "    z_{23} = a_{12}w_{00} + a_{13}w_{01} + a_{14}w_{02} + a_{22}w_{10} + a_{23}w_{11} + a_{24}w_{12} + a_{32}w_{20} + a_{33}w_{21} + a_{34}w_{22} \\\\\n",
    "    z_{24} = a_{13}w_{00} + a_{14}w_{01} + \\ \\  0w_{02} + a_{23}w_{10} + a_{24}w_{11} + \\ \\  0w_{12} + a_{33}w_{20} + a_{34}w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{30} = \\ \\  0w_{00} + a_{20}w_{01} + a_{21}w_{02} + \\ \\  0w_{10} + a_{30}w_{11} + a_{31}w_{12} + \\ \\  0w_{20} + a_{40}w_{21} + a_{41}w_{22} \\\\\n",
    "    z_{31} = a_{20}w_{00} + a_{21}w_{01} + a_{22}w_{02} + a_{30}w_{10} + a_{31}w_{11} + a_{32}w_{12} + a_{40}w_{20} + a_{41}w_{21} + a_{42}w_{22} \\\\\n",
    "    z_{32} = a_{21}w_{00} + a_{22}w_{01} + a_{23}w_{02} + a_{31}w_{10} + a_{32}w_{11} + a_{33}w_{12} + a_{41}w_{20} + a_{42}w_{21} + a_{43}w_{22} \\\\\n",
    "    z_{33} = a_{22}w_{00} + a_{23}w_{01} + a_{24}w_{02} + a_{32}w_{10} + a_{33}w_{11} + a_{34}w_{12} + a_{42}w_{20} + a_{43}w_{21} + a_{44}w_{22} \\\\\n",
    "    z_{34} = a_{23}w_{00} + a_{24}w_{01} + \\ \\  0w_{02} + a_{33}w_{10} + a_{34}w_{11} + \\ \\  0w_{12} + a_{43}w_{20} + a_{44}w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{40} = \\ \\  0w_{00} + a_{30}w_{01} + a_{31}w_{02} + \\ \\  0w_{10} + a_{40}w_{11} + a_{41}w_{12} + \\ \\  0w_{20} + \\ \\  0w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{41} = a_{30}w_{00} + a_{31}w_{01} + a_{32}w_{02} + a_{40}w_{10} + a_{41}w_{11} + a_{42}w_{12} + \\ \\  0w_{20} + \\ \\  0w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{42} = a_{31}w_{00} + a_{32}w_{01} + a_{33}w_{02} + a_{41}w_{10} + a_{42}w_{11} + a_{43}w_{12} + \\ \\  0w_{20} + \\ \\  0w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{43} = a_{32}w_{00} + a_{33}w_{01} + a_{34}w_{02} + a_{42}w_{10} + a_{43}w_{11} + a_{44}w_{12} + \\ \\  0w_{20} + \\ \\  0w_{21} + \\ \\  0w_{22} \\\\\n",
    "    z_{44} = a_{33}w_{00} + a_{34}w_{01} + \\ \\  0w_{02} + a_{43}w_{10} + a_{44}w_{11} + \\ \\  0w_{12} + \\ \\  0w_{20} + \\ \\  0w_{21} + \\ \\  0w_{22} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "## Backward\n",
    "\n",
    "### The weights gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & \\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & \\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & \\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "    \\begin{cases}\n",
    "    \\delta{w_{00}} = \\ \\  0\\delta{z_{00}} + \\ \\  0\\delta{z_{01}} + \\ \\  0\\delta{z_{02}} + \\ \\  0\\delta{z_{03}} + \\ \\  0\\delta{z_{04}} + \\ \\  0\\delta{z_{10}} + a_{00}\\delta{z_{11}} + a_{01}\\delta{z_{12}} + a_{02}\\delta{z_{13}} + a_{03}\\delta{z_{14}} + \\ \\  0\\delta{z_{20}} + a_{10}\\delta{z_{21}} + a_{11}\\delta{z_{22}} + a_{12}\\delta{z_{23}} + a_{13}\\delta{z_{24}} + \\ \\  0\\delta{z_{30}} + a_{20}\\delta{z_{31}} + a_{21}\\delta{z_{32}} + a_{22}\\delta{z_{33}} + a_{23}\\delta{z_{34}} + \\ \\  0\\delta{z_{40}} + a_{30}\\delta{z_{41}} + a_{31}\\delta{z_{42}} + a_{32}\\delta{z_{43}} + a_{33}\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{01}} = \\ \\  0\\delta{z_{00}} + \\ \\  0\\delta{z_{01}} + \\ \\  0\\delta{z_{02}} + \\ \\  0\\delta{z_{03}} + \\ \\  0\\delta{z_{04}} + a_{00}\\delta{z_{10}} + a_{01}\\delta{z_{11}} + a_{02}\\delta{z_{12}} + a_{03}\\delta{z_{13}} + a_{04}\\delta{z_{14}} + a_{10}\\delta{z_{20}} + a_{11}\\delta{z_{21}} + a_{12}\\delta{z_{22}} + a_{13}\\delta{z_{23}} + a_{14}\\delta{z_{24}} + a_{20}\\delta{z_{30}} + a_{21}\\delta{z_{31}} + a_{22}\\delta{z_{32}} + a_{23}\\delta{z_{33}} + a_{24}\\delta{z_{34}} + a_{30}\\delta{z_{40}} + a_{31}\\delta{z_{41}} + a_{32}\\delta{z_{42}} + a_{33}\\delta{z_{43}} + a_{34}\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{02}} = \\ \\  0\\delta{z_{00}} + \\ \\  0\\delta{z_{01}} + \\ \\  0\\delta{z_{02}} + \\ \\  0\\delta{z_{03}} + \\ \\  0\\delta{z_{04}} + a_{01}\\delta{z_{10}} + a_{02}\\delta{z_{11}} + a_{03}\\delta{z_{12}} + a_{04}\\delta{z_{13}} + \\ \\  0\\delta{z_{14}} + a_{11}\\delta{z_{20}} + a_{12}\\delta{z_{21}} + a_{13}\\delta{z_{22}} + a_{14}\\delta{z_{23}} + \\ \\  0\\delta{z_{24}} + a_{21}\\delta{z_{30}} + a_{22}\\delta{z_{31}} + a_{23}\\delta{z_{32}} + a_{24}\\delta{z_{33}} + \\ \\  0\\delta{z_{34}} + a_{31}\\delta{z_{40}} + a_{32}\\delta{z_{41}} + a_{33}\\delta{z_{42}} + a_{34}\\delta{z_{43}} + \\ \\  0\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{10}} = \\ \\  0\\delta{z_{00}} + a_{00}\\delta{z_{01}} + a_{01}\\delta{z_{02}} + a_{02}\\delta{z_{03}} + a_{03}\\delta{z_{04}} + \\ \\  0\\delta{z_{10}} + a_{10}\\delta{z_{11}} + a_{11}\\delta{z_{12}} + a_{12}\\delta{z_{13}} + a_{13}\\delta{z_{14}} + \\ \\  0\\delta{z_{20}} + a_{20}\\delta{z_{21}} + a_{21}\\delta{z_{22}} + a_{22}\\delta{z_{23}} + a_{23}\\delta{z_{24}} + \\ \\  0\\delta{z_{30}} + a_{30}\\delta{z_{31}} + a_{31}\\delta{z_{32}} + a_{32}\\delta{z_{33}} + a_{33}\\delta{z_{34}} + \\ \\  0\\delta{z_{40}} + a_{40}\\delta{z_{41}} + a_{41}\\delta{z_{42}} + a_{42}\\delta{z_{43}} + a_{43}\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{11}} = a_{00}\\delta{z_{00}} + a_{01}\\delta{z_{01}} + a_{02}\\delta{z_{02}} + a_{03}\\delta{z_{03}} + a_{04}\\delta{z_{04}} + a_{10}\\delta{z_{10}} + a_{11}\\delta{z_{11}} + a_{12}\\delta{z_{12}} + a_{13}\\delta{z_{13}} + a_{14}\\delta{z_{14}} + a_{20}\\delta{z_{20}} + a_{21}\\delta{z_{21}} + a_{22}\\delta{z_{22}} + a_{23}\\delta{z_{23}} + a_{24}\\delta{z_{24}} + a_{30}\\delta{z_{30}} + a_{31}\\delta{z_{31}} + a_{32}\\delta{z_{32}} + a_{33}\\delta{z_{33}} + a_{34}\\delta{z_{34}} + a_{40}\\delta{z_{40}} + a_{41}\\delta{z_{41}} + a_{42}\\delta{z_{42}} + a_{43}\\delta{z_{43}} + a_{44}\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{12}} = a_{01}\\delta{z_{00}} + a_{02}\\delta{z_{01}} + a_{03}\\delta{z_{02}} + a_{04}\\delta{z_{03}} + \\ \\  0\\delta{z_{04}} + a_{11}\\delta{z_{10}} + a_{12}\\delta{z_{11}} + a_{13}\\delta{z_{12}} + a_{14}\\delta{z_{13}} + \\ \\  0\\delta{z_{14}} + a_{21}\\delta{z_{20}} + a_{22}\\delta{z_{21}} + a_{23}\\delta{z_{22}} + a_{24}\\delta{z_{23}} + \\ \\  0\\delta{z_{24}} + a_{31}\\delta{z_{30}} + a_{32}\\delta{z_{31}} + a_{33}\\delta{z_{32}} + a_{34}\\delta{z_{33}} + \\ \\  0\\delta{z_{34}} + a_{41}\\delta{z_{40}} + a_{42}\\delta{z_{41}} + a_{43}\\delta{z_{42}} + a_{44}\\delta{z_{43}} + \\ \\  0\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{20}} = \\ \\  0\\delta{z_{00}} + a_{10}\\delta{z_{01}} + a_{11}\\delta{z_{02}} + a_{12}\\delta{z_{03}} + a_{13}\\delta{z_{04}} + \\ \\  0\\delta{z_{10}} + a_{20}\\delta{z_{11}} + a_{21}\\delta{z_{12}} + a_{22}\\delta{z_{13}} + a_{23}\\delta{z_{14}} + \\ \\  0\\delta{z_{20}} + a_{30}\\delta{z_{21}} + a_{31}\\delta{z_{22}} + a_{32}\\delta{z_{23}} + a_{33}\\delta{z_{24}} + \\ \\  0\\delta{z_{30}} + a_{40}\\delta{z_{31}} + a_{41}\\delta{z_{32}} + a_{42}\\delta{z_{33}} + a_{43}\\delta{z_{34}} + \\ \\  0\\delta{z_{40}} + \\ \\  0\\delta{z_{41}} + \\ \\  0\\delta{z_{42}} + \\ \\  0\\delta{z_{43}} + \\ \\  0\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{21}} = a_{10}\\delta{z_{00}} + a_{11}\\delta{z_{01}} + a_{12}\\delta{z_{02}} + a_{13}\\delta{z_{03}} + a_{14}\\delta{z_{04}} + a_{20}\\delta{z_{10}} + a_{21}\\delta{z_{11}} + a_{22}\\delta{z_{12}} + a_{23}\\delta{z_{13}} + a_{24}\\delta{z_{14}} + a_{30}\\delta{z_{20}} + a_{31}\\delta{z_{21}} + a_{32}\\delta{z_{22}} + a_{33}\\delta{z_{23}} + a_{34}\\delta{z_{24}} + a_{40}\\delta{z_{30}} + a_{41}\\delta{z_{31}} + a_{42}\\delta{z_{32}} + a_{43}\\delta{z_{33}} + a_{44}\\delta{z_{34}} + \\ \\  0\\delta{z_{40}} + \\ \\  0\\delta{z_{41}} + \\ \\  0\\delta{z_{42}} + \\ \\  0\\delta{z_{43}} + \\ \\  0\\delta{z_{44}} \\\\\n",
    "    \\delta{w_{22}} = a_{11}\\delta{z_{00}} + a_{12}\\delta{z_{01}} + a_{13}\\delta{z_{02}} + a_{14}\\delta{z_{03}} + \\ \\  0\\delta{z_{04}} + a_{21}\\delta{z_{10}} + a_{22}\\delta{z_{11}} + a_{23}\\delta{z_{12}} + a_{24}\\delta{z_{13}} + \\ \\  0\\delta{z_{14}} + a_{31}\\delta{z_{20}} + a_{32}\\delta{z_{21}} + a_{33}\\delta{z_{22}} + a_{34}\\delta{z_{23}} + \\ \\  0\\delta{z_{24}} + a_{41}\\delta{z_{30}} + a_{42}\\delta{z_{31}} + a_{43}\\delta{z_{32}} + a_{44}\\delta{z_{33}} + \\ \\  0\\delta{z_{34}} + \\ \\  0\\delta{z_{40}} + \\ \\  0\\delta{z_{41}} + \\ \\  0\\delta{z_{42}} + \\ \\  0\\delta{z_{43}} + \\ \\  0\\delta{z_{44}} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & z\\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & z\\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & z\\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " 0      & 0      & 0      & 0      & 0      & 0      & 0      \\\\\n",
    " 0      & a_{00} & a_{01} & a_{02} & a_{03} & a_{04} & 0      \\\\\n",
    " 0      & a_{10} & a_{11} & a_{12} & a_{13} & a_{14} & 0      \\\\\n",
    " 0      & a_{20} & a_{21} & a_{22} & a_{23} & a_{24} & 0      \\\\\n",
    " 0      & a_{30} & a_{31} & a_{32} & a_{33} & a_{34} & 0      \\\\\n",
    " 0      & a_{40} & a_{41} & a_{42} & a_{43} & a_{44} & 0      \\\\\n",
    " 0      & 0      & 0      & 0      & 0      & 0      & 0      \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    "    \\delta{z_{00}} & \\delta{z_{01}} & \\delta{z_{02}} & \\delta{z_{03}} & \\delta{z_{04}} \\\\\n",
    "    \\delta{z_{10}} & \\delta{z_{11}} & \\delta{z_{12}} & \\delta{z_{13}} & \\delta{z_{14}} \\\\\n",
    "    \\delta{z_{20}} & \\delta{z_{21}} & \\delta{z_{22}} & \\delta{z_{23}} & \\delta{z_{24}} \\\\\n",
    "    \\delta{z_{30}} & \\delta{z_{31}} & \\delta{z_{32}} & \\delta{z_{33}} & \\delta{z_{34}} \\\\\n",
    "    \\delta{z_{40}} & \\delta{z_{41}} & \\delta{z_{42}} & \\delta{z_{43}} & \\delta{z_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}} = A^{[l]} * \\delta{Z^{[l]}} \\tag{B1}\n",
    "$$\n",
    "\n",
    "### The activations gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    0              & 0              & 0              & 0              & 0              & 0               & 0              \\\\\n",
    "    0              & \\delta{z_{00}} & \\delta{z_{01}} & \\delta{z_{02}} & \\delta{z_{03}} & \\delta{z_{04}}  & 0              \\\\\n",
    "    0              & \\delta{z_{10}} & \\delta{z_{11}} & \\delta{z_{12}} & \\delta{z_{13}} & \\delta{z_{14}}  & 0              \\\\\n",
    "    0              & \\delta{z_{20}} & \\delta{z_{21}} & \\delta{z_{22}} & \\delta{z_{23}} & \\delta{z_{24}}  & 0              \\\\\n",
    "    0              & \\delta{z_{30}} & \\delta{z_{31}} & \\delta{z_{32}} & \\delta{z_{33}} & \\delta{z_{34}}  & 0              \\\\\n",
    "    0              & \\delta{z_{40}} & \\delta{z_{41}} & \\delta{z_{42}} & \\delta{z_{43}} & \\delta{z_{44}}  & 0              \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0               & 0              \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " w_{22} & w_{21} & w_{20} \\\\\n",
    " w_{12} & w_{11} & w_{10} \\\\\n",
    " w_{02} & w_{01} & w_{00} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}} = PAD(\\delta{Z^{[l]}}) * ROT180(W^{[l]}) \\tag{B2}\n",
    "$$"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "c9b51abc6cb5de99"
  },
  {
   "cell_type": "markdown",
   "id": "579ebaab-19eb-45dc-89e1-0098e449716a",
   "metadata": {},
   "source": [
    "# stride = 2, pad = 0\n",
    "\n",
    "## Forward\n",
    "\n",
    "The forward pass of layer $l$: $Z^{[l]} = A^{[l]}*W^{[l]} + \\mathbf{b}^{[l]}$, when ignore $\\mathbf{b}$ for clearity.\n",
    "\n",
    "$$\n",
    "Z^{[l]} = A^{[l]}*W^{[l]} \\tag{F1}\n",
    "$$\n",
    "\n",
    "or expanded as:\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    "    a_{00} & a_{01} & a_{02} & a_{03} & a_{04} \\\\\n",
    "    a_{10} & a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
    "    a_{20} & a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
    "    a_{30} & a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
    "    a_{40} & a_{41} & a_{42} & a_{43} & a_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    "    w_{00} & w_{01} & w_{02} \\\\\n",
    "    w_{10} & w_{11} & w_{12} \\\\\n",
    "    w_{20} & w_{21} & w_{22} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    z_{00} & z_{01} \\\\\n",
    "    z_{10} & z_{11} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    " \\begin{cases}\n",
    "    z_{00} = a_{00}w_{00} + a_{01}w_{01} + a_{02}w_{02} + a_{10}w_{10} + a_{11}w_{11} + a_{12}w_{12} + a_{20}w_{20} + a_{21}w_{21} + a_{22}w_{22} \\\\\n",
    "    z_{01} = a_{02}w_{00} + a_{03}w_{01} + a_{04}w_{02} + a_{12}w_{10} + a_{13}w_{11} + a_{14}w_{12} + a_{22}w_{20} + a_{23}w_{21} + a_{24}w_{22} \\\\\n",
    "    z_{10} = a_{20}w_{00} + a_{21}w_{01} + a_{22}w_{02} + a_{30}w_{10} + a_{31}w_{11} + a_{32}w_{12} + a_{40}w_{20} + a_{41}w_{21} + a_{42}w_{22} \\\\\n",
    "    z_{11} = a_{22}w_{00} + a_{23}w_{01} + a_{24}w_{02} + a_{32}w_{10} + a_{33}w_{11} + a_{34}w_{12} + a_{42}w_{20} + a_{43}w_{21} + a_{44}w_{22} \\\\\n",
    " \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "## Backward\n",
    "\n",
    "### The weights gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & \\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & \\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & \\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{cases}\n",
    "    \\delta{w_{00}} = a_{00}\\delta{z_{00}} + a_{02}\\delta{z_{01}} + a_{20}\\delta{z_{10}} + a_{22}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{01}} = a_{01}\\delta{z_{00}} + a_{03}\\delta{z_{01}} + a_{21}\\delta{z_{10}} + a_{23}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{02}} = a_{02}\\delta{z_{00}} + a_{04}\\delta{z_{01}} + a_{22}\\delta{z_{10}} + a_{24}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{10}} = a_{10}\\delta{z_{00}} + a_{12}\\delta{z_{01}} + a_{30}\\delta{z_{10}} + a_{32}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{11}} = a_{11}\\delta{z_{00}} + a_{13}\\delta{z_{01}} + a_{31}\\delta{z_{10}} + a_{33}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{12}} = a_{12}\\delta{z_{00}} + a_{14}\\delta{z_{01}} + a_{32}\\delta{z_{10}} + a_{34}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{20}} = a_{20}\\delta{z_{00}} + a_{22}\\delta{z_{01}} + a_{40}\\delta{z_{10}} + a_{42}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{21}} = a_{21}\\delta{z_{00}} + a_{23}\\delta{z_{01}} + a_{41}\\delta{z_{10}} + a_{43}\\delta{z_{11}} \\\\\n",
    "    \\delta{w_{22}} = a_{22}\\delta{z_{00}} + a_{24}\\delta{z_{01}} + a_{42}\\delta{z_{10}} + a_{44}\\delta{z_{11}} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{w_{00}} & \\delta{w_{01}} & \\delta{w_{02}} \\\\\n",
    " \\delta{w_{10}} & \\delta{w_{11}} & \\delta{w_{12}} \\\\\n",
    " \\delta{w_{20}} & \\delta{w_{21}} & \\delta{w_{22}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " a_{00} & a_{01} & a_{02} & a_{03} & a_{04} \\\\\n",
    " a_{10} & a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
    " a_{20} & a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
    " a_{30} & a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
    " a_{40} & a_{41} & a_{42} & a_{43} & a_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    "    \\delta{z_{00}} & 0              & \\delta{z_{01}} \\\\\n",
    "    0              & 0              & 0              \\\\\n",
    "    \\delta{z_{10}} & 0              & \\delta{z_{11}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{W^{[l]}} = A^{[l]} * DIL(\\delta{Z^{[l]}}) \\tag{B1}\n",
    "$$\n",
    "\n",
    "### The activations gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{cases}\n",
    "    \\delta{a_{00}} = w_{00}\\partial{z_{00}} \\\\\n",
    "    \\delta{a_{01}} = w_{01}\\partial{z_{00}} \\\\\n",
    "    \\delta{a_{02}} = w_{02}\\partial{z_{00}} + w_{00}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{03}} = w_{01}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{04}} = w_{02}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{10}} = w_{10}\\partial{z_{00}} \\\\\n",
    "    \\delta{a_{11}} = w_{11}\\partial{z_{00}} \\\\\n",
    "    \\delta{a_{12}} = w_{12}\\partial{z_{00}} + w_{10}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{13}} = w_{11}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{14}} = w_{12}\\partial{z_{01}} \\\\\n",
    "    \\delta{a_{20}} = w_{20}\\partial{z_{00}} + w_{00}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{21}} = w_{21}\\partial{z_{00}} + w_{01}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{22}} = w_{22}\\partial{z_{00}} + w_{20}\\partial{z_{01}} + w_{02}\\partial{z_{10}} + w_{00}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{23}} = w_{21}\\partial{z_{01}} + w_{01}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{24}} = w_{22}\\partial{z_{01}} + w_{02}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{30}} = w_{10}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{31}} = w_{11}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{32}} = w_{12}\\partial{z_{10}} + w_{10}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{33}} = w_{11}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{34}} = w_{12}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{40}} = w_{20}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{41}} = w_{21}\\partial{z_{10}} \\\\\n",
    "    \\delta{a_{42}} = w_{22}\\partial{z_{10}} + w_{20}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{43}} = w_{21}\\partial{z_{11}} \\\\\n",
    "    \\delta{a_{44}} = w_{22}\\partial{z_{11}} \\\\\n",
    "    \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore,\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & \\delta{z_{00}} & 0              & \\delta{z_{01}} & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & \\delta{z_{10}} & 0              & \\delta{z_{11}} & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "    0              & 0              & 0              & 0              & 0              & 0              & 0               \\\\\n",
    "\\end{bmatrix}\n",
    "*\n",
    "\\begin{bmatrix}\n",
    " w_{22} & w_{21} & w_{20} \\\\\n",
    " w_{12} & w_{11} & w_{10} \\\\\n",
    " w_{02} & w_{01} & w_{00} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}} = PAD\\left(DIL(\\delta{Z^{[l]}})\\right) * ROT180(W^{[l]}) \\tag{B2}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2081dce3-94c8-4535-b103-6b57a1feca76",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Max Pooling 2D\n",
    "\n",
    "## Forward\n",
    "\n",
    "The forward pass of layer $l$: $Z^{[l]} = MAX\\left(WINDOWED(A^{[l]})\\right)$. where bold represents max value inside the sliding window.\n",
    "\n",
    "$$\n",
    "Z^{[l]} = MAX\\left(WINDOWED(A^{[l]})\\right) \\tag{F1}\n",
    "$$\n",
    "\n",
    "or expanded as:\n",
    "\n",
    "$$\n",
    "MAX\\left(\n",
    "\\begin{bmatrix}\n",
    " a_{00} & a_{01} & a_{02} & \\boldsymbol{a_{03}} & a_{04} \\\\\n",
    " a_{10} & \\boldsymbol{a_{11}} & a_{12} & a_{13} & a_{14} \\\\\n",
    " a_{20} & a_{21} & \\boldsymbol{a_{22}} & a_{23} & a_{24} \\\\\n",
    " \\boldsymbol{a_{30}} & a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
    " a_{40} & a_{41} & a_{42} & a_{43} & a_{44} \\\\\n",
    "\\end{bmatrix}\n",
    "\\right)\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "    z_{00} & z_{01} \\\\\n",
    "    z_{10} & z_{11} \\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Where,\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    " \\begin{cases}\n",
    "    z_{00} = a_{11} \\\\\n",
    "    z_{01} = a_{03} \\\\\n",
    "    z_{10} = a_{30} \\\\\n",
    "    z_{11} = a_{22} \\\\\n",
    " \\end{cases}\n",
    "\\end{equation}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf41db9-6928-4555-88f9-cb1133608f25",
   "metadata": {},
   "source": [
    "## Backward\n",
    "\n",
    "The activations gradient.\n",
    "\n",
    "$$\n",
    "\\frac{\\partial{Z^{[l]}}}{A^{[l]}}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " \\delta{a_{00}} & \\delta{a_{01}} & \\delta{a_{02}} & \\delta{a_{03}} & \\delta{a_{04}} \\\\\n",
    " \\delta{a_{10}} & \\delta{a_{11}} & \\delta{a_{12}} & \\delta{a_{13}} & \\delta{a_{14}} \\\\\n",
    " \\delta{a_{20}} & \\delta{a_{21}} & \\delta{a_{22}} & \\delta{a_{23}} & \\delta{a_{24}} \\\\\n",
    " \\delta{a_{30}} & \\delta{a_{31}} & \\delta{a_{32}} & \\delta{a_{33}} & \\delta{a_{34}} \\\\\n",
    " \\delta{a_{40}} & \\delta{a_{41}} & \\delta{a_{42}} & \\delta{a_{43}} & \\delta{a_{44}} \\\\\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    " 0                & 0                & 0                & \\partial{z_{01}} & 0 \\\\\n",
    " 0                & \\partial{z_{00}} & 0                & 0.               & 0 \\\\\n",
    " 0                & 0                & \\partial{z_{11}} & 0                & 0 \\\\\n",
    " \\partial{z_{10}} & 0                & 0                & 0                & 0 \\\\\n",
    " 0                & 0                & 0                & 0                & 0 \\\\\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1242c3ba-36a6-4f11-a31f-8df97f3f6da0",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- [Dumoulin & Visin (2016). \"A guide to convolution arithmetic for deep learning.\"](https://arxiv.org/pdf/1603.07285.pdf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}