Vector Calculas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "90a0d702",
   "metadata": {},
   "source": [
    "# Vector Calculus | Matrix Calculus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b297340",
   "metadata": {},
   "source": [
    "![vector-calc](./images/vector_calculus.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "028d36ef",
   "metadata": {},
   "source": [
    "### SISO\n",
    "![siso](./images/siso.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b056ee00",
   "metadata": {},
   "source": [
    "### MISO\n",
    "\n",
    "![miso](./images/miso.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b2373d8",
   "metadata": {},
   "source": [
    "$Y=f(X) \\quad X \\in \\mathbb{R} \\quad Y \\in \\mathbb{R} $\n",
    "\n",
    "$\\text{Any incremental change in X result change in Y, SO } f\\colon X+\\delta x \\to  Y+\\delta Y  $\n",
    "\n",
    "$\\delta Y = \\frac{df}{dx} \\delta X$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb48e0b",
   "metadata": {},
   "source": [
    "### input output space\n",
    "![miso-1](./images/miso_cor.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e1a35e9",
   "metadata": {},
   "source": [
    "$y = f(x_{1},x_{2}, x_{3}, \\dots ,x_{n}) \\quad  \n",
    "\\delta x = \\begin{bmatrix}\n",
    "\\delta x_{1}\\\\\n",
    "\\delta x_{2}\\\\\n",
    "\\delta x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta x_{n}\\\\\n",
    "\\end{bmatrix}\n",
    "X = \\begin{bmatrix}\n",
    "x_{1}\\\\\n",
    "x_{2}\\\\\n",
    "x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "x_{n}\\\\\n",
    "\\end{bmatrix}\n",
    "\\quad \\quad X \\in \\mathbb{R}^{n} , Y \\in \\mathbb{R},  \\delta x \\in \\mathbb{R}^{n}, \\delta y \\in \\mathbb{R}$\n",
    "\n",
    "$\\text{change in Y due to change in X is } X+\\delta x \\to Y+\\delta y $\n",
    "\n",
    "$\\delta y = \\frac{\\partial{f}}{\\partial{x_{1}}} \\delta x_{1}+ \\frac{\\partial{f}}{\\partial{x_{2}}} \\delta x_{2}+ \\frac{\\partial{f}}{\\partial{x_{3}}} \\delta x_{3}+ \\dots+ \\frac{\\partial{f}}{\\partial{x_{n}}} \\delta x_{n}  $\n",
    "\n",
    "$\\delta y = \\underbrace{\\begin{bmatrix} \\delta x_{1}& \\delta x_{2}& \\delta x_{3} &\\dots&\\delta x_{n} \\end{bmatrix} \\begin{bmatrix}\n",
    "\\frac{\\partial{f}}{\\partial{x_{1}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{2}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{3}}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{n}}}\n",
    "\\end{bmatrix}}_{\\text{Matrix Multiplication}}= \\underbrace{\\begin{bmatrix}\n",
    "\\frac{\\partial{f}}{\\partial{x_{1}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{2}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{3}}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{n}}}\n",
    "\\end{bmatrix}\\cdot \\begin{bmatrix} \\delta x_{1}\\\\ \\delta x_{2}\\\\ \\delta x_{3} \\\\ \\dots\\\\ \\delta x_{n} \\end{bmatrix}}_{\\text{Vector dot Products}} = \\nabla f  \\cdot  \\delta x \\quad \\text { where } \\nabla f = \\begin{bmatrix}\n",
    "\\frac{\\partial{f}}{\\partial{x_{1}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{2}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{3}}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{n}}}\n",
    "\\end{bmatrix} \\quad \\text{ and } \\quad \\delta x = \\begin{bmatrix}\n",
    "\\delta x_{1}\\\\\n",
    "\\delta x_{2}\\\\\n",
    "\\delta x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta x_{n}\\\\\n",
    "\\end{bmatrix}$\n",
    "\n",
    "$ \\text{So Equation from Figure 1 and Figure 2 can be compared } \\frac{df}{dX} \\equiv \\nabla f = \\nabla Y = \\begin{bmatrix}\n",
    "\\frac{\\partial{f}}{\\partial{x_{1}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{2}}}\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{3}}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_{n}}}\n",
    "\\end{bmatrix} \\quad \\nabla f \\text{ is having n componnet, } \\quad \\nabla f \\in \\mathbb{R}^{n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e409592",
   "metadata": {},
   "source": [
    "### MIMO \n",
    "![mimo](./images/mimo.png)\n",
    "\n",
    "![mimo-1](./images/mimo_cor.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aaf20c6",
   "metadata": {},
   "source": [
    "$X \\in \\mathbb{R}^{n} , Y \\in \\mathbb{R}^{m},  \\delta x \\in \\mathbb{R}^{n}, \\delta y \\in \\mathbb{R}^{m}$\n",
    "\n",
    "$Y =\\begin{bmatrix} y_{1}\\\\y_{2}\\\\y_{3}\\\\ \\vdots \\\\y_{m}\\end{bmatrix} = \\begin{bmatrix} \n",
    "f_{1}(x_{1},x_{2}, x_{3}, \\dots ,x_{n})\\\\\n",
    "f_{2}(x_{1},x_{2}, x_{3}, \\dots ,x_{n})\\\\\n",
    "f_{3}(x_{1},x_{2}, x_{3}, \\dots ,x_{n})\\\\ \n",
    "\\vdots \\\\\n",
    "f_{m}(x_{1},x_{2}, x_{3}, \\dots ,x_{n})\\end{bmatrix}_{mx1}  \n",
    "\\underbrace{\\begin{bmatrix}\n",
    "x_{1}\\\\\n",
    "x_{2}\\\\\n",
    "x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "x_{n}\\\\\n",
    "\\end{bmatrix}_{nx1}}_{X} \\quad\n",
    "\\underbrace{\\begin{bmatrix}\n",
    "\\delta x_{1}\\\\\n",
    "\\delta x_{2}\\\\\n",
    "\\delta x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta x_{n}\\\\\n",
    "\\end{bmatrix}_{nx1}}_{\\delta x}\\quad\n",
    "\\underbrace{\\begin{bmatrix}\n",
    "\\delta y_{1}\\\\\n",
    "\\delta y_{2}\\\\\n",
    "\\delta y_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta y_{m}\\\\\n",
    "\\end{bmatrix}_{mx1}}_{\\delta y}$\n",
    "\n",
    "\n",
    "$\\text{change in Y due to change in X is } X+\\delta x \\to Y+\\delta y $\n",
    "\n",
    "$\\delta y_{1} = \\frac{\\partial{f_1}}{\\partial{x_{1}}} \\delta x_{1}+ \\frac{\\partial{f_1}}{\\partial{x_{2}}} \\delta x_{2}+ \\frac{\\partial{f_1}}{\\partial{x_{3}}} \\delta x_{3}+ \\dots+ \\frac{\\partial{f_1}}{\\partial{x_{n}}} \\delta x_{n} \\\\\n",
    "\\delta y_{2} = \\frac{\\partial{f_2}}{\\partial{x_{1}}} \\delta x_{1}+ \\frac{\\partial{f_2}}{\\partial{x_{2}}} \\delta x_{2}+ \\frac{\\partial{f_2}}{\\partial{x_{3}}} \\delta x_{3}+ \\dots+ \\frac{\\partial{f_2}}{\\partial{x_{n}}} \\delta x_{n} \\\\\n",
    "\\vdots\\\\\n",
    "\\delta y_{m} = \\frac{\\partial{f_m}}{\\partial{x_{1}}} \\delta x_{1}+ \\frac{\\partial{f_m}}{\\partial{x_{2}}} \\delta x_{2}+ \\frac{\\partial{f_m}}{\\partial{x_{3}}} \\delta x_{3}+ \\dots+ \\frac{\\partial{f_m}}{\\partial{x_{n}}} \\delta x_{n} \\\\\n",
    "\\quad\\\\\n",
    "\\begin{bmatrix}\n",
    "\\delta y_{1}\\\\\n",
    "\\delta y_{2}\\\\\n",
    "\\delta y_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta y_{m}\\\\\n",
    "\\end{bmatrix} = \\begin{bmatrix} \n",
    "\\frac{\\partial{f_1}}{\\partial{x_1}} & \\frac{\\partial{f_1}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_1}}{\\partial{x_n}}\\\\\n",
    "\\frac{\\partial{f_2}}{\\partial{x_1}} & \\frac{\\partial{f_2}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_2}}{\\partial{x_n}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f_m}}{\\partial{x_1}} & \\frac{\\partial{f_m}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_m}}{\\partial{x_n}}\\\\\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "\\delta x_{1}\\\\\n",
    "\\delta x_{2}\\\\\n",
    "\\delta x_{3}\\\\\n",
    "\\vdots\\\\\n",
    "\\delta x_{n}\\\\\n",
    "\\end{bmatrix}\n",
    "= J(Y)  \\cdot  \\delta x $\n",
    "\n",
    "$\\text{So Equation from Figure 1 , Figure 2 and Figure 3 can be compared }$\n",
    "\n",
    "$\\frac{df}{dX} \\equiv \\nabla Y \\equiv  J(Y) \\quad J(Y) \\text{ is having mxn componnet, } \\quad J(Y) \\in \\mathbb{R}^{mxn}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f74ebf94",
   "metadata": {},
   "source": [
    "### Second Order Derivatives "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b13e1df3",
   "metadata": {},
   "source": [
    "\n",
    "$Y = f(X) \\quad X \\in \\mathbb{R}^{n}  \\quad \\frac{d^2Y}{dX^2} = \\frac{d}{dX} \\frac{dY}{dX} \\quad  \\text{let, } g = \\frac{dY}{dX} $\n",
    "\n",
    "$\\text{if } Y \\in \\mathbb{R} \\quad \\text{ then , } g \\in \\mathbb{R}^{n} \\quad \\quad \\quad \\text{if } Y \\in \\mathbb{R}^{m} \\quad \\text{ then , } g \\in \\mathbb{R}^{mxn}$\n",
    "\n",
    "$\\underbrace{\\frac{d^2Y}{dX^2} = \\frac{dg}{dX} \\in \\mathbb{R}^{nxn}}_{Hessian Matrix}  \\quad \\quad \\quad \\quad \\quad \\quad \\underbrace{\\frac{d^2Y}{dX^2} = \\frac{dg}{dX} \\in \\mathbb{R}^{mxnxn}}_{Tensor }$\n",
    "\n",
    "$\\frac{d^2Y}{dX^2} =  \\frac{d}{dX}\\begin{bmatrix} \n",
    "\\frac{\\partial{f}}{\\partial{x_1}} \\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_2}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f}}{\\partial{x_n}}\\\\\n",
    "\\end{bmatrix} \n",
    "\\quad \\quad \\quad \\quad \\quad \\quad \\frac{d^2Y}{dX^2} = \\frac{d}{dX} \\begin{bmatrix} \n",
    "\\frac{\\partial{f_1}}{\\partial{x_1}} & \\frac{\\partial{f_1}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_1}}{\\partial{x_n}}\\\\\n",
    "\\frac{\\partial{f_2}}{\\partial{x_1}} & \\frac{\\partial{f_2}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_2}}{\\partial{x_n}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial{f_m}}{\\partial{x_1}} & \\frac{\\partial{f_m}}{\\partial{x_2}} & \\cdots &\\frac{\\partial{f_m}}{\\partial{x_n}}\\\\\n",
    "\\end{bmatrix} $\n",
    "\n",
    "$\\underbrace{\\begin{bmatrix} \n",
    "\\frac{\\partial^{2}{f}}{\\partial{x_1}^{2}} & \\frac{\\partial^{2}{f}}{\\partial{x_1}\\partial{x_2}} & \\cdots &\\frac{\\partial^{2}{f}}{\\partial{x_1}\\partial{x_n}}\\\\\n",
    "\\frac{\\partial^{2}{f}}{\\partial{x_2}\\partial{x_1}} & \\frac{\\partial^{2}{f}}{\\partial{x_2}^{2}} & \\cdots &\\frac{\\partial^{2}{f}}{\\partial{x_2}\\partial{x_n}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial^{2}{f}}{\\partial{x_n}\\partial{x_1}} & \\frac{\\partial^{2}{f}}{\\partial{x_n}\\partial{x_2}} & \\cdots &\\frac{\\partial^{2}{f}}{\\partial{x_n}^{2}}\\\\\n",
    "\\end{bmatrix}}_{Hessian Matrix} \n",
    "\\quad \n",
    "\\underbrace{\n",
    "\\left [\n",
    "\\begin{bmatrix} \n",
    "\\frac{\\partial^{2}{f_1}}{\\partial{x_1}^{2}} & \\frac{\\partial^{2}{f_1}}{\\partial{x_1}\\partial{x_2}} & \\cdots &\\frac{\\partial^{2}{f_1}}{\\partial{x_1}\\partial{x_n}}\\\\\n",
    "\\frac{\\partial^{2}{f_1}}{\\partial{x_2}\\partial{x_1}} & \\frac{\\partial^{2}{f_1}}{\\partial{x_2}^{2}} & \\cdots &\\frac{\\partial^{2}{f_1}}{\\partial{x_2}\\partial{x_n}}\\\\\n",
    "\\vdots\\\\\n",
    "\\frac{\\partial^{2}{f_1}}{\\partial{x_n}\\partial{x_1}} & \\frac{\\partial^{2}{f_1}}{\\partial{x_n}\\partial{x_2}} & \\cdots &\\frac{\\partial^{2}{f_1}}{\\partial{x_n}^{2}}\\\\\n",
    "\\end{bmatrix}\n",
    "\\cdots \\text{Hessian of } f_2 \\text{ to } f_m\n",
    "\\right ]}_{tensor}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d102c0a",
   "metadata": {},
   "source": [
    "### Tables\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8902c1",
   "metadata": {},
   "source": [
    "|Function|Input & Output|1st Differentiation|2nd Differentiation|\n",
    "|:----------------|:------|:------|:-------| \n",
    "|$\\text{SISO }\\quad f\\colon \\mathbb{R} \\to \\mathbb{R}\\quad$|$X \\in \\mathbb{R} ,Y \\in \\mathbb{R}\\quad$ |$\\frac{df}{dX} \\in \\mathbb{R}\\quad$|$\\frac{d^2f}{dX^2} \\in \\mathbb{R}$|\n",
    "|$\\text{MISO }\\quad f\\colon \\mathbb{R}^{n} \\to \\mathbb{R}\\quad$|$X \\in \\mathbb{R}^{n},Y \\in \\mathbb{R}\\quad$ |$\\text{gradient }$$\\nabla  \\in \\mathbb{R}^{n}\\quad$|$\\text{Hassian Matrix}$ $H \\in \\mathbb{R}^{nxn}\\quad$ |\n",
    "|$\\text{MIMO }\\quad f\\colon \\mathbb{R}^{n} \\to \\mathbb{R}^{m}\\quad$|$X \\in \\mathbb{R}^{n},Y \\in \\mathbb{R}^{m}$| $\\text{Jacobian Matrix}$ $J \\in \\mathbb{R}^{mxn}$ |$\\text{Tensor} \\in \\mathbb{R}^{mxnxn}$ \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7b2b68c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}