02_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/ascarrambad/ml-21-22/blob/main/02_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nu4HJeN686y2"
   },
   "source": [
    "# Machine Learning SP 2021/2022\n",
    "\n",
    "Prof. Cesare Alippi<br>\n",
    "Giorgia Adorni ([`giorgia.adorni@usi.ch`](mailto:giorgia.adorni@usi.ch))<br>\n",
    "Luca Butera ([`luca.butera@usi.ch`](mailto:luca.butera@usi.ch))<br>\n",
    "Matteo Riva ([`matteo.riva@usi.ch`](mailto:matteo.riva@usi.ch))\n",
    "\n",
    "---\n",
    "# Lab 02: Classification\n",
    "\n",
    "We have a $d$-dimensional input vector $X \\in \\mathbb{R}^d$ and a set of $k$ possible classes, $C_1, \\dots, C_k$.\n",
    "Our goal in classification is to assign $X$ to the correct class. \n",
    "\n",
    "In particular, our goal is to determine a __discriminant__ function to parition the input space. In this session we will focus on binary classification.\n",
    "\n",
    "![alt text](https://jakelearnsdatascience.files.wordpress.com/2017/02/lda_binary.png)\n",
    "\n",
    "---\n",
    "# Logistic Regression\n",
    "\n",
    "Consider a binary classification problem $\\{(x_1, y_1), \\dots, (x_1, y_n)\\}$ and a Logistic Regression model, then:\n",
    "\n",
    "$$Pr(y_i=1\\vert x_i, \\boldsymbol \\theta) = \\sigma(x_i^\\top\\boldsymbol \\theta) = \\frac{1}{1+e^{-x_i^\\top\\boldsymbol \\theta}}\n",
    "$$\n",
    "\n",
    "Where $\\sigma({}\\cdot{})$ is the _sigmoid_ function:\n",
    "\n",
    "<img style=\"text-align: center\" src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/2560px-Logistic-curve.svg.png\" width=\"400\">\n",
    "\n",
    "Let's get some data and use _scikit-learn_ to fit a Logistic Regression model on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4IgAvdbC1482"
   },
   "outputs": [],
   "source": [
    "# first we define some helper functions to generate data and plot results\n",
    "import numpy as np\n",
    "from sklearn.datasets import make_classification, make_circles, make_moons\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "# color_maps\n",
    "cm = plt.cm.RdBu\n",
    "cm_bright = ListedColormap(['#FF0000', '#0000FF'])  \n",
    "\n",
    "# function to generate classification problems\n",
    "def get_data(n, ctype='simple'):\n",
    "  if ctype == 'simple':\n",
    "    x, y = make_classification(n_features=2, \n",
    "                               n_redundant=0, \n",
    "                               n_informative=2, \n",
    "                               n_clusters_per_class=1)\n",
    "    x += np.random.uniform(size=x.shape) # add some noise\n",
    "  elif ctype == 'circles':\n",
    "    x, y = make_circles(n, noise=0.1, factor=0.5)\n",
    "  \n",
    "  elif ctype == 'moons':\n",
    "    x, y = make_moons(n, noise=0.1)\n",
    "  else:\n",
    "    raise ValueError\n",
    "  return x, y\n",
    "\n",
    "# function to plot decision boundaries\n",
    "def plot_decision_surface(model, x, y, transform=lambda x:x):    \n",
    "  #init figure\n",
    "  fig = plt.figure()\n",
    "\n",
    "  # Create mesh\n",
    "  h = .01  # step size in the mesh\n",
    "  x_min, x_max = x[:, 0].min() - .5, x[:, 0].max() + .5\n",
    "  y_min, y_max = x[:, 1].min() - .5, x[:, 1].max() + .5\n",
    "  xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                        np.arange(y_min, y_max, h))\n",
    "\n",
    "  # plot train data\n",
    "  plt.scatter(x[:, 0], x[:, 1], c=y, cmap=cm_bright,\n",
    "              edgecolors='k')\n",
    "  plt.xlim(xx.min(), xx.max())\n",
    "  plt.ylim(yy.min(), yy.max())\n",
    "\n",
    "  plt.xlabel(r'$x_1$')\n",
    "  plt.ylabel(r'$x_2$');\n",
    "\n",
    "  y_pred = model.predict(transform(np.c_[xx.ravel(), yy.ravel()]))\n",
    "\n",
    "  y_pred = y_pred.reshape(xx.shape)\n",
    "  plt.contourf(xx, yy, y_pred > 0.5, cmap=cm, alpha=.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-SjvNW28BxGR"
   },
   "outputs": [],
   "source": [
    "np.random.seed(78)\n",
    "\n",
    "# Create a classification problem\n",
    "x, y = get_data(100)\n",
    "\n",
    "# Let's look at the data\n",
    "plt.scatter(x[:, 0], x[:, 1], c=y, cmap=cm_bright, edgecolors='k')\n",
    "plt.xlabel(r'$x_1$')\n",
    "plt.ylabel(r'$x_2$');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ayq8nv8xojjG"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "classifier = LogisticRegression() # create an instance of the model\n",
    "classifier.fit(x, y)              # fit the data\n",
    "\n",
    "plot_decision_surface(classifier, x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2_4giXAY4TPV"
   },
   "source": [
    "Logistic Regression is often referred to as a **generalized linear method**. In fact, even if the model is nonlinear, the predicted class depends only on the linear combination $x_i^\\top \\boldsymbol \\theta$ of the input variables. In other words, the decision surfaces are **linear**.\n",
    "\n",
    "In fact, the decision boundary of the binary logistic regression model that we built can be written as:\n",
    "\n",
    "$$x_2 = -\\frac{\\theta_0}{\\theta_2} - \\frac{\\theta_1}{\\theta_2} x_1$$\n",
    "\n",
    "_Homework: show why this is true._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wi0MniB74Q7a"
   },
   "outputs": [],
   "source": [
    "print(f'Intercept (shape: {classifier.intercept_.shape}: {classifier.intercept_}')\n",
    "print(f'Angular coeffs. (shape: {classifier.coef_.shape}: {classifier.coef_}')\n",
    "\n",
    "theta = np.c_[classifier.intercept_, classifier.coef_].ravel()\n",
    "\n",
    "print(f'Theta (shape: {theta.shape}: {theta}')\n",
    "\n",
    "b = -theta[0]/theta[2]\n",
    "m = -theta[1]/theta[2]\n",
    "\n",
    "x1 = np.array([x[:,0].min(), x[:,0].max()])\n",
    "x2 = b + m * x1\n",
    "\n",
    "plt.plot(x1, x2, c='black')\n",
    "plt.scatter(x[:, 0], x[:, 1], c=y, cmap=cm_bright, edgecolors='k')\n",
    "plt.xlabel(r'$x_1$')\n",
    "plt.ylabel(r'$x_2$');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EVwVoxFoEi1F"
   },
   "source": [
    "What happens if the data are not linerly separable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2rlSbxu8_jM5"
   },
   "outputs": [],
   "source": [
    "np.random.seed(20)\n",
    "\n",
    "x,y = get_data(120, 'circles')\n",
    "\n",
    "classifier = LogisticRegression()\n",
    "classifier.fit(x, y)\n",
    "\n",
    "plot_decision_surface(classifier, x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BcvcieFmGv5T"
   },
   "source": [
    "Remember: the linearity is in the input variables! We can use some nonlinear features to project the data in a space where they are separable with a straight line.\n",
    "\n",
    "Let's try with polar coordinates.\n",
    "$$\n",
    "\\left\\{\\begin{array}{rl}\n",
    "x&=r \\cos \\phi \\\\\n",
    "y&=r \\sin \\phi\n",
    "\\end{array}\n",
    "\\right. \\implies\n",
    "\\left\\{\n",
    "  \\begin{array}{rl}\n",
    "r&=\\sqrt{x^2 + y^2}\\\\\n",
    "\\phi&= {atan2}\\left(y,x\\right) \n",
    "\\end{array}\n",
    "\\right.\n",
    "$$\n",
    "\n",
    "<img src=\"https://vvvv.org/sites/default/files/imagecache/large/images/283px-Polar_coordinate_components.svg_.png\" width=\"300\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rs15ChyY_kTA"
   },
   "outputs": [],
   "source": [
    "def to_polar(x):\n",
    "  r = np.sqrt(np.square(x).sum(axis=1))\n",
    "  phi = np.arctan2(x[:,1], x[:,0])\n",
    "  return np.c_[r, phi]\n",
    "\n",
    "x_polar = to_polar(x)\n",
    "\n",
    "plt.scatter(x_polar[:, 0], x_polar[:, 1], c=y, cmap=cm_bright, edgecolors='k')\n",
    "plt.xlabel(r'$r$')\n",
    "plt.ylabel(r'$\\phi$');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cO6nKbEKIyRn"
   },
   "source": [
    "Nice! Let's fit the model using the polar features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-A5AKM1GAMah"
   },
   "outputs": [],
   "source": [
    "classifier = LogisticRegression()\n",
    "classifier.fit(x_polar, y)\n",
    "\n",
    "plot_decision_surface(classifier, x, y, transform=to_polar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IzlJtIhDJ4cO"
   },
   "source": [
    "It is not always easy to find suitable features/projections by hand. That's why we need nonlinearity.\n",
    "\n",
    "In the next part we will see how to use neural networks to solve classification problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8gttJ8kwLVdh"
   },
   "source": [
    "## K-Nearest Neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "igyFScSyLehY"
   },
   "outputs": [],
   "source": [
    "# try it on your own!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sQ5s722RHacM"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "02_classification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}