support_vector_machines.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:866862dfbd9e10b81d5418021b399fead5bcd1d64bd8d6caf51f465dd8bc9520"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#<span style=\"background-color:#66FF99)\">Support Vector Machines</span>\n",
      "* One of the best 'out of box' classifers\n",
      "\n",
      "###<span style=\"background-color:#FF99FF)\">Maximal Margin Classifer</span>\n",
      "\n",
      "* **What is a Hyperplane?**\n",
      "    * Flat subspace of dimension p - 1 (in 3D, hyperplane is a flat 2D)\n",
      "    * Hyperplane equation:\n",
      "    ##$\\beta_0 + \\beta_1X_1 + \\beta_2X_2 + ... \\beta_pX_p = 0$\n",
      "\n",
      "* **Classification using Separating Hyperplane**\n",
      "    * A test observation is assigned a class depending on which side of the hyperplane it is located\n",
      "    * If f(x) is positive, assign class 1, if negative, assign class -1\n",
      "    * The magnitude of the f(x) shows how far it lies from the hyperplane\n",
      "    * Separating hyperplane has the property:\n",
      "    ##$\\beta_0 + \\beta_1X_1 + \\beta_2X_2 + ... \\beta_pX_p > 0$ if $y_i = 1$\n",
      "    ##$\\beta_0 + \\beta_1X_1 + \\beta_2X_2 + ... \\beta_pX_p < 0$ if $y_i = -1$\n",
      "* **Maximal Margin Classifer**\n",
      "    * Which of the hyperplanes to use? Maximal margin hyperplane (aka. optimal separting hyperplane) where the margin is the largest\n",
      "    * Margin is the smallest distance the observation is from the hyperplane\n",
      "    * Support vectors: if these points are moved slighly, the maximal margin hyperplane would move\n",
      "\n",
      "* **Construction of the Maximal Margin Classifier**\n",
      "    * Optimization (using Math)!\n",
      "\n",
      "* **Non-separable Case**\n",
      "    * Maximal Margin Classifer is a natural way to perform classification, only if a separating hyperplane exists! Sometimes it does not\n",
      "    * Develop a hyperplane that ALMOST separates the classes, Soft-Margin\n",
      "    * Generalization of the maximal margin classifer to non-separable case is called ``Support Vector Classifer``"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###<span style=\"background-color:#FF99FF)\">Support Vector Classifers</span>\n",
      "\n",
      "* **Overview**\n",
      "    * A hyperplane which does not *perfectly* separate the 2 classes, just for *most* of training observations\n",
      "    * Support Vector Classifers (Soft margin classifer) does this\n",
      "* **Details of Support Vector Classifer**\n",
      "    * Uses a nonnegative tuning parameter (C) which determines how much mistake we can tolerate (budget for the amount which can be violated\n",
      "    * If C = 0, no tolerance, will be same as maximal margin hyperplane\n",
      "    * If C > 0, no more than C observations can be on the wrong side\n",
      "    * As C increases, the more tolerant, the bigger the margin\n",
      "    * C is used as a tuning parameter and chosen via cross-validation  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###<span style=\"background-color:#FF99FF)\">Support Vector Machines</span>\n",
      "\n",
      "* **Classification with Non-linear Decision Boundaries**\n",
      "    * Support Vector Classifer useful for linear class boundaries. How about non-linear boundaries?\n",
      "    * By using quadratic, cubic, higher-order polynomial\n",
      "    * There are unlimited options and could end up with a huge number of features, then will be computationally expensive\n",
      "    * SVM allows us to enlarge feature space which leads to efficient computations\n",
      "    \n",
      "* **Support Vector Machine**\n",
      "    * Extension of the SVC using **Kernels** -  enlarge our feature space to accomodate a non-linear boundary\n",
      "    * Kernel approach - efficient computational approach for enlarging\n",
      "    * In representing the inner product and computing its coefficients, all we need are inner products (?)\n",
      "    * Generalization of the inner product $K(x_i, x_i)$ where K is some function referred to as a kernel\n",
      "    * Kernel is a function which quantifies the similarity of 2 observations!\n",
      "    * Linear kernel quantifies the similarity of a pair of observations using Pearson (standard) correlation\n",
      "    * Can also use polynomial kernel, radial kernel\n",
      "    * How does radial kernel work? If given test observation is far from training observation, K will be very small. So training observations which are far will play no role is predicting the class. Radial kernel has local behavior, only nearby training observations wil have effect\n",
      "    * What's the advantage of using Kernel as opposed to enlarging feature space? Computational less expensive!\n",
      "    * Radial kernel takes various values of $\\gamma$ (positive constant) : as $\\gamma$ increase, fit becomes more linear\n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###4. SVMs with more than 2 classes\n",
      "\n",
      "* One vs One Classification\n",
      "* One vs All Classification"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}