support_vector_machine_SVM_scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is a Support Vector Machine?\n",
    "\n",
    "It's a supervised machine learning algorithm which can be used for both classification or regression problems. But it's usually used for classification. Given 2 or more labeled classes of data, it acts as a discriminative classifier, formally defined by an optimal hyperplane that seperates all the classes. New examples that are then mapped into that same space can then be categorized based on on which side of the gap they fall.\n",
    "\n",
    "## What are Support Vectors?\n",
    "\n",
    "![alt text](https://www.dtreg.com/uploaded/pageimg/SvmMargin2.jpg \"Logo Title Text 1\")\n",
    " \n",
    "Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set, they are what help us build our SVM. \n",
    "\n",
    "## Whats a hyperplane?\n",
    "\n",
    "![alt text](http://slideplayer.com/slide/1579281/5/images/32/Hyperplanes+as+decision+surfaces.jpg \"Logo Title Text 1\")\n",
    "\n",
    "Geometry tells us that a hyperplane is a subspace of one dimension less than its ambient space. For instance, a hyperplane of an n-dimensional space is a flat subset with dimension n − 1. By its nature, it separates the space into two half spaces.\n",
    "\n",
    "## Linear vs nonlinear classification?\n",
    "\n",
    "Sometimes our data is linearly seperable. That means for N classes with M features, we can learn a mapping that is a linear combination. (like y = mx + b). Or even a multidimensional hyperplane (y = x + z + b + q). No matter how many dimensions/features a set of classes have, we can represent the mapping using a linear function.\n",
    "\n",
    "But sometimes its not. Like if there was a quadratic mapping. Luckily for us SVMs can can efficiently perform a non-linear classification using what is called the kernel trick. I'll talk about that lter on in the course. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2187ffa0b08>]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3RVZfr28e+dkBB6F2kSeq+GGkgsBJCq2LCgYgGVmlhmnHEcp9h+jqGpiNgLoCIgigLBwYTQE3pv0lvovT/vH4F5GQckgZzsc5Lrs1YWIWezz7UoF0+es/d9zDmHiIj4ryCvA4iIyO9TUYuI+DkVtYiIn1NRi4j4ORW1iIify+OLk5YsWdKFh4f74tQiIjlSamrqHudcqUs95pOiDg8PJyUlxRenFhHJkcxs0+Ue09aHiIifU1GLiPg5FbWIiJ9TUUuW6zm5Jz0n9/Q6hkiOoaIWEfFzGSpqMytqZmPNbJWZrTSzFr4OJiIi6TJ6ed4QYLJz7i4zCwXy+zCTiIhc5IpFbWaFgSjgEQDn3CnglC/CDP15LdHVS9GgQlFfnF585Lf70Sm7Ui759Y/bf5xtmURykoxsfVQG0oCPzWyhmX1gZgV+e5CZ9TKzFDNLSUtLy3SQA8dOMWruZu54dyav/riS46fOZvocIiI5kV3pjQPMLAKYA0Q65+aa2RDgkHPuL5f7NREREe5q7kw8dOI0r/24itHzNhNeIj+vdatPiyolMn0e8daFlbRW0CIZZ2apzrmISz2WkRX1VmCrc27u+Z+PBRpnVbiLFQ4L4bVu9Rj1RDMccN/IOfxp/FIOnTjti6cTEQkIVyxq59xOYIuZ1Tj/pVuBFb4M1bJKSSYPiOKJ1pUYM28zbeOT+HnlLl8+pYiI38roddT9gC/NbAnQEHjVd5HS5QsN5s8dazPu6UiK5AvhsU9T6D96IXuPnPT1U4uI+JUr7lFfjavdo76cU2fO8e4v63hn+joKhYXw18616dKgLGaWZc8hIuKla92j9lxoniAGtqnOD/1aU6F4fgaMWcTjn6aw4+Bxr6OJiPhcQBT1BTWuL8S4p1ryYsdazFy/h7bxSYyau5lz57L+uwIREX8RUEUNEBxkPN66MlMGRlG3XBH+NH4p938wh417jnodTUTEJwKuqC+oWKIAo55oxuvd6rF82yHaDU7i/aT1nDl7zutoIiJZKmCLGsDM6N70BhLiomldrSSv/riKO4fPYtXOQ15HExHJMgFd1BdcXySMkQ9FMOy+Rmzdf5xOQ5OJT1jDyTO6DV1EAl+OKGpIX113blCWhLhoOtUvw9Cf19J5WDILN+/3OpqIyDXJMUV9QfECoQzu3oiPHong8IkzdBs+i3/8sIJjp854HU1E5KrkuKK+4JaapZkaG8UDzW7gw+RfaTc4iZnr9ngdS0Qk03JsUQMUCgvhn7fXY0yv5gSb8cAHc/njt0s4eFxDnkQkcOToor6geeUSTB4YRe/oynydsoWY+ESmLt/pdSwRkQzJFUUNEBYSzAu31WJCn0iKFwil1+ep9B21gD0a8iQifi7XFPUF9csXZWLfVjwTU52py3fRJj6R8Qu34ovhVCIiWSHXFTWkD3nqd2s1JvVvRaWSBYj9ajGPfjKf7Qc05ElE/E+uLOoLqpUuxNgnW/JSp9rM2bCPmPhEPp+zSUOeRMSv5OqihvQhT4+2qsTU2Cga3VCMv0xYRvf357Ah7YjX0UREABX1f1Qonp/PH2vK/91Zn5U7D3HbkBm8l6ghTyLiPRX1RcyMe5pUYFpcNNHVS/H6T6u4/d2ZrNiuIU8i4h0V9SWULhzGiB438u4Djdl58ARd3k7mramrNeRJRDyhor4MM6NDvTIkxEbTpWFZhv17HR2HJpO6aZ/X0UQkl1FRX0GxAqHE39OQT3o24fips9z13mxenricoyc15ElEsoeKOoNuqnEdU2Kj6NG8Ip/M2ki7wUnMWJvmdSwRyQVU1JlQMG8e/t61Ll/3bkFocBA9PpzHc98s5uAxDXkSEd9RUV+FppWK8+OA1jx9UxXGLdxGm0GJTF6mIU8i4hsq6qsUFhLM8+1r8l2fSEoVzMuTX6Ty9Jep7D58wutoIpLDqKivUd1yRfiubyTPtavBtJW7iYlPYmyqhjyJSNZRUWeBkOAg+txclR/7t6bqdQV59pvFPPzxfLbuP+Z1NBHJATJU1Ga20cyWmtkiM0vxdahAVfW6gnzTuwV/61KHlI37aDsoiU9nbdSQJxG5JplZUd/snGvonIvwWZocICjIeLhlOFNjo4gIL85fJy7nnhGzWa8hTyJylbT14SPli+Xn055N+NfdDVi7+wi3DZnBO9PXcVpDnkQkkzJa1A6YamapZtbrUgeYWS8zSzGzlLQ03QgC6beh33VjeRLiomhT6zrenLKarm/PZNm2g15HE5EAYhm5OsHMyjrntpvZdUAC0M85l3S54yMiIlxKirayf2vysh28OGE5+4+doldUZQbcWo2wkGCvY4mIHzCz1MttLWdoRe2c237+x93AeKBp1sXLPdrXLcPPcdF0a1SO4b+sp8OQGczfqCFPIvL7rljUZlbAzApd+BxoCyzzdbCcqkj+EN68uwGfPdqUk2fOcfd7s3npu2Uc0ZAnEbmMjKyoSwPJZrYYmAdMcs5N9m2snC+qeimmxkbxSMtwPp+ziXaDkkhco719EflfGdqjziztUWdO6qZ9PD92CevTjtKtcTle6lSbovlDvY4lItnomveoxbdurFicSf1b0/fmqkxctJ028Yn8uHSHbkMXEUBF7TfCQoJ5tl0NvusbyfVFwnj6ywU8+UUquw9pyJNIbqei9jN1yhZhwtOR/KF9TaavTqNNfCJfp2zR6lokF1NR+6E8wUE8dVMVJg9oTc3rC/P82CX0+HAeW/ZpyJNIbqSi9mOVSxVkTK/m/OP2uizcvJ+2g5L4eOavnNWQJ5FcRUXt54KCjB7NKzI1LppmlYvzt+9XcPd7s1i3+7DX0UQkm6ioA0S5ovn4+JEmDLq3ARv2HKXDkGSG/bxWQ55EcgEVdQAxM+5oVJ5pcdHE1CnNWwlr6DwsmaVbNeRJJCdTUQegkgXz8s79jRnR40b2HT1F13eSee2nlZw4fdbraCLiAyrqANauzvUkxEVzT0QFRiRu4LYhM5i7Ya/XsUQki6moA1yRfCG8fmd9vny8GWfOnePe9+fw4oSlHD5x2utoIpJFVNQ5RGTVkkwZGMVjrSrx5dzNtBuUxPRVu72OJSJZQEWdg+QPzcNfOtXm26daUiBvHnp+Mp/Yrxax7+gpr6OJyDVQUedAjW8oxg/9W9H/1mp8v3g7MfGJfL94u25DFwlQKuocKm+eYOJiqvN9v1aUK5aPfqMX8sRnqezSkCeRgKOizuFqlSnMuKda8qcONZmxNn3I05h5m7W6FgkgKupcIE9wEL2iqjBlYBS1yxTmj+OW8sAHc9m8V0OeRAKBijoXCS9ZgNFPNOfVO+qxZOtB2g5O5IMZGzTkScTPqahzmaAg4/5mN5AQF0XLKiX556SVdBs+i9U7NeRJxF+pqHOpMkXy8eHDEQzp3pAt+47RadgMBk9bw6kzGvIk4m9U1LmYmdG1YTkSYqPoUK8Mg6etpfOwZBZvOeB1NBG5iIpaKFEwL0O6N+KDhyI4ePw0d7w7k1cmreD4KQ15EvEHKmr5jza1SzM1LoruTW9g5IxfaT8kidnrNeRJxGsqavkvhcNCePWOeox6ohkA942cwwvjlnJIQ55EPKOilktqWaUkkwdE0SuqMl/N30xMfCLTVuzyOpZIrqSilsvKFxrMnzrUYtzTkRTNF8rjn6XQf/RC9h456XW0q7Kpx0Ns6vGQ1zFEMk1FLVfUsEJRvu/Xitg21flp2Q7axCfy3aJtug1dJJtkuKjNLNjMFprZD74MJP4pNE8QA9pUY1L/1lQsUYABYxbx+Kcp7Dh43OtoIjleZlbUA4CVvgoigaF66UJ8+1RLXuxYi5nr9xATn8SXczdxTrehi/iMZeTbVzMrD3wKvALEOec6/d7xERERLiUl5YrnvXfE7AyF/Kp3iwwdJ9lr895j/HHcEmat30vzysV5vVt9wksW8DrWf/x2P/rY/PkA5G/S5L++XvHzz7Itk8jlmFmqcy7iUo9ldEU9GHgeuOz9xWbWy8xSzCwlLS3tKmJKoLmhRH6+fLwZr3erx/Jth2g3OIn3k9Zz5qxuQxfJSldcUZtZJ6CDc+5pM7sJeDarVtSSc+w8eIIXJyxj2spd1C9fhDfurE+tMoW9jvVfLqywtYIWf3StK+pIoIuZbQTGALeY2RdZmE9ygOuLhDHyoRt5+/5GbNt/nM7DkolPWMPJM7oNXeRaXbGonXMvOOfKO+fCge7Av51zD/o8mQQcM6NT/bJMi4umc4OyDP15LZ2GJrNg836vo4kENF1HLVmuWIFQBt3bkI8facKRk2e4c/gs/vHDCo6dOuN1NJGAlKGrPjJLe9RyweETp3lj8iq+mLOZCsXz8Xq3+kRWLel1LBG/kxVXfYhclUJhIfzz9np81as5eYKCeOCDufxh7BIOHteQJ5GMUlFLtmhWuQQ/DWjNk9FVGLtgKzHxiUxdvtPrWCIBQUUt2SYsJJg/3laTCU9HUqJgXnp9nkqfUQtIOxyYQ55EsouKWrJdvfJFmNg3kmfbVidh+S5iBiUyfuFWDXkSuQwVtXgiJDiIvrdU48cBrahcsgCxXy2m5yfz2XZAQ55EfktFLZ6qel0hvnmyJX/tXJu5G/bRNj6Rz2dv1JAnkYuoqMVzwUFGz8hKTI2NonHFYvzlu+V0f38OG9KOeB1NxC+oqMVvVCien88ebcqbd9Vn1c5DtB8yg+G/aMiTiIpa/IqZcXdEBabFRXNzjVK8MXkVt787kxXbD3kdTcQzKmrxS9cVDmNEjwiGP9CYnQdP0uXtZP41ZTUnTmvIk+Q+Kmrxa7fVK8O0uCi6NizH29PX0XHoDFI37fM6lki2UlGL3yuaP5S37mnAp4825cTpc9z13mxenricoyc15ElyBxW1BIzo6qWYEhvFQ80r8unsjbQdlETSGr2bkOR8KmoJKAXz5uFvXevyde8W5A0J4qGP5vHsN4s5eExDniTnUlFLQGoSXpwf+7fm6ZuqMH7hNtoMSmTysh1exxLxCRW1BKywkGCeb1+T7/pEUqpgXp78YgFPfZHK7sMnvI4mkqVU1BLw6pYrwnd9I3muXQ1+XrWbmPgkxqZqyJPkHCpqyRFCgoPoc3NVfuzfmmrXFeTZbxbz0Efz2LLvmNfRRK6ZilpylKrXFeTr3i34e9c6LNi0n3aDk/hk5q8a8iQBTUUtOU5QkPFQi3CmxEYREV6cl79fwT0jZrNut4Y8SWBSUUuOVb5Yfj7t2YS37m7A2t1H6DBkBu9MX8dpDXmSAKOilhzNzLjzxvJMi4umTe3reHPKarq+PZNl2w56HU0kw1TUkiuUKpSXdx+4kfcebEzakZN0fWcmb0xepSFPEhBU1JKrtK9bhmmx0dzZuBzDf1lPhyEzmL9RQ57Ev6moJdcpkj+E/7urAV881oxTZ89x93uzeem7ZRzRkCfxUypqybVaVSvJlIFR9IwM5/M5m2g3KIlfVu/2OpbI/1BRS65WIG8e/tq5DmOfbEm+0GAe+Xg+cV8vYv/RU15HE/mPKxa1mYWZ2TwzW2xmy83sb9kRTCQ73VixGJP6t6LfLVWZuGg7MYMSmbRkh25DF7+QkRX1SeAW51wDoCHQ3sya+zaWSPbLmyeYZ9rWYGLfVpQpko8+oxbQ+/NUdh/SkCfx1hWL2qW7cEtXyPkPLTMkx6pdtjDjn27JC7fVJHFNGrfGJ/L1/C1aXYtnMrRHbWbBZrYI2A0kOOfmXuKYXmaWYmYpaWl61w0JbHmCg+gdXYWfBrSmVpnCPP/tEnp8qCFP4g3LzCrBzIoC44F+zrlllzsuIiLCpaSkZEE8Ee+dO+cYNW8zr/+0irPnHM+1q8HDLcMJDjKvo0kOYmapzrmISz2Wqas+nHMHgF+A9lmQSyQgBAUZDzavyNTYKJpVLs7ff1jB3e/NYu2uw15Hk1wiI1d9lDq/ksbM8gFtgFW+Dibib8oWzcfHjzRh8L0N+XXPUToOTWbYz2s5dUZDnsS3MrKiLgNMN7MlwHzS96h/8G0sEf9kZtzeqBwJcdG0q3s9byWsocvbySzZesDraJKDZWqPOqO0Ry25RcKKXbw4YSlph0/yROvKxMZUJywk2OtYEoCybI9aRP5bTO3STI2N5t4mFRiRtIH2g5OYs2Gv17Ekh1FRi1yjIvlCeK1bfUY93oxzDrq/P4c/j1/K4ROnvY4mOYSKWiSLtKxakskDW/N4q0qMnreZtoOSmL5KQ57k2qmoRbJQ/tA8vNipNt8+1ZKCefPQ85P5DByzkH0a8iTXQEUt4gONbijGD/1bMeDWakxauoM28YlMXLxdt6HLVVFRi/hI3jzBxMZU5/t+rahQLB/9Ry/kic9S2XlQQ54kc1TUIj5W8/rCjHs6kj93qEXyujRi4hMZPW+zVteSYSpqkWwQHGQ8EVWZyQOiqFOuMC+MW8r9I+eyae9Rr6NJAFBRi2Sj8JIFGPV4c169ox7Lth2k3eAkPpixgbPntLqWy1NRi2SzoCDj/mY3MDUuisgqJfnnpJV0Gz6L1Ts15EkuTUUt4pEyRfLxwcMRDL2vEVv2HaPTsBkMnrZGQ57kf6ioRTxkZnRpUJZpcdF0qFeGwdPW0nlYMou2aMiT/H8qahE/ULxAKEO6N+LDhyM4ePw03d6dySuTVnD81Fmvo4kfUFGL+JFba5VmalwU3ZvewMgZv9JucBKz1u/xOpZ4TEUt4mcKh4Xw6h31GP1Ec8zg/pFzeWHcUg5pyFOupaIW8VMtqpRg8oAoekdV5qv5m4mJT2Tail1exxIPqKhF/Fi+0GBe6FCLCX0iKZY/lMc/S6Hf6IXsPXLS62iSjVTUIgGgfvmiTOzbiriY6kxelj7k6btF23Qbei6hohYJEKF5guh/azUm9W9NxRIFGDBmEY99msL2A8e9jiY+pqIWCTDVSxfi26da8pdOtZm9fi9tByXx5dxNnNNt6DmWilokAAUHGY+1qsSUgVE0qFCEP49fxn0j5/DrHg15yolU1CIB7IYS+fnisWa8cWc9Vuw4RPvBSYxIXM+Zs7oNPSdRUYsEODPj3iY3MC0umqjqpXjtp1V0Gz6LlTsOeR1NsoiKWiSHKF04jPd73Mg79zdm+4HjdB6WTPzU1Zw8o9vQA52KWiQHMTM61i9DQmw0XRqUZei/19FpaDILNu/3OppcAxW1SA5UrEAo8fc25OOeTTh68gx3Dp/F379fwbFTZ7yOlmONf2sB499a4JNzq6hFcrCba1zHlNgoHmxWkY9mpg95Sl6rIU+B5opFbWYVzGy6ma00s+VmNiA7golI1igUFsI/bq/L171bkCcoiAc/nMvzYxdz8LiGPAWKjKyozwDPOOdqAc2BPmZW27exRCSrNa1UnJ8GtOapm6rw7YJtxMQnMmX5Tq9jSQZYZmcFmNl3wNvOuYTLHRMREeFSUlKuNZuI+MjSrQd5/tslrNxxiI71yvBylzqUKpTX61gB5bf70dvXpr8rT9lqRf/r63c80zhD5zOzVOdcxKUey9QetZmFA42AuZd4rJeZpZhZSlpaWmZOKyLZrF75IkzsG8lz7WqQsGIXMYMSGbdgq4Y8+akMr6jNrCCQCLzinBv3e8dqRS0SONbtPszzY5ewYPMBoquX4tVu9ShXNJ/XsQLOhRV2RlfQv3XNK2ozCwG+Bb68UkmLSGCpel0hvnmyJS93rs38jftoG5/IZ7M3asiTH8nIVR8GfAisdM7F+z6SiGS34CDjkcj0IU+NKxbjpe+Wc+/7s1mfdsTraELGVtSRQA/gFjNbdP6jg49ziYgHKhTPz2ePNuXNu+qzeudhbhsyg3d/WachTx7L9FUfGaE9apHAt/vwCV6asJzJy3dSt1xh3rizPnXKFvE6Vo6VZVd9iEjucV2hMN7rcSPDH2jMzoMn6fL2TN6csooTpzXkKbupqEXkd91WrwzT4qK4o1E53pm+no5DZ5CycZ/XsXIVFbWIXFHR/KH86+4GfPZoU06cPsfdI2bz8sTlHD2pIU/ZQUUtIhkWVb0UU2OjeLhFOJ/O3kjbQUkkrdENbr6mohaRTCmQNw8vd6nDN71bkDckiIc+msez3yzmwLFTXkfLsVTUInJVIsKL82P/1vS5uQrjF26jTXwSPy3d4XWsHElFLSJXLSwkmOfa1WRi30hKF87LU18u4KkvUtl9+ITX0XIUFbWIXLM6ZYswoU8kf2hfk59X7SYmPolvUrZoyFMWUVGLSJYICQ7iqZuq8NOA1lQvXZDnxi7hoY/msWXfMa+jBTwVtYhkqSqlCvJVrxb8o2sdFmzaT7vBSXwy81cNeboGKmoRyXJBQUaPFuFMiY2iSXhxXv5+BXePmM263Ye9jhaQVNQi4jPli+Xnk55NiL+nAevTjtBhSDLvTF/HaQ15yhQVtYj4lJnRrXF5EmKjialTmjenrKbr2zNZtu2g19EChopaRLJFqUJ5eef+xozocSNpR07S9Z2ZvDFZQ54yQkUtItmqXZ3rmRYbzV2NyzP8l/V0GDKDeb9qyNPvUVGLSLYrkj+EN+6qzxePNePU2XPcM2I2f5mwjCMa8nRJKmoR8UyraiWZGhvFo5GV+GLuJtrGJzJ99W6vY/kdFbWIeCp/aB5e6lybsU+2JH/ePPT8eD5xXy1i/1ENebpARS0ifuHGisWY1L8V/W+pysTF24kZlMikJTt0GzoqahHxI3nzBBPXtgbf92tFmSL56DNqAb0/T2XXodw95ElFLSJ+p1aZwox/uiUv3FaTxDVptIlP5Kv5m3Pt6lpFLSJ+KU9wEL2jqzB5YBS1yhTmD98u5cEP57J5b+4b8qSiFhG/VqlkAcY80Zx/3l6XxVsO0m5wEh8m/8rZXDTkSUUtIn4vKMh4sHlFpsZG0bxycf7xwwruem8Wa3fljiFPKmoRCRhli+bjo0eaMKR7QzbuOUrHockM/Xktp87k7CFPKmoRCShmRteG5ZgWF027utcTn7CGLm8ns3jLAa+j+YyKWkQCUomCeRl2XyNGPhTB/mOnuOPdmbz240qOn8p5Q56uWNRm9pGZ7TazZdkRSEQkM2JqlyYhLpp7m1RgRNIGbhuSxJwNe72OlaUysqL+BGjv4xwiIletcFgIr3Wrz6jHm3HOQff35/Dn8Us5fOK019GyRJ4rHeCcSzKzcN9HERFPfdwxY8f1nOTbHNegZdWSTBkYRXzCaj5M/pV/r9rNK3fU5Zaapb2Odk2ybI/azHqZWYqZpaSlpWXVaUVEMiVfaDB/7libcU9HUjgshEc/SWHAmIXsPXLS62hXzTJyS+b5FfUPzrm6GTlpRESES0lJubZkIiLX6NSZc7z7yzremb6OQmEhvNylDp3rl8HMvI72P8ws1TkXcanHdNWHiORYoXmCGNimOj/0a02F4vnpP3ohT3yWws6DgTXkSUUtIjlejesLMe6plrzYsRbJ6/YQE5/I6HmBM+QpI5fnjQZmAzXMbKuZPeb7WCIiWSs4yHi8dWWmDIyibrkivDBuKfePnMumvUe9jnZFGdqjziztUYuIP3POMWb+Fl6dtJLT587xTEwNHm1VieAg7/autUctInIRM+O+pjeQEBdNq6oleeXHlXR7dyard/rnkCcVtYjkWtcXCWPkQxEMu68RW/cfp9OwGQxKWON3Q55U1CKSq5kZnRuUJSEumo71yjDk57V0GjaDRX405ElFLSICFC8QyuDujfjokQgOnzhDt3dn8s8fVvjFkCcVtYjIRW6pWZqpsVHc1/QGPkj+lXaDk5i1fo+nmVTUIiK/USgshFfuqMeYXs0JMrh/5FxeGLeEg8e9GfKkohYRuYzmlUsweWAUvaMr89X8LbQdlEjCil3ZnkNFLSLyO8JCgnnhtlpM6BNJsfyhPPFZCn1HLWBPNg55UlGLiGRA/fJFmdi3Fc/EVGfq8l3ExCcyYeG2bLkNXUUtIpJBoXmC6HdrNSb1b0V4yQIM/GoRj32awvYDx336vCpqEZFMqla6EGOfbMlLnWoze/1e2g5K4os5mzh3zjeraxW1iMhVCA4yHm1ViamxUTSsUJQXJyyj+8g5HDt1Jsuf64pvxSUiIpdXoXh+Pn+sKd+kbCV1037yh2Z9raqoRUSukZlxT5MK3NOkgk/Or60PERE/p6IWEfFzKmoRET+nohYR8XMqahERP6eiFhHxcypqERE/p6IWEfFz5ovJT2aWBmy6yl9eEvD27RQuTbkyR7kyR7kyJyfmquicK3WpB3xS1NfCzFKccxFe5/gt5coc5coc5cqc3JZLWx8iIn5ORS0i4uf8sajf9zrAZShX5ihX5ihX5uSqXH63Ry0iIv/NH1fUIiJyERW1iIif88uiNrM3zWyVmS0xs/FmVtTrTABmdreZLTezc2bm6aVBZtbezFab2Toz+6OXWS5mZh+Z2W4zW+Z1louZWQUzm25mK8//GQ7wOhOAmYWZ2TwzW3w+19+8znSBmQWb2UIz+8HrLBczs41mttTMFplZitd5LjCzomY29nx3rTSzFll1br8saiABqOucqw+sAV7wOM8Fy4BuQJKXIcwsGHgHuA2oDdxnZrW9zHSRT4D2Xoe4hDPAM865WkBzoI+f/J6dBG5xzjUAGgLtzay5x5kuGACs9DrEZdzsnGvoZ9dSDwEmO+dqAg3Iwt87vyxq59xU59yFd4icA5T3Ms8FzrmVzrnVXucAmgLrnHMbnHOngDFAV48zAeCcSwL2eZ3jt5xzO5xzC85/fpj0f0TlvE0FLt2R8z8NOf/h+Sv8ZlYe6Ah84HWWQGBmhYEo4EMA59wp59yBrDq/Xxb1bzwK/OR1CD9TDthy0c+34gelEyjMLBxoBMz1Nkm681sMi4DdQIJzzh9yDQaeB855HeQSHDDVzFLNrJfXYc6rDKQBH5/fLvrAzApk1ck9K2ozm2Zmyy7x0fWiY/5M+resX/pTLj9gl/ia56uwQGBmBYFvgYHOuUNe5wFwzp11zjUk/TvHpmZW18s8Zv415igAAAGySURBVNYJ2O2cS/Uyx++IdM41Jn3rr4+ZRXkdiPQ3Cm8MDHfONQKOAln22pFn70LunGvze4+b2cNAJ+BWl40Xe18pl5/YClz8dsflge0eZQkYZhZCekl/6Zwb53We33LOHTCzX0jf4/fyxdhIoIuZdQDCgMJm9oVz7kEPM/2Hc277+R93m9l40rcCPX3diPR/k1sv+m5oLFlY1H659WFm7YE/AF2cc8e8zuOH5gPVzKySmYUC3YGJHmfya2ZmpO8frnTOxXud5wIzK3XhqiYzywe0AVZ5mck594JzrrxzLpz0v1v/9peSNrMCZlbowudAW7z9Tw0A59xOYIuZ1Tj/pVuBFVl1fr8sauBtoBCQcP4SnPe8DgRgZneY2VagBTDJzKZ4keP8C619gSmkvyj2tXNuuRdZfsvMRgOzgRpmttXMHvM603mRQA/glvN/pxadXzF6rQww3cyWkP4fcIJzzq8uh/MzpYFkM1sMzAMmOecme5zpgn7Al+f/LBsCr2bViXULuYiIn/PXFbWIiJynohYR8XMqahERP6eiFhHxcypqERE/p6IWEfFzKmoRET/3/wBeQxNkG6tjhQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#To help us perform math operations\n",
    "import numpy as np\n",
    "#to plot our data and model visually\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "#Step 1 - Define our data\n",
    "\n",
    "#Input data - Of the form [X value, Y value, Bias term]\n",
    "X = np.array([\n",
    "    [-2,4,-1],\n",
    "    [4,1,-1],\n",
    "    [1, 6, -1],\n",
    "    [2, 4, -1],\n",
    "    [6, 2, -1],\n",
    "])\n",
    "\n",
    "#Associated output labels - First 2 examples are labeled '-1' and last 3 are labeled '+1'\n",
    "y = np.array([-1,-1,1,1,1])\n",
    "\n",
    "#lets plot these examples on a 2D graph!\n",
    "#for each example\n",
    "for d, sample in enumerate(X):\n",
    "    # Plot the negative samples (the first 2)\n",
    "    if d < 2:\n",
    "        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n",
    "    # Plot the positive samples (the last 3)\n",
    "    else:\n",
    "        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)\n",
    "\n",
    "# Print a possible hyperplane, that is seperating the two classes.\n",
    "#we'll two points and draw the line between them (naive guess)\n",
    "plt.plot([-2,6],[6,0.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's define our loss function (what to minimize) and our objective function (what to optimize)\n",
    "\n",
    "#### Loss function\n",
    "\n",
    "We'll use the Hinge loss. This is a loss function used for training classifiers. The hinge loss is used for \"maximum-margin\" classification, most notably for support vector machines (SVMs).\n",
    "\n",
    "![alt text](http://i.imgur.com/OzCwzyN.png \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "c is the loss function, x the sample, y is the true label, f(x) the predicted label.\n",
    "\n",
    "![alt text](http://i.imgur.com/FZ7JcG3.png \"Logo Title Text 1\")\n",
    "\n",
    " \n",
    "#### Objective Function\n",
    "\n",
    "![alt text](http://i.imgur.com/I5NNu44.png \"Logo Title Text 1\")\n",
    "\n",
    "As you can see, our objective of a SVM consists of two terms. The first term is a regularizer, the heart of the SVM, the second term the loss. The regularizer balances between margin maximization and loss. We want to find the decision surface that is maximally far away from any data points.\n",
    "\n",
    "How do we minimize our loss/optimize for our objective (i.e learn)?\n",
    "\n",
    "We have to derive our objective function to get the gradients! Gradient descent ftw.  As we have two terms, we will derive them seperately using the sum rule in differentiation.\n",
    "\n",
    "\n",
    "![alt text](http://i.imgur.com/6uK3BnH.png \"Logo Title Text 1\")\n",
    "\n",
    "This means, if we have a misclassified sample, we update the weight vector w using the gradients of both terms, else if classified correctly,we just update w by the gradient of the regularizer.\n",
    "\n",
    "\n",
    "\n",
    "Misclassification condition \n",
    "\n",
    "![alt text](http://i.imgur.com/g9QLAyn.png \"Logo Title Text 1\")\n",
    "\n",
    "Update rule for our weights (misclassified)\n",
    "\n",
    "![alt text](http://i.imgur.com/rkdPpTZ.png \"Logo Title Text 1\")\n",
    "\n",
    "including the learning rate η and the regularizer λ\n",
    "The learning rate is the length of the steps the algorithm makes down the gradient on the error curve.\n",
    "- Learning rate too high? The algorithm might overshoot the optimal point.\n",
    "- Learning rate too low? Could take too long to converge. Or never converge.\n",
    "\n",
    "The regularizer controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data. As a regulizing parameter we choose 1/epochs, so this parameter will decrease, as the number of epochs increases.\n",
    "- Regularizer too high? overfit (large testing error) \n",
    "- Regularizer too low? underfit (large training error) \n",
    "\n",
    "Update rule for our weights (correctly classified)\n",
    "\n",
    "![alt text](http://i.imgur.com/xTKbvZ6.png \"Logo Title Text 1\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lets perform stochastic gradient descent to learn the seperating hyperplane between both classes\n",
    "\n",
    "def svm_sgd_plot(X, Y):\n",
    "    #Initialize our SVMs weight vector with zeros (3 values)\n",
    "    w = np.zeros(len(X[0]))\n",
    "    #The learning rate\n",
    "    eta = 1\n",
    "    #how many iterations to train for\n",
    "    epochs = 100000\n",
    "    #store misclassifications so we can plot how they change over time\n",
    "    errors = []\n",
    "\n",
    "    #training part, gradient descent part\n",
    "    for epoch in range(1,epochs):\n",
    "        error = 0\n",
    "        for i, x in enumerate(X):\n",
    "            #misclassification\n",
    "            if (Y[i]*np.dot(X[i], w)) < 1:\n",
    "                #misclassified update for ours weights\n",
    "                w = w + eta * ( (X[i] * Y[i]) + (-2  *(1/epoch)* w) )\n",
    "                error = 1\n",
    "            else:\n",
    "                #correct classification, update our weights\n",
    "                w = w + eta * (-2  *(1/epoch)* w)\n",
    "        errors.append(error)\n",
    "        \n",
    "\n",
    "    #lets plot the rate of classification errors during training for our SVM\n",
    "    plt.plot(errors, '|')\n",
    "    plt.ylim(0.5,1.5)\n",
    "    plt.axes().set_yticklabels([])\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Misclassified')\n",
    "    plt.show()\n",
    "    \n",
    "    return w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:31: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEGCAYAAABxfL6kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAPDklEQVR4nO3df4xsdXnH8fdHLiBqQZCt3gqyXIs11FS9bhrQxlhUWn9EEzQqLS1S622hraiJCvEPo/UfiRpDbYpXC/FXqWJtC1hBgkJj2gJ7LVgoUBCxUqEstWo1VBGf/jFny3Ddu3N2dmd297vvV7KZOd9zZs7z3B+fnP2eM2dSVUiS2vGI9S5AkrS2DHZJaozBLkmNMdglqTEGuyQ1Ztu0d3j44YfX7OzstHcrSZvanj177quqmT7bTj3YZ2dnmZ+fn/ZuJWlTS/KNvts6FSNJjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMQa7JDXGYJekxhjsktQYg12SGmOwS1JjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMZMJdiT7Eoyn2R+YWFhGruUpC1rKsFeVburaq6q5mZmZqaxS0naspyKkaTGGOyS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMQa7JDXGYJekxhjsktQYg12SGmOwS1JjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMYY7JLUGINdkhpjsEtSYwx2SWqMwS5JjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMQa7JDXGYJekxhjsktQYg12SGmOwS1JjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMYY7JLUmKkEe5JdSeaTzC8sLExjl5K0ZU0l2Ktqd1XNVdXczMzMNHYpSVuWUzGS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMQa7JDXGYJekxhjsktQYg12SGmOwS1JjDHZJaozBLkmN2bbcyiSXALWv9VX1sjWvSJK0KssGO/De7vEk4AnAJ7rlk4E7J1STJGkVlg32qroaIMkfV9Vzh1ZdkuTvJ1qZJGksfefYZ5LsWFxIcjTgN2ZI0gY0aipm0ZuAq5Lc0S3PAr83kYokSavSK9ir6rIkxwBP7YZuqaofTq4sSdK4ek3FJHkU8BbgD6vqBuBJSV460cokSWPpO8d+AfAj4Phu+S7g3ROpSJK0Kn2D/clVdQ7wAEBV3Q9kYlVJksbWN9h/lOQgug8rJXky4By7JG1Afa+KeQdwGXBkkk8CzwFeO6miJEnj63tVzBVJvgIcx2AK5syqum+ilUmSxrLsVEySp3aPO4GjgLuBbzG4Kmbn5MuTJK3UqCP2NwO7gPctsa6AE9a8IknSqowK9iu6x9dV1R3LbilJ2hBGXRVzdvf4mUkXIklaG6OO2P8ryZeAo5NcvPdK78cuSRvPqGB/CbAT+DhLz7NLkjaYUfdj/xHwT0meXVULU6pJkrQKo74a7wNV9Ubg/CQ/9RV5TsVI0sYzairm493je5fdSpK0YYyaitnTPV69OJbkUODIqvrqhGuTJI2h7/3Yr0pycJLDgBuAC5K8f7KlSZLG0ffujodU1feAk4ALqupZwAsmV5YkaVx9g31bku3Aq4BLJ1iPJGmV+gb7u4DLgdur6rokO4DbJlfW0mbP+tyqf/Z+n+Hlxef7Gtt7/b7W9X2+FsurHVvL8VHr1mL9Wm6zku0mue0426/X61b72rV4/Vq9xyTeaxLvN46+t+29CLhoaPkO4BWTKkqSNL6+J0/P6U6e7p/kyiT3JTll0sVJklau71TMid3J05cy+CLrpwBv6buTJLuSzCeZX1jwA6ySNEl9g33/7vHFwIVV9e2V7KSqdlfVXFXNzczMrKhASdLK9P3O00uS3ALcD5yRZAb438mVJUkaV68j9qo6CzgemKuqB4AfAC+fZGGSpPGk6qfu7bX0hsnTgGOBRy6OVdXHVrrDubm5mp+fX+nLJGlLS7Knqub6bNtrKibJO4DnMQj2vwNeBHwZWHGwS5Imq+/J01cCzwfuqarTgKcDB06sKknS2PoG+/1V9RPgx0kOBu4FdkyuLEnSuPpeFTOf5LHAh4E9wPeBaydWlSRpbH1vKXBG9/S8JJcBB3s/dknamEZ9Nd7O5dZV1VfWviRJ0mqMOmJ/3zLrCjhhDWuRJK2BUV+N96vTKkSStDb63t3xD7qTp4vLhyY5Y7nXSJLWR9/LHV9fVd9ZXKiq/wZeP5mSJEmr0TfYH5EkiwtJ9gMOmExJkqTV6Hsd++XAp5Ocx+Ck6e8Dl02sKknS2PoG+9uAXcDpQIAvAB+ZVFGSpPH1/YDST4DzGHxA6TDgiKp6cKKVSZLG0veqmKu67zw9DLgeuCDJ+ydbmiRpHH1Pnh7SfefpScAFVfUs4AWTK0uSNK6+wb4tyXbgVcClE6xHkrRKfYP9XQyujLm9qq5LsgO4bXJlSZLG1ffk6UXARUPLdwCvmFRRkqTxjbq741ur6pwkf8Lg+vWHqao3TKwySdJYRh2x39w9+u3TkrRJjLq74yXd40enU44kabVGTcVcvNz6qnrZ2pYjSVqtUVMxxwPfBC4ErmFwOwFJ0gY2KtifALwQOBn4DeBzwIVVddOkC5MkjWfZ69ir6sGquqyqTgWOA24HrkryR1OpTpK0YiOvY09yIPASBkfts8C5wGcnW5YkaVyjTp5+FHga8HngnVV141SqkiSNbdQR+28BPwCeArxh+EuUgKqqgydYmyRpDKOuY+97LxlJ0gZhcEtSYwx2SWqMwS5JjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmOmEuxJdiWZTzK/sLAwjV1K0pY1lWCvqt1VNVdVczMzM9PYpSRtWU7FSFJjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMYY7JLUGINdkhpjsEtSYwx2SWqMwS5JjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMQa7JDXGYJekxhjsktQYg12SGmOwS1JjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMYY7JLUGINdkhpjsEtSYwx2SWqMwS5JjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmOmEuxJdiWZTzK/sLAwjV1K0pY1lWCvqt1VNVdVczMzM9PYpSRtWU7FSFJjDHZJaozBLkmNMdglqTEGuyQ1xmCXpMYY7JLUGINdkhpjsEtSYwx2SWqMwS5JjTHYJakxBrskNcZgl6TGGOyS1BiDXZIaY7BLUmMMdklqjMEuSY0x2CWpMamq6e4wWQC+MebLDwfuW8NyNgN73hrseWtYTc9HVdVMnw2nHuyrkWS+qubWu45psuetwZ63hmn17FSMJDXGYJekxmy2YN+93gWsA3veGux5a5hKz5tqjl2SNNpmO2KXJI1gsEtSYzZNsCf59SS3Jrk9yVnrXc9KJDkyyZeS3JzkpiRnduOHJbkiyW3d46FDrzm76/XWJL82NP6sJP/SrTs3SbrxA5N8qhu/JsnstPtcSpL9kvxzkku75aZ7TvLYJJ9Jckv39338Fuj5Td2/6xuTXJjkka31nOT8JPcmuXFobCo9Jjm128dtSU7tVXBVbfgfYD/ga8AO4ADgBuDY9a5rBfVvB3Z2z38G+DfgWOAc4Kxu/CzgPd3zY7seDwSO7nrfr1t3LXA8EODzwIu68TOA87rnrwE+td59d7W8GfgL4NJuuemegY8Cv9s9PwB4bMs9A08Evg4c1C1/Gnhtaz0DzwV2AjcOjU28R+Aw4I7u8dDu+aEj613v/wg9/1CPBy4fWj4bOHu961pFP38LvBC4FdjejW0Hbl2qP+Dy7s9gO3DL0PjJwIeGt+meb2Pw6basc59HAFcCJ/BQsDfbM3Awg5DLXuMt9/xE4Jtd8GwDLgVObLFnYJaHB/vEexzeplv3IeDkUbVulqmYxX88i+7qxjad7lesZwLXAI+vqrsBusef7TbbV79P7J7vPf6w11TVj4HvAo+bRA8r8AHgrcBPhsZa7nkHsABc0E0/fSTJo2m456r6D+C9wL8DdwPfraov0HDPQ6bR41jZt1mCPUuMbbrrNJM8Bvgr4I1V9b3lNl1irJYZX+416yLJS4F7q2pP35csMbapemZwpLUT+LOqeibwAwa/ou/Lpu+5m1d+OYMph58DHp3klOVessTYpuq5h7XscazeN0uw3wUcObR8BPCtdaplLEn2ZxDqn6yqz3bD/5lke7d+O3BvN76vfu/qnu89/rDXJNkGHAJ8e+076e05wMuS3An8JXBCkk/Qds93AXdV1TXd8mcYBH3LPb8A+HpVLVTVA8BngWfTds+LptHjWNm3WYL9OuCYJEcnOYDByYWL17mm3roz338O3FxV7x9adTGweJb7VAZz74vjr+nOlB8NHANc2/269z9Jjuve87f3es3ie70S+GJ1k3LroarOrqojqmqWwd/XF6vqFNru+R7gm0l+oRt6PvCvNNwzgymY45I8qqv1+cDNtN3zomn0eDlwYpJDu9+OTuzGljftExCrOHHxYgZXk3wNePt617PC2n+Fwa9PXwWu735ezGAO7Urgtu7xsKHXvL3r9Va6M+fd+BxwY7fugzz06eFHAhcBtzM4875jvfseqvl5PHTytOmegWcA893f9d8wuJKh9Z7fCdzS1ftxBleDNNUzcCGDcwgPMDiKft20egR+pxu/HTitT73eUkCSGrNZpmIkST0Z7JLUGINdkhpjsEtSYwx2SWqMwa7mJHkwyfVDP2t2N9Aks8N3+JM2om3rXYA0AfdX1TPWuwhpvXjEri0jyZ1J3pPk2u7n57vxo5JcmeSr3eOTuvHHJ/nrJDd0P8/u3mq/JB/O4B7kX0hy0Lo1JS3BYFeLDtprKubVQ+u+V1W/zOBTfx/oxj4IfKyqfgn4JHBuN34ucHVVPZ3BPV9u6saPAf60qn4R+A7wign3I62InzxVc5J8v6oes8T4ncAJVXVHd1O2e6rqcUnuY3Bf7Qe68bur6vAkC8ARVfXDofeYBa6oqmO65bcB+1fVuyffmdSPR+zaamofz/e1zVJ+OPT8QTxXpQ3GYNdW8+qhx3/snv8DgztQAvwm8OXu+ZXA6fD/39168LSKlFbDIw216KAk1w8tX1ZVi5c8HpjkGgYHNSd3Y28Azk/yFgbfgHRaN34msDvJ6xgcmZ/O4A5/0obmHLu2jG6Ofa6q7lvvWqRJcipGkhrjEbskNcYjdklqjMEuSY0x2CWpMQa7JDXGYJekxvwf8SSrOfiPGIUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "w = svm_sgd_plot(X,y)\n",
    "#they decrease over time! Our SVM is learning the optimal hyperplane"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.quiver.Quiver at 0x21806ff3a48>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUnklEQVR4nO3de5RVdd3H8c83ud/EC5kJCiqS4AXxaLpYy8xaLVLT7KolCmpYIaJSluIlU0wrLyimEAialJVKFy3LZ6W1WqvUg6KC3Jkx0JQpc9nzUBDxff7YZztnrmefmbNn/84579das2D2ObPnK8KbH7/ZZ7a5uwAA4XpX1gMAADpHqAEgcIQaAAJHqAEgcIQaAALXK42T7r333j5y5Mg0Tg0ANWn58uV/c/dh7T2WSqhHjhypfD6fxqkBoCaZ2SsdPcbWBwAEjlADQOAINQAEjlCj4qY+PlVTH5+a9RhAzSDUABC4RKE2s6Fm9pCZrTGz1WZ2fNqDAQAiSS/PmyvpcXf/lJn1kTQgxZkAAEVKhtrMhkg6QdIUSXL3HZJ2pDsWqknr/ej8G/l2jy+etLjHZgJqSZKtjwMlNUlabGbPm9lCMxvY+klmNs3M8maWb2pqqvigAFCvrNSNA8wsJ+nPkia6+9NmNlfS2+5+dUcfk8vlnFcm1q94Jc0KGkjOzJa7e669x5KsqLdI2uLuTxfef0jShEoNBwDoXMlQu/vrkjab2ZjCoQ9JejnVqQAA70h61ccMSUsLV3xsksSrGQCghyQKtbuvkNTu3gnQGnvTQGXxykQACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDAEWoACByhBoDApRLqXbvSOCsA1KdUQv3SS9Ktt0rbtqVxdgCoL6mEeudOadYsadQogg0A3ZUo1GbWaGYvmdkKM8uXen7fvtGPW7dGwT7wQOm22wg2AHRFOSvqD7r7eHfPlXriuHHS/fdLo0dH77/xhnTZZQQbALoila0PM2nyZOnll6NgH3xwdJxgA0D5kobaJf3WzJab2bT2nmBm08wsb2b5pqYmSVKvXlGwV6+W7ruPYANAVyQN9UR3nyDpo5Kmm9kJrZ/g7gvcPefuuWHDhrV4rFcv6ZxzOg72QQdJt98u/etf3fuPAYBalCjU7v5a4cetkpZJOrYrn6yjYL/+unTppdEKm2ADQEslQ21mA81scPxzSR+RtLI7n5RgA0BySVbU+0j6o5m9IOkZSY+5++OV+OTFwV6yJNoCkQg2ABQrGWp33+TuRxbexrn7nEoP0auXdO650po1HQd77lyCDaA+BfVNmToL9iWXEGwA9SmoUMcINgA0CzLUMYINAIGHOlYc7MWL2wb7oIOkO+4g2ABqU1WEOtarlzRlSnOwDzwwOv7Xv0ozZxJsALWpqkIdI9gA6klVhjrWuzfBBlD7qjrUseJg33tv+8G+807p3//OdEwA6JKaCHWsd29p6tT2g33xxdH7BBtAtampUMdKBZsVNoBqUpOhjrUO9qhR0fHXXiPYAKpHTYc6Fgd77Vpp0aL2gz1vHsEGEKa6CHWsd2/pvPPaD/aMGQQbQJjqKtQxgg2gmtRlqGMEG0A1qOtQx4qDvXChNHJkdDwO9sEHS3fdRbABZINQF+ndWzr/fGndupbBfvVV6aKLCDaAbBDqdhBsACEh1J0g2ABCQKgTiIO9dq30/e+3H+zvfU/avj3TMQHUKEJdhj59pAsuaA72AQdEx199VZo+PbpKhGADqDRC3QVxsNetaz/YrLABVBKh7oaOgr1lC8EGUDmEugKSBPvuuwk2gK4h1BVUHOwFC1oG+8tfJtgAuoZQp6BPH+kLXyDYACqDUKeIYAOoBELdA1oHe//9o+NxsEePlu65h2ADaB+h7kFxsNevl+bPbw725s3Sl75EsNP2yuRz9Mrkc7IeAygboc5Anz7StGkEG0AyiUNtZruZ2fNm9miaA9UTgg0giXJW1DMlrU5rkHqWJNjz50s7dmQ7J4BsmLuXfpLZcEn3SZoj6TJ3P7Wz5+dyOc/n8yXP+9n5f0o05I8vPD7R82rFjh3S4sXSnDlRrGMjRkizZ0c36u3TJ7v5qkXr/ehtzz4rSRpwzDEtjh/wg/t7bCagI2a23N1z7T2WdEV9u6TLJe3q5JNMM7O8meWbmpq6MCZiffpIF14obdgQbX2MGBEd37xZ+uIXWWED9abkitrMTpV0srt/2cxOlPSVSq2okcz27dKSJW1X2PvvL115JSvspOIVNitohKi7K+qJkk4zs0ZJD0o6ycweqOB8KKFv32iFvX59yxX2X/7SvMJesIAVNlCrSoba3a9w9+HuPlLSmZJ+5+5npz4Z2igO9t13twz2hRcSbKBWcR11FerbN1pJE2ygPiS66qNc7FH3rO3bo6tEbryx7R727NnSlCnsYQOhq8RVHwhY6xX28OHR8XiFfcgh0ffJZoUNVCdCXUPiYG/YEN1ZJg72K69EL6gh2EB1ItQ1qG/f6BWNBBuoDYS6hhFsoDYQ6jpQKthjxkgLFxJsIFSEuo50FOzGxuj7ZMfB/s9/Mh0TQCuEug4VB/uuu6T99ouOx8E+5BCCDYSEUNexvn2jW4Ft3EiwgZARaiQK9qJFBBvICqHGOzoL9gUXEGwgK4QabcTB3rBBmjePYANZI9ToUL9+0vTpBBvIGqFGSaWCPWaMdO+9BBtIC6FGYq2D/d73RscbGqTzzyfYQFoINcoWB3vjRunOOwk2kDZCjS7r10+66CKCDaSNUKPbSgX7fe+LbmxAsIGuIdSomI6CvWmTdN55BBvoKkKNiisO9h13SPvuGx0n2EDXEGqkpl8/acaMKNAEG+g6Qo3UlQr2oYdKS5ZIO3dmOiYQLEKNHtNRsDdulKZOjVbYBBtoi1Cjx8XB3rhRmjuXYAOlEGpkpn9/6eKLCTZQCqFG5pIE+777CDbqF6FGMFoH+z3viY5v3ChNmRIF+/77CTbqD6FGcOJgb9ok3X57y2AvWsQKG/WHUCNY/ftLM2c2B3vUKGnVqpYrbIKNekCoEbw42CtXSldf3XZL5NBD2RJBbSPUqBoDBrRcYcfB3rBBOvdcgl1pJ54YvSF7hBpVp/WWCMFGrSsZajPrZ2bPmNkLZrbKzK7ricGAUoqDfdttBBu1K8mKerukk9z9SEnjJU0ys+PSHQtIrn9/6ZJLmoO9zz7RcYKNWmHunvzJZgMk/VHSl9z96Y6el8vlPJ/PV2A8oHzbtkkLFkg33SS98Ubz8dGjoy9GnnWW1KtXdvOFqvV+9O9/H/34gQ+0PP7UUz0xTf0xs+XunmvvsUR71Ga2m5mtkLRV0hPtRdrMpplZ3szyTU1N3ZsY6IYBA9pfYa9fL51zjjR2rPSDH7DCRvUod0U9VNIySTPcfWVHz2NFjZBs2ybNny/dfDMr7HLEK2xW0D2j2yvqmLu/JekpSZMqMBfQIwYMkC69NFph33orK2xUnyRXfQwrrKRlZv0lfVjSmrQHAyqtVLDHjZMeeIBgIzxJVtT7SnrSzF6U9KyiPepH0x0LSE9HwV63Tpo8mWAjPGXtUSfFHjWqybZt0j33RHvYW7c2Hz/kkGgP+8wz2cNG+iq2Rw3UogEDpMsukxoapFtukd797ug4K2yEglADBUmD/d//Zjsn6g+hBlopFeyxY6WlSwk2eg6hBjoQB3vTJum7320Z7LPPJtjoOYQaKGHgQGnWrI6DPW4cwUa6CDWQUEfBXruWYCNdhBooU+tgDxsWHS8O9g9/SLBROYQa6KI42A0N0ne+0zLYn/88wUblEGqgmwYOlL7yFYKN9BBqoEIINtJCqIEKKxXsww6TfvQjgo3kCDWQkuJgf/vb0t57R8fXrJE+9zmCjeQINZCygQOlr35Vamwk2OgaQg30EIKNriLUQA+Lg82WCJIi1EBGBg3qPNiHHy49+CDBBqEGMlcc7Jtvbg726tXRjXcJNgg1EIhBg6TLLyfYaItQA4Eh2GiNUAOBKhXsI46Qfvxjgl0PCDUQuOJg33STtNde0fGXX45uvEuwax+hBqrEoEHS174WXYdNsOsLoQaqTNJg79qV6ZioIEINVKlSwT78cOknPyHYtYBQA1UuDnZDg/Stb7UM9mc/S7BrAaEGasTgwdLXv95xsI84gmBXK0IN1JiOgr1qFcGuVoQaqFGtg73nntFxgl19CDVQ4+JgNzZKN97YfrB/+lOC3V3LbnlOy255LpVzE2qgTgweLF1xRfvB/sxnCHbISobazEaY2ZNmttrMVpnZzJ4YDEA6CHb1SbKi3ilplrsfKuk4SdPNbGy6YwFIG8GuHubu5X2A2c8lzXP3Jzp6Ti6X83w+393ZgG46MeHznkpxhurx9tvSvHnSLbdIb77ZfPyww6Rrr5U+8QnpXWyWvqP1fvRr69+SJL139NAWx8+YNSHR+cxsubvn2nusrF92Mxsp6ShJT7fz2DQzy5tZvqmpqZzTAgjAkCHSlVdGV4nMmdO8wl65Uvr0p6Ujj5QeeogVdhYSr6jNbJCk30ua4+6PdPZcVtRA9WOFXZ54hZ10Bd1at1fUZtZb0sOSlpaKNIDawAo7HEmu+jBJiyStdvdb0x8JQEhaB3uPPaLjcbDHj5cefphgpynJinqipMmSTjKzFYW3k1OeC0Bg4mA3Nko33NAc7Jdekj71KYKdprKv+kiCPWqg9r39tnTnndEe9j/+0Xz88MOjPewzzmAPuxwVu+oDAGJDhkizZ7PC7gmEGkC3EOz0EWoAFREHu6FBuv76tsE+6ijpkUcIdlcQagAVtfvu0lVXNQd7aOGFei++KH3ykwS7Kwg1gFTEwW5sJNjdRagBpIpgdx+hBtAjioP9zW+2DfaECdKyZQS7PYQaQI/afXfp6qvbBvuFF6LvH0Kw2yLUADJBsJMj1AAyRbBLI9QAghAHu6FBuu66tsE++mjpZz+TUviuF8Ej1ACCMnSodM01zcHefffo+IoV0fcPmTCh/oJNqAEEKQ52YyPBJtQAgkawCTWAKlEc7G98o76CTagBVJWhQ6Pvd91ZsH/+89oKNqEGUJU6C/bHP15bwSbUAKpaqWAffXT1B5tQA6gJxcG+9trmYD//fPUHm1ADqClDh0Yr6zjYQ4ZEx4uD/YtfVFewCTWAmtRZsE8/vbqCTagB1LQ99qj+YBNqAHWhmoNNqAHUleJgX3NN22DnctIvfxlWsAk1gLq0xx7RS9JbB/u556TTTgsr2IQaQF2rhmATagBQ2MEm1ABQJA52Q0N0I4MQgk2oAaAde+4Z3RosDvbgwdHxONjHHCM9+mjPBJtQA0An4mA3NrYM9vLl0sc+1jPBJtQAkECWwS4ZajO718y2mtnKyn96AKguxcG+6qqeCXaSFfUSSZMq9ykBoPrtuad0/fUdB/vYY6XHHqtMsM0TnMXMRkp61N0PS3LSXC7n+Xy+e5MB6FmLT0n2vKmPpTtHlXrzTem226S5c6V//rP5eC4XvRLy5JMls44/3syWu3uuvccqtkdtZtPMLG9m+aampkqdFgCqQkcr7HxeOvXU7q2wWVEDQAr+/vdohX3HHclW2D2yogYANNtrL+mGG6LrsGfPlgYNio7HK+z3v1/61a+SrbAJNQCkKA52Y2PLYD/7rHTKKc3B7kySy/N+JOlPksaY2RYzO7/bkwNAnSkV7M6UDLW7n+Xu+7p7b3cf7u6LKjAzANSl4mBfeWVzsDvD1gcAZGCvvaQ5c5qD3RlCDQAZioPdGUINAIEj1AAQuEQveCn7pGZNkl7p4ofvLelvFRynUpirPMxVHuYqTy3OdYC7D2vvgVRC3R1mlu/o1TlZYq7yMFd5mKs89TYXWx8AEDhCDQCBCzHUC7IeoAPMVR7mKg9zlaeu5gpujxoA0FKIK2oAQBFCDQCBCzLUZvYdM1tjZi+a2TIzG5r1TJJkZp82s1VmtsvMMr00yMwmmdlaM9tgZl/PcpZiod4M2cxGmNmTZra68P9wZtYzSZKZ9TOzZ8zshcJc12U9U8zMdjOz583s0axnKWZmjWb2kpmtMLNg7lBiZkPN7KFCu1ab2fGVOneQoZb0hKTD3P0ISeskXZHxPLGVkj4h6Q9ZDmFmu0m6S9JHJY2VdJaZjc1ypiJLFObNkHdKmuXuh0o6TtL0QH7Ntks6yd2PlDRe0iQzOy7jmWIzJa3OeogOfNDdxwd2LfVcSY+7+/skHakK/toFGWp3/6277yy8+2dJw7OcJ+buq919bdZzSDpW0gZ33+TuOyQ9KOn0jGeSJLn7HyS9mfUcrbn7X939ucLP/6noD9F+2U4leeR/C+/2Lrxl/hV+Mxsu6RRJC7OepRqY2RBJJ0haJEnuvsPd36rU+YMMdSvnSfp11kMEZj9Jm4ve36IAolMtCvcAPUrS09lOEilsMayQtFXSE+4ewly3S7pc0q6sB2mHS/qtmS03s2lZD1NwoKQmSYsL20ULzWxgpU6eWajN7H/MbGU7b6cXPWe2on+yLg1prgC0d9P5zFdh1cDMBkl6WNIl7v521vNIkrv/193HK/qX47Fmlugm0mkxs1MlbXX35VnO0YmJ7j5B0dbfdDM7IeuBJPWSNEHS3e5+lKT/k1Sxrx31qtSJyuXuH+7scTM7V9Kpkj7kPXixd6m5ArFF0oii94dLei2jWaqGmfVWFOml7v5I1vO05u5vmdlTivb4s/xi7ERJp5nZyZL6SRpiZg+4+9kZzvQOd3+t8ONWM1umaCsw068bKfozuaXoX0MPqYKhDnLrw8wmSfqapNPcfVvW8wToWUmjzWyUmfWRdKakX2Q8U9DMzBTtH65291uznidmZsPiq5rMrL+kD0tak+VM7n5F4bZ7IxX93vpdKJE2s4FmNjj+uaSPKNu/1CRJ7v66pM1mNqZw6EOSXq7U+YMMtaR5kgZLeqJwCc49WQ8kSWZ2hpltkXS8pMfM7DdZzFH4QutFkn6j6ItiP3H3VVnM0lrAN0OeKGmypJMKv6dWFFaMWdtX0pNm9qKiv4CfcPegLocLzD6S/mhmL0h6RtJj7v54xjPFZkhaWvh/OV7SjZU6MS8hB4DAhbqiBgAUEGoACByhBoDAEWoACByhBoDAEWoACByhBoDA/T+/eLc+WPCXVgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for d, sample in enumerate(X):\n",
    "    # Plot the negative samples\n",
    "    if d < 2:\n",
    "        plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n",
    "    # Plot the positive samples\n",
    "    else:\n",
    "        plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)\n",
    "\n",
    "# Add our test samples\n",
    "plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')\n",
    "plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')\n",
    "\n",
    "# Print the hyperplane calculated by svm_sgd()\n",
    "x2=[w[0],w[1],-w[1],w[0]]\n",
    "x3=[w[0],w[1],w[1],-w[0]]\n",
    "\n",
    "x2x3 =np.array([x2,x3])\n",
    "X,Y,U,V = zip(*x2x3)\n",
    "ax = plt.gca()\n",
    "ax.quiver(X,Y,U,V,scale=1, color='blue')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}