slides/lecture14-kernels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i_f5u2x9nn6I",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "\n",
    "# Lecture 14: Kernels\n",
    "\n",
    "### Applied Machine Learning\n",
    "\n",
    "__Volodymyr Kuleshov__<br>Cornell Tech"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Announcements\n",
    "\n",
    "* Homework 3 is due this week\n",
    "* We have released a practice prelim\n",
    "* Prelim is in-class next Wednesday, review session on Monday"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# Part 1: The Kernel Trick in SVMs\n",
    "\n",
    "So far, the majority of the machine learning models we have seen have been *linear*.\n",
    "\n",
    "In this lecture, we will see a general way to make many of these models *non-linear*. We start with SVMs as a motivating example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Binary Classification\n",
    "\n",
    "Consider a training dataset $\\mathcal{D} = \\{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \\ldots, (x^{(n)}, y^{(n)})\\}$.\n",
    "\n",
    "We distinguish between two types of supervised learning problems depending on the targets $y^{(i)}$. \n",
    "\n",
    "1. __Regression__: The target variable $y \\in \\mathcal{Y}$ is continuous:  $\\mathcal{Y} \\subseteq \\mathbb{R}$.\n",
    "2. __Binary Classification__: The target variable $y$ is discrete and takes on one of $K=2$ possible values.\n",
    "\n",
    "In this lecture, we assume $\\mathcal{Y} = \\{-1, +1\\}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: SVM Model Family\n",
    "\n",
    "We will consider models of the form\n",
    "\n",
    "\\begin{align*}\n",
    "f_\\theta(x) = \\theta^\\top \\phi(x) + \\theta_0\n",
    "\\end{align*}\n",
    "\n",
    "where $x$ is the input and $y \\in \\{-1, 1\\}$ is the target. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: Primal and Dual Formulations\n",
    "\n",
    "Recall that the the max-margin hyperplane can be formulated as the solution to the following *primal* optimization problem.\n",
    "\\begin{align*}\n",
    "\\min_{\\theta,\\theta_0, \\xi}\\; & \\frac{1}{2}||\\theta||^2 + C \\sum_{i=1}^n \\xi_i \\;  \\\\\n",
    "\\text{subject to } \\; & y^{(i)}((x^{(i)})^\\top\\theta+\\theta_0)\\geq 1 - \\xi_i \\; \\text{for all $i$} \\\\\n",
    "& \\xi_i \\geq 0\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The solution to this problem also happens to be given by the following *dual* problem:\n",
    "\\begin{align*}\n",
    "\\max_{\\lambda} & \\sum_{i=1}^n \\lambda_i - \\frac{1}{2} \\sum_{i=1}^n \\sum_{k=1}^n \\lambda_i \\lambda_k y^{(i)} y^{(k)} (x^{(i)})^\\top x^{(k)}  \\\\\n",
    "\\text{subject to } \\; & \\sum_{i=1}^n \\lambda_i y^{(i)} = 0 \\\\\n",
    "& C \\geq \\lambda_i \\geq 0 \\; \\text{for all $i$}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Primal Solution\n",
    "\n",
    "We can obtain a primal solution from the dual via the following equation:\n",
    "$$\n",
    "\\theta^* = \\sum_{i=1}^n \\lambda_i^* y^{(i)} x^{(i)}.\n",
    "$$\n",
    "\n",
    "Ignoring the $\\theta_0$ term for now, the score at a new point $x'$ will equal\n",
    "$$\n",
    "(\\theta^*)^\\top x' = \\sum_{i=1}^n \\lambda_i^* y^{(i)}(x^{(i)})^\\top x'.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Kernel Trick in SVMs\n",
    "\n",
    "Notice that in both equations, the features $x$ are never used directly. Only their *dot product* is used.\n",
    "\\begin{align*}\n",
    "J(\\lambda) &= \\sum_{i=1}^n \\lambda_i - \\frac{1}{2} \\sum_{i=1}^n \\sum_{k=1}^n \\lambda_i \\lambda_k y^{(i)} y^{(k)} (x^{(i)})^\\top x^{(k)}  \\\\\n",
    "(\\theta^*)^\\top x' & = \\sum_{i=1}^n \\lambda_i^* y^{(i)}(x^{(i)})^\\top x'.\n",
    "\\end{align*}\n",
    "\n",
    "If we can compute the dot product efficiently, we can potentially use very complex features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: Polynomial Regression\n",
    "\n",
    "Recall that a $p$-th degree polynomial\n",
    "\n",
    "$$\n",
    "a_p x^p + a_{p-1} x^{p-1} + ... + a_{1} x + a_0.\n",
    "$$\n",
    "\n",
    "forms a linear model with parameters $a_p, a_{p-1}, ..., a_0$.\n",
    "This means we can use our algorithms for linear models to learn non-linear features!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Specifically, given a one-dimensional continuous variable $x$, we can define a feature function $\\phi : \\mathbb{R} \\to \\mathbb{R}^p$ as\n",
    "\n",
    "$$\\phi(x) = \\begin{bmatrix}\n",
    "1 \\\\\n",
    "x \\\\\n",
    "x^2 \\\\\n",
    "\\vdots \\\\\n",
    "x^p\n",
    "\\end{bmatrix}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Then the class of models of the form\n",
    "$$ f_\\theta(x) := \\sum_{j=0}^p \\theta_j x^j = \\theta^\\top \\phi(x) $$\n",
    "with parameters $\\theta$ encompasses the set of $p$-degree polynomials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* It is non-linear in the input variable $x$, meaning that we can represent complex functions.\n",
    "* It is a linear model as a function of the parameters $\\theta$, meaning that we can use our familiar ordinary least squares algorithm to learn these features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Kernel Trick: A First Example\n",
    "\n",
    "Can we compute the dot product $\\phi(x)^\\top \\phi(x')$ of polynomial features $\\phi(x)$ more efficiently than using the standard definition of a dot product? \n",
    "\n",
    "Let's look at an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To start, consider polynomial features $\\phi : \\mathbb{R}^d \\to \\mathbb{R}^{d^2}$ of the form\n",
    "$$ \\phi(x)_{ij} = x_i x_j \\;\\text{ for $i,j \\in \\{1,2,\\ldots,d\\}$}. $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For $d=3$ this looks like\n",
    "$$ \\small \\phi(x) = \\begin{bmatrix}\n",
    "x_1 x_1 \\\\\n",
    "x_1 x_2 \\\\\n",
    "x_1 x_3 \\\\\n",
    "x_2 x_1 \\\\\n",
    "x_2 x_2 \\\\\n",
    "x_3 x_3 \\\\\n",
    "x_3 x_1 \\\\\n",
    "x_3 x_2 \\\\\n",
    "x_3 x_3 \\\\\n",
    "\\end{bmatrix}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The product of $x$ and $z$ in feature space equals:\n",
    "$$ \\phi(x)^\\top \\phi(z) = \\sum_{i=1}^d \\sum_{j=1}^d x_i x_j z_i z_j $$\n",
    "Normally, computing this dot product involves a sum over $d^2$ terms and takes $O(d^2)$ time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "An alternative way of computing the dot product $\\phi(x)^\\top \\phi(z)$ is to instead compute $(x^\\top z)^2$. One can check that this has the same result:\n",
    "\\begin{align*}\n",
    "(x^\\top z)^2 & = (\\sum_{i=1}^d x_i z_i)^2 \\\\\n",
    "& = (\\sum_{i=1}^d x_i z_i) \\cdot (\\sum_{j=1}^d x_j z_j) \\\\\n",
    "& = \\sum_{i=1}^d \\sum_{j=1}^d x_i z_i x_j z_j \\\\\n",
    "& = \\phi(x)^\\top \\phi(z)\n",
    "\\end{align*}\n",
    "What is the big-O runtime for computing this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Computing $(x^\\top z)^2$ can be done in only $O(d)$ time! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This is a very powerful idea:\n",
    "* We can compute the dot product between $O(d^2)$ features in only $O(d)$ time.\n",
    "* We can use high-dimensional features within ML algorithms that only rely on dot products without incurring extra costs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Kernel Trick: Polynomial Features\n",
    "\n",
    "The number of polynomial features $\\phi_p$ of degree $p$ when $x \\in \\mathbb{R}^d$ \n",
    "\n",
    "$$ \\phi_p(x)_{i_1, i_2, \\ldots, i_p} = x_{i_1} x_{i_2} \\cdots x_{i_p} \\;\\text{ for $i_1, i_2,  \\ldots, i_p \\in \\{1,2,\\ldots,d\\}$} $$\n",
    "\n",
    "scales as $O(d^p)$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, we can compute the dot product $\\phi_p(x)^\\top \\phi_p(z)$ in this feature space in only $O(d)$ time for any $p$ as:\n",
    "$$\\phi_p(x)^\\top \\phi_p(z) = (x^\\top z)^p.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Kernel Trick in SVMs\n",
    "\n",
    "More generally, given features $\\phi(x)$, suppose that we have a function $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ that outputs dot products between vectors in $\\mathcal{X}$\n",
    "\n",
    "$$ K(x, z) = \\phi(x)^\\top \\phi(z). $$\n",
    "\n",
    "We will call $K$ the *kernel function*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Recall that an example of a useful kernel function is\n",
    "$$K(x,z) = (x \\cdot z)^p$$\n",
    "because it computes the dot product of polynomial features of degree $p$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Notice that we can rewrite the dual of the SVM as\n",
    "\\begin{align*}\n",
    "\\max_{\\lambda} & \\sum_{i=1}^n \\lambda_i - \\frac{1}{2} \\sum_{i=1}^n \\sum_{k=1}^n \\lambda_i \\lambda_k y^{(i)} y^{(k)} K(x^{(i)}, x^{(k)})  \\\\\n",
    "\\text{subject to } \\; & \\sum_{i=1}^n \\lambda_i y^{(i)} = 0 \\\\\n",
    "& C \\geq \\lambda_i \\geq 0 \\; \\text{for all $i$}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Also, the predictions at a new point $x'$ are given by \n",
    "$$(\\theta^*)^\\top \\phi(x') = \\sum_{i=1}^n \\lambda_i^* y^{(i)}\\phi(x^{(i)})^\\top \\phi(x') = \\sum_{i=1}^n \\lambda_i^* y^{(i)} K(x^{(i)}, x').$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "__Note__: This is a non-parametric algorithm like k-NN. In fact, an SVM outputs the \"weighted average label\" $\\sum_{i=1}^n \\lambda_i^* y^{(i)} K(x^{(i)}, x')$, with weights $\\lambda_i^* K(x^{(i)}, x')$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Algorithm: Kernelized Support Vector Machine Classification (Dual Form)\n",
    "\n",
    "* __Type__: Supervised learning (binary classification)\n",
    "* __Model family__: Non-linear decision boundaries.\n",
    "* __Objective function__: Dual of SVM optimization problem.\n",
    "* __Optimizer__: Sequential minimal optimization.\n",
    "* __Probabilistic interpretation__: No simple interpretation!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Kernel Trick: General Idea\n",
    "\n",
    "Many types of features $\\phi(x)$ have the property that their dot product $\\phi(x)^\\top \\phi(z)$ can be computed more efficiently than if we had to form these features explicitly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Also, we will see that many algorithms in machine learning can be written down as optimization problems in which the features $\\phi(x)$ only appear as dot products $\\phi(x)^\\top \\phi(z)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The *Kernel Trick* means that we can use complex non-linear features within  these algorithms with little additional computational cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Examples of algorithms in which we can use the Kernel trick:\n",
    "* Supervised learning algorithms: linear regression, logistic regression, support vector machines, etc.\n",
    "* Unsupervised learning algorithms: PCA, density estimation.\n",
    "\n",
    "We will look at more examples shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "\n",
    "# Part 2: Kernelized Ridge Regression\n",
    "\n",
    "Support vector machines are far from being the only algorithm that benefits from kernels.\n",
    "\n",
    "Another algorithm that supports kernels is Ridge regression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: Ridge Regression\n",
    "\n",
    "Recall that a linear model has the form\n",
    "$$ f_\\theta(x) = \\theta^\\top \\phi(x). $$\n",
    "where $\\phi(x)$ is a vector of features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We pick $\\theta$ to minimize the (L2-regularized) mean squared error (MSE):\n",
    "$$J(\\theta)= \\frac{1}{2n} \\sum_{i=1}^n(y^{(i)} - \\theta^\\top \\phi(x^{(i)}))^2 + \\frac{\\lambda}{2}\\sum_{j=1}^d \\theta_j^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Featurized Design Matrix\n",
    "\n",
    "It is useful to represent the featurized dataset as a matrix $\\Phi \\in \\mathbb{R}^{n \\times p}$:\n",
    "\n",
    "$$ \\Phi = \\begin{bmatrix}\n",
    "\\phi(x^{(1)})_1 & \\phi(x^{(1)})_2 & \\ldots & \\phi(x^{(1)})_p \\\\\n",
    "\\phi(x^{(2)})_1 & \\phi(x^{(2)})_2 & \\ldots & \\phi(x^{(2)})_p \\\\\n",
    "\\vdots \\\\\n",
    "\\phi(x^{(n)})_1 & \\phi(x^{(n)})_2 & \\ldots & \\phi(x^{(n)})_p\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\begin{bmatrix}\n",
    "- & \\phi(x^{(1)})^\\top & - \\\\\n",
    "- & \\phi(x^{(2)})^\\top & - \\\\\n",
    "& \\vdots & \\\\\n",
    "- & \\phi(x^{(n)})^\\top & - \\\\\n",
    "\\end{bmatrix}\n",
    ".$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Featurized Normal Equations\n",
    "\n",
    "The normal equations provide a closed-form solution for $\\theta$.\n",
    "<!-- $$ \\theta = (X^\\top X  + \\lambda I)^{-1} X^\\top y.$$ -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When the vectors of attributes $x^{(i)}$ are featurized, we can write this as\n",
    "$$ \\theta = (\\Phi^\\top \\Phi + \\lambda I)^{-1} \\Phi^\\top y.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Push-Through Matrix Identity\n",
    "\n",
    "We can modify this expression by using a version of the [push-through matrix identity](https://en.wikipedia.org/wiki/Woodbury_matrix_identity#Discussion):\n",
    "\n",
    "$$ (\\lambda I + U V)^{-1} U = U (\\lambda I + V U)^{-1} $$\n",
    "\n",
    "where $U \\in \\mathbb{R}^{n \\times m}$ and $V \\in \\mathbb{R}^{m \\times n}$ and $\\lambda \\neq 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Proof sketch: Start with $U (\\lambda I + V U) = (\\lambda I + U V) U$ and multiply both sides by $(\\lambda I + V U)^{-1}$ on the right and $(\\lambda I + U V)^{-1}$ on the left."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Normal Equations: Dual Form\n",
    "\n",
    "We can apply the identity $(\\lambda I + U V)^{-1} U = U (\\lambda I + V U)^{-1}$ to the normal equations with $U=\\Phi^\\top$ and $V=\\Phi$.\n",
    "\n",
    "$$ \\theta = (\\Phi^\\top \\Phi + \\lambda I)^{-1} \\Phi^\\top y$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "to obtain the *dual* form:\n",
    "\n",
    "$$ \\theta = \\Phi^\\top (\\Phi \\Phi^\\top + \\lambda I)^{-1} y.$$\n",
    "\n",
    "The first approach takes $O(p^3)$ time; the second is $O(n^3)$ and is faster when $p > n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Feature Representations for Parameters\n",
    "\n",
    "An interesting corollary of the dual form\n",
    "$$ \\theta = \\Phi^\\top \\underbrace{(\\Phi \\Phi^\\top + \\lambda I)^{-1} y}_\\text{call this vector $\\alpha$} = \\Phi^T \\alpha$$\n",
    "is that the optimal $\\theta$ is a linear combination of the $n$ training set features:\n",
    "$$ \\theta = \\sum_{i=1}^n \\alpha_i \\phi(x^{(i)}). $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Here, the weights $\\alpha_i$ are derived from $(\\Phi \\Phi^\\top + \\lambda I)^{-1} y$ and equal\n",
    "$$\\alpha_i = \\sum_{j=1}^n L_{ij} y_j$$\n",
    "where $L = (\\Phi \\Phi^\\top + \\lambda I)^{-1}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Predictions From Feature Products\n",
    "\n",
    "Consider now a prediction $\\phi(x')^\\top \\theta$ at a new input $x'$:\n",
    "$$\\phi(x')^\\top \\theta = \\sum_{i=1}^n \\alpha_i \\phi(x')^\\top \\phi(x^{(i)}).$$\n",
    "\n",
    "The crucial observation is that the features $\\phi(x)$ are never used directly in this equation. Only their dot product is used!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning From Feature Products\n",
    "\n",
    "We also don't need features $\\phi$ for learning $\\theta$, just their dot product! \n",
    "First, recall that each row $i$ of $\\Phi$ is the $i$-th featurized input $\\phi(x^{(i)})^\\top$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Thus $K = \\Phi \\Phi^\\top$ is a matrix of all dot products between all the $\\phi(x^{(i)})$\n",
    "$$K_{ij} = \\phi(x^{(i)})^\\top \\phi(x^{(j)}).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can compute $\\alpha = (K+\\lambda I)^{-1}y$ and use it for predictions\n",
    "$$\\phi(x')^\\top \\theta = \\sum_{i=1}^n \\alpha_i \\phi(x')^\\top \\phi(x^{(i)}).$$\n",
    "and all this only requires dot products, not features $\\phi$!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Algorithm: Kernelized Ridge Regression\n",
    "\n",
    "* __Type__: Supervised learning (Regression)\n",
    "* __Model family__: Polynomials.\n",
    "* __Objective function__: $L2$-regularized ridge regression.\n",
    "* __Optimizer__: Normal equations (dual form)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "# Part 3: Types of Kernels\n",
    "\n",
    "We have seen two examples of kernelized algorithms. Let's now look at some additional examples of kernels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Kernelized Ridge Regression\n",
    "\n",
    "The normal equations provide a closed-form solution for $\\theta$:\n",
    "\n",
    "$$ \\theta = (\\Phi^\\top \\Phi + \\lambda I)^{-1} \\Phi^\\top y.$$\n",
    "\n",
    "They also can be written in this form:\n",
    "\n",
    "$$ \\theta = \\Phi^\\top (\\Phi \\Phi^\\top + \\lambda I)^{-1} y.$$\n",
    "\n",
    "The first approach takes $O(d^3)$ time; the second is $O(n^3)$ and is faster when $d > n$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "An interesting corollary is that the optimal $\\theta$ is a linear combination of the $n$ training set features:\n",
    "$$ \\theta = \\sum_{i=1}^n \\alpha_i \\phi(x^{(i)}). $$\n",
    "We can compute a prediction $\\phi(x')^\\top \\theta$ for $x'$ without ever using the features (only their dot products):\n",
    "$$\\phi(x')^\\top \\theta = \\sum_{i=1}^n \\alpha_i \\phi(x')^\\top \\phi(x^{(i)}).$$\n",
    "Equally importantly, we can learn $\\theta$ from only dot products."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Review: Kernelized SVMs\n",
    "\n",
    "We can rewrite the dual of the SVM as\n",
    "\\begin{align*}\n",
    "\\max_{\\lambda} & \\sum_{i=1}^n \\lambda_i - \\frac{1}{2} \\sum_{i=1}^n \\sum_{k=1}^n \\lambda_i \\lambda_k y^{(i)} y^{(k)} K(x^{(i)}, x^{(k)})  \\\\\n",
    "\\text{subject to } \\; & \\sum_{i=1}^n \\lambda_i y^{(i)} = 0 \\\\\n",
    "& C \\geq \\lambda_i \\geq 0 \\; \\text{for all $i$}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Also, the predictions at a new point $x'$ are given by $$\\sum_{i=1}^n \\lambda_i^* y^{(i)} K(x^{(i)}, x').$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Definition: Kernels\n",
    "\n",
    "The *kernel* corresponding to features $\\phi(x)$ is a function $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ that outputs dot products between vectors in $\\mathcal{X}$\n",
    "$$ K(x, z) = \\phi(x)^\\top \\phi(z). $$\n",
    "\n",
    "We will also consider general functions $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ and call these *kernel functions*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Kernels have multiple interpretations:\n",
    "* The dot product or geometrical angle between $x$ and $z$\n",
    "* A notion of similarity between $x$ and $z$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We will look at a few examples of kernels using the following dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-3.0, 3.0)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfZklEQVR4nO3dd3gVZd7G8e8vhRBCD6EnBgXEDhIpiiuroKgoi4JixbKLYsG66oqKDRVRUbEgdrFR7NjABcSyKEFBqYLSWxIIoSQQkvO8f5DllU2Fc3ImmXN/rivXxTnznJl7BG6HmWfmmHMOERHxjyivA4iISGip2EVEfEbFLiLiMyp2ERGfUbGLiPiMil1ExGeCLnYzq2lmP5rZPDNbYGb3hSKYiIgcGAt2HruZGZDgnNtuZrHAt8ANzrlZoQgoIiL7JybYFbg9/2fYXvQytuhHdz2JiHgk6GIHMLNoYA7QGnjWOfdDCWMGAYMAEhISOrZr1y4UmxYRiRhz5szJcs4llTcu6FMx+6zMrD7wAXC9c25+aePS0tJcenp6yLYrIhIJzGyOcy6tvHEhnRXjnNsCTAd6hXK9IiJScaGYFZNUdKSOmcUDPYHFwa5XREQOTCjOsTcDXi86zx4FTHDOTQ7BekVE5ACEYlbML0CHEGQREZEQ0J2nIiI+o2IXEfEZFbuIiM+o2EVEfEbFLiLiMyp2ERGfUbGLiPiMil1ExGdU7CIiPqNiFxHxGRW7iIjPqNhFRHxGxS4i4jMqdhERn1Gxi4j4jIpdRMRnVOwiIj6jYhcR8RkVu4iIz6jYRUR8RsUuIuIzKnYREZ9RsVdD27Zt49FHH+WQNm2JiYmhYWIig6+5lmXLlnkdTXwoEAjw9ttv06Vje2rExlKrZhxnn9GLmTNneh1NSqFir2ays7M5vtuJfPjVN1x29+O8+v1v3PvGp2QWxNK5S1e+++47ryOKjwQCAS696ALuv/U6TknI4q2+B/NS74NomTWP8/r0ZvRTT3odUUpgzrmwbzQtLc2lp6eHfbt+cMFFF5Hj4rj41vsws32Wzft+Bq88cCurV64kLi7Oo4TiJ2PHjuWp++7g3hOSiIvZ9zgwY8du7pixgX/P/I5jjjnGo4SRxczmOOfSyhunI/ZqZOPGjXz66af0vermYqUOcMzx3Wl5SDsmTZrkQTrxG+ccTz42ggGH1i5W6gCNE2Lp1SqB0U+O8iCdlEXFXo2kp6fT9qhjqV23fqljjjmxBzO+1rlPCd62bdtYsWo1RzepVeqYTs1q8c3X08OYSioi6GI3s2Qzm25mC81sgZndEIpgUpyZ4QKBMscEAoESj+ZF9td//xyVdbI24NCftyooFEfsBcAtzrnDgS7AtWZ2eAjWK/+jc+fOLJ3/M1uzN5c6Zu7XX3LKyX8NYyrxqzp16tDm4IOZu35HqWNmrc+j+yk9wphKKiLoYnfOrXfO/VT0623AIqBFsOuV4hITE+l7zrlMfOYRSrroPXva52SuWUnfvn09SCd+dPPt/+LtJdvJ3V1YbNm6bflMXb6dITfe7EEyKUtIZ8WYWSowEzjSObe1tHGaFXPgtm7dyik9TyUQG89pF/2D1HZHsSVrIzM/Gs8PUz7m888+JS2t3IvmIhXinOPqf/ydqZ+8T5+D4+nQLIHdhY5vV29n8h87eOTxUVx55d+9jhkxKjorJmTFbma1ga+B4c6590tYPggYBJCSktJx5cqVIdluJMrLy+O1117j+RdeZNXK5dSpW48LBwzguuuuJTk52et44jPOOT755BOefmIkP/08l+joaE477TRuvOWfOogIs7AWu5nFApOBL51zT5Q3XkfsIiL7L2zz2G3PJfGXgUUVKXUREalcoZgVcwJwCXCymc0t+jkjBOsVEZEDEBPsCpxz3wKayCoiUkXozlMREZ9RsYuI+IyKXUTEZ1TsIiI+o2IXEfEZFbuIiM+o2EVEfEbFLiLiMyp2ERGfUbGLiPiMil1ExGdU7CIiPqNiFxHxGRW7iIjPqNhFRHxGxS4i4jMqdhERn1Gxi4j4TNBfjSdSGfLz85k2bRoZGRm0bNmSk046iejoaK9jiVQLKnapcp4fM4Zhw+6lccuDaNSsJetX/k7u1i088dhI+vfv73U8kSpPxS5Vyqgnn+SJp5/hltHjSGlz2N73l8ydzbVDriUQCHD++ed7mFCk6jPnXNg3mpaW5tLT08O+XanacnJySDkolfvGTaZJy4OKLV82/2eev2Mwq1auICZGxyQSecxsjnMurbxxungqVcbEiRM5stMJJZY6QOsjO9CgSTOmTJkS5mQi1YuKXaqMlStX0rRV2zLHtDjkUFatWhWmRCLVk4pdqoxGjRqxJXN9mWOyN64nMTExTIlEqicVu1QZ/fv3Z/a0z9mxLafE5ZnrVrN0/s+cccYZYU5W/TnnyMjIYPXq1RQUFHgdRyqZil2qjObNm3PJxZfwzO2Dyd2+bZ9lOZsyGX3b1dz2z3+SkJDgUcLq6e2336b9kYfRutVBHHvU4TRvksRdQ+8kNzfX62hSSTQrRqqUgoICrrt+CO+++y6dTz2LxGbJbFy5jNnTv2DIddfzwAP3Y2Zex6w27rrzDt56aQyXHl6HDs0SiDJjVc4uJizZRn79ZKbN/JZatWp5HVMqKKyzYszsFTPLMLP5oVifRK6YmBjGPP8c8+b+TPf27WhiuZx5YieWLlnCgw8+oFLfD+np6bz4/LPc3y2Jjs1rE1X03y6lXhy3HJdIbPZqRjzysMcppTKE5IjdzP4CbAfecM4dWd54HbGLVL7LLrkIFkzjnHb1S1y+cssuhs/OYc36jbovoJoI6xG7c24msDkU6xKR0Pg5fTZHJcWVuvyg+nEU5O8iIyMjjKkkHMJ28dTMBplZupmlZ2ZmhmuzIhErtkYNdhYESl1eGHDs2l1AXFzp5S/VU9iK3Tk31jmX5pxLS0pKCtdmRSJWn3P78936/FKXz1m/nXZt2+q+AB/SdEcRn/rHoKv4cV0e8zbsKLZsS14B4xZu57ahd1doXTNnzqR/3z6ktmxG64OSuW7w1SxZsiTUkSVEVOwiPtW0aVPe++hjnvoph+d/zuaXDTtYkpXHxEXZ3DJ9A1dcM6TcxyA757j9n7dyQd+zaLj6R25vX4shh8eQ9f2HdD2uIxMnTgzT3sj+CMmlcDN7B+gONDKzNcAw59zLoVi3iBy4k046iQVLfuOlF8fy0XsTyc/PJ63TSUx99Sbat29f7ucnTpzIhNdfYkT3JtSN+/8vOkltUJOuzeO56srL6dChA61bt67EvZD9pRuURKRUnTocQ886m+jcsk6Jy9/4NZvkv/Zn1NOjw5wsMumxvSISlO3bt/PrwkWkNa9d6pjjW8Tz5eefhjGVVISKXURKVFhYSFSUEVXGzb4xUeihYlWQil1ESlS3bl2aNWnCoqy8Usf8tCGPzl2PD2MqqQgVu4iUyMwYcvOtjF+yg4JA8Wtx2XkFfL58B9ffeLMH6aQsKnYRKdU111xD8pFp3P99Jr9u3IFzjt2Fjpkrt3LXNxlcd+MtdOrUyeuY8j/05B8RKVVMTAzvfzyZMWPGMHrU46z45ndcwNGlU0eee/UuzjrrLK8jSgk03VFESuScK/aY5Ly8PKKjo6lRo4ZHqSKbpjuKyH5bsGABl11yEXUS4omOjubglJaMfPRRduzY81iC+Ph4lXo1oGIXEQCmTJnCiV27EJj/b545tSXvndeWwe1i+OiFkZzYtTM5OSV/F61UPSp2EWHbtm1ccF4/buvckH6HNaBBfAzRUcahjeL5Z6dEmu7O5OYbrvc6plSQil1EGDduHIcnxXN4UvHvPzUzBhxWl0mTJpGdne1BOtlfKnYR4dsZ0+iQWHod1K8ZQ6tGdZg7d274QskBU7GLCGZGCfcg7SPgICpKlVEd6HdJROh5+pnMziwsdXlW7m5Wbd5Bx44dw5hKDpSKXUQ4//zzWbG1gPS124stKww43liwlUsHXkrt2qU/6VGqDt15KiLEx8fz0eTP6H36aXTblM/JybWoVzOGpZvzmLx8J/VT2jJi5ONex5QK0hG7iADQpUsXfpr3K217XcwTv+zklukZTN2WyI33P8aUaTOIj4/3OqJUkB4pICJSTeiRAiIiEUrFLiLiMyp2ERGfUbGLiPiMil1ExGdU7CIiPqNiFxHxGRW7iIjPqNhFRHwmJMVuZr3MbImZLTOzO0KxThEROTBBF7uZRQPPAqcDhwMXmNnhwa5XREQOTCiO2DsBy5xzfzjn8oF3gT4hWK+IiByAUBR7C2D1n16vKXpvH2Y2yMzSzSw9MzMzBJsVEZGShO3iqXNurHMuzTmXlpSUFK7NiohEnFB80cZaIPlPr1sWvSfVUG5uLu+88w5ff/MNZlH0POVk+vXrR82aNb2OJiIVFIoj9tlAGzNrZWY1gAHAxyFYr4TZ9OnTSUlNZeybE4hLPpzY5m0ZNeZlUg8+mB9++MHreCJSQUEfsTvnCszsOuBLIBp4xTm3IOhkElaLFy+mX//zGPzQsxxx3PF73z+l38XM+XoqZ/Y+i59/mkNycnIZaxGRqiAk59idc58559o65w5xzg0PxTolvB57/AlO6T9wn1L/r44n9aRLr78xevQzHiQTkf2lO08FgEmTJvGXPueXuvwvZ5/HO+PHhzGRiBwoFbsAsG1rDvUaNip1eb3EJLZv2xrGRCJyoFTsAkDqwYewfNGvpS5fvvAXWh18SBgTiciBUrFHsNzcXD788ENeeeUVTvlrdz5743mcc8XGBQIBvnjrRa4dfHX4Q4rIflOxRyDnHI+MGEGL5GTuHzmKtz7+kunf/Yf5s79n5JCBbM3evHfslqwMXrz3ZmpFOy6++GIPU4tIRYXiBiWpZu6+Zxjj3/+QYa9+RJPkVGBP2S/+aRZP3jqIm3p3pe3RHXDOsWLJAi644EIef/d14uLivA0uIhViJf3Tu7KlpaW59PT0sG9XYMOGDbRt145H35tR4sXSX3/4hndG3s0Lzz9HVFQUxx13HPXq1fMgqYj8LzOb45xLK2+cjtgjzLhx4+jco3epM2CO7NQNomOpVasW3bp1C3M6EQkFnWOPMCtWrqJZautSl5sZLQ9py5o1a8KYSkRCScUeYRonNWLzxnVljslav5bExMQwJRKRUFOxR5gLL7yQ7z//gF15eSUuX/nbQrI3rqN79+7hDSYiIaNijzBt2rShd+/ePHfnNezM3bHPssx1a3juzmu5995hxMbGepRQRIKli6cR6KWxLzDo6sHcdNbxdDn1LOonNWXd74uZ+90M7rnnbq6+6iqvI4pIEDTdMYItX76cCRMmsGnzZlqlpjJgwAAaNGjgdSwRKUVFpzuq2EUq0Y4dO5g0aRJLly6lfv369OvXj9TUVK9jSTVV0WLXOXaRSvLqa6/RMiWF515/m8VZeUyd/Svtj+3IxZcOZOfOnV7HEx/TEbtIJZg4cSLX3XATtz79Oi0POXTv+zvzchk77Caa10/gvYkTPEwo1ZGO2EU84pzjjjuHMui+UfuUOkDN+FoMfvBpvv3uO+bOnetNQPE9FbtIiKWnp1MQcBzWsUuJy2NrxNGtd3/eeGNcmJNJpFCxi4RYZmYmSc2TMbNSxzRqnszGzIwwppJIomIXCbEWLVqwbsXvBAKBUsdsWLGMlOTkMKaSSKJi95ndu3cza9Yspk2bxtq1a72OE5GOPvpoEhMb8tPMqSUuz92+jW8mT+Lyyy4LbzCJGLrz1Cecczz+xBM89tjj1G7QkIQ69VixZCEnnngio59+ilatWnkdMWKYGaMeG8n5F1xIzfgEjuh0wt7TMtmZGxlz1/X079ePtm3bepxU/ErF7hNDbriRqV9/w01Pvk5K28MA2Jm7g6njX+P4E7rxn++/040xYdSjRw/efON1rr7mWmrUqk1qu6PYmp3F4p9+4OqrB/PQ8Ae9jig+pnnsPjBv3jx69jqdh8ZPJaFO8W87en/sKKK2rGP8O+94kC6yBQIBpk2bxu+//07dunU544wz9I1UcsD0DUo+t2LFCl5+5RWWLl3GwoULOelvF5ZY6gCnnn8Zt/TpRnZ2tp4FE2ZRUVH06NGDHj16eB1FIogunlYzzjnuHHoX7Y/tyM/L19Pg8E5k79jJIUe2L/Uztes1oFHTFqxatSp8QUXEM0EdsZtZf+Be4DCgk3NO51cq2ZNPPcWEDz5ixKRp1G2w51uO5syYwtbNm0r9TCAQYNuWzdSpUydcMUXEQ8Eesc8HzgFmhiCLlGP37t088sgIBt03am+pA3TueSYzP5lY6ud++c/XNGveXDNjRCJEUMXunFvknFsSqjBStm+//ZYGTZqR3LrdPu93Ovl0sjM3MvmNMcU+k7luNeMevZthdw0t805IEfGPsF08NbNBwCCAlJSUcG3WV7Zu3Uq9ho2KvR8TW4M7nn2TkTcM5PsvPqJ7n/OpVacey+b9yH+mfMLwBx7g3HPP9SCxiHih3CN2M/vKzOaX8NNnfzbknBvrnEtzzqUlJSUdeOII1rp1a5Yvnk9hQUGxZY2ateDhd76kXoNEFkyfTMYv39Ht6ENZsmgR119/vQdpRcQr5R6xO+c0T6uKOOKII0hNPYjvv/iIE3sXPwLPzsrgj4VzWbpkCU2aNNn7vnOO9PR0fv31V+Li4ujZsyeNGzcOZ3QRCSPNY69mnhs9mlNP60UgUMgJp/clJjYW5xy/z5/Li/fdzNA779yn1OfNm8fAy68gI2sThx3bmZ07tjP42mvp3/88nh39NDVr1vRwb0SkMgR156mZ9QVGA0nAFmCuc+608j6nO0+DM2fOHIbcdDNLFi8hpc2hbMnMoCB/J8PuuZsrr7hi77jFixdzwoknct6QoXQ74xyiovaceduek81rDw+lTnQhn03+ZO/7IlK16cusI8DSpUv5448/qF+/Pscdd1yxgu533vnENW9D74FXF/tswe7d3HvpmYwZ/RQ9e/YMV2QRCYKKPcLl5OTQMjmFJyf/h1p16pY45t+T3mTL0p+ZNGF8mNOJyIHQd55GuI0bN1KvYWKppQ7Q4uA2rFq9OoypRCQcVOw+1bBhQ3KyN5G/M6/UMVnr15LUqPi8eBGp3lTsPtWoUSO6dO7K9198VOJy5xxff/g2lw+8NMzJRKSyqdh97P77hjHx2REs/vnHfd4vLCjg3aeGE+sK6NNnv+4zE5FqQPPYfaxr1668/eY4Lhk4kBYHt6VNh87syt3Bj1M/4bB27Zj65RfExsZ6HVNEQkzF7nO9evVi7erVfPDBB8ybN4+aTety/6eTad++vdfRRKSSaLqjiEg1oemOIiIRSsUuIuIzKnYREZ9RsYuI+IyKXUTEZ1TsIiI+o2IXEfEZFbuIiM+o2EVEfEbFLiLiMyp2ERGfUbGLiPiMil1ExGdU7CIiPqNiFxHxGRW7iIjPqNhFRHxGX40nsp/y8/P54osv2LBhA02aNKFXr17ExcV5HUtkLxW7yH4YO/YF7rrjDprXiaVZQjQbcgv5e04+9w9/iMHXXON1PBEgyGI3s5HAWUA+8DtwuXNuSwhyiVQ5Y8Y8z4ND7+Cuzg1JbVBz7/urcnbx8LB/UVhQwHVDhniYUGSPoL7M2sxOBaY55wrMbASAc+728j6nL7OW6iYvL48WTRvzQLckkusVP+2ybls+//p6I2vWbyQhIcGDhBIJwvJl1s65Kc65gqKXs4CWwaxPpKr6+OOPaZ1Yq8RSB2hepwbtkhL44IMPwpxMpLhQzoq5Avg8hOsTqTLWrFlD81plj2kev2eciNfKPcduZl8BTUtYNNQ591HRmKFAAfBWGesZBAwCSElJOaCwIl5JSkpi0y4rc0xWvpGUlBSmRCKlC+ocO4CZXQZcBZzinMutyGd0jl2qm5ycHFJaNGdUj2Y0qhVbbPnmvAKGTFnLyjVradCggQcJJRKE5Ry7mfUCbgPOrmipi1RH9erV4+ZbbuHRHzazJa9gn2U5Owt49MdNDBkyRKUuVUKw89ifAeKAqWYGMMs5d3XQqUSqoHvuvY+Cgt1c/9RTHNeiDk3iAmTsiuLHtdsYfM213D/8Ia8jigAhOBVzIHQqRqqzrKws3n33XdatW0uzZs0ZMGCAzq1LWITlVIxIJAoEAnt+Cgv3/lqkKlGxi1SQc447b7+NNgen8slzw1n7xat89vxDtD2kFbfdeosKXqoMPStGpILuvecuPhj3EqNPbUH9mv//V+eSI+ry8DuvEhsTw/BHRniYUGQPnWMXqYD/Tnd8skczEkuY7pidV8CQqWtZvmoNDRs29CChRAKdYxcJoQ8//JCjm9UpsdQBGsTH0KF5Xd5///1y17VixQpuuelGUls2p0mjhpx0fBcmTJhAYWFhqGNLhFKxi1RAVlYWiXFl/+s2sYZj06ZNZY6ZMWMGxx5zFCu+epebj47jkW4N6Ry1mmE3Deacs3uze/fuUMaWCKViF6mA5ORk1pZzC966PGjZsvTn4G3ZsoV+fftwc8cGDDyqAan1a5JYK5ZuKXV5sFsS6xfMZviDD4Q4uUQiFbtIBZx11ln8sTmPVTm7Sly+dms+izNz6du3b6nreP311zkyqSZHNy3+WN/YaOPSI+ry3DOjyc/PD1luiUwqdpEKiI+P5+FHR/LwrCz+yN65z7IVW3by0KwsHnzoYWrVKv0RkFM++4TOjUufiJZSL47asVEsXLgwZLklMmm6o0gFDRp0FTHRMfzr9n/SNGEHzRKiWb+jkPXbd/Pgw48yaNBVZX4+UFhIlJX9hMjoKNN8eAmail1kP1xx5ZVcfMklTJ06lfXr19O0aVNOPfVUatSoUe5nT/zrKcx8cxFdk0tevnF7Ppt27KJdu3YhTi2RRsUusp9q1KjBmWeeud+f+/s/BvHoIw9zemo8rf70nakAAed4Z/E2Bg68rMzTOSIVoXPsImHSuHFjXnzlNe7/PpNPlmSzPb+QgHMsysxlxKxN5NVL5sGHH/E6pviAil0kjPr378+X/55BTmoXLv94OeeO/42xvwXod93tTJv5rb4IW0JCp2JEijjn2LVrF3FxcVg5FzmDkZaWxoT3PsQ5R2FhITEx+msooaUjdol48+fP55ILB1ArviZ1atemWeNG3H3XULKzsyt1u2amUpdKoWKXiPbVV1/xlxO6Er14Bi+emcJ757Xhrk71mDXxRTp37EBGRobXEUX2m4pdIlZubi4D+p/Lrcc15Jx2Dagbt+foOaVeHNd3TOTohJ1cM+jvHqcU2X8qdolY48ePp3XDmhzZuOTpheceWpepU79i/fr1YU4mEhwVu0Ss72bOoH3D0i+SJtSIpl3TesyZMyeMqUSCp2KXiBUdHUNhoOxH8RYEAkRHR4cpkUhoqNglYp12xpn8mFn6c1my8wpYmrGNrl27hjGVSPBU7BKxzj77bLYEYpmxYmuxZYUBx+sLcrjgggHUr18//OFEgqBJtBKxYmJimPz5l/Q8uTuLsgs5ObkmDeJj+H3zTj5dsZOGB7Vl1NPPeB1TZL/piF0i2lFHHcUvCxZxwoXX8MofUQyblcPMXU257eEnmTJthh7IJdWSOVf2xaPKkJaW5tLT08O+XRGR6szM5jjn0sobpyN2ERGfUbGLiPhMUMVuZg+Y2S9mNtfMpphZ81AFExGRAxPsEftI59zRzrn2wGTgnuAjiYhIMIIqdufcnycAJwDhvxIrIiL7CHoeu5kNBy4FcoC/ljFuEDCo6OUuM5sf7LarsEZAltchKpGf98/P+wbav+ru0IoMKne6o5l9BTQtYdFQ59xHfxr3L6Cmc25YuRs1S6/IlJ3qSvtXffl530D7V91VdP/KPWJ3zvWo4DbfAj4Dyi12ERGpPMHOimnzp5d9gMXBxRERkWAFe479ETM7FAgAK4GrK/i5sUFut6rT/lVfft430P5VdxXaP08eKSAiIpVHd56KiPiMil1ExGc8K3Y/P47AzEaa2eKi/fvAzOp7nSmUzKy/mS0ws4CZ+WZqmZn1MrMlZrbMzO7wOk8omdkrZpbh1/tHzCzZzKab2cKiP5s3eJ0pVMysppn9aGbzivbtvnI/49U5djOr+987V81sCHC4c66iF1+rNDM7FZjmnCswsxEAzrnbPY4VMmZ2GHsumL8A3Oqcq/bPYDazaOA3oCewBpgNXOCcW+hpsBAxs78A24E3nHNHep0n1MysGdDMOfeTmdUB5gB/88Pvn5kZkOCc225mscC3wA3OuVmlfcazI3Y/P47AOTfFOVdQ9HIW0NLLPKHmnFvknFvidY4Q6wQsc8794ZzLB95lzxReX3DOzQQ2e52jsjjn1jvnfir69TZgEdDC21Sh4fbYXvQytuinzL709By7mQ03s9XARfj3AWJXAJ97HULK1QJY/afXa/BJMUQaM0sFOgA/eBwlZMws2szmAhnAVOdcmftWqcVuZl+Z2fwSfvoAOOeGOueS2XPX6nWVmSXUytu3ojFDgQL27F+1UpH9E6lqzKw28B5w4/+cFajWnHOFRU/RbQl0MrMyT6dV6pdZ+/lxBOXtm5ldBvQGTnHV8GaB/fi984u1QPKfXrcsek+qiaLzz+8Bbznn3vc6T2Vwzm0xs+lAL6DUC+Fezorx7eMIzKwXcBtwtnMu1+s8UiGzgTZm1srMagADgI89ziQVVHSB8WVgkXPuCa/zhJKZJf13Zp2ZxbPnAn+ZfenlrJj32PMIyr2PI3DO+eIIycyWAXHApqK3Zvllxg+AmfUFRgNJwBZgrnPuNE9DhYCZnQE8CUQDrzjnhnubKHTM7B2gO3sea7sRGOace9nTUCFkZt2Ab4Bf2dMpAHc65z7zLlVomNnRwOvs+XMZBUxwzt1f5meq4VkCEREpg+48FRHxGRW7iIjPqNhFRHxGxS4i4jMqdhERn1Gxi4j4jIpdRMRn/g/Srlf8VbUDQQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# https://scikit-learn.org/stable/auto_examples/svm/plot_svm_kernels.html\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import svm\n",
    "\n",
    "# Our dataset and targets\n",
    "X = np.c_[(.4, -.7), (-1.5, -1), (-1.4, -.9), (-1.3, -1.2), (-1.1, -.2), (-1.2, -.4), (-.5, 1.2), (-1.5, 2.1), (1, 1),\n",
    "          (1.3, .8), (1.2, .5), (.2, -2), (.5, -2.4), (.2, -2.3), (0, -2.7), (1.3, 2.1)].T\n",
    "Y = [0] * 8 + [1] * 8\n",
    "\n",
    "x_min, x_max = -3, 3\n",
    "y_min, y_max = -3, 3\n",
    "plt.scatter(X[:, 0], X[:, 1], c=Y, zorder=10, cmap=plt.cm.Paired, edgecolors='k', s=80)\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Linear Kernel\n",
    "\n",
    "The simplest kind of kernel that exists is called the linear kernel.\n",
    "This simply corresponds to dot product multiplication of the features:\n",
    "$$K(x,z) = x^\\top z$$\n",
    "\n",
    "Applied to an SVM, this corresponds to a linear decision boundary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Below is an example of how we can use the SVM implementation in `sklearn` with a linear kernel.\n",
    "\n",
    "Internally, this solves the dual SVM optimization problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-3.0, 3.0)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAAA690lEQVR4nO3deXzU1b3/8deZyb5vkISEEMK+yCIRZRERCLK5oIBa19pbbK2tXW6t3v7a2mu99VarrW2vrdYlIAJhE2RN2Ins+04SAgkJIWSbLGSZzMz5/RFAcWOZE5IMn+fjweNhFt5zRiaf+eZ8z/kcpbVGCCGE57C09ACEEEKYJYVdCCE8jBR2IYTwMFLYhRDCw0hhF0IIDyOFXQghPIzbhV0p5aeU2q6U2qeUOqSU+r2JgQkhhLg2yt117EopBQRqrWuUUt5AJvCc1nqriQEKIYS4Ol7uBuimd4aa8x96n/8ju56EEKKFuF3YAZRSVmAX0BX4h9Z629d8z3RgOoCvf8CgDoldTDy0EK2arfQsTqeDyOgORnOLC/IIctUR7Gs1lulwaorP2YkK8MbXS26/tUbHy+tLtdbtLvd9bk/FXBKmVBiwCPix1vrgN31fUu9++pVZy409rhCtUaO9gWfHDab7gGR+8cZ7xnKz9u3kpe9O5oe3RDOua7ix3Fn7S5h3qIx/3Z1EdJCPsVxhzr2zj+7SWidf7vuMvi1rrW3AOmCcyVwh2qJtGcuotpUzdtoTRnPT01LxDwrmjk6hxjIbnZqM4zYGdQiUou4BTKyKaXf+Sh2llD+QAhx1N1eIti49LZXYTkn0GTzcWGZleSnbMpYxYtIU/L3NXZdtLaimot7J+G7mfgMQLcfEKyMWWKeU2g/sADK01ksN5ArRZpUWFXLy6EHGTH0ci8VcAV7/yRycjkbuqV5vLBNge2ENMUHe3BwbaDRXtAwTq2L2AwMNjEUIjxEVG8ffV27Hx9fPWKbT4WDN/I/oFx1AfIivsVyAnw2JpbTWgUUpo7miZcitbyEMc7lcAISER+IXYO4KePem1ZSeKWSC4ekSl9ZYlKJ9oLfRXNFypLALYdiKj//NS9+dTH3tOaO56157jkh/LwbHBRnLPGd38vSnuXyWX2UsU7Q8KexCGORyuVg9byYoZfRqvSgvl71nahnXNQyrxdx0ybqTlZw91yhX6x5GCrsQBh3YsoHiUycZO+1xo7kZaTPwssDYLmHGMrXWrMi20S3Cj26R/sZyRcuTwi6EQelpMwiJiGLw6AnGMuvratn46TyGxAcT5m9kszgA+4trKaiyG5+zFy1PCrsQhpScPsXezDWMmvwwXt7mNvlsXvEJtTVVTOhutgCvyK4g2MfK8E7BRnNFyzP39i/EDS44LIInX/gDN98+xlim1pqMtFQSw3zpFWV2uuS+XpEMS2jExyrXd55GCrsQhvgFBJIy1ezceta+neRlHeaZW2JQhteY94zyp6fhNwvROshbtRAG7Nm0hox5M3A0NhrNzbjQFyYxxFhmo1Pz/u5iTlfbjWWK1kUKuxAGLHr3r6yc/T5WL3O/BFeWlbBt9XJG3D0VP4NtdLecqmbxsQrOSGH3WFLYhXBT7uH95Bzcw5gpjxmdLlm3aHZTX5iqdcYyAVbkVBAT5M0A6QvjsaSwC+GmjLRUfP38GXH3FGOZToeD1QvM94U5WVHP4ZI6xncNk74wHkwKuxBuqKmsYPOqxQyf+ACBweb6o+/emEF5cZHxNebLs234WBWjk8KM5orWRQq7EG6wlZWQ2KOv8dUwGfNmEhnTwWhfGAB/bwspXcKMHqknWh9Z7iiEG+KTuvP7Dz8xmnn65HEObtvEtGd+ibXiU6PZ3x3Y3mieaJ3kil2Ia1R86iRVFeXGczPmNfWFmXh6kbFMrTVZZXWYPONYtF5S2IW4RjNe/z3/79GJF/uvm3ChL8zQjiHG+8L8Mj2PbYU1xjJF6yWFXYhrcLYwn72Zaxg+8X6jR999tnwRdTXVTOgWZiwTYFl2BSG+Vjn67gYhhV2Ia7Bm/kcoi4XR9z9iLPNCX5jOYb5Gt/qXnGtkR2ENY5JCpS/MDUL+lYW4SvaGetYtnsOgESlExnQwlnts7w7ys48woVu40Y1Oq47b0BrGdQ0zlilaNynsQlylo7u3UWOrIGXaE0ZzM9JSCQgKYYTBvjBaa7YVVJPcIZDoIHOthEXrJssdhbhK/YbcwZuLN9E+vpOxTFvpWbavWUHK1Mfx4zNjuUopXh+bSI3daSxTtH5S2IW4ClprlFJEd0w0mrvuk6a+MHdXrQWDLQS01vh6WfB1o4mYS2s25VWxIttGdnk9VgUDYwO5p0cEfdoHGBurMEcKexvlcjrZ+9k6jh/aS0T7WIaMvZuAYHO/wouv9+8//IrGhgZ++PJfjM2DOx0O1iyYRX/DfWFOVNTzamYhvxwWR9cIv2vKcGnNW1uLyK+yMyIhmJuiA7AqhdUCf95ymgd6RTLR8MlOwn0yx94G1dfV8oenH2TBv95AKQsHt2fynw/cSX7WkZYemkertlWQuXwhvv4BRm9uXuwLY7hALs+uoLzOQUyQ9zVnrD5eSWFVAz0j/ViSVYHDpbHVO1h8tJzJPSNIO1TKiYp6g6MWJsgVexu0/KN3CAmP5CfvpF1cQ73+kzm8+4fneXmG2S3o4nMblsylsaGBlGlm+8Kkp80gMqYDt3Qw1xemxu5kw8kqRnQKIcjn2vrCaK1ZmlXBqM4hrMix8db4zhezJnQP54WMPMZ3C2NFto1nBscYG7twn1yxt0E71q5kwqPfv2RjzIi7p3K2IJ+y4qIWHJnncrlcrJn/ET0G3EJCt17GcgtP5HBoeyajH3gUq8VgL/cTlTQ4tVvdIescLs7U2CmpbSSlS9glbxAJob70bR9AgJeFwyW1JoYsDHK7sCulOiql1imlDiulDimlnjMxMPHNlLLgcl66ykFrjdYuLAaLg/jc/i0bKC7II+VBs0scV1/oC1O40FimS2uWZ9voEelHl2ucW/8ihcL1NT1mnFo3fVVecq2OiSt2B/ALrXVv4DbgR0qp3gZyxTe4bewkPk39J06H4+Ln1iz4iA6duxHeTn4lbg6duvfmwWd/xeBR441l1teeY+PS+cb7wmgNj/dvxyP92rmVE+BtJT7Eh3YBXmQct1HV8Pnr7Xh5PUdL66i2O2RlTCvk9qtJa10EFJ3/72ql1BEgDjjsbrb4euMefooju7byq2kpDBh+J4W52RTmZvOrf3zU0kPzSFprwttFc+9TzxrNvdgX5rYEo7lWi2JIx2AjWZO6R/BpVjnDE0J4dtkJhnYMprbRxa6iGh7p1445B0r5w2iz4xfuM3rzVCmVCAwEtpnMFZfy8fXj+bdSObp7KzkH9pLUux/JI+/Cx89cfxHxufWL5xIcFk7yyLuMZWqtSW+mvjCrc21M7B5OiK/7P953dg4hq6yOHYU13Nszgkaniwh/L6KDwkk7WMrjA9qREGpuiaYww1hhV0oFAQuAn2qtq77m69OB6QBRMXGmHvaGpZSi16Ah9Bo0pKWH4tHsDfXMfut/6D1oiNHCfmzPdk7lHOVHg2OMLp1cmWNj4ZEyRieFGdnnpJTi6eRothfWsCLbRk55PVYLDIwJ5Ncj4ukWKRcTrZGRwq6U8qapqM/SWn/tXSCt9TvAOwBJvftJt3/RJmxN/7SpL4zho+/SL/SF6WRuU1mj00XGcRuDOgTRPvDa165/mVKKW+ODuTXezPSOaH4mVsUo4D3giNb6DfeHJETrkZ6WSlznbvS+ZaixTFvpWXasXcEd90zFz42t/l+2+VQ1lQ1OJho+AFu0PSZeVcOAx4BRSqm95/9MMJArRIs6fmgvuYf2MWbq40anS9Yu+hinw8HdleuMZQIsz7YRG+RN/xhZpXKjM7EqJhOQlazC49RU2ujUow+3T3rAWKbT4WDtglkMiAkkLsRcG12700WYn5XhCcFYZGH5DU9aCgjxDfoPHUn/oSONZu7akEH52TP84HazCwh8rBZevD3eaKZou6SlgBBfo+D4Mez1dcZzM9JSiYqJI9lgX5jaRienq+3G8kTbJ4VdiC9xOZ289tOneOuFZ4zmFuZmc2jHZ4yeYrYvzOrjlfxwaS5FUtzFeVLYhfiSfZvXU1KYz/AJ9xvNzZg3Ay+LYkLhAmOZLq1ZkVNBjyh/YoPl6DvRRAq7EF+SnpZKWFR7ku8cZyyzvvYcm5bOZ1jHYML8zN3a2nemltPVjUzoFmYsU7R9UtiF+ILiUyfZv3k9oyZ/By9vc5t8MpcvpO5cTbMcphHqa2WYod4wwjNIYRfiC3asW4myWBj1wHeMZWqtyUibQVK4Lz0i3W+je0GN3cm+M+dI6RKGt1V+lMXnZLmjEF8w8bGnGXTHWCLaxxrLPLp7G6dyjvKs4b4wQT5W/n1vV2N5wnNIYRfiPK01SiliOyUZzc1ISyUgONRoX5gLYw3xvbZj74Rnk9/fhDjvlacfZPH7fzeaWVFSzI51Kxl57zR8DfaF2ZBXxfPpJ7HVOy7/zeKGI4VdCCDn4B4O79yCf6C5jUMA6xbNbuoLY1tjNHd5to1qu1Ou2MXXksIuBJCRNgO/gECGTzTXF8bR2MiaBR9xc2wgHQyuMc8tr+dYaR3ju4ZLXxjxtaSwixteVUU5W9M/ZfjE+wkIMrdscNeGdCpKihlveI358pwKfK2KUUmhRnOF55DCLm54GxbPpdHewNhpTxjNzUibQVRsPINizU3v1NidbDhZxYjEEIJ8ZBpGfD1ZFSNueANHjMZitRDfpYexzILcLA7v3MxDP34Ba+knxnK9LYqnBranT3vpuS6+mRR2ccOLT+pOfFJ3o5kZaef7whTMB4MtBHy9LIyXE5LEZchUjLihffrh2+Qc3GM0s+5cDZnLFjA8IZhQg0X9SEkty7MrsDtdxjKFZ5LCLm5YxadOMudvf2Rvptkj6i72hTF8Zb3wSDlzDpTKcWXismQqRrRKjkY7h7Z/RmVFGZHtY+k16DYsVrM3C1fPn4nFamW08b4wqXSJ8KO7wb4wZ881svN0Dff3ipS+MOKypLCLVmfN/I9Y8M6btI/vRLvYeE6fzKGm0sZ3fvprbh0z0chj2OvrWL94Lsl3jiO8XYyRTICju7dScDyLHxvuC7MyuwKAcV3DjGUKzyVv/aJVWfHxeyyf9S73PvUsFouFwzs3ExwWwcTHpjPz9ZfYmv6pkcfZvGoJ56oqSZn6uJG8CzLSZhAYEsrtBvvC2J0uMnIruSUuiHaB5loJC88lhV20GrXVVSz415ukPPgES2f8k3EPP8V/py5h6Pj7WPjuX7nnqWf56M2XcTrc74/iaLTTa9Bt9Bp0m4GRN/m8L8yDRvvCVNQ5iAv2MT5nLzyXTMWIVmPb6mX0uWUoGWkzeOblv9Br0BAARkyagpeXF2sXziYyugP7t25k4PBRbj3WmCmPMWbKYyaGfdHahR839YWpWA0GWwhEB/nwakonY3nC88kVu2g1Ss8UEpvYhdLTBfS8+dIr6X5D7uDksUN07NqTsjOFbj1OftYRXC6zSwYv9IUZGBNo9OzR0tpGKqWDo7hKUthFqxEUGk5l6VkCQ0IpzM265Gsnjx6ifVxHyopPExR67VMSVRVl/L/HJjH/7dfdHe4ldq5fha30LBMNH303+0ApP1qWS6OsXRdXQQq7aDVuS5nIjnWrGDPlUf750i8oyj8BwMljh/jg1V8zfOL9HD+4hwFuTMOsXzwXR6OdoePuNTVsoOkwjajYeG6ODTSWWd3gZGNeFUM7hri9xFFrja3eQcm5RpwubWiEorWSOXbRaoS3i+H2ifdzeNc2Bgy7k99/dzJOpwNfP3/ueugpMpctYtLjP8DP/9r6pLicTlbPm0nv5CFm+8IcP8aRXVt56CcvYi1ZZCx3Ta4Nu1O73R1yw8lKFh8t50xNIz5eFtCalC5hTOkdafQmr2g95F9VtCqP/vy3xCV2IX3uhySPGsddDz1F31uHs/iDfzDw9lHc/eQz15y9J3MtpUUFjGmGJY7eFsXEgnnGMl1asyLHRq8ofzqHX/tGp1n7S5h3qIw7EkO4vVMI/aMDuKdnBAWVDfx23SkaHDLF44mMFHal1PtKqbNKqYMm8sSNy+rlxXdffIVXZq2gfYeO1NfWkNCtN39euI6pP/xPtzb9bMtYSni7aJJH3mVsvLU11WxatoBhCcGE+Jr7Bfh4eT1nahrdWuKYXVbHmtxK7kgMYfGxCqIDvenTPoBtBTU0ODVhflYWHSk3NmbReph6JX4I/B2YYShP3ODadYjnnu/+yGjm0y/9mbOF+Xh5m9vk89nyhdTXnmNCN7PLEbtF+vPPSUlEBVz7WFfm2BidFMqiI+X8ZXxn2p/f3DS6cygvrsnntvZBfJpVwdQ+kVgt0oHGkxi5YtdabwTkrV+0WlprrF5exHZKMpqZMW8GnXvdRI8of6O5ALHBPnhbr73gnqiox89L0bud/8WiDmC1KEZ1DuFUVQOu8zdVhWe5bnPsSqnpSqmdSqmd1RXyHiCun4a6Op6fOoatGUuN5h7Z1dQXZlJIidHcj/aX8uqmArdXr3hZFApFVYPzK1+ranDia7Vgd2ppKuaBrtu/qNb6Ha11stY6OTg84no9rBBsXrWYwtwsQiOijOamp6US5GPh9gSzfWFW5djQ4Pb0SHKHIE5X2ymrc/BZftXFzxfX2FmebSMm2JuOIT6E+MoRe55GljsKj9bURncG8V260/PmW43lVpScYee6ldzTLczoksHM/Gqq7U4jpySldAnjuRUnePimKD7Yc5ZFR8sJ8bFytLSOB3pHkp5TySP9zL7ZidZBCrvwaDkH9nDy6AG+++IrRtvorlnwMdrpNN5Gd0V2BXHBPvSPdv9M03B/L54fHsefMgtJjgukY7AvLqBzuC+fZlUwvmsYw67it42y2kb2FJ3Dx2ohOS6QAG+50m+tTC13nA1sAXoopQqUUt8zkSuEuzLSUvEPDGL4hPuNZToaG1m7cBY3x5rtC5NTXk9WWT3ju4UZexPq2z6At8Z3JjbIh62FNWw+VU11g4vf3hHPg32v/Gp98dFyfrLiBPuKa9mYV8n0JbnsPXPOyBiFeUau2LXWD5vIEcK0kfc9RO/kIfgHBhnL3LluJbbSs0wYEW8sEyAqwIvv3BTFqM6hRnPD/L2Y2ieKqX2ubdolp7yeT46W89fxnS8uvzx0tpZXMwt5954u+Mnu1VZHpmKER+udPITeyUOMZmbMm0G7uAQGxppb4ggQ5ud1VVfR18umvCru6hJ2yZr6Pu0D6BLux+6iGoZ2NHfzWJghb7XCI7mcTua9/TrFBXlGcy/0hRnzwKNGN/VsPlXFZ/lVF9ewtyZ2pwtfr68+Vz8vhd3Z+sYrpLALD7Uncw2L3v0reccOGc1NT0vF26KYUJBmLNOlNal7S1iaVWH0Bq8pg+OCWZ1beUlfmTM1dvYX1zIwxlw3S2GOTMUIj5SRNoOI9jEMumOssczammoyly3k9k5m+8LsKTrHmZpGHunXzlimSQNiAugZ5c/PVp5kVOdQahudrM6t5PEB7Qj1kxLSGsm/ivA4Rfkn2L9lA1N+8AusXuZe4pnLFjRLX5jl2RWE+VkZEh9sNNcUpRTPDo5hf3EtO07X4Gu18PKoBDqF+bb00MQ3kMIuPM7qeTOwenlx52Rzi7Wa+sLMJKl3P7pF2o3lFtfY2XX6HFP7RLrVF6a5KaXoHxNIf5l6aRNkjl14pGHjJxPeLtpY3uGdWyjMzWJS8FljmQBldQ7iQny4y/BGJ3Fjkyt24XEe+8XvjK8uyUhLJdjHwnCDfWEAercL4O8TOrfKm6ai7ZIrduExtNbkZx8BMFooy88WsXP9KkYnme0LU1Rtp8HhanVFvarBydKscmbuK2FHYY2ckdoGSWEXHiN7/25eeHAsW9M/NZq79nxfGHfPHv2yP285zW/XnTKa6a5jpXU8uyyX7LJ6fKyK2QdLeWm9HKHX1khhFx4jIy0V/6BgBgwfZSzT0Whn7aKPGdQhkJggc31hssvqyC6r5/ZOrWcljNaav20r4unkaH42pAMP9o3itZRO+FoVy7MrWnp44ipIYRceobK8lG2rl3H7xAfwCzC3cmPH2qa+MOO7ut9G94uWZ9vw81LG+8K443R1I3UOF0M7fv5mY7UoJvWIYGtBTQuOTFwtKezCI6z/ZA6ORjspUx83mvt5XxhzbxZVDU4y86sYmRjaqlrfWhQ4NXx5Rt3p0siRqG2LFHbhEbasWkKfW4YRl9TNWOapnKMc3b2NMVMeM9oXZntBNXanNj5n766YIG8i/L1Yk1t58XONTheLjpQbXw0kmpcsdxQe4aUPFlFZXmo0Mz0tFR+rYsKpNDB4fNzopFC6RvqRGOZnLNMEpRQ/vS2Wl9af4rP8ajqE+LDrdA1dwv1knX0bI4VdtHlaa/wCAo3OrddWV5G5bCHDE4KNngmqtUYp1eqK+gUJob68PTGJbYU1VNQ5+PmQDnSP9Gt1SzLFt5OpGNGmFeWf4FfTxnD80F6juZuWLaChrpYJBs4e/aLXPjvNzH0lRjNN8/WyMKJTCPf2jKBHlL8U9TZICrto01anpVKUd4LI6A7GMi8cgJ3Upz/dIs0dpnGmxs7mU9VyI1I0Oynsos2qr6tlw5J5DB49nrCo9sZyD+/YzOmTOdwdZLYvzMpsG0ph/ABsIb5MCrtoszav+ITamipSpj1hNDc9LZVgHyvDDW4eanC4WJ1r49b4YCK/cMScEM1BCrtok5qmS1JJ6NaLHgNuMZZbVlzErg3ppHQJxcdq7scjM7+KaruLCa1siaPwTLIqRrRJWmvu/u4z+PoFGL25t3bhLLTTaXy6pGdUAA/1jeSm9gFGc4X4OlLYRZtksVgYete9RjMdjXbWLZrNoA6BRBvsCwMQF+LDwze1zqPvhOeRqRjR5lSWl/LJe3+j2ma2MdX2tSuwlZ41vsRxRXYFR0pqjWYK8W2ksIs2Z93Cj0n7x5+oqigzmpsxN5X28eb7wry3+yzrT1YZyxTicqSwizbF6XCwZuEs+gweTlznrsZy87OOcGzvDsZMeQyLwTn71bk2Gl2try+M8GxS2EWbsnvTasrOnGbsNPNdHH2sign5acYynS7Nymwbfdr5t9oWAsIzGSnsSqlxSqljSqkcpdQLJjKF+DoZaTOIiI7l5hEpxjJrq6vIXL6Q2xNCCDbYF2ZP0TmKzzUyobvZOXshLsftwq6UsgL/AMYDvYGHlVK93c0V4sscjXa8vX2a2uh6mVvQtXHpfBrqao1Pl1TbnSSG+XJbfOs5JUncGEz8dAwGcrTWuQBKqTnAvcBhA9lCXOTl7cMv3/oQrc0drqy1JmPeDLr2HUi3yDpjuQB3dg5lZGKINNES152JqZg44Isn8hac/9wllFLTlVI7lVI7qyvKDTysuJE01NVRcroAwGihPLg9k6KTx5kUdMZYJkBBVQNOl5aiLlrEdbt5qrV+R2udrLVODg6PuF4PKzxE5vKF/PSeYRTkZhnNzUhLJcTXyrAEs31hXsjI491dxcYyhbgaJgp7IdDxCx/Hn/+cEEZorVk9bwYJXXsS19nc0XdlZ06za0MGY5Kapy+MyTcLIa6GiVfzDqCbUqqzUsoHeAhYYiBXtBCtNcUFeRenPlpa1r6d5GUdZszUx41ObaxZ8BG4XEb7wmitWZ5to2OID32lL4xoIW7fPNVaO5RSzwKrACvwvtb6kNsjEy0i9/B+3n35eSrLStBaExUbx9Mv/Zn4pO4tNqaMtFQCgkIYNmGyscxGewNrF35MsuG+MFll9eSU1zN9ULTMr4sWY2TNmNZ6ObDcRJZoOTVVNl577kke/flvGHLXvaA16z6Zw6s/epQ3Fm3Ax8/caUJXqr72HLs2ZHDnfQ/j52/uCnj7muVUVZQxoV+8sUyAjXlV+HlZuLNziNFcIa6GdHcUF21ZtYTeyUMYNv7zK+PRDzzCznUr2bl+FUPH3Xfdx+QXEMibizcZz82YN5PojokMiPU1mvvUwPakdAklwNvcRichrpa0FBAXVZQUE9upy1c+3yGxC+VnzS4HvBphUe2NHn2Xl3WYrL07GDPlUaN9YQCsFiXtA0SLk8IuLup2083s3rgal9N58XOOxkb2ZK6lW79B1308O9at5OXvT6WixOybyup5M/GxKsbnzTWW6XRpnk8/yZpcm7FMIa6VFHZxUf+hIwkICubNX07nyK4tHNyWyWvPPUlc565075983ceTPjeVs4X5hIRHGcs8V11J5rIFjOhkti/M7qJzHCurx89LfqREy5M5dnGRxWrl+bc+ZOWcD5j1l1ewWKwMGXs3KdPMLjO8EoUncji0PZNpz/zSbF+YT+fTUF/H+G7RxjIBlmdXEOHvxa3SF0a0AlLYb2CbVy1mxax/U3K6gM49+zL5+z+le/9B3PPkM9zz5DMtOrbV82Zg9fLmzskPG8t0uVysnj+Trn0H0jXCXF+Yomo7u4vO8VDfSLwsssRRtDz5vfEGtXr+TOa//WemPP1z/jh7BbeOmcgbP/8eOQf3tPTQqK89x8al87k1ZSKhkebOCT2047Nm6QuzMseGVcHYLmFGc4W4VnLFfgNyOhwsevev/OpvM0no3guAkfc9hNPpZPH7f+cXb7zXouNTysLUH/6n8Xn95ugLA3BbfBCR/l5EBngbzRXiWskV+w2oqqIMp9N5sahf0PfW4eQda/luy77+/ox7+CmSevczlnmhL0yK4b4wAL3aBXBPT2lsJ1oPKew3oKDQMFxOx1d6wZw4coDo+E4tNKrPx7DukznYG+qN5q5Z8BFozV2G+8IsOFzG6Wq7sUwhTJDCfgPy9vElZdoTvP2b5ziTfwKtNUf3bGfWmy8z4bHpLTq2pTP+yaw3XsblchnLbLQ3sHbR7GbpCzNjXwn7zpwzlimECTLHfoN6YPrPWGSx8rsn76PR3kBoRBQP/+RFBg4f1WJjspWeZfuaFYyd9rj5vjDlpUwYabYvzPLsCvy9LNyRKH1hROsihf0GZbFaeeDpn3Hf935MQ10t/kHBLd6NcN0ns3E6Ghkz9XGjuRlpM5r6wsSY6wtTWe8gM7+asdIXRrRCMhVzg7N6eREQ3PLncjodDtYsmMVNt91ObKckY7knjx0ia99OUqY+brQvTEZuJQ6XZny3cGOZQpgiV+yiVagoKSY0IoqUaU8Yzc1Im3G+L8xs8DF3Zd3gcDGoQyAJod/+W0C9w8Xm/GqKauwEeFsY2jHY6Dy/EF9HCrtoFaJi4/jDR8uMZp6rrmTzikWM6BRCkMGiDvBIv3Zorb/1e9bmVvL+nrP0iPKjc5gfZ6rt/GJVHrfEBfLDW2KML7sU4gJ5ZYkWV1leSk2VDaWU0SmhjUvm0VBfxwTD0yWFVU3LG79trJ/lVzHrQAnf6RdFicObRVk2PitqIKV7JLV2J3/ZWmR0TEJ8kRR20eI++fdb/PTu4TTUmevfcqEvTLd+g+gSYa4/elG1nWeW5bIiu+Ibv0drzUf7S5ncM4I5WbU8+Ju3+HBrLv89O4Ms7wSiQgM5WlpHboXZtfpCXCCFXbSoC31hBg4fha+/uaP3Dm3PpCgvl0mBp41lAqzIrsCq+NYujjnl9SjgQIVmyrMvMmDYnVgsFqI7JvKj195l3YkqRiQEs+FkldGxCXGBFHbRojKXL6Suptr4TdP0tFRCfa0M62iuL0yDw8WaE5Xc1jGYCP9vvj1V2eCkfaA3ReccdOk78JKvhUZEERIaSoCPlcp6h7GxCfFFUthFi9Fak5E2g8SefenW72ZjuaVFhezeuJqULmF4G7xBuTGvihq7i4mXmbOP9PeisNpOp2AvDm3/7JKvlZwuoLqqkqp6pzQNE81GCrtoMfnZRziVc5SUaU8YvWm6ZsFHAIwz2BcGYP3JKjqF+tK73bdPGSWG+RLkY6F7qGLJu2+wbtFsqirKOLp7G28+9xgTuoayMa+KUZ1DjY5PiAtkuaNoMZ269+a1+WuJijW31b/R3sC6RbO5JTaAdoFmr4h/c0c8JecaL/smpJTiiQHteWPzab7TN5IdqX9i9uu/JSLIj5FxPuwqrGJox2DiQmQ9u2geUtg9zI61K1i7aDY1lRX0uWUYEx79PiHhkS09rG8Ul9TNaN621cupqihjQv+ORnMB/LwsdLzMhqQLBsQE8txtsbyzq5gAbwtDOvhhq3ew4FAV47uF852bzJ3jKsSXyVSMB1n8/t+Z8/f/5faJ9/PIz37DuepKfvfkfdRUfvPSvJay5IN/8NYLz+B0mL2BmJGWSnTHRPrHmGsiVlnv4KcrTnDwbO1V/b1BHYJ4e1ISTwxoT9cIP0YkhPDuPV14rH87rHKEnmhGcsXuIWqqbHya+k/+Ny2DyOhYAHoOHMw/f/czVs+byX3/8ZMWHuHnHI2NrJrzAR279jR6UPXJowfJ3r+LR37+GyxF84zlZhyv5IStgVDfq9+9alGKATGBQKCx8QhxOXLF3obZG+opyM2i2lZB3rHDJHTtcbGoX3DLqAkc27ujhUb49XZvzKCipJiUaWa7OKanpeJjVUzIm2Ms0+nSrMypoF90wBVPwwjR0ty6XFJKTQVeAnoBg7XWO00MSlxe+twPWfCvNwkKC6eyrJTeg26luDAfp8NxyVXwmfxcwttFt+BIvyp9bipRsfEMHD7aWGZNlY3NKz/hjkSzfWF2nq6hpNbBUze3N5YpRHNz94r9IHA/sNHAWMQV2rFuJSs+fo/fvb+QPy9cz1vLtuDl44vV6sWcv7168Vi53MP7WTbzHUY98EgLj/hzBblZHN65mdEPPILFaq4Ab1wyD3t9vfG+MMuzbUT6e3FrnNkDsIVoTm5dsWutj8C3N0MS5qXPTeWhH79Ah8QuAAQEBfPUf/0PP5lwG/nZh3l2/GCCwyKoq6nm0Z//hq5f2v3YkoJCwrj3qWcZed9DxjIv9IXp3j+ZpPAaY7kAKV1CcTi13OwUbcp1u3mqlJoOTAeIiom7Xg/rkWylZ4lNuPQwiqCQMIJCwnjqv/6It48PNZU2OiR2xcu7de1uDItqz4PP/spo5oGtGzmTf4LvxNUDZjf9DE+QY+9E23PZqRil1Gql1MGv+XPv1TyQ1vodrXWy1jo5ODzi2kcs6NbvZnauX3XJ504cOYDT6SAqJo6I9rEkdOvV6or6nsy17NqQcdk+5ldr9bwZhPpaGWq4L8z8w2XSz0W0SZe9Ytdaj7keAxFX7p4nn+H333sAp9PBzSPGUJibzby3X+fBH79gdPmgSVpr5rz1P1i9fLh5hLmXVMnpAnZvWsMDPcON9oXZkFfFzH0l9IryJ9Svdf4/FeKbyHLHNigmoTMvfbCIqooy3nvlRXasXcH0373OiElTWnpo3+jYnu2cyjnG2GmPt/q+MFprlmdVXFFfGCFaI3eXO04G/ga0A5YppfZqre8yMjLxraLjO/G9//rjFX1vcUEe89/+Mwe2bSIwOIQ77n2QiY9Ov65X9+lzPyQgOJQhd13VDN63arQ3sP6TOQw23BfmaGkdJ2wN/CA5WhYGiDbJrSt2rfUirXW81tpXax0tRb31qSwr4eX/mEKHzl145aNl/OiVtziwdRPv/c+L120MFSXF7Fi3kpH3TjN6mMbFvjCGlziuyLYR4G1hZKJ0XxRtk0zFeLg1C2Yx8PYxTP6P54iM6UBS7/784o1/s3P9KkpOF1yXMZwtzCciugNjpjxmNDcjLZXYTkn0M9gXRmuN3aUZ3TkUf2/58RBtk7xyPVxe1mH6DB52yef8AgLp0mcABcePXZcx9BhwC39ZkklMQmdjmSeOHCB7/y7GTH0ci8HpEqUULwyP43uy01S0YVLYPVx0fCdOHDlwyeccjY3kZx2mfVxCsz9+2ZnTNNobjM9VZ8ybga9VMe7ELGOZTpemuMYOyKY70bZJYfdwo6c8yobFc/lsxSKcDgeV5aX8+w/Pk9ijr/Fe6F/n7d/+lP/+D7OrdZqzL8zTn+ZypOTq2vMK0dpIYfdw0fGd+Pkb75E+N5Wnbu/Jz+4ZjtXLh2f/+Pdmf+ymvjBbuGXUeKO5G5akNVNfmAoi/L3oHilLHEXbJjsvbgDd+w/i9x9+Qn3tOby8vfHyvj5HsmXMTcXL24eR9xruCzOvqS9MZ4N9YQqr7Ow9U8t3boqSvjCizZMr9huIX0DgdSvqdedqyFy+kNvG3k2IwRYSB7ZupPjUSSYFFBrLBFiRXYGXBcZ2CTOaK0RLkMIumsXOdauoO1fD2GlPGM3NSEs13hfG6dJszKtiSHww4f7yS6xo++RVLJrF8In3E5uYRJc+A4xllpw+xZ5Na3igV4TRvjBWi+Kt8Z1pcLqMZQrRkqSwi2ahlDLeB37Nglkoi8VoX5gLwuRKXXgQmYoRxr3z378k7f9eM5ppb6hn/cf/ZHCsv9G+MEdKanlxdR5F1XZjmUK0NCnswqiKkjNs/HQejeeP5zNlW8YyqhqczXL03Ulbg8ytC48ihV0YtWbBx2iXq1n6wsQF+9Av2lxfGFudg82nqhjVORQ/L/lREJ5DXs3CGEdjI2sXzqLf0JFEd0w0lpt7eB85B/cwvluY0a3+Gbk2HC4Y3y3MWKYQrYEUdmHMzvWrsJWeJWXq40ZzM9Ka+sKM6myuja7TpVmZY6N/dADxIb7GcoVoDWRiURjTsUsPJj72NAOG3Wkss6aygs2rFjMqMZRAg31hXFozuVcEHaWoCw8khV0YE5fUjUd+9v+MZm5YkkZjQwMTusUazfW2WpjUXQ5VF55JpmKEEZuWLuD4ob1GMy/0hekx4BYSw/2M5RZV20nPsdHgkA1JwjNJYRduq62p5oNXf01GWqrR3ANbNlBckMckf7N9YZZmVfCvXWeoa5TCLjyTFHbhtsxlC6ivPccYwzdN09NmEOZnZYjBvjB1jS7WnqhkaMcQ2W0qPJYUduEWrTUZ82aS1Luf0RYCZwvz2Zu5hrFdwvC2mlviuDGvitpGFxNkiaPwYFLYhVuO7NpKYW4WKYa7OK6Z/xHKYuEug31htNYsz66gc5gvPaPkMA3hueR3UeGWsjOniY7vxJCxdxvLtDfUs372OwyO9ScqwFxfmBq7C2+rYmyS2Y1OQrQ2UtiFW26f9ADDJkzGYjH3y9/WjKVU2833hQn2tfL62ERcWhvNFaK1kakYcc3OFubjcrmMFnVo6gsTH2K2L0yN3Ul1gxMAi1ytCw8nhV1cE0ejnZe+O5n3XnnRaG7u4X0cP7jXeF+Y5VkVfG9xDpX1DmOZQrRWUtjFNdmxdiW20rMk33mX0dz0uan4+gdwZ6LhvjDHbfRq50+on8w+Cs/n1qtcKfUacDdgB44D39Va2wyMS7RyGWmptItLoP+QO4xlVtsq2JK+hNEd/Y32hdleWENZrYOnB0Ubydt5uoYFh8s4VdlAXIgvU3pHcktckJFsIUxw94o9A+irte4HZAFmfy8XrVJ+9hGO7tnOmCmPYbGaK8AblsylsaHBeBvd5dkVRAV4kdzB/eK7raCa/9t+hnt6RPCPiUnc3yuCt3eeYcupagMjFcIMt67YtdbpX/hwKzDFveGItmDT0gV4+/gy8t4HjWW6XC5Wz59Jz4GDSQyrMpZbcq6RA8W1PNIvCqvF/Tn7uYfKeGZwzMU3iVvjg/G1Wvhg71mjO2SFcIfJCcengLkG80Qr9eCzv+LWlIkEh5lbjrhv83rOFuTzeIIDCDGW2y7Qm79P7EyIr5mXep6t/iurdW6KDiDP1oDWWtbHi1bhsq92pdRqIOZrvvRrrfXi89/za8ABzPqWnOnAdIComLhrGqxoHby8vY22DwBYfb4vzG3x5q96TR6kERPkQ3ZZPX3af17cc8rriQnylqIuWo3LzrFrrcdorft+zZ8LRf1JYBLwiNbfvPNDa/2O1jpZa50cHC59sNsirTV/fOY7bPx0ntHcs4X57P1srfG+MKuP2/jfzEKjXRwn94rg/3acIae86bDu3PJ6/ratiMm9Io09hhDucndVzDjgeeAOrXWtmSGJ1urwzi0c2LqJYeMnG81dPW9ms/SFWXKsAosCPy9zbxZjksJwuDSvbiqgssFJsK+VKb0jGdvF3PJMIdzl7sTj3wFfIOP8r6FbtdY/cHtUolXKSEslKDSM21ImGcu0N9SzYc67DLpjLFEBJ4zlHi6pI6+ygR8NjjE+RTKuazh3dQmj3qHx9VKyk1W0Ou6uiulqaiCidSs/W8TO9auY8Mj38fEz1xlxa/qnVNud3KMPAoHGcldkVxDobeGOTuZuxH6RUgp/bynoonWSnafiiqxd8DHa5WLM1MeM5qaf7wtzk8G+MOV1DjafqmZ0Uii+XvISFzce2V8trkjvW4Zi9faifVyCsczjh/aSe2gf3x/U3uh0iZdF8UDvSO7s3Hzz3i6tqXe48POyyFSMaHWksIsr0jt5CL2ThxjNzEgz3xcGIMTXyiP92hnNvEBrzcocG/MOl3HO7iTQ28oDvSOZYLhpmRDukMIuLmvVnA8ZePsoo1fr1bYKtqz6lNEJZvvCHDpbS43dSXKHICM7Tb9sTW4lS7Mq+H+3x5MU4ceJinre3FKE1dJ0U1WI1kAmIMW3ys8+QuqffsP2NcuN5q5fPJdGu/m+MHMOlvLurmKjmV+06Gg5PxocQ1KEHwCdw/149tYYFh0pb7bHFOJqSWEX3ypj3gy8fX254x6DfWGczqa+MDffSmKYn7HcU5UN7C+uZVzX8Ga5Wgc4U2Ona8SlY+4S7kdxTSPfsj/vKxocLjaerGJpVjknbfWmhylucFLYxTeqra4ic9lCht51r9m+MFs2UFKYz93+p4xlAqzIseFlUaQ042ahxDA/9p25dC/evuJzJIb5XvEc+/Hyep5emsv6k5XkV9r5/foC/m/7GTmyTxgjc+ziG21atoCGulpSpj1uNDcjLZVww31hahudrM2tZFjH4GY9TOPBvpH8344zfN/Vnt5RARwprePdXcV8/wp7vbu05vXNp/newPbcfn6Nfd0AF79em09mXjUjEptn3b24sUhhF9+osqyEnjffSlLv/sYyiwvy2PfZOqb1icDL4HRJcU0jIb5WJnZv3huYg+OC8VKK+YfLeHtHMfEhPjxzSwzJV3jQxomKBhQwPOHzNzV/bwv39YhgU36VFHZhhBR28Y2m/eh5XC5zDbTgC31huoQZze0c7sc/707ieiw4vLlDEDdf46EdDpfGy6K+Mm3jbVU4XDIVI8yQOXbxtUqLCgGwWMy9ROz1dWxYMpfkkXcRGeBtLNdW56DR6cKivlowW5uuEX7UNDrZX3zu4uecLs2yrIpmaVksbkxS2MVXlBUX8dN7hpGRlmo0d0v6p9RU2rhbHzCa+87uYn6y4mSbuPlotSh+cmssf8o8zT+2F5F2qJRfrDqJr5diVDPulBU3FpmKEV+xZsFHaJeLAcNHGcvUWpORlkrHEB9uam+2L8zWU9VM7B7eZrb2D4gJ5K/jE9mQV0V1g5PH+rdjYGxgmxm/aP2ksItLOBrtrFs0mwHDR9OuQ0djuccP7SX38H6mD4o2Ol2SftyGU8P4bm1r12dkgDf3y+EcopnIVIy4xPa1K6gsK2mGJY4z8AsI5M7O5lZ9OFyaVTk2BsYE0iHYx+08rTV2p+uqNhoJ0RrJFbu4xIbFabSPT6DfkDuMZVZVlLM1/VPGJPgT4G2uL8zeM+cor3Pwg+QrW0P+TfJsDXxytIzM/GpcWhPi60VKl1Du6RFBkME+NkJcL1LYxSWe+9M/OVuYb3Q1zIaLfWE6GMsEGBQbyKtjEugeee0Hf+w9c44/bz7N+K5hTOkdQY3dRYS/Fydt9fwqI49XRicQ1owbnoRoDjIVIy4REBRMYo8+xvIu9IXpNeg2OoX5GsuFplOMerULuOa+MA0OF29sPs0j/aJYmWOjuMZBhL8XOwprKKxqZGBMIO80Y0MxIZqLFHYBNPWF+d2T93F45xajuXs/W0fJ6VPc7ZdvNHfG3rO8v9u9orspv4quEb6szLbxH4Oi+cltsUzuFckroxPoEOyDl0Wxt6hpukeItkQKuwBg49L5ZO/fhV+AuaWI0NQdMtzfi1sN94VZnm2jssHpVk5WaT092wVQXue4ZIu/UoqJ3cPZe+YcPaL8OV4u3RdF2yKFXTStMZ83g659B5rtC3PqJPs3r+euLqFG+8JsOFlFncPFBDeXOFpU03N3ao3zS50TGpwuvCxN2/ybqQOwEM1GCrvg4PZMik4eJ2XaE0ZzV8+ficVqZazBvjBaa1Zk2+gS7kv3SPd6ufePCWTvmVoSw/xYmvX5QRmNTs2Cw2UkdwjieHk9PaKu/easEC1BbvcLVqfNICgsnFtTJhrLtNfXsX7xXJLvHEdkQI6x3EMldeRVNvDjwTFub3QaHBfEB3vOMqF7GKtybGwvrKFjiC97ztTQNcKfU1UNjOgUIkseRZsjhV0weMwE+g25Ax9fc6cZbV61hHNVldzt3AcEGsuN9PdiYrewi73M3WG1KP5rRBy/X1/AwNgAOob40ujUTOgWzvbCGrzO93URoq2Rwi4YOu4+o3kX+sIkhPrQ12BfGIDYYB+mJ8cYy0sM8+PNcYmkH7ex7kQV5xqdRAd6c1eXMIYlhOBtlQl20fZIYb+BORrtZMybyfAJ9xs9+u74wb2cOHKAp5PN9oXZWlBNuJ+X8TnvMD8vpvWJYlqfKKO5QrQUuXl6A9u+ZjkzX3+J3EN7jeZmzJuBf2AQIw2eBuRwaf61s5g5B0uNZQrhqaSw38DS02YQHd+Jm4z2hSljy6oljOzgZbQvzLaCasrrHIzvFmYsUwhP5VZhV0q9rJTar5Taq5RKV0qZbQYims3JY4fI2ruDMdMeN9oXZv3iuTga7W6vMf+y5dkVtA/0ZlDstR1JJ8SNxN2f6Ne01v201gOApcBv3R+SuB4y0mbg7evLHXdPNZbpcjpZM/8jeicPISHUXF+Y/MoGDp6tY1zXsGvuCyPEjcStm6da66ovfBgISCPrNkBrTbWtjGHjJxMUau7Kuqy4CJ/qYib5OgFz8+uFVXbC/Zta6QohLk+5e6iAUuoV4HGgErhTa13yDd83HZh+/sO+wEG3Hrh1iwI8+S6fJz8/T35uIM+vreuhtb5s46XLFnal1Grg6xYO/1prvfgL3/ci4Ke1/t1lH1SpnVrr5Mt9X1slz6/t8uTnBvL82rorfX6XnYrRWo+5wsecBSwHLlvYhRBCNB93V8V0+8KH9wJH3RuOEEIId7m78/RVpVQPwAXkAT+4wr/3jpuP29rJ82u7PPm5gTy/tu6Knp/bN0+FEEK0LrLzVAghPIwUdiGE8DAtVtg9uR2BUuo1pdTR889vkVIqrKXHZJJSaqpS6pBSyqWU8pilZUqpcUqpY0qpHKXUCy09HpOUUu8rpc4qpTxy/4hSqqNSap1S6vD51+ZzLT0mU5RSfkqp7Uqpfeef2+8v+3daao5dKRVyYeeqUuonQG+t9ZXefG3VlFJjgbVaa4dS6n8BtNa/auFhGaOU6kXTDfN/Af+ptd7ZwkNym1LKCmQBKUABsAN4WGt9uEUHZohSagRQA8zQWvdt6fGYppSKBWK11ruVUsHALuA+T/j3U029rwO11jVKKW8gE3hOa731m/5Oi12xe3I7Aq11utbacf7DrUB8S47HNK31Ea31sZYeh2GDgRytda7W2g7MoWkJr0fQWm8Eyi/7jW2U1rpIa737/H9XA0eAuJYdlRm6Sc35D73P//nWetmic+xKqVeUUqeAR/DcBmJPAStaehDisuKAU1/4uAAPKQw3GqVUIjAQ2NbCQzFGKWVVSu0FzgIZWutvfW7NWtiVUquVUge/5s+9AFrrX2utO9K0a/XZ5hyLaZd7bue/59eAg6bn16ZcyfMTorVRSgUBC4CffmlWoE3TWjvPd9GNBwYrpb51Oq1Zj8bz5HYEl3tuSqkngUnAaN0GNwtcxb+dpygEOn7h4/jznxNtxPn55wXALK31wpYeT3PQWtuUUuuAcXxLI8WWXBXjse0IlFLjgOeBe7TWtS09HnFFdgDdlFKdlVI+wEPAkhYek7hC528wvgcc0Vq/0dLjMUkp1e7CyjqllD9NN/i/tV625KqYBcAl7Qi01h5xhaSUygF8gbLzn9rqKSt+AJRSk4G/Ae0AG7BXa31Xiw7KAKXUBOAvgBV4X2v9SsuOyByl1GxgJE1tbYuB32mt32vRQRmklBoObAIO0FRTAP5La7285UZlhlKqH5BK0+vSAqRprf/7W/9OG5wlEEII8S1k56kQQngYKexCCOFhpLALIYSHkcIuhBAeRgq7EEJ4GCnsQgjhYaSwCyGEh/n/HALXz8W8jb8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# https://scikit-learn.org/stable/auto_examples/svm/plot_svm_kernels.html\n",
    "clf = svm.SVC(kernel='linear' , gamma=2)\n",
    "clf.fit(X, Y)\n",
    "\n",
    "# plot the line, the points, and the nearest vectors to the plane\n",
    "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=80, facecolors='none', zorder=10, edgecolors='k')\n",
    "plt.scatter(X[:, 0], X[:, 1], c=Y, zorder=10, cmap=plt.cm.Paired, edgecolors='k')\n",
    "XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]\n",
    "Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()])\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(XX.shape)\n",
    "plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.Paired)\n",
    "plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'],levels=[-.5, 0, .5])\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Polynomial Kernel\n",
    "\n",
    "A more interesting example is the polynomial kernel of degree $p$, of which we have already seen a simple example:\n",
    "$$K(x,z) = (x^\\top z + c)^p.$$\n",
    "\n",
    "This corresponds to a mapping to a feature space of dimension $d+p \\choose p$ that has all monomials $x_{i_1}x_{i_2}\\cdots x_{i_p}$ of degree at most $p$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For $d=3$ this feature map looks like\n",
    "$$ \\small \\phi(x) = \\begin{bmatrix}\n",
    "x_1 x_1 \\\\\n",
    "x_1 x_2 \\\\\n",
    "x_1 x_3 \\\\\n",
    "x_2 x_1 \\\\\n",
    "x_2 x_2 \\\\\n",
    "x_2 x_3 \\\\\n",
    "x_3 x_1 \\\\\n",
    "x_3 x_2 \\\\\n",
    "x_3 x_3 \\\\\n",
    "\\sqrt{2}c x_1 \\\\\n",
    "\\sqrt{2}c x_2 \\\\\n",
    "\\sqrt{2}c x_3 \\\\\n",
    "c\n",
    "\\end{bmatrix}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The polynomial kernel allows us to compute dot products in a $O(d^p)$-dimensional space in time $O(d)$.\n",
    "\n",
    "Let's see how it would be implemented in `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-3.0, 3.0)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABLp0lEQVR4nO3dd3RURfvA8e9sTe+9FwKE3qSDgCJFFLG3V319FazYsYv1J/aGXewKKorYAKWD9N5reiG9J5tt8/sjMRITSCQJuwnzOSfnsHfn3n2WJM9O5s48I6SUKIqiKB2HxtEBKIqiKK1LJXZFUZQORiV2RVGUDkYldkVRlA5GJXZFUZQORiV2RVGUDqbFiV0I4SKE2CSE2CmE2CuEeKo1AlMURVFOjWjpPHYhhADcpZTlQgg9sBa4S0q5oTUCVBRFUf4dXUsvIGs+GcprH+prv9SqJ0VRFAdpcWIHEEJoga1AJ+BtKeXGRtpMBaYCGF3d+ofFxLfGSyuKAthtNnIyUvELCqEiOwWdRqDVCKosdgLdddT8Ya20d0cLTflSysCm2rV4KKbexYTwARYAd0op95yoXVy3XvK5r35rtddVlDPdsvlfMuf/HuaJOd8za+qljI33IafcTEapmfcvUJ2ojmLy3ANbpZQDmmrXqrNipJTFwApgfGteV1GUk/tz0QLCYxMozs/FbJMMCnfnYH4V/ULdHR2a4gCtMSsmsLanjhDCFRgLHGjpdRVFaZ6SgjwO7tzCsAkXsXXl73gZtfQIcuejyZ24okeAo8NTHKA1xthDgc9qx9k1wLdSyl9a4bqKojSDt38gr//0Jzq9jgcuPZfBYR5oa8fYXXRqqcqZqDVmxewC+rZCLIqinKLAsAj2bFxLZVkJg/uE88aGbHoGuTEmztvRoSkOoD7OFaUdSzu8n5fv/i/ZaclsXf0HBq0g1teF5ckl5FdaHB2e4iAqsStKO7Zu8UJ2/LkCV3cPtq78nd4h7hwqqAKgd4i6cXqmUoldUdopKSXrf/+ZHgOHU1qQT352BgPDPdiVU4mrTkMnPxdHh6g4iErsitJOHd61jbzMNIaOn8y2NUsBGBDmwe6cSroHuaLVqEVJZyqV2BWlnVq/ZCF6g5EBo8axfc0yYhN74mnQEOqpZ0CYh6PDUxxIJXZFaadCouI474rrsdlsHNmznb7Dz0Gv1fDE2ZFMSPB1dHiKA7VKrRhFUU6/cVfeANSsOpV2O4NTFmDxNaLXqv7amU79BChKO5R6aB/mahMAO9auwMuoJcHfhfuWpPLu5mMOjk5xNJXYFaWdsVmtPH/r1Xz07EPY7XZ2rV9F3xB3Ksx2UkuqCXBTf4if6VRiV5R2Zt/W9ZQWFTBg1DhSDuymrLiQfqE1Rb8Auga4OjhCxdFUYleUdmbT0t8wurrRZ9godq1fDUCfUHcOFFShEZDgrxL7mU79zaYo7YjdZmPzisX0HXEOBhdX9mxcQ3Tnbvi42DmYX0Wsj1EV/lJUYleU9uTQzi2UFuYz8JyJVFdVcWjbBiZ19gWCODvGC43aKUlBJXZFaVe69B3Is1/+QnhsZw5s34jVLukd4gbAuXE+jg1OcRoqsStKOyKEIK5bbwD2bvoTnQa6B7qRU27GLiHEQ6/2N1XUzVNFaS+S9u3io2cfoiivZp76vi3r6OzvilGnYeHBIu5enIy99bYwVtoxldgVpZ3Y8PvPrPrpWwwurlSWlZJ8YA89gmqGYZKLTMT6uKjCXwqgEruitAtSSrau+p1uAwbj7unNwZ1bkHY7PYLcsEtJclE1sb5GR4epOAmV2BWlHchMPkx2ahIDRo0D4OD2TWh1OroEuJJXYaHKaifGR9VfV2qoxK4o7cDWlb8D0H/UeQAc2L6JeG8dLjoNKcXVAMT4qB67UkMldkVpB7Q6Hf3PHotfUCjmahNJe3fSLbBmfL1rgCsPDQ9XiV2po6Y7Kko7MOm6W5h03S0ApBzYg9VirqsJ4+2iY0ikpyPDU5yM6rEripOrLC9Dyr/nMR7evQ2ALrWJfW1aKUlFJofEpjgnldjbKbvNxrbVS/nu3ZdZ9v1XVJaVOjokpY18+MwMHrv2/LrHR/fsICA0Aj9XHVJK3tp4jKVHi9s8jiOFJubtyeeH/QXkVlja/PWUU6cSeztkqqrk2WlX8P37ryKEhj2b1nL/JaNJO7Tf0aEprcxqsbBr/Sqiu3SvO5a0bydx3XoBUGKyYbLaCfMytFkMUkrmbMvh/9ZkUG21k1tu4d7FyaxKKWmz11RaRo2xt0O/ffkBXr7+TP/gWzSams/mlT/O48NnZ/DM5z87ODqlNR3etYWq8jL6DBsDQHlpMbkZaYyZcjXkJ3Gstucc4t52iX1vXhWbMst5c0IsHgYtABM7+/LQH6n0D/OoO6Y4D9Vjb4c2L1/MxGtvrkvqACMvuIzcjDQKcrIdGJnS2nauW4VWp6PHoOEApB7YC0BM1x4A5JbXJPYgD32bxbA+vYyx8T71EniUt5EeQW5sy6pos9dVTl2LE7sQIlIIsUIIsU8IsVcIcVdrBKacmBAa7DZbvWNSSqS0o1FLyjuUXetXkdCzH24eNbNeUg7uAaD7sicByKusSextuR2eEGCXDYvQ2KRE1RtzTq3RY7cC90kpuwGDgduFEN1a4brKCQw+bxI/f/YeNqu17tiy778kLDYB38AQB0amtLZLpt3L5P/dWfc49dB+/F11eBlrEvn4Tj68Nj4GN33bDYcMi/Tkj6PFlFb//fN2tNDEgfwq+oe5t9nrKqeuxR/zUspsILv232VCiP1AOLCvpddWGjf+qhvZv3UDD14+lj7DR5OZdJjMpMM8+PaXjg5NaWX9zx5b73H64f31FiK5G7TEtfEYd2KgG6Njvbnj12SGRnpSabGzNbucOweFtukHinLqWvXvNyFEDNAX2Nia11XqMxhdmPHmZxzYtoEju3cQ160XA0aNw+Ci9rrsSLavXY6PfyCxiT0BsFmtZKUcpW/c373kpUnFeBm1DAxv2wVKV/cMZGS0F1uzKjBoBTf2C8LHRc29cFat9p0RQngA3wN3SykbTKoWQkwFpgIEhIS31suesYQQJPYfQmL/IY4ORWkjn734OJHxXbjvtY8ByM1Kx2KuJtLLt67N9/sKiPN1afPEDhDhZSTCS5UtaA9aZVaMEEJPTVL/Skr5Q2NtpJQfSCkHSCkHePr6tcbLKkqHlZ+dSW5GGt3OGlZ3LDvlKADhx81ZL6224WVUwyFKfa0xK0YAc4D9UspXWx6SoigHttWMZib2H1x3LCc9Bfg7sdulpMJsV/PIlQZao8c+DPgPMEYIsaP2a2IrXFdRzlgHtm/CzcOLqE5d647lZKTi6uGJZ20iN1ntSMBNr5ajKPW1xqyYtYCazaoorejonu0k9O6PRvt3bzwvK4PA0AiEqJlTXmWxA+CiU4ldqU/d1lYUJ/TUpz9SVlJc71jBsUwCQsOBDAB8XXV8PqUTRpXYlX9QiV1RnJDBxRX/f0xfLUo9SHeZDeE1i9A0QuCtphwqjVAf9YriZFb/Mp95b82qV4PdXG2izGzH77jSAUVVVr7clUdaSbUjwlScmErsiuJkNvz+M9tW/4E4rhBLSUE+AL7H9dCLTVa+21tAZqn5tMeoODeV2BXFySTv311Xb/0vZcUFAPXmrP+V9hsr0KWc2VRiVxQnUpSXQ0lBHtFdetQ7Xl57I9XzuMSura3kabOftvCUdkIldkVxImmHa3bBiu5cv0BqZXkZAO7HFd0yaGsSu8WuMrtSn0rsiuJEKspK8PTxI/K4hUkApsqaDS1cdH+Pu/81f91kVUMxSn1qrpSiOJGh4yYzdNzkBsfNpioAjNq/+2KeRi3zLu1cL9krCqjErijtgtVSs1OSTvt3EtcIgateJXWlITUUoyhO5Nmpl7NsfsMNU+z2mq0Q/7nz4fx9BSw5UnwaIlPaE5XYFcVJVJaXsW/L+robpc2xMaOMP9MabH+gnOFUYlcUJ5GXlQ5AYHhkg+c0mprZMPZ/3Cf1d9VRUGVt0F45s6nErihOouBYFtD4DmN6Q00NdqutfmYPdNeTW2GpV35AUVRiVxQnUZR7DADfoJAGzxld3QAw/WM1UoiHAbNNUmSytX2ASruhEruiOAlXT0869eiLt59/w+fcPQCoNNdP7OGeBtz0GgrVcIxyHDXdUVGcxInmsAN4ePsAUGau3zPvFeLG15ck1CsYpigqsStKO+DpW9OLL/nHkItGJXSlEWooRlGcxEfPPsRr993c6HO+gcEAFFRZGjy38EAhr6zLatPYlPZFJXZFcRL52RkU5eU0+pybhxeuOg15lQ3H0svMNtamlWKyqmJgSg01FKM4pW2rl/LL5++Rk55CZKeuTL7xDhL7D3Z0WG3KarGg1TX+KymEICiuK1lBoUBmvecSA1yxSziYX0XvEPfTEKni7FSPXXE6G37/mU9mPcr4q27k6c9+YuiEi3jjwVvZu3mdo0NrU1LaEZoT/0oGR8WSnZbU4HhioCsaAXtyK9syPKUdUT12xalIKfnuvVe47ZnXSew/BICRky5Fp9Ox4MM36H7WUAdH2HaE0GC3NRxD/0t4bCc2L1+E2dYJw3FVHt30WuL9XNiZU8k1pyNQxempHrviVMwmE/lZGXTtV3/YpdeQs0k5uNdBUZ0enXv3p1PPfid8PiohEWm3k1rccPPqEVFeRHga1ApUBVA9dsXJ6I1G3L28yUw6RER8l7rjKQf2EtRIDZWO5PLbZ5z0+ZiuNdvlHSk0keDvWu+5yV392iwupf1RPXbFqWg0GiZcczPvPXkf2WnJAKQc3Msnsx5l4jWNTwU8UwSFR+Hl68/B/KpGn5dSklfR+FCOlJJik5W8Cgu2f1YSUzoc1WNXnM75/5mK3Wblqf9OwWazYnRx5aKbpjP8/IsdHVqb+umTt1ky7xNmL97c6EpSIQRd+pzFvk3LGj3/6935/HigkM+mdMLtuL1RV6WUsPBAIcfKLRh0GpCSsfE+XNrNH6NO9e06IpXYFaej0WiYfOMdTLruFirKSvHw8kaj1TZ9Yjun1espysuhoqwEDy+fRtt0O2som1cs5li5mRAPQ73n+oa68+3eAjZllDMq1huAr3blsT69jLHx3mSVWTBZ7UT7GDmcX8UTK9J5enSkSu4dUKt8R4UQHwshcoUQe1rjeooCoNXp8PL1OyOSOoB/cBjwd/nexvQcPBKADT3+1+C5rgGuBLnrWJFSAsDhgiqWJZVwdowXCw8WEeyup3uQGxszyqm2SXxctCzYX9gG70RxtNb6qP4UGN9K11KUM1JQeBQAOempJ2wTGh1HUEQUO9Ysb/CcRgjGxHqz81glOeVmFh8p5pw4bxbsL2TWudFc3M2f8+J9+L9zoii32Oni78qSo8VqzL0DapXELqVcDaiPfkVpgbCYeAAykw+dsI0Qgv5nn8eeTWuptDSswX5unA9CwLLkEpKLTLjoBN0CXQly19e10WoEY2K9SC+txl57U1XpWE7b4JoQYqoQYosQYktZkfoMUJR/cnFz55xLriUirvNJ2w08ZyIWczWbMsobPBforufJUZFckuiPTiMQCEqrG34AlFbbMGo1mG0SvVaNsXc0p+07KqX8QEo5QEo5wNNXzblVlMb879HnOWvMhJO2SejVn4DQCFamNL6Jde8Qd4w6DQPCPMgqM1NQZa234XVOuZnfDhcT4qkn0suAl/HMuIdxJlGzYhTFyZQU5mN0dcOldju8f9JoNAw//2IWznmL/EoLAW76Bm3WppWy81gFaaVmru4ZwCfbc1lwoBAvg5YD+VVc0s2f34+UcE2vgLZ+O4oDqL/BFMWJpB7ax63n9mX76qUnbXf2hZcjpWRpUkmjz0sJe/KqmJjgw9zd+fQKcWNYhCfdg9yYkODDz4eKODvGi2FRXs2OraDSwtKjxaxOKW10fF9xHq013XEusB7oIoTIEEI0nIulKEqTIuI64+bhxe6Na07aLjgimp6DR/L7kWKsjcxqGRrpSbingQ0Z5bw+IYZQDwMbMstZl15GWbWdJ86O4Ioeze+tLzxQyPRFyezMqWR1aglTf0pix7GKf/3+lNOjVYZipJRXtcZ1FOVMp9Xp6DlkJDvWLsdut6M5SRnf8VfdyEsbVvNnWilnx3jXv45GcHl3f17bkM2+vCou6x7AZd1PbdjlSKGJHw8U8saE2Lphn725lcxam8mHF8bjohY4OR31HVEUJ9P/7PMozs/l8K6tJ23Xe9howmMTWLC/sNGqjiOivYjyNvDlzrxGe/XNtSa1lHHxPvXG8rsHuRHv68K27IYzcxTHU4ldUZxMv5HnojcaWbd44UnbaTQazr9uGsnF1WzNbjgsotUIpvYP5qb+wWhbsOe12WbHqGt4ARedwGxTi5uckUrsiuJk3Dw8mT7rHabcNL3JtsMnTiEgNIJ5u/Mb7bX3DHZnQJhHo0XFmmtguCdLk0qoPm5P1WPlZnblVNJXbcXnlFRiVxQn1P/s8/AJCGqynU5v4OKb7+JwoYmNmY0Pi0gp+Xp3Hp9szz2lWPqEuNE1wJV7Fqcwf28Bn+/IZcbvqVzXJxBvFzVj2hmpxK4oTmr72uV8+sLjTbYbMelSQmPi+SzdtdG6L0LUrD796WAhSUWmfx2HEII7BoYwbUAwxdVWhBA8MyaK8Z18//W1lNNDJXZFcVJZKUf4/ZtPObp3x0nbaXU6rrzzIbJSjrDkaHGjba7tFYinQcs7m4+dUtEvIQS9Q9y5qV8w/+kdSLSP8V9fQzl9VGJXFCc1ZsrVuHl48fNn7zXZdsCocXQbMJSvd+U1WhvGw6Dlxn5BHC4w8dvhoma9vto/tf1SiV1RnJSruwfnXXE9m5f9RvqRAydtK4TgugeepMIm+HJnXqNtzo72on+oO3N351NlsTfaJq2kmrc2ZnPld4eYMu8gt/5ylAX7CzBZG2+vOCeV2BXFiU245mZc3D349u0Xm2wblZDIuCv/y5KjxezPq2zwvBCCW88K4dlzonDVN/zV355dwaPL0vBz0XFxoh9TEv04L96H/XmVPLosjQqzKiPQXqjErihOzNPHl6umP0y/s89r1tDIpbfcR0BIOG8dNWKxNexlB7rrifN1AWqqPP6l0mLj1fVZXNMrgCVHi8ksM+Oi07AmrYxKiyTG28inO05tVo1y+qnErihO7txL/8Poi65s1lx0V3cP/vfo82QmH2benoITtlt0uIjbf00mtbgagFUppXQLcOGXg0VMGxDMPUPCuKJHAC+NjcZFJ/B31/FnWhnlqtfeLqjErijtgN1uZ8m8T1n983dNtu09bDQjL7iM7w8UsXN840M4QyI9cTdoeHldJtVWOwfyq+jk70qV1c7QSM+6dlqNYFIXP3YeqyTax3hK0yWV008ldkVpB4QQbF6+iM9ffoqSgsZvjh7vuvufxD8olHcev6vRG58+LjruGhxKWomZOdtz+etvAZuEfw742OwSjQC7rNlXVXF+KrErSjsghOC/Dz+H2VTFZy8+0WR7N08vpj31KjnpKXy0LafRNv1CPZjS1Y8lR4oxaDXsy63Ez1XHsuNqvFtsdhbsL6RviDsZpdXE147PK85NJXZFaSfCYztx8c13seGPX9jwxy9Ntu9+1lAuuOE2/jhawtq0xrfRu7Z3IH1D3EkMdCW5uJpzY72ZuyefJ1ek88HWHO74LRlPo5bkYhOjY70bnU2jOB/1XVKUduSCG24jrntvPp31GKaqhlMa/+nSW+6jU89+zN5ZRlaZucHzOo1g5qgIRsd68/CICObtyWdQuAe9gl3xMmgZ38mHEpOFcrOd//QKbIu3pLQBVcFHUdoRrU7HrU+/TnFezgn3RD2eTq9n+qx3eOTq8bywNpMXx0Zj/MfGGH/NtkkrqaZLgCsuOg1LjpZQZbET6mngvHgfRkZ7o29J7V/ltFKJXVHamfDYToTHdgKgMDcbv6DQk7YPCA3n9mff5MXp1/Pu5mPcNTi00amTFWYbmzLLSfAP4P0L4tskduX0UEMxitJObVz6K3dfMJwD2zc12bb3sNFcMu1eVqSU8suhxmvFTO7qx8hoL77elc/GjLLWDlc5jVRiV5R2quegEQSEhvPmg7dSnN/0qtCLbppO/1Hn8fHOAnY2shH1X+V54/1ceHV9Nilqznq7pRK7orRTbp5e3P3S+1SWl/LGjFuwWhreHD2eRqPh1qdfJywmnhf/zGz0ZqpRp+GREeG46zXsymn65qzinFRiV5R2LCohkakzX+bgjs18MuvxJuvJuHl4cv/rnyDcfHh6j2i0RIC/m563JsZyYVe/tgpbaWMqsStKOzd03GQuumk6EfGdm9U+KDyKu196n9yMNJ5LDcDSyIbU7gYtAPvzKnlrY/Ypbc6hOI5K7IrSAVx+2wNMuPp/CCEwVzc9Np7YfzDTZr7Mvi3rmb0p+4Q9/aNFJpYmlTBnW47aeKMdUYldUTqQ/Vs3cPcFw0g5sKfJtsPPv5hLb72PlSmlfLU7v9E2kzr7cVFXP349XMyCA4WtHa7SRlRiV5QOJCQqBq1Ox4vTrycvK6PJ9lNuuovRU67iu70FLDrBlnnX9wlkeJQnn+3IY0VySaNtFOeiEruidCC+gSE8+NbnmE0mZt1xLaVFJ+9lCyG48eH/o++Ic3l/Sw7r0xvOX9cIwd2DQ+kV7MbWrHI1JNMOtEpiF0KMF0IcFEIcEUI81BrXVBTl1ETEd+H+1z8mPzuDl+66nuqqqpO21+p0TJ/1Dp169efljXnsamSOu16r4dGREdwzJAwhhEruTq7FiV0IoQXeBiYA3YCrhBDdWnpdRVFOXdd+g7jz+beJjO+CTq9vsr3R1ZUH3viE0OhYnluTyeGChh8GLjoNWo2goNLCw0vT1AImJ9YaPfaBwBEpZZKU0gzMAya3wnUVRWmBAaPGMXXmy2h1OkqLCrHbTr6tnYe3Lw+9/SWeQeE8tTKDtJLqRtvZJORWWHhiRXqji5wUx2uNxB4OpB/3OKP2WD1CiKlCiC1CiC1lTYz7KYrSeirLy5h5/YV8+MwM7PaGuykdzzcwhEfe/Rqtlz+Pb6rgWHnDxB3krufpMZFIYOaKNPIqLG0UuXKqTtvNUynlB1LKAVLKAZ6+akWbopwubh6eDJt4Mat++pbPX5rZ5Ph4cGQMD7/zFVaLmceXp5Nf2TBxR3gZeXJUJOVmO0+sSKOoytpW4SunoDUSeyYQedzjiNpjiqI4iUum3cP5/5nK7998ytevP9dkco/s1JWH3v6SMuHCYztko4k73s+FJ86OwLN2lariPFojsW8GEoQQsUIIA3Al8FMrXFdxECklORmpzZoHrbQPQgiuvvsxxl52Hb9+8T5/fPtZk+fEdevNjDc/ozD3GI/u0lBiapjcEwPdeGFsNL6uOqx2SaXl5OP4yunR4o02pJRWIcQdwBJAC3wspdzb4sgUh0jat4sPn5lBSUEeUkoCQsOZ9uQrRMQ1rw6J4ryEEFz/4DP4BgYzZFzz5jd06XMW97/+CS9Ov46ZhfDMmCg8jfV76H9t2vH6+ixyKiw8NToSN73qxTtSq4yxSyl/k1J2llLGSymfa41rKqdfeWkxL911A5Oum8bsxZt5e/FmRk2+klm3X4vZdPK50Er7oNFouOim6Xj6+GIxV7N+SdN/XHc/ayj3vTqHjEp4fL9LoxUhAUZEe3G00MQzqzKospz8Jq3SttTKU6XO+iU/0W3AEIZNmIJGo0Gj1XLOJdcQGd+FLSuXODo8pZUtnf8lbz18Ows/nt1k215Dzubulz8g7fABnlyR3mhyHxThyX1DwziQX8Wzq9MxWVVydxSV2JU6RXk5hEY33OsyLCaewtxjDohIaUvjrriBoRMu4pvZLzQrufcdPoa7X3qf5GITT61Mb3Q8fViUF/cMDmNfXhWvrs9qi7CVZlCJXamT0LMf21YvrbeQxWqxsH3tchJ69XdgZEpb0Gi13PrUawybMIVvZr/Aj3PeavKc/mePZfrLH3G0xMrMFY0n95ExNcn9kkT/tghbaQaV2JU6vYeOws3Dk9cemMr+revZs3EtL911A+Gxnejce4Cjw1PagFan49anX2P4xIv59Yv3m7V36oBR45j+wrscLbHy5Ip0KhoZlhkZ40WXAFcA1qaVUq2GZU4r4YhiPnHdesnnvvrttL+u0jSzqYrF8z5h07Lf0Gi0DDnvAsZefh06vcHRoSltyG6zkZeVTnBkTLPP2bxiMW/efzPxfi48OSqybtel46WVVDP9t2R6Bbvx6MgIjDrVl2yJyXMPbJVSNtnLUon9DLZuyUIWffUReVkZxHbtwZSb76ZzbzXkcqZb8NEb2Kw2Lpl2T91UxhPZvGIxbz54K/HeOmaOisSjkeS+LKmYtzYeU8m9FTQ3sav/4TPU0vlfMP/dV7h02r08P3cRg849n1fv/R9H9mx3dGiKA0kpyc1I44cPXuO7d19ucoXqWaPH1w7L2Hj8gFujs2XOifNh+qBQduVU8tzqDDUscxqoxH4GslmtLPjwDe5+8X16DxuNb2AIoy66kstue6BZsyOUjksIwc1PvMToKVfx40dv8u3bLzYrud/90vukHtzHEyeYCjkmzpvpg0LZnVvJjkbqvSutSyX2M1BpUQE2m42ozon1jvcYNJzUg/scFJXiLDQaDf97dBajp1zNwo9n883sF5o8p//ZY7n3lQ9JLbPzxPI0yqobT+5vnx/HoAjPtghbOY5K7GcgD28f7DZrg1owyft3ExwR7aCoFGdSk9yfZ+xl1xESFdusc/qOOId7XvmQ1HLJo3sNlDaS3MM8a27C786p4JlVahFTW1GJ/QykNxgZe/n1vPv4XRxLS0ZKyYHtm/jqtWeY+J+pjg5PcRIajYb/PvwcoyZfAUBm8pEmh2X6Dh/Dfa9+RFbyER5fnkZpdePlfIuqbGzLruCplemq/EAbUIn9DHXJ1HvoPnA4M2+4iP+NSOT9mfdy1fSH6Tt8jKNDU5xQRtIhHrlqPF+++nSTyb33sNHc99ocssrMPLYsvdGqkCNjvLh3SE35gadXNb7QSTl1arrjGc5mtVJdVYmrh2eTU9uUM5eUks9ffpIlcz9m4rU3c809jzf587J74xpeuedGQox2nhkThY9Lw2Kyf6aV8vK6LDr7u/LkqEhc9aqveTJquqPSLFqdDjdPL5XUlZMSQnDd/U8y7qob+e3LD/n69Web7Ln3HDSCB974jGPVGh7ZqaG4kc06hkV5MWNYOBFeBgxa9TPYWlpcj11RlBMzVVWyaemvHEtPwc3Dk0Hnnk9gWGTTJzqhv5K7tNv59YsPSOjVn4HnTDzpOd3PGsqMNz/npenX82i+jWfHROHrWj/tDIn0ZEhkzUyZgkoLRp2m0YVOSvOpHruitJHVP3/H9PMHs3Hpr1C7K9Wj157PezPvwVxtcnR4p0QIwfUznuaO52czYPT4Zp3TbcAQZrz1OXlmLY8tT6PwBPuj2uySmSvSmXmCufBK86nErihtYOPSX/nu3Ze57Jb7KDiWxS+fv8/Gpb8y+qKrqCwr470n7nF0iKdMCMHQcZPRaDTkZaWzeO7HTZ6T2H8wD87+gjyLjkd30OgeqlqN4Po+QaQUV59wLrzSPCqxK0ork1Ly7dsvMum6W/jhwze4cvrDfLr+MM989hNJ+3YREBrOoV1bSTnY/neQXDr/Sz5/aSY/ffpOk2279hvEg7O/oCAni8eWpzWa3M8K9+DhEeGklZh5YoVK7qdKJXZFaWVJ+3aCEOzbsp5Lp91Ln2Gj0Wg0BEfGcMf/vcXqX75n6PjJ/PnbD44OtcWuuH0GQydcxLw3n29Wz71r34E8WDss8+hO0egN1QFhHjwyIpz0EjNztuW0RdgdnkrsitLKSosKCQyNICc9hfgefeo95+0XgJevH24eXpQWFTomwFb012YdZ40ez+cvzWTlwm+aPKdrv0HMeOsz8rMzeGSXhuJG5rn3C/PgydGR3NgvuC3C7vBUYleUVuYXFEJ2ahIRnbqwd8u6es/lZWVQXlJEWVEBfkEhDoqwdWl1Ou54fjY9B49g+fdf1duB60QS+w9hxlufkZeVxhPL0xtdodojyA0voxaLzc4n23PVsMy/oBK7orSyqIRE3L28ie/em58+fpsVC+ZSWlTAgW0bee3+mzn3suv4c9GPjLzgMkeH2mr0BiP3vPIRD73zFRpt86YqJvYfwv2vfUKWCR7bazxh4k4urubXQ0XMVGPuzaYSu6K0MiEEV931CAs/fptLpt3LpuWLuG/K2Xwy61EGn3cBB7ZtZOC5EwmNjnN0qK3KxdUNNw9PTJUVvPHgrSTt29XkOT0GDefeVz4iM+kwT65sfJu9zv6uPDwinNQSMzNXqqmQzaESewezefkiXrjzOh6/7gLmvTWL0qICR4d0Ruo5aAS3PPUqv331ASUFeQw8ZyKBYZH89Mk7dO49gBtmPOPoENtMVUU5R/fs4MU7r+NYWnKT7XsPHcXdL71PcpGJZ1ZlNFrxsX9YzWyZ1OJqnlTz3JukasV0IAs/ns3qX+ZzydS78QsO489FC9izcS3PfP4THt6+jg7vjGS329m7+U9y01Nx9fCgz7AxuHl6OTqsNpeVcpQnb5yCh5cPT326EE+fpn/+NvzxC289fDs9A114/OwIDNqG/c5NmWW8vekYT42KJMbXpS1Cd2pqz9MzTHlpMXdfMJwXvv0D/+DQuuPvzbyHkMhYLrppugOjU85EB7Zv4vlbryauWy8eeW8ueoOxyXPW/PI97z5xNwPDPXhweDg6TcP6MSarHZfafVNtdom2kTYdlSoCdgYwV5vISDpEWXERqQf3EdWpS72kDnDWmIkc3LHZQREqZ7KufQdyy1OvkpORSm5merPOGTHpEv770LNsyiznjQ3Z2BvpeP6V1L/bm8/MFelqD9VGtKgImBDiMuBJIBEYKKXc0hpBKU37/ZtP+f791/Dw8aWkIJ9u/QeRk5mGzWpFq/v723osLQnfQDUXWHGMIeMupO+Ic3Bxc2/2OWMvv57K8jK+mf0C7gYN0/oHN1p9NNBNz57cSmatzeSREeHoGxm6OVO19H9iD3AxsLoVYlGaafOKxSz6eg4zP/6BV35YyZu/rkdnMKLV6pj31qy6AlNJ+3bx6xcfMOaSaxwcsXImc3Fzx26z8c3sF9i9cU2zzrnwv7dz/n+msuhwMV94n99om1Gx3tw2MIRt2RW8vC4Lm/30Dys7qxb12KWU+wFVy/s0+/2bz7jyzocIi4kHwM3Dkxsf+T+mTxxM2uF93DFhIJ4+flSVl3HtvY/TqUdfB0esnOnM5mq2rf6DZT98xfNfL8Y/JOyk7YUQXH33Y5QVF/H9+68SdFYw4zs1vAF7XrwP1VY7H23L5c2N2dw9OFTlI05jPXYhxFRgKkBASPjpetkOqTg/l9Co+nOgPbx88PDy4cZHnkdvMFBeUkxYTCd0er2DolSUv7m4unH3Sx/w6DUTeevh23n8w+/qDRk2RgjBTY+9QFlRIe+tW4Gvi45BEZ4N2l3QxY9qq8THVauSeq0mh2KEEEuFEHsa+Zr8b15ISvmBlHKAlHKAp6/fqUeskNCrH1tWLql3LHn/bmw2KwEh4fgFhRKVkKiSuuJUQqPjuOmxWRzauYUfPny9Wefo9HrufOEd4rr14uV1WRzMr2q03aXd/Tk3zgeAnHJzK0XcfjWZ2KWU50opezTytfB0BKg0dOENt/HHd5/z3bsvc3TvDlb//B2v3ncTV9z5UJO9IEVxpKHjL2LEpEv55fP3KM7PbdY5Lq5u3P/6J/iERvLslnKOnSRxJxWauP3XZBYeaP8F1lpC3UZuh0KiYnnykwWUFhUw57mH2bx8EVNnvszISZc6OjRFadL1M57mmc9/xicgqNnnePsFMOPNz7HZrDyzKuOEK0+jfYycFe7Bx9tzWZlc0lohtzstWqAkhJgCvAUEAsXADinluKbOUwuUTq+cjFTmv/sKuzeuwd3Ti7MnX8H5105VvXvF4Qpzs/ELCm26Ya19W9bz/G1X0/2sYTwZdazRxUkWm52nVmawL6+Sx0ZG0C/MozVDdqjTskBJSrlAShkhpTRKKYObk9SV06ukII9nbrqUsNh4nvvyV25/7k12b1jDnP972NGhKWe4X7/8gPumjKIor/mbaXQbMIQbH/4/dq1fxac7Gh/K0Ws1PDIynChvIy/8mUlm6Zk35q6GYjq4Zd9/Rd8R5zLlprvwDwkjrltv7nv1I7asXEJeVoajw1POYP1HjsVqMfPjnLf+1Xmjp1zFeVfcwE8Hi1iV0vhwi5teyxOjIrk40Z9QzzNvEoFK7B1c6qF9dB84rN4xFzd34rv3IePoQQdFpSg194pGXnAZK3+cR0lB3r8699p7n6Br34G8vekYKcWmRtv4ueq4okcAGiHIr7ScURUhVWLv4IIjoknev7veMavFQtqhfQSFRzkoKkWpMem6W7CYq1mxYO6/Ok+n1zP9hXdxM2h5YW0WVZYT14ux2Ow8sjSNF9ZmYj1DVqeqxN7BnXPptaxa+A1/LlqAzWqlpDCfj56dQUyXHoTHJTg6vA6hsryMrav+YO2vP7Ds+69YMu9TFs/9uO4vovLSYjYt+43dG9eQvH83Bcey6so+nOlCo+Po2m8QG5f9+8kUPgFB3P7G12SXW/hg67ETttNrNVzZM4BdOZV8sCUHR1S0Pd3UtIgOLjgimntfncNXrz3DB08/gFarY8i4ydzx/GxHh9aulBYVcnDHJo7u2UHqwb1kpRxl4rVTGXflDRTl5fDKPTc2OOfmx18kIr4L2SlJvP7AtAbP3/n82wwZdyGZSYdZ8s0nBIVHERwZQ2h0HMER0c0qc9sR3PzES3j7+p/Sud0GDGHKTXfxw4ev0zfUg5HRjde6HxPrTWapmfn7CojyMTCpc8deJKkS+xmgc+/+PPXpj5gqK9Dp9ej0BkeH5PSklFRVlOPm4Ul5SRG3ju2LtNvR6nRExHUmoVc/QqJiAAgKj+Tpz3/C3dMbo6srOr0BjUaDwcUVqNkDdda836koK6G8tJjy4iJKCvKI7tIdgPxjmaxf8hMVpX/fCBRC8PhH8+nadyAZRw+StG8XkZ26Eh6XgMHYsTaYCI2KbdH5U26+i13rV/HeniR6BLnh59p4WrumVwDpJdXM2ZZLrI8L3YPcWvS6zkxttKEoxykvLWbNz/NZvmAuASFhPDj7CwBWLJhLWGwnYhN7tllirSgrISc9hezUJLJTkjjvyhvw8vVn4cez+Wb2CwBotFpCImOJSujKjY/8Hx7evlgtlnZdPqK8tJjfvviAfmePPeWCddmpSTx05Xn0C9DzyMiIE7artNj4clc+1/QMwN3QvE23nUlz57GrHruiUFNY7ZfP32P5D19jqqygU89+DDz373Kxo6dc1eYxuHt6E9etN3Hdetc7Pum6WzhrzATSDu8n7fB+0o8cIPXwflzdawpiffHKU2xZsZiYLj2I7tqd2K49iE3sRUBo+yi2l5uRxo9z3iKma0/ocWrXCI2O49Jb7mfuG8+xIaOMwY0UC4OaaZBT+9fsT2Cx1dxw7Yh13FViVxRg5Y/zWDz3YwafdwGTrruFmNphEmeg1ekIi4knLCaewWMnNXg+sd8gqirKST24h53rV2K32fALDmX2ok0ArP31B3QGA7FdexAUEe10FRD3bl4HQGziKWb1WhOu/h9rf53PnG1H6RvijlF34oRtsdl5eGkaCf4uTBsQ0qLXdUYqsStnrO1rlqHT6+k5eCTjr/4fg867oMXjvY4w+LwLGHzeBQCYTVWkHT5AWUlR3fMLPnqD7NQkANw8vYnp0p2zxoxn3JX/BcBus6HROmZYouBYFr99+QFd+w0iMCyyRdfS6fVc98DTPDftCn4+WMSl3U98Q1av1dAt0JWFB4voFezOkMjGe/jtlUrsyhmnsryML15+klU/fUvfEefSc/BIXNzc22VS/yeDiyudetYfp571ze+kHzlIyoE9JO/fTfKB3eSkpwI1SX3aOb0JDIskKqErEfFdiIzvQkxiT7z9Ato0Vou5mmenXo652sT1M55ulWt2P2sofUecyw87NzOxsw03/Yk/sP7TO4g9eVXM3pRNgr8LAW7t9z7FP6mbp8oZJfXQPt6YcQs5GalceMNtXDLtnjN6lpCpqpIf3n+N1EP7SD9yoK6U7iXT7uWSafdQVlzEF688RWh0LMER0QSGRREYFoGXXwAazb8fmy4vLWbf5vWcNWY8QggWfjybnoNHNLiv0BJJ+3by2LWTuL5PIBcnnnwaZVaZmbsXJZMY6MqToyKdbpjqn9TNU0X5h/QjB5h5w2TcPLx47P1vSOw/2NEhOZyLqxtX3/1o3ePSokIykw7hF1Qz7lycn8O+zX+y9tfv6513y1OvMvKCy0g7vJ/5772Cp48fbh6eGF3d0en1DJswhcCwCJL372b5D19TkJNFVsoRcjPSAHh+3hKiO3dj8o13tPp7iuvWm24DhvLbvk1M7uLXaAXIv4R5Gvhv3yB+PVREicmGzwmmSrY3HeNdKEozhMd15sIbbuecS67B2z/Q0eE4JS9fP7yO+8CL7NSV2Ys3U11VRV5WGnlZGeRmppPQu6bTWFFaM0XzyO7tVJWXUW2q2eEovkcfAsMiKMjJYvPyRfgFhxKb2ItRF15B4oAhRMR1btP3Mfby63hjxjp251TSJ9T9pG3Hd/Lh3DjvDjU7Rg3FKB2alJIf57zJsAlT2qQ2jqmqktz0VHIyUynMOUZxfg5lxUVUlJVgqijHXG3CarEg7XYQAo1Wi15vwGB0wcXdHVd3T9y9vPH08cXL1x9vvwB8AoPxDQzG08fP6YcG/slut2O1mNHp9A67IQtgrjYxbUxvRky6hDt1m5t1ToXZxoaMMs6p3WLPGamhGOWMJ6Xk01mP8cd3nwMw5aa7WnQ9s6mKo3t3cHD7ZpL27yL14D7ystLrtdEI8DJq8TBocdVpMGgFbhpRm6AlNjtY7ZISm50cq50Ks50Kiw1rIzWs9AYj/iFhBIZFEBhWU24gJCqG0KhYgiKinXIFqkajcYq4DEYXuvYbxIFtG2Fg83riS44U89nOPALd9PQKOXkv39mpxK50SFJKvnj5Kf747nMmXXcLF/1v+ildp+BYFptXLGb7mqUc2LYJi7kaqBmb7eJr5LyeAYR5GQh21xPorsfLqEXzL3vZUkqqrHZKTDaKTVaKTFYKKq3kV1rJq8gn91A2m7eto+y4srNCoyEwLIKwmE6ExyYQEd+ZyE5dCI/tjNHV9ZTea0cT06U7uzesxjqgE7qTjLP/5fzOviw6UszH23N5dXzMv/4+OhOV2JUOacGHb7B47hwmXHMTV931yL8a0qiuqmL9koWs/mV+TY8PiPAyMCHWnV7BgSQGuuLRisvRhRC46bW46bWEep54hk652UZ2mZms2q+M0iIy9vzJnnUr6srRCiEIiogmKiGR6M7diO7SjejO3fEPCWt3wzot5RsYjN1mo7y6eTdFjToN1/YK4NX12axPL2NYVOMFxdoDldiVDsdirmbbmqWMvOBSrr33iWYntMLcbBZ9NYcVC+ZSWV5KuKeBa3oGMDTKkwgvx1da9DBoSfB3JcG/fo/cZpdkl5tJLa4mraSa1OJ8UjcvZcvyRci6czVE9RpMTNfuxHTtSWzXHoRGxzl0HLyt2Ww1f+H8m1mZw6O8+GZPAd/uLWBopGe7/TBUiV3pcPQGI4+9/w06vaFZv5hFecf48aO3WPH959glDI30ZOKgKLoFuraLX2ytRhDhZSTCy8jxe2VVWeykllSTVGQiuchEctI2lm5fj9lWk+5ddIKo7v2J7tKd2MSexCX2IjwuocNscp6VcgRXdw88/8VfV1qN4NJu/qzLKKPCYm/Vv8xOp47xHVQUIDstme/fe4UbH3keN4+ml4hXV1Xx6xfv8/OHr2K1S86N9+GSRD+CPTrGgiVXvYauAa50Dfi7h2+zSzJKzRwtNHGkyERy1l7W7N3GH7V3bw1aQVRin9piYj2J6dqDyE5d2l1teIu5mq0rf6f7wGEIkfqvzh0T582YOO82iuz0UIld6RBMVZW8dt9NFBfkUVFa3GRi37V+FXP+7xHyMtMYGunJ9X0CCWmjhC6lRAIaIZBSUlpdM0Sg1Qi0QmDQipMuomlNWo0g2sdItI+RMdQkL7uUZJXVJPujhSaO5h5k/cGdLJtfk+x1GojyNhI18iKiu3QnunM3ojp3a9aHp6PMe2sWxfm5XMg+4NRmuORWWPA2ak9aTMxZqcSutHtSSj569kEyk4/w0OwvTlpMqqqinC9ffZoVC+YS7mng2TGR9Aw+9altdinJKbdgtUsivY1IKXnpzywKqqyUmKyUm+1UWmyc18mHWwaEIIHrFhxpcJ2Luvrx375BVFvtPLwsDU+DFh8XLd5GLT6uOnoGuZHg74pdSmx22aqLaTTi76Gcs2P+TvY55RaSimqTfZGJ7Yvns3LhN3XnhXjoiTzrHCITEolKSCSyUxeCI6IdOm5vtZiZ/96rLPrqIyYm+ND7FKctHsyvYsYfqTw6MpyB4c77AXYiKrEr7d6KBXNZt+hHLr/tAXoOHnnCdikH9/LGA9PIzUzj4kQ/ru4ZcEoJcmNGGXtzKzlUYCK52ITJKhkU7sEjIyMQQlBksmLQCjr5ueBh1OKu19IloGZut4C6euB2KbHYJRabpEvtDVGLXeJj1FJqtpFVZqbEZKXaJrmhTyAJ/q7kVli45eck/Fx1BHvoCfHQE+phYHCkJ1HerTdcohGCUE8DoZ6GutkhUkoKq6wkF1WTXGwiqaiapKMH2bJySd0+onqjkbCYTkTGdyEsthNhMfF1W/39taNUW7BazGxd9Qfz33uVzKRDjJ5yFTcZtp3y9WJ9jeg0sD+vSiV2RTndrBYzCz95mx6DRnDhSeqOrFz4DZ88/yheWhvPjYls9rZoUkqSi6tJLjLVrUj8YX8hRwtNxPu5cE6cDzE+Rjr5/b0o5/lzo094PSEE53f2PeHzHgYtT4yq/xdHpeXv+etGrYYreviTW2HhWLmF7ccqWV5VSpiXgShvI/vzKnl5XRYRXkYivWuORXkbifUxtnhIQQiBv5sefzc9A8I96o5XWxNIK/lrRk41aSUpHFh5kLW/Weud6xsUQnBENEHhUQSGRxIQEo5vUAi+AUF4+QXi4e3T7MJileVlHEtLJu3QPvZt3cCOP5dTXlxEYHgUj4+MYIDLdmo+Rk+NQash3NNIekn1KV/DkVRiV9o1nd7A05/+CNBoUrDbbHz9xnP89uWH9A52476hkXi7NP1jn1dhYXlyCStTSsgqs2DQCoZHeWHUabh3SChHC6spM9vwcdHSL9S9TeuMHF961tdVx1U969e5MVntdSnMqNPQI8iN9JJqlhyprJsBM+vcKBID3diXV8m27ApifYzE+LgQ4qFv8fi+UadpdBpmlcVOVpmZzFIzWeVmjpVVkJ26k137tlBUZW1wHaHR4O7phZunFy6u7hiMLugMNTObbDYbZpOJyvISyoqLqCovqzvPw8eXft42RvWOoE+Ia6vdr/Bz1VFssjXd0AmpxK60W/u3rqdLn4EnLOhlNlUx+9E72bJiCeOu/C/T5Lpm/dIvTy7hzQ3ZSKBnkBtTuvozKMIDo07D70eL+WpXHuGeBsK8DGSXmXl70zGu7hXA+E4n7om3JZfjeuJxvi7cMyQMqJkBk1dpIbW4mljfmr8ojhaa+H5fAbXrmTBqBTE+Rh4/OxJPo5bSahuuOtEqH1Sueg3xfi7E+zUsMWC22SmotFJYZa27H1FSbaPcbKPCXIypshBzmcRqr92+Tgi8tDULubzCdAS4BRLsoa/7cGrPq0TbQosSuxDiJeACwAwcBf4rpSxuhbgU5aQO7dzCs9Ou5PLbHmi09GtlWSkv3XUDh3Zu4eb+QUwS6+EEv/xSSjZnleNp0JIY6EbPIDcu6ebPefHe9aY+LjlSzA/7C7i8uz9r08rYkF5GuJeRK3sEsPBAITY7Jx1mOd20GkGIh6HebJ8LuvgxrpMPaSVmUopNJBdVk1lqxsNQk8g/3Z7LypQSIr2NxPm60Mmv5qtLQOuOjxu0mroxfGc1qYsvFtvpL5LYGlraY/8DeFhKaRVCvAA8DDzY8rAU5cTM1SY+eOoB/INCGXv59Q2eLy8tZtbt15K2fyf3Dw1j+EmWhu/Lq+ST7bkcKjAxKsaLxEA3At31/Kd3/b8Cqq12vtiZx1U9/fl+XyE39w+mW6ArB/KreH9rDpck+jN3dx7nxHnX60E7I4NWU5ew/2lkjBe+rjqSikxszSpneXIJYZ4G3p0UB8CC/QVohSDez4VYX+NJdyhq7waEeTTdyEm1KLFLKX8/7uEG4NKWhaMoTVvw4RtkpRzhobe/bDCXuqKshOdvvZr0g3t4cPiJp6oVVFr4eHsua9PK8HfVccfAEEbHnnhRyqbMcjr5ubAsuZTbBobU/dIPivDEqNXwyY5cugS4siG9jFEnuY6z6xPiTp/aKYJSSgqqrPXGw1enlpJUVHNDUQChngZGx3hxeY+abfQqLSffjq69+PVQET2D3Vp1ptHp1Jpj7DcC3zTZSlFaIO3Qfn7+7F1GXnApvYacXe85U2UFL02/gbTDB3hkeHi9mRv/tCq1lI0Z5VzVI4ApiX5NzhgpqLQQ4W1g0eEiegXXn1HTM9iN1OJq+gS7kd/ITcH2SghBgJu+3l6gr42PpajKWrdyNanQVDfCZbHZ+c8Ph/F10RHnW9Ojj/V1oYu/K77taGeiP9NK+XBrDud39uXm2qmp7U2T/9tCiKVASCNPPSqlXFjb5lHACnx1kutMBaYCBISEn1KwimK1muncewDX3PNE/eMWC68/MI0ju7YwY1jjSb2g0kJepZWuAa5c2MWPoZGezV5t6uWiY39+FSEeBg4XmOpNlzxSaCLEQ09epZWIdtrD+zd8XXUMCPdo8H9stcPVPQNJKjKRUlzNpsxyJHB970Au7uZPcZWVeXvzifY2EuNjJNLb6FS1WGx2ya+HivhkRy6Jga4NhuPakxbvoCSEuAGYBpwjpaxszjlqByWlNUkpeW/mvaz5ZT53DAxhbLxPgzabM8t5Y0M2bnoN706K+9dT4irMNm7+6SiXd/fnj6QS7hkSRic/F5IKTby6Posxcd7M31vABxfGO1WyciST1U5qcTV+rjoC3fUczK9i5op0qo7bVcTfTcfdg0PpFexOQaWFzDIz4Z4G/Fx1p7UAW6XFxmPL0zlaaOKsMHfuHRrmlENKp2UHJSHEeGAGcHZzk7qinIrSokJ++/IDLrzhNtw8698MXfDRG6z5ZT5X9QxokNRtdsm8Pfl8u7eAWB8j9w8LO6V5zu4GLZO7+rE6rYxzY72ZtSaDkmobnkYt5yf4si6tlPM7+6qkfhwXnabebJouAa7MvTSB/EorqcXVpJRUk15Sk/gBtmVXMHvTsdpzBaEeBoI99NzUL5hAdz15FRZKqm34u+rwMmpPeb66zV5TG+dwoYkqi53zO/viqtMQ4KZjcpdQRkZ7tYuqnifT0oGv2YAR+KP2P2KDlPKWFkelKP/wzewXWPXTNwyfOKVeYt/wxy/Mf/cVRsV4cUV3/3rnVFvtvLQui82Z5Zwb583U/sEtWn15eXd/bFLy3b4CBoS5E+RuIK/CzPf7CpiQ4MtVPQNO+dpnCiEEgbW7Tf1zKGdQhCeB7vqajURKzWSXm8koNaPX1iTZZUklzN2TD/y9BaG3UcessVG46bWsTi1lT04lOg1o/kr6Em6qHSf/alcea1JLya2w8NcsxiB3PRMTfBBC8MiIiNPzn3AatHRWTKfWCkRRTiRp305W/jiX8VfdSER8l7rjaYf2897Me+jcewB3dClt0MvSawUuWsHU/sGtMr9cCMHVPQOZ1NmXNallFFZZSQhw5cZ+wc1azaqcnJdRW29Wzj+NjvUixtdIQaWVYlPNV0m1DWPtYqr0kmo2ZpZhsdVU05SypgzxX4ndy6glzs+FYVFehHsa6OTvQrhn82r2tzctHmM/FWqMXWkuKSVP3Xgxx9JTeHXBqrreenlpMY9dez6W6mpeG+ZZb9ZFRmk1Rq2GQHc9UsoO+YurnJmaO8bu3CsplDPeht9/5tDOLVxx+4y6pC6l5L0n7qHgWDZ3vfh+vaSeUmTikaVpvL4hC6DNkrpdSiotNuwO6BgpSlPU34+KU4vu0p1xV93I2RdeXnfsty8/YNvqpdzUL4i+vz9Ud/xooYknVqRh1Gm4fWBom8QjpWTxkWK+21dAhdmGu17LJd3868ZpFcUZqMSuOLWwmHiuf+CpusdH9+5g3luzOGv0eCYFJ9cdTykyMXNFGm56Dc+OiWqz7e2WJZXwy6EiHhsRQZyfC8lFJl5bn41Wg8OKgCnKP6mhGMUplRYV8s7jd5OXlV53zFRZwexH7sTXAHf7Ha3rIUsp+WhbLgadhmfaMKkDLDhQyO0DQ4irrbMS6+vCHYNCWLC/sM1eU1H+LZXYFae04MPXWbf4R8wmU92xz19+ktyMVO4ZElZvvrgQggeHh/PcmKg227f0L8fKzQ2KZ8X7upBTbuHfTESottpZnVLKL4cKSSk2NX2CovwLKrErTicnI5Wl879g1OQrCY9LAGD7mmWs/HEeF3f1pUftcv5Ki40vduZhttnxNGpPSwnYGB8Xdh6rvxZvZ04FMT7GZo+xHy00Me2XJFamlJBWYuaplRm8s+mYuhGrtBo1xq44ne/efRmtTscl0+4GoKy4iA+fmUG0t7FuEZDVLnlxbRa7cioYEOZOYmDztrprqSt6+PPO5mPcbA+iW4Ab+/Or+HBrTrOLRdml5OV1WfyvbxAjomtm+VT1sfPo8jTWppYxMubEJYYVpblUYlecSsrBvaxb9CMX/vd2fANras998fKTlBXkMnNcTN3OPh9ty2H7sQruGBhy2pI6wMBwT3RCMH9fAe9uziHCy8BtZ4WctJLk8ZKLqhHA8Ki/ywm76jVc1MWPNWmlKrErrUIldsWp+AYGc8H1t3LBDbcCsH3tctb+9gNX9PAnrnZ7t8VHilh0uJiLuvo1WvCrrfUL86DfKW7CYLVLdBrR6CpZq10NxSitQ42xK07F2y+Aq+56BHdPbyrLy5jz3ENExHfmsm41dWAqLTa+3pVPv1B3rmuHZVU7+blQbrGxK6ei7thf5WIHRzS+KYii/Fuqx644BSklX7z8FGeNGU9i/8FAzVh7UU42D/UyoNfWVAl002t5YWw0ni2o7udIWo1g+qBQXlybxZBIDwLd9axLK8PfTceYdrzzkuJcVI9dcQp7N/3J4rlzSD20D6hZiPT7vE+YkOBDlwBXbHbJ2rRSpJSEehradXncPiHuvDEhhlBPA1UWO//pHcijIyPqqhgqSkupHrvicFJK5r//Kn5BIYy5+GrsNhtznnsYb/9Aru1V04v9bm8Bc/fk4zEqkj6hjVf/a0/83fRcnOjfdENFOQWqx6443O4Nqzm0YzOTb7wDg9GFpfO/IOXAHm7uosPdoGV3TgXz9uQzKsarQyR1RWlrKrErDiWlZP67rxAQEs6oi66kpDCfb995mV7BbgyL9KTcbOO1DdmEeuq5ZUBjW+8qivJPaihGcSi7zcaQcRfiHRCI3mBk3puPYK4oYdrIWIQQvLv5GMVVVl4YG42rvu36ISUmKytTSsmvtJDg78qQCE815q20WyqxKw6l1emYcM1NQM0N01U/fcuUrn5EeBkBODfOh+5BbiT4u57sMi1yuKCKZ1ZlMCDMg0hvA4sPF/HTgUKeHhPplBsaK0pTVGJXHObAto0cS09hxPmXIDQaPnvhCbz9A7m8hw92KdEIQd9Qd/rSduPqUkre3nSMm/oHM7J2if9FXf14bX02C/YXck2v9jdXXlHUGLviEFJKvpn9AvPfewW73ca6RT9yZM92ru+kxVWn4f/WZPLt3vw2jyO/0kphlbXeEn8hBOd39mVzZnmbv76itAWV2BWH2L91Awd3bOaC62/DbrPzzdsvEO/nwuhYb1Ykl7I5sxxXXdv/eOo0ApuU2Oz1j1fb7Oja4QIoRQGV2BUHWfDhG/gEBDH6oiv49cv3KThWU/GwxGRjzvYcEgNcOb9z2+9I5OuqI8bHhV8O/b1RhsUm+X5fQV31RUVpb9QYu3LaHdq5hb2b/+Saex+nsryMnz99l4HnTKR7UBKz1mZSbZXcMSgEzWnaQ/TOQSE8tTKdTZnlRHoZ2X6snAS/0/PBoihtQSV25bSzWa10GzCUcy65lq9efQarqZIbXQ+QWQqbMsq4qmdA3ayY0yHEw8BbE+PYll1OQaWV8zr5NNglSVHaE5XYldMusf9gHvvgGzKTDrPix7lMSPCt2/3ojQmxbb69XWN0GsHAcFVdUekY1Bi7clptWvYbFWUlAMx763mMrm5c2cOf7DIzAJHeRrUwSFFaSCV25bTJTk3ijRm38Ovn73Nwx2a2rvqDi+OMFFZaue3XJJYcKXZ0iIrSIajErpw2v3z+Hjq9gfOuuIF5bz6PT0AQF3T25b0tObjrtQyJVEMhitIaWpTYhRDPCCF2CSF2CCF+F0KEtVZgSsdSmJvNml++Z9TkK0jev5uDOzZzZYxgQ0Y5+/OruL5PIF5GtXxfUVpDS3vsL0kpe0kp+wC/AE+0PCSlI/rtyw+x221M/M9Uvn37RUI89AyL8uSzHbkk+LtwTpzaPUhRWkuLEruUsvS4h+6A2o1XaUBKSf6xLIaOm4xWq8NUWcFVPQPIKDVjtcO0/sGnbc66opwJhJQty8VCiOeA64ASYLSUMu8E7aYCU2sf9gD2tOiFnVsA0PaFThynI7+/jvzeQL2/9q6LlLLJm1FNJnYhxFKgsR0OHpVSLjyu3cOAi5RyZpMvKsQWKeWAptq1V+r9tV8d+b2Ben/tXXPfX5MLlKSU5zbzNb8CfgOaTOyKoihK22nprJiE4x5OBg60LBxFURSlpVpaUmCWEKILYAdSgVuaed4HLXxdZ6feX/vVkd8bqPfX3jXr/bX45qmiKIriXNTKU0VRlA5GJXZFUZQOxmGJvSOXIxBCvCSEOFD7/hYIIXwcHVNrEkJcJoTYK4SwCyE6zNQyIcR4IcRBIcQRIcRDjo6nNQkhPhZC5AohOuT6ESFEpBBihRBiX+3P5l2Ojqm1CCFchBCbhBA7a9/bU02e46gxdiGE118rV4UQ04FuUsrm3nx1akKI84DlUkqrEOIFACnlgw4Oq9UIIRKpuWH+PnC/lHKLg0NqMSGEFjgEjAUygM3AVVLKfQ4NrJUIIUYC5cDnUsoejo6ntQkhQoFQKeU2IYQnsBW4qCN8/4QQAnCXUpYLIfTAWuAuKeWGE53jsB57Ry5HIKX8XUpprX24AYhwZDytTUq5X0p50NFxtLKBwBEpZZKU0gzMo2YKb4cgpVwNFDbZsJ2SUmZLKbfV/rsM2A+EOzaq1iFrlNc+1Nd+nTRfOnSMXQjxnBAiHbiGjltA7EZgkaODUJoUDqQf9ziDDpIYzjRCiBigL7DRwaG0GiGEVgixA8gF/pBSnvS9tWliF0IsFULsaeRrMoCU8lEpZSQ1q1bvaMtYWltT7622zaOAlZr316405/0pirMRQngA3wN3/2NUoF2TUtpqq+hGAAOFECcdTmvTPU87cjmCpt6bEOIGYBJwjmyHiwX+xfeuo8gEIo97HFF7TGknasefvwe+klL+4Oh42oKUslgIsQIYz0kKKTpyVkyHLUcghBgPzAAulFJWOjoepVk2AwlCiFghhAG4EvjJwTEpzVR7g3EOsF9K+aqj42lNQojAv2bWCSFcqbnBf9J86chZMd8D9coRSCk7RA9JCHEEMAIFtYc2dJQZPwBCiCnAW0AgUAzskFKOc2hQrUAIMRF4HdACH0spn3NsRK1HCDEXGEVNWdscYKaUco5Dg2pFQojhwBpgNzU5BeARKeVvjouqdQghegGfUfNzqQG+lVI+fdJz2uEogaIoinISauWpoihKB6MSu6IoSgejEruiKEoHoxK7oihKB6MSu6IoSgejEruiKEoHoxK7oihKB/P/Lq0u+4sAWS4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# https://scikit-learn.org/stable/auto_examples/svm/plot_svm_kernels.html\n",
    "clf = svm.SVC(kernel='poly', degree=3, gamma=2)\n",
    "clf.fit(X, Y)\n",
    "\n",
    "# plot the line, the points, and the nearest vectors to the plane\n",
    "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=80, facecolors='none', zorder=10, edgecolors='k')\n",
    "plt.scatter(X[:, 0], X[:, 1], c=Y, zorder=10, cmap=plt.cm.Paired, edgecolors='k')\n",
    "XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]\n",
    "Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()])\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(XX.shape)\n",
    "plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.Paired)\n",
    "plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'],levels=[-.5, 0, .5])\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Radial Basis Function Kernel\n",
    "\n",
    "Another example is the Radial Basis Function (RBF; sometimes called Gaussian) kernel\n",
    "$$K(x,z) = \\exp \\left(-\\frac{||x - z||^2}{2\\sigma^2}\\right),$$\n",
    "where $\\sigma$ is a hyper-parameter. It's easiest to understand this kernel by viewing it as a similarity measure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can show that this kernel corresponds to an *infinite-dimensional* feature map and the limit of the polynomial kernel as $p \\to \\infty$. \n",
    "\n",
    "To see why that's intuitively the case, consider the Taylor expansion\n",
    "\n",
    "$$\n",
    "\\exp \\left(-\\frac{||x - z||^2}{2\\sigma^2}\\right) \\approx 1 - \\frac{||x - z||^2}{2\\sigma^2} + \\frac{||x - z||^4}{2! \\cdot 4\\sigma^4} - \\frac{||x - z||^6}{3! \\cdot 8\\sigma^6} + \\ldots\n",
    "$$\n",
    "\n",
    "Each term on the right hand side can be expanded into a polynomial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can look at the `sklearn` implementation again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-3.0, 3.0)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy86wFpkAAAACXBIWXMAAAsTAAALEwEAmpwYAABfaklEQVR4nO3ddXRUV9fA4d8ZycxEJu6ekADBHYpr0QJV2lJvKW/djXrfvnV36qWUFi3Fi1ux4m4hCQlJiHsydr8/AnxUkZnMHbnPWl2rSWbO3UMy+545so+QJAmFQqFQeA6V3AEoFAqFwrGUxK5QKBQeRknsCoVC4WGUxK5QKBQeRknsCoVC4WGUxK5QKBQexu7ELoTQCyE2CyF2CiH2CiFecERgCoVCobg4wt517EIIAfhJklQthNAC64D7JUna6IgAFQqFQnFhNPY2IDXeGapPfak99Z+y60mhUChkYndiBxBCqIGtQDPgI0mSNv3NYyYAEwB0Bt9OMUmpjri0e5Kgvq6G2uoq6mqqMNXXo1KpSWyeIXdk5y0v8zCmhnq0PjoM/gH4+hsx+PqBkDsyx6upqqCmsoIoTYPcoTQJi1Wixmylzmyj3mLDKoFKQHKQDkTT/ULzKhuotzT2AXVqgU6jwlerws9H3WTXdHdHS+uLJUkKP9fj7B6K+UNjQgQBc4B7JUna80+PS8loK708daHDrusOLGYTKrUGlUrFjx+8yi9ff4RaoyGtbSfadO9Ny049SG/XGZXKPeazM/ftYu+W39izaQ37t25CCMFnK3ai9/Wj6EQuweERaLQ+cofpEAumfMbUd/7L95enEaBz/6QjSRJZ5Q2E+2nx91Gz6HAZn/5eSISfllbhBpqHGWgWoqdZiB7RpIndxLHyejJL6zlSWs+hknpaRRh4pm88ANP3FJMYpKN9lB86jXu8L5ra6GkHtkqS1Plcj3NIj/00SZLKhRArgaHAPyZ2byFJEsf272L13OlsWDqPB9/4jJadetBrxOUkt2xDm+598PUPkDvMi5KS0ZaUjLaMumki9XW15Bzaj97XD4D3Hp/IybwcLrl0NP3GjCOpeSuZo7VPVEIKACeqTDTXGWSO5uLVmq2sPFbJ0qPlHCtv4O4uUQxpFkTvRCPto/yIDnDujTjW6EOs0YdeCUYArLbGTw4ANSYrs/aXUm+x4aMWtI/y45L4ALrF+eOrdf+ba1OzO7ELIcIB86mkbgAGA6/ZHZkbM5saWLdgNktnfEfWgT1odTo69R2Cb0AgAHEp6cSlpMscpePoDb6kt+sENN7MLr/jAdYtnM3KOdP49advSGnVjivufIgOvQbIHOnFOT1smFdponmY+yV2i03iux0n+fVoBXUWG6kheu7sHEn3eH8A/H3U+LvA8IdaJTDqGlOSn4+a7y9PY29RLZvzqtl4vIrNedVM7BzJsLRgzFYJlWh8juKvHNFjjwa+PTXOrgKmS5I03wHtuh2b1YpKrUaSJH768DUCQ8O55Yn/csmwMfidSuqeTghBxz6D6NhnENWV5aydP4vlM7+noqQIgIa6OqwWM74BRpkjPX/hMfGoNVpyK91rjN1ik9CoBGoBR8vq6RLrz2XNg0kLdY+bk/ZUT719lB+3d4zgYHEd8YE6AJYfK2fm3hIubRbE4NQggvQOHXxwew4dYz9fnjbGXlZUyLxvPmHvlnX874fFqDUaSgrzCYmIatIxSndhs9mQbDbUGg0Lv/+c2ZPfZeh1tzH8+tvdJsE/esUA4kz5PNUnTu5QzskmSSw5Us6MfSW8MTiRUF8tVpvkUb3bXQU1zNxXws7CWjQqQb8kI2NahJxJ/J5KljF2b1NbVckv33zM4mlfYjGb6TX8curravALCCQ0Mlru8FyGSqWCU5PCrbr25MD2Tcye/A6/Tv+GMbfey5BrbnL5idaY5GYc35ItdxjnlF9l4t2N+RworqN1hC8WW2PHzZOSOkDbKD/aRvmRW9nA/INlLD9WQXZFA28OSZI7NJegJPaLdCLrKC/cejlV5aVcMmwMV975EFEJyXKH5fIS0zN46K0vyNy3i58+fJXv336RI3u2c9+rH8sd2r+KTUlj68pFmK02tGrXXKGxIrOCT38vQKMS3N89mv5JRo//xBhn1DGxSxTXtQ2jrK5x4rWywcpX2wq5pnWY0yeEXYWS2C9QfV0teoMvUfFJdB88kr6jryElo63cYbmdlIy2PPnxD+xYv5KAoGCgcb24qb6O4PAomaP7q7iUNGwS5FWZSArSyx3O39pVWENaqIEHe0QT5quVOxynMuo0ZyZeM0vr2ZBbxZrsSoanBTOuTZhLTA47k2t2PVyQxWxm5idv8dDo3lSUFqNSq7nlyZeVpG6n9j37k9qqPQA/vv8qj14xkDXzZuBqRzbGnlrFdLzCJHMkf1RrtnKyxgzAXV2jeLF/vNcl9T9rH+3HpyNTGZQSxILDZfxnfibLMstd7m+qKSmJ/Tzk5xzj+VvGMPvzd2ndtRdqtXfd/Z1l+A0TSEhrwafPPcR7j02kqrxM7pDOiE5MQSUgp8J1VsZUm6w8s+I4L646jtUm4aNWedxY+sUKNmi4q2sUb12aRGyAD5tyqz1+WOpsylDMOWxatoDJLzyCSq3m/tc/pdugEXKH5LGiE5J5+vMZLJgymekfvc7RvTt46K0vSG7ZRu7Q8NHpifb3cZnEXmu28vyq42SVN/BEr1glof+DlGA9/xuUQL3FBkBBtYmtJ2oYlhaEyoMTvdJj/xeSJLHh13nEpqTzyrQlSlJ3ApVKxaibJvLCN3MIDAnDPzBI7pDOiOkykCzVOct0NDmLTeLVtXlkltbzeK8YusT6yx2SS1MJcWa36tKjFUzeWsiLq3Ipr7PIHFnTURL73zDV11FefBIhBBNfeJtnv5hBWHSs3GF5lZSMdrw0ZT7hMfHYbDZW/fwjVou8b8T41OYU5mZjstpkjWP63mJ2FtZyd9cousa6Z0kKuYxvG8bEzpHsLarlgcXH2HOyVu6QmoSS2P+ktqqSV+8ez2v33IDVYkFv8HX5Ndae6vSY6J5Na5n84qO888gdNNTVyRZPXGo6ks1GrswTqKPSQ7i3WxQDU4JkjcMdCSEYlhbMG0MS8dWqeWZFDr+fqD73E92MktjPUlNVwf/+cy2Hd2/nslvuRq1RpiBcQdsefbnlif+yfe1yXrnrOmqrKmWJI65ZcwCyZRpnL6+3YLFJBOjUDFKSul2SgvS8eWkilzUPoVW4r9zhOJyS2E+pq6nm1bvHk31oPw++OZkel14md0iKswy++ibue+0Tju7dycsTx1Fd4fwVM1Hxyag1WlkmUG2SxOvr8nhh1XGvWrbXlHy1am7pEIFBq6LObGPy1kJqT1WXdHdKYj/luzee49j+3dz/+id07DNI7nAUf6PboBE89NbnFB7PJufwAadfX6PVEpOUKktiX3Gsgr1FdfRJ9PzdpHI4WFzH4sNlPL08h4p6959UVRL7KePufYIH3viMzv0ulTsUxb/o0Hsg7y34jYzOPQCc3nuNS013emKvNVv5bkcRzcMMDEzxjiqhztY+2o+n+sRxvNLEU8tzKHXzFTNen9gPbN+M1WIhMDRcSepu4nQJ5FVzf+KDJ+7CZnXex+e41HRO1lic+pH95wOlVDRYuaNjhEevvZZb5xh/nusXT3GtmaeX51DmxsndqxN75r5d/G/itcz+/F25Q1FchLrqKjYunc+09//ntGuePiAlt9I5K2MkSWJbfg094vzdpo66O2sd4cuzfeMRAuos8i5rtYfXLvsw1dfx0dP3YQwOYei1t8kdjuIiDLv+dgpzs1kwZTIJaRn0HnlFk18zLrVxZUxORQPpTki0QgheG5RIrdl9k4y7aRXhy/vDklGrBJIkYZVA42Y7e722xz7z07fJzzrKnS+8faa6oML9jH/oWTI69+CLlx8n6+DeJr9eRGwCWh8dmemjm/xaVptEg8WGWiU84hBtd3I6qX+2tZA31+dhc7OVSF6Z2HMO72fh1M/pP/Za2nTrLXc4CjtotFruffVjAoJC2P/7hia/nlqjISohmbxjh5v8Wpvyqrht7hG3O5LPUwghiPb3YUNuNVN3FcsdzgXxyqEYs6mBlp16MO7eJ+UOReEAgSFhvD5jOb7+ztleH5vcjKP7dkF8025sWZ5ZgY9aRbS/svNZLpc1Dya3soGZ+0pIDdFxSbx7HOXolT321FbtmfTpNGUIxoOcTuoHtm1i98Y1TXqt2JR0ik8cp6EJJ9cqGyxsy6+hb5JRqdwoIyEEEzpF0jxUz3sbC8hz0qS5vbwusS+fNZXKshK5w1A0AZvNxndvPs8nzz5ITVVFk10nJikVSZI4UdV0b/KNudXYJOid4B49RE+mVat4tGcsgTo1+dVKYnc5R3Zv58uXn2D9wjlyh6JoAiqVituffo2K0mKmvfdKk10nNjkNaNoljxtzq4j005IcrGuyayjOX7iflo9HptA5xj1KJHvVGPvSGd9h8POn35hxcodil7qaapbN/J5VP0+jMDcb3wAjPYZcxrDrbvP6A7VTMtoy9NpbWfzDl/QfO+7MsXuOFJWQhBCCvCbssV/RMpRqk9UlygfYJIm12ZUsOlzO4dJ61AI6RPs1FtCK8LwCWv9Ec2qlzOIj5SQH62kR5rr7Crymx15XU82mZfO5ZOgYDH7ucdf9O9WV5bx4+5Uc2bOdAZdfz6ib/sOI8RPQGQw8f8tYDu7YIneIsrvizocwhoQx5c0XmqTkgI/eQGhkTJOOt7aK8KVbnPy11m2SxPsb85l7sIzucf5c3jKEy1uG0ixEz1sbTrDgkOscX+gMDVaJWftK+GBTPmaZ6/L/G69J7FtXL8VUX0+vEZfLHYpdvnv9OZq16UBVWQm/Lf4ZtUZL1sG9rFswmysmNp4VajZ59/I4X/8Axj/0DF0HDkeyNc2bLyoxhfwm6rHvKKhhd2FNk7R9oZYdrSCvsoEWoXp+OVSGxSZRXm9h7oFSxrYIYfreYo6V1csdptPoNSru6hJFbqWJOftL5Q7nH3nNUEzh8SzComJJa9tJ7lAuWkVJEdvWLmfQVeMxBody3+TpqFSN9+ZVP//I8tlTiW/WnE3LFtJr+FiZo5VXz2FN+/qjEpLYsG09kiQ5fLjkpz3FmG0Sbw7xc2i7F0qSJOYfKmNAspFFR8p5f1gy/j6NG6WGpwfzxNJshqUFsehwOXd1jZI1VmfqGONPz/gAZuwroV9yIBF+WrlD+guv6bFfceeDvD13zZlE6I4y9+2iWev27Fy/iuHj7/jDa+kz6ipO5ubQokM3Dm7fLGOUrsNmtbL6l+n8vmqJw9uOik+mxmyjyuTYTwQWm8SR0npausD4bZ3FRkG1iaJaM4NTg84kdYCEQB2tI3zx1ajYV+SZx8v9m1s6RADw7Y6TMkfy9+zusQsh4oHvgEhAAiZLkvSeve02Bbc/4k4IJMmGEKq/VDSUJAlJsiEhgfzzbS5BqFQs/uFLTA31dOw9CJXacdvyI+MTAcivMmHUOS4J51Q0YLJKpIXIn9hPE4i/3VJvlaTGn3rh31u4n5bbO0YS7uuagx6O6L5agIclScoAugN3CyEyHNCuw6ya+xPP33o5tdVVcodil2atO3B0z0469BrAvG8//cPhzstnfU9Mchr7tvxGRqceMkbpOoQQjL7tXvKzMx3ea4+Ma0zsBQ5e15xZ2jhe3SxU79B2L4avVk2c0YdwXw1Lj5ZT2fD/f29HS+s5UFxHlcniVStjznZpsyA6uujyR7tvN5Ik5QP5p/6/SgixH4gF9tnbtqMc3buD3KOH3Ho1DEBAUDBdBw6juPAENpuVx68eTPte/cnLPExe5mFG3DiBed9+SpcBQ+UO1WV0HTCM8NgEFk79gq4Dhzus3fCYBAAKa8wOaxMgq7wBvUYQ5e8a47Yj00OYd6iUXglG7llwjEviA6g129iaX831bcP5cXcx/x2YIHeYsqk1W5m5t4SucQEutfzRoQPOQogkoAOwyZHt2is/6yixyc1cYk2wvcY//Bz5WUexWa30HzMOva8fGZ170LHPIH7+8kMeeutz9x9yciCVWs2l19zMoR1byDqwx2Ht6gwGAkPDKax2bGK/qX0E7w5NdpkDNfonG2keamBLXjWjW4QQpG/sxQ9PC2b6nmJubB9OQqD3bqJSCcHSzAqm73GtImEOGyASQvgDs4AHJEn6yzHyQogJwASAsKhYR132vBTl55HWpoNTr9lUfP0DePqzH1kzfyYrZk+j6EQOvv5Gelx6Gf+dMp/QqBi5Q3Q5fUdfze8rF1Nf69glhOEx8Zw86dizV7VqQXSA69yYhRDc2TmSzXnVLDpczpHSetQq6BDlx6Q+cV5/+Ideo2JUejBTdxeTXd5AYpBr3OQcktiFEFoak/pUSZJm/91jJEmaDEwGSMlo69TixhUlJwkKi3DmJZuUj97AoCtvYNCVN8gdilvwCwjk2S9nObzdsOhYso7tdlh7DRYbU3YW0TfJ6FIJUwhBt7gAl9gw5YqGpgUzY18J8w6Vck/XaLnDARwwFCMaxze+BPZLkvS2/SE5ltVioWWnHiQ1by13KAqZVVeWU5BzzGHthUfHUVxrcdghDIU1ZuYdKiO/yrHDO4qmZdSp6ZdkZHVWJdUm552F+28c0WPvCdwA7BZC7Dj1vackSVrogLbtptZoePyD7+QOQyEzSZJ47qbRRMQm8PiHUxzSZkhkDBabREW9lWCD/W+lk6fG6yNcZOJUcf6GpQVTZ7ZRZ7b9Yb2/XByxKmYdyspphYsTQtC536UsnPo5VeVlDqnFHxrV+LG7uNbskMReVNuY2F11bbTin6UE63mkp3PnDv+N+27DPE+11VXcO7wby2dNlTsUhcy6DR6B1WJh25qlDmkvNLJxorqkznKOR56f0joLKgFBeiWxu6ucigaKa+UfSvP4xO6j01FScIIq5XANr5fcsi0hkdFsW+2YxB4S0VgfpaTWMYm9xmQlUK9RTkxyU5UNVu5fdIzFR8rlDsXzE7tG64NOb6C6slzuUBQyE0LQoddAdm9ai8Vs/47RgOBQ1KKxp+0IEzpH8cVlqQ5pS+F8Rp2a1hG+/JYj/w53j0/sAMbQcMpLiuQOQ+ECRt40kZenLkStsX+CUqVSEWzQUFrnuI/eGqW37tZ6xAeQV2Uit1Le0tlekdhDI6Mpyc+TOwyFC4iMSyQ6McVhu5AbE7tjlrh9ua2QZZnlDmlLIY8up2rH/J5XLWscXpHYO/QeRIuO3eQOQ+Eitq9dztyvPnRIW8bWvSnxj3dIW6uyKjlc4j2HVniicD8tCYE+bC+Qt5SxV0y/j7ppotwhKFzI3i3rWTr9O4aPvwOtj31bwANDIzi8axsQbndctWYbvlqv6Gt5tEd7xhLqgOWv9vCavyKL2Ux9nfcdCKD4q/R2XTCbGsg6sNfutoJCw6kqL8Vqs2/3qdkqYbFJGDRe85b0WAmBOvxk3qTkFX9FdTXV3N43g19//FruUBQuoFmb9kBjOWd7BYaGIUkSlQ32jbM3nDoYWackdrcnSRLT9xazLucvtRCdxiv+igx+/oRGxXJAOTJOAYRERBMUFsGx/fYX8DKGhAFQXm/fkkeLTcKoUytDMR5ACMHKY5WsyZIvsXvFGDtARqce/Lb4Z6wWC2qN17xsxT9IatGaqnL7T5k3BocCUFFvX489SK9hyuVpdsejcA3NQvTsPSnf0K/XdA9ad+tFXU01R/ZslzsUhQt45J2veOz9b+1u50yPvcExm5QUniElWEdJneUPxwk6k1cldpVazfa1y+UOReECHHWwdWCIY3rspXUWXlmbK2svT+E4iadOlcqpcOyZuOfLaxK7X0AgtzzxMj0uvUzuUBQuoKyogDfuv4VdG1bb1Y6fMQi1RmP35Gm9xcbG3GqKHHyGqkIecYE6dGpBhZ1zLxfLqwabB15xvdwhKFyEwS+A7WuXkda2I2179L3odoQQBASFUF5v3xby05UE7Fw1qXAR4b4afrwqXbaza72mx37a3s3r2bJysdxhKGSm9/UjKCyCwuNZdrcVGBJGhZ09dvWpBGBx0GlMCnkJIWQ9kNzrEvvPX33I1HdeQlLeQF4vMi6Rwtxsu9sxhoTa/ZFbq25MAmar8nfpKWbvK+HT3wtkubbXJfbeI6/gZG4OB7ZtkjsUhcwi4hI5mZdjdzvG4DDK7Zw81WtUhPlq8FEr1R09xfHKBjbLVAzM6xJ714HDMfj5s/LnH+UORfEvLGYTO9evZM38mezdvB6b1fGHBCemZxARE4/NZrOrnaCwcMrrLXZ9CtRrVHw5uhmDU4PsikXhOgJ1GirtvOFfLK+aPAXQG3y5ZNgY1sybwY2PPId/oP1nXyoca/nM75k1+R0i4hIJj47jRNYRqivKue6BSXQbNMJh1xk+/g6Gj7/D7nYCQ8MxWSXqLDZ8tfIfZKxwDf46NWabRIPF5vRSEV7XYwcYfOWN+PobyTt2RO5QFH+y6IcvWTj1c0bfeg8qlYp9v/9GQFAII26YwJQ3n2fjr/PkDvEvgkIjAPtPUvpwcz4/7i52REgKF3C6PESd2b5PhBfDKxN7QnpLPli4iebtu8gdiuIstVWVzPrsHQZfcxPzv/uUodfeyovf/sIlw8Yw+/P3uOzWe/j+nZewWhyzNrjoRC5PXDOErat/taudoLDGxF5mZ2LPLm9gf7GyQclTBOs1xBl9sMqwUMPrhmJO02i1WC0WKstKCA6PlDscBbBp2QJadbmEpdO/466X3qVlpx4A9Bl5JRqNhhWzpxEaGcOujWvo0GuA3dcz+PmRc3g/BXYueQw+fai1nYk9WK+hoFrZoOQpesQH0CM+QJZre2WP/bT//edaPnzqHrnDUJxSXJBHdFIqxSdyadGx+x9+1rZHX7IO7iW+WQtKChxzzKGfMQitTkd50Um72gmNjAYgv8vN9rXjq6G4VknsCvt5dWLv2GcQ+7duJHPfLrlDUQD+gcFUFJ/EzxhIXuahP/ws68BeImLjKSk84bAJbyEEQWGRlBfbl9j1vn74BgTafa5uhJ+WGrONapM8KykUjrWjoIYnl2XLUibCqxN7/zHXYvDzZ8GUz+QORQF0HzyCLSuXMOjK8Xz6/MPk5xwDIOvgXr5+dRK9RlzO0T3bae+AYZjTgkLD7U7sAOHRsRTbmdgTAnWkh+qpaYLELkkS5fUWimrMdp/2pDg/pbUW9hXVYZHh39trx9gBfAOMDLj8ehZO/Zxr7nmciNgEuUPyasHhUfQecTn7tm6ifc/+vHDLWKxWCzq9gUvH3cq6BXMYeeNE9AZfh12zVdeeNDjgyMTw2HjyszIh/uKXO3aK8afTqVPuHWl1VgVzD5RSUG3GR6MCSWJwahBXZoQqJzY1odOnYsmx6czrf6vDrr8NlUrNml9myB2KAhj/0LPEJqXy60/f0HnAUC4ddyutu/Vi7tcf0aH3AEbdfJdDr3f1XY9yw8PP2d1O5KldrDYXK1UxdVcRM/aW0DfJSO9EI+0ifbmsRQi5FQ08u/I4DRbnL8XzFrWnljnKsbfBIYldCPGVEOKkEGKPI9pzppCIaF78di6X3/mg3KEoALVGwy1PvszLUxcRERNPfW01CWkZvDV7JVf95xGEjIWV/k1UfDJmUwPFtfatjHl3wwleXeeYyeHDJXUsz6ygb5KRuQfLiPTT0irCl0251TRYJYL0aubst/8UKcXfqzZZ0agEeo3z/2YdNRTzDfAh8J2D2nOq5JZtALDZbKhUXv8hxiWEx8Rx2S13N/l1tqxYxOcvPcZLU+YTGZd40e3EJKcCkFdpIsJPe9HtqFWCPSdqkCTJ7pvY4iPlDEwJZM7+Ut4dlnwmroHJgTy5PIfuEf7MO1TGVa1CUatc84bpzow6NRnhBlk6Iw7JYpIkrQHc+ta/YckvPDy2L/W1NXKHonAincGX6opyyk7aV4UvJrnxvNKcCvvqsjcPNVBlspJXZf/JO8fK6tFrBBnhhj/cbNQqwYBkI8crG7CdmlRVON7YlqG8NECeeTundU+FEBOEEL8LIX6vKnO9e0BoVCyFx7OU4mBeJuTUGvTSk/l2tRMYEoYxJIyscvsSe0aEAYA9DjgiT6MSCMTfnu5U2WBFp1Zhskpo1cqnVE/jtN+oJEmTJUnqLElS54DgEGdd9rylt+tE8/ZdWDT1C4dtWVe4vtDIGABKCuxL7NBYLfJYWb1dbcQG+BDqq2FHgf2JvXOMPyeqTJTUWVifU3nm+4XVJhYeLicqQEu80QejTilc5mgWm8Sd846y9Gi5LNdXbtVnGX7DBIrzc9m8fKHcoSicxODnj29AIMX5uXa3ldyyDTkVDXatNBFCMLp5CO0i7V/SOTg1iM151VyZEcrX20/yyK9ZvLjqOA8uzmJUejC/HqlgdAvX62R5ghNVJgqqzWhkmrtQEvtZOvUdQmR8Egunfq6csORF+o8ZR3JGW7vbadamA1YJ9g571a52RrcIYVia/btrgw0aHusVy7TdxbSN8qVnXACtInwZlhbEvENl9E0y0jPBeN7tldSaWXa0nDVZldSald2x/+b0kFxikE6W6ztkVYwQYhrQDwgTQuQCz0mS9KUj2nYmlUrFLU/8F72vn9yhKJzo+gefdkg7aW07AXBox+90tLOtapOVnIoGMsLt67m3jvDl/WHJLM0sZ2NeNRabRGqwnmf7xpESrD/vduYeKGX63mI6RvtTZ7YyeWshj/SMoX2U8l75O5ml9WhUgnijGyd2SZKudUQ7rsCeE+sV7quuphqdwdeu5a6BIWFEJ6Wyf9tGiLcvnq+3n2R9ThXfjm1m9+7QIIOGq1qFcVWrsIt6/pHSen4+UMp7w5IJ821cXbP3ZC2vrsvj88tS0Su7V//iYEkdKcG6M2fZOpvyG/kbRSdy+fqVSVSWlcgdisIJ1syfyW29WzqkamSrLpdwYNsmuw+lHpAcSJ3FxrqzJj3lsja7kktTg84kdYBWEb6kBuvZli/PmZ6uLiPcl75J5z/M5WhKYv8bDfW1LJ3xHWvmKWUGvEFETGP3OjfzsN1tteneh/raGrsPzMgINxBn9GHB4XLZ53tMVhu6v9k9qdcITHbewDzVDe3CGZku38S0ktj/RlxKOuntOrN67nTZ31SKpheXmg5A7pGDdrfVumsvNFoffrfzdHohBCPTgzlaWs/eojq747JH19gAlmVW/GG1T0G1iV2FtXRQxtj/Ir/KZPcnNnspif0f9Bl1FXnHDnNsv1Kr3dP5BwYTEhlN9qG9drdl8POndbde/Fblb3enYEByIIE6Ndvz5d0N3T7KlxZhBh5cnMXMvSV8t+Mkj/2azY3twwnUe3WB2L+QJInnVh7nrd8cU+/nYimJ/R90HTQctUbLhiW/yB2KwgmSW7Tm2P7dDmmr64BhFJ04zuFS+zYr6TQq3h+ezA3twh0S18USQnBP1yju7BxJeYMFIQQvDUhgaDPHHHjiSY6U1lNYY26S8ssXQrnd/gN/YxA9h41Go/WROxSFEwy4/HrKigodUnyry8BhfPXKJFZlVZIearCrraBTPeLCahPhflpUMlW3FELQLsqPdsrQy79anV2JRoVsZ52epiT2fzHxhXfkDkHhJB16D3RYW34BgXTqN4Q1qxdxc/twfOysxXKktJ7Hl2ZxV5coBqYEOSbI8+CIm5w3MVttrM6qpEtsAP4+8pZpUIZizkGSJOpqlCVd3qDweBZH9+5wSFsDxl5LlcnKb8er7G4rJVhHaoiBb3YUUdnQtHWMcioa+GBTPuNmHGLsjwf5z/yjzNlfQr1yIMc5/X6ihsoGK0NTg+QORUns5/LKXdfz3mMT5Q5D4QSfPPcQ37z2jEPaatW1F1EJycyrirK7LZUQ3NUlkhqTlc+32n8+6z/Znl/DpOU5hOg1XN4yhLEtQxiSGsT+olomLc9pkrNYPUn3OH9eHphA2yjHHd14sZTEfg6R8Ykc3rUVm1X5o/Z0rbv2InPfLqrKy+xuS6VScem4WzmyZzv7i+yv1JgUpOfq1mGsya5kTZbjNy3Vmq28veEE17cNY8nRcvKqTOg1KtbmVFFrlkgK1PHNjqa7qbi708NWrSN8ZZsHOZuS2M+hWesO1NVUk59zTO5QFE2sfc/+SDYbO39b5ZD2+o6+moCgEGbsc8wO5qsyQskIN1BYY/8hHH+2OquSjDA98w+WcWfnSB7sEcM1rcN4Y3Aieo0g1E/D+pwqqpVe+19IksTzq47z837X2amuJPZzSEzPAOD4kQMyR6Joaimt2hEUFsHvKxc7pD29wZfh4+9g64kadg593e721KrGZYYXW/Pl3xworqNZqIE6i41LzlrRoVYJRjYPYWdBLYlBOjLtrDfviTbnVbOjoBaDDIdW/xMlsZ9DVGIKAAVKj93jqVQqOve/lN0b12A22XcS0mlDrrmZgKAQfvrwNYfsYj5d33vPyVo+2JSPzUE7o08PHlgl+HOLVpuESoBNwiWGGVyJ2Wrj6+0niTP6MCglUO5wzlAS+znoDb5cfdejpLfrLHcoCicYfeu9vD13LVofx5RbNfj5M/aO+9m7ZT1bHbiD9FhZPcsyK/hq20mH3DDaRPqy72QtIQYNyzMrznzfbLUxZ38pHaL8yK1sIPUCSv16gzn7S8mvNnNbxwiXOhBcWcd+Hsbcfp/cISicJPTUGaiONOjK8Syd/i1fbsulXaSfQ0q5jmoeQmGNmXkHyzDq1Fzd2r7hmV4JRqbsLOLqVmFM21PM+pwqYow+bD1RTUqwnmPl9fRPDsSgVfqCp1WbrMzaX0rP+AA6Rsu70/TPlN/SeagoLaasyL5T7BXuI/vQPv474WqK8x1T70Oj9eGGR57nRJWJuQccd5D7rR0i6JdkZOruYmbbOUGr06h4snccP+4pplusP20jDRh91AxtFkRFvZlqk40b2spb2sDV+PuoeWVgAnd0ipQ7lL9QEvt5eO+xiXz41L1yh6FwEl9/I/u3bmTF7KkOa7N9z/50HTicn/YWk1fpmFUtKiG4r1s0vRMCyCyrt3tIpnmYgbcuTUKvUbHkaAULD5exMbeagSlBPNs33u4DPzxJTkXjHExKiJ5gg+sNfLheRC7IZrOhVWrGeI3wmDg69B7Iitk/MOb2+/DROWZc+abHXmTP5vW8mxfKq/6VDhmTVasED/aIARrruVTUWwjQqS96kjPcT8tN7SO4qX2E3bF5qo25Vby6No9HesbQ6wLOjHUm5RZ8Hhpqa9AZ7CvmpHAvQ6+9jcqyEn5b9LPD2gwOj+TGR5/n0I4tDh2SUasEapWgzmzjiWU5vL4u7w+10xWOk1VWzzsb8mkWqqdrrGuNq59NSeznoaK0GGNwqNxhKJyoVdeeJDZvxbzvPnXoruPeI66g68DhTN1dxOESxx6godcIhjYLYmNuNU8uy6aoxuzQ9r1dUY2ZF9fk4qtV8WSvWLuLuzUl143MRZhNDVSUFBESGSN3KAonEkJw9d2Pcek1N2OzOa73K4Tg9qdfJSgiltf3SQ7dySmEYHSLECb1iSO/2sxDS7LYIfMhHZ7CbG3cXVpntvFs3zhCzzr/1RUpif0cJJuN2595nU59B8sdisLJOvQawJBrbkajdeyb2D8wmHte+YiSghO8v9Fxm4xO6xLrzxtDEgnSq/lxT7FyvKMDaNWCsS1DeLpPHMlusJZfSezn4KM30H/MOJJbtpE7FIUMrBYLy2d+z9bVSx3abnq7Tlz/4DNsyqtm+l7H1xiJM+p4c0gSj/eKRQhBeb2F3ErH7Kb1JoXVJnYVNH7qGZQSRKsI+Ss3ng8lsZ/DoZ2/k5+dKXcYCpkIIfh1+rd8//aLWMyOHbO+dNwt9B55JdN2F7Oy44MObRsa16afXor37Y6TPLAoi5n7SrDalB78+ThcUsdjS7N5d2M+Jqt7TUYrif0cPn/pcYfV6Fa4H5Vazbh7n6DweBYr5/zg0LaFENw26RWatenIx8/c7/DJ1LPd2C6CzjF+TNlZxENLsjhQ3HTX8gSrjlXw5LIcfNQqXugf79ITpX/HvaJ1sqITueRlHqJtjz5yh6KQUfteA2jZqTuzPnuH2irH1kL30el5+J0vCQyN4KXVuRRUO74kL0CwQcMTveN4olcsVSYrjy/NZllmeZNcy53ZJImPNxfwzsZ8mofpeXNIIvGBjqkb5ExKYv8Xp8u3duw7ROZIFHISQjD+oWepKi/l5y8/cHj7gSFhPPb+t1j1ATy73Up5XdMdf9cjPoCPhqdwdatQOsc0rsMuqDZRa1bqrEPjbl4h4PKWIbzYP4FAvXvu4XRIYhdCDBVCHBRCHBFCPOGINl3Bb0vmkpieQXRCstyhKGSW3LINV9z5EK279WqS9mOTm/HIu99QVlzIC6uPN+mBFgatiuvbhhOk1yBJEu9uyGfCL0eZua/EKxN8nbmx9O7pobCJnSO5qb1rVWu8UHYndiGEGvgIGAZkANcKITLsbVduZUUFZB3YS68Rl8sdisJFXD7hAdr26Ntk7ae368SDb0wmu8rGS6tzqTM3/YSdEILbO0WSHmZgys4i7vjlKNN2F1FR37SHZrsCmySxKquCuxdk8vOBUnYWNh5hKDyg5rwjeuxdgSOSJGVKkmQCfgRGO6BdWQWHR/Hh4s30H3Ot3KEoXEhDXR3T3n/FYcfn/Vm7nv2595UPOVRSx8trcp1SGqBZiJ5n+8bzxpBEWoX78uOeEtZkN84leOoa+B35NTy8JIt3NuQTZNDw6qAErszwnN3ljkjsscDxs77OPfW9PxBCTBBC/C6E+L2qzHF1MprC6Z2GgSFh+Aa4ZpEfhTzUGjVbViziuzeew2JumonOrgOHM/Gl99lT5LzkDpAeauCpPnF8ODyZQSlBAPx6tILHTk20uvswjdUmnVnqmVlWT63ZxgPdo3lzSCItw91jffr5ctrkqSRJkyVJ6ixJUueA4BBnXfaiLJ3+LS/edgW11VVyh6JwMRqtDzc++gL52Zks+uGrJrtOr+FjufOFt9l1so4XsiOcWtQrPlB35kANg1ZFdYOVDzYVcPOcI7z92wm2nqh2WiyOUFRj5qc9xdwx7+iZTyKjmgfz0YgU+icHeuRxf46Y8s0D4s/6Ou7U99ySxWxmwZTJhERE4esfcO4nKLxO+5796dhnEHM+f5dew8cSHN40By30GXklAJ89/zAvZumY1CcOXycfmNwn0UjvhAAOFNex8lgl649XUlhjptOpFTW7CmpIDtYToHOdg5yhcQhpWWYFa7Ir2XOyFpsE7SJ9CfdrLA+hdbN16RfKEYl9C5AmhEimMaGPA65zQLuy+G3xzxTn53LTYy/KHYpsJEniZF4OKpWa8Jg4ucNxSeMfepZHrxzI9I9e587n32qy6/QZeSVqtZpPnn2Q51fl8mzfOPx9nJtEhRC0DPelZbgvd3SKpLSucQdurdnKC6uPY7U1jtO3ifSlVYQvLcIMTo/RYpM4UlrPyRozfRKNCCFYcKiMOouNKzNCGZgSSJS/95ypYHdilyTJIoS4B1gCqIGvJEnaa3dkMrCYzcz5/D0Sm7eiY59Bcocji8x9u/j8pceoKClCkiTComO58/m3iEtJlzs0lxKVkMztT79Kauv2TX6tnsPGovXR88GTdzNpr44XW9bLtr5aqxZEnkqQOrWK/w1MZHt+DTsKavjlYCmz95dyS/twxrQMpazOwvJjFSQG6ogz+hDhp3XoEsI9J2vZklfNkdI6DpfU02CV0GtUXBIfgEYleL5/PIE6tUescrlQDvnrkCRpIbDQEW3J6bfFP1OYm81Db3/hlX8M1ZXlvHH/zYx/6Bl6XDoaJImVP//Iq3eP5+05q/HRK4eNnK3vZVc77VpdBw7jkXe+5J1HJ/DUCRsv9I8nTObSsWqVoHmYgeZhBsa1CaPBYuNQSd2ZnvGx8nqm7Cw683iVgDBfLQ/1iKZluC/HyupZf7wKg0aFj1qgVQsEgkviAwjQqTlYXMfmvGqqTFYq6y2U1lk4WWvh/WHJGHVqdhbUsOBQGUlBOganBpERbqBVhC+aUzePIDfdXOQI3vvK/0a3QSOx2Wx08tKdphuW/EJG5x70HDb2zPcGXnE9v69czO+rlnDJ0DHyBeeiqsrL+PrVSfQZdRXte/Zv0mu169mfJz78njcmXs0TS7N5oX8CsUbXGV7QaVS0ifQ783XHaH+mXpHG8YoG8qpMFFabKaw2Y9Q1pp2s8gZm7SvhzzXJ0kIbx+wzy+qZvb+EAB81Rp2aYIOGTtF+Z1a2XN4ylHGtw9x6I1FTURL7WXQGA/1GXyN3GLIpKyokOjH1L9+PSUql9GSBDBG5PoOfH5n7dlKQc4y2PfqiUjXtpFyLjt14+tv5vHbPDTy5LJvn+8WTEuK69cH9fdRnxuf/rH9yIP2SjDRYJUxWG2ZrY8I+nfiHpAYxtFnQP356Pr1yR/FXyr8MUFtVyTM3jmLv5vVyhyKrtDYd2bZm2R+OgrOYzWxft4K0tp1kjMx1abQ+XH7HA2Qd2MP2tcuccs3klm149stZaIOjmLQihz0na51y3aYghECvUWHUaQj11RLqq0WrbkzkapXwyiFRR1ASOzD/u085umcHBi9f3tjukn74+gfwzqMT2L91A3s2reON+28mNrkZ6e06yx2ey+o5bCzhsQn88vXHTrtmTFIqz309h+C4VJ5fW8jGXGXPheL/eX1iLysqYOHUz7lk6GhSMtrKHY6sVGo1j73/DWltOzH13ZeZ/vEbtO/Zn/tf/1TpOf0LtUbD8Otv5/CurRzdu8Np1w2NjObZL2eRmJ7Ba+vylDK8ijO8fox95qdvY7VYufrux+QOxel+WzKXRVO/oOhELsktWjP2jgdIb9eJy26+i8tuvkvu8NxK75FXNh56HhHl1OsGBAXz1KfTeOeRO/hg41qqG6yMaek5NU8UF8ere+z5OcdYNfcnhlx9IxGxCXKH41TLZk5h5idvceWdD/HKtEV0GzSCtx+6jSN7tssdmlvy9Q/g6rsfIzjcuYkdQO/rx6PvfUO3wSP5ekcRU3YWeWzxLsX58eoee2RcIve8/AGtujZNjW1XZbVYmPP5ezz+wRQS0lsC0G/MOKxWK3O/+pCH3/5S5gjdk8VsZudvKwmPiSchraVTr63R+nDv/z7ELyCQmbOnUmO2MqFTpEfWQVGcm1f32FUqFT0uvQyjixclc7TKshKsVuuZpH5a6269yD64T6ao3J/VYuHdRyeyftHPslxfpVZz26RXGHnjRBYdLue9jfnKwdVeyisTuyRJvPPwHfz60zdyhyIL/8AgbFYLRSdy//D9Y/t3ExmXKFNU7k9nMJCY3pLMfbtki0EIwbX3P8VVdz3CqqxKXitNw6Ikd6/jlYl97+b1bFm5+EzddW+j9dEx+Oqb+OSZ+ynIOYYkSRzYvpmp77zE8BsmyB2eW4tOTKXweJasMQghGHv7/Vz3wCQ2LZ3Pm+vzlOTuZbxujF2SJGZ++hYhkdEMvOJ6ucORzRUTHmSOSs1zN4/BbGogMCSMa+97kg69BsgdmlszBodQU1kudxgAjLxxImq1hilvvcCbv53g0UtilO33XsLrEvvezes5tPN3bnniv2h9dHKHIxuVWs0Vdz7ImNvupaGuFoN/gLJW3QMNu/52bJKNqW+/xAeb8rmve7QyoeoFvC6xz/3qQ4LCIujrxTVhzqbWaJTj/xzoslvvof9Y1zond8T4CTTU1TLzk7fw16m5vWPTHAyicB1el9iv/M/DVJWX4qNz3cJJCvcVGBJGYEjYma/r62rZvGwBBcez8PUPoNugEYTHxP9LC01j7O33U11ezrxpXxLpp2VUc+9aCeZtvG7ytHn7LnTud6ncYSg8UH52JlPeeoGyokIA1sybwX0jurNp2QKQJApzs5k0fgSfPvcgpoZ6p8YmhGD8Q8/Qpf9QvtxexDY3O7dUcWG8JrGXFRXw9SuTKCk4IXcoCg/160/fsOTHrwHYtGwBMz55k6smPkxJwQnmf/cZm5YtoP+Ya6mtquLTZx90enwqtZr//Pc9EtJa8taGExTVmJ0eg8I5vCaxr/z5R5bO+A6L2SR3KAoPdCLrKMtnTaXPqKsICotg+kevM/LGicz+/D3G3fck32w4zEvf/kLmvl2ERcdyaNdWsg46/wRJvcGX+17/FKvWlzePh2JTSg94JK9I7DabjVVzfqR1t95ExifJHY7Cw1jMJj599kF89AauvutRMvftBCHY9/sGrrzzIdr37I9KpSIyPol7/vcBa+bP4pKho1m/cLYs8UYnJHPjoy9wYPtmlhwplyUGRdPyisS+f+tGigvy6DfaeWdUKrzHzE/f5sie7dzxzGsEhUVQWVZKeHQchcez/nLYdWBIGMbgEHz9jVSWlcoTMNBn1FW07NSd74+YqDVbz/0EhVvxisT+2+K56Ay+dOqrTJoqHG/otbdy26RX6TZoBAAhEVHkZ2cS16w5e3//7Q+PLTqRS3VFGVVlJU4v8Xs2IQTXPTCJ6vIypdfugbwisfv6+9Nn5JXoDAa5Q1F4iNrqKn7+8gOsFgtBYRF/2MWckNYSP2Mgqa3a8ctXH7FyzjQqy0o4sG0T7zxyB4OuupH1i36mz6irZHwFkNqqPentu7DiWIWscSgczyvWsV//4DNyh6DwIMf27+aDJ+/mZF4OaW070arLJX/4+elCXB9Nuo8r7nyIzSsW8cN7LxMSEUXP4ZezY90Kug4aTnRiikyv4P916X8pU3dsobTOQojBK9KBV/D432RNVQW+/kav2S6/ZcUiVsyZRnVFGa269GT4+DswBisn6jiC2dTA3K8+Yu5XH2AMDmXSpz/SslP3v31sm269mfjC23zz2tP4+hvpOnA4FSVF/PL1xwy6cjxX3/Wok6P/e/HNWgBQUG1SErsH8fjf5Bv330JAYBAPv/OV3KE0ublffcia+TO5YsIDhETGsH7RHJ67eQwvffcL/oHBcofn9j586l62rFjEJcPGcPNjL57z37R9z/68/fNa9m5Zz8nj2Rj8/bnrpfdcqoSDUDWOxirFHz2LRyd2s6mBzL07uXTcLXKH4nBFJ46zeu50CnOz8Q0w0r7XAOZ9+wmvTV9GaGQ0AC06dOXT5x5k2YwpjLn9Ppkjdk85h/YTEhmFf2AwI2+cyIDLr6PdJf3O+/kqlYo23XpDt95NF6Qdsg40rqWP9tfKHInCkTx68vT4kYNYzCaatekgdygOI0kS0z9+g0njR1BdWUZsajr+gUF8+fKTRMQmnEnqp3UZMJyDO7bIFK37ys08xPuP/4cnxg1h3jefAJDWtuMFJXVXZzGbWD5rCmkhekJ9lcTuSezqsQshrgKeB1oCXSVJ+t0RQTlKzuH9ACSkZ8gcieMsnvYV29cuZ+QNd7JgymT8g4KpKCmmZaduHNq5FavFglrz/7/WgpxMgsOVan7nK3PfLuZ+9SFbVixC7+vHmNvu9cjDRyRJYsqbL3AyN4f/9IuTOxyFg9nbY98DXA6scUAsDpefnYlaoyVChmp6TcFiNjPvm4/pO+oqVv78I899NZu3Zq/i/QUb0Pro0OkNTH3npTMFpjL37WLBlMkM8OIDRc6HdNa2+vnffsLezesZc/t9vDvvN66++zH8jUHyBdcErBYLX786iaUzvmNMixA6RvvLHZLCwezqsUuStB9w2RUnbbr1xj8w+A89WHd2cMcWQiKj2bpmGePufYKYpFQAfP0DuPWp/3Hf8O7s2rCG1fNmEBQWQV11FeMfeoZmrT1nKMqR6mtrWP3LDH6d/g0PvvEZcanNuf6hZ7nd18+lJjgdKfvQPj5/8VEy9+3i8pYh3NAuXO6QFE3AaRlPCDEBmAAQFhXrlGu27taL1t16OeVazlBXU0VgSDgn83KITvjjGmh/YxD+xiB6DhtL3rFDXHbL3cQkNUOjVcZO/yzv2BGWzfiONfNnUlddRWrr9tTX1gL8ZY7CUxQez2LOF++zbv4MAnzUPNYzhp4JnnnzUpxHYhdCLAP+bu/zJEmS5p7vhSRJmgxMBkjJaOuUxVVFJ47jozf84eADdxYVn0T2wT206d6H31ctISG95ZmfHdu/G6vVwskTOSSmtyIhreW/tOS9GurqeHr8CKwWC90GjWDINTeT1raj3GE1CZvVyq6Na1g5+wd+X/0rWiExIj2Ya1qFEaBTyx2eogmdM7FLkjTIGYE0hVfvuYHEtJbc99oncofiEHGpzQmLjiM2OY0F30/GarXQsc8g8jIPM+OTNxlx40RmT36Xt2av/MPzJEkic99Ojh8+gMZHR5vuvT3mZncuRSdyWfnzNLIP7OWR975GZzBw32ufkJLR1iP/DWw2G0d2b2PTsgVsnvMNJbUWjDo1Y5sHMTI9WFn94iU8Y/D5n0gSuOj4/8W68dHnee2eGxl540QKc7P48uUnCYmMZuSNE/l1+reMvvUeAkP/f9w0+9A+Jr/wCNWVFbTs2I26mmq+ee0Zug8eyY2PPu+RRwRaLRZ2rF/JillT2bF+BQDtew6gvrYGg58/HXoNkDlCx6quKGPP5vXs2rCanUtmUlZnQaMSdIjy5bYOEXSNDUCr9qz3geLf2bvccSzwARAOLBBC7JAkyWVKKGq0WqwWi9xhOFRKRjse/3AKU995qbGCYGpzTuZms/DIQcbefh/9zjqkO+/YEV75z3WMvGkiWQf3smP9SvwCjAy99hZyjx7ivUcn8vC7X6FSedZ2hg2/zuPjp+8jKCyC0bfeQ/+x1xEe4zlL+qoryzm043cObt/MgXlfc6S0Hgnw06poH+VHtzh/OsX44++jDLd4K3tXxcwB5jgoFocz+AVQW1UpdxgOl5LRlmc+n0F+zjFO5mbjFxBISqt2f0nQsz57mwFXjGfxD18y8MrxXHvvk1SUFjHt/VcJi4ohPzuTvZvX0aZ7H5leif1sNhu7N65hxayptOzcg6HX3krXAUPxeWMyHfsMcvvJY0mSKDiexZFd2zi8eysHt28h9+hBJElCrdGSHqThmtahtI/yIz3UgFql9MwVHj4UExgSxonsTLnDaDLRCclEJyT/7c9qqyrZ+dsqImMT6NB7EGNvvx+A0KgYHn77C+4b2YPh19/Byp9/dMvEXllWyupfprN81hRO5uYQEBRCy849APDRG+g6cJjMEV6cyrISju7dSebenRzZs52je7ZTXVEOgEGjonmYnl6tQ2kVbiA91IBO41mfthSO4dGJfcDl19FQXyd3GLKoKC3GGBTCiexMelx62R9+pvf1I7VVe1RqNaUnC2SK0D4fP30fuzaspkWHrlx916N0GTAMrY9O7rAuSG1VJcf27yZz3y6O7mtM5sX5uQAIICFQR/dQPc2bR5EeqifeqFN65Irz4tGJvV3P/nKHIBs/YxBV5WWERcdybP9uug8eeeZnFrOZnEP7aNmxOwFBrl/10VRfx29LfmH5rO956K3PCQ6P4pp7Huf6B58+U3bW1Znq68g6uI+je3eQuXcnmft2kn/Wp8lIPy3NQ/WMbB9OWqiBlGAdvlpljFxxcTw6sdusVjL37cLPGOgShxo4kzE4hLS2HQkMCWPh95+TkNaC7oNHUV1ZzrT3XiaxeSu2r13G0OtukzvUf1R0IpdlM75j5c/TqK4oJy41ndKTBQSHR5Hcso3c4f0jm81GftZRjuzZzpHd2zm6dwfHjxw4M5EfYtCQFqJnQNswmgXraRaqx6jz6Leiwsk8+q/JZrPy0oSrGHj5eG589Hm5w3G6sXc8wNsP3cYVEx/m15++ZfKLj6JWa+g+ZBSRcYkc3rWVTn2HyB3m36oqL+PhsX2xWi106T+UwVffREbnHi5ZvqK2uooju7dxaOdWDu/aytE9O6itbpy099WqSAvR0zk9kLRQvVJJUeEUHp3YNVofWnToxs4Nq+QORRbp7Trxnxff4dPnHyYuJZ0RN0zAVFfPpuULiE5M5bEPprjMqhGL2cRvS34h++Bebnj4OQKCgrnjmddp2ak7YdHOKUFxvsqLT3Jg2yYObN/Mwe2byTlyAMlmQyUax8V7RxlID40iPdRAnNEHlQvejBSezaMTO0DHPoP59vVnyM08RFxKutzhOF27nv35YOFGtqxYTM7h/fgajTz8zlckNW8ld2hAY898+cwp/Dr9W8qLTxKXmk5DXR06g4HeI6+QOzygcSJ635bf2Pf7BvZt3UB+1lEA9BpB81ADPTKCaRnmS3qYXhkXV7gEcXbJUmdJyWgrvTx1oVOuVV58kruHduGym+/imnsed8o1Fednx/qVvPvoBEz19bTt0Zdh199O2x59ZR9uMdXXsX/bJnZvXMOeTevO1PX31arICDfQKsKX1hG+pAbrlVUqCqcaPe3AVkmSOp/rcR7fYw8Ki6B9zwFsX7ucq+9+TPak4e0O7dyKJNlo3r4Lqa3a03PY5Qy77lbiUpvLGldhbjbb1y5n5/qV7N+0GpNVQqMSZIQb6NM2jLaRfjQLURK5wj14fGIHuOWJ/2IMDlGSukxsNhvb1y5n/refcHDHFtp0782TH/9wahz9NdliOrpnB1tX/8r2WZ9zvNIEQEyAD0NSg+gY7UfrCF9lA5DCLXlFYj89+WY2NaBSqT3m4A138PuqJfz04evkZR4iLCqWGx994Q/1bJzJZrVyYPtmNi1bwNZ5Uymts6AS0DrClyGpQXSJ9Sc6wEeW2BQKR/KaDFdRUsRzt4xhxPgJDL76JrnD8Wj1tTWo1Gp8dHoqSopRq9Xc9d/36T54pNNX4UiSxOFd29iwZC6bli2gvPgkPno9HcP09IgLoHOsUixL4Xm8JrEbQ8IIi4pj+sdv0HXQCI+sxS23yrJSfv3xa5b89DVX/ecRhlxzM/3HjGPA5dc5fRis8HgWa+fPYt2i2ZzMzUHro6NzhJZePWPoFO2PQasMsSg8l9ckdiEEtzzxX568dihfv/o097/2iTLm7iDF+Xks+H4yK+f8gKm+ns79L6VZm8ZTiVRq5/WG6+tq2bxsAat+/pED2zcjgLaRvozrFkWP+ABlKaLCa3hNYgeITUnjyokP8eMHr7J67k/0GzNO7pA8wsfP3M/hXVu5ZOgYRt38H6fvF8g5vJ/lM79n3aI51FVXEZ2Ywg1tw+mXbCRM2eWp8EJeldgBRt44kd0b17JlxWL6jr5G6bVfhEM7t7Jw6ufc8sR/CQwJ46bHXsQvINCpO0QtZjNbVizi15++4eCOLWhVgp4JAQzplkBGuBYhQp0Wi0LharwusavUah548zN0el8lqV8Am83GjnUrmPfNxxzcsQX/wCCOHz5AYLdeJKZnOC2O6ooyls+aytLp31J6soCIuARuaR/OgJQgjMoBzQoPZruAzaRel9gB/AICgcZDDb5/+yWuf/BpZTL1X5ga6nnmhpEcP3KQ0KgYbnjkefqPvRa9wddpMZzMy2Hh95+zeu5PNNTX0T7Kl7v7xNExxoBKOC8OhUIOmWX1vLE+77wf75WJ/bSCnGNsXr6AY/t389QnPxAcHil3SC7j9AHJ3QePxEenp33PAYy66S66Dxnl1CWLuUcP8svXH/PbkrmoJCt9EgMZ3SKSpCDXPYS7wWKjtM5yZk38tvxq9hXVUW2yUmOyUWexYbNJPNsvHoCpu4rYkleNEKAWAo1KYNSpeapP4zmtq45VcLLGTKBeQ7BBTZhBS4S/Vlmm6UUWHymnuPb8z2/26sSe3q4zj73/HW8+cAvP33o5j73/LbHJzeQOS1b52Zks/uFL1sybgcnUQHq7ToRERHPt/U85NY7sQ/uYPfldtqxYhF4jGNUsiNEtQlym5K1NkhA0rrball/N6qxK8qvNFFSZqGiwAjD9qnR0GhXb82uYf6gMP60aPx8VBo3qD8stjTo14X5aJEnCKoHFJv3hY/fG3Co25Fb/4fqRflomX5YKwC8HSrHYJJKCdaQG6wnUe/Xb2uOYrRLrsiu5JD6AVVnnd4azxxcBOx9H9+7gjftvwWI28+h7X9O8fRe5Q3K6/JxjfPfGc+xcvxKN1oeew8Yw7LrbSUhv6dQ4jh85wMxP3mLLysX4aVWMSA9mVPMQWcfPJUmisMbMgeI6DpfUc6S0nqzyet4dmkx0gA8LDpUxa18JMQE+RAVoifDTEu6r5ZL4AHQaFWarDbVK2FW+12y1UdFgpbTOQkmtBask0SvBCMATS7PZX/z/R0BG+Gnpm2RkfNtwu1+7Qn47C2p4duVxnuody//W5ilFwM5Xaqv2vPTdL3z1v6eIjEuUOxynqa4oo6K0hNjkZvj6B5CXeZgr7nyIQVeOJzDUuUmh8HgWMz59iw2L52LQCK5pHcplzUNkGW6QJImCajMGrYogvYaNudW8uq5xfFOnFqSG6BmYEsTpemDD04IYkf7PRwxq1fZvhtKqVYT5qhqXb/5pwc+rgxOpNlk5VlbP0dJ6DpbUozl1E7HaJB5ekkVGuIGucQG0ifBVCpm5mV2FtagFtI30O+/nKD32v2GzWvn61acZdNUNTl3x4QySJHFw+2ZWzJnGpmXzSW7Zlue/mg00rnxRqZy7I7OitJh5/+nPkiPlqFWCUenBjG0ZSoCTe+hmq43dhbVsOVHN1hM1FNaYubVDBKNbhFBRb+G341W0CDOQEOheB0qX11v4cHMBOwtqMFkljDo1PeMDGNU8hFijUhfHHTyzIoeqkFRenrqQ6zrGKz32i1VwPIstKxax8udpXHrNzVx+54NnVtK4s7XzZ/Hzl++Tn52Jwc+fPqOuYuAV48/83JlJ3VRfx8KpXzDvm49pqK1mSGoQ17QOI8Tg/D/JBouNW+ceodpkQ6cWtI3yY3SLELrE+gMQqNcwLM31D/3+O0F6DU/3iaPBYmNbfg3rcipZfqyCXolGYo0+VDVY8VELpYqlC8upaKB99wvrYCqJ/W/EJKXy5uyV/PTh6yye9hVr5s/kspvvZtj1t6HRuk8vp7qynC0rFtNlwFD8jUHUVlcRGBLGqJsbV7c4c7niaZIkseHXX/jxvVcoLsijW6w/N/ZLJs6oc9r19xXVseJYBTUmK0/0jkOnUXFlRijxRh1to3zxccDQiavRaVT0iA+gR3wANSbrmcnbH3YXsS6nipHpwYxMD8ZPWWnjUuotNsrrrUTGJ13Q85TE/g/8A4O5bdIrDLrqBn784FVW/jyNETdMABp7mz56g8wR/r2K0mK2rVnG5uUL2bNpHVaLGZ1ezyVDxzDkmpu4dNzNssWWdXAv3772DAd3bCE5SMcDA+JpcwHjhvaobLCwLLOCpUfLOVFlRq9R0SshAJskoRKCsS3Pf6eq2SqxLb+a8norQXo1HaP9HDKO7ixnJ+/eiUZO1pj5YXcx8w6WckVGKCPTg93q9XiyklNLHEMioi/oeUpiP4fE9Awe/+A7qivLUanVmOrruH9UT9LbdqL3qCtpd0k/tD7O6W3+HZvVSk1VJQFBwZQVFXDP0K5IkkR4bALDrruN7kNGktyyLYBsO22rK8uZ8fGbLJs5hQCt4O4uUQxMCXTKWLUkSQghWJZZwbc7isgIN3BVqzAuiQ9AfxHDD78eLWfqriJiA3yIMfqQX2Xio80FXNc2jKHN3G+4JiPcl4y+vhwpref7nUV8s6OIoloLEzopezpcQWVDY2I3Bodc0PPsSuxCiDeAUYAJOArcIklSuT1tuip/YxAAJlMDvYaPZe2C2WxZuRiDnz/tew1g1M13OeWA6Ia6Oo4d2M3hnVs5uGMzB7Ztpk2PPtz/2icEh0dxwyPP07JTdxLSWspeMkGSJNYtmM3Ud/9LVXkpw1ONXNc2vMlXukiSxN6iOmbuLaFfspF+SYEMSQ2ic4w/CYEXfxNecqSc2ftLuLpVKOtyqth4vIpYo45xrcOYe6AUq41/XR3jypqF6Hm+fzzb82uIOzWpWlJrRqdRKRuhZFRrtgFg8Dde0PPs7bEvBZ6UJMkihHgNeBLw6BOj/Y1BXP/gM1xzzxPs3byOTcsXsn3tci4ddysAuzetZfXcn0hq0Zq41OZEJyQTGhV7wbs1a6urKD6RS2FuFjWVFWcqUb585zUc2bMdgOjEFLoPGUX7Xv3PPG/otbc66JXaJz/nGF+9/CR7t6wnLVTPC4MTSAlp+t2iuwpqmLanmH1FdQTp1fSxNb4h/H3UdiWoBouNKTuLuLZNKLP2lXJHp0gywg0cKK7js62FXNEylGm7ixiYEnhRnwRcRYfo/x8ae3djPvlVJh7tGUvzMNccevR0JmvjqkWd4cL+/e1K7JIk/XrWlxuBK+1pz51otFra9exPu579sdlsZ75fUVLMwR1b+G3x3DPfEyoVHy3eQlBYBKt/mc6WFYvw0ekRKhWSJGGzWrn/9U8RQjD1nf+yfNb31NfWnHm+3tfvTCXK0bfdC0g0a9PRJevbWMxmFn4/mVmfvYOPZGZi50gubRZk1+ac8/Xh5nyWHq0gxKBhQqdIBqUEOmy1x+a8apqF6Fl+rJK7ukbROaZxxUy3uAB0ahVf7zhJ8zADG49X0S/Z/VdQAdzQLpw31p/gqeXZ3Nk5iiGpQXKH5HXMtsbErtFcWMfQkWPstwI/ObA9t3H2MsFew8fSa/hYqivKyD16iIKcYxTl5xIQ1DhGVltVSUlhPqaGeqRTNwStjw9WixmN1ofE5q3oN3ocIZFRhEbGEBmfSFRCyplhlU59Bzv/BZ6nrAN7+OyFR8g+uJce8QFM6BTf5MsXS2rN+Puo0WlUdInxJ8GoY2hakMNXtpTUmokL9GHR4TLaRv5xNVGbSF+yyxtoH+lLcd351/NwdemhBt4ZmsSb60/w0eYCTtaYub5NmOxDfF7l1DYjcYFLkc/5rhNCLAOi/uZHkyRJmnvqMZMACzD1X9qZAEwACItyXt1uufgHBtOiYzdadOz2h+8Pu/52hl1/+z8+7/SNwZ2YTQ3M+eJ9fvn6IwK18ESvWHrEBzTpNS02iV8OlvLTnmKuyAjl6lZhdItrumsa9Rr2F9cR5e/D4ZJ6WkX8f3I/UlpPlL+WoloLcXaM4bsifx81z/SN49PfC1ifU8XYFiHKkkgZXOhG0nMmdkmSBv3bz4UQNwMjgYHSv1xdkqTJwGRo3Hl6QVEqXFbWgT188uyDHD9ygP5JRm7vFNnkk22HSur4cFMB2RUNdI31p0/ihU0sXYxusf58sbWQq1uF8vGWAh7sEUOzED2ZpfV8sCmfIc2CmLm3hLu6/l0fyL2pVYK7ukRRZbLh56M+s9JI0fROd9Qlm/WCnmfvqpihwGNAX0mSau1pS+FerBYLc7/+kDmfv4dRC0/3iTuzU7MpLThUxhfbCgnWa3iqd2yT9tLP5uejZnSLENbkVDEoOZBX1+ZS0WAlQKdmRFowv+VUMiI92GNXkAjRWErYbJX4YFM+bSJ9GayMuTc59akbqM1qO8cj/8jeAdAPAR2w9NQdfKMkSRPtbFPh4vJzjvHJ0/dzZM92Lhk6mrt89zmttkvLMAODU4O4qV2404cErm4VilWSmLGvhM4xfkT4+VBUY2LWvhKGpQVzbRvXm8x2NJWAigYrn/5eSFKQjrRQZbVMUzo9VWSxmC/oefauivHu4uVeRpIkVv38I9+9+TwarZaHL4mhT/BBoGkT7LKj5WSWNzChUyQpIXruCpFnuEMIwXVtwhmZHsza7CpK6yykhRm4tWOk19RAV6sED18SwwOLj/H2hnzeHZqk1JlpQmeqdFoubFJe+Y0ozktVeRkfjs7g85ceo7kR3u8f1uRj2yarjQ825fPB5gKOVzRgvsCPo03FJjUetCFJEpIEMhRIlZVRp+aBbtGcqDIxfW+J3OF4NNWp3dkWs+mCnucd3QyFXfZv3cBHk+6jsriam9qHM6ZFSJOvSy+rs/DK2lwOltRzVatQrm0dJnu5XEmSmLq7mIWHyuga60e4n5bM0nqm7S5maLMgxrcLd8p6fVfQNsqP/klGlhwp56pWoW69KcuVaVSnx9idOHmq8GxWi4U5X7zPnC/eI9pPzdODk0h1wu5Rq03i2ZU5FFabebxXDJfEN/2ql/Px054Stp+o5oqMUBYcLqPGVI2fVs0VGSFsPF7FtN2C673o1KKbO0RwcweUpN6ETvdlbM5cFaPwXGVFBXzw5D0c2LaJfklGJnaO+sM5nU1JrRLc3D4Co07tMpNzNSYrvxws5epWoSzNrODp3nGkhOg5VlbPOxvy6ZdkZOa+Ei5rHuL0Q0LkEnTWvILVJsn+icoTnf4XPXt3+/lQbrWKv9i1YTVPjhvKsf27ub97NA/2iHFKUt97spZlmeUAdIrxd5mkDrApt5o2kb4szazg7q5RZ+reJAfruadbFEuOltM+2o8NuVUyR+pcDRYbjyzJYu7BUrlD8Uhn9gtc4ESOktgVZ9isVmZ88iav3XMDwbZq3uofyQAn1T3ZV1TLC6uOM/dAKWar681GVjZYCPfTUlBtotmfhqNSg/UUVpsJN2ioaji/j8wNFhtrsiqZf6iUrPL6pgjZKXQaFQhYl+NdNzRnObPn8wLnbpTErgAaD+h45e7xzPn8PfonGXljSCLxTtoef7S0nhdX5RLmq+Wl/glo1a73kT7MT8vxChNJQXp2FvxxL97OwhqSgnTkVpkI9T336ObR0nrunJ/JqqwKcipMvLAql483F2Bz0+U13WL9OVpaT7kH1clxFaf7OBdaBExJ7AoO79rGpOuGcej39dzbNYr7u0c7bW1yYbWJF1cfx99HxYsD4gmS4czT89Elxp/MsnoGJAfy8ZYCfjteSXmdhQ3Hq/hwUwGDU4M4WFxHj3PshLVJEm/+doLbOkTwbL947uoSxccjUjhSVs+6bPfs9Z4+BWt/cZ3MkXgey6nMrtZc2PtCSexeTJIkfv3pG166dTTamhJeH5LIICdvE9+aX4PZKvFcv3jCfC+sV+JMOo2KG9qFM/dAKVdmhDD/YBn3LjrGLwdLuSIjhHkHS7m+bfg5b4jHyhoQQK+E/78BGLQqxjQPYW1OZRO/iqaREqxDJSCzzH2HlFyV6dSkqY/+wlajuWb3SNHkTPV1fPXKU6yZN5POMX482CNGljonw9OCuSQuwGV76mcbkhqESsD3u4qJ9tfSJcaf/GoTM/aWcF3b8POqV26xSWhU4i9FtLRqgcXmnkMxPmoVY1qEOGUprLepO3WCks5wYWcDu/67SeFwxfl5vP3wHWQd2M241qFc0zrM6RtrVh2rICrAhxZhBrdI6qcNSgmib2IgOwtqKK230CM+gPZRfuc9L9AsRE+12cquwhranhrCsNokFhwqo7cTqlQ2lZvaR8gdgkc6czSen5LYFf9i/9YNvPvoRKw15UzqE0vXWOdURzxbXqWJj7YU0C7Kj6f7xDn9+vbSqgWdL7KSpVoluK9bNK+vO0GPeH/C/bT8llNFqK/GaSuQmoLJasMmKZuVHK3G1JjYT5+5fL6UxO4lJEli6fRvmfLWC0TGJTGpZyBxRucfCmGTJD7cnI+PWvCfzpFOv74raB/lx3vDklidXUlVg5Ub2oXTIdrPrcsRvLwmj1qzlTeGJMkdikcpb7Cg1wh89E4881ThHixmE9++/hzLZ31P5xg/Huok8POR56SfVVmV7Cuq496uUYS68GRpUwv11XJ5y1C5w3CYeotN6a03gfI6yx92+J4vJbF7uMqyEt599E4ObNvEFS1DZC1U1WCx8f3OItJC9AxIcb1hB0mSMNsktH8zuan4d+X1FtJdaKewpyiutRB2Hnsj/kxJ7B4s9+hB3rj/FsoLcnm4Rwx9kuSdnFMJGNMihGahepcadsgub+DnAyWsy6nCJkkYdRoGpwZyWfMQjz0RyZHMVhsna8xuPfnrqk7WmGkX5XvuB/6J8tnJQ21ft4Lnbh6D2dTA/wYlyJ7UAbRqFZe1CCEj/ML/UJvKjoIanl6RQ7ivliszQhieFsxlzYMprDbx+NJsyuuV3ZTnkl1hwiZBoocd5C23BouNkjoLUf4+F/xcJbF7oCU/fsObD9xClNbMm5cEuMRH5N2FNfx6tNyl1mo3WGy8/dsJrm8bxuIj5RRWWwgxaNiSV01epZkOUX5M3lood5guL1Cn5vq2YWSEy/935knyKhsP14g1XnhiV4ZiPIjNamXKWy+w5Mev6RLrz8NOqsp4PuYdKuNQST0DXWhJ39qcSpqF6Fh8uJzbO0WeORFqTIsQ3tmQj0Yl2JFfQ2ldY8JX/L1wPy1Xt/L8816dLbuiAYDwB6dc8HNd412vsFt9bQ1vPXgbS378mtHNg3myV6zLJPV6i41tJ2roGR/gUjW7DxXX0yLcl9I6yx+2+AshGJEezI6CGpqHGThaqmyV/yfVJisbjlfRYHGNYws9SVZ5A1ofHdEJKRf8XNd45yvsUlZUyMuj2rFz/XImdo7k1o6RLpVA952sxWyT6BxzcZt6mopKNK6EsUoSfz5OtcFqQ6Nq3ObvQv+ULmdtdiWvrsvjeOWFncmpOLfMsnrimzW/4AJgoCR2t5d37AjP3TyG3EoTk3rHMSwtWO6Q/mJ/cR0qAS1dbAy2XZQfOwpqSQrSM//Q/x8UYbZKzNpXQueYxnK0zcNcK25XIUkSiw6XkxykIzVYmTh1JKtN4nBJPSmt2l3U85WBQzd2YNsm3nroNtQaLf8blPiXAyBcRUWDlXijzuU2sHSN9efr7ScZnh7EkiPlbM6rJt6oY3tBNc1CDByvbKBPolFZ8vgPtubXkF3RwH3dopR1/w6WU9FAncVGetvOF/V8JbG7qS0rFvHhpHsJi47j+TbSRS2Jcpa7ukRhdaHVMKepVYKn+sTywqpcOkT7Em/UYbZKDE8LZnNeNZpTdV0Uf2WTJH7YXUyEn4Y+ia4zIe4p9pxsPMylRcduF/V8JbG7oeWzpvLV/54kLUTHMx0FRp3rb813pTH/syUF6XlnaBK/Hi1n5bFKasxWIv20XJoaRM8Eo0ue5uQKyuos1JttXNsmXPk3agI7CmqICdASFh17Uc9XErsbkSSJn798nxkfv0n7XgN4KjrXaScd2eO9jflkhBsY7ORDPM5XkF7D1a3ClCV7FyDUV8t7w5JRu/6fn9tpsNjYVVhr1/tF+bW4CZvNxvdvv8iMj9+kf5KRZ2Lz3CKpA+wqrGFfUe25H6hwCyuPVVBntqFVC5cqDeEpdp+sxWSVaH3f+xfdhntkBi9ns1qZ/MIjLJr6BSPTg7mvezQaFx3a+DuBOjUV9Va5w1A4wPqcSt7dmM+iI2Vyh+KxNuVWoff1I6Nzj4tuw67ELoR4SQixSwixQwjxqxAixp72FH9lMZv44Kl7WDNvBte2DuP2jhFu10uK8vchr0pZ5+zuTlSZ+GhzAWmhekalh8gdjkey2CQ25lbTofdAtHaU1ra3x/6GJEltJUlqD8wHnrWzPcVZzKYG3n30TjYtnc8t7cMZ1ybMLZeVpQTrKag2U9mgFNRyVzUmK/9bk4tKCB7rGatMmDaRHQU1VDZY6XHpZXa1Y1dilyTp7GPV/QDXW9PmxjL37WLXumXc2TmSMW58KEObSF/ijD4U1yiJ3V0dKK7jZI2Zx3rFEOHn+quw3FVprYUofy3te/a3qx0hSfblYiHEy8CNQAXQX5Kkon943ARgwqkvWwN77LqwawsDiuUOogl58uvz5NcGyutzd80lSTrnQcXnTOxCiGVA1N/8aJIkSXPPetyTgF6SpOfOeVEhfpck6eK2VLkB5fW5L09+baC8Pnd3vq/vnOvYJUkadJ7XnAosBM6Z2BUKhULRdOxdFZN21pejgQP2haNQKBQKe9m78/RVIURzwAZkAxPP83mT7byuq1Nen/vy5NcGyutzd+f1+uyePFUoFAqFa1F2nioUCoWHURK7QqFQeBjZErsnlyMQQrwhhDhw6vXNEUIEyR2TIwkhrhJC7BVC2IQQHrO0TAgxVAhxUAhxRAjxhNzxOJIQ4ishxEkhhEfuHxFCxAshVgoh9p3627xf7pgcRQihF0JsFkLsPPXaXjjnc+QaYxdCGE/vXBVC3AdkSJJ0vpOvLk0IMQRYIUmSRQjxGoAkSY/LHJbDCCFa0jhh/hnwiCRJv8sckt2EEGrgEDAYyAW2ANdKkrRP1sAcRAjRB6gGvpMkqbXc8TiaECIaiJYkaZsQIgDYCozxhN+faKwj4idJUrUQQgusA+6XJGnjPz1Hth67J5cjkCTpV0mSTu+f3wjEyRmPo0mStF+SpINyx+FgXYEjkiRlSpJkAn6kcQmvR5AkaQ1Qes4HuilJkvIlSdp26v+rgP3AxZ1S4WKkRtWnvtSe+u9f86WsY+xCiJeFEMeB6/HcAmK3AovkDkJxTrHA8bO+zsVDEoO3EUIkAR2ATTKH4jBCCLUQYgdwElgqSdK/vrYmTexCiGVCiD1/899oAEmSJkmSFE/jrtV7mjIWRzvXazv1mEmAhcbX51bO5/UpFK5GCOEPzAIe+NOogFuTJMl6qopuHNBVCPGvw2lNejSeJ5cjONdrE0LcDIwEBkpuuFngAn53niIPiD/r67hT31O4iVPjz7OAqZIkzZY7nqYgSVK5EGIlMJR/KaQo56oYjy1HIIQYCjwGXCZJknImnHvYAqQJIZKFED7AOOAXmWNSnKdTE4xfAvslSXpb7ngcSQgRfnplnRDCQOME/7/mSzlXxcwC/lCOQJIkj+ghCSGOADqg5NS3NnrKih8AIcRY4AMgHCgHdkiSdKmsQTmAEGI48C6gBr6SJOlleSNyHCHENKAfjWVtC4HnJEn6UtagHEgI0QtYC+ymMacAPCVJ0kL5onIMIURb4Fsa/y5VwHRJkl781+e44SiBQqFQKP6FsvNUoVAoPIyS2BUKhcLDKIldoVAoPIyS2BUKhcLDKIldoVAoPIyS2BUKhcLDKIldoVAoPMz/AQnVjCOYuf80AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# https://scikit-learn.org/stable/auto_examples/svm/plot_svm_kernels.html\n",
    "clf = svm.SVC(kernel='rbf', gamma=.5)\n",
    "clf.fit(X, Y)\n",
    "\n",
    "# plot the line, the points, and the nearest vectors to the plane\n",
    "plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=80, facecolors='none', zorder=10, edgecolors='k')\n",
    "plt.scatter(X[:, 0], X[:, 1], c=Y, zorder=10, cmap=plt.cm.Paired, edgecolors='k')\n",
    "XX, YY = np.mgrid[x_min:x_max:200j, y_min:y_max:200j]\n",
    "Z = clf.decision_function(np.c_[XX.ravel(), YY.ravel()])\n",
    "\n",
    "# Put the result into a color plot\n",
    "Z = Z.reshape(XX.shape)\n",
    "plt.pcolormesh(XX, YY, Z > 0, cmap=plt.cm.Paired)\n",
    "plt.contour(XX, YY, Z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'],levels=[-.5, 0, .5])\n",
    "plt.xlim(x_min, x_max)\n",
    "plt.ylim(y_min, y_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# When is $K$ A Kernel?\n",
    "\n",
    "We've seen that for many features $\\phi$ we can define a kernel function $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ that efficiently computes $\\phi(x)^\\top \\phi(x)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Suppose now that we use some kernel function $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ in an ML algorithm. Is there an implicit feature mapping $\\phi$ that corresponds to using K?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Let's start by defining a necessary condition for $K : \\mathcal{X} \\times \\mathcal{X} \\to [0, \\infty]$ to be associated with a feature map."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Suppose that $K$ is a kernel for some feature map $\\phi$, and consider an arbitrary set of $n$ points $\\{x^{(1)}, x^{(2)}, \\ldots, x^{(n)}\\}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Consider the matrix $L \\in \\mathbb{R}^{n\\times n}$ defined as $L_{ij} = K(x^{(i)}, x^{(j)}) = \\phi(x^{(i)})^\\top \\phi(x^{(j)})$. We claim that $L$ must be symmetric and positive semidefinite."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Indeed, it $L$ is symmetric because the dot product $\\phi(x^{(i)})^\\top \\phi(x^{(j)})$ is symmetric. Moreover, for any $z$,\n",
    "\n",
    "\\begin{align*}\n",
    "z^\\top L z\n",
    "& = \\sum_{i=1}^n \\sum_{j=1}^n z_i L_{ij} z_j \n",
    " = \\sum_{i=1}^n \\sum_{j=1}^n z_i \\phi(x^{(i)})^\\top \\phi(x^{(j)}) z_j \\\\\n",
    "& = \\sum_{i=1}^n \\sum_{j=1}^n z_i (\\sum_{k=1}^n \\phi(x^{(i)})_k \\phi(x^{(j)})_k ) z_j \\\\\n",
    "& = \\sum_{k=1}^n \\sum_{i=1}^n \\sum_{j=1}^n z_i \\phi(x^{(i)})_k \\phi(x^{(j)})_k z_j \\\\\n",
    "& = \\sum_{k=1}^n \\sum_{i=1}^n \\left( z_i \\phi(x^{(i)})_k \\right)^2 \\geq 0\n",
    "\\end{align*}\n",
    "\n",
    "Thus if $K$ is a kernel, $L$ must be positive semidefinite for any $n$ points $x^{(i)}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Mercer's Theorem\n",
    "\n",
    "if $K$ is a kernel, $L$ must be positive semidefinite for any set of $n$ points $x^{(i)}$. It turns out that it is is also a sufficient condition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "__Theorem.__ (Mercer) Let $K: \\mathcal{X} \\times \\mathcal{X} \\to [0,\\infty]$ be a kernel function. There exists a mapping $\\phi$ associated with $K$ if for any $n$ and any dataset $\\{x^{(1)}, x^{(2)}, \\ldots, x^{(n)}\\}$ of size $n \\geq 1$, if and only if the matrix $L$ defined as $L_{ij} = K(x^{(i)}, x^{(j)})$ is symmetric and positive semidefinite.\n",
    "\n",
    "This characterizes precisely which kernel functions correspond to some $\\phi$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pros and Cons of Kernels\n",
    "\n",
    "Are kernels a free lunch? Not quite.\n",
    "* Kernels allow us to use features $\\phi$ of very large dimension $d$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* However computation is at least $O(n^2)$, where $n$ is the dataset size. We need to compute distances $K(x^{(i)}, x^{(j)})$, for all $i,j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Approximate solutions can be found more quickly, but in practice kernel methods are not used with today's massive datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* However, on small and medium-sized data, kernel methods will be at least as good as neural nets and probably much easier to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary: Kernels\n",
    "\n",
    "* A kernel is a function $K: \\mathcal{X} \\times \\mathcal{X} \\to [0,\\infty]$ that defines a notion of similarity over pairs of vectors in $\\mathcal{X}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Kernels are often associated with high-dimensional features $\\phi$ and implicitly map inputs to this feature space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Kernels can be incorporated into many machine learning algorithms, which enables them to learn highly nonlinear models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Examples of algorithms in which we can use kernels include:\n",
    "* Supervised learning algorithms: linear regression, logistic regression, support vector machines, etc.\n",
    "* Unsupervised learning algorithms: PCA, density estimation.\n",
    "\n",
    "Kernels are very powerful because they can be used throughout machine learning. "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "celltoolbar": "Slideshow",
  "colab": {
   "collapsed_sections": [],
   "name": "neural-ode.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "rise": {
   "controlsTutorial": false,
   "enable_chalkboard": true,
   "height": 900,
   "help": false,
   "margin": 0,
   "maxScale": 2,
   "minScale": 0.2,
   "progress": true,
   "scroll": true,
   "theme": "simple",
   "width": 1200
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}