slides/lecture07-naive-bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i_f5u2x9nn6I",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "\n",
    "# Lecture 7: Naive Bayes\n",
    "\n",
    "### Applied Machine Learning\n",
    "\n",
    "__Volodymyr Kuleshov__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Announcements\n",
    "\n",
    "+ Project proposals due Wed\n",
    "+ Homework 2 is out tonight, see also the late day policy\n",
    "+ Canvas Discussions should now be working "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Part 1: Text Classification\n",
    "\n",
    "We will now do a quick detour to talk about an important application area of machine learning: text classification. \n",
    "\n",
    "Afterwards, we will see how text classification motivates new classification algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Classification\n",
    "\n",
    "Consider a training dataset $\\mathcal{D} = \\{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \\ldots, (x^{(n)}, y^{(n)})\\}$.\n",
    "\n",
    "We distinguish between two types of supervised learning problems depending on the targets $y^{(i)}$. \n",
    "\n",
    "1. __Regression__: The target variable $y \\in \\mathcal{Y}$ is continuous:  $\\mathcal{Y} \\subseteq \\mathbb{R}$.\n",
    "2. __Classification__: The target variable $y$ is discrete and takes on one of $K$ possible values:  $\\mathcal{Y} = \\{y_1, y_2, \\ldots y_K\\}$. Each discrete value corresponds to a *class* that we want to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text Classification\n",
    "\n",
    "An interesting instance of a classification problem is classifying text.\n",
    "* **Many applications:** spam filtering, fraud detection, medical record classification.\n",
    "* **Inputs** $x$ are sequences of words of an arbitrary length.\n",
    "* **Limitation:** Our algorithms take as input column vectors of a fixed dimension.\n",
    "\n",
    "*What do we do?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Feature Representations for Text\n",
    "\n",
    "Each data point $x$ in this dataset is a sequence of characters of an arbitrary length.\n",
    "\n",
    "How do we transform these into $d$-dimensional features $\\phi(x)$ that can be used with our machine learning algorithms?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* We may devise hand-crafted features by inspecting the data:\n",
    "    * Does the message contain the word \"church\"? Does the email of the user originate outside the United States? Is the organization a university? etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* We can count the number of occurrences of each word:\n",
    "    * Does this message contain \"Aardvark\", yes or no?\n",
    "    * Does this message contain \"Apple\", yes or no?\n",
    "    * ... Does this message contain \"Zebra\", yes or no?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* Finally, many modern deep learning methods can directly work with sequences of characters of an arbitrary length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bag of Words Representations\n",
    "\n",
    "A widely-used approach to representing text documents is called \"*bag of words*\".\n",
    "\n",
    "We start by defining a **vocabulary** $V$ containing all the possible words we are interested in, e.g.:\n",
    "$$ V = \\{\\text{church}, \\text{doctor}, \\text{fervently}, \\text{purple}, \\text{slow}, ...\\} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A bag of words representation of a document $x$ is a function $\\phi(x) \\to \\{0,1\\}^{|V|}$ that outputs a feature vector\n",
    "$$\n",
    "\\phi(x) = \\left( \n",
    "\\begin{array}{c}\n",
    "0 \\\\\n",
    "1 \\\\\n",
    "0 \\\\\n",
    "\\vdots \\\\\n",
    "1 \\\\\n",
    "\\vdots \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "\\begin{array}{l}\n",
    "\\;\\text{church} \\\\\n",
    "\\;\\text{doctor} \\\\\n",
    "\\;\\text{fervently} \\\\\n",
    "\\vdots \\\\\n",
    "\\;\\text{purple} \\\\\n",
    "\\vdots \\\\\n",
    "\\end{array}\n",
    "$$\n",
    "of dimension $V$. The $j$-th component $\\phi(x)_j$ equals $1$ if $x$ contains the $j$-th word in $V$ and $0$ otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification Dataset: Twenty Newsgroups\n",
    "\n",
    "To illustrate the text classification problem, we will use a popular dataset called `20-newsgroups`. \n",
    "* It contains ~20,000 documents collected approximately evenly from 20 different online newsgroups.\n",
    "* Each newgroup covers a different topic such as medicine, computer graphics, or religion.\n",
    "* This dataset is often used to benchmark text classification and other types of algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's load this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       ".. _20newsgroups_dataset:\n",
       "\n",
       "The 20 newsgroups text dataset\n",
       "------------------------------\n",
       "\n",
       "The 20 newsgroups dataset comprises around 18000 newsgroups posts on\n",
       "20 topics split in two subsets: one for training (or development)\n",
       "and the other one for testing (or for performance evaluation). The split\n",
       "between the train and test set is based upon a messages posted before\n",
       "and after a specific date.\n",
       "\n",
       "This module contains two loaders. The first one,\n",
       ":func:`sklearn.datasets.fetch_20newsgroups`,\n",
       "returns a list of the raw texts that can be fed to text feature\n",
       "extractors such as :class:`~sklearn.feature_extraction.text.CountVectorizer`\n",
       "with custom parameters so as to extract feature vectors.\n",
       "The second one, :func:`sklearn.datasets.fetch_20newsgroups_vectorized`,\n",
       "returns ready-to-use features, i.e., it is not necessary to use a feature\n",
       "extractor.\n",
       "\n",
       "**Data Set Characteristics:**\n",
       "\n",
       "    =================   ==========\n",
       "    Classes                     20\n",
       "    Samples total            18846\n",
       "    Dimensionality               1\n",
       "    Features                  text\n",
       "    =================   ==========\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html\n",
    "    \n",
    "from IPython.display import Markdown, display\n",
    "import numpy as np; np.set_printoptions(precision=2)\n",
    "import pandas as pd; pd.options.display.float_format = \"{:,.2f}\".format\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "# for this lecture, we will restrict our attention to just 4 different newsgroups:\n",
    "categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']\n",
    "\n",
    "# load the dataset\n",
    "twenty_train = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)\n",
    "\n",
    "# print some information on it\n",
    "Markdown(twenty_train.DESCR[:1088])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The set of targets in this dataset are the newgroup topics:\n",
    "twenty_train.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: s0612596@let.rug.nl (M.M. Zwart)\n",
      "Subject: catholic church poland\n",
      "Organization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\n",
      "Lines: 10\n",
      "\n",
      "Hello,\n",
      "\n",
      "I'm writing a paper on the role of the catholic church in Poland after 1989. \n",
      "Can anyone tell me more about this, or fill me in on recent books/articles(\n",
      "in english, german or french). Most important for me is the role of the \n",
      "church concerning the abortion-law, religious education at schools,\n",
      "birth-control and the relation church-state(government). Thanx,\n",
      "\n",
      "                                                 Masja,\n",
      "\"M.M.Zwart\"<s0612596@let.rug.nl>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's examine one data point\n",
    "print(twenty_train.data[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2257\n"
     ]
    }
   ],
   "source": [
    "# We have about 2k data points in total\n",
    "print(len(twenty_train.data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's see an example of Bag-of-Words on `20-newsgroups`.\n",
    "\n",
    "We start by computing these features using the `sklearn` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35788)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# vectorize the training set\n",
    "count_vect = CountVectorizer(binary=True)\n",
    "X_train = count_vect.fit_transform(twenty_train.data)\n",
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `sklearn` count vectorizer gives us mappings between words and indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word to index `.vocabulary`:  {'from': 14887, 'sd345': 29022, 'city': 8696, 'ac': 4017, 'uk': 33256, 'michael': 21661, 'coll ...\n",
      "Index to word:  ['00' '000' '0000' '0000001200' '000005102000' '0001' '000100255pixel'\n",
      " '00014' '000406' '0007']\n"
     ]
    }
   ],
   "source": [
    "print('Word to index `.vocabulary`: ', str(count_vect.vocabulary_)[:94] + ' ...')\n",
    "feature_names = count_vect.get_feature_names_out()\n",
    "print('Index to word: ', feature_names[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can retrieve the index of $\\phi(x)$ associated with each `word` using these mappings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index for the word \"church\":  8609\n",
      "Index for the word \"computer\":  9338\n",
      "Word for the index '8609':  church\n",
      "Word for the index '10000':  counseling\n"
     ]
    }
   ],
   "source": [
    "# The CountVectorizer class records the index j associated with each word in V\n",
    "print('Index for the word \"church\": ', count_vect.vocabulary_.get(u'church'))\n",
    "print('Index for the word \"computer\": ', count_vect.vocabulary_.get(u'computer'))\n",
    "\n",
    "# And we can map the indices back to words\n",
    "print(\"Word for the index '8609': \", feature_names[8609])\n",
    "print(\"Word for the index '10000': \", feature_names[10000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Our featurized dataset is in the matrix `X_train`. We can use the above indices to retrieve the 0-1 value that has been computed for each word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From: s0612596@let.rug.nl (M.M. Zwart)\n",
      "Subject: catholic church poland\n",
      "Organization: Faculteit der Letteren, Rijksuniversiteit Groningen, NL\n",
      "Lines: 10\n",
      "\n",
      "Hello,\n",
      "\n",
      "I'm writing a paper on the role of the catholic church in Poland after 1989. \n",
      "Can anyone tell me more about this, or fill me in on recent books/articles(\n",
      "in english, german or french). Most important for me is the role of the \n",
      "church concerning the abortion-law, religious education at schools,\n",
      "birth-control and the relation church-state(government). Thanx,\n",
      "\n",
      "                                                 Masja,\n",
      "\"M.M.Zwart\"<s0612596@let.rug.nl>\n",
      "\n",
      "------------------------------------------------------------\n",
      "Value at the index for the word \"church\":  1\n",
      "Value at the index for the word \"computer\":  0\n",
      "Value at the index for the word \"doctor\":  0\n",
      "Value at the index for the word \"important\":  1\n"
     ]
    }
   ],
   "source": [
    "# We can examine if any of these words are present in our previous datapoint\n",
    "print(twenty_train.data[3])\n",
    "\n",
    "# let's see if it contains these two words?\n",
    "print('---'*20)\n",
    "print('Value at the index for the word \"church\": ', X_train[3, count_vect.vocabulary_.get(u'church')])\n",
    "print('Value at the index for the word \"computer\": ', X_train[3, count_vect.vocabulary_.get(u'computer')])\n",
    "print('Value at the index for the word \"doctor\": ', X_train[3, count_vect.vocabulary_.get(u'doctor')])\n",
    "print('Value at the index for the word \"important\": ', X_train[3, count_vect.vocabulary_.get(u'important')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Practical Considerations\n",
    "\n",
    "In practice, we may use some additional modifications of this technique:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Sometimes, the feature $\\phi(x)_j$ for the $j$-th word holds the count of occurrences of word $j$ instead of just the binary occurrence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* The raw text is usually preprocessed. One common technique is *stemming*, in which we only keep the root of the word.\n",
    "    * e.g. \"slowly\", \"slowness\", both map to \"slow\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Filtering for common *stopwords* such as \"the\", \"a\", \"and\". Similarly, rare words are also typically excluded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Classification Using BoW Features\n",
    "\n",
    "Let's now have a look at the performance of classification over bag of words features.\n",
    "\n",
    "Now that we have a feature representation $\\phi(x)$, we can apply the classifier of our choice, such as logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      " This problem is unconstrained.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RUNNING THE L-BFGS-B CODE\n",
      "\n",
      "           * * *\n",
      "\n",
      "Machine precision = 2.220D-16\n",
      " N =       143156     M =           10\n",
      "\n",
      "At X0         0 variables are exactly at the bounds\n",
      "\n",
      "At iterate    0    f=  3.12887D+03    |proj g|=  2.12500D+02\n",
      "\n",
      "           * * *\n",
      "\n",
      "Tit   = total number of iterations\n",
      "Tnf   = total number of function evaluations\n",
      "Tnint = total number of segments explored during Cauchy searches\n",
      "Skip  = number of BFGS updates skipped\n",
      "Nact  = number of active bounds at final generalized Cauchy point\n",
      "Projg = norm of the final projected gradient\n",
      "F     = final function value\n",
      "\n",
      "           * * *\n",
      "\n",
      "   N    Tit     Tnf  Tnint  Skip  Nact     Projg        F\n",
      "*****     38     43      1     0     0   9.100D-05   1.095D-02\n",
      "  F =   1.0948389171855248E-002\n",
      "\n",
      "CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL            \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.6s finished\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=100000.0, multi_class=&#x27;multinomial&#x27;, verbose=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=100000.0, multi_class=&#x27;multinomial&#x27;, verbose=True)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=100000.0, multi_class='multinomial', verbose=True)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Create an instance of Softmax and fit the data.\n",
    "logreg = LogisticRegression(C=1e5, multi_class='multinomial', verbose=True)\n",
    "logreg.fit(X_train, twenty_train.target) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "And now we can use this model for predicting on new inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'God is love' => soc.religion.christian\n",
      "'OpenGL on the GPU is fast' => comp.graphics\n"
     ]
    }
   ],
   "source": [
    "docs_new = ['God is love', 'OpenGL on the GPU is fast']\n",
    "\n",
    "X_new = count_vect.transform(docs_new)\n",
    "predicted = logreg.predict(X_new)\n",
    "\n",
    "for doc, category in zip(docs_new, predicted):\n",
    "    print('%r => %s' % (doc, twenty_train.target_names[category]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary of Text Classification\n",
    "\n",
    "* Classifying text normally requires specifying features over the raw data.\n",
    "* A widely used representation is \"bag of words\", in which features are occurrences or counts of words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Once text is featurized, any off-the-shelf supervised learning algorithm can be applied, but some work better than others, as we will see next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "# Part 2: Generative Classifers for Text\n",
    "\n",
    "In this lecture, we are going to look at generative classifiers and their applications to text classification.\n",
    "\n",
    "<!-- We will start by defining the concept of a generative *model*. -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Supervised Learning Models\n",
    "\n",
    "A supervised learning model can be seen as directly predicting a one-hot embedding of the target class:\n",
    "\n",
    "$$\n",
    "\\underbrace{\\left[ \n",
    "\\begin{array}{c}\n",
    "x_1 \\\\\n",
    "x_2 \\\\\n",
    "x_3 \\\\\n",
    "x_4 \\\\\n",
    "\\end{array}\n",
    "\\right]}_\\text{input}\n",
    "{\\to}\\text{model $f_\\theta(x)$}{\\to}\n",
    "\\underbrace{\\left[ \n",
    "\\begin{array}{c}\n",
    "0 \\\\\n",
    "1 \\\\\n",
    "0 \\\\\n",
    "\\end{array}\n",
    "\\right]}_\\text{output}\n",
    "$$\n",
    "\n",
    "In this example, the $f_\\theta(x)$ predicts the second class for input $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Probabilistic Models\n",
    "\n",
    "A probabilistic model outputs a vector of class probabilities\n",
    "\n",
    "$$\n",
    "\\underbrace{\\left[ \n",
    "\\begin{array}{c}\n",
    "x_1 \\\\\n",
    "x_2 \\\\\n",
    "x_3 \\\\\n",
    "x_4 \\\\\n",
    "\\end{array}\n",
    "\\right]}_\\text{input}\n",
    "{\\to}\\text{model $P_\\theta(y|x)$}{\\to}\n",
    "\\underbrace{\\left[ \n",
    "\\begin{array}{c}\n",
    "0.1 \\\\\n",
    "0.7 \\\\\n",
    "0.2 \\\\\n",
    "\\end{array}\n",
    "\\right]}_\\text{output}\n",
    "$$\n",
    "\n",
    "Here, the $P_\\theta(y|x)$ still predicts the second class for input $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Logistic Regression\n",
    "\n",
    "For example, a logistic (softmax) model outputs a probability distribution \n",
    "\n",
    "$$\n",
    "P_\\theta(y| x) \n",
    "= \\left[ \\begin{array}{c}\n",
    "P_\\theta(y=0| x) \\\\\n",
    "P_\\theta(y=1| x) \\\\\n",
    "\\end{array} \\right]\n",
    "= \\left[ \\begin{array}{c}\n",
    "1-\\sigma(\\theta^\\top x) \\\\\n",
    "\\sigma(\\theta^\\top x) \\\\\n",
    "\\end{array} \\right]\n",
    "$$\n",
    "\n",
    "where $\\theta^\\top x$ is a linear model and\n",
    "$$\\sigma(z) = \\frac{1}{1 + \\exp(-z)}$$\n",
    "is the *sigmoid* or *logistic* function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Discriminative vs. Generative Classifers\n",
    "\n",
    "Logistic regression is an example of a *discriminative* machine learning classifier.\n",
    "$$\n",
    "P_\\theta(y| x) \n",
    "= \\left[ \\begin{array}{c}\n",
    "P_\\theta(y=0| x) \\\\\n",
    "P_\\theta(y=1| x) \\\\\n",
    "\\end{array} \\right]\n",
    "$$\n",
    "* It directly transforms $x$ into a score for each class $y$ (e.g., via the formula $\\sigma(\\theta^\\top x)$)\n",
    "* It can be interpreted as defining a *conditional* probability $P_\\theta(y|x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Generative classifier instead define a __joint__ distribution $P_\\theta(x,y)$ over $x,y$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generative Classifiers: Intuition\n",
    "\n",
    "Imagine we have a dataset of, e.g., flowers. We fit a Gaussian to each class:\n",
    "\n",
    "<center><img width=50% src=\"img/gda_example.png\"></center>\n",
    "\n",
    "At test time, we predict the class of the Gaussian that is most likely to have generated the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example: Gaussian Discriminant Analysis\n",
    "\n",
    "Suppose we want to train a classifier over Iris flowers. In GDA, we define __Gaussian__ probabilities:\n",
    "\n",
    "\\begin{align*}\n",
    "P_\\theta(x|y=\\text{setosa}) && P_\\theta(x|y=\\text{non-setosa})\n",
    "\\end{align*}\n",
    "\n",
    "as well as prior probabilities $P_\\theta(y=\\text{setosa}), P_\\theta(y=\\text{non-setosa})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Each Gaussian $P_\\theta(x|y=\\text{setosa/non-setosa})$ matches the mean and variance of the class. The $P(y)$ measures the frequency of each class $y$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Predictions From Generative Classifiers\n",
    "\n",
    "Given a new $x'$, we would compare the probabilities of both models:\n",
    "\n",
    "\\begin{align*}\n",
    "P_\\theta(x'|y=\\text{0})P_\\theta(y=\\text{0}) && \\text{vs.} && P_\\theta(x'|y=\\text{1})P_\\theta(y=\\text{1})\n",
    "\\end{align*}\n",
    "\n",
    "We output the class that's more likely to have generated $x'$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Formally, given a new $x'$, we return the most likely class to have generated it:\n",
    "\n",
    "\\begin{align*}\n",
    "\\arg \\max_k P_\\theta(y=k | x') & = \\arg \\max_k  \\frac{P_\\theta(x' | y=k) P_\\theta(y=k)}{P_\\theta(x')} \\\\\n",
    "& = \\arg \\max_k P_\\theta(x' | y=k) P_\\theta(y=k),\n",
    "\\end{align*}\n",
    "\n",
    "where we have applied Bayes' rule in the first line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Generative Text Classifier\n",
    "\n",
    "Can we apply GDA to text classification?\n",
    "\n",
    "* In Iris flower classification, the data $x$ is real-valued, hence we fit a Gaussian to it.\n",
    "* When $x$ is discrete, we want to choose a more appropriate distribution (this lecture!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Maximum Likelihood Learning\n",
    "\n",
    "We can learn a generative model $P_\\theta(x, y)$ by maximizing the *likelihood*:\n",
    "\n",
    "$$ \\max_\\theta \\frac{1}{n}\\sum_{i=1}^n \\log P_\\theta(x={x}^{(i)}, y=y^{(i)}). $$\n",
    "\n",
    "This says that we should choose parameters $\\theta$ such that the model $P_\\theta$ assigns a high probability to each training example $(x^{(i)}, y^{(i)})$ in the dataset $\\mathcal{D}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "How did we get this formula? The likelihood of the data under is defined as:\n",
    "\n",
    "$$\n",
    "L(\\theta) = \\prod_{i=1}^n P_\\theta({x}^{(i)}, y^{(i)})\n",
    "$$\n",
    "\n",
    "This is the probability of observing the dataset $\\mathcal{D} = \\{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \\ldots, (x^{(n)}, y^{(n)})\\}$ if each training instance were to be an independent sample from $P_\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The principle of maximum likelihood says that we should find model parameters $\\theta$ that yield high likelihood.\n",
    "\n",
    "In other words, this means that we want to find $\\theta$ such that the training dataset $\\mathcal{D}$ has a high probability under the probability distribution $P_\\theta$ induced by the model. To put it differently, the model $P_\\theta$ is likely to have generated the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "A lot of mathematical derivations and numerical calculations become simpler if we instead maximize the log of the likelihood:\n",
    "\\begin{align*}\n",
    "\\ell(\\theta) = \\log L(\\theta) & = \\log \\prod_{i=1}^n P_\\theta({x}^{(i)}, y^{(i)}) = \\sum_{i=1}^n \\log P_\\theta({x}^{(i)}, y^{(i)}).\n",
    "\\end{align*}\n",
    "Note that since the log is a monotonically increasing function, it doesn't change the optimal $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Finally, it's often simpler to take the average, instead of the sum. This gives us the following learning principle, known as *maximum log-likelihood*:\n",
    "$$\n",
    "\\max_\\theta \\ell(\\theta) = \\max_{\\theta} \\frac{1}{n}\\sum_{i=1}^n \\log P_\\theta({x}^{(i)}, y^{(i)}).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "# Part 3: Naive Bayes\n",
    "\n",
    "Next, we are going to look at Naive Bayes, a generative classification algorithm. \n",
    "\n",
    "We will apply Naive Bayes to the text classification problem.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Review: Text Classification\n",
    "\n",
    "An common type of classification problem is classifying text.\n",
    "* Includes a lot applied problems: spam filtering, fraud detection, medical record classification, etc.\n",
    "* Inputs $x$ are sequences of words of an arbitrary length.\n",
    "* The dimensionality of text inputs is usually very large, proportional to the size of the vocabulary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A Generative Model for Text Classification\n",
    "\n",
    "In binary text classification, we fit two models on a labeled corpus:\n",
    "\\begin{align*}\n",
    "P_\\theta(x|y=\\text{0}) && \\text{and} && P_\\theta(x|y=\\text{1})\n",
    "\\end{align*}\n",
    "<!-- We also define priors $P_\\theta(y=\\text{0}), P_\\theta(y=\\text{1})$. -->\n",
    "\n",
    "Each model $P_\\theta(x | y=k)$ *scores* $x$ based on how much it looks like class $k$.\n",
    "The documents $x$ are in __bag-of-words__ representation.\n",
    "\n",
    "How do we choose $P_\\theta(x|y=k)$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: Categorical Distribution\n",
    "\n",
    "A [Categorical](https://en.wikipedia.org/wiki/Categorical_distribution) distribution with parameters $\\theta$ is a probability\n",
    "over $K$ discrete outcomes $x \\in \\{1,2,...,K\\}$:\n",
    "\n",
    "$$\n",
    "P_\\theta(x = j) = \\theta_j.\n",
    "$$\n",
    "\n",
    "When $K=2$ this is called the [Bernoulli](https://en.wikipedia.org/wiki/Bernoulli_distribution)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# First Attempt at a Generative Model\n",
    "\n",
    "<!-- Note there is a finite number of $x$'s: each is a binary vector of size $d$. -->\n",
    "\n",
    "A first solution is to assume that $P(x|y=\\text{spam})$ is a categorical distribution that assigns a probability to each possible word $x'=[0,1,0,...,0]$:\n",
    "$$\n",
    "P(x=x'|y=\\text{spam}) = P \\left( \n",
    "\\begin{array}{c}\n",
    "0 \\\\\n",
    "1 \\\\\n",
    "0 \\\\\n",
    "\\vdots \\\\\n",
    "0 \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\left.\n",
    "\\begin{array}{l}\n",
    "\\;\\text{church} \\\\\n",
    "\\;\\text{doctor} \\\\\n",
    "\\;\\text{fervently} \\\\\n",
    "\\vdots \\\\\n",
    "\\;\\text{purple}\n",
    "\\end{array}\n",
    "\\right) = \\theta_{x'} = 0.0012\n",
    "$$\n",
    "The $\\theta_{x'}$ is the probability of $x'$ under class $\\text{spam}$. \n",
    "We need to specify $\\theta_{x}$ for all $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Problem: High Dimensionality\n",
    "\n",
    "How many parameters does a Categorical model $P(x|y=\\text{spam})$ have?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* If the dimensionality $d$ of $x$ is high (e.g., vocabulary has size 10,000), $x$ can take a huge number of values ($2^{10000}$ in our example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* We need to specify $2^{d}-1$ parameters for the Categorical distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For comparison, there are $\\approx 10^{82}$ atoms in the universe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Naive Bayes Assumption \n",
    "The Naive Bayes assumption is a __general technique__ that can be used with any $d$-dimensional $x$ to construct tractable models $P_\\theta(x|y)$ as follows:\n",
    "\n",
    "$$\n",
    " P_{\\theta} \\left( x=\n",
    "\\begin{array}{c}\n",
    "0 \\\\\n",
    "1 \\\\\n",
    "0 \\\\\n",
    "\\vdots \\\\\n",
    "0 \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\left.\n",
    "\\begin{array}{l}\n",
    "\\;\\text{church} \\\\\n",
    "\\;\\text{doctor} \\\\\n",
    "\\;\\text{fervently} \\\\\n",
    "\\vdots \\\\\n",
    "\\;\\text{purple}\n",
    "\\end{array}\n",
    "\\right) = P_\\theta(x_1=0) \\cdot P_\\theta(x_2=1) \\cdot ... \\cdot P_\\theta(x_d=0)\n",
    "$$\n",
    "\n",
    "Each $P_\\theta(x_j=0)$ is a Bernoulli. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How many parameters do we need to specify this probability?\n",
    "\n",
    "$$\n",
    " P_{\\theta} \\left( x=\n",
    "\\begin{array}{c}\n",
    "0 \\\\\n",
    "1 \\\\\n",
    "0 \\\\\n",
    "\\vdots \\\\\n",
    "0 \n",
    "\\end{array}\n",
    "\\right.\n",
    "\\left.\n",
    "\\begin{array}{l}\n",
    "\\;\\text{church} \\\\\n",
    "\\;\\text{doctor} \\\\\n",
    "\\;\\text{fervently} \\\\\n",
    "\\vdots \\\\\n",
    "\\;\\text{purple}\n",
    "\\end{array}\n",
    "\\right) = P_\\theta(x_1=0) \\cdot P_\\theta(x_2=1) \\cdot ... \\cdot P_\\theta(x_d=0)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This typically makes the number of parameters linear instead of exponential in $d$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Formally, the Naive Bayes assumptions is defined as follows:\n",
    "<!-- The Naive Bayes assumption is a __general technique__ that can be used with any $d$-dimensional $x$ to construct tractable models $P_\\theta(x|y)$. -->\n",
    "* We define the probability that $x$ takes value $x'$ to be a product of independent terms (one for each attribute $x_j$):\n",
    "$$ P_\\theta(x=x'|y) = \\prod_{j=1}^d P_\\theta(x_j = x_j' \\mid y) $$\n",
    "* Each $P_\\theta(x_j = x_j' \\mid y)$ is typically defined to be a simple distribution like a Bernoulli."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Naive Bayes Assumption for Bag of Words \n",
    "\n",
    "To deal with high-dimensional $x$, we choose a simpler model for $P_\\theta(x|y)$:\n",
    "1. We define the model $P_\\theta(x|y=k)$ for documents $x$ as the product of the occurrence probabilities of each of its words $x_j$:\n",
    "    $$ P_\\theta(x = x'|y=k) = \\prod_{j=1}^d P_\\theta(x_j = x_j' \\mid y=k) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "2. We define a (Bernoulli) model with one parameter $\\psi_{jk} \\in [0,1]$ for the occurrence of each word $j$ in class $k$:\n",
    "    $$P_\\theta(x_j = 1 \\mid y=k) = \\psi_{jk} \\;\\;\\;\\;\\; P_\\theta(x_j = 0 \\mid y=k) = 1-\\psi_{jk}$$\n",
    "    $\\psi_{jk}$ is the probability that a document of class $k$ contains word $j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How many parameters does this new model have?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* We have $K$ distributions of the form $P_\\theta(x|y=k)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* The distribution $P_\\theta(x|y=k) = \\prod_{j=1}^d P_\\theta(x_j \\mid y=k)$ is the product of $d$ such one-parameter distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* We have a distribution $P_\\theta(x_j = 1 \\mid y=k)$ for each word $j$ and each distribution has one parameter $\\psi_{jk}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Thus, we only need $Kd$ parameters instead of $K(2^d-1)$!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Is Naive Bayes a Good Assumption?\n",
    "\n",
    "Naive Bayes assumes that words are uncorrelated, but in reality they are.\n",
    "  * If spam email contains \"bank\", it probably contains \"account\"\n",
    "  \n",
    "As a result, the probabilities estimated by Naive Bayes can be over- under under-confident."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In practice, however, Naive Bayes is a very useful assumption that gives very good classification accuracy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Defining Prior Distributions \n",
    "\n",
    "A full generative model still requires defining the distribution $P_\\theta(y=k)$.\n",
    "* This encodes our prior belief about $y$ before we see $x$.\n",
    "* It can also be learned from data.\n",
    "\n",
    "Since we have a small number of classes $K$, we may use a Categorical distribution with parameters $\\vec\\phi = (\\phi_1,...,\\phi_K)$ and learn $\\vec\\phi$ from data:\n",
    "\n",
    "$$ P_\\theta(y=k) = \\phi_k.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Bernoulli Naive Bayes Model\n",
    "\n",
    "The *Bernoulli Naive Bayes* model $P_\\theta(x,y)$ is defined for *binary data* $x \\in \\{0,1\\}^d$ (e.g., bag-of-words documents).\n",
    "\n",
    "The $\\theta$ contains prior parameters $\\vec\\phi = (\\phi_1,...,\\phi_K)$ and $K$ sets of per-class parameters $\\psi_k = (\\psi_{1k},...,\\psi_{dk})$.\n",
    "\n",
    "* The probability of the data $x$ for each class equals\n",
    "$$P_\\theta(x|y=k) = \\prod_{j=1}^d P(x_j \\mid y=k),$$\n",
    "where each $P_\\theta(x_j \\mid y=k)$ is a $\\text{Bernoulli}(\\psi_{jk})$.\n",
    "\n",
    "* The probability over $y$ is Categorical:\n",
    "$P_\\theta(y=k) = \\phi_k$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Formally, we have:\n",
    "\\begin{align*}\n",
    "P_\\theta(y) & = \\text{Categorical}(\\phi_1,\\phi_2,\\ldots,\\phi_K) \\\\\n",
    "P_\\theta(x_j=1|y=k) & = \\text{Bernoulli}(\\psi_{jk}) \\\\\n",
    "P_\\theta(x|y=k) & = \\prod_{j=1}^d P_\\theta(x_j|y=k)\n",
    "\\end{align*}\n",
    "The parameters of the model are $\\theta = (\\phi_1,...,\\phi_K, \\psi_{11}, ...,\\psi_{dK})$.\n",
    "There are exactly $K(d+1)$ parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "# Part 4: Naive Bayes: Learning\n",
    "\n",
    "We will now turn our attention to learning the parameters of the Naive Bayes model and using them to make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The Parameters of a Naive Bayes Model\n",
    "\n",
    "We need to learn the parameters of two sets of distributions:\n",
    "* The distribution over classes is [Categorical](https://en.wikipedia.org/wiki/Categorical_distribution), denoted $\\text{Categorical}(\\phi_1, \\phi_2, ..., \\phi_K)$. Thus, $P_\\theta(y=k) = \\phi_k$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* The conditional probability $P(x\\mid y=k)$ of the data under class $k$ is a product of Bernoullis $$P_\\theta(x|y=k) = \\prod_{j=1}^d P(x_j \\mid y=k),$$\n",
    "where each $P_\\theta(x_j \\mid y=k)$ is a $\\text{Bernoulli}(\\psi_{jk})$.\n",
    "<!-- with parameter (i.e., probability that $x_j=1$ given class $k$) of $\\psi_{jk}$. -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning the Parameters $\\phi$\n",
    "\n",
    "Consider learning $\\vec \\phi = (\\phi_1, \\phi_2, \\ldots, \\phi_K)$. \n",
    "* We have $n$ datapoints. Each point has a label $k\\in\\{1,2,...,K\\}$.\n",
    "* Our model is a categorical and assigns a probability $\\phi_k$ to each outcome $k\\in\\{1,2,...,K\\}$.\n",
    "* We want to infer $\\phi_k$ assuming our dataset is sampled from the model.\n",
    "\n",
    "What are the maximum likelihood $\\phi_k$ that are most likely to have generated our data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Intuitively, the class probabilities $\\phi$ should just be the class proportions that we see in the data. \n",
    "$$ \\phi_k = \\frac{n_k}{n}$$\n",
    "for each $k$, where $n_k = |\\{i : y^{(i)} = k\\}|$ is the number of training targets with class $k$.\n",
    "\n",
    "Thus, the optimal $\\phi_k$ is just the proportion of data points with class $k$ in the training set!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning the Parameters $\\psi_{jk}$\n",
    "\n",
    "Next we want to learn the parameter $\\psi_{jk}$ (i.e., probability that $x_j=1$ given class $k$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* For each $k$, consider all the inputs $x$ for which $y=k$.\n",
    "* We seek  the probability $\\psi_{jk}$ of a word $j$ being present in an $x$.\n",
    "\n",
    "What is the optimal $\\psi_{jk}$ in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each $\\psi_{jk}$ is simply the proportion of documents in class $k$ that contain the word $j$.\n",
    "\\begin{align*}\n",
    "\\psi_{jk} = \\frac{n_{jk}}{n_k}.\n",
    "\\end{align*}\n",
    "where $|\\{i : x^{(i)}_j = 1 \\text{ and } y^{(i)} = k\\}|$ is the number of $x^{(i)}$ with label $k$ and a positive occurrence of word $j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Querying the Model\n",
    "\n",
    "How do we ask the model for predictions? As discussed earlier, we can apply Bayes' rule:\n",
    "$$\\arg\\max_y P_\\theta(y|x) = \\arg\\max_y P_\\theta(x|y)P(y).$$\n",
    "Thus, we can estimate the probability of $x$ and under each $P_\\theta(x|y=k)P(y=k)$ and choose the class that explains the data best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification Dataset: Twenty Newsgroups\n",
    "\n",
    "To illustrate the text classification problem, we will use a popular dataset called `20-newsgroups`. \n",
    "* It contains ~20,000 documents collected approximately evenly from 20 different online newsgroups.\n",
    "* Each newgroup covers a different topic such as medicine, computer graphics, or religion.\n",
    "* This dataset is widely used to benchmark text classification and other types of algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's load this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "# for this lecture, we will restrict our attention to just 4 different newsgroups:\n",
    "categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']\n",
    "twenty_train = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)\n",
    "Markdown(twenty_train.DESCR[:1088]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Example: Text Classification\n",
    "\n",
    "Let's see how this approach can be used in practice on the text classification dataset.\n",
    "* We will learn a good set of parameters for a Bernoulli Naive Bayes model\n",
    "* We will compare the outputs to the true predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's see an example of Naive Bayes on `20-newsgroups`.\n",
    "\n",
    "We start by computing these features using the `sklearn` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 1000)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# vectorize the training set\n",
    "count_vect = CountVectorizer(binary=True, max_features=1000)\n",
    "y_train = twenty_train.target\n",
    "X_train = count_vect.fit_transform(twenty_train.data).toarray()\n",
    "feature_names = count_vect.get_feature_names_out()\n",
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's compute the maximum likelihood model parameters on our dataset. This is done in closed-form by just estimating statistics on the data, so we don't need to do gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.21 0.26 0.26 0.27]\n"
     ]
    }
   ],
   "source": [
    "n = X_train.shape[0] # size of the dataset\n",
    "d = X_train.shape[1] # number of features in our dataset\n",
    "K = 4 # number of clases\n",
    "\n",
    "# these are the shapes of the parameters\n",
    "psis = np.zeros([K,d])\n",
    "phis = np.zeros([K])\n",
    "\n",
    "# we now compute the parameters\n",
    "for k in range(K):\n",
    "    X_k = X_train[y_train == k]\n",
    "    psis[k] = np.mean(X_k, axis=0)\n",
    "    phis[k] = X_k.shape[0] / float(n)\n",
    "\n",
    "# print out the class proportions\n",
    "print(phis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The learned parameters contain information about the **most frequent words**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# top 10 ~unique words occurring in alt.atheism:\n",
      "['someone', 'again', 'allan', 'political', 'schneider', 'atheism', 'caltech', 'cco', 'keith', 'atheists']\n",
      "\n",
      "# top 10 ~unique words occurring in comp.graphics:\n",
      "['files', 'version', 'file', 'mail', 'image', 'keywords', 'program', 'looking', 'please', 'graphics']\n",
      "\n",
      "# top 10 ~unique words occurring in sci.med:\n",
      "['soon', 'univ', 'pittsburgh', 'disease', 'geb', 'banks', 'medical', 'years', 'gordon', 'pitt']\n",
      "\n",
      "# top 10 ~unique words occurring in soc.religion.christian:\n",
      "['faith', 'athos', 'church', 'bible', 'christ', 'jesus', 'apr', 'christians', '1993', 'rutgers']\n"
     ]
    }
   ],
   "source": [
    "top_words = []\n",
    "for category in range(4):\n",
    "    top_words.append(feature_names[np.argsort(psis[category])[-200:]])\n",
    "\n",
    "for category in range(4):\n",
    "    words_from_other_categories = np.concatenate(top_words[:category] + top_words[category+1:])\n",
    "    unique_words = [word for word in top_words[category] if word not in words_from_other_categories]\n",
    "    print(f'\\n# top 10 ~unique words occurring in {twenty_train.target_names[category]}:')\n",
    "    print(unique_words[-10:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can compute **predictions** using Bayes' rule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 3 0 3 3 3 2 2 2]\n"
     ]
    }
   ],
   "source": [
    "def nb_predictions(x, psis, phis):\n",
    "    \"\"\"This returns class assignments and scores under the NB model.\n",
    "    \n",
    "    We compute \\arg\\max_y p(y|x) as \\arg\\max_y p(x|y)p(y)\n",
    "    \"\"\"\n",
    "    # adjust shapes\n",
    "    n, d = x.shape\n",
    "    x = np.reshape(x, (1, n, d))\n",
    "    psis = np.reshape(psis, (K, 1, d))\n",
    "    \n",
    "    psis = psis.clip(1e-14, 1-1e-14) # clip probabilities to avoid log(0)\n",
    "    \n",
    "    # compute log-probabilities\n",
    "    logpy = np.log(phis).reshape([K,1])\n",
    "    logpxy = x * np.log(psis) + (1-x) * np.log(1-psis)\n",
    "    logpyx = logpxy.sum(axis=2) + logpy\n",
    "\n",
    "    return logpyx.argmax(axis=0).flatten(), logpyx.reshape([K,n])\n",
    "\n",
    "idx_train, logpyx = nb_predictions(X_train, psis, phis)\n",
    "print(idx_train[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can measure the **accuracy**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8692955250332299"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(idx_train==y_train).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can also **make up new queries** and **ask the model** to label them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'OpenGL on the GPU is fast' => comp.graphics\n"
     ]
    }
   ],
   "source": [
    "docs_new = ['OpenGL on the GPU is fast']\n",
    "\n",
    "X_new = count_vect.transform(docs_new).toarray()\n",
    "predicted, logpyx_new = nb_predictions(X_new, psis, phis)\n",
    "\n",
    "for doc, category in zip(docs_new, predicted):\n",
    "    print('%r => %s' % (doc, twenty_train.target_names[category]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Algorithm: Bernoulli Naive Bayes\n",
    "\n",
    "* __Type__: Supervised learning (multi-class classification)\n",
    "* __Model family__: Products of Bernoulli distributions, categorical priors\n",
    "* __Objective function__: Log-likelihood.\n",
    "* __Optimizer__: Closed form solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<left><img width=25% src=\"img/cornell_tech2.svg\"></left>\n",
    "# Part 4: Naive Bayes: Learning (Advanced)\n",
    "\n",
    "We conclude by deriving the formula for Naive Bayes from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Review: Maximum Likelihood Learning\n",
    "\n",
    "We can learn a generative model $P_\\theta(x, y)$ by maximizing the *maximum likelihood*:\n",
    "\n",
    "$$ \\max_\\theta \\sum_{i=1}^n \\log P_\\theta({x}^{(i)}, y^{(i)}). $$\n",
    "\n",
    "This seeks to find parameters $\\theta$ such that the model assigns high probability to the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's use maximum likelihood to fit the Bernoulli Naive Bayes model. Note that model parameters $\\theta$ are the union of the parameters of each sub-model:\n",
    "$$\\theta = (\\phi_1, \\phi_2,\\ldots, \\phi_K, \\psi_{11}, \\psi_{21}, \\ldots, \\psi_{dK}).$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning a Bernoulli Naive Bayes Model\n",
    "\n",
    "Given a dataset $\\mathcal{D} = \\{(x^{(i)}, y^{(i)})\\mid i=1,2,\\ldots,n\\}$, we want to optimize the log-likelihood $\\ell(\\theta) = \\log L(\\theta)$:\n",
    "\\begin{align*}\n",
    "\\ell(\\theta) & = \\sum_{i=1}^n \\sum_{j=1}^d \\log P_\\theta(x^{(i)}_j | y^{(i)}) + \\sum_{i=1}^n \\log P_\\theta(y^{(i)}) \\\\\n",
    "& = \\sum_{k=1}^K \\sum_{j=1}^d \\underbrace{\\sum_{i :y^{(i)} =k} \\log P(x^{(i)}_j | y^{(i)} ; \\psi_{jk})}_\\text{all the terms that involve $\\psi_{jk}$} + \\underbrace{\\sum_{i=1}^n \\log P(y^{(i)} ; \\vec \\phi)}_\\text{all the terms that involve $\\vec \\phi$}.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In equality #2, we use Naive Bayes: $P_\\theta(x,y)=P_\\theta(y) \\prod_{i=1}^d P(x_j|y)$; in the third one, we change the order of summation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each $\\psi_{jk}$ for $k=1,2,\\ldots,K$ is found in only the following terms:\n",
    "$$ \\max_{\\psi_{jk}} \\ell(\\theta) = \\max_{\\psi_{jk}} \\sum_{i :y^{(i)} =k} \\log P(x^{(i)}_j | y^{(i)} ; \\psi_{jk}). $$\n",
    "Thus, optimization over $\\psi_{jk}$ can be carried out independently of all the other parameters by just looking at these terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Similarly, optimizing for $\\vec \\phi = (\\phi_1, \\phi_2, \\ldots, \\phi_K)$ only involves a few terms:\n",
    "$$ \\max_{\\vec \\phi} \\sum_{i=1}^n \\log P_\\theta(x^{(i)}, y^{(i)} ; \\theta) = \\max_{\\vec\\phi} \\sum_{i=1}^n  \\log P_\\theta(y^{(i)} ; \\vec \\phi). $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning the Parameters $\\phi$\n",
    "\n",
    "Let's first consider the optimization over $\\vec \\phi = (\\phi_1, \\phi_2, \\ldots, \\phi_K)$. \n",
    "$$ \\max_{\\vec \\phi} \\sum_{i=1}^n  \\log P_\\theta(y=y^{(i)} ; \\vec \\phi). $$\n",
    "* We have $n$ datapoints, each having one of $K$ classes\n",
    "* We want to learn the most likely class probabilities $\\phi_k$ that generated this data\n",
    "\n",
    "What is the maximum likelihood $\\phi$ in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Intuitively, the maximum likelihood class probabilities $\\phi$ should just be the class proportions that we see in the data. \n",
    "\n",
    "Let's calculate this formally. Our objective $J(\\vec \\phi)$ equals\n",
    "\\begin{align*}\n",
    "J(\\vec\\phi) & = \\sum_{i=1}^n  \\log P_\\theta(y^{(i)} ; \\vec \\phi) = \\sum_{i=1}^n  \\log \\left( \\frac{\\phi_{y^{(i)}}}{\\sum_{k=1}^K \\phi_k}\\right) \\\\\n",
    "& = \\sum_{i=1}^n \\log \\phi_{y^{(i)}} - n \\cdot \\log \\sum_{k=1}^K \\phi_k \\\\ \n",
    "& = \\sum_{k=1}^K \\sum_{i : y^{(i)} = k} \\log \\phi_k - n \\cdot \\log \\sum_{k=1}^K \\phi_k\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Taking the derivative and setting it to zero, we obtain \n",
    "$$ \\frac{\\phi_k}{\\sum_l \\phi_l} = \\frac{n_k}{n}$$\n",
    "for each $k$, where $n_k = |\\{i : y^{(i)} = k\\}|$ is the number of training targets with class $k$.\n",
    "\n",
    "Thus, the optimal $\\phi_k$ is just the proportion of data points with class $k$ in the training set!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Learning the Parameters $\\psi_{jk}$\n",
    "\n",
    "Next, let's look at the maximum likelihood term\n",
    "$$ \\arg\\max_{\\psi_{jk}} \\sum_{i :y^{(i)} =k} \\log P(x^{(i)}_j | y^{(i)} ; \\psi_{jk}). $$\n",
    "over the word parameters $\\psi_{jk}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Our dataset are all the inputs $x$ for which $y=k$.\n",
    "* We seek  the probability $\\psi_{jk}$ of a word $j$ being present in a $x$.\n",
    "\n",
    "What is the maximum likelihood $\\psi_{jk}$ in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each $\\psi_{jk}$ is simply the proportion of documents in class $k$ that contain the word $j$.\n",
    "\n",
    "We can maximize the likelihood exactly like we did for $\\phi$ to obtain **closed-form** solutions:\n",
    "\\begin{align*}\n",
    "\\psi_{jk} = \\frac{n_{jk}}{n_k}.\n",
    "\\end{align*}\n",
    "where $|\\{i : x^{(i)}_j = 1 \\text{ and } y^{(i)} = k\\}|$ is the number of $x^{(i)}$ with label $k$ and a positive occurrence of word $j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generative vs. Discriminative Approaches\n",
    "\n",
    "What are the pros and cons of generative and discriminative methods?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* If we only care about prediction, we don't need a model of $P(x)$. It's simpler to only model $P(y|x)$ (what we care about).\n",
    "    * In practice, discriminative models are often be more accurate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* If we care about other tasks (generation, dealing with missing values, etc.) or if we care about computation time, or we know the true model is generative, we want to use the generative approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary\n",
    "\n",
    "1. We can now handle text-based inputs with bag of words features <br> <span style='color:grey'>More on text and language processing later in the course</span>\n",
    "2. **Discriminative models** directly estimate $P_\\theta(y|x)$: distinguishes classes directly\n",
    "3. **Generative models** estimate $P_\\theta(x,y)$ via class-conditional generations $P_\\theta(x|y)$\n",
    "\n",
    "**Next class**: more generative models, beyond the Naive Bayes limitations, fitting mixtures to data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "celltoolbar": "Slideshow",
  "colab": {
   "collapsed_sections": [],
   "name": "neural-ode.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "rise": {
   "controlsTutorial": false,
   "height": 900,
   "help": false,
   "margin": 0,
   "maxScale": 2,
   "minScale": 0.2,
   "progress": true,
   "scroll": true,
   "theme": "simple",
   "width": 1200
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}