Language-model-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# N-gram language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Formal grammars (e.g. regular, context free) give a hard “binary” model of the legal  sentences in a language.\n",
    "\n",
    "- A *probabilistic* model of a language that gives a probability that a string is a member of a language is more useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tasks:\n",
    "- Speech recognition: \n",
    "“I ate a cherry” is a more likely sentence than “Eye eight uh Jerry” \n",
    "- OCR & Handwriting recognition: \n",
    "More probable sentences are more likely correct readings\n",
    "- Machine translation: \n",
    "More likely sentences are probably better translations.\n",
    "- Generation: \n",
    "More likely sentences are probably better NL generations.  \n",
    "- Context sensitive spelling correction: \n",
    "“Their are problems wit this sentence.”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A language model also supports predicting the completion of a sentence.  \n",
    "Please turn off your cell _____  \n",
    "Your program does not ______  \n",
    "\n",
    "Predictive text input systems can guess what you are typing and give choices on how to complete it.\n",
    "\n",
    "- Estimate probability of each word given prior context.\n",
    "P(phone | Please turn off your cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An N-gram model uses only N-1 words of prior context.\n",
    "- Unigram:  P(phone)\n",
    "- Bigram:  P(phone | cell)\n",
    "- Trigram:  P(phone | your cell)\n",
    "    \n",
    "The **Markov assumption** is the presumption that the future behavior of a dynamical system only depends on its recent history.  In particular, in a kth-order Markov model, the next state only depends on the k most recent states, therefore an N-gram model is a (N-1)-order Markov model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## N-gram model formulas\n",
    "- Word sequences: $$w_1^n = w_1 ... w_n$$\n",
    "\n",
    "- Chain rule of probability: $$P(w_1^n) = P(w_1)P(w_2|w_1)P(w_3|w_1^2)...P(w_n|w_1^{n-1}) = \\prod_{k=1}^n P(w_k|w_1^{k-1}) $$\n",
    "\n",
    "- Bigram approximation: $$P(w_1^n) = \\prod_{k=1}^n P(w_k|w_{k-1}) $$\n",
    "\n",
    "- N-gram approximation: $$P(w_1^n) = \\prod_{k=1}^n P(w_k|w_{k-N+1}^{k-1}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do you estimate the probabilities?\n",
    "Use counts in your corpus:  \n",
    "\n",
    "- Bigram: $$P(w_n|w_{n-1}) = \\frac{C(w_{n-1}w_n)}{C(w_{n-1})}$$\n",
    "\n",
    "- N-gram: $$P(w_n|w_{n-N+1}^{n-1}) = \\frac{C(w_{n-N+1}^{n-1}w_n)}{C(w_{n-N+1}^{n-1})}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Demo!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the same code as previous notebook. But now you can specify higher-order n-grams in `CountVectorizer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use both unigrams *and* bigrams:  \n",
    "    `vectorizer = CountVectorizer(tokenizer=tokenize_text, ngram_range=(1,2), encoding='latin-1')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q: Does the accuracy improve?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Q: What happens when you specify higher and higher order ngrams?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Fun resource: Google ngram viewer\n",
    "- Google provides 1 through 5-gram counts of all their Google books that you can download\n",
    "- Can also view the frequency of n-grams across time: https://books.google.com/ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"900\"\n",
       "            height=\"500\"\n",
       "            src=\"https://books.google.com/ngrams/interactive_chart?content=natural+language+processing&case_insensitive=on&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t4%3B%2Cnatural%20language%20processing%3B%2Cc0%3B%2Cs0%3B%3Bnatural%20language%20processing%3B%2Cc0%3B%3BNatural%20Language%20Processing%3B%2Cc0%3B%3BNatural%20language%20processing%3B%2Cc0%3B%3BNATURAL%20LANGUAGE%20PROCESSING%3B%2Cc0?vspace=0&marginheight=0&hspace=0&marginwidth=0&frameborder=0\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1193a9d50>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame(src=\"https://books.google.com/ngrams/interactive_chart?content=natural+language+processing&case_insensitive=on&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t4%3B%2Cnatural%20language%20processing%3B%2Cc0%3B%2Cs0%3B%3Bnatural%20language%20processing%3B%2Cc0%3B%3BNatural%20Language%20Processing%3B%2Cc0%3B%3BNatural%20language%20processing%3B%2Cc0%3B%3BNATURAL%20LANGUAGE%20PROCESSING%3B%2Cc0\", width=900, height=500, marginwidth=0, marginheight=0, hspace=0, vspace=0, frameborder=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But is local context enough?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No. Many times, you need **long-distance** dependencies:\n",
    "- Syntactic dependencies  \n",
    "The man next to the large oak tree **is** tall.  \n",
    "The men next to the large oak tree near the grocery store on the corner **are** tall.  \n",
    "- Semantic dependencies  \n",
    "The bird next to the large oak tree **flies** rapidly.  \n",
    "The man next to the large oak tree **talks** rapidly.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** You need more complex models like syntactic, semantic and discourse parsers. To be continued..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}