NLP session.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "# NLP session 29th Oct 2019\n",
    "\n",
    "\n",
    "## What is NLP?\n",
    "\n",
    "**Neuro-linguistic programming** -  is a pseudoscientific approach to communication, personal development, and psychotherapy created by Richard Bandler and John Grinder in California, United States in the 1970s. NLP's creators claim there is a connection between neurological processes (neuro-), language (linguistic) and behavioral patterns learned through experience (programming), and that these can be changed to achieve specific goals in life.[[1]](http://web.archive.org/web/20190103020411/http://www.som.surrey.ac.uk/NLP/Resources/IntroducingNLP.pdf)\n",
    "[[2]](https://en.wikipedia.org/wiki/Neuro-linguistic_programming)\n",
    "\n",
    "**Natural language processing** -  is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.\n",
    "Challenges in natural language processing frequently involve *speech recognition*, *natural language understanding*, and *natural language generation*. \n",
    "[[3]](https://en.wikipedia.org/wiki/Natural_language_processing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%%html\n",
    "<style>\n",
    "table {float: left}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Seting Up env\n",
    "Let us import and check what version of **spacy** we have, this should be ```2.2.1```. I am also using **displacy** for visualizations.\n",
    "\n",
    "To check which version of the language models we have installed, in terminall run also\n",
    "```bash\n",
    "spacy validate\n",
    "```\n",
    "\n",
    "We will be using some advanced tools, so we need large model ```en_core_web_lg```. Lastly, we import also **Matcher**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spaCy version:  2.2.2\n"
     ]
    },
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'en_core_web_lg'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-bbf36d35f3df>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mspacy\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdisplacy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0men_core_web_lg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0men_core_web_sm\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'en_core_web_lg'"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import random\n",
    "import nltk\n",
    "\n",
    "import spacy\n",
    "print(\"spaCy version: \",spacy.__version__)\n",
    "from spacy import displacy\n",
    "\n",
    "import en_core_web_lg\n",
    "import en_core_web_sm\n",
    "\n",
    "from spacy.matcher import Matcher\n",
    "from spacy.matcher import PhraseMatcher\n",
    "from spacy.tokens import Doc, Token, Span\n",
    "from spacy.lang.en import English\n",
    "\n",
    "import json\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Strings\n",
    "\n",
    "Strings in python are **immutable arrays**. We can use both positive and negative index to slice them\n",
    "```python\n",
    "b = \"Hello, World!\"\n",
    "print(b[-5:-2])\n",
    "```\n",
    "\n",
    "We can concatinate them \n",
    "```python\n",
    "a = \"Hello\"\n",
    "b = \"World\"\n",
    "c = a + b\n",
    "print(c)\n",
    "```\n",
    "... format\n",
    "```python\n",
    "quantity = 3\n",
    "itemno = 567\n",
    "price = 49.95\n",
    "myorder = \"I want {} pieces of item {} for {} dollars.\"\n",
    "print(myorder.format(quantity, itemno, price))\n",
    "```\n",
    "\n",
    "...split\n",
    "```python\n",
    "a = \"Hello, World!\"\n",
    "print(a.split(\",\")) \n",
    "```\n",
    "\n",
    "... apply methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length:  6\n",
      "Upper case:  KONRAD\n",
      "Lower case:  konrad\n",
      "Title case:  Konrad\n",
      "False\n",
      "True\n",
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "string = \"Konrad\"\n",
    "print(\"Length: \",len(string))\n",
    "print(\"Upper case: \",string.upper())\n",
    "print(\"Lower case: \",string.lower())\n",
    "print(\"Title case: \",string.lower().title())\n",
    "print(string.startswith('ko'))\n",
    "print(string.lower().startswith('ko'))\n",
    "print(string.endswith('D'))\n",
    "print(string.upper().endswith('D'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "filename = \"cisco_article.txt\"\n",
    "\n",
    "with open(filename, 'r') as f:\n",
    "    text_whole=f.read().replace('\\n', '').replace('    ','')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regex\n",
    "\n",
    "Webpage: [Regex 101](https://regex101.com/)\n",
    "\n",
    "**\"If someone told you that they know regex, they lied to you.\"**\n",
    "\n",
    "Pattern for words (including ```It's```, ```AI/ML``` or ```SD-WAN```)\n",
    "```\n",
    "(\\w*[-'—/]*\\w+)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## Tokenization\n",
    "Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens , perhaps at the same time throwing away certain characters, such as punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from nltk import RegexpTokenizer\n",
    "tokenizer = RegexpTokenizer(\"(\\w*[-'—/]*\\w+)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1467\n",
      "baseline\n"
     ]
    }
   ],
   "source": [
    "nltk_tokenized_words = tokenizer.tokenize(text_whole)\n",
    "print(len(nltk_tokenized_words))\n",
    "print(nltk_tokenized_words[666])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence tokenize\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35\n",
      "in the cisco dna center and the cloud.cisco ai network analytics in the cloudfor years now, cisco has been integrating ai/ml into many operational and security components, with cisco dna center the focal point for insights and actions.\n"
     ]
    }
   ],
   "source": [
    "corpus = nltk.sent_tokenize(text_whole.lower())\n",
    "print(len(corpus))\n",
    "print(corpus[12])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency\n",
    "\n",
    "We understand now the term of the token and we know how to tokenize the text. Next step checok how frequently tokens appears in our document. To count it we will use ```Counter``` from ```collections``` built in python package. To see most common X objects we run ```Counter().most_common(X)```. \n",
    "\n",
    "We will be monitoring this frequency at our next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('and', 74),\n",
       " ('the', 46),\n",
       " ('of', 44),\n",
       " ('to', 44),\n",
       " ('a', 28),\n",
       " ('for', 25),\n",
       " ('Cisco', 25),\n",
       " ('in', 22),\n",
       " ('AI', 20),\n",
       " ('network', 19),\n",
       " ('with', 15),\n",
       " ('that', 14),\n",
       " ('Analytics', 14),\n",
       " ('data', 13),\n",
       " ('Network', 13),\n",
       " ('is', 12),\n",
       " ('performance', 11),\n",
       " ('can', 11),\n",
       " ('DNA', 11),\n",
       " ('issues', 10)]"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(nltk_tokenized_words).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop words\n",
    "**Stop words** are words which are filtered out before processing of natural language data (text).\n",
    "\n",
    "Stop words are generally the most common words in a language; there is no single universal list of stop words used by all natural language processing tools, and indeed not all tools even use such a list. Some tools avoid removing stop words to support phrase search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "985\n"
     ]
    }
   ],
   "source": [
    "nltk_filtered_sentence = [w for w in nltk_tokenized_words if not w in stop_words]\n",
    "print(len(nltk_filtered_sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Cisco', 25),\n",
       " ('AI', 20),\n",
       " ('network', 19),\n",
       " ('Analytics', 14),\n",
       " ('data', 13),\n",
       " ('Network', 13),\n",
       " ('performance', 11),\n",
       " ('DNA', 11),\n",
       " ('issues', 10),\n",
       " ('NetOps', 8),\n",
       " ('Worldwide', 8),\n",
       " ('Data', 8),\n",
       " ('Platform', 8),\n",
       " ('networks', 6),\n",
       " ('branch', 6),\n",
       " ('devices', 6),\n",
       " ('IT', 6),\n",
       " ('patterns', 6),\n",
       " ('Center', 6),\n",
       " ('offices', 5)]"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(nltk_filtered_sentence).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming\n",
    "\n",
    "**Stemming** is the process of reducing inflection in words to their root forms such as mapping a group of words to the same stem even if the stem itself is not a valid word in the Language.\n",
    "\n",
    "Stem (root) is the part of the word to which you add inflectional (changing/deriving) affixes such as (-ed,-ize, -s,-de,mis). So stemming a word or sentence may result in words that are not actual words. Stems are created by removing the suffixes or prefixes used with a word.\n",
    "\n",
    "**Information**: Removing suffixes from a word is called Suffix Stripping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\n",
      "troubl\n",
      "troubl\n",
      "troubl\n",
      "destabil\n"
     ]
    }
   ],
   "source": [
    "from nltk import PorterStemmer\n",
    "porter = PorterStemmer()\n",
    "#proide a word to be stemmed\n",
    "print(porter.stem(\"cats\"))\n",
    "print(porter.stem(\"trouble\"))\n",
    "print(porter.stem(\"troubling\"))\n",
    "print(porter.stem(\"troubled\"))\n",
    "print(porter.stem(\"destabilized\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\n",
      "troubl\n",
      "troubl\n",
      "troubl\n",
      "dest\n",
      "the\n"
     ]
    }
   ],
   "source": [
    "from nltk import LancasterStemmer\n",
    "lancaster=LancasterStemmer()\n",
    "print(lancaster.stem(\"cats\"))\n",
    "print(lancaster.stem(\"trouble\"))\n",
    "print(lancaster.stem(\"troubling\"))\n",
    "print(lancaster.stem(\"troubled\"))\n",
    "print(lancaster.stem(\"destabilized\"))\n",
    "print(lancaster.stem(\"the\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\n",
      "troubl\n",
      "troubl\n",
      "troubl\n",
      "destabil\n"
     ]
    }
   ],
   "source": [
    "from nltk import SnowballStemmer\n",
    "snowball = SnowballStemmer('english', ignore_stopwords=\"yes\")\n",
    "#proide a word to be stemmed\n",
    "print(snowball.stem(\"cats\"))\n",
    "print(snowball.stem(\"trouble\"))\n",
    "print(snowball.stem(\"troubling\"))\n",
    "print(snowball.stem(\"troubled\"))\n",
    "print(snowball.stem(\"destabilized\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "985\n"
     ]
    }
   ],
   "source": [
    "nltk_snowball_stemmer = [snowball.stem(w) for w in nltk_filtered_sentence]\n",
    "print(len(nltk_snowball_words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('network', 46),\n",
       " ('cisco', 25),\n",
       " ('data', 21),\n",
       " ('ai', 20),\n",
       " ('analyt', 15),\n",
       " ('perform', 12),\n",
       " ('center', 11),\n",
       " ('dna', 11),\n",
       " ('issu', 11),\n",
       " ('action', 9),\n",
       " ('platform', 9),\n",
       " ('netop', 8),\n",
       " ('insight', 8),\n",
       " ('worldwid', 8),\n",
       " ('branch', 7),\n",
       " ('offic', 7),\n",
       " ('devic', 7),\n",
       " ('connect', 7),\n",
       " ('cloud', 7),\n",
       " ('time', 7)]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(nltk_snowball_stemmer).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lemmatization\n",
    "Lemmatization, unlike Stemming, reduces the inflected words properly ensuring that the root word belongs to the language. In Lemmatization root word is called Lemma. A lemma (plural lemmas or lemmata) is the canonical form, dictionary form, or citation form of a set of words.\n",
    "\n",
    "Consider the below two cells. **Did the lemmatizer work as expected?**\n",
    "\n",
    "In the next step let us check the most frequent words. **Do you see some issue here?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer \n",
    "from nltk.corpus import wordnet\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\n",
      "trouble\n",
      "troubling\n",
      "troubled\n",
      "destabilized\n"
     ]
    }
   ],
   "source": [
    "print(lemmatizer.lemmatize(\"cats\"))\n",
    "print(lemmatizer.lemmatize(\"trouble\"))\n",
    "print(lemmatizer.lemmatize(\"troubling\"))\n",
    "print(lemmatizer.lemmatize(\"troubled\"))\n",
    "print(lemmatizer.lemmatize(\"destabilized\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def get_wordnet_pos(word):\n",
    "    \"\"\"Map POS tag to first character lemmatize() accepts\"\"\"\n",
    "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
    "    tag_dict = {\"J\": wordnet.ADJ,\n",
    "                \"N\": wordnet.NOUN,\n",
    "                \"V\": wordnet.VERB,\n",
    "                \"R\": wordnet.ADV}\n",
    "\n",
    "    return tag_dict.get(tag, wordnet.NOUN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "985\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('network', 25),\n",
       " ('Cisco', 25),\n",
       " ('AI', 20),\n",
       " ('Analytics', 14),\n",
       " ('data', 13),\n",
       " ('Network', 13),\n",
       " ('performance', 11),\n",
       " ('DNA', 11),\n",
       " ('issue', 11),\n",
       " ('NetOps', 8),\n",
       " ('Worldwide', 8),\n",
       " ('Data', 8),\n",
       " ('Platform', 8),\n",
       " ('branch', 7),\n",
       " ('office', 7),\n",
       " ('device', 7),\n",
       " ('pattern', 7),\n",
       " ('action', 7),\n",
       " ('application', 6),\n",
       " ('time', 6)]"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk_lemmatized_words = [lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in nltk_filtered_sentence]\n",
    "print(len(nltk_lemmatized_words))\n",
    "Counter(nltk_lemmatized_words).most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "985\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('network', 44),\n",
       " ('cisco', 25),\n",
       " ('data', 21),\n",
       " ('ai', 20),\n",
       " ('analytics', 15),\n",
       " ('center', 11),\n",
       " ('performance', 11),\n",
       " ('dna', 11),\n",
       " ('issue', 11),\n",
       " ('platform', 9),\n",
       " ('netops', 8),\n",
       " ('insight', 8),\n",
       " ('action', 8),\n",
       " ('worldwide', 8),\n",
       " ('branch', 7),\n",
       " ('office', 7),\n",
       " ('device', 7),\n",
       " ('cloud', 7),\n",
       " ('pattern', 7),\n",
       " ('baseline', 7)]"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk_lemmatized_words = [lemmatizer.lemmatize(w.lower(), get_wordnet_pos(w)) for w in nltk_filtered_sentence]\n",
    "print(len(nltk_lemmatized_words))\n",
    "Counter(nltk_lemmatized_words).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lexical Diversity \n",
    "Let’s define a short function to identify an introductory metric for our story. The Lexical Diversity represents the ratio of unique words used to the total number of words in the story."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def lexical_diversity(text):\n",
    "    return (len(set(text)) / len(text), len(set(text)), len(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lexical diversity of NLTK:  (0.4178595773687798, 613, 1467)\n",
      "Lexical diversity of NLTK without stepwords:  (0.5604060913705584, 552, 985)\n",
      "Lexical diversity of NLTK Snowball:  (0.44568527918781725, 439, 985)\n",
      "Lexical diversity of NLTK Lemmantized:  (0.47411167512690355, 467, 985)\n"
     ]
    }
   ],
   "source": [
    "print(\"Lexical diversity of NLTK: \",lexical_diversity(nltk_tokenized_words))\n",
    "print(\"Lexical diversity of NLTK without stepwords: \",lexical_diversity(nltk_filtered_sentence))\n",
    "print(\"Lexical diversity of NLTK Snowball: \",lexical_diversity(nltk_snowball_stemmer))\n",
    "print(\"Lexical diversity of NLTK Lemmantized: \",lexical_diversity(nltk_lemmatized_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words (BOW)\n",
    "\n",
    "The bag-of-words model is simple to understand and implement. It is a way of extracting features from the text for use in machine learning algorithms.\n",
    "\n",
    "If you want to make an NLP application that classifies documents in different categories, then you can use BOW. BOW is also used to generate frequency count and vocabulary from a dataset. These derived attributes are then used in NLP applications such as sentiment analysis, Word2vec, and so on.\n",
    "\n",
    "In this approach, each word or token is called a “gram”. Creating a vocabulary of two-word pairs is called a bigram model. For example, the bigrams in the first document : ```It was the best of times``` are as follows:\n",
    "* \"it was\"\n",
    "* \"was the\"\n",
    "* \"the best\"\n",
    "* \"best of\"\n",
    "* \"of times\"\n",
    "\n",
    "We treat each sentence as a separate document and we make a list of all words from all the four documents excluding the punctuation.\n",
    "\n",
    "The next step is the create vectors. Vectors convert text that can be used by the machine learning algorithm.\n",
    "\n",
    "### n-grams\n",
    "*\\\"In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. The n-grams typically are collected from a text or speech corpus. When the items are words, n-grams may also be called shingles[clarification needed].\\\"* \n",
    "\n",
    "[source](https://en.wikipedia.org/wiki/N-gram)\n",
    "\n",
    "In n-grams, word order is important, whereas in BOW it is not important to maintain word order. During the NLP application, n-gram is used to consider words in their real order so we can get an idea about the context of the particular word; BOW is used to build vocabulary for your text dataset.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['network',\n",
       " 'cisco',\n",
       " 'data',\n",
       " 'ai',\n",
       " 'analytics',\n",
       " 'center',\n",
       " 'performance',\n",
       " 'dna',\n",
       " 'issue',\n",
       " 'platform',\n",
       " 'netops',\n",
       " 'insight',\n",
       " 'action',\n",
       " 'worldwide',\n",
       " 'branch',\n",
       " 'office',\n",
       " 'device',\n",
       " 'cloud',\n",
       " 'pattern',\n",
       " 'baseline']"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_freq_words = Counter(nltk_lemmatized_words).most_common(20)\n",
    "most_freq_words = [freq[0] for freq in most_freq_words]\n",
    "most_freq_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "sentence_vectors = []\n",
    "for sentence in corpus:\n",
    "    sentence_tokens = tokenizer.tokenize(sentence)\n",
    "    lemmatized_sentence_tokens = [lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in sentence_tokens]\n",
    "    sent_vec = []\n",
    "    for token in most_freq_words:\n",
    "        if token in lemmatized_sentence_tokens:\n",
    "            sent_vec.append(1)\n",
    "        else:\n",
    "            sent_vec.append(0)\n",
    "    sentence_vectors.append(sent_vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
       "       [0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
       "       [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0],\n",
       "       [1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0],\n",
       "       [1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0],\n",
       "       [1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1],\n",
       "       [1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0],\n",
       "       [1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0],\n",
       "       [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1],\n",
       "       [1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1],\n",
       "       [1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1],\n",
       "       [1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0],\n",
       "       [1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1],\n",
       "       [1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0],\n",
       "       [1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0],\n",
       "       [1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_vectors = np.asarray(sentence_vectors)\n",
    "sentence_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF\n",
    "\n",
    "Term Frequency - Inverse Document Frequency. This methodology will help finding important, but seldom words.\n",
    "\n",
    "\n",
    "$$\\text{TF-IDF} = TF * IDF$$\n",
    "$$TF = tf_{i,j}$$ \n",
    "$$IDF = log(\\frac{N}{df_{i}})$$ \n",
    "\n",
    "tf - number of times word occurs in the sentence divided by total number of terms in the sentence\n",
    "\n",
    "N - total number of sentences\n",
    "\n",
    "dfi - number of sentences containing word\n",
    "\n",
    "\n",
    "### Example\n",
    "Consider 20 sentences. We are looking for the word *NLP*, that occurs only once in the sentence *I love NLP*. \n",
    "$$tf_{NLP,j} = \\frac{\\text{# occurences of word}}{\\text{# words in the sentence}} = \\frac{1}{3} = 0.33$$\n",
    "$$\\text{TF-IDF}_{NLP} = tf_{i,j} * log(\\frac{\\text{total # sentences}}{\\text{# sentences containing word}}) = 0.33 * log(\\frac{20}{1}) = 0.33 * 1.301 = 0.43$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'sklearn'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-901db3a446a6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_extraction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTfidfVectorizer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtfidf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'sklearn'"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer()\n",
    "vectors = vectorizer.fit_transform(corpus)\n",
    "feature_names = vectorizer.get_feature_names()\n",
    "dense = vectors.todense()\n",
    "denselist = dense.tolist()\n",
    "\n",
    "## Compare both ways\n",
    "df = pd.DataFrame(denselist, columns=feature_names)\n",
    "df_1 = pd.DataFrame(vectors.toarray(), columns=feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## Word2vec\n",
    "The Word to Vec model produces a vocabulary, with each word being represented by an n-dimensional numpy array (100 values in this example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "corpus_tokenised = []\n",
    "for sentence in corpus:\n",
    "    sentence_tokens = tokenizer.tokenize(sentence)\n",
    "    filtered_sentence = [w.lower() for w in sentence_tokens if not w in stop_words] \n",
    "    lemmatized_sentence_tokens = [lemmatizer.lemmatize(w, get_wordnet_pos(w)) for w in filtered_sentence]\n",
    "    corpus_tokenised.append(lemmatized_sentence_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "model = word2vec.Word2Vec(corpus_tokenised, size=300, window=3, min_count=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.6817740e-04, -2.8518352e-05, -3.3628449e-04, -4.9796910e-04,\n",
       "        7.3395413e-04,  1.4484520e-03, -6.6038156e-05,  2.1801452e-04,\n",
       "        5.9292431e-04, -5.1206538e-05, -4.2694263e-04, -1.7206463e-03,\n",
       "       -1.3506947e-03, -9.1160729e-04,  1.4229257e-03, -6.0548878e-04,\n",
       "       -8.1224879e-04,  9.8901649e-04,  5.9237989e-04, -3.1332063e-04,\n",
       "        7.5117685e-05,  3.5242827e-04, -1.6659006e-04, -1.2199940e-04,\n",
       "       -7.9429876e-05, -2.3054839e-04, -3.0415252e-04, -9.7894762e-04,\n",
       "        1.5105685e-03,  9.4492879e-04, -1.3115926e-03, -2.9610924e-04,\n",
       "        1.5793691e-03,  1.1521649e-03, -1.4870856e-03, -1.0963898e-03,\n",
       "        1.3932979e-03, -2.1771775e-04,  9.9827885e-04, -4.8045994e-04,\n",
       "        1.6082077e-03,  1.1020086e-03,  2.9957460e-05, -3.3649329e-05,\n",
       "       -3.9492518e-04, -1.7665951e-04,  2.2322120e-04,  8.9955184e-04,\n",
       "       -1.3561846e-03,  8.0665498e-04,  8.7253470e-04, -1.1191472e-03,\n",
       "       -1.1536921e-03,  1.3400570e-03,  1.0289421e-03, -2.2954899e-05,\n",
       "        8.0299209e-04, -6.9745671e-04,  1.1748864e-03, -7.0365309e-04,\n",
       "       -1.2306117e-04, -7.8533648e-04,  6.8606774e-04,  8.6464868e-05,\n",
       "        2.1127715e-04, -6.7812821e-04, -3.2928435e-04,  1.4677199e-03,\n",
       "        1.1561929e-03,  1.1236183e-04, -1.2416021e-03,  2.2500554e-04,\n",
       "        1.4692555e-03,  7.5024139e-04,  9.9940086e-04, -1.4603883e-05,\n",
       "       -8.2105608e-04,  1.5598470e-03,  6.2732631e-04,  1.6144870e-03,\n",
       "       -5.2334537e-04, -3.7737540e-04,  1.0386008e-03, -1.7379385e-03,\n",
       "        2.8085828e-04, -5.1257672e-04,  7.4820872e-04,  7.3657825e-04,\n",
       "        6.0495199e-04, -8.2960317e-04,  1.5968032e-03, -1.8093133e-04,\n",
       "        1.4476675e-03, -8.7931514e-04,  1.4348766e-03,  5.3520076e-04,\n",
       "        1.4336923e-03, -1.1210168e-03, -2.0242079e-04, -9.4290808e-05,\n",
       "        1.3475766e-03, -1.1800174e-03, -7.4590591e-04, -7.9662853e-04,\n",
       "        1.1184271e-03, -1.0811962e-03,  3.5504563e-05, -9.5484115e-04,\n",
       "        1.5473957e-03,  2.3068207e-04,  7.4552977e-04, -9.8199223e-04,\n",
       "        9.5415761e-04,  1.5178970e-03,  2.0413034e-04,  2.0063479e-05,\n",
       "        7.9177023e-04,  9.9409139e-04, -7.7784283e-04,  5.3562515e-04,\n",
       "        4.8945629e-04,  5.1312967e-05, -9.4614911e-04,  1.5218372e-03,\n",
       "        3.2153420e-04, -9.1106439e-04,  2.1606578e-04,  1.6322099e-03,\n",
       "        1.2662896e-03, -9.4347388e-06, -1.4532052e-04,  2.9097922e-05,\n",
       "        1.2719260e-03, -1.1370251e-03,  1.4205147e-03,  3.6980209e-04,\n",
       "        1.5408266e-03, -1.4009580e-03,  5.8613357e-04, -1.1491592e-03,\n",
       "        1.5402971e-03, -9.4334275e-04, -1.3546260e-03,  1.3284226e-03,\n",
       "        1.1959790e-03, -2.3870599e-04,  1.5555563e-03, -1.1472136e-03,\n",
       "        7.7011302e-04,  9.0411959e-05,  8.2136149e-04, -3.7261224e-04,\n",
       "       -1.2576684e-03, -7.9787005e-04, -1.0559147e-03, -4.7048993e-04,\n",
       "        1.6264472e-03,  2.6611760e-04, -1.1532810e-03,  6.4774021e-04,\n",
       "       -1.4093040e-03, -1.2092260e-03,  1.1526952e-03,  2.5521583e-04,\n",
       "        4.4981681e-04, -2.9787063e-04, -1.0776533e-03,  1.0977259e-04,\n",
       "       -2.7216194e-04, -5.5590150e-04,  9.2792994e-04,  9.6968113e-04,\n",
       "       -1.0755470e-03,  1.5157886e-03, -1.3785021e-03, -7.5975322e-04,\n",
       "       -8.2238327e-04, -1.3290929e-03,  1.4907481e-03,  7.7726909e-05,\n",
       "       -8.1096857e-04,  8.4111576e-05, -2.5640926e-04, -1.5473460e-03,\n",
       "       -8.3176716e-04, -3.9426918e-04, -1.5335197e-03, -1.5013367e-03,\n",
       "       -1.1271997e-03,  3.6901882e-04, -5.9927040e-04,  1.6392386e-03,\n",
       "        1.3476629e-03,  3.2462040e-04, -3.8016881e-04,  1.0742854e-03,\n",
       "        1.5884818e-03, -3.2097218e-04, -1.5277563e-03,  9.9272851e-04,\n",
       "       -1.5211021e-04, -7.1442354e-04, -1.3763221e-03,  1.1956843e-03,\n",
       "       -3.0808311e-04, -7.9529936e-04,  4.1256755e-04, -1.2433105e-03,\n",
       "       -1.3510424e-03, -7.4077438e-04, -1.1447578e-03, -1.5276074e-03,\n",
       "        1.4991972e-03,  1.6004650e-03,  5.1909889e-04,  1.3435754e-03,\n",
       "        7.4845122e-04, -2.2113934e-04,  1.2753414e-03,  9.6666865e-04,\n",
       "        6.6311279e-04, -1.5289658e-03, -1.4568364e-03,  7.9311524e-04,\n",
       "       -9.7862887e-04, -1.2355034e-03, -8.5860770e-04,  7.5415155e-04,\n",
       "        5.5313885e-04,  1.5786634e-04, -1.1024412e-03,  1.4111176e-03,\n",
       "       -2.6551349e-04,  1.4178617e-03, -3.9460600e-04,  5.0429697e-04,\n",
       "       -7.6053775e-04,  6.3093286e-04, -1.3160682e-03,  6.0824020e-04,\n",
       "       -7.9200638e-04, -6.0292985e-04, -1.2955959e-03, -1.3110916e-03,\n",
       "        1.1219118e-03,  3.6452213e-04,  3.5767455e-04, -2.1110290e-04,\n",
       "       -1.0038174e-03, -1.9707035e-04,  1.3965664e-03, -9.9983905e-04,\n",
       "       -1.1236710e-03,  1.5237289e-03,  2.4526057e-04,  7.6278148e-04,\n",
       "       -2.7892186e-04, -5.1041786e-04,  4.7978139e-04, -1.0895465e-03,\n",
       "        1.6063574e-04,  5.1975943e-04,  1.7809158e-04, -1.4950550e-03,\n",
       "        5.2059232e-04,  5.8371952e-04, -1.4898011e-03, -7.9204398e-04,\n",
       "        1.9730983e-04,  1.1609987e-04,  9.4495408e-05, -1.3960039e-03,\n",
       "        6.4867083e-05,  7.7561580e-04,  1.0738074e-03, -7.5955722e-05,\n",
       "        6.9983484e-04,  1.6181953e-03,  1.5116147e-03,  9.9418068e-04,\n",
       "        4.8329603e-04,  5.5762171e-04,  1.7269631e-03, -1.0702189e-03,\n",
       "       -1.0440754e-03,  2.2129316e-04,  1.1035531e-03, -1.2221056e-03,\n",
       "       -1.3651849e-03, -1.6182315e-03, -1.4895047e-03,  8.2411635e-04,\n",
       "       -1.3779261e-03,  1.1350497e-03,  1.4901856e-03,  4.9431517e-04,\n",
       "       -1.4371662e-03, -8.0880820e-04, -7.5281935e-04,  8.1902678e-04],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv['ai']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "### Similarity \n",
    "Now we could even use Word2vec to compute the similarity between two Make Models in the vocabulary by invoking the model.similarity( ) and passing in the relevant words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0069827754"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.similarity('ai','cisco')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('gather', 0.18863195180892944),\n",
       " ('application', 0.182594895362854),\n",
       " ('abnormal', 0.14176899194717407),\n",
       " ('blizzard', 0.14128705859184265),\n",
       " ('operating', 0.14054402709007263),\n",
       " ('problem', 0.1390073001384735),\n",
       " ('switch', 0.13393692672252655),\n",
       " ('change', 0.1276453733444214),\n",
       " ('human', 0.12112872302532196),\n",
       " ('compare', 0.12103308737277985)]"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('cisco')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "# Spacy\n",
    "\n",
    "Before going here please familiarise yourself with Preworkout :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = en_core_web_lg.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n",
      "Do something to the doc here!\n"
     ]
    }
   ],
   "source": [
    "doc1 = nlp(\"I like cats\")\n",
    "doc2 = nlp(\"I like dogs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc similarity:  0.957709143352323\n",
      "Words similarity:  0.83117634\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"a184f9a885234951a8efe0ea8cdad518-0\" class=\"displacy\" width=\"575\" height=\"224.5\" direction=\"ltr\" style=\"max-width: none; height: 224.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">like</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">cats</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-a184f9a885234951a8efe0ea8cdad518-0-0\" stroke-width=\"2px\" d=\"M70,89.5 C70,2.0 225.0,2.0 225.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-a184f9a885234951a8efe0ea8cdad518-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,91.5 L62,79.5 78,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-a184f9a885234951a8efe0ea8cdad518-0-1\" stroke-width=\"2px\" d=\"M245,89.5 C245,2.0 400.0,2.0 400.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-a184f9a885234951a8efe0ea8cdad518-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400.0,91.5 L408.0,79.5 392.0,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cats\n",
      "dogs\n"
     ]
    }
   ],
   "source": [
    "print(\"Doc similarity: \", doc1.similarity(doc2))\n",
    "print(\"Words similarity: \",doc1[2].similarity(doc2[2]))\n",
    "\n",
    "displacy.render(doc1, style=\"dep\")\n",
    "print(doc1[2])\n",
    "print(doc2[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"a2d0aa77123f484bb373df316d222bc1-0\" class=\"displacy\" width=\"575\" height=\"224.5\" direction=\"ltr\" style=\"max-width: none; height: 224.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">like</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"134.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">dogs</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-a2d0aa77123f484bb373df316d222bc1-0-0\" stroke-width=\"2px\" d=\"M70,89.5 C70,2.0 225.0,2.0 225.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-a2d0aa77123f484bb373df316d222bc1-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,91.5 L62,79.5 78,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-a2d0aa77123f484bb373df316d222bc1-0-1\" stroke-width=\"2px\" d=\"M245,89.5 C245,2.0 400.0,2.0 400.0,89.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-a2d0aa77123f484bb373df316d222bc1-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M400.0,91.5 L408.0,79.5 392.0,79.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kbujak/anaconda3/envs/nlp/lib/python3.7/runpy.py:193: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">I like cats</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc2, style=\"dep\")\n",
    "displacy.render(doc1, style=\"ent\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.26763    0.029846  -0.3437    -0.54409   -0.49919    0.15928\n",
      " -0.35278   -0.2036     0.23482    1.5671    -0.36458   -0.028713\n",
      " -0.27053    0.2504    -0.18126    0.13453    0.25795    0.93213\n",
      " -0.12841   -0.18505   -0.57597    0.18538   -0.19147   -0.38465\n",
      "  0.21656   -0.4387    -0.27846   -0.41339    0.37859   -0.2199\n",
      " -0.25907   -0.019796  -0.31885    0.12921    0.22168    0.32671\n",
      "  0.46943   -0.81922   -0.20031    0.013561  -0.14663    0.14438\n",
      "  0.0098044 -0.15439    0.21146   -0.28409   -0.4036     0.45355\n",
      "  0.12173   -0.11516   -0.12235   -0.096467  -0.26991    0.028776\n",
      " -0.11307    0.37219   -0.054718  -0.20297   -0.23974    0.86271\n",
      "  0.25602   -0.3064     0.014714  -0.086497  -0.079054  -0.33109\n",
      "  0.54892    0.20076    0.28064    0.037788   0.0076729 -0.0050123\n",
      " -0.11619   -0.23804    0.33027    0.26034   -0.20615   -0.35744\n",
      "  0.54125   -0.3239     0.093441   0.17113   -0.41533    0.13702\n",
      " -0.21765   -0.65442    0.75733    0.359      0.62492    0.019685\n",
      "  0.21156    0.28125    0.22288    0.026787  -0.1019     0.11178\n",
      "  0.17202   -0.20403   -0.01767   -0.34351    0.11926    0.73156\n",
      "  0.11094    0.12576    0.64825   -0.80004    0.62074   -0.38557\n",
      "  0.015614   0.2664     0.18254    0.11678    0.58919   -1.0639\n",
      " -0.29969    0.14827   -0.42925   -0.090766   0.12313   -0.024253\n",
      " -0.21265   -0.10331    0.91988   -1.4097    -0.0542    -0.071201\n",
      "  0.66878   -0.24651   -0.46788   -0.23991   -0.14138   -0.038911\n",
      " -0.48678    0.22975    0.36074    0.13024   -0.40091    0.19673\n",
      "  0.016017   0.30575   -2.1901    -0.55468    0.26955    0.63815\n",
      "  0.42724   -0.070186  -0.11196    0.14079   -0.022228   0.070456\n",
      "  0.17229    0.099383  -0.12258   -0.23416   -0.26525   -0.088991\n",
      " -0.061554   0.26582   -0.53112   -0.4106     0.45211   -0.39669\n",
      " -0.43746   -0.6632    -0.048135   0.23171   -0.37665   -0.38261\n",
      " -0.29286   -0.036613   0.25354    0.49775    0.3359    -0.11285\n",
      " -0.17228    0.85991   -0.34081    0.27959    0.03698    0.61782\n",
      "  0.23739   -0.32049   -0.073717   0.015991  -0.37395   -0.4152\n",
      "  0.049221  -0.3137     0.091128  -0.38258   -0.036783   0.10902\n",
      " -0.38332   -0.74754    0.016473   0.55256   -0.29053   -0.50617\n",
      "  0.83599   -0.31783   -0.77465   -0.0049272 -0.17103   -0.38067\n",
      "  0.44987   -0.12497    0.60263   -0.12026    0.37368   -0.079952\n",
      " -0.15785    0.37684   -0.18679    0.18855   -0.4759    -0.11708\n",
      "  0.36999    0.54134    0.42752    0.038618   0.043483   0.31435\n",
      " -0.24491   -0.67818   -0.33833    0.039218  -0.11964    0.8474\n",
      "  0.09451    0.070523  -0.2806     0.296     -0.17554   -0.41087\n",
      "  0.70748    0.17686    0.043479  -0.31902    0.64584   -0.45268\n",
      " -0.7967     0.099817  -0.1734     0.11404   -0.36809    0.12035\n",
      " -0.048582   0.55945   -0.51508    0.072704   0.18106    0.07802\n",
      " -0.31526    0.38189    0.092801  -0.044227  -0.66154   -0.020428\n",
      "  0.059836  -0.23628   -0.017592  -0.56481   -0.52934   -0.16392\n",
      "  0.077331   0.24583   -0.32195   -0.36811   -0.037208   0.26702\n",
      " -0.57907    0.46457   -0.54636    0.11855    0.092475  -0.10469\n",
      "  0.03319    0.62616   -0.33684    0.045742   0.25089    0.28973\n",
      "  0.060633  -0.4096     0.39198    0.58276    0.496     -0.75881\n",
      "  0.13655    0.21704   -0.37978   -0.54051   -0.22813    0.28393\n",
      " -0.58739    1.0472    -0.13318   -0.07325    0.12991   -0.44999  ]\n"
     ]
    }
   ],
   "source": [
    "print(doc1[2].vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(doc2[2].vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector Norm and OOV\n",
    "```token.vector_norm``` is L2 norm of the token (the square root of the sum of the values squared) while ```token.is_ovv``` checks if the token is Out-Of-Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22.897898\n",
      "21.888851\n"
     ]
    }
   ],
   "source": [
    "print(doc1[2].vector_norm)\n",
    "print(doc2[2].vector_norm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(doc1[2].is_oov)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Statistical Models vs Rule-based systems\n",
    "Statistical models are useful if your application needs to be able to generalize based on a few examples.\n",
    "\n",
    "Rule-based approaches on the other hand come in handy if there's a more or less finite number of instances you want to find. For example, all countries or cities of the world, drug names or even dog breeds.\n",
    "\n",
    "|                     | Statistical models                                          | Rule-based systems                                     |\n",
    "|:--------------------|:------------------------------------------------------------|:-------------------------------------------------------|\n",
    "| Use cases           | application needs to generalize based on examples           | dictionary with finite number of examples              |\n",
    "| Real-world examples | product names, person names, subject/object relationships   | countries of the world, cities, drug names, dog breeds |\n",
    "| spaCy features      | entity recognizer, dependency parser, part-of-speech tagger | tokenizer, Matcher, PhraseMatcher                      |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n",
      "Matched span: Golden Retriever\n",
      "Root token: Retriever\n",
      "Root head token: have\n",
      "Previous token: a DET\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add('DOG', None, [{'LOWER': 'golden'}, {'LOWER': 'retriever'}])\n",
    "doc = nlp(\"I have a Golden Retriever\")\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    span = doc[start:end]\n",
    "    print('Matched span:', span.text)\n",
    "    print('Root token:', span.root.text)\n",
    "    print('Root head token:', span.root.head.text)\n",
    "    print('Previous token:', doc[start - 1].text, doc[start - 1].pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 1\n",
    "Why does this pattern not match the tokens “Silicon Valley” in the doc?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n"
     ]
    }
   ],
   "source": [
    "pattern = [{'LOWER': 'silicon'}, {'TEXT': ' '}, {'LOWER': 'valley'}]\n",
    "doc = nlp(\"Can silicon valley workers rein in big tech from within?\")\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"SILICON_VALLEY\", None, pattern)\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(\"Matched based on token shape:\", doc[start:end])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n",
      "Matched based on token shape: Silicon Valley\n"
     ]
    }
   ],
   "source": [
    "# The tokenizer doesn't create tokens for single spaces, so there's no token with the value ' ' in between. \n",
    "# The tokenizer already takes care of splitting off whitespace and each dictionary in the pattern describes one token.\n",
    "pattern = [{'LOWER': 'silicon'}, {'LOWER': 'valley'}]\n",
    "doc = nlp(\"Can Silicon Valley workers rein in big tech from within?\")\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"SILICON_VALLEY\", None, pattern)\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(\"Matched based on token shape:\", doc[start:end])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task 2\n",
    "Both patterns in this exercise contain mistakes and won’t match as expected. Can you fix them? \n",
    "* ```pattern1``` so that it correctly matches all case-insensitive mentions of \"Amazon\" plus a title-cased proper noun.\n",
    "* ```pattern2``` so that it correctly matches all case-insensitive mentions of \"ad-free\", plus the following noun.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\n",
    "    \"Twitch Prime, the perks program for Amazon Prime members offering free \"\n",
    "    \"loot, games and other benefits, is ditching one of its best features: \"\n",
    "    \"ad-free viewing. According to an email sent out to Amazon Prime members \"\n",
    "    \"today, ad-free viewing will no longer be included as a part of Twitch \"\n",
    "    \"Prime for new members, beginning on September 14. However, members with \"\n",
    "    \"existing annual subscriptions will be able to continue to enjoy ad-free \"\n",
    "    \"viewing until their subscription comes up for renewal. Those with \"\n",
    "    \"monthly subscriptions will have access to ad-free viewing until October 15.\"\n",
    ")\n",
    "\n",
    "pattern1 = [{\"LOWER\": \"Amazon\"}, {\"IS_TITLE\": True, \"POS\": \"PROPN\"}]\n",
    "pattern2 = [{\"LOWER\": \"ad-free\"}, {\"POS\": \"NOUN\"}]\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"PATTERN1\", None, pattern1)\n",
    "matcher.add(\"PATTERN2\", None, pattern2)\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(doc.vocab.strings[match_id], doc[start:end].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PATTERN1 Amazon Prime\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN1 Amazon Prime\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN2 ad-free viewing\n",
      "PATTERN2 ad-free viewing\n"
     ]
    }
   ],
   "source": [
    "pattern1 = [{\"LOWER\": \"amazon\"}, {\"IS_TITLE\": True, \"POS\": \"PROPN\"}]\n",
    "pattern2 = [{\"LOWER\": \"ad\"}, {\"TEXT\": \"-\"}, {\"LOWER\": \"free\"}, {\"POS\": \"NOUN\"}]\n",
    "\n",
    "matcher = Matcher(nlp.vocab)\n",
    "matcher.add(\"PATTERN1\", None, pattern1)\n",
    "matcher.add(\"PATTERN2\", None, pattern2)\n",
    "\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(doc.vocab.strings[match_id], doc[start:end].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exact string matching\n",
    "Sometimes it’s more efficient to match exact strings instead of writing patterns describing the individual tokens. This is especially true for finite categories of things – like all countries of the world. We already have a list of countries, so let’s use this as the basis of our information extraction script. A list of string names is available as the variable COUNTRIES."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Czech Republic may help Slovakia protect its airspace\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "Do something to the doc here!\n",
      "[Czech Republic, Slovakia]\n"
     ]
    }
   ],
   "source": [
    "with open(\"data/countries.json\") as f:\n",
    "    COUNTRIES = json.loads(f.read())\n",
    "\n",
    "patterns = list(nlp.pipe(COUNTRIES))\n",
    "\n",
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "matcher.add(\"COUNTRY\", None, *patterns)\n",
    "\n",
    "print([doc[start:end] for match_id, start, end in matcher(doc)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Pipelines\n",
    "spaCy ships with the following built-in pipeline components. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What happens when we call nlp\n",
    "First, the tokenizer is applied to turn the string of text into a Doc object. Next, a series of pipeline components is applied to the Doc in order. In this case, the tagger, then the parser, then the entity recognizer. Finally, the processed Doc is returned, so you can work with it.\n",
    "\n",
    "<img src=\"img/pipeline.png\" >\n",
    "<br clear=\"left\"/>\n",
    "\n",
    "The part-of-speech tagger sets the ```token.tag``` attribute. The dependency parser adds the token dot dep and token dot head attributes and is also responsible for detecting sentences and base noun phrases, also known as noun chunks. The named entity recognizer adds the detected entities to the| doc dot ents property. It also sets entity type attributes on the tokens that indicate if a token is part of an entity or not. Finally, the text classifier sets category labels that apply to the whole text, and adds them to the doc dot cats property. Because text categories are always very specific, the text classifier is not included in any of the pre-trained models by default. But you can use it to train your own system.\n",
    "\n",
    "\n",
    "| Name    | Description             | Creates                                           |\n",
    "|:--------|:------------------------|:--------------------------------------------------|\n",
    "| tagger  | Part-of-speech tagger   | Token.tag                                         |\n",
    "| parser  | Dependency parser       | Token.dep, Token.head, Doc.sents, Doc.noun_chunks |\n",
    "| ner     | Named entity recognizer | Doc.ents, Token.ent_iob, Token.ent_type           |\n",
    "| textcat | Text classifier         | Doc.cats                                          |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tagger', 'parser', 'ner']"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = en_core_web_sm.load()\n",
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('tagger', <spacy.pipeline.pipes.Tagger at 0x7fc24c9579d0>),\n",
       " ('parser', <spacy.pipeline.pipes.DependencyParser at 0x7fc1ea430d70>),\n",
       " ('ner', <spacy.pipeline.pipes.EntityRecognizer at 0x7fc1ea430f30>)]"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "### Custom pipeline components\n",
    "\n",
    "After the text is tokenized and a ```Doc``` object has been created, pipeline components are applied in order. spaCy supports a range of built-in components, but also lets you define your own. Custom components are executed automatically when you call the ```nlp``` object on a text. They're especially useful for adding your own custom metadata to documents and tokens. You can also use them to update built-in attributes, like the named entity spans.\n",
    "\n",
    "Fundamentally, a pipeline component is a function or callable that takes a ```doc```, modifies it and returns it, so it can be processed by the next component in the pipeline. Components can be added to the pipeline using the ```nlp.add_pipe``` method. The method takes at least one argument: the component function. \n",
    "\n",
    "Don't forget to return the ```Doc``` so it can be processed by the next component in the pipeline! The Doc created by the tokenizer is passed through all components, so it's important that they all return the modified doc.\n",
    "\n",
    "| Argument | Description          | Example                                 |\n",
    "|:---------|:---------------------|:----------------------------------------|\n",
    "| last     | If True, add last    | nlp.add_pipe(component, last=True)      |\n",
    "| first    | If True, add first   | nlp.add_pipe(component, first=True)     |\n",
    "| before   | Add before component | nlp.add_pipe(component, before='ner')   |\n",
    "| after    | Add after component  | nlp.add_pipe(component, after='tagger') |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def custom_component(doc):\n",
    "    print(\"Do something to the doc here!\")\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp.add_pipe(custom_component, first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('custom_component', <function custom_component at 0x7fc24670f560>), ('tagger', <spacy.pipeline.pipes.Tagger object at 0x7fc24c9579d0>), ('parser', <spacy.pipeline.pipes.DependencyParser object at 0x7fc1ea430d70>), ('ner', <spacy.pipeline.pipes.EntityRecognizer object at 0x7fc1ea430f30>)]\n"
     ]
    }
   ],
   "source": [
    "print([pip for pip in nlp.pipeline])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = en_core_web_lg.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excercise 1 (Simple component)\n",
    "Before we run the tagger, we want to know the length of ```Doc``` object in tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['length_component', 'tagger', 'parser', 'ner']\n"
     ]
    }
   ],
   "source": [
    "# Load the small English model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "def length_component(doc):\n",
    "    doc_length = len(doc)\n",
    "    print(\"This document is {} tokens long.\".format(doc_length))\n",
    "    return doc\n",
    "\n",
    "nlp.add_pipe(length_component, first=True)\n",
    "print(nlp.pipe_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This document is 5 tokens long.\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"This is a sentence.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excercise 2 (Complex Component)\n",
    "Use the ```PhraseMatcher``` to find animal names in the document and adds the matched spans to the ```doc.ents```. A ```PhraseMatcher``` with the animal patterns has already been created as the variable matcher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This document is 2 tokens long.\n",
      "This document is 1 tokens long.\n",
      "This document is 1 tokens long.\n",
      "This document is 2 tokens long.\n",
      "animal_patterns: [Golden Retriever, cat, turtle, Rattus norvegicus]\n",
      "['length_component', 'tagger', 'parser', 'ner', 'animal_component']\n",
      "This document is 8 tokens long.\n",
      "[('cat', 'ANIMAL'), ('golden Retriever', 'ANIMAL')]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">I have a \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    cat\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ANIMAL</span>\n",
       "</mark>\n",
       " and a \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    golden Retriever\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ANIMAL</span>\n",
       "</mark>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "animals = [\"Golden Retriever\", \"cat\", \"turtle\", \"Rattus norvegicus\"]\n",
    "animal_patterns = list(nlp.pipe(animals))\n",
    "print(\"animal_patterns:\", animal_patterns)\n",
    "matcher = PhraseMatcher(nlp.vocab, attr=\"LOWER\")\n",
    "matcher.add(\"ANIMAL\", None, *animal_patterns)\n",
    "\n",
    "def animal_component(doc):\n",
    "    matches = matcher(doc)\n",
    "    # Create a Span for each match and assign the label 'ANIMAL'\n",
    "    spans = [Span(doc, start, end, label=\"ANIMAL\") for match_id, start, end in matches]\n",
    "    # Overwrite the doc.ents with the matched spans\n",
    "    doc.ents = spans\n",
    "    return doc\n",
    "\n",
    "nlp.add_pipe(animal_component, after=\"ner\")\n",
    "print(nlp.pipe_names)\n",
    "\n",
    "doc = nlp(\"I have a cat and a golden Retriever\")\n",
    "print([(ent.text, ent.label_) for ent in doc.ents])\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "### Extension attributes\n",
    "Custom attributes let you add any meta data to ```Docs```, ```Tokens``` and ```Spans```. The data can be added once, or it can be computed dynamically. Custom attributes are available via the ```._``` property. \n",
    "```python\n",
    "doc._.title = 'My document'\n",
    "token._.is_color = True\n",
    "span._.has_color = False\n",
    "```\n",
    "This makes it clear that they were added by the user, and not built into spaCy, like ```token.text```.  Attributes need to be registered on the global ```Doc```, ```Token``` and ```Span``` classes you can import from ```spacy.tokens```. To register a custom attribute on the ```Doc```, ```Token``` or ```Span```, you can use the ```set_extension``` method.\n",
    "\n",
    "The first argument is the attribute name. Keyword arguments let you define how the value should be computed. In this case, it has a default value and can be overwritten."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "Doc.set_extension('title', default=None)\n",
    "Token.set_extension('is_color', default=False)\n",
    "Span.set_extension('has_color', default=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attribute extensions\n",
    "Attribute extensions set a default value that can be overwritten. For example, a custom ```is_color``` attribute on the token that defaults to ```False```. We can also add ```force=True``` to force the process of overwrite. \n",
    "\n",
    "On individual tokens, its value can be changed by overwriting it – in this case, ```True``` for the token ```blue```.\n",
    "\n",
    "```python\n",
    "Token.set_extension('is_color', default=False, force=True)\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "doc[3]._.is_color = True\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "#### Property extensions\n",
    "\n",
    "Property extensions work like properties in Python: they can define a ```getter``` function and an optional ```setter```. The ```getter``` function is only called when you retrieve the attribute. This lets you compute the value dynamically, and even take other custom attributes into account. ```Getter``` functions take one argument: the object, in this case, the token. In this example, the function returns whether the token text is in our list of colors. We can then provide the function via the getter keyword argument when we register the extension. \n",
    "\n",
    "The token \"blue\" now returns True for \"is color\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "def get_is_color(token):\n",
    "    colors = ['red', 'yellow', 'blue', 'green']\n",
    "    return token.text in colors\n",
    "\n",
    "# Set extension on the Token with getter\n",
    "Token.set_extension('is_color', getter=get_is_color, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue. Roses are red. Grass is green.\")\n",
    "print([str(token._.is_color) + ' - ' + token.text for token in doc if token._.is_color == True])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Spans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from spacy.tokens import Span\n",
    "\n",
    "def get_has_color(span):\n",
    "    colors = ['red', 'yellow', 'blue']\n",
    "    return any(token.text in colors for token in span)\n",
    "\n",
    "Span.set_extension('has_color', getter=get_has_color, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "print(doc[1:4]._.has_color, '-', doc[1:4].text)\n",
    "print(doc[0:2]._.has_color, '-', doc[0:2].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method extensions\n",
    "Method extensions make the extension attribute a callable method. You can then pass one or more arguments to it, and compute attribute values dynamically – for example, based on a certain argument or setting.\n",
    "\n",
    "In this example, the method function checks whether the ```doc``` contains a ```token``` with a given text. The first argument of the method is always the object itself – in this case, the ```Doc```. It's passed in automatically when the method is called. All other function arguments will be arguments on the method extension. In this case, ```token_text```.\n",
    "\n",
    "Here, the custom ```has_token``` method returns ```True``` for the word \"blue\" and ```False``` for the word \"cloud\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def has_token(doc, token_text):\n",
    "    in_doc = token_text in [token.text for token in doc]\n",
    "    return in_doc\n",
    "\n",
    "Doc.set_extension('has_token', method=has_token, force=True)\n",
    "\n",
    "doc = nlp(\"The sky is blue.\")\n",
    "print(doc._.has_token('blue'), '- blue')\n",
    "print(doc._.has_token('cloud'), '- cloud')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excercise 3\n",
    "Check if the ```Doc``` objecxt has number inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = English()\n",
    "\n",
    "def get_has_number(doc):\n",
    "    return any(token.like_num for token in doc)\n",
    "\n",
    "Doc.set_extension(\"has_number\", getter=get_has_number, force=True)\n",
    "\n",
    "doc = nlp(\"The museum closed for five years in 2012.\")\n",
    "print(\"has_number:\", doc._.has_number)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excercise 4\n",
    "Wrap ```Span``` into XML tags with ```to_html``` attribute and return it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = English()\n",
    "\n",
    "def to_html(span, tag):\n",
    "    return \"<{tag}>{text}</{tag}>\".format(tag=tag, text=span.text)\n",
    "\n",
    "Span.set_extension(\"to_html\", method=to_html, force=True)\n",
    "\n",
    "doc = nlp(\"Hello world, this is a sentence.\")\n",
    "span = doc[0:2]\n",
    "print(span._.to_html(\"strong\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Excerice 5\n",
    "Components with extensions. Extension attributes are especially powerful if they’re combined with custom pipeline components. Write a pipeline component that finds country names and a custom extension attribute that returns a country’s capital, if available.\n",
    "List of countries is in ```data/countries.json```. Capitals are in ```data/capitals.json``` and the text to check is in ```data/country_text.txt```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "with open(\"data/countries.json\") as f:\n",
    "    COUNTRIES = json.loads(f.read())\n",
    "\n",
    "with open(\"data/capitals.json\") as f:\n",
    "    CAPITALS = json.loads(f.read())\n",
    "    \n",
    "with open(\"data/country_text.txt\") as f:\n",
    "    TEXT = f.read()\n",
    "\n",
    "nlp = English()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "matcher = PhraseMatcher(nlp.vocab)\n",
    "matcher.add(\"COUNTRY\", None, *list(nlp.pipe(COUNTRIES)))\n",
    "\n",
    "def countries_component(doc):\n",
    "    matches = matcher(doc)\n",
    "    doc.ents = [Span(doc, start, end, label=\"GPE\") for match_id, start, end in matches]\n",
    "    return doc\n",
    "\n",
    "nlp.add_pipe(countries_component)\n",
    "print(nlp.pipe_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "get_capital = lambda span: CAPITALS.get(span.text)\n",
    "Span.set_extension(\"capital\", getter=get_capital, force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(TEXT)\n",
    "print([(ent.text, ent.label_, ent._.capital) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling\n",
    "If you need to process a lot of texts and create a lot of ```Doc``` objects in a row, the ```nlp.pipe``` method can speed this up significantly. It processes the texts as a stream and yields ```Doc``` objects. It is much faster than just calling ```nlp``` on each text, because **it batches up the texts**. ```nlp.pipe``` is a generator that yields ```Doc``` objects, so in order to get a list of ```Docs```, remember to call the list method around it.\n",
    "\n",
    "**BAD**:\n",
    "```python\n",
    "docs = [nlp(text) for text in LOTS_OF_TEXTS]\n",
    "```\n",
    "**GOOD**:\n",
    "```python\n",
    "docs = list(nlp.pipe(LOTS_OF_TEXTS))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Passing in context\n",
    "```nlp.pipe``` also supports passing in tuples of text / context if you set ```as_tuples=True```. The method will then yield doc / context tuples. This is useful for passing in additional metadata, like an ID associated with the text, or a page number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "Doc.set_extension('id', default=None, force=True)\n",
    "Doc.set_extension('page_number', default=None, force=True)\n",
    "\n",
    "data = [\n",
    "    ('This is a text', {'id': 1, 'page_number': 15}),\n",
    "    ('And another text', {'id': 2, 'page_number': 16}),\n",
    "]\n",
    "\n",
    "for doc, context in nlp.pipe(data, as_tuples=True):\n",
    "    doc._.id = context['id']\n",
    "    doc._.page_number = context['page_number']\n",
    "    print(\"ID: {0}, Page Number: {1}, Content: {2}\".format(doc._.id, doc._.page_number, doc.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "### Performance\n",
    "Another common scenario: Sometimes you already have a model loaded to do other processing, but you only need the tokenizer for one particular text. Running the whole pipeline is unnecessarily slow, because you'll be getting a bunch of predictions from the model that you don't need.\n",
    "<img src=\"img/pipeline.png\" >\n",
    "<br clear=\"left\"/>\n",
    "\n",
    "If you only need a tokenized ```Doc``` object, you can use the ```nlp.make_doc``` method instead, which takes a text and returns a ```Doc```. This is also how spaCy does it behind the scenes: ```nlp.make_doc``` turns the text into a ```Doc``` before the pipeline components are called.\n",
    "\n",
    "**BAD**: \n",
    "```python\n",
    "doc = nlp(\"Hello world\")\n",
    "```\n",
    "**GOOD**:\n",
    "```python\n",
    "doc = nlp.make_doc(\"Hello world!\")\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<spacy.lang.en.English object at 0x7fc232fc8550> complete 0.6604928970336914\n",
      "<spacy.lang.en.English object at 0x7fc232fc8550> make_doc 0.00038242340087890625\n",
      "<spacy.lang.en.English object at 0x7fc20c765c50> complete 0.007763385772705078\n",
      "<spacy.lang.en.English object at 0x7fc20c765c50> make_doc 0.0002598762512207031\n",
      "<spacy.lang.en.English object at 0x7fc20c765590> complete 0.00023317337036132812\n",
      "<spacy.lang.en.English object at 0x7fc20c765590> make_doc 0.00016069412231445312\n"
     ]
    }
   ],
   "source": [
    "nlp1 = en_core_web_lg.load()\n",
    "nlp2 = en_core_web_sm.load()\n",
    "nlp3 = English()\n",
    "\n",
    "for nlp in [nlp1, nlp2, nlp3]:\n",
    "    for method in ['complete', 'make_doc']:\n",
    "        if method == 'complete':\n",
    "            start_time = time.time()\n",
    "            doc = nlp(\"Hello world\")\n",
    "            stop_time = time.time() - start_time\n",
    "            print(nlp, method ,stop_time)\n",
    "        else:\n",
    "            start_time = time.time()\n",
    "            doc = nlp.make_doc(\"Hello world!\")\n",
    "            stop_time = time.time() - start_time\n",
    "            print(nlp, method ,stop_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Disabling pipeline components\n",
    "spaCy also allows you to temporarily disable pipeline components using the ```nlp.disable_pipes``` context manager. It takes a variable number of arguments, the string names of the pipeline components to disable. For example, if you only want to use the entity recognizer to process a document, you can temporarily disable the tagger and parser. After the with block, the disabled pipeline components are automatically restored. In the with block, spaCy will only run the remaining components.\n",
    "\n",
    "```python\n",
    "with nlp.disable_pipes('tagger', 'parser'):\n",
    "    doc = nlp(text)\n",
    "    print(doc.ents)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "with open(\"data/tweets.json\") as f:\n",
    "    TEXTS = json.loads(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "for doc in nlp.pipe(TEXTS):\n",
    "    print([token.text for token in doc if token.pos_ == \"ADJ\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "docs = list(nlp.pipe(TEXTS))\n",
    "entities = [doc.ents for doc in docs]\n",
    "print(*entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and updating model\n",
    "\n",
    "spaCy’s models are **statistical** and every “decision” they make – for example, which ```part-of-speech``` tag to assign, or whether a word is a named entity – is a **prediction**. This prediction is based on the examples the model has seen during **training**. To train a model, you first need training data – examples of text, and the labels you want the model to predict. This could be a part-of-speech tag, a named entity or any other information.\n",
    "\n",
    "The model is then shown the unlabelled text and will make a prediction. Because we know the correct answer, we can give the model feedback on its prediction in the form of an **error gradient** of the **loss function** that calculates the difference between the training example and the expected output. The greater the difference, the more significant the gradient and the updates to our model.\n",
    "\n",
    "<img src=\"img/training.png\" >\n",
    "<br clear=\"left\"/>\n",
    "\n",
    "* **Training data**: Examples and their annotations.\n",
    "* **Text**: The input text the model should predict a label for.\n",
    "* **Label**: The label the model should predict.\n",
    "* **Gradient**: How to change the weights.\n",
    "\n",
    "**Why updating the model?**\n",
    "* Better results on your specific domain\n",
    "* Learn classification schemes specifically for your problem\n",
    "* Essential for text classification\n",
    "* Very useful for named entity recognition\n",
    "* Less critical for part-of-speech tagging and dependency parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where to get training data?\n",
    "Collecting training data may sound incredibly painful – and it can be, if you’re planning a large-scale annotation project. \n",
    "\n",
    "spaCy’s rule-based ```Matcher``` is a great way to quickly create training data for named entity models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "with open(\"data/iphone.json\") as f:\n",
    "    TEXTS = json.loads(f.read())\n",
    "\n",
    "nlp = English()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern1 = [{\"LOWER\": \"iphone\"}, {\"LOWER\": \"x\"}]\n",
    "pattern2 = [{\"LOWER\": \"iphone\"}, {\"IS_DIGIT\": True, \"OP\": \"?\"}]\n",
    "matcher.add(\"GADGET\", None, pattern1, pattern2)\n",
    "\n",
    "TRAINING_DATA = []\n",
    "\n",
    "# Create a Doc object for each text in TEXTS\n",
    "for doc in nlp.pipe(TEXTS):\n",
    "    # Match on the doc and create a list of matched spans\n",
    "    spans = [doc[start:end] for match_id, start, end in matcher(doc)]\n",
    "    # Get (start character, end character, label) tuples of matches\n",
    "    entities = [(span.start_char, span.end_char, \"GADGET\") for span in spans]\n",
    "    # Format the matches as a (doc.text, entities) tuple\n",
    "    training_example = (doc.text, {\"entities\": entities})\n",
    "    # Append the example to the training data\n",
    "    TRAINING_DATA.append(training_example)\n",
    "\n",
    "print(*TRAINING_DATA, sep=\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "with open(\"data/gadgets.json\") as f:\n",
    "    TRAINING_DATA = json.loads(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create new pipe\n",
    "We start off with a blank English model using the spacy dot blank method. The blank model doesn't have any pipeline components, only the language data and tokenization rules. We then create a blank entity recognizer and add it to the pipeline. Using the \"add label\" method, we can add new string labels to the model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "nlp = spacy.blank(\"en\")\n",
    "ner = nlp.create_pipe(\"ner\")\n",
    "nlp.add_pipe(ner)\n",
    "ner.add_label(\"GADGET\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problems with training\n",
    "When you start running your own experiments, you might find that a lot of things just don't work the way you want them to. And that's okay. Training models is an iterative process, and you have to try different things until you find out what works best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 1: Models can \"forget\" things\n",
    "\n",
    "Statistical models can learn lots of things – but it doesn't mean that they won't unlearn them. If you're updating an existing model with new data, especially new labels, it can **overfit** and **adjust too much to the new examples**. For instance, if you're only updating it with examples of ```WEBSITE```, it may \"forget\" other labels it previously predicted correctly – like ```PERSON```. This is also known as the catastrophic forgetting problem.\n",
    "\n",
    "**TL;DR**\n",
    "* Existing model can overfit on new data e.g.: if you only update it with ```WEBSITE```, it can \"unlearn\" what a ```PERSON``` is\n",
    "* Also known as **catastrophic forgetting** problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Solution 1\n",
    "To prevent this, make sure to always mix in examples of what the model previously got correct. If you're training a new category ```WEBSITE```, also include examples of ```PERSON```. You can create those additional examples by running the existing model over data and extracting the entity spans you care about. You can then mix those examples in with your existing data and update the model with annotations of all labels.\n",
    "\n",
    "**BAD**:\n",
    "```json\n",
    "TRAINING_DATA = [\n",
    "    ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]})\n",
    "]\n",
    "```\n",
    "**GOOD**:\n",
    "```json\n",
    "TRAINING_DATA = [\n",
    "    ('Reddit is a website', {'entities': [(0, 6, 'WEBSITE')]}),\n",
    "    ('Obama is a person', {'entities': [(0, 5, 'PERSON')]})\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2: Models can't learn everything\n",
    "\n",
    "Another common problem is that your model just won't learn what you want it to. spaCy's models make **predictions based on the local context** – for example, for named entities, the surrounding words are most important. **If the decision is difficult to make based on the context, the model can struggle to learn it**. The label scheme also needs to be consistent and not too specific. For example, it may be very difficult to teach a model to predict whether something is ```ADULT_CLOTHING``` or ```CHILDRENS_CLOTHING``` based on the context. However, just predicting the label ```CLOTHING``` may work better.\n",
    "\n",
    "* spaCy's models make predictions based on local context\n",
    "* Model can struggle to learn if decision is difficult to make based on context\n",
    "* Label scheme needs to be consistent and not too specific. For example: ```CLOTHING``` is better than ```ADULT_CLOTHING``` and ```CHILDRENS_CLOTHING```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Solution 2\n",
    "Before you start training and updating models, it's worth taking a step back and planning your label scheme. Try to pick categories that are reflected in the local context and make them more generic if possible. You can always add a rule-based system later to go from generic to specific. Generic categories like \"clothing\" or \"band\" are both easier to label and easier to learn.\n",
    "\n",
    "**BAD**:\n",
    "```json\n",
    "LABELS = ['ADULT_SHOES', 'CHILDRENS_SHOES', 'BANDS_I_LIKE']\n",
    "```\n",
    "**GOOD**:\n",
    "```json\n",
    "LABELS = ['CLOTHING', 'BAND']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TEST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question I\n",
    "\n",
    "Which sentence is correct about Strings in python:\n",
    "\n",
    "1) Strings are fancy underwear\n",
    "\n",
    "**2) Strings are immutable arrays**\n",
    "\n",
    "3) Strings are mutable\n",
    "\n",
    "4) We can use only double quote (\") for Strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question II\n",
    "\n",
    "What is required for Lemmatization (greedy)?\n",
    "\n",
    "1) Sentence tokenization\n",
    "\n",
    "2) Words tokenization\n",
    "\n",
    "3) Stemming\n",
    "\n",
    "**4) Words tokenization and Parts of speech**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question III \n",
    "\n",
    "What are the examples of Stop Words in english:\n",
    "\n",
    "1) 'yes', 'no', 'gimme more'\n",
    "\n",
    "2) 'pneumonoultramicroscopicsilicovolcanoconiosis'\n",
    "\n",
    "3) 'stop'\n",
    "\n",
    "**4) 'a', 'the'**\n",
    "\n",
    "https://en.wikipedia.org/wiki/Pneumonoultramicroscopicsilicovolcanoconiosis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question IV\n",
    "\n",
    "How many bi-grams can we generate from the given sentence:\n",
    "“Cisco is a great place to work”\n",
    "\n",
    "1) 7\n",
    "\n",
    "**2) 6**\n",
    "\n",
    "3) 5\n",
    "\n",
    "4) 10\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question V \n",
    "\n",
    "Collaborative Filtering and Content Based Models are the two popular recommendation engines, what role does NLP play in building such algorithm?\n",
    "\n",
    "1) Feature Extraction from text\n",
    "\n",
    "2) Measuring Feature Similarity\n",
    "\n",
    "3) Engineering Features for vector space learning model\n",
    "\n",
    "**4) All of these**\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question VII\n",
    "What’s not included in a model package that you can load into spaCy?\n",
    "\n",
    "1) A meta file including the language, pipeline and license.\n",
    "\n",
    "2) Binary weights to make statistical predictions.\n",
    "\n",
    "**3) The labelled data that the model was trained on.**\n",
    "\n",
    "4) Strings of the model's vocabulary and their hashes.\n",
    "\n",
    "**Note:** Statistical models allow you to generalize based on a set of training examples. Once they’re trained, they use binary weights to make predictions. That’s why it’s not necessary to ship them with their training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question VIII \n",
    "Why does this code throw an error?\n",
    "```python\n",
    "from spacy.lang.en import English\n",
    "from spacy.lang.de import German\n",
    "\n",
    "# Create an English and German nlp object\n",
    "nlp = English()\n",
    "nlp_de = German()\n",
    "\n",
    "# Get the ID for the string 'Bowie'\n",
    "bowie_id = nlp.vocab.strings['Bowie']\n",
    "print(bowie_id)\n",
    "\n",
    "# Look up the ID for 'Bowie' in the vocab\n",
    "print(nlp_de.vocab.strings[bowie_id])\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "**1) The string ```'Bowie'``` isn't in the German vocab, so the hash can't be resolved in the string store.**\n",
    "\n",
    "2) ```'Bowie'``` is not a regular word in the English or German dictionary, so it can't be hashed.\n",
    "\n",
    "3) ```nlp_de``` is not a valid name. The vocab can only be shared if the ```nlp``` objects have the same name.\n",
    "\n",
    "**Note:** Hashes can’t be reversed. To prevent this problem, add the word to the new vocab by processing a text or looking up the string, or use the same vocab to resolve the hash back to a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question IX\n",
    "\n",
    "What does spaCy do when you call ```nlp``` on a string of text?\n",
    "```python\n",
    "doc = nlp(\"This is a sentence.\")\n",
    "```\n",
    "1) Run the tagger, parser and entity recognizer and then the tokenizer.\n",
    "\n",
    "**2) Tokenize the text and apply each pipeline component in order.**\n",
    "\n",
    "3) Connect to the spaCy server to compute the result and return it.\n",
    "\n",
    "4) Initialize the language, add the pipeline and load in the binary model weights.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question X\n",
    "\n",
    "Here’s an excerpt from a training set that labels the entity type TOURIST_DESTINATION in traveler reviews.\n",
    "```json\n",
    "TRAINING_DATA = [\n",
    "    (\n",
    "        \"i went to amsterdem last year and the canals were beautiful\",\n",
    "        {\"entities\": [(10, 19, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "    (\n",
    "        \"You should visit Paris once in your life, but the Eiffel Tower is kinda boring\",\n",
    "        {\"entities\": [(17, 22, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "    (\"There's also a Paris in Arkansas, lol\", {\"entities\": []}),\n",
    "    (\n",
    "        \"Berlin is perfect for summer holiday: lots of parks, great nightlife, cheap beer!\",\n",
    "        {\"entities\": [(0, 6, \"TOURIST_DESTINATION\")]},\n",
    "    ),\n",
    "]\n",
    "```\n",
    "Why is this data and label scheme problematic?\n",
    "\n",
    "**1) Whether a place is a tourist destination is a subjective judgement and not a definitive category. It will be very difficult for the entity recognizer to learn.**\n",
    "\n",
    "2) Paris and Arkansas should also be labelled as tourist destinations for consistency. Otherwise, the model will be confused.\n",
    "\n",
    "3) Rare out-of-vocabulary words like the misspelled 'amsterdem' shouldn't be labelled as entities.\n",
    "\n",
    "**Note:** A much better approach would be to only label ```GPE``` (geopolitical entity) or ```LOCATION``` and then use a rule-based system to determine whether the entity is a tourist destination in this context. For example, you could resolve the entities types back to a knowledge base or look them up in a travel wiki. \n",
    "\n",
    "Even very uncommon words or misspellings can be labelled as entities. In fact, **being able to predict categories in misspelled text based on the context is one of the big advantages of statistical named entity recognition**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question XI \n",
    "\n",
    "A model was trained with the data you just labelled, plus a few thousand similar examples. After training, it’s doing great on ```WEBSITE```, but doesn’t recognize ```PERSON``` anymore. Why could this be happening?\n",
    "\n",
    "1) It's very difficult for the model to learn about different categories like PERSON and WEBSITE.\n",
    "\n",
    "**2) The training data included no examples of PERSON, so the model learned that this label is incorrect.** \n",
    "\n",
    "3) The hyperparameters need to be retuned so that both entity types can be recognized.\n",
    "\n",
    "**Note:** If ```PERSON``` entities occur in the training data but aren’t labelled, the model will learn that they shouldn’t be predicted. Similarly, if an existing entity type isn’t present in the training data, the model may ”forget” and stop predicting it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "[0. urllib and bs4 and textacy]()\n",
    "\n",
    "[1. Rule-based entity](https://spacy.io/usage/rule-based-matching#entityruler)\n",
    "\n",
    "[2. Pipelines](https://spacy.io/usage/processing-pipelines)\n",
    "\n",
    "[3. Training models](https://spacy.io/usage/training#basics)\n",
    "\n",
    "[4. Stanfords CS224n](http://web.stanford.edu/class/cs224n/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}