notebook/06_spaCy_Matcher.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. HOw to use the spaCy Matcher\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy \n",
    "from spacy.matcher import Matcher"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Basic Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(16571425990740197027, 5, 6)]\n"
     ]
    }
   ],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "# print(nlp)\n",
    "matcher = Matcher(nlp.vocab)\n",
    "# print(matcher)\n",
    "pattern = [{\"LIKE_EMAIL\":True}]\n",
    "# print(pattern)\n",
    "matcher.add(\"EMAIL_ADDRESS\", [pattern])\n",
    "# print(matcher)\n",
    "doc = nlp(\"This is email address: manish@outlook.com\")\n",
    "matches = matcher(doc)\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line of code matcher = Matcher(nlp.vocab) is used in the context of spaCy, an advanced Natural Language Processing (NLP) library in Python. Here’s what it does:\n",
    "\n",
    "**Matcher**: This is a class provided by spaCy that allows you to match sequences of tokens based on patterns you define. It’s a powerful tool for finding words and phrases in text using rule-based matching.\n",
    "**nlp.vocab**: The nlp object is typically a spaCy Language instance that contains the processing pipeline and various NLP tools. The .vocab attribute of nlp is a storage container for vocabulary items and has data on word types, including their lexical attributes.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output [(16571425990740197027, 5, 6)] is a list of tuples that you would typically receive from using spaCy’s Matcher object. Here’s what each element of the tuple represents:\n",
    "\n",
    "* 16571425990740197027: This is the match ID, a unique identifier for the matched rule. In spaCy, match IDs are usually the hash value of the rule name added to the Matcher. For example, if you added a rule named “EMAIL_ADDRESS”, this number would correspond to the hash of that string.\n",
    "* 5: This is the start index of the match in the Doc object. It indicates the position of the first token in the document that matches the pattern.\n",
    "* 6: This is the end index of the match in the Doc object. It indicates the position right after the last token in the document that matches the pattern.\n",
    "* So, in the context of your previous code, this output means that the Matcher found one match for an email address in the text, starting at the fifth token and ending before the sixth token. Given the input text “This is email address: manish@outlook.com”, the match would be “manish@outlook.com”, which is the email address identified by the pattern [{\"LIKE_EMAIL\":True}]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EMAIL_ADDRESS\n"
     ]
    }
   ],
   "source": [
    "print(nlp.vocab[matches[0][0]].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2. Attributes Taken by Matcher¶\n",
    "* ORTH - The exact verbatim of a token (str)\n",
    "\n",
    "* TEXT - The exact verbatim of a token (str)\n",
    "\n",
    "* LOWER - The lowercase form of the token text (str)\n",
    "\n",
    "* LENGTH - The length of the token text (int)\n",
    "\n",
    "* IS_ALPHA\n",
    "\n",
    "* IS_ASCII\n",
    "\n",
    "* IS_DIGIT\n",
    "\n",
    "* IS_LOWER\n",
    "\n",
    "* IS_UPPER\n",
    "\n",
    "* IS_TITLE\n",
    "\n",
    "* IS_PUNCT\n",
    "\n",
    "* IS_SPACE\n",
    "\n",
    "* IS_STOP\n",
    "\n",
    "* IS_SENT_START\n",
    "\n",
    "* LIKE_NUM\n",
    "\n",
    "* LIKE_URL\n",
    "\n",
    "* LIKE_EMAIL\n",
    "\n",
    "* SPACY\n",
    "\n",
    "* POS\n",
    "\n",
    "* TAG\n",
    "\n",
    "* MORPH\n",
    "\n",
    "* DEP\n",
    "\n",
    "* LEMMA\n",
    "\n",
    "* SHAPE\n",
    "\n",
    "* ENT_TYPE\n",
    "\n",
    "_ - Custom extension attributes (Dict[str, Any])\n",
    "\n",
    "* OP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 Applied Matcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<>:1: SyntaxWarning: invalid escape sequence '\\s'\n",
      "<>:1: SyntaxWarning: invalid escape sequence '\\s'\n",
      "C:\\Users\\Manjusha Kumari\\AppData\\Local\\Temp\\ipykernel_8252\\2756658730.py:1: SyntaxWarning: invalid escape sequence '\\s'\n",
      "  with open (\"F:\\spaCy-master\\data\\wiki_mlk.txt\", \"r\") as f:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<spacy.lang.en.English object at 0x0000029FAFD303E0>\n"
     ]
    }
   ],
   "source": [
    "with open (\"F:\\spaCy-master\\data\\wiki_mlk.txt\", \"r\") as f:\n",
    "    text = f.read()\n",
    "    # print(text)\n",
    "    nlp = spacy.load(\"en_core_web_sm\")\n",
    "    # print(nlp)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3232560085755078826, 0, 1) Martin\n",
      "(3232560085755078826, 1, 2) Luther\n",
      "(3232560085755078826, 2, 3) King\n",
      "(3232560085755078826, 3, 4) Jr.\n",
      "(3232560085755078826, 6, 7) Michael\n",
      "(3232560085755078826, 7, 8) King\n",
      "(3232560085755078826, 8, 9) Jr.\n",
      "(3232560085755078826, 10, 11) January\n",
      "(3232560085755078826, 15, 16) April\n",
      "(3232560085755078826, 23, 24) Baptist\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"POS\":\"PROPN\"}]  # proper noun \n",
    "matcher.add(\"PROPER_NOUNS\", [pattern])\n",
    "doc = nlp(text)\n",
    "matches = matcher(doc)\n",
    "# print(len(matches)\n",
    "for match in matches[:10]:\n",
    "    print(match, doc[match[1]: match[2]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4.1 Improving it with Multi-Word Tokens \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(451313080118390996, 0, 1) Martin\n",
      "(451313080118390996, 0, 2) Martin Luther\n",
      "(451313080118390996, 1, 2) Luther\n",
      "(451313080118390996, 0, 3) Martin Luther King\n",
      "(451313080118390996, 1, 3) Luther King\n",
      "(451313080118390996, 2, 3) King\n",
      "(451313080118390996, 0, 4) Martin Luther King Jr.\n",
      "(451313080118390996, 1, 4) Luther King Jr.\n",
      "(451313080118390996, 2, 4) King Jr.\n",
      "(451313080118390996, 3, 4) Jr.\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"POS\":\"PROPN\", \"OP\":\"+\"}]\n",
    "matcher.add(\"PROPER_NOUN\",[pattern])\n",
    "doc = nlp(text)\n",
    "# print(doc)\n",
    "matches = matcher(doc)\n",
    "# print(matches)\n",
    "for match in matches[:10]:\n",
    "    print(match, doc[match[1]: match[2]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* pattern = [{\"POS\":\"PROPN\", \"OP\":\"+\"}]: This line defines a pattern that the Matcher will use to find matches in the text. The pattern is looking for one or more tokens with the part-of-speech tag PROPN, which stands for “proper noun”. The OP key specifies the operator, with \"+\" meaning “one or more times”.\n",
    "\n",
    "* matcher.add(\"PROPER_NOUN\",[pattern]): This line adds the defined pattern to the Matcher under the rule name “PROPER_NOUN”. When the Matcher finds a sequence of tokens that matches the pattern, it will label the sequence with this rule name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4.3 Sorting it to Apperance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(3232560085755078826, 0, 4), (3232560085755078826, 6, 9), (3232560085755078826, 10, 11), (3232560085755078826, 15, 16), (3232560085755078826, 23, 24), (3232560085755078826, 49, 50), (3232560085755078826, 69, 71), (3232560085755078826, 83, 88), (3232560085755078826, 89, 90), (3232560085755078826, 113, 114), (3232560085755078826, 117, 118), (3232560085755078826, 128, 132), (3232560085755078826, 133, 134), (3232560085755078826, 140, 141), (3232560085755078826, 146, 148), (3232560085755078826, 149, 150), (3232560085755078826, 151, 152), (3232560085755078826, 163, 164), (3232560085755078826, 165, 166), (3232560085755078826, 167, 168), (3232560085755078826, 172, 173), (3232560085755078826, 174, 175), (3232560085755078826, 185, 186), (3232560085755078826, 193, 195), (3232560085755078826, 240, 242), (3232560085755078826, 243, 244), (3232560085755078826, 245, 246), (3232560085755078826, 247, 251), (3232560085755078826, 252, 253), (3232560085755078826, 262, 263), (3232560085755078826, 270, 271), (3232560085755078826, 297, 298), (3232560085755078826, 317, 318), (3232560085755078826, 322, 323), (3232560085755078826, 325, 328), (3232560085755078826, 346, 347), (3232560085755078826, 348, 349), (3232560085755078826, 370, 372), (3232560085755078826, 377, 378), (3232560085755078826, 384, 385), (3232560085755078826, 386, 387), (3232560085755078826, 392, 394), (3232560085755078826, 402, 403), (3232560085755078826, 405, 406), (3232560085755078826, 407, 408), (3232560085755078826, 417, 418), (3232560085755078826, 422, 425), (3232560085755078826, 431, 432), (3232560085755078826, 450, 451), (3232560085755078826, 455, 457), (3232560085755078826, 458, 459), (3232560085755078826, 463, 466), (3232560085755078826, 469, 474), (3232560085755078826, 485, 487), (3232560085755078826, 503, 506), (3232560085755078826, 514, 515), (3232560085755078826, 528, 530), (3232560085755078826, 536, 541), (3232560085755078826, 543, 545), (3232560085755078826, 546, 547), (3232560085755078826, 548, 549)]\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"POS\":\"PROPN\", \"OP\":\"+\"}]\n",
    "matcher.add(\"PROPER_NOUNS\", [pattern], greedy=\"LONGEST\")\n",
    "doc = nlp(text)\n",
    "matches = matcher(doc)\n",
    "matches.sort(key = lambda x: x[1])\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* pattern = [{\"POS\":\"PROPN\", \"OP\":\"+\"}]: This line defines a pattern that the Matcher will use to find matches in the text. The pattern is looking for one or more tokens with the part-of-speech tag PROPN, which stands for “proper noun”. The OP key specifies the operator, with \"+\" meaning “one or more times”.\n",
    "\n",
    "* matcher.add(\"PROPER_NOUNS\", [pattern], greedy=\"LONGEST\"): This line adds the defined pattern to the Matcher under the rule name “PROPER_NOUNS”. The greedy=\"LONGEST\" parameter tells the Matcher to prefer longer matches over shorter ones when patterns overlap.\n",
    "\n",
    "* matches.sort(key = lambda x: x[1]): This line sorts the matches by their start index in ascending order. This is useful if you want to process or display the matches in the order they appear in the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4.4 Adding in Sequunces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n",
      "(3232560085755078826, 49, 51) King advanced\n",
      "(3232560085755078826, 89, 91) King participated\n",
      "(3232560085755078826, 113, 115) King led\n",
      "(3232560085755078826, 167, 169) King helped\n",
      "(3232560085755078826, 247, 252) Director J. Edgar Hoover considered\n",
      "(3232560085755078826, 322, 324) King won\n",
      "(3232560085755078826, 485, 488) United States beginning\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"POS\": \"PROPN\", \"OP\": \"+\"}, {\"POS\": \"VERB\"}]\n",
    "matcher.add(\"PROPER_NOUNS\", [pattern], greedy='LONGEST')\n",
    "doc = nlp(text)\n",
    "matches = matcher(doc)\n",
    "matches.sort(key = lambda x: x[1])\n",
    "print (len(matches))\n",
    "for match in matches[:10]:\n",
    "    print (match, doc[match[1]:match[2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.5. Finding Quotes and Speakers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, `and what is the use of a book,' thought Alice `without pictures or conversation?'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<>:2: SyntaxWarning: invalid escape sequence '\\s'\n",
      "<>:2: SyntaxWarning: invalid escape sequence '\\s'\n",
      "C:\\Users\\Manjusha Kumari\\AppData\\Local\\Temp\\ipykernel_8252\\1590009246.py:2: SyntaxWarning: invalid escape sequence '\\s'\n",
      "  with open(\"F:\\spaCy-master\\data\\\\alice.json\", \"r\") as f:\n"
     ]
    }
   ],
   "source": [
    "import json \n",
    "with open(\"F:\\spaCy-master\\data\\\\alice.json\", \"r\") as f:\n",
    "    data = json.load(f)\n",
    "    text = data[0][2][0]\n",
    "    print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, 'and what is the use of a book,' thought Alice 'without pictures or conversation?'\n"
     ]
    }
   ],
   "source": [
    "text = text.replace(\"`\", \"'\")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3232560085755078826, 47, 58) 'and what is the use of a book,'\n",
      "(3232560085755078826, 60, 67) 'without pictures or conversation?'\n"
     ]
    }
   ],
   "source": [
    "matcher = Matcher(nlp.vocab)\n",
    "pattern = [{\"ORTH\":\"'\"}, {'IS_ALPHA':True, \"OP\":\"+\"}, {\"IS_PUNCT\":True, \"OP\":\"*\"}, {\"ORTH\":\"'\"}]\n",
    "matcher.add(\"PROPER_NOUNS\", [pattern], greedy='LONGEST')\n",
    "doc = nlp(text)\n",
    "matches = matcher(doc)\n",
    "matches.sort(key = lambda x: x[1])\n",
    "\n",
    "for match in matches[:10]:\n",
    "    print(match, doc[match[1]:match[2]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.5.1. Find Speaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(3232560085755078826, 47, 67) 'and what is the use of a book,' thought Alice 'without pictures or conversation?'\n"
     ]
    }
   ],
   "source": [
    "speak_lemmas = [\"think\", \"say\"]\n",
    "text = data[0][2][0].replace( \"`\", \"'\")\n",
    "matcher = Matcher(nlp.vocab)\n",
    "pattern1 = [{'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}, {\"POS\": \"VERB\", \"LEMMA\": {\"IN\": speak_lemmas}}, {\"POS\": \"PROPN\", \"OP\": \"+\"}, {'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}]\n",
    "matcher.add(\"PROPER_NOUNS\", [pattern1], greedy='LONGEST')\n",
    "doc = nlp(text)\n",
    "matches = matcher(doc)\n",
    "matches.sort(key = lambda x: x[1])\n",
    "print (len(matches))\n",
    "for match in matches[:10]:\n",
    "    print (match, doc[match[1]:match[2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem with this Approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(3232560085755078826, 47, 67) 'and what is the use of a book,' thought Alice 'without pictures or conversation?'\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "1\n",
      "(3232560085755078826, 0, 6) 'Well!' thought Alice\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "1\n",
      "(3232560085755078826, 57, 68) 'which certainly was not here before,' said Alice\n",
      "0\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "for text in data[0][2]:\n",
    "    text = text.replace(\"`\", \"'\")\n",
    "    doc = nlp(text)\n",
    "    matches = matcher(doc)\n",
    "    matches.sort(key = lambda x: x[1])\n",
    "    print (len(matches))\n",
    "    for match in matches[:10]:\n",
    "        print (match, doc[match[1]:match[2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.5.3. Adding More Patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(3232560085755078826, 47, 67) 'and what is the use of a book,' thought Alice 'without pictures or conversation?'\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "1\n",
      "(3232560085755078826, 0, 6) 'Well!' thought Alice\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "0\n",
      "1\n",
      "(3232560085755078826, 57, 68) 'which certainly was not here before,' said Alice\n",
      "0\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "speak_lemmas = [\"think\", \"say\"]\n",
    "text = data[0][2][0].replace( \"`\", \"'\")\n",
    "matcher = Matcher(nlp.vocab)\n",
    "pattern1 = [{'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}, {\"POS\": \"VERB\", \"LEMMA\": {\"IN\": speak_lemmas}}, {\"POS\": \"PROPN\", \"OP\": \"+\"}, {'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}]\n",
    "pattern2 = [{'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}, {\"POS\": \"VERB\", \"LEMMA\": {\"IN\": speak_lemmas}}, {\"POS\": \"PROPN\", \"OP\": \"+\"}]\n",
    "pattern3 = [{\"POS\": \"PROPN\", \"OP\": \"+\"},{\"POS\": \"VERB\", \"LEMMA\": {\"IN\": speak_lemmas}}, {'ORTH': \"'\"}, {'IS_ALPHA': True, \"OP\": \"+\"}, {'IS_PUNCT': True, \"OP\": \"*\"}, {'ORTH': \"'\"}]\n",
    "matcher.add(\"PROPER_NOUNS\", [pattern1, pattern2, pattern3], greedy='LONGEST')\n",
    "for text in data[0][2]:\n",
    "    text = text.replace(\"`\", \"'\")\n",
    "    doc = nlp(text)\n",
    "    matches = matcher(doc)\n",
    "    matches.sort(key = lambda x: x[1])\n",
    "    print (len(matches))\n",
    "    for match in matches[:10]:\n",
    "        print (match, doc[match[1]:match[2]])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "manish",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}