MiniProject1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3b8ddd63",
   "metadata": {},
   "source": [
    "# Developing an Information Retrieval System with Advanced Boolean Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0732c5e5",
   "metadata": {},
   "source": [
    "<div style=\"line-height: 2;\">This project aims to develop an Information Retrieval (IR) system that supports both standard Boolean queries and proximity queries. The system is designed to handle a collection of text documents, building an Inverted Index and a Positional Index to facilitate efficient document retrieval.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46b86a61",
   "metadata": {},
   "source": [
    "<div style=\"line-height: 2;\">For developing this system, the first step is reading document files which are prepared. But before that, we need to import some neccessary libraries:\n",
    "    <ul>\n",
    "        <li>\n",
    "            <b>NLTK</b>: Natural Language ToolKit: A leading platform for building Python programs to work with human language data\n",
    "        </li>\n",
    "        <li>\n",
    "            <b>OS</b>: A built-in module that provides a wide range of functions for interacting with the operating system. It allows you to work with file systems, directories, paths, environment variables, and more.\n",
    "        </li>\n",
    "        <li>\n",
    "            <b>String</b>: A package which provides a wide range of string manipulation functions and methods\n",
    "        </li>\n",
    "    </ul>\n",
    "\n",
    "Also we import some specific parts of NLTK which are described later:\n",
    "<ul>\n",
    "    <li>\n",
    "        <b>Stopwords</b>\n",
    "    </li>\n",
    "    <li>\n",
    "        <b>Regexp_tokenize</b>\n",
    "    </li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (3.8.1)\n",
      "Requirement already satisfied: click in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from nltk) (8.1.7)\n",
      "Requirement already satisfied: joblib in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from nltk) (1.3.2)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from nltk) (2023.10.3)\n",
      "Requirement already satisfied: tqdm in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (from nltk) (4.66.1)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip3 install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "6ff567bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import os\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c151972",
   "metadata": {},
   "source": [
    "In the current directory, we have a file named **data** which contains our text files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "98b9e293",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define path of data files here\n",
    "folder_path = \"data\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41a8279",
   "metadata": {},
   "source": [
    "With `os.listdir()` we make a list of all files and directories in the `folder_path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8a4d4d2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make a list of all files in \"data\" directory which are our text files\n",
    "file_list = os.listdir(folder_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267e1135",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "As there might exist some files which are not text or some directories, we use the below code to extract `.txt` files.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "96e6ae80",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "document titles:\n",
      "----------------------------------------\n",
      "Jerry Decided To Buy a Gun.txt\n",
      "Rentals at the Oceanside Community.txt\n",
      "Gasoline Prices Hit Record High.txt\n",
      "Cloning Pets.txt\n",
      "Crazy Housing Prices.txt\n",
      "Man Injured at Fast Food Place.txt\n",
      "A Festival of Books.txt\n",
      "Food Fight Erupted in Prison.txt\n",
      "Better To Be Unlucky.txt\n",
      "Sara Went Shopping.txt\n",
      "Freeway Chase Ends at Newsstand.txt\n",
      "Trees Are a Threat.txt\n",
      "A Murder-Suicide.txt\n",
      "Happy and Unhappy Renters.txt\n",
      "Pulling Out Nine Tons of Trash.txt\n",
      "----------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# text_files stores text files' names\n",
    "text_files = [file for file in file_list if file.endswith('.txt')]\n",
    "\n",
    "# print text files titles\n",
    "print(\"document titles:\\n\" + \"-\"*40)\n",
    "for text_file in text_files:\n",
    "    print(text_file)\n",
    "print( \"-\"*40)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2502e710",
   "metadata": {},
   "source": [
    "\n",
    "<div style=\"line-height: 2;\">\n",
    "As we mentioned above, some specific functions of NLTK package are needed.\n",
    "\n",
    "- **regexp_tokenize**: we use this function for tokenization, specifically for tokenizing text based on a regular expression pattern.\n",
    "- **stopwords**: This module specifically contains a list of common stop words for multiple languages, which in this project we set English.Some examples of English stopwords are: \"the\", \"and\", \"or\", \"is\", \"was\", ... .\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "777589df",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import regexp_tokenize\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50293929",
   "metadata": {},
   "source": [
    "Also, we need to download stopwords for preprocessing instructons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "665f8c2f",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Error loading stopwords: <urlopen error [SSL:\n",
      "[nltk_data]     CERTIFICATE_VERIFY_FAILED] certificate verify failed:\n",
      "[nltk_data]     unable to get local issuer certificate (_ssl.c:992)>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# nltk.download('punkt')\n",
    "# download stopwords here\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff84957c",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a9cb60a",
   "metadata": {},
   "source": [
    "<div style=\"line-height: 2;\">\n",
    "In this step, we want to preprocess text files functionally. Below steps have to be implemented:\n",
    "    <ol>\n",
    "        <li>\n",
    "            <b>tokenization</b>: which we will do it using `regexp_tokenize()` function from `NLTK` library. For this job, we need to specify a pattern for natural language. As in our documents we have words, numbers with less or equal to 3 digits, numbers with more than 3 digits which are seperated 3 by 3, and floating point numbers, we have to use a pattern to handle all these types concurrently. We used `r'\\d{1,3}(?:,\\d{3})*(?:\\.\\d+)?|\\w+'` which is decoded below: <br>\n",
    "            <ul>\n",
    "                <li>\n",
    "                    \\d{1,3}: This part matches one to three-digit numbers like \"4\", \"32\", \"879\", not \"1234\" or \"1,000\".\n",
    "                </li>\n",
    "                <li>\n",
    "                (?:,\\d{3})*: This part matches comma-separated thousands separators for numbers like \"75,000\".\n",
    "                </li>\n",
    "                <li>\n",
    "                (?:\\.\\d+)?: This part handles floating point numbers.\n",
    "                </li>\n",
    "                <li>\n",
    "                |: This symbol acts as an OR operator, allowing the regular expression to match either a number (as described above) or its next part.\n",
    "                </li>\n",
    "                <li>\n",
    "                \\w+: This part matches words, which consist of one or more word characters (letters, digits, and underscores). It's not limited to numbers and can match words like \"apple,\" \"word123,\" and \"my_variable.\"\n",
    "                </li>\n",
    "            </ul>\n",
    "        </li>\n",
    "      <li>\n",
    "          <b>Lowercase tokens</b>: as step of preprocessing, we have to convert all words' characters to lowercase. So, after query preprocessing, our job is easier.      \n",
    "      </li>\n",
    "        <li>\n",
    "         <b>Delete Stopwrds</b>: We have to delete stopwords(These are the words which are use frequently in documents and not neccessary in Information Retrival. There is a list of stopwords which is prepared for English and we downloaded above.). So, the process of retriving will be faster because there are less words for checking.\n",
    "        </li>\n",
    "      <li>\n",
    "          <b>Delete punctuations</b>: Trivially, punctuations are not needed when we want to retrive information. So, it is needed to omit all punctuations from all documents.\n",
    "      <\\li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "42a8f6d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing(text):\n",
    "    '''\n",
    "    Define a regular expression pattern for tokenization \n",
    "    (matching words, 3-digit numbers, numbers with more than 3 digits and \n",
    "    have thousands seperators in which n > 3, and floating-point numbers)\n",
    "    '''\n",
    "    pattern = r'\\d{1,3}(?:,\\d{3})*(?:\\.\\d+)?|\\w+'\n",
    "    \n",
    "    # Tokenize the text using the regular expression pattern    \n",
    "    tokens = regexp_tokenize(text, pattern)\n",
    "    \n",
    "    # Lowercase the tokens\n",
    "    tokens = [word.lower() for word in tokens]\n",
    "    \n",
    "    # Get the set of English stopwords\n",
    "    stop_words = set(stopwords.words('english'))\n",
    "    \n",
    "    # Remove stopwords\n",
    "    tokens = [word for word in tokens if word not in stop_words]\n",
    "    \n",
    "    # Remove punctuation\n",
    "    tokens = [word for word in tokens if word not in string.punctuation]\n",
    "    \n",
    "    # Join tokens to return a string for each document\n",
    "    return ' '.join(tokens)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b692ab57",
   "metadata": {},
   "source": [
    "There is another function we have to import from `collections` module named `defaultdict` which creates dictionaries with default values for new intered keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "dad40236",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9892a83c",
   "metadata": {},
   "source": [
    "## Inverted index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c55fd50d",
   "metadata": {},
   "source": [
    "<div style=\"line-height: 2;\">Now, after preprocessing, it is time to create our inverted index.<br>\n",
    "    An inverted index is a kind of data structure which is particularly a dictionary that stores distinct words of all documnet files as dictionary keys and the values of dictionay keys are the docIDs of documents which have the key word in theirselves. Also, for each docID, we store a list which contains the index of that specific word in the document which is the word appeared in.<br>\n",
    "    <span style=\"color:red;\">Note: </span>I used try-execpt because as I wanted to work with files and read them all using UTF-8 endoding, I confronted the error \"UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 394: invalid start byte\". That means there exists a file that contains non-UTF-8 encoded characters. So, I used latin-1 encoding for these type of files.<br>\n",
    "    <span style=\"color:blue;\">Warnin: </span>code in line 53 through 55 should be only run once because in the fisrt time, the processed files are created and the content is added. If we run it more than one time, the content of each, will be duplicate which is a trash!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "43142492",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jerry\n",
      "{1: [0, 19, 37, 44, 73, 85, 130, 133]}\n",
      "baldwin\n",
      "{1: [1]}\n",
      "30\n",
      "{1: [2], 3: [17], 10: [6]}\n",
      "years\n",
      "{1: [3, 96], 2: [48], 3: [46], 5: [46, 163], 7: [136], 9: [16], 10: [20, 25], 12: [12, 26], 13: [24, 72, 98], 15: [32]}\n",
      "old\n",
      "{1: [4, 50], 6: [2], 9: [17, 127], 10: [26], 11: [2], 12: [140], 13: [12, 18], 15: [135]}\n",
      "manager\n",
      "{1: [5], 2: [32], 3: [35], 6: [53]}\n",
      "pizza\n",
      "{1: [6]}\n",
      "restaurant\n",
      "{1: [7, 13], 6: [15, 52, 66]}\n",
      "lived\n",
      "{1: [8], 10: [7], 13: [68]}\n",
      "apartment\n",
      "{1: [9]}\n",
      "one\n",
      "{1: [10, 24], 2: [126], 5: [20, 78], 6: [61, 65], 7: [2, 54, 113, 154, 158], 9: [77], 11: [12, 34, 73, 112], 12: [56, 124], 13: [48, 54], 14: [6, 95], 15: [29, 127]}\n",
      "mile\n",
      "{1: [11], 10: [45], 15: [110]}\n",
      "north\n",
      "{1: [12], 11: [90, 130]}\n",
      "walked\n",
      "{1: [14]}\n",
      "work\n",
      "{1: [15], 2: [95, 134], 3: [131], 12: [47], 15: [0, 77, 106]}\n",
      "raining\n",
      "{1: [16]}\n",
      "took\n",
      "{1: [17]}\n",
      "bus\n",
      "{1: [18], 11: [93, 98, 100, 115]}\n",
      "loved\n",
      "{1: [20, 60]}\n",
      "gangster\n",
      "{1: [21, 48]}\n",
      "movies\n",
      "{1: [22], 7: [12]}\n",
      "new\n",
      "{1: [23, 30, 52, 118], 4: [42, 44, 72], 5: [76], 6: [9, 71], 14: [23]}\n",
      "came\n",
      "{1: [25], 7: [111], 9: [109], 13: [101, 159], 14: [37], 15: [41]}\n",
      "would\n",
      "{1: [26, 38, 74, 78, 88], 2: [57, 63, 72, 78, 82, 154, 168, 176], 6: [45, 80], 7: [140, 149], 8: [109], 9: [81, 96], 13: [118, 129, 133], 14: [51, 141, 147], 15: [146, 148]}\n",
      "go\n",
      "{1: [27], 3: [100], 5: [5, 7, 142, 158], 6: [74, 86], 13: [130], 14: [119]}\n",
      "theater\n",
      "{1: [28]}\n",
      "watch\n",
      "{1: [29, 81], 15: [91, 152]}\n",
      "movie\n",
      "{1: [31, 64, 77], 7: [18]}\n",
      "three\n",
      "{1: [32], 2: [53, 60], 9: [100], 10: [28], 11: [70], 15: [31]}\n",
      "four\n",
      "{1: [33], 2: [55], 10: [33, 52], 12: [146], 14: [64]}\n",
      "times\n",
      "{1: [34], 3: [64], 9: [93, 120, 124, 133]}\n",
      "went\n",
      "{1: [35, 97, 114, 127, 129], 8: [65], 10: [4], 11: [127]}\n",
      "video\n",
      "{1: [36, 40, 42]}\n",
      "buy\n",
      "{1: [39], 5: [103, 105, 112, 116], 10: [68], 15: [156]}\n",
      "barney\n",
      "{1: [41]}\n",
      "store\n",
      "{1: [43, 99], 3: [134], 10: [44]}\n",
      "home\n",
      "{1: [45], 10: [64, 82], 14: [38, 40]}\n",
      "collection\n",
      "{1: [46]}\n",
      "1,000\n",
      "{1: [47], 12: [121]}\n",
      "videos\n",
      "{1: [49], 7: [10]}\n",
      "ones\n",
      "{1: [51, 53], 14: [28]}\n",
      "color\n",
      "{1: [54], 14: [151]}\n",
      "black\n",
      "{1: [55]}\n",
      "white\n",
      "{1: [56]}\n",
      "english\n",
      "{1: [57]}\n",
      "spanish\n",
      "{1: [58]}\n",
      "japanese\n",
      "{1: [59]}\n",
      "could\n",
      "{1: [61], 13: [145]}\n",
      "tell\n",
      "{1: [62], 3: [72], 13: [146]}\n",
      "name\n",
      "{1: [63]}\n",
      "director\n",
      "{1: [65], 5: [146], 15: [24]}\n",
      "stars\n",
      "{1: [66]}\n",
      "plot\n",
      "{1: [67]}\n",
      "say\n",
      "{1: [68], 5: [139], 12: [127], 13: [144]}\n",
      "liked\n",
      "{1: [69]}\n",
      "pulp\n",
      "{1: [70]}\n",
      "fiction\n",
      "{1: [71]}\n",
      "well\n",
      "{1: [72], 4: [66], 12: [134], 15: [19]}\n",
      "rattle\n",
      "{1: [75]}\n",
      "details\n",
      "{1: [76]}\n",
      "invite\n",
      "{1: [79]}\n",
      "place\n",
      "{1: [80], 5: [36], 13: [115]}\n",
      "time\n",
      "{1: [82], 4: [54], 13: [123], 15: [30, 39]}\n",
      "nice\n",
      "{1: [83], 4: [12], 5: [51], 13: [63, 179]}\n",
      "guy\n",
      "{1: [84], 14: [73]}\n",
      "finally\n",
      "{1: [86], 9: [105]}\n",
      "decided\n",
      "{1: [87]}\n",
      "like\n",
      "{1: [89, 91, 137], 5: [68, 108, 114], 8: [59], 13: [79], 14: [1, 11]}\n",
      "gun\n",
      "{1: [90, 98, 115, 126]}\n",
      "gangsters\n",
      "{1: [92, 138]}\n",
      "saved\n",
      "{1: [93]}\n",
      "money\n",
      "{1: [94], 2: [188], 3: [89], 9: [66], 12: [152]}\n",
      "couple\n",
      "{1: [95, 108], 7: [114], 13: [21, 38, 64]}\n",
      "bought\n",
      "{1: [100, 107], 3: [42, 108], 5: [34], 10: [51]}\n",
      "used\n",
      "{1: [101], 3: [5, 135], 13: [76, 105, 113]}\n",
      "38\n",
      "{1: [102]}\n",
      "caliber\n",
      "{1: [103]}\n",
      "revolver\n",
      "{1: [104, 119]}\n",
      "300\n",
      "{1: [105]}\n",
      "also\n",
      "{1: [106], 2: [40], 9: [172], 11: [80], 14: [81]}\n",
      "boxes\n",
      "{1: [109]}\n",
      "ammunition\n",
      "{1: [110]}\n",
      "following\n",
      "{1: [111], 7: [75]}\n",
      "saturday\n",
      "{1: [112], 6: [6], 7: [60], 14: [49], 15: [10]}\n",
      "morning\n",
      "{1: [113]}\n",
      "club\n",
      "{1: [116, 120]}\n",
      "practice\n",
      "{1: [117]}\n",
      "10\n",
      "{1: [121], 2: [47], 3: [58], 5: [71], 7: [24], 9: [40], 10: [83]}\n",
      "minutes\n",
      "{1: [122], 3: [98]}\n",
      "accidentally\n",
      "{1: [123], 6: [28]}\n",
      "dropped\n",
      "{1: [124]}\n",
      "pistol\n",
      "{1: [125]}\n",
      "bullet\n",
      "{1: [128]}\n",
      "right\n",
      "{1: [131], 3: [28], 9: [115]}\n",
      "knee\n",
      "{1: [132]}\n",
      "walks\n",
      "{1: [134]}\n",
      "limp\n",
      "{1: [135]}\n",
      "cane\n",
      "{1: [136]}\n",
      "oceanside\n",
      "{2: [0]}\n",
      "community\n",
      "{2: [1, 119, 183], 15: [80]}\n",
      "lozano\n",
      "{2: [2]}\n",
      "beach\n",
      "{2: [3], 5: [31, 82]}\n",
      "debating\n",
      "{2: [4]}\n",
      "whether\n",
      "{2: [5], 5: [101]}\n",
      "allow\n",
      "{2: [6]}\n",
      "homeowners\n",
      "{2: [7, 34, 175]}\n",
      "rent\n",
      "{2: [8, 148, 163], 14: [9, 17]}\n",
      "homes\n",
      "{2: [9], 12: [84]}\n",
      "weekly\n",
      "{2: [10, 45]}\n",
      "basis\n",
      "{2: [11]}\n",
      "summer\n",
      "{2: [12, 33]}\n",
      "rentals\n",
      "{2: [13, 39, 46]}\n",
      "produce\n",
      "{2: [14, 67], 4: [94]}\n",
      "high\n",
      "{2: [15], 5: [6], 12: [120]}\n",
      "incomes\n",
      "{2: [16]}\n",
      "owners\n",
      "{2: [17]}\n",
      "city\n",
      "{2: [18, 31, 42, 96, 110, 116, 135, 153, 167, 173], 7: [22]}\n",
      "gets\n",
      "{2: [19]}\n",
      "part\n",
      "{2: [20]}\n",
      "income\n",
      "{2: [21]}\n",
      "15\n",
      "{2: [22, 157]}\n",
      "percent\n",
      "{2: [23, 158], 3: [18], 5: [17, 72], 7: [25], 8: [115], 12: [105], 14: [19]}\n",
      "surcharge\n",
      "{2: [24]}\n",
      "owner\n",
      "{2: [25], 5: [77, 85], 9: [166]}\n",
      "boon\n",
      "{2: [26]}\n",
      "coffers\n",
      "{2: [27]}\n",
      "said\n",
      "{2: [28], 3: [37, 71, 94, 104], 4: [7, 86], 5: [26, 83, 144], 6: [34, 79, 87], 7: [121, 157], 8: [49, 108], 9: [4], 12: [55], 13: [65, 151, 165, 181], 14: [30, 59, 87], 15: [145]}\n",
      "rick\n",
      "{2: [29]}\n",
      "brown\n",
      "{2: [30]}\n",
      "bring\n",
      "{2: [35], 5: [138], 14: [98]}\n",
      "2,000\n",
      "{2: [36]}\n",
      "week\n",
      "{2: [37, 122], 8: [6, 14, 98], 13: [117, 132]}\n",
      "however\n",
      "{2: [38], 11: [103]}\n",
      "worms\n",
      "{2: [41]}\n",
      "stopped\n",
      "{2: [43], 10: [66]}\n",
      "allowing\n",
      "{2: [44]}\n",
      "ago\n",
      "{2: [49], 3: [23, 47], 7: [137], 8: [15], 13: [31, 99]}\n",
      "problems\n",
      "{2: [50, 109, 112], 12: [54]}\n",
      "generating\n",
      "{2: [51]}\n",
      "two\n",
      "{2: [52, 59], 3: [45], 5: [37], 6: [57], 8: [2, 16, 76], 9: [55, 75], 10: [21], 11: [76, 104, 128], 12: [53], 13: [167], 14: [46, 75], 15: [79, 86, 109]}\n",
      "even\n",
      "{2: [54, 161], 3: [116], 4: [14], 5: [133], 7: [101], 13: [152, 160], 15: [68]}\n",
      "families\n",
      "{2: [56]}\n",
      "pile\n",
      "{2: [58]}\n",
      "bedroom\n",
      "{2: [61], 5: [38, 79]}\n",
      "house\n",
      "{2: [62], 5: [29, 132, 165], 13: [131]}\n",
      "park\n",
      "{2: [64]}\n",
      "cars\n",
      "{2: [65], 3: [65], 9: [80], 11: [72, 77, 106, 122]}\n",
      "lawn\n",
      "{2: [66]}\n",
      "huge\n",
      "{2: [68]}\n",
      "amounts\n",
      "{2: [69]}\n",
      "trash\n",
      "{2: [70, 74, 107], 15: [119, 124]}\n",
      "sometimes\n",
      "{2: [71]}\n",
      "toss\n",
      "{2: [73]}\n",
      "streets\n",
      "{2: [75]}\n",
      "sidewalks\n",
      "{2: [76]}\n",
      "noise\n",
      "{2: [77, 105]}\n",
      "another\n",
      "{2: [79], 14: [20, 102, 108]}\n",
      "problem\n",
      "{2: [80], 6: [42]}\n",
      "people\n",
      "{2: [81, 149], 3: [73], 7: [0, 39, 102, 110], 12: [164], 13: [180], 14: [140]}\n",
      "party\n",
      "{2: [83]}\n",
      "late\n",
      "{2: [84]}\n",
      "loud\n",
      "{2: [85]}\n",
      "every\n",
      "{2: [86], 3: [105], 7: [52], 13: [90], 14: [13, 121], 15: [162]}\n",
      "night\n",
      "{2: [87], 13: [7, 170]}\n",
      "abuse\n",
      "{2: [88]}\n",
      "created\n",
      "{2: [89]}\n",
      "lot\n",
      "{2: [90], 3: [43], 13: [147]}\n",
      "friction\n",
      "{2: [91]}\n",
      "neighbors\n",
      "{2: [92, 152], 14: [22, 24, 92]}\n",
      "resulted\n",
      "{2: [93]}\n",
      "extra\n",
      "{2: [94], 6: [62], 12: [58]}\n",
      "maintenance\n",
      "{2: [97]}\n",
      "crews\n",
      "{2: [98]}\n",
      "police\n",
      "{2: [99], 3: [62], 11: [11, 35, 83, 105, 152, 159], 13: [176], 14: [58], 15: [96]}\n",
      "respond\n",
      "{2: [100, 136]}\n",
      "almost\n",
      "{2: [101], 3: [16], 13: [60, 156], 15: [14]}\n",
      "hourly\n",
      "{2: [102]}\n",
      "residents\n",
      "{2: [103], 3: [0], 7: [152], 12: [44, 94, 117, 128]}\n",
      "complaints\n",
      "{2: [104, 137]}\n",
      "music\n",
      "{2: [106], 14: [107]}\n",
      "parking\n",
      "{2: [108], 7: [43, 97]}\n",
      "budget\n",
      "{2: [111]}\n",
      "making\n",
      "{2: [113], 3: [88]}\n",
      "reconsider\n",
      "{2: [114]}\n",
      "ban\n",
      "{2: [115, 180]}\n",
      "officials\n",
      "{2: [117, 178], 5: [135], 8: [48, 86], 12: [126]}\n",
      "hold\n",
      "{2: [118]}\n",
      "meeting\n",
      "{2: [120]}\n",
      "next\n",
      "{2: [121], 12: [24], 13: [33, 69], 15: [38]}\n",
      "listen\n",
      "{2: [123], 14: [54]}\n",
      "pros\n",
      "{2: [124]}\n",
      "cons\n",
      "{2: [125]}\n",
      "official\n",
      "{2: [127]}\n",
      "already\n",
      "{2: [128, 184], 4: [19], 7: [145], 13: [53]}\n",
      "suggested\n",
      "{2: [129]}\n",
      "proposal\n",
      "{2: [130]}\n",
      "thinks\n",
      "{2: [131]}\n",
      "fine\n",
      "{2: [132, 143]}\n",
      "might\n",
      "{2: [133, 144], 6: [92], 15: [140]}\n",
      "homeowner\n",
      "{2: [138, 146, 162]}\n",
      "charged\n",
      "{2: [139], 11: [155]}\n",
      "200\n",
      "{2: [140]}\n",
      "per\n",
      "{2: [141], 3: [121]}\n",
      "response\n",
      "{2: [142]}\n",
      "cause\n",
      "{2: [145, 191]}\n",
      "careful\n",
      "{2: [147]}\n",
      "sure\n",
      "{2: [150], 12: [68]}\n",
      "considerate\n",
      "{2: [151]}\n",
      "still\n",
      "{2: [155], 5: [106], 9: [73, 174], 10: [86]}\n",
      "get\n",
      "{2: [156], 3: [4, 118], 5: [99], 7: [130], 12: [96], 13: [110], 14: [53]}\n",
      "rental\n",
      "{2: [159]}\n",
      "fee\n",
      "{2: [160], 7: [98]}\n",
      "totally\n",
      "{2: [164]}\n",
      "offset\n",
      "{2: [165]}\n",
      "fines\n",
      "{2: [166]}\n",
      "post\n",
      "{2: [169]}\n",
      "inconsiderate\n",
      "{2: [170], 14: [27]}\n",
      "renters\n",
      "{2: [171], 14: [3, 80, 83]}\n",
      "names\n",
      "{2: [172]}\n",
      "website\n",
      "{2: [174]}\n",
      "know\n",
      "{2: [177], 5: [151], 12: [95, 153]}\n",
      "think\n",
      "{2: [179], 12: [162]}\n",
      "continued\n",
      "{2: [181]}\n",
      "visitors\n",
      "{2: [182]}\n",
      "proven\n",
      "{2: [185]}\n",
      "consideration\n",
      "{2: [186]}\n",
      "others\n",
      "{2: [187]}\n",
      "worth\n",
      "{2: [189], 5: [167], 9: [170], 15: [141]}\n",
      "headaches\n",
      "{2: [190]}\n",
      "southern\n",
      "{3: [1]}\n",
      "california\n",
      "{3: [2], 5: [24], 7: [112]}\n",
      "trying\n",
      "{3: [3]}\n",
      "skyrocketing\n",
      "{3: [6]}\n",
      "prices\n",
      "{3: [7, 49, 128], 5: [4, 153], 7: [106], 12: [118]}\n",
      "gasoline\n",
      "{3: [8]}\n",
      "average\n",
      "{3: [9]}\n",
      "price\n",
      "{3: [10, 26], 4: [25], 5: [74], 10: [59]}\n",
      "87\n",
      "{3: [11]}\n",
      "octane\n",
      "{3: [12]}\n",
      "economy\n",
      "{3: [13]}\n",
      "gas\n",
      "{3: [14, 25, 39, 44, 93, 111]}\n",
      "2.22\n",
      "{3: [15]}\n",
      "higher\n",
      "{3: [19], 5: [157]}\n",
      "today\n",
      "{3: [20, 127]}\n",
      "12\n",
      "{3: [21]}\n",
      "months\n",
      "{3: [22]}\n",
      "lowest\n",
      "{3: [24]}\n",
      "southland\n",
      "{3: [27]}\n",
      "2.09\n",
      "{3: [29]}\n",
      "gallon\n",
      "{3: [30, 113, 122], 10: [69]}\n",
      "seashell\n",
      "{3: [31, 54]}\n",
      "station\n",
      "{3: [32, 34, 55, 76]}\n",
      "arcadia\n",
      "{3: [33]}\n",
      "everett\n",
      "{3: [36, 70]}\n",
      "reason\n",
      "{3: [38], 14: [7, 21]}\n",
      "cheaper\n",
      "{3: [40]}\n",
      "elsewhere\n",
      "{3: [41]}\n",
      "reduced\n",
      "{3: [48]}\n",
      "passing\n",
      "{3: [50], 5: [119]}\n",
      "savings\n",
      "{3: [51]}\n",
      "customers\n",
      "{3: [52], 11: [59]}\n",
      "lines\n",
      "{3: [53, 103]}\n",
      "often\n",
      "{3: [56]}\n",
      "run\n",
      "{3: [57], 5: [161], 9: [82, 118], 11: [82]}\n",
      "20\n",
      "{3: [59, 97], 13: [71]}\n",
      "vehicles\n",
      "{3: [60]}\n",
      "long\n",
      "{3: [61], 5: [11, 160]}\n",
      "several\n",
      "{3: [63], 11: [53]}\n",
      "block\n",
      "{3: [66, 77, 101]}\n",
      "traffic\n",
      "{3: [67], 7: [26], 12: [62]}\n",
      "horsetrail\n",
      "{3: [68]}\n",
      "drive\n",
      "{3: [69, 130, 132, 136], 6: [11]}\n",
      "line\n",
      "{3: [74, 96]}\n",
      "barco\n",
      "{3: [75]}\n",
      "away\n",
      "{3: [78, 102], 10: [46]}\n",
      "2.14\n",
      "{3: [79]}\n",
      "rather\n",
      "{3: [80]}\n",
      "wait\n",
      "{3: [81], 4: [15], 7: [40]}\n",
      "save\n",
      "{3: [82], 15: [89]}\n",
      "5\n",
      "{3: [83], 5: [16], 10: [32]}\n",
      "cents\n",
      "{3: [84], 10: [77]}\n",
      "ok\n",
      "{3: [85]}\n",
      "course\n",
      "{3: [86], 15: [33]}\n",
      "mind\n",
      "{3: [87]}\n",
      "young\n",
      "{3: [90], 6: [25]}\n",
      "man\n",
      "{3: [91], 6: [3, 88], 11: [5], 13: [0, 9], 14: [91]}\n",
      "pumping\n",
      "{3: [92]}\n",
      "waited\n",
      "{3: [95]}\n",
      "asked\n",
      "{3: [99], 12: [136]}\n",
      "penny\n",
      "{3: [106]}\n",
      "counts\n",
      "{3: [107]}\n",
      "99\n",
      "{3: [109]}\n",
      "bummer\n",
      "{3: [110]}\n",
      "1\n",
      "{3: [112], 9: [38]}\n",
      "pretty\n",
      "{3: [114], 12: [63]}\n",
      "cheap\n",
      "{3: [115]}\n",
      "though\n",
      "{3: [117]}\n",
      "eight\n",
      "{3: [119], 4: [75], 13: [97], 15: [6]}\n",
      "miles\n",
      "{3: [120]}\n",
      "paying\n",
      "{3: [123]}\n",
      "much\n",
      "{3: [124], 9: [18], 15: [75]}\n",
      "fill\n",
      "{3: [125]}\n",
      "tank\n",
      "{3: [126]}\n",
      "killing\n",
      "{3: [129]}\n",
      "grocery\n",
      "{3: [133]}\n",
      "around\n",
      "{3: [137], 5: [15]}\n",
      "neighborhood\n",
      "{3: [138], 13: [88]}\n",
      "show\n",
      "{3: [139]}\n",
      "wheels\n",
      "{3: [140]}\n",
      "company\n",
      "{4: [0, 85, 100], 9: [162]}\n",
      "phoenix\n",
      "{4: [1]}\n",
      "arizona\n",
      "{4: [2]}\n",
      "says\n",
      "{4: [3], 5: [97, 123], 9: [183]}\n",
      "clone\n",
      "{4: [4, 22, 27, 53]}\n",
      "cat\n",
      "{4: [5, 17, 28, 35, 63]}\n",
      "actually\n",
      "{4: [6]}\n",
      "felix\n",
      "{4: [8]}\n",
      "lee\n",
      "{4: [9, 87, 111]}\n",
      "president\n",
      "{4: [10]}\n",
      "twice\n",
      "{4: [11], 12: [98], 13: [116]}\n",
      "inc\n",
      "{4: [13, 38, 51, 78]}\n",
      "beloved\n",
      "{4: [16]}\n",
      "dies\n",
      "{4: [18]}\n",
      "clients\n",
      "{4: [20]}\n",
      "whose\n",
      "{4: [21]}\n",
      "lives\n",
      "{4: [23], 8: [94]}\n",
      "donor\n",
      "{4: [24]}\n",
      "steep\n",
      "{4: [26]}\n",
      "cost\n",
      "{4: [29], 12: [28]}\n",
      "50,000\n",
      "{4: [30]}\n",
      "first\n",
      "{4: [31], 5: [35], 9: [9, 29], 12: [33], 14: [31]}\n",
      "veterinarian\n",
      "{4: [32]}\n",
      "must\n",
      "{4: [33], 5: [149], 9: [28, 53], 12: [115]}\n",
      "biopsy\n",
      "{4: [34]}\n",
      "sent\n",
      "{4: [36], 8: [83]}\n",
      "twin\n",
      "{4: [37, 50, 77]}\n",
      "cultured\n",
      "{4: [39, 59]}\n",
      "grow\n",
      "{4: [40]}\n",
      "fresh\n",
      "{4: [41], 6: [48], 13: [95]}\n",
      "cells\n",
      "{4: [43, 45]}\n",
      "stored\n",
      "{4: [46]}\n",
      "liquid\n",
      "{4: [47]}\n",
      "nitrogen\n",
      "{4: [48]}\n",
      "notify\n",
      "{4: [49]}\n",
      "ready\n",
      "{4: [52]}\n",
      "pay\n",
      "{4: [55], 5: [92], 9: [175], 12: [155]}\n",
      "half\n",
      "{4: [56], 7: [41], 8: [33], 13: [172], 15: [150, 155]}\n",
      "amount\n",
      "{4: [57]}\n",
      "25,000\n",
      "{4: [58, 83]}\n",
      "cell\n",
      "{4: [60], 9: [143]}\n",
      "implanted\n",
      "{4: [61]}\n",
      "female\n",
      "{4: [62], 6: [26]}\n",
      "estrus\n",
      "{4: [64]}\n",
      "goes\n",
      "{4: [65], 5: [148], 7: [11], 14: [10]}\n",
      "kitten\n",
      "{4: [67, 73, 80]}\n",
      "born\n",
      "{4: [68]}\n",
      "60\n",
      "{4: [69]}\n",
      "days\n",
      "{4: [70]}\n",
      "later\n",
      "{4: [71], 5: [47], 6: [75]}\n",
      "weaned\n",
      "{4: [74]}\n",
      "weeks\n",
      "{4: [76]}\n",
      "delivers\n",
      "{4: [79]}\n",
      "receives\n",
      "{4: [81]}\n",
      "remaining\n",
      "{4: [82]}\n",
      "growing\n",
      "{4: [84]}\n",
      "facility\n",
      "{4: [88]}\n",
      "handle\n",
      "{4: [89]}\n",
      "dozen\n",
      "{4: [90]}\n",
      "births\n",
      "{4: [91]}\n",
      "year\n",
      "{4: [92, 99, 116], 5: [54], 6: [1], 7: [37, 38, 56, 119], 9: [126], 11: [1], 12: [139], 13: [11, 17, 157], 14: [14], 15: [134]}\n",
      "goal\n",
      "{4: [93]}\n",
      "50\n",
      "{4: [95, 97], 9: [42], 10: [76], 13: [23, 26]}\n",
      "kittens\n",
      "{4: [96]}\n",
      "puppies\n",
      "{4: [98]}\n",
      "currently\n",
      "{4: [101]}\n",
      "experimenting\n",
      "{4: [102]}\n",
      "stray\n",
      "{4: [103]}\n",
      "dogs\n",
      "{4: [104, 115]}\n",
      "canine\n",
      "{4: [105]}\n",
      "clones\n",
      "{4: [106]}\n",
      "seem\n",
      "{4: [107], 14: [26]}\n",
      "perfect\n",
      "{4: [108]}\n",
      "bizarre\n",
      "{4: [109]}\n",
      "nevertheless\n",
      "{4: [110]}\n",
      "believes\n",
      "{4: [112]}\n",
      "successfully\n",
      "{4: [113]}\n",
      "cloning\n",
      "{4: [114]}\n",
      "homebuyers\n",
      "{5: [0, 66]}\n",
      "nationwide\n",
      "{5: [1]}\n",
      "watching\n",
      "{5: [2]}\n",
      "housing\n",
      "{5: [3, 147, 152]}\n",
      "question\n",
      "{5: [8], 7: [72]}\n",
      "everyone\n",
      "{5: [9, 122, 127], 7: [6], 10: [93], 15: [100]}\n",
      "lips\n",
      "{5: [10]}\n",
      "interest\n",
      "{5: [12], 13: [141]}\n",
      "rates\n",
      "{5: [13]}\n",
      "stay\n",
      "{5: [14, 155]}\n",
      "telling\n",
      "{5: [18]}\n",
      "remarked\n",
      "{5: [19]}\n",
      "realtor\n",
      "{5: [21, 93]}\n",
      "santa\n",
      "{5: [22]}\n",
      "monica\n",
      "{5: [23]}\n",
      "crazy\n",
      "{5: [25]}\n",
      "tim\n",
      "{5: [27, 96]}\n",
      "looking\n",
      "{5: [28, 95], 12: [99]}\n",
      "near\n",
      "{5: [30]}\n",
      "199\n",
      "{5: [32], 10: [14, 31]}\n",
      "3\n",
      "{5: [33], 12: [29]}\n",
      "condominium\n",
      "{5: [39]}\n",
      "venice\n",
      "{5: [40, 81]}\n",
      "70,000\n",
      "{5: [41], 7: [59]}\n",
      "friends\n",
      "{5: [42, 56], 13: [41], 14: [50]}\n",
      "thought\n",
      "{5: [43], 15: [139]}\n",
      "overpaying\n",
      "{5: [44]}\n",
      "five\n",
      "{5: [45], 10: [24], 12: [25], 14: [18], 15: [133]}\n",
      "move\n",
      "{5: [48]}\n",
      "sold\n",
      "{5: [49], 7: [36, 84]}\n",
      "230,000\n",
      "{5: [50]}\n",
      "profit\n",
      "{5: [52]}\n",
      "last\n",
      "{5: [53], 13: [6, 70, 169]}\n",
      "visiting\n",
      "{5: [55]}\n",
      "saw\n",
      "{5: [57], 14: [129, 135]}\n",
      "local\n",
      "{5: [58], 13: [163]}\n",
      "paper\n",
      "{5: [59], 8: [63]}\n",
      "exact\n",
      "{5: [60]}\n",
      "condo\n",
      "{5: [61, 80]}\n",
      "sale\n",
      "{5: [62], 10: [49]}\n",
      "510,000\n",
      "{5: [63]}\n",
      "seller\n",
      "{5: [64]}\n",
      "market\n",
      "{5: [65]}\n",
      "feel\n",
      "{5: [67, 107, 113]}\n",
      "offer\n",
      "{5: [69]}\n",
      "least\n",
      "{5: [70]}\n",
      "asking\n",
      "{5: [73]}\n",
      "donna\n",
      "{5: [75]}\n",
      "told\n",
      "{5: [84], 14: [71]}\n",
      "whatever\n",
      "{5: [86]}\n",
      "anyone\n",
      "{5: [87]}\n",
      "offers\n",
      "{5: [88]}\n",
      "give\n",
      "{5: [89]}\n",
      "20,000\n",
      "{5: [90]}\n",
      "table\n",
      "{5: [91]}\n",
      "tired\n",
      "{5: [94], 14: [4]}\n",
      "hopes\n",
      "{5: [98, 128]}\n",
      "desperate\n",
      "{5: [100]}\n",
      "decide\n",
      "{5: [102, 104]}\n",
      "made\n",
      "{5: [109], 10: [89]}\n",
      "wrong\n",
      "{5: [110], 14: [150]}\n",
      "decision\n",
      "{5: [111], 9: [104]}\n",
      "overpaid\n",
      "{5: [115]}\n",
      "want\n",
      "{5: [117]}\n",
      "kick\n",
      "{5: [118]}\n",
      "great\n",
      "{5: [120], 7: [124], 15: [130]}\n",
      "opportunity\n",
      "{5: [121]}\n",
      "bubble\n",
      "{5: [124]}\n",
      "burst\n",
      "{5: [125, 129]}\n",
      "sometime\n",
      "{5: [126]}\n",
      "day\n",
      "{5: [130], 14: [48, 66, 77], 15: [163]}\n",
      "sell\n",
      "{5: [131], 15: [147]}\n",
      "government\n",
      "{5: [134], 12: [149]}\n",
      "idea\n",
      "{5: [136], 7: [133]}\n",
      "future\n",
      "{5: [137], 12: [51]}\n",
      "inevitably\n",
      "{5: [140]}\n",
      "things\n",
      "{5: [141], 11: [40], 13: [78, 103, 121]}\n",
      "cycles\n",
      "{5: [143]}\n",
      "state\n",
      "{5: [145], 8: [8], 9: [23, 181], 12: [130]}\n",
      "come\n",
      "{5: [150], 11: [133], 13: [114], 14: [52], 15: [27]}\n",
      "always\n",
      "{5: [154, 166], 13: [75, 86], 14: [25, 35]}\n",
      "little\n",
      "{5: [156], 9: [149, 163], 12: [163]}\n",
      "lose\n",
      "{5: [159], 13: [140]}\n",
      "twenty\n",
      "{5: [162]}\n",
      "road\n",
      "{5: [164]}\n",
      "paid\n",
      "{5: [168], 10: [61], 12: [108, 116]}\n",
      "79\n",
      "{6: [0]}\n",
      "slightly\n",
      "{6: [4]}\n",
      "injured\n",
      "{6: [5]}\n",
      "waiting\n",
      "{6: [7]}\n",
      "brand\n",
      "{6: [8]}\n",
      "convertible\n",
      "{6: [10]}\n",
      "lane\n",
      "{6: [12]}\n",
      "burger\n",
      "{6: [13, 83]}\n",
      "prince\n",
      "{6: [14, 84]}\n",
      "herman\n",
      "{6: [16]}\n",
      "sherman\n",
      "{6: [17, 33, 50]}\n",
      "northville\n",
      "{6: [18]}\n",
      "suffered\n",
      "{6: [19]}\n",
      "mild\n",
      "{6: [20]}\n",
      "burn\n",
      "{6: [21]}\n",
      "9\n",
      "{6: [22], 10: [37]}\n",
      "00\n",
      "{6: [23]}\n",
      "p\n",
      "{6: [24]}\n",
      "employee\n",
      "{6: [27, 70]}\n",
      "spilled\n",
      "{6: [29]}\n",
      "cup\n",
      "{6: [30]}\n",
      "coffee\n",
      "{6: [31, 35, 64]}\n",
      "lap\n",
      "{6: [32]}\n",
      "hot\n",
      "{6: [36], 10: [90]}\n",
      "scalding\n",
      "{6: [37]}\n",
      "refused\n",
      "{6: [38]}\n",
      "medical\n",
      "{6: [39]}\n",
      "aid\n",
      "{6: [40]}\n",
      "saying\n",
      "{6: [41]}\n",
      "stain\n",
      "{6: [43]}\n",
      "slacks\n",
      "{6: [44]}\n",
      "wash\n",
      "{6: [46]}\n",
      "given\n",
      "{6: [47], 7: [69]}\n",
      "refill\n",
      "{6: [49]}\n",
      "drove\n",
      "{6: [51], 7: [115], 10: [41]}\n",
      "john\n",
      "{6: [54], 10: [18, 84]}\n",
      "johnson\n",
      "{6: [55]}\n",
      "gave\n",
      "{6: [56]}\n",
      "free\n",
      "{6: [58], 7: [100]}\n",
      "gift\n",
      "{6: [59]}\n",
      "certificates\n",
      "{6: [60]}\n",
      "large\n",
      "{6: [63]}\n",
      "newest\n",
      "{6: [67]}\n",
      "sandwich\n",
      "{6: [68]}\n",
      "mcrap\n",
      "{6: [69]}\n",
      "hire\n",
      "{6: [72]}\n",
      "let\n",
      "{6: [73]}\n",
      "evening\n",
      "{6: [76]}\n",
      "quite\n",
      "{6: [77]}\n",
      "upset\n",
      "{6: [78]}\n",
      "probably\n",
      "{6: [81]}\n",
      "sue\n",
      "{6: [82]}\n",
      "letting\n",
      "{6: [85]}\n",
      "fault\n",
      "{6: [89]}\n",
      "ordering\n",
      "{6: [90], 11: [87]}\n",
      "something\n",
      "{6: [91]}\n",
      "able\n",
      "{6: [93]}\n",
      "spill\n",
      "{6: [94]}\n",
      "joke\n",
      "{7: [1]}\n",
      "los\n",
      "{7: [3, 34, 150], 11: [3, 21]}\n",
      "angeles\n",
      "{7: [4, 35, 151], 11: [4, 22]}\n",
      "reads\n",
      "{7: [5]}\n",
      "watches\n",
      "{7: [7]}\n",
      "tv\n",
      "{7: [8, 20], 9: [98], 15: [64]}\n",
      "rents\n",
      "{7: [9]}\n",
      "popular\n",
      "{7: [13, 86, 146], 8: [35]}\n",
      "reading\n",
      "{7: [14]}\n",
      "material\n",
      "{7: [15]}\n",
      "comic\n",
      "{7: [16]}\n",
      "books\n",
      "{7: [17, 128], 11: [52]}\n",
      "magazines\n",
      "{7: [19], 8: [42]}\n",
      "guides\n",
      "{7: [21]}\n",
      "libraries\n",
      "{7: [23]}\n",
      "car\n",
      "{7: [27], 9: [151, 165], 10: [40], 14: [115], 15: [48]}\n",
      "washes\n",
      "{7: [28]}\n",
      "explain\n",
      "{7: [29]}\n",
      "annual\n",
      "{7: [30], 14: [8]}\n",
      "book\n",
      "{7: [31, 143]}\n",
      "festival\n",
      "{7: [32, 48, 63, 99, 134, 159]}\n",
      "west\n",
      "{7: [33], 11: [64]}\n",
      "hour\n",
      "{7: [42], 9: [101], 11: [13]}\n",
      "space\n",
      "{7: [44]}\n",
      "become\n",
      "{7: [45]}\n",
      "available\n",
      "{7: [46]}\n",
      "outdoor\n",
      "{7: [47]}\n",
      "sponsored\n",
      "{7: [49]}\n",
      "newspaper\n",
      "{7: [50], 11: [137]}\n",
      "occurs\n",
      "{7: [51]}\n",
      "april\n",
      "{7: [53]}\n",
      "weekend\n",
      "{7: [55], 14: [122]}\n",
      "attendance\n",
      "{7: [57]}\n",
      "estimated\n",
      "{7: [58]}\n",
      "75,000\n",
      "{7: [61]}\n",
      "sunday\n",
      "{7: [62]}\n",
      "featured\n",
      "{7: [64]}\n",
      "280\n",
      "{7: [65]}\n",
      "exhibitors\n",
      "{7: [66]}\n",
      "90\n",
      "{7: [67]}\n",
      "talks\n",
      "{7: [68]}\n",
      "authors\n",
      "{7: [70, 81, 129]}\n",
      "audience\n",
      "{7: [71]}\n",
      "answer\n",
      "{7: [73]}\n",
      "period\n",
      "{7: [74], 9: [102]}\n",
      "talk\n",
      "{7: [76], 13: [119, 134]}\n",
      "autograph\n",
      "{7: [77]}\n",
      "seekers\n",
      "{7: [78]}\n",
      "sought\n",
      "{7: [79]}\n",
      "150\n",
      "{7: [80], 15: [3]}\n",
      "food\n",
      "{7: [82, 104], 8: [20, 25, 69], 13: [161]}\n",
      "court\n",
      "{7: [83, 105]}\n",
      "kinds\n",
      "{7: [85], 13: [120]}\n",
      "ethnic\n",
      "{7: [87]}\n",
      "foods\n",
      "{7: [88]}\n",
      "american\n",
      "{7: [89]}\n",
      "hamburgers\n",
      "{7: [90]}\n",
      "hawaiian\n",
      "{7: [91]}\n",
      "shave\n",
      "{7: [92]}\n",
      "ice\n",
      "{7: [93], 15: [159]}\n",
      "drinks\n",
      "{7: [94, 109]}\n",
      "except\n",
      "{7: [95], 11: [28]}\n",
      "7\n",
      "{7: [96], 10: [9]}\n",
      "avoided\n",
      "{7: [103]}\n",
      "sneaking\n",
      "{7: [107]}\n",
      "sandwiches\n",
      "{7: [108]}\n",
      "san\n",
      "{7: [116]}\n",
      "francisco\n",
      "{7: [117]}\n",
      "sixth\n",
      "{7: [118]}\n",
      "love\n",
      "{7: [120]}\n",
      "husband\n",
      "{7: [122]}\n",
      "fantastic\n",
      "{7: [123]}\n",
      "outdoors\n",
      "{7: [125]}\n",
      "among\n",
      "{7: [126]}\n",
      "many\n",
      "{7: [127], 14: [2, 78]}\n",
      "good\n",
      "{7: [131], 12: [135]}\n",
      "deals\n",
      "{7: [132]}\n",
      "occurred\n",
      "{7: [135], 13: [29], 15: [107]}\n",
      "nobody\n",
      "{7: [138]}\n",
      "knew\n",
      "{7: [139], 14: [36]}\n",
      "succeed\n",
      "{7: [141]}\n",
      "although\n",
      "{7: [142], 15: [132]}\n",
      "festivals\n",
      "{7: [144]}\n",
      "us\n",
      "{7: [147]}\n",
      "cities\n",
      "{7: [148]}\n",
      "embrace\n",
      "{7: [153]}\n",
      "angelenos\n",
      "{7: [155]}\n",
      "unpredictable\n",
      "{7: [156]}\n",
      "founders\n",
      "{7: [160]}\n",
      "inmates\n",
      "{8: [0, 11, 38]}\n",
      "released\n",
      "{8: [1]}\n",
      "correctional\n",
      "{8: [3]}\n",
      "officers\n",
      "{8: [4, 13, 17], 11: [124]}\n",
      "held\n",
      "{8: [5]}\n",
      "tower\n",
      "{8: [7, 79, 80]}\n",
      "prison\n",
      "{8: [9, 47, 85]}\n",
      "complex\n",
      "{8: [10]}\n",
      "captured\n",
      "{8: [12]}\n",
      "tried\n",
      "{8: [18], 11: [81]}\n",
      "quell\n",
      "{8: [19]}\n",
      "fight\n",
      "{8: [21, 26]}\n",
      "main\n",
      "{8: [22]}\n",
      "dining\n",
      "{8: [23]}\n",
      "room\n",
      "{8: [24]}\n",
      "erupted\n",
      "{8: [27]}\n",
      "prisoners\n",
      "{8: [28, 64, 100]}\n",
      "discovered\n",
      "{8: [29]}\n",
      "candy\n",
      "{8: [30, 34, 90, 104], 13: [94]}\n",
      "ration\n",
      "{8: [31, 105]}\n",
      "cut\n",
      "{8: [32, 51], 12: [122]}\n",
      "bartering\n",
      "{8: [36]}\n",
      "item\n",
      "{8: [37, 54]}\n",
      "trade\n",
      "{8: [39]}\n",
      "cigarettes\n",
      "{8: [40]}\n",
      "cigars\n",
      "{8: [41]}\n",
      "stationery\n",
      "{8: [43]}\n",
      "legal\n",
      "{8: [44]}\n",
      "dictionaries\n",
      "{8: [45]}\n",
      "items\n",
      "{8: [46, 58]}\n",
      "necessary\n",
      "{8: [50], 12: [133]}\n",
      "back\n",
      "{8: [52], 10: [78], 11: [116], 12: [156], 15: [28]}\n",
      "luxury\n",
      "{8: [53]}\n",
      "order\n",
      "{8: [55]}\n",
      "provide\n",
      "{8: [56]}\n",
      "basic\n",
      "{8: [57]}\n",
      "soap\n",
      "{8: [60, 112]}\n",
      "razors\n",
      "{8: [61]}\n",
      "toilet\n",
      "{8: [62]}\n",
      "berserk\n",
      "{8: [66]}\n",
      "reduction\n",
      "{8: [67]}\n",
      "threw\n",
      "{8: [68], 11: [33]}\n",
      "plates\n",
      "{8: [70]}\n",
      "silverware\n",
      "{8: [71]}\n",
      "doors\n",
      "{8: [72]}\n",
      "windows\n",
      "{8: [73]}\n",
      "guards\n",
      "{8: [74, 77, 93]}\n",
      "grabbed\n",
      "{8: [75]}\n",
      "hauled\n",
      "{8: [78], 15: [114]}\n",
      "door\n",
      "{8: [81], 10: [34], 13: [34], 14: [33]}\n",
      "secured\n",
      "{8: [82]}\n",
      "messages\n",
      "{8: [84]}\n",
      "demanding\n",
      "{8: [87]}\n",
      "big\n",
      "{8: [88], 9: [21], 12: [159], 15: [122]}\n",
      "bags\n",
      "{8: [89], 15: [60, 125]}\n",
      "exchange\n",
      "{8: [91]}\n",
      "sparing\n",
      "{8: [92]}\n",
      "warden\n",
      "{8: [95]}\n",
      "complied\n",
      "{8: [96]}\n",
      "demands\n",
      "{8: [97]}\n",
      "negotiations\n",
      "{8: [99]}\n",
      "approved\n",
      "{8: [101]}\n",
      "deal\n",
      "{8: [102]}\n",
      "restored\n",
      "{8: [103]}\n",
      "return\n",
      "{8: [106]}\n",
      "administration\n",
      "{8: [107]}\n",
      "reduce\n",
      "{8: [110]}\n",
      "daily\n",
      "{8: [111]}\n",
      "allotments\n",
      "{8: [113]}\n",
      "75\n",
      "{8: [114]}\n",
      "sam\n",
      "{9: [0, 67, 106, 111, 128, 136, 152, 168, 173, 179]}\n",
      "unemployed\n",
      "{9: [1]}\n",
      "piano\n",
      "{9: [2], 14: [136]}\n",
      "tuner\n",
      "{9: [3]}\n",
      "second\n",
      "{9: [5]}\n",
      "thing\n",
      "{9: [6, 10]}\n",
      "ever\n",
      "{9: [7]}\n",
      "life\n",
      "{9: [8], 13: [108], 14: [86], 15: [165]}\n",
      "afghan\n",
      "{9: [11]}\n",
      "blanket\n",
      "{9: [12]}\n",
      "church\n",
      "{9: [13]}\n",
      "raffle\n",
      "{9: [14]}\n",
      "25\n",
      "{9: [15]}\n",
      "bigger\n",
      "{9: [19]}\n",
      "120,000\n",
      "{9: [20, 135]}\n",
      "cube\n",
      "{9: [22, 33, 35, 63]}\n",
      "lottery\n",
      "{9: [24, 139, 182]}\n",
      "game\n",
      "{9: [25]}\n",
      "win\n",
      "{9: [26, 65]}\n",
      "contestant\n",
      "{9: [27, 52]}\n",
      "guess\n",
      "{9: [30, 54]}\n",
      "number\n",
      "{9: [31, 61, 79, 92]}\n",
      "spinning\n",
      "{9: [32]}\n",
      "stop\n",
      "{9: [34, 83, 116], 11: [135]}\n",
      "six\n",
      "{9: [36, 89], 12: [11]}\n",
      "numbers\n",
      "{9: [37]}\n",
      "x\n",
      "{9: [39, 41, 43, 45, 47, 50, 72]}\n",
      "100\n",
      "{9: [44, 48, 70, 131]}\n",
      "500\n",
      "{9: [46], 15: [121]}\n",
      "0\n",
      "{9: [49, 71, 132]}\n",
      "correct\n",
      "{9: [51]}\n",
      "selected\n",
      "{9: [56]}\n",
      "variables\n",
      "{9: [57, 76]}\n",
      "going\n",
      "{9: [58], 12: [22, 60, 150]}\n",
      "greater\n",
      "{9: [59]}\n",
      "guessing\n",
      "{9: [60]}\n",
      "appears\n",
      "{9: [62]}\n",
      "guarantee\n",
      "{9: [64]}\n",
      "correctly\n",
      "{9: [68]}\n",
      "guessed\n",
      "{9: [69]}\n",
      "choose\n",
      "{9: [74]}\n",
      "variable\n",
      "{9: [78, 91]}\n",
      "sign\n",
      "{9: [84, 117]}\n",
      "hill\n",
      "{9: [85]}\n",
      "street\n",
      "{9: [86, 146], 10: [12], 11: [75]}\n",
      "lake\n",
      "{9: [87]}\n",
      "avenue\n",
      "{9: [88], 11: [67]}\n",
      "hours\n",
      "{9: [90], 14: [47, 65, 76], 15: [7]}\n",
      "teenage\n",
      "{9: [94]}\n",
      "boy\n",
      "{9: [95], 15: [84, 136]}\n",
      "change\n",
      "{9: [97], 10: [79]}\n",
      "channels\n",
      "{9: [99]}\n",
      "tough\n",
      "{9: [103]}\n",
      "flipped\n",
      "{9: [107]}\n",
      "coin\n",
      "{9: [108]}\n",
      "heads\n",
      "{9: [110]}\n",
      "picked\n",
      "{9: [112, 114], 14: [94]}\n",
      "teenager\n",
      "{9: [113]}\n",
      "76\n",
      "{9: [119]}\n",
      "teen\n",
      "{9: [121]}\n",
      "clicked\n",
      "{9: [122]}\n",
      "120\n",
      "{9: [123, 134]}\n",
      "sixty\n",
      "{9: [125]}\n",
      "jumped\n",
      "{9: [129]}\n",
      "joy\n",
      "{9: [130]}\n",
      "dreamily\n",
      "{9: [137]}\n",
      "left\n",
      "{9: [138], 14: [39]}\n",
      "studio\n",
      "{9: [140]}\n",
      "talking\n",
      "{9: [141], 13: [143]}\n",
      "excitedly\n",
      "{9: [142]}\n",
      "phone\n",
      "{9: [144]}\n",
      "crossing\n",
      "{9: [145]}\n",
      "got\n",
      "{9: [147], 10: [39, 75], 11: [38], 12: [145], 13: [127, 137]}\n",
      "hit\n",
      "{9: [148]}\n",
      "sports\n",
      "{9: [150, 164]}\n",
      "slowly\n",
      "{9: [153]}\n",
      "getting\n",
      "{9: [154]}\n",
      "better\n",
      "{9: [155, 184], 14: [93]}\n",
      "hospital\n",
      "{9: [156, 158], 11: [7]}\n",
      "month\n",
      "{9: [157], 13: [30]}\n",
      "bill\n",
      "{9: [159]}\n",
      "110,000\n",
      "{9: [160]}\n",
      "insurance\n",
      "{9: [161]}\n",
      "sued\n",
      "{9: [167]}\n",
      "9,000\n",
      "{9: [169]}\n",
      "repairs\n",
      "{9: [171]}\n",
      "federal\n",
      "{9: [176], 12: [109, 131]}\n",
      "taxes\n",
      "{9: [177]}\n",
      "winnings\n",
      "{9: [178]}\n",
      "play\n",
      "{9: [180]}\n",
      "unlucky\n",
      "{9: [185]}\n",
      "sara\n",
      "{10: [0, 16, 29, 38, 50, 65, 74, 80]}\n",
      "smith\n",
      "{10: [1]}\n",
      "pasadena\n",
      "{10: [2]}\n",
      "resident\n",
      "{10: [3], 12: [57]}\n",
      "shopping\n",
      "{10: [5], 15: [55]}\n",
      "303\n",
      "{10: [8]}\n",
      "n\n",
      "{10: [10]}\n",
      "foothill\n",
      "{10: [11]}\n",
      "since\n",
      "{10: [13]}\n",
      "2\n",
      "{10: [15]}\n",
      "married\n",
      "{10: [17], 13: [22]}\n",
      "seven\n",
      "{10: [19]}\n",
      "children\n",
      "{10: [22]}\n",
      "bob\n",
      "{10: [23]}\n",
      "nancy\n",
      "{10: [27]}\n",
      "owns\n",
      "{10: [30]}\n",
      "blue\n",
      "{10: [35]}\n",
      "toyola\n",
      "{10: [36]}\n",
      "barget\n",
      "{10: [42, 47]}\n",
      "department\n",
      "{10: [43]}\n",
      "holiday\n",
      "{10: [48]}\n",
      "slice\n",
      "{10: [53]}\n",
      "toaster\n",
      "{10: [54]}\n",
      "29.95\n",
      "{10: [55]}\n",
      "plus\n",
      "{10: [56]}\n",
      "tax\n",
      "{10: [57]}\n",
      "regular\n",
      "{10: [58]}\n",
      "39.95\n",
      "{10: [60]}\n",
      "check\n",
      "{10: [62]}\n",
      "way\n",
      "{10: [63], 12: [154]}\n",
      "milkplus\n",
      "{10: [67]}\n",
      "nonfat\n",
      "{10: [70]}\n",
      "milk\n",
      "{10: [71, 72]}\n",
      "3.50\n",
      "{10: [73]}\n",
      "arrived\n",
      "{10: [81]}\n",
      "kids\n",
      "{10: [85], 13: [85, 89]}\n",
      "sleeping\n",
      "{10: [87]}\n",
      "woke\n",
      "{10: [88]}\n",
      "nutritious\n",
      "{10: [91]}\n",
      "breakfast\n",
      "{10: [92]}\n",
      "24\n",
      "{11: [0]}\n",
      "taken\n",
      "{11: [6]}\n",
      "county\n",
      "{11: [8]}\n",
      "jail\n",
      "{11: [9]}\n",
      "leading\n",
      "{11: [10]}\n",
      "freeway\n",
      "{11: [14]}\n",
      "chase\n",
      "{11: [15, 18, 26, 126]}\n",
      "stolen\n",
      "{11: [16]}\n",
      "suv\n",
      "{11: [17, 108, 132]}\n",
      "ended\n",
      "{11: [19]}\n",
      "downtown\n",
      "{11: [20, 39]}\n",
      "front\n",
      "{11: [23, 114]}\n",
      "spring\n",
      "{11: [24]}\n",
      "hotel\n",
      "{11: [25]}\n",
      "uneventful\n",
      "{11: [27]}\n",
      "empty\n",
      "{11: [29, 99]}\n",
      "bottle\n",
      "{11: [30]}\n",
      "whiskey\n",
      "{11: [31]}\n",
      "driver\n",
      "{11: [32, 37, 61, 79, 91, 101, 139, 149, 156]}\n",
      "vehicle\n",
      "{11: [36]}\n",
      "started\n",
      "{11: [41]}\n",
      "happen\n",
      "{11: [42]}\n",
      "ran\n",
      "{11: [43, 113]}\n",
      "fire\n",
      "{11: [44], 12: [16, 78, 87], 15: [97]}\n",
      "hydrant\n",
      "{11: [45, 48]}\n",
      "water\n",
      "{11: [46]}\n",
      "spewed\n",
      "{11: [47]}\n",
      "causing\n",
      "{11: [49]}\n",
      "geyser\n",
      "{11: [50]}\n",
      "ruined\n",
      "{11: [51]}\n",
      "carts\n",
      "{11: [54], 15: [56]}\n",
      "vendor\n",
      "{11: [55]}\n",
      "put\n",
      "{11: [56], 12: [93]}\n",
      "outside\n",
      "{11: [57]}\n",
      "attract\n",
      "{11: [58]}\n",
      "bookstore\n",
      "{11: [60]}\n",
      "hurriedly\n",
      "{11: [62]}\n",
      "turned\n",
      "{11: [63]}\n",
      "onto\n",
      "{11: [65]}\n",
      "grand\n",
      "{11: [66]}\n",
      "managed\n",
      "{11: [68]}\n",
      "bang\n",
      "{11: [69]}\n",
      "parked\n",
      "{11: [71]}\n",
      "side\n",
      "{11: [74, 78]}\n",
      "officer\n",
      "{11: [84, 160]}\n",
      "standing\n",
      "{11: [85]}\n",
      "crosswalk\n",
      "{11: [86]}\n",
      "halt\n",
      "{11: [88]}\n",
      "turning\n",
      "{11: [89]}\n",
      "caused\n",
      "{11: [92]}\n",
      "slam\n",
      "{11: [94]}\n",
      "brakes\n",
      "{11: [95]}\n",
      "avoid\n",
      "{11: [96]}\n",
      "collision\n",
      "{11: [97]}\n",
      "uninjured\n",
      "{11: [102]}\n",
      "pursuing\n",
      "{11: [107]}\n",
      "different\n",
      "{11: [109], 14: [124, 126]}\n",
      "directions\n",
      "{11: [110]}\n",
      "lucky\n",
      "{11: [111], 12: [97], 14: [72, 84]}\n",
      "drivers\n",
      "{11: [117]}\n",
      "braked\n",
      "{11: [118]}\n",
      "early\n",
      "{11: [119]}\n",
      "enough\n",
      "{11: [120]}\n",
      "damage\n",
      "{11: [121]}\n",
      "minor\n",
      "{11: [123]}\n",
      "resumed\n",
      "{11: [125]}\n",
      "blocks\n",
      "{11: [129]}\n",
      "find\n",
      "{11: [131]}\n",
      "full\n",
      "{11: [134], 13: [107], 15: [71]}\n",
      "plowed\n",
      "{11: [136]}\n",
      "stand\n",
      "{11: [138]}\n",
      "wearing\n",
      "{11: [140]}\n",
      "seatbelt\n",
      "{11: [141]}\n",
      "slumped\n",
      "{11: [142]}\n",
      "behind\n",
      "{11: [143]}\n",
      "steering\n",
      "{11: [144]}\n",
      "wheel\n",
      "{11: [145]}\n",
      "proprietor\n",
      "{11: [146]}\n",
      "newsstand\n",
      "{11: [147]}\n",
      "yelling\n",
      "{11: [148]}\n",
      "shaking\n",
      "{11: [150]}\n",
      "magazine\n",
      "{11: [151]}\n",
      "called\n",
      "{11: [153], 13: [175], 14: [57]}\n",
      "ambulance\n",
      "{11: [154]}\n",
      "failure\n",
      "{11: [157]}\n",
      "yield\n",
      "{11: [158]}\n",
      "driving\n",
      "{11: [161]}\n",
      "influence\n",
      "{11: [162]}\n",
      "mountain\n",
      "{12: [0]}\n",
      "town\n",
      "{12: [1, 42, 49]}\n",
      "canton\n",
      "{12: [2, 90]}\n",
      "elevation\n",
      "{12: [3]}\n",
      "6,000\n",
      "{12: [4]}\n",
      "feet\n",
      "{12: [5]}\n",
      "surrounded\n",
      "{12: [6]}\n",
      "thick\n",
      "{12: [7]}\n",
      "underbrush\n",
      "{12: [8, 21]}\n",
      "pine\n",
      "{12: [9]}\n",
      "trees\n",
      "{12: [10, 19, 34, 112, 147]}\n",
      "drought\n",
      "{12: [13]}\n",
      "plants\n",
      "{12: [14]}\n",
      "major\n",
      "{12: [15]}\n",
      "hazard\n",
      "{12: [17]}\n",
      "thousands\n",
      "{12: [18]}\n",
      "tons\n",
      "{12: [20], 13: [93], 15: [16, 36]}\n",
      "removed\n",
      "{12: [23, 32, 36]}\n",
      "minimum\n",
      "{12: [27]}\n",
      "million\n",
      "{12: [30], 15: [142]}\n",
      "brush\n",
      "{12: [31]}\n",
      "toppled\n",
      "{12: [35]}\n",
      "cleared\n",
      "{12: [37, 66, 73]}\n",
      "nonflammable\n",
      "{12: [38]}\n",
      "area\n",
      "{12: [39, 65, 74]}\n",
      "safely\n",
      "{12: [40]}\n",
      "surround\n",
      "{12: [41]}\n",
      "4,000\n",
      "{12: [43, 80]}\n",
      "look\n",
      "{12: [45]}\n",
      "forward\n",
      "{12: [46, 100]}\n",
      "help\n",
      "{12: [48], 13: [77]}\n",
      "survive\n",
      "{12: [50]}\n",
      "inferno\n",
      "{12: [52]}\n",
      "trucks\n",
      "{12: [59], 15: [116]}\n",
      "make\n",
      "{12: [61, 67, 72]}\n",
      "bad\n",
      "{12: [64]}\n",
      "dirt\n",
      "{12: [69]}\n",
      "bikers\n",
      "{12: [70]}\n",
      "try\n",
      "{12: [71]}\n",
      "personal\n",
      "{12: [75]}\n",
      "playground\n",
      "{12: [76]}\n",
      "recent\n",
      "{12: [77]}\n",
      "burned\n",
      "{12: [79]}\n",
      "acres\n",
      "{12: [81]}\n",
      "destroyed\n",
      "{12: [82]}\n",
      "11\n",
      "{12: [83]}\n",
      "nearby\n",
      "{12: [85]}\n",
      "hamilton\n",
      "{12: [86]}\n",
      "raging\n",
      "{12: [88]}\n",
      "toward\n",
      "{12: [89]}\n",
      "sudden\n",
      "{12: [91]}\n",
      "rainstorm\n",
      "{12: [92]}\n",
      "massive\n",
      "{12: [101]}\n",
      "clearing\n",
      "{12: [102, 107]}\n",
      "operation\n",
      "{12: [103]}\n",
      "ninety\n",
      "{12: [104]}\n",
      "cutting\n",
      "{12: [106]}\n",
      "funds\n",
      "{12: [110]}\n",
      "unfortunately\n",
      "{12: [111]}\n",
      "private\n",
      "{12: [113]}\n",
      "property\n",
      "{12: [114, 148]}\n",
      "range\n",
      "{12: [119]}\n",
      "remove\n",
      "{12: [123]}\n",
      "tree\n",
      "{12: [125]}\n",
      "apply\n",
      "{12: [129]}\n",
      "loans\n",
      "{12: [132]}\n",
      "thelma\n",
      "{12: [137]}\n",
      "65\n",
      "{12: [138]}\n",
      "widow\n",
      "{12: [141]}\n",
      "living\n",
      "{12: [142]}\n",
      "social\n",
      "{12: [143]}\n",
      "security\n",
      "{12: [144]}\n",
      "loan\n",
      "{12: [151]}\n",
      "supposed\n",
      "{12: [157]}\n",
      "planners\n",
      "{12: [158]}\n",
      "ideas\n",
      "{12: [160]}\n",
      "ought\n",
      "{12: [161]}\n",
      "woman\n",
      "{13: [1, 15]}\n",
      "died\n",
      "{13: [2]}\n",
      "apparent\n",
      "{13: [3]}\n",
      "murder\n",
      "{13: [4]}\n",
      "suicide\n",
      "{13: [5]}\n",
      "altadena\n",
      "{13: [8]}\n",
      "74\n",
      "{13: [10]}\n",
      "dominic\n",
      "{13: [13]}\n",
      "vittorio\n",
      "{13: [14, 43]}\n",
      "70\n",
      "{13: [16]}\n",
      "wife\n",
      "{13: [19, 51]}\n",
      "victoria\n",
      "{13: [20]}\n",
      "fact\n",
      "{13: [25]}\n",
      "th\n",
      "{13: [27]}\n",
      "anniversary\n",
      "{13: [28]}\n",
      "according\n",
      "{13: [32]}\n",
      "neighbor\n",
      "{13: [35], 14: [32, 96, 103, 109, 117]}\n",
      "mrs\n",
      "{13: [36, 66, 149]}\n",
      "allen\n",
      "{13: [37, 67, 150]}\n",
      "childless\n",
      "{13: [39]}\n",
      "close\n",
      "{13: [40]}\n",
      "mr\n",
      "{13: [42]}\n",
      "retired\n",
      "{13: [44]}\n",
      "carpenter\n",
      "{13: [45]}\n",
      "emphysema\n",
      "{13: [46]}\n",
      "blind\n",
      "{13: [47]}\n",
      "eye\n",
      "{13: [49]}\n",
      "cataract\n",
      "{13: [50]}\n",
      "diabetic\n",
      "{13: [52]}\n",
      "foot\n",
      "{13: [55]}\n",
      "amputated\n",
      "{13: [56]}\n",
      "complications\n",
      "{13: [57]}\n",
      "disease\n",
      "{13: [58]}\n",
      "eyesight\n",
      "{13: [59]}\n",
      "completely\n",
      "{13: [61]}\n",
      "gone\n",
      "{13: [62]}\n",
      "widowed\n",
      "{13: [73]}\n",
      "dom\n",
      "{13: [74, 104]}\n",
      "changing\n",
      "{13: [80]}\n",
      "light\n",
      "{13: [81], 15: [8]}\n",
      "bulbs\n",
      "{13: [82]}\n",
      "fixing\n",
      "{13: [83]}\n",
      "appliances\n",
      "{13: [84]}\n",
      "friendly\n",
      "{13: [87, 106]}\n",
      "halloween\n",
      "{13: [91]}\n",
      "handed\n",
      "{13: [92]}\n",
      "fruit\n",
      "{13: [96]}\n",
      "vicky\n",
      "{13: [100]}\n",
      "diabetes\n",
      "{13: [102]}\n",
      "seemed\n",
      "{13: [109, 139]}\n",
      "quieter\n",
      "{13: [111, 112]}\n",
      "nicest\n",
      "{13: [122]}\n",
      "happened\n",
      "{13: [124]}\n",
      "less\n",
      "{13: [125, 126]}\n",
      "sicker\n",
      "{13: [128]}\n",
      "conversations\n",
      "{13: [135]}\n",
      "steadily\n",
      "{13: [136]}\n",
      "shorter\n",
      "{13: [138]}\n",
      "listening\n",
      "{13: [142]}\n",
      "pain\n",
      "{13: [148]}\n",
      "talked\n",
      "{13: [153]}\n",
      "either\n",
      "{13: [154]}\n",
      "vittorios\n",
      "{13: [155]}\n",
      "never\n",
      "{13: [158]}\n",
      "delivered\n",
      "{13: [162]}\n",
      "agency\n",
      "{13: [164]}\n",
      "heard\n",
      "{13: [166]}\n",
      "gunshots\n",
      "{13: [168]}\n",
      "scared\n",
      "{13: [171]}\n",
      "death\n",
      "{13: [173]}\n",
      "immediately\n",
      "{13: [174]}\n",
      "sad\n",
      "{13: [177]}\n",
      "ending\n",
      "{13: [178]}\n",
      "together\n",
      "{13: [182]}\n",
      "sickness\n",
      "{13: [183]}\n",
      "alone\n",
      "{13: [184]}\n",
      "world\n",
      "{13: [185], 14: [101]}\n",
      "samantha\n",
      "{14: [0, 128]}\n",
      "renting\n",
      "{14: [5]}\n",
      "clockwork\n",
      "{14: [12]}\n",
      "landlord\n",
      "{14: [15]}\n",
      "raises\n",
      "{14: [16]}\n",
      "moved\n",
      "{14: [29, 41, 43]}\n",
      "slammer\n",
      "{14: [34]}\n",
      "saxophonist\n",
      "{14: [42, 44]}\n",
      "practiced\n",
      "{14: [45]}\n",
      "whole\n",
      "{14: [55, 85]}\n",
      "band\n",
      "{14: [56]}\n",
      "saxophone\n",
      "{14: [60, 67, 130]}\n",
      "playing\n",
      "{14: [61, 68, 74]}\n",
      "permitted\n",
      "{14: [62]}\n",
      "apartments\n",
      "{14: [63]}\n",
      "job\n",
      "{14: [69], 15: [18]}\n",
      "related\n",
      "{14: [70]}\n",
      "unhappy\n",
      "{14: [79]}\n",
      "happy\n",
      "{14: [82, 142]}\n",
      "howard\n",
      "{14: [88, 134]}\n",
      "middle\n",
      "{14: [89]}\n",
      "aged\n",
      "{14: [90]}\n",
      "chef\n",
      "{14: [97]}\n",
      "best\n",
      "{14: [99]}\n",
      "leftovers\n",
      "{14: [100]}\n",
      "pianist\n",
      "{14: [104]}\n",
      "played\n",
      "{14: [105]}\n",
      "delightful\n",
      "{14: [106, 138]}\n",
      "mechanic\n",
      "{14: [110]}\n",
      "tune\n",
      "{14: [111]}\n",
      "ups\n",
      "{14: [112]}\n",
      "changed\n",
      "{14: [113]}\n",
      "oil\n",
      "{14: [114]}\n",
      "latest\n",
      "{14: [116]}\n",
      "birder\n",
      "{14: [118]}\n",
      "birding\n",
      "{14: [120]}\n",
      "binoculars\n",
      "{14: [123]}\n",
      "persons\n",
      "{14: [125]}\n",
      "attitudes\n",
      "{14: [127]}\n",
      "player\n",
      "{14: [131, 137]}\n",
      "irritating\n",
      "{14: [132]}\n",
      "yet\n",
      "{14: [133, 145]}\n",
      "millions\n",
      "{14: [139, 146]}\n",
      "roof\n",
      "{14: [143, 149]}\n",
      "head\n",
      "{14: [144]}\n",
      "complain\n",
      "{14: [148]}\n",
      "crew\n",
      "{15: [1]}\n",
      "consisting\n",
      "{15: [2]}\n",
      "volunteers\n",
      "{15: [4, 95]}\n",
      "worked\n",
      "{15: [5]}\n",
      "drizzle\n",
      "{15: [9]}\n",
      "clean\n",
      "{15: [11]}\n",
      "carson\n",
      "{15: [12]}\n",
      "creek\n",
      "{15: [13]}\n",
      "nine\n",
      "{15: [15, 35]}\n",
      "debris\n",
      "{15: [17, 113]}\n",
      "done\n",
      "{15: [20, 78]}\n",
      "smiled\n",
      "{15: [21]}\n",
      "alan\n",
      "{15: [22]}\n",
      "specter\n",
      "{15: [23]}\n",
      "event\n",
      "{15: [25]}\n",
      "scheduled\n",
      "{15: [26]}\n",
      "hope\n",
      "{15: [34]}\n",
      "garbage\n",
      "{15: [37, 40]}\n",
      "shapes\n",
      "{15: [42]}\n",
      "sizes\n",
      "{15: [43]}\n",
      "colors\n",
      "{15: [44]}\n",
      "cans\n",
      "{15: [45]}\n",
      "bottles\n",
      "{15: [46]}\n",
      "bicycles\n",
      "{15: [47]}\n",
      "tires\n",
      "{15: [49]}\n",
      "auto\n",
      "{15: [50]}\n",
      "batteries\n",
      "{15: [51]}\n",
      "sofas\n",
      "{15: [52]}\n",
      "furniture\n",
      "{15: [53]}\n",
      "clothing\n",
      "{15: [54]}\n",
      "bowling\n",
      "{15: [57]}\n",
      "balls\n",
      "{15: [58]}\n",
      "plastic\n",
      "{15: [59]}\n",
      "dolls\n",
      "{15: [61]}\n",
      "baby\n",
      "{15: [62]}\n",
      "carriages\n",
      "{15: [63]}\n",
      "antennas\n",
      "{15: [65]}\n",
      "portable\n",
      "{15: [66]}\n",
      "radios\n",
      "{15: [67]}\n",
      "golf\n",
      "{15: [69, 73]}\n",
      "bag\n",
      "{15: [70]}\n",
      "set\n",
      "{15: [72]}\n",
      "clubs\n",
      "{15: [74]}\n",
      "backbreaking\n",
      "{15: [76]}\n",
      "groups\n",
      "{15: [81, 88]}\n",
      "cub\n",
      "{15: [82]}\n",
      "scouts\n",
      "{15: [83, 85]}\n",
      "environmental\n",
      "{15: [87]}\n",
      "bay\n",
      "{15: [90]}\n",
      "whales\n",
      "{15: [92, 153]}\n",
      "concerned\n",
      "{15: [93]}\n",
      "retirees\n",
      "{15: [94]}\n",
      "departments\n",
      "{15: [98]}\n",
      "assisted\n",
      "{15: [99]}\n",
      "issued\n",
      "{15: [101]}\n",
      "boots\n",
      "{15: [102]}\n",
      "gloves\n",
      "{15: [103]}\n",
      "rain\n",
      "{15: [104]}\n",
      "gear\n",
      "{15: [105]}\n",
      "along\n",
      "{15: [108]}\n",
      "stretch\n",
      "{15: [111]}\n",
      "streambed\n",
      "{15: [112]}\n",
      "roadside\n",
      "{15: [115]}\n",
      "lined\n",
      "{15: [117]}\n",
      "take\n",
      "{15: [118]}\n",
      "landfill\n",
      "{15: [120]}\n",
      "yellow\n",
      "{15: [123]}\n",
      "filled\n",
      "{15: [126]}\n",
      "found\n",
      "{15: [128, 137]}\n",
      "anything\n",
      "{15: [129]}\n",
      "value\n",
      "{15: [131]}\n",
      "earring\n",
      "{15: [138]}\n",
      "dollars\n",
      "{15: [143]}\n",
      "shiny\n",
      "{15: [144]}\n",
      "donate\n",
      "{15: [149]}\n",
      "proceeds\n",
      "{15: [151]}\n",
      "use\n",
      "{15: [154]}\n",
      "triple\n",
      "{15: [157]}\n",
      "scoop\n",
      "{15: [158]}\n",
      "cream\n",
      "{15: [160]}\n",
      "cone\n",
      "{15: [161]}\n",
      "rest\n",
      "{15: [164]}\n"
     ]
    }
   ],
   "source": [
    "# Create the inverted index using defaultdict() for simplicity\n",
    "inverted_index = defaultdict(dict)\n",
    "\n",
    "# Create an empty list which will contain docIDs\n",
    "docIDs = []\n",
    "\n",
    "# Define a variable for allocating docIDs to the files\n",
    "id = 1\n",
    "\n",
    "# Start to read document files\n",
    "for text_file in text_files:\n",
    "    \n",
    "    # Define docID to for the current file we want to read\n",
    "    file_id = id \n",
    "    \n",
    "    # Append the docID to the corresponding list\n",
    "    docIDs.append(id)\n",
    "    \n",
    "    # Join file_path and file name to make a full file path\n",
    "    file_path = os.path.join(folder_path, text_file)\n",
    "    \n",
    "    # Open files, read them, and store their content in file_content variable\n",
    "    try:\n",
    "        with open(file_path, 'r', encoding='utf-8') as file:\n",
    "            file_content = file.read()\n",
    "    except UnicodeDecodeError:\n",
    "        with open(file_path, 'r', encoding='latin-1') as file:\n",
    "            file_content = file.read()\n",
    "\n",
    "    # Preprocess each file content using preprocessing \n",
    "    # function whivch is defined above\n",
    "    processed_text = preprocessing(file_content)\n",
    "    \n",
    "    # Create inverted index: check if each word exists in processed document or not\n",
    "    for index, token in enumerate(processed_text.split()):\n",
    "\n",
    "        # If the word is in the dictionary, its file_id may or \n",
    "        # may not exist in the inverted index\n",
    "        if token in inverted_index:\n",
    "            \n",
    "            # If file_id of the token is in the corresponding list of the \n",
    "            # token in dictionary, just add its index to the corresponding list\n",
    "            if file_id in inverted_index[token]:\n",
    "                inverted_index[token][file_id].append(index)\n",
    "         \n",
    "            # Else, add the file_id and the token's index in the file to the dictionary\n",
    "            else:\n",
    "                inverted_index[token][file_id] = [index]\n",
    "\n",
    "        # If token is not in the dictionary, add token, its file id,\n",
    "        # and its index in the corresponding file to the dictionary\n",
    "        else:\n",
    "            inverted_index[token][file_id] = [index]\n",
    "            \n",
    "    # To store processed texts, create some new .txt \n",
    "    # files an append the processed_text to them\n",
    "    # Note: run this part of the code only once.\n",
    "    #with open(\"data/processed/processed%i.txt\" % (id), \"a\") as processed_file:\n",
    "    #    # Write text to the file\n",
    "    #    processed_file.write(processed_text)\n",
    "    \n",
    "    # Prepare id variable for next file\n",
    "    id = id + 1    \n",
    "\n",
    "# Print the inverted index:\n",
    "for key in inverted_index:\n",
    "    print(key)\n",
    "    print(inverted_index[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d534b71f",
   "metadata": {},
   "source": [
    "## Query Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369f6ff6",
   "metadata": {},
   "source": [
    "<div style=\"line-height: 2;\">This is the last section which contains query processing.<br>\n",
    "In this part, I defined a while loop to get the queries from user until he/she inters \"Quit\" keyword to interrup the seatch engine.<br>\n",
    "    As it is guaranteed that the queries are which of the forms below, I did the needed process.\n",
    "    <ul>\n",
    "        <li>\n",
    "            x OR y\n",
    "        </li>\n",
    "        <li>\n",
    "            x AND y\n",
    "        </li>\n",
    "        <li>\n",
    "            NOT x\n",
    "        </li>\n",
    "    </ul>\n",
    "    For this job, first of all, I defined 2 lists to store docIDs of query tokens(I mean x and y in the list above, not OR, NOT, and ANd) which contain docIDs of the files which have the token. Another list I defined, is \"terms\". It contains tokens of the query after splitting. And the last list is \"result\" which contains docIDs which we have to print as a response to each query. <br>\n",
    "    If the query contains \"NOT\" in it, do the preprocessing fo the single-word after \"NOT\" and store it in the term1 variable. Then, append the values of the word in dictionary to docIDs_1. Then, for implementing \"NOT\" instruction, search in docIDs list for IDs which are not it docIDs_1 and print it as the result.<br>\n",
    "    If the query contains \"OR\", \"AND\", or \"NEAR\\\", we have to words in our query which we have to do the neccessary procces for them. But the common part for all these 3 cases is that we have to seperate 2 words and store them in term1 and term2 variables. The, append docIDs corresponding to each term to its corresponding docIDs list(for term1, store in docIDs_1 and for term2 store in docIDs_2) to store docIDs which each word appead in the corresponding document. Now, if \"OR\" is in the query, clearly we have to do the logical or on docIDs_1 and docIDs_2 and output the result. But if there is \"AND\" or \"NEAR/\" in the query, we have to intersect the docIDs_1 and cocIDs_2. If there is \"AND\", we obtained the result and give it to output. <br>\n",
    "    But if there is \"NEAR/\", after intersecting, we have to do more. Tracing on intersected documents and find the indecies which the words appeared in. If the distance is less or equal to the number after / in \"NEAR/\", add it to the result. If not, do the process for next document.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59160b20",
   "metadata": {},
   "source": [
    "Also, there is a merge function which is used for computing union of 2 lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "da454347",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge 2 sorted lists which will give a sorted array of docIDs\n",
    "def merge(docID1, docID2):\n",
    "    result = []\n",
    "    i = 0\n",
    "    j = 0\n",
    "    flag = 1\n",
    "    while i < len(docID1) and j < len(docID2):\n",
    "        if docID1[i] == docID2[j]:\n",
    "            result.append(docID1[i])\n",
    "            i += 1\n",
    "            j += 1\n",
    "        elif docID1[i] < docID2[j]:\n",
    "            result.append(docID1[i])\n",
    "            i += 1\n",
    "        else:\n",
    "            result.append(docID2[j])\n",
    "            j += 1\n",
    "    if i == len(docID1):\n",
    "        if j < len(docID2):\n",
    "            for k in range(j, len(docID2)):\n",
    "                result.append(docID2[k])\n",
    "    if j == len(docID2):\n",
    "        if i < len(docID1):\n",
    "            for k in range(i, len(docID1)):\n",
    "                result.append(docID1[k])\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68a67c48",
   "metadata": {},
   "source": [
    "# Query processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "a0142293",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 6]\n",
      "Jerry Decided To Buy a Gun.txt\n",
      "Man Injured at Fast Food Place.txt\n"
     ]
    }
   ],
   "source": [
    "query = \"\"\n",
    "\n",
    "# Define a true while to give the query from user\n",
    "while True:\n",
    "    docIDs_1 = []\n",
    "    docIDs_2 = []\n",
    "    terms = []\n",
    "    result = []\n",
    "    query = input()\n",
    "    \n",
    "    if query == \"Quit\":\n",
    "        break\n",
    "        \n",
    "    else:\n",
    "        terms = query.split()\n",
    "\n",
    "        if \"NOT\" in query:\n",
    "            term1 = preprocessing(terms[1])\n",
    "            for key in inverted_index[term1]:\n",
    "                docIDs_1.append(key)\n",
    "            result = [docID for docID in docIDs if docID not in docIDs_1]\n",
    "\n",
    "        else:\n",
    "            term1 = preprocessing(terms[0])\n",
    "            term2 = preprocessing(terms[2])\n",
    "\n",
    "            for key in inverted_index[term1]:\n",
    "                docIDs_1.append(key)\n",
    "            for key in inverted_index[term2]:\n",
    "                docIDs_2.append(key)\n",
    "\n",
    "            if \"OR\" in query:\n",
    "                result = merge(docIDs_1, docIDs_2)\n",
    "\n",
    "            else:\n",
    "                intersection = list(filter(lambda value: value in docIDs_2, docIDs_1))\n",
    "\n",
    "                if \"NEAR/\" in query:\n",
    "                    # Calculate number of words between terms[0] and terms[2]\n",
    "                    near = terms[1][5:]\n",
    "                    near = int(near)\n",
    "\n",
    "                    # Check if terms[0] and terms[2] are near atmost \"near\" words\n",
    "                    for i in range(len(intersection)):\n",
    "                        for index1 in inverted_index[term1][intersection[i]]:\n",
    "                            for index2 in inverted_index[term2][intersection[i]]:\n",
    "                                diff = abs(index1 - index2) - 1\n",
    "                                if diff <= near and diff >= 0:\n",
    "                                    result.append(intersection[i])\n",
    "                else:\n",
    "                    result = intersection\n",
    "\n",
    "        # Print result docIDs\n",
    "        print(result)\n",
    "        \n",
    "        # Print result documents' titles\n",
    "        for docID in result:\n",
    "            print(text_files[docID - 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}