Boolean Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Boolean Model from Scratch\n",
    "\n",
    "### Boolean model is used in information retrieval systems to retrieve relevant documents from the corpus of documents. Query is fed into the model, and using the set operations based on the query model evaluates it and return back the relevant documents.\n",
    "\n",
    "*make_doc()*  function initializes the corpus of the boolean model. Here we are taking six documents for the demonstration purpose. <br>\n",
    "doc 1: MS Dhoni <br>\n",
    "doc 2: Persistent Systems <br>\n",
    "doc 3: Indian Army <br>\n",
    "doc 4: Question Answering Systems <br>\n",
    "doc 5: GATE <br>\n",
    "doc 6: Internals Best Talk Show <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_doc():\n",
    "    doc1 = \"MS Dhoni is former Indian Cricketer and plays in IPL\"\n",
    "    doc2 = \"Persistent systems is the only software company which comes for placements\"\n",
    "    doc3 = \"Personnel who serve in the Para (SF) are allowed to wear the Balidan (Sacrifice) patch on their right pocket\"\n",
    "    doc4 = \"Question Answering System can pull of answers from unstructured collection of natural language\"\n",
    "    doc5 = \"It is an examination that primarily tests the comprehensive understanding of various undergraduate subjects\"\n",
    "    doc6 = \"Internals is the best talk show youtube have ever hosted\"\n",
    "    return (doc1, doc2, doc3, doc4, doc5, doc6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing required libraries for preprocessing of documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from collections import deque\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.tokenize import SpaceTokenizer\n",
    "from nltk.tokenize import TweetTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*process_doc()* function reduces a document into list of words after preprocessing it with stemmer and removing stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_doc(doc):\n",
    "    \"\"\"Process doc function.\n",
    "    Input:\n",
    "        doc: a string containing a information\n",
    "    Output:\n",
    "        doc_clean: a list of words containing the processed doc\n",
    "\n",
    "    \"\"\"\n",
    "    stemmer = PorterStemmer()\n",
    "    stopwords_english = stopwords.words('english')\n",
    "    tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True,\n",
    "                               reduce_len=True)\n",
    "    doc_tokens = tokenizer.tokenize(doc)\n",
    "\n",
    "    doc_clean = []\n",
    "    for word in doc_tokens:\n",
    "        if (word not in stopwords_english and  # remove stopwords\n",
    "                word not in string.punctuation):  # remove punctuation\n",
    "            # doc_clean.append(word)\n",
    "            stem_word = stemmer.stem(word)  # stemming word\n",
    "            doc_clean.append(stem_word)\n",
    "\n",
    "    return doc_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function *process_query()* is used to get the list of words in the query after preprocessing on it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_query(query):\n",
    "    stemmer = PorterStemmer()\n",
    "    stopwords_english = stopwords.words('english')\n",
    "    tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True,\n",
    "                               reduce_len=True)\n",
    "    doc_tokens = tokenizer.tokenize(query)\n",
    "\n",
    "    doc_clean = []\n",
    "    for word in doc_tokens:\n",
    "        if (word not in string.punctuation):  # remove punctuation\n",
    "            # doc_clean.append(word)\n",
    "            stem_word = stemmer.stem(word)  # stemming word\n",
    "            doc_clean.append(stem_word)\n",
    "\n",
    "    return doc_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*build_bag()* function is useful to collect list of all word in all documents.\n",
    "\n",
    "* Build bag function returns back the list of words present in the corpus, documents and peprocessed documents\n",
    "* These returned values are then useful to make the dataframe of the corpus "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_bag():\n",
    "    word_bag = []\n",
    "    docs_clean = []\n",
    "    docs = make_doc()\n",
    "    for doc in make_doc():\n",
    "        doc = process_doc(doc)\n",
    "        for word in doc:\n",
    "            if word not in word_bag:\n",
    "                word_bag.append(word)\n",
    "            else:\n",
    "                pass\n",
    "        docs_clean.append(doc)\n",
    "    return word_bag, docs, docs_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*make_data()* function is the driver function for making dataframe out of the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_data():\n",
    "    data = {}\n",
    "    word_bag, docs, docs_clean = build_bag()\n",
    "    for i in range(len(make_doc())):\n",
    "        data[i] = []\n",
    "        for word in word_bag:\n",
    "            if word in docs_clean[i]:\n",
    "                data[i].append(1)\n",
    "            else:\n",
    "                data[i].append(0)\n",
    "    return data, word_bag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, labels = make_data()\n",
    "df = pd.DataFrame(data = data, index = labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*predict()* function resonates with the *boolean_model()* function and solves the **infix** of the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(query, df):\n",
    "    query = process_query(query)\n",
    "    stack = deque()\n",
    "    flag = 0\n",
    "    i = 0\n",
    "    while i < len(query):\n",
    "        word = query[i]\n",
    "        if word in ['not', 'or', 'and']:\n",
    "            if word == 'not':\n",
    "                flag = 1\n",
    "                op1 = list(df.loc[query[i + 1]] > 0)\n",
    "                result = boolean_model(op1, word)\n",
    "                stack.append(result)\n",
    "            else:\n",
    "                op1 = stack.pop()\n",
    "                if query[i + 1] == 'not':\n",
    "                    op2 = boolean_model(list(df.loc[query[i + 2]] > 0), 'not')\n",
    "                    i += 1\n",
    "                else:\n",
    "                    op2 = list(df.loc[query[i + 1]] > 0)\n",
    "                result = boolean_model(op1, word, op2)\n",
    "                stack.append(result)\n",
    "            i += 1\n",
    "        else:\n",
    "            op1 = list(df.loc[word] > 0)\n",
    "            stack.append(op1)\n",
    "        i += 1\n",
    "    output = stack.pop()\n",
    "    result = []\n",
    "    for i in range(len(output)):\n",
    "        if output[i]:\n",
    "            result.append('doc ' + str(i + 1))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*boolean_model()* is the model function to perform all the set operations.\n",
    "\n",
    "### Inputs are \n",
    "* operand 1 = [1,0,0,0,1,0]\n",
    "* operator = and\n",
    "* operand 2 = [1,1,0,0,1,0]\n",
    "\n",
    "### Output is\n",
    "* similar list like operand 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def boolean_model(op1, oper, op2 = []):\n",
    "    print(oper)\n",
    "    if oper == 'not':\n",
    "        op1 = [not i for i in op1]\n",
    "        print(op1)\n",
    "        return op1\n",
    "    elif oper == 'and':\n",
    "        for i in range(len(op1)):\n",
    "            op1[i] = op1[i] and op2[i]\n",
    "    else:\n",
    "        for i in range(len(op1)):\n",
    "            op1[i] = op1[i] or op2[i]\n",
    "    print(op1)\n",
    "    return op1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we get the user input and feed it to the boolean model\n",
    "\n",
    "* we get the result list with boolean values corresponding to the documents where query matches to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "youtube and ms\n",
      "and\n",
      "[False, False, False, False, False, False]\n",
      "Query does not retrieve document\n"
     ]
    }
   ],
   "source": [
    "query = input()\n",
    "result = predict(query, df)\n",
    "if len(result) == 0:\n",
    "    print('Query does not retrieve document')\n",
    "else:\n",
    "    for i in result:\n",
    "        print('{} matches'.format(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Queries which are tested on the dataset\n",
    "\n",
    "* ipl or persistent and not systems\n",
    "* ms and persistent or not systems\n",
    "* ms and balidan\n",
    "* ms or balidan\n",
    "* examination\n",
    "* youtube and ms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}