Using ML algorithms.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import nltk\n",
    "nltk.download() # we need corpora package\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intro\n",
    "\n",
    "The sklearn.datasets.fetch_20newsgroups function is a data fetching / caching functions that downloads the data archive from the original 20 newsgroups website, extracts the archive contents in the ~/scikit_learn_data/20news_home folder and calls the sklearn.datasets.load_files on either the training or testing set folder, or both of them.\n",
    "\n",
    "Also lets check the first article from the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alt.atheism',\n",
      " 'comp.graphics',\n",
      " 'comp.os.ms-windows.misc',\n",
      " 'comp.sys.ibm.pc.hardware',\n",
      " 'comp.sys.mac.hardware',\n",
      " 'comp.windows.x',\n",
      " 'misc.forsale',\n",
      " 'rec.autos',\n",
      " 'rec.motorcycles',\n",
      " 'rec.sport.baseball',\n",
      " 'rec.sport.hockey',\n",
      " 'sci.crypt',\n",
      " 'sci.electronics',\n",
      " 'sci.med',\n",
      " 'sci.space',\n",
      " 'soc.religion.christian',\n",
      " 'talk.politics.guns',\n",
      " 'talk.politics.mideast',\n",
      " 'talk.politics.misc',\n",
      " 'talk.religion.misc']\n",
      "\n",
      "First article is:\n",
      "From: lerxst@wam.umd.edu (where's my thing)\n",
      "Subject: WHAT car is this!?\n",
      "Nntp-Posting-Host: rac3.wam.umd.edu\n"
     ]
    }
   ],
   "source": [
    "twenty_train = fetch_20newsgroups(subset='train', shuffle=True) #there is about 14MB\n",
    "twenty_test = fetch_20newsgroups(subset='test', shuffle=True)\n",
    "pprint(list(twenty_train.target_names))\n",
    "\n",
    "print(\"\\nFirst article is:\\n\" + \"\\n\".join(twenty_train.data[0].split(\"\\n\")[:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting features from text files\n",
    "\n",
    "In order to run machine learning algorithms we need to convert the text files into numerical feature vectors. I will use BagOfWords model. Briefly, we segment each text file into words (for English splitting by space), and count number of times each word occurs in each document and finally assign each word an integer id. Each unique word in our dictionary will correspond to a feature (descriptive feature).\n",
    "\n",
    "Scikit-learn has a high level component which will create feature vectors for us ‘CountVectorizer’.\n",
    "\n",
    "Here by doing ‘count_vect.fit_transform(twenty_train.data)’, we are learning the vocabulary dictionary and it returns a Document-Term matrix. [n_samples, n_features]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11314, 130107)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect = CountVectorizer()\n",
    "X_train_counts = count_vect.fit_transform(twenty_train.data)\n",
    "X_train_counts.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TF**: Just counting the number of words in each document has 1 issue: it will give more weightage to longer documents than shorter documents. To avoid this, we can use frequency (TF - Term Frequencies) i.e. #count(word) / #Total words, in each document.\n",
    "\n",
    "**TF-IDF**: Finally, we can even reduce the weightage of more common words like (the, is, an etc.) which occurs in all document. This is called as TF-IDF i.e Term Frequency times inverse document frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11314, 130107)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_transformer = TfidfTransformer()\n",
    "X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
    "X_train_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bias\n",
    "\n",
    "You can easily build a NBclassifier in scikit using below 2 lines of code: (note - there are many variants of NB, but discussion about them is out of scope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline\n",
    "\n",
    "Building a pipeline. We can write less code and do all of the above, by building a pipeline as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')),\n",
    "                     ('tfidf', TfidfTransformer()),\n",
    "                     ('clf', MultinomialNB())])\n",
    "\n",
    "text_clf = text_clf.fit(twenty_train.data, twenty_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing model #1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.81691449814126393"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted = text_clf.predict(twenty_test.data)\n",
    "np.mean(predicted == twenty_test.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grid Search\n",
    "\n",
    "Almost all the classifiers will have various parameters which can be tuned to obtain optimal performance. Scikit gives an extremely useful tool ‘GridSearchCV’.\n",
    "\n",
    "Here, we are creating a list of parameters for which we would like to do performance tuning. All the parameters name start with the classifier name (remember the arbitrary name we gave). E.g. vect__ngram_range; here we are telling to use unigram and bigrams and choose the one which is optimal.\n",
    "\n",
    "Next, we create an instance of the grid search by passing the classifier, parameters and n_jobs=-1 which tells to use multiple cores from user machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters = {'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'tfidf__use_idf': (True, False),\n",
    "              'clf__alpha': (1e-2, 1e-3),}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=-1)\n",
    "gs_clf = gs_clf.fit(twenty_train.data, twenty_train.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clf__alpha': 0.01, 'tfidf__use_idf': False, 'vect__ngram_range': (1, 1)}"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_clf.best_score_\n",
    "gs_clf.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Including NTLK\n",
    "\n",
    "Things learned from previous notebooks now will get handy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer(\"english\", ignore_stopwords=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class StemmedCountVectorizer(CountVectorizer):\n",
    "    def build_analyzer(self):\n",
    "        analyzer = super(StemmedCountVectorizer, self).build_analyzer()\n",
    "        return lambda doc: ([stemmer.stem(w) for w in analyzer(doc)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stemmed_count_vect = StemmedCountVectorizer(stop_words='english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text_mnb_stemmed = Pipeline([('vect', stemmed_count_vect), ('tfidf', TfidfTransformer()), \n",
    "                             ('mnb', MultinomialNB(fit_prior=False))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_mnb_stemmed = text_mnb_stemmed.fit(twenty_train.data, twenty_train.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predicted_mnb_stemmed = text_mnb_stemmed.predict(twenty_test.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.81678173127987252"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(predicted_mnb_stemmed == twenty_test.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('second_text.txt', 'r') as f:\n",
    "    new_document=f.read().replace('\\n', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'sci.electronics'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_own_doc = text_mnb_stemmed.predict([new_document])\n",
    "print(predicted_own_doc)\n",
    "twenty_train.target_names[predicted_own_doc[0]+1]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}