naive_bayes_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification\n",
    "\n",
    "Using Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from nlp.nb import NBClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading index from stored file (113239 bytes)...\n",
      "Counts:\n",
      "1    7518\n",
      "0    6617\n",
      "Name: label, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10624</th>\n",
       "      <td>Во Франции цементный концерн подозревают в спо...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10193</th>\n",
       "      <td>Полиция Ирландии арестовала двух человек после...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2894</th>\n",
       "      <td>Российские медицинские туристы, которые ездили...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9087</th>\n",
       "      <td>Следователь московского полицейского главка вы...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7888</th>\n",
       "      <td>МЧС предупредило москвичей об ухудшении погоды...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    body  label\n",
       "10624  Во Франции цементный концерн подозревают в спо...      1\n",
       "10193  Полиция Ирландии арестовала двух человек после...      1\n",
       "2894   Российские медицинские туристы, которые ездили...      0\n",
       "9087   Следователь московского полицейского главка вы...      1\n",
       "7888   МЧС предупредило москвичей об ухудшении погоды...      1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "STORAGE_PATH = './storage/nb.model'\n",
    "\n",
    "good_articles = pd.read_table('./articles/good.articles', sep='\\n', names=['body'])\n",
    "good_articles['label'] = 0 # good\n",
    "bad_articles  = pd.read_table('./articles/bad.articles', sep='\\n', names=['body'])\n",
    "bad_articles['label'] = 1 # bad\n",
    "\n",
    "# joining\n",
    "articles = pd.concat([good_articles, bad_articles], ignore_index=True)\n",
    "\n",
    "# persistently shuffling\n",
    "index_path = f\"{STORAGE_PATH}.shuffled_index\"\n",
    "if Path(index_path).is_file():\n",
    "  file_size = os.path.getsize(index_path)\n",
    "  print(f\"Loading index from stored file ({file_size} bytes)...\")\n",
    "  with open(index_path, 'rb') as fp:\n",
    "    shuffled_index = pickle.load(fp)\n",
    "else:\n",
    "  print('Shuffling index for the first time ...')\n",
    "  shuffled_index = np.random.permutation(articles.index)\n",
    "  print('Saving index on disk for further access...')\n",
    "  with open(index_path, 'wb') as fp:\n",
    "    pickle.dump(shuffled_index, fp)\n",
    "  file_size = os.path.getsize(index_path)\n",
    "  print(f\"Done. It took {file_size} bytes on the disk.\")\n",
    "\n",
    "articles = articles.reindex(shuffled_index)\n",
    "\n",
    "print(f\"Counts:\\n{articles['label'].value_counts()}\")\n",
    "articles.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting between train and test\n",
    "\n",
    "Initial dataset was made up roughly by two categories: accidents and else otherwise. This is our best approximation to what we wanna see as bad and good news (cold start). Later on initial dataset is to be replaced by human-decided data from any categories.\n",
    "\n",
    "That is why we use just as little as 2k (15%) articles to initially train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Train dataset:\n",
      "label\n",
      "0     991\n",
      "1    1129\n",
      "dtype: int64\n",
      "(2120, 2)\n",
      "\n",
      "Test dataset:\n",
      "label\n",
      "0    5626\n",
      "1    6389\n",
      "dtype: int64\n",
      "(12015, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>body</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8298</th>\n",
       "      <td>Шесть детей госпитализировали из петербургской...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5947</th>\n",
       "      <td>Французский модный дом Balenciaga представил р...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4395</th>\n",
       "      <td>В Ненецком автономном округе частные детсады, ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10142</th>\n",
       "      <td>Число жертв взрыва в дипломатическом квартале ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11498</th>\n",
       "      <td>На юго-востоке Москвы нашли еще один снаряд. П...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    body  label\n",
       "8298   Шесть детей госпитализировали из петербургской...      1\n",
       "5947   Французский модный дом Balenciaga представил р...      0\n",
       "4395   В Ненецком автономном округе частные детсады, ...      0\n",
       "10142  Число жертв взрыва в дипломатическом квартале ...      1\n",
       "11498  На юго-востоке Москвы нашли еще один снаряд. П...      1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data, test_data = train_test_split(articles, train_size = 0.15)\n",
    "\n",
    "print('\\nTrain dataset:')\n",
    "print(train_data.groupby('label').size())\n",
    "print(train_data.shape)\n",
    "\n",
    "print('\\nTest dataset:')\n",
    "print(test_data.groupby('label').size())\n",
    "print(test_data.shape)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Removed highly correlated words: ,, нача, не, к, num, британск, с, юбк, труд, средств, руководител, нам, привод, глав, изначальн, работ, чечн, все, рассказа, церкв, чувств, том, продолж, установк, измерен, тысяч, крайн, осуществлен, устройств, проход, немн, кипр, отел, мин, через, нег, волк, занят, демонстрирова, отказа, зат, нью, мог, кров, разбира, проведен, заработн, владивосток, судьб, граждан, сбыва, остальн, помоч, забрасыва, образова, добра, забира, пас, пенсионер, девушк, шок, реконструкц, впоследств, замет, исполнен, попрос, огранич, сиден, воева, первичн, наруша, пугачев, проб, бумажн, ирин, паспорт, барс, пороча, нац, приостанов, заперт, стремительн, ура, самостоятельн, телефон, действова, великобритан, матер, утвержд, фрг, выкат, казанск, высажива, ялт, юстиц, генера, кат, настаива, тунисск, штраф, концерт, буквальн, снежн, размест, горн, принадлежат, денежн, эфиопск, нормальн, предел, ушл, монсон, пояс, жан, мот, захотел, заказа, принадлежа, электрон, крут, противоположн, усилен, поставк, компьютер, демонстрац, воспитан, стекл, телеведущ, джан, надея, налож, кондитерск, ужин, персона, неудач, бетон, разговарива, коммунальн, авиалайнер, адлер, переписк, белк, невад, кольцев, инженерн, зачислен, мемориа, почв, крепк, раздач, злоупотреблен, отопительн, собеседник, стриптиз, расов, салют, магистрал, монреал.\n",
      "Removed highly correlated words: ,, нача, не, к, num, британск, с, юбк, труд, средств, руководител, нам, привод, глав, изначальн, работ, чечн, все, рассказа, церкв, чувств, том, продолж, установк, измерен, тысяч, крайн, осуществлен, устройств, проход, немн, кипр, отел, мин, через, нег, волк, занят, демонстрирова, отказа, зат, нью, мог, кров, разбира, проведен, заработн, владивосток, судьб, граждан, сбыва, остальн, помоч, забрасыва, образова, добра, забира, пас, пенсионер, девушк, шок, реконструкц, впоследств, замет, исполнен, попрос, огранич, сиден, воева, первичн, наруша, пугачев, проб, бумажн, ирин, паспорт, барс, пороча, нац, приостанов, заперт, стремительн, ура, самостоятельн, телефон, действова, великобритан, матер, утвержд, фрг, выкат, казанск, высажива, ялт, юстиц, генера, кат, настаива, тунисск, штраф, концерт, буквальн, снежн, размест, горн, принадлежат, денежн, эфиопск, нормальн, предел, ушл, монсон, пояс, жан, мот, захотел, заказа, принадлежа, электрон, крут, противоположн, усилен, поставк, компьютер, демонстрац, воспитан, стекл, телеведущ, джан, надея, налож, кондитерск, ужин, персона, неудач, бетон, разговарива, коммунальн, авиалайнер, адлер, переписк, белк, невад, кольцев, инженерн, зачислен, мемориа, почв, крепк, раздач, злоупотреблен, отопительн, собеседник, стриптиз, расов, салют, магистрал, монреал.\n",
      "Tokenized 2120 documents\n",
      "Built IDF\n",
      "Removing word . having zero idf\n"
     ]
    }
   ],
   "source": [
    "nbcs = []\n",
    "for strategy in ['tf', 'tfidf']:\n",
    "  nbc = NBClassifier(strategy = strategy)\n",
    "  nbc.train(train_data)\n",
    "  nbcs.append(nbc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_by_strategy = [[nbc.predict(article).label for article in test_data['body']] for nbc in nbcs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.97      0.98      0.98      5626\n",
      "          1       0.98      0.97      0.98      6389\n",
      "\n",
      "avg / total       0.98      0.98      0.98     12015\n",
      "\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.98      0.97      0.98      5626\n",
      "          1       0.98      0.98      0.98      6389\n",
      "\n",
      "avg / total       0.98      0.98      0.98     12015\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for predictions in predictions_by_strategy:\n",
    "  report = metrics.classification_report(test_data['label'], predictions)\n",
    "  print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Naive Bayes makes accurate predictions trained just on a few thousand of documents. It performs well comparing to neural networks, in particular to deep ones, which are a way more greedy for data and require more time to train.\n",
    "\n",
    "We used two ways of representing probability of a word given the class:\n",
    "  - using **Term Frequency**, where `P(w|C) = TF(w|C) / sum_for_words_in_C( TF(w) )`\n",
    "  - and **TFIDF**, where `P(w|C) = TF(w|C) * IDF(w) / sum_for_words_in_C( TF(w) * IDF(w) )`\n",
    "  \n",
    "We also added Laplace smoothing normalized by the average document length of a class (_not shown in the formula above_).\n",
    "\n",
    "Both gave us approximately the same precision, recall and f1-scores.\n",
    "\n",
    "However, for both scenarios a considerable improvement was reached by utilizing an advanced definition of **Term Frequency**: raw count of words in a document adjusted by document length. It can be attributed to greatly-varied document lengths in the dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}