TFIDF Features with NB, LR, LSVM, and RF Classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oGe6H--LKgwp"
   },
   "source": [
    "## Experimenting with an Naive Bayes, Logistic Regression, Linear SVM, and RandomForest Classifiers:\n",
    "\n",
    "* **NB**: Naive Bayes, BernoulliNB implements the naive Bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions\n",
    "* **LR**:Logistic Regression (aka logit, MaxEnt) classifier.\n",
    "* **LSVM**:Linear Support Vector Classification.\n",
    "* **RF**  : A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_X1dJopWKBe2"
   },
   "outputs": [],
   "source": [
    "import os \n",
    "datasets = [\"testdata.manual.2009.06.14.csv\", \"training.1600000.processed.noemoticon.csv\"]\n",
    "train_path = os.path.join(\"dataset\", datasets[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f7aiONiPDpEA"
   },
   "source": [
    "**Loading Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wF9kDXGtKVtC"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from tqdm import trange\n",
    "from dataloader import DataLoader\n",
    "\n",
    "data_loader = DataLoader()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 111
    },
    "colab_type": "code",
    "id": "tFtv0FOwKZ8X",
    "outputId": "58e949ac-d0a8-469a-f25b-dfdef035266e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>ID</th>\n",
       "      <th>Date</th>\n",
       "      <th>Query</th>\n",
       "      <th>UserID</th>\n",
       "      <th>Tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810369</td>\n",
       "      <td>Mon Apr 06 22:19:45 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810672</td>\n",
       "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sentiment  ...                                              Tweet\n",
       "0          0  ...  @switchfoot http://twitpic.com/2y1zl - Awww, t...\n",
       "1          0  ...  is upset that he can't update his Facebook by ...\n",
       "\n",
       "[2 rows x 6 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data_loader.read_df(train_path, \n",
    "                           df_type='csv', encoding='latin-1',\n",
    "                           names=[\"Sentiment\", \"ID\", \"Date\", \"Query\",\"UserID\",\"Tweet\"])\n",
    "data.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "00mncymVEF-P"
   },
   "source": [
    "**preprocessing using texthero**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 145
    },
    "colab_type": "code",
    "id": "bFVkO-9FLIBL",
    "outputId": "2fa5ce01-e044-4b0f-a5b5-99acf789d100"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>ID</th>\n",
       "      <th>Date</th>\n",
       "      <th>Query</th>\n",
       "      <th>UserID</th>\n",
       "      <th>Tweet</th>\n",
       "      <th>CleanTweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810369</td>\n",
       "      <td>Mon Apr 06 22:19:45 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "      <td>switchfoot http twitpic com 2y1zl awww bummer ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810672</td>\n",
       "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "      <td>upset update facebook texting might cry result...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sentiment  ...                                         CleanTweet\n",
       "0          0  ...  switchfoot http twitpic com 2y1zl awww bummer ...\n",
       "1          0  ...  upset update facebook texting might cry result...\n",
       "\n",
       "[2 rows x 7 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import texthero as hero\n",
    "\n",
    "data['CleanTweet'] = data['Tweet'].pipe(hero.clean)\n",
    "\n",
    "data.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_elxGoYpZepe"
   },
   "source": [
    "**Extracting top words for TF-IDF representation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "u97ekW5ta02g",
    "outputId": "441bb9a8-9bf3-4349-e013-9a86136f6e2f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ft:5, vocab lenght:51758\n"
     ]
    }
   ],
   "source": [
    "def get_vocabs(pos_words_dic, neg_words_dic, ft):\n",
    "    vocabs = []\n",
    "    for word,freq in pos_words_dic.items():\n",
    "        if freq > ft and word not in vocabs:\n",
    "            vocabs.append(word)\n",
    "\n",
    "    for word,freq in neg_words_dic.items():\n",
    "        if freq > ft and word not in vocabs:\n",
    "            vocabs.append(word)\n",
    "    return vocabs\n",
    "    \n",
    "ft = 5\n",
    "top_pos_words = hero.top_words(data[data['Sentiment'] == 4]['CleanTweet'])\n",
    "top_neg_words = hero.top_words(data[data['Sentiment'] == 0]['CleanTweet'])\n",
    "vocabs5 = get_vocabs(top_pos_words.to_dict(), top_neg_words.to_dict(), ft=ft)\n",
    "print( \"ft:{}, vocab lenght:{}\".format(ft, len(vocabs5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1kZUR8dRqGTH"
   },
   "source": [
    "**Train test split**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9DPS-no1gJOb"
   },
   "outputs": [],
   "source": [
    "def transform_labels(label):\n",
    "    if label==4:\n",
    "        return 1\n",
    "    return label\n",
    "\n",
    "data['Sentiment'] = data['Sentiment'].apply(lambda x:transform_labels(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "KCi0IbftgWkF",
    "outputId": "78173c8b-c0f3-4fb4-f8f4-402b2ed8e702"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 1120000\n",
      "Test size: 480000\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(data['CleanTweet'].tolist(), \n",
    "                                                    data['Sentiment'].tolist(),\n",
    "                                                    test_size=0.3, random_state=40)\n",
    "\n",
    "print(\"Train size:\", len(x_train))\n",
    "print(\"Test size:\", len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f2hXfUwiLC5H"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from model import ModelPipeline\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "09CfpnXvgm8i"
   },
   "source": [
    "## TFIDFNB Model\n",
    "TF-IDF representation with Naive Bays model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "znIviv3CbEtz"
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import BernoulliNB\n",
    "\n",
    "model=ModelPipeline(estimator=BernoulliNB(),\n",
    "                    transformer=TfidfVectorizer(vocabulary=vocabs5) )\n",
    "\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "NO1U08klgItt",
    "outputId": "c62fae90-a945-4c47-b769-33ad7b36ae90"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.77      0.77      0.77    239943\n",
      "           1       0.77      0.77      0.77    240057\n",
      "\n",
      "    accuracy                           0.77    480000\n",
      "   macro avg       0.77      0.77      0.77    480000\n",
      "weighted avg       0.77      0.77      0.77    480000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict(x_test)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KhBCD1JgIKU_"
   },
   "source": [
    "## TFIDFLR Model\n",
    "TF-IDF representation with LogisticRegression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3Lj3DAb6IOXI"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model=ModelPipeline(estimator=LogisticRegression(C=0.1, max_iter=200),\n",
    "                    transformer=TfidfVectorizer(vocabulary=vocabs5) )\n",
    "\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "0_8yxXpRIRLA",
    "outputId": "d6a17875-515d-49b6-ecbb-6b4d50bc08c5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.79      0.75      0.77    239943\n",
      "           1       0.76      0.80      0.78    240057\n",
      "\n",
      "    accuracy                           0.78    480000\n",
      "   macro avg       0.78      0.78      0.78    480000\n",
      "weighted avg       0.78      0.78      0.78    480000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict(x_test)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "smE6Lg7OIbLR"
   },
   "source": [
    "## TFIDFLSVM Model\n",
    "TF-IDF representation with Linear SVM model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U7N-kjj7IHwF"
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "model=ModelPipeline(estimator=LinearSVC(C=1),\n",
    "                    transformer=TfidfVectorizer(vocabulary=vocabs5) )\n",
    "\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "EfBV-jIPqymQ",
    "outputId": "d7ae617b-9031-4347-9e4b-a9f630972873"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.79      0.76      0.77    239943\n",
      "           1       0.77      0.80      0.78    240057\n",
      "\n",
      "    accuracy                           0.78    480000\n",
      "   macro avg       0.78      0.78      0.78    480000\n",
      "weighted avg       0.78      0.78      0.78    480000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict(x_test)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8Y-DwA5JKZ1U"
   },
   "source": [
    "## TFIDFRF Model\n",
    "TF-IDF representation with RandomForest model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yjkk18oOJZz_"
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model=ModelPipeline(estimator=RandomForestClassifier(max_depth=100),\n",
    "                    transformer=TfidfVectorizer(vocabulary=vocabs5) )\n",
    "\n",
    "model.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "colab_type": "code",
    "id": "udoz3sWiKftR",
    "outputId": "39bc9978-6af5-4c3e-924d-7bcf2fa4390a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.70      0.74    239943\n",
      "           1       0.73      0.80      0.76    240057\n",
      "\n",
      "    accuracy                           0.75    480000\n",
      "   macro avg       0.75      0.75      0.75    480000\n",
      "weighted avg       0.75      0.75      0.75    480000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict(x_test)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Dd0jMlIIN3LV"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "collapsed_sections": [],
   "name": "TFIDF Features with NB, LR, LSVM, and RF Models .ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}