text_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DnLV1HUefFtW"
   },
   "source": [
    "# Text Features In CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0UAHpnD8fFtZ"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/events/2020_06_04_catboost_tutorial/text_features.ipynb)\n",
    "\n",
    "**Set GPU as hardware accelerator**\n",
    "\n",
    "First of all, you need to select GPU as hardware accelerator. There are two simple steps to do so:\n",
    "Step 1. Navigate to **Runtime** menu and select **Change runtime type**\n",
    "Step 2. Choose **GPU** as hardware accelerator.\n",
    "That's all!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9FM0IRyi8NOw"
   },
   "source": [
    "Let's install CatBoost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 361
    },
    "colab_type": "code",
    "id": "TpJdgt63fSOv",
    "outputId": "d62a776e-f741-4192-b919-91903ea0441b"
   },
   "outputs": [],
   "source": [
    "!pip install catboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "viF18QJqfFtd"
   },
   "source": [
    "In this tutorial we will use dataset **IMDB** from [Kaggle](https://www.kaggle.com) competition for our experiments. Data can be downloaded [here](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "MNC1tP0UfFtd",
    "outputId": "2c0abe55-df9c-4a0f-daa4-dc8c8d858f63"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.24.2\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=4)\n",
    "\n",
    "import catboost\n",
    "print(catboost.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OkexL1k7fFti"
   },
   "source": [
    "## Preparing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-11-17 21:11:35--  https://transfersh.com/ou7jB/imdb.csv\n",
      "Resolving transfersh.com (transfersh.com)... 64:ff9b::6bb2:6ca6\n",
      "Connecting to transfersh.com (transfersh.com)|64:ff9b::6bb2:6ca6|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 66212309 (63M) [text/csv]\n",
      "Saving to: ‘imdb.csv’\n",
      "\n",
      "imdb.csv            100%[===================>]  63.14M  16.4MB/s    in 4.7s    \n",
      "\n",
      "2020-11-17 21:11:41 (13.4 MB/s) - ‘imdb.csv’ saved [66212309/66212309]\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>One of the other reviewers has mentioned that ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A wonderful little production. &lt;br /&gt;&lt;br /&gt;The...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I thought this was a wonderful way to spend ti...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Basically there's a family where a little boy ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Petter Mattei's \"Love in the Time of Money\" is...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  label\n",
       "0  One of the other reviewers has mentioned that ...      1\n",
       "1  A wonderful little production. <br /><br />The...      1\n",
       "2  I thought this was a wonderful way to spend ti...      1\n",
       "3  Basically there's a family where a little boy ...      0\n",
       "4  Petter Mattei's \"Love in the Time of Money\" is...      1"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!wget https://transfersh.com/ou7jB/imdb.csv -O imdb.csv\n",
    "df = pd.read_csv('imdb.csv')\n",
    "df['label'] = (df['sentiment'] == 'positive').astype(int)\n",
    "df.drop(['sentiment'], axis=1, inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nikitxskv/.local/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train dataset shape: (40000, 1)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from catboost import Pool\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_df, test_df = train_test_split(df, train_size=0.8, random_state=0)\n",
    "y_train, X_train = train_df['label'], train_df.drop(['label'], axis=1)\n",
    "y_test, X_test = test_df['label'], test_df.drop(['label'], axis=1)\n",
    "\n",
    "train_pool = Pool(data=X_train, label=y_train, text_features=['review'])\n",
    "test_pool = Pool(data=X_test, label=y_test, text_features=['review'])\n",
    "\n",
    "print('Train dataset shape: {}\\n'.format(train_pool.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.to_csv('imdb_train.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "id": "VTi3eN58fFt6",
    "outputId": "e694fed2-1341-45a3-c799-334b32fbc01e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.8742963\tbest: 0.8742963 (0)\ttotal: 224ms\tremaining: 3m 43s\n",
      "100:\ttest: 0.9385780\tbest: 0.9385780 (100)\ttotal: 23.5s\tremaining: 3m 28s\n",
      "200:\ttest: 0.9460053\tbest: 0.9460053 (200)\ttotal: 46.2s\tremaining: 3m 3s\n",
      "300:\ttest: 0.9505063\tbest: 0.9505063 (300)\ttotal: 1m 9s\tremaining: 2m 41s\n",
      "400:\ttest: 0.9529535\tbest: 0.9529535 (400)\ttotal: 1m 31s\tremaining: 2m 17s\n",
      "500:\ttest: 0.9546771\tbest: 0.9546771 (500)\ttotal: 1m 56s\tremaining: 1m 55s\n",
      "600:\ttest: 0.9556983\tbest: 0.9556983 (600)\ttotal: 2m 21s\tremaining: 1m 33s\n",
      "700:\ttest: 0.9564580\tbest: 0.9564580 (700)\ttotal: 2m 45s\tremaining: 1m 10s\n",
      "800:\ttest: 0.9570723\tbest: 0.9570723 (800)\ttotal: 3m 7s\tremaining: 46.6s\n",
      "900:\ttest: 0.9575270\tbest: 0.9575386 (892)\ttotal: 3m 29s\tremaining: 23.1s\n",
      "999:\ttest: 0.9579518\tbest: 0.9579518 (999)\ttotal: 3m 51s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.9579518196\n",
      "bestIteration = 999\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "\n",
    "def fit_model(train_pool, test_pool, **kwargs):\n",
    "    model = CatBoostClassifier(\n",
    "        iterations=1000,\n",
    "        learning_rate=0.05,\n",
    "        eval_metric='AUC',\n",
    "#         task_type='GPU',\n",
    "        **kwargs\n",
    "    )\n",
    "\n",
    "    return model.fit(\n",
    "        train_pool,\n",
    "        eval_set=test_pool,\n",
    "        verbose=100,\n",
    "    )\n",
    "\n",
    "model = fit_model(train_pool, test_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IiHpTGfbfFuV"
   },
   "source": [
    "## How it works?\n",
    "\n",
    "1. **Text Tokenization**\n",
    "2. **Dictionary Creation**\n",
    "3. **Feature Calculation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MszSnbqH8NR3"
   },
   "source": [
    "## Text Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mOBGuexjb8tr"
   },
   "source": [
    "Usually we get our text as a sequence of Unicode symbols. So, if the task isn't a DNA classification we don't need such granularity, moreover, we need to extract more complicated entities, e.g. words. The process of extraction tokens -- words, numbers, punctuation symbols or special symbols which defines emoji from a sequence is called **tokenization**.<br>\n",
    "\n",
    "Tokenization is the first part of text preprocessing in CatBoost and performed as a simple splitting a sequence on a string pattern (e.g. space)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NAeELULufFuV"
   },
   "outputs": [],
   "source": [
    "text_small = [\n",
    "    \"Cats are so cute :)\",\n",
    "    \"Mouse scare...\",\n",
    "    \"The cat defeated the mouse\",\n",
    "    \"Cute: Mice gather an army!\",\n",
    "    \"Army of mice defeated the cat :(\",\n",
    "    \"Cat offers peace\",\n",
    "    \"Cat is scared :(\",\n",
    "    \"Cat and mouse live in peace :)\"\n",
    "]\n",
    "\n",
    "target_small = [1, 0, 1, 1, 0, 1, 0, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "E21CQ8ocfFuX",
    "outputId": "f78b995b-29fc-41c9-b28c-b3adee167ba7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Cats', 'are', 'so', 'cute', ':)'],\n",
       " ['Mouse', 'scare...'],\n",
       " ['The', 'cat', 'defeated', 'the', 'mouse'],\n",
       " ['Cute:', 'Mice', 'gather', 'an', 'army!'],\n",
       " ['Army', 'of', 'mice', 'defeated', 'the', 'cat', ':('],\n",
       " ['Cat', 'offers', 'peace'],\n",
       " ['Cat', 'is', 'scared', ':('],\n",
       " ['Cat', 'and', 'mouse', 'live', 'in', 'peace', ':)']]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from catboost.text_processing import Tokenizer\n",
    "\n",
    "simple_tokenizer = Tokenizer()\n",
    "\n",
    "def tokenize_texts(texts):\n",
    "    return [simple_tokenizer.tokenize(text) for text in texts]\n",
    "\n",
    "simple_tokenized_text = tokenize_texts(text_small)\n",
    "simple_tokenized_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ChZQ5cpJfFuZ"
   },
   "source": [
    "### More preprocessing!\n",
    "\n",
    "Lets take a closer look on the tokenization result of small text example -- the tokens contains a lot of mistakes:\n",
    "\n",
    "1. They are glued with punctuation 'Cute:', 'army!', 'skare...'.\n",
    "2. The words 'Cat' and 'cat', 'Mice' and 'mice' seems to have same meaning, perhaps they should be the same tokens.\n",
    "3. The same problem with tokens 'are'/'is' -- they are inflected forms of same token 'be'.\n",
    "\n",
    "**Punctuation handling**, **lowercasing**, and **lemmatization** processes help to overcome these problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qaoTjEmR8NSM"
   },
   "source": [
    "### Punctuation handling and lowercasing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "6cPpYpmtfFuZ",
    "outputId": "2bc7abef-5828-43af-d588-48edb490eed9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['cats', 'are', 'so', 'cute'],\n",
       " ['mouse', 'scare'],\n",
       " ['the', 'cat', 'defeated', 'the', 'mouse'],\n",
       " ['cute', 'mice', 'gather', 'an', 'army'],\n",
       " ['army', 'of', 'mice', 'defeated', 'the', 'cat'],\n",
       " ['cat', 'offers', 'peace'],\n",
       " ['cat', 'is', 'scared'],\n",
       " ['cat', 'and', 'mouse', 'live', 'in', 'peace']]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = Tokenizer(\n",
    "    lowercasing=True,\n",
    "    separator_type='BySense',\n",
    "    token_types=['Word', 'Number']\n",
    ")\n",
    "\n",
    "tokenized_text = [tokenizer.tokenize(text) for text in text_small]\n",
    "tokenized_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JDhBkZzJfFua"
   },
   "source": [
    "### Removing stop words\n",
    "\n",
    "**Stop words** - the words that are considered to be uninformative in this task, e.g. function words such as *the, is, at, which, on*.\n",
    "Usually stop words are removed during text preprocessing to reduce the amount of information that is considered for further algorithms.\n",
    "Stop words are collected manually (in dictionary form) or automatically, for example taking the most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "d1MYzKgTfFub",
    "outputId": "865f655e-0cb9-4626-9d40-e459b9487b0f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['cats', 'so', 'cute'],\n",
       " ['mouse', 'scare'],\n",
       " ['cat', 'defeated', 'mouse'],\n",
       " ['cute', 'mice', 'gather', 'army'],\n",
       " ['army', 'mice', 'defeated', 'cat'],\n",
       " ['cat', 'offers', 'peace'],\n",
       " ['cat', 'scared'],\n",
       " ['cat', 'mouse', 'live', 'peace']]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stop_words = set(('be', 'is', 'are', 'the', 'an', 'of', 'and', 'in'))\n",
    "\n",
    "def filter_stop_words(tokens):\n",
    "    return list(filter(lambda x: x not in stop_words, tokens))\n",
    "    \n",
    "tokenized_text_no_stop = [filter_stop_words(tokens) for tokens in tokenized_text]\n",
    "tokenized_text_no_stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vxofPVc1fFuc"
   },
   "source": [
    "### Lemmatization\n",
    "\n",
    "Lemma (Wikipedia) -- is the canonical form, dictionary form, or citation form of a set of words.<br>\n",
    "For example, the lemma \"go\" represents the inflected forms \"go\", \"goes\", \"going\", \"went\", and \"gone\".<br>\n",
    "The process of convertation word to its lemma called **lemmatization**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "id": "HWrijpMGfFud",
    "outputId": "1b6b8015-8cf9-47c5-89cf-5d5fc8b5f794"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     /home/nikitxskv/.local/lib/python3.5/site-\n",
      "[nltk_data]     packages/nltk/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk_data_path = os.path.join(os.path.dirname(nltk.__file__), 'nltk_data')\n",
    "nltk.data.path.append(nltk_data_path)\n",
    "nltk.download('wordnet', nltk_data_path)\n",
    "\n",
    "lemmatizer = nltk.stem.WordNetLemmatizer()\n",
    "\n",
    "def lemmatize_tokens_nltk(tokens):\n",
    "    return list(map(lambda t: lemmatizer.lemmatize(t), tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "XfyhV9ONfFuf",
    "outputId": "4b0568c9-3bb8-483a-8f86-dd358c6fd2c5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['cat', 'so', 'cute'],\n",
       " ['mouse', 'scare'],\n",
       " ['cat', 'defeated', 'mouse'],\n",
       " ['cute', 'mouse', 'gather', 'army'],\n",
       " ['army', 'mouse', 'defeated', 'cat'],\n",
       " ['cat', 'offer', 'peace'],\n",
       " ['cat', 'scared'],\n",
       " ['cat', 'mouse', 'live', 'peace']]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_small_lemmatized_nltk = [lemmatize_tokens_nltk(tokens) for tokens in tokenized_text_no_stop]\n",
    "text_small_lemmatized_nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y63KVna4fFui"
   },
   "source": [
    "Now words with same meaning represented by the same token, tokens are not glued with punctuation.\n",
    "\n",
    "<span style=\"color:red\">Be carefull.</span> You should verify for your own task:<br>\n",
    "Is it realy necessary to remove punctuation, lowercasing sentences or performing a lemmatization and/or by word tokenization?<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qFWoSX-kfFui"
   },
   "source": [
    "### Let's check up accuracy with new text preprocessing\n",
    "\n",
    "Since CatBoost doesn't perform spacing punctuation, lowercasing letters and lemmatization, we need to preprocess text manually and then pass it to learning algorithm.\n",
    "\n",
    "Since the natural text features is only synopsis and review, we will preprocess only them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "colab_type": "code",
    "id": "ZHL3x7NwfFuj",
    "outputId": "85135452-02ea-4644-882d-726fcc568605"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 49s, sys: 652 ms, total: 3min 50s\n",
      "Wall time: 3min 51s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "def preprocess_data(X):\n",
    "    X_preprocessed = X.copy()\n",
    "    X_preprocessed['review'] = X['review'].apply(lambda x: ' '.join(lemmatize_tokens_nltk(tokenizer.tokenize(x))))\n",
    "    return X_preprocessed\n",
    "\n",
    "X_preprocessed_train = preprocess_data(X_train)\n",
    "X_preprocessed_test = preprocess_data(X_test)\n",
    "\n",
    "train_processed_pool = Pool(\n",
    "    X_preprocessed_train, y_train, \n",
    "    text_features=['review'],\n",
    ")\n",
    "\n",
    "test_processed_pool = Pool(\n",
    "    X_preprocessed_test, y_test, \n",
    "    text_features=['review'],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "id": "0jJJSrFJfFuk",
    "outputId": "6baeef42-d430-4793-fc33-556095416a9b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.8772679\tbest: 0.8772679 (0)\ttotal: 114ms\tremaining: 1m 54s\n",
      "100:\ttest: 0.9422074\tbest: 0.9422074 (100)\ttotal: 12.2s\tremaining: 1m 48s\n",
      "200:\ttest: 0.9513095\tbest: 0.9513095 (200)\ttotal: 24.2s\tremaining: 1m 36s\n",
      "300:\ttest: 0.9566328\tbest: 0.9566328 (300)\ttotal: 36.5s\tremaining: 1m 24s\n",
      "400:\ttest: 0.9593129\tbest: 0.9593129 (400)\ttotal: 48.9s\tremaining: 1m 13s\n",
      "500:\ttest: 0.9610368\tbest: 0.9610368 (500)\ttotal: 1m 1s\tremaining: 1m\n",
      "600:\ttest: 0.9622586\tbest: 0.9622586 (600)\ttotal: 1m 13s\tremaining: 48.6s\n",
      "700:\ttest: 0.9631853\tbest: 0.9631853 (700)\ttotal: 1m 25s\tremaining: 36.5s\n",
      "800:\ttest: 0.9637900\tbest: 0.9637900 (800)\ttotal: 1m 37s\tremaining: 24.2s\n",
      "900:\ttest: 0.9642351\tbest: 0.9642422 (898)\ttotal: 1m 49s\tremaining: 12s\n",
      "999:\ttest: 0.9646141\tbest: 0.9646180 (998)\ttotal: 2m 1s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.9646179863\n",
      "bestIteration = 998\n",
      "\n",
      "Shrink model to first 999 iterations.\n"
     ]
    }
   ],
   "source": [
    "model_on_processed_data = fit_model(train_processed_pool, test_processed_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "AXDdPAgyfFum",
    "outputId": "61e26e81-b858-4675-ab58-aaf3384428ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9579518196391623 vs 0.9646179862813278 (+0.70%)\n"
     ]
    }
   ],
   "source": [
    "def print_score_diff(first_model, second_model):\n",
    "    first_accuracy = first_model.best_score_['validation']['AUC']\n",
    "    second_accuracy = second_model.best_score_['validation']['AUC']\n",
    "\n",
    "    gap = (second_accuracy - first_accuracy) / first_accuracy * 100\n",
    "\n",
    "    print('{} vs {} ({:+.2f}%)'.format(first_accuracy, second_accuracy, gap))\n",
    "    \n",
    "print_score_diff(model, model_on_processed_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CJr7fXN7fFun"
   },
   "source": [
    "## Dictionary Creation\n",
    "\n",
    "After the first stage, preprocessing of text and tokenization, the second stage starts. The second stage uses the prepared text to select a set of units, which will be used for building new numerical features.\n",
    "\n",
    "A set of selected units is called dictionary. It might contain words, word bigramms, or character n-gramms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D6H1MXf9fFuo"
   },
   "outputs": [],
   "source": [
    "from catboost.text_processing import Dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Rn402k78fFuq"
   },
   "outputs": [],
   "source": [
    "dictionary = Dictionary(occurence_lower_bound=0, max_dictionary_size=10)\n",
    "\n",
    "dictionary.fit(text_small_lemmatized_nltk);\n",
    "#dictionary.fit(text_small, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 253
    },
    "colab_type": "code",
    "id": "KJr0UBzOfFur",
    "outputId": "4ab23b42-0fb7-4ac4-c878-63da839c8635"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"end_of_word_token_policy\":\"Insert\",\"skip_step\":\"0\",\"start_token_id\":\"0\",\"token_level_type\":\"Word\",\"dictionary_format\":\"id_count_token\",\"end_of_sentence_token_policy\":\"Skip\",\"gram_order\":\"1\"}\r\n",
      "10\r\n",
      "0\t6\tcat\r\n",
      "1\t5\tmouse\r\n",
      "2\t2\tarmy\r\n",
      "3\t2\tcute\r\n",
      "4\t2\tdefeated\r\n",
      "5\t2\tpeace\r\n",
      "6\t1\tgather\r\n",
      "7\t1\tlive\r\n",
      "8\t1\toffer\r\n",
      "9\t1\tscare\r\n"
     ]
    }
   ],
   "source": [
    "dictionary.save('dictionary.tsv')\n",
    "!cat dictionary.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U1wLb5MX8NTY"
   },
   "source": [
    "## Feature Calculation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KYzNqXgcfFut"
   },
   "source": [
    "### Convertation into fixed size vectors\n",
    "\n",
    "The majority of classic ML algorithms are computing and performing predictions on a fixed number of features $F$.<br>\n",
    "That means that learning set $X = \\{x_i\\}$ contains vectors $x_i = (a_0, a_1, ..., a_F)$ where $F$ is constant.\n",
    "\n",
    "Since text object $x$ is not a fixed length vector, we need to perform preprocessing of the origin set $D$.<br>\n",
    "One of the simplest text to vector encoding technique is **Bag of words (BoW)**.\n",
    "\n",
    "### Bag of words algorithm\n",
    "\n",
    "The algorithm takes in a dictionary and a text.<br>\n",
    "During the algorithm text $x = (a_0, a_1, ..., a_k)$ converted into vector $\\tilde x = (b_0, b_1, ..., b_F)$,<br> where $b_i$ is 0/1 (depending on whether there is a word with id=$i$ from dictionary into text $x$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 161
    },
    "colab_type": "code",
    "id": "7Ea944JbfFuu",
    "outputId": "5f788c52-345c-4703-957a-4f57dd29c418"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['cat', 'so', 'cute'],\n",
       " ['mouse', 'scare'],\n",
       " ['cat', 'defeated', 'mouse'],\n",
       " ['cute', 'mouse', 'gather', 'army'],\n",
       " ['army', 'mouse', 'defeated', 'cat'],\n",
       " ['cat', 'offer', 'peace'],\n",
       " ['cat', 'scared'],\n",
       " ['cat', 'mouse', 'live', 'peace']]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_small_lemmatized_nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "bRm5Cf5qkzlJ",
    "outputId": "6226eea1-ab2b-4924-df6c-a006e71965f5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0, 3]]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary.apply([text_small_lemmatized_nltk[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 305
    },
    "colab_type": "code",
    "id": "ga0AfpT8fFuv",
    "outputId": "6b6e9abb-3e2a-4a8e-eac9-dacbac3c33fd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
       "       [1., 1., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 1., 1., 0., 0., 1., 0., 0., 0.],\n",
       "       [1., 1., 1., 0., 1., 0., 0., 0., 0., 0.],\n",
       "       [1., 0., 0., 0., 0., 1., 0., 0., 1., 0.],\n",
       "       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [1., 1., 0., 0., 0., 1., 0., 1., 0., 0.]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def bag_of_words(tokenized_text, dictionary):\n",
    "    features = np.zeros((len(tokenized_text), dictionary.size))\n",
    "    for i, tokenized_sentence in enumerate(tokenized_text):\n",
    "        indices = np.array(dictionary.apply([tokenized_sentence])[0])\n",
    "        features[i, indices] = 1\n",
    "    return features\n",
    "\n",
    "bow_features = bag_of_words(text_small_lemmatized_nltk, dictionary)\n",
    "bow_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vhr-EyPyfFuy"
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from scipy.sparse import csr_matrix\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "def fit_linear_model(X, c):\n",
    "    model = LogisticRegression()\n",
    "    model.fit(X, c)\n",
    "    return model\n",
    "\n",
    "def fit_naive_bayes(X, c):\n",
    "    clf = MultinomialNB()\n",
    "    if isinstance(X, csr_matrix):\n",
    "        X.eliminate_zeros()\n",
    "    clf.fit(X, c)\n",
    "    return clf\n",
    "\n",
    "def evaluate_model_logloss(model, X, y):\n",
    "    y_pred = model.predict_proba(X)[:,1]\n",
    "    metric = log_loss(y, y_pred)\n",
    "    print('Logloss: ' + str(metric))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "colab_type": "code",
    "id": "GekNCx5ofFuz",
    "outputId": "5b218b73-c7fd-4628-f218-29d0d30686eb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear model\n",
      "Logloss: 0.49830123911420365\n",
      "Naive bayes\n",
      "Logloss: 0.4528488772318392\n",
      "Comparing to constant prediction\n",
      "Logloss: 0.6931471805599453\n"
     ]
    }
   ],
   "source": [
    "def evaluate_models(X, y):\n",
    "    linear_model = fit_linear_model(bow_features, target_small)\n",
    "    naive_bayes = fit_naive_bayes(bow_features, target_small)\n",
    "        \n",
    "    print('Linear model')\n",
    "    evaluate_model_logloss(linear_model, X, y)\n",
    "    print('Naive bayes')\n",
    "    evaluate_model_logloss(naive_bayes, X, y)\n",
    "    print('Comparing to constant prediction')\n",
    "    logloss_constant_prediction = log_loss(y, np.ones(shape=(len(text_small), 2)) * 0.5)\n",
    "    print('Logloss: ' + str(logloss_constant_prediction))\n",
    "    \n",
    "evaluate_models(bow_features, target_small)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "colab_type": "code",
    "id": "uFsAWNE9fFu2",
    "outputId": "7197acdf-71ac-4c81-b507-4f06cafdbea8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear model\n",
      "Logloss: 0.46346010998469667\n",
      "Naive bayes\n",
      "Logloss: 0.3680393546716464\n",
      "Comparing to constant prediction\n",
      "Logloss: 0.6931471805599453\n"
     ]
    }
   ],
   "source": [
    "dictionary = Dictionary(occurence_lower_bound=0)\n",
    "dictionary.fit(text_small_lemmatized_nltk)\n",
    "\n",
    "bow_features = bag_of_words(text_small_lemmatized_nltk, dictionary)\n",
    "evaluate_models(bow_features, target_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yvjUACB_fFu6"
   },
   "source": [
    "### Looking at sequences of letters / words\n",
    "\n",
    "Let's look at the example: texts 'The cat defeated the mouse' and 'Army of mice defeated the cat :('<br>\n",
    "Simplifying it we have three tokens in each sentence 'cat defeat mouse' and 'mouse defeat cat'.<br>\n",
    "After applying BoW we get two equal vectors with the opposite meaning:\n",
    "\n",
    "| cat | mouse | defeat |\n",
    "|-----|-------|--------|\n",
    "| 1   | 1     | 1      |\n",
    "| 1   | 1     | 1      |\n",
    "\n",
    "How to distinguish them?\n",
    "Lets add sequences of words as a single tokens into our dictionary:\n",
    "\n",
    "| cat | mouse | defeat | cat_defeat | mouse_defeat | defeat_cat | defeat_mouse |\n",
    "|-----|-------|--------|------------|--------------|------------|--------------|\n",
    "| 1   | 1     | 1      | 1          | 0            | 0          | 1            |\n",
    "| 1   | 1     | 1      | 0          | 1            | 1          | 0            |\n",
    "\n",
    "**N-gram** is a continguous sequence of $n$ items from a given sample of text or speech (Wikipedia).<br>\n",
    "In example above Bi-gram (Bigram) = 2-gram of words.\n",
    "\n",
    "Ngrams help to add into vectors more information about text structure, moreover there are n-grams has no meanings in separation, for example, 'Mickey Mouse company'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 379
    },
    "colab_type": "code",
    "id": "WU6iWFPZClrf",
    "outputId": "b666b9a2-0782-472a-a729-0fa1b15bd9f2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"end_of_word_token_policy\":\"Insert\",\"skip_step\":\"0\",\"start_token_id\":\"0\",\"token_level_type\":\"Word\",\"dictionary_format\":\"id_count_token\",\"end_of_sentence_token_policy\":\"Skip\",\"gram_order\":\"2\"}\r\n",
      "17\r\n",
      "0\t1\tarmy mouse\r\n",
      "1\t1\tcat defeated\r\n",
      "2\t1\tcat mouse\r\n",
      "3\t1\tcat offer\r\n",
      "4\t1\tcat scared\r\n",
      "5\t1\tcat so\r\n",
      "6\t1\tcute mouse\r\n",
      "7\t1\tdefeated cat\r\n",
      "8\t1\tdefeated mouse\r\n",
      "9\t1\tgather army\r\n",
      "10\t1\tlive peace\r\n",
      "11\t1\tmouse defeated\r\n",
      "12\t1\tmouse gather\r\n",
      "13\t1\tmouse live\r\n",
      "14\t1\tmouse scare\r\n",
      "15\t1\toffer peace\r\n",
      "16\t1\tso cute\r\n"
     ]
    }
   ],
   "source": [
    "dictionary = Dictionary(occurence_lower_bound=0, gram_order=2)\n",
    "dictionary.fit(text_small_lemmatized_nltk)\n",
    "\n",
    "dictionary.save('dictionary.tsv')\n",
    "!cat dictionary.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "colab_type": "code",
    "id": "ypPTi_XXfFu7",
    "outputId": "59136696-c457-4f99-b884-cf1e2e68fb80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear model\n",
      "Logloss: 0.4084388990666391\n",
      "Naive bayes\n",
      "Logloss: 0.25985126095069233\n",
      "Comparing to constant prediction\n",
      "Logloss: 0.6931471805599453\n"
     ]
    }
   ],
   "source": [
    "bow_features = bag_of_words(text_small_lemmatized_nltk, dictionary)\n",
    "evaluate_models(bow_features, target_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1uLlIfJHodEL"
   },
   "source": [
    "### Unigram + Bigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "colab_type": "code",
    "id": "XaRC74kNfFu8",
    "outputId": "f67a5ea4-0795-4b16-db80-2bff733109e9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear model\n",
      "Logloss: 0.32129697521334455\n",
      "Naive bayes\n",
      "Logloss: 0.13103685350656918\n",
      "Comparing to constant prediction\n",
      "Logloss: 0.6931471805599453\n"
     ]
    }
   ],
   "source": [
    "dictionary1 = Dictionary(occurence_lower_bound=0)\n",
    "dictionary1.fit(text_small_lemmatized_nltk)\n",
    "\n",
    "bow_features1 = bag_of_words(text_small_lemmatized_nltk, dictionary1)\n",
    "\n",
    "dictionary2 = Dictionary(occurence_lower_bound=0, gram_order=2)\n",
    "dictionary2.fit(text_small_lemmatized_nltk)\n",
    "\n",
    "bow_features2 = bag_of_words(text_small_lemmatized_nltk, dictionary2)\n",
    "\n",
    "bow_features = np.concatenate((bow_features1, bow_features2), axis=1)\n",
    "evaluate_models(bow_features, target_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oFR_rMfH8NT_"
   },
   "source": [
    "## CatBoost Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8xoFAOiz8NT_"
   },
   "source": [
    "Parameter names:\n",
    "\n",
    "1. **Text Tokenization** - `tokenizers`\n",
    "2. **Dictionary Creation** - `dictionaries`\n",
    "3. **Feature Calculation** - `feature_calcers`\n",
    "\n",
    "\\* More complex configuration with `text_processing` parameter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wntt3XrYgkhf"
   },
   "source": [
    "### `tokenizers`\n",
    "\n",
    "Tokenizers used to preprocess Text type feature columns before creating the dictionary.\n",
    "\n",
    "[Documentation](https://catboost.ai/docs/references/tokenizer_options.html).\n",
    "\n",
    "```\n",
    "tokenizers = [{\n",
    "\t'tokenizerId': 'Space',\n",
    "\t'delimiter': ' ',\n",
    "\t'separator_type': 'ByDelimiter',\n",
    "},{\n",
    "\t'tokenizerId': 'Sense',\n",
    "\t'separator_type': 'BySense',\n",
    "}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aKqHyav7fFu-"
   },
   "source": [
    "### `dictionaries`\n",
    "\n",
    "Dictionaries used to preprocess Text type feature columns.\n",
    "\n",
    "[Documentation](https://catboost.ai/docs/references/dictionaries_options.html).\n",
    "\n",
    "```\n",
    "dictionaries = [{\n",
    "\t'dictionaryId': 'Unigram',\n",
    "\t'max_dictionary_size': '50000',\n",
    "\t'gram_count': '1',\n",
    "},{\n",
    "\t'dictionaryId': 'Bigram',\n",
    "\t'max_dictionary_size': '50000',\n",
    "\t'gram_count': '2',\n",
    "},{\n",
    "\t'dictionaryId': 'Trigram',\n",
    "\t'token_level_type': 'Letter',\n",
    "\t'max_dictionary_size': '50000',\n",
    "\t'gram_count': '3',\n",
    "}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JT6I_LN98NUC"
   },
   "source": [
    "### `feature_calcers`\n",
    "\n",
    "Feature calcers used to calculate new features based on preprocessed Text type feature columns.\n",
    "\n",
    "1. **`BoW`**<br>\n",
    "Bag of words: 0/1 features (text sample has or not token_id).<br>\n",
    "Number of produced numeric features = dictionary size.<br>\n",
    "Parameters: `top_tokens_count` - maximum number of tokens that will be used for vectorization in bag of words, the most frequent $n$ tokens are taken (**highly affect both on CPU ang GPU RAM usage**).\n",
    "\n",
    "2. **`NaiveBayes`**<br>\n",
    "NaiveBayes: [Multinomial naive bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes) model. As many new features as classes are added. This feature is calculated by analogy with counters in CatBoost by permutation ([estimation of CTRs](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html)). In other words, a random permutation is made and then we go from top to bottom on the dataset and calculate the probability of its belonging to this class for each object.\n",
    "\n",
    "3. **`BM25`**<br>\n",
    "[BM25](https://en.wikipedia.org/wiki/Okapi_BM25). As many new features as classes are added. The idea is the same as in Naive Bayes, but for each class we calculate not the conditional probability, but a certain relevance, which is similar to tf-idf, where the tokens instead of the words and the classes instead of the documents (or rather, the unification of all texts of this class). Only the tf multiplier in BM25 is replaced with another multiplier, which gives an advantage to classes that contain rare tokens.\n",
    "\n",
    "```\n",
    "feature_calcers = [\n",
    "\t'BoW:top_tokens_count=1000',\n",
    "\t'NaiveBayes',\n",
    "\t'BM25',\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "02lH5f1PgpYM"
   },
   "source": [
    "### `text_processing`\n",
    "\n",
    "```\n",
    "text_processing = {\n",
    "    \"tokenizers\" : [{\n",
    "        \"tokenizer_id\" : \"Space\",\n",
    "        \"separator_type\" : \"ByDelimiter\",\n",
    "        \"delimiter\" : \" \"\n",
    "    }],\n",
    "\n",
    "    \"dictionaries\" : [{\n",
    "        \"dictionary_id\" : \"BiGram\",\n",
    "        \"max_dictionary_size\" : \"50000\",\n",
    "        \"occurrence_lower_bound\" : \"3\",\n",
    "        \"gram_order\" : \"2\"\n",
    "    }, {\n",
    "        \"dictionary_id\" : \"Word\",\n",
    "        \"max_dictionary_size\" : \"50000\",\n",
    "        \"occurrence_lower_bound\" : \"3\",\n",
    "        \"gram_order\" : \"1\"\n",
    "    }],\n",
    "\n",
    "    \"feature_processing\" : {\n",
    "        \"default\" : [{\n",
    "            \"dictionaries_names\" : [\"BiGram\", \"Word\"],\n",
    "            \"feature_calcers\" : [\"BoW\"],\n",
    "            \"tokenizers_names\" : [\"Space\"]\n",
    "        }, {\n",
    "            \"dictionaries_names\" : [\"Word\"],\n",
    "            \"feature_calcers\" : [\"NaiveBayes\"],\n",
    "            \"tokenizers_names\" : [\"Space\"]\n",
    "        }],\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "id": "-HOhMr-ffFu_",
    "outputId": "d28394e4-fb6e-4a63-9090-8860a3a27333"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.8772679\tbest: 0.8772679 (0)\ttotal: 172ms\tremaining: 2m 51s\n",
      "100:\ttest: 0.9422074\tbest: 0.9422074 (100)\ttotal: 19.1s\tremaining: 2m 49s\n",
      "200:\ttest: 0.9513095\tbest: 0.9513095 (200)\ttotal: 37.3s\tremaining: 2m 28s\n",
      "300:\ttest: 0.9566328\tbest: 0.9566328 (300)\ttotal: 56.2s\tremaining: 2m 10s\n",
      "400:\ttest: 0.9593129\tbest: 0.9593129 (400)\ttotal: 1m 14s\tremaining: 1m 50s\n",
      "500:\ttest: 0.9610368\tbest: 0.9610368 (500)\ttotal: 1m 32s\tremaining: 1m 32s\n",
      "600:\ttest: 0.9622586\tbest: 0.9622586 (600)\ttotal: 1m 51s\tremaining: 1m 13s\n",
      "700:\ttest: 0.9631853\tbest: 0.9631853 (700)\ttotal: 2m 9s\tremaining: 55.1s\n",
      "800:\ttest: 0.9637900\tbest: 0.9637900 (800)\ttotal: 2m 27s\tremaining: 36.6s\n",
      "900:\ttest: 0.9642351\tbest: 0.9642422 (898)\ttotal: 2m 46s\tremaining: 18.3s\n",
      "999:\ttest: 0.9646141\tbest: 0.9646180 (998)\ttotal: 3m 5s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.9646179863\n",
      "bestIteration = 998\n",
      "\n",
      "Shrink model to first 999 iterations.\n"
     ]
    }
   ],
   "source": [
    "model_on_processed_data_2 = fit_model(\n",
    "    train_processed_pool,\n",
    "    test_processed_pool,\n",
    "    text_processing = {\n",
    "        \"tokenizers\" : [{\n",
    "            \"tokenizer_id\" : \"Space\",\n",
    "            \"separator_type\" : \"ByDelimiter\",\n",
    "            \"delimiter\" : \" \"\n",
    "        }],\n",
    "    \n",
    "        \"dictionaries\" : [{\n",
    "            \"dictionary_id\" : \"BiGram\",\n",
    "            \"max_dictionary_size\" : \"50000\",\n",
    "            \"occurrence_lower_bound\" : \"3\",\n",
    "            \"gram_order\" : \"2\"\n",
    "        }, {\n",
    "            \"dictionary_id\" : \"Word\",\n",
    "            \"max_dictionary_size\" : \"50000\",\n",
    "            \"occurrence_lower_bound\" : \"3\",\n",
    "            \"gram_order\" : \"1\"\n",
    "        }],\n",
    "    \n",
    "        \"feature_processing\" : {\n",
    "            \"default\" : [{\n",
    "                \"dictionaries_names\" : [\"BiGram\", \"Word\"],\n",
    "                \"feature_calcers\" : [\"BoW\"],\n",
    "                \"tokenizers_names\" : [\"Space\"]\n",
    "            }, {\n",
    "                \"dictionaries_names\" : [\"Word\"],\n",
    "                \"feature_calcers\" : [\"NaiveBayes\"],\n",
    "                \"tokenizers_names\" : [\"Space\"]\n",
    "            }],\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "HFJRD9RofFvC",
    "outputId": "08f27541-75fe-4c0e-dd88-3b6e9a716035"
   },
   "outputs": [],
   "source": [
    "print_score_diff(model_no_text, model_on_processed_data_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xlo77dzufFvE"
   },
   "source": [
    "## Summary: Text features in CatBoost\n",
    "\n",
    "### The algorithm:\n",
    "1. Input text is loaded as a usual column. ``text_column: [string]``.\n",
    "2. Each text sample is tokenized via splitting by space. ``tokenized_column: [[string]]``.\n",
    "3. Dictionary estimation.\n",
    "4. Each string in tokenized column is converted into token_id from dictionary. ``text: [[token_id]]``.\n",
    "5. Depending on the parameters CatBoost produce features basing on the resulting text column: Bag of words, Multinomial naive bayes or Bm25.\n",
    "6. Computed float features are passed into the usual CatBoost learning algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_A87DhGF8SIa"
   },
   "source": [
    "# Embeddings In CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-11-17 17:22:02--  https://transfersh.com/HDHxy/embedded_train.npy\n",
      "Resolving transfersh.com (transfersh.com)... 64:ff9b::6bb2:6ca6\n",
      "Connecting to transfersh.com (transfersh.com)|64:ff9b::6bb2:6ca6|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 163840128 (156M) []\n",
      "Saving to: ‘embedded_train.npy’\n",
      "\n",
      "embedded_train.npy  100%[===================>] 156.25M  18.8MB/s    in 9.7s    \n",
      "\n",
      "2020-11-17 17:22:13 (16.0 MB/s) - ‘embedded_train.npy’ saved [163840128/163840128]\n",
      "\n",
      "--2020-11-17 17:22:18--  https://transfersh.com/whOm3/embedded_test.npy\n",
      "Resolving transfersh.com (transfersh.com)... 64:ff9b::6bb2:6ca6\n",
      "Connecting to transfersh.com (transfersh.com)|64:ff9b::6bb2:6ca6|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 40960128 (39M) []\n",
      "Saving to: ‘embedded_test.npy’\n",
      "\n",
      "embedded_test.npy   100%[===================>]  39.06M  11.9MB/s    in 3.3s    \n",
      "\n",
      "2020-11-17 17:22:21 (11.9 MB/s) - ‘embedded_test.npy’ saved [40960128/40960128]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# from sentence_transformers import SentenceTransformer\n",
    "# big_model = SentenceTransformer('roberta-large-nli-stsb-mean-tokens')\n",
    "# X_embed_train = big_model.encode(X_train['review'].to_list())\n",
    "# X_embed_test = big_model.encode(X_test['review'].to_list())\n",
    "\n",
    "!wget https://transfersh.com/HDHxy/embedded_train.npy -O embedded_train.npy\n",
    "X_embed_train = np.load('embedded_train.npy')\n",
    "\n",
    "!wget https://transfersh.com/whOm3/embedded_test.npy -O embedded_test.npy\n",
    "X_embed_test = np.load('embedded_test.npy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_embed_train_small, y_train_small = X_embed_train[:1000], y_train[:1000]\n",
    "X_embed_test_small, y_test_small = X_embed_test[:1000], y_test[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pure embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logloss: 0.7683870896879872\n"
     ]
    }
   ],
   "source": [
    "linmodel = fit_linear_model(X_embed_train_small, y_train_small)\n",
    "evaluate_model_logloss(linmodel, X_embed_test_small, y_test_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear Discriminant Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logloss: 0.812982481413373\n"
     ]
    }
   ],
   "source": [
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "\n",
    "clf = LinearDiscriminantAnalysis()\n",
    "clf.fit(X_embed_train_small[:500], y_train_small[:500])\n",
    "\n",
    "X_lda_train_small = clf.transform(X_embed_train_small[500:])\n",
    "X_embed_lda_train_small = np.concatenate([X_embed_train_small[500:], X_lda_train_small], axis=1)\n",
    "\n",
    "X_lda_test_small = clf.transform(X_embed_test_small)\n",
    "X_embed_lda_test_small = np.concatenate([X_embed_test_small, X_lda_test_small], axis=1)\n",
    "\n",
    "\n",
    "linmodel = fit_linear_model(X_embed_lda_train_small, y_train_small[500:])\n",
    "evaluate_model_logloss(linmodel, X_embed_lda_test_small, y_test_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings in CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "with open('train_embed_text.tsv', 'w') as f:\n",
    "    writer = csv.writer(f, delimiter='\\t', quotechar='\"')\n",
    "    for y, text, row in zip(y_train, X_preprocessed_train['review'].to_list(), X_embed_train):\n",
    "        writer.writerow((str(y), text, ';'.join(map(str, row))))\n",
    "\n",
    "with open('test_embed_text.tsv', 'w') as f:\n",
    "    writer = csv.writer(f, delimiter='\\t', quotechar='\"')\n",
    "    for y, text, row in zip(y_test, X_preprocessed_test['review'].to_list(), X_embed_test):\n",
    "        writer.writerow((str(y), text, ';'.join(map(str, row))))\n",
    "        \n",
    "with open('pool_text.cd', 'w') as f:\n",
    "    f.write(\n",
    "        '0\\tLabel\\n'\\\n",
    "        '1\\tText\\n'\\\n",
    "        '2\\tNumVector'\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import Pool\n",
    "train_embed_pool = Pool('train_embed_text.tsv', column_description='pool_text.cd')\n",
    "test_embed_pool = Pool('test_embed_text.tsv', column_description='pool_text.cd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.9189093\tbest: 0.9189093 (0)\ttotal: 92.2ms\tremaining: 1m 32s\n",
      "100:\ttest: 0.9573114\tbest: 0.9573114 (100)\ttotal: 9.45s\tremaining: 1m 24s\n",
      "200:\ttest: 0.9623914\tbest: 0.9623914 (200)\ttotal: 18.8s\tremaining: 1m 14s\n",
      "300:\ttest: 0.9650703\tbest: 0.9650703 (300)\ttotal: 28.2s\tremaining: 1m 5s\n",
      "400:\ttest: 0.9665432\tbest: 0.9665432 (400)\ttotal: 37.6s\tremaining: 56.2s\n",
      "500:\ttest: 0.9675662\tbest: 0.9675662 (500)\ttotal: 47.4s\tremaining: 47.2s\n",
      "600:\ttest: 0.9683025\tbest: 0.9683025 (600)\ttotal: 57s\tremaining: 37.8s\n",
      "700:\ttest: 0.9688488\tbest: 0.9688488 (700)\ttotal: 1m 6s\tremaining: 28.2s\n",
      "800:\ttest: 0.9691871\tbest: 0.9691871 (800)\ttotal: 1m 15s\tremaining: 18.8s\n",
      "900:\ttest: 0.9695822\tbest: 0.9695822 (900)\ttotal: 1m 24s\tremaining: 9.33s\n",
      "999:\ttest: 0.9697098\tbest: 0.9697265 (992)\ttotal: 1m 34s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.9697265166\n",
      "bestIteration = 992\n",
      "\n",
      "Shrink model to first 993 iterations.\n"
     ]
    }
   ],
   "source": [
    "model_text_embeddings = fit_model(train_embed_pool, test_embed_pool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9579518196391623 vs 0.9697265165993134 (+1.23%)\n"
     ]
    }
   ],
   "source": [
    "print_score_diff(model, model_text_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}