week5-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final project: StackOverflow assistant bot\n",
    "\n",
    "Congratulations on coming this far and solving the programming assignments! In this final project, we will combine everything we have learned about Natural Language Processing to construct a *dialogue chat bot*, which will be able to:\n",
    "* answer programming-related questions (using StackOverflow dataset);\n",
    "* chit-chat and simulate dialogue on all non programming-related questions.\n",
    "\n",
    "For a chit-chat mode we will use a pre-trained neural network engine available from [ChatterBot](https://github.com/gunthercox/ChatterBot).\n",
    "Those who aim at honor certificates for our course or are just curious, will train their own models for chit-chat.\n",
    "![](https://imgs.xkcd.com/comics/twitter_bot.png)\n",
    "©[xkcd](https://xkcd.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data description\n",
    "\n",
    "To detect *intent* of users questions we will need two text collections:\n",
    "- `tagged_posts.tsv` — StackOverflow posts, tagged with one programming language (*positive samples*).\n",
    "- `dialogues.tsv` — dialogue phrases from movie subtitles (*negative samples*).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File data/dialogues.tsv is already downloaded.\n",
      "File data/tagged_posts.tsv is already downloaded.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "from common.download_utils import download_project_resources\n",
    "\n",
    "download_project_resources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those questions, that have programming-related intent, we will proceed as follow predict programming language (only one tag per question allowed here) and rank candidates within the tag using embeddings.\n",
    "For the ranking part, you will need:\n",
    "- `word_embeddings.tsv` — word embeddings, that you  trained with StarSpace in the 3rd assignment. It's not a problem if you didn't do it, because we can offer an alternative solution for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result of this notebook, you should obtain the following new objects that you will then use in the running bot:\n",
    "\n",
    "- `intent_recognizer.pkl` — intent recognition model;\n",
    "- `tag_classifier.pkl` — programming language classification model;\n",
    "- `tfidf_vectorizer.pkl` — vectorizer used during training;\n",
    "- `thread_embeddings_by_tags` — folder with thread embeddings, arranged by tags.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some functions will be reused by this notebook and the scripts, so we put them into *utils.py* file. Don't forget to open it and fill in the gaps!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/aimilios/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part I. Intent and language recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to write a bot, which will not only **answer programming-related questions**, but also will be able to **maintain a dialogue**. We would also like to detect the *intent* of the user from the question (we could have had a 'Question answering mode' check-box in the bot, but it wouldn't fun at all, would it?). So the first thing we need to do is to **distinguish programming-related questions from general ones**.\n",
    "\n",
    "It would also be good to predict which programming language a particular question referees to. By doing so, we will speed up question search by a factor of the number of languages (10 here), and exercise our *text classification* skill a bit. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import re\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first assignment (Predict tags on StackOverflow with linear models), you have already learnt how to preprocess texts and do TF-IDF tranformations. Reuse your code here. In addition, you will also need to [dump](https://docs.python.org/3/library/pickle.html#pickle.dump) the TF-IDF vectorizer with pickle to use it later in the running bot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tfidf_features(X_train, X_test, vectorizer_path):\n",
    "    \"\"\"Performs TF-IDF transformation and dumps the model.\"\"\"\n",
    "    \n",
    "    # Train a vectorizer on X_train data.\n",
    "    # Transform X_train and X_test data.\n",
    "    \n",
    "    # Pickle the trained vectorizer to 'vectorizer_path'\n",
    "    # Don't forget to open the file in writing bytes mode.\n",
    "    \n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################\n",
    "    vectorizer = TfidfVectorizer(min_df=5, max_df=0.9, ngram_range=(1, 2), token_pattern='(\\S+)')\n",
    "    X_train = vectorizer.fit_transform(X_train)\n",
    "    X_test = vectorizer.transform(X_test)\n",
    "    with open(vectorizer_path, 'wb') as f:\n",
    "        pickle.dump(vectorizer, f)\n",
    "    \n",
    "    \n",
    "    return X_train, X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, load examples of two classes. Use a subsample of stackoverflow data to balance the classes. You will need the full data later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_size = 200000\n",
    "\n",
    "dialogue_df = pd.read_csv('data/dialogues.tsv', sep='\\t').sample(sample_size, random_state=0)\n",
    "stackoverflow_df = pd.read_csv('data/tagged_posts.tsv', sep='\\t').sample(sample_size, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check how the data look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>82925</th>\n",
       "      <td>Donna, you are a muffin.</td>\n",
       "      <td>dialogue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48774</th>\n",
       "      <td>He was here last night till about two o'clock....</td>\n",
       "      <td>dialogue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55394</th>\n",
       "      <td>All right, then make an appointment with her s...</td>\n",
       "      <td>dialogue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90806</th>\n",
       "      <td>Hey, what is this-an interview? We're supposed...</td>\n",
       "      <td>dialogue</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>107758</th>\n",
       "      <td>Yeah. He's just a friend of mine I was trying ...</td>\n",
       "      <td>dialogue</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     text       tag\n",
       "82925                            Donna, you are a muffin.  dialogue\n",
       "48774   He was here last night till about two o'clock....  dialogue\n",
       "55394   All right, then make an appointment with her s...  dialogue\n",
       "90806   Hey, what is this-an interview? We're supposed...  dialogue\n",
       "107758  Yeah. He's just a friend of mine I was trying ...  dialogue"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dialogue_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2168983</th>\n",
       "      <td>43837842</td>\n",
       "      <td>Efficient Algorithm to compose valid expressio...</td>\n",
       "      <td>python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1084095</th>\n",
       "      <td>15747223</td>\n",
       "      <td>Why does this basic thread program fail with C...</td>\n",
       "      <td>c_cpp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1049020</th>\n",
       "      <td>15189594</td>\n",
       "      <td>Link to scroll to top not working</td>\n",
       "      <td>javascript</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200466</th>\n",
       "      <td>3273927</td>\n",
       "      <td>Is it possible to implement ping on windows ph...</td>\n",
       "      <td>c#</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1200249</th>\n",
       "      <td>17684551</td>\n",
       "      <td>GLSL normal mapping issue</td>\n",
       "      <td>c_cpp</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          post_id                                              title  \\\n",
       "2168983  43837842  Efficient Algorithm to compose valid expressio...   \n",
       "1084095  15747223  Why does this basic thread program fail with C...   \n",
       "1049020  15189594                  Link to scroll to top not working   \n",
       "200466    3273927  Is it possible to implement ping on windows ph...   \n",
       "1200249  17684551                          GLSL normal mapping issue   \n",
       "\n",
       "                tag  \n",
       "2168983      python  \n",
       "1084095       c_cpp  \n",
       "1049020  javascript  \n",
       "200466           c#  \n",
       "1200249       c_cpp  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stackoverflow_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply *text_prepare* function to preprocess the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import text_prepare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "dialogue_df['text'] = dialogue_df['text'].apply(text_prepare)\n",
    "stackoverflow_df['title'] = stackoverflow_df['title'].apply(text_prepare)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intent recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will do a binary classification on TF-IDF representations of texts. Labels will be either `dialogue` for general questions or `stackoverflow` for programming-related questions. First, prepare the data for this task:\n",
    "- concatenate `dialogue` and `stackoverflow` examples into one sample\n",
    "- split it into train and test in proportion 9:1, use *random_state=0* for reproducibility\n",
    "- transform it into TF-IDF features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size = 360000, test size = 40000\n"
     ]
    }
   ],
   "source": [
    "X = np.concatenate([dialogue_df['text'].values, stackoverflow_df['title'].values])\n",
    "y = ['dialogue'] * dialogue_df.shape[0] + ['stackoverflow'] * stackoverflow_df.shape[0]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=0)\n",
    "print('Train size = {}, test size = {}'.format(len(X_train), len(X_test)))\n",
    "\n",
    "X_train_tfidf, X_test_tfidf = tfidf_features(X_train, X_test, RESOURCE_PATH['TFIDF_VECTORIZER'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the **intent recognizer** using LogisticRegression on the train set with the following parameters: *penalty='l2'*, *C=10*, *random_state=0*. Print out the accuracy on the test set to check whether everything looks good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=10, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=0, solver='liblinear',\n",
       "          tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "######################################\n",
    "######### YOUR CODE HERE #############\n",
    "######################################\n",
    "intent_recognizer = LogisticRegression(penalty='l2', C=10, random_state=0, solver='liblinear')\n",
    "intent_recognizer.fit(X_train_tfidf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy = 0.991575\n"
     ]
    }
   ],
   "source": [
    "# Check test accuracy.\n",
    "y_test_pred = intent_recognizer.predict(X_test_tfidf)\n",
    "test_accuracy = accuracy_score(y_test, y_test_pred)\n",
    "print('Test accuracy = {}'.format(test_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dump the classifier to use it in the running bot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(intent_recognizer, open(RESOURCE_PATH['INTENT_RECOGNIZER'], 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Programming language classification "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train one more classifier for the programming-related questions. It will predict exactly one tag (=programming language) and will be also based on Logistic Regression with TF-IDF features. \n",
    "\n",
    "First, let us prepare the data for this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stackoverflow_df['title'].values\n",
    "y = stackoverflow_df['tag'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size = 160000, test size = 40000\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
    "print('Train size = {}, test size = {}'.format(len(X_train), len(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us reuse the TF-IDF vectorizer that we have already created above. It should not make a huge difference which data was used to train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['uitableview lags scrolling' 'use dbnullvalue null c#net'\n",
      " 'load crystal report wpf c# using entity data model'\n",
      " 'nsstring property making shallow copy' 'create list object swift']\n"
     ]
    }
   ],
   "source": [
    "print(X_train[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = pickle.load(open(RESOURCE_PATH['TFIDF_VECTORIZER'], 'rb'))\n",
    "\n",
    "X_train_tfidf, X_test_tfidf = vectorizer.transform(X_train), vectorizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the **tag classifier** using OneVsRestClassifier wrapper over LogisticRegression. Use the following parameters: *penalty='l2'*, *C=5*, *random_state=0*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.multiclass import OneVsRestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneVsRestClassifier(estimator=LogisticRegression(C=5, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='warn',\n",
       "          n_jobs=None, penalty='l2', random_state=0, solver='liblinear',\n",
       "          tol=0.0001, verbose=0, warm_start=False),\n",
       "          n_jobs=None)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "######################################\n",
    "######### YOUR CODE HERE #############\n",
    "######################################\n",
    "tag_classifier = OneVsRestClassifier(LogisticRegression(penalty='l2', C=5, random_state=0, solver='liblinear'))\n",
    "tag_classifier.fit(X_train_tfidf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy = 0.800725\n"
     ]
    }
   ],
   "source": [
    "# Check test accuracy.\n",
    "y_test_pred = tag_classifier.predict(X_test_tfidf)\n",
    "test_accuracy = accuracy_score(y_test, y_test_pred)\n",
    "print('Test accuracy = {}'.format(test_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dump the classifier to use it in the running bot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(tag_classifier, open(RESOURCE_PATH['TAG_CLASSIFIER'], 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part II. Ranking  questions with embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find a relevant answer (a thread from StackOverflow) on a question you will use vector representations to calculate similarity between the question and existing threads. We already had `question_to_vec` function from the assignment 3, which can create such a representation based on word vectors. \n",
    "\n",
    "However, it would be costly to compute such a representation for all possible answers in *online mode* of the bot (e.g. when bot is running and answering questions from many users). This is the reason why you will create a *database* with pre-computed representations. These representations will be arranged by non-overlaping tags (programming languages), so that the search of the answer can be performed only within one tag each time. This will make our bot even more efficient and allow not to store all the database in RAM. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load StarSpace embeddings which were trained on Stack Overflow posts. These embeddings were trained in *supervised mode* for duplicates detection on the same corpus that is used in search. We can account on that these representations will allow us to find closely related answers for a question. \n",
    "\n",
    "If for some reasons you didn't train StarSpace embeddings in the assignment 3, you can use [pre-trained word vectors](https://code.google.com/archive/p/word2vec/) from Google. All instructions about how to work with these vectors were provided in the same assignment. However, we highly recommend to use StartSpace's embeddings, because it contains more appropriate embeddings. If you chose to use Google's embeddings, delete the words, which is not in Stackoverflow data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "starspace_embeddings, embeddings_dim = load_embeddings('data/StarSpace_embeddings.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to precompute representations for all possible answers, we need to load the whole posts dataset, unlike we did for the intent classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "posts_df = pd.read_csv('data/tagged_posts.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>post_id</th>\n",
       "      <th>title</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9</td>\n",
       "      <td>Calculate age in C#</td>\n",
       "      <td>c#</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16</td>\n",
       "      <td>Filling a DataSet or DataTable from a LINQ que...</td>\n",
       "      <td>c#</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>39</td>\n",
       "      <td>Reliable timer in a console application</td>\n",
       "      <td>c#</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42</td>\n",
       "      <td>Best way to allow plugins for a PHP application</td>\n",
       "      <td>php</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59</td>\n",
       "      <td>How do I get a distinct, ordered list of names...</td>\n",
       "      <td>c#</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   post_id                                              title  tag\n",
       "0        9                                Calculate age in C#   c#\n",
       "1       16  Filling a DataSet or DataTable from a LINQ que...   c#\n",
       "2       39            Reliable timer in a console application   c#\n",
       "3       42    Best way to allow plugins for a PHP application  php\n",
       "4       59  How do I get a distinct, ordered list of names...   c#"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "posts_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the distribution of posts for programming languages (tags) and find the most common ones. \n",
    "You might want to use pandas [groupby](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html) and [count](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.count.html) methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tag\n",
       "c#            394451\n",
       "c_cpp         281300\n",
       "java          383456\n",
       "javascript    375867\n",
       "php           321752\n",
       "python        208607\n",
       "r              36359\n",
       "ruby           99930\n",
       "swift          34809\n",
       "vb             35044\n",
       "Name: tag, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts_by_tag = posts_df.groupby('tag')['tag'].count()\n",
    "# counts_by_tag = posts_df.groupby(['tag']).count()\n",
    "\n",
    "\n",
    "counts_by_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for each `tag` you need to create two data structures, which will serve as online search index:\n",
    "* `tag_post_ids` — a list of post_ids with shape `(counts_by_tag[tag],)`. It will be needed to show the title and link to the thread;\n",
    "* `tag_vectors` — a matrix with shape `(counts_by_tag[tag], embeddings_dim)` where embeddings for each answer are stored.\n",
    "\n",
    "Implement the code which will calculate the mentioned structures and dump it to files. It should take several minutes to compute it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[       9       16       39 ... 45712096 45812002 45872841]\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.makedirs(RESOURCE_PATH['THREAD_EMBEDDINGS_FOLDER'], exist_ok=True)\n",
    "\n",
    "for tag, count in counts_by_tag.items():\n",
    "    tag_posts = posts_df[posts_df['tag'] == tag]\n",
    "    \n",
    "    tag_post_ids = tag_posts['post_id'].values\n",
    "    \n",
    "    tag_vectors = np.zeros((count, embeddings_dim), dtype=np.float32)\n",
    "    for i, title in enumerate(tag_posts['title']):\n",
    "        tag_vectors[i, :] = question_to_vec(title, starspace_embeddings, embeddings_dim)\n",
    "\n",
    "    # Dump post ids and vectors to a file.\n",
    "    filename = os.path.join(RESOURCE_PATH['THREAD_EMBEDDINGS_FOLDER'], os.path.normpath('%s.pkl' % tag))\n",
    "    pickle.dump((tag_post_ids, tag_vectors), open(filename, 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "latex_envs": {
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}