week1-MultilabelClassification-checkpoint.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict tags on StackOverflow with linear models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment you will learn how to predict tags for posts from [StackOverflow](https://stackoverflow.com). To solve this task you will use multilabel classification approach.\n",
    "\n",
    "### Libraries\n",
    "\n",
    "In this task you will need the following libraries:\n",
    "- [Numpy](http://www.numpy.org) — a package for scientific computing.\n",
    "- [Pandas](https://pandas.pydata.org) — a library providing high-performance, easy-to-use data structures and data analysis tools for the Python\n",
    "- [scikit-learn](http://scikit-learn.org/stable/index.html) — a tool for data mining and data analysis.\n",
    "- [NLTK](http://www.nltk.org) — a platform to work with natural language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "The following cell will download all data required for this assignment into the folder `week1/data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'requests'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-f3f9bca8fbe1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0msys\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0msys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"..\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mcommon\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdownload_utils\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mdownload_week1_resources\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[0mdownload_week1_resources\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Documents\\NLP_Coursera\\natural-language-processing\\common\\download_utils.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mos\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mshutil\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mrequests\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mcommon\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mtqdm_utils\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'requests'"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "from common.download_utils import download_week1_resources\n",
    "\n",
    "download_week1_resources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grading\n",
    "We will create a grader instance below and use it to collect your answers. Note that these outputs will be stored locally inside grader and will be uploaded to platform only after running submitting function in the last part of this assignment. If you want to make partial submission, you can run that cell any time you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from grader import Grader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grader = Grader()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this and most of the following assignments you will need to use a list of stop words. It can be downloaded from *nltk*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this task you will deal with a dataset of post titles from StackOverflow. You are provided a split to 3 sets: *train*, *validation* and *test*. All corpora (except for *test*) contain titles of the posts and corresponding tags (100 tags are available). The *test* set is provided for Coursera's grading and doesn't contain answers. Upload the corpora using *pandas* and look at the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ast import literal_eval\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def read_data(filename):\n",
    "    data = pd.read_csv(filename, sep='\\t')\n",
    "    data['tags'] = data['tags'].apply(literal_eval)\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = read_data('data/train.tsv')\n",
    "validation = read_data('data/validation.tsv')\n",
    "test = pd.read_csv('data/test.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, *title* column contains titles of the posts and *tags* column contains the tags. It could be noticed that a number of tags for a post is not fixed and could be as many as necessary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more comfortable usage, initialize *X_train*, *X_val*, *X_test*, *y_train*, *y_val*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train = train['title'].values, train['tags'].values\n",
    "X_val, y_val = validation['title'].values, validation['tags'].values\n",
    "X_test = test['title'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most known difficulties when working with natural data is that it's unstructured. For example, if you use it \"as is\" and extract tokens just by splitting the titles by whitespaces, you will see that there are many \"weird\" tokens like *3.5?*, *\"Flip*, etc. To prevent the problems, it's usually useful to prepare the data somehow. In this task you'll write a function, which will be also used in the other assignments. \n",
    "\n",
    "**Task 1 (TextPrepare).** Implement the function *text_prepare* following the instructions. After that, run the function *test_test_prepare* to test it on tiny cases and submit it to Coursera."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "REPLACE_BY_SPACE_RE = re.compile('[/(){}\\[\\]\\|@,;]')\n",
    "BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')\n",
    "STOPWORDS = set(stopwords.words('english'))\n",
    "\n",
    "def text_prepare(text):\n",
    "    \"\"\"\n",
    "        text: a string\n",
    "        \n",
    "        return: modified initial string\n",
    "    \"\"\"\n",
    "    text = # lowercase text\n",
    "    text = # replace REPLACE_BY_SPACE_RE symbols by space in text\n",
    "    text = # delete symbols which are in BAD_SYMBOLS_RE from text\n",
    "    text = # delete stopwords from text\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_text_prepare():\n",
    "    examples = [\"SQL Server - any equivalent of Excel's CHOOSE function?\",\n",
    "                \"How to free c++ memory vector<int> * arr?\"]\n",
    "    answers = [\"sql server equivalent excels choose function\", \n",
    "               \"free c++ memory vectorint arr\"]\n",
    "    for ex, ans in zip(examples, answers):\n",
    "        if text_prepare(ex) != ans:\n",
    "            return \"Wrong answer for the case: '%s'\" % ex\n",
    "    return 'Basic tests are passed.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(test_text_prepare())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run your implementation for questions from file *text_prepare_tests.tsv* to earn the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prepared_questions = []\n",
    "for line in open('data/text_prepare_tests.tsv', encoding='utf-8'):\n",
    "    line = text_prepare(line.strip())\n",
    "    prepared_questions.append(line)\n",
    "text_prepare_results = '\\n'.join(prepared_questions)\n",
    "\n",
    "grader.submit_tag('TextPrepare', text_prepare_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can preprocess the titles using function *text_prepare* and  making sure that the headers don't have bad symbols:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = [text_prepare(x) for x in X_train]\n",
    "X_val = [text_prepare(x) for x in X_val]\n",
    "X_test = [text_prepare(x) for x in X_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each tag and for each word calculate how many times they occur in the train corpus. \n",
    "\n",
    "**Task 2 (WordsTagsCount).** Find 3 most popular tags and 3 most popular words in the train data and submit the results to earn the points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Dictionary of all tags from train corpus with their counts.\n",
    "tags_counts = {}\n",
    "# Dictionary of all words from train corpus with their counts.\n",
    "words_counts = {}\n",
    "\n",
    "######################################\n",
    "######### YOUR CODE HERE #############\n",
    "######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are assuming that *tags_counts* and *words_counts* are dictionaries like `{'some_word_or_tag': frequency}`. After applying the sorting procedure, results will be look like this: `[('most_popular_word_or_tag', frequency), ('less_popular_word_or_tag', frequency), ...]`. The grader gets the results in the following format (two comma-separated strings with line break):\n",
    "\n",
    "    tag1,tag2,tag3\n",
    "    word1,word2,word3\n",
    "\n",
    "Pay attention that in this assignment you should not submit frequencies or some additional information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "most_common_tags = sorted(tags_counts.items(), key=lambda x: x[1], reverse=True)[:3]\n",
    "most_common_words = sorted(words_counts.items(), key=lambda x: x[1], reverse=True)[:3]\n",
    "\n",
    "grader.submit_tag('WordsTagsCount', '%s\\n%s' % (','.join(tag for tag, _ in most_common_tags), \n",
    "                                                ','.join(word for word, _ in most_common_words)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming text to a vector\n",
    "\n",
    "Machine Learning algorithms work with numeric data and we cannot use the provided text data \"as is\". There are many ways to transform text data to numeric vectors. In this task you will try to use two of them.\n",
    "\n",
    "#### Bag of words\n",
    "\n",
    "One of the well-known approaches is a *bag-of-words* representation. To create this transformation, follow the steps:\n",
    "1. Find *N* most popular words in train corpus and numerate them. Now we have a dictionary of the most popular words.\n",
    "2. For each title in the corpora create a zero vector with the dimension equals to *N*.\n",
    "3. For each text in the corpora iterate over words which are in the dictionary and increase by 1 the corresponding coordinate.\n",
    "\n",
    "Let's try to do it for a toy example. Imagine that we have *N* = 4 and the list of the most popular words is \n",
    "\n",
    "    ['hi', 'you', 'me', 'are']\n",
    "\n",
    "Then we need to numerate them, for example, like this: \n",
    "\n",
    "    {'hi': 0, 'you': 1, 'me': 2, 'are': 3}\n",
    "\n",
    "And we have the text, which we want to transform to the vector:\n",
    "\n",
    "    'hi how are you'\n",
    "\n",
    "For this text we create a corresponding zero vector \n",
    "\n",
    "    [0, 0, 0, 0]\n",
    "    \n",
    "And iterate over all words, and if the word is in the dictionary, we increase the value of the corresponding position in the vector:\n",
    "\n",
    "    'hi':  [1, 0, 0, 0]\n",
    "    'how': [1, 0, 0, 0] # word 'how' is not in our dictionary\n",
    "    'are': [1, 0, 0, 1]\n",
    "    'you': [1, 1, 0, 1]\n",
    "\n",
    "The resulting vector will be \n",
    "\n",
    "    [1, 1, 0, 1]\n",
    "   \n",
    "Implement the described encoding in the function *my_bag_of_words* with the size of the dictionary equals to 5000. To find the most common words use train data. You can test your code using the function *test_my_bag_of_words*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DICT_SIZE = 5000\n",
    "WORDS_TO_INDEX = ####### YOUR CODE HERE #######\n",
    "INDEX_TO_WORDS = ####### YOUR CODE HERE #######\n",
    "ALL_WORDS = WORDS_TO_INDEX.keys()\n",
    "\n",
    "def my_bag_of_words(text, words_to_index, dict_size):\n",
    "    \"\"\"\n",
    "        text: a string\n",
    "        dict_size: size of the dictionary\n",
    "        \n",
    "        return a vector which is a bag-of-words representation of 'text'\n",
    "    \"\"\"\n",
    "    result_vector = np.zeros(dict_size)\n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################\n",
    "    return result_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_my_bag_of_words():\n",
    "    words_to_index = {'hi': 0, 'you': 1, 'me': 2, 'are': 3}\n",
    "    examples = ['hi how are you']\n",
    "    answers = [[1, 1, 0, 1]]\n",
    "    for ex, ans in zip(examples, answers):\n",
    "        if (my_bag_of_words(ex, words_to_index, 4) != ans).any():\n",
    "            return \"Wrong answer for the case: '%s'\" % ex\n",
    "    return 'Basic tests are passed.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(test_my_bag_of_words())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now apply the implemented function to all samples (this might take up to a minute):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import sparse as sp_sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_train])\n",
    "X_val_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_val])\n",
    "X_test_mybag = sp_sparse.vstack([sp_sparse.csr_matrix(my_bag_of_words(text, WORDS_TO_INDEX, DICT_SIZE)) for text in X_test])\n",
    "print('X_train shape ', X_train_mybag.shape)\n",
    "print('X_val shape ', X_val_mybag.shape)\n",
    "print('X_test shape ', X_test_mybag.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might notice, we transform the data to sparse representation, to store the useful information efficiently. There are many [types](https://docs.scipy.org/doc/scipy/reference/sparse.html) of such representations, however sklearn algorithms can work only with [csr](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html#scipy.sparse.csr_matrix) matrix, so we will use this one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 3 (BagOfWords).** For the 11th row in *X_train_mybag* find how many non-zero elements it has. In this task the answer (variable *non_zero_elements_count*) should be a number, e.g. 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "row = X_train_mybag[10].toarray()[0]\n",
    "non_zero_elements_count = ####### YOUR CODE HERE #######\n",
    "\n",
    "grader.submit_tag('BagOfWords', str(non_zero_elements_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-IDF\n",
    "\n",
    "The second approach extends the bag-of-words framework by taking into account total frequencies of words in the corpora. It helps to penalize too frequent words and provide better features space. \n",
    "\n",
    "Implement function *tfidf_features* using class [TfidfVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) from *scikit-learn*. Use *train* corpus to train a vectorizer. Don't forget to take a look into the arguments that you can pass to it. We suggest that you filter out too rare words (occur less than in 5 titles) and too frequent words (occur more than in 90% of the titles). Also, use bigrams along with unigrams in your vocabulary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tfidf_features(X_train, X_val, X_test):\n",
    "    \"\"\"\n",
    "        X_train, X_val, X_test — samples        \n",
    "        return TF-IDF vectorized representation of each sample and vocabulary\n",
    "    \"\"\"\n",
    "    # Create TF-IDF vectorizer with a proper parameters choice\n",
    "    # Fit the vectorizer on the train set\n",
    "    # Transform the train, test, and val sets and return the result\n",
    "    \n",
    "    \n",
    "    tfidf_vectorizer = ####### YOUR CODE HERE #######\n",
    "    \n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################\n",
    "    \n",
    "    return X_train, X_val, X_test, tfidf_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have done text preprocessing, always have a look at the results. Be very careful at this step, because the performance of future models will drastically depend on it. \n",
    "\n",
    "In this case, check whether you have c++ or c# in your vocabulary, as they are obviously important tokens in our tags prediction task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_tfidf, X_val_tfidf, X_test_tfidf, tfidf_vocab = tfidf_features(X_train, X_val, X_test)\n",
    "tfidf_reversed_vocab = {i:word for word,i in tfidf_vocab.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "######### YOUR CODE HERE #############"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can't find it, we need to understand how did it happen that we lost them? It happened during the built-in tokenization of TfidfVectorizer. Luckily, we can influence on this process. Get back to the function above and use '(\\S+)' regexp as a *token_pattern* in the constructor of the vectorizer.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, use this transormation for the data and check again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "######### YOUR CODE HERE #############"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MultiLabel classifier\n",
    "\n",
    "As we have noticed before, in this task each example can have multiple tags. To deal with such kind of prediction, we need to transform labels in a binary form and the prediction will be a mask of 0s and 1s. For this purpose it is convenient to use [MultiLabelBinarizer](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html) from *sklearn*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MultiLabelBinarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlb = MultiLabelBinarizer(classes=sorted(tags_counts.keys()))\n",
    "y_train = mlb.fit_transform(y_train)\n",
    "y_val = mlb.fit_transform(y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement the function *train_classifier* for training a classifier. In this task we suggest to use One-vs-Rest approach, which is implemented in [OneVsRestClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html) class. In this approach *k* classifiers (= number of tags) are trained. As a basic classifier, use [LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html). It is one of the simplest methods, but often it performs good enough in text classification tasks. It might take some time, because a number of classifiers to train is large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.linear_model import LogisticRegression, RidgeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_classifier(X_train, y_train):\n",
    "    \"\"\"\n",
    "      X_train, y_train — training data\n",
    "      \n",
    "      return: trained classifier\n",
    "    \"\"\"\n",
    "    \n",
    "    # Create and fit LogisticRegression wraped into OneVsRestClassifier.\n",
    "\n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the classifiers for different data transformations: *bag-of-words* and *tf-idf*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "classifier_mybag = train_classifier(X_train_mybag, y_train)\n",
    "classifier_tfidf = train_classifier(X_train_tfidf, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can create predictions for the data. You will need two types of predictions: labels and scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_val_predicted_labels_mybag = classifier_mybag.predict(X_val_mybag)\n",
    "y_val_predicted_scores_mybag = classifier_mybag.decision_function(X_val_mybag)\n",
    "\n",
    "y_val_predicted_labels_tfidf = classifier_tfidf.predict(X_val_tfidf)\n",
    "y_val_predicted_scores_tfidf = classifier_tfidf.decision_function(X_val_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now take a look at how classifier, which uses TF-IDF, works for a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_val_pred_inversed = mlb.inverse_transform(y_val_predicted_labels_tfidf)\n",
    "y_val_inversed = mlb.inverse_transform(y_val)\n",
    "for i in range(3):\n",
    "    print('Title:\\t{}\\nTrue labels:\\t{}\\nPredicted labels:\\t{}\\n\\n'.format(\n",
    "        X_val[i],\n",
    "        ','.join(y_val_inversed[i]),\n",
    "        ','.join(y_val_pred_inversed[i])\n",
    "    ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we would need to compare the results of different predictions, e.g. to see whether TF-IDF transformation helps or to try different regularization techniques in logistic regression. For all these experiments, we need to setup evaluation procedure. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "To evaluate the results we will use several classification metrics:\n",
    " - [Accuracy](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)\n",
    " - [F1-score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)\n",
    " - [Area under ROC-curve](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html)\n",
    " - [Area under precision-recall curve](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score) \n",
    " \n",
    "Make sure you are familiar with all of them. How would you expect the things work for the multi-label scenario? Read about micro/macro/weighted averaging following the sklearn links provided above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import roc_auc_score \n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement the function *print_evaluation_scores* which calculates and prints to stdout:\n",
    " - *accuracy*\n",
    " - *F1-score macro/micro/weighted*\n",
    " - *Precision macro/micro/weighted*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_evaluation_scores(y_val, predicted):\n",
    "    \n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print('Bag-of-words')\n",
    "print_evaluation_scores(y_val, y_val_predicted_labels_mybag)\n",
    "print('Tfidf')\n",
    "print_evaluation_scores(y_val, y_val_predicted_labels_tfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might also want to plot some generalization of the [ROC curve](http://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc) for the case of multi-label classification. Provided function *roc_auc* can make it for you. The input parameters of this function are:\n",
    " - true labels\n",
    " - decision functions scores\n",
    " - number of classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from metrics import roc_auc\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_classes = len(tags_counts)\n",
    "roc_auc(y_val, y_val_predicted_scores_mybag, n_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_classes = len(tags_counts)\n",
    "roc_auc(y_val, y_val_predicted_scores_tfidf, n_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 4 (MultilabelClassification).** Once we have the evaluation set up, we suggest that you experiment a bit with training your classifiers. We will use *F1-score weighted* as an evaluation metric. Our recommendation:\n",
    "- compare the quality of the bag-of-words and TF-IDF approaches and chose one of them.\n",
    "- for the chosen one, try *L1* and *L2*-regularization techniques in Logistic Regression with different coefficients (e.g. C equal to 0.1, 1, 10, 100).\n",
    "\n",
    "You also could try other improvements of the preprocessing / model, if you want. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "######################################\n",
    "######### YOUR CODE HERE #############\n",
    "######################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you are happy with the quality, create predictions for *test* set, which you will submit to Coursera."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_predictions = ######### YOUR CODE HERE #############\n",
    "test_pred_inversed = mlb.inverse_transform(test_predictions)\n",
    "\n",
    "test_predictions_for_submission = '\\n'.join('%i\\t%s' % (i, ','.join(row)) for i, row in enumerate(test_pred_inversed))\n",
    "grader.submit_tag('MultilabelClassification', test_predictions_for_submission)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis of the most important features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, it is usually a good idea to look at the features (words or n-grams) that are used with the largest weigths in your logistic regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement the function *print_words_for_tag* to find them. Get back to sklearn documentation on [OneVsRestClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html) and [LogisticRegression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_words_for_tag(classifier, tag, tags_classes, index_to_words, all_words):\n",
    "    \"\"\"\n",
    "        classifier: trained classifier\n",
    "        tag: particular tag\n",
    "        tags_classes: a list of classes names from MultiLabelBinarizer\n",
    "        index_to_words: index_to_words transformation\n",
    "        all_words: all words in the dictionary\n",
    "        \n",
    "        return nothing, just print top 5 positive and top 5 negative words for current tag\n",
    "    \"\"\"\n",
    "    print('Tag:\\t{}'.format(tag))\n",
    "    \n",
    "    # Extract an estimator from the classifier for the given tag.\n",
    "    # Extract feature coefficients from the estimator. \n",
    "    \n",
    "    ######################################\n",
    "    ######### YOUR CODE HERE #############\n",
    "    ######################################\n",
    "    \n",
    "    top_positive_words = # top-5 words sorted by the coefficiens.\n",
    "    top_negative_words = # bottom-5 words  sorted by the coefficients.\n",
    "    print('Top positive words:\\t{}'.format(', '.join(top_positive_words)))\n",
    "    print('Top negative words:\\t{}\\n'.format(', '.join(top_negative_words)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print_words_for_tag(classifier_tfidf, 'c', mlb.classes, tfidf_reversed_vocab, ALL_WORDS)\n",
    "print_words_for_tag(classifier_tfidf, 'c++', mlb.classes, tfidf_reversed_vocab, ALL_WORDS)\n",
    "print_words_for_tag(classifier_tfidf, 'linux', mlb.classes, tfidf_reversed_vocab, ALL_WORDS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authorization & Submission\n",
    "To submit assignment parts to Cousera platform, please, enter your e-mail and token into variables below. You can generate token on this programming assignment page. <b>Note:</b> Token expires 30 minutes after generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grader.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "STUDENT_EMAIL = # EMAIL \n",
    "STUDENT_TOKEN = # TOKEN \n",
    "grader.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to submit these answers, run cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grader.submit(STUDENT_EMAIL, STUDENT_TOKEN)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}