L4.Discovering interpretable features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/ML-Challenge/week4-unsupervised-learning/blob/master/L4.Discovering%20interpretable%20features.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson, we'll learn about a dimension reduction technique called \"Non-negative matrix factorization\" (\"NMF\") that expresses samples as combinations of interpretable parts. For example, it expresses documents as combinations of topics, and images in terms of commonly occurring visual patterns. We'll also learn to use NMF to build recommender systems that can find us similar articles to read, or musical artists that match our listening history!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:43.400823Z",
     "start_time": "2020-02-15T19:00:43.387719Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Download lesson datasets\n",
    "# Required if you're using Google Colab\n",
    "!wget \"https://github.com/ML-Challenge/week4-unsupervised-learning/raw/master/datasets.zip\"\n",
    "!unzip -o datasets.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.057125Z",
     "start_time": "2020-02-15T19:00:43.403715Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import utils\n",
    "# We'll be using this module throughout the lesson\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.276482Z",
     "start_time": "2020-02-15T19:00:44.058977Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import dependencies\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "# and setting the size of all plots.\n",
    "plt.rcParams['figure.figsize'] = [11, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Non-negative matrix factorization (NMF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "NMF stands for non-negative matrix factorization. NMF, like PCA, is a dimension reduction technique. In contrast to PCA, however, NMF models are interpretable. This means an NMF models are easier to understand, and much easier for us to explain to others. NMF can't be applied to every dataset, however. It is required that the sample features be non-negative, so greater than or equal to 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Interpretable parts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "NMF achieves its interpretability by decomposing samples as sums of their parts. For example, NMF decomposes documents as combinations of the common topics, and images as combinations of common patterns.\n",
    "\n",
    "![Parts](assets/4-1.png)\n",
    "\n",
    "We'll learn about both these examples in detail later. For now, let's focus on getting started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Using scikit-learn NMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "NMF is available in scikit learn, and follows the same fit/transform pattern as PCA. However, unlike PCA, the desired number of components must always be specified. NMF works both with numpy arrays and sparse arrays in the `csr_matrix` format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's see an application of NMF to a toy example of a word-frequency array. In this toy dataset, there are only 4 words in the vocabulary, and these correspond to the four columns of the word-frequency array.\n",
    "\n",
    "![Toy dataset](assets/4-2.png)\n",
    "\n",
    "Each row represents a document, and the entries of the array measure the frequency of each word in the document using what's known as `tf-idf`. \n",
    "\n",
    "* `tf` is the frequency of the word in the document.\n",
    "So if 10% of the words in the document are `datacamp`, then the tf of datacamp for that document is 0.1. \n",
    "* `idf` is a weighting scheme that reduces the influence of frequent words like `the`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's now see how to use NMF in Python. Firstly, import NMF.\n",
    "\n",
    "```\n",
    "from sklearn.decomposition import NMF\n",
    "```\n",
    "\n",
    "Create a model, specifying the desired number of components. Let's specify 2.\n",
    "\n",
    "```\n",
    "model = NMF(n_components=2)\n",
    "```\n",
    "\n",
    "Fit the model to the samples, then use the fit model to perform the transformation.\n",
    "\n",
    "```\n",
    "model.fit(samples)\n",
    "\n",
    "nmf_features = model.transform(samples)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**NMF components**\n",
    "\n",
    "Just as PCA has principal components, NMF has components which it learns from the samples, and as with PCA, the dimension of the components is the same as the dimension of the samples. In our example, for instance, there are 2 components, and they live in 4 dimensional space, corresponding to the 4 words in the vocabulary. The entries of the NMF components are always non-negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**NMF features**\n",
    "\n",
    "The NMF feature values are non-negative, as well. As we saw with PCA, our transformed data in this example will have two columns, corresponding to our two new features. The features and the components of an NMF model can be combined to approximately reconstruct the original data samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Reconstrution of a sample**\n",
    "\n",
    "If we multiply each NMF components by the corresponding NMF feature value, and add up each column, we get something very close to the original sample. So a sample can be reconstructed by multiplying the NMF components by the NMF feature values of the sample, and adding up. This calculation also can be expressed as what is known as a product of matrices. We won't be using that point of view, but that's where the matrix factorisation, or MF, in NMF comes from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Finally, remember that NMF can only be applied to arrays of non-negative data, such as word-frequency arrays. In the next section, we construct another example by encoding collections of images as non-negative arrays. There are many other great examples as well, such as arrays encoding audio spectrograms, and arrays representing the purchase histories on e-Commerce sites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## NMF applied to Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's apply NMF, using the tf-idf word-frequency array of Wikipedia articles, given as a csr matrix `articles`. Here, we fit the model and transform the articles. In the next example, we'll explore the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.718351Z",
     "start_time": "2020-02-15T19:00:44.278477Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import NMF\n",
    "from sklearn.decomposition import NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.734285Z",
     "start_time": "2020-02-15T19:00:44.720295Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create an NMF instance: model\n",
    "model = NMF(n_components=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.844962Z",
     "start_time": "2020-02-15T19:00:44.736252Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Fit the model to articles\n",
    "model.fit(utils.articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.860919Z",
     "start_time": "2020-02-15T19:00:44.848951Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Transform the articles: nmf_features\n",
    "nmf_features = model.transform(utils.articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.892834Z",
     "start_time": "2020-02-15T19:00:44.864909Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print the NMF features\n",
    "print(nmf_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## NMF features of the Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now we will explore the NMF features we created in the previous example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.908791Z",
     "start_time": "2020-02-15T19:00:44.895826Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a pandas DataFrame: df\n",
    "df = pd.DataFrame(nmf_features, index=utils.titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.923751Z",
     "start_time": "2020-02-15T19:00:44.911783Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print the row for 'Anne Hathaway'\n",
    "print(df.loc['Anne Hathaway'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.939708Z",
     "start_time": "2020-02-15T19:00:44.926743Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print the row for 'Denzel Washington'\n",
    "print(df.loc['Denzel Washington'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "When investigating the features, notice that for both actors, the NMF feature 3 has by far the highest value. This means that both articles are reconstructed using mainly the 3rd NMF component. In the next section, we'll see why: NMF components represent topics (for instance, acting!)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# NMF learns interpretable parts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this section, we'll learn that the components of NMF represent patterns that frequently occur in the samples. Let's consider a concrete example, where scientific articles are represented by their word frequencies. There are 20000 articles, and 800 words. So the array has 800 columns.\n",
    "\n",
    "![Articles](assets/4-3.png)\n",
    "\n",
    "Let's fit an NMF model with 10 components to the articles.\n",
    "\n",
    "```\n",
    "from sklearn.decomposition import NMF\n",
    "\n",
    "nmf = NMF(n_components=10)\n",
    "\n",
    "nmf.fit(articles)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The 10 components are stored as the 10 rows of a 2-dimensional numpy array. The rows, or components, live in an 800-dimensional space - there is one dimension for each of the words.\n",
    "\n",
    "![NMF Components](assets/4-4.png)\n",
    "\n",
    "Aligning the words of our vocabulary with the columns of the NMF components allows them to be interpreted.  Choosing a component, such as this one, and looking at which words have the highest values, we see that they fit a theme: the words are 'species', 'plant', 'plants', 'genetic', 'evolution' and 'life'.\n",
    "\n",
    "![NMF Component](assets/4-6.png)\n",
    "\n",
    "The same happens if any other component is considered.\n",
    "\n",
    "![NMF Component](assets/4-7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "So if NMF is applied to documents, then the components correspond to topics, and the NMF features reconstruct the documents from the topics.  If NMF is applied to a collection of images, on the other hand, then. the NMF components represent patterns that frequently occur in the images.\n",
    "In this example, for instance, NMF decomposes images from an LCD display into the individual cells of the display.\n",
    "\n",
    "![NMF Images](assets/4-8.png)\n",
    "\n",
    "This example we'll investigate for in the examples. To do this, we'll need to know how to represent a collection of images as a non-negative array. \n",
    "\n",
    "**Grayscale images**\n",
    "\n",
    "An image in which all the pixels are shades of gray ranging from black to white is called a grayscale image. Since there are only shades of grey, a grayscale image can be encoded by the brightness of every pixel. Representing the brightness as a number between 0 and 1, where 0 is totally black and 1 is totally white, the image can be represented as 2-dimensional array of numbers.\n",
    "\n",
    "![NMF Images](assets/4-9.png)\n",
    "\n",
    "Here, for example, is a grayscale photo of the moon!\n",
    "\n",
    "![Moon](assets/4-10.png)\n",
    "\n",
    "These 2-dimensional arrays of numbers can then be flattened by enumerating the entries. For instance, we could read-off the values row-by-row, from left-to-right, and from top-to-bottom.\n",
    "\n",
    "![Moon](assets/4-12.png)\n",
    "\n",
    "The grayscale image is now represented by a flat array of non-negative numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Encoding a collection of images**\n",
    "\n",
    "A collection of grayscale images of the same size can thus be encoded as a 2-dimensional\n",
    "array, in which each row represents an image as a flattened array, and each column represents a pixel. Viewing the images as samples, and the pixels as features, we see that the data is arranged similarly to the word frequency array. \n",
    "\n",
    "![Collection](assets/4-13.png)\n",
    "\n",
    "Indeed, the entries of this array are non-negative, so NMF can be used to learn the parts of the images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**Visualizing samples**\n",
    "\n",
    "It's difficult to visualize an image by just looking at the flattened array. To recover the image, use the reshape method of the sample, specifying the dimensions of the original image as a tuple. This yields the 2-dimensional array of pixel brightnesses.\n",
    "\n",
    "```\n",
    "bitmap = sample.reshape((2,3))\n",
    "```\n",
    "\n",
    "To display the corresponding image, import `pyplot`, and pass the 2-dimensional array to the plt dot `imshow` function.\n",
    "\n",
    "```\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## NMF learns topics of documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We learned that when NMF is applied to documents, the components correspond to topics of documents, and the NMF features reconstruct the documents from the topics. Let's verify this for the NMF model that we built earlier using the Wikipedia articles. Previously, we saw that the 3rd NMF feature value was high for the articles about actors Anne Hathaway and Denzel Washington. In this example, let's identify the topic of the corresponding NMF component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.955666Z",
     "start_time": "2020-02-15T19:00:44.942701Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame: components_df\n",
    "components_df = pd.DataFrame(model.components_, columns=utils.words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.970627Z",
     "start_time": "2020-02-15T19:00:44.957660Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print the shape of the DataFrame\n",
    "print(components_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:44.986583Z",
     "start_time": "2020-02-15T19:00:44.973618Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Select row 3: component\n",
    "component = components_df.iloc[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.002541Z",
     "start_time": "2020-02-15T19:00:44.989575Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print result of nlargest\n",
    "print(component.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Take a moment to recognise the topic that the articles about Anne Hathaway and Denzel Washington have in common"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Explore the LED digits dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In the following examples, we'll use NMF to decompose grayscale images into their commonly occurring patterns. Firstly, we'll explore the image dataset and see how it is encoded as an array. We are given 100 images as a 2D array 'leds', where each row represents a single 13x8 image. The images in our dataset are pictures of a LED digital display."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.017500Z",
     "start_time": "2020-02-15T19:00:45.004534Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Select the 0th row: digit\n",
    "digit = utils.leds[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.032462Z",
     "start_time": "2020-02-15T19:00:45.020493Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print digit\n",
    "print(digit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.048418Z",
     "start_time": "2020-02-15T19:00:45.034455Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Reshape digit to a 13x8 array: bitmap\n",
    "bitmap = digit.reshape(13,8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.063378Z",
     "start_time": "2020-02-15T19:00:45.051409Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print bitmap\n",
    "print(bitmap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.270847Z",
     "start_time": "2020-02-15T19:00:45.065373Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Use plt.imshow to display bitmap\n",
    "plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## NMF learns the parts of images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now let's use what we've learned about NMF to decompose the digits dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.286806Z",
     "start_time": "2020-02-15T19:00:45.272818Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def show_as_image(digit):\n",
    "    bitmap = digit.reshape((13, 8))\n",
    "    plt.figure()\n",
    "    plt.imshow(bitmap, cmap='gray', interpolation='nearest')\n",
    "    plt.colorbar()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.302777Z",
     "start_time": "2020-02-15T19:00:45.290769Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create an NMF model: model\n",
    "model = NMF(n_components=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:45.318382Z",
     "start_time": "2020-02-15T19:00:45.306734Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(utils.leds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.744953Z",
     "start_time": "2020-02-15T19:00:45.320376Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.760878Z",
     "start_time": "2020-02-15T19:00:46.746917Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Assign the 0th row of features: digit_features\n",
    "digit_features = features[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.776834Z",
     "start_time": "2020-02-15T19:00:46.763868Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print digit_features\n",
    "print(digit_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Take a moment to look through the plots and notice how NMF has expressed the digit as a sum of the components!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## PCA doesn't learn parts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Unlike NMF, PCA doesn't learn the parts of things. Its components do not correspond to topics (in the case of documents) or to parts of images, when trained on images. Let's verify this by inspecting the components of a PCA model fit to the dataset of LED digit images from the previous example. The images are available as a 2D array `leds`. Also available is a modified version of the `show_as_image()` function which colors a pixel red if the value is negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.792791Z",
     "start_time": "2020-02-15T19:00:46.780826Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def show_as_image(vector):\n",
    "    \"\"\"\n",
    "    Given a 1d vector representing an image, display that image in \n",
    "    black and white.  If there are negative values, then use red for \n",
    "    that pixel.\n",
    "    \"\"\"\n",
    "    bitmap = vector.reshape((13, 8))  # make a square array\n",
    "    bitmap /= np.abs(vector).max()  # normalise\n",
    "    bitmap = bitmap[:,:,np.newaxis]\n",
    "    rgb_layers = [np.abs(bitmap)] + [bitmap.clip(0)] * 2\n",
    "    rgb_bitmap = np.concatenate(rgb_layers, axis=-1)\n",
    "    plt.imshow(rgb_bitmap, interpolation='nearest')\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.807752Z",
     "start_time": "2020-02-15T19:00:46.795802Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.823714Z",
     "start_time": "2020-02-15T19:00:46.809749Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a PCA instance: model\n",
    "model = PCA(n_components=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:46.839699Z",
     "start_time": "2020-02-15T19:00:46.825705Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply fit_transform to samples: features\n",
    "features = model.fit_transform(utils.leds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.282586Z",
     "start_time": "2020-02-15T19:00:46.841660Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Call show_as_image on each component\n",
    "for component in model.components_:\n",
    "    show_as_image(component)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Notice that the components of PCA do not represent meaningful parts of images of LED digits!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building recommender systems using NMF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that we are engineers at a large online newspaper. We've been given the task of recommending articles that are similar to the article currently being read by a customer. Given an article, how can we find articles that have similar topics? In this section, we'll learn how to solve this problem, and others like it, by using NMF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Strategy**\n",
    "\n",
    "Our strategy for solving this problem is to apply NMF to the word-frequency array of the articles, and to use the resulting NMF features. We learned in the previous sections these NMF features describe the topic mixture of an article. So similar articles will have similar NMF features. But how can two articles be compared using their NMF features? Before answering this question, let's set the scene by doing the first step.\n",
    "\n",
    "We are given a word frequency array articles corresponding to the collection of newspaper articles in question. Import NMF, create the model, and use the fit_transform method to obtain the transformed articles.\n",
    "\n",
    "```\n",
    "from sklearn.decomposition import NMF\n",
    "\n",
    "nmf = NMF(n_components=6)\n",
    "\n",
    "nmf_features = nmf.fit_transform(articles)\n",
    "```\n",
    "\n",
    "Now we've got NMF features for every article, given by the columns of the new array. Now we need to define how to compare articles using their NMF features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version of articles**\n",
    "\n",
    "Similar documents have similar topics, but it isn't always the case that the NMF feature values are exactly the same. For instance, one version of a document might use very direct language, whereas other versions might interleave the same content with meaningless chatter.\n",
    "\n",
    "![Chatter](assets/4-14.png)\n",
    "\n",
    "Meaningless chatter reduces the frequency of the topic words overall, which reduces the values of the NMF features representing the topics. However, on a scatter plot of the NMF features, all these versions lie on a single line passing through the origin.\n",
    "\n",
    "![Scatter plot](assets/4-15.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cosine similarity**\n",
    "\n",
    "For this reason, when comparing two documents, it's a good idea to compare these lines. We'll compare them using what is known as the cosine similarity, which uses the angle between the two lines.\n",
    "\n",
    "![Cosine](assets/4-16.png)\n",
    "\n",
    "Higher values indicate greater similarity. The technical definition of the cosine similarity is out the scope of this lesson, but we've already gained an intuition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculating the cosine similarities**\n",
    "\n",
    "Let's see now how to compute the cosine similarity. Firstly, import the normalize function, and apply it to the array of all NMF features.\n",
    "\n",
    "```\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "norm_features = normalize(nmf_features)\n",
    "```\n",
    "\n",
    "Now select the row corresponding to the current article, and pass it to the dot method of the array of all normalized features. This results in the cosine similarities.\n",
    "\n",
    "```\n",
    "current_article = norm_features[23, :] # if has index 23\n",
    "\n",
    "similarities = norm_features.dot(current_article)\n",
    "```\n",
    "\n",
    "With the help of a pandas DataFrame, we can label the similarities with the article titles. Start by importing pandas.\n",
    "\n",
    "```\n",
    "import pandas as pd\n",
    "```\n",
    "\n",
    "After normalizing the NMF features, create a DataFrame whose rows are the normalized features, using the titles as an index.\n",
    "\n",
    "```\n",
    "df = pd.DataFrame(norm_features, index=titles)\n",
    "```\n",
    "\n",
    "Now use the `.loc()` method of the dataframe to select the normalized feature values for the current article, using its title 'Dog bites man'.\n",
    "\n",
    "```\n",
    "current_article = df.loc['Dog bites man']\n",
    "```\n",
    "\n",
    "Calculate the cosine similarities using the `.dot()` method of the dataframe.\n",
    "\n",
    "```\n",
    "similarities = df.dot(current_article)\n",
    "```\n",
    "\n",
    "Finally, use the nlargest method of the resulting pandas Series to find the articles with the highest cosine similarity.\n",
    "\n",
    "```\n",
    "print(similarities.nlargest())\n",
    "```\n",
    "| Article                        | Similarity |\n",
    "| ------------------------------ | ---------: |\n",
    "| Dog Bites man                  | 1.000000   |\n",
    "| Hound mauls cat                | 0.979946   |\n",
    "| Pets go wild!                  | 0.979708   |\n",
    "| Dachshunds are dangerous       | 0.949641   |\n",
    "| Our streets are no longer safe | 0.900474   |\n",
    "\n",
    "We see that all of them are concerned with 'domestic animals' and/or 'danger'!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Which articles are similar to 'Cristiano Ronaldo'?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We learned how to use NMF features and the cosine similarity to find similar articles. Let's apply this to our NMF model for popular Wikipedia articles, by finding the articles most similar to the article about the footballer Cristiano Ronaldo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.298508Z",
     "start_time": "2020-02-15T19:00:47.284544Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.313467Z",
     "start_time": "2020-02-15T19:00:47.302502Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Normalize the NMF features: norm_features\n",
    "norm_features = normalize(nmf_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.329425Z",
     "start_time": "2020-02-15T19:00:47.317456Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=utils.titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.344385Z",
     "start_time": "2020-02-15T19:00:47.332421Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Select the row corresponding to 'Cristiano Ronaldo': article\n",
    "article = df.loc['Cristiano Ronaldo']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.360375Z",
     "start_time": "2020-02-15T19:00:47.347385Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Compute the dot products: similarities\n",
    "similarities = df.dot(article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.376332Z",
     "start_time": "2020-02-15T19:00:47.362342Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Display those with the largest cosine similarity\n",
    "print(similarities.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Recommend musical artists part I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this example and the next, we'll use what we've learned about NMF to recommend popular music artists! We are given a sparse array `artists`, the rows correspond to artists and the column to users. The entries give the number of times each artist was listened to by each user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this example, we'll build a pipeline and transform the array into normalized NMF features. The first step in the pipeline, `MaxAbsScaler`, transforms the data so that all users have the same influence on the model, regardless of how many different artists they've listened to. In the next example, we'll use the resulting normalized NMF features for recommendation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.392290Z",
     "start_time": "2020-02-15T19:00:47.378294Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.decomposition import NMF\n",
    "from sklearn.preprocessing import Normalizer, MaxAbsScaler\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.408239Z",
     "start_time": "2020-02-15T19:00:47.395248Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a MaxAbsScaler: scaler\n",
    "scaler = MaxAbsScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.424198Z",
     "start_time": "2020-02-15T19:00:47.410210Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create an NMF model: nmf\n",
    "nmf = NMF(n_components=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.440154Z",
     "start_time": "2020-02-15T19:00:47.426167Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a Normalizer: normalizer\n",
    "normalizer = Normalizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.456086Z",
     "start_time": "2020-02-15T19:00:47.442124Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, nmf, normalizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:00:47.501964Z",
     "start_time": "2020-02-15T19:00:47.458081Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply fit_transform to artists: norm_features\n",
    "norm_features = pipeline.fit_transform(utils.artists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Now that we've computed the normalized NMF features, we'll use them in the next example to recommend musical artists!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommend musical artists part II"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we are big fans of Dr. Dre - which other musicial artists might we like? Let's use the NMF features from the previous example and the cosine similarity to find similar musical artists. The names of the musical artists are available as the list `artist_names`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:02:31.266697Z",
     "start_time": "2020-02-15T19:02:31.259737Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame: df\n",
    "df = pd.DataFrame(norm_features, index=utils.artist_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:02:34.888896Z",
     "start_time": "2020-02-15T19:02:34.880940Z"
    }
   },
   "outputs": [],
   "source": [
    "# Select row of 'Dr. Dre': artist\n",
    "artist = df.loc['Dr. Dre']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:02:35.999807Z",
     "start_time": "2020-02-15T19:02:35.986665Z"
    }
   },
   "outputs": [],
   "source": [
    "# Compute cosine similarities: similarities\n",
    "similarities = df.dot(artist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T19:02:37.033753Z",
     "start_time": "2020-02-15T19:02:37.020798Z"
    }
   },
   "outputs": [],
   "source": [
    "# Display those with highest cosine similarity\n",
    "print(similarities.nlargest())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Week 4 - Unsupervised Learning](https://radu-enuca.gitbook.io/ml-challenge/unsupervised-learning)**\n",
    "\n",
    "*Have questions or comments? Visit the ML Challenge Mattermost Channel.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}