L3.Decorrelating data and dimension reduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/ML-Challenge/week4-unsupervised-learning/blob/master/L3.Decorrelating%20data%20and%20dimension%20reduction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dimension reduction summarizes a dataset using its common occuring patterns. In this lesson, we'll learn about the most fundamental of dimension reduction techniques, \"Principal Component Analysis\" (\"PCA\"). PCA is often used before supervised learning to improve model performance and generalization. It can also be useful for unsupervised learning. For example, we'll employ a variant of PCA that will allow us to cluster Wikipedia articles by their content!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:45.729289Z",
     "start_time": "2020-02-15T17:33:45.716324Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Download lesson datasets\n",
    "# Required if you're using Google Colab\n",
    "!wget \"https://github.com/ML-Challenge/week4-unsupervised-learning/raw/master/datasets.zip\"\n",
    "!unzip -o datasets.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:46.387661Z",
     "start_time": "2020-02-15T17:33:45.732279Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import utils\n",
    "# We'll be using this module throughout the lesson\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:46.607084Z",
     "start_time": "2020-02-15T17:33:46.389641Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import dependencies\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "# and setting the size of all plots.\n",
    "plt.rcParams['figure.figsize'] = [11, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Visualizing the PCA transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In the next two lessons we'll learn techniques for dimension reduction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Dimension reduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Dimension reduction finds patterns in data, and uses these patterns to re-express it in a compressed form. This makes subsequent computation with the data much more efficient, and this can be a big deal in a world of big datasets. However, the most important function of dimension reduction is to reduce a dataset to its bare bones, discarding noisy features that cause big problems for supervised learning tasks like regression and classification. In many real-world applications, it's dimension reduction that makes prediction possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Principal Component Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this lesson, we'll learn about the most fundamental of dimension reduction techniques. It's called Principal Component Analysis, or `PCA` for short. PCA performs dimension reduction in two steps, and the first one, called de-correlation, doesn't change the dimension of the data at all. It's this first step that we'll focus on next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this first step, PCA rotates the samples so that they are aligned with the coordinate axes. In fact, it does more than this: PCA also shifts the samples so that they have mean zero. These scatter plots show the effect of PCA applied to two features of the wine dataset.\n",
    "\n",
    "![PCA](assets/3-1.png)\n",
    "\n",
    "Notice that no information is lost - this is true no matter how many features our dataset has."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "`scikit-learn` has an implementation of PCA, and it has fit and transform methods just like StandardScaler: \n",
    "\n",
    "* The `fit` method learns how to shift and how to rotate the samples, but doesn't actually change them. \n",
    "* The `transform` method, on the other hand, applies the transformation that fit learned. In particular, the transform method can be applied to new, unseen samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Using scikit-learn PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's see PCA in action on the some features of the wine dataset. Firstly, import PCA.\n",
    "\n",
    "```\n",
    "from sklearn.decomposition import PCA\n",
    "```\n",
    "\n",
    "Now create a PCA object, and fit it to the samples.\n",
    "\n",
    "```\n",
    "model = PCA()\n",
    "model.fit(utils.wine)\n",
    "```\n",
    "\n",
    "Then use the fit PCA object to transform the samples. This returns a new array of transformed samples.\n",
    "\n",
    "```\n",
    "transformed = model.transform(utils.wine)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## PCA features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This new array has the same number of rows and columns as the original sample array. In particular, there is one row for each transformed sample. The columns of the new array correspond to `PCA features`, just as the original features corresponded to columns of the original array.\n",
    "\n",
    "It is often the case that the features of a dataset are correlated. This is the case with many of the features of the wine dataset, for instance. However, PCA, due to the rotation it performs, de-correlates the data, in the sense that the columns of the transformed array are not linearly correlated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Pearson correlation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Linear correlation can be measured with the Pearson correlation. It takes values between -1 and 1, where larger values indicate a stronger correlation, and 0 indicates no linear correlation. Here are some examples of features with varying degrees of correlation.\n",
    "\n",
    "![Decorrelated](assets/3-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Principal components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Finally, PCA is called principal component analysis because it learns the principal components of the data. These are the directions in which the samples vary the most, depicted here in red. \n",
    "\n",
    "![PCs](assets/3-3.png)\n",
    "\n",
    "It is the principal components that PCA aligns with the co-ordinate axes. After a PCA model has been fit, the principal components are available as the `components_` attribute. This is numpy array with one row for each principal component."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Correlated data in nature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We have an array of grains giving the width and length of samples of grain. We suspect that width and length will be correlated. To confirm this, let's make a scatter plot of width vs length and measure their Pearson correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:46.905454Z",
     "start_time": "2020-02-15T17:33:46.608076Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from scipy.stats import pearsonr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:46.921352Z",
     "start_time": "2020-02-15T17:33:46.906398Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Assign the 0th column of grains: width\n",
    "width = utils.grains[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:46.936313Z",
     "start_time": "2020-02-15T17:33:46.923346Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Assign the 1st column of grains: length\n",
    "length = utils.grains[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.122814Z",
     "start_time": "2020-02-15T17:33:46.939320Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Scatter plot width vs length\n",
    "plt.scatter(width, length)\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.138772Z",
     "start_time": "2020-02-15T17:33:47.124808Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Calculate the Pearson correlation\n",
    "correlation, pvalue = pearsonr(width, length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.154729Z",
     "start_time": "2020-02-15T17:33:47.141764Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Display the correlation\n",
    "print(correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "As we would expect, the width and length of the grain samples are highly correlated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Decorrelating the grain measurements with PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We observed in the previous example that the width and length measurements of the grain are correlated. Now, we'll use PCA to decorrelate these measurements, then plot the decorrelated points and measure their Pearson correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.293358Z",
     "start_time": "2020-02-15T17:33:47.158719Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import PCA\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.308318Z",
     "start_time": "2020-02-15T17:33:47.295351Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create PCA instance: model\n",
    "model = PCA()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.324273Z",
     "start_time": "2020-02-15T17:33:47.311309Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply the fit_transform method of model to grains: pca_features\n",
    "pca_features = model.fit_transform(utils.grains)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.340230Z",
     "start_time": "2020-02-15T17:33:47.326268Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Assign 0th column of pca_features: xs\n",
    "xs = pca_features[:,0]\n",
    "\n",
    "# Assign 1st column of pca_features: ys\n",
    "ys = pca_features[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.557652Z",
     "start_time": "2020-02-15T17:33:47.343223Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Scatter plot xs vs ys\n",
    "plt.scatter(xs, ys)\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.573610Z",
     "start_time": "2020-02-15T17:33:47.559645Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Calculate the Pearson correlation of xs and ys\n",
    "correlation, pvalue = pearsonr(xs, ys)\n",
    "\n",
    "# Display the correlation\n",
    "print(correlation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Excellent! We've successfully decorrelated the grain measurements with PCA!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Intrinsic dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Consider a dataset with two features: `latitude` and `longitude`. These two features might track the flight of an airplane, for example. This dataset is two dimensional, yet it turns out that it can be closely approximated using only one feature: the displacement along the flight path. This dataset is intrinsically one dimensional.\n",
    "\n",
    "![Flight Path](assets/3-4.png)\n",
    "\n",
    "The intrinsic dimension of a dataset is the number of features required to approximate it. The intrinsic dimension informs dimension reduction, because it tells us how much a dataset can be compressed.\n",
    "\n",
    "Next, we'll gain a solid understanding of the intrinsic dimension, and be able to use PCA to identify it in real-world datasets that have thousands of features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To better illustrate the intrinsic dimension, let's consider an example dataset containing only some of the samples from the iris dataset. Specifically, let's take three measurements from the iris versicolor samples: `sepal length`, `sepal width`, and `petal width`. So each sample is represented as a point in 3-dimensional space.\n",
    "\n",
    "However, if we make a 3d scatter plot of the samples, we see that they all lie very close to a flat, 2-dimensional sheet. This means that the data can be approximated by using only two co-ordinates, without losing much information. So this dataset has intrinsic dimension 2.\n",
    "\n",
    "![3D Scatter](assets/3-5.png)\n",
    "\n",
    "But scatter plots are only possible if there are 3 features or less. So how can the intrinsic dimension be identified, even if there are many features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This is where PCA is really helpful. The intrinsic dimension can be identified by counting the PCA features that have high variance. To see how, let's see what happens when PCA is applied to the dataset of versicolor samples.\n",
    "\n",
    "![PCA 3D](assets/3-6.png)\n",
    "\n",
    "PCA rotates and shifts the samples to align them with the co-ordinate axes. This expresses the samples using three PCA features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The PCA features are in a special order. Here is a bar graph showing the variance of each of the PCA features.\n",
    "\n",
    "![PCA Bar](assets/3-7.png)\n",
    "\n",
    "As we can see, each PCA feature has less variance than the last, and in this case the last PCA feature has very low variance. This agrees with the scatter plot of the PCA features, where the samples don't vary much in the vertical direction. In the other two directions, however, the variance is apparent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The intrinsic dimension is the number of PCA features that have significant variance. In our example, only the first two PCA features have significant variance. So this dataset has intrinsic dimension 2, which agrees with what we observed when inspecting the scatter plot.\n",
    "\n",
    "![PCA Reduction](assets/3-8.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's see how to plot the variances of the PCA features in practice. Firstly, make the necessary imports.\n",
    "\n",
    "```\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "```\n",
    "\n",
    "Then create a PCA model, and fit it to the samples.\n",
    "\n",
    "```\n",
    "pca = PCA()\n",
    "pca.fit(samples)\n",
    "```\n",
    "\n",
    "Now create a range enumerating the PCA features,\n",
    "\n",
    "```\n",
    "features = range(pca.n_components_)\n",
    "```\n",
    "\n",
    "and make a bar plot of the variances; the variances are available as the `explained_variance_` attribute of the PCA model.\n",
    "\n",
    "```\n",
    "plt.bar(features, pca.explained_variance_)\n",
    "\n",
    "plt.xticks(features)\n",
    "plt.ylabel('variance')\n",
    "plt.xlabel('PCA feature')\n",
    "\n",
    "plt.show()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The intrinsic dimension is a useful idea that helps to guide dimension reduction. However, it is not always unambiguous. Here is a graph of the variances of the PCA features for the wine dataset.\n",
    "\n",
    "![Wine PCA](assets/3-9.png)\n",
    "\n",
    "We could argue for an intrinsic dimension of 2, of 3, or even more, depending upon the threshold we chose. In the next section, we'll learn to use the intrinsic dimension for dimension reduction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## The first principal component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The first principal component of the data is the direction in which the data varies the most. In this example, our job is to use PCA to find the first principal component of the length and width measurements of the grain samples, and represent it as an arrow on the scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.793023Z",
     "start_time": "2020-02-15T17:33:47.575605Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Make a scatter plot of the untransformed points\n",
    "plt.scatter(width, length)\n",
    "\n",
    "# Create a PCA instance: model\n",
    "model = PCA()\n",
    "\n",
    "# Fit model to points\n",
    "model.fit(utils.grains)\n",
    "\n",
    "# Get the mean of the grain samples: mean\n",
    "mean = model.mean_\n",
    "\n",
    "# Get the first principal component: first_pc\n",
    "first_pc = model.components_[0,:]\n",
    "\n",
    "# Plot first_pc as an arrow, starting at mean\n",
    "plt.arrow(mean[0], mean[1], first_pc[0], first_pc[1], color='red', width=0.01)\n",
    "\n",
    "# Keep axes on same scale\n",
    "#plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This is the direction in which the grain data varies the most."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Variance of the PCA features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The fish dataset is 6-dimensional. But what is its intrinsic dimension? Let's make a plot of the variances of the PCA features to find out. As before, `fish_samples` is a 2D array, where each row represents a fish. We'll need to standardize the features first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.807984Z",
     "start_time": "2020-02-15T17:33:47.795018Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.823461Z",
     "start_time": "2020-02-15T17:33:47.810491Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create scaler: scaler\n",
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.839416Z",
     "start_time": "2020-02-15T17:33:47.825449Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a PCA instance: pca\n",
    "pca = PCA()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.855376Z",
     "start_time": "2020-02-15T17:33:47.841407Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, pca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:47.871325Z",
     "start_time": "2020-02-15T17:33:47.857364Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Fit the pipeline to 'samples'\n",
    "pipeline.fit(utils.fish_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.041870Z",
     "start_time": "2020-02-15T17:33:47.878306Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Plot the explained variances\n",
    "features = range(pca.n_components_)\n",
    "plt.bar(features, pca.explained_variance_)\n",
    "plt.xlabel('PCA feature')\n",
    "plt.ylabel('variance')\n",
    "plt.xticks(features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    " It looks like PCA features 0 and 1 have significant variance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dimension reduction with PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dimension reduction represents the same data using less features and is vital for building machine learning pipelines using real-world data. Finally, in this lesson, we'll learn how to perform dimension reduction using PCA.\n",
    "\n",
    "We've seen already that the PCA features are in decreasing order of variance. PCA performs dimension reduction by discarding the PCA features with lower variance, which it assumes to be noise, and retaining the higher variance PCA features, which it assumes to be informative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use PCA for dimension reduction, we need to specify how many PCA features to keep. For example, specifying `n_components=2` when creating a PCA model tells it to keep only the first two PCA features. A good choice is the intrinsic dimension of the dataset, if we know it. Let's consider the following example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The iris dataset has 4 features representing the 4 measurements. Here, the measurements are in a numpy array called `samples`. Let's use PCA to reduce the dimension of the iris dataset to only 2. Begin by importing PCA as usual.\n",
    "\n",
    "```\n",
    "from sklearn.decomposition import PCA\n",
    "```\n",
    "\n",
    "Create a PCA model specifying `n_components=2`, and then fit the model and transform the samples as usual.\n",
    "\n",
    "```\n",
    "pca = PCA(n_components=2)\n",
    "\n",
    "pca.fit(samples)\n",
    "transformed = pca.transform(samples)\n",
    "```\n",
    "\n",
    "Here is a scatterplot of the two PCA features, where the colors represent the three species of iris.\n",
    "\n",
    "![Iris PCA](assets/3-14.png)\n",
    "\n",
    "Remarkably, despite having reduced the dimension from 4 to 2, the species can still be distinguished. Remember that PCA didn't even know that there were distinct species. PCA simply took the 2 PCA features with highest variance. As we can see, these two features are very informative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PCA discards the low variance features, and assumes that the higher variance features are informative. Like all assumptions, there are cases where this doesn't hold. As we saw with the iris dataset, however, it often does in practice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word frequency arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases, an alternative implementation of PCA needs to be used. Word frequency arrays are a great example. In a word-frequency array, each row corresponds to a document, and each column corresponds to a word from a fixed vocabulary. The entries of the word-frequency array measure how often each word appears in each document.\n",
    "\n",
    "![Word Frequency Array](assets/3-13.png)\n",
    "\n",
    "Only some of the words from the vocabulary appear in any one document, so most entries of the word frequency array are zero. Arrays like this are said to be sparse, and are often represented using a special type of array called a `csr_matrix`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`csr_matrices` save space by remembering only the non-zero entries of the array. Scikit-learn's PCA doesn't support csr_matrices, and we'll need to use `TruncatedSVD` instead. `TruncatedSVD` performs the same transformation as PCA, but accepts csr matrices as input. Other than that, we interact with `TruncatedSVD` and PCA in exactly the same way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Dimension reduction of the fish measurements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In the previous example, we saw that 2 was a reasonable choice for the \"intrinsic dimension\" of the fish measurements. Now let's use PCA for dimensionality reduction of the fish measurements, retaining only the 2 most important components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.056830Z",
     "start_time": "2020-02-15T17:33:48.044861Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a PCA model with 2 components: pca\n",
    "pca = PCA(n_components=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.071790Z",
     "start_time": "2020-02-15T17:33:48.059821Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create pipeline: pipeline\n",
    "pipeline = make_pipeline(scaler, pca)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.087748Z",
     "start_time": "2020-02-15T17:33:48.073785Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Fit the pipeline to 'samples'\n",
    "pipeline.fit(utils.fish_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.102707Z",
     "start_time": "2020-02-15T17:33:48.090741Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Transform the scaled samples: pca_features\n",
    "pca_features = pipeline.transform(utils.fish_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.117668Z",
     "start_time": "2020-02-15T17:33:48.105698Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print the shape of pca_features\n",
    "print(utils.fish_samples.shape)\n",
    "print(pca_features.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Superb! We've successfully reduced the dimensionality from 6 to 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## A tf-idf word-frequency array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In this example, we'll create a tf-idf word frequency array for a toy collection of documents. For this, we use the `TfidfVectorizer` from sklearn. It transforms a list of documents into a word frequency array, which it outputs as a `csr_matrix`. It has `fit()` and `transform()` methods like other sklearn objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.133625Z",
     "start_time": "2020-02-15T17:33:48.120659Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "documents = ['cats say meow', 'dogs say woof', 'dogs chase cats']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.163545Z",
     "start_time": "2020-02-15T17:33:48.136616Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Import TfidfVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.178504Z",
     "start_time": "2020-02-15T17:33:48.165538Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a TfidfVectorizer: tfidf\n",
    "tfidf = TfidfVectorizer() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.194463Z",
     "start_time": "2020-02-15T17:33:48.181497Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Apply fit_transform to document: csr_mat\n",
    "csr_mat = tfidf.fit_transform(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.209786Z",
     "start_time": "2020-02-15T17:33:48.196456Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Print result of toarray() method\n",
    "print(csr_mat.toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.225621Z",
     "start_time": "2020-02-15T17:33:48.211659Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Get the words: words\n",
    "words = tfidf.get_feature_names()\n",
    "\n",
    "# Print words\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Clustering Wikipedia part I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We saw that `TruncatedSVD` is able to perform PCA on sparse arrays in csr_matrix format, such as word-frequency arrays. Let's combine our knowledge of TruncatedSVD and k-means to cluster some popular pages from Wikipedia. In this example, we build the pipeline. In the next example, we'll apply it to the word-frequency array of some Wikipedia articles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's create a Pipeline object consisting of a TruncatedSVD followed by KMeans. The word-frequency matrix is already computed, so there's no need for a `TfidfVectorizer`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The Wikipedia dataset we'll be working with was obtained from [here](https://blog.lateral.io/2015/06/the-unknown-perils-of-mining-wikipedia/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.288454Z",
     "start_time": "2020-02-15T17:33:48.227616Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Perform the necessary imports\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.303415Z",
     "start_time": "2020-02-15T17:33:48.290449Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a TruncatedSVD instance: svd\n",
    "svd = TruncatedSVD(n_components=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.318375Z",
     "start_time": "2020-02-15T17:33:48.305409Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a KMeans instance: kmeans\n",
    "kmeans = KMeans(n_clusters=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:33:48.333335Z",
     "start_time": "2020-02-15T17:33:48.320371Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a pipeline: pipeline\n",
    "pipeline = make_pipeline(svd, kmeans)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Clustering Wikipedia part II"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It is now time to put our pipeline from the previous example to work! We have an array `articles` of tf-idf word-frequencies of some popular Wikipedia articles, and a list `titles` of their titles. Let's use the pipeline to cluster the Wikipedia articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:34:14.670048Z",
     "start_time": "2020-02-15T17:34:14.466621Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Fit the pipeline to articles\n",
    "pipeline.fit(utils.articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:34:14.951590Z",
     "start_time": "2020-02-15T17:34:14.938635Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Calculate the cluster labels: labels\n",
    "labels = pipeline.predict(utils.articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:34:15.781556Z",
     "start_time": "2020-02-15T17:34:15.767578Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame aligning labels and titles: df\n",
    "df = pd.DataFrame({'label': labels, 'article': utils.titles})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-15T17:34:16.480252Z",
     "start_time": "2020-02-15T17:34:16.463304Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Display df sorted by cluster label\n",
    "print(df.sort_values('label'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "---\n",
    "**[Week 4 - Unsupervised Learning](https://radu-enuca.gitbook.io/ml-challenge/unsupervised-learning)**\n",
    "\n",
    "*Have questions or comments? Visit the ML Challenge Mattermost Channel.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}