machine_learning_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning\n",
    "\n",
    "**Objectives**: This notebook review some basic machine learning algorithms. Machine learning is a rich and exciting area of computer science that is often considered a subset of the broader field of artificial intelligence. At its most basic level, machine learning seeks to build algorithms that allow systems to perform actions without explicitly being programmed. This notebook: \n",
    "  \n",
    "* Reviews applications of machine learning\n",
    "* Uses a supervised learning model\n",
    "* Uses an unsupervised learning model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning\n",
    "\n",
    "Machine learning is a core component of many data-driven products. Recommendation engines, self-driving cars, credit scoring, spam filtering, product ranking, customer support, and even medical diagnoses are all examples of where machine learning is being applied. \n",
    "\n",
    "Machine learning is in the top five skills identified in LinkedIn profiles of current data scientists. \n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/azbones/big_data/master/images/rjmetrics-ml.png\">\n",
    "(source: https://rjmetrics.com/resources/reports/the-state-of-data-science/)\n",
    "\n",
    "\n",
    "Amazon Web Services has a machine learning product which can be used for batch or real time analysis on their infrastructure.\n",
    "\n",
    "* [Amazon Web Services Machine Learning](https://aws.amazon.com/machine-learning/)\n",
    "\n",
    "Google  open-sourced part of their machine learning libraries that are available at:\n",
    "\n",
    "* \"[Tensorflow](http://www.tensorflow.org/) is designed to facilitate research in machine learning, and to make it quick and easy to transition from research prototype to production system.\"\n",
    "\n",
    "Microsoft's Azure platform now has a machine learning product that is part of their Cortana Analytics Suite which has a drag and drop interface (ML Studio) for deploying machine learning in their cloud using R and Python.\n",
    "\n",
    "* [Azure Machine Learning](https://azure.microsoft.com/en-us/services/machine-learning/) is \"a fully managed big data and advanced analytics suite that enables you to transform your data into intelligent action.\"\n",
    "\n",
    "\n",
    "Even non-profits like Kahn Academy are using machine learning in their production systems:\n",
    "\n",
    "* [Khan Academy Machine Learning → Measurable Learning](http://derandomized.com/post/51729670543/khan-academy-machine-learning-measurable)\n",
    "\n",
    "If you are interested in learning more about Machine Learning, I suggest the course by Andrew Ng from Stanford which is available on Coursera:\n",
    "\n",
    "* https://www.coursera.org/learn/machine-learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised Learning\n",
    "\n",
    "One way to subdivide the machine learning domain is by supervised or unsupervised learning. \n",
    "\n",
    "In supervised learning, a training dataset is used to build a model that is then applied to new observations to make predictions. The training dataset has both the predictor variables (often called features in ML parlance) and the \"correct\" answers or outcome that you are seeking to predict. In traditional statistics classes, the model features will be called independent variables and the predicted values are dependent variables.\n",
    "\n",
    "Often after developing the model, a separate, test dataset is used to measure how well the model performs with new data.\n",
    "\n",
    "Predictions can include:\n",
    "\n",
    "* **Classification:** The model is used to classify observations into categorical values or classes (e.g.- for a cancer test, there might be the categories of \"benign\", \"precancerous\", and \"malignant\").\n",
    "\n",
    "* **Regression:** The model is used to predict values that are continuous in nature (e.g.- measures with infinite measures like temperature).\n",
    "\n",
    "Supervised machine learning algorithms can include:\n",
    "\n",
    "* Decision trees\n",
    "* Discriminant analysis\n",
    "* Naïve Bayes classifiers\n",
    "* Nearest neighbors\n",
    "* Neural networks\n",
    "* Support machine vectors\n",
    "\n",
    "Today, we are going to use the Nearest Neighbor or K-Nearest Neighbor (KNN) algorithm to explore supervised learning in a classification example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**KNN Algorithm**\n",
    "\n",
    "KNN is one of the simplest and most used supervised learning algorithms. While it can be used for both regression and classification, we are going to use it for classification.\n",
    "\n",
    "Simply, the algorithm uses the distance between an observation's feature vector (the values of the features that describe it) and the training dataset to determine group membership. \n",
    "\n",
    "The K refers to the number of neighbors you set to evaluate versus each observation. The algorithm uses distance (most often Euclidian although there are many options here) to determine the K closest neighbors and then classify the observation based on the class that results in the smallest misclassification cost. So, if you select a K of five and three of the five closest points are in group A, that observation would be classified as part of group A. You should select odd values of K to avoid any ties when calculating the lowest misclassification cost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create Synthetic Dataset**\n",
    "\n",
    "To begin, we are going to create a synthetic dataset using the scikit-learn machine learning library. The <code>make_blobs</code> function allows us to create distinct groups by drawing random samples form a Gaussian distribution.\n",
    "\n",
    "Our synthetic dataset will have 1000 samples, 5 defined centers, and 2 features.\n",
    "\n",
    "```\n",
    "X -upper case X- represents the matrix of feature values\n",
    "y -lower case y- represents the 1-dimensional vector of outcome categories\n",
    "```\n",
    "Run the code in the following block to generate the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create synthetic data.\n",
    "\n",
    "from sklearn.datasets import make_blobs\n",
    "X, y = make_blobs(n_samples=1000, centers=5, n_features=2,\n",
    "                  random_state=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Explore X and y to see the shape of the arrays and their values\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, while we could work with the numpy arrays directly, we are going to read them into a pandas dataframe for convenience. Do that in the next codeblock."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dataframe from numpy arrays\n",
    "\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(data=X, index=y, columns=['feature 1', 'feature 2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the dataframe structure to the numpy array below. Where are X and y represented?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Where are X and y in the dataframe?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the Synthetic Data**\n",
    "\n",
    "Now, we are going to plot the features as x and y coordinates and the outcome groupings by color. Remember this data is our training dataset, so we have both the predictors (x and y coordinates on our graph) and the thing we want to predict, group membership. Practically, an example of this might be features of e-mail and whether we, as experts, classified those individual messages as spam or non-spam.\n",
    "\n",
    "Plot the data below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import matplotlib as mpl\n",
    "\n",
    "# Plot data\n",
    "sns.set_style(\"darkgrid\")\n",
    "mpl.rcParams['figure.figsize'] = (10.0, 10.0)\n",
    "group1 = plt.scatter(df['feature 1'].loc[0], df['feature 2'].loc[0], c='b', label='group 1')\n",
    "group2 = plt.scatter(df['feature 1'].loc[1], df['feature 2'].loc[1], c='c', label='group 2')\n",
    "group3 = plt.scatter(df['feature 1'].loc[2], df['feature 2'].loc[2], c='r', label='group 3')\n",
    "group4 = plt.scatter(df['feature 1'].loc[3], df['feature 2'].loc[3], c='k', label='group 4')\n",
    "group5 = plt.scatter(df['feature 1'].loc[4], df['feature 2'].loc[4], c='w', label='group 5')\n",
    "legend = plt.legend(handles=[group1, group2, group3, group4, group5], frameon=1)\n",
    "plt.xlabel('X Values')\n",
    "plt.ylabel('Y values')\n",
    "legend.get_frame().set_facecolor('#ffffff')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Build the KNN Model**\n",
    "\n",
    "Next, we are going to build our KNN classifier and fit the model to the training data.  For this initial attempt, we will use a k value of 1, so set k represented by <code>n_neighbors</code> to 1 and run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create KNN classifier and fit model\n",
    "\n",
    "from sklearn import neighbors, datasets\n",
    "\n",
    "X, y = df.values, df.index\n",
    "n_neighbors = 1\n",
    "knn = neighbors.KNeighborsClassifier(n_neighbors=n_neighbors)\n",
    "knn.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Predict Group Membership for a New Observation**\n",
    "\n",
    "Now we have a classifier that can take any two-feature inputs and predict group membership. Populate the features with some example values and run the prediction in the next block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict group membership based values entered in predict fucntion.\n",
    "\n",
    "classes = ['Group 1', 'Group 2', 'Group 3', 'Group 4', 'Group 5']\n",
    "obs_one = 1 # try any number here\n",
    "obs_two = 2 # try any number here\n",
    "\n",
    "prediction = classes[knn.predict([[obs_one, obs_two]])[0]]\n",
    "        \n",
    "print(\"An observation with feature-one value of {} and feature-two value of {} is a member of {}\".format(obs_one,obs_two,prediction))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**N-dimensional Features**\n",
    "\n",
    "While our example using two features was instructive as a visualization, most machine learning problems will have many more features. Unfortunately, plotting the feature space of predictions is not possible beyond three dimensions (even three dimensional plots are difficult to use given they are rendered in two dimensions). The KNN models work exactly the same way, regardless of the number of features.\n",
    "\n",
    "Build a synthetic dataset with three features below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# New synthetic data with 3 features.\n",
    "\n",
    "X2, y2 = make_blobs(n_samples=1000, centers=4, n_features=3,\n",
    "                  random_state=4)\n",
    "df2 = pd.DataFrame(data=X2, index=y2, columns=['feature 1', 'feature 2', 'feature 3'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training values with a 3D projection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3D plot of three features from training data.\n",
    "\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "sns.set_style(\"whitegrid\")\n",
    "mpl.rcParams['figure.figsize'] = (10.0, 10.0)\n",
    "fig = plt.figure('3D')\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "group1 = ax.scatter(df2['feature 1'].loc[0], df2['feature 2'].loc[0], df2['feature 3'].loc[0], c='b', label='group 1')\n",
    "group2 = ax.scatter(df2['feature 1'].loc[1], df2['feature 2'].loc[1], df2['feature 3'].loc[1], c='c', label='group 2')\n",
    "group3 = ax.scatter(df2['feature 1'].loc[2], df2['feature 2'].loc[2], df2['feature 3'].loc[2], c='r', label='group 3')\n",
    "group4 = ax.scatter(df2['feature 1'].loc[3], df2['feature 2'].loc[3], df2['feature 3'].loc[3], c='k', label='group 4')\n",
    "ax.set_xlabel('X Values')\n",
    "ax.set_ylabel('Y Values')\n",
    "ax.set_zlabel('Z Values')\n",
    "legend = plt.legend(handles=[group1,group2,group3,group4], frameon=1)\n",
    "legend.get_frame().set_facecolor('#ffffff')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Build the KNN Model for Three Features**\n",
    "\n",
    "Next, we are going to build our KNN classifier and fit the model to the new training data.  For this initial attempt, we will use a k value of 1 again, so set this in the code and run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3 Feature model fitting.\n",
    "\n",
    "classes_3d = ['Group 1','Group 2','Group 3','Group 4']\n",
    "n_neighbors = 1\n",
    "knn_3d = neighbors.KNeighborsClassifier(n_neighbors=n_neighbors)\n",
    "knn_3d.fit(X2, y2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Predict Group Membership with Three Values**\n",
    "\n",
    "Populate the observation values and predict group membership below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another prediction based on 3 features.\n",
    "\n",
    "obs_3d_one = 1 # try any number here\n",
    "obs_3d_two = 2 # try any number here\n",
    "obs_3d_three = 3 # try any number here\n",
    "\n",
    "prediction_3d = classes_3d[knn_3d.predict([[obs_3d_one, obs_3d_two, obs_3d_three]])[0]]\n",
    "\n",
    "print(\"An observation with a feature-one value of {}, a feature-two value of \\\n",
    "{}, and a feature-three value of {} is a member of {}\".format(obs_3d_one, obs_3d_two, \n",
    "                                                              obs_3d_three, prediction_3d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised Learning\n",
    "\n",
    "As you may have guessed, unsupervised learning does not use a training set, but instead draws inferences about the data from the data itself without defined labels.\n",
    "\n",
    "<img src=\"http://imgs.xkcd.com/comics/watson_medical_algorithm.png\">\n",
    "(source: http://xkcd.com/1619/)\n",
    "\n",
    "The most common use case for unsupervised learning is cluster analysis. Examples of cluster analysis include:\n",
    "\n",
    "* **Genetic Classification:** Genome sequencing dataset can be used to group individuals or organisms with similar properties.\n",
    "\n",
    "* **Document Classification:** Documents with similar topics, based on NLP, can be grouped together. \n",
    "* **Behavioral Cohort Analysis:** Clustering users together based on their behavior.\n",
    "\n",
    "\n",
    "Unsupervised machine learning algorithms can include:\n",
    "\n",
    "* Gaussian mixture models\n",
    "* Hidden Markov models\n",
    "* Hierarchical clustering\n",
    "* K-Means clustering\n",
    "\n",
    "We are going to use the k-means algorithm, again from the sci-kit learn package, to explore unsupervised learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**K-Means Algorithm**\n",
    "\n",
    "The k-means algorithm is an iterative algorithm that starts by locating the k cluster centers in arbitrary locations within the multi-dimensional feature space. It then assigns each observation to the mean center that it is closest to using a distance calculation (usually Euclidian). It then iterates through randomly assigned cluster centers until has minimized the distance from each observation to its cluster center. This is calculated by looking at the sum of distances for all observations to the selected center over all the centers. \n",
    "\n",
    "Given its simplicity, k-means also has higher performance than other clustering techniques and is therefore quite popular for larger datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create Synthetic Dataset**\n",
    "\n",
    "We are again going to create a synthetic dataset, but we are not going to use the group membership in our model. This is the definition of unsupervised learning. While we will know apriori how many clusters our data has, this will allow us to better understand the algorithm as we evaluate different models with the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Synthetic data with 2 features for k-means.\n",
    "\n",
    "Xkm, ykm = make_blobs(n_samples=5000, centers=10, n_features=2,\n",
    "                  random_state=7)\n",
    "kmeans_df = pd.DataFrame(data=Xkm, index=None, columns=['feature 1', 'feature 2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plot the Synthetic Data**\n",
    "\n",
    "Now we will plot the data.  Notice we are not using group membership in our dataframe or our plot. Our synthetic dataset will have some well defined groups that are clearly seen in a two dimensional plot and some clusters that are overlapping the same feature space. \n",
    "\n",
    "Most real datasets will be more complex in that they have many more features and less differentiation among these features, so clusters are not as likely to be seen in plots given less uniform distributions.\n",
    "\n",
    "Run the code below to plot the data. Count the number of clusters that are clearly visible in the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot observation data.\n",
    "\n",
    "sns.set_style(\"darkgrid\")\n",
    "mpl.rcParams['figure.figsize'] = (10.0, 10.0)\n",
    "plt.figure('Unsupervised Data')\n",
    "plt.title('Unsupervised Data')\n",
    "plt.scatter(kmeans_df['feature 1'], kmeans_df['feature 2'], c='b')\n",
    "plt.xlabel('Feature 1')\n",
    "plt.ylabel('Feature 2')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Build the K-Means Model**\n",
    "\n",
    "We are again going to use the sci-kit learn package to build our model. Other than the data, the key input into k-means is k or the number of clusters you want to model. We will review how to evaluate different values for k after we build our initial model. Other unsupervised algorithms can derive the number of clusters on their own, but also come with more complexity.\n",
    "\n",
    "Depending on the dataset, the number of iterations defined by the <code>init</code> parameter can also be important as k-means can be subject to local minima which will result in poor cluster results. Given k-means uses random inputs, it is often useful to run it multiple times to assure that it is consistently converging.\n",
    "\n",
    "We will start with three clusters to see how the algorithm handles the synthetic data.\n",
    "\n",
    "Define the number of clusters in the <code>cluster_num</code> varaiable and run the codeblock below to create the model with three centers and fit it to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit k-means model to data.\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "cluster_num = 3\n",
    "\n",
    "k_means = KMeans(n_clusters=cluster_num, init='k-means++', n_init=1000)\n",
    "k_means.fit(kmeans_df)\n",
    "kmeans_df['cluster'] = k_means.labels_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, plot the data using the cluster numbers that we added to the dataframe. Remember, the actual cluster membership, or ground truth in machine learning terms, is not part of our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot observations with color coding for each cluster.\n",
    "\n",
    "import matplotlib.cm as cm\n",
    "\n",
    "plt.figure('Unsupervised Data w Clusters')\n",
    "plt.title('Unsupervised Data With K-Means Clusters by Color with k={}'.format(cluster_num))\n",
    "plt.scatter(kmeans_df['feature 1'], kmeans_df['feature 2'], c=kmeans_df['cluster'], cmap=cm.Oranges)\n",
    "plt.xlabel('Feature 1')\n",
    "plt.ylabel('Feature 2')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Print Cluster Centers**\n",
    "\n",
    "We can directly access the cluster center coordinates from our model by calling the <code>cluster\\_centers\\_</code> attribute.\n",
    "\n",
    "Print those below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (k_means.cluster_centers_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's replot the data with the color-coded cluster membership along with the cluster centers as large x's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now add cluster centers to same plot.\n",
    "\n",
    "plt.figure('Unsupervised Data w Clusters')\n",
    "plt.title('Unsupervised Data, K-Means Clusters by Color, Cluster Centers with k={}'.format(cluster_num))\n",
    "plt.scatter(kmeans_df['feature 1'], kmeans_df['feature 2'], c=kmeans_df['cluster'], cmap=cm.Oranges)\n",
    "plt.scatter(k_means.cluster_centers_[:,0], k_means.cluster_centers_[:,1], marker='x', s=500, linewidths=10)\n",
    "plt.xlabel('Feature 1')\n",
    "plt.ylabel('Feature 2')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change K**\n",
    "\n",
    "Now, go back through the code and change K to different values and rerun the model and plots to see how the algorithm performs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Determining Optimal K**\n",
    "\n",
    "Unfortunately, there is not an exact way to determine the right number of clusters given part of the value of clusters is they have some meaning that can be put to use. The most accurate number of clusters to represent all the observations would be a cluster for every observation. Or course, that approach completely loses the value of having a much smaller set of groups that can be used to generalize across many cases.\n",
    "\n",
    "So, the challenge is having a relatively small set of clusters that have small average distances to the hypothetical mean centroid values while also having each centroid sufficiently distant from the others to provide for distinctness among clusters. Ideally, these cluster can also be understood as a segment or profile (rather than just the mathematical means) such that the clusters can aid in analysis or decision making. User behavioral cohorts, for example, might be given titles that sum up the multiple features that comprise the cluster like \"casual users\", \"power users\", etc."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}