2-machine-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. The “Friends and Family” data set\n",
    "\n",
    "> An experiment was designed in 2011 to study (a) how people make decisions, with emphasis on the social aspects involved, and (b) how we can empower people to make better decisions using personal and social tools. The data set was collected by Nadav Aharony, Wei Pan, Cory Ip, Inas Khayal, and Alex Pentland.\n",
    "\n",
    "More details are available on http://realitycommons.media.mit.edu/friendsdataset.html\n",
    "\n",
    "> The subjects were members of a young-family residential living community adjacent to a major research university in North America. All members of the community are couples, and at least one of the members is affiliated with the university. The community is composed of over 400 residents, approximately half whom have children. A pilot phase of 55 participants was launched in March 2010. In September 2010, phase two of the study included 130 participants, approximately 64 families. Participants were selected out of approximately 200 applicants in a way that would achieve a representative sample of the community and sub-communities.\n",
    "\n",
    "In the ``data-fnf`` directory, we provide a data set with 129 users interacting with each other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!ls data-fnf/records/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each CSV file contains call and text metadata records belonging to a single user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!head \"data-fnf/records/fa10-01-01.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Export indicators for every user\n",
    "\n",
    "With bandicoot, it is easy to load all the users and automatically compute their indicators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bandicoot as bc\n",
    "from tqdm import tqdm_notebook as tqdm  # Interactive progress bar\n",
    "import glob\n",
    "import os\n",
    "\n",
    "import pandas\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load a user and returns all its indicators\n",
    "\n",
    "def make_features(user_id):\n",
    "    user = bc.read_csv(user_id, \"data-fnf/records/\",\n",
    "                       attributes_path=\"data-fnf/attributes/\",\n",
    "                       describe=False, warnings=False)\n",
    "\n",
    "    return bc.utils.all(user, summary='extended', split_day=True, split_week=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Loop over all CSV files in /data-fnf/records and call make_features\n",
    "\n",
    "all_features = []\n",
    "\n",
    "for f in tqdm(glob.glob(\"data-fnf/records/*.csv\")):\n",
    "    user_id = os.path.basename(f[:-4])  # Remove .csv extension\n",
    "    all_features.append(make_features(user_id))\n",
    "\n",
    "# Export all features in one file (fnf_features.csv)\n",
    "bc.io.to_csv(all_features, 'fnf_features.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Gender classification\n",
    "\n",
    "The data set provided contains both metadata records and gender for each user. Let's try to predict the gender from the indicators we computed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Load the features and attributes in a table, using the pandas library\n",
    "\n",
    "df = pandas.read_csv('fnf_features.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create two objects:\n",
    "\n",
    "- the array ``y`` contains the labels we want to predict (male/female),\n",
    "- the matrix ``X`` contains the features for all users (one column for one feature, one line for one user)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 1. We convert gender labels to binary values (zero or one):\n",
    "y = (df.attributes__gender == 'male').values.astype(np.int)\n",
    "\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 2. We remove columns with reporting variables and attributes (the first 39 and the last 2):\n",
    "df = df[df.columns[39:-2]]\n",
    "X = df.values\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn import svm, linear_model, ensemble, neighbors, tree\n",
    "from sklearn import metrics, cross_validation, preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 3. We impute missing values in the features\n",
    "imp = preprocessing.Imputer(missing_values='NaN', strategy='mean', axis=0)\n",
    "imp.fit(df)\n",
    "\n",
    "X = imp.transform(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 4. Preprocess data (center around 0 and scale to remove the variance)\n",
    "scaler = preprocessing.StandardScaler()\n",
    "X = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification with cross-validation\n",
    "\n",
    "> Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.\n",
    "\n",
    "> The advantages of support vector machines are:\n",
    "> - Effective in high dimensional spaces.\n",
    "> - Still effective in cases where number of dimensions is greater than the number of samples.\n",
    "> - Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.\n",
    "> - Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels.\n",
    "\n",
    "> Source: http://scikit-learn.org/stable/modules/svm.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 5. Divide records in training and testing\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.3)\n",
    "\n",
    "# 6. Create an SVM classifier and train it on 70% of the data set\n",
    "clf = svm.SVC()\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# 7. Analyze accuracy of predictions on 30% of the data set\n",
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" role=\"alert\">\n",
    "    <strong>Question:</strong> Is it a good score? Why?\n",
    "</div>\n",
    "### Performance of the algorithm\n",
    "\n",
    "The [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) helps visualize the performance of the algorithm. Each line corresponds to actual classes (male/female), and each column to predicted classes (male/female)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y_pred = clf.fit(X_train, y_train).predict(X_test)\n",
    "cm = metrics.confusion_matrix(y_test, y_pred)\n",
    "\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(X_train), len(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use other classifiers\n",
    "\n",
    "You can easily use different classifiers in scikit-learn, such as:\n",
    "\n",
    "- SVM with ``svm.SVC()`` (see above),\n",
    "- k-nearest neighbors with ``neighbors.KNeighborsClassifier()``,\n",
    "- random forests with ``ensemble.RandomForestClassifier()``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "classifier = ensemble.RandomForestClassifier(random_state=0)\n",
    "\n",
    "classifier.fit(X_train, y_train)\n",
    "classifier.score(X_test, y_test)                   "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  },
  "toc": {
   "toc_cell": false,
   "toc_number_sections": false,
   "toc_threshold": 6,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}