Decision trees and random forests (Python).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decion trees and random forests\n",
    "\n",
    "## Data\n",
    "\n",
    "Use the [World Value Survey](http://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp) datafiles and corresponding questionaire and codebook files to understand what is in the data.\n",
    "\n",
    "## Overarching research question\n",
    "\n",
    "Explain what variables effect happiness (`V10`) using decison-tree learning.\n",
    "\n",
    "## Method\n",
    "\n",
    "There are many tools used for this, we apply [SciKit learn](https://scikit-learn.org/stable/modules/tree.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## create new data matrix for decision tree analysis\n",
    "\n",
    "import csv\n",
    "\n",
    "dependent = 'V10'\n",
    "independent = [ 'V4', 'V5', 'V6', 'V7', 'V8', 'V9']\n",
    "\n",
    "dependent_data = []\n",
    "independent_data = []\n",
    "\n",
    "for line in csv.DictReader( open('data/wvs.csv') ):\n",
    "    \n",
    "    dependent_data.append( line[ dependent ] )\n",
    "    \n",
    "    dd = []\n",
    "    for variable in independent:\n",
    "        dd.append( line[variable] )\n",
    "    independent_data.append( dd )\n",
    "    \n",
    "\n",
    "print( len( dependent_data ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "\n",
    "decision_tree = tree.DecisionTreeClassifier( max_depth = 3) # tree.DecisionTreeRegressor() # for regression model\n",
    "model = decision_tree.fit( independent_data, dependent_data )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import graphviz\n",
    "from IPython.display import display\n",
    "image = tree.export_graphviz( model, class_names = True )\n",
    "display( graphviz.Source( image ) ) ## or you could write the graphviz format to a file and visualise it using some other format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model analysis\n",
    "\n",
    "As we spoke, there are [many different approaches to evaluate the quality of a model](https://scikit-learn.org/stable/modules/model_evaluation.html#model-evaluation). Beyond single metrics (like accuracy score, kappa score), confusion matrix may be beneficial to understand model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "predicted_values = model.predict( independent_data )\n",
    "metrics.confusion_matrix( predicted_values, dependent_data )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tasks\n",
    "\n",
    "* Choose better or more interesting values to be modelled.\n",
    "* Improve the data preprocessing (remove missing values etc.)\n",
    "* Apply training data - test data split in the data analysis stage. Does that improve the analysis at all?\n",
    "* Increase the maximum depth of the decision tree. Does it improve the analysis at all?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random forests -  a decision tree on steroids.\n",
    "\n",
    "The challenge with decision trees is that they run a single model on the dataset.\n",
    "However, like many other machine learning algorihms (both supervised and unsupervised), it relies on a single random state.\n",
    "It can overfit or does not lead to best accuracy.\n",
    "[Random forests](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) fix these issues through creating more trees, examining their differences and creating a classifier through combining different decision trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import ensemble\n",
    "\n",
    "random_forest = ensemble.RandomForestClassifier()\n",
    "model = random_forest.fit( independent_data, dependent_data )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_values = model.predict( independent_data )\n",
    "metrics.confusion_matrix( predicted_values, dependent_data )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}