Data science - statistics (R).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data science - quantitative analysis\n",
    "\n",
    "Data science can be used to numerical data as well.\n",
    "We do not need to modify it to suite computer analysis, so we can read the data directly in.\n",
    "Here we examine World Value Survey dataset.\n",
    "\n",
    "[Narrated code walkthough](https://www.youtube.com/watch?v=aCbSM4Qiz_Q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data <- read.csv('./data/wvs.csv')\n",
    "data$V10 <- as.factor( data$V10 )\n",
    "table( data$V10 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised machine learning\n",
    "\n",
    "We use a general purpose library [caret](https://www.rdocumentation.org/packages/caret/) in developing and workingwith these exercises.\n",
    "1. splitting the data into train and test datasets\n",
    "1. fit a model to the data\n",
    "1. examine the model performance using test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library( caret )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = createDataPartition(y=data$V10, p=0.7, list=FALSE)\n",
    "train = data[index,]\n",
    "test = data[-index,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table( train$V10 )\n",
    "table( test$V10 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = train( V10 ~ V4 + V5 + V6 + V7 + V8 + V9,  data=train, method=\"rpart\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = predict( model , newdata = test )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tab_class <- table( test$V10 , pred )\n",
    "confusionMatrix(tab_class, mode = \"everything\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "* `V10` has several unwanted values: `-5`, `-2` and `-1`. Remove them from the data and rerun the analysis.\n",
    "* What other variables would you add to the analysis? Do they improve accuracy? See [survey documentation](https://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp) for the meaning of variables.\n",
    "* What other methods than `rpart` exists (see [caret documentation](http://topepo.github.io/caret/train-models-by-tag.html) on available models)? Try out them. Do you get better results?\n",
    "* What does cross validation mean? Try out cross validation and test out these things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised machine learning\n",
    "\n",
    "Beyond seeking to classify data based on existing attributes, in some questions you may want to find groups of similar entries from the data.\n",
    "This is unsupervised machine learning; several methods excists for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clusters <- kmeans( data[1:500,], centers = 5 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## cluster values per variable\n",
    "aggregate( data[1:500,], by=list(cluster=clusters$cluster), mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "* Which cluster has highest number of data points\n",
    "* How does changing the number of clusters change there results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}