Python Workshop (SKELETON).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3CWRGRFjzadp"
   },
   "source": [
    "# Python Workshop\n",
    "\n",
    "Importing different libraries allows us to access and use the functions stored inside them. These six lines are the most commonly used data science packages and settings, so it's good practice to include at the top of every notebook you open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:51:13.565359Z",
     "start_time": "2018-04-17T05:51:12.679565Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "Ckirx_M7zqmi"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c43cKMTBzyfj"
   },
   "source": [
    "## Section 1: Variables, types, and basic math operations\n",
    "---\n",
    "So far, we've added the following *operations* to our toolbox:\n",
    "- `+` , `-` , `*` , `/` : Add, subtract, multiply, divide\n",
    "- `=` : Assign variables\n",
    "- `<`, `>`, `<=`, `>=`, `==`: Compare values\n",
    "---\n",
    "\n",
    "Given that we have the following sales values for 2016 and 2017:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZYfk-b9N2fni"
   },
   "source": [
    "<header><h4 align='center'>Sales</h4></header>\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "    <thead>\n",
    "        <tr>\n",
    "            <td><b>Year</b></td>\n",
    "            <td><b>Product</b></td>\n",
    "            <td><b>Revenue</b></td>\n",
    "            <td><b>Cost</b></td>\n",
    "        </tr>\n",
    "    </thead>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>Phone</td>\n",
    "        <td>320000</td>\n",
    "        <td>254000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>Laptop</td>\n",
    "        <td>120000</td>\n",
    "        <td>80000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2017</td>\n",
    "        <td>Phone</td>\n",
    "        <td>465000</td>\n",
    "        <td>362000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2017</td>\n",
    "        <td>Laptop</td>\n",
    "        <td>105000</td>\n",
    "        <td>67300</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "imwZ3SLr6T3m"
   },
   "source": [
    "**Task:** \n",
    "- Assign four variables to represent phone revenues and costs in 2016 and 2017, respectively.\n",
    "- Use these four variables to create two new variables representing profit in 2016 and 2017, respectively.\n",
    "- Use the two profit variables to calculate total combined profit in 2016 and 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.664892Z",
     "start_time": "2018-04-17T05:50:58.659727Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 266,
     "status": "ok",
     "timestamp": 1523924871734,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "-nh2Wffz0944",
    "outputId": "f1abd589-bb2e-420a-e868-02bfb42be1af"
   },
   "outputs": [],
   "source": [
    "# Phone revenues and costs in 2016 and 2017\n",
    "revenue_2016 = ...\n",
    "cost_2016 = ...\n",
    "revenue_2017 = ...\n",
    "cost_2017 = ...\n",
    "\n",
    "print(revenue_2016, cost_2016, revenue_2017, cost_2017)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.692632Z",
     "start_time": "2018-04-17T05:50:58.667484Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 420,
     "status": "ok",
     "timestamp": 1523924882960,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "y1mV5ZJ6C1fb",
    "outputId": "6527d7bd-b701-4eeb-a7cc-2671af8ecbed"
   },
   "outputs": [],
   "source": [
    "# Profit in 2016 and 2017\n",
    "profit_2016 = ...\n",
    "profit_2017 = ...\n",
    "\n",
    "print(profit_2016, profit_2017)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.709543Z",
     "start_time": "2018-04-17T05:50:58.696947Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 526,
     "status": "ok",
     "timestamp": 1523924889934,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "q_8qmmxn7pLR",
    "outputId": "f88d78e7-6e23-4412-9bf5-316e0440336b"
   },
   "outputs": [],
   "source": [
    "# Combined total profit\n",
    "total_profit = ...\n",
    "\n",
    "total_profit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0tEvAd-J7yUN"
   },
   "source": [
    "**Task:**\n",
    "  - What were average monthly profit figures over 2016 and 2017, combined?\n",
    "  - Our goal was to achieve $10,000 in average monthly profit over the last year. Did we achieve this? Format the answer as a boolean (`True` or `False`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.729019Z",
     "start_time": "2018-04-17T05:50:58.720115Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 294,
     "status": "ok",
     "timestamp": 1523924637026,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "zX_7nJ8Hz-Cd",
    "outputId": "cee45243-aabf-4e2e-be54-e363032e2705"
   },
   "outputs": [],
   "source": [
    "# Monthly profit figures\n",
    "monthly_profit = ...\n",
    "\n",
    "monthly_profit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.742821Z",
     "start_time": "2018-04-17T05:50:58.732014Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 384,
     "status": "ok",
     "timestamp": 1523924637530,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "rHcp7bxJ2Bvc",
    "outputId": "62ca8e04-5334-488f-a341-863274e73400"
   },
   "outputs": [],
   "source": [
    "# Success?\n",
    "success = ...\n",
    "\n",
    "success"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EQAPXRQsAEQV"
   },
   "source": [
    "## Section 2: Lists and loops\n",
    "---\n",
    "We now have additional tools in our toolbox:\n",
    "- Lists allow us to store multiple values in one variable\n",
    "- Loops allow us to operate on lists by *iterating* through each value in the list\n",
    "---\n",
    "\n",
    "Ideally, all of our raw datasets would be cleanly organized and formatted exactly to our needs. For example, if we were looking for data regarding annual revenue, cost, and profit, our table from Section 1 contains exactly the information that we want. However, not all data is this cleanly formatted and aggregated into rows; furthermore, this may not necessarily be in our best interest! (Why not?)\n",
    "\n",
    "Instead, we might have more granular data in the form of monthly values:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NbpPDpt8I5BX"
   },
   "source": [
    "<header><h4 align='center'>2016 Phone Sales</h4></header>\n",
    "<table border=\"1\" class=\"dataframe\">\n",
    "    <thead>\n",
    "        <tr>\n",
    "            <td><b>Year</b></td>\n",
    "            <td><b>Month</b></td>\n",
    "            <td><b>Revenue</b></td>\n",
    "            <td><b>Cost</b></td>\n",
    "        </tr>\n",
    "    </thead>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>1</td>\n",
    "        <td>33000</td>\n",
    "        <td>26800</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>2</td>\n",
    "        <td>24000</td>\n",
    "        <td>19200</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>3</td>\n",
    "        <td>19000</td>\n",
    "        <td>15900</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>4</td>\n",
    "        <td>20000</td>\n",
    "        <td>16300</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>5</td>\n",
    "        <td>21000</td>\n",
    "        <td>15000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>6</td>\n",
    "        <td>23000</td>\n",
    "        <td>18000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>7</td>\n",
    "        <td>21000</td>\n",
    "        <td>16700</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>8</td>\n",
    "        <td>26000</td>\n",
    "        <td>21300</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>9</td>\n",
    "        <td>24000</td>\n",
    "        <td>21000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>10</td>\n",
    "        <td>28000</td>\n",
    "        <td>23000</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>11</td>\n",
    "        <td>43000</td>\n",
    "        <td>35700</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>2016</td>\n",
    "        <td>12</td>\n",
    "        <td>38000</td>\n",
    "        <td>30100</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ULjh_XjFN9bQ"
   },
   "source": [
    "Then the revenue and cost columns can be represented using two lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-09-18T05:55:50.239776Z",
     "start_time": "2018-09-18T05:55:50.233336Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "WOg-PPb4H6eZ"
   },
   "outputs": [],
   "source": [
    "monthly_revenue_2016 = [33000, 24000, 19000, 20000, 21000, 23000, 21000, 26000, 24000, 28000, 43000, 38000]\n",
    "monthly_cost_2016 = [26800, 19200, 15900, 16300, 15000, 18000, 16700, 21300, 21000, 23000, 35700, 30100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gZQQPu3UOVj6"
   },
   "source": [
    "**Task:**\n",
    "- For 2016 phone sales' monthly revenue and monthly cost, find each of the following:\n",
    "  - Mean\n",
    "  - Standard deviation ([`np.std`](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.std.html))\n",
    "  - 25, 50, 75th percentiles ([`np.percentile`](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.percentile.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.775172Z",
     "start_time": "2018-04-17T05:50:58.760855Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 347,
     "status": "ok",
     "timestamp": 1523925323264,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "0JnRybSXRWjD",
    "outputId": "75cf3cc3-8a0d-40e1-aba0-0380fc441d3d"
   },
   "outputs": [],
   "source": [
    "# Mean\n",
    "revenue_2016_mean = ...\n",
    "cost_2016_mean = ...\n",
    "\n",
    "print(revenue_2016_mean, cost_2016_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.790994Z",
     "start_time": "2018-04-17T05:50:58.777759Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 490,
     "status": "ok",
     "timestamp": 1523925329276,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "hGIed2_ZSW8T",
    "outputId": "f12f7e57-0f28-4123-f5cc-f40d0408c0c4"
   },
   "outputs": [],
   "source": [
    "# Standard deviation\n",
    "revenue_2016_std = ...\n",
    "cost_2016_std = ...\n",
    "\n",
    "print(revenue_2016_std, cost_2016_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.804355Z",
     "start_time": "2018-04-17T05:50:58.794668Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 362,
     "status": "ok",
     "timestamp": 1523865205656,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "rNObXdCNSgCr",
    "outputId": "38ff66b5-43f5-4b19-ca75-ef80b7fea1b6"
   },
   "outputs": [],
   "source": [
    "# 25, 50, 75th percentiles\n",
    "revenue_2016_perc = ...\n",
    "cost_2016_perc = ...\n",
    "\n",
    "print(revenue_2016_perc, cost_2016_perc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CMyW0myjRgWU"
   },
   "source": [
    "Before we complete the next task, observe the following two cells to observe the difference between the two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:51:22.414645Z",
     "start_time": "2018-04-17T05:51:22.408805Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 283,
     "status": "ok",
     "timestamp": 1523865206206,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "_dK5tdu8SIpA",
    "outputId": "5bfabe9e-e8d4-41d5-91cb-4085787e2930"
   },
   "outputs": [],
   "source": [
    "[1, 2, 3] + [1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:51:23.459095Z",
     "start_time": "2018-04-17T05:51:23.453909Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 373,
     "status": "ok",
     "timestamp": 1523865206807,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "GoRNamX4Spqk",
    "outputId": "1c23a1fc-7869-4a44-a443-8f87bfc1ffae"
   },
   "outputs": [],
   "source": [
    "np.add([1, 2, 3], [1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EFF0-Q5HStUn"
   },
   "source": [
    "We see that the `+` operator *concatenates* the two lists, whereas [`np.add`](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.add.html) performs *element-wise* addition. Similarly, [`np.subtract`](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.subtract.html) will perform *element-wise* subtraction.\n",
    "\n",
    "**Task:**\n",
    "- Use [`np.subtract`](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.subtract.html) to calculate the profit for each month in 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:58.850443Z",
     "start_time": "2018-04-17T05:50:58.838044Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 316,
     "status": "ok",
     "timestamp": 1523865207435,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "6cb-LxJ_XR2f",
    "outputId": "24ee3c3d-407f-4870-bd51-6a57d59002d4"
   },
   "outputs": [],
   "source": [
    "# Monthly profit 2016\n",
    "monthly_profit_2016 = ...\n",
    "\n",
    "monthly_profit_2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ns7PAYGmXa3q"
   },
   "source": [
    "**Task:**\n",
    "- (Challenging!) Use a loop to calculate the month-over-month raw change (in dollars, *not* percentages) for 2016 profits.\n",
    "  - Hint: Since there are 12 months (`len(monthly_profit_2016)`), there are 11 values that we will want to calculate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.078673Z",
     "start_time": "2018-04-17T05:50:58.853307Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 315,
     "status": "ok",
     "timestamp": 1523865207981,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "7kJNxx49X5vv",
    "outputId": "7a44cefd-ec7f-4a88-f1ed-e141482218ea"
   },
   "outputs": [],
   "source": [
    "# Month-over-month profit change\n",
    "monthly_profit_change = []\n",
    "\n",
    "for i in ...:\n",
    "    ...\n",
    "    monthly_profit_change.append(...)\n",
    "    \n",
    "monthly_profit_change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:31:45.231164Z",
     "start_time": "2018-04-17T05:31:45.219015Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 221,
     "status": "ok",
     "timestamp": 1523865208509,
     "user": {
      "displayName": "Jun Seo Park",
      "photoUrl": "//lh6.googleusercontent.com/-KIeDOii6NOY/AAAAAAAAAAI/AAAAAAAAAA4/BO9dUEBtd8E/s50-c-k-no/photo.jpg",
      "userId": "116991900080281348598"
     },
     "user_tz": 420
    },
    "id": "T6WCoDBNZkU3",
    "outputId": "eb2f0553-b932-459b-a83c-7635b9de17a2"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jH6am5vjCZbU"
   },
   "source": [
    "## Section 3: Using Pandas to Manipulate Data\n",
    "---\n",
    "New tools (assume `df` is a DataFrame, `colname` is the name of a column):\n",
    "- [`pd.read_csv`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) to read CSV files\n",
    "- `df.head()` to view the first 5 lines\n",
    "- `df['colname']` to view a column as a `Series`\n",
    "- `df[['colname1', 'colname2']]` to view multiple columns as a `DataFrame` \n",
    "- `df.loc[_____, _____]` to view certain rows and columns based on index names\n",
    "- `df.iloc[_____, _____]` to view certain rows and columns based on numerical indices\n",
    "---\n",
    "\n",
    "Switching gears, we'll move to a real-life dataset of Kiva loans.\n",
    "\n",
    "**Task:**\n",
    "- Read in the Kiva dataset; let's call it `loans`.\n",
    "- View the first five lines to get a grasp of what the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.079479Z",
     "start_time": "2018-04-17T05:50:57.747Z"
    },
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "tHByP7ObFVyL"
   },
   "outputs": [],
   "source": [
    "# Read in dataset\n",
    "loans = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.080926Z",
     "start_time": "2018-04-17T05:50:57.759Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# View first five lines\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task:**\n",
    "\n",
    "It's always good practice to do two things once we first load a dataset:\n",
    "- Use `.shape` to find the dimensions of the data\n",
    "- Use `.isnull().sum()` to find how many values are missing from each column\n",
    "    - (Optional) If you'd like to know how/why this works, check out the documentation for [`df.isnull()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.isnull.html) and [`df.sum()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sum.html), and try each separately on `loans`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.082290Z",
     "start_time": "2018-04-17T05:50:57.766Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Dimensions of data\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.083425Z",
     "start_time": "2018-04-17T05:50:57.769Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of missing values from each column\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task:**\n",
    "\n",
    "For these next two tasks, use only square brackets (not `.loc` or `.iloc`).\n",
    "- Take a look at the `use` column by itself, as a `Series`\n",
    "- Take a look at the `sector` and `use` columns together, as a `DataFrame`\n",
    "    - Do you notice anything odd about the `sector` column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.084609Z",
     "start_time": "2018-04-17T05:50:57.772Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 'use' column\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.085833Z",
     "start_time": "2018-04-17T05:50:57.779Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 'sector' and 'use' columns\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T01:12:59.992705Z",
     "start_time": "2018-04-17T01:12:59.362425Z"
    }
   },
   "source": [
    "**Task:**\n",
    "\n",
    "For these next three tasks, use only square brackets (not `.loc` or `.iloc`).\n",
    "- View the rows that have a `sector` value of `Tranpotation`. \n",
    "    - What format is this?\n",
    "- Replace the `Tranpotation` values with `Transportation`.\n",
    "    - Are you seeing a warning? Why might this happen?\n",
    "- Check `loans.head()`. Did we correctly replace the `Tranpotation` values?\n",
    "- Hint: a pandas series has a .replace() method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.087223Z",
     "start_time": "2018-04-17T05:50:57.791Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# View rows with the sector Tranpotation\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.088378Z",
     "start_time": "2018-04-17T05:50:57.795Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Attempt to replace Tranpotation with Transportation using square brackets\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.089412Z",
     "start_time": "2018-04-17T05:50:57.798Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Verify correct(?) replacement\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task:**\n",
    "- Using `.loc[_____, ______]`, replace the `Tranpotation` values with `Transportation`.\n",
    "- Verify using `loans.head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.090227Z",
     "start_time": "2018-04-17T05:50:57.801Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Replace Tranpotation with Transportation using .loc\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.091234Z",
     "start_time": "2018-04-17T05:50:57.803Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Verify correct replacement\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Working with Aggregate Data\n",
    "---\n",
    "More tools in our toolbox! Again, assume `df` is a DataFrame and `colname` is the name of a column.\n",
    "- [`df.groupby(colname)`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html): Places each row of a DataFrame into a group based on certain column values\n",
    "- `df.groupby(colname).function()` applies `function()` to each group. Possible functions include:\n",
    "    - `count()`: number of non-NA rows in each group\n",
    "    - `mean()`: mean of all values in each group\n",
    "    - `std()`: standard deviation of all values in each group\n",
    "    - `min()`: minimum of all values in each group\n",
    "    - `max()`: maximum of all values in each group\n",
    "    - `median()`: median of all values in each group\n",
    "---\n",
    "\n",
    "**Task:**\n",
    "- Find the number of loans for each sector\n",
    "- Find the average loan amount for each country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.094540Z",
     "start_time": "2018-04-17T05:50:57.806Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of loans for each sector\n",
    "loans_by_sector = ...\n",
    "\n",
    "loans_by_sector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.095700Z",
     "start_time": "2018-04-17T05:50:57.810Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Average loan amount for each country\n",
    "average_loan_by_country = ...\n",
    "\n",
    "average_loan_by_country"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 5: Graphing\n",
    "---\n",
    "Basic graphing tools:\n",
    "- [`plt.plot(x, y)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html): Line graph\n",
    "- [`plt.scatter(x, y)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.scatter.html): Scatter plot\n",
    "- [`plt.bar(x, height)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.bar.html): Bar chart\n",
    "- [`plt.hist(x)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.hist.html): Histogram\n",
    "- [`plt.figure(figsize=(__, __))`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figure.html): Set figure size\n",
    "- [`plt.xlabel(_____)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.xlabel.html), [`plt.ylabel(_____)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.xlabel.html): Set x-axis and y-axis labels\n",
    "- [`plt.xlim(_____)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.xlim.html), [`plt.ylim(_____)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.ylim.html): Set x-axis and y-axis limits\n",
    "- [`plt.title(_____)`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.title.html): Set title\n",
    "- [`plt.legend()`](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html): Show legend (assuming you have assigned labels; e.g. `plt.scatter(x, y, label='United States')`)\n",
    "- Note that you can control almost any small detail on the graph! Color, line width/scatterpoint size, rotating axis tick-mark labels, etc. Chances are, someone else has had that question too! Use Google liberally.\n",
    "---\n",
    "\n",
    "**Task:**\n",
    "- Create a scatterplot of `lender_count` versus `loan_amount`.\n",
    "- Create a histogram of the `term_in_months` to view the distribution of loan terms.\n",
    "- Using `loans_by_sector` (the grouped DataFrame created in the last section), create a bar plot of the number of loans in each sector. \n",
    "    - Tip: You can use `df.index` to access the index names.\n",
    "    - Try looking [here](https://stackoverflow.com/questions/10998621/rotate-axis-text-in-python-matplotlib/23009503?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa) for tips on how to rotate tick labels. This was the third Google search result; the first two returned results for more specific scenarios, and the third was exactly what we needed!\n",
    "    - Do we need to include an x-axis label for this chart? Why or why not?\n",
    "- Create a histogram of `loan_amount` to view the distribution of loan amounts.\n",
    "    - Interpret the graph; why did the plotting function set these x-limits?\n",
    "    - How can we work around this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.096523Z",
     "start_time": "2018-04-17T05:50:57.812Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Scatterplot, lender_count vs. loan_amount\n",
    "plt.figure(figsize=(10, 7))\n",
    "plt.scatter(...)\n",
    "plt.xlabel('Funded Amount')\n",
    "plt.ylabel('Loan Amount')\n",
    "plt.title('Loan amount vs. Funded amount')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.097550Z",
     "start_time": "2018-04-17T05:50:57.815Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Histogram, distribution of loan terms\n",
    "plt.figure(figsize=(10, 7))\n",
    "plt.hist(...)\n",
    "# Set an xlabel and a ylabel below!\n",
    "...\n",
    "...\n",
    "plt.title('Distribution of loan term length')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.098483Z",
     "start_time": "2018-04-17T05:50:57.817Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Bar plot, number of loans in each sector\n",
    "plt.figure(figsize=(10, 7))\n",
    "# Try setting up the barplot below. (Not necessarily limited to one line of code)\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.099331Z",
     "start_time": "2018-04-17T05:50:57.820Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Histogram, distribution of loan amount\n",
    "plt.figure(figsize=(10, 7))\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why do we get the above result? We can check the numerical distribution of a column using [`df[colname].describe()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.describe.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.100259Z",
     "start_time": "2018-04-17T05:50:57.825Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Taking a look at the data:\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the max loan is 100000, while the 75th percentile is 1000. This is why Matplotlib has decided to extend out our x-limits so widely, to fit in that max data point! In order to avoid this, we should subset the loan amounts in which most of our data lies; 5000 seems like a reasonable estimate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-04-17T05:50:59.101052Z",
     "start_time": "2018-04-17T05:50:57.827Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Histogram, distribution of loan amount (without outliers):\n",
    "plt.figure(figsize=(10, 7))\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge/Join Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Merge_ and _Join_ are two separate pandas functions but perform similar actions. For the difference between the two: https://stackoverflow.com/questions/22676081/what-is-the-difference-between-join-and-merge-in-pandas\n",
    "\n",
    "The main idea is to combine two dataframes and only keep the rows which have the same certain column values across both dataframes. \n",
    "\n",
    "Walk through the example by running the cells below. \n",
    "\n",
    "\n",
    "We make a dataframe of gdp of 5 countries and a dataframe of the unemployment rate of 3 countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gdp = pd.DataFrame(columns = ['Countries', 'Price (billions of $)' ])\n",
    "gdp['Countries'] = pd.Series(['United States', 'China', 'India', 'Mexico', 'Austria'])\n",
    "gdp['Price (billions of $)'] = pd.Series([20412.87, 14092.51, 2848.23, 1212.83, 477.67])\n",
    "gdp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unemployment = pd.DataFrame(columns = ['Country Name', 'Unemployment Rate' ])\n",
    "unemployment['Country Name'] = pd.Series(['India', 'United States', 'Austria'])\n",
    "unemployment['Unemployment Rate'] = pd.Series([3.7, 5.6, 7.0])\n",
    "unemployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to find the countries for which we have information of both gdp and unemployment rate. \n",
    "\n",
    "So we combine, or _merge_, the gdp and unemployment dataframes based on the names of the countries. Since we are doing gdp _Right Merge/Join_ with unemployment, we keep the rows of the unemployment dataframe where the __Country Name__ column in unemployment = __Countries__ column of gdp.\n",
    "\n",
    "Notice how we had the countries 'India', 'United States', and 'Austria' in our unemployment dataframe and our gdp dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "right_merge_df = pd.merge(gdp, unemployment, how='right', left_on = 'Countries', right_on ='Country Name')\n",
    "right_merge_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then drop the unnecessary __Country Name__ column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "right_merge_df.drop('Country Name', 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "Python Workshop (MASTER).ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}