01-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis and Visualization in Python\n",
    "## Starting With Data\n",
    "Questions\n",
    "* How can I import data in Python?\n",
    "* What is Pandas?\n",
    "* Why should I use Pandas to work with data?\n",
    "\n",
    "Objectives\n",
    "* Load the Python Data Analysis Library (Pandas).\n",
    "* Use `read_csv` to read tabular data into Python.\n",
    "* Describe what a DataFrame is in Python.\n",
    "* Access and summarize data stored in a DataFrame.\n",
    "* Perform basic mathematical operations and summary statistics on data in a Pandas DataFrame.\n",
    "* Create simple plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working With Pandas DataFrames in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas in Python\n",
    "One of the best options for working with tabular data in Python is to use the Python Data Analysis Library (a.k.a. Pandas). The Pandas library provides data structures, produces high quality plots with matplotlib and integrates nicely with other libraries that use NumPy (which is another Python library) arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ### as ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading CSV Data Using Pandas\n",
    "### So What’s a DataFrame?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that pd.read_csv is used because we imported pandas as pd\n",
    "###(\"###/surveys.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### = pd.read_csv(\"data/surveys.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df### # Displays the first several rows of a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df.###\n",
    "# What format does it return the shape of the DataFrame in?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df.###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute descriptive statistics per column\n",
    "print(\"Count:    \", surveys_df['weight'].###())\n",
    "print(\"Mean:     \", surveys_df['weight'].###())\n",
    "print(\"Std Dev.: \", surveys_df['weight'].###())\n",
    "print(\"Min:      \", surveys_df['weight'].###())\n",
    "print(\"Max:      \", surveys_df['weight'].###())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# New column - Convert all weights from grams to kilograms\n",
    "surveys_df[###] = surveys_df[###] ###\n",
    "surveys_df.###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of Data\n",
    "### Checking the format of our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['month']###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Native Python Type | Pandas Type | Description\n",
    "-------------------|-------------|------------\n",
    "`str`              | `object`    | The most general dtype. Will be assigned to your column if column has mixed types (numbers and strings).\n",
    "`int`              | `int64`     | 64 bits integer\n",
    "`float`            | `float64`   | Numeric characters with decimals. If a column contains numbers and NaNs(see below), pandas will default to float64.\n",
    " N/A               | `datetime64`| Values meant to hold time data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working With Our Survey Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summary of descriptive statistics\n",
    "surveys_df###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['month'] = surveys_df['month'].###('###')\n",
    "surveys_df['month'].###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['month'].###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['month'].###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df[###].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Calculating Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1`. What happens if we try to convert `weight` values to `int64` integers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['weight'].###('int64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2`. Try converting the column `plot_id` to native Python `float` data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['plot_id'] ### surveys_df['###'].###(\"###\")\n",
    "surveys_df['plot_id'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3`. Create a list of unique site ID’s (`plot_id`) found in the surveys data. Call it `site_names`. How many unique sites are there in the data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "site_names = surveys_df[###]###\n",
    "site_names.###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4`. What is the difference between `len(site_names)` and `surveys_df['plot_id'].nunique()`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(len(site_names))\n",
    "# print(surveys_df['plot_id'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groups in Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group data by sex\n",
    "### = surveys_df.###('###')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summary statistics for all numeric columns by sex\n",
    "by_sex.###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Provide the mean for each numeric column by sex\n",
    "by_sex.###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Grouping\n",
    "`1`. How many recorded individuals are female `F`, and how many male `M`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_sex###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2`. What happens when you group by two columns using the following syntax and then grab mean values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_site_sex = surveys_df.groupby(['plot_id','sex'])\n",
    "by_site_sex.###()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3`. Summarize `weight` values for each site (`plot_id`) in your data. HINT: it is possible to select a column once the data has been grouped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_site = surveys_df.###(['###'])\n",
    "by_site['###'].###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the Number of Records of One Species"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df['species_id'].###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df.groupby('species_id')['record_id']#.count()###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick & Easy Plotting Data Using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "by_site['record_id'].###().###(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise - Plotting Challenge\n",
    "Create a `line` plot of the median `weight` per month."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Why is the following line necessary?\n",
    "# surveys_df['month'] = surveys_df['month'].astype('int64')\n",
    "surveys_df.###('###')['###'].###().plot(kind='###')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "site_sex_totalw = by_site_sex['weight'].###()\n",
    "site_sex_totalw###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change the right-most categorical values into columns\n",
    "sst = site_sex_totalw.unstack()\n",
    "sst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s_plot = sst.plot(kind='bar', ###,\n",
    "                  title=\"Total weight by site and sex\")\n",
    "s_plot.set_xlabel(\"Site\")\n",
    "s_plot.set_ylabel(\"Weight (g)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}