2017-11-30-Getting-Data-Using-APIs-Answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Women in Data Science ATX Meetup - November 30, 2017\n",
    "\n",
    "# Getting Data Using APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we introduce a few Python libraries for getting and dealing with data using web Application Programming Interfaces (APIs), then work through some examples. Here is a quick overview of the topic directed toward data scientists:\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-apis-application-programming-interfaces-5-apis-a-data-scientist-must-know/\n",
    "\n",
    "In the author's words:\n",
    "> \"In simple words, an API is a (hypothetical) contract between 2 softwares saying if the user software provides input in a pre-defined format, the later with extend its functionality and provide the outcome to the user software. Think of it like this, Graphical user interface (GUI) or command line interface (CLI) allows humans to Interact with code, where as an Application programmable interface (API) allows one piece of code to interact with other code.\n",
    "...\"\n",
    "\n",
    "> \"An API is a set of rules with which the interaction between various entities is defined. We are specifically talking about interaction between two software.\"\n",
    "\n",
    "We'll be accessing data from the following APIs:\n",
    "- [USNO Astronomical Applications Department API](http://aa.usno.navy.mil/data/docs/api.php)\n",
    "- [Google Maps APIs](https://developers.google.com/maps/web-services/)\n",
    "- [OpenWeatherMap API](https://openweathermap.org/api)\n",
    "- [Planet Data API](https://www.planet.com/docs/reference/data-api/)\n",
    "- [Twitter API](https://developer.twitter.com/en/docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Useful Packages\n",
    "We'll use the following packages to make requests, parse responses, and read in authorization credentials for accessing APIs that require them:\n",
    "- [requests](http://docs.python-requests.org/en/master/) \"is the only Non-GMO HTTP library for Python, safe for human consumption\"\n",
    "- [json](https://docs.python.org/3/library/json.html) is a common package for encoding and decoding JSON (JavaScript Object Notation)\n",
    "- [pyyaml](https://github.com/yaml/pyyaml) is a package for reading YAML (Yet Another Markup Language) formatted files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON Package\n",
    "Many APIs these days return data in JavaScript Object Notation (JSON) format. It looks a lot like a Python `dict`, but is a language-agnostic serialization standard. We will use the `json` package for converting JSON-formatted responses to Python data structures such as dictionaries. Double-quotes are a big deal to JSON, whereas Python is all like, \"meh\" (or, 'meh'?).\n",
    "\n",
    "Let's try an example of parsing JSON, taken from \"Data Science from Scratch\" by Joel Grus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Serialized: <class 'str'>\n",
      "{ \"title\" : \"Data Science Book\",\n",
      "                  \"author\" : \"Joel Grus\",\n",
      "                  \"publicationYear\" : 2014,\n",
      "                  \"topics\" : [ \"data\", \"science\", \"data science\"] }\n",
      "\n",
      "...parsing json...\n",
      "\n",
      "Deserialized: <class 'dict'>\n",
      "{'title': 'Data Science Book', 'author': 'Joel Grus', 'publicationYear': 2014, 'topics': ['data', 'science', 'data science']}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "serialized = \"\"\"{ \"title\" : \"Data Science Book\",\n",
    "                  \"author\" : \"Joel Grus\",\n",
    "                  \"publicationYear\" : 2014,\n",
    "                  \"topics\" : [ \"data\", \"science\", \"data science\"] }\"\"\"\n",
    "print(\"Serialized: {}\".format(type(serialized)))\n",
    "print(serialized)\n",
    "print(\"\")\n",
    "\n",
    "# parse the JSON to create a Python object\n",
    "print(\"...parsing json...\")\n",
    "print(\"\")\n",
    "deserialized = json.loads(serialized)\n",
    "if \"data science\" in deserialized[\"topics\"]:\n",
    "    print(\"Deserialized: {}\".format(type(deserialized)))\n",
    "    print(deserialized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requests Package\n",
    "We will use the Python Requests library for making HTTP requests.\n",
    "- http://docs.python-requests.org/en/master/\n",
    "- http://docs.python-requests.org/en/master/user/quickstart/\n",
    "\n",
    "This package is not (yet) in the Python standard library, but it's really nice to use and well-documented. Other packages are `http.client` and `urllib`. See here: https://docs.python.org/3/library/internet.html\n",
    "\n",
    "Let's try another example from \"Data Science from Scratch\" for making a `GET` `HTTP` request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "\n",
    "endpoint = \"https://api.github.com/users/joelgrus/repos\"\n",
    "\n",
    "response = requests.get(endpoint)\n",
    "\n",
    "repos = json.loads(response.text)  # \"loads\" means \"load string\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `json.loads()` for decoding **from** JSON and `json.dumps()` for encoding **to** JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise:** Replace `json.loads(response.text)` with `response.json()` and verify that they return the same thing.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 11189868, 'name': 'bitstarter', 'full_name': 'joelgrus/bitstarter', 'owner': {'login': 'joelgrus', 'id': 1308313, 'avatar_url': 'https://avatars1.githubusercontent.com/u/1308313?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/joelgrus', 'html_url': 'https://github.com/joelgrus', 'followers_url': 'https://api.github.com/users/joelgrus/followers', 'following_url': 'https://api.github.com/users/joelgrus/following{/other_user}', 'gists_url': 'https://api.github.com/users/joelgrus/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/joelgrus/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/joelgrus/subscriptions', 'organizations_url': 'https://api.github.com/users/joelgrus/orgs', 'repos_url': 'https://api.github.com/users/joelgrus/repos', 'events_url': 'https://api.github.com/users/joelgrus/events{/privacy}', 'received_events_url': 'https://api.github.com/users/joelgrus/received_events', 'type': 'User', 'site_admin': False}, 'private': False, 'html_url': 'https://github.com/joelgrus/bitstarter', 'description': None, 'fork': True, 'url': 'https://api.github.com/repos/joelgrus/bitstarter', 'forks_url': 'https://api.github.com/repos/joelgrus/bitstarter/forks', 'keys_url': 'https://api.github.com/repos/joelgrus/bitstarter/keys{/key_id}', 'collaborators_url': 'https://api.github.com/repos/joelgrus/bitstarter/collaborators{/collaborator}', 'teams_url': 'https://api.github.com/repos/joelgrus/bitstarter/teams', 'hooks_url': 'https://api.github.com/repos/joelgrus/bitstarter/hooks', 'issue_events_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues/events{/number}', 'events_url': 'https://api.github.com/repos/joelgrus/bitstarter/events', 'assignees_url': 'https://api.github.com/repos/joelgrus/bitstarter/assignees{/user}', 'branches_url': 'https://api.github.com/repos/joelgrus/bitstarter/branches{/branch}', 'tags_url': 'https://api.github.com/repos/joelgrus/bitstarter/tags', 'blobs_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/blobs{/sha}', 'git_tags_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/tags{/sha}', 'git_refs_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/refs{/sha}', 'trees_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/trees{/sha}', 'statuses_url': 'https://api.github.com/repos/joelgrus/bitstarter/statuses/{sha}', 'languages_url': 'https://api.github.com/repos/joelgrus/bitstarter/languages', 'stargazers_url': 'https://api.github.com/repos/joelgrus/bitstarter/stargazers', 'contributors_url': 'https://api.github.com/repos/joelgrus/bitstarter/contributors', 'subscribers_url': 'https://api.github.com/repos/joelgrus/bitstarter/subscribers', 'subscription_url': 'https://api.github.com/repos/joelgrus/bitstarter/subscription', 'commits_url': 'https://api.github.com/repos/joelgrus/bitstarter/commits{/sha}', 'git_commits_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/commits{/sha}', 'comments_url': 'https://api.github.com/repos/joelgrus/bitstarter/comments{/number}', 'issue_comment_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues/comments{/number}', 'contents_url': 'https://api.github.com/repos/joelgrus/bitstarter/contents/{+path}', 'compare_url': 'https://api.github.com/repos/joelgrus/bitstarter/compare/{base}...{head}', 'merges_url': 'https://api.github.com/repos/joelgrus/bitstarter/merges', 'archive_url': 'https://api.github.com/repos/joelgrus/bitstarter/{archive_format}{/ref}', 'downloads_url': 'https://api.github.com/repos/joelgrus/bitstarter/downloads', 'issues_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues{/number}', 'pulls_url': 'https://api.github.com/repos/joelgrus/bitstarter/pulls{/number}', 'milestones_url': 'https://api.github.com/repos/joelgrus/bitstarter/milestones{/number}', 'notifications_url': 'https://api.github.com/repos/joelgrus/bitstarter/notifications{?since,all,participating}', 'labels_url': 'https://api.github.com/repos/joelgrus/bitstarter/labels{/name}', 'releases_url': 'https://api.github.com/repos/joelgrus/bitstarter/releases{/id}', 'deployments_url': 'https://api.github.com/repos/joelgrus/bitstarter/deployments', 'created_at': '2013-07-05T02:02:28Z', 'updated_at': '2013-07-17T17:59:24Z', 'pushed_at': '2013-07-17T17:59:11Z', 'git_url': 'git://github.com/joelgrus/bitstarter.git', 'ssh_url': 'git@github.com:joelgrus/bitstarter.git', 'clone_url': 'https://github.com/joelgrus/bitstarter.git', 'svn_url': 'https://github.com/joelgrus/bitstarter', 'homepage': None, 'size': 161, 'stargazers_count': 0, 'watchers_count': 0, 'language': 'JavaScript', 'has_issues': False, 'has_projects': True, 'has_downloads': True, 'has_wiki': True, 'has_pages': False, 'forks_count': 0, 'mirror_url': None, 'archived': False, 'open_issues_count': 0, 'license': None, 'forks': 0, 'open_issues': 0, 'watchers': 0, 'default_branch': 'master'}\n"
     ]
    }
   ],
   "source": [
    "print(json.loads(response.text)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 11189868, 'name': 'bitstarter', 'full_name': 'joelgrus/bitstarter', 'owner': {'login': 'joelgrus', 'id': 1308313, 'avatar_url': 'https://avatars1.githubusercontent.com/u/1308313?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/joelgrus', 'html_url': 'https://github.com/joelgrus', 'followers_url': 'https://api.github.com/users/joelgrus/followers', 'following_url': 'https://api.github.com/users/joelgrus/following{/other_user}', 'gists_url': 'https://api.github.com/users/joelgrus/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/joelgrus/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/joelgrus/subscriptions', 'organizations_url': 'https://api.github.com/users/joelgrus/orgs', 'repos_url': 'https://api.github.com/users/joelgrus/repos', 'events_url': 'https://api.github.com/users/joelgrus/events{/privacy}', 'received_events_url': 'https://api.github.com/users/joelgrus/received_events', 'type': 'User', 'site_admin': False}, 'private': False, 'html_url': 'https://github.com/joelgrus/bitstarter', 'description': None, 'fork': True, 'url': 'https://api.github.com/repos/joelgrus/bitstarter', 'forks_url': 'https://api.github.com/repos/joelgrus/bitstarter/forks', 'keys_url': 'https://api.github.com/repos/joelgrus/bitstarter/keys{/key_id}', 'collaborators_url': 'https://api.github.com/repos/joelgrus/bitstarter/collaborators{/collaborator}', 'teams_url': 'https://api.github.com/repos/joelgrus/bitstarter/teams', 'hooks_url': 'https://api.github.com/repos/joelgrus/bitstarter/hooks', 'issue_events_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues/events{/number}', 'events_url': 'https://api.github.com/repos/joelgrus/bitstarter/events', 'assignees_url': 'https://api.github.com/repos/joelgrus/bitstarter/assignees{/user}', 'branches_url': 'https://api.github.com/repos/joelgrus/bitstarter/branches{/branch}', 'tags_url': 'https://api.github.com/repos/joelgrus/bitstarter/tags', 'blobs_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/blobs{/sha}', 'git_tags_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/tags{/sha}', 'git_refs_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/refs{/sha}', 'trees_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/trees{/sha}', 'statuses_url': 'https://api.github.com/repos/joelgrus/bitstarter/statuses/{sha}', 'languages_url': 'https://api.github.com/repos/joelgrus/bitstarter/languages', 'stargazers_url': 'https://api.github.com/repos/joelgrus/bitstarter/stargazers', 'contributors_url': 'https://api.github.com/repos/joelgrus/bitstarter/contributors', 'subscribers_url': 'https://api.github.com/repos/joelgrus/bitstarter/subscribers', 'subscription_url': 'https://api.github.com/repos/joelgrus/bitstarter/subscription', 'commits_url': 'https://api.github.com/repos/joelgrus/bitstarter/commits{/sha}', 'git_commits_url': 'https://api.github.com/repos/joelgrus/bitstarter/git/commits{/sha}', 'comments_url': 'https://api.github.com/repos/joelgrus/bitstarter/comments{/number}', 'issue_comment_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues/comments{/number}', 'contents_url': 'https://api.github.com/repos/joelgrus/bitstarter/contents/{+path}', 'compare_url': 'https://api.github.com/repos/joelgrus/bitstarter/compare/{base}...{head}', 'merges_url': 'https://api.github.com/repos/joelgrus/bitstarter/merges', 'archive_url': 'https://api.github.com/repos/joelgrus/bitstarter/{archive_format}{/ref}', 'downloads_url': 'https://api.github.com/repos/joelgrus/bitstarter/downloads', 'issues_url': 'https://api.github.com/repos/joelgrus/bitstarter/issues{/number}', 'pulls_url': 'https://api.github.com/repos/joelgrus/bitstarter/pulls{/number}', 'milestones_url': 'https://api.github.com/repos/joelgrus/bitstarter/milestones{/number}', 'notifications_url': 'https://api.github.com/repos/joelgrus/bitstarter/notifications{?since,all,participating}', 'labels_url': 'https://api.github.com/repos/joelgrus/bitstarter/labels{/name}', 'releases_url': 'https://api.github.com/repos/joelgrus/bitstarter/releases{/id}', 'deployments_url': 'https://api.github.com/repos/joelgrus/bitstarter/deployments', 'created_at': '2013-07-05T02:02:28Z', 'updated_at': '2013-07-17T17:59:24Z', 'pushed_at': '2013-07-17T17:59:11Z', 'git_url': 'git://github.com/joelgrus/bitstarter.git', 'ssh_url': 'git@github.com:joelgrus/bitstarter.git', 'clone_url': 'https://github.com/joelgrus/bitstarter.git', 'svn_url': 'https://github.com/joelgrus/bitstarter', 'homepage': None, 'size': 161, 'stargazers_count': 0, 'watchers_count': 0, 'language': 'JavaScript', 'has_issues': False, 'has_projects': True, 'has_downloads': True, 'has_wiki': True, 'has_pages': False, 'forks_count': 0, 'mirror_url': None, 'archived': False, 'open_issues_count': 0, 'license': None, 'forks': 0, 'open_issues': 0, 'watchers': 0, 'default_branch': 'master'}\n"
     ]
    }
   ],
   "source": [
    "print(response.json()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise:** If you have a GitHub account, replace `joelgrus` with your username and explore.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering\n",
      "data-science-from-scratch-workshop\n",
      "Decision-Trees\n",
      "KNIME_2017_Workshop\n",
      "logistic-regression\n",
      "Meetup-Slides\n",
      "multiple-linear-regression-and-gradient-descent\n",
      "NaiveBayesPresentation\n",
      "Neural_Networks\n",
      "PCA_SVD\n",
      "RecommenderSystems\n",
      "SQLworkshop\n",
      "stats_and_regression\n",
      "titanic-data-munging\n",
      "titanic-EDA\n",
      "titanic-hypothesis-testing\n",
      "wids-github\n",
      "widsatx-mapreduce\n",
      "widsatx-nlp\n",
      "widsatx-python\n",
      "wids_network_analysis\n"
     ]
    }
   ],
   "source": [
    "endpoint = \"https://api.github.com/users/womenindatascienceatx/repos\"\n",
    "response = requests.get(endpoint)\n",
    "repos = json.loads(response.text)  # \"loads\" means \"load string\"\n",
    "[print(repo['name']) for repo in repos];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise:** The `repos` object above is a list of dicts. What do you get when you make a `GET` request to the endpoints `https://api.github.com` and `https://api.github.com/users`?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "endpoint = \"https://api.github.com\"\n",
    "response = requests.get(endpoint)\n",
    "print(type(response.json()))\n",
    "\n",
    "endpoint = \"https://api.github.com/users\"\n",
    "response = requests.get(endpoint)\n",
    "print(type(response.json()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look at what got returned (a list of dicts where each element of the list is a repo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['archive_url',\n",
       " 'archived',\n",
       " 'assignees_url',\n",
       " 'blobs_url',\n",
       " 'branches_url',\n",
       " 'clone_url',\n",
       " 'collaborators_url',\n",
       " 'comments_url',\n",
       " 'commits_url',\n",
       " 'compare_url',\n",
       " 'contents_url',\n",
       " 'contributors_url',\n",
       " 'created_at',\n",
       " 'default_branch',\n",
       " 'deployments_url',\n",
       " 'description',\n",
       " 'downloads_url',\n",
       " 'events_url',\n",
       " 'fork',\n",
       " 'forks',\n",
       " 'forks_count',\n",
       " 'forks_url',\n",
       " 'full_name',\n",
       " 'git_commits_url',\n",
       " 'git_refs_url',\n",
       " 'git_tags_url',\n",
       " 'git_url',\n",
       " 'has_downloads',\n",
       " 'has_issues',\n",
       " 'has_pages',\n",
       " 'has_projects',\n",
       " 'has_wiki',\n",
       " 'homepage',\n",
       " 'hooks_url',\n",
       " 'html_url',\n",
       " 'id',\n",
       " 'issue_comment_url',\n",
       " 'issue_events_url',\n",
       " 'issues_url',\n",
       " 'keys_url',\n",
       " 'labels_url',\n",
       " 'language',\n",
       " 'languages_url',\n",
       " 'license',\n",
       " 'merges_url',\n",
       " 'milestones_url',\n",
       " 'mirror_url',\n",
       " 'name',\n",
       " 'notifications_url',\n",
       " 'open_issues',\n",
       " 'open_issues_count',\n",
       " 'owner',\n",
       " 'private',\n",
       " 'pulls_url',\n",
       " 'pushed_at',\n",
       " 'releases_url',\n",
       " 'size',\n",
       " 'ssh_url',\n",
       " 'stargazers_count',\n",
       " 'stargazers_url',\n",
       " 'statuses_url',\n",
       " 'subscribers_url',\n",
       " 'subscription_url',\n",
       " 'svn_url',\n",
       " 'tags_url',\n",
       " 'teams_url',\n",
       " 'trees_url',\n",
       " 'updated_at',\n",
       " 'url',\n",
       " 'watchers',\n",
       " 'watchers_count']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(repos[0].keys())  # dictionary keys of the first repo in the list of repos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will use `dateutil.parser` to convert from ISO datetime format to a datetime object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2017-02-23T21:58:30Z'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repos[0]['created_at']  # ISO datetime format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2017, 2, 23, 21, 58, 30, tzinfo=tzutc())"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "from dateutil.parser import parse\n",
    "\n",
    "parse(repos[0]['created_at'])  # datetime object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, count up the number of repos created by month and by weekday using a `Counter` object from the `collections` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dates = [parse(repo[\"created_at\"]) for repo in repos]\n",
    "month_counts = Counter(date.month for date in dates)\n",
    "weekday_counts = Counter(date.weekday() for date in dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of repos created by month: [(1, 1), (2, 1), (3, 2), (4, 2), (6, 4), (7, 2), (8, 2), (9, 4), (10, 2), (11, 1)]\n",
      "Number of repos created by weekday: [(0, 1), (1, 1), (2, 3), (3, 13), (4, 3)]\n"
     ]
    }
   ],
   "source": [
    "# Sort by month/weekday\n",
    "print(\"Number of repos created by month:\", sorted(month_counts.items()))\n",
    "print(\"Number of repos created by weekday:\", sorted(weekday_counts.items()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(9, 4), (6, 4), (3, 2), (10, 2), (8, 2), (7, 2), (4, 2), (2, 1), (11, 1), (1, 1)]\n",
      "[(3, 13), (2, 3), (4, 3), (0, 1), (1, 1)]\n"
     ]
    }
   ],
   "source": [
    "# Sort by counts\n",
    "print(month_counts.most_common())  # What month were the most repos created?\n",
    "print(weekday_counts.most_common())  # which day(s) of the week were the most repos created?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyYaml Library\n",
    "We will use a YAML-formatted (Yet Another Markup Language) file which we call `config.yml` for keeping configuration data all in one place, and for keeping API keys and tokens secret. \n",
    "\n",
    "You will find a file named `config.yml.template` in the same directory as this notebook. It contains entries like this:\n",
    "\n",
    "```\n",
    "...\n",
    "planet:\n",
    "    url: https://api.planet.com/data/v1\n",
    "    key: {PLANET_API_KEY}\n",
    "...\n",
    "```\n",
    "\n",
    "Please keep that file open in an editor and, as we go through the tutorial, replace the values in curly braces with your personal API keys.\n",
    "\n",
    "For example, when we get to the section on on the Planet API you would replace `{PLANET_API_KEY}` with a key looking something like this: `a3a64774d30c4749826b6be445489d3b` (not a real key, but you can generate one by signing up for an account).\n",
    "\n",
    "**IMPORTANT NOTE: DO NOT COMMIT config.yml TO GITHUB!**\n",
    "If you plan to commit any code from this tutorial to GitHub, ensure that `config.yml` is in the repository's `.gitignore` file.\n",
    "\n",
    "Next, let's make a function for loading our configuration file. We can call this whenever we make a change to `config.yml` and want to re-load the file to memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "def load_config():\n",
    "    \"\"\"load the configuration file as a python dictionary\"\"\"\n",
    "    with open(\"config.yml\", 'r') as ymlfile:\n",
    "        cfg = yaml.load(ymlfile)\n",
    "    return cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## USNO Astronomical Applications Department API\n",
    "http://aa.usno.navy.mil/data/docs/api.php\n",
    "\n",
    "This API does not require any authentication credentials, so we can immediately start doing `GET` requests.\n",
    "\n",
    "In general, an API request takes the form:\n",
    "```\n",
    "http://api.usno.navy.mil/<web_service>?<parameters>\n",
    "```\n",
    "\n",
    "The available web services are:\n",
    "- `imagery`: synthetic images of astronomical bodies under a set of conditions\n",
    "- `moon`: dates and times of a list of primary moon phases\n",
    "- `rstt`: rise, set, and transit times for the Sun and Moon\n",
    "- `sidtime`: Greenwich mean and apparent sidereal time, local mean and apparent sidereal time, and the Equation of the Equinoxes\n",
    "- `eclipses/solar`: local circumstances for solar eclipses\n",
    "- `christian`: selected Christian observances\n",
    "- `jewish`: selected Jewish observances\n",
    "- `islamic`: selected Islamic observcances\n",
    "- `jdconverter`: converts dates between the Julian/Gregorian calendar and Julian date\n",
    "\n",
    "You can find a description of the various parameters in the docs link above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "configuration = load_config()  # load config.yml\n",
    "\n",
    "ECLIPSE_API_URL = configuration['usno_solar_eclipse']['url']\n",
    "## NOTE: This API doesn't require a key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_local_eclipse_data(geocode):\n",
    "    \"\"\"Get data on the solar eclipse at a particular location.\"\"\"\n",
    "    query_params = {\n",
    "        \"date\": \"8/21/2017\",\n",
    "        \"coords\": geocode,\n",
    "        \"format\": \"json\"\n",
    "    }\n",
    "    response = requests.get(\n",
    "        ECLIPSE_API_URL,\n",
    "        params=query_params\n",
    "    )\n",
    "    return response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'apiversion': '2.0.0',\n",
       " 'day': 21,\n",
       " 'deltaT': '69.4s',\n",
       " 'description': 'Sun in Partial Eclipse at this Location',\n",
       " 'duration': '0h 11m 08.6s',\n",
       " 'error': False,\n",
       " 'event': 'Solar Eclipse of 2017 Aug. 21',\n",
       " 'height': '0m',\n",
       " 'lat': '46.670000',\n",
       " 'local_data': [{'altitude': '1.0',\n",
       "   'azimuth': '286.3',\n",
       "   'day': '21',\n",
       "   'phenomenon': 'Eclipse Begins',\n",
       "   'position_angle': '229.3',\n",
       "   'time': '18:42:26.8',\n",
       "   'vertex_angle': '187.0'},\n",
       "  {'altitude': '----',\n",
       "   'azimuth': '288.3',\n",
       "   'day': '21',\n",
       "   'phenomenon': 'Sunset',\n",
       "   'time': '18:54'}],\n",
       " 'lon': '1.480000',\n",
       " 'magnitude': '0.158',\n",
       " 'month': 8,\n",
       " 'obscuration': '7.4%',\n",
       " 'tz': '0',\n",
       " 'year': 2017}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geocode = \"46.67,1.48\"\n",
    "get_local_eclipse_data(geocode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Maps APIs\n",
    "\n",
    "Google provides lots of useful public APIs, including their Maps APIs:\n",
    "\n",
    "https://developers.google.com/maps/web-services/\n",
    "\n",
    "We will use Google Map's geocoding API to get latitude and longitude coordinates (a geocode) for a given postcode.\n",
    "\n",
    "For contrast, we'll use the `geopy` Python client wrapper for getting a timezone ID after getting a geocode from a postcode.\n",
    "\n",
    "From their GitHub page: \"geopy is a Python 2 and 3 client for several popular geocoding web services.\"\n",
    "\n",
    "https://github.com/geopy/geopy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "GEOCODING_API_URL = configuration['google_geocoding']['url']\n",
    "GEOCODING_API_KEY = configuration['google_geocoding']['key']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "def geocode_postcode_data(postcode):\n",
    "    \"\"\"Use Google geocode API to get geocode from postcode.\"\"\"\n",
    "    print(\"get the geocode for postcode {}\".format(postcode))\n",
    "    if not postcode:\n",
    "        return None\n",
    "    time.sleep(1)  # avoid rate limiting if calling this function in a loop\n",
    "    url = \"{}/json?components=postal_code:{}&key={}\".format(\n",
    "        GEOCODING_API_URL,\n",
    "        postcode,\n",
    "        GEOCODING_API_KEY\n",
    "    )\n",
    "    r = requests.get(url)\n",
    "    postcode_geocode = None\n",
    "    if r.status_code == 200:\n",
    "        results = r.json()['results']\n",
    "        if len(results) > 0:\n",
    "            location = results[0]['geometry']['location']\n",
    "            postcode_geocode = \"{},{}\".format(location['lat'], location['lng'])\n",
    "    return postcode_geocode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from geopy import geocoders\n",
    "\n",
    "\n",
    "def get_timezone(geocode):\n",
    "    \"\"\"Use geopy client to get timezone from lat/lon.\"\"\"\n",
    "    # Localize the event times\n",
    "    lat = float(geocode.split(',')[0])\n",
    "    lon = float(geocode.split(',')[1])\n",
    "    # Get timezone from lat/lon\n",
    "    g = geocoders.GoogleV3()\n",
    "    timezone = str(g.timezone((lat, lon)))\n",
    "    return timezone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's write a function to localize a datetime object to a particular timezone and format it as a more easily human-readable string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import pytz\n",
    "\n",
    "\n",
    "def localize_time(time_string, timezone_id):\n",
    "    time_split = time_string.split(':')\n",
    "    datetime = dt.datetime(2017, 8, 21, int(time_split[0]), int(time_split[1]))\n",
    "    utc_datetime = datetime.replace(tzinfo=pytz.utc)\n",
    "    time_format = '%-I:%M %p %Z'  # e.g., '1:44 PM EST'\n",
    "\n",
    "    # Convert timezone_id to pytz timezone object\n",
    "    timezone = pytz.timezone(timezone_id)\n",
    "\n",
    "    # Localize timezone-aware datetime object\n",
    "    localized_datetime = utc_datetime.astimezone(timezone)\n",
    "\n",
    "    return localized_datetime.strftime(time_format)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'timezone_id' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-b98958117d62>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlocalize_time\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"16:41:01.2\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimezone_id\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'timezone_id' is not defined"
     ]
    }
   ],
   "source": [
    "localize_time(\"16:41:01.2\", timezone_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise:** Use the above functions to get and print the begin time, the end time, and the maximum obscuration for the August 21, 2017 eclipse in your zip code and your time zone.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MY_POSTCODE = 78731\n",
    "geocode = geocode_postcode_data(MY_POSTCODE)\n",
    "print(geocode)\n",
    "print(\"\")\n",
    "\n",
    "dict_eclipse_data = get_local_eclipse_data(geocode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_eclipse_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for local in dict_eclipse_data['local_data']:\n",
    "    print(local['phenomenon'], local['time'])\n",
    "    \n",
    "obscuration = dict_eclipse_data['obscuration']\n",
    "print(\"Obscuration at Maximum Eclipse: {}\".format(obscuration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timezone_id = get_timezone(geocode)\n",
    "print(timezone_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for local in dict_eclipse_data['local_data']:\n",
    "    print(local['phenomenon'], localize_time(local['time'], timezone_id))\n",
    "    \n",
    "obscuration = dict_eclipse_data['obscuration']\n",
    "print(\"Obscuration at Maximum Eclipse: {}\".format(obscuration))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Planet Data API\n",
    "\n",
    "Sign up for an Planet account [here](https://www.planet.com/account/#/). Log in and copy/paste your API key to your `config.yml` file.\n",
    "\n",
    "Docs: https://www.planet.com/docs/reference/data-api/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the needed configuration file variables\n",
    "cfg = load_config()\n",
    "PLANET_API_URL = cfg['planet']['url']\n",
    "PLANET_API_KEY = cfg['planet']['key']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example is taken from here:\n",
    "- https://github.com/planetlabs/notebooks/blob/master/jupyter-notebooks/data-api-tutorials/search_and_download_quickstart.ipynb."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define an Area of Interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Stockton, CA bounding box (created via geojson.io) \n",
    "geojson_geometry = {\n",
    "  \"type\": \"Polygon\",\n",
    "  \"coordinates\": [\n",
    "    [ \n",
    "      [-121.59290313720705, 37.93444993515032],\n",
    "      [-121.27017974853516, 37.93444993515032],\n",
    "      [-121.27017974853516, 38.065932950547484],\n",
    "      [-121.59290313720705, 38.065932950547484],\n",
    "      [-121.59290313720705, 37.93444993515032]\n",
    "    ]\n",
    "  ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# # Austin, TX bounding box (created via geojson.io)\n",
    "# geojson_geometry = {\n",
    "#     \"type\": \"Polygon\",\n",
    "#     \"coordinates\": [\n",
    "#       [\n",
    "#         [-97.84698486328125, 30.115433670851925],\n",
    "#         [-97.63481140136719, 30.115433670851925],\n",
    "#         [-97.63481140136719, 30.433281874927655],\n",
    "#         [-97.84698486328125, 30.433281874927655],\n",
    "#         [-97.84698486328125, 30.115433670851925]\n",
    "#       ]\n",
    "#     ]\n",
    "# }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get images that overlap with our AOI \n",
    "geometry_filter = {\n",
    "  \"type\": \"GeometryFilter\",\n",
    "  \"field_name\": \"geometry\",\n",
    "  \"config\": geojson_geometry\n",
    "}\n",
    "\n",
    "# get images acquired within a date range\n",
    "date_range_filter = {\n",
    "  \"type\": \"DateRangeFilter\",\n",
    "  \"field_name\": \"acquired\",\n",
    "  \"config\": {\n",
    "    \"gte\": \"2016-08-31T00:00:00.000Z\",\n",
    "    \"lte\": \"2016-09-01T00:00:00.000Z\"\n",
    "  }\n",
    "}\n",
    "\n",
    "# only get images which have <50% cloud coverage\n",
    "cloud_cover_filter = {\n",
    "  \"type\": \"RangeFilter\",\n",
    "  \"field_name\": \"cloud_cover\",\n",
    "  \"config\": {\n",
    "    \"lte\": 0.5\n",
    "  }\n",
    "}\n",
    "\n",
    "# combine our geo, date, cloud filters\n",
    "combined_filter = {\n",
    "  \"type\": \"AndFilter\",\n",
    "  \"config\": [geometry_filter, date_range_filter, cloud_cover_filter]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching: Items and Assets\n",
    "You can learn more about item & asset types in Planet's Data API [here](https://www.planet.com/docs/reference/data-api/items-assets/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import requests\n",
    "from requests.auth import HTTPBasicAuth\n",
    "\n",
    "\n",
    "item_type = \"PSScene3Band\"\n",
    "\n",
    "# API request object\n",
    "search_request = {\n",
    "  \"interval\": \"day\",\n",
    "  \"item_types\": [item_type], \n",
    "  \"filter\": combined_filter\n",
    "}\n",
    "\n",
    "# fire off the POST request\n",
    "search_result = \\\n",
    "  requests.post(\n",
    "    'https://api.planet.com/data/v1/quick-search',\n",
    "    auth=HTTPBasicAuth(PLANET_API_KEY, ''),\n",
    "    json=search_request)\n",
    "\n",
    "print(json.dumps(search_result.json(), indent=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract image IDs only\n",
    "image_ids = [feature['id'] for feature in search_result.json()['features']]\n",
    "print(image_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For demo purposes, just grab the first image ID\n",
    "id0 = image_ids[0]\n",
    "id0_url = 'https://api.planet.com/data/v1/item-types/{}/items/{}/assets'.format(item_type, id0)\n",
    "\n",
    "# Returns JSON metadata for assets in this ID. Learn more: planet.com/docs/reference/data-api/items-assets/#asset\n",
    "result = \\\n",
    "  requests.get(\n",
    "    id0_url,\n",
    "    auth=HTTPBasicAuth(PLANET_API_KEY, '')\n",
    "  )\n",
    "\n",
    "# List of asset types available for this particular satellite image\n",
    "print(result.json().keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activation and Downloading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is \"inactive\" if the \"visual\" asset has not yet been activated; otherwise 'active'\n",
    "print(result.json()['visual']['status'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Parse out useful links\n",
    "links = result.json()[u\"visual\"][\"_links\"]\n",
    "self_link = links[\"_self\"]\n",
    "activation_link = links[\"activate\"]\n",
    "\n",
    "# Request activation of the 'visual' asset:\n",
    "activate_result = \\\n",
    "  requests.get(\n",
    "    activation_link,\n",
    "    auth=HTTPBasicAuth(PLANET_API_KEY, '')\n",
    "  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activation_status_result = \\\n",
    "  requests.get(\n",
    "    self_link,\n",
    "    auth=HTTPBasicAuth(PLANET_API_KEY, '')\n",
    "  )\n",
    "    \n",
    "print(activation_status_result.json()[\"status\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image can be downloaded by making a GET with your Planet API key, from here:\n",
    "download_link = activation_status_result.json()[\"location\"]\n",
    "print(download_link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Twitter APIs\n",
    "https://developer.twitter.com/en/docs\n",
    "\n",
    "In this case there is a well-written Python client wrapper called `twython` that can be used in lieue of the lower level `requests` library (or the lower-level `http.client` library).\n",
    "\n",
    "This paragraph from \"Data Science from Scratch\" is on point:\n",
    "\n",
    "> \"Typically we won’t be working with APIs at this low “make the requests and parse the\n",
    "responses ourselves” level. One of the benefits of using Python is that someone has\n",
    "already built a library for pretty much any API you’re interested in accessing. When\n",
    "they’re done well, these libraries can save you a lot of the trouble of figuring out the\n",
    "hairier details of API access. (When they’re not done well, or when it turns out they’re\n",
    "based on defunct versions of the corresponding APIs, they can cause you enormous\n",
    "headaches.)\n",
    "\n",
    "> Nonetheless, you’ll occasionally have to roll your own API-access library (or, more\n",
    "likely, debug why someone else’s isn’t working), so it’s good to know some of the\n",
    "details.\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get  Your Twitter Credentials\n",
    "Sign in to your Twitter account and go to https://apps.twitter.com/. We'll go over this part together.\n",
    "\n",
    "The following example is taken from here:\n",
    "- https://github.com/joelgrus/data-science-from-scratch/blob/master/code/getting_data.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "configuration = load_config()\n",
    "\n",
    "# these should be in your \n",
    "CONSUMER_KEY = configuration['twitter']['consumer_key']\n",
    "CONSUMER_SECRET = configuration['twitter']['consumer_secret']\n",
    "ACCESS_TOKEN = configuration['twitter']['access_token']\n",
    "ACCESS_TOKEN_SECRET = configuration['twitter']['access_token_secret']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twitter Search API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from twython import Twython\n",
    "\n",
    "\n",
    "def call_twitter_search_api():\n",
    "\n",
    "    twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET)\n",
    "\n",
    "    # search for tweets containing the phrase \"data science\"\n",
    "    for status in twitter.search(q='\"data science\"')[\"statuses\"]:\n",
    "        user = status[\"user\"][\"screen_name\"].encode('utf-8')\n",
    "        text = status[\"text\"].encode('utf-8')\n",
    "        print(user, \":\", text)\n",
    "        print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for tweets containing the phrase \"data science\"\n",
    "call_twitter_search_api()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twitter Streaming API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from twython import TwythonStreamer\n",
    "\n",
    "\n",
    "# appending data to a global variable is pretty poor form\n",
    "# but it makes the example much simpler\n",
    "tweets = [] \n",
    "\n",
    "class MyStreamer(TwythonStreamer):\n",
    "    \"\"\"our own subclass of TwythonStreamer that specifies\n",
    "    how to interact with the stream\"\"\"\n",
    "\n",
    "    def on_success(self, data):\n",
    "        \"\"\"what do we do when twitter sends us data?\n",
    "        here data will be a Python object representing a tweet\"\"\"\n",
    "\n",
    "        # only want to collect English-language tweets\n",
    "        if data['lang'] == 'en':\n",
    "            tweets.append(data)\n",
    "\n",
    "        # stop when we've collected enough\n",
    "        if len(tweets) >= 100:\n",
    "            self.disconnect()\n",
    "\n",
    "    def on_error(self, status_code, data):\n",
    "        print(status_code, data)\n",
    "        self.disconnect()\n",
    "\n",
    "def call_twitter_streaming_api():\n",
    "    stream = MyStreamer(CONSUMER_KEY, CONSUMER_SECRET, \n",
    "                        ACCESS_TOKEN, ACCESS_TOKEN_SECRET)\n",
    "\n",
    "    # starts consuming public statuses that contain the keyword 'data'\n",
    "    stream.statuses.filter(track='data')\n",
    "    \n",
    "    ## if instead we wanted to start consuming a sample of *all* public statuses\n",
    "    # stream.statuses.sample()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "call_twitter_streaming_api()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(tweets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise:** Find the most common hashtags in `tweets`.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_hashtags = Counter(\n",
    "    hashtag['text'].lower()\n",
    "    for tweet in tweets\n",
    "    for hashtag in tweet[\"entities\"][\"hashtags\"]\n",
    ")\n",
    "print(top_hashtags.most_common(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thanks for Coming!\n",
    "\n",
    "- agarwal.meghann@gmail.com\n",
    "- https://www.linkedin.com/in/meghann-agarwal/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}