scraping-html.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting data from HTML\n",
    "\n",
    "By [Allison Parrish](http://www.decontextualize.com/)\n",
    "\n",
    "HTML stands for \"hypertext markup language.\" Most of the documents you see when you're browsing the web are written in this format.  Our goal today is to discuss and put into practice basic methods for extracting data from documents written in HTML.\n",
    "\n",
    "In most browsers, there's a \"View Source\" option that allows you to see the HTML source code for any page you're looking at. For example, in Chrome, you can CTRL-click anywhere on the page, or go to `View > Developer > View Source`:\n",
    "\n",
    "<a href=\"http://static.decontextualize.com/snaps/nytimes-view-source.png\"><img src=\"http://static.decontextualize.com/snaps/nytimes-view-source.png\" alt=\"nytimes-view-source\"/></a>\n",
    "\n",
    "You'll see something that looks like this, a mish-mash of angle brackets and quotes and slashes and text. This is HTML.\n",
    "\n",
    "<a href=\"http://static.decontextualize.com/snaps/nytimes-source.png\"><img src=\"http://static.decontextualize.com/snaps/nytimes-source.png\" alt=\"nytimes-source\"/></a>\n",
    "\n",
    "### What HTML looks like\n",
    "\n",
    "HTML consists of a series of *tags*. Tags have a *name*, a series of key/value pairs called *attributes*, and some textual *content*. Attributes are optional. Here's a simple example, using the HTML `<p>` tag (`p` means \"paragraph\"):\n",
    "\n",
    "    <p>Mother said there'd be days like these.</p>\n",
    "    \n",
    "This example has just one tag in it: a `<p>` tag. The source code for a tag has two parts, its opening tag (`<p>`) and its closing tag (`</p>`). In between the opening and closing tag, you see the tag's contents (in this case, the text `Mother said there'd be days like these.`).\n",
    "\n",
    "Here's another example, using the HTML `<div>` tag:\n",
    "\n",
    "    <div class=\"header\" style=\"background: blue;\">Mammoth Falls</div>\n",
    "    \n",
    "In this example, the tag's name is `div`. The tag has two attributes: `class`, with value `header`, and `style`, with value `background: blue;`. The contents of this tag is `Mammoth Falls`.\n",
    "\n",
    "Tags can contain other tags, in a hierarchical relationship. For example, here's some HTML to make a bulletted list:\n",
    "\n",
    "    <ul>\n",
    "      <li>Item one</li>\n",
    "      <li>Item two</li>\n",
    "      <li>Item three</li>\n",
    "    </ul>\n",
    "\n",
    "The `<ul>` tag (`ul` stands for \"unordered list\") in this example has three other `<li>` tags inside of it (`li` stands for \"list item\"). The `<ul>` tag is said to be the \"parent\" of the `<li>` tags, and the `<li>` tags are the \"children\" of the `<ul>` tag. All tags grouped under a particular parent tag are called \"siblings.\"\n",
    "\n",
    "### HTML's shortcomings\n",
    "\n",
    "HTML documents are intended to add \"markup\" to text to add information that allows browsers to display the text in different ways---e.g., HTML markup might tell the browser to make the font of the text a particular size, or to position it in a particular place on the screen.\n",
    "\n",
    "Because the primary purpose of HTML is to change the appearance of text, HTML markup usually does *not* tell us anything useful about what the text means, or what kind of data it contains. When you look at a web page in the browser, it might appear to contain a list of newspaper articles, or a table with birth rates, or a series of names with associated biographies, or whatever. But that's information that we get, as humans, from reading the page. There's (usually) no easy way to extract this information with a computer program.\n",
    "\n",
    "HTML is also notoriously messy---web browsers are very forgiving of syntax errors and other irregularities in HTML (like mismatched or unclosed tags). (This is in contrast to data formats like JSON, where even small errors will fail to be parsed.) For this reason, we need special libraries to parse HTML into data structures that our Python programs can use, libraries that can make a \"good guess\" about what the structure of an HTML document is, even when that structure is written incorrectly or inconsistently.\n",
    "\n",
    "[Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/) is a Python library that parses HTML (even if it's poorly formatted) and allows us to extract and manipulate its contents. We'll be using this library in the examples that follow.\n",
    "\n",
    "Keep in mind that there are only sketchy rules for what HTML elements \"mean\"---semantic information you figure out for one web page might not apply to the next. Values for `class` attributes especially are meaningful only in the context of a single page.\n",
    "\n",
    "> Note: There's an effort to add semantic information to HTML markup called [HTML Microformats](http://microformats.org/). If sites added microformats to their markup, you'd be able to write code that could more reliably extract information from web pages, because there would be a common language for what tags with particular classes and attributes mean. Alas, microformats remain unpopular, and until the anarcho-collectivists win a greater mindshare, we can count only on our own individual readings of individual HTML documents.\n",
    "\n",
    "### Inspecting HTML's anatomy with Developer Tools\n",
    "\n",
    "I've crafted a very simple example of HTML for us to work with. It concerns kittens. [Here's the rendered version](http://static.decontextualize.com/kittens.html), and [here's the HTML source code](https://raw.githubusercontent.com/ledeprogram/courses/master/databases/data/kittens.html).\n",
    "\n",
    "Now we're going to use Developer Tools in Chrome to take a look at how `kittens.html` is organized. Click on the \"rendered version\" link above. In Chrome, ctrl-click (or right click) anywhere on the page and select \"Inspect Element.\" This will open Chrome's Developer Tools. Your screen should look (something) like this:\n",
    "\n",
    "<a href=\"http://static.decontextualize.com/snaps/kittens-dev-tools.png\"><img src=\"http://static.decontextualize.com/snaps/kittens-dev-tools.png\" alt=\"kittens-dev-tools\"/></a>\n",
    "\n",
    "In the upper panel, you see the web page you're inspecting. In the lower panel, you see a version of the HTML source code, with little arrows next to some of the lines. (The little arrows allow you to collapse parts of the HTML source that are hierarchically related.) As you move your mouse over the elements in the top panel, different parts of the source code will be highlighted. Chrome is showing you which parts of the source code are causing which parts of the page to show up. Pretty spiffy!\n",
    "\n",
    "This relationship also works in reverse: you can move your mouse over some part of the source code in the lower panel, which will highlight in the top panel what that source code corresponds to on the page. We'll be using this later to visually identify the parts of the page that are interesting to us, so we can write code that extracts the contents of those parts automatically.\n",
    "\n",
    "### Characterizing the structure of kittens\n",
    "\n",
    "Here's what the source code of kittens.html looks like:\n",
    "\n",
    "\t<!doctype html>\n",
    "\t<html>\n",
    "\t  <head>\n",
    "\t    <title>Kittens!</title>\n",
    "\t  </head>\n",
    "\t  <body>\n",
    "\t    <h1>Kittens and the TV Shows They Love</h1>\n",
    "\t    <div class=\"kitten\">\n",
    "\t      <h2>Fluffy</h2>\n",
    "\t      <div><img src=\"http://placekitten.com/100/100\"></div>\n",
    "\t      <ul class=\"tvshows\">\n",
    "\t        <li><a href=\"http://www.imdb.com/title/tt0106145/\">Deep Space Nine</a></li>\n",
    "\t        <li><a href=\"http://www.imdb.com/title/tt0088576/\">Mr. Belvedere</a></li>\n",
    "\t      </ul>\n",
    "\t      Last check-up: <span class=\"lastcheckup\">2014-01-17</span>\n",
    "\t    </div>\n",
    "\t    <div class=\"kitten\">\n",
    "\t      <h2>Monsieur Whiskeurs</h2>\n",
    "\t      <div><img src=\"http://placekitten.com/150/100\"></div>\n",
    "\t      <ul class=\"tvshows\">\n",
    "\t        <li><a href=\"http://www.imdb.com/title/tt0106179/\">The X-Files</a></li>\n",
    "\t        <li><a href=\"http://www.imdb.com/title/tt0098800/\">Fresh Prince</a></li>\n",
    "\t      </ul>\n",
    "\t      Last check-up: <span class=\"lastcheckup\">2013-11-02</span>\n",
    "\t    </div>\n",
    "\t  </body>\n",
    "\t</html>\n",
    "\n",
    "This is pretty well organized HTML, but if you don't know how to read HTML, it will still look like a big jumble. Here's how I would characterize the structure of this HTML, reading in my own idea of what the meaning of the elements are.\n",
    "\n",
    "* We have two \"kittens,\" both of which are contained in `<div>` tags with class `kitten`.\n",
    "* Each \"kitten\" `<div>` has an `<h2>` tag with that kitten's name.\n",
    "* There's an image for each kitten, specified with an `<img>` tag.\n",
    "* Each kitten has a list (a `<ul>` with class `tvshows`) of television shows, contained within `<li>` tags.\n",
    "* Those list items themselves have links (`<a>` tags) with an `href` attribute that contains a link to an IMDB entry for that show.\n",
    "\n",
    "> BONUS QUIZ: What's the parent tag of `<a href=\"http://www.imdb.com/title/tt0088576/\">Mr. Belvedere</a>`? Both `<div class=\"kitten\">` tags share a parent tag---what is it? What attributes are present on both `<img>` tags?\n",
    "\n",
    "### Scraping kittens with Beautiful Soup\n",
    "\n",
    "We've examined `kittens.html` a bit now. What we'd like to do is write some code that is going to extract information from the HTML, like \"what is the last checkup date for each of these kittens?\" or \"what are Monsieur Whiskeur's favorite TV shows?\" To do so, we need to *parse* the HTML, and create a representation of it in our program that we can manipulate with Python.\n",
    "\n",
    "As mentioned above, HTML is hard to parse by hand. (Don't even try it. In particular, [don't parse HTML with regular expressions](http://stackoverflow.com/a/1732454).)\n",
    "\n",
    "Beautiful Soup is a Python library that will parse the HTML for us, and give us some Python objects that we can call methods on to poke at the data contained therein. So instead of working with strings and bytes, we can work with Python objects, methods and data structures.\n",
    "\n",
    "If you're using Anaconda, Beautiful Soup comes pre-installed, so there's no extra installation step! If you're not using Anaconda, you may need to install the library with `pip`:\n",
    "\n",
    "    pip3 install beautifulsoup4\n",
    "\n",
    "Beautiful Soup only parses HTML. It's left to us to actually *get* the HTML from somewhere. Because we're doing web scraping, most likely we'll want to download the HTML directly from the web site that hosts it. For that, we'll use the `urlopen` function in the built-in Python library `urllib.request`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "resp = requests.get(\"http://static.decontextualize.com/kittens.html\")\n",
    "html_str = resp.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `html_str` is a string that contains the HTML source code of the page in question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html>\\n<html>\\n\\t<head>\\n\\t\\t<title>Kittens!</title>\\n\\t\\t<style type=\"text/css\">\\n\\t\\t\\tspan.lastcheckup { font-family: \"Courier\", fixed; font-size: 11px; }\\n\\t\\t</style>\\n\\t</head>\\n\\t<body>\\n\\t\\t<h1>Kittens and the TV Shows They Love</h1>\\n\\t\\t<div class=\"kitten\">\\n\\t\\t\\t<h2>Fluffy</h2>\\n\\t\\t\\t<div><img src=\"http://placekitten.com/120/120\"></div>\\n\\t\\t\\t<ul class=\"tvshows\">\\n\\t\\t\\t\\t<li>\\n\\t\\t\\t\\t\\t<a href=\"http://www.imdb.com/title/tt0106145/\">Deep Space Nine</a>\\n\\t\\t\\t\\t</li>\\n\\t\\t\\t\\t<li>\\n\\t\\t\\t\\t\\t<a href=\"http://www.imdb.com/title/tt0088576/\">Mr. Belvedere</a>\\n\\t\\t\\t\\t</li>\\n\\t\\t\\t</ul>\\n\\t\\t\\tLast check-up: <span class=\"lastcheckup\">2014-01-17</span>\\n\\t\\t</div>\\n\\t\\t<div class=\"kitten\">\\n\\t\\t\\t<h2>Monsieur Whiskeurs</h2>\\n\\t\\t\\t<div><img src=\"http://placekitten.com/110/110\"></div>\\n\\t\\t\\t<ul class=\"tvshows\">\\n\\t\\t\\t\\t<li>\\n\\t\\t\\t\\t\\t<a href=\"http://www.imdb.com/title/tt0106179/\">The X-Files</a>\\n\\t\\t\\t\\t</li>\\n\\t\\t\\t\\t<li>\\n\\t\\t\\t\\t\\t<a href=\"http://www.imdb.com/title/tt0098800/\">Fresh Prince</a>\\n\\t\\t\\t\\t</li>\\n\\t\\t\\t</ul>\\n\\t\\t\\tLast check-up: <span class=\"lastcheckup\">2013-11-02</span>\\n\\t\\t</div>\\n\\t</body>\\n</html>\\n\\n'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks like a mess but it's apparent that we've obtained the data as desired.\n",
    "\n",
    "Now we need to create a Beautiful Soup object from that data. Here's how to go about doing that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.BeautifulSoup"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "document = BeautifulSoup(html_str, \"html.parser\")\n",
    "type(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BeautifulSoup` function creates a new Beautiful Soup object. It takes two parameters: the string containing the HTML data, and a string that designates which underlying parser to use to build the parsed version of the document. (Leave this as `\"html.parser\"`.) I've assigned this object to the variable `document`. This object supports a number of interesting methods that allow us to dig into the contents of the HTML. Primarily what we'll be working with are `Tag` objects and `ResultSet` objects, which are essentially just lists of `Tag` objects.\n",
    "\n",
    "### Finding a tag\n",
    "\n",
    "HTML documents are composed of tags. To represent this, Beautiful Soup has a type of value that represents tags. We can use the `.find()` method of the `BeautifulSoup` object to find a tag that matches a particular tag name. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.Tag"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h1_tag = document.find('h1')\n",
    "type(h1_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `Tag` object has several interesting attributes and methods. The `string` attribute of a `Tag` object, for example, returns a string representing that tag's contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Kittens and the TV Shows They Love'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h1_tag.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the attributes of a tag by treating the tag object as though it were a dictionary, using the square-bracket index syntax, with the name of the attribute whose value you want as a string inside the brackets. For example, to print out the `src` attribute of the first `<img>` tag in the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://placekitten.com/120/120'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img_tag = document.find('img')\n",
    "img_tag['src']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: You might have noticed that there is more than one `<img>` tag in `kittens.html`! If more than one tag matches the name you pass to `.find()`, it returns only the *first* matching tag. (A better name for `.find()` might be `find_first`.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding multiple tags\n",
    "\n",
    "It's often the case that we want to find not just one tag that matches particular criteria, but ALL tags matching those criteria. For that, we use the `.find_all()` method of the `BeautifulSoup` object. For example, to find all `h2` tags in the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.ResultSet"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2_tags = document.find_all('h2')\n",
    "type(h2_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Fluffy', 'Monsieur Whiskeurs']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[tag.string for tag in h2_tags]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the `.find()` and `.find_all()` methods can search not just for tags with particular names, but also for tags that have particular attributes. For that, we use the `attrs` keyword argument, giving it a dictionary that associates attribute names as keys and the desired attribute value as values. For example, to find all `span` tags with a `class` attribute of `lastcheckup`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2014-01-17', '2013-11-02']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "checkup_tags = document.find_all('span', attrs={'class': 'lastcheckup'})\n",
    "[tag.string for tag in checkup_tags]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: Beautiful Soup's `.find()` and `.find_all()` methods are actually more powerful than we're letting on here. [Check out the details in the official Beautiful Soup documentation.](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding tags within tags\n",
    "\n",
    "Let's say that we wanted to print out a list of the name of each kitten, along with a list of the names of that kitten's favorite TV shows. In other words, we want to print out something that looks like this:\n",
    "\n",
    "    Fluffy: Deep Space Nine, Mr. Belvedere\n",
    "    Monsieur Whiskeurs: The X-Files, Fresh Prince\n",
    "    \n",
    "In order to do this, we need to find *not just* tags with particular names, but tags with *particular hierarchical relationships* with other tags. I.e., we need to identify all of the kittens, and then find the shows that belong to that kitten. This kind of search is made easy by the fact that you can use `.find()` and `.find_all()` methods not just on the entire document, but on *individual tags*. When you use these methods on tags, they search for matching tags that are specifically *children of* the tag that you call them on.\n",
    "\n",
    "In our kittens example, we can see that information about individual kittens is grouped together under `<div>` tags with a `class` attribute of `kitten`. So, to find a list of all `<div>` tags with `class` set to `kitten`, we might do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "kitten_tags = document.find_all(\"div\", attrs={\"class\": \"kitten\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll loop over that list of tags and find, inside each of them, the `<h2>` tag that is its child:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fluffy\n",
      "Monsieur Whiskeurs\n"
     ]
    }
   ],
   "source": [
    "for kitten_tag in kitten_tags:\n",
    "    h2_tag = kitten_tag.find('h2')\n",
    "    print(h2_tag.string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll go one extra step. Looping over all of the kitten tags, we'll find not just the `<h2>` tag with the kitten's name, but all `<a>` tags (which contain the names of the TV shows that we were looking for):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fluffy: Deep Space Nine, Mr. Belvedere\n",
      "Monsieur Whiskeurs: The X-Files, Fresh Prince\n"
     ]
    }
   ],
   "source": [
    "for kitten_tag in kitten_tags:\n",
    "    h2_tag = kitten_tag.find('h2')\n",
    "    a_tags = kitten_tag.find_all('a')\n",
    "    a_tag_strings = [tag.string for tag in a_tags]\n",
    "    a_tag_strings_joined = \", \".join(a_tag_strings)\n",
    "    print(h2_tag.string + \": \" + a_tag_strings_joined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXERCISE: Modify the code above to print out a list of kitten names along with the last check-up date for that kitten."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> EXTRA FUN EXERCISE: Rewrite the code above to create a dictionary that maps kitten names to a list of links to that kitten's favorite shows. I.e., you should end up with a dictionary that looks like this:\n",
    "\n",
    "    {u'Fluffy': [u'http://www.imdb.com/title/tt0106145/',\n",
    "      u'http://www.imdb.com/title/tt0088576/'],\n",
    "     u'Monsieur Whiskeurs': [u'http://www.imdb.com/title/tt0106179/',\n",
    "      u'http://www.imdb.com/title/tt0098800/']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding sibling tags\n",
    "\n",
    "Often, the tags we're looking for don't have a distinguishing characteristic, like a `class` attribute, that allows us to find them using `.find()` and `.find_all()`, and the tags also aren't in a parent-child relationship. This can be tricky! Take the following HTML snippet, for example:    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cheese_html = \"\"\"\n",
    "<h2>Camembert</h2>\n",
    "<p>A soft cheese made in the Camembert region of France.</p>\n",
    "\n",
    "<h2>Cheddar</h2>\n",
    "<p>A yellow cheese made in the Cheddar region of... France, probably, idk whatevs.</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our task was to create a dictionary that maps the name of the cheese to the description that follows in the `<p>` tag directly afterward, we'd be out of luck. Fortunately, Beautiful Soup has a `.find_next_sibling()` method, which allows us to search for the next tag that is a *sibling* of the tag you're calling it on (i.e., the two tags share a parent), that also matches particular criteria. So, for example, to accomplish the task outlined above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Camembert': 'A soft cheese made in the Camembert region of France.',\n",
       " 'Cheddar': 'A yellow cheese made in the Cheddar region of... France, probably, idk whatevs.'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document = BeautifulSoup(cheese_html, \"html.parser\")\n",
    "cheese_dict = {}\n",
    "for h2_tag in document.find_all('h2'):\n",
    "    cheese_name = h2_tag.string\n",
    "    cheese_desc_tag = h2_tag.find_next_sibling('p')\n",
    "    cheese_dict[cheese_name] = cheese_desc_tag.string\n",
    "\n",
    "cheese_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now know most of what you need to know to scrape web pages effectively. Good job!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When things go wrong with Beautiful Soup\n",
    "\n",
    "A number of things might go wrong with Beautiful Soup. You might, for example, search for a tag that doesn't exist in the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "footer_tag = document.find(\"footer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup doesn't return an error if it can't find the tag you want. Instead, it returns `None`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(footer_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you try to call a method on the object that Beautiful Soup returned anyway, you might end up with an error like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'find'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-25-68ae6cd15b29>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfooter_tag\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"p\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'find'"
     ]
    }
   ],
   "source": [
    "footer_tag.find(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might also inadvertently try to get an attribute of a tag that wasn't actually found. You'll get a similar error in that case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not subscriptable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-26-914072fc1329>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfooter_tag\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'title'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not subscriptable"
     ]
    }
   ],
   "source": [
    "footer_tag['title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whenever you see something like `TypeError: 'NoneType' object is not subscriptable`, it's a good idea to check to see whether your method calls are indeed finding the thing you were looking for.\n",
    "\n",
    "However, the `.find_all()` method will return an empty list if it doesn't find any of the tags you wanted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "footer_tags = document.find_all(\"footer\")\n",
    "print(footer_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you attempt to access one of the elements of this regardless..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-35baeffa5004>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfooter_tags\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstring\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "print(footer_tags[0].string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...you'll get an `IndexError`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "* [Chapter 11](https://automatetheboringstuff.com/chapter11/) from Al Sweigart's [Automate the Boring Stuff with Python](https://automatetheboringstuff.com/) is another good take on this material (and discusses a wider range of techniques).\n",
    "* [The official Beautiful Soup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) provides a systematic walkthrough of the library's functionality. If you find yourself thinking, \"it really should be easy to do the thing that I want to do, why isn't it easier?\" then check the documentation! Leonard's probably already thought of a way to make it easier and implemented a feature in the code to help you out.\n",
    "* Beautiful Soup is the best scraping library out there for quick jobs, but if you have a larger site that you need to scrape, you might look into [Scrapy](http://scrapy.org/), which bundles a good parser with a framework for writing web \"spiders\" (i.e., programs that parse web pages and follow the links found there, in order to make a catalog of an entire web site, not just a single web page)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}