2B. Web Scraping .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "trY1sIDAuWve"
   },
   "source": [
    "# 11.3 Web Scraping\n",
    "\n",
    "**HTML**, which stands for \"hypertext markup language\", is an XML-like language for specifying the appearance of web pages. Each tag in HTML corresponds to a specific page element. For example:\n",
    "\n",
    "- `<img>` specifies an image. The path to the image file is specified in the `src=` attribute.\n",
    "- `<a>` specifies a hyperlink. The text enclosed between `<a>` and `</a>` is the text of the link that appears, while the URL is specified in the `href=` attribute of the tag.\n",
    "- `<table>` specifies a table. The rows of the table are specified by `<tr>` tags nested inside the `<table>` tag, while the cells in each row are specified by `<td>` tags nested inside each `<tr>` tag.\n",
    "\n",
    "Our goal in this section is not to teach you HTML to make a web page. You will learn just enough HTML to be able to scrape data programmatically from a web page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_cq2Gt5ZuWvh"
   },
   "source": [
    "# Inspecting HTML Source Code\n",
    "\n",
    "Suppose we want to scrape faculty information from the [100 Best Films Website](https://www.hollywoodreporter.com/lists/100-best-films-ever-hollywood-favorites-818512) (`https://www.hollywoodreporter.com/lists/100-best-films-ever-hollywood-favorites-818512`). Once we have identified a web page that we want to scrape, the next step is to study the HTML source code. All web browsers have a \"View Source\" or \"Page Source\" feature that will display the HTML source of a web page. \n",
    "\n",
    "Visit the web page above, and view the HTML source of that page. (You may have to search online to figure out how to view the page source in your favorite browser.) Scroll down until you find the HTML code for the table containing information about the name, office, phone, e-mail, and office hours of the faculty members.\n",
    "\n",
    "Notice how difficult it can be to find a page element in the HTML source. Many browsers allow you to right-click on a page element and jump to the part of the HTML source corresponding to that element."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ppv1jhtTuWvi"
   },
   "source": [
    "# Web Scraping Using BeautifulSoup\n",
    "\n",
    "`BeautifulSoup` is a Python library that makes it easy to navigate an HTML document. Like with `lxml`, we can query tags by name or attribute, and we can narrow our search to the ancestors and descendants of specific tags. In fact, it is possible to use `lxml` with HTML documents, but many web sites have malformed HTML, and `lxml` is not very forgiving. `BeautifulSoup` handles malformed HTML more gracefully and is thus the library of choice.\n",
    "\n",
    "First, we issue an HTTP request to the URL to get the HTML source code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hMmOJP-ZuWvj"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "response = requests.get(\"https://www.hollywoodreporter.com/lists/100-best-films-ever-hollywood-favorites-818512\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lcuyq9gpuWvo"
   },
   "source": [
    "The HTML source is stored in the `.content` attribute of the response object. We pass this HTML source into `BeautifulSoup` to obtain a tree-like representation of the HTML document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9qEyIz__uWvp"
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(response.content, \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M5XBlMhquWvt"
   },
   "source": [
    "Now we can search for tags within this HTML document, using tags like `.find_all()`. For example, we can find all headings on this page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8RpjIxeGuWvu"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "102"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "headings = soup.find_all(\"header\")\n",
    "len(headings)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G_q2JhIOuWv3"
   },
   "source": [
    "There is one movie per heading except the first heading item can be ignored. We iterate over all rows, extracting information about each movie to append to `rows`, which we will eventually turn into a `DataFrame`. As you read the code below, refer to the HTML source above, so that you understand what each line is doing. In particular, ask yourself the following questions:\n",
    "\n",
    "You are encouraged to add `print()` statements inside the `for` loop to check your understanding of each line of code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kAVQaQFWuWvx"
   },
   "source": [
    "As a visual inspection of [the web page](https://statistics.calpoly.edu/content/StatisticsDirectory%26Office%20Hours) would confirm, there are 2 tables on the page, and we are interested in the second one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XbRSPnisuWv4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'title': \"Hollywood's 100 Favorite Films\", 'year': None}, {'title': 'The Seven Samurai', 'year': '(1954)'}, {'title': 'Bonnie and Clyde', 'year': '(1967)'}, {'title': 'Reservoir Dogs', 'year': '(1992)'}, {'title': 'Airplane!', 'year': '(1980)'}]\n"
     ]
    }
   ],
   "source": [
    "rows = []\n",
    "for movie in headings[1:]:\n",
    "    # Get all the cells in the row.\n",
    "    movieTitle = movie.find(\"h1\").text\n",
    "    try :\n",
    "        movieYear = movie.find(\"h2\").text\n",
    "    except:\n",
    "        movieYear = None\n",
    "    # Append this data.\n",
    "    rows.append({\n",
    "        \"title\": movieTitle,\n",
    "        \"year\": movieYear\n",
    "    })\n",
    "    \n",
    "print (rows[:5]) # print first 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rYIyExFTuWv8"
   },
   "source": [
    "In the code above, observe that `.find_all()` returns a list with all matching tags, while `.find()` returns only the first matching tag. If no matching tags are found, then `.find_all()` will return an empty list `[]`, while `.find()` will return `None`.\n",
    "\n",
    "Finally, we turn `rows` into a `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9daEVF15uWv9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hollywood's 100 Favorite Films</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Seven Samurai</td>\n",
       "      <td>(1954)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bonnie and Clyde</td>\n",
       "      <td>(1967)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Reservoir Dogs</td>\n",
       "      <td>(1992)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Airplane!</td>\n",
       "      <td>(1980)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>(1994)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>(1994)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>Citizen Kane</td>\n",
       "      <td>(1941)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>The Wizard of Oz</td>\n",
       "      <td>(1939)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>(1972)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>101 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                              title    year\n",
       "0    Hollywood's 100 Favorite Films    None\n",
       "1                 The Seven Samurai  (1954)\n",
       "2                  Bonnie and Clyde  (1967)\n",
       "3                    Reservoir Dogs  (1992)\n",
       "4                         Airplane!  (1980)\n",
       "..                              ...     ...\n",
       "96                     Pulp Fiction  (1994)\n",
       "97         The Shawshank Redemption  (1994)\n",
       "98                     Citizen Kane  (1941)\n",
       "99                 The Wizard of Oz  (1939)\n",
       "100                   The Godfather  (1972)\n",
       "\n",
       "[101 rows x 2 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame(rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1_Etb0wBuWwB"
   },
   "source": [
    "Now this data is ready for further processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yN7cN1IluWwC"
   },
   "source": [
    "# Ethical Enlightenment: `robots.txt`\n",
    "\n",
    "Web robots are crawling the web all the time. A website may want to restrict the robots from crawling specific pages. One reason is financial: each visit to a web page, by a human or a robot, costs the website money, and the website may prefer to save their limited bandwidth for human visitors. Another reason is privacy: a website may not want a search engine to preserve a snapshot of a page for all eternity.\n",
    "\n",
    "To specify what a web robot is and isn't allowed to crawl, most websites will place a text file named `robots.txt` in the top-level directory of the web server. For example, the Statistics department web page has a `robots.txt` file: https://statistics.calpoly.edu/robots.txt\n",
    "\n",
    "The format of the `robots.txt` file should be self-explanatory, but you can read a full specification of the standard here: http://www.robotstxt.org/robotstxt.html. As you scrape websites using your web robot, always check the `robots.txt` file first, to make sure that you are respecting the wishes of the website owner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LKAvgpN6uWwD"
   },
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KK6rJQbuuWwF"
   },
   "source": [
    "Find an easy website to scrape. What do I mean by easy?? Well, the CSE UCR Faculty website is developed in such a way to deter scrappers. Click on the link and checkout the source code. Can you understand it?? https://www1.cs.ucr.edu/people/faculty/\n",
    "\n",
    "Find another website and extract some elements from that website. Try the following websites as a suggestion:\n",
    "1. https://www.imdb.com/list/ls050274118/\n",
    "2. https://www.imdb.com/list/ls000055475/\n",
    "3. https://www.forbes.com/best-states-for-business/list/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ThisAfternoon\n",
      "Sunny\n",
      "High: 91 °F\n",
      "           period    short_descriptions temperatures\n",
      "0   ThisAfternoon                 Sunny  High: 91 °F\n",
      "1         Tonight          Mostly Clear   Low: 64 °F\n",
      "2         Tuesday  Patchy Fogthen Sunny  High: 88 °F\n",
      "3    TuesdayNight            Patchy Fog   Low: 62 °F\n",
      "4       Wednesday  Patchy Fogthen Sunny  High: 83 °F\n",
      "5  WednesdayNight          Mostly Clear   Low: 62 °F\n",
      "6        Thursday                 Sunny  High: 84 °F\n",
      "7   ThursdayNight          Mostly Clear   Low: 63 °F\n",
      "8          Friday                 Sunny  High: 90 °F\n"
     ]
    }
   ],
   "source": [
    "#Write code here\n",
    "import bs4\n",
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "page = requests.get('https://forecast.weather.gov/MapClick.php?lat=33.853966413390374&lon=-117.79324044763452#.XvvQOJNKh0t')\n",
    "page.raise_for_status()\n",
    "\n",
    "soup = bs4.BeautifulSoup(page.content, 'html.parser')\n",
    "week = soup.find(id='seven-day-forecast')\n",
    "\n",
    "items = week.find_all(class_='tombstone-container')\n",
    "\n",
    "print(items[0].find(class_='period-name').get_text())\n",
    "print(items[0].find(class_='short-desc').get_text())\n",
    "print(items[0].find(class_='temp').get_text())\n",
    "\n",
    "period_names = [item.find(class_='period-name').get_text() for item in items]\n",
    "short_descriptions = [item.find(class_='short-desc').get_text() for item in items]\n",
    "temperatures = [item.find(class_='temp').get_text() for item in items]\n",
    "\n",
    "weather_stuff = pd.DataFrame(\n",
    "    {\n",
    "        'period' : period_names,\n",
    "        'short_descriptions' : short_descriptions,\n",
    "        'temperatures' : temperatures,\n",
    "    })\n",
    "print(weather_stuff)\n",
    "weather_stuff.to_csv('weather.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the web scraping of this weather forecast website gives us a short description, the high and low temperature of the next 5 days."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example of web scraping is below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4\n",
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "response = requests.get('https://www.imdb.com/list/ls000055475/')\n",
    "response.raise_for_status()\n",
    "\n",
    "soup = bs4.BeautifulSoup(response.content, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Meryl Streep\n",
      " \n",
      " The Devil Wears Prada\n",
      " \n",
      " Ingrid Bergman\n",
      " \n",
      " Casablanca\n",
      " \n",
      " Vivien Leigh\n",
      " \n",
      " A Streetcar Named Desire\n",
      " \n"
     ]
    }
   ],
   "source": [
    "items = soup.findAll(\"div\", {\"class\": \"lister-item-content\"})\n",
    "\n",
    "for actress in items[0:3]:\n",
    "    name = actress.find('a').contents[0]\n",
    "    movie = actress.find(\"p\",\"text-muted text-small\").text.split(\"|\")[1]\n",
    "    \n",
    "    print(name, movie)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "11.3 Web Scraping.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}