1_WEBSCRAPE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Gabriela Tanumihardja**</br>\n",
    "**Capstone Project - Part I** </br>\n",
    "**Data acquisition - webscraping**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "1. [Introduction](#intro)\n",
    "2. [Beaverton](#bvt)\n",
    "3. [The Globe and Mail](#gm)\n",
    "4. [The Onion](#onion)\n",
    "5. [The New York Times](#nyt)\n",
    "6. [The New York Time - rescrape](#nyt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.common.exceptions import NoSuchElementException\n",
    "import time\n",
    "import requests\n",
    "import shutil\n",
    "import urllib\n",
    "import re\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "<a id='intro'></a> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Satire is a literary device which draws the reader's attention to shortcomings and vices of people, organizations, or society. Though often humourus, the purpose of satire is to make the readers think and maybe induce a change in the system itself. There has been a lot of satirical novels that have been published throughout history; for example 1984 by George Orwell and A Clockwork Orange by Anthony Burgess. Satire is also commonly used in news production, especially on the web. Though satirical news are never meant to delude their readers, often times it is very difficult to distinguish them from legitimate news. Website like the Onion and the Beaverton are very similar in appearance to legitimate news sources. I found that if I didn't know that the Onion exclusively publish satirical news pieces, I would have a hard time distinguishing articles just based on their headlines. Based on this, I would like to build a model that could potentially distinguish whether or not a news headline is satirical or legitimate. It also would be very interesting to see if a model could predict the source of the news headline. </br>\n",
    "\n",
    "In starting this project, I would need to obtain some news headlines, satirical and legitimate, from news websites. I have chosen the Beaverton, the Onion, the Globe and Mail, and the New York Times. I opted for these sources as they are well established and their articles are of high quality. I chose to limit my news sources to be Canadian and American to eliminate any biases related to customs, tradition, or local events. I would like to make sure that the data I use is relatively even in terms to sources, especially in satire/legitimate class. I will now move on to scrape these sites, starting with **the Beaverton**. I will be using BeautifulSoup package to scrape website information and Selenium to automate my browser. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up browser in Chrome for Selenium:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save path to chromedriver executable file to variable\n",
    "\n",
    "chrome_path = '/Users/gabrielatanumihardja/opt/chromedriver'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the chrome webdriver to variable\n",
    "# Open browser\n",
    "\n",
    "browser = webdriver.Chrome(chrome_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beaverton 🐿️\n",
    "<a id='bvt'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the website, I will scrape the articles from `National`, `World`, `Sports`, `Business`, and `Culture` tabs. Each page has a next button that has a class name of `next`. I will use this to specify Selenium's click task. Each tab has different number of pages, so I will insert a `try` to click on the next button unless there is a `NoSuchELementException` thrown, in which case I will break out of the current loop and go to the next tab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the tabs\n",
    "\n",
    "topic_list_bvt = ['national', 'world', 'sports', 'business', 'culture']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create empty lists for title, topics, date, and source (will be beaverton)\n",
    "\n",
    "titles_list_bvt = []\n",
    "topics_bvt = []\n",
    "date_bvt = []\n",
    "source_bvt = []\n",
    "\n",
    "# Function that will go to the first website, scrape all the data, append all the articles, date, 'beaverton' as source, and topics to empty lists.\n",
    "\n",
    "def scrape_bvt(topic):\n",
    "    websource_bvt = browser.page_source\n",
    "    \n",
    "    # get websource\n",
    "    \n",
    "    soup = BeautifulSoup(websource_bvt)\n",
    "\n",
    "    articles = soup.find_all('h3', itemprop = 'headline')\n",
    "        \n",
    "    dates = soup.find_all('time', datetime = True)\n",
    "        \n",
    "    for article in articles:\n",
    "        titles_list_bvt.append(article.text)\n",
    "        topics_bvt.append(topic)\n",
    "        source_bvt.append('beaverton')\n",
    "            \n",
    "    for date in dates:\n",
    "        date_bvt.append(date['content'])\n",
    "\n",
    "# Loop through pages and tabs with try and except built in\n",
    "\n",
    "for topic in topic_list_bvt:\n",
    "    browser.get(f'https://www.thebeaverton.com/news/{topic}/')\n",
    "    \n",
    "    while True:\n",
    "        try:\n",
    "            scrape_bvt(topic)\n",
    "            browser.find_element_by_class_name('next').click()\n",
    "            time.sleep(5)\n",
    "        except NoSuchElementException:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything seems to work! I will put the scraped data into a df which I could export later as csv's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make df\n",
    "\n",
    "articles_df = pd.DataFrame(\n",
    "    {'title': titles_list_bvt,\n",
    "     'topic': topics_bvt,\n",
    "     'date_published': date_bvt,\n",
    "     'source': source_bvt\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>topic</th>\n",
       "      <th>date_published</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Guy who has definitely gotten into a fight at ...</td>\n",
       "      <td>national</td>\n",
       "      <td>2020-08-24T12:20:03-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Party that wants to manage government can’t ma...</td>\n",
       "      <td>national</td>\n",
       "      <td>2020-08-24T09:49:25-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Canada searches for new country to compare our...</td>\n",
       "      <td>national</td>\n",
       "      <td>2020-08-21T15:14:56-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Trudeau hopes giving Parliament five week vaca...</td>\n",
       "      <td>national</td>\n",
       "      <td>2020-08-18T19:08:42-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Highlights of Andrew Scheer’s tenure as Conser...</td>\n",
       "      <td>national</td>\n",
       "      <td>2020-08-17T11:34:45-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2717</th>\n",
       "      <td>Beatles fan unimpressed by rest of humanity</td>\n",
       "      <td>culture</td>\n",
       "      <td>2011-10-02T19:03:09-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2718</th>\n",
       "      <td>CBC to change fall programming</td>\n",
       "      <td>culture</td>\n",
       "      <td>2011-07-08T20:22:07-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2719</th>\n",
       "      <td>Local Yeti captures rare photograph of graffit...</td>\n",
       "      <td>culture</td>\n",
       "      <td>2011-05-16T06:05:16-04:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2720</th>\n",
       "      <td>Eminem makes another mean face for photo shoot</td>\n",
       "      <td>culture</td>\n",
       "      <td>2011-01-21T09:20:49-05:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2721</th>\n",
       "      <td>Brad Pitt looks a little like Gabriel Dumont</td>\n",
       "      <td>culture</td>\n",
       "      <td>2010-11-25T22:40:23-05:00</td>\n",
       "      <td>beaverton</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2722 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title     topic  \\\n",
       "0     Guy who has definitely gotten into a fight at ...  national   \n",
       "1     Party that wants to manage government can’t ma...  national   \n",
       "2     Canada searches for new country to compare our...  national   \n",
       "3     Trudeau hopes giving Parliament five week vaca...  national   \n",
       "4     Highlights of Andrew Scheer’s tenure as Conser...  national   \n",
       "...                                                 ...       ...   \n",
       "2717        Beatles fan unimpressed by rest of humanity   culture   \n",
       "2718                     CBC to change fall programming   culture   \n",
       "2719  Local Yeti captures rare photograph of graffit...   culture   \n",
       "2720     Eminem makes another mean face for photo shoot   culture   \n",
       "2721       Brad Pitt looks a little like Gabriel Dumont   culture   \n",
       "\n",
       "                 date_published     source  \n",
       "0     2020-08-24T12:20:03-04:00  beaverton  \n",
       "1     2020-08-24T09:49:25-04:00  beaverton  \n",
       "2     2020-08-21T15:14:56-04:00  beaverton  \n",
       "3     2020-08-18T19:08:42-04:00  beaverton  \n",
       "4     2020-08-17T11:34:45-04:00  beaverton  \n",
       "...                         ...        ...  \n",
       "2717  2011-10-02T19:03:09-04:00  beaverton  \n",
       "2718  2011-07-08T20:22:07-04:00  beaverton  \n",
       "2719  2011-05-16T06:05:16-04:00  beaverton  \n",
       "2720  2011-01-21T09:20:49-05:00  beaverton  \n",
       "2721  2010-11-25T22:40:23-05:00  beaverton  \n",
       "\n",
       "[2722 rows x 4 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check!\n",
    "\n",
    "articles_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export df to a csv\n",
    "\n",
    "articles_df.to_csv('data/beaverton.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check!\n",
    "\n",
    "bvt = pd.read_csv('data/beaverton.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2722"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double check!\n",
    "\n",
    "len(bvt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Globe and Mail 🌎\n",
    "<a id='gm'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the Globe and Mail I will scrape headlines from the `Canada`, `World`, `Sports`, `Arts`, and `Politics` sections of the websites."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify topic list and empty lists\n",
    "\n",
    "topic_list_gm = ['canada', 'world', 'sports', 'arts', 'politics']\n",
    "titles_list_gm = []\n",
    "topics_gm = []\n",
    "date_gm = []\n",
    "source_gm = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Globe and Mail website is constructed differently to the Beaverton's website. It seems that with each click of the `View More` button, the page lengthen and new stories are appended at the end of the list. For the sake of simplicity, I will scrape articles below the `Latest` tag. The dates are quite tricky to obtain as the tags vary between articles, so I will pass in both classes into my loop. Because the page lengthen for each click, I will click the button a number of times and scrape the whole page after. I will click the button 80 times, as each page generally puts up 20 new articles. With 80 clicks, 20 articles per click, and 5 topics, I will have more than 8000 articles. I think this will be sufficient for my project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Click 0\n",
      "Click 5\n",
      "Click 10\n",
      "Click 15\n",
      "Click 20\n",
      "Click 25\n",
      "Click 30\n",
      "Click 35\n",
      "Click 40\n",
      "Click 45\n",
      "Click 50\n",
      "Click 55\n",
      "Click 60\n",
      "Click 65\n",
      "Click 70\n",
      "Click 75\n",
      "Click 0\n",
      "Click 5\n",
      "Click 10\n",
      "Click 15\n",
      "Click 20\n",
      "Click 25\n",
      "Click 30\n",
      "Click 35\n",
      "Click 40\n",
      "Click 45\n",
      "Click 50\n",
      "Click 55\n",
      "Click 60\n",
      "Click 65\n",
      "Click 70\n",
      "Click 75\n",
      "Click 0\n",
      "Click 5\n",
      "Click 10\n",
      "Click 15\n",
      "Click 20\n",
      "Click 25\n",
      "Click 30\n",
      "Click 35\n",
      "Click 40\n",
      "Click 45\n",
      "Click 50\n",
      "Click 55\n",
      "Click 60\n",
      "Click 65\n",
      "Click 70\n",
      "Click 75\n",
      "Click 0\n",
      "Click 5\n",
      "Click 10\n",
      "Click 15\n",
      "Click 20\n",
      "Click 25\n",
      "Click 30\n",
      "Click 35\n",
      "Click 40\n",
      "Click 45\n",
      "Click 50\n",
      "Click 55\n",
      "Click 60\n",
      "Click 65\n",
      "Click 70\n",
      "Click 75\n",
      "Click 0\n",
      "Click 5\n",
      "Click 10\n",
      "Click 15\n",
      "Click 20\n",
      "Click 25\n",
      "Click 30\n",
      "Click 35\n",
      "Click 40\n",
      "Click 45\n",
      "Click 50\n",
      "Click 55\n",
      "Click 60\n",
      "Click 65\n",
      "Click 70\n",
      "Click 75\n"
     ]
    }
   ],
   "source": [
    "# Starting loop\n",
    "\n",
    "for topic in topic_list_gm:\n",
    "    \n",
    "    browser.get(f'https://www.theglobeandmail.com/{topic}/')\n",
    "    \n",
    "    button = browser.find_element_by_xpath(f'//button[normalize-space()= \"View More {topic.title()}\"]')\n",
    "    \n",
    "    # 80 clicks should be sufficient\n",
    "    for i in range(80):\n",
    "        browser.execute_script(\"arguments[0].click();\", button)\n",
    "        \n",
    "        # Print to make sure that it's still clickin' away\n",
    "        if i%5 == 0:\n",
    "            print(f'Click {i}')\n",
    "        \n",
    "        time.sleep(10)\n",
    "\n",
    "    websource_gm = browser.page_source\n",
    "\n",
    "    soup = BeautifulSoup(websource_gm)\n",
    "    \n",
    "    # Scrape articles from **below** the latest tag\n",
    "    latest = soup.find('div', class_ = 'u-wrapper pb-feature pb-layout-item pb-f-global-story-feed')\n",
    "\n",
    "    articles_gm = latest.findChildren('div', class_ = 'c-card__hed-text')\n",
    "\n",
    "    dates_gm = latest.findChildren('time', class_ = ['c-timestamp u-no-wrap js-story-moment',\n",
    "                                                     'c-timestamp u-no-wrap'])\n",
    "    \n",
    "    # Append the articles, dates, source, and topics to empty lists\n",
    "    for article in articles_gm:\n",
    "            titles_list_gm.append(article.text)\n",
    "            topics_gm.append(topic)\n",
    "            source_gm.append('the globe and mail')\n",
    "            \n",
    "    for date in dates_gm:\n",
    "                date_gm.append(date['datetime'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All seem to works as expected, I will now convert the data into a df and save the csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make df\n",
    "\n",
    "articles_df_gm = pd.DataFrame(\n",
    "    {'title': titles_list_gm,\n",
    "     'topic': topics_gm,\n",
    "     'date_published': date_gm,\n",
    "     'source': source_gm\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>topic</th>\n",
       "      <th>date_published</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Artificial intelligence needed to fight child...</td>\n",
       "      <td>canada</td>\n",
       "      <td>2020-08-24T22:28:28.391Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Evening Update: First documented coronavirus ...</td>\n",
       "      <td>canada</td>\n",
       "      <td>2020-08-24T20:54:37.240Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>New study calls for fresh approach to tacklin...</td>\n",
       "      <td>canada</td>\n",
       "      <td>2020-08-24T20:47:44.753Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>‘Nerve-racking’: Staff talk about stress of f...</td>\n",
       "      <td>canada</td>\n",
       "      <td>2020-08-24T20:41:33.986Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alberta Health Minister still talking with do...</td>\n",
       "      <td>canada</td>\n",
       "      <td>2020-08-24T19:52:20.261Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8175</th>\n",
       "      <td>Trudeau invites Scheer, Blanchet, Singh and M...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2019-11-04T00:12:04.506Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8176</th>\n",
       "      <td>Don’t blame Ottawa for Encana’s loss</td>\n",
       "      <td>politics</td>\n",
       "      <td>2019-11-03T23:55:14.036Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8177</th>\n",
       "      <td>Will backbench MPs seize the power of a minor...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2019-11-03T23:51:38.475Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8178</th>\n",
       "      <td>Refugee advocates set to challenge Canada’s b...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2019-11-03T23:43:31.132Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8179</th>\n",
       "      <td>‘It’s time to get back that hungry, competiti...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2019-11-03T23:34:09.022Z</td>\n",
       "      <td>the globe and mail</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8180 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  title     topic  \\\n",
       "0      Artificial intelligence needed to fight child...    canada   \n",
       "1      Evening Update: First documented coronavirus ...    canada   \n",
       "2      New study calls for fresh approach to tacklin...    canada   \n",
       "3      ‘Nerve-racking’: Staff talk about stress of f...    canada   \n",
       "4      Alberta Health Minister still talking with do...    canada   \n",
       "...                                                 ...       ...   \n",
       "8175   Trudeau invites Scheer, Blanchet, Singh and M...  politics   \n",
       "8176               Don’t blame Ottawa for Encana’s loss  politics   \n",
       "8177   Will backbench MPs seize the power of a minor...  politics   \n",
       "8178   Refugee advocates set to challenge Canada’s b...  politics   \n",
       "8179   ‘It’s time to get back that hungry, competiti...  politics   \n",
       "\n",
       "                date_published              source  \n",
       "0     2020-08-24T22:28:28.391Z  the globe and mail  \n",
       "1     2020-08-24T20:54:37.240Z  the globe and mail  \n",
       "2     2020-08-24T20:47:44.753Z  the globe and mail  \n",
       "3     2020-08-24T20:41:33.986Z  the globe and mail  \n",
       "4     2020-08-24T19:52:20.261Z  the globe and mail  \n",
       "...                        ...                 ...  \n",
       "8175  2019-11-04T00:12:04.506Z  the globe and mail  \n",
       "8176  2019-11-03T23:55:14.036Z  the globe and mail  \n",
       "8177  2019-11-03T23:51:38.475Z  the globe and mail  \n",
       "8178  2019-11-03T23:43:31.132Z  the globe and mail  \n",
       "8179  2019-11-03T23:34:09.022Z  the globe and mail  \n",
       "\n",
       "[8180 rows x 4 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check!\n",
    "\n",
    "articles_df_gm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to csv\n",
    "\n",
    "articles_df_gm.to_csv('data/globeandmail.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Onion 🧅\n",
    "<a id='onion'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the Onion, I will scrape headlines from `Politics`, `Sports`, `Local`, and `Entertainment` sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify a list of topic\n",
    "topic_list_o = ['politics', 'sports', 'local', 'entertainment']\n",
    "\n",
    "titles_list_o = []\n",
    "topics_o = []\n",
    "date_o = []\n",
    "source_o = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will extract headline information from the panel section of each page. I will then click on the `More Stories` button. Similar to the Beaverton code, I will build in the try and except code. The timestamp for the articles are quite peculiar as there are 2 elements under the same class. Each iteration extracts 2 duplicate dates. I could not find the difference between the two class_ specification, so I will create a new list that just drops duplicated date as it is very unlikely that the Onion would publish 2 articles at the exact same time in the same topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function that will go to the first website, scrape all the data, append all the articles, date, 'onion' as source, and topics to empty lists.\n",
    "\n",
    "def scrape_onion(topic):\n",
    "    \n",
    "        websource_o = browser.page_source\n",
    "\n",
    "        soup = BeautifulSoup(websource_o)\n",
    "\n",
    "        articles = soup.find_all('h2', class_ = ['sc-759qgu-0 cYlVdn cw4lnv-6 eXwNRE',\n",
    "                                                 'sc-759qgu-0 sc-759qgu-1 jmAmn sc-3kpz0l-8 bbgWOc'])\n",
    "\n",
    "        dates = soup.find_all('time', class_ = 'uhd9ir-0 gWMcOL cjg713-0 jElrJy')\n",
    "        \n",
    "        # Drop duplicated timestamptb\n",
    "        clean_dates = list(pd.Series(dates).drop_duplicates())\n",
    "\n",
    "        for article in articles:\n",
    "            titles_list_o.append(article.text)\n",
    "            topics_o.append(topic)\n",
    "            source_o.append('the onion')\n",
    "\n",
    "\n",
    "        for date in clean_dates:\n",
    "            date_o.append(date['datetime'])\n",
    "\n",
    "    \n",
    "for topic in topic_list_o:\n",
    "    browser.get(f'https://{topic}.theonion.com/')\n",
    "    \n",
    "    while True:\n",
    "        try:\n",
    "            scrape_onion(topic)\n",
    "            browser.find_element_by_xpath('/html/body/div[3]/div[4]/main/div/div[5]/div/a/button').click()\n",
    "            time.sleep(5)\n",
    "        except NoSuchElementException:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that it was working just fine. I will create a df from the lists, and hope that the length of dates match up with the rest of the lists 🤞. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the df\n",
    "\n",
    "articles_df_o = pd.DataFrame(\n",
    "    {'title': titles_list_o,\n",
    "     'topic': topics_o,\n",
    "     'date_published': date_o,\n",
    "     'source': source_o\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked 🙌! I assume the dates for each of these articles are correct, since they line up over topics as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>topic</th>\n",
       "      <th>date_published</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Highlights Of The 2020 Democratic National Con...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2020-08-21T13:35:00-05:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Congressional Republicans Grill Postmaster Gen...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2020-08-21T13:00:00-05:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Bloomberg Looks Straight Into Camera, Silently...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2020-08-20T21:20:00-05:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>‘Milwaukee Is A Great City On A Great Lake,’ S...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2020-08-20T11:20:00-05:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DNC Speakers Can’t Believe They’re Giving Prim...</td>\n",
       "      <td>politics</td>\n",
       "      <td>2020-08-20T11:00:00-05:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20195</th>\n",
       "      <td>Hugh Hefner Comes Out of Retirement, Changes P...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>1996-02-20T18:18:00-06:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20196</th>\n",
       "      <td>Nine Drawn and Quartered at Renaissance Fair</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>1995-12-18T18:38:00-06:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20197</th>\n",
       "      <td>Congress Hires Drummer</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>1995-12-11T18:55:00-06:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20198</th>\n",
       "      <td>Sonic Booms</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>1993-12-06T18:09:00-06:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20199</th>\n",
       "      <td>Your Real Horoscope</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>1993-12-06T18:01:00-06:00</td>\n",
       "      <td>the onion</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>20200 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title          topic  \\\n",
       "0      Highlights Of The 2020 Democratic National Con...       politics   \n",
       "1      Congressional Republicans Grill Postmaster Gen...       politics   \n",
       "2      Bloomberg Looks Straight Into Camera, Silently...       politics   \n",
       "3      ‘Milwaukee Is A Great City On A Great Lake,’ S...       politics   \n",
       "4      DNC Speakers Can’t Believe They’re Giving Prim...       politics   \n",
       "...                                                  ...            ...   \n",
       "20195  Hugh Hefner Comes Out of Retirement, Changes P...  entertainment   \n",
       "20196       Nine Drawn and Quartered at Renaissance Fair  entertainment   \n",
       "20197                             Congress Hires Drummer  entertainment   \n",
       "20198                                        Sonic Booms  entertainment   \n",
       "20199                                Your Real Horoscope  entertainment   \n",
       "\n",
       "                  date_published     source  \n",
       "0      2020-08-21T13:35:00-05:00  the onion  \n",
       "1      2020-08-21T13:00:00-05:00  the onion  \n",
       "2      2020-08-20T21:20:00-05:00  the onion  \n",
       "3      2020-08-20T11:20:00-05:00  the onion  \n",
       "4      2020-08-20T11:00:00-05:00  the onion  \n",
       "...                          ...        ...  \n",
       "20195  1996-02-20T18:18:00-06:00  the onion  \n",
       "20196  1995-12-18T18:38:00-06:00  the onion  \n",
       "20197  1995-12-11T18:55:00-06:00  the onion  \n",
       "20198  1993-12-06T18:09:00-06:00  the onion  \n",
       "20199  1993-12-06T18:01:00-06:00  the onion  \n",
       "\n",
       "[20200 rows x 4 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check!\n",
    "\n",
    "articles_df_o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save to csv\n",
    "\n",
    "articles_df_o.to_csv('data/theonion.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The New York Times 🗽\n",
    "<a id='nyt'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The New York Times has an API from which I could scrape all of the articles published in each specified month and year. For the sake of simplicity, I will scrape the latest 4 months of NYT articles... Each request returns a gigantic amount of data, so 4 API requests return more data than I have from the other sources. This may create an imbalance in my dataset in terms of time, however it may be good enough get going for now. I will return and rescrape once I re-evaluate. The API request returns a json, from which I could extract my preferred information by calling the keys in each dictionary entries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5/2020 compeleted\n",
      "6/2020 compeleted\n",
      "7/2020 compeleted\n",
      "8/2020 compeleted\n"
     ]
    }
   ],
   "source": [
    "# Specify months to scrape\n",
    "months = [5, 6, 7, 8]\n",
    "topic = []\n",
    "headlines = []\n",
    "date = []\n",
    "\n",
    "# Looping over months and getting info from each article through dictionary's keys\n",
    "for mon in months:\n",
    "    \n",
    "    req = requests.get(f'https://api.nytimes.com/svc/archive/v1/2020/{mon}.json?api-key=q8mol0Visv97gwH0dPI7BDMW5SYTegT0')\n",
    "    \n",
    "    full_month = req.json()\n",
    "    \n",
    "    # Get to the meat of the json\n",
    "    \n",
    "    docs = full_month['response']['docs']\n",
    "    \n",
    "    for article in docs:\n",
    "        headlines.append(article['headline']['main'])\n",
    "    \n",
    "    for article in docs:\n",
    "        topic.append(article['news_desk'])\n",
    "        \n",
    "    for article in docs:\n",
    "        date.append(article['pub_date'])\n",
    "        \n",
    "    print(f'{mon}/2020 compeleted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything seemed to work fine. I will now create a df from the lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make df\n",
    "\n",
    "articles_df_nyt = pd.DataFrame(\n",
    "    {'title': headlines,\n",
    "     'topic': topic,\n",
    "     'date_published': date,\n",
    "     'source': 'nyt'\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>topic</th>\n",
       "      <th>date_published</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Seven States to Coordinate on Amassing Medical...</td>\n",
       "      <td>Metro</td>\n",
       "      <td>2020-05-03T13:10:38+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Courage to Be Alone</td>\n",
       "      <td>OpEd</td>\n",
       "      <td>2020-05-01T19:30:09+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tennis Coming Back Slowly With Exhibition Matches</td>\n",
       "      <td>Sports</td>\n",
       "      <td>2020-05-01T05:00:06+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Best Part of the Campaign Trail (the Food!...</td>\n",
       "      <td>Dining</td>\n",
       "      <td>2020-05-01T18:24:58+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Go Ahead, Blow Out the Candles on Zoom</td>\n",
       "      <td>AtHome</td>\n",
       "      <td>2020-05-02T16:00:07+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24818</th>\n",
       "      <td>To Test Spread of Coronavirus, These Scientist...</td>\n",
       "      <td>Culture</td>\n",
       "      <td>2020-08-23T18:45:59+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24819</th>\n",
       "      <td>Map: Tracking Tropical Storms Laura and Marco</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>2020-08-23T03:29:21+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24820</th>\n",
       "      <td>How is the Coronavirus Affecting Low-Income Fa...</td>\n",
       "      <td>Reader Center</td>\n",
       "      <td>2020-08-23T05:25:15+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24821</th>\n",
       "      <td>A Film Pantheon That Omits Black Directors</td>\n",
       "      <td>Movies</td>\n",
       "      <td>2020-08-23T14:55:57+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24822</th>\n",
       "      <td>How Decades of Racist Housing Policy Left Neig...</td>\n",
       "      <td>Climate</td>\n",
       "      <td>2020-08-24T04:26:00+0000</td>\n",
       "      <td>nyt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>24823 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title          topic  \\\n",
       "0      Seven States to Coordinate on Amassing Medical...          Metro   \n",
       "1                                The Courage to Be Alone           OpEd   \n",
       "2      Tennis Coming Back Slowly With Exhibition Matches         Sports   \n",
       "3      The Best Part of the Campaign Trail (the Food!...         Dining   \n",
       "4                 Go Ahead, Blow Out the Candles on Zoom         AtHome   \n",
       "...                                                  ...            ...   \n",
       "24818  To Test Spread of Coronavirus, These Scientist...        Culture   \n",
       "24819      Map: Tracking Tropical Storms Laura and Marco           U.S.   \n",
       "24820  How is the Coronavirus Affecting Low-Income Fa...  Reader Center   \n",
       "24821         A Film Pantheon That Omits Black Directors         Movies   \n",
       "24822  How Decades of Racist Housing Policy Left Neig...        Climate   \n",
       "\n",
       "                 date_published source  \n",
       "0      2020-05-03T13:10:38+0000    nyt  \n",
       "1      2020-05-01T19:30:09+0000    nyt  \n",
       "2      2020-05-01T05:00:06+0000    nyt  \n",
       "3      2020-05-01T18:24:58+0000    nyt  \n",
       "4      2020-05-02T16:00:07+0000    nyt  \n",
       "...                         ...    ...  \n",
       "24818  2020-08-23T18:45:59+0000    nyt  \n",
       "24819  2020-08-23T03:29:21+0000    nyt  \n",
       "24820  2020-08-23T05:25:15+0000    nyt  \n",
       "24821  2020-08-23T14:55:57+0000    nyt  \n",
       "24822  2020-08-24T04:26:00+0000    nyt  \n",
       "\n",
       "[24823 rows x 4 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check!!\n",
    "\n",
    "articles_df_nyt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles_df_nyt.to_csv('data/nyt.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__In building this notebook, I attempted my code a little at the time and I scaled up the code once I am sure that it would work. It took many trial and errors to make a code that would move smoothly over the whole thing and in finding the right classes and tags.__</br>\n",
    "\n",
    "Now I will go on to clean the data... pt. 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The New York Times pt 2\n",
    "<a id='nyt2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From evaluating the initial models, I found out that the dates from the NYT is actually causing significant bias and skew my model significantly. From earlier EDA, I know that the overall published date ranges from 1993 to 2020. I will scrape all NYT headlines I could from 1993 to 2020. Once I am done with this, I will randomly downsample the v. large amount of data to equal the first iteration (20,000 articles). In preliminary trials, the API throws a `KeyError` randomly. Since we have a lot of data, I will just pass a `continue` through this error and move on. 2020 has only gone on for 8 months, so I will separately scrape 2020 data separately. The rest of the code is very similar to the previous code, with the addition of try, except, and finally!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the months and empty lists\n",
    "\n",
    "months = np.arange(1,13)\n",
    "months_2020 = np.arange(1,9)\n",
    "years = np.arrange[1993, 2021]\n",
    "topic = []\n",
    "headlines = []\n",
    "date = []\n",
    "\n",
    "# Loops - through years\n",
    "for year in years:\n",
    "    if year != 2020:\n",
    "        # - through months\n",
    "        for mon in months:\n",
    "\n",
    "            req = requests.get(f'https://api.nytimes.com/svc/archive/v1/{year}/{mon}.json?api-key=q8mol0Visv97gwH0dPI7BDMW5SYTegT0')\n",
    "\n",
    "            full_month = req.json()\n",
    "\n",
    "            try:\n",
    "                docs = full_month['response']['docs']\n",
    "\n",
    "                for article in docs:\n",
    "                    topic.append(article['section_name'])\n",
    "                    headlines.append(article['headline']['main'])\n",
    "                    date.append(article['pub_date'])\n",
    "                    \n",
    "                time.sleep(5)\n",
    "                                \n",
    "             # where am I getting a key error?       \n",
    "            except KeyError:\n",
    "                print(f'key error {mon}/{year}')\n",
    "                continue\n",
    "                \n",
    "            # Show me where I'm at!    \n",
    "            finally:\n",
    "                print(f'{mon}/{year} compeleted')\n",
    "                \n",
    "    else:\n",
    "        # year == 2020\n",
    "        for mon in months_2020:\n",
    "\n",
    "            req = requests.get(f'https://api.nytimes.com/svc/archive/v1/{year}/{mon}.json?api-key=q8mol0Visv97gwH0dPI7BDMW5SYTegT0')\n",
    "\n",
    "            full_month = req.json()\n",
    "\n",
    "            try:\n",
    "                docs = full_month['response']['docs']\n",
    "\n",
    "                for article in docs:\n",
    "                    topic.append(article['section_name'])\n",
    "                    headlines.append(article['headline']['main'])\n",
    "                    date.append(article['pub_date'])\n",
    "                    \n",
    "                time.sleep(5)\n",
    "                                \n",
    "            except KeyError:\n",
    "                print(f'key error {mon}/{year}')\n",
    "                continue\n",
    "                \n",
    "            finally:\n",
    "                print(f'{mon}/{year} compeleted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, everything seems to be in order! Now putting together the dataframe and sampling time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55719"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check!\n",
    "\n",
    "len(date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whew 55K articles, definitely need to resample! After creating df!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create df\n",
    "\n",
    "articles_df_nyt = pd.DataFrame(\n",
    "    {'title': headlines,\n",
    "     'topic': topic,\n",
    "     'date_published': date,\n",
    "     'source': 'nyt'\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the whole data to csv\n",
    "\n",
    "articles_df_nyt.to_csv('data/nyt1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I will simply use the sample function to pare down the massive amount of data to 20,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sampling\n",
    "\n",
    "sampled_nyt = articles_df_nyt.sample(20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save sampled data to csv\n",
    "\n",
    "sampled_nyt.to_csv('data/sampled_nyt.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on to cleaning and adjusting the data... Then to retrain the models... (continue to pt. 4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "webscraping",
   "language": "python",
   "name": "webscraping"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}