notebooks/01_data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Collection\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of contents\n",
    "\n",
    "1. [Project Introduction](#intro)\n",
    "    * 1.1 [Workflow sample](#workflow_sample)\n",
    "2. [Using Pushshift's API to pull data from subreddits](#API)\n",
    "3. [Pulling data from r/DMAcademy and r/truezelda](#pull-data-1)\n",
    "4. [Pulling data from r/PoliticalDiscussion](#pull-data-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='intro'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Project Introduction\n",
    "\n",
    "This project is all about Natural Language Processing (NLP) and binary classification models. The goals are to use [Pushshift's](https://github.com/pushshift/api) API to collect posts from two subreddits, then to create and compare two different models that can predict which subreddit a given post came from. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subreddit selection\n",
    "\n",
    "This project requires text-rich subreddit posts in order for the final models to be well-informed. Initially, I considered using [r/oceanography](https://www.reddit.com/r/oceanography/), but found too many posts containing only images or videos. I then considered [r/DnD](https://www.reddit.com/r/DnD/) and [r/legendofzelda](https://www.reddit.com/r/legendofzelda/), but again found too many images and not enough text. I settled on using the slightly smaller but more discussion-focused subreddits [r/DMAcademy](https://www.reddit.com/r/DMAcademy/) and [r/truezelda](https://www.reddit.com/r/truezelda/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Breaking this project down into steps:\n",
    "1. Use Pushshift's API to collect subreddit posts - collect 5,000 posts from [r/DMAcademy](https://www.reddit.com/r/DMAcademy/) and 5,000 posts from [r/truezelda](https://www.reddit.com/r/truezelda/)\n",
    "2. Data cleaning and preprocessing - drop removed or deleted posts from our dataset, drop null values; remove hyperlinks, digits, punctuation, and bot messages from posts\n",
    "3. Conduct EDA - investigate word count per post, character count per post, and post sentiment averaged across each subreddit individually \n",
    "4. Fit models to data - train a Bernoulli Naive Bayes model and a Support Vector Machine (SVM) classifier to predict which subreddit a post came from\n",
    "5. Compare model results and metrics - compare misclassification rates and accuracy scores of each model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After accomplishing the initial goal\n",
    "\n",
    "After completing the minimum goals for this project, I cleaned up my workflow and applied it to a new goal: collecting posts from one subreddits, then creating and comparing two models that can **predict which year** a given post from that subreddit came from. For this goal, I needed a subreddit which would see a significant enough change in vocabulary across time for a model to use in its predictions. I chose [r/PoliticalDiscussion](https://www.reddit.com/r/PoliticalDiscussion/) for this because it met the main requirement of being a text-rich subreddit, and I thought political eras would see a significant enough change in discussion topics and vocabulary for my models to pick up. This subreddit was created in 2011 and focuses on US politics so I chose to pull posts from two years where presidential elections were held, 2012 and 2020. I was working on data collection on March 31, 2021, so I pulled data from March 31, 2012 and March 31, 2020."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='workflow_sample'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Workflow sample\n",
    "\n",
    "Before jumping into the deep end and pull 20,000 subreddit posts, we can pull a small sample of posts from just a single subreddit to prove that we can indeed use this API! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import requests\n",
    "import pandas as pd\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define reddit's URL for requests\n",
    "url = 'https://api.pushshift.io/reddit/search/' + 'submission'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create parameters \n",
    "params = {\n",
    "'subreddit': 'DMAcademy',\n",
    "'size': 10}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initiate pull request\n",
    "res = requests.get(url, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out request's status code\n",
    "res.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A status code of 200 indicates a successful pull! An example of an error at this stage of the process would be getting a 404 status code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get data from json file\n",
    "data = res.json()\n",
    "posts = data['data']\n",
    "df = pd.DataFrame(posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>selftext</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Looking to read or possibly use older modules....</td>\n",
       "      <td>Favorite non-5e module?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Hey folks, I hope you're having an amazing day...</td>\n",
       "      <td>Need some help balancing encounters</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Got a an Eladrin Vamp BBEG(al) who serves Kann...</td>\n",
       "      <td>Eladrin Vampire that serves Kannoth. Autumn or...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>So, Ive been running my own homebrew campaign ...</td>\n",
       "      <td>So i need advice on what to do with my 5e Camp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>My players are about to enter a dungeon that h...</td>\n",
       "      <td>Complex Traps Question</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Now, before people tear me limb from limb. I d...</td>\n",
       "      <td>Tell your players how their characters feel.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>It feels like a treasure hunt trying to find t...</td>\n",
       "      <td>Third party prewritten campaign modules spanni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Easy question:\\n\\nI’ve been playing DnD for tw...</td>\n",
       "      <td>Trying to DM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>tl;dr: players never search or investigate any...</td>\n",
       "      <td>I'd like to reward my players' curiousity, but...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Hey there! I'm running a D&amp;amp;D game where th...</td>\n",
       "      <td>Some Moving Castles would be nice.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            selftext  \\\n",
       "0  Looking to read or possibly use older modules....   \n",
       "1  Hey folks, I hope you're having an amazing day...   \n",
       "2  Got a an Eladrin Vamp BBEG(al) who serves Kann...   \n",
       "3  So, Ive been running my own homebrew campaign ...   \n",
       "4  My players are about to enter a dungeon that h...   \n",
       "5  Now, before people tear me limb from limb. I d...   \n",
       "6  It feels like a treasure hunt trying to find t...   \n",
       "7  Easy question:\\n\\nI’ve been playing DnD for tw...   \n",
       "8  tl;dr: players never search or investigate any...   \n",
       "9  Hey there! I'm running a D&amp;D game where th...   \n",
       "\n",
       "                                               title  \n",
       "0                            Favorite non-5e module?  \n",
       "1                Need some help balancing encounters  \n",
       "2  Eladrin Vampire that serves Kannoth. Autumn or...  \n",
       "3  So i need advice on what to do with my 5e Camp...  \n",
       "4                             Complex Traps Question  \n",
       "5       Tell your players how their characters feel.  \n",
       "6  Third party prewritten campaign modules spanni...  \n",
       "7                                       Trying to DM  \n",
       "8  I'd like to reward my players' curiousity, but...  \n",
       "9                 Some Moving Castles would be nice.  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the posts we just pulled\n",
    "df[['selftext', 'title']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Easy question:\\n\\nI’ve been playing DnD for two years but am curious about DMing. When it comes to creating a small town and the NPCs there within. Is it necessary to create every individual character with stats, race, abilities, etc... I ask because if a PC starts beef with a random NPC do I need to be prepared for a battle or should I just focus on key NPCs that are meant to progress the story. \\n\\nThanks!'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['selftext'][7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printing out our first post makes it easy to see that we'll have a bit of tidying up to do before these posts are ready for our models! We'll tackle that in notebook 02_text_cleaning.ipynb."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='API'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Using Pushshift's API to pull data from subreddits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've done a proof of concept, we can put the above code into a function to pull many more posts over an extended period of time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Warning**\n",
    "Pulling large volumes of data through Pushshift's API puts you at risk of getting banned by the server host. Therefore, it's recommended that you pull small amounts of posts over an extended period of time. I set up a function with a built-in delay that pulls 100 posts per minute for a user-specified number of iterations. To get 5,000 posts, I ran this function for 50 iterations for each subreddit/time period. I've since altered the code in this notebook so that it only pulls 2 iterations, to demonstrate the process without forcing readers to run four 50-minute data pulls. You can read more about Pushshift's API on this [GitHub page](https://github.com/pushshift/api). There is also a [YouTube video](https://www.youtube.com/watch?v=AcrjEWsMi_E) walkthrough of setting up this API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a function to pull data using Pushshift's API\n",
    "# Input: type of data to pull (can be 'submission' or 'comment), \n",
    "# desired number of pull iterations, desired subreddit, desired time\n",
    "\n",
    "def get_posts(pull_type, iters, subreddit, desired_time):\n",
    "    \n",
    "    # Define reddit's URL for requests\n",
    "    url = 'https://api.pushshift.io/reddit/search/' + pull_type\n",
    "        \n",
    "    # Create empty master dataframe to fill\n",
    "    master_df = pd.DataFrame()\n",
    "    \n",
    "    # Loop through specified number \n",
    "    for i in range(iters):\n",
    "        # Set API parameters\n",
    "        params = {\n",
    "        'subreddit': subreddit,\n",
    "        'size': 100,\n",
    "        'before':desired_time}\n",
    "        \n",
    "        # Pull data\n",
    "        res = requests.get(url, params)\n",
    "        data = res.json()\n",
    "        posts = data['data']\n",
    "        df = pd.DataFrame(posts)\n",
    "        \n",
    "        # Concatenate data to master dataframe\n",
    "        frames = [df, master_df]\n",
    "        master_df = pd.concat(frames, axis=0, ignore_index=True)\n",
    "        \n",
    "        # Get time of oldest post in this data\n",
    "        # This resets the API parameters so that you pull older posts every iteration\n",
    "        desired_time = df['created_utc'].min()\n",
    "        print(f'Completed {i+1} iterations, {iters-i-1} iterations remaining')\n",
    "        \n",
    "        # Time delay so you don't get banned by Pushshift\n",
    "        time.sleep(60)\n",
    "    \n",
    "    # Return dataframe containing all collected posts\n",
    "    return master_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pull-data-1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Pulling data from r/DMAcademy and r/truezelda\n",
    "\n",
    "First goal: create and compare models that can predict whether a post came from subreddit A or subreddit B.\n",
    "\n",
    "Second goal: create and compare models that can predict whether a post from one subreddit came from year A or year B. \n",
    "\n",
    "Thus, I pulled data from three different subreddits but did four total pulls: one from [r/DMAcademy](https://www.reddit.com/r/DMAcademy/), one from [r/truezelda](https://www.reddit.com/r/truezelda/), one from [r/PoliticalDiscussion](https://www.reddit.com/r/PoliticalDiscussion/) in the year 2012, and one from r/PoliticalDiscussion in the year 2020. \n",
    "\n",
    "I chose to pull 5,000 posts from each subreddit/time period to ensure that my models would be well-informed. It was recommended that my models be trained on a minimum of 2,000 posts from each subreddit, so I pulled well over that to ensure that I would have enough posts to work with even after dropping potentially hundreds of unusable posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, I pulled 5,000 posts from r/DMAcademy and stored it in a dataframe. The `get_posts` function pulls 100 posts per iteration, so passing it 50 will produce 50 * 100, or 5,000 posts. I passed `int(time.time())` to the function to pull the 5,000 most recent posts at the time of writing. When I pulled posts from r/PoliticalDiscussion, I passed the function a specific time in [Unix or Epoch time](https://en.wikipedia.org/wiki/Unix_time) (formatted as number of seconds since 00:00:00 Jan 1, 1970, an arbitrary date) to pull posts from a specific date and time in 2012 and 2020."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 1 iterations, 1 iterations remaining\n",
      "Completed 2 iterations, 0 iterations remaining\n"
     ]
    }
   ],
   "source": [
    "# pull data from r/DMAcademy\n",
    "dmacademy_df = get_posts('submission', 2, 'DMAcademy', int(time.time()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>all_awardings</th>\n",
       "      <th>allow_live_comments</th>\n",
       "      <th>author</th>\n",
       "      <th>author_flair_css_class</th>\n",
       "      <th>author_flair_richtext</th>\n",
       "      <th>author_flair_text</th>\n",
       "      <th>author_flair_type</th>\n",
       "      <th>author_fullname</th>\n",
       "      <th>author_patreon_flair</th>\n",
       "      <th>author_premium</th>\n",
       "      <th>...</th>\n",
       "      <th>url</th>\n",
       "      <th>whitelist_status</th>\n",
       "      <th>wls</th>\n",
       "      <th>link_flair_css_class</th>\n",
       "      <th>post_hint</th>\n",
       "      <th>preview</th>\n",
       "      <th>author_cakeday</th>\n",
       "      <th>author_flair_background_color</th>\n",
       "      <th>author_flair_text_color</th>\n",
       "      <th>removed_by_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>CORBICULACC</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_9ylxmsj9</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Dragonboy233</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_4enr9fgd</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Fadinglight656</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_95mbl9ol</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Miner49r10</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_pmomu</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>Guide</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Wabafeedle</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_xhccq</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Donutslayer4137</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_oseao</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Voracious-Herbivore</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_5a67v8zo</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Dungeons_and_Doctors</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_88ollbtg</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>KalosKagathos98</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_58bpivyp</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>self</td>\n",
       "      <td>{'enabled': False, 'images': [{'id': 'LHMtPlsc...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>pucksrage</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_710zy1h</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>https://www.reddit.com/r/DMAcademy/comments/mi...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>self</td>\n",
       "      <td>{'enabled': False, 'images': [{'id': 'KNSJmVJC...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 66 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    all_awardings  allow_live_comments                author  \\\n",
       "0              []                False           CORBICULACC   \n",
       "1              []                False          Dragonboy233   \n",
       "2              []                False        Fadinglight656   \n",
       "3              []                False            Miner49r10   \n",
       "4              []                False            Wabafeedle   \n",
       "..            ...                  ...                   ...   \n",
       "195            []                False       Donutslayer4137   \n",
       "196            []                False   Voracious-Herbivore   \n",
       "197            []                False  Dungeons_and_Doctors   \n",
       "198            []                False       KalosKagathos98   \n",
       "199            []                False             pucksrage   \n",
       "\n",
       "    author_flair_css_class author_flair_richtext author_flair_text  \\\n",
       "0                     None                    []              None   \n",
       "1                     None                    []              None   \n",
       "2                     None                    []              None   \n",
       "3                     None                    []              None   \n",
       "4                     None                    []              None   \n",
       "..                     ...                   ...               ...   \n",
       "195                   None                    []              None   \n",
       "196                   None                    []              None   \n",
       "197                   None                    []              None   \n",
       "198                   None                    []              None   \n",
       "199                   None                    []              None   \n",
       "\n",
       "    author_flair_type author_fullname author_patreon_flair author_premium  \\\n",
       "0                text     t2_9ylxmsj9                False          False   \n",
       "1                text     t2_4enr9fgd                False          False   \n",
       "2                text     t2_95mbl9ol                False          False   \n",
       "3                text        t2_pmomu                False          False   \n",
       "4                text        t2_xhccq                False          False   \n",
       "..                ...             ...                  ...            ...   \n",
       "195              text        t2_oseao                False          False   \n",
       "196              text     t2_5a67v8zo                False          False   \n",
       "197              text     t2_88ollbtg                False          False   \n",
       "198              text     t2_58bpivyp                False          False   \n",
       "199              text      t2_710zy1h                False          False   \n",
       "\n",
       "     ...                                                url  whitelist_status  \\\n",
       "0    ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "1    ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "2    ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "3    ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "4    ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "..   ...                                                ...               ...   \n",
       "195  ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "196  ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "197  ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "198  ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "199  ...  https://www.reddit.com/r/DMAcademy/comments/mi...           all_ads   \n",
       "\n",
       "     wls  link_flair_css_class post_hint  \\\n",
       "0      6                   NaN       NaN   \n",
       "1      6                   NaN       NaN   \n",
       "2      6                   NaN       NaN   \n",
       "3      6                 Guide       NaN   \n",
       "4      6                   NaN       NaN   \n",
       "..   ...                   ...       ...   \n",
       "195    6                   NaN       NaN   \n",
       "196    6                   NaN       NaN   \n",
       "197    6                   NaN       NaN   \n",
       "198    6                   NaN      self   \n",
       "199    6                   NaN      self   \n",
       "\n",
       "                                               preview author_cakeday  \\\n",
       "0                                                  NaN            NaN   \n",
       "1                                                  NaN            NaN   \n",
       "2                                                  NaN            NaN   \n",
       "3                                                  NaN            NaN   \n",
       "4                                                  NaN            NaN   \n",
       "..                                                 ...            ...   \n",
       "195                                                NaN            NaN   \n",
       "196                                                NaN            NaN   \n",
       "197                                                NaN            NaN   \n",
       "198  {'enabled': False, 'images': [{'id': 'LHMtPlsc...            NaN   \n",
       "199  {'enabled': False, 'images': [{'id': 'KNSJmVJC...            NaN   \n",
       "\n",
       "    author_flair_background_color  author_flair_text_color  \\\n",
       "0                             NaN                      NaN   \n",
       "1                             NaN                      NaN   \n",
       "2                             NaN                      NaN   \n",
       "3                             NaN                      NaN   \n",
       "4                             NaN                      NaN   \n",
       "..                            ...                      ...   \n",
       "195                           NaN                      NaN   \n",
       "196                           NaN                      NaN   \n",
       "197                           NaN                      NaN   \n",
       "198                           NaN                      NaN   \n",
       "199                           NaN                      NaN   \n",
       "\n",
       "     removed_by_category  \n",
       "0                    NaN  \n",
       "1                    NaN  \n",
       "2                    NaN  \n",
       "3                    NaN  \n",
       "4                    NaN  \n",
       "..                   ...  \n",
       "195                  NaN  \n",
       "196                  NaN  \n",
       "197                  NaN  \n",
       "198                  NaN  \n",
       "199                  NaN  \n",
       "\n",
       "[200 rows x 66 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the dataframe\n",
    "dmacademy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm the number of unique posts we just pulled\n",
    "dmacademy_df['id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The printout above shows us the number of **unique reddit links** contained in our dataframe. This tells us that we didn't pull any duplicate posts, hooray! \n",
    "\n",
    "This all looks good, so now we can pull posts from r/truezelda."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 1 iterations, 1 iterations remaining\n",
      "Completed 2 iterations, 0 iterations remaining\n"
     ]
    }
   ],
   "source": [
    "# Pull posts from r/truezelda\n",
    "truezelda_df = get_posts('submission', 2, 'truezelda', int(time.time()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>all_awardings</th>\n",
       "      <th>allow_live_comments</th>\n",
       "      <th>author</th>\n",
       "      <th>author_flair_css_class</th>\n",
       "      <th>author_flair_richtext</th>\n",
       "      <th>author_flair_text</th>\n",
       "      <th>author_flair_type</th>\n",
       "      <th>author_fullname</th>\n",
       "      <th>author_patreon_flair</th>\n",
       "      <th>author_premium</th>\n",
       "      <th>...</th>\n",
       "      <th>whitelist_status</th>\n",
       "      <th>wls</th>\n",
       "      <th>post_hint</th>\n",
       "      <th>preview</th>\n",
       "      <th>author_flair_background_color</th>\n",
       "      <th>author_flair_text_color</th>\n",
       "      <th>banned_by</th>\n",
       "      <th>edited</th>\n",
       "      <th>suggested_sort</th>\n",
       "      <th>author_cakeday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>ebrennan123</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_641so1bp</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>lutyrannus</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_89rcoije</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>self</td>\n",
       "      <td>{'enabled': False, 'images': [{'id': 'O-MO_jJf...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>b100mpanda</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_8grakdiw</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>MarcMars82</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_48efq32a</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Magus_Sisters</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_amnlc1yt</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>TheDerpThatDerps</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_4t71o7tm</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Dragenby</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_5c40z7wo</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>self</td>\n",
       "      <td>{'enabled': False, 'images': [{'id': 'u3Oz3nfH...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Mankiz</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_7eyebbx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>in_the_shire</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_6nwdti6n</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Watson_Rosewater</td>\n",
       "      <td>None</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_52kdmfo3</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>all_ads</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 68 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    all_awardings  allow_live_comments            author  \\\n",
       "0              []                False       ebrennan123   \n",
       "1              []                False        lutyrannus   \n",
       "2              []                False        b100mpanda   \n",
       "3              []                False        MarcMars82   \n",
       "4              []                False     Magus_Sisters   \n",
       "..            ...                  ...               ...   \n",
       "195            []                False  TheDerpThatDerps   \n",
       "196            []                False          Dragenby   \n",
       "197            []                False            Mankiz   \n",
       "198            []                False      in_the_shire   \n",
       "199            []                False  Watson_Rosewater   \n",
       "\n",
       "    author_flair_css_class author_flair_richtext author_flair_text  \\\n",
       "0                     None                    []              None   \n",
       "1                     None                    []              None   \n",
       "2                     None                    []              None   \n",
       "3                     None                    []              None   \n",
       "4                     None                    []              None   \n",
       "..                     ...                   ...               ...   \n",
       "195                   None                    []              None   \n",
       "196                   None                    []              None   \n",
       "197                   None                    []              None   \n",
       "198                   None                    []              None   \n",
       "199                   None                    []              None   \n",
       "\n",
       "    author_flair_type author_fullname author_patreon_flair author_premium  \\\n",
       "0                text     t2_641so1bp                False          False   \n",
       "1                text     t2_89rcoije                False          False   \n",
       "2                text     t2_8grakdiw                False           True   \n",
       "3                text     t2_48efq32a                False          False   \n",
       "4                text     t2_amnlc1yt                False          False   \n",
       "..                ...             ...                  ...            ...   \n",
       "195              text     t2_4t71o7tm                False          False   \n",
       "196              text     t2_5c40z7wo                False          False   \n",
       "197              text      t2_7eyebbx                False          False   \n",
       "198              text     t2_6nwdti6n                False          False   \n",
       "199              text     t2_52kdmfo3                False          False   \n",
       "\n",
       "     ... whitelist_status  wls  post_hint  \\\n",
       "0    ...          all_ads    6        NaN   \n",
       "1    ...          all_ads    6       self   \n",
       "2    ...          all_ads    6        NaN   \n",
       "3    ...          all_ads    6        NaN   \n",
       "4    ...          all_ads    6        NaN   \n",
       "..   ...              ...  ...        ...   \n",
       "195  ...          all_ads    6        NaN   \n",
       "196  ...          all_ads    6       self   \n",
       "197  ...          all_ads    6        NaN   \n",
       "198  ...          all_ads    6        NaN   \n",
       "199  ...          all_ads    6        NaN   \n",
       "\n",
       "                                               preview  \\\n",
       "0                                                  NaN   \n",
       "1    {'enabled': False, 'images': [{'id': 'O-MO_jJf...   \n",
       "2                                                  NaN   \n",
       "3                                                  NaN   \n",
       "4                                                  NaN   \n",
       "..                                                 ...   \n",
       "195                                                NaN   \n",
       "196  {'enabled': False, 'images': [{'id': 'u3Oz3nfH...   \n",
       "197                                                NaN   \n",
       "198                                                NaN   \n",
       "199                                                NaN   \n",
       "\n",
       "    author_flair_background_color author_flair_text_color banned_by edited  \\\n",
       "0                             NaN                     NaN       NaN    NaN   \n",
       "1                             NaN                     NaN       NaN    NaN   \n",
       "2                             NaN                     NaN       NaN    NaN   \n",
       "3                             NaN                     NaN       NaN    NaN   \n",
       "4                             NaN                     NaN       NaN    NaN   \n",
       "..                            ...                     ...       ...    ...   \n",
       "195                           NaN                     NaN       NaN    NaN   \n",
       "196                           NaN                     NaN       NaN    NaN   \n",
       "197                           NaN                     NaN       NaN    NaN   \n",
       "198                           NaN                     NaN       NaN    NaN   \n",
       "199                           NaN                     NaN       NaN    NaN   \n",
       "\n",
       "     suggested_sort  author_cakeday  \n",
       "0               NaN             NaN  \n",
       "1               NaN             NaN  \n",
       "2               NaN             NaN  \n",
       "3               NaN             NaN  \n",
       "4               NaN             NaN  \n",
       "..              ...             ...  \n",
       "195             NaN             NaN  \n",
       "196             NaN             NaN  \n",
       "197             NaN             NaN  \n",
       "198             NaN             NaN  \n",
       "199             NaN             NaN  \n",
       "\n",
       "[200 rows x 68 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the dataframe\n",
    "truezelda_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for number of unique posts\n",
    "truezelda_df['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Are the missing ids nulls?\n",
    "truezelda_df['id'].isnull().sum()\n",
    "# Nope! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, you may pull duplicate posts. I'm not sure why this happens, but I've only seen it turn up at a very small fraction of the data, so we can dismiss it and use what we have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to investigate the full datasets without waiting pulling it through the API, run the cell below to read them in as csv's and check the numbers of unique posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000\n",
      "4996\n"
     ]
    }
   ],
   "source": [
    "dmacademy_df = pd.read_csv('../data/dmacademy.csv')\n",
    "truezelda_df = pd.read_csv('../data/truezelda.csv')\n",
    "\n",
    "print(dmacademy_df['id'].nunique())\n",
    "print(truezelda_df['id'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See what I mean about pulling duplicate posts? Since it only appears to be 4 duplicates out of 5,000, I'm going to keep what I have and move on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save data to .csv files\n",
    "Now that we've pulled the data needed for the first goal of this project, let's save it all as `.csv`s. I commented out the lines below to prevent myself from accidentally overwriting my datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set index=False to avoid creating an unnecessary index column\n",
    "# dmacademy_df.to_csv('../data/dmacademy.csv', index=False)\n",
    "# truezelda_df.to_csv('../data/truezelda.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pull-data-2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Pulling data from r/PoliticalDiscussion\n",
    "\n",
    "Now let's pull posts from r/PoliticalDiscussion. I'm pulling data from different years using an [Epoch time converter](https://www.epochconverter.com/). \n",
    "\n",
    "The specific Epoch times I used are 1333169208, which is Saturday, March 31, 2012 4:46:48 AM, and 1585630008, which is Tuesday, March 31, 2020 4:46:48 AM. Same date and time, but different years! \n",
    "\n",
    "I abbreviated the name political discussion to poli_dis for the dataframes below. I will continue to use this abbreviation throughout the rest of the notebooks in this project.\n",
    "\n",
    "**Note**: For whatever reason, pulling subreddit submissions from r/PoliticalDiscussion results in a dataframe full of posts containing only the word 'removed'. This isn't helpful to our modeling, so I decided to pull comments from this subreddit, instead of posts (called 'submissions') like I did for the previous two subreddits. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 1 iterations, 1 iterations remaining\n",
      "Completed 2 iterations, 0 iterations remaining\n"
     ]
    }
   ],
   "source": [
    "poli_dis_2012_df = get_posts('comment', 2, 'PoliticalDiscussion', 1333169208) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2012_df['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 1 iterations, 1 iterations remaining\n",
      "Completed 2 iterations, 0 iterations remaining\n"
     ]
    }
   ],
   "source": [
    "poli_dis_2020_df = get_posts('comment', 2, 'PoliticalDiscussion', 1585630008)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2020_df['id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the goal here isn't to predict on the 'subreddit' column but to predict on the year a post was made, we should add our target variable to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "poli_dis_2012_df['year'] = '2012'\n",
    "poli_dis_2020_df['year'] = '2020'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "200\n"
     ]
    }
   ],
   "source": [
    "print(poli_dis_2012_df['id'].nunique())\n",
    "print(poli_dis_2020_df['id'].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200, 23)\n",
      "(200, 35)\n"
     ]
    }
   ],
   "source": [
    "# Note that the datasets from different years have different numbers of columns\n",
    "print(poli_dis_2012_df.shape)\n",
    "print(poli_dis_2020_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to investigate the full datasets without waiting pulling it through the API, run the cell below to read them in as csv's and check the numbers of unique posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "poli_dis_2012_df = pd.read_csv('../data/poli_dis_2012.csv')\n",
    "poli_dis_2020_df = pd.read_csv('../data/poli_dis_2020.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5000\n",
      "5000\n"
     ]
    }
   ],
   "source": [
    "print(poli_dis_2012_df['id'].nunique())\n",
    "print(poli_dis_2020_df['id'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save data to .csv files\n",
    "Now that we've pulled the data needed for the second model, let's save it as `.csv`s. I commented out the lines below to prevent myself from accidentally overwriting my datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# poli_dis_2012_df.to_csv('../data/poli_dis_2012.csv', index=False)\n",
    "# poli_dis_2020_df.to_csv('../data/poli_dis_2020.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to move on to notebook 2, text cleaning!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}