notebooks/02_text_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Cleaning\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of contents\n",
    "\n",
    "1. [Identifying removed or deleted posts](#rem-del-posts)\n",
    "2. [Dropping unusable posts](#drop-posts)\n",
    "3. [Identifying bot messages](#bot-messages)\n",
    "4. [Isolating desired features](#isolate-features)\n",
    "5. [Inspect text for odd characters](#odd-chars)\n",
    "6. [Use regex to remove unwanted characters](#tidy-text)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Read in the data files\n",
    "We can read in the `.csv`s we created in the first notebook. We're working with four datasets total, so we'll do each step in the cleaning process four times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmacademy_df = pd.read_csv('../data/dmacademy.csv')\n",
    "truezelda_df = pd.read_csv('../data/truezelda.csv')\n",
    "poli_dis_2012_df = pd.read_csv('../data/poli_dis_2012.csv')\n",
    "poli_dis_2020_df = pd.read_csv('../data/poli_dis_2020.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing we should fix before we get going is changing the column names in our dataframes. The submissions have a column called `'selftext'`, while the comments have a column called `'body'`. We want to work with each of these, so let's just rename them all to the same thing so that we only have to write code for one column name instead of two. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmacademy_df.rename(columns={'selftext': 'text'}, inplace=True)\n",
    "truezelda_df.rename(columns={'selftext': 'text'}, inplace=True)\n",
    "poli_dis_2012_df.rename(columns={'body': 'text'}, inplace=True)\n",
    "poli_dis_2020_df.rename(columns={'body': 'text'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='rem-del-posts'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Check for removed or deleted posts\n",
    "\n",
    "One column that caught my eye in these dataframes is the `removed_by_category` column. Each row of this column was found to contain one of six unique values: 'reddit', 'moderator', 'author', 'deleted', 'automod_filtered', or a null value. Rows where this column contains anything other than a null are posts that have been removed from the subreddit. This means the body of the post only contains the word '[removed]' or '[deleted]'. \n",
    "\n",
    "Let's see how many rows in each dataframe might be posts that have been removed or deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>all_awardings</th>\n",
       "      <th>allow_live_comments</th>\n",
       "      <th>author</th>\n",
       "      <th>author_flair_css_class</th>\n",
       "      <th>author_flair_richtext</th>\n",
       "      <th>author_flair_text</th>\n",
       "      <th>author_flair_type</th>\n",
       "      <th>author_fullname</th>\n",
       "      <th>author_patreon_flair</th>\n",
       "      <th>author_premium</th>\n",
       "      <th>...</th>\n",
       "      <th>link_flair_css_class</th>\n",
       "      <th>post_hint</th>\n",
       "      <th>preview</th>\n",
       "      <th>author_flair_background_color</th>\n",
       "      <th>author_flair_text_color</th>\n",
       "      <th>removed_by_category</th>\n",
       "      <th>gilded</th>\n",
       "      <th>author_cakeday</th>\n",
       "      <th>suggested_sort</th>\n",
       "      <th>banned_by</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Vashael</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_7p5eovfe</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Atarihero76</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_48oeurxu</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>Guide</td>\n",
       "      <td>self</td>\n",
       "      <td>{'enabled': False, 'images': [{'id': '9XSNOgfA...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>JethroBuldean</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_1plx2r4z</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Mechaaniac</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_a04c8tpe</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Hungerforhuman</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_a2ouksrx</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>AngelsJos</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_9571c28u</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Light_of_Avalon</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_871qw</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Randoff-Runemaker</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_ab1yxy0e</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>Shatyel</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_1v4sccw1</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>[]</td>\n",
       "      <td>False</td>\n",
       "      <td>casualaudience</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text</td>\n",
       "      <td>t2_2jjmhv7b</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 70 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     all_awardings  allow_live_comments             author  \\\n",
       "0               []                False            Vashael   \n",
       "1               []                False        Atarihero76   \n",
       "2               []                False      JethroBuldean   \n",
       "3               []                False         Mechaaniac   \n",
       "4               []                False     Hungerforhuman   \n",
       "...            ...                  ...                ...   \n",
       "4995            []                False          AngelsJos   \n",
       "4996            []                False    Light_of_Avalon   \n",
       "4997            []                False  Randoff-Runemaker   \n",
       "4998            []                False            Shatyel   \n",
       "4999            []                False     casualaudience   \n",
       "\n",
       "     author_flair_css_class author_flair_richtext author_flair_text  \\\n",
       "0                       NaN                    []               NaN   \n",
       "1                       NaN                    []               NaN   \n",
       "2                       NaN                    []               NaN   \n",
       "3                       NaN                    []               NaN   \n",
       "4                       NaN                    []               NaN   \n",
       "...                     ...                   ...               ...   \n",
       "4995                    NaN                    []               NaN   \n",
       "4996                    NaN                    []               NaN   \n",
       "4997                    NaN                    []               NaN   \n",
       "4998                    NaN                    []               NaN   \n",
       "4999                    NaN                    []               NaN   \n",
       "\n",
       "     author_flair_type author_fullname author_patreon_flair author_premium  \\\n",
       "0                 text     t2_7p5eovfe                False          False   \n",
       "1                 text     t2_48oeurxu                False          False   \n",
       "2                 text     t2_1plx2r4z                False          False   \n",
       "3                 text     t2_a04c8tpe                False          False   \n",
       "4                 text     t2_a2ouksrx                False          False   \n",
       "...                ...             ...                  ...            ...   \n",
       "4995              text     t2_9571c28u                False           True   \n",
       "4996              text        t2_871qw                False          False   \n",
       "4997              text     t2_ab1yxy0e                False          False   \n",
       "4998              text     t2_1v4sccw1                False          False   \n",
       "4999              text     t2_2jjmhv7b                False          False   \n",
       "\n",
       "      ... link_flair_css_class  post_hint  \\\n",
       "0     ...                  NaN        NaN   \n",
       "1     ...                Guide       self   \n",
       "2     ...                  NaN        NaN   \n",
       "3     ...                  NaN        NaN   \n",
       "4     ...                  NaN        NaN   \n",
       "...   ...                  ...        ...   \n",
       "4995  ...                  NaN        NaN   \n",
       "4996  ...                  NaN        NaN   \n",
       "4997  ...                  NaN        NaN   \n",
       "4998  ...                  NaN        NaN   \n",
       "4999  ...                  NaN        NaN   \n",
       "\n",
       "                                                preview  \\\n",
       "0                                                   NaN   \n",
       "1     {'enabled': False, 'images': [{'id': '9XSNOgfA...   \n",
       "2                                                   NaN   \n",
       "3                                                   NaN   \n",
       "4                                                   NaN   \n",
       "...                                                 ...   \n",
       "4995                                                NaN   \n",
       "4996                                                NaN   \n",
       "4997                                                NaN   \n",
       "4998                                                NaN   \n",
       "4999                                                NaN   \n",
       "\n",
       "      author_flair_background_color author_flair_text_color  \\\n",
       "0                               NaN                     NaN   \n",
       "1                               NaN                     NaN   \n",
       "2                               NaN                     NaN   \n",
       "3                               NaN                     NaN   \n",
       "4                               NaN                     NaN   \n",
       "...                             ...                     ...   \n",
       "4995                            NaN                     NaN   \n",
       "4996                            NaN                     NaN   \n",
       "4997                            NaN                     NaN   \n",
       "4998                            NaN                     NaN   \n",
       "4999                            NaN                     NaN   \n",
       "\n",
       "      removed_by_category gilded author_cakeday suggested_sort  banned_by  \n",
       "0                     NaN    NaN            NaN            NaN        NaN  \n",
       "1                     NaN    NaN            NaN            NaN        NaN  \n",
       "2                     NaN    NaN            NaN            NaN        NaN  \n",
       "3                     NaN    NaN            NaN            NaN        NaN  \n",
       "4                     NaN    NaN            NaN            NaN        NaN  \n",
       "...                   ...    ...            ...            ...        ...  \n",
       "4995                  NaN    NaN            NaN            NaN        NaN  \n",
       "4996                  NaN    NaN            NaN            NaN        NaN  \n",
       "4997                  NaN    NaN            NaN            NaN        NaN  \n",
       "4998                  NaN    NaN            NaN            NaN        NaN  \n",
       "4999                  NaN    NaN            NaN            NaN        NaN  \n",
       "\n",
       "[5000 rows x 70 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect one of our dataframes\n",
    "dmacademy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['[deleted]' '[removed]' nan]\n",
      "['[removed]' '[deleted]' nan]\n"
     ]
    }
   ],
   "source": [
    "for df in [dmacademy_df, truezelda_df]:\n",
    "    # Identify all the unique values in the 'removed_by_category' column\n",
    "    # print(df['removed_by_category'].unique())\n",
    "    \n",
    "    # Investigate unique values in posts that were removed\n",
    "    print(df[df['removed_by_category'].notnull()]['text'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this information, I decided to remove all rows where `selftext` or `body` is deleted, removed, or nan. Unforunately, this means dropping hundreds of rows, but I intentionally pulled more data than I thought necessary to account for this possibility. \n",
    "\n",
    "Below, I print out examples of nulls, deleted, and removed posts in the DMAcademy data. All of these rows still have titles, but including the titles will give us the impression later that our classes are more balanced than they truly are, so I'd rather drop them, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>removed_by_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>775</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Giving depth to your random encounters</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>937</th>\n",
       "      <td>NaN</td>\n",
       "      <td>New DM</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3190</th>\n",
       "      <td>NaN</td>\n",
       "      <td>How should I balance low stake/high stake enco...</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3318</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Тhеrе is а Yоutubеr cаlled \"Lеon Farаdаy\" that...</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     text                                              title  \\\n",
       "775   NaN             Giving depth to your random encounters   \n",
       "937   NaN                                             New DM   \n",
       "3190  NaN  How should I balance low stake/high stake enco...   \n",
       "3318  NaN  Тhеrе is а Yоutubеr cаlled \"Lеon Farаdаy\" that...   \n",
       "\n",
       "     removed_by_category  \n",
       "775              deleted  \n",
       "937              deleted  \n",
       "3190             deleted  \n",
       "3318             deleted  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dmacademy_df[dmacademy_df['text'].isnull()][['text','title','removed_by_category']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>removed_by_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>I need help can I ask some assistance.</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>What to do with an unhelpful player.</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>(popular question) Playing after LMoP for newb...</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>how should i go about this session if there I'...</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>Menacing music for Villains?</td>\n",
       "      <td>deleted</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         text                                              title  \\\n",
       "21  [deleted]             I need help can I ask some assistance.   \n",
       "38  [deleted]               What to do with an unhelpful player.   \n",
       "62  [deleted]  (popular question) Playing after LMoP for newb...   \n",
       "78  [deleted]  how should i go about this session if there I'...   \n",
       "81  [deleted]                       Menacing music for Villains?   \n",
       "\n",
       "   removed_by_category  \n",
       "21             deleted  \n",
       "38             deleted  \n",
       "62             deleted  \n",
       "78             deleted  \n",
       "81             deleted  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dmacademy_df.loc[dmacademy_df['text']=='[deleted]'][['text','title','removed_by_category']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>title</th>\n",
       "      <th>removed_by_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>[removed]</td>\n",
       "      <td>Who is the Queen of Air and Darkness? What hap...</td>\n",
       "      <td>moderator</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>[removed]</td>\n",
       "      <td>A helpful video about how to design a balanced...</td>\n",
       "      <td>moderator</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>[removed]</td>\n",
       "      <td>Character backstory assist.</td>\n",
       "      <td>moderator</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>227</th>\n",
       "      <td>[removed]</td>\n",
       "      <td>New player to Dnd/ build help pls</td>\n",
       "      <td>moderator</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>[removed]</td>\n",
       "      <td>My friend asked me if I knew much about the Ba...</td>\n",
       "      <td>moderator</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          text                                              title  \\\n",
       "47   [removed]  Who is the Queen of Air and Darkness? What hap...   \n",
       "89   [removed]  A helpful video about how to design a balanced...   \n",
       "224  [removed]                        Character backstory assist.   \n",
       "227  [removed]                  New player to Dnd/ build help pls   \n",
       "241  [removed]  My friend asked me if I knew much about the Ba...   \n",
       "\n",
       "    removed_by_category  \n",
       "47            moderator  \n",
       "89            moderator  \n",
       "224           moderator  \n",
       "227           moderator  \n",
       "241           moderator  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dmacademy_df.loc[dmacademy_df['text']=='[removed]'][['text','title','removed_by_category']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's see jut how many rows we'll be dropping from each dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create lists of dataframe names and dataframes for easier iteration\n",
    "df_names_list = ['dmacademy_df', 'truezelda_df', 'poli_dis_2012_df', 'poli_dis_2020_df']\n",
    "df_list = [dmacademy_df, truezelda_df, poli_dis_2012_df, poli_dis_2020_df]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dmacademy_df contains 113 empty posts\n",
      "truezelda_df contains 884 empty posts\n",
      "poli_dis_2012_df contains 267 empty posts\n",
      "poli_dis_2020_df contains 238 empty posts\n"
     ]
    }
   ],
   "source": [
    "for i, dataframe in enumerate(df_list):    \n",
    "    # add up number of rows that contain 'deleted', 'removed' , or nan\n",
    "    num_nulls = 0\n",
    "    num_nulls += len(dataframe.loc[(dataframe['text']=='[deleted]') | (dataframe['text']=='[removed]')])\n",
    "    num_nulls += dataframe['text'].isnull().sum()\n",
    "    \n",
    "    print(f'{df_names_list[i]} contains {num_nulls} empty posts')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='drop-posts'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yup, looks like we'll be dropping a considerable amount."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Drop rows with removed, deleted, or null posts or comments\n",
    "\n",
    "Now that we feel confident we've identified all the empty posts, let's drop them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, df in enumerate(df_list):\n",
    "    \n",
    "    # Drop rows where posts are nulls\n",
    "    df.dropna(axis=0, inplace=True, subset=['text'])\n",
    "\n",
    "    # Drop rows where posts were deleted\n",
    "    deleted_rows = df.loc[df['text']=='[deleted]'].index\n",
    "    df.drop(deleted_rows, inplace=True, axis=0)\n",
    "    \n",
    "    # Drop rows where posts were removed\n",
    "    removed_rows = df.loc[df['text']=='[removed]'].index\n",
    "    df.drop(removed_rows, inplace=True, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can confirm our work here was successful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dmacademy_df contains 0 empty posts\n",
      "truezelda_df contains 0 empty posts\n",
      "poli_dis_2012_df contains 0 empty posts\n",
      "poli_dis_2020_df contains 0 empty posts\n"
     ]
    }
   ],
   "source": [
    "for i, dataframe in enumerate(df_list):    \n",
    "    # add up number of rows that contain 'deleted', 'removed' or nan\n",
    "    num_nulls = 0\n",
    "    num_nulls += len(dataframe.loc[(dataframe['text']=='[deleted]') | (dataframe['text']=='[removed]')])\n",
    "    num_nulls += dataframe['text'].isnull().sum()\n",
    "    \n",
    "    print(f'{df_names_list[i]} contains {num_nulls} empty posts')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4887\n",
      "4112\n",
      "4733\n",
      "4762\n"
     ]
    }
   ],
   "source": [
    "# Inspect the new size of our dataframes\n",
    "for df in df_list:\n",
    "    print(len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataframes are significantly smaller now, but so much cleaner! Yay!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='bot-messages'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Check for bot messages\n",
    "I don't anticipate finding any bot messages in the r/truezelda and r/DMAcademy posts, but I've already noticed some in the comments from the r/PoliticalDiscussion. This makes sense; many subreddits have automated bots to post informative comments.\n",
    "\n",
    "Unfortunately, I don't know all the forms that bot messages can come in, so I may not be able to find them all, but I removed what I was able to find so that my models aren't informed by repetitive bot messages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dmacademy_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "dmacademy_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "truezelda_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "truezelda_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "poli_dis_2012_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "poli_dis_2012_df\n",
      "Series([], Name: text, dtype: object) \n",
      "\n",
      "poli_dis_2020_df\n",
      "37      Hello, /u/PMmeURsprintPROGRAMS. Thanks for con...\n",
      "38      Hello, /u/pleasedontbullyme_. Thanks for contr...\n",
      "39      Hello, /u/chickenman86. Thanks for contributin...\n",
      "40      Hello, /u/dumbirds. Thanks for contributing! U...\n",
      "67      Hello, /u/Theduder89. Thanks for contributing!...\n",
      "                              ...                        \n",
      "4987    Hello, /u/cc_hk. Thanks for contributing! Unfo...\n",
      "4994    Hello, /u/Realtalkdo3. Thanks for contributing...\n",
      "4995    Hello, /u/Gransanto102. Thanks for contributin...\n",
      "4996    Hello, /u/SocialObserver3802. Thanks for contr...\n",
      "4997    Hello, /u/SthenicFreeze. Thanks for contributi...\n",
      "Name: text, Length: 182, dtype: object \n",
      "\n",
      "poli_dis_2020_df\n",
      "1       [A reminder for everyone](https://www.reddit.c...\n",
      "3       [A reminder for everyone](https://www.reddit.c...\n",
      "19      [A reminder for everyone](https://www.reddit.c...\n",
      "42      [A reminder for everyone](https://www.reddit.c...\n",
      "45      [A reminder for everyone](https://www.reddit.c...\n",
      "                              ...                        \n",
      "4909    Your submission was automatically removed.\\n\\n...\n",
      "4947    [A reminder for everyone](https://www.reddit.c...\n",
      "4962    [A reminder for everyone](https://www.reddit.c...\n",
      "4989    [A reminder for everyone](https://www.reddit.c...\n",
      "4990    [A reminder for everyone](https://www.reddit.c...\n",
      "Name: text, Length: 271, dtype: object \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, df in enumerate(df_list):\n",
    "    print(df_names_list[i])\n",
    "    print(df.loc[df['text'].str.contains('Hello, /u/')]['text'], '\\n')\n",
    "    print(df_names_list[i])\n",
    "    print(df.loc[df['text'].str.contains('I am a bot')]['text'], '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate a couple of these messages to make sure they're just bots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, /u/PMmeURsprintPROGRAMS. Thanks for contributing! Unfortunately your submission has been removed per the following rules:\\n\\n\\n* Please do not submit low investment content. Low investment content can be, but is not limited to DAE, ELI5, CMV, TIL, polls, requests for users to educate you on a subject or assist you with an outside argument, bald speculation, and discussion prompts that boil down to \"thoughts\", \"how does this affect the election\", \"discuss\", and other similar permutations.\\n\\n* Submissions must discuss a political topic and you must ask a question on that topic.\\n\\n\\n\\nPlease note that our submission guidelines are intended to maintain the high quality of discussion on the subreddit. Except in rare circumstances, removal of your submission does not count as a \\'warning\\', and we hope you feel encouraged to redraft within our guidelines per the sidebar and our [wiki guide to posting](https://www.reddit.com/r/PoliticalDiscussion/wiki/posts). If you feel this was done in error, would like clarification, or need further assistance, please [message the moderators via modmail](https://www.reddit.com/message/compose?to=/r/PoliticalDiscussion&amp;subject=Messaging regarding the removal of this submission by /u/PMmeURsprintPROGRAMS&amp;message=Messaging about [this submission.](https://www.reddit.com/r/PoliticalDiscussion/comments/fp08ag/-/\\\\)). **Do not PM individual moderators.** Please check with the moderators first before reposting this topic. Thank you!'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2020_df['text'][37]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"[A reminder for everyone](https://www.reddit.com/r/PoliticalDiscussion/comments/4479er/rules_explanations_and_reminders/). This is a subreddit for genuine discussion:\\n\\n* Don't post low effort comments like joke threads, memes, slogans, or links without context.\\n* Help prevent this subreddit from becoming an echo chamber. Please don't downvote comments with which you disagree.\\n* The downvote and report buttons are not disagree buttons.  Please don't use them that way.\\n\\nViolators will be fed to the bear.\\n\\n---\\n\\n*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/PoliticalDiscussion) if you have any questions or concerns.*\""
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2020_df['text'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yup, looks like nonsense that we don't want! Let's drop these rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dmacademy_df is 4887 rows\n",
      "truezelda_df is 4112 rows\n",
      "poli_dis_2012_df is 4733 rows\n",
      "poli_dis_2020_df is 4309 rows\n"
     ]
    }
   ],
   "source": [
    "for i, df in enumerate(df_list):\n",
    "\n",
    "    # Drop rows where posts were deleted\n",
    "    hello_bots = df.loc[df['text'].str.contains('Hello, /u/')]['text'].index\n",
    "    df.drop(hello_bots, inplace=True, axis=0)\n",
    "    \n",
    "    # Drop rows where posts were removed\n",
    "    iamabot_bots = df.loc[df['text'].str.contains('I am a bot')]['text'].index\n",
    "    df.drop(iamabot_bots, inplace=True, axis=0)\n",
    "    \n",
    "    # Inspect the final sizes of our dataframes\n",
    "    print(f'{df_names_list[i]} is {len(df)} rows')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the final sizes of our datasets! They've changed a lot from the initial 5,000, but we've still got plenty to work with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='isolate-features'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Isolate the subreddit, title, and text columns \n",
    "After dropping empty posts, our dataframes still contain a lot of arbitrary data in all of those extra columns. Since this is a project on Natural Language Processing (NLP), I'm only going to use the title and main text (stored in the `title` and `text` columns, respectively) for every post. Comments don't have titles, so for the data from r/PoliticalDiscussion we can simply use the `text` column. We can isolate these columns into new dataframes so that we aren't working with so much extraneous data, without losing or overwriting any of our original data, just in case we end up wanting it for something later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmacademy_df = dmacademy_df[['subreddit', 'title', 'text']].copy()\n",
    "truezelda_df = truezelda_df[['subreddit', 'title', 'text']].copy()\n",
    "poli_dis_2012_df = poli_dis_2012_df[['subreddit', 'text']].copy()\n",
    "poli_dis_2020_df = poli_dis_2020_df[['subreddit', 'text']].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subreddit</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Seeking seasoned DMs to be guests on interview...</td>\n",
       "      <td>**Edit (UPDATE): Thank you for the robust resp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>TERRAIN, and Using it Effectively</td>\n",
       "      <td>TERRAIN, and using it Effectively – DM Tips\\n\\...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Know the exact location of something</td>\n",
       "      <td>The players are planning on dropping an evil a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>How to run military basic as a session</td>\n",
       "      <td>I am running a campaign for all intents and pu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Best time to take breaks/how long they should be</td>\n",
       "      <td>Hey just a newbie DM .My sessions are usually ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>🐟 Urgently need DM support to appropriately en...</td>\n",
       "      <td>Hi all, I am hoping for some advice please! I ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Do we help with plot here?</td>\n",
       "      <td>Mouseketeers go away\\n\\nSo i’m stuck on a plot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Where to go next? Open to suggestions.</td>\n",
       "      <td>I have my first group about to finish Lost Min...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>NPCs - Playing against type</td>\n",
       "      <td>I'm pretty new to DMing and take very long to ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>DMAcademy</td>\n",
       "      <td>Need Help World Building // Populating a Magic...</td>\n",
       "      <td>Hello!\\n\\nDragon Maidens if any of you are rea...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4887 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      subreddit                                              title  \\\n",
       "0     DMAcademy  Seeking seasoned DMs to be guests on interview...   \n",
       "1     DMAcademy                  TERRAIN, and Using it Effectively   \n",
       "2     DMAcademy               Know the exact location of something   \n",
       "3     DMAcademy             How to run military basic as a session   \n",
       "4     DMAcademy   Best time to take breaks/how long they should be   \n",
       "...         ...                                                ...   \n",
       "4995  DMAcademy  🐟 Urgently need DM support to appropriately en...   \n",
       "4996  DMAcademy                         Do we help with plot here?   \n",
       "4997  DMAcademy             Where to go next? Open to suggestions.   \n",
       "4998  DMAcademy                        NPCs - Playing against type   \n",
       "4999  DMAcademy  Need Help World Building // Populating a Magic...   \n",
       "\n",
       "                                                   text  \n",
       "0     **Edit (UPDATE): Thank you for the robust resp...  \n",
       "1     TERRAIN, and using it Effectively – DM Tips\\n\\...  \n",
       "2     The players are planning on dropping an evil a...  \n",
       "3     I am running a campaign for all intents and pu...  \n",
       "4     Hey just a newbie DM .My sessions are usually ...  \n",
       "...                                                 ...  \n",
       "4995  Hi all, I am hoping for some advice please! I ...  \n",
       "4996  Mouseketeers go away\\n\\nSo i’m stuck on a plot...  \n",
       "4997  I have my first group about to finish Lost Min...  \n",
       "4998  I'm pretty new to DMing and take very long to ...  \n",
       "4999  Hello!\\n\\nDragon Maidens if any of you are rea...  \n",
       "\n",
       "[4887 rows x 3 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect one new dataframe\n",
    "dmacademy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='odd-chars'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Inspect text for odd characters\n",
    "\n",
    "There are a lot of funky characters and strings in our text that we might want to consider removing before we tokenize and vectorize our data for model fitting. This includes typical relics like `\\n` and `&amp;`, but there's also some unseemly strings like `#x200B;`, as well as a lot of [YouTube links](https://www.youtube.com/watch?v=xvFZjo5PgG0). Some of these characters might be filtered out under the hood in our models, but I'm not an expert in NLP classification models so I'm going to do a bit of cleaning myself, or else I won't be able to say with confidence that this text data was properly processed. However, since I'm not expert, there is probably something I've missed.\n",
    "\n",
    "First, let's print out a post from a single dataframe to skim for funky strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TERRAIN, and using it Effectively – DM Tips\n",
      "\n",
      "Using terrain Effectively Video:\n",
      "\n",
      "[https://youtu.be/AnpNtWTIX2Q](https://youtu.be/AnpNtWTIX2Q)\n",
      "\n",
      "Hey folks, I’d like to share with you some advice, in video and written form, on the use of Terrain in your Tabletop RPGs.\n",
      "\n",
      "I see a lot of questions and suggestions on adding terrain to your combats and skill challenges, but just plopping down some environment features is not the end of the technique, it is the beginning. Here I will spell out definitions and techniques for how to actually go about making terrain that is effective and will add drama to your encounters/scenes.\n",
      "\n",
      "The official dictionary definition of TERRAIN is: a stretch of land, especially with regard to its physical features\n",
      "\n",
      "How that relates to DnD (and other RPG settings) in my mind, is to redefine Terrain as: Anything in a scene or setting that is not a creature, (but also sometimes a creature can be a terrain feature...).\n",
      "\n",
      "So Terrain in an RPG is not only the swamps, treetops, bushes, boulders, weather, it is anything and every physical thing in a scene, Think of it all connected. \n",
      "\n",
      "HAVE A PLAN! Don’t just make the terrain a backdrop, make it an active force in the encounter.\n",
      "\n",
      "There are 4 categories of Terrain, and some of them can overlap, but this gives you a good breakdown of options when creating Terrain and designing encounters: \n",
      "\n",
      "**Set Pieces**, like scenes from Indiana Jones, \n",
      "\n",
      "**Monster tactics and/or habitats they can exploit**, \n",
      "\n",
      "**Terrain that moves** (rivers, landslides, lava, wind), \n",
      "\n",
      "or even \n",
      "\n",
      "**Monsters that cause terrain changes**, throwing trees, carving swaths out of the ground, etc…\n",
      "\n",
      "A Set-piece is a pre-planned situation, scenario, and/or set-up, that will drastically change the landscape and danger once it triggers. Indiana Jones boulder, Fast and the Furious car scenes, falling bridges, explosions, pretty much all the action scenes in movies are based off set-pieces. Something will change the world in a way that makes the protagonist have to act quickly to avoid certain disaster, while trying to do something else.\n",
      "\n",
      "Examples of Set-pieces: [Shark vs Rowboat](https://www.youtube.com/watch?v=c-xQej68BfI), [Goblin Bridge](https://www.youtube.com/watch?v=r9rhPAS3r1U),  [Zombie Shark and Drow Escape](https://www.youtube.com/watch?v=goyXZOPW1qI)\n",
      "\n",
      "Monster Tactics that use terrain and their natural habitat/defenses. This is where you plan your environment and the creatures/enemies in it to their own advantage. Goblins know to attack then retreat behind walls for full cover, a chameleon stays in the trees to avoid detection, luring players into traps, or other pitfalls/kill-zones, or terrain the monster knows it can move better or fight better in.\n",
      "\n",
      "Examples of Monster Tactics using terrain: [Moose and Dryads,](https://www.youtube.com/watch?v=cCKulrLta8k) [Scorpion Pitfall](https://www.youtube.com/watch?v=GctMlSSwMlw), [MegOctopuss Cavern](https://www.youtube.com/watch?v=N40RIslV5vY)\n",
      "\n",
      "Moving Terrain can be classified as rivers/oceans, landslides, floods, earthquakes, basically anything that keeps the actual ground/water under them in constant motion. This type of terrain tends to work itself out, since it is organic and dangerous by its very nature. But it is still a good idea to have a plan and think of ways to use this environment. The river is moving quickly, so a simple skill challenge to cross, add in some swimming creatures and now you have a very dangerous situation that requires players to plan and really work out.\n",
      "\n",
      "Examples of Moving Terrain: [KoboGnolls at Sea](https://www.youtube.com/watch?v=JTAf58OYoUo), \n",
      "\n",
      "Sometimes it’s Monsters that create Terrain changes. Giants ripping up trees and throwing them, causing damage, and eventually cover/concealment. An Ankheg digging holes/tunnels under player’s feet, a massive leviathan causing a tidal wave. Etc… In these instances you can make the monster’s abilities/tactics tie into the environment to basically create an organic set-piece, caused by the actions of the creature. Fight a Red Dragon in a thick forest? One fire breath now makes a forest an inferno of danger.\n",
      "\n",
      "Some examples of Monsters that cause Terrain changes: [Troll Bridge](https://www.youtube.com/watch?v=cITJbEOqXXM), [Sandworm Plateau](https://www.youtube.com/watch?v=s9IHObatL2U), [Goblin Donkey Kong](https://www.youtube.com/watch?v=_zMte08VlQc)\n",
      "\n",
      "&amp;#x200B;\n",
      "\n",
      "I hope this helps some DMs with not only adding terrain but using it effectively in your own games. it doesn't take much to make a flat combat dynamic, you can use just 1 or multiple techniques from above, the simplest tweak to otherwise mundane combat can make a HUGE difference in fun and drama.\n"
     ]
    }
   ],
   "source": [
    "print(dmacademy_df['text'][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The post above showcases many of the examples I was describing before, including several YouTube links and one `&amp;#x200B;`. Let's use some regular expressions to replace these with empty strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Because this man was part of an invading force. If Russia were occupying our country and all you\\'d already heard stories of torture and murder of innocents, had relatives kicked out of their houses while the military searched for people who weren\\'t there, and heard of entire towns blown up as part of a \"clearing exercise\", you wouldn\\'t blame the invading force instead of just the individual that killed your family?\\n\\nEdit: fixed a typo'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2012_df['text'][14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've also got a lot of backslashes and \\n. Now that we've seen some of the things we want to remove, let's get to it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='tidy-text'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Use regex to remove links, digits, and other unwanted characters\n",
    "\n",
    "I still have a lot to learn about [regular expressions](https://docs.python.org/3/library/re.html) and string processing, so I relied on my classmate Amir Semsarzadeh and my instructor John Hazard to help me with the code below. I found this [Regex Tester](https://www.regextester.com/) to be very helpful in building these expressions! I also relied on many of the answers people have shared on stackoverflow, which I source in the cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Redefine dataframe list to ensure it's updated with just the desired columns we isolated earlier\n",
    "df_list = [dmacademy_df, truezelda_df, poli_dis_2012_df, poli_dis_2020_df]\n",
    "\n",
    "# Iterate over each dataframe\n",
    "for df in df_list:\n",
    "    \n",
    "    # remove links\n",
    "    # https://stackoverflow.com/questions/51994254/removing-url-from-a-column-in-pandas-dataframe/51994366\n",
    "    df.replace('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', ' ', regex=True, inplace=True)\n",
    "    \n",
    "    # remove digits, '&amp;', and forward slashes\n",
    "    # https://stackoverflow.com/questions/41719259/how-to-remove-numbers-from-string-terms-in-a-pandas-dataframe\n",
    "    df.replace('\\d+', '', regex=True, inplace=True) # help from Amir! \n",
    "    df.replace('&amp;', ' ', regex=True, inplace=True)\n",
    "    df.replace('/', ' ', regex=True, inplace=True)\n",
    "    \n",
    "    # remove new line characters\n",
    "    # https://gist.github.com/smram/d6ded3c9028272360eb65bcab564a18a\n",
    "    df.replace(to_replace=[r'\\\\t|\\\\n|\\\\r', '\\t|\\n|\\r'], value=[' ',' '], regex=True, inplace=True)\n",
    "    \n",
    "    # removes punctuation\n",
    "    # From Amir! \n",
    "    df.replace(r'[^a-zA-Z ]\\s?',' ',regex=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TERRAIN and using it Effectively  DM Tips  Using terrain Effectively Video     Hey folks I d like to share with you some advice in video and written form on the use of Terrain in your Tabletop RPGs  I see a lot of questions and suggestions on adding terrain to your combats and skill challenges but just plopping down some environment features is not the end of the technique it is the beginning Here I will spell out definitions and techniques for how to actually go about making terrain that is effective and will add drama to your encounters scenes  The official dictionary definition of TERRAIN is a stretch of land especially with regard to its physical features  How that relates to DnD  and other RPG settings in my mind is to redefine Terrain as Anything in a scene or setting that is not a creature  but also sometimes a creature can be a terrain feature      So Terrain in an RPG is not only the swamps treetops bushes boulders weather it is anything and every physical thing in a scene Think of it all connected   HAVE A PLAN Don t just make the terrain a backdrop make it an active force in the encounter  There are  categories of Terrain and some of them can overlap but this gives you a good breakdown of options when creating Terrain and designing encounters     Set Pieces   like scenes from Indiana Jones     Monster tactics and or habitats they can exploit       Terrain that moves   rivers landslides lava wind    or even     Monsters that cause terrain changes   throwing trees carving swaths out of the ground etc  A Set piece is a pre planned situation scenario and or set up that will drastically change the landscape and danger once it triggers Indiana Jones boulder Fast and the Furious car scenes falling bridges explosions pretty much all the action scenes in movies are based off set pieces Something will change the world in a way that makes the protagonist have to act quickly to avoid certain disaster while trying to do something else  Examples of Set pieces  Shark vs Rowboat    Goblin Bridge     Zombie Shark and Drow Escape    Monster Tactics that use terrain and their natural habitat defenses This is where you plan your environment and the creatures enemies in it to their own advantage Goblins know to attack then retreat behind walls for full cover a chameleon stays in the trees to avoid detection luring players into traps or other pitfalls kill zones or terrain the monster knows it can move better or fight better in  Examples of Monster Tactics using terrain  Moose and Dryads     Scorpion Pitfall    MegOctopuss Cavern    Moving Terrain can be classified as rivers oceans landslides floods earthquakes basically anything that keeps the actual ground water under them in constant motion This type of terrain tends to work itself out since it is organic and dangerous by its very nature But it is still a good idea to have a plan and think of ways to use this environment The river is moving quickly so a simple skill challenge to cross add in some swimming creatures and now you have a very dangerous situation that requires players to plan and really work out  Examples of Moving Terrain  KoboGnolls at Sea     Sometimes it s Monsters that create Terrain changes Giants ripping up trees and throwing them causing damage and eventually cover concealment An Ankheg digging holes tunnels under player s feet a massive leviathan causing a tidal wave Etc In these instances you can make the monster s abilities tactics tie into the environment to basically create an organic set piece caused by the actions of the creature Fight a Red Dragon in a thick forest One fire breath now makes a forest an inferno of danger  Some examples of Monsters that cause Terrain changes  Troll Bridge    Sandworm Plateau    Goblin Donkey Kong      xB  I hope this helps some DMs with not only adding terrain but using it effectively in your own games it doesn t take much to make a flat combat dynamic you can use just  or multiple techniques from above the simplest tweak to otherwise mundane combat can make a HUGE difference in fun and drama \n"
     ]
    }
   ],
   "source": [
    "print(dmacademy_df['text'][1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Because this man was part of an invading force If Russia were occupying our country and all you d already heard stories of torture and murder of innocents had relatives kicked out of their houses while the military searched for people who weren t there and heard of entire towns blown up as part of a  clearing exercise  you wouldn t blame the invading force instead of just the individual that killed your family  Edit fixed a typo'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poli_dis_2012_df['text'][14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This may not be a perfect job, but you can tell that we cleaned out a lot of things! This will pay off when we build our models later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Save processed data for modeling\n",
    "\n",
    "Now that we've finished tidying up our data, we can save it as new `.csv`s to read in in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "dmacademy_df.to_csv('../data/clean_dmacademy.csv', index=False)\n",
    "truezelda_df.to_csv('../data/clean_truezelda.csv', index=False)\n",
    "\n",
    "poli_dis_2012_df.to_csv('../data/clean_poli_dis_2012.csv', index=False)\n",
    "poli_dis_2020_df.to_csv('../data/clean_poli_dis_2020.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready to move on to notebook 3, exploratory data analysis! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}