FinalProject_group071.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COGS 108 - Final Project "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From venting about a broken heart, about a racist president, or about being sent to fight a war that you don’t agree with, music is easily one of the most common forms of expression and catharsis. Given music’s power to express one’s perspective, it’s no surprise that the music surrounding the Vietnam War era is iconic for the music (amongst other media) that arose from that time. \n",
    "\tWhen President Donald Trump was elected into office, many people and music forums were discussing how artists would be inspired by the president’s election, suggesting that countless protest songs and political anthems would arise from this time period. Despite this suggestion, the vast majority of popular music seems to be apolitical.\n",
    "\tHas music become less political or have people stopped listening to political music? Does our nostalgia overemphasize the music of the Vietnam War era? Our goal is to expose the indirect influences of major events like the Vietnam War on music and to make connects (if any) between politics and music. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names\n",
    "\n",
    "- Albert Putra Purnama\n",
    "- Austin Moss-Ennis\n",
    "- Chandler Ennis\n",
    "- Xirui He"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Compared to the Vietnam War’s influence on popular music, how did Donald Trump’s election win and campaign platform influence popular music in America?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background and Prior Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;From venting about a broken heart, about a racist president, or about being sent to fight a war that you don’t agree with, music is easily one of the most common forms of expression and catharsis. Given music’s power to express one’s perspective, it’s no surprise that the music surrounding the Vietnam War era is iconic for the music (amongst other media) that arose from that time. <br>\n",
    "&emsp;&emsp;When President Donald Trump was elected into office, many people and music forums were discussing how artists would be inspired by the president’s election, suggesting that countless protest songs and political anthems would arise from this time period. Despite this suggestion, the vast majority of popular music seems to be apolitical. <br>\n",
    "&emsp;&emsp;Has music become less political or have people stopped listening to political music? Does our nostalgia overemphasize the music of the Vietnam War era? Our goal is to expose the indirect influences of major events like the Vietnam War on music and to make connects (if any) between politics and music. \n",
    "\n",
    "References (include links):\n",
    "- [What has America been singing about?](https://journals.sagepub.com/doi/full/10.1177/0305735617748205)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of popular political/protest songs during the Vietnam War will be drastically higher than the number of political songs nowadays because youth participation in politics has been much lower than it was during the Vietnam War."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset 1**\n",
    "- Dataset Name: Kaggle 380000 Metro Lyrics\n",
    "- Link to the dataset: https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics\n",
    "- Number of observations: 380000+\n",
    "- Description: This dataset is the initial simple modeling on a clean data provided by kaggle. The data is in csv format, ready to be analysed by simple statistical model.\n",
    "\n",
    "**Dataset 2**\n",
    "- Dataset Name: Top Songs from 1958 to 2019\n",
    "- Link to the dataset: https://www.billboard.com\n",
    "- Number of observations: \n",
    "- Description: This dataset is taken by scraping the information from billboard.com. Further explanation in the data cleaning section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "Please make sure you are using `python3` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "File b'Clean_Country_Data.csv' does not exist",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-3705446494de>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# because it takes ~5s to load the data on a Macbook Pro.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m# imagine if you load it in inferior computers.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mcountry_songs_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Clean_Country_Data.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mhot_songs_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Clean_Hot_100_Chart.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mcountry_songs_data\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'year'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcountry_songs_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Last Week Charted'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'-'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/site-packages/pandas-0.23.4-py3.7-macosx-10.13-x86_64.egg/pandas/io/parsers.py\u001b[0m in \u001b[0;36mparser_f\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    676\u001b[0m                     skip_blank_lines=skip_blank_lines)\n\u001b[1;32m    677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 678\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    679\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    680\u001b[0m     \u001b[0mparser_f\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/site-packages/pandas-0.23.4-py3.7-macosx-10.13-x86_64.egg/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    438\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    439\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 440\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    441\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    442\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/site-packages/pandas-0.23.4-py3.7-macosx-10.13-x86_64.egg/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    785\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'has_index_names'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'has_index_names'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    786\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 787\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    788\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    789\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/site-packages/pandas-0.23.4-py3.7-macosx-10.13-x86_64.egg/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1012\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'c'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1013\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'c'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1014\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1015\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1016\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'python'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.7/site-packages/pandas-0.23.4-py3.7-macosx-10.13-x86_64.egg/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1706\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'usecols'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1707\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1708\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1709\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1710\u001b[0m         \u001b[0mpassed_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnames\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: File b'Clean_Country_Data.csv' does not exist"
     ]
    }
   ],
   "source": [
    "# Read data to a variable, this cell need to be separated\n",
    "# because it takes ~5s to load the data on a Macbook Pro.\n",
    "# imagine if you load it in inferior computers.\n",
    "country_songs_data = pd.read_csv('Clean_Country_Data.csv')\n",
    "hot_songs_data = pd.read_csv('Clean_Hot_100_Chart.csv')\n",
    "country_songs_data['year'] = country_songs_data.apply (lambda row: int(row['Last Week Charted'].split('-')[0]), axis=1)\n",
    "hot_songs_data['year'] = country_songs_data.apply (lambda row: int(row['Last Week Charted'].split('-')[0]), axis=1)\n",
    "#hip_songs_data['year'] = country_songs_data.apply (lambda row: int(row['Last Week Charted'].split('-')[0]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure that the data is large enough\n",
    "assert len(country_songs_data) > 1000, \"Country data not large enough\"\n",
    "assert len(hot_songs_data) > 1000, \"Hot 100 data not large enough\"\n",
    "#assert len(hip_songs_data) > 1000, \"Hip Hop data not large enough\"\n",
    "\n",
    "# visually check the if the data is parsed correctly\n",
    "display(country_songs_data.head())\n",
    "display(hot_songs_data.head())\n",
    "#display(hip_songs_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Explnations on the lyrics data** <br>\n",
    "\n",
    "| Column name | Description |\n",
    "|-------------|-------------|\n",
    "| Artist | The artist, singer, or creator of the song |\n",
    "| Song | The title of the song |\n",
    "| Date | The date the song got released |\n",
    "| Current Rank | The rank of this song on the billboard during the release |\n",
    "| Last Weeks Position | The rank of this song on the billboard last week |\n",
    "| Weeks on Chart | The number of weeks the song is in the billboard top 100 |\n",
    "| Peak Position | The highest position of the song ever achieved in the chart |\n",
    "\n",
    "**Explanations on the song data** <br>\n",
    "\n",
    "| Column name | Description |\n",
    "|-------------|-------------|\n",
    "| Artist | The artist, singer, or creator of the song |\n",
    "| Song | The title of the song |\n",
    "| Date | The date the song got released |\n",
    "| Years on Chart | The range of years the song is in the billboard top 100 |\n",
    "| Weeks on Chart | The number of weeks the song is in the billboard top 100 |\n",
    "| Peak Position | The highest position of the song ever achieved in the chart |\n",
    "| Political Value | A numerical representation of how political the song is <br>as determined by our sentiment analysis program |\n",
    "\n",
    "This marks the end of input data. By this point, all inputs should have been loaded<br>\n",
    " into notebook's memory. Next thing we have to do is define some constants. <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Declare constants\n",
    "CONST_RAW_DATA_USED = 0.5 # Percentage of input data we want to use\n",
    "                    # 0.5 means 50% of the entire set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Get billboard data from 1960s to date.\n",
    "2. Get lyrics from billboard songs (1960s to date) from genius.com\n",
    "3. Get popular protests, war, or politics related songs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE\n",
    "## FEEL FREE TO ADD MULTIPLE CELLS PER SECTION\n",
    "country_songs_data = country_songs_data.dropna()\n",
    "hot_songs_data = hot_songs_data.dropna()\n",
    "#hip_songs_data = hip_songs_data.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_lyrics(lyrics):\n",
    "    #Normalize case sensitive names before lowercasing them\n",
    "    lyrics = lyrics.replace(\"Johnson\", \"president\")\n",
    "\n",
    "    lyrics = lyrics.lower()\n",
    "    lyrics = lyrics.strip()\n",
    "    lyrics = lyrics.replace(\"can't\", \"can not\")\n",
    "    lyrics = lyrics.replace(\"won't\", \"will not\")\n",
    "    lyrics = lyrics.replace(\"ain't\", \"aint\")\n",
    "    lyrics = lyrics.replace(\"n't\", \" not\")\n",
    "    lyrics = lyrics.replace(\"'ll\", \" will\")\n",
    "    lyrics = lyrics.replace(\"'re\", \" are\")\n",
    "    lyrics = lyrics.replace(\"'ve\", \" have\")\n",
    "    lyrics = lyrics.replace(\"'m\", \" am\")\n",
    "    lyrics = lyrics.replace(\"how'd\", \"how did\")\n",
    "    lyrics = lyrics.replace(\"'d\", \" would\")\n",
    "    lyrics = lyrics.replace(\"it's\", \"it is\")\n",
    "    lyrics = lyrics.replace(\"'til\", \"until\")\n",
    "    lyrics = lyrics.replace(\"'s\", \"s\")\n",
    "    lyrics = lyrics.replace(\"in'\", \"ing\")\n",
    "    lyrics = lyrics.replace(\"'cause\", \"because\")\n",
    "    lyrics = lyrics.replace(\"gon'\", \"going to\")\n",
    "    lyrics = lyrics.replace(\"gonna\", \"going to\")\n",
    "    lyrics = lyrics.replace(\"'bout\", \"about\")\n",
    "    lyrics = lyrics.replace(\"y'all\", \"you all\")\n",
    "    lyrics = lyrics.replace(\"tryna'\", \"trying to\")\n",
    "    lyrics = lyrics.replace(\"lil'\", \"little\")\n",
    "    lyrics = lyrics.replace(\"'em\", \"them\")\n",
    "    lyrics = lyrics.replace(\"'im\", \"him\")\n",
    "    lyrics = lyrics.replace(\"wanna'\", \"want to\")\n",
    "    lyrics = re.sub(\"[\\[].*?[\\]]\", \"\", lyrics)\n",
    "    lyrics = lyrics.translate(str.maketrans('', '', string.punctuation))\n",
    "    lyrics = lyrics.replace(\" ya \", \" you \")\n",
    "\n",
    "    # Normalizing terms for `gun`\n",
    "    lyrics = lyrics.replace(\"bullet\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"guns\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"aa-12\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ak-47\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ak-5\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ak-74\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ar-15\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"calico\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"caliber\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"desert eagle\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"draco\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"famas\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"five-seven\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"five seven\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"p90\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"glock\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"luger\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"m16\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"m1\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"m21\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"m4\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"m9\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"mac-10\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"mac-11\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"mac-12\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"master key\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ruger\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"blackhawk\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"sks\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"tec-9\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"uzi\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"colt\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"armalite\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"beretta\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"glock\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"heckler & koch\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"kel-tec\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"intratec\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"mossberg\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"ruger\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"sig\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"taurus\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"sinchester\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"smith & wesson\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"smith and wesson\", \"gun\")\n",
    "    lyrics = lyrics.replace(\"revolver\", \"gun\")\n",
    "\n",
    "    # Normalizing terms for `war`\n",
    "    lyrics = lyrics.replace(\"combat\", \"war\")\n",
    "    lyrics = lyrics.replace(\"fighting\", \"war\")\n",
    "    lyrics = lyrics.replace(\"fightin'\", \"war\")\n",
    "    lyrics = lyrics.replace(\"battlefield\", \"war\")\n",
    "    lyrics = lyrics.replace(\"battle\", \"war\")\n",
    "    lyrics = lyrics.replace(\"warfare\", \"war\")\n",
    "    lyrics = lyrics.replace(\"war-worn\", \"war\")\n",
    "    lyrics = lyrics.replace(\"war-torn\", \"war\")\n",
    "    lyrics = lyrics.replace(\"bloodshed\", \"war\")\n",
    "\n",
    "    # Normalizing terms for `death`\n",
    "    lyrics = lyrics.replace(\"die\", \"death\")\n",
    "    lyrics = lyrics.replace(\"dying\", \"death\")\n",
    "    lyrics = lyrics.replace(\"dead\", \"death\")\n",
    "    lyrics = lyrics.replace(\"kill\", \"death\")\n",
    "    lyrics = lyrics.replace(\"killed\", \"death\")\n",
    "\n",
    "    # Normalizing terms for `soldiers`\n",
    "    lyrics = lyrics.replace(\"soldier\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"private\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"trooper\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"troops\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"army\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"military\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"navy\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"veteran\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"veterans\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"vet\", \"soldiers\")\n",
    "    lyrics = lyrics.replace(\"vets\", \"soldiers\")\n",
    "\n",
    "    # Normalizing terms for `protest`\n",
    "    lyrics = lyrics.replace(\"protesting\", \"protest\")\n",
    "    lyrics = lyrics.replace(\"protests\", \"protest\")\n",
    "    lyrics = lyrics.replace(\"picket line\", \"protest\")\n",
    "    lyrics = lyrics.replace(\"picket sign\", \"protest\")\n",
    "\n",
    "    # Normalizing terms for `Vietnam`\n",
    "    lyrics = lyrics.replace(\"viet cong\", \"vietnam\")\n",
    "    lyrics = lyrics.replace(\" nam \", \"vietnam\")\n",
    "    lyrics = lyrics.replace(\"'nam\", \"vietnam\")\n",
    "    lyrics = lyrics.replace(\"vietnamese\", \"vietnam\")\n",
    "\n",
    "    # Normalizing terms for `president`\n",
    "    lyrics = lyrics.replace(\"chief\", \"president\")\n",
    "    lyrics = lyrics.replace(\"presidential\", \"president\")\n",
    "    lyrics = lyrics.replace(\"lyndon johnson\", \"president\")\n",
    "    lyrics = lyrics.replace(\"lbj\", \"president\")\n",
    "    lyrics = lyrics.replace(\"nixon\", \"president\")\n",
    "    lyrics = lyrics.replace(\"richard nixon\", \"president\")\n",
    "    lyrics = lyrics.replace(\"donald trump\", \"president\")\n",
    "\n",
    "    # Normalizing terms for `communism`\n",
    "    lyrics = lyrics.replace(\"communist\", \"communism\")\n",
    "    lyrics = lyrics.replace(\"commie\", \"communism\")\n",
    "\n",
    "    # Normalizing terms for `America`\n",
    "    lyrics = lyrics.replace(\"usa\", \"america\")\n",
    "    lyrics = lyrics.replace(\"united states\", \"america\")\n",
    "    lyrics = lyrics.replace(\"uncle sam\", \"america\")\n",
    "    lyrics = lyrics.replace(\"american\", \"america\")\n",
    "    lyrics = lyrics.replace(\"red white and blue\", \"america\")\n",
    "    lyrics = lyrics.replace(\"red white blue\", \"america\")\n",
    "\n",
    "    # Normalziing terms for `MAGA`\n",
    "    lyrics = lyrics.replace(\"make america great\", \"maga\")\n",
    "    lyrics = lyrics.replace(\"making america great\", \"maga\")\n",
    "\n",
    "    # Normalizing terms for `red hat`\n",
    "    lyrics = lyrics.replace(\"maga hat\", \"red hat\")\n",
    "    lyrics = lyrics.replace(\"red cap\", \"red hat\")\n",
    "    lyrics = lyrics.replace(\"racist hat\", \"red hat\")\n",
    "\n",
    "    # Normalizing terms for `the wall`\n",
    "    lyrics = lyrics.replace(\"walls\", \"the wall\")\n",
    "    lyrics = lyrics.replace(\"wall\", \"the wall\")\n",
    "\n",
    "    # Normalizing terms for `Mexico`\n",
    "    lyrics = lyrics.replace(\"mexican\", \"mexico\")\n",
    "    lyrics = lyrics.replace(\"mexicans\", \"mexico\")\n",
    "\n",
    "    # Normalizing terms for `racism`\n",
    "    lyrics = lyrics.replace(\"racist\", \"racism\")\n",
    "\n",
    "    # Normalizing terms for `fascism`\n",
    "    lyrics = lyrics.replace(\"fascist\", \"fascism\")\n",
    "    \n",
    "    #Removing special characters and numerics\n",
    "    lyrics = lyrics.replace('‘', '')\n",
    "    lyrics = lyrics.replace('’', '')\n",
    "    lyrics = lyrics.replace('…', ' ')\n",
    "    lyrics = lyrics.replace('\\r', ' ')\n",
    "    lyrics = lyrics.replace('\\n', ' ')\n",
    "    lyrics = lyrics.replace('\\xa0', ' ')\n",
    "    lyrics = lyrics.replace('0', '')\n",
    "    lyrics = lyrics.replace('1', '')\n",
    "    lyrics = lyrics.replace('2', '')\n",
    "    lyrics = lyrics.replace('3', '')\n",
    "    lyrics = lyrics.replace('4', '')\n",
    "    lyrics = lyrics.replace('5', '')\n",
    "    lyrics = lyrics.replace('6', '')\n",
    "    lyrics = lyrics.replace('7', '')\n",
    "    lyrics = lyrics.replace('8', '')\n",
    "    lyrics = lyrics.replace('9', '')\n",
    "    \n",
    "    #Removing instances of multiple spaces between words\n",
    "    for i in range(0, 10):\n",
    "        lyrics = lyrics.replace('  ', ' ')\n",
    "    \n",
    "    lyrics = lyrics.strip()\n",
    "    \n",
    "    return lyrics\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis & Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Get the frequently used words in popular protest, politics, or war songs\n",
    "2. Get the statistics of the data based on groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get protests songs\n",
    "\n",
    "We will use this later for the baseline of war related words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import nltk\n",
    "#from string import punctuation\n",
    "import operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "country_songs_data['Lyrics'] = country_songs_data.apply (lambda row: clean_lyrics(row['Lyrics']), axis=1)\n",
    "hot_songs_data['Lyrics'] = hot_songs_data.apply (lambda row: clean_lyrics(row['Lyrics']), axis=1)\n",
    "#hip_songs_data['Lyrics'] = hip_songs_data.apply (lambda row: clean_lyrics(row['Lyrics']), axis=1)\n",
    "display(country_songs_data.head())\n",
    "display(hot_songs_data.head())\n",
    "#hip_songs_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# freq = dict()\n",
    "# #print (pol_df['Lyrics'])\n",
    "# for lyrics in pol_df['lyrics']:\n",
    "#     lyrics = [w for w in lyrics.split()]\n",
    "#     for word in lyrics:\n",
    "#         if word in freq.keys():\n",
    "#             freq[word] += 1\n",
    "#         else:\n",
    "#             freq[word] = 1\n",
    "\n",
    "# sorted_freq = sorted(freq.items(), key=operator.itemgetter(1))\n",
    "# print(sorted_freq[-40:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analytics\n",
    "\n",
    "- Group by year\n",
    "- Group by genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "war_related_words = [\n",
    "    'fight', 'stand', 'war', 'hate', 'president', 'propaganda', 'gun', 'death', \n",
    "    'protest', 'soldiers', 'vietnam', 'communism', 'america'\n",
    "]\n",
    "race_related_words = [\n",
    "    'city', 'stand', 'man', 'black', 'hate', 'politics', 'gun', 'death', 'racism', 'mexico'\n",
    "]\n",
    "election_related_words = [\n",
    "    'politics', 'president', 'propaganda', 'protest', 'america'\n",
    "]\n",
    "trump_related_words = [\n",
    "    'trump', ' maga ', 'red hat', 'the wall', 'fascicm'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_group_by_year = country_songs_data.groupby('year')\n",
    "hot_group_by_year = hot_songs_data.groupby('year')\n",
    "#hip_group_by_year = hip_songs_data.groupby('year')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getYearlyValues(songs_group_by_year):\n",
    "    thres = 4\n",
    "    y_r_w_s = dict()\n",
    "    y_rat_w_s = dict()\n",
    "    y_s_w_t = dict()\n",
    "    y_r_r_s = dict()\n",
    "    y_rat_r_s = dict()\n",
    "    y_s_r_t = dict()\n",
    "    y_r_e_s = dict()\n",
    "    y_rat_e_s = dict()\n",
    "    y_s_e_t = dict()\n",
    "    y_r_t_s = dict()\n",
    "    y_rat_t_s = dict()\n",
    "    y_s_t_t = dict()\n",
    "\n",
    "    for year, group in songs_group_by_year:\n",
    "        war_matrix = [[w in l for w in war_related_words] for l in group['Lyrics']]\n",
    "        race_matrix = [[w in l for w in race_related_words] for l in group['Lyrics']]\n",
    "        elec_matrix = [[w in l for w in election_related_words] for l in group['Lyrics']]\n",
    "        war_related_sum = sum([any(m) for m in war_matrix])\n",
    "        war_related_sum_t = sum([1 if sum(m) > thres else 0 for m in war_matrix])\n",
    "        race_related_sum = sum([any(m) for m in race_matrix])\n",
    "        race_related_sum_t = sum([1 if sum(m) > thres else 0 for m in race_matrix])\n",
    "        elec_related_sum = sum([any(m) for m in elec_matrix])\n",
    "        elec_related_sum_t = sum([1 if sum(m) > thres else 0 for m in elec_matrix])\n",
    "        if year > 2014:\n",
    "            trump_matrix = [[w in l for w in trump_related_words] for l in group['Lyrics']]\n",
    "            trump_related_sum = sum([any(m) for m in trump_matrix])\n",
    "            trump_related_sum_t = sum([1 if sum(m) > 1 else 0 for m in trump_matrix])\n",
    "        else:\n",
    "            trump_related_sum = 0\n",
    "            trump_related_sum_t = 0\n",
    "        if year < 1000:\n",
    "            continue\n",
    "        y_r_w_s[year] = war_related_sum\n",
    "        y_rat_w_s[year] = war_related_sum / len(group)\n",
    "        y_s_w_t[year] = war_related_sum_t\n",
    "        y_r_r_s[year] = race_related_sum\n",
    "        y_rat_r_s[year] = race_related_sum / len(group)\n",
    "        y_s_r_t[year] = race_related_sum_t\n",
    "        y_r_e_s[year] = elec_related_sum\n",
    "        y_rat_e_s[year] = elec_related_sum / len(group)\n",
    "        y_s_e_t[year] = elec_related_sum_t\n",
    "        y_r_t_s[year] = trump_related_sum\n",
    "        y_rat_t_s[year] = trump_related_sum / len(group)\n",
    "        y_s_t_t[year] = trump_related_sum_t\n",
    "    return [y_r_w_s, y_rat_w_s, y_s_w_t, y_r_r_s, y_rat_r_s, y_s_r_t, y_r_e_s, y_rat_e_s, y_s_e_t, y_r_t_s, y_rat_t_s, y_s_t_t]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_vals_by_year = getYearlyValues(country_group_by_year)\n",
    "hot_vals_by_year = getYearlyValues(hot_group_by_year)\n",
    "#hip_vals_by_year = getYearlyValues(hip_songs_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[0].keys()), list(country_vals_by_year[0].values()))\n",
    "\n",
    "fig.suptitle('Sum of war related country songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[1].keys()), list(country_vals_by_year[1].values()))\n",
    "fig.suptitle('Ratio of war related country songs to country songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[2].keys()), list(country_vals_by_year[2].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[3].keys()), list(country_vals_by_year[3].values()))\n",
    "fig.suptitle('Sum of race related country songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[4].keys()), list(country_vals_by_year[4].values()))\n",
    "fig.suptitle('Ratio of race related country songs to country songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[5].keys()), list(country_vals_by_year[5].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[6].keys()), list(country_vals_by_year[6].values()))\n",
    "fig.suptitle('Sum of election related country songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[7].keys()), list(country_vals_by_year[7].values()))\n",
    "fig.suptitle('Ratio of election related country songs to country songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[8].keys()), list(country_vals_by_year[8].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[9].keys()), list(country_vals_by_year[9].values()))\n",
    "fig.suptitle('Sum of Trump related country songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[10].keys()), list(country_vals_by_year[10].values()))\n",
    "fig.suptitle('Ratio of Trump related country songs to country songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(country_vals_by_year[11]), list(country_vals_by_year[11].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[0].keys()), list(hot_vals_by_year[0].values()))\n",
    "\n",
    "fig.suptitle('Sum of war related Hot 100 songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[1].keys()), list(hot_vals_by_year[1].values()))\n",
    "fig.suptitle('Ratio of war related Hot 100 songs to Hot 100 songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[2].keys()), list(hot_vals_by_year[2].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('War related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[3].keys()), list(hot_vals_by_year[3].values()))\n",
    "fig.suptitle('Sum of race related Hot 100 songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[4].keys()), list(hot_vals_by_year[4].values()))\n",
    "fig.suptitle('Ratio of race related Hot 100 songs to Hot 100 songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[5].keys()), list(hot_vals_by_year[5].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Race related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[6].keys()), list(hot_vals_by_year[6].values()))\n",
    "fig.suptitle('Sum of election related Hot 100 songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[7].keys()), list(hot_vals_by_year[7].values()))\n",
    "fig.suptitle('Ratio of election related Hot 100 songs to Hot 100songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[8].keys()), list(hot_vals_by_year[8].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Election related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[9].keys()), list(hot_vals_by_year[9].values()))\n",
    "fig.suptitle('Sum of Trump related Hot 100 songs over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[10].keys()), list(hot_vals_by_year[10].values()))\n",
    "fig.suptitle('Ratio of Trump related Hot 100 songs and songs produced over the years', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related / total songs', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "plt.xlim(1950,2020)\n",
    "plt.bar(list(hot_vals_by_year[11]), list(hot_vals_by_year[11].values()))\n",
    "fig.suptitle('With threshold', fontsize=20)\n",
    "plt.xlabel('Year', fontsize=18)\n",
    "plt.ylabel('Trump related', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Year based analysis\n",
    "\n",
    "The initial idea of grouping based on year is to see how does politics impact the songs released that year.  \n",
    "The first figure shows that there are significant amount of war related songs in the year 2006. In 2006,  \n",
    "American people realize that war in Iraq was about getting Iraq's earth resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2\n",
    "\n",
    "In this data we will use different data set to analyze. The main purpose of the following part is to analyze   curated data instead of some provided data from kaggle. The reasons why we want to curate the data ourselves are two-fold:\n",
    "1. The data provided by kaggle user that is being used above also does not go beyond 2016, in this new dataset, we include songs up to March 2019.\n",
    "2. The songs in the kaggle dataset may not be popular songs. However, the data we scraped are from top 100 charts on billboard (popular).\n",
    "\n",
    "---\n",
    "## Data retrieval\n",
    "\n",
    "The following is the code that we used to get the data from [billboard](https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics) and [genius](https://genius.com/)\n",
    "\n",
    "> Please note that the following code is not the complete version of the code. Please refer to the github repo [here](https://github.com/albertputrapurnama/FinalProject108) for the complete python script used for actual dataset retrieval.\n",
    "\n",
    "We will guide you briefly on how it works\n",
    "\n",
    "```python\n",
    "    # filename: DataWrangler_toCSV.py\n",
    "    from datetime import date, datetime, timedelta\n",
    "    import csv\n",
    "    import inquirer\n",
    "    from bs4 import BeautifulSoup\n",
    "    import requests\n",
    "    import time\n",
    "    import urllib2\n",
    "    import json\n",
    "    import re\n",
    "    import unicodedata\n",
    "\n",
    "    lyric_dict = dict()\n",
    "    nan = 0\n",
    "\n",
    "    charts = {'Hot 100': [\"https://www.billboard.com/charts/hot-100/\", 1958, 8, 4],\n",
    "              'Pop Songs': [\"https://www.billboard.com/charts/pop-songs/\", 1992, 10, 3],\n",
    "              'Rock Songs': [\"https://www.billboard.com/charts/rock-songs/\", 2009, 6, 20],\n",
    "              'Hip-Hop/R&B Songs': [\"https://www.billboard.com/charts/r-b-hip-hop-songs/\", 1958, 10, 20],\n",
    "              'Country Songs': [\"https://www.billboard.com/charts/country-songs/\", 1958, 10, 20]\n",
    "              }\n",
    "```\n",
    "\n",
    "*The above code is all about defining our constants and importing libraries.*\n",
    "\n",
    "---\n",
    "```python\n",
    "    \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "    # filename: DataWrangler_toCSV.py\n",
    "    def info(s, dt):\n",
    "        artist = s.get(\"data-artist\").encode('ascii', 'ignore').decode('ascii')\n",
    "        song = s.get(\"data-title\").encode('ascii', 'ignore').decode('ascii')\n",
    "        rank = s.get(\"data-rank\")\n",
    "        lyrics = get_lyrics(song, artist).encode('ascii', 'ignore').decode('ascii')\n",
    "        if s.find(class_=\"chart-list-item__last-week\") is None:\n",
    "            last = '0'\n",
    "        else:\n",
    "            last = s.find(class_=\"chart-list-item__last-week\").contents[0]\n",
    "        if s.find(class_=\"chart-list-item__weeks-at-one\") is None:\n",
    "            peak = '0'\n",
    "        else:\n",
    "            peak = s.find(class_=\"chart-list-item__weeks-at-one\").contents[0]\n",
    "        if s.find(class_=\"chart-list-item__weeks-on-chart\") is None:\n",
    "            weeks = '0'\n",
    "        else:\n",
    "            weeks = s.find(class_=\"chart-list-item__weeks-on-chart\").contents[0]\n",
    "        return [artist, song, dt.isoformat(), rank, last, peak, weeks, lyrics]\n",
    "```\n",
    "\n",
    "`info` function is a helper function used to scrape the data we want from the page passed into it\n",
    "\n",
    "`info` function takes in 2 params:\n",
    "1. **s** — which is a soup page. soup page is a page given by BeautifulSoup, more documentation on BeautifulSoup [here](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)\n",
    "2. **dt** — the date of the chart page passed in.\n",
    "\n",
    "`info` returns an array of scraped data in the desired format for further analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "    def scrape_site(writer, url, chart_date):\n",
    "        global nan\n",
    "        start_time = time.time()\n",
    "        new_url = url + chart_date.isoformat()\n",
    "        req = requests.get(new_url)\n",
    "        soup = BeautifulSoup(req.text, \"html.parser\")\n",
    "        songs = soup.find_all(class_=\"chart-list-item\")\n",
    "        hundred_list = [info(x, chart_date) for x in songs]\n",
    "        print(\"Time taken: \" + str(time.time() - start_time))\n",
    "        print(\"NaNs = \" + str(nan))\n",
    "        writer.writerows(hundred_list)\n",
    "```\n",
    "`scrape` site will take use `info` function to scrape data out of a given url.\n",
    "\n",
    "`scrape_site` function takes in 3 params:\n",
    "1. **writer** — The object or entity which writes the scraped data to a dataframe.\n",
    "2. **url** — The URL of the page soon to be scraped\n",
    "3. **chart_date** — The publishing date of the chart."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "    def get_lyrics(song, artist):\n",
    "        global nan\n",
    "        global lyric_dict\n",
    "        # Makes a URL that the genius API is capable of reading/requesting\n",
    "        genius_url = \"https://api.genius.com/search?q=\"\n",
    "\n",
    "        ...\n",
    "        More codes here\n",
    "        ...\n",
    "        \n",
    "            lyrics = get_lyrics_from_url(lyric_url)\n",
    "            lyric_dict[url] = lyrics\n",
    "            return lyrics\n",
    "```\n",
    "\n",
    "`get_lyrics` is a huge function that takes in song title and artists and spit out a nice well-formatted lyrics.\n",
    "\n",
    "There are many other helper function that come into play here, but for the sake of simplifying the report, we will not be covering those functions here. Feel free to download the complete python script on the github repository [here](https://github.com/albertputrapurnama/FinalProject108)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "The following code block is the main code\n",
    "\n",
    "```python\n",
    "    # filename: DataWrangler_toCSV.py\n",
    "    billboard_url, base_year, base_month, base_day = inquire()\n",
    "\n",
    "    curr_date = date(base_year, base_month, base_day)\n",
    "    end_date = last_day()\n",
    "    # Ask user for input to name the spreadsheet.\n",
    "    sheetTitle = raw_input(\"Enter spreadsheet title: \").replace(\" \", \"_\")\n",
    "    sheetTitle = sheetTitle + \".csv\"\n",
    "    with open(sheetTitle, mode='w') as billboard_chart:\n",
    "        charter = csv.writer(billboard_chart, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL)\n",
    "        charter.writerow(\n",
    "            [\"Artist\", \"Song\", \"Date\", \"Current Rank\", \"Last Weeks Position\", \"Weeks on Chart\", \"Peak Position\"])\n",
    "        #************Section 2**********************\n",
    "        while curr_date != end_date:\n",
    "            scrape_site(charter, billboard_url, curr_date)\n",
    "            print(\"Finished charting the week of: \" + curr_date.isoformat())\n",
    "            curr_date = date_increment(curr_date)\n",
    "        #*******************************************\n",
    "```\n",
    "\n",
    "This is where we put all the function definitions to work. We will upen up a file to write. Then, we will run the scraper on the dates we specified. This is shown specifically on code marked as **Section 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ethics & Privacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;Since we are going to focus our attention to music and politics, most of our data will be available publicly and less likely to have any privacy issues.<br>\n",
    "&emsp;&emsp;Most of our ethical considerations revolves around the community and political figures itself. We live in an era where social construct is a huge thing we cannot ignore. This includes race, gender, and sexual orientations. We have to be careful about the possible race discrimination that may be revealed in the data that we collected. One way to mitigate this is by evaluating the data source and look for race discrimination before using the data. This method can also be used to mitigate any gender discrimination. For sexual orientation, the data is not given from the dataset, although we can find the sexual orientation of the musician, and that is a very complicated and exhaustive work to find sexual orientation of an artist. Furthermore, there are thousands of songs with thousands of different writers and musicians.<br>\n",
    "&emsp;&emsp;Since we analyze artist’s response toward political activity around them, we have to be careful not proposing conclusions that may endanger the artist’s career or life. Likewise, political figures should also be protected. This can be solved by anonymizing any entity (artists or public figures) that we will analyze. This does two-fold: protecting artist’s and public figure’s identity, and push the research to a more generalized political and musical scope.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion & Discussion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Fill in your discussion information here*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}