FinalProject_group058.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COGS 108 - Final Project "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People now intensively communicate on the social platform such as twitter, and it becomes the one of the most common ways for people to share, gather, and get influenced by information. We as a team wanted to explore how much different media may related to their follower's sentiment on twitter. We used text mining, AB test, and classification to carry out this project.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Names\n",
    "\n",
    "- Zirui Wang\n",
    "- Yang Kuang\n",
    "- Hengyu Liu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Research Question"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To what extent are mainstream medias' impact on their recent followers' speech on Twitter? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background and Prior Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### References (include links):\n",
    "- Garrett RK (2019) Social media’s contribution to political misperceptions in U.S. Presidential elections. PLoS ONE 14(3): e0213500. https://urldefense.com/v3/__https://doi.org/10.1371/journal.pone.0213500*5Cn__;JQ!!Mih3wA!QlQUFTK24Z4yWuHSjpnaVlfV8_Os401B_7fRaOqps2CZOe77QdZOeES4PhXe9ZE$ - Tajvidi, R., &amp; Karami, A. (2017, September 20). The effect of social media on firm performance. Retrieved October 24, 2020, from https://urldefense.com/v3/__https://www.sciencedirect.com/science/article/abs/pii/S0747563217305514?via=ihub*5Cn__;JQ!!Mih3wA!QlQUFTK24Z4yWuHSjpnaVlfV8_Os401B_7fRaOqps2CZOe77QdZOeES4Gfj2K2Q$ \n",
    "#### Background\n",
    "- 1) People now are heavily rely on the technology and it becomes the one of the most common ways for people to gather information. However, there are so many different media sources. Different media sources have different stances and different ways to express or describe a contentious event. If one person only has one source of information, he or she very likely will believe that is the case. Later on, if he or she keeps receiving information only from this source, he or she starts to stay in the comfort zone/echo chambor. It becomes harder and harder for he/she to hear so many other voices addressed in other sources. Then, these people's views may closely related to the his/her main media source's view.\n",
    "- 2) This topic becomes interesting to me because people often consider themselves as objective. However, their views may depend on their sources of information. Misinformation leads a lot of people to judge a contentious event from a narrow perspective.\n",
    "- 3) This is important because this is a way that a nation can utilize its media to manipulate people's views/stances. Also, this is what happening in the world.\n",
    "- 4) For real world example, media have great influence on U.S. Presidential elections. Certainly to some degree media becoming tools of political propaganda, but how effective they really are, and does this pattern also exist in other fields?\n",
    "- 5) Our proposed research question is - To what extent are people's stances on contentious events on social media platforms related to their main media resources? We planned to randomly select 10K Twitter users and determine their main media source from its following list. However, it becomes so time consuming that it is impossible to access every user's following and follwer list. So, we changed our stretegy and tried to start with 10 mainstream media. As for each media, we collect its 1k recent followers. As for each follower, we collect its 200 recent tweets. In this way, we will ideally have 200K (200 * 1K) tweets for each media. Since there are cases like someone's tweets are unable to access or someone doesn't have at least 200 tweets, we didn't obtain 200K tweets for each media. They are roughly around 180K for each media. Now we have 10 groups of Twitter users' tweets according to each media. Going forward with our project, we will contact three different analyses on these tweets to see the impact of media on people's speech. Since our strategy changed a little bit, we rephrased our research question - To what extent are mainstream medias' impact on their recent followers' speech on Twitter? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hypothesis\n",
    "We expect that some media will significantly influence its recent follower's views/stances/speech in a way that the users who followed different media will have different measurement of sentimental expression.\n",
    "\n",
    "#### Explanation\n",
    "There are so many media sources now. People usually choose their main media sources based on their preference in the beginning by following some media twitter accounts or subscribe to some newspaper. Later on, people tend to stay in their comfort zone/echo chamber and see posts/news which expressed in a way that they accepted/liked. When a controversial event happens, there may be various kinds of perspectives expressed by different media sources. Even there are cases like some specifc media are censoring specific topics they don't like. If people highly rely on their main media sources, it is possible that they only view that event through only the perspective of their main media source. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import math\n",
    "import json\n",
    "import string\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import string\n",
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "from collections import defaultdict\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction import  text\n",
    "from scipy.sparse import hstack, csr_matrix\n",
    "import tweepy as tw\n",
    "import datetime\n",
    "import json\n",
    "import pickle\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.feature_extraction import  text\n",
    "from sklearn.linear_model import LogisticRegression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Fill in your dataset information here*\n",
    "\n",
    "(Copy this information for each dataset)\n",
    "- Dataset Name:\n",
    "- Link to the dataset:\n",
    "- Number of observations:\n",
    "\n",
    "1-2 sentences describing each dataset. \n",
    "\n",
    "If you plan to use multiple datasets, add 1-2 sentences about how you plan to combine these datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Since these datasets are saved after pulling from the twitter api and too large to upload to Github, please download them here:* https://drive.google.com/file/d/1b94bDMC8Og_tUrG7AOOA4PqmJaEVqK1z/view?usp=sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset 1\n",
    "- Dataset Name: Media list\n",
    "- Link to the dataset: media_ten\n",
    "- Number of observations: 10\n",
    "- Description: These are ten media that we started with. It contains the name of the media and their Twitter ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>news_media_twitter_id</th>\n",
       "      <th>news_media_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>nytimes</td>\n",
       "      <td>The New York Times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CNN</td>\n",
       "      <td>CNN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>WSJ</td>\n",
       "      <td>The Wall Street Journal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>washingtonpost</td>\n",
       "      <td>The Washington Post</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>EpochTimes</td>\n",
       "      <td>The Epoch Times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>OANN</td>\n",
       "      <td>One America News</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>news_ntd</td>\n",
       "      <td>NTD News</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>FoxNews</td>\n",
       "      <td>Fox News</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>newsmax</td>\n",
       "      <td>Newsmax</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>nypost</td>\n",
       "      <td>New York Post</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  news_media_twitter_id          news_media_name\n",
       "0               nytimes       The New York Times\n",
       "1                   CNN                      CNN\n",
       "2                   WSJ  The Wall Street Journal\n",
       "3        washingtonpost      The Washington Post\n",
       "4            EpochTimes          The Epoch Times\n",
       "5                  OANN         One America News\n",
       "6              news_ntd                 NTD News\n",
       "7               FoxNews                 Fox News\n",
       "8               newsmax                  Newsmax\n",
       "9                nypost            New York Post"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "media = pd.read_csv('new_media.csv')\n",
    "media_ten = media[:10]\n",
    "media_ten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset 2  \n",
    "- Dataset Name: Recent 1000 followers for each media\n",
    "- Link to the dataset: media_followers\n",
    "- Number of observations: 10K (10 media * 1K followers)\n",
    "- Description: This dataset contains 1K users' information for each media. Obtained previously and saved as \"user_101000.txt\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saved data file for \"media_followers\"\n",
    "with open(\"user_101000.txt\", \"r\") as fp:\n",
    "    media_followers = json.load(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of followers: 1000\t Media: nytimes\n",
      "Number of followers: 1000\t Media: CNN\n",
      "Number of followers: 1000\t Media: WSJ\n",
      "Number of followers: 1000\t Media: washingtonpost\n",
      "Number of followers: 1000\t Media: EpochTimes\n",
      "Number of followers: 1000\t Media: OANN\n",
      "Number of followers: 1000\t Media: news_ntd\n",
      "Number of followers: 1000\t Media: FoxNews\n",
      "Number of followers: 1000\t Media: newsmax\n",
      "Number of followers: 1000\t Media: nypost\n"
     ]
    }
   ],
   "source": [
    "for i in media_followers:\n",
    "    print(\"Number of followers: \" + str(len(media_followers[i])) + \"\\t Media: \" + i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset 3 \n",
    "- Dataset Name: Recent 200 tweets for each follower for each media\n",
    "- Link to the dataset: final\n",
    "- Number of observations: 1723916\n",
    "- Description: This dataset contains most recent 200 tweets for each follower in each media group. Ideally we will get 200 tweets for each follower, but some followers don't have at least 200 tweets posted. Noticed that OANN(One America News) has dramatically lower tweets that being collectable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saved data file for \"final\"\n",
    "with open(\"final1.txt\", \"r\") as fp:\n",
    "    final = json.load(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of tweets in total: 169000 \t Media: nytimes\n",
      "Number of tweets in total: 196000 \t Media: CNN\n",
      "Number of tweets in total: 197000 \t Media: WSJ\n",
      "Number of tweets in total: 194000 \t Media: washingtonpost\n",
      "Number of tweets in total: 108000 \t Media: EpochTimes\n",
      "Number of tweets in total: 83916 \t Media: OANN\n",
      "Number of tweets in total: 186000 \t Media: news_ntd\n",
      "Number of tweets in total: 194000 \t Media: FoxNews\n",
      "Number of tweets in total: 197000 \t Media: newsmax\n",
      "Number of tweets in total: 199000 \t Media: nypost\n",
      "Number of observations: 1723916\n"
     ]
    }
   ],
   "source": [
    "#collected tweets from each media\n",
    "sum = 0\n",
    "for i in final:\n",
    "    print(\"Number of tweets in total: \" + str(len(final[i])) + \" \\t Media: \" + i)\n",
    "    sum += len(final[i])\n",
    "    \n",
    "print(\"Number of observations: \" + str(sum))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Cleaning/Processing (10%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Describe your data cleaning steps here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read json saved data in to dataframe\n",
    "def read_data(data):\n",
    "    with open(data) as f:\n",
    "        dict_1 = json.load(f)\n",
    "    temp=[]\n",
    "    for i in dict_1.keys():\n",
    "        df_new=pd.DataFrame(dict_1[i],columns=['Post'])\n",
    "        df_new['Media']=i\n",
    "        temp.append(df_new)\n",
    "    return pd.concat(temp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#clean data, remove https, /n.\n",
    "def clean_data(df):\n",
    "    df['Post']=df['Post'].str.replace('\\n','')\n",
    "    df['Post']=df['Post'].apply(lambda x: x+' ')\n",
    "    df['Post']=df['Post'].apply(lambda x: re.sub(r'http[s]?://.* ','',x))\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Read data\n",
    "Originally data is in json format. The json dictionary has 10 keys, each of them is one media. Each key correspondes to a list where it contains all the tweets of that media. For every media, we have a dataframe. We merge all the dataframes together. In the end, we return the output. ('final1.txt' is larger than 100MB. We cannot upload it to GitHub.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Post</th>\n",
       "      <th>Media</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>President-elect Joe Biden will nominate Pete B...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>President-elect Joe Biden intends to name Gina...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Japanese prosecutors charged a man with arson ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A man was sentenced to four weeks in jail for ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sweden has been an outlier in the global pande...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Here's a look at some of the highlights worth ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Tom Cruise erupted at crew members on the set ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Delivery workers in South Korea say they feel ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Lawmakers in Mexico approved legislation Tuesd...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>MacKenzie Scott announced on Tuesday that she ...</td>\n",
       "      <td>nytimes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Post    Media\n",
       "0  President-elect Joe Biden will nominate Pete B...  nytimes\n",
       "1  President-elect Joe Biden intends to name Gina...  nytimes\n",
       "2  Japanese prosecutors charged a man with arson ...  nytimes\n",
       "3  A man was sentenced to four weeks in jail for ...  nytimes\n",
       "4  Sweden has been an outlier in the global pande...  nytimes\n",
       "5  Here's a look at some of the highlights worth ...  nytimes\n",
       "6  Tom Cruise erupted at crew members on the set ...  nytimes\n",
       "7  Delivery workers in South Korea say they feel ...  nytimes\n",
       "8  Lawmakers in Mexico approved legislation Tuesd...  nytimes\n",
       "9  MacKenzie Scott announced on Tuesday that she ...  nytimes"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=read_data('final1.txt')\n",
    "df[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Clean data\n",
    "Firstly, we removed new line characters. Then, we removed the hyperlinks. At the end of the post, some of them has a space, but some of them doesn't. So we added a space at the end of each post so that we make sure every post has at least a space at the end of the post. In this way, it's easier to remove the hyperlinks which are usually placed by the followers in the end of the post by using regex."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=clean_data(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3\n",
    "Sample 100000 tweets from each media's followers. Use all if less than 100000 tweets available. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(1)\n",
    "medias = list(df['Media'].unique())\n",
    "results = [0]*10\n",
    "for i in range(10):\n",
    "    media = medias[i]\n",
    "    temp = df[df['Media']==media]\n",
    "    results[i] = temp.sample(n=min(100000, temp.shape[0]), random_state=1, replace=False)\n",
    "df = pd.concat(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4: Text Cleasing/Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenalize\n",
    "df['Token'] = df['Post'].apply(word_tokenize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove stop words and punctuation\n",
    "punctuation = set(string.punctuation)\n",
    "def remove_punt(l):\n",
    "    return [i for i in l if not i in punctuation]\n",
    "punctuation.update(['``', '’', \"''\", '”', '—', '“', \"'s\"])\n",
    "\n",
    "stop_words = set(stopwords.words('english'))\n",
    "def remove_sw(l):\n",
    "    return [c for c in l if not c.lower() in punctuation and not c.lower() in stop_words]\n",
    "df['Stop'] = df['Token'].apply(remove_sw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PorterStemmer\n",
    "from nltk.stem import PorterStemmer\n",
    "ps = PorterStemmer()\n",
    "df['Stem'] = df['Stop'].apply(lambda x: [ps.stem(y) for y in x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 5: Text Sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sentimental analysis\n",
    "tweets = defaultdict(pd.Series)\n",
    "for i in df['Media'].unique():\n",
    "    stem_tweets[i] = df[df['Media']==i]['Stem']\n",
    "    tweets[i] = df[df['Media']==i]['Post'].apply(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer \n",
    "analyser = SentimentIntensityAnalyzer()\n",
    "scores = defaultdict(pd.Series)\n",
    "for i in df['Media'].unique():\n",
    "    scores[i] = tweets[i].apply(analyser.polarity_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Visualization (15%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1\n",
    "Created a dataframe that contains the media and each post's compound score. Used the same method with read_data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'media_sent_scores.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-2-dc4087ab6054>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mget_compound_score\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0md\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0md\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'compound'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[1;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"media_sent_scores.txt\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"r\"\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mfp\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     11\u001b[0m     \u001b[0mscores_j\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m \u001b[0mscores\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSeries\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'media_sent_scores.txt'"
     ]
    }
   ],
   "source": [
    "# font\n",
    "font1 = {'family' : 'Times New Roman',\n",
    "'weight' : 'normal',\n",
    "'size'   : 15,\n",
    "}\n",
    "\n",
    "# read in saved sentimental data\n",
    "def get_compound_score(d):\n",
    "    return d['compound']\n",
    "with open(\"media_sent_scores.txt\", \"r\") as fp:\n",
    "    scores_j = json.load(fp)\n",
    "scores = defaultdict(pd.Series)\n",
    "for i in scores_j:\n",
    "    scores[i] = pd.Series(json.loads(scores_j[i]))\n",
    "compound_scores = defaultdict(pd.Series)\n",
    "for i in scores:\n",
    "    compound_scores[i] = scores[i].apply(get_compound_score)\n",
    "\n",
    "temp=[]\n",
    "for i in compound_scores.keys():\n",
    "    df_new=pd.DataFrame(compound_scores[i],columns=['Compound'])\n",
    "    df_new['Media']=i\n",
    "    temp.append(df_new)\n",
    "df_c=pd.concat(temp)\n",
    "df_c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2\n",
    "Check each media compound score's summary. (count mean std min 25% 50% 75% max) and sort the media by the mean compound score from largest to smallest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ow=df_c.groupby('Media').describe()['Compound']\n",
    "df_sort=df_ow.sort_values(by='mean',ascending= False)\n",
    "df_sort"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3\n",
    "Plot histgram for the compound score of each post of 10 media respectively. Used subplot to plot a 5x2 matritx plot. The graphs is also sorted by the mean compound score from largest to smallest, which corresponds to the previous step's sorting. From the graphs shown below, we can see that some media have concentrated netraul i.e. nypost, while others have relatively extremed distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_list=df_sort.index\n",
    "plt.figure(figsize=(1000, 1000))\n",
    "fig, axs = plt.subplots(2, 5,figsize=(40,20),sharey=True,sharex=False)\n",
    "for i in range(10):\n",
    "    axs[i//5, i%5].hist(compound_scores[plot_list[i]],color=sns.color_palette(\"tab10\")[i])\n",
    "    axs[i//5, i%5].set_ylabel('Frequency',font1)\n",
    "    axs[i//5, i%5].set_xlabel('Compound_scores',font1)\n",
    "    axs[i//5, i%5].set_title(plot_list[i],font1)\n",
    "    axs[i//5, i%5].tick_params(labelsize=25,width=0.2)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"graphs/hist.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4\n",
    "Used boxplot to plot a 5x2 matritx plot. The graphs here are one-to-one corresponds to the step 3 graphs in the same colors and positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(50, 50))\n",
    "fig, ax1 = plt.subplots(2,figsize=(20,20))\n",
    "ax1[0] = sns.boxplot(ax=ax1[0],x=\"Media\", y=\"Compound\",data=df_c[df_c['Media'].isin(plot_list[:5])],order=plot_list[:5],\n",
    "                     palette=sns.color_palette(\"tab10\")[:5])\n",
    "ax1[1] = sns.boxplot(ax=ax1[1],x=\"Media\", y=\"Compound\",data=df_c[df_c['Media'].isin(plot_list[5:])],order=plot_list[5:],\n",
    "                     palette=sns.color_palette(\"tab10\")[5:])\n",
    "ax1[0].set_xlabel('Compound_score',font1)\n",
    "ax1[0].set_ylabel('Media',font1)\n",
    "ax1[1].set_xlabel('Compound_score',font1)\n",
    "ax1[1].set_ylabel('Media',font1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"graphs/box.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis & Results (25%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1 AB Test\n",
    "AB testing with α=0.05. We could use the compound score as a metric to check the influence of media on people since it is a continuous variable and also measures the sentiment of a sentence. And we can only test the two media which have the greatest difference in their mean score. According to the histogram, boxplot, and mean, we found that news_ntd and nypost have the greatest difference ( news_ntd-nypost = 0.083883 -0.097072=0.1809556239999914)in any pairs of media. So we can only run an AB test on these two with the largest differences. If there is no statistically significant difference between them. We can conclude that the media does not influence people's posts. Our H0: there are no differences between the compound score for the two media. H1: nypost has a lower compound score than news_ntd. We sample the data 500 times, randomly assign them to media to create data under H0. And we calculate the differences and compare them with our observed difference 0.1809556239999914. Then we calculate late the p-value which is the probability of seeing a difference of means at least as extreme as the observed, under the null hypothesis.\n",
    "The p-value is 0.00<0.05 We reject the H0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_repetitions = 500\n",
    "df_t=df_c[df_c['Media'].isin(['news_ntd','nypost'])].reset_index(drop=True)\n",
    "differences = []\n",
    "for _ in range(n_repetitions):\n",
    "    \n",
    "    # shuffle the compound\n",
    "    shuffled_compound = (\n",
    "       df_t['Compound']\n",
    "        .sample(replace=False, frac=1)\n",
    "        .reset_index(drop=True)\n",
    "    )\n",
    "    \n",
    "    # put them in a table\n",
    "    shuffled = (\n",
    "        df_t\n",
    "        .assign(**{'Shuffled Compound': shuffled_compound})\n",
    "    )\n",
    "    \n",
    "    # compute the group differences (test statistic!)\n",
    "    group_means = (\n",
    "        shuffled\n",
    "        .groupby('Media')\n",
    "        .mean()\n",
    "        .loc[:, 'Shuffled Compound']\n",
    "    )\n",
    "    difference = group_means.diff().iloc[-1]\n",
    "    \n",
    "    # add it to the list of results\n",
    "    differences.append(difference)\n",
    "\n",
    "observed_difference = (\n",
    "    df_t\n",
    "    .groupby('Media')['Compound']\n",
    "    .mean()\n",
    "    .diff()\n",
    "    .iloc[-1]\n",
    ")\n",
    "\n",
    "p_value=np.count_nonzero(differences <= observed_difference) / n_repetitions\n",
    "p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2 Classification\n",
    "Since we obtain such significant difference in sentimential expression between 2 medias, we want to use classification as a predictive tool, featured on unigram of non-neutral sentiment post, try to predict whether this post come from one of these two medias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"graphs/non-n.jpg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \"classification\" contains all non-neutral sentiment post from nypost(New York Post) and news_ntd(NTD News).\n",
    "c = pd.read_csv('classification.csv')\n",
    "\n",
    "# get data from selected 2 medias\n",
    "media_list = ['news_ntd','nypost']\n",
    "d = c[c['Media'] == 'news_ntd']\n",
    "e = c[c['Media'] == 'nypost']\n",
    "f = pd.concat([d,e])\n",
    "\n",
    "# process dataframe\n",
    "tokenizer = nltk.RegexpTokenizer(r\"\\w+\")\n",
    "def get_Mids(m):\n",
    "    return m == \"news_ntd\"\n",
    "f['is_ntd'] = f['Media'].apply(get_Mids)\n",
    "def unigrams(s):\n",
    "    return list(tokenizer.tokenize(s))\n",
    "f['Unigrams'] = f['Post'].apply(unigrams)\n",
    "\n",
    "\n",
    "# shuffle and divide sets\n",
    "shuffled = f.sample(frac=1, random_state=26)\n",
    "train = shuffled[:100000]\n",
    "validation = shuffled[100000:120000]\n",
    "test = shuffled[120000:]\n",
    "\n",
    "# training unigram\n",
    "uC = defaultdict(int)\n",
    "for i in train['Unigrams']:\n",
    "    for j in i:\n",
    "        uC[j]+=1\n",
    "uniwords = sorted(list(uC.items()), key = lambda x:x[1], reverse = True)\n",
    "# use 500 most frequent uniwords as one-hot coding features\n",
    "uniword = [x[0] for x in uniwords[:500]]\n",
    "uniID = dict(zip(uniword, range(len(uniword))))\n",
    "\n",
    "# feature engeineering\n",
    "def feature(datum):\n",
    "    feat = [0]*len(uniword)\n",
    "    uns = list(tokenizer.tokenize(datum))\n",
    "\n",
    "    for un in uns:\n",
    "        if un in uniword:\n",
    "            feat[uniID[un]] += 1\n",
    "    feat.append(1) #offset\n",
    "    return feat\n",
    "\n",
    "# training data\n",
    "trainX = list(train['Post'].apply(feature))\n",
    "trainy = list(train['is_ntd'])\n",
    "valiX = list(validation['Post'].apply(feature))\n",
    "valiy = list(validation['is_ntd'])\n",
    "testX = list(test['Post'].apply(feature))\n",
    "testy = list(test['is_ntd'])\n",
    "\n",
    "solver = LogisticRegression(solver=\"liblinear\",C=0.1)\n",
    "solver.fit(trainX,trainy)\n",
    "\n",
    "# prediction evaluation. Validation set result not included\n",
    "predictions = solver.predict(testX)\n",
    "accuracy = sum(predictions==testy)/len(testy)\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As for result, we got 100% accuracy on our classification model, meaning that there's clear difference about sentimental expression between two medias. This result confirms that certain media will strongly correlated to their followers in terms of emotions and characteristics of twittering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ethics & Privacy (15%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Acquire data from personal social media. Personal information might be involved.\n",
    "- Generalize people’s viewpoint, might have misunderstandings or misinterpretations in comparing similarity.\n",
    "- User's post or media may contain severely controversial topics.\n",
    "- We didn't acquire the Twitter users consent on whether we can use/collect their tweets.\n",
    "- We dropped the names and specific locaitons of any users to protect users' privacy. Even though we didn't show their names/id's, we exposed their tweets information/stances on different events.\n",
    "- We used recent tweets from the Twitter. And recently the U.S. presidential election is still a hot topic. The liberals and conservatives have strongly opposing opinions. The news media are also exhibiting their stances, which may influence their followers in a greater impact than usual.\n",
    "- Our results may expose Twitter users' political stances by analyzing their recent posts which may largely involve in the discussion of presidential election.\n",
    "- Our data involves different people's tweets. Different people have different views on different events. Since our project and datasets may be published online, by look at our data, some people may feel offended by some of the tweets we collected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion & Discussion (15%)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By our EDA result, we found that selected 10 known medias have obvious difference sentimental distribution among their recent follower's tweets. This indicates that we can reasonably expect certain media will be followed by different user group, or their followers will be influenced by their recently followed media. Certain causation is not studied in this project, but we further carries out AB test to confirm this relationship. For two representitively selected medias, we found p-value=0 in AB test shows that such difference between medias are forceful/statistically significant. \n",
    "We also applied classification technic, examing the possibility to determine users' recently followed media by their recent post. Although its bi-outcome prediction, 100% accuracy testified such possibility that we can further applied on multiple medias.\n",
    "\n",
    "Several concerns exist for this project, the most essential one is twitter API limitation. It has a strict limits on request betweens users relationship, users post, and other identity access. This project can be carried to boarder field with higher API access, such as followers' trend by given time, sample all followers instead of recent ones, generate users sentimental tread, etc. \n",
    "Due to time limits, we only do 10 media for EDA, and testifies 2 of them which expected to have distinctive results. Same precedure can be applied to larger variable pools, and by our EDA result variation, we can reasonably expect that such differences among medias on twitter are common phenomenon.\n",
    "For collected data, we noticed that OANN(One America News) has dramatically lower tweets that being collectable. It's possibly because most of its new followers are inactive or private-post users, but reason behind that requires more study to find out. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Team Contributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Yang Kuang\n",
    "Yang worked on the data collection, data analyses, and conclusion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hengyu Liu\n",
    "Hengyu worked on the data visulization and data analyses of the experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Zirui Wang\n",
    "Zirui Wang coordinated the team work. Zirui worked on the design, data collection, and report of the experiment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}