Swinburne_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Swinburne_project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project is to understand community attitudes towards violence against women in Australia, provided by IT departments in Swinburne University of Technology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data are from the Twitter and Instagram.\n",
    "\n",
    "Research scope is Melbourne metropolitan region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Source code: https://github.com/Tiangewang0524/Swinburne_project\n",
    "\n",
    "\n",
    "** There is the shortage on the raw data. Will be discussed in the end. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Requisite package:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib\n",
    "\n",
    "vaderSentiment\n",
    "\n",
    "json\n",
    "\n",
    "wordcloud\n",
    "\n",
    "nltk\n",
    "\n",
    "* If you are doing this for first time you need to use nltk.download('punkt') for word_tokenize to work and nltk.download('averaged_perceptron_tagger') for pos_tag to work. \n",
    "\n",
    "\n",
    "* Part-of-speech tags list:\n",
    "\n",
    "  https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data collect "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "raw_data.sh\n",
    "\n",
    "A shell script to collect Twitter and Instagram raw data from AURIN/tweet-infra: https://github.com/AURIN/tweet-infra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "curl \"http://45.113.232.90/couchdbro/instagram/_design/instagram/_view/summary\" \\\n",
    "-G \\\n",
    "--data-urlencode 'start_key=[\"melbourne\",2017,1,1]' \\\n",
    "--data-urlencode 'end_key=[\"melbourne\",2018,3,12]' \\\n",
    "--data-urlencode 'reduce=false' \\\n",
    "--data-urlencode 'include_docs=true' \\\n",
    "--user \"readonly:ween7ighai9gahR6\" \\\n",
    "-o /tmp/raw_instagram_data.json\n",
    "\n",
    "curl \"http://45.113.232.90/couchdbro/twitter/_design/twitter/_view/summary\" \\\n",
    "-G \\\n",
    "--data-urlencode 'start_key=[\"melbourne\",2017,6,15]' \\\n",
    "--data-urlencode 'end_key=[\"melbourne\",2018,5,4]' \\\n",
    "--data-urlencode 'reduce=false' \\\n",
    "--data-urlencode 'include_docs=true' \\\n",
    "--user \"readonly:ween7ighai9gahR6\" \\\n",
    "-o /tmp/raw_twitter_data.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'wordcloud'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-d6c7a513f0ab>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mvaderSentiment\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvaderSentiment\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSentimentIntensityAnalyzer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mwordcloud\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mWordCloud\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# import matplotlib.ticker as mtick\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'wordcloud'"
     ]
    }
   ],
   "source": [
    "import json\n",
    "# from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "import nltk\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt\n",
    "# import matplotlib.ticker as mtick"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* convert:\n",
    "\n",
    "To convert numbers to months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert numbers to months\n",
    "def convert(x):\n",
    "    return{\n",
    "        1:'Jan',\n",
    "        2:'Feb',\n",
    "        3:'Mar',\n",
    "        4:'Apr',\n",
    "        5:'May',\n",
    "        6:'Jun',\n",
    "        7:'Jul',\n",
    "        8:'Aug',\n",
    "        9:'Sep',\n",
    "        10:'Oct',\n",
    "        11:'Nov',\n",
    "        12:'Dec'\n",
    "    }.get(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* filtered:\n",
    "\n",
    "Clean and filter Twitter/Instagram raw data with hashtag #metoo & #timesup and reserve 5 attributes: \"tweet_id\", \"user_id\", \"user_name\", \"post_text\" and \"post_time\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter Twitter data\n",
    "def twi_filtered(read_file, write_file):\n",
    "    info = dict()\n",
    "    write_file.write('{\"rows\":[\\n')\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            # print(item)\n",
    "            sub_post = item[\"doc\"][\"text\"].split()\n",
    "            if key_hashtag_1 in sub_post or key_hashtag_2 in sub_post:\n",
    "                info[\"post_time\"] = item[\"doc\"][\"created_at\"]\n",
    "                info[\"tweet_id\"] = item[\"doc\"][\"id_str\"]\n",
    "                info[\"user_id\"] = item[\"doc\"][\"user\"][\"id_str\"]\n",
    "                info[\"user_name\"] = item[\"doc\"][\"user\"][\"name\"]\n",
    "                info[\"post_text\"] = item[\"doc\"][\"text\"]\n",
    "                each = json.dumps(info)\n",
    "                write_file.write(each + ',' + '\\n')\n",
    "                write_file.flush()\n",
    "                print(\"User \" + info[\"user_name\"] + \"'s filtered task done!\")\n",
    "        except:\n",
    "            continue\n",
    "    write_file.seek(write_file.tell() - 2)\n",
    "    write_file.write(']}')\n",
    "\n",
    "\n",
    "# filter Instagram data\n",
    "def ins_filtered(read_file, write_file):\n",
    "    list_a = []\n",
    "    info = dict()\n",
    "    write_file.write('{\"rows\":[\\n')\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            # Removed '#' from hashtag\n",
    "            if ins_tag_1 in item[\"doc\"][\"tags\"] or ins_tag_2 in item[\"doc\"][\"tags\"]:\n",
    "                # created_time is a string of numbers which are hard to understand\n",
    "                # and no document to interpret what they represent.\n",
    "                # info[\"post_time\"] = item[\"doc\"][\"created_time\"]\n",
    "                list_a.append(item[\"key\"][1])\n",
    "                list_a.append(convert(item[\"key\"][2]))\n",
    "                list_a.append(item[\"key\"][3])\n",
    "                info[\"post_time\"] = list_a\n",
    "                list_a = []\n",
    "                info[\"ins_post_id\"] = item[\"doc\"][\"caption\"][\"id\"]\n",
    "                info[\"user_id\"] = item[\"doc\"][\"user\"][\"id\"]\n",
    "                info[\"user_name\"] = item[\"doc\"][\"user\"][\"username\"]\n",
    "                info[\"post_text\"] = item[\"doc\"][\"caption\"][\"text\"]\n",
    "                each = json.dumps(info)\n",
    "                print(each)\n",
    "                write_file.write(each + ',' + '\\n')\n",
    "                write_file.flush()\n",
    "                print(\"User \" + info[\"user_name\"] + \"'s filtered task done!\")\n",
    "        except:\n",
    "            continue\n",
    "    write_file.seek(write_file.tell() - 2)\n",
    "    write_file.write(']}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* sentiment:\n",
    "\n",
    "Sentiment analysis on Twitter/Instagram filtered data based on vaderSentiment package. To evaluate the attitude (positive or negative) of each Twitter/Instagram post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# twitter sentiment analysis\n",
    "def twi_sentiment(read_file, write_file):\n",
    "    month_attitude = dict()\n",
    "    sid = SentimentIntensityAnalyzer()\n",
    "    info = dict()\n",
    "    write_file.write('{\"rows\":[\\n')\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            post_time = item[\"post_time\"].split()\n",
    "            date = post_time[5] + \",\" + post_time[1]\n",
    "            if date not in month_attitude.keys():\n",
    "                month_attitude[date] = dict()\n",
    "                month_attitude[date][\"count\"] = 0\n",
    "                month_attitude[date][\"pos_count\"] = 0\n",
    "                month_attitude[date][\"neg_count\"] = 0\n",
    "            month_attitude[date][\"count\"] += 1\n",
    "            sentence = item[\"post_text\"]\n",
    "\n",
    "            # evaluate the sentiment for each tweet and save as a new file for further analysis.\n",
    "            info = dict(item)\n",
    "\n",
    "            # 1. Positive score – Negative score > 0 => Positive sentiment.\n",
    "            # 2. Positive score – Negative score < 0 => Negative sentiment.\n",
    "            if sid.polarity_scores(sentence)['neg'] - sid.polarity_scores(sentence)['pos'] > 0:\n",
    "                month_attitude[date][\"neg_count\"] += 1\n",
    "                info[\"attitude\"] = \"negative\"\n",
    "            elif sid.polarity_scores(sentence)['neg'] - sid.polarity_scores(sentence)['pos'] < 0:\n",
    "                month_attitude[date][\"pos_count\"] += 1\n",
    "                info[\"attitude\"] = \"positive\"\n",
    "\n",
    "            each = json.dumps(info)\n",
    "            write_file.write(each + ',' + '\\n')\n",
    "            write_file.flush()\n",
    "            print(\"User \" + info[\"user_name\"] + \"'s attitude task done!\")\n",
    "        except:\n",
    "            continue\n",
    "    write_file.seek(write_file.tell() - 2)\n",
    "    write_file.write(']}')\n",
    "    return month_attitude\n",
    "\n",
    "\n",
    "# instagram sentiment analysis\n",
    "def ins_sentiment(read_file, write_file):\n",
    "    month_attitude = dict()\n",
    "    sid = SentimentIntensityAnalyzer()\n",
    "    info = dict()\n",
    "    write_file.write('{\"rows\":[\\n')\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            date = str(item[\"post_time\"][0]) + \",\" + str(item[\"post_time\"][1])\n",
    "            if date not in month_attitude.keys():\n",
    "                month_attitude[date] = dict()\n",
    "                month_attitude[date][\"count\"] = 0\n",
    "                month_attitude[date][\"pos_count\"] = 0\n",
    "                month_attitude[date][\"neg_count\"] = 0\n",
    "            month_attitude[date][\"count\"] += 1\n",
    "            sentence = item[\"post_text\"]\n",
    "\n",
    "            # evaluate the sentiment for each tweet and save as a new file for further analysis.\n",
    "            info = dict(item)\n",
    "\n",
    "            # 1. Positive score – Negative score > 0 => Positive sentiment.\n",
    "            # 2. Positive score – Negative score < 0 => Negative sentiment.\n",
    "            if sid.polarity_scores(sentence)['neg'] - sid.polarity_scores(sentence)['pos'] > 0:\n",
    "                month_attitude[date][\"neg_count\"] += 1\n",
    "                info[\"attitude\"] = \"negative\"\n",
    "            elif sid.polarity_scores(sentence)['neg'] - sid.polarity_scores(sentence)['pos'] < 0:\n",
    "                month_attitude[date][\"pos_count\"] += 1\n",
    "                info[\"attitude\"] = \"positive\"\n",
    "\n",
    "            each = json.dumps(info)\n",
    "            write_file.write(each + ',' + '\\n')\n",
    "            write_file.flush()\n",
    "            print(\"User \" + info[\"user_name\"] + \"'s attitude task done!\")\n",
    "        except:\n",
    "            continue\n",
    "    write_file.seek(write_file.tell() - 2)\n",
    "    write_file.write(']}')\n",
    "    return month_attitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* draw_sentiment:\n",
    "\n",
    "Draw diagrams based on the result above via matplotlib package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# draw the sentiment analysis\n",
    "def draw_sentiment(attitude):\n",
    "    # y axis related\n",
    "    y_axis_pos = list()\n",
    "    y_axis_neg = list()\n",
    "    y_axis_difference = list()\n",
    "    for each_data in attitude.values():\n",
    "        y_axis_pos.append(int(-(each_data['pos_count'])))\n",
    "        y_axis_neg.append(int(each_data['neg_count']))\n",
    "        y_axis_difference.append(int(each_data['neg_count']) - int(each_data['pos_count']))\n",
    "\n",
    "    # X axis related\n",
    "    x_axis = list()\n",
    "    for month in attitude.keys():\n",
    "        x_axis.append(month)\n",
    "    x_length = [i for i in range(len(x_axis))]\n",
    "\n",
    "    fig = plt.figure()\n",
    "    ax1 = fig.add_subplot(111)\n",
    "    ax1.plot(x_length, y_axis_difference, 'or-', label='difference')\n",
    "    ax1.set_title('The comparison between positive and negative comments\\n', fontsize=12, color='r')\n",
    "    for i, (_x, _y) in enumerate(zip(x_length, y_axis_difference)):\n",
    "        plt.text(_x, _y, y_axis_difference[i], color='black', fontsize=10)\n",
    "\n",
    "    # left legend\n",
    "    ax1.legend(loc=1)\n",
    "    ax1.set_ylim([-500, 1000])\n",
    "    ax1.set_ylabel('difference')\n",
    "    ax2 = ax1.twinx()\n",
    "    plt.bar(x_length, y_axis_pos, alpha=0.3, color='blue', label='pos_count')\n",
    "    plt.bar(x_length, y_axis_neg, alpha=0.3, color='red', label='neg_count')\n",
    "\n",
    "    # right legend\n",
    "    ax2.legend(loc=2)\n",
    "    ax2.set_ylim([-500, 1000])\n",
    "    ax2.set_ylabel('number of count')\n",
    "\n",
    "    for i, (_x, _y) in enumerate(zip(x_length, y_axis_pos)):\n",
    "        plt.text(_x, _y - 50, y_axis_pos[i], color='maroon', fontsize=8, ha='center', va='bottom')\n",
    "    for i, (_x, _y) in enumerate(zip(x_length, y_axis_neg)):\n",
    "        plt.text(_x, _y + 5, y_axis_neg[i], color='maroon', fontsize=8, ha='center', va='bottom')\n",
    "\n",
    "    plt.xticks(x_length, x_axis)\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment results are shown below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img style=\"float: middle;\" src=\"https://github.com/Tiangewang0524/Swinburne_project/blob/master/dataset/twi_sentiment.png?raw=true\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<img style=\"float: middle;\" src=\"https://github.com/Tiangewang0524/Swinburne_project/blob/master/dataset/ins_sentiment.png?raw=true\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* find_trollers:\n",
    "\n",
    "Find trollers (who always have negative comments on women)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find trollers\n",
    "def find_troller(read_file, write_file):\n",
    "    # count positive and negative comments for all users\n",
    "    user_list = dict()\n",
    "    nega_user = dict()\n",
    "    troller_user = dict()\n",
    "    write_file.write('{\"Users\":[\\n')\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            user_id = item[\"user_id\"]\n",
    "            if user_id not in user_list.keys():\n",
    "                user_list[user_id] = dict()\n",
    "                user_list[user_id][\"count\"] = 0\n",
    "                user_list[user_id][\"pos_count\"] = 0\n",
    "                user_list[user_id][\"neg_count\"] = 0\n",
    "            user_list[user_id][\"count\"] += 1\n",
    "            if item[\"attitude\"] == \"negative\":\n",
    "                user_list[user_id][\"neg_count\"] += 1\n",
    "            elif item[\"attitude\"] == \"positive\":\n",
    "                user_list[user_id][\"pos_count\"] += 1\n",
    "        except:\n",
    "            continue\n",
    "\n",
    "    # for better sort, count negative comments for users and save as a dict.\n",
    "    for each_user in user_list.keys():\n",
    "        nega_user[each_user] = user_list[each_user][\"neg_count\"]\n",
    "\n",
    "    # Select top 10 trollers who posted most negative commments on women.\n",
    "    sort_user = sorted(nega_user.items(), key=lambda item:item[1], reverse=True)\n",
    "    # count for 10 times\n",
    "    count_a = 0\n",
    "    read_file.seek(0)\n",
    "    for each in sort_user:\n",
    "        # each[0] is the user_id, each[1] is the number of negative comments.\n",
    "        troller_user[\"user_id\"] = each[0]\n",
    "        for each_row in read_file:\n",
    "            try:\n",
    "                # Remove the blank and comma in the end of each row.\n",
    "                item = json.loads(each_row[:-2])\n",
    "                if each[0] == item[\"user_id\"]:\n",
    "                    troller_user[\"user_name\"] = item[\"user_name\"]\n",
    "                    read_file.seek(0)\n",
    "                    break\n",
    "            except:\n",
    "                continue\n",
    "        troller_user[\"negative_comments\"] = each[1]\n",
    "        each = json.dumps(troller_user)\n",
    "        write_file.write(each + ',' + '\\n')\n",
    "        write_file.flush()\n",
    "        print(\"Troller \" + troller_user[\"user_name\"] + \" has been found!\")\n",
    "        troller_user.clear()\n",
    "\n",
    "        count_a += 1\n",
    "        if count_a == 10:\n",
    "            break\n",
    "\n",
    "    write_file.seek(write_file.tell() - 2)\n",
    "    write_file.write(']}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* word_identify:\n",
    "\n",
    "Collect all the tweet and instagram posts, split them and identify the part-of-speech of the words. Then store as a txt file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect all the tweet and instagram posts, split them and identify the part-of-speech of the words.\n",
    "# Then store as a txt file.\n",
    "def word_identify(read_file, write_file):\n",
    "    text = list()\n",
    "    for each_row in read_file:\n",
    "        try:\n",
    "            # Remove the blank and comma in the end of each row.\n",
    "            item = json.loads(each_row[:-2])\n",
    "            # Remove the identical retweets\n",
    "            if item[\"post_text\"] not in text:\n",
    "                text.append(item[\"post_text\"])\n",
    "        except:\n",
    "            continue\n",
    "    str_text = \" \".join(text)\n",
    "\n",
    "    # split the str_text\n",
    "    tokens = nltk.word_tokenize(str_text)\n",
    "    # Add part-of-speech tag for each word\n",
    "    tagged = nltk.pos_tag(tokens)\n",
    "    text = list()\n",
    "    for each in tagged:\n",
    "        # each format : (word, 'tag')\n",
    "        # Reserve nouns, verbs and adjectives, remove RT, @, https, SSS and URL。\n",
    "        if each[1][0] == 'N' or each[1][0] == 'J' or each[1] == 'VB':\n",
    "            if each[0] != '@' and each[0] != 'metoo' and each[0] != 'timesup' and each[0][0] != '/':\n",
    "                if each[0] != 'RT' and each[0] != 'https' and each[0] != 'SSS':\n",
    "                    text.append(each[0])\n",
    "\n",
    "    str_text = \" \".join(text)\n",
    "    write_file.write(str_text)\n",
    "    print(\"Word_identify task done!\")\n",
    "    return str_text\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* word_cloud:\n",
    "\n",
    "Generate word cloud for Twitter/Instagram filtered data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word cloud display\n",
    "def word_cloud(str_text):\n",
    "    wordcloud = WordCloud(background_color=\"white\", width=1000, height=860, margin=2).generate(str_text)\n",
    "    plt.imshow(wordcloud)\n",
    "    plt.axis(\"off\")\n",
    "    plt.show()\n",
    "\n",
    "    file_name = input(\"input file name:\")\n",
    "    wordcloud.to_file(file_name)\n",
    "    print(\"Word_cloud task done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WordCloud results are shown below:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img style=\"float: middle;\" src=\"https://github.com/Tiangewang0524/Swinburne_project/blob/master/dataset/twi_word_cloud.png?raw=true\" width=\"60%\">"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img style=\"float: middle;\" src=\"https://github.com/Tiangewang0524/Swinburne_project/blob/master/dataset/ins_word_cloud.png?raw=true\" width=\"60%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Reflection "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the Twitter API has 7-day limit which means no tweets will be found for a date older than one week. So I had to use Prof. Richard's cloud to obtain raw Twitter data. Unfortunately, the data of Melbourne tweets stored in the cloud are from 15th JUN, 2017 to 4th MAY, 2018. Thus, I only gained 140 GB raw Twitter data.\n",
    "\n",
    "Similarly, the Instagram API also has strict requirement when signing up for the 'access_token'. It asked me to demonstrate some reasons of using their API and record a video to display what I'm researching for (Very strange). So I also use Richard's cloud but only got 12 GB raw Instagram data from 1ST JAN, 2017 to 12th MAR, 2018. \n",
    "\n",
    "After testing, these functions doesn't show any bugs and fully satify with the requirements of tasks. However, as the crawler step is not created by myself and I have no way of knowing the detail, I can't figure out why the numbers of positive/negative counts in some months are too small. \n",
    "\n",
    "I guess:\n",
    "\n",
    "1. The data for some days in some months in Richard's cloud are missing. \n",
    "\n",
    "2. Richard's cloud will not record those tweets/Instagram posts without the coordinates although this kind of users who posted them in Melbourne region. \n",
    "\n",
    "Functions can be use for any kinds and any period of Twitter/Instagram data. Once we have the chance to get more  abundant data, we could gain more accurate result."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}