twitter_pretagging.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "twitter-pretagging",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMP0FGFNwzfXgTOzV7Gt7Kh",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dieko95/AlgoTrading/blob/master/twitter_pretagging.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nYeFRTD02JJd",
        "colab_type": "text"
      },
      "source": [
        "# Training Dataset Creation - Tagging \n",
        "\n",
        "This notebook aims to create the dataset for Code for Venezuela's Blackout Project. \n",
        "\n",
        "This dataset is going to be consumed by an ML model that will aim to predict: \n",
        "\n",
        "- If a tweet is from Venezuela\n",
        "- If so from which state(s) \n",
        "- About what public service the user is reporting (sinluz)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rPjvaTP1pN6_",
        "colab_type": "text"
      },
      "source": [
        "## Libraries\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YESX9HG2pMWC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "\n",
        "# For better visualization of text in Pandas DF\n",
        "pd.set_option('display.max_colwidth', None)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUGVhRHs5OZn",
        "colab_type": "text"
      },
      "source": [
        "## Accessing Data\n",
        "\n",
        "The untagged dataset originates from scraped tweets by Code For Venezuela's Angostura ETL. A subset of tweets (11,000) was queried from the etl in order for them to be tagged. The first 4,000 tweets have already been tagged. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rN37zlHcfbj0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Read CSV from github \n",
        "tags_df = pd.read_csv('https://raw.githubusercontent.com/dieko95/blackouts-C4V/diego-first-iter/tagging-set-original_for_jupyter_tagging.csv')\n",
        "\n",
        "\n",
        "# Fill Nas in label_country to 0\n",
        "tags_df.label_country.fillna('0', \n",
        "                             inplace = True)\n",
        "\n",
        "\n",
        "# Tagged Tweets\n",
        "pre_tag_df = tags_df[(tags_df.label_country != '0')].copy()\n",
        "\n",
        "\n",
        "# Tweets to tag\n",
        "to_tag_df = tags_df[~tags_df.index.isin(tags_df[(tags_df.label_country != '0')].index)].copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aROWrNLK52aq",
        "colab_type": "text"
      },
      "source": [
        "## Cleaning Text\n",
        "\n",
        "This is a helper function to quickly clean text.\n",
        "\n",
        "- Converts all text to low caps. \n",
        "- Strips all spanish accents\n",
        "\n",
        "Pending:\n",
        "\n",
        "- Strip dots and links (@ and # must remain) "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6ec-4OIX6BAi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cleaner(df,text_col):\n",
        "  # to lower\n",
        "\n",
        "  df[text_col] = df[text_col].str.lower()\n",
        "\n",
        "  # Convert common spanish accents\n",
        "\n",
        "  df[text_col] = df[text_col].str.replace(\"ú\", \"u\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ù\", \"u\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ü\", \"u\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ó\", \"o\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ò\", \"o\")\n",
        "  df[text_col] = df[text_col].str.replace(\"í\", \"i\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ì\", \"i\")\n",
        "  df[text_col] = df[text_col].str.replace(\"é\", \"e\")\n",
        "  df[text_col] = df[text_col].str.replace(\"è\", \"e\")\n",
        "  df[text_col] = df[text_col].str.replace(\"á\", \"a\")\n",
        "  df[text_col] = df[text_col].str.replace(\"à\", \"a\")\n",
        "  df[text_col] = df[text_col].str.replace(\"ñ\", \"gn\")\n",
        "\n",
        "  return df\n",
        "\n",
        "\n",
        "to_tag_df = cleaner(to_tag_df, 'concat_text_user_description')\n",
        "to_tag_df = cleaner(to_tag_df, 'full_text')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtogO9m3jv2n",
        "colab_type": "text"
      },
      "source": [
        "## Sections to Tag \n",
        "\n",
        "- Tag label_type (service reported)\n",
        "  - Extracting pound signs (\\#)\n",
        "\n",
        "- Tag Country\n",
        "  - Matches any state? \n",
        "  - has keyword 'edo' or 'estado' in it?\n",
        "  - Follows any of the common accounts?\n",
        "- Tag State\n",
        "  - Match with list of venezuela states\n",
        "  - We can use a list of venezuelan cities as well "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJQo6Q725duV",
        "colab_type": "text"
      },
      "source": [
        "## Tagging Label Type"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aEVa39Xe5ogc",
        "colab_type": "text"
      },
      "source": [
        "### Hashtags\n",
        "\n",
        "* \\#SinLuz\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QOMhF-6eCFxY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "clJfc4I_vpdH",
        "colab_type": "text"
      },
      "source": [
        "## Tagging Country\n",
        "\n",
        "  - Matches any state? \n",
        "  - has keyword 'edo' or 'estado' in it?\n",
        "  - Follows any of the common accounts?\n",
        "\n",
        "*Notes*\n",
        "  - For this section I will use the tweet's original text. If I include the user description it can add noise because a user can be reporting about a power outage of another state (e.g., I'm from caracas and reporting a power outage in Zulia)\n",
        "  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LXRzNfuXGk-G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMZGyQZR0cmE",
        "colab_type": "text"
      },
      "source": [
        "## Tagging State"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_sEMAHvA3ToW",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Incluimos cuentas que reportan a nivel nacional? Es ruido porque lo que hacen es repetir lo que otros usuarios dicen? O captura señal porque son reportes de fallas de luz?\n",
        "\n",
        "~~~\n",
        "print(tags_df.loc[8122,'full_text'])\n",
        "\n",
        "#Ahora Reportan más zonas #SinLuz: \n",
        "\n",
        "Catia, Distrito Capital ❌💡\n",
        "Guatire y Guarenas, Edo. Miranda ❌💡\n",
        "Estado Mérida ❌💡\n",
        "Estado Aragua ❌💡\n",
        "\n",
        "Comenta si hay fallas en tu zona #2Oct\n",
        "\n",
        "~~~"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3yimdsAJVOBU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}