04_Data-Preparation_limited_merged_no-nulls.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e046c399-9e3c-42e2-a970-a981f2ef14b4",
   "metadata": {},
   "source": [
    "# Data Preparation\n",
    "**Author**: Andrea Cass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1873e523-46fc-45e3-aca8-a9c40dfbc863",
   "metadata": {},
   "source": [
    "## 1. About this notebook\n",
    "The purpose of this notebook is to make some final preparations before the data can be explored and visualized. The data used is:\n",
    "> *03_Sentiment-analysis_limited_merged.csv*\n",
    "\n",
    "Goals:\n",
    "* Seperate the set of probability scores into individual columns (Negative, Neutral, Positive)\n",
    "* Create a new categorical column, CLASS, based on which sentiment classification has the largest probability score\n",
    "* Create a new numerical column, _____, based on class (-1 for NEGATIVE, 0 for NEUTRAL, 1 for POSITIVE)\n",
    "\n",
    "The output will be a single dataset saved as a csv filed titled,\n",
    "> *04_Prepared-data_limited_merged.csv*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b299562c-6687-4894-950f-dace29d02c8b",
   "metadata": {},
   "source": [
    "## 2. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "56e3dce7-aeff-420d-8ae8-25acab67dc15",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import csv\n",
    "import os\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cde82a60-c3e1-4076-ac87-9a0029956d72",
   "metadata": {},
   "source": [
    "## 3. Working directory & file paths\n",
    "\n",
    "Before beginning data and pre-processing, the working directory needs to be set up. You should have already created a folder called \"CASS_thesis\" within your desired working directory.\n",
    "\n",
    "Two objects will be named:\n",
    "\n",
    "* **cwd**: the current working directory (e.g., your Desktop)\n",
    "* **CASS_thesis**: the folder where all data from my Notebooks will be saved"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f45b61-6981-454d-9b53-14e3827d5c99",
   "metadata": {},
   "source": [
    "### 3.1. Current working directory\n",
    "Use the code below to find out what your current working directory is set to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "412c69dc-155f-4dca-8263-5e12d375ce40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/andycass/Desktop/Thesis_data-and-code'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find current working directory\n",
    "\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55843a9f-72a2-487d-b825-235499ee69b0",
   "metadata": {},
   "source": [
    "If your current working directory is not your desired directory, follow the subsequent steps to change the working diectory by:\n",
    "1. deciding where you would like your working directory to be (e.g., your Desktop)\n",
    "2. entering the file path of your desired working directory into the code below\n",
    "\n",
    "**NOTE**: If you are satisfied with your working directory and do NOT wish to change it, skip the block of code underneath **3.1.1. Changing current working directory** and, instead, proceed from the block of code underneath **3.1.2. Naming current working directory**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418064cb-6dd9-42e1-90a5-03f7543e1507",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 3.1.1. Changing current working directory\n",
    "**NOTE**: The code below contains the path to **my** desired working directory to serve as an example. You must alter it to the path of **your** desired working directory. Keep in mind that my example is formatted according to Macbook standards, and Windows formatting differs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b18d73ab-952a-4f05-93f5-b9d548d3dd7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# changing current working directory\n",
    "\n",
    "os.chdir('/Users/andycass/Desktop/Thesis_data-and-code')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "044c6bc0-de8f-40d1-913e-496931db2058",
   "metadata": {},
   "source": [
    "#### 3.1.2. Naming current working directory\n",
    "Now that your current working directory is established, use the code below to name it \"cwd\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d9c2f68b-0535-4c5c-a0db-8006c7456735",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the current working directory\n",
    "\n",
    "cwd = Path.cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "90415f05-93aa-4825-9c3b-f1a80d628858",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the current working directory location\n",
    "\n",
    "cwd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d691682a-81f5-4b90-b2cb-cb40b9f17816",
   "metadata": {},
   "source": [
    "### 3.2 CASS_thesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ec0b24c3-6784-473f-a78b-cc1c9ce831e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the CASS_thesis folder\n",
    "\n",
    "CASS_thesis = cwd / 'CASS_thesis'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a1aabfae-d77f-44a4-8c32-8dc51f800d12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code/CASS_thesis')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the CASS_thesis location\n",
    "\n",
    "CASS_thesis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab242360-4d41-4f89-982b-1fcb1fb22f82",
   "metadata": {},
   "source": [
    "## 4. Separating scores\n",
    "### 4.1. Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "573a3f79-ca2d-4581-9965-40afe292940a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_1548/2123999987.py:1: DtypeWarning: Columns (0) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  df = pd.read_csv(CASS_thesis / '03_Sentiment-analysis_limited_merged.csv', index_col=[0])\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(CASS_thesis / '03_Sentiment-analysis_limited_merged.csv', index_col=[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14713f66-f4ba-4581-9738-040d7f118b6a",
   "metadata": {},
   "source": [
    "### 4.2. Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3b44e02d-6627-4b39-82b8-fd94759b9d3b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844  0.88380396 0.0482117 ]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313  0.5833739  0.05059479]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568 0.4430802  0.25394407]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253 0.07604674 0.9013506 ]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864  0.27545568 0.02625783]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66260</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766 0.82208896 0.14240335]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986  0.09380516 0.01299612]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968 0.5201203  0.32332003]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664 0.06375945 0.02303378]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373    0.06414817 0.01212183]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66266 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66260  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66261  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66262  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66263  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66264  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66260  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66261  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66262  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66263  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66264  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66260                           0.0                         1.0   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         0.0   \n",
       "66264                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66260                        2.0                         0.0   \n",
       "66261                        6.0                         0.0   \n",
       "66262                       17.0                         0.0   \n",
       "66263                        1.0                         0.0   \n",
       "66264                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ...  week  month    year  year-week  \\\n",
       "0                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "1                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "2                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "3                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "4                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "...                                ...  ...   ...    ...     ...        ...   \n",
       "66260                              0.0  ...  26.0    6.0  2021.0    2021-26   \n",
       "66261                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
       "66262                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
       "66263                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
       "66264                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
       "\n",
       "       year-month  Language                       date      inflow  \\\n",
       "0         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "1         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "2         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "3         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "4         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "...           ...       ...                        ...         ...   \n",
       "66260     2021-06   English  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66261     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66262     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263     2021-06   English  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66264     2021-06   English  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66260  For day 1 of week 2, @user discusses the Socia...   \n",
       "66261  @user Europe is racist, but in a different way...   \n",
       "66262  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66263  @user I must have missed the great flight of t...   \n",
       "66264  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                   scores  \n",
       "0      [0.0679844  0.88380396 0.0482117 ]  \n",
       "1      [0.3660313  0.5833739  0.05059479]  \n",
       "2      [0.30297568 0.4430802  0.25394407]  \n",
       "3      [0.02260253 0.07604674 0.9013506 ]  \n",
       "4      [0.6982864  0.27545568 0.02625783]  \n",
       "...                                   ...  \n",
       "66260  [0.03550766 0.82208896 0.14240335]  \n",
       "66261  [0.8931986  0.09380516 0.01299612]  \n",
       "66262  [0.15655968 0.5201203  0.32332003]  \n",
       "66263  [0.91320664 0.06375945 0.02303378]  \n",
       "66264  [0.92373    0.06414817 0.01212183]  \n",
       "\n",
       "[66266 rows x 23 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ee77b10c-0b5f-4832-b87d-53cd5954e8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# resetting index\n",
    "\n",
    "df = df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "28654757-af27-4f04-a65a-6cecd5a8e49d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844  0.88380396 0.0482117 ]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313  0.5833739  0.05059479]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568 0.4430802  0.25394407]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253 0.07604674 0.9013506 ]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864  0.27545568 0.02625783]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766 0.82208896 0.14240335]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986  0.09380516 0.01299612]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968 0.5201203  0.32332003]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664 0.06375945 0.02303378]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66265</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373    0.06414817 0.01212183]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66266 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66261  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66262  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66264  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66265  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66261  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66262  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66263  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66264  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66265  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         1.0   \n",
       "66264                           0.0                         0.0   \n",
       "66265                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66261                        2.0                         0.0   \n",
       "66262                        6.0                         0.0   \n",
       "66263                       17.0                         0.0   \n",
       "66264                        1.0                         0.0   \n",
       "66265                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ...  week  month    year  year-week  \\\n",
       "0                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "1                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "2                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "3                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "4                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
       "...                                ...  ...   ...    ...     ...        ...   \n",
       "66261                              0.0  ...  26.0    6.0  2021.0    2021-26   \n",
       "66262                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
       "66263                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
       "66264                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
       "66265                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
       "\n",
       "       year-month  Language                       date      inflow  \\\n",
       "0         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "1         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "2         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "3         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "4         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
       "...           ...       ...                        ...         ...   \n",
       "66261     2021-06   English  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66262     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66264     2021-06   English  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66265     2021-06   English  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66261  For day 1 of week 2, @user discusses the Socia...   \n",
       "66262  @user Europe is racist, but in a different way...   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66264  @user I must have missed the great flight of t...   \n",
       "66265  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                   scores  \n",
       "0      [0.0679844  0.88380396 0.0482117 ]  \n",
       "1      [0.3660313  0.5833739  0.05059479]  \n",
       "2      [0.30297568 0.4430802  0.25394407]  \n",
       "3      [0.02260253 0.07604674 0.9013506 ]  \n",
       "4      [0.6982864  0.27545568 0.02625783]  \n",
       "...                                   ...  \n",
       "66261  [0.03550766 0.82208896 0.14240335]  \n",
       "66262  [0.8931986  0.09380516 0.01299612]  \n",
       "66263  [0.15655968 0.5201203  0.32332003]  \n",
       "66264  [0.91320664 0.06375945 0.02303378]  \n",
       "66265  [0.92373    0.06414817 0.01212183]  \n",
       "\n",
       "[66266 rows x 23 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8705d2aa-3db2-49c9-93ab-ba97301d4c22",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 66266 entries, 0 to 66265\n",
      "Data columns (total 23 columns):\n",
      " #   Column                           Non-Null Count  Dtype  \n",
      "---  ------                           --------------  -----  \n",
      " 0   text                             66265 non-null  object \n",
      " 1   author_id                        66262 non-null  float64\n",
      " 2   created_at                       38612 non-null  object \n",
      " 3   lang                             66262 non-null  object \n",
      " 4   geo.place_id                     38226 non-null  object \n",
      " 5   public_metrics.retweet_count     38612 non-null  float64\n",
      " 6   public_metrics.reply_count       38612 non-null  float64\n",
      " 7   public_metrics.like_count        38612 non-null  float64\n",
      " 8   public_metrics.quote_count       38612 non-null  float64\n",
      " 9   public_metrics.impression_count  38612 non-null  float64\n",
      " 10  geo.coordinates.coordinates      4348 non-null   object \n",
      " 11  in_reply_to_user_id              12326 non-null  float64\n",
      " 12  entities.hashtags                15560 non-null  object \n",
      " 13  week                             38612 non-null  float64\n",
      " 14  month                            38612 non-null  float64\n",
      " 15  year                             38612 non-null  float64\n",
      " 16  year-week                        38612 non-null  object \n",
      " 17  year-month                       38612 non-null  object \n",
      " 18  Language                         66265 non-null  object \n",
      " 19  date                             38612 non-null  object \n",
      " 20  inflow                           66265 non-null  object \n",
      " 21  new text                         66264 non-null  object \n",
      " 22  scores                           66264 non-null  object \n",
      "dtypes: float64(10), object(13)\n",
      "memory usage: 11.6+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36885422-9583-4fce-8bc8-21604a2d663d",
   "metadata": {},
   "source": [
    "### 4.3. Converting scores to string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f12f75c6-2bb4-443e-b2d3-98b8f9503e31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking what dtype scores is\n",
    "\n",
    "df.scores.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "5640e882-26f3-4263-9466-41c7f625ebf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting scores to string\n",
    "\n",
    "df[\"scores\"]=df[\"scores\"].values.astype('str')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1f2a9bf-c219-4adf-bc58-39002e5434f3",
   "metadata": {},
   "source": [
    "### 4.4 Splitting the scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3fa97042-8b63-4fd8-aec3-cbafd8544f7e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0        [0.0679844  0.88380396 0.0482117 ]\n",
      "1        [0.3660313  0.5833739  0.05059479]\n",
      "2        [0.30297568 0.4430802  0.25394407]\n",
      "3        [0.02260253 0.07604674 0.9013506 ]\n",
      "4        [0.6982864  0.27545568 0.02625783]\n",
      "                        ...                \n",
      "66261    [0.03550766 0.82208896 0.14240335]\n",
      "66262    [0.8931986  0.09380516 0.01299612]\n",
      "66263    [0.15655968 0.5201203  0.32332003]\n",
      "66264    [0.91320664 0.06375945 0.02303378]\n",
      "66265    [0.92373    0.06414817 0.01212183]\n",
      "Name: scores, Length: 66266, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# viewing the scores\n",
    "\n",
    "print(df['scores'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4bffd92-882b-4649-892b-54043a235bcb",
   "metadata": {},
   "source": [
    "Upon viewing the scores, it is apparant that there are some--seemingly random--extra spaces. These spaces need to be removed. Once this is done, the string can be split at each space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3f748968-23e9-4506-b70f-cb95c8df2360",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replacing multiple spcaes with a single space\n",
    "\n",
    "df.scores = df.scores.replace(r'\\s+', ' ', regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3c18ba27-634e-406b-aac5-b8aaadbf4497",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0         [0.0679844 0.88380396 0.0482117 ]\n",
      "1          [0.3660313 0.5833739 0.05059479]\n",
      "2         [0.30297568 0.4430802 0.25394407]\n",
      "3        [0.02260253 0.07604674 0.9013506 ]\n",
      "4         [0.6982864 0.27545568 0.02625783]\n",
      "                        ...                \n",
      "66261    [0.03550766 0.82208896 0.14240335]\n",
      "66262     [0.8931986 0.09380516 0.01299612]\n",
      "66263     [0.15655968 0.5201203 0.32332003]\n",
      "66264    [0.91320664 0.06375945 0.02303378]\n",
      "66265       [0.92373 0.06414817 0.01212183]\n",
      "Name: scores, Length: 66266, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# viewing the scores again\n",
    "\n",
    "print(df['scores'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc49b4d-35a4-4beb-9114-dc40686be610",
   "metadata": {},
   "source": [
    "The extra spaces have been removed. Now, the string can be turned into a list, splitting at each space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "a7b2793b-05cb-4100-9db6-ba3a075af2c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.scores=df.scores.str[1:-1].str.split(' ').tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a6334171-c14e-4f81-aefb-34aa36245bf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0         [0.0679844, 0.88380396, 0.0482117, ]\n",
      "1           [0.3660313, 0.5833739, 0.05059479]\n",
      "2          [0.30297568, 0.4430802, 0.25394407]\n",
      "3        [0.02260253, 0.07604674, 0.9013506, ]\n",
      "4          [0.6982864, 0.27545568, 0.02625783]\n",
      "                         ...                  \n",
      "66261     [0.03550766, 0.82208896, 0.14240335]\n",
      "66262      [0.8931986, 0.09380516, 0.01299612]\n",
      "66263      [0.15655968, 0.5201203, 0.32332003]\n",
      "66264     [0.91320664, 0.06375945, 0.02303378]\n",
      "66265        [0.92373, 0.06414817, 0.01212183]\n",
      "Name: scores, Length: 66266, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# viewing the scores again\n",
    "\n",
    "print(df['scores'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5276a4e5-5964-43b9-8b80-4a67006c91ef",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: ['0.0679844', '0.88380396', '0.0482117', ''],\n",
       " 1: ['0.3660313', '0.5833739', '0.05059479'],\n",
       " 2: ['0.30297568', '0.4430802', '0.25394407'],\n",
       " 3: ['0.02260253', '0.07604674', '0.9013506', ''],\n",
       " 4: ['0.6982864', '0.27545568', '0.02625783'],\n",
       " 5: ['0.54107517', '0.37931463', '0.07961021'],\n",
       " 6: ['0.7315472', '0.24781975', '0.02063307'],\n",
       " 7: ['0.6039582', '0.37551644', '0.02052534'],\n",
       " 8: ['0.8756516', '0.11219172', '0.01215654'],\n",
       " 9: ['0.13284624', '0.8060908', '0.06106308'],\n",
       " 10: ['0.4532086', '0.50662094', '0.0401704', ''],\n",
       " 11: ['0.7565991', '0.22713704', '0.01626382'],\n",
       " 12: ['0.7629791', '0.1803103', '0.05671053'],\n",
       " 13: ['0.03224549', '0.91901845', '0.04873606'],\n",
       " 14: ['0.716869', '0.26023984', '0.02289111'],\n",
       " 15: ['0.0262996', '0.13661228', '0.8370881', ''],\n",
       " 16: ['0.02549446', '0.897955', '0.07655057'],\n",
       " 17: ['0.2620131', '0.70915294', '0.02883392'],\n",
       " 18: ['0.01405706', '0.95670676', '0.02923623'],\n",
       " 19: ['0.27915767', '0.7049082', '0.01593406'],\n",
       " 20: ['0.76053226', '0.21511602', '0.02435167']}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the scores as a dictionary\n",
    "\n",
    "df.loc[0:20, \"scores\"].to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1108a70b-54ae-40c9-8fbd-4edd47a21c74",
   "metadata": {},
   "source": [
    "When viewing the scores as a dictionary, there appear to be some extra empty list items at the end. To solve this, I will create 4 new columns--1 for each sentiment and 1 empty (i.e., Negative, Neutral, Positive, empty). The empty column will then be checked to verify that it is indeed empty and consequently be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8b66aa94-2a72-4460-88d5-8ee7801d2163",
   "metadata": {},
   "outputs": [],
   "source": [
    "# separating the list of scores into individual columns and saving it as a new dataframe, df2\n",
    "\n",
    "df2 = pd.DataFrame(df['scores'].to_list(), columns=['Negative', 'Neutral', 'Positive', 'empty'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e126f8f3-4f99-416e-bff8-3885fdfe610a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# merging df2 to df\n",
    "\n",
    "df = pd.concat([df, df2], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1514358b-5917-4685-b9e8-1270691cffdf",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>empty</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844, 0.88380396, 0.0482117, ]</td>\n",
       "      <td>0.0679844</td>\n",
       "      <td>0.88380396</td>\n",
       "      <td>0.0482117</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313, 0.5833739, 0.05059479]</td>\n",
       "      <td>0.3660313</td>\n",
       "      <td>0.5833739</td>\n",
       "      <td>0.05059479</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568, 0.4430802, 0.25394407]</td>\n",
       "      <td>0.30297568</td>\n",
       "      <td>0.4430802</td>\n",
       "      <td>0.25394407</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253, 0.07604674, 0.9013506, ]</td>\n",
       "      <td>0.02260253</td>\n",
       "      <td>0.07604674</td>\n",
       "      <td>0.9013506</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864, 0.27545568, 0.02625783]</td>\n",
       "      <td>0.6982864</td>\n",
       "      <td>0.27545568</td>\n",
       "      <td>0.02625783</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766, 0.82208896, 0.14240335]</td>\n",
       "      <td>0.03550766</td>\n",
       "      <td>0.82208896</td>\n",
       "      <td>0.14240335</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986, 0.09380516, 0.01299612]</td>\n",
       "      <td>0.8931986</td>\n",
       "      <td>0.09380516</td>\n",
       "      <td>0.01299612</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968, 0.5201203, 0.32332003]</td>\n",
       "      <td>0.15655968</td>\n",
       "      <td>0.5201203</td>\n",
       "      <td>0.32332003</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664, 0.06375945, 0.02303378]</td>\n",
       "      <td>0.91320664</td>\n",
       "      <td>0.06375945</td>\n",
       "      <td>0.02303378</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66265</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373, 0.06414817, 0.01212183]</td>\n",
       "      <td>0.92373</td>\n",
       "      <td>0.06414817</td>\n",
       "      <td>0.01212183</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66266 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66261  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66262  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66264  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66265  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66261  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66262  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66263  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66264  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66265  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         1.0   \n",
       "66264                           0.0                         0.0   \n",
       "66265                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66261                        2.0                         0.0   \n",
       "66262                        6.0                         0.0   \n",
       "66263                       17.0                         0.0   \n",
       "66264                        1.0                         0.0   \n",
       "66265                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ... year-month  Language  \\\n",
       "0                                  0.0  ...    2016-04    German   \n",
       "1                                  0.0  ...    2016-04    German   \n",
       "2                                  0.0  ...    2016-04    German   \n",
       "3                                  0.0  ...    2016-04    German   \n",
       "4                                  0.0  ...    2016-04    German   \n",
       "...                                ...  ...        ...       ...   \n",
       "66261                              0.0  ...    2021-06   English   \n",
       "66262                              0.0  ...    2021-06   English   \n",
       "66263                              0.0  ...    2021-06   English   \n",
       "66264                              0.0  ...    2021-06   English   \n",
       "66265                              0.0  ...    2021-06   English   \n",
       "\n",
       "                            date      inflow  \\\n",
       "0      2016-04-20 00:00:00+00:00     Syrians   \n",
       "1      2016-04-20 00:00:00+00:00     Syrians   \n",
       "2      2016-04-20 00:00:00+00:00     Syrians   \n",
       "3      2016-04-20 00:00:00+00:00     Syrians   \n",
       "4      2016-04-20 00:00:00+00:00     Syrians   \n",
       "...                          ...         ...   \n",
       "66261  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66262  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66264  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66265  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66261  For day 1 of week 2, @user discusses the Socia...   \n",
       "66262  @user Europe is racist, but in a different way...   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66264  @user I must have missed the great flight of t...   \n",
       "66265  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                      scores    Negative     Neutral  \\\n",
       "0       [0.0679844, 0.88380396, 0.0482117, ]   0.0679844  0.88380396   \n",
       "1         [0.3660313, 0.5833739, 0.05059479]   0.3660313   0.5833739   \n",
       "2        [0.30297568, 0.4430802, 0.25394407]  0.30297568   0.4430802   \n",
       "3      [0.02260253, 0.07604674, 0.9013506, ]  0.02260253  0.07604674   \n",
       "4        [0.6982864, 0.27545568, 0.02625783]   0.6982864  0.27545568   \n",
       "...                                      ...         ...         ...   \n",
       "66261   [0.03550766, 0.82208896, 0.14240335]  0.03550766  0.82208896   \n",
       "66262    [0.8931986, 0.09380516, 0.01299612]   0.8931986  0.09380516   \n",
       "66263    [0.15655968, 0.5201203, 0.32332003]  0.15655968   0.5201203   \n",
       "66264   [0.91320664, 0.06375945, 0.02303378]  0.91320664  0.06375945   \n",
       "66265      [0.92373, 0.06414817, 0.01212183]     0.92373  0.06414817   \n",
       "\n",
       "         Positive empty  \n",
       "0       0.0482117        \n",
       "1      0.05059479  None  \n",
       "2      0.25394407  None  \n",
       "3       0.9013506        \n",
       "4      0.02625783  None  \n",
       "...           ...   ...  \n",
       "66261  0.14240335  None  \n",
       "66262  0.01299612  None  \n",
       "66263  0.32332003  None  \n",
       "66264  0.02303378  None  \n",
       "66265  0.01212183  None  \n",
       "\n",
       "[66266 rows x 27 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the dataframe\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "81a70a10-f4c6-44ea-ae3c-6d572f24dab3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['' None]\n"
     ]
    }
   ],
   "source": [
    "# checking the unique values of the empty column\n",
    "\n",
    "print(df['empty'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f3a6b69-d628-48cf-b0f6-d2cce9172276",
   "metadata": {},
   "source": [
    "The only values inside the empty column include 'None' or a space (i.e., ' '). Therefore, there are no scores in this column and it can be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "4c0e26bf-a880-4926-9eb7-26a40550585d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(columns=['empty'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "9ef99738-b344-4ed7-94bd-623b40470e43",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844, 0.88380396, 0.0482117, ]</td>\n",
       "      <td>0.0679844</td>\n",
       "      <td>0.88380396</td>\n",
       "      <td>0.0482117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313, 0.5833739, 0.05059479]</td>\n",
       "      <td>0.3660313</td>\n",
       "      <td>0.5833739</td>\n",
       "      <td>0.05059479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568, 0.4430802, 0.25394407]</td>\n",
       "      <td>0.30297568</td>\n",
       "      <td>0.4430802</td>\n",
       "      <td>0.25394407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253, 0.07604674, 0.9013506, ]</td>\n",
       "      <td>0.02260253</td>\n",
       "      <td>0.07604674</td>\n",
       "      <td>0.9013506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864, 0.27545568, 0.02625783]</td>\n",
       "      <td>0.6982864</td>\n",
       "      <td>0.27545568</td>\n",
       "      <td>0.02625783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766, 0.82208896, 0.14240335]</td>\n",
       "      <td>0.03550766</td>\n",
       "      <td>0.82208896</td>\n",
       "      <td>0.14240335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986, 0.09380516, 0.01299612]</td>\n",
       "      <td>0.8931986</td>\n",
       "      <td>0.09380516</td>\n",
       "      <td>0.01299612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968, 0.5201203, 0.32332003]</td>\n",
       "      <td>0.15655968</td>\n",
       "      <td>0.5201203</td>\n",
       "      <td>0.32332003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664, 0.06375945, 0.02303378]</td>\n",
       "      <td>0.91320664</td>\n",
       "      <td>0.06375945</td>\n",
       "      <td>0.02303378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66265</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373, 0.06414817, 0.01212183]</td>\n",
       "      <td>0.92373</td>\n",
       "      <td>0.06414817</td>\n",
       "      <td>0.01212183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66266 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66261  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66262  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66264  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66265  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66261  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66262  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66263  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66264  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66265  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         1.0   \n",
       "66264                           0.0                         0.0   \n",
       "66265                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66261                        2.0                         0.0   \n",
       "66262                        6.0                         0.0   \n",
       "66263                       17.0                         0.0   \n",
       "66264                        1.0                         0.0   \n",
       "66265                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ... year-week  year-month Language  \\\n",
       "0                                  0.0  ...   2016-16     2016-04   German   \n",
       "1                                  0.0  ...   2016-16     2016-04   German   \n",
       "2                                  0.0  ...   2016-16     2016-04   German   \n",
       "3                                  0.0  ...   2016-16     2016-04   German   \n",
       "4                                  0.0  ...   2016-16     2016-04   German   \n",
       "...                                ...  ...       ...         ...      ...   \n",
       "66261                              0.0  ...   2021-26     2021-06  English   \n",
       "66262                              0.0  ...   2021-26     2021-06  English   \n",
       "66263                              0.0  ...   2021-26     2021-06  English   \n",
       "66264                              0.0  ...   2021-25     2021-06  English   \n",
       "66265                              0.0  ...   2021-25     2021-06  English   \n",
       "\n",
       "                            date      inflow  \\\n",
       "0      2016-04-20 00:00:00+00:00     Syrians   \n",
       "1      2016-04-20 00:00:00+00:00     Syrians   \n",
       "2      2016-04-20 00:00:00+00:00     Syrians   \n",
       "3      2016-04-20 00:00:00+00:00     Syrians   \n",
       "4      2016-04-20 00:00:00+00:00     Syrians   \n",
       "...                          ...         ...   \n",
       "66261  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66262  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66264  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66265  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66261  For day 1 of week 2, @user discusses the Socia...   \n",
       "66262  @user Europe is racist, but in a different way...   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66264  @user I must have missed the great flight of t...   \n",
       "66265  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                      scores    Negative     Neutral  \\\n",
       "0       [0.0679844, 0.88380396, 0.0482117, ]   0.0679844  0.88380396   \n",
       "1         [0.3660313, 0.5833739, 0.05059479]   0.3660313   0.5833739   \n",
       "2        [0.30297568, 0.4430802, 0.25394407]  0.30297568   0.4430802   \n",
       "3      [0.02260253, 0.07604674, 0.9013506, ]  0.02260253  0.07604674   \n",
       "4        [0.6982864, 0.27545568, 0.02625783]   0.6982864  0.27545568   \n",
       "...                                      ...         ...         ...   \n",
       "66261   [0.03550766, 0.82208896, 0.14240335]  0.03550766  0.82208896   \n",
       "66262    [0.8931986, 0.09380516, 0.01299612]   0.8931986  0.09380516   \n",
       "66263    [0.15655968, 0.5201203, 0.32332003]  0.15655968   0.5201203   \n",
       "66264   [0.91320664, 0.06375945, 0.02303378]  0.91320664  0.06375945   \n",
       "66265      [0.92373, 0.06414817, 0.01212183]     0.92373  0.06414817   \n",
       "\n",
       "         Positive  \n",
       "0       0.0482117  \n",
       "1      0.05059479  \n",
       "2      0.25394407  \n",
       "3       0.9013506  \n",
       "4      0.02625783  \n",
       "...           ...  \n",
       "66261  0.14240335  \n",
       "66262  0.01299612  \n",
       "66263  0.32332003  \n",
       "66264  0.02303378  \n",
       "66265  0.01212183  \n",
       "\n",
       "[66266 rows x 26 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the dataframe\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3832cd3-735d-4d3b-9845-be637a6d2527",
   "metadata": {},
   "source": [
    "## 5. Creating CLASS column\n",
    "\n",
    "### 5.1. Converting sentiment columns to float\n",
    "\n",
    "#### 5.1.1. Solving error\n",
    "Before creating the CLASS column, the sentiment columns (Negative, Neutral, Positive) need to be converted from string to float. However, intitial attemps to do so revealed that the Negative column has a couple of strange values. Specifically, two entries have merely the letter 'a'. The code below solves this issue and then continues converting sentiment columns to float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "54b3e506-e3b3-4a8f-8495-04ca58b6ce5f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5238    a\n",
       "5239    a\n",
       "Name: Negative, dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# locating which entries have the value 'a' for the Negative column\n",
    "\n",
    "df.loc[df['Negative'] == 'a', 'Negative']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d138c46f-7c72-493f-b18e-295df4efedf7",
   "metadata": {},
   "source": [
    "Indeces 5238 and 5239 have been identified. They will be viewed and dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "39af1a16-6415-4629-b703-5e7e05491b96",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': '❗❗\\r#Letzte #Chance! Noch schnell @freifunkmainz beim Online-Voting für #freies #WLAN für #Flüchtlinge unterstützen https://t.co/OhCeAL77jH\"',\n",
       " 'author_id': 3239113792.0,\n",
       " 'created_at': '2016-01-08 22:52:29+00:00',\n",
       " 'lang': 'de',\n",
       " 'geo.place_id': '8abc99434d4f5d28',\n",
       " 'public_metrics.retweet_count': 0.0,\n",
       " 'public_metrics.reply_count': 1.0,\n",
       " 'public_metrics.like_count': 0.0,\n",
       " 'public_metrics.quote_count': 0.0,\n",
       " 'public_metrics.impression_count': 0.0,\n",
       " 'geo.coordinates.coordinates': '[7.00845283, 50.97120189]',\n",
       " 'in_reply_to_user_id': 176171694.0,\n",
       " 'entities.hashtags': \"[{'start': 3, 'end': 10, 'tag': 'Letzte'}, {'start': 11, 'end': 18, 'tag': 'Chance'}, {'start': 71, 'end': 78, 'tag': 'freies'}, {'start': 79, 'end': 84, 'tag': 'WLAN'}, {'start': 89, 'end': 101, 'tag': 'Flüchtlinge'}]\",\n",
       " 'week': 1.0,\n",
       " 'month': 1.0,\n",
       " 'year': 2016.0,\n",
       " 'year-week': '2016-01',\n",
       " 'year-month': '2016-01',\n",
       " 'Language': 'German',\n",
       " 'date': '2016-01-08 00:00:00+00:00',\n",
       " 'inflow': 'Syrians',\n",
       " 'new text': '❗❗',\n",
       " 'scores': ['a'],\n",
       " 'Negative': 'a',\n",
       " 'Neutral': None,\n",
       " 'Positive': None}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing index 5238\n",
    "\n",
    "df.iloc[5238].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "8d0afc5a-5198-4887-944a-d427e1c39f38",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': '[0.03929918 0.3001709  0.66053   ]',\n",
       " 'author_id': nan,\n",
       " 'created_at': nan,\n",
       " 'lang': nan,\n",
       " 'geo.place_id': nan,\n",
       " 'public_metrics.retweet_count': nan,\n",
       " 'public_metrics.reply_count': nan,\n",
       " 'public_metrics.like_count': nan,\n",
       " 'public_metrics.quote_count': nan,\n",
       " 'public_metrics.impression_count': nan,\n",
       " 'geo.coordinates.coordinates': nan,\n",
       " 'in_reply_to_user_id': nan,\n",
       " 'entities.hashtags': nan,\n",
       " 'week': nan,\n",
       " 'month': nan,\n",
       " 'year': nan,\n",
       " 'year-week': nan,\n",
       " 'year-month': nan,\n",
       " 'Language': nan,\n",
       " 'date': nan,\n",
       " 'inflow': nan,\n",
       " 'new text': nan,\n",
       " 'scores': ['a'],\n",
       " 'Negative': 'a',\n",
       " 'Neutral': None,\n",
       " 'Positive': None}"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing index 5239\n",
    "\n",
    "df.iloc[5239].to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aaf46ba-518e-4480-a833-95239efbd6b4",
   "metadata": {},
   "source": [
    "Both entries reveal nearly identical tweets that do not have any sentiment scores. These need to be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "932b34c1-73e5-47fd-945b-18548ccfb09c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping index 5238\n",
    "\n",
    "df = df.drop(df.index[5238])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a39f65-8bfd-4253-9f58-b15a8a325c8e",
   "metadata": {},
   "source": [
    "**NOTE**: After dropping index 5238, index 5239 moved forward one index, hence the code below using 5238"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b33783f3-f797-4e5b-af78-069854f7afae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping index 5239\n",
    "\n",
    "df = df.drop(df.index[5238])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "946a6ab4-c985-4c7f-999b-4322d79415d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], Name: Negative, dtype: object)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checking which entries still have the value 'a' for the Negative column\n",
    "\n",
    "df.loc[df['Negative'] == 'a', 'Negative']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a26461db-e8e1-47a1-a356-6562bcc39d01",
   "metadata": {},
   "source": [
    "All entries with the value 'a' for the Negative column have been dropped. Conversion of the sentiment columns from string to float can continue now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48aa5bc1-ad4e-421d-8d10-34cf6b8cb41a",
   "metadata": {},
   "source": [
    "#### 5.1.2. Converting Negative to float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "3c6de8c9-56cc-4f1e-ad6d-fb21103a3950",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Negative'] = df['Negative'].astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a17951-e11e-4a9b-a78d-d4999422aae3",
   "metadata": {},
   "source": [
    "#### 5.1.3. Converting Neutral to float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6e167fe4-78fa-4441-9278-afc2c9dff9bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Neutral'] = df['Neutral'].astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c506fa6-6836-45b8-9f3b-ff2099cd6b4d",
   "metadata": {},
   "source": [
    "#### 5.1.4. Converting Positive to float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "8a5ad80d-08ac-4570-acff-07cba9f41882",
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Positive'] = df['Positive'].astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7817f7e7-91aa-4580-90cd-b322285da6a2",
   "metadata": {},
   "source": [
    "### 5.2. Deriving CLASS column from largest sentiment column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "52013572-dd14-4eb3-8ca8-b2f9078c9482",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a new empty column, CLASS\n",
    "\n",
    "df['CLASS'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "7b50e381-aee5-4558-80e9-3281b3435b04",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a function to derive CLASS based on which sentiment column is largest\n",
    "\n",
    "def calc_CLASS(Negative, Neutral, Positive):\n",
    "    if Negative > Neutral and Negative > Positive:\n",
    "        CLASS = 'NEGATIVE'\n",
    "    elif Neutral > Negative and Neutral > Positive:\n",
    "        CLASS = 'NEUTRAL'\n",
    "    elif Positive > Negative and Positive > Neutral:\n",
    "        CLASS = 'POSITIVE'\n",
    "    return CLASS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7d3cba39-bf24-4de0-b83d-18e9ffdf3b6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# applying function\n",
    "\n",
    "df['CLASS'] = df.apply(lambda x: calc_CLASS(x['Negative'], x['Neutral'], x['Positive']), \n",
    "                        axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "276ff1d4-acc2-4685-be37-ae9fd68ad59e",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>CLASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844, 0.88380396, 0.0482117, ]</td>\n",
       "      <td>0.067984</td>\n",
       "      <td>0.883804</td>\n",
       "      <td>0.048212</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313, 0.5833739, 0.05059479]</td>\n",
       "      <td>0.366031</td>\n",
       "      <td>0.583374</td>\n",
       "      <td>0.050595</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568, 0.4430802, 0.25394407]</td>\n",
       "      <td>0.302976</td>\n",
       "      <td>0.443080</td>\n",
       "      <td>0.253944</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253, 0.07604674, 0.9013506, ]</td>\n",
       "      <td>0.022603</td>\n",
       "      <td>0.076047</td>\n",
       "      <td>0.901351</td>\n",
       "      <td>POSITIVE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864, 0.27545568, 0.02625783]</td>\n",
       "      <td>0.698286</td>\n",
       "      <td>0.275456</td>\n",
       "      <td>0.026258</td>\n",
       "      <td>NEGATIVE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766, 0.82208896, 0.14240335]</td>\n",
       "      <td>0.035508</td>\n",
       "      <td>0.822089</td>\n",
       "      <td>0.142403</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986, 0.09380516, 0.01299612]</td>\n",
       "      <td>0.893199</td>\n",
       "      <td>0.093805</td>\n",
       "      <td>0.012996</td>\n",
       "      <td>NEGATIVE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968, 0.5201203, 0.32332003]</td>\n",
       "      <td>0.156560</td>\n",
       "      <td>0.520120</td>\n",
       "      <td>0.323320</td>\n",
       "      <td>NEUTRAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664, 0.06375945, 0.02303378]</td>\n",
       "      <td>0.913207</td>\n",
       "      <td>0.063759</td>\n",
       "      <td>0.023034</td>\n",
       "      <td>NEGATIVE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66265</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373, 0.06414817, 0.01212183]</td>\n",
       "      <td>0.923730</td>\n",
       "      <td>0.064148</td>\n",
       "      <td>0.012122</td>\n",
       "      <td>NEGATIVE</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66264 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66261  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66262  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66264  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66265  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66261  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66262  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66263  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66264  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66265  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         1.0   \n",
       "66264                           0.0                         0.0   \n",
       "66265                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66261                        2.0                         0.0   \n",
       "66262                        6.0                         0.0   \n",
       "66263                       17.0                         0.0   \n",
       "66264                        1.0                         0.0   \n",
       "66265                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ... year-month  Language  \\\n",
       "0                                  0.0  ...    2016-04    German   \n",
       "1                                  0.0  ...    2016-04    German   \n",
       "2                                  0.0  ...    2016-04    German   \n",
       "3                                  0.0  ...    2016-04    German   \n",
       "4                                  0.0  ...    2016-04    German   \n",
       "...                                ...  ...        ...       ...   \n",
       "66261                              0.0  ...    2021-06   English   \n",
       "66262                              0.0  ...    2021-06   English   \n",
       "66263                              0.0  ...    2021-06   English   \n",
       "66264                              0.0  ...    2021-06   English   \n",
       "66265                              0.0  ...    2021-06   English   \n",
       "\n",
       "                            date      inflow  \\\n",
       "0      2016-04-20 00:00:00+00:00     Syrians   \n",
       "1      2016-04-20 00:00:00+00:00     Syrians   \n",
       "2      2016-04-20 00:00:00+00:00     Syrians   \n",
       "3      2016-04-20 00:00:00+00:00     Syrians   \n",
       "4      2016-04-20 00:00:00+00:00     Syrians   \n",
       "...                          ...         ...   \n",
       "66261  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66262  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66264  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66265  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66261  For day 1 of week 2, @user discusses the Socia...   \n",
       "66262  @user Europe is racist, but in a different way...   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66264  @user I must have missed the great flight of t...   \n",
       "66265  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                      scores  Negative   Neutral  Positive  \\\n",
       "0       [0.0679844, 0.88380396, 0.0482117, ]  0.067984  0.883804  0.048212   \n",
       "1         [0.3660313, 0.5833739, 0.05059479]  0.366031  0.583374  0.050595   \n",
       "2        [0.30297568, 0.4430802, 0.25394407]  0.302976  0.443080  0.253944   \n",
       "3      [0.02260253, 0.07604674, 0.9013506, ]  0.022603  0.076047  0.901351   \n",
       "4        [0.6982864, 0.27545568, 0.02625783]  0.698286  0.275456  0.026258   \n",
       "...                                      ...       ...       ...       ...   \n",
       "66261   [0.03550766, 0.82208896, 0.14240335]  0.035508  0.822089  0.142403   \n",
       "66262    [0.8931986, 0.09380516, 0.01299612]  0.893199  0.093805  0.012996   \n",
       "66263    [0.15655968, 0.5201203, 0.32332003]  0.156560  0.520120  0.323320   \n",
       "66264   [0.91320664, 0.06375945, 0.02303378]  0.913207  0.063759  0.023034   \n",
       "66265      [0.92373, 0.06414817, 0.01212183]  0.923730  0.064148  0.012122   \n",
       "\n",
       "          CLASS  \n",
       "0       NEUTRAL  \n",
       "1       NEUTRAL  \n",
       "2       NEUTRAL  \n",
       "3      POSITIVE  \n",
       "4      NEGATIVE  \n",
       "...         ...  \n",
       "66261   NEUTRAL  \n",
       "66262  NEGATIVE  \n",
       "66263   NEUTRAL  \n",
       "66264  NEGATIVE  \n",
       "66265  NEGATIVE  \n",
       "\n",
       "[66264 rows x 27 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing dataframe\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a96a3cc-4f6e-46f3-a80a-f2b539c8d191",
   "metadata": {},
   "source": [
    "## 6. Creating num_CLASS column\n",
    "\n",
    "In addition to a categorical class column, CLASS, a numerical class column, num_CLASS, will be created. The value of num_CLASS will be:\n",
    "* -1 if the tweet is most likely negative\n",
    "* 0 if the tweet is most likely neutral\n",
    "* 1 if the tweet is most likely positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "22c03b20-93cf-4e86-b465-fbf58a4d7ed0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a new empty column, num_CLASS\n",
    "\n",
    "df['num_CLASS'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "78f6a840-79bd-4cfd-9a67-d4b644b1ce60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a function to derive num_CLASS based on which sentiment column is largest\n",
    "\n",
    "def calc_CLASS(Negative, Neutral, Positive):\n",
    "    if Negative > Neutral and Negative > Positive:\n",
    "        num_CLASS = -1\n",
    "    elif Neutral > Negative and Neutral > Positive:\n",
    "        num_CLASS = 0\n",
    "    elif Positive > Negative and Positive > Neutral:\n",
    "        num_CLASS = 1\n",
    "    return num_CLASS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "a362fefd-33ed-4624-b4c4-ba4ea284caef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# applying function\n",
    "\n",
    "df['num_CLASS'] = df.apply(lambda x: calc_CLASS(x['Negative'], x['Neutral'], x['Positive']), \n",
    "                        axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "86a10946-7bc6-4ed6-9b7a-cdb934a1a9a1",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>CLASS</th>\n",
       "      <th>num_CLASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4.122038e+09</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>[0.0679844, 0.88380396, 0.0482117, ]</td>\n",
       "      <td>0.067984</td>\n",
       "      <td>0.883804</td>\n",
       "      <td>0.048212</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1.179544e+09</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>[0.3660313, 0.5833739, 0.05059479]</td>\n",
       "      <td>0.366031</td>\n",
       "      <td>0.583374</td>\n",
       "      <td>0.050595</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>2.246076e+08</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>[0.30297568, 0.4430802, 0.25394407]</td>\n",
       "      <td>0.302976</td>\n",
       "      <td>0.443080</td>\n",
       "      <td>0.253944</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2.480764e+09</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>[0.02260253, 0.07604674, 0.9013506, ]</td>\n",
       "      <td>0.022603</td>\n",
       "      <td>0.076047</td>\n",
       "      <td>0.901351</td>\n",
       "      <td>POSITIVE</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>6.062653e+08</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>[0.6982864, 0.27545568, 0.02625783]</td>\n",
       "      <td>0.698286</td>\n",
       "      <td>0.275456</td>\n",
       "      <td>0.026258</td>\n",
       "      <td>NEGATIVE</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
       "      <td>[0.03550766, 0.82208896, 0.14240335]</td>\n",
       "      <td>0.035508</td>\n",
       "      <td>0.822089</td>\n",
       "      <td>0.142403</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2.521809e+09</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user Europe is racist, but in a different way...</td>\n",
       "      <td>[0.8931986, 0.09380516, 0.01299612]</td>\n",
       "      <td>0.893199</td>\n",
       "      <td>0.093805</td>\n",
       "      <td>0.012996</td>\n",
       "      <td>NEGATIVE</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>[0.15655968, 0.5201203, 0.32332003]</td>\n",
       "      <td>0.156560</td>\n",
       "      <td>0.520120</td>\n",
       "      <td>0.323320</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2.591892e+09</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>@user I must have missed the great flight of t...</td>\n",
       "      <td>[0.91320664, 0.06375945, 0.02303378]</td>\n",
       "      <td>0.913207</td>\n",
       "      <td>0.063759</td>\n",
       "      <td>0.023034</td>\n",
       "      <td>NEGATIVE</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66265</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1.344017e+08</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>[0.92373, 0.06414817, 0.01212183]</td>\n",
       "      <td>0.923730</td>\n",
       "      <td>0.064148</td>\n",
       "      <td>0.012122</td>\n",
       "      <td>NEGATIVE</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66264 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text     author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
       "...                                                  ...           ...   \n",
       "66261  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
       "66262  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
       "66264  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
       "66265  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
       "\n",
       "                      created_at lang      geo.place_id  \\\n",
       "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                          ...  ...               ...   \n",
       "66261  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66262  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66263  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66264  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66265  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         1.0   \n",
       "66264                           0.0                         0.0   \n",
       "66265                           8.0                         6.0   \n",
       "\n",
       "       public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                            0.0                         0.0   \n",
       "1                            0.0                         0.0   \n",
       "2                            0.0                         0.0   \n",
       "3                            4.0                         0.0   \n",
       "4                            0.0                         0.0   \n",
       "...                          ...                         ...   \n",
       "66261                        2.0                         0.0   \n",
       "66262                        6.0                         0.0   \n",
       "66263                       17.0                         0.0   \n",
       "66264                        1.0                         0.0   \n",
       "66265                       38.0                         6.0   \n",
       "\n",
       "       public_metrics.impression_count  ... Language  \\\n",
       "0                                  0.0  ...   German   \n",
       "1                                  0.0  ...   German   \n",
       "2                                  0.0  ...   German   \n",
       "3                                  0.0  ...   German   \n",
       "4                                  0.0  ...   German   \n",
       "...                                ...  ...      ...   \n",
       "66261                              0.0  ...  English   \n",
       "66262                              0.0  ...  English   \n",
       "66263                              0.0  ...  English   \n",
       "66264                              0.0  ...  English   \n",
       "66265                              0.0  ...  English   \n",
       "\n",
       "                            date      inflow  \\\n",
       "0      2016-04-20 00:00:00+00:00     Syrians   \n",
       "1      2016-04-20 00:00:00+00:00     Syrians   \n",
       "2      2016-04-20 00:00:00+00:00     Syrians   \n",
       "3      2016-04-20 00:00:00+00:00     Syrians   \n",
       "4      2016-04-20 00:00:00+00:00     Syrians   \n",
       "...                          ...         ...   \n",
       "66261  2021-06-28 00:00:00+00:00  Ukrainians   \n",
       "66262  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66263  2021-06-27 00:00:00+00:00  Ukrainians   \n",
       "66264  2021-06-26 00:00:00+00:00  Ukrainians   \n",
       "66265  2021-06-25 00:00:00+00:00  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66261  For day 1 of week 2, @user discusses the Socia...   \n",
       "66262  @user Europe is racist, but in a different way...   \n",
       "66263  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66264  @user I must have missed the great flight of t...   \n",
       "66265  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "                                      scores  Negative   Neutral  Positive  \\\n",
       "0       [0.0679844, 0.88380396, 0.0482117, ]  0.067984  0.883804  0.048212   \n",
       "1         [0.3660313, 0.5833739, 0.05059479]  0.366031  0.583374  0.050595   \n",
       "2        [0.30297568, 0.4430802, 0.25394407]  0.302976  0.443080  0.253944   \n",
       "3      [0.02260253, 0.07604674, 0.9013506, ]  0.022603  0.076047  0.901351   \n",
       "4        [0.6982864, 0.27545568, 0.02625783]  0.698286  0.275456  0.026258   \n",
       "...                                      ...       ...       ...       ...   \n",
       "66261   [0.03550766, 0.82208896, 0.14240335]  0.035508  0.822089  0.142403   \n",
       "66262    [0.8931986, 0.09380516, 0.01299612]  0.893199  0.093805  0.012996   \n",
       "66263    [0.15655968, 0.5201203, 0.32332003]  0.156560  0.520120  0.323320   \n",
       "66264   [0.91320664, 0.06375945, 0.02303378]  0.913207  0.063759  0.023034   \n",
       "66265      [0.92373, 0.06414817, 0.01212183]  0.923730  0.064148  0.012122   \n",
       "\n",
       "          CLASS num_CLASS  \n",
       "0       NEUTRAL         0  \n",
       "1       NEUTRAL         0  \n",
       "2       NEUTRAL         0  \n",
       "3      POSITIVE         1  \n",
       "4      NEGATIVE        -1  \n",
       "...         ...       ...  \n",
       "66261   NEUTRAL         0  \n",
       "66262  NEGATIVE        -1  \n",
       "66263   NEUTRAL         0  \n",
       "66264  NEGATIVE        -1  \n",
       "66265  NEGATIVE        -1  \n",
       "\n",
       "[66264 rows x 28 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the dataframe\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2791fdf-705f-49cd-9c93-65ffc5fae5d2",
   "metadata": {},
   "source": [
    "## 7. Handling nulls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "024882d2-dcb4-4ea3-a1f7-05723e0f48fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 66264 entries, 0 to 66265\n",
      "Data columns (total 28 columns):\n",
      " #   Column                           Non-Null Count  Dtype  \n",
      "---  ------                           --------------  -----  \n",
      " 0   text                             66263 non-null  object \n",
      " 1   author_id                        66261 non-null  float64\n",
      " 2   created_at                       38611 non-null  object \n",
      " 3   lang                             66261 non-null  object \n",
      " 4   geo.place_id                     38225 non-null  object \n",
      " 5   public_metrics.retweet_count     38611 non-null  float64\n",
      " 6   public_metrics.reply_count       38611 non-null  float64\n",
      " 7   public_metrics.like_count        38611 non-null  float64\n",
      " 8   public_metrics.quote_count       38611 non-null  float64\n",
      " 9   public_metrics.impression_count  38611 non-null  float64\n",
      " 10  geo.coordinates.coordinates      4347 non-null   object \n",
      " 11  in_reply_to_user_id              12325 non-null  float64\n",
      " 12  entities.hashtags                15559 non-null  object \n",
      " 13  week                             38611 non-null  float64\n",
      " 14  month                            38611 non-null  float64\n",
      " 15  year                             38611 non-null  float64\n",
      " 16  year-week                        38611 non-null  object \n",
      " 17  year-month                       38611 non-null  object \n",
      " 18  Language                         66264 non-null  object \n",
      " 19  date                             38611 non-null  object \n",
      " 20  inflow                           66264 non-null  object \n",
      " 21  new text                         66263 non-null  object \n",
      " 22  scores                           66264 non-null  object \n",
      " 23  Negative                         66264 non-null  float64\n",
      " 24  Neutral                          66264 non-null  float64\n",
      " 25  Positive                         66264 non-null  float64\n",
      " 26  CLASS                            66264 non-null  object \n",
      " 27  num_CLASS                        66264 non-null  int64  \n",
      "dtypes: float64(13), int64(1), object(14)\n",
      "memory usage: 14.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "4f410e41-4e0d-4db3-83f8-434ac194184b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27653\n"
     ]
    }
   ],
   "source": [
    "print(66264-38611)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f095744-aa08-4c99-9c36-90d359df828f",
   "metadata": {},
   "source": [
    "There are 27,653 rows with a missing entry for \"created_at\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "03d28ea9-52e1-4cf8-8331-d4728e160602",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                    text   author_id  \\\n",
      "8642                                               @N24          NaN   \n",
      "8643                                                 NaN         NaN   \n",
      "13732  #refugeeswelcome #1209HH #Flüchtling #Syrien #...         NaN   \n",
      "29750  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "29751  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "...                                                  ...         ...   \n",
      "57395  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "57396  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "57397  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "57398  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "57399  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
      "\n",
      "      created_at lang geo.place_id  public_metrics.retweet_count  \\\n",
      "8642         NaN  NaN          NaN                           NaN   \n",
      "8643         NaN  NaN          NaN                           NaN   \n",
      "13732        NaN  NaN          NaN                           NaN   \n",
      "29750        NaN   de          NaN                           NaN   \n",
      "29751        NaN   de          NaN                           NaN   \n",
      "...          ...  ...          ...                           ...   \n",
      "57395        NaN   de          NaN                           NaN   \n",
      "57396        NaN   de          NaN                           NaN   \n",
      "57397        NaN   de          NaN                           NaN   \n",
      "57398        NaN   de          NaN                           NaN   \n",
      "57399        NaN   de          NaN                           NaN   \n",
      "\n",
      "       public_metrics.reply_count  public_metrics.like_count  \\\n",
      "8642                          NaN                        NaN   \n",
      "8643                          NaN                        NaN   \n",
      "13732                         NaN                        NaN   \n",
      "29750                         NaN                        NaN   \n",
      "29751                         NaN                        NaN   \n",
      "...                           ...                        ...   \n",
      "57395                         NaN                        NaN   \n",
      "57396                         NaN                        NaN   \n",
      "57397                         NaN                        NaN   \n",
      "57398                         NaN                        NaN   \n",
      "57399                         NaN                        NaN   \n",
      "\n",
      "       public_metrics.quote_count  public_metrics.impression_count  ...  \\\n",
      "8642                          NaN                              NaN  ...   \n",
      "8643                          NaN                              NaN  ...   \n",
      "13732                         NaN                              NaN  ...   \n",
      "29750                         NaN                              NaN  ...   \n",
      "29751                         NaN                              NaN  ...   \n",
      "...                           ...                              ...  ...   \n",
      "57395                         NaN                              NaN  ...   \n",
      "57396                         NaN                              NaN  ...   \n",
      "57397                         NaN                              NaN  ...   \n",
      "57398                         NaN                              NaN  ...   \n",
      "57399                         NaN                              NaN  ...   \n",
      "\n",
      "      Language  date      inflow  \\\n",
      "8642    German   NaN     Syrians   \n",
      "8643    German   NaN     Syrians   \n",
      "13732   German   NaN     Syrians   \n",
      "29750   German   NaN  Ukrainians   \n",
      "29751   German   NaN  Ukrainians   \n",
      "...        ...   ...         ...   \n",
      "57395   German   NaN  Ukrainians   \n",
      "57396   German   NaN  Ukrainians   \n",
      "57397   German   NaN  Ukrainians   \n",
      "57398   German   NaN  Ukrainians   \n",
      "57399   German   NaN  Ukrainians   \n",
      "\n",
      "                                                new text  \\\n",
      "8642                                              @user    \n",
      "8643                                                 NaN   \n",
      "13732  #refugeeswelcome #1209HH #Flüchtling #Syrien #...   \n",
      "29750  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "29751  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "...                                                  ...   \n",
      "57395  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "57396  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "57397  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "57398  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "57399  Die Belarus-Route ist nach wie vor offen via @...   \n",
      "\n",
      "                                      scores  Negative   Neutral  Positive  \\\n",
      "8642    [0.23433569, 0.4841369, 0.2815274, ]  0.234336  0.484137  0.281527   \n",
      "8643   [0.34819144, 0.39594722, 0.2558613, ]  0.348191  0.395947  0.255861   \n",
      "13732    [0.11344611, 0.7992048, 0.08734913]  0.113446  0.799205  0.087349   \n",
      "29750     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "29751     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "...                                      ...       ...       ...       ...   \n",
      "57395     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "57396     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "57397     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "57398     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "57399     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
      "\n",
      "         CLASS num_CLASS  \n",
      "8642   NEUTRAL         0  \n",
      "8643   NEUTRAL         0  \n",
      "13732  NEUTRAL         0  \n",
      "29750  NEUTRAL         0  \n",
      "29751  NEUTRAL         0  \n",
      "...        ...       ...  \n",
      "57395  NEUTRAL         0  \n",
      "57396  NEUTRAL         0  \n",
      "57397  NEUTRAL         0  \n",
      "57398  NEUTRAL         0  \n",
      "57399  NEUTRAL         0  \n",
      "\n",
      "[27653 rows x 28 columns]\n"
     ]
    }
   ],
   "source": [
    "# use isnull() function to check for null values in the 'created_at' column\n",
    "\n",
    "null_rows = df[df['created_at'].isnull()]\n",
    "\n",
    "# print the rows with null values in the 'created_at' column\n",
    "\n",
    "print(null_rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "c9f03375-ee14-4bd8-a02c-b47e6286855e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new dataframe with only null values in the 'created_at' column\n",
    "\n",
    "null_rows_df = df[df['created_at'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "57c9e3f7-ff0b-4d9e-821e-d42d7b2bbb82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>CLASS</th>\n",
       "      <th>num_CLASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8642</th>\n",
       "      <td>@N24</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>@user</td>\n",
       "      <td>[0.23433569, 0.4841369, 0.2815274, ]</td>\n",
       "      <td>0.234336</td>\n",
       "      <td>0.484137</td>\n",
       "      <td>0.281527</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8643</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[0.34819144, 0.39594722, 0.2558613, ]</td>\n",
       "      <td>0.348191</td>\n",
       "      <td>0.395947</td>\n",
       "      <td>0.255861</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13732</th>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>[0.11344611, 0.7992048, 0.08734913]</td>\n",
       "      <td>0.113446</td>\n",
       "      <td>0.799205</td>\n",
       "      <td>0.087349</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29750</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29751</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57395</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57396</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57397</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57398</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57399</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27653 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text   author_id  \\\n",
       "8642                                               @N24          NaN   \n",
       "8643                                                 NaN         NaN   \n",
       "13732  #refugeeswelcome #1209HH #Flüchtling #Syrien #...         NaN   \n",
       "29750  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "29751  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "...                                                  ...         ...   \n",
       "57395  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "57396  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "57397  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "57398  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "57399  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "\n",
       "      created_at lang geo.place_id  public_metrics.retweet_count  \\\n",
       "8642         NaN  NaN          NaN                           NaN   \n",
       "8643         NaN  NaN          NaN                           NaN   \n",
       "13732        NaN  NaN          NaN                           NaN   \n",
       "29750        NaN   de          NaN                           NaN   \n",
       "29751        NaN   de          NaN                           NaN   \n",
       "...          ...  ...          ...                           ...   \n",
       "57395        NaN   de          NaN                           NaN   \n",
       "57396        NaN   de          NaN                           NaN   \n",
       "57397        NaN   de          NaN                           NaN   \n",
       "57398        NaN   de          NaN                           NaN   \n",
       "57399        NaN   de          NaN                           NaN   \n",
       "\n",
       "       public_metrics.reply_count  public_metrics.like_count  \\\n",
       "8642                          NaN                        NaN   \n",
       "8643                          NaN                        NaN   \n",
       "13732                         NaN                        NaN   \n",
       "29750                         NaN                        NaN   \n",
       "29751                         NaN                        NaN   \n",
       "...                           ...                        ...   \n",
       "57395                         NaN                        NaN   \n",
       "57396                         NaN                        NaN   \n",
       "57397                         NaN                        NaN   \n",
       "57398                         NaN                        NaN   \n",
       "57399                         NaN                        NaN   \n",
       "\n",
       "       public_metrics.quote_count  public_metrics.impression_count  ...  \\\n",
       "8642                          NaN                              NaN  ...   \n",
       "8643                          NaN                              NaN  ...   \n",
       "13732                         NaN                              NaN  ...   \n",
       "29750                         NaN                              NaN  ...   \n",
       "29751                         NaN                              NaN  ...   \n",
       "...                           ...                              ...  ...   \n",
       "57395                         NaN                              NaN  ...   \n",
       "57396                         NaN                              NaN  ...   \n",
       "57397                         NaN                              NaN  ...   \n",
       "57398                         NaN                              NaN  ...   \n",
       "57399                         NaN                              NaN  ...   \n",
       "\n",
       "      Language  date      inflow  \\\n",
       "8642    German   NaN     Syrians   \n",
       "8643    German   NaN     Syrians   \n",
       "13732   German   NaN     Syrians   \n",
       "29750   German   NaN  Ukrainians   \n",
       "29751   German   NaN  Ukrainians   \n",
       "...        ...   ...         ...   \n",
       "57395   German   NaN  Ukrainians   \n",
       "57396   German   NaN  Ukrainians   \n",
       "57397   German   NaN  Ukrainians   \n",
       "57398   German   NaN  Ukrainians   \n",
       "57399   German   NaN  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "8642                                              @user    \n",
       "8643                                                 NaN   \n",
       "13732  #refugeeswelcome #1209HH #Flüchtling #Syrien #...   \n",
       "29750  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "29751  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "...                                                  ...   \n",
       "57395  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "57396  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "57397  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "57398  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "57399  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "\n",
       "                                      scores  Negative   Neutral  Positive  \\\n",
       "8642    [0.23433569, 0.4841369, 0.2815274, ]  0.234336  0.484137  0.281527   \n",
       "8643   [0.34819144, 0.39594722, 0.2558613, ]  0.348191  0.395947  0.255861   \n",
       "13732    [0.11344611, 0.7992048, 0.08734913]  0.113446  0.799205  0.087349   \n",
       "29750     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "29751     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "...                                      ...       ...       ...       ...   \n",
       "57395     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "57396     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "57397     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "57398     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "57399     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "\n",
       "         CLASS num_CLASS  \n",
       "8642   NEUTRAL         0  \n",
       "8643   NEUTRAL         0  \n",
       "13732  NEUTRAL         0  \n",
       "29750  NEUTRAL         0  \n",
       "29751  NEUTRAL         0  \n",
       "...        ...       ...  \n",
       "57395  NEUTRAL         0  \n",
       "57396  NEUTRAL         0  \n",
       "57397  NEUTRAL         0  \n",
       "57398  NEUTRAL         0  \n",
       "57399  NEUTRAL         0  \n",
       "\n",
       "[27653 rows x 28 columns]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "null_rows_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "4a03a247-31f1-4a8b-8896-427ead51b565",
   "metadata": {},
   "outputs": [],
   "source": [
    "null_rows_df = null_rows_df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "ef2ca2a5-ec74-44ea-988e-aa5891a1c956",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>CLASS</th>\n",
       "      <th>num_CLASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@N24</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>@user</td>\n",
       "      <td>[0.23433569, 0.4841369, 0.2815274, ]</td>\n",
       "      <td>0.234336</td>\n",
       "      <td>0.484137</td>\n",
       "      <td>0.281527</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[0.34819144, 0.39594722, 0.2558613, ]</td>\n",
       "      <td>0.348191</td>\n",
       "      <td>0.395947</td>\n",
       "      <td>0.255861</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>[0.11344611, 0.7992048, 0.08734913]</td>\n",
       "      <td>0.113446</td>\n",
       "      <td>0.799205</td>\n",
       "      <td>0.087349</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27648</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27649</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27650</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27651</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27652</th>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>19836074.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "      <td>Die Belarus-Route ist nach wie vor offen via @...</td>\n",
       "      <td>[0.0282606, 0.7912057, 0.18053377]</td>\n",
       "      <td>0.028261</td>\n",
       "      <td>0.791206</td>\n",
       "      <td>0.180534</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27653 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text   author_id  \\\n",
       "0                                                  @N24          NaN   \n",
       "1                                                    NaN         NaN   \n",
       "2      #refugeeswelcome #1209HH #Flüchtling #Syrien #...         NaN   \n",
       "3      Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "4      Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "...                                                  ...         ...   \n",
       "27648  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "27649  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "27650  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "27651  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "27652  Die Belarus-Route ist nach wie vor offen via @...  19836074.0   \n",
       "\n",
       "      created_at lang geo.place_id  public_metrics.retweet_count  \\\n",
       "0            NaN  NaN          NaN                           NaN   \n",
       "1            NaN  NaN          NaN                           NaN   \n",
       "2            NaN  NaN          NaN                           NaN   \n",
       "3            NaN   de          NaN                           NaN   \n",
       "4            NaN   de          NaN                           NaN   \n",
       "...          ...  ...          ...                           ...   \n",
       "27648        NaN   de          NaN                           NaN   \n",
       "27649        NaN   de          NaN                           NaN   \n",
       "27650        NaN   de          NaN                           NaN   \n",
       "27651        NaN   de          NaN                           NaN   \n",
       "27652        NaN   de          NaN                           NaN   \n",
       "\n",
       "       public_metrics.reply_count  public_metrics.like_count  \\\n",
       "0                             NaN                        NaN   \n",
       "1                             NaN                        NaN   \n",
       "2                             NaN                        NaN   \n",
       "3                             NaN                        NaN   \n",
       "4                             NaN                        NaN   \n",
       "...                           ...                        ...   \n",
       "27648                         NaN                        NaN   \n",
       "27649                         NaN                        NaN   \n",
       "27650                         NaN                        NaN   \n",
       "27651                         NaN                        NaN   \n",
       "27652                         NaN                        NaN   \n",
       "\n",
       "       public_metrics.quote_count  public_metrics.impression_count  ...  \\\n",
       "0                             NaN                              NaN  ...   \n",
       "1                             NaN                              NaN  ...   \n",
       "2                             NaN                              NaN  ...   \n",
       "3                             NaN                              NaN  ...   \n",
       "4                             NaN                              NaN  ...   \n",
       "...                           ...                              ...  ...   \n",
       "27648                         NaN                              NaN  ...   \n",
       "27649                         NaN                              NaN  ...   \n",
       "27650                         NaN                              NaN  ...   \n",
       "27651                         NaN                              NaN  ...   \n",
       "27652                         NaN                              NaN  ...   \n",
       "\n",
       "      Language  date      inflow  \\\n",
       "0       German   NaN     Syrians   \n",
       "1       German   NaN     Syrians   \n",
       "2       German   NaN     Syrians   \n",
       "3       German   NaN  Ukrainians   \n",
       "4       German   NaN  Ukrainians   \n",
       "...        ...   ...         ...   \n",
       "27648   German   NaN  Ukrainians   \n",
       "27649   German   NaN  Ukrainians   \n",
       "27650   German   NaN  Ukrainians   \n",
       "27651   German   NaN  Ukrainians   \n",
       "27652   German   NaN  Ukrainians   \n",
       "\n",
       "                                                new text  \\\n",
       "0                                                 @user    \n",
       "1                                                    NaN   \n",
       "2      #refugeeswelcome #1209HH #Flüchtling #Syrien #...   \n",
       "3      Die Belarus-Route ist nach wie vor offen via @...   \n",
       "4      Die Belarus-Route ist nach wie vor offen via @...   \n",
       "...                                                  ...   \n",
       "27648  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "27649  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "27650  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "27651  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "27652  Die Belarus-Route ist nach wie vor offen via @...   \n",
       "\n",
       "                                      scores  Negative   Neutral  Positive  \\\n",
       "0       [0.23433569, 0.4841369, 0.2815274, ]  0.234336  0.484137  0.281527   \n",
       "1      [0.34819144, 0.39594722, 0.2558613, ]  0.348191  0.395947  0.255861   \n",
       "2        [0.11344611, 0.7992048, 0.08734913]  0.113446  0.799205  0.087349   \n",
       "3         [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "4         [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "...                                      ...       ...       ...       ...   \n",
       "27648     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "27649     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "27650     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "27651     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "27652     [0.0282606, 0.7912057, 0.18053377]  0.028261  0.791206  0.180534   \n",
       "\n",
       "         CLASS num_CLASS  \n",
       "0      NEUTRAL         0  \n",
       "1      NEUTRAL         0  \n",
       "2      NEUTRAL         0  \n",
       "3      NEUTRAL         0  \n",
       "4      NEUTRAL         0  \n",
       "...        ...       ...  \n",
       "27648  NEUTRAL         0  \n",
       "27649  NEUTRAL         0  \n",
       "27650  NEUTRAL         0  \n",
       "27651  NEUTRAL         0  \n",
       "27652  NEUTRAL         0  \n",
       "\n",
       "[27653 rows x 28 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "null_rows_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aef57da-848c-48bf-ab6e-7cee9430f7c4",
   "metadata": {},
   "source": [
    "There is at least one row (index 1) that contains no text. I will now check if there are others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "86640f3d-9d64-4a49-990f-667afc94a824",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  text  author_id created_at lang geo.place_id  public_metrics.retweet_count  \\\n",
      "1  NaN        NaN        NaN  NaN          NaN                           NaN   \n",
      "\n",
      "   public_metrics.reply_count  public_metrics.like_count  \\\n",
      "1                         NaN                        NaN   \n",
      "\n",
      "   public_metrics.quote_count  public_metrics.impression_count  ... Language  \\\n",
      "1                         NaN                              NaN  ...   German   \n",
      "\n",
      "   date   inflow  new text                                 scores  Negative  \\\n",
      "1   NaN  Syrians       NaN  [0.34819144, 0.39594722, 0.2558613, ]  0.348191   \n",
      "\n",
      "    Neutral  Positive    CLASS num_CLASS  \n",
      "1  0.395947  0.255861  NEUTRAL         0  \n",
      "\n",
      "[1 rows x 28 columns]\n"
     ]
    }
   ],
   "source": [
    "# use isnull() function to check for null values in the 'text' column\n",
    "null_text = null_rows_df[null_rows_df['text'].isnull()]\n",
    "\n",
    "# print the rows with null values in the 'created_at' column\n",
    "print(null_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "0bbf33a4-da44-4fe9-a1d7-3ae93df495aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@N24 \n"
     ]
    }
   ],
   "source": [
    "# view the \"text\" entry for the row with index 0\n",
    "\n",
    "print(null_rows_df.loc[0, 'text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a893d8-c94f-4f18-bac8-1f95e26ea656",
   "metadata": {},
   "source": [
    "Index 0 contains essentially no text either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "10c35371-7726-4ac5-a754-971565d38b36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#refugeeswelcome #1209HH #Flüchtling #Syrien #Wilhelmsburg \n"
     ]
    }
   ],
   "source": [
    "# view the \"text\" entry for the row with index 2\n",
    "\n",
    "print(null_rows_df.loc[2, 'text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "d70545bb-61ae-42e6-9dfd-81e430ce15db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text                               #refugeeswelcome #1209HH #Flüchtling #Syrien #...\n",
      "author_id                                                                        NaN\n",
      "created_at                                                                       NaN\n",
      "lang                                                                             NaN\n",
      "geo.place_id                                                                     NaN\n",
      "public_metrics.retweet_count                                                     NaN\n",
      "public_metrics.reply_count                                                       NaN\n",
      "public_metrics.like_count                                                        NaN\n",
      "public_metrics.quote_count                                                       NaN\n",
      "public_metrics.impression_count                                                  NaN\n",
      "geo.coordinates.coordinates                                                      NaN\n",
      "in_reply_to_user_id                                                              NaN\n",
      "entities.hashtags                                                                NaN\n",
      "week                                                                             NaN\n",
      "month                                                                            NaN\n",
      "year                                                                             NaN\n",
      "year-week                                                                        NaN\n",
      "year-month                                                                       NaN\n",
      "Language                                                                      German\n",
      "date                                                                             NaN\n",
      "inflow                                                                       Syrians\n",
      "new text                           #refugeeswelcome #1209HH #Flüchtling #Syrien #...\n",
      "scores                                           [0.11344611, 0.7992048, 0.08734913]\n",
      "Negative                                                                    0.113446\n",
      "Neutral                                                                     0.799205\n",
      "Positive                                                                    0.087349\n",
      "CLASS                                                                        NEUTRAL\n",
      "num_CLASS                                                                          0\n",
      "Name: 2, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(null_rows_df.iloc[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "90085588-d59d-411c-a19f-6213753d5aa2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Die Belarus-Route ist nach wie vor offen via @RND_de\n"
     ]
    }
   ],
   "source": [
    "# view the \"text\" entry for the row with index 2\n",
    "\n",
    "print(null_rows_df.loc[3, 'text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "e136400e-9582-45c6-ada9-de4d056f249d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text                               Die Belarus-Route ist nach wie vor offen via @...\n",
      "author_id                                                                 19836074.0\n",
      "created_at                                                                       NaN\n",
      "lang                                                                              de\n",
      "geo.place_id                                                                     NaN\n",
      "public_metrics.retweet_count                                                     NaN\n",
      "public_metrics.reply_count                                                       NaN\n",
      "public_metrics.like_count                                                        NaN\n",
      "public_metrics.quote_count                                                       NaN\n",
      "public_metrics.impression_count                                                  NaN\n",
      "geo.coordinates.coordinates                                                      NaN\n",
      "in_reply_to_user_id                                                              NaN\n",
      "entities.hashtags                                                                NaN\n",
      "week                                                                             NaN\n",
      "month                                                                            NaN\n",
      "year                                                                             NaN\n",
      "year-week                                                                        NaN\n",
      "year-month                                                                       NaN\n",
      "Language                                                                      German\n",
      "date                                                                             NaN\n",
      "inflow                                                                    Ukrainians\n",
      "new text                           Die Belarus-Route ist nach wie vor offen via @...\n",
      "scores                                            [0.0282606, 0.7912057, 0.18053377]\n",
      "Negative                                                                    0.028261\n",
      "Neutral                                                                     0.791206\n",
      "Positive                                                                    0.180534\n",
      "CLASS                                                                        NEUTRAL\n",
      "num_CLASS                                                                          0\n",
      "Name: 3, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(null_rows_df.iloc[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f36bae-09a4-4390-9c08-af7f22d3a247",
   "metadata": {},
   "source": [
    "The tweet as seen in index 3 appears to have been mistakenly copied over and over and excluded a lot of other information (such as created_at, etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "d24f0f34-3171-41c6-b657-410126b82353",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows that contain 'Die Belarus-Route ist nach wie vor offen via @RND_de': 27650\n"
     ]
    }
   ],
   "source": [
    "# fill NaN values within 'text' with an empty string\n",
    "\n",
    "null_rows_df['text'] = null_rows_df['text'].fillna('')\n",
    "\n",
    "# count the number of rows that contain the specified text\n",
    "\n",
    "count = len(null_rows_df[null_rows_df['text'].str.contains('Die Belarus-Route ist nach wie vor offen via @RND_de')])\n",
    "\n",
    "# print the count\n",
    "\n",
    "print(\"Number of rows that contain 'Die Belarus-Route ist nach wie vor offen via @RND_de':\", count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d742dd4d-0437-431c-9af4-77b78a4cdc32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "print(27653-27650)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ef1c0b7-3d1c-4451-b7cf-6404ebed0eae",
   "metadata": {},
   "source": [
    "Out of the 27,653 entries with a missing value for created_at, 27,650 of them contain the text \"Die Belarus-Route ist nach wie vor offen via @RND_de\"\n",
    "\n",
    "I want to now see whether this tweet exists within the original dataframe but has an actual value for created_at. To do so, I will create a dataframe without any nulls for created_at and check if any row contains \"Die Belarus-Route ist nach wie vor offen via @RND_de\" under the column 'text'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "e8938c6b-a9b4-4582-96d9-9d0154835bc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new dataframe without null values in the 'created_at' column\n",
    "\n",
    "no_nulls_df = df.dropna(subset=['created_at'], how='any')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "30a5c5a4-0014-4c26-8f78-18cf808cbcb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 38611 entries, 0 to 66265\n",
      "Data columns (total 28 columns):\n",
      " #   Column                           Non-Null Count  Dtype  \n",
      "---  ------                           --------------  -----  \n",
      " 0   text                             38611 non-null  object \n",
      " 1   author_id                        38611 non-null  float64\n",
      " 2   created_at                       38611 non-null  object \n",
      " 3   lang                             38611 non-null  object \n",
      " 4   geo.place_id                     38225 non-null  object \n",
      " 5   public_metrics.retweet_count     38611 non-null  float64\n",
      " 6   public_metrics.reply_count       38611 non-null  float64\n",
      " 7   public_metrics.like_count        38611 non-null  float64\n",
      " 8   public_metrics.quote_count       38611 non-null  float64\n",
      " 9   public_metrics.impression_count  38611 non-null  float64\n",
      " 10  geo.coordinates.coordinates      4347 non-null   object \n",
      " 11  in_reply_to_user_id              12325 non-null  float64\n",
      " 12  entities.hashtags                15559 non-null  object \n",
      " 13  week                             38611 non-null  float64\n",
      " 14  month                            38611 non-null  float64\n",
      " 15  year                             38611 non-null  float64\n",
      " 16  year-week                        38611 non-null  object \n",
      " 17  year-month                       38611 non-null  object \n",
      " 18  Language                         38611 non-null  object \n",
      " 19  date                             38611 non-null  object \n",
      " 20  inflow                           38611 non-null  object \n",
      " 21  new text                         38611 non-null  object \n",
      " 22  scores                           38611 non-null  object \n",
      " 23  Negative                         38611 non-null  float64\n",
      " 24  Neutral                          38611 non-null  float64\n",
      " 25  Positive                         38611 non-null  float64\n",
      " 26  CLASS                            38611 non-null  object \n",
      " 27  num_CLASS                        38611 non-null  int64  \n",
      "dtypes: float64(13), int64(1), object(14)\n",
      "memory usage: 8.5+ MB\n"
     ]
    }
   ],
   "source": [
    "no_nulls_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "0a7c1a73-54bc-46b9-9316-d663e680327c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows that contain 'Die Belarus-Route ist nach wie vor offen via @RND_de': 0\n"
     ]
    }
   ],
   "source": [
    "# count the number of rows that contain the specified text\n",
    "\n",
    "count = len(no_nulls_df[no_nulls_df['text'].str.contains('Die Belarus-Route ist nach wie vor offen via @RND_de')])\n",
    "\n",
    "# print the count\n",
    "\n",
    "print(\"Number of rows that contain 'Die Belarus-Route ist nach wie vor offen via @RND_de':\", count)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46c42008-7f2a-4ffe-bc1a-1738dbb21958",
   "metadata": {},
   "source": [
    "Zero rows contain that tweet.\n",
    "\n",
    "I want to see what the other 3 tweets were."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "f5f6b9b9-1807-4989-9bf4-e3e25937d2fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing all observations that contain 'Die Belarus-Route ist nach wie vor offen via @RND_de' as its entry for 'text'\n",
    "\n",
    "null_rows_df = null_rows_df[~null_rows_df['text'].str.contains('Die Belarus-Route ist nach wie vor offen via @RND_de')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "a736ec0a-daae-4fbf-baed-0f063ba1d878",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 3 entries, 0 to 2\n",
      "Data columns (total 28 columns):\n",
      " #   Column                           Non-Null Count  Dtype  \n",
      "---  ------                           --------------  -----  \n",
      " 0   text                             3 non-null      object \n",
      " 1   author_id                        0 non-null      float64\n",
      " 2   created_at                       0 non-null      object \n",
      " 3   lang                             0 non-null      object \n",
      " 4   geo.place_id                     0 non-null      object \n",
      " 5   public_metrics.retweet_count     0 non-null      float64\n",
      " 6   public_metrics.reply_count       0 non-null      float64\n",
      " 7   public_metrics.like_count        0 non-null      float64\n",
      " 8   public_metrics.quote_count       0 non-null      float64\n",
      " 9   public_metrics.impression_count  0 non-null      float64\n",
      " 10  geo.coordinates.coordinates      0 non-null      object \n",
      " 11  in_reply_to_user_id              0 non-null      float64\n",
      " 12  entities.hashtags                0 non-null      object \n",
      " 13  week                             0 non-null      float64\n",
      " 14  month                            0 non-null      float64\n",
      " 15  year                             0 non-null      float64\n",
      " 16  year-week                        0 non-null      object \n",
      " 17  year-month                       0 non-null      object \n",
      " 18  Language                         3 non-null      object \n",
      " 19  date                             0 non-null      object \n",
      " 20  inflow                           3 non-null      object \n",
      " 21  new text                         2 non-null      object \n",
      " 22  scores                           3 non-null      object \n",
      " 23  Negative                         3 non-null      float64\n",
      " 24  Neutral                          3 non-null      float64\n",
      " 25  Positive                         3 non-null      float64\n",
      " 26  CLASS                            3 non-null      object \n",
      " 27  num_CLASS                        3 non-null      int64  \n",
      "dtypes: float64(13), int64(1), object(14)\n",
      "memory usage: 696.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "null_rows_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "b51d7228-e0cd-46c9-9b03-8a0eb53bf65e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "      <th>new text</th>\n",
       "      <th>scores</th>\n",
       "      <th>Negative</th>\n",
       "      <th>Neutral</th>\n",
       "      <th>Positive</th>\n",
       "      <th>CLASS</th>\n",
       "      <th>num_CLASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>@N24</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>@user</td>\n",
       "      <td>[0.23433569, 0.4841369, 0.2815274, ]</td>\n",
       "      <td>0.234336</td>\n",
       "      <td>0.484137</td>\n",
       "      <td>0.281527</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[0.34819144, 0.39594722, 0.2558613, ]</td>\n",
       "      <td>0.348191</td>\n",
       "      <td>0.395947</td>\n",
       "      <td>0.255861</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "      <td>#refugeeswelcome #1209HH #Flüchtling #Syrien #...</td>\n",
       "      <td>[0.11344611, 0.7992048, 0.08734913]</td>\n",
       "      <td>0.113446</td>\n",
       "      <td>0.799205</td>\n",
       "      <td>0.087349</td>\n",
       "      <td>NEUTRAL</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  author_id created_at  \\\n",
       "0                                              @N24         NaN        NaN   \n",
       "1                                                           NaN        NaN   \n",
       "2  #refugeeswelcome #1209HH #Flüchtling #Syrien #...        NaN        NaN   \n",
       "\n",
       "  lang geo.place_id  public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0  NaN          NaN                           NaN                         NaN   \n",
       "1  NaN          NaN                           NaN                         NaN   \n",
       "2  NaN          NaN                           NaN                         NaN   \n",
       "\n",
       "   public_metrics.like_count  public_metrics.quote_count  \\\n",
       "0                        NaN                         NaN   \n",
       "1                        NaN                         NaN   \n",
       "2                        NaN                         NaN   \n",
       "\n",
       "   public_metrics.impression_count  ... Language  date   inflow  \\\n",
       "0                              NaN  ...   German   NaN  Syrians   \n",
       "1                              NaN  ...   German   NaN  Syrians   \n",
       "2                              NaN  ...   German   NaN  Syrians   \n",
       "\n",
       "                                            new text  \\\n",
       "0                                             @user    \n",
       "1                                                NaN   \n",
       "2  #refugeeswelcome #1209HH #Flüchtling #Syrien #...   \n",
       "\n",
       "                                  scores  Negative   Neutral  Positive  \\\n",
       "0   [0.23433569, 0.4841369, 0.2815274, ]  0.234336  0.484137  0.281527   \n",
       "1  [0.34819144, 0.39594722, 0.2558613, ]  0.348191  0.395947  0.255861   \n",
       "2    [0.11344611, 0.7992048, 0.08734913]  0.113446  0.799205  0.087349   \n",
       "\n",
       "     CLASS num_CLASS  \n",
       "0  NEUTRAL         0  \n",
       "1  NEUTRAL         0  \n",
       "2  NEUTRAL         0  \n",
       "\n",
       "[3 rows x 28 columns]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "null_rows_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "ff4b1791-3314-46c7-b62d-298634ffd237",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# view the \"text\" entry for the row with index 0\n",
    "\n",
    "print(null_rows_df.loc[1, 'text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "107d11ce-5ce2-43f5-af57-c9d27ca93b70",
   "metadata": {},
   "source": [
    "The remaining tweets contain either no text, only \"@N24\", or only a series of hashtags. Furthermore, they do not contain any other important info, such as created_at.\n",
    "Therefore, EVERY observation that had a null for created_at needs to be dropped from the original dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "efc74b01-c693-4447-b618-58a6b700f3b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping all observations that have null for created_at from original dataframe\n",
    "\n",
    "df = df.dropna(subset=['created_at'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "635167d6-5337-451f-a949-9d7d22b96a05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 38611 entries, 0 to 66265\n",
      "Data columns (total 28 columns):\n",
      " #   Column                           Non-Null Count  Dtype  \n",
      "---  ------                           --------------  -----  \n",
      " 0   text                             38611 non-null  object \n",
      " 1   author_id                        38611 non-null  float64\n",
      " 2   created_at                       38611 non-null  object \n",
      " 3   lang                             38611 non-null  object \n",
      " 4   geo.place_id                     38225 non-null  object \n",
      " 5   public_metrics.retweet_count     38611 non-null  float64\n",
      " 6   public_metrics.reply_count       38611 non-null  float64\n",
      " 7   public_metrics.like_count        38611 non-null  float64\n",
      " 8   public_metrics.quote_count       38611 non-null  float64\n",
      " 9   public_metrics.impression_count  38611 non-null  float64\n",
      " 10  geo.coordinates.coordinates      4347 non-null   object \n",
      " 11  in_reply_to_user_id              12325 non-null  float64\n",
      " 12  entities.hashtags                15559 non-null  object \n",
      " 13  week                             38611 non-null  float64\n",
      " 14  month                            38611 non-null  float64\n",
      " 15  year                             38611 non-null  float64\n",
      " 16  year-week                        38611 non-null  object \n",
      " 17  year-month                       38611 non-null  object \n",
      " 18  Language                         38611 non-null  object \n",
      " 19  date                             38611 non-null  object \n",
      " 20  inflow                           38611 non-null  object \n",
      " 21  new text                         38611 non-null  object \n",
      " 22  scores                           38611 non-null  object \n",
      " 23  Negative                         38611 non-null  float64\n",
      " 24  Neutral                          38611 non-null  float64\n",
      " 25  Positive                         38611 non-null  float64\n",
      " 26  CLASS                            38611 non-null  object \n",
      " 27  num_CLASS                        38611 non-null  int64  \n",
      "dtypes: float64(13), int64(1), object(14)\n",
      "memory usage: 8.5+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a45c9667-b944-470d-bb04-0975e6075b04",
   "metadata": {},
   "source": [
    "38611 observations remain.\n",
    "\n",
    "To summarize, 27653 observations without any useful information were dropped, leaving 38611 for the remaining dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc0974e0-4894-4960-8e3f-ecb10bf07878",
   "metadata": {},
   "source": [
    "## 8. Saving the data\n",
    "The data will be saved in the CASS_thesis folder as a csv titled,\n",
    "\n",
    "> *04_Prepared-data_limited_merged.csv*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "1558fb49-7bf4-4335-baaf-8d086c8a05e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(CASS_thesis / '04_Prepared-data_limited_merged.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a18d1a-6cb5-4f03-80bf-b5f0c11ec002",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}