Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cascade Influence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This repository contains:\n",
    " - The scripts to estimate user influence from Twitter information cascades (i.e. Cas.In);\n",
    " - A small dataset of 20 cascades for testing Cas.In;\n",
    " - A hands-on tutorial to walk you through running Cas.In on real cascades."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citation\n",
    "The algorithm was introduced in the paper:\n",
    "\n",
    "Rizoiu, M.-A., Graham, T., Zhang, R., Zhang, Y., Ackland, R., & Xie, L. (2018). **#DebateNight: The Role and Influence of Socialbots on Twitter During the 1st 2016 U.S. Presidential Debate**. In *Proc. International AAAI Conference on Web and Social Media (ICWSM ’18)* (pp. 1–10). Stanford, CA, USA.  \n",
    "[pdf at arxiv with supplementary material](https://arxiv.org/abs/1802.09808)\n",
    "\n",
    "**Bibtex**\n",
    "```\n",
    "@inproceedings{rizoiu2018debatenight,\n",
    "    address = {Stanford, CA, USA},\n",
    "    author = {Rizoiu, Marian-Andrei and Graham, Timothy and Zhang, Rui and Zhang, Yifei and Ackland, Robert and Xie, Lexing},\n",
    "    booktitle = {International AAAI Conference on Web and Social Media (ICWSM '18)},\n",
    "    title = {{{\\#}DebateNight: The Role and Influence of Socialbots on Twitter During the 1st 2016 U.S. Presidential Debate}},\n",
    "    url = {https://arxiv.org/abs/1802.09808},\n",
    "    year = {2018}\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### License\n",
    "Both dataset and code are distributed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license, a copy of which can be obtained following this link. If you require a different license, please contact [Yifei Zhang](mailto:yifeiacc@gmail.com), [Marian-Andrei Rizoiu](mailto:Marian-Andrei@rizoiu.eu) or [Lexing Xie](mailto:Lexing.Xie@anu.edu.au)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to run Cas.In in a terminal:\n",
    "\n",
    "### Required packages:\n",
    "  - python3\n",
    "  - numpy\n",
    "  - pandas\n",
    "    \n",
    "### Arguments of Cas.In:\n",
    "\n",
    "*--cascade_path* : the path of cascade file (see the format here below). \n",
    "\n",
    "*--time_decay* : the coefficient value of time decay (hyperparameter $r$ in the paper). **Default**:-0.000068\n",
    "\n",
    "*--save2csv* : save result to csv file. **Default**: False\n",
    "\n",
    "### Command:\n",
    "```bash\n",
    "cd scripts\n",
    "python3 influence.py --cascade_path path/to/file\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File format and toy dataset\n",
    "\n",
    "### Dataset\n",
    "We provide a toy dataset -- dubbed SMH -- for testing Cas.In.\n",
    "It was collected in 2017 by following the Twitter handle of the Sydney Morning Herald newspaper (tweets and retweets mentioning SMH or linking to an article from SMH).\n",
    "The data contains 20 cascades (one file per cascade).\n",
    "We annonymized the `user_id` (as per Twitter's ToS) by mapping original values to a sequence from 0 to n, while preserving the identity of users across cascades.\n",
    "\n",
    "### The format cascade files:\n",
    " - A csv file with 3 columns (`time`, `magnitude`, `user_id`), where each row is a tweet in the cascade:\n",
    "    - `time` represents the timestamp of tweet -- the first tweet is always at time zero, for the following retweets it shows the offset in seconds from the initial tweet;\n",
    "    - `magnitude` is the local influence of the user (here the number of followers);\n",
    "    - `user_id` the id of the user emitting the tweet (here annonymized).\n",
    " - The rows in the file (i.e. the tweets) are sorted by the timestamp;\n",
    " \n",
    "eg:\n",
    "```\n",
    "time,magnitude,user_id \n",
    "0,4674,\"0\"\n",
    "321,1327,\"1\"\n",
    "339,976,\"2\"\n",
    "383,477,\"3\"\n",
    "699,1209,\"4\"\n",
    "824,119,\"5\"\n",
    "835,1408,\"6\"\n",
    "1049,896,\"7\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cascade influence tutorial\n",
    "\n",
    "Next, we drive you through using Cas.In for estimating user influence starting from a single cascade.\n",
    "\n",
    "###  Preliminary\n",
    "We need to first load all required packages of cascade influence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/yifei/Desktop/cascade-influence/scripts\n"
     ]
    }
   ],
   "source": [
    "cd scripts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from casIn.user_influence import P,influence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute influence in one cascade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "###  Read data\n",
    "Load the first cascade in the SMH toy dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>magnitude</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>991</td>\n",
       "      <td>419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>127</td>\n",
       "      <td>1352</td>\n",
       "      <td>658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2149</td>\n",
       "      <td>2057</td>\n",
       "      <td>264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2465</td>\n",
       "      <td>1155</td>\n",
       "      <td>1016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2485</td>\n",
       "      <td>1917</td>\n",
       "      <td>790</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time  magnitude  user_id\n",
       "0     0        991      419\n",
       "1   127       1352      658\n",
       "2  2149       2057      264\n",
       "3  2465       1155     1016\n",
       "4  2485       1917      790"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cascade = pd.read_csv(\"../data/SMH/SMH-cascade-0.csv\")\n",
    "cascade.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Compute matrix P\n",
    "\n",
    "We first need to compute the probabilities $p_{ij}$, where $p_{ij}$ is the probability that $j^{th}$ tweet is a direct retweet of the $i^{th}$ (see the paper for more details).\n",
    "We need to specify the hyper-parameter $r$,  the time decay coefficient. \n",
    "Here we choose $r = -0.000068$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p_ij = P(cascade,r = -0.000068)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Compute user influence and matrix M\n",
    "The function `influence()` will return an array of influences for each user and the matrix $M = m_{ij}$, where $m_{ij}$ is the influence of the $i^{th}$ tweet of the $j^{th}$ tweet (direct and indirect)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inf, m_ij = influence(p_ij)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Link influence with user_id\n",
    "\n",
    "Now, we add the computed user influence back to the pandas data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cascade[\"influence\"] = pd.Series(inf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>magnitude</th>\n",
       "      <th>user_id</th>\n",
       "      <th>influence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>991</td>\n",
       "      <td>419</td>\n",
       "      <td>60.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>127</td>\n",
       "      <td>1352</td>\n",
       "      <td>658</td>\n",
       "      <td>34.590370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2149</td>\n",
       "      <td>2057</td>\n",
       "      <td>264</td>\n",
       "      <td>29.656122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2465</td>\n",
       "      <td>1155</td>\n",
       "      <td>1016</td>\n",
       "      <td>13.535845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2485</td>\n",
       "      <td>1917</td>\n",
       "      <td>790</td>\n",
       "      <td>15.913873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time  magnitude  user_id  influence\n",
       "0     0        991      419  60.000000\n",
       "1   127       1352      658  34.590370\n",
       "2  2149       2057      264  29.656122\n",
       "3  2465       1155     1016  13.535845\n",
       "4  2485       1917      790  15.913873"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cascade.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute influence over multiple cascades\n",
    "### Load function\n",
    "The function *casIn()* compute influence in one cascade, which basically contain all the steps described above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from casIn.user_influence import casIn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time</th>\n",
       "      <th>magnitude</th>\n",
       "      <th>user_id</th>\n",
       "      <th>influence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>991</td>\n",
       "      <td>419</td>\n",
       "      <td>60.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>127</td>\n",
       "      <td>1352</td>\n",
       "      <td>658</td>\n",
       "      <td>34.590370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2149</td>\n",
       "      <td>2057</td>\n",
       "      <td>264</td>\n",
       "      <td>29.656122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2465</td>\n",
       "      <td>1155</td>\n",
       "      <td>1016</td>\n",
       "      <td>13.535845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2485</td>\n",
       "      <td>1917</td>\n",
       "      <td>790</td>\n",
       "      <td>15.913873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   time  magnitude  user_id  influence\n",
       "0     0        991      419  60.000000\n",
       "1   127       1352      658  34.590370\n",
       "2  2149       2057      264  29.656122\n",
       "3  2465       1155     1016  13.535845\n",
       "4  2485       1917      790  15.913873"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "influence = casIn(cascade_path=\"../data/SMH/SMH-cascade-0.csv\",time_decay=-0.000068)\n",
    "influence.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load multiple cascades\n",
    "\n",
    "The SMH toy dataset contains 20 cascades for testing out Cas.In.\n",
    "Let's load all of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cascades = []\n",
    "for i in range(20):\n",
    "    inf = casIn(cascade_path=\"../data/SMH/SMH-cascade-%d.csv\" % i,time_decay=-0.000068)\n",
    "    cascades.append(inf)\n",
    "cascades = pd.concat(cascades)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute user influence in multiple cascades\n",
    "\n",
    "The influence of a user is by definition the mean influence of the tweets they emit.\n",
    "We compute the user influence as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result = cascades.groupby(\"user_id\").agg({\"influence\" : \"mean\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>influence</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>734</th>\n",
       "      <td>214.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1225</th>\n",
       "      <td>205.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>755</th>\n",
       "      <td>190.554571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>189.557461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>581</th>\n",
       "      <td>141.033129</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          influence\n",
       "user_id            \n",
       "734      214.000000\n",
       "1225     205.000000\n",
       "755      190.554571\n",
       "60       189.557461\n",
       "581      141.033129"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.sort_values(\"influence\",ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}