Twitter/Twitter_Gettingpast_32K_Limit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will show how to use `Selenium` and `Tweepy` to get past the `32,000` Tweet limit. \n",
    "\n",
    "We use `Selenium` to open up a browser and visit Twitter's search page. From the Twiter search page we can obtain the tweet IDs for a given user, then using `tweepy` we can obtain the contents for all tweet IDs obtained.\n",
    "\n",
    "\n",
    "* Adapted from [Twitter Scraping](https://github.com/bpb27/twitter_scraping).\n",
    "* The Authentication process follows [Intro_Collecting_Tweets](https://github.com/Data4Democracy/assemble/blob/master/tutorials/Intro_Collecting_Tweets.ipynb).\n",
    "* In order to get this working you need to install [ChromeDriver]( https://sites.google.com/a/chromium.org/chromedriver/) \n",
    " * For Ubuntu ([source](https://christopher.su/2015/selenium-chromedriver-ubuntu/)):\n",
    "\n",
    "```\n",
    "wget -N http://chromedriver.storage.googleapis.com/2.26/chromedriver_linux64.zip\n",
    "unzip chromedriver_linux64.zip\n",
    "chmod +x chromedriver\n",
    "\n",
    "sudo mv -f chromedriver /usr/local/share/chromedriver\n",
    "sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver\n",
    "sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver\n",
    " ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import time\n",
    "import datetime\n",
    "\n",
    "from tweepy import API\n",
    "from tweepy import OAuthHandler\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.common.exceptions import NoSuchElementException \n",
    "from selenium.common.exceptions import StaleElementReferenceException"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user = 'kdnuggets'\n",
    "\n",
    "start = datetime.datetime(2017, 1, 15)  \n",
    "end = datetime.datetime(2017, 1, 16)    \n",
    "\n",
    "twitter_ids_filename = 'all_ids.json'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from config import *\n",
    "\n",
    "def get_twitter_auth():\n",
    "    \"\"\"Setup Twitter Authentication.\n",
    "    \n",
    "    Return: tweepy.OAuthHandler object\n",
    "    \"\"\"\n",
    "    auth = OAuthHandler(consumer_key, consumer_secret)\n",
    "    auth.set_access_token(access_token, access_secret)\n",
    "    return auth\n",
    "    \n",
    "def get_twitter_client():\n",
    "    \"\"\"Setup Twitter API Client.\n",
    "    \n",
    "    Return: tweepy.API object\n",
    "    \"\"\"\n",
    "    auth = get_twitter_auth()\n",
    "    client = API(auth)\n",
    "    return client\n",
    "\n",
    "client = get_twitter_client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def twitter_url(user, start, end):\n",
    "    \"\"\"Form url to access tweets via Twitter's search page.\n",
    "    \n",
    "    Return: string\n",
    "    \"\"\"\n",
    "    url1 = 'https://twitter.com/search?f=tweets&q=from%3A'\n",
    "    url2 = user + '%20since%3A' + start.strftime('%Y-%m-%d') \n",
    "    url3 = '%20until%3A' + end.strftime('%Y-%m-%d') + '%20include%3Aretweets&src=typd'\n",
    "    return url1 + url2 + url3\n",
    "    \n",
    "def increment_day(date, i):\n",
    "    \"\"\"Increment day object by i days.\n",
    "    \n",
    "    Return: datetime object\n",
    "    \"\"\"\n",
    "    return date + datetime.timedelta(days=i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Tweet IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scrolling down to load more tweets\n",
      "scrolling down to load more tweets\n",
      "20 tweets found, 0 total\n",
      "9 tweets found, 20 total\n"
     ]
    }
   ],
   "source": [
    "# Adapted from https://github.com/bpb27/twitter_scraping\n",
    "\n",
    "delay = 1  # time to wait on each page load before reading the page\n",
    "driver = webdriver.Chrome() \n",
    "\n",
    "tweet_selector = 'li.js-stream-item'\n",
    "id_selector = '.time a.tweet-timestamp'\n",
    "\n",
    "ids = list()\n",
    "for day in range((end - start).days + 1):\n",
    "    # Get Twitter search url\n",
    "    startDate = increment_day(start, 0)\n",
    "    endDate = increment_day(start, 1)\n",
    "    url = twitter_url(user, startDate, endDate)\n",
    "\n",
    "    driver.get(url)\n",
    "    time.sleep(delay)\n",
    "\n",
    "    try:\n",
    "        found_tweets = driver.find_elements_by_css_selector(tweet_selector)\n",
    "        increment = 10\n",
    "\n",
    "        # Scroll through the Twitter search page\n",
    "        while len(found_tweets) >= increment:\n",
    "            print('scrolling down to load more tweets')\n",
    "            driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')\n",
    "            time.sleep(delay)\n",
    "            found_tweets = driver.find_elements_by_css_selector(tweet_selector)\n",
    "            increment += 10\n",
    "        print('{} tweets found, {} total'.format(len(found_tweets), len(ids)))\n",
    "\n",
    "        # Get the IDs for all Tweets\n",
    "        for tweet in found_tweets:\n",
    "            try:\n",
    "                id = tweet.find_element_by_css_selector(id_selector).get_attribute('href').split('/')[-1]\n",
    "                ids.append(id)\n",
    "            except StaleElementReferenceException as e:\n",
    "                print('lost element reference', tweet)\n",
    "\n",
    "    except NoSuchElementException:\n",
    "        print('no tweets on this day')\n",
    "\n",
    "    start = increment_day(start, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tweets found on this scrape:  29\n",
      "total tweet count:  29\n",
      "Tweets Scraped!\n"
     ]
    }
   ],
   "source": [
    "# Adapted from https://github.com/bpb27/twitter_scraping\n",
    "\n",
    "try:\n",
    "    with open(twitter_ids_filename) as f:\n",
    "        all_ids = ids + json.load(f)\n",
    "        data_to_write = list(set(all_ids))\n",
    "        print('tweets found on this scrape: ', len(ids))\n",
    "        print('total tweet count: ', len(data_to_write))\n",
    "except FileNotFoundError:\n",
    "    with open(twitter_ids_filename, 'w') as f:\n",
    "        all_ids = ids\n",
    "        data_to_write = list(set(all_ids))\n",
    "        print('tweets found on this scrape: ', len(ids))\n",
    "        print('total tweet count: ', len(data_to_write))\n",
    "\n",
    "with open(twitter_ids_filename, 'w') as outfile:\n",
    "    json.dump(data_to_write, outfile)\n",
    "\n",
    "print('Tweets Scraped!')\n",
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Tweet Info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "820722556936351744\n",
      "How Can Lean Six Sigma Help #MachineLearning? #KDN https://t.co/iPQJiQQhEz\n",
      "820741858326409224\n",
      "Solid Collection of \"Top #MachineLearning Books\" https://t.co/Dw4rL7ZzVF https://t.co/TC0PYFf8yK\n",
      "821019653698940928\n",
      "#ICYMI Game Theory Reveals the Future of #DeepLearning https://t.co/Dgq1dJI96h https://t.co/pB6hcmeG1c\n",
      "820725725359652864\n",
      "How IBM Is Using #ArtificialIntelligence to Provide #Cybersecurity https://t.co/b15TylAUrH #AI https://t.co/RJhBqxFTyc\n",
      "820642932084637698\n",
      "Exclusive Interview with top #DataScientist @Jeremyphoward on #DeepLearning, @Kaggle, #DataScience, and more… https://t.co/yUwleeOu0h\n",
      "820767777808125952\n",
      "What is the Role of the Activation Function in a Neural Network? #KDN https://t.co/jxzx8QChFl\n",
      "821024768375869442\n",
      "The Best Metric to Measure Accuracy of Classification Models #KDN https://t.co/9npH84wJ4S\n",
      "821012639102996480\n",
      "Poker Play Begins in \"Brains Vs. AI: Upping the Ante\" | Carnegie Mellon School of Computer Science… https://t.co/xwSoyPvXki\n",
      "820769790860075008\n",
      "Stanford courseware: #DataMining for Cyber #Security https://t.co/1UWpYtlN0l https://t.co/ZyZeVmUwOt\n",
      "820666882877427712\n",
      "#ICYMI The Major Advancements in #DeepLearning in 2016 https://t.co/qqXSid2Rvs https://t.co/F1N9bM1jyo\n",
      "820707623892877313\n",
      "10 Steps to Success in #Kaggle #DataScience Competitions #KDN https://t.co/1q65aIOF1j\n",
      "821005348706418688\n",
      "#ICYMI 3 methods to deal with outliers https://t.co/dr78u2lWYy https://t.co/6zjTfutcVD\n",
      "820692370874388481\n",
      "How to Choose a Data Format #KDN https://t.co/uCkVDUK8iT\n",
      "820685718502670336\n",
      "A Concise Overview of Recent Advances in the Internet of Things (#IoT) https://t.co/2HgaQbuzn4 https://t.co/6HO48dBJYA\n",
      "820696573558616066\n",
      "#DeepLearning for Self-Driving Cars: MIT Courseware https://t.co/NirWPBqs9K https://t.co/CNbcS4TVZg\n",
      "820650254773133313\n",
      "#ICYMI Ten Myths About Machine Learning, by Pedro Domingos https://t.co/4hMm2ojYwO https://t.co/2V36Lw8VxT\n",
      "820998297053630464\n",
      "#ICYMI Generative Adversarial Networks – Hot Topic in Machine Learning https://t.co/9DgHuiPHxO https://t.co/j2ufpQsqRR\n",
      "820994580531343361\n",
      "Shortcomings of #DeepLearning #KDN https://t.co/s13yap2ep6\n",
      "820703318678896640\n",
      "Baidu launches medical #chatbot to help Chinese doctors diagnose patients https://t.co/odsJnRd9Eh https://t.co/IymHSXb0x1\n",
      "820781844065120256\n",
      "Deriving Euclidean Distance Matrices on GPU, with Theano https://t.co/KZzrv5g5VG #GPU #programming #Python https://t.co/93BD3dqdA6\n",
      "820734522018897920\n",
      "A Concise Overview of Recent Advances in Vehicle Technologies https://t.co/ngOGz2oNSs https://t.co/dxEiwXIIuJ\n",
      "820662180861407232\n",
      "10 Tips to Improve your #DataScience #Interview #KDN https://t.co/94quRjNW5C\n",
      "821039939987111936\n",
      "The #DataScience Process, Rediscovered #KDN https://t.co/UdScyRHYQi\n",
      "820757455101640704\n",
      "A Concise Overview of Recent Advances in Chatbot Technologies https://t.co/c2DkUDikkc https://t.co/F8icjFXwKZ\n",
      "821029291194458112\n",
      "Learning to reconstruct an image from pixel coordinates: #NeuralNetworks map from x,y of an images pixels to R,G,B… https://t.co/HbbRX6zqlD\n",
      "820657265564852224\n",
      "Deep Reinforcement Learning: Playing a Racing Game https://t.co/w85vYoXN7g https://t.co/C42uCJupzE\n",
      "820721469487513600\n",
      "Cluster Analysis Puzzle: Learn by Doing! Quick clustering overview https://t.co/qjnNBhVqV5 https://t.co/8guPUlZg4C\n",
      "820647115206643713\n",
      "Predicting Future Human Behavior with #DeepLearning #KDN https://t.co/a1gBc7MJyq\n",
      "821048152266567681\n",
      "5 Big Predictions for #AI in 2017: #Reinforcement Learning, #Adversarial nets, #China,  Language learning, #Hype https://t.co/fQ2SuCwrC2\n"
     ]
    }
   ],
   "source": [
    "with open(twitter_ids_filename) as f:\n",
    "    ids = json.load(f)\n",
    "    for tweetId in ids:\n",
    "        print(tweetId)\n",
    "        tweet = client.get_status(tweetId)\n",
    "        print(tweet.text)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}