tweetsniff.py

Grabs a user's Twitter feed and tweets with specific keywords via the Twitter API for local processing (storing to ElasticSearch, regex highlighting, etc)

Installation

The following Python modules are required:

• twitter
• elasticsearch
• termcolor
• dateutil
• hashlib
• urllib
• httplib

Install them with: pip install

Valid Twitter API key & token are required, see https://apps.twitter.com/

Usage

usage: tweetsniff.py [-h] [-c CONFIG]

Display a Tweet feed

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        configuration file (default: /etc/tweetsniff.conf)

Expand URLs

• Enable in config with "process_urls: True"
• urllib2.url provides the same features but gets sometimes rejected by sites (403)
• URLs have to be part of the twitter URL object, otherwise they do not get recognized
• not yet in writeCEFEvent
• ES object contains URL, expanded URL, MD5 and SHA1 of the (expanded) URL

Todo

• Add more statistics