CertStreamMonitor

Monitor certificates generated for specific domain strings and associated, store data into sqlite3 database, alert you when sites come online.

CertStreamMonitor architecture relies on 3 scripts :

• certstreammonitor.py
  • this script runs as a daemon.
  • reading the certstream feed from CertStream service provided by Calidog Security (thanks so much for this!), it selects hostnames covered by certificates that match your criteria (SearchKeyWords parameter in conf).
  • it writes these hostnames along with its certificate relevant informations to the database.
• scanhost.py
  • this script can be executed as often as you like.
  • it checks if site corresponding to the hostanme stored in DB is UP ot not.
  • it collects informations about the sites that are up to DB and to a JSON file.
• gethost.py
  • Due to @nbeguier contribution, the project has also a gethost.py script that provides a way for security operators to request the last hostnames which have been detected by certstreammointor.py since X seconds.
• check_rules.py
  • Due to @nbeguier contribution, the project has also a check_rules.py script that provides a way for security operators to check their own rules, especially SearchKeywords, DetectionThreshold and BlacklistKeywords.

Features

• Monitoring:
  • monitor wss://certstream.calidog.io CT logs aggregator server with certstream-python (see certstream-python), but you can choose, and operate, your own server (see certstream-server).
  • choose strings you want to monitor in Subject Alt Names field of certificates
• Storing:
  • store hostnames found along with its certificate relevant data into a sqlite3 database
• Alerting:
  • for each hostname not already flagged as up : check if corresponding site is up or not
    • if it's up :
      • collects informations (IP address, AS informations, HTTP code, web page title, abuse email, (optional) google safe browsing status)
      • write them to a JSON file in the /alerts directory (default value) to push forward investigation.
      • (optional) push them to a destination (through apprise package) such as an email address, a Slack channel or even Twitter account
      • flags the hostname in the DB as up
    • if it's not :
      • the hostname will be checked until the Alert_Monitor_timelapse configuration variable (in days) will be reached
      • when the limit is reached a string like Stop checking on 2019-09-27 is included into the StillInvestig column

Requirements

• Python 3
• certstream
• sqlite3
• ipwhois
• PySocks
• hues
• websocket-client
• apprise

Install

Install the requirements

$ pip3 install -r requirements.txt

Configuration file

You can find a configuration file example in 'conf' directory. Configurable parameters are:

• SearchKeywords: Keywords to look for (with '|' (or) as separator)
• DetectionThreshold: set the minimum number of detected SearchKeywords in a hostname before writing it to DB. Under this value but above zero, detected hostnames are only written to logfile. Default value: 2.
• DBFile: SQLite3 database file (the path and file will be created if don't exist)
• TABLEname: The name of the database table
• LogFile: The logging file (the path and file will be created if don't exist)
• UAfile: you can provide a User-Agent file to masquerade this value of requests (random change for each request)
• Alerts_dir: you can specify where JSON alert files are written You can use the following strings to add time/date hashed based subdirectories: %%m -> month, %%d -> day, %%Y -> year, %%H -> hour, %%M -> minute. Example: Alerts_dir = ./alerts/%%Y/%%m/%%d

Optional:

• Proxy: allows to give a SOCKS or HTTP proxy to process your scanhost.py's requests (as Tor)
• Proxy_* parameters : allow you to specify HTTP proxy informations (server, port[, user, password]) for CertStreamMonitor.py script to connect to the CT logs aggregator server.
• ACTServer: you can specify the CT logs aggregator server of your choice. By default, it is the server run by Calidog Security.
• Safe_Browsing_API_Key: indicate (if you want) your Google Safe Browsing API key in order to check hostnames that are UP against Google Safe Browsing Lookup API (How-To get an API key for the Safe Browsing Lookup API).
• Notification_Destination: specify a notification destination as attended by apprise package. Documentation about the format of this parameter is available on the apprise Github page.
• BlacklistKeywords: Keywords to ignore matched hosts (with '|' (or) as separator)

Usage

CertStreamMonitor.py

$ python3 ./CertStreamMonitor.py -c conf/example.conf
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
Connection established to CertStream! Listening for events...
[2018-03-12T11:40:15] cpanel.my-appleid-apple.net (SAN: mail.my-appleid-apple.net,my-appleid-apple.net,webdisk.my-appleid-apple.net,webmail.my-appleid-apple.net,www.my-appleid-apple.net) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 45:11:51:2D:24:D3:04:6E:DF:49:46:6D:64:56:67:4B:0A:48:8D:93) (StartTime: 2018-03-12T10:39:40)
[2018-03-12T11:41:19] cpanel.verification-account-apple-now.com (SAN: mail.verification-account-apple-now.com,verification-account-apple-now.com,webdisk.verification-account-apple-now.com,webmail.verification-account-apple-now.com,www.verification-account-apple-now.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 2D:90:F9:F7:83:F6:48:26:EF:C9:72:50:4B:06:FA:36:53:94:3C:8B) (StartTime: 2018-03-12T10:40:49)
[2018-03-12T11:41:36] login-apple.sytes.net (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: C7:78:2F:08:1E:CC:83:6C:06:EF:77:14:D2:1A:4E:06:A8:B3:F9:77) (StartTime: 2018-03-12T10:41:08)
[2018-03-12T11:42:26] cpanel.restore-account-apple.com (SAN: mail.restore-account-apple.com,restore-account-apple.com,webdisk.restore-account-apple.com,webmail.restore-account-apple.com,www.restore-account-apple.com) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: F3:CA:B1:C6:DE:4F:05:16:FD:06:F3:FF:29:8A:D3:1F:10:9D:50:1A) (StartTime: 2018-03-12T10:41:59)
[2018-03-12T11:49:37] securelogin.here.att.thysseankrupp.com (SAN: ) (Issuer: /C=US/CN=Let's Encrypt Authority X3/O=Let's Encrypt) (Fingerprint: 8F:9B:98:8D:5D:9B:03:0B:4F:62:56:40:C1:DE:9A:A4:FB:2D:A3:3E) (StartTime: 2018-03-12T09:22:41)
...

scanhost.py

$ python3 scanhost.py --help

    -h --help		Print this help
    -c --config		Configuration file to use
    -f --fqdn-dirs      Store JSON files in sub-directories based on the hostname

$ python3 ./scanhost.py -c conf/example.conf
Test all domains in DB for Internet Presence:
*********************************************
14:30:12 - ERROR -   https://socialparadiseweb.cf.socialparadise.cf - Connection error
14:32:18 - ERROR -   https://rapportannuel-assurancemaladie.paris - Connection error
14:32:23 - SUCCESS - HTTP 200 - socialmediaforsocialaction.com
Creating ./alerts/socialmediaforsocialaction.com.json : {'hostname': 'socialmediaforsocialaction.com', 'http_code': 200, 'cert_serial_number': '89:6C:03:F6:82:57:03:2A:A8:D0:E1:2F:E8:56:0E:32:83:E5:EC:29', 'webpage_title': 'Social Media for Social Action', 'ip_addr': '198.49.23.145', 'asn': '53831', 'asn_cidr': '198.49.23.0/24', 'asn_country_code': 'US', 'asn_description': 'SQUARESPACE - Squarespace, Inc., US', 'asn_abuse_email': 'abuse-network@squarespace.com'}
14:32:25 - ERROR -   https://social.socialbride.co.za - Connection error
14:32:34 - SUCCESS - HTTP 503 - assurances-sociales.com
Creating ./alerts/assurances-sociales.com.json : {'hostname': 'assurances-sociales.com', 'http_code': 503, 'cert_serial_number': '1A:0D:45:D9:05:15:DC:17:6C:9F:9E:47:A5:62:03:D9:25:02:F9:3C', 'webpage_title': 'Accueil', 'ip_addr': '164.132.235.17', 'asn': '16276', 'asn_cidr': '164.132.0.0/16', 'asn_country_code': 'FR', 'asn_description': 'OVH, FR', 'asn_abuse_email': 'lir@ovh.net'}

gethost.py

$ python3 gethost.py --help

    -h --help   Print this help
    -c --config Configuration file to use
    --since     Since when it displays findings (seconds)

$ python3 ./gethost.py -c conf/example.conf --since 36000 # 10 hours
Display all domains in DB for Internet Presence:
************************************************
socialparadiseweb.cf.socialparadise.cf None
rapportannuel-assurancemaladie.paris None
socialmediaforsocialaction.com 2019-06-12T15:54:31
social.socialbride.co.za None

check_rules.py

$ python3 check_rules.py --help

    -h --help       Print this help
    -c --config     Configuration file to use
    -d --domain     Domain name to check

# No match - Keywords not found.
$ ./check_rules.py -c conf/example.conf -d www.google.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Keywords not found.

# No match - Detection threashold not reached.
$ ./check_rules.py -c conf/example.conf -d paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Detection threashold not reached.

# No match - Blacklisted keywords.
$ ./check_rules.py -c conf/example.conf -d login-paypal.gouv
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
No match - Blacklisted keywords.

# This is a match, detection threashold reached.
$ ./check_rules.py -c conf/example.conf -d login-paypal.com
Looking for these strings: paypal|apple|account|secure|login, detection threshold: 2
This is a match, detection threashold reached.

Authors

• Thomas Damonneville (thomas.damonneville@assurance-maladie.fr)
• Christophe Brocas (christophe.brocas@assurance-maladie.fr)
• And of course, a big shout out to Calidog Security and its fantastic CertStream service without which nothing would have been possible!

Contributors

• @ant1: #3
• @xme: #8, #9
• @nbeguier: #20, #21, #28

Presentations

• SSTIC 2018 | June 2018 - C.Brocas, T. Damonneville: "Certificate Transparency ou comment un nouveau standard peut aider votre veille sur certaines menaces". Slides (fr), full article (fr), video (fr).
• Hack-it-n 2018bis | December 2018 - C. Brocas, T. Damonneville (given by C. Brocas): "CertStreamMonitor, use Certificate Transparency to improve your threats detection". Slides (en).
• Toulouse hacking Convention 2019 | March 2019 - C. Brocas, T. Damonneville (given by C. Brocas): "Certificate Transparency & threats detection, 24 months later". Slides (en), video (fr).

License

GNU GENERAL PUBLIC LICENSE (GPL) Version 3