Parsing Feeds Tutorial
This is a walk-through of how to create a feed configuration file to add new threat intelligence feeds to your CIF installation. It explains the commonly used configuration values and how they affect feed generation. If you are already familiar with feed configuration and just need details about all of the configuration parameters, see the parsing feeds page.
In this example, we'll walk through creating a single feed configuration file to pull two feeds from the Dragon Research Group:
YAML is the syntax used to generate CIF feed configuration files.
All parameters can be a Global parameter or a Feed parameter. If the parameter is specified twice, the Feed parameter will supersede the Global parameter.
parser: <value>
defaults:
<parameter>: <value>
<parameter>: <value>
<parameter>:
- <value>
- <value>
feeds:
<parameter>: <value>
<parameter>: <value>
<parameter>: <value>
<parameter>: <value>
CIF feed configuration files can be found in /etc/cif/rules:
- default - feeds shipped with a standard CIF installation
- disabled - feeds that have been found to have issues
- example - feed configurations files to be used as example configurations
Note: To browse /etc/cif/rules you'll need to be the 'cif' user (e.g. sudo su - cif).
CIF will load all feed configuration files found in /etc/cif/rules/default with the file extension .yml. Any files without the extension of .yml are ignored.
Configuration files can contain multiple feeds which provides a way to group related feeds and make use of global values. When adding a feed source not shipped by default with CIF, it is recommended to create a new config file to avoid the process of merging configs when existing feed configuration files are updated.
Both of Dragon Research Group feeds are sourced from the same provider so they inevitably share many similar configuration values, these are placed at the top of the file.
parser: pipe
defaults:
tags: scanner
protocol: tcp
provider: dragonresearchgroup.org
altid_tlp: green
tlp: amber
confidence: 85
values:
- asn
- asn_desc
- observable
- lasttime
- null
- parser: pipe - these are pipe delimited feeds
- defaults: - this is a list of feed configuration values that can be shared across both feeds
- tags: scanner - scanner is the Tag associated with this type of feed data
- protocol: tcp - this is network traffic using the TCP protocol
- provider: dragonresearchgroup.org - domain of where the feeds can be found
- altid_tlp: green - the URL to the feed data is publicly available
- tlp: amber - the feed data is free for non-commercial use only
- confidence: 85 - 85 percent confident the data is as it is described
- values: - this is a list of the pipe delimited columns
- asn - asn number provided by the feed provider
- asn_desc - asn description provided by the feed provider
- observable - the indicator being shared, usually a IP address, FQDN or URL
- lasttime - timestamp of last time seen
- null - null is used to discard data, it's similar to sending data to /dev/null
feeds:
ssh:
remote: http://dragonresearchgroup.org/insight/sshpwauth.txt
application: ssh
portlist: 22
vnc:
remote: http://dragonresearchgroup.org/insight/vncprobe.txt
application: vnc
portlist: 5900-5904
- feeds - this is a list of feed specific configuration values
- ssh - name of feed section, used with -f in cif-smrt
- vnc - name of feed section, used with -f in cif-smrt
- remote - URL to the providers feed
- application - application associated with the listening port
- portlist - port(s) associated with application
Test the ssh configuration is this config
sudo su - cif -c "csirtg-smrt r /etc/cif/rules/default/drg.yml -f ssh"
Test both the feed configurations in this config
sudo su - cif -c "csirtg-smrt r /etc/cif/rules/default/drg.yml"
