FAQ: Parsing Feeds

Why are some records in a feed not picked up by cif-smrt?

cif-smrt is configured by default to only ingest records with a timestamp that matches the same day of the feed being parsed. Example:

Feed:

1.1.1.1, 2015-12-15
2.2.2.2, 2015-12-16

Config:

parser: csv
defaults:
  provider: example.com
  tlp: green
  altid_tlp: white
  confidence: 85
  alt_tlp: green
  tags:
    - botnet

feeds:
  scanners:
    remote: <url>
    values:
      - observable
      - lasttime

If you were to parse that feed on 2015-12-16, the records with a timestamp (lasttime) of 2015-12-15 would be skipped. A reason for this is, some people create feeds that never expire records. Once you parse that feed, you do not need to ingest records that have already been ingested in previous days.

What are some ways around this?

1. You can not parse out the timestamp and cif-smrt will stamp with records with the current day.

...

feeds:
  scanners:
    remote: <url>
    values:
      - observable
      - null

2. You can instruct cif-smrt to via /etc/cif/cif-smrt.yml to ingest records with a timestamp X days ago.

Example:

$ sudo cat /etc/cif/cif-smrt.yml 
---
client:
  remote: http://localhost:5000
  token: <token>
  notbefore: '7 days ago'

I have a feed with multiple observables in the same record, how do I correctly parse that record?

You need to parse the feed multiple times parsing out the different observable each time. Example:

Feed:

# IP, FQDN, Timestamp
1.1.1.1, one.example.com, 2015-12-16

Config:

parser: csv
defaults:
  provider: example.com
  tlp: green
  altid_tlp: white
  confidence: 85
  alt_tlp: green
  tags:
    - botnet

feeds:
  botnet-ip:
    remote: hxxp://example.com/botnet.csv
    values:
      - observable
      - null
      - lasttime

feeds:
  botnet-fqdn:
    remote: hxxp://example.com/botnet.csv
    values:
      - null
      - observable
      - lasttime