This repository has been archived by the owner on May 23, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 62
FAQ: Parsing Feeds
Gabriel Iovino edited this page Dec 17, 2015
·
2 revisions
cif-smrt is configured by default to only ingest records with a timestamp that matches the same day of the feed being parsed. Example:
Feed:
1.1.1.1, 2015-12-15
2.2.2.2, 2015-12-16
Config:
parser: csv
defaults:
provider: example.com
tlp: green
altid_tlp: white
confidence: 85
alt_tlp: green
tags:
- botnet
feeds:
scanners:
remote: <url>
values:
- observable
- lasttime
If you were to parse that feed on 2015-12-16, the records with a timestamp (lasttime) of 2015-12-15 would be skipped. A reason for this is, some people create feeds that never expire records. Once you parse that feed, you do not need to ingest records that have already been ingested in previous days.
What are some ways around this?
- You can not parse out the timestamp and cif-smrt will stamp with records with the current day.
...
feeds:
scanners:
remote: <url>
values:
- observable
- null
- You can instruct cif-smrt to via /etc/cif/cif-smrt.yml to ingest records with a timestamp X days ago.
Example:
$ sudo cat /etc/cif/cif-smrt.yml
---
client:
remote: http://localhost:5000
token: <token>
notbefore: '7 days ago'
You need to parse the feed multiple times parsing out the different observable each time. Example:
Feed:
# IP, FQDN, Timestamp
1.1.1.1, one.example.com, 2015-12-16
Config:
parser: csv
defaults:
provider: example.com
tlp: green
altid_tlp: white
confidence: 85
alt_tlp: green
tags:
- botnet
feeds:
botnet-ip:
remote: hxxp://example.com/botnet.csv
values:
- observable
- null
- lasttime
feeds:
botnet-fqdn:
remote: hxxp://example.com/botnet.csv
values:
- null
- observable
- lasttime