Alternate methods for fetching and parsing data sets
CIF ships with a utility named cif-smrt to fetch and parse threat intelligence feeds. There will be times when cif-smrt does not have the features or capabilities that are needed to fetch or normalize the data appropriately. Here are a few alternate solutions for those scenarios.
You can create a BASH, Python or Perl script to fetch, normalize (if needed) and write the file to the local file system. You can place this script in the CIF users crontab and write the file locally (e.g. /home/cif/data/data.csv). You would then create a cif-smrt configuration file and place the local file in the remote parameter of the feed configuration file (e.g. remote: /home/cif/data/data.csv).
cif-smrt is a tool that fetches, parses and injects data to CIF using the CIF API. You can just as easily leverage the API to ingest data into CIF. Here are two example projects that demonstrate how to do this: py-cifapwg and py-cifcleanmx
