Timestamp

Timestamps

CIF supports three separate timestamps per record or observation: (reporttime, lasttime, firsttime). A record should have at least one timestamp associated with it and could have up to three timestamps.

Definitions

reporttime

This is the timestamp of when the record or observation was given to you.

lasttime

This is a machine generated timestamp of the last time the source observed the behavior. This would be the most recent timestamp found in machine generated logs where the host is leveraging clock synchronization (NTP).

firsttime

This is a machine generated timestamp of the first time the source observed the behavior. This would be the earliest timestamp found in machine generated logs where the host is leveraging clock synchronization (NTP).

Example

An information sharing partner may give you the following intelligence:

address      portlist  protocol  firsttime             lasttime              description
192.168.1.1  22        tcp       2016-06-18T00:00:00Z  2016-06-18T10:10:00Z  scanner

If you were to ingest this record into CIF at 2016-06-18T12:00:00Z, you could associate these three timestamps with this single observation:

 firsttime: 2016-06-18T00:00:00Z
  lasttime: 2016-06-18T10:10:00Z
reporttime: 2016-06-18T12:00:00Z

CIF-SMRT

When using cif-smrt to ingest intelligence into CIF, cif-smrt will automatically fill in lastime, reporttime and firsttime if those values are not specified. It's not uncommon to see the exact same timestamp when dealing with a feed that does not give any timestamps. Here's an example of the alienvault feed:

{
    "lasttime" : "2016-05-24T13:01:52Z",
    "firsttime" : "2016-05-24T13:01:52Z",
    "reporttime" : "2016-05-24T13:01:51Z",
    "tlp" : "white",
    "tags" : ["suspicious"],
    "altid" : "https://reputation.alienvault.com/reputation.data",
    "description" : "Scanning Host",
    "altid_tlp" : "white",
    "asn" : "8075",
    "confidence" : 65,
    "group" : ["everyone"],
    "provider" : "reputation.alienvault.com",
    "observable" : "13.84.219.191",
    "otype" : "ipv4",
}

Query Examples

A typical CIF query is to return x data over y period. The "y period" can be nuanced as you have two common choices:

1. lasttime
2. reporttime

If the set of records have lasttime and reporttime specified and the delta between those values is large, the data returned could be rather different when choosing to filter on lasttime vs reporttime.

The CIFv2 CLI clients default timestamp choice is almost always reporttime. When the lasttime and reporttime values are the same as the alienvault example above, the returned results are very likely what you expect. In the scenario where you know there is a large delta between lasttime and reporttime and you know you want the period to be based on lasttime you'll want to make sure you are being specific in your queries.

Perl CLI examples

Queries using reporttime

Under the hood these queries use the API parameters reporttime and reporttimeend

1. --today return results for the current day starting at T00:00:00Z

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --today

1. --last-hour return results for the current day and the current hour between 00:00Z - 59:59Z

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --last-hour

1. --last-day return results for the previous 24 hours from the current time.

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --last-day

1. --days [int] return results for the previous two days from the current time.

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --days 2

Queries using lasttime and firsttime

When you know that you want to query on the machine generated timestamp (lasttime) you will need to leverage the filters lasttime and firsttime.

1. return results for the current day starting at T00:00:00Z

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --firsttime 2016-05-24T00:00:00Z --lasttime 2016-05-24T23:59:59Z

1. return results for the current day and the current hour between 00:00Z - 59:59Z

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --firsttime 2016-05-24T15:00:00Z --lasttime 2016-05-24T15:59:59Z

1. return results for the previous 24 hours from the current time.

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --firsttime 2016-05-23T15:13:59Z --lasttime 2016-05-24T15:14:00Z

1. return results for the previous two days from the current time.

$ cif --otype fqdn -c 85 --provider osint.bambenekconsulting.com --firsttime 2016-05-22T15:13:59Z --lasttime 2016-05-24T15:14:00Z