[SIEM][Detection Engine] Signal ECS Mappings

[FrankHassanabad]

current signal schema - 10/18/2019

In the new index all the ECS fields are being copied over and the new signal meta object tracks the additions listed below:

field	description	example	data type
@timestamp	time the rule ran if available	2019-10-01T08:20:04.323Z	date
parent.index	the index the parent event came from	auditbeat-8.0.0	keyword
parent.id	original document id	CsFDe5df	keyword
parent.type	If the parent is an event or another signal	event	keyword
parent.depth	current depth of the tree the parent is at	1	long
original_time	original date time of the event	2019-10-01T08:20:04.323Z	date
rule_revision	current rule revision	21	long
rule_id	rule id from the UI	siem-rule-1	keyword
rule_type	type of the rule	KQL	keyword
rule_query	query of the rule	host.name: someHost	keyword
index_patterns	index patterns the rule was run against	[winglogbeat-*, auditbeat-*]	keyword
name	name of the rule or origin of the signal such as ML	ml-anomaly-hosts	keyword
description	description of the rule or origin	Host KQL Signal	keyword
severity	How severe the signal is. string of low, medium, high, critical	low	keyword
references	Any url links for references.	example.com	keyword

History

version 0 - 10/01/2019

field	description	example	data type
parent.id	original document id	CsFDe5df	keyword
parent.type	If the parent is an event or another signal	event	keyword
parent.depth	current depth of the tree the parent is at	1	long
time_detected	time the rule ran if available	2019-10-01T08:20:04.323Z	date
rule_revision	current rule revision	21	long
rule_id	rule id from the UI	siem-rule-1	keyword
rule_type	type of the rule	KQL	keyword
rule_query	query of the rule	host.name: someHost	keyword
index_patterns	index patterns the rule was run against	winglogbeat-, auditbeat-	keyword]
name	name of the rule or origin of the signal such as ML	ml-anomaly-hosts	keyword
description	description of the rule or origin	Host KQL Signal	keyword
severity	How severe the signal is. Number from 1 to ... N	1	long
references	Any url links for references	example.com	keyword
error.code	Any custom code for error handling specific to signal creation	5	keyword
error.id	Any custom id for error handling specific to signal creation	1	keyword
error.message	Any custom message for error handling specific to signal creation	Could not generate signal	text
error.type	Any custom error type specific to signal creation	elastic_error	keyword

Why a signals sub-object?

This is the current Signal ECS mapping that the SIEM UI team is using. Since some events already have severity filled in them:

  "event": {
    "sequence": 240,
    "category": "network_traffic",
    "outcome": "deny",
    "dataset": "cisco.ios",
    "code": "IPACCESSLOGRP",
    "type": "firewall",
    "severity": 6,
    "module": "cisco"
  },

and some have errors already filled in

  "error": {
    "message": "failed to hash executable /usr/local/go/pkg/tool/linux_amd64/vet for PID 144885: failed to stat file /usr/local/go/pkg/tool/linux_amd64/vet: stat /usr/local/go/pkg/tool/linux_amd64/vet: no such file or directory"
  },

we have chosen to not overwrite the fields of a document that a signal is being created from when copying the document to the new index by adding a sub-object called signal.

[elasticmachine]

Pinging @elastic/siem

[MikePaquette]

Initial Brain Dump Reaction for Discussion:

Thanks for getting this ball rolling! An excellent starting list.

1. What about event.risk_score and event.risk_score_norm ? These are event fields, but in practice, we're only going to update them in signals. Should we allow updating some event.* fields in the signal document? Or should we always try to preserve the original values of the underlying events? (Realizing that there maybe zero or more events associated with a signal.)
2. Is the signal.severity field necessary, or should we switch to a risk-based scoring system? (see comment above) . We could make this a default risk score for the signal, or the rule could contain code that allowed this field to be modified based on other factors.
3. We should populate event.kind: "signal"
4. What about the threat.* fields such as threat.tactic.references ? These might be populated by the event-generating entity, but also might be relevant to the rule. Do we overwrite them? or create a copy of them in the signal.* ?
5. parent.type should define a way to represent "none" for the case when there are no identifiable events or signals to associate with the newly created signal.
6. Re: time_detected, I think we should populate @timestamp with this information, so we can use time-histograms, ML, and other standard tools to display and visualize signals without specifying a custom time field for the index pattern.

[MikePaquette]

I think the idea of a signal.* field set makes great sense.

There is one fundamental question about signals, which is what should @timestamp represent. I think signals are a separate logical entity and separate document vs. the source events that might trigger their creation, and therefore the @timestamp field in a signal document should be populated with the time that the signal is created so we can use time-histograms, ML, and other standard tools to display and visualize signals without specifying a custom time field for the index pattern. Also, doing so will facilitate correctly time-sequenced displays of source events vs. signals, and parent signals vs. child signals.

Other recommendations:

1. Remove signal.severity and replace it with signal.risk.score and signal.risk.tag as proposed in Risk Fields ecs#518. For SIEM, users prefer signal prioritization based on risk rather than severity. SIEM detection rules will specify how the signal.risk.* fields should be populated, for example, by writing a constant value, or by calculating an aggregate score based on various risk scores contained within other ECS field sets.
2. Populate event.kind: "signal"
3. Define signal.threat.* fields per threat fields ecs#505. SIEM detection rules can specify how to populate them e.g., {copy_original, copy_original_and_append_rule_specified_values, use_rule_specified_values}
4. 'signal.parent.type` should define a way to represent "none" for the case when there are no identifiable events or signals to associate with the newly created signal.

[randomuserid]

1. Usually severity is set in a rule and as the opinion of the rule author and is the same everywhere unless someone decides to adjust it. Risk scores are organization-specific and adjusted based on factors like asset business values and compensating controls.
2. Yes we need some way to discern a signal from an atomic event
3. Yes the ML signals - both ours and Endgame's - may not come from a document, or the EQL signals may come from a search result on a series of documents, or may be even a set of signals.

[randomuserid]

One more thing - I don't see all of the decorations people want from the Detail view field list in the original design doc: https://docs.google.com/document/d/1cwith7P19asZJ178cZBB2TJ6XT6DZvCcKPBEi8gl-TU/edit#
I take it these are going in "threat" fields. I had broken them out on the assumption people would want to search by type - either for ATT&CK categories or threat intel IOC matches. And will be be able to do aggregations and counts if these are all put into arrays? Are they going in arrays?

[MikePaquette]

@randomuserid you are correct, we've not addressed the IOC Matching fields anywhere yet. I think we'll need a follow-on PR after elastic/ecs#505

[randomuserid]

What about the severity rating - are we agreed on keeping it or should I elaborate further

[randomuserid]

Also, severity usually gets mapped to labels like critical, high, medium, low, informational. I used to map numbers to labels in searches but it can be simpler to use labels than to maintain mappings between numbers and names.

[MikePaquette]

@FrankHassanabad

A few more items:

• Per [SIEM][Detection Engine] Signal ECS Mappings #47015 (comment) can we change event.kind to "signal" ?
• FYI, I am proposing a new re-usable ECS field set rule.* Add Rule Fields for Observers and Agents ecs#598 and if approved and merged, we should replace the signal.rule* fields with the new ECS rule fields.
• Question: should the signal write its own value for ecs.version ?
• Question: why do we need signal.index_pattern? Should this be part of the "rule" details instead?
• Outstanding in ECS, but needed once ECS defines: event.category and event.type - are these specific to signals, or should they be derived from any source events?
• Question: Is `signal.references' meant to be a link to the signal itself, or the rule that caused the signal to be generated? Do we need both?

Thanks

[randomuserid]

signal.reference is typically a link to some research or discourse which resulted in the signal being created and the nature of the TTPs that are being detected

[FrankHassanabad]

This ticket is no longer being updated. We will have a few minor tweaks moving forward to the signals schema but for the most part this is completed.