Add network directions ingress and egress

[webmat]

Adding these two values based on the discussion in #791.

I've only lightly modified the description to associate which values to use for host based, and which ones should be associated to perimeter based. Feedback welcome, if folks would like more adjustments to the description.

I think we should discuss whether this should be considered a breaking change. My opinion is that for folks that have dutifully been populating inbound/outbound, this will feel like a breaking change. On the other hand based on feedback so far, the usefulness of the field was hampered by the difficulty of using the same pair of values "depending on context". So I would lean towards getting this into ECS in the 1.x line, and not waiting for 2.0.

Closes #791

[webmat]

ping @dcode @leehinman @dainperkins

[dcode]

I like it. LGTM

[webmat]

ping @andrewkroh

[webmat]

I'd like to get your thoughts about implementation in Beats, @andrewkroh or @leehinman. Any concerns?

[andrewstucki]

So, a couple of thoughts. For our current ECS implementation, it seems like we already have the semantics for defining traffic in internal ip space (internal) v. something involving external ips (external)--which really seem to be what the re-definition of inbound v. outbound are all about (basically a network boundary traversal == external).

The thing that appears to be missing from internal v. external is, interestingly enough, a sense of "directionality" (the irony isn't lost on me that this is network.direction we're talking about).

If you wanted to indicate that network coming into a host/network was from public ip space, it seems to me that the current definition of [inbound, external] would be functionally equivalent to and require re-categorization as [inbound, ingress]. Likewise [outbound, external] would turn into [outbound, egress].

So, overall I'm somewhat indifferent about/against this change for a couple of reasons:

1. we already have the means for tagging network boundary traversals
2. the directionality components of inbound/ingress and outbound/egress are basically always tied together
3. in common usage, inbound and ingress (as well as outbound and egress) are often interchangeable (for example AWS Security Group Rules), so from an implementation, and even threat-hunting perspective, I would imagine this would get confusing.

That said, if we do see this as a substantive change, I'd want to consider deprecating internal and external because they seem to serve roughly the same purpose as the re-definition.

Alternatively, what about just shoring up the documentation and saying explicitly internal/external should be used for referring to network traversals while inbound/outbound are about host-level traffic direction? Or even just renaming inbound/outbound as ingress/egress? Would either of those two alternatives address the concern?

[ebeahan]

Looking back at the issue posed in #791, I think both adding ingress and egress as recommended values and the guidance on the usage of ingress/egress vs. internal/external/inbound/outbound should help address the current ambiguity.

Is the use of unknown acceptable for either host-based or network-based situations?

We could provide more context around the expected use of internal, external, inbound, outbound, and unknown from the network perimeter's perspective. Based on my understanding, if you have some concept of a "home or "local" network defined, that info can be used to determine if your originator and responder are "local" or not.

originator local?	responder local?	direction
false	false	external
false	true	inbound
true	false	outbound
true	true	internal
undefined	undefined	unknown

[webmat]

@andrewstucki Let me take another stab at trying to clarify this. The current definition really doesn't expand on internal & external. But I think it's clear we'll need to expand more in the actual docs based on this.

Host monitoring perspective (sources could be EDR, Packetbeat on a host)

• ingress: traffic originating from outside the host, to an address local to the host
• egress: traffic originating from the host, towards a remote address

Perimeter monitoring perspective (sources could be firewall, router, Packetbeat on a network tap)

• inbound: traffic originating from outside the network perimeter, towards a host on the network being monitored.
• outbound: traffic originating from a host on the network being monitored, to the outside of the network perimeter.
• internal: traffic between two hosts on subnets that are known to be inside the perimeter.
• external: traffic between two hosts on external networks.
  • This would typically only be useful to folks monitoring backbone, ISPs, public VPN services, or other situations where one's own network is acting as a conduit between two untrusted networks.

Prior to this PR, "inbound" and "outbound" were meant to capture both kinds of incoming and outgoing traffic, and the distinction between the different situations was a hand-wavy "interpret this depending on the data source".

So splitting off ingress/egress out of inbound/outbound is meant to make it more straightforward to interpret either case, when writing detection rules and the like, without having to rely on another factor to determine "is this host-based or perimeter-based?" Now it's entirely captured in the values in network.direction.

Unknown: we should keep it. It can be useful on high traffic lossy taps that miss the SYN/ACK on some flows. It may also be useful for some event sources that don't give a clear indication of who's the initiator. For example some auditd events only refer to local/remote IPs.

[webmat]

Another note that's unrelated to the schema itself.

To reliably determine what's inside/outside the network perimeter, user configuration may be needed.

Heuristics based on standard private IP ranges may be fine as a default baseline a product can start from. However private IP addresses can be considered "outside" the perimeter in some situations, and public IP addresses can be considered "inside" the perimeter in some other situations.

Although I don't think the point I'm making here needs to be explicit in the network.direction field definition.

[webmat]

@elasticmachine, run elasticsearch-ci/docs

[webmat]

I rebased this on the most recent master branch for merging.

I've also added a paragraph to the description to clarify the meaning of "internal" and "external".

We can cover usage in more detail after #988 merges, and we work on the usage docs for the network field set.

This is ready for a final review.

[ebeahan]

LGTM