[Filebeat] Improve ECS categorization in iptables module #16637

leehinman · 2020-02-26T20:21:30Z

event.category
event.kind
event.type
related.ip
convert pipeline to yaml

elasticmachine · 2020-02-26T20:21:32Z

Pinging @elastic/siem (Team:SIEM)

andrewkroh · 2020-02-27T19:48:31Z

x-pack/filebeat/module/iptables/log/test/geo.log-expected.json

        "event.module": "iptables",
-        "event.outcome": "deny",


I think it still makes sense to populate this if we know what action was taken, but use success or failure.

I'll ask for some ECS input. My understanding was that allowed/denied would move to event.type. So you could have:
event.type = ["connection", "denied"]

and then event.outcome could be success or failure, based on if you were successful in denying the connection.

@leehinman that's correct.

and then event.outcome could be success or failure, based on if you were successful in denying the connection.

So isn't the fact that iptables says it blocked the connection evidence enough to set the event.outcome to failure?

Yeah I'd have to agree with @andrewkroh here. When we model the iptables events as category "network" and type "connection", looking for event.outcome:failure I would expect to find the connections denied. Not events where a firewall "failed to block a connection" -- can this actually happen?

If iptables failed to block the connection I'd be very surprised if we got something in the iptables log saying so. At best we might get something from kernel logger. So I'd expect every allowed/denied to end up with event.outcome = success. Which doesn't make event.outcome very useful.

out of band yes, its possible (websense & some IPS's used to work like that - it sees badness - sends a spoofed reset oob to kill the connection, and it didn't always work).

from an observers standpoint wouldn't the "deny" be event.action? with outcome being success/deny (with a strong default to succeed as oob is likely fairly limited in production these days).

iptables is an ugly one as it can be e.g. ubiquiti as a network firewall, or a host based firewall - where the connection event (e.g. from auditbeat) would have a event.outcome as fail, but e.g. filebeat pulling in the iptables event would be event.action:deny | block | reset, etc. with the event.outcome being specific to the action being taken (in this case the block by the firewall component)

sorry - "denied" can be one of the values in the event.type (in addition to e.g. connection), as well event.action: deny

Yeah I'm retreating from my previous comment. I agree with setting event.outcome:successful when a firewall denies a connection.

Examples of event.outcome:failure would be cases like @dainperkins mentions, where for example sending a TCP rst fails to kill the connection. Whether or not a given firewall correctly reports this is not relevant in the bigger picture (we'd simply never set it in this case).

There's other well known cases where the agent's action is not successful, like Endpoint trying to kill a process, but the process is already gone.

webmat

Here's a few suggestions that are applicable now:

Please make event.category an array of values like event.type, even when there's only one value.
I would recommend populating event.action with any raw value as per iptable. Perhaps lowercase it first to make it easier on the eyes. But essentially the goal of event.action not having "allowed values" is for allowing to relay the information as specified by the source. So event.action should be "allow", "deny", and if I understand correctly "drop_input" or any value that's in this position in the logs.
Then also map these event actions to an additional value in the event.type array, whichever fits best, either "allowed" or "denied". The value "connection" already in the array is good 👍.

Your PR didn't modify this, so feel free to address here or not. But it looks like the fields could be array fields: iptables.fragment_flags and iptables.tcp.flags?

Here's a few additional suggestions that are applicable to ECS 1.5 (so totally fine to address in a later PR, IMO):

iptables.ubiquiti.*_zone likely map to observer.ingress/egress.zone
iptables.ubiquiti.rule_* likely map to fields in the rule.* field set

So overall, I'd say only the first 3 bullet points need to be addressed here, and I'd give it my 👍 for mapping to ECS 1.4. The rest of the feedback can be addressed later :-)

andrewkroh · 2020-03-10T15:54:22Z

x-pack/filebeat/module/iptables/log/test/geo.log-expected.json

@@ -3,20 +3,32 @@
        "destination.ip": "10.4.0.5",
        "destination.mac": "90:10:20:76:8d:20",
        "destination.port": 443,
+        "event.action": "d",


Looks like this should be deny or something more than d.

This is implementing @webmat 's suggestion. The original value from ubiquiti iptables is "d" or "a"

I think expanding it to a word will be more user friendly. I have a feeling users will think it's some kind of bug like me.

sure. switched to accept/drop which is what is displayed in ubiquiti gui

Agree with switching to being more readable.

The ECS recommendation is only a high level for "this should be driven by the source". But when the underlying source transport is too compact like this "a" & "d", I think it's totally fine to expand it. Expanding it to what's in their UI is actually a great resolution here.

adriansr · 2020-03-13T16:52:27Z

libbeat/_meta/fields.ecs.yml

@@ -2074,6 +2074,8 @@
      type: keyword
      ignore_above: 1024
      description: Hostname of the observer.
+    - name: egress.zone
+    - name: ingress.zone


These fields already exist in master since the upgrade to ECS 1.5.

@adriansr Thanks for catching that. Should be fixed now.

andrewkroh

LGTM

- event.category - event.kind - event.type - related.ip - convert pipeline to yaml Closes elastic#16166

- event.action, map to accept/drop like gui - lowercase event.action - fix tcp_flags grok to get all entries - make iptables.tcp.flags an array - make iptables.fragment_flags an array - make event.category an array - map iptables.ubiquiti.output_zone -> oberserver.egress.zone - map iptables.ubiquiti.input_zone -> observer.ingress.zone - map iptables.ubiquiti.rule_set -> rule.name - map iptables.ubiquiti.rule_number -> rule.id - rebase to master

* Improve ECS categorization in iptables module - event.action, map to accept/drop like gui - event.category - event.kind - event.type - observer.egress.zone - observer.ingress.zone - related.ip - rule.id - rule.name - convert pipeline to yaml - fix tcp_flags grok to get all entries - make iptables.tcp.flags an array - make iptables.fragment_flags an array Closes elastic#16166 (cherry picked from commit d9c83df)

…7064) * Improve ECS categorization in iptables module - event.action, map to accept/drop like gui - event.category - event.kind - event.type - observer.egress.zone - observer.ingress.zone - related.ip - rule.id - rule.name - convert pipeline to yaml - fix tcp_flags grok to get all entries - make iptables.tcp.flags an array - make iptables.fragment_flags an array Closes #16166 (cherry picked from commit d9c83df)

leehinman added enhancement Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM ecs labels Feb 26, 2020

leehinman requested a review from a team as a code owner February 26, 2020 20:21

leehinman force-pushed the 16166_iptables_ecs_1.4 branch from 02ceb0c to 1b2970a Compare February 26, 2020 20:22

andrewkroh reviewed Feb 27, 2020

View reviewed changes

webmat reviewed Mar 5, 2020

View reviewed changes

leehinman force-pushed the 16166_iptables_ecs_1.4 branch from 1b2970a to afad22c Compare March 10, 2020 14:40

andrewkroh reviewed Mar 10, 2020

View reviewed changes

adriansr suggested changes Mar 13, 2020

View reviewed changes

leehinman force-pushed the 16166_iptables_ecs_1.4 branch from 506a7a4 to 2d2d5cc Compare March 14, 2020 03:25

andrewkroh approved these changes Mar 14, 2020

View reviewed changes

leehinman added 2 commits March 13, 2020 22:45

Improve ECS categorization in iptables module

ee6ee40

- event.category - event.kind - event.type - related.ip - convert pipeline to yaml Closes elastic#16166

leehinman force-pushed the 16166_iptables_ecs_1.4 branch from 2d2d5cc to 3a184a6 Compare March 14, 2020 03:55

Merge branch 'master' into 16166_iptables_ecs_1.4

b566069

leehinman merged commit d9c83df into elastic:master Mar 17, 2020

leehinman deleted the 16166_iptables_ecs_1.4 branch March 17, 2020 19:53

leehinman mentioned this pull request Mar 17, 2020

Cherry-pick #16637 to 7.x: [Filebeat] Improve ECS categorization in iptables module #17064

Merged

leehinman added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 17, 2020

leehinman mentioned this pull request Apr 15, 2020

Checkpoint Syslog Filebeat module #17682

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Improve ECS categorization in iptables module #16637

[Filebeat] Improve ECS categorization in iptables module #16637

leehinman commented Feb 26, 2020 •

edited by zube bot

Loading

elasticmachine commented Feb 26, 2020

andrewkroh Feb 27, 2020

leehinman Feb 27, 2020

MikePaquette Feb 28, 2020

andrewkroh Mar 2, 2020

webmat Mar 5, 2020

leehinman Mar 5, 2020

dainperkins Mar 5, 2020

dainperkins Mar 5, 2020 •

edited

Loading

webmat Mar 5, 2020

webmat left a comment

andrewkroh Mar 10, 2020

leehinman Mar 12, 2020

andrewkroh Mar 13, 2020

leehinman Mar 13, 2020

webmat Mar 17, 2020

adriansr Mar 13, 2020

leehinman Mar 14, 2020

andrewkroh left a comment

[Filebeat] Improve ECS categorization in iptables module #16637

[Filebeat] Improve ECS categorization in iptables module #16637

Conversation

leehinman commented Feb 26, 2020 • edited by zube bot Loading

elasticmachine commented Feb 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dainperkins Mar 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

leehinman commented Feb 26, 2020 •

edited by zube bot

Loading

dainperkins Mar 5, 2020 •

edited

Loading