Populate more ECS fields in the Suricata module #10006
Conversation
|
Pinging @elastic/secops |
adriansr
added
ecs
Filebeat
There was a problem hiding this comment.
There are few more fields that could be populated (not sure what the degree of difficulty will be for some of them w.r.t. doing this with ingest node).
- source.bytes
- source.packets
- destination.bytes
- destination.packets
- destination.domain = suricata.eve.http.hostname
- event.start = suricata.eve.flow.start
- event.end = suricata.eve.timestamp (assuming this is marking the end)
- event.duration = end - start
- network.protocol = app_proto
- url.path = suricata.eve.http.url (not sure if this also includes the query string though)
- http.request.method should be lowercased (I think that’s what the ECS docs say, double check).
- network.type - ipv4/ipv6
- network.bytes
- network.packets
| @@ -93,7 +100,39 @@ | |||
| , "value": "{{suricata.eve.event_type}}" | |||
| } | |||
| } | |||
|
|
|||
| , {"convert": | |||
There was a problem hiding this comment.
Do we need type conversion? Otherwise we could just use the rename processor: https://www.elastic.co/guide/en/elasticsearch/reference/current/rename-processor.html
There was a problem hiding this comment.
Good point. I wonder why it was done this way. Did I do it this way initially? (If so, I'm sorry, this was my very first module LOL)
There was a problem hiding this comment.
Never mind, it just occurred to me that this was the simplest way to copy the data out (instead of renaming).
We had decided that since Suricata had a JSON format which is already very familiar to people, we would start for now by leaving the full untouched event in place, and only copy out data, instead of renaming the fields.
This is different from most modules, but most modules have field names determined by the grok patterns (us), not determined by the tool creator (and familiar to the tool's community).
(My first coffee just kicked in, haha)
There was a problem hiding this comment.
Back in the day convert was the only way we found to copy a field and failing silently if the source field doesn't exist.
That, or an ugly gigantic piece of painless.
Original discussion: #8550 (comment)
|
A few additional suggestions
And btw @ruflin for Suricata, data is copied out, not moved out of the event. So no need for aliases at this time. |
|
@webmat I think also in suricata we should follow the same logic as we do in all other modules, meaning using alias instead of copy. There are quite a few other modules where we have JSON docs and we converted them all to ECS. copy_to increases the data size and I rather have everyone rely on ECS instead of using the "old" fields. Perhaps later on we can even allow to turn on alias for a single module if the "original" fields are needed, but let's not start to duplicate data. |
|
jenkins, test this. |
|
@webmat |
There was a problem hiding this comment.
Left a few comments for things to fix, or code blocks to cut out.
One of the big Painless blocks could have been done with grok, I think (unless I'm missing something?). The other one for network stats is bugging me a bit, but it seems to work fine, so ignore my whining there ;-)
| } | ||
| , { "script": | ||
| { "lang": "painless" | ||
| , "source": "long g(def map, def key) { if(map!=null && map[key]!=null) { return map[key]; } return 0; } def n=ctx['network']; if (n==null){n=new HashMap(); ctx['network']=n;}def s=ctx['source'], d=ctx['destination']; def sp=g(s,'packets'), sb=g(s,'bytes'), dp=g(d,'packets'), db=g(d,'bytes'); if(sb+db>0)n['bytes']=sb+db; if(sp+dp>0)n['packets']=sp+dp;" |
There was a problem hiding this comment.
I'm not a big fan of this much compact Painless. If only we could do indentation here, or use saved scripts :-)
I'm surprised that this code didn't create an empty network block for the stats event. It doesn't look like this code guards against running when there's no bytes or packets involved.
In other words, this should fine after all, despite my Spidey sense tingling :-)
There was a problem hiding this comment.
"I'm surprised that this code didn't create an empty network block"
Me too. I rewrote it a bit.
|
@ruflin I'd be curious to know which tools have JSON logs this way, that we are translating instead of copying out. Note that I'm not counting the tools from the Elastic Stack, since for those, the logs are bound to move to ECS eventually at the source (so I have no problem butchering those into shape ;-) ) The reason why we decided to translate Suricata this way is that it has quite a rich community of users and tools, used to the EVE log format. So having 30 to 50% of the fields moved out of the original event would look weird. So we decided to only copy out the data, and keep the EVE event untouched. But I'm on the fence. I wouldn't make it a requirement for this PR, but eventually we could do straight renames, or introduce an option to get rid of the original fields that have been translated. I'd like to get some user feedback about this. @MikePaquette Are you aware of anyone who's been waiting for this module, who'd be willing to give us feedback about this? |
There was a problem hiding this comment.
For users that like the old mapping, we can have migration aliases or perhaps a similar feature in the future. But we should not duplicate data because of ECS. If we keep all the old fields around we don't push users in the direction of ECS and to learn ECS which decreases the overall benefit.
For now there are mainly the stack logs that are in json format, but I expect more to come. And this here would set a precedent I prefer not to have.
If we think the fields are really that important, we could introduce them as constant alias but that is something we can still add later if needed as it is not breaking.
|
@andrewkroh @tsg @MikePaquette @webmat Could you chime in here? My suggestion is to follow with the suricata module the same migration path as we do for all the other modules and have migration aliases. |
|
From my understanding of the discussion, the Suricata module already did copying, and it's released in 6.5, so we are already on that path. Note that this is not so much about ECS migration, but more that we consider that we might want to keep the duplication long term. The reason is to make the life of the Suricata users a bit easier (they might be confused that the suricata category is called Another thought that I had was that in some cases we might want to do some data manipulation when copying, say, from I don't feel strongly about it, but I'd favour copying in this particular case. |
adriansr
force-pushed
the
suricata_ecs_update
branch
from
31f6c24
to
f578e42
Compare
|
@tsg Thanks for chiming in. If the team wants to really keep the original suricata names I would suggest we then use alias for it (without the To make sure we talk about the same version: My assumption is the above should happen for 7.0 and not be backported. |
|
For new users (users not using Filebeat+Suricata in 6.x) I don't really see much benefit to adding aliases from a usability perspective. They don't show up in Kibana so they're only useful to someone that knows to search for So that leaves the decision as whether or not we should duplicate fields or remove the original Suricata field when something is copied. Users familiar with the Suricata fields will find is useful if we keep the original Suricata fields. So I'd keep the fields the around. Some users (like me) that are more familiar with meaning behind ECS fields might like an option to drop the EVE fields that have been coped elsewhere. |
|
I'm introducing alias support in Kibana / Index patterns here: #10075 This should solve the issue around not seeing the fields in Kibana. If we have that, are you ok with just having the aliases? |
|
Interesting, I'd say if that works well, we're good with aliases. Is the Kibana team on board with having index pattern entries for aliases? Just thinking if there will be any unexpected side effects. |
|
If the index pattern is generated automatically by Kibana, it's what Kibana generates. I did some testing around it and there is no issue with it. In the future I hope Kibana adds support for an alias type in the index pattern as it would allow a bit more magic: elastic/kibana#28570 |
|
Thanks for the details @ruflin. Good from my side. Any concerns @andrewkroh? |
|
jenkins, test this |
From http.url
This assumes client=source server=destination.
Populates
- source.{packets|bytes}
- destination.{packets|bytes}
- network.{packets|bytes}
Replace ugly painless code.
Lowercase processor can have a target field so its not neccesary to copy the field in a previous step.
adriansr
force-pushed
the
suricata_ecs_update
branch
from
95d014a
to
af68418
Compare
* Populate more ECS fields in the Suricata module
A few more ECS fields are populated by the ingest pipeline that enriches
Suricata's eve.json events.
Additions:
- http.request.referrer (from suricata.eve.http.http_refer)
- event.action (from suricata.eve.alert.category)
describes the action that caused the event.
Examples: "Attempted Denial of Service", "Successful Administrator Privilege Gain"
- event.outcome (from suricata.eve.alert.action)
Possible values: "allowed", "blocked"
- event.severity (from suricata.eve.alert.severity)
Possible values: 1, 2 or 3.
- network.transport (from suricata.eve.proto)
Examples: "tcp", "udp", "ipv6-icmp"
* Use message for suricata.eve.alert.category
Instead of event.action, which is expected to have a fixed set of
enumeration values.
* Populate destination.domain
When http.hostname is present.
* Populate event.{start,end,duration}
* populate network.protocol
* url.hostname is url.domain
* Populate url.path, url.fragment, url.query
From http.url
* Lowercase http request method
* Source/Destination and aggregated counters
This assumes client=source server=destination.
Populates
- source.{packets|bytes}
- destination.{packets|bytes}
- network.{packets|bytes}
* Updated golden files
* Populate ECS field `http.response.body.bytes`
* Use grok pattern to parse url fields
Replace ugly painless code.
* Avoid pairs of convert/lowercase
Lowercase processor can have a target field so its not neccesary to copy
the field in a previous step.
* Cleanup painless script
* Fix golden data
* Fix golden data (2)
* Copy timestamp to event.end instead of parsing date again
(cherry picked from commit 184149f4a18b4162b0d6c89adba3bb924a2db0b8)
adriansr
added
v6.7.0
and removed
needs_backport
…odule (#10537) Cherry-pick of PR #10006 to 6.x branch. Original message: A few more ECS fields are populated by the ingest pipeline that enriches Suricata's eve.json events. Additions: - http.request.referrer (from suricata.eve.http.http_refer) - event.action (from suricata.eve.alert.category) describes the action that caused the event. Examples: "Attempted Denial of Service", "Successful Administrator Privilege Gain" - event.outcome (from suricata.eve.alert.action) Possible values: "allowed", "blocked" - event.severity (from suricata.eve.alert.severity) Possible values: 1, 2 or 3. - network.transport (from suricata.eve.proto) Examples: "tcp", "udp", "ipv6-icmp" (cherry picked from commit 184149f4a18b4162b0d6c89adba3bb924a2db0b8)
(cherry picked from commit 2091dc4)
Previous PR (elastic#10565) put this entry in the wrong section. It's an added feature, not a breaking change.
Previous PR (elastic#10565) put this entry in the wrong section. It's an added feature, not a breaking change. (cherry picked from commit 220d053)
A few more ECS fields are populated by the ingest pipeline that enriches Suricata's eve.json events.
Additions:
http.request.referrer (from suricata.eve.http.http_refer)
event.action (from suricata.eve.alert.category)
describes the action that caused the event.
Examples: "Attempted Denial of Service", "Successful Administrator Privilege Gain"
event.outcome (from suricata.eve.alert.action)
Possible values: "allowed", "blocked"
event.severity (from suricata.eve.alert.severity)
Possible values: 1, 2 or 3.
network.transport (from suricata.eve.proto)
Examples: "tcp", "udp", "ipv6-icmp"