[Suricata] Update fields and paths #8550

adriansr · 2018-10-03T17:39:12Z

This PR updates the suricata module to:

Add top-level source_ecs to compensate for source in 6.x.
Use ECS fields where appropriate.
Include mappings for src/dst geo points.
Use the correct path for eve.json
- Windows: Path used by msi package install.
- macOS: Path used by brew package install.
Fields from eve.json added to template

x-pack/filebeat/module/suricata/eve/_meta/fields.yml

andrewkroh · 2018-10-03T22:44:53Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

      , "ignore_missing": true
      }
    }

  , { "geoip":
      { "field": "suricata.eve.src_ip"
-      , "target_field": "ecs.source.geo"
+      , "target_field": "suricata.eve.src_geo"


ECS doesn't yet have a source.geo or a destination.geo but it will. So for master, where we don't have the source field conflict, we should put the geo data under those fields.

Do you already have a plan for dealing with the ECS source conflict with the Filebeat source for 6.x?

Perhaps we could just rename all uses of the ECS source field to source_ecs when we backport.

My idea was to have this branch directly mergeable into 6.x to speed up things, so no source usage yet.

I thought it makes more sense to not create an intermediate field (i.e. source_ecs) just to have it replaced with source soon. So I leave this under suricata.eve for now and keep those fields too for 7.x, adding also source.

Makes any sense?

I follow your thinking.

My idea with source_ecs is that this would be the only difference between the module in 6.x vs 7.x. My hope is that this would make forward upgrades easier for early adopters of the module. We can even warn users in advance that we will definitely be breaking the source_ecs field in 7.x by renaming it to source.

We can't guarantee that we won't break other fields, but if they build out any security content (dashboards, alerts) it will be easier to update this content. I guess this will even make it easier for us with the dashboards we create for 6.x.

andrewkroh · 2018-10-03T22:47:39Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

-      { "field": "ecs.user_agent.os_minor"
-      , "target_field": "ecs.user_agent.os.minor"
+      { "field": "user_agent.os_minor"
+      , "target_field": "user_agent.os.minor"
      , "ignore_missing": true
      }
    }

  , { "geoip":
      { "field": "suricata.eve.src_ip"


It looks like we should "promote" this up to the source.ip.

Same answer as above, I was refraining from using source.

webmat

It's looking good overall, but I have a few questions I've added here and there.

filebeat/_meta/fields.common.yml

x-pack/filebeat/module/suricata/eve/_meta/fields.yml

webmat · 2018-10-04T14:25:43Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+
+  , { "set":
+      { "field": "http.request.method"
+      , "value": "{{suricata.eve.http.http_method}}"


The set operations I had done initially were on fields that were always present. How does it behave for missing fields? Does it create an empty http.request.method field? If that's the case, we must find a way that doesn't create empty fields. Because the mappings you just added are for http events, but you still have dns, tls, ftp, smtp and a whole bunch more to map. We can't end up with all of them defined (and empty) on each event :-)

All of these mappings remind me that we should likely add a new processor that can do mass rename/copies with light type coercion (e.g. string to int or float, mostly). I could have sworn I had opened an issue for this. Because this is very painful at the moment.

I wonder whether Beats or Ingest Node would be a better place to have this. Perhaps Ingest Node? @adriansr @andrewkroh @tsg What do you think?

adriansr

(I'm having some github issues with my messages disappearing / showing up later, hope you can follow up the discussion)

adriansr · 2018-10-04T16:30:56Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+      { "lang": "painless"
+      , "source": "if (ctx.suricata.eve.src_ip != null) { if (ctx.source_ecs == null) ctx.source_ecs = new HashMap(); ctx.source_ecs.ip = ctx.suricata.eve.src_ip; }"
+      }
+    }


@webmat what do you think of this way to conditionally set a field only when the source field exists? I'm not familiar with ingest pipelines or painless

Also, it looks like we have the option (not tested) of using the convert processor. Set type to be the original type and use ignore_missing and target_field. It's hacky but slightly cleaner than the above painless.

There's a lie hidden in the name "Painless". I cannot imagine myself doing this for 50+ fields. And that's just Suricata...

I haven't looked at the convert processor. I'll go have a look.

But I do think we need a more declarative and compact way to do it. Imagine this was an Ingest Node plugin:

{ "normalize": [ { "source": "suricata.eve.event_type", "target": "event.type", "error_missing": true }, { "source": "suricata.eve.http.status", "target": "http.response.status_code", "conversion": "integer" }, { ... } ] },

By default error_missing = false, so the long list of fields we may have to convert would not need to specify that. But we could still force an error to be added to the event when really critical fields are missing, as I'm showing in the first event.

This example is loosely based on a plugin I was proposing we add to Logstash elastic/logstash#9768 (for Suricata, when I started it as a LS module). Although I didn't keep pushing it when we moved Suricata to Beats. So it's still in the idea phase at this time for LS.

Yeah convert could work, and it's not that hacky. But I feel that having one processor definition per field is pretty heavy. Plus the default is not to ignore missing, which makes the pipeline definition heavier still. And since it's specifically tailored to convert, the type field is mandatory, when in this case I'd say 80%+ of the fields are straight renames that should remain as strings 😂

But I'd say let's go this way for now, it's the simplest approach, I think. The important part is to not have tons of empty fields :-)

{ "convert": { "field": "suricata.eve.event_type", "target_field": "event.type", "type": "string" } }, { "convert": { "field": "suricata.eve.http.status", "target_field": "http.response.status_code", "type": "integer", "ignore_missing": true } }, ...

Bad news, conversion doesn't seem to work:

Error loading pipeline: Error loading pipeline for fileset suricata/eve: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"},"suppressed":[{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [text] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}}]}],"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"},"suppressed":[{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [text] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}}]},"status":400}. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"},"suppressed":[{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [text] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}}]}],"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"},"suppressed":[{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [ip] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [text] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}},{"type":"parse_exception","reason":"[type] type [keyword] not supported, cannot convert field.","header":{"processor_type":"convert","property_name":"type"}}]},"status":400}```

My mistake, I was using elasticsearch types when it expects JSON types. It works

adriansr · 2018-10-04T23:56:17Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+      { "lang": "painless"
+      , "source": "void createAndSet(def ctx, List components, def value) { int n = components.getLength(); try { def map = ctx; for (int i = 0; i < n - 1; i++) { map = map.computeIfAbsent(components.get(i), x -> new HashMap()); } map[components.get(n-1)] = value; } catch (Exception e) { } } def getValue(def ctx, List components) { int n = components.getLength(); try { def map = ctx; for (int i = 0; i < n - 1; i++) { map = map.get(components.get(i)); if (map == null) return null; } return map[components.get(n-1)]; }catch(Exception e) { return null; } } List split(String path) { List res = new ArrayList(); int pos = 0; int next = -1; while ( (next=path.indexOf('.',pos))>=0) { if (pos < next - 1) { res.add(path.substring(pos, next)); } pos = next + 1; } if (pos < path.length()) { res.add(path.substring(pos)); } return res; } void copyIfPresent(def ctx, String source, String dest) { def val = getValue(ctx, split(source)); if (val != null) { createAndSet(ctx, split(dest), val); } } int n = params.list.getLength(); for (int i=0; i<n; i+=2) { copyIfPresent(ctx, params.list.get(i), params.list.get(i+1)); }"
+      , "params": {
+      "list": [


This ugly painless does the job of creating an ECS copy of a field only if it exists in the event. Too bad it can't be formatted or documented properly, but at least I've been able to parametrize the field names.

@webmat WDYT?

Now that I see that convert works, I guess convert is clearer than this ugly script?

Alternative rename on the Beats side could be used.

I'd go for Ingest Node convert.

If we want to keep the original Suricata object around, we can't use Beats rename

webmat

LGTM

filebeat/_meta/fields.common.yml

adriansr · 2018-10-09T15:40:33Z

@webmat @andrewkroh I've pushed some fixes, please have a look

outdated

webmat

Some of the fields that are being copied over to the ECS field names are not being used everywhere. Examples of that are: both the IP and port of the source and destination in both of the saved searches. We're still using the suricata.eve.* fields there.

Some well defined ECS fields are not being copied to their ECS location from the original Suricata object. However I think it's ok for now, as the massive amount of copying required to move most fields will be easier once we have a better way than the current field by field approach with the convert filter. So we'll need to revisit anyway.

Ingestion and dashboards all look fine to me otherwise.

So all in all I'm good with this as is, since we'll need to revisit it again later.

I'd rather use the ECS IP & port fields in both saved searches, but since we need to revisit anyway, perhaps it doesn't matter. You can fix those now or keep them for later, I'm good with both approaches.

I've left a few comments here and there, but since I'm not sure of the team's usual approach, I'm not marking them as blockers for this PR.

webmat · 2018-10-15T14:19:47Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+      {"field": "suricata.eve.http.http_user_agent"
+      ,"target_field": "user_agent.original"
+      ,"type": "string"
+      ,"ignore_missing": true


Previous was more compact (hadn't noticed the params initially). But the single line of Painless was inscrutable. I much prefer this, even though it's more verbose. It's really hard to make a programming error in this JSON :-)

Plus when we get a massive rename/coerce processor in IN we can make all of this more compact again.

webmat · 2018-10-15T14:23:20Z

x-pack/filebeat/module/suricata/eve/_meta/fields.yml

@@ -6,6 +6,54 @@
    - name: event_type
      type: keyword

+    - name: vars


I think trying to define the booleans under vars is a losing game. Isn't this super arbitrary depending on the kinds of messages we're getting? And actually, are they detected as booleans without this, or are they interpreted as strings? Does the JSON contain "true" or true?

Yes, they are true (not a string).

These fields definitions come from a generator, so it outputs all the fields it finds in the provided eve.json.

Ok, I'm happy you didn't have write them all by hand, then :-)

This doesn't change the fact that I get the sense that these fields are very message-specific. So if we didn't happen to generate messages containing other fields from our 2 respective computers, then we're missing an unknown amount of fields here. It's in that sense that I'd rather not declare them at all.

But perhaps this is just my OCD. Perhaps nobody will notice and perhaps this is fine.

webmat · 2018-10-15T19:25:36Z

x-pack/filebeat/module/suricata/eve/_meta/fields.yml

+        - name: software_version
+          type: keyword
+
+    - name: stats


I'd be inclined to skip all of the integers under stats. They're already numerics in the JSON logs, and will therefore be autodetected as long.

Do we have a policy of trying to define as many fields as possible in modules (even when it's basically a no-op)? I understand that if the defaults change this could cause problems, but this seems like a lot of overhead for something that doesn't change the outcome.

I understand that if they are removed from the template, the fields won't be available in Kibana i.e. for Dashboards?

That's incorrect. This is just declaring these field types explicitly. They will all end up in ES the same whether this is there or not. Kibana will let you use them in dashboards etc regardless of this. If they are in an ElasticSearch index, you can use them in Kibana, there's no other dependency.

As we just discussed, we'll keep stats and get rid of vars. Keeping stats around will ensure the Kibana index pattern is pre-populated with the numeric stats, which could eventually be used in visualizations we ship with the module. So it makes sense to pre-populate those with the index template.

Ingest pipeline updated accordingly

These come from a somewhat boring eve.json

Some painless magic

adriansr added the review label Oct 3, 2018

adriansr requested review from webmat and andrewkroh October 3, 2018 17:39

andrewkroh reviewed Oct 3, 2018

View reviewed changes

adriansr force-pushed the feature-suricata-fields branch from 91cb8f1 to 8c4c4c0 Compare October 4, 2018 11:01

webmat reviewed Oct 4, 2018

View reviewed changes

adriansr commented Oct 4, 2018

View reviewed changes

webmat previously approved these changes Oct 5, 2018

View reviewed changes

filebeat/_meta/fields.common.yml Show resolved Hide resolved

webmat approved these changes Oct 15, 2018

View reviewed changes

adriansr added 15 commits October 16, 2018 13:51

Update eve.json paths

957aa9a

Remove ecs prefix from fields, some renames and http fields

d5e1b31

Ingest pipeline updated accordingly

Added region_iso_code to geolocation fields

91044eb

Create source_ecs for 6.x compatibility

8b1b1a6

Update dashboards fields

1eb261a

Restore src_ip and dest_ip fields

d17d60c

Added fields to suricata.eve

9953493

These come from a somewhat boring eve.json

Replace usages of set/painless with convert

7e0c95a

Populate ECS fields only if source exists

a2ced16

Some painless magic

Replace painless with convert

9c74b65

More fields from long running suricata

04d1704

Update fields in Alert dashboard

2ce28b8

Update overview dashboard

2233e01

Missing source and destination port fields

26bd7c5

Use ECS fields in suricata dashboards' saved searches

60c6532

adriansr force-pushed the feature-suricata-fields branch from 3db862b to 60c6532 Compare October 16, 2018 11:51

Remove uninteresting suricata fields

dd0d4e9

adriansr force-pushed the feature-suricata-fields branch from 03e97f6 to dd0d4e9 Compare October 16, 2018 14:25

adriansr merged commit 4d60173 into elastic:feature-suricata Oct 16, 2018

adriansr mentioned this pull request Jan 11, 2019

Populate more ECS fields in the Suricata module #10006

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Suricata] Update fields and paths #8550

[Suricata] Update fields and paths #8550

adriansr commented Oct 3, 2018 •

edited

Loading

andrewkroh Oct 3, 2018

andrewkroh Oct 3, 2018

adriansr Oct 3, 2018 •

edited

Loading

andrewkroh Oct 3, 2018 •

edited

Loading

andrewkroh Oct 3, 2018

adriansr Oct 3, 2018

webmat left a comment

webmat Oct 4, 2018

webmat Oct 4, 2018

adriansr left a comment

adriansr Oct 4, 2018

adriansr Oct 4, 2018

webmat Oct 4, 2018

webmat Oct 4, 2018 •

edited

Loading

adriansr Oct 4, 2018

adriansr Oct 5, 2018

adriansr Oct 4, 2018

adriansr Oct 5, 2018

ruflin Oct 5, 2018

webmat Oct 5, 2018

webmat left a comment

adriansr commented Oct 9, 2018

webmat left a comment

webmat Oct 15, 2018

webmat Oct 15, 2018

adriansr Oct 16, 2018

webmat Oct 16, 2018

webmat Oct 15, 2018

adriansr Oct 16, 2018

webmat Oct 16, 2018

webmat Oct 16, 2018

[Suricata] Update fields and paths #8550

[Suricata] Update fields and paths #8550

Conversation

adriansr commented Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

andrewkroh Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

adriansr commented Oct 9, 2018

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr commented Oct 3, 2018 •

edited

Loading

adriansr Oct 3, 2018 •

edited

Loading

andrewkroh Oct 3, 2018 •

edited

Loading

webmat Oct 4, 2018 •

edited

Loading