Re-introduce a simplified version of user_agent.

[webmat]

This is an implementation of my proposal in #235.

You'll notice the version breakdown fields are gone for now. I get the feeling that people either don't care about version breakdowns, or they want to see all version fields they see broken down (not just the UA version). So I'm holding off on adding them for now, unless someone feels strongly about this. This will let us discuss ideas about generalizing the concept of breaking down arbitrary version fields in a separate discussion.

In line with the discussion in #235, for now this is a simple field set, but may grow to cover more concepts:

• agent type (desktop/mobile browser, email client, feed reader, robot)
• client architecture
• rendering engine details
• producer of the user agent (Chrome, Mozilla)
• screen dimensions

This kind of reverts #172 and closes #235.

[ruflin]

+1 on not adding all the versions but have them in one string as I think Elasticsearch with the prefix query can do most necessary queries already.

So this output of the user_agent is identical to what the user_agent processor does besides the version part? https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-user-agent.html

[ruflin]

How does this correlate with #238 ?

[webmat]

It correlates in absolutely no way. Here user_agent.device.name would contain things like "iPad", "iPhone"...

[webmat]

@ruflin No, it does not match what IN's user_agent parser gives us. Here's the format it outputs, and the ECS format, side by side:

IN's user_agent result:

"user_agent" : {
  "name" : "Mobile Safari",
  "major" : "12",
  "minor" : "0",
  "device" : "iPhone",
  "os": "iOS 12.1",
  "os_minor" : "1",
  "os_major" : "12",
  "os_name" : "iOS",
}

If we include ECS' reusable os field in, the ECS user_agent from this proposal looks like this:

"user_agent" : {
  "name" : "Mobile Safari",
  "version": "12.0",
  "device": {
    "name": "iPhone",
    "version": "???"
  },
  "os": {
    "name": "iOS",
    "version": "12.1"
  },
  "original": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"
}

Takeaways:

• This just made me realize that I'm not sure if user agents typically give a device.version (different from the OS version). I will remove it for now
• IN's UA parser gives us a full OS version field, and a broken up agent version field. Initially we may have to reconstruct it from that, to have user_agent.version. This is not perfect.
• Most of the other differences I'm already working around with a bunch of field renames, for now. See any of the Beats access log modules: Convert apache2.access to ECS - Take 2 beats#9245, Convert Filebeat iis.access to ECS beats#9084, Convert Filebeat nginx.access to ECS beats#9081, Convert Filebeat's traefik.access to ECS. beats#9005.
• os currently doesn't have the concept of "full_name" ("iOS 12.1"), which would be nice to have.

[ruflin]

Proposal LGTM.

[ruflin]

@webmat As a follow up we should open an Issue in the Github repo so the processor gets adjusted to ECS format and potentially relies by default on user_agent.original for the processing. For backward compatiblity the processor could have an ecs option.

[MikePaquette]

LGTM

[ruflin]

@webmat I still think we should turn indexing off here: #138

[webmat]

I don't mind turning it off, actually. But didn't we get a request in Beats for precisely the reverse?

[webmat]

Of course people can override that, so not a big issue

[ruflin]

I think that request was for having it as a text field? Having is a keyword would still not make ti searchable. I wonder now if not index if a user can still add a .text multi-field which is indexed?

[webmat]

I don't think it's possible, no. But if that person wants indexing, they can totally re-enable keyword indexing on the canonical field, and add a multi-field for text indexing.

If ECS makes it non-indexed, it means that no application that will depend on the official ECS spec can ever perform searches in that field. So whatever users do with it (re-enabling indexing) will not conflict. Because ECS will say it should not be indexed, so the applications can't depend on this being a keyword or a text field.

What I want to avoid at all cost is to make this canonical field text. If we do text here, it must be a multi-field.

[webmat]

user_agent.original is now marked as non indexed. This gives us absolute freedom to change it in the future, if we decide to do so. Whereas going from indexed to not is a breaking change.