Convert Filebeat logstash.* to ECS

[webmat]

Caveats

• logstash.log.module is parsed with trailing spaced included. Seems like a good time to fix this and strip those spaces.
• logstash.slowlog.message is not useful, I would take the opportunity to remove it. It always just contains "event processing time". In the plain logs, it also contains a duplicate of the nested event, which is also at logstash.slowlog.event)
• This module has 4 pipelines, 2 per fileset. Each fileset has a plain log and a JSON log pipeline parser. However the JSON log pipeline has no tests file and no expected file. I will add some and transition them as well.
  • Integration tests now run the JSON pipelines

Renames

• logstash.log.level => log.level
• logstash.log.message => message
• logstash.slowlog.level => log.level
• logstash.slowlog.took_in_nanos => event.duration
• read_timestamp => event.created

TODO

• Conversion of the plain logs
• Add test log for LS JSON logs
• Conversion of the JSON logs
• Strip trailing spaces in logstash.log.module
• Alias renamed fields to their ECS counterpart
• Document field migrations in ecs-migration.yml
• remove logstash.slowlog.message
• Changelog
  • Normal ECS changelog
  • logstash.slowlog.message removed

[webmat]

@ruflin Quick question. This module has 2 pipelines per fileset. One for Logstash' plain log format, the other for the JSON format.

Right now, only plain logs are tested. When trying to add a sample JSON log for logstash.log, I'm facing two problems:

• The integration tests run the plain log pipeline instead of the JSON log pipeline. The module itself probably does the right thing when used in real life, thanks to the format option (see manifest). But I don't see an obvious way to instruct test_modules.py to do the right thing, here. WDYT?
• Also, when adding these files, git tells me they're ignored. But I can't for the life or me figure out where this ignore takes place. My find / ag / grep are failing me :-)

[ruflin]

Can you provide me with some JSON logs here. Will check what I can hack in to make this happen in test_modules.py somehow.

@ycombinator I assume you face the same problem for the ES json logs. How did you solve this?

[webmat]

@ruflin Awesome, thanks.

Here's a few lines for logstash.log:

{"level":"INFO","loggerName":"logstash.agent","timeMillis":1546896321871,"thread":"Ruby-0-Thread-1: /Users/mat/work/elastic/releases/6.5.1/logstash/lib/bootstrap/environment.rb:6","logEvent":{"message":"Pipelines running","count":1,"running_pipelines":[{"metaClass":{"metaClass":{"metaClass":{"running_pipelines":"[:main]","non_running_pipelines":[]}}}}]}}
{"level":"INFO","loggerName":"logstash.pipeline","timeMillis":1546896322538,"thread":"[main]>worker7","logEvent":{"message":"Pipeline has terminated","pipeline_id":"main","thread":"#<Thread:0x7d16ffef run>"}}
{"level":"INFO","loggerName":"logstash.agent","timeMillis":1546896322594,"thread":"Api Webserver","logEvent":{"message":"Successfully started Logstash API endpoint","port":9600}}

I'll get you slow logs shortly

[webmat]

@ruflin Here's the slow log example:

{"level":"INFO","loggerName":"slowlog.logstash.filters.ruby","timeMillis":1546959515605,"thread":"Ruby-0-Thread-6: :1","logEvent":{"message":"event processing time","plugin_params":{"code":"sleep(5)","id":"9746eeec777fcb4c7ea30366dab5445873a8560b7f1766ee1272e59b08d731fb"},"took_in_nanos":5026401704,"took_in_millis":5026,"event":"{\"sequence\":0,\"host\":\"matbook-pro.lan\",\"message\":\"Hello world!\",\"@version\":\"1\",\"@timestamp\":\"2019-01-08T14:58:30.399Z\"}"}}

[ruflin]

@webmat I opened #9959 for this

[webmat]

@ruflin @ycombinator Ready for final review.

Both unchecked caveats in the description are currently done in this PR, but I'm fine with reverting these changes, if you think that's best. Thanks!

[ycombinator]

Nit: I'd make this "Pipeline or parsing logstash slow logs", just so it's not the same as the logs fileset's pipeline description.

[ycombinator]

One small naming nit 🙄, otherwise LGTM.

[webmat]

Fixed the pipeline names for slowlog.

To my great surprise, the Travis CI tests were failing for the LS slow log. They work on my machine. Will investigate, to see if this was a transient problem or something legit.

[webmat]

Legit failure. Was able to reproduce locally after a rebuild:

{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [logstash.slowlog.took_in_nanos]."}}

[webmat]

Tests were perfectly green. Rebased for a single conflict in ecs-migration.yml. Merging right away.