Amazon Elasticsearch support

[s12v]

Adds optional Support for Amazon Elasticsearch Service (request signing using AWS-SDK).

Example configuration:

output.elasticsearch:
  hosts: ["https://search-test1-abcdef.eu-central-1.es.amazonaws.com:443"]
  aws:
    enabled: true
    region: eu-central-1

PR looks huge because of SDK, but actually it's only 100 lines.

TODO:

• Tests
• Pin AWS-SDK version

[elasticmachine]

Can one of the admins verify this patch?

[tsg]

Hi @s12v, thank you for opening a PR. Unfortunately at the moment we don't want to maintain non-standard ES outputs in the main Beats repo. Is the request signing strictly required for the AWS ES service?

We regularly test the Beats output with the Elastic Cloud service, which also runs on AWS, in case that is a viable alternative for you. Any other hosted Elasticsearch service that doesn't modify the API should be fine as well.

If you want, you can also maintain the modified output as an external plugin (in its own repo) and load it using the -plugin option (available on Linux). See this repo for some examples of run-time loadable plugins, but note that the API is changing slightly in 6.0.

[s12v]

:-(

Is the request signing strictly required for the AWS ES service?

You can also restrict access by IP, but it's not an option for dynamically created/terminated VMs. Signing is the best option.

If you want, you can also maintain the modified output as an external plugin

Thanks for the information. Probably, I don't want to have entirely new Elasticsearch output (it will be 100% copy-paste), I only need to somehow extend the existing ES output and sign the request (add auth headers). In the PR I do it in execHTTPRequest(). Maybe you have an idea how to extend an existing output?

By the way, I can see that plugin support is a relatively new future and it's experimental. Are you planning to support this feature? Is there a list of available plugins where I can check for more (live) examples?

Also, I'm wondering, if I create a plugin, I'll need to compile/publish it for each libbeats version using the same compiler, right?

[tsg]

Yes, the plugin feature is experimental because we're not sure about the support for it in the Go language itself. It was added in 1.8 for Linux, but then no other platform support was added in 1.9 so I'm not sure what the plans for it are. Also yes, you will need to compile against each version of libbeat using the same compiler version and you'll need to be prepared for occasional changes in the libbeat APIs. It's not ideal, but that's what we can offer at the moment with regards to run-time loadable plugins.

[s12v]

I tried it out. Well... it's experimental indeed :-)

If you're interested, some feedback: after some prodding, it worked - required the latest golang@master (because of golang/go#19529), and also go-structform didn't work well... But finally it compiled and sent some signed requests to Amazon using -plugin. Yay :)
I put files on https://github.com/s12v/beats-aws-elasticsearch, should somebody need it.

However, I still don't see a good way to reuse existing functionality (I mean, elasticsearch output). I copied the entire package, but that would be a nightmare to support. I guess functionality from this PR can't be implemented in a sane way using plugins system.

Implementing a plugin for Kinesis Firehose output would be another solution (it's like a queue in front of AWS Elasticsearch). Maybe I'll try to work on that later... But it's much more work and in the meantime I'll have to search for other shippers :(