Using specific pass for each host in redis output

[rvillablanca]

What does this PR do?

This PR adds support to redis output for multiple password

Why is it important?

It is useful when you have a multiple redis instances with different credentials.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes Redis output multiple password support. #16058

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[urso]

Thank you for the PR. I wonder if we want to enforce the : at the beginning, or rather want to make it optional. WDYT?

[rvillablanca]

Yeah as I said before, for me it is ok both way but the second one (whithout :) seems the pass could be the username part.

[rvillablanca]

So what will we do with the colon?

[urso]

Using redis:// like an URL seems to be quite common practice. Let's keep your implementation treating it as an URL. But let's check for the scheme as well, in case some other scheme like 'rediss' is used.

The redis output supports connecting to redis via TLS as well. The URL scheme would be rediss:// then. Depending on the scheme defined we should enable TLS for the connection. This is what the Elasticsearch output does as well (hidden in the go stdlib, though). If we use https even if TLS is not enabled, then the Elasticsearch output will still attempt to connect via TLS using default settings and the system certificate pool.

You want to give it a try introducing the 'rediss' scheme as well? Or shall follow up later?

[rvillablanca]

Sure let's do it

[rvillablanca]

@urso at the moment I am validating that a valid tls configuration is required when using rediss scheme, but I know this is not the expected way so I need to know where I can see for that default settings in elasticsearch output using https without tls configuration

[urso]

If TLS is used the Elasticsearch output itself does not much. The output only configures a TLS dialer. The net/http package will use the TLS dialer if the "https" schema is used. If the user did not explicitely configure TLS, then the dialer will use system (and go net) defaults.

The output creates a TCP and TLS dialer first, and assignes those to the http.Transport.

If "https" is used net/http will use the TLS DLS dialer we did set up. In case TLS was not configured, then the crypto/tls package will fallback to its default configuration.

All in all, it is more a matter of configuring and selecting the correct dialer based on the scheme used.

[urso]

Please add some integration tests as well. Check out redis_integration_test.go. Thanks.

[rvillablanca]

I made all your suggestions, only integration test remain but I need some time, thanks for your mentoring @urso 👌

[rvillablanca]

Before writing integration test I'd like to make sure that they are actually passing so, I suppose I can do that by running make integration-tests-environment right ?

[rvillablanca]

Running make integration-tests-environment takes a long time but finishes successfully, I wonder if there is a way to run integration tests for redis output only, with the environment in docker.

I tried it by running make start-environment and then go test -tags integration . in redis output directory, but It couldn't connect with dockerized redis (no docker port mapping).

[urso]

Jenkins, test this.

[rvillablanca]

I had to add the license header manually because every time I have to run some script related with python I have different problems. I really expect the migration to python 3 is finished.

We are done with this @urso 👍

[rvillablanca]

CI is failing and I still having problems running tasks with python, the message is /usr/bin/python2.7: No module named venv when I run make fmt update in the libbeat directory.

If I try to configure python with make python-env I'm receiving the same message. I tried to install virtualenv but it is already installed 🤔

rodrigo@vivobook:~/go/src/github.com/elastic/beats (specific-password-redis)*$ sudo pip2 install virtualenv
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: virtualenv in /usr/lib/python2.7/site-packages (16.1.0)

[rvillablanca]

According to this https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html migration is done so I will try it again

[urso]

Jenkins, test this.

[urso]

Approved. I started an internal CI run and will merge + prepare backports to release branches tomorrow. I expect the enhancement to become available with 7.7.0.

[urso]

PR and backports are merged. Thank you for taking the time to improve the redis output!