[Elastic Log Driver] How to supply custom certs to the Docker plugin

[fearful-symmetry]

In #13990, the top issue in the Elastic Log Driver right now is how to supply custom SSL certs (AKA, output.elasticsearch.ssl.certificate_authorities and so on). This is easier said than done, as a docker plugin is basically a runc container with no easy way to bindmount directories from the outside world. As a result, any beats option that takes a filepath is going to cause issues. As far as SSL certs go, there are three options right now:

• Pass a raw string to to the plugin via the normal config channels. A user would pass the raw string (i.e. cat cert.pem) to the plugin via daemon.json or --log-opts. This is the worst solution, as .pem certificates are problematically long and ugly, and most shells have character limits when added via --log-opts.

• Have the user rebuild the plugin. This is in line with what other solutions seem to require. A "custom" rebuild of the container would either add a bindmount to the config.json or build the root certificate into the plugin rootfs. There's some minimally painful ways to do this, either by building entirely inside docker, or by hosting an "intermediate" container of some kind on docker.elastic.co, and pulling it down to add the user's supplied cert and building the plugin.

• Something with environment variables. Users can stop a plugin and add env variables via docker plugin env. I'm not sure if it's desirable or possible, but we could have them pass a raw string of the cert via this. We run into some of the usability issues discussed above.

[jsoriano]

any beats option that takes a filepath is going to cause issues

This is in line with what other solutions seem to require.

I think they are key points to consider the rebuild of the plugin as the best option. Other options may require to add flags or environment variables that do hacky things to retrieve data from files. And these flags would need extra documentation that wouldn't be aligned neither with other Beats nor with what other plugins do.
I think we could document this rebuild by now and maybe provide some tooling to ease the build and installation or publication of the images with custom files.

Another option for the future might be to add some flags to connect to a remote keystore, for example the certificates or other files could be available in a Vault server, and the plugin connects to this server to retrieve them (though connecting to these keystores might also require certificate files, so it can become an chicken-egg problem).

Somehow similar problems are discussed in these issues:

• When deploying in kubernetes - Enable keystore for autodiscover static configuration #16306
• In functionbeat - Add more keystore backends to support Functionbeat cloud providers #15879

[ph]

@fearful-symmetry and I have been discussing a bit about this problem, the rebuild is not ideal and cumbersome. But after looking at other projects I think this is probably the only solution that we currently can use which works. As far as I can understand this is a limitation with the docker plugin API itself.

It would be nice to be able to mount a volume inside a docker plugin but doesn't seem to be possible either.

[adnsimona]

Hi it is bugging me currently. It would be nice to add container-like privileges to log drivers / docker plugins:

• be able to mount a volume to it, and use the certs from there
• also not this topic, but be able to add log driver to a docker network to use the host from there.