[elasticsearch] fix node roles for clients nodes

[jmlrt]

This commit fix the node.roles variable for client role with
Elasticsearch version > 8.3.0.

As client nodes are nodes that don't have any other roles,
setting the client roles is done by configuring node.roles: []
(empty list).

Elasticsearch chart usually define node.roles as an environment
variable, however, for client nodes, setting an empty list as value of
an environment variable isn't recognized by Elasticsearch so we were
required to also add it to the elasticsearch.yaml config file (more
details in
#1186 (comment)).

Starting with Elasticsearch 8.3.0 this is not working anymore and
Elasticsearch fails to start is a node.roles environment variable is
defined with an empty list as value.

This commit define the node.roles environment variable only if the
roles list isn't empty.

Fixes also the tests for the multi example.

Relates to #1186

[jmlrt]

cc @mark-vieira FYI as this is a behavior that is changing starting from 8.3.0.

This can be reproduced with Docker containers:

• ✅ With 8.2.0:

$ docker run -e 'node.roles=' docker.elastic.co/elasticsearch/elasticsearch:8.2.0 | grep roles
{"@timestamp":"2022-09-08T17:35:06.073Z", "log.level": "INFO", "message":"node name [74aebdb8202a], node ID [z8ZQRg7RSNWh5DcMXc2BgA], cluster name [docker-cluster], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"74aebdb8202a","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2022-09-08T17:35:16.502Z", "log.level": "INFO", "message":"parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.authz.store.FileRolesStore","elasticsearch.node.name":"74aebdb8202a","elasticsearch.cluster.name":"docker-cluster"}
...

• ❌ With 8.3.0 and later versions:

$ docker run -e 'node.roles=' docker.elastic.co/elasticsearch/elasticsearch:8.3.0 | grep roles
Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.roles] found at line number [1], column number [12]
        at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:768)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:743)
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:687)
        at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1190)
        at org.elasticsearch.node.InternalSettingsPreparer.loadOverrides(InternalSettingsPreparer.java:143)
        at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:53)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
        at org.elasticsearch.cli.Command.main(Command.java:50)
        at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

[mark-vieira]

Ok, so as I understand it, the elasticsearch.yml file is defaulted to set node.roles: [], essentially making the default a coordinating node. Here we override this value via an environment variable, only if it's set, yes? If so, this should work.

It's also worth noting that starting with 8.2 we support envirnoment variable substitution here, so that's an alternative way of doign it. Essentially we could make the yaml node.roles: [${NODE_ROLES}] and then set the NODE_ROLES env var. This should work with empty values correctly, as the final generated yaml (before parsing) would just be node.roles: [].

I'm good with this solution though. I didn't go so far as to investigate what change in 8.3 broke this.