Bootstrap pod does not install plugins from the general spec, fails if plugin specific settings are configured in opensearch.yml

[dobharweim]

Description

If I want to provide an S3 API backed repository for opensearch snapshots, but not use S3 itself, I need to provide the s3.client.default.endpoint configuration. However this causes the bootstrap pod to fail as it doesn't have the plugin installed and does not recognise the setting.

Steps to reproduce

1. Install the latest version of the operator.
2. Deploy the following YAML:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-first-cluster
spec:
  security:
    tls:
       http:
         generate: true 
       transport:
         generate: true
         perNode: true
  general:
    httpPort: 9200
    serviceName: my-first-cluster
    version: 2.3.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
    additionalConfig:
      s3.client.default.endpoint: "example.s3.com"
  dashboards:
    tls:
      enable: true
      generate: true
    version: 2.3.0
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      resources:
         requests:
            memory: "4Gi"
            cpu: "1000m"
         limits:
            memory: "4Gi"
            cpu: "1000m"
      roles:
        - "data"
        - "cluster_manager"
      persistence:
         emptyDir: {}

Expected Outcome

I expect the bootstrap pod to also install all required plugins so that it can verify the additional config settings.

Actual outcome

Bootstrap pod fails with the following error:

[2023-02-12T11:27:04,774][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [development-bootstrap-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [s3.client.default.endpoint] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.5.0.jar:2.5.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.5.0.jar:2.5.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [s3.client.default.endpoint] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549) ~[opensearch-2.5.0.jar:2.5.0]
uncaught exception in thread [main]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:170) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.node.Node.<init>(Node.java:506) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.node.Node.<init>(Node.java:356) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.5.0.jar:2.5.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.5.0.jar:2.5.0]
        ... 6 more
java.lang.IllegalArgumentException: unknown setting [s3.client.default.endpoint] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608)
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549)
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519)
        at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489)
        at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:170)
        at org.opensearch.node.Node.<init>(Node.java:506)
        at org.opensearch.node.Node.<init>(Node.java:356)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)```

[swoehrl-mw]

Hi @dobharweim. You are correct that currently the operator does not install any plugins on the bootstrap pod. We may need to do that in the future. Or we could also look into not giving the bootstrap pod the additionalConfig.

Edit: Due to similar issues raised, the solution will have to be installing the plugins in the bootstrap pod as well.

As a workaround: Can you try to set spec.bootstrap.additionalConfig to set some dummy configuration? That should prevent the general additionalConfig from being added to the bootstrap pod.

[AndersBennedsgaard]

@swoehrl-mw nice workaround.
I simply set bootstrap.additionalConfig."network.bind_host"="0.0.0.0" and that prevented passing the plugin settings to the bootstrap container 👍

[Akila-I]

I also faced this issue and used the workaround mentioned in #430 (comment). When can we expect the Helm Chart release for the fix?