[Bug] Temporal error when using Elasticsearch V8

[danielserrao]

What are you really trying to do?

I'm trying to use Elasticsearch as visibility store of Temporal.

I deployed temporal using elasticsearch v8 but I keep getting errors and it seems that Temporal is not even supporting version 8 of elasticsearch.

Describe the bug and Minimal Reproduction

When I deploy the latest version of this helm chart with the following user supplied helm values it works fine.

elasticsearch:
  enabled: true
  external: false
  host: "elasticsearch-master-headless"
  port: "9200"
  version: "v7"
  imageTag: 7.17.3
  scheme: "http"
  logLevel: "error"
prometheus:
  enabled: false
grafana:
  enabled: false

But when I change version to v8 and imageTag to 8.14.0 I get the following log error on the elasticsearch pods:

{"@timestamp":"2024-10-10T08:57:48.084Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-master-1","elasticsearch.cluster.name":"elasticsearch","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:563)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:509)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:479)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:50)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:503)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:256)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)\n\tat org.elasticsearch.server@8.14.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 12 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 12 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ingest] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 12 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 12 more\n"}

And when I try to use an external elasticsearch by changing:

• enabled to false
• external to true
• version to v8
• host to elasticsearch-es-http.elasticsearch.svc
• imageTag to 8.14.0

I get the following error:

temporal-schema-1-8gjzr setup-visibility-store curl: Failed to open schema/elasticsearch/visibility/index_template_v8.json
temporal-schema-1-8gjzr setup-visibility-store curl: option --data-binary: error encountered when reading a file
temporal-schema-1-8gjzr setup-visibility-store curl: try 'curl --help' or 'curl --manual' for more information

which makes sense because the v8 version is not available at https://github.com/temporalio/temporal/tree/main/schema/elasticsearch/visibility/versioned.

According to the documentation at https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch, v8 should be supported but doesn't appear to be the case but maybe I'm missing something.

Environment/Versions

• Docker image: temporalio/admin-tools:1.25.0-tctl-1.18.1-cli-1.0.0
• Temporal Version: 1.25.0
• Kubernetes Client Version: v1.31.0
• Kubernetes Server Version: v1.29.9