[TASK] Increase Elasticsearch nodes from 2 to 3 by default

[aabouzaid]

Related to

Epic: https://github.com/camunda/distribution/issues/230

Overview

The ideal setup for Elasticseaech is 3 nodes. For local setup, we reduce the ES nodes to 1 anyway.

Relevant documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html

Because it’s not resilient to failures, we do not recommend deploying a two-node cluster in production.

Actions

Sub-tasks

Preview Give feedback

1. Change ES nodes from 2 to 3 by default for the Camunda 8.6 chart (alpha now).
2. Add elasticsearch.master.podAntiAffinityPreset: hard
3. Add the change to the official docs announcement and upgrade pages.

PRs

Preview Give feedback

1. feat: increase elasticsearch nodes to 3 #2298
   kind/enhancement size/xs +2
   
2. docs(self-managed): Helm chart Elasticsearch nodes number changed camunda-docs#4255
   component:self-managed target:8.6 +2
   

[aabouzaid]

All actions are done ✔️