Streamlining three_data_hall Replication Setup Process

[simenl]

What would you like to be added/changed?

Issue Description

When configuring a cluster with three_data_hall replication as described in the documentation, the current process involves several manual steps. This issue proposes enhancements to simplify and automate the setup.

Current Process

1. Create the primary fdb cluster in one availability zone, specifying redundancyMode: triple.
2. Wait for the primary fdb cluster to reconcile
3. Copy the connection string of the primary cluster.
4. Create two secondary fdb clusters in the other two availability zones.
5. Update the fdb clusters to use redundancyMode: three_data_hall.

Proposed Changes

Option 1: Config Map Enhancement
The operator currently stores the connection string for the primary cluster in a config map named <primary-cluster-name>-config under the config-map key. To simplify the process:

• Introduce an option to automatically set seedConnectionString for the secondary clusters using the value from the config map.

This enhancement eliminates the need for step 3, streamlining the setup process by removing the dependency on dynamic values that are not known upfront.

Option 2: Await Reconciliation
Alternatively, consider enhancing the operator to wait for the primary cluster to reconcile before initiating the secondary clusters with the seedConnectionString. This would make the setup process even more straightforward:

1. Create the primary and secondary fdb clusters.
2. Wait for all fdb clusters to reconcile.
3. Update to redundancyMode: three_data_hall.

[johscheuer]

I appreciate the feedback on the setup process, Personally I would lean towards not implementing those changes inside the operator itself as the proposed solutions will only work for deployments on the same Kubernetes cluster (the operator doesn't support cross Kubernetes interactions yet, only locking with the FDB cluster is supported). We have plans to improve the operator for such deployments and this would also include deployments in the same Kubernetes cluster, either across different namespaces or in the same namespace. But this will require a new design and some fundamental changes in the operator logic, hopefully we can get the design out by next year.

Probably a better short/mid-term solution is to implement support for this in the kubectl fdb plugin, we actually have an old design for this here: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/design/plugin_multi_fdb_support.md, we never implemented that because of resources/time restrictions. That way the "complexity" of the setup would be coded into the plugin and a user could do something like kubectl fdb create ... to bootstrap such a cluster. For the plugin I would be fine to start implementing this support only for a single Kubernetes cluster. What are your thoughts on such a solution?

The referenced design must be verified before implementing, since we learned a lot since the time we've written this design :)

Again, thanks for your feedback!