using consul-k8s install to install a secondary dc incorrectly fails due to presence of the consul-federation secret

[rboyer]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

If attempting to use consul-k8s install to setup a secondary datacenter it fails due to the presence of the consul-federation secret. This secret is currently required to be ferried between clusters by an administrator in advance of setting up the cluster. This check should likely be made conditional depending upon whether or not the provided values.yaml would have tried to create one of these secrets, instead of merely consuming it.

Reproduction Steps

Steps to reproduce this issue, eg:

1. Follow existing helm oriented wan federation guides to setup a primary datacenter.
2. Extract the federation secret from the primary cluster.
3. In the secondary cluster, apply the secret:

$ kubectl apply -f consul-federation-secret.yaml 
secret/consul-federation created

4. When running consul-k8s install with the following values.yml, following whatever the older helm guides imply you need:

global:
  federation:
    enabled: true
...snip remainder...

$ consul-k8s install -f secondary-consul.yaml -namespace default

5. Get following unfortunate output:

==> Checking if Consul can be installed
 ✓ No existing Consul installations found.
 ✓ No existing Consul persistent volume claims found
 ! found Consul secret from previous installation: "consul-federation" in namespace "default". Use the command `kubectl delete secret consul-federation --namespace default` to delete

Expected behavior

The secondary cluster would likely do the opposite and check for the presence of the secret (and fail otherwise).

[david-yu]

Thanks @rboyer that is something we need to address to fully support federation via the Consul K8s CLI. We currently recommend only using Helm installs for Federation (docs)

We recommend using the Consul K8s CLI to install Consul on Kubernetes for single-cluster deployments.

[t-eckert]

@rboyer, I'm still smoothing the edges down, but this work should solve this problem if you want to give it a try: #1126