More than 1 vault pod is failing as asking for more than 5th key

[manik9910]

Using vault crd:
https://raw.githubusercontent.com/banzaicloud/bank-vaults/1.15.3/operator/deploy/cr-raft-ha-storage.yaml

when size is 1, it is working fine
when size is more than 1, it is failing

1st vault pod has taken 3/5 keys and working fine
2nd vault pod is taking 4th and 5th key and failed. Ideally it should take any of 3 keys from total of 5 keys.

Environment details:

• Kubernetes version v1.23.6
• Cloud-provider/provisioner capi/Openstack
• bank-vaults version (e.g. 0.4.17): ghcr.io/banzaicloud/bank-vaults:1.19.0
• Install method (e.g. helm or static manifests): Applied kustomization on https://raw.githubusercontent.com/banzaicloud/bank-vaults/1.15.3/operator/deploy/cr-raft-ha-storage.yaml

Error in vault container:

kubectl logs -n vault vault-1 vault
2023-05-29T09:46:53.109Z [INFO]  core: security barrier not initialized
2023-05-29T09:46:53.635Z [ERROR] core: failed to get raft challenge: leader_addr=https://vault:8200/ error="error during raft bootstrap init call: Put \"https://vault:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp 10.43.68.122:8200: connect: connection refused"
2023-05-29T09:46:53.635Z [ERROR] core: failed to join raft cluster: error="failed to get raft challenge"

Error in bank-vaults container:

root@management-cluster-control-plane-dhdzl:~# kubectl logs -n vault vault-1 bank-vaults
{"level":"info","msg":"joining leader vault...","time":"2023-05-29T09:46:47Z"}
{"level":"info","msg":"vault metrics exporter enabled: :9091/metrics","time":"2023-05-29T09:46:47Z"}
{"level":"info","msg":"joining raft cluster...","time":"2023-05-29T09:46:47Z"}
{"level":"fatal","msg":"error joining leader vault: error joining raft cluster: Error making API request.\n\nURL: POST https://127.0.0.1:8200/v1/sys/storage/raft/join\nCode: 500. Errors:\n\n* failed to join raft cluster: failed to get raft challenge","time":"2023-05-29T09:46:53Z"}

Pod's Status:

root@management-cluster-control-plane-dhdzl:~# k get pods -n vault -o wide
NAME                                     READY   STATUS             RESTARTS         AGE     IP             NODE                                     NOMINATED NODE   READINESS GATES
vault-0                                  2/2     Running            0                32m     10.42.171.19   management-cluster-control-plane-t8fw7   <none>           <none>
vault-1                                  0/2     CrashLoopBackOff   10 (4m14s ago)   32m     10.42.220.83   management-cluster-control-plane-dhdzl   <none>           <none>
vault-config-operator-7987fbdc4c-vx96v   2/2     Running            0                72m     10.42.171.7    management-cluster-control-plane-t8fw7   <none>           <none>
vault-configurer-86f766dbbb-pksst        1/1     Running            0                6m40s   10.42.171.18   management-cluster-control-plane-t8fw7   <none>           <none>
vault-operator-876f4b9d-gt2sc            1/1     Running            0                72m     10.42.171.61   management-cluster-control-plane-t8fw7   <none>           <none>

I also added and try:

vaultEnvsConfig:

• name: POD_NAME
  valueFrom:
  fieldRef:
  fieldPath: metadata.name

When I tried to modify cluster_addr as below.
cluster_addr: "https://${.Env.POD_NAME}:8201"
It is failing by error that converting YAMLtoJSON conversion failed. So it's giving syntax issue here.

kubectl get kustomizations -A
`default     vault                           2d1h   False   var substitution failed for 'vault': YAMLToJSON: variable substitution failed: unable to parse variable name`

My requirement is to create HA cluster with replicas which is failing with above.

/kind bug

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.

[github-actions]

This issue has been marked stale for 20 days, and is now closed due to inactivity. If the issue is still relevant, please re-open this issue or file a new one. Thank you!