Nomad client does not connect to the Consul server if it was rebootstrapped

[EugenKon]

Nomad version

v1.8.5

Operating system and Environment details

Linux ip-172-31-18-51 6.8.0-1010-aws #10-Ubuntu SMP Thu Jun 13 17:36:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Issue

Nomad/Consul servers are deployed and bootstrapped. Nomad/Consul clients are deployed too. No special actions are taken, because they are auto-joined as described at https://github.com/hashicorp/learn-nomad-cluster-setup/
If server EC2 instance was terminated it is redeployed and thus rebootstrapped (automatically during provisioning).

Nomad/Consul client does not connect to the new Nomad/Consul server automatically, until their restart on a client EC2 instance:

sudo systemctl restart nomad
sudo systemctl restart consul

Reproduction steps

1. Create server node
2. Create client node
3. Terminate server node
4. Redeploy server node

Expected Result

Client EC2 Nomad/consul should reconnect automatically.

Actual Result

It does not connect to consul/nomad server until nomad/consul client are restarted.

[tgross]

@EugenKon cloud autojoin is for "dicovery"; that is, discovering the Consul/Nomad server exists in the first place. Once it's found, there's no fallback to discovery again. Closing as a duplicate of #17872