Do not bootstrap with non voters

[dadgar]

Fixes an issue a cluster could be bootstrapped by promoting non voting servers
to voters. The fix is to not bootstrap unless bootstrap expect is reached with
voting servers only.

[shantanugadgil]

sorry to barge in into this, but I am observing a somewhat similar situation and was wondering if it could be due to this.

Nomad 0.8.5
Consul 1.2.3

I'll try to explain my scenario in brief:

3 (three) Nomad Servers in an AWS ASG using Terraform. (create before destroy set on the ASG and LC)
termination policy set to "OldestLaunchConfiguration" and "OldestInstance"
retry join of Nomad set using "autojoin" using EC2 tags.
bootstrap_expect is set to "3"

When I change the AMI to a new version and terraform apply (it goes to deposed) (all ok so far)
Now I start the dance of "increment by 1 and decrement by 1" (for the nomad servers)
counts go: 4,3,4,3,4,3
** This works fine if I use an ample delay during the up-down sequence.**

Sometimes (not all the time) if I take out the server which was the leader, then sometimes a leader is never chosen and all servers go into an endless loop of "unable to elect a leader"

The "no leader" situation also occurs if I do the counts in the following way: (starting with 3)
4,1,3
(I was just hoping for the 4,1,3 is an optimized way of evacuating the older EC2 instances)

[shantanugadgil]

@dadgar bump?

[preetapan]

@shantanugadgil what you describe seems unrelated to this. This was fixing bootstrapping with non voting servers which is only possible in nomad enterprise.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.