Step down leadership on establishLeader failures

[notnoop]

Raft currently manages nomad cluster leadership: Raft conducts the leadership election and then informs nomad when leadership is gained or lost. This works very well the vast majority of the time. However, if nomad fails unexpectedly in establishLeadership method, it's possible that raft believes a server to be the leader, while the nomad server does not perform its leadership duties, and the cluster is wedged. It's possible to fix the cluster by terminating the server to force an election.

On failure, the leadership loop should instead force a step down. The raft library offers a raft.LeadershipTransfer, which nomad can use in these cases. The documentation indicates that we need to upgrade to Raft 3 as well, but it's not very clear. (See #7208 for related issue on defaulting to Raft v3)

Consul hit the same issue: it's noted in hashicorp/consul#5047 and fixed in hashicorp/consul#5247

[ketzacoatl]

Has this gotten triage review or is the raftv3 migration on the roadmap in any capacity?

[lgfa29]

Hi @ketzacoatl 👋

Yes, the plan is to have this in the next Nomad release.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.