You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Raft currently manages nomad cluster leadership: Raft conducts the leadership election and then informs nomad when leadership is gained or lost. This works very well the vast majority of the time. However, if nomad fails unexpectedly in establishLeadership method, it's possible that raft believes a server to be the leader, while the nomad server does not perform its leadership duties, and the cluster is wedged. It's possible to fix the cluster by terminating the server to force an election.
On failure, the leadership loop should instead force a step down. The raft library offers a raft.LeadershipTransfer, which nomad can use in these cases. The documentation indicates that we need to upgrade to Raft 3 as well, but it's not very clear. (See #7208 for related issue on defaulting to Raft v3)
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Raft currently manages nomad cluster leadership: Raft conducts the leadership election and then informs nomad when leadership is gained or lost. This works very well the vast majority of the time. However, if nomad fails unexpectedly in
establishLeadership
method, it's possible that raft believes a server to be the leader, while the nomad server does not perform its leadership duties, and the cluster is wedged. It's possible to fix the cluster by terminating the server to force an election.On failure, the leadership loop should instead force a step down. The raft library offers a
raft.LeadershipTransfer
, which nomad can use in these cases. The documentation indicates that we need to upgrade to Raft 3 as well, but it's not very clear. (See #7208 for related issue on defaulting to Raft v3)Consul hit the same issue: it's noted in hashicorp/consul#5047 and fixed in hashicorp/consul#5247
The text was updated successfully, but these errors were encountered: