-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[0.7] all evaluations blocked due to "failed to dequeue evaluation: eval broker disabled" #3333
Comments
@jippi Did you have a leader? That error is a sign that there was no cluster leader. |
@dadgar the cluster showed an active leader, the log in the OP was from the cluster leader. Asking raft at the time pointed to this node as active leader too, and state otherwise was updated when i did evaluations and job submissions. With and without "AllowStale" query option |
@jippi Do you have server logs from this period? Has this happened before the beta? Nothing has changed that would effect that system for a while now! |
@dadgar it's the same cluster i shared logs for on another issue on gitter, so the logs would be in there. The error happened 1h before opening this ticket |
@jippi Hey had a chance to read through your logs. As suspected, there are leadership transitions causing the evaluation broker to be disabled. The leader is the only server handing out work to the other schedulers, so when there is no leader, the eval broker" is disabled and you get those logs.
I am not sure if you were having an unusual network event but the timing between the servers is excessively large. Your latency between servers should be ~10ms. |
@dadgar we run each nomad server in an AZ, other than that there isn't anything special about it. We've only seen this once in the last 8 months, so might be a network fluke somewhere - thank you! |
We just saw this happen two days in a row. The symptom is stalled deployments without anything being blocked. To fix this, we've been:
Its not clear if restarting the leader without killing deployments will work, we've just been doing that as a safety precaution, though the next time this appears, we'll be trying that. Is there anything we can do to further diagnose this issue? /cc @jippi in case there was something in the logs you saw that stuck out. |
@josegonzalez I would suggest you look at the logs to see what the latency is between servers and whether you have a leader. This error is only emitted when there is no leader. |
@dadgar the weird part is that there was an election at this time and then things hit the fan:
Thats from the node that was the new leader. I ended up restarting that server and then my following deployments succeeded. |
@josegonzalez We have a fix for Nomad leader loop in PR #3402 that is going to be in the 0.7.1 release. It fixes a bug in the leader loop that used to cause leader to be stuck until they restart. From your logs, it appears that you might have run into this. We can watch this ticket for a while. If you are able to upgrade to 0.7.1 after we release we should see this not happen anymore.. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.7.0-beta1
Operating system and Environment details
Issue
All evaluations in my cluster was in
pending
state.Tailing the nomad server output I saw a lot of errors like
2017/10/06 08:37:25.583510 [ERR] worker: failed to dequeue evaluation: eval broker disabled
.Restarting the server (which was leader) made all evaluations get processed correctly from the backlog right away (after new leader election)
Reproduction steps
N/A - not sure how it got into this state
Nomad Server logs (if appropriate)
N/A - let me know if there is anything you need, the instance is running in debug mode
Nomad Client logs (if appropriate)
N/A - let me know if there is anything you need, the instance is running in debug mode
The text was updated successfully, but these errors were encountered: