-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad server crash after raft quorum on 0.9.3, 0.10.1 #6787
Comments
Quick hotfix to at least allow the cluster to come back up:
|
Fixes #6787 In ProposedAllocs the proposed alloc slice was being copied while its contents were not. Since RemoveAllocs nils elements of the proposed alloc slice and is called twice, it could panic on the second call when erroneously accessing a nil'd alloc. The fix is to not copy the proposed alloc slice and pass the slice returned by the 1st RemoveAllocs call to the 2nd call, thus maintaining the trimmed length.
Hi @johngilden ! Thanks for reporting this serious bug! The fix is up in #6792 and you can download Linux or Windows amd64 binaries to test it. |
Seems to work fine! Would like to get a build of this with UI enabled to put into production. |
(Appreciate the quick turn-around on this issue btw) |
Fixes #6787 In ProposedAllocs the proposed alloc slice was being copied while its contents were not. Since RemoveAllocs nils elements of the proposed alloc slice and is called twice, it could panic on the second call when erroneously accessing a nil'd alloc. The fix is to not copy the proposed alloc slice and pass the slice returned by the 1st RemoveAllocs call to the 2nd call, thus maintaining the trimmed length.
Thank you for the report and testing! We're hoping to get the official 0.10.2 release out tomorrow (Dec 4th). |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.10.2-rc1 (33d411d)
Same issue on 0.9.3 and 0.10.1.
Operating system and Environment details
Ubuntu 16.04
Issue
Three-node nomad server cluster, has been working fine forever. Suddenly started crashing with SIGSEGV after completing Raft election. Single node works fine but as soon as another node joins the cluster all nodes crash with the below error messages. Probably bad alloc state handling?
Reproduction steps
No idea.
Nomad Server logs (if appropriate)
The text was updated successfully, but these errors were encountered: