-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad server 1.3.5 isn't able to join 1.4.0 #14819
Comments
Hi @freef4ll and thanks for raising this. We will look into this straight away but could benefit from a little more detail. Could you share information regarding the upgrade steps you are taking in order to see this panic, with attention taken to which server is the leader at each stage?
Could you clarify whether this means 1.3.5 and 1.3.6 together is running fine, or whether this is 1.3.6 and 1.4.0? |
Steps is a new cluster:
The 1.3.5 runs on a Mac, and is installed via brew. |
Hi @freef4ll! Adding an "old" server to a current cluster isn't supported. But I was able to reproduce a crash with a more realistic upgrade scenario:
I think we're often at risk of this scenario when adding new raft entries but because the leader transition itself is writing a new kind of raft entry in Nomad 1.4.0, that's why we're seeing this. We're having a chat internally about how we can fix this and we'll circle back shortly. |
@tgross , great thank you! |
We'll leave this issue pinned so that other folks see it at least until 1.4.1 comes out (which should be quickly), and perhaps sometime after so that as well. |
Nomad 1.4.1 has been released with a fix for this issue. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad running on Linux is 1.4.0 and another is 1.3.5, which panics upon trying to upgrade:
Operating system and Environment details
1.4.0:
1.3.5 from brew:
Issue
Trying to bring up a cluster causes the 1.3.5 to panic:
If 1.3.6 is used, this comes up fine.
The text was updated successfully, but these errors were encountered: