fix panic from keyring raft entries being written during upgrade #14821
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
During an upgrade to Nomad 1.4.0, if a server running 1.4.0 becomes the leader before one of the 1.3.x servers, the old server will crash because the keyring is initialized and writes a raft entry. Wait until all members are on a version that supports the keyring before initializing it.
jrasell
approved these changes
Oct 6, 2022
There was a problem hiding this comment.
LGTM.
I tested the patch locally through a previously failing scenario which didn't cause the non-1.4.0 servers to panic when a 1.4.0 server become leader.
$ nomad server members
Name Address Port Status Leader Raft Version Build Datacenter Region
server1.global 192.168.0.14 4648 alive true 3 1.4.0-dev dc1 global
server2.global 192.168.0.14 5648 alive false 3 1.3.5 dc1 global
server3.global 192.168.0.14 6648 alive false 3 1.4.0-dev dc1 global
jrasell
reviewed
Oct 6, 2022
This was referenced Oct 6, 2022
This was referenced Oct 6, 2022
Merged
schmichael
reviewed
Oct 6, 2022
schmichael
reviewed
Oct 6, 2022
schmichael
approved these changes
Oct 6, 2022
tgross
added a commit
that referenced
this pull request
Oct 17, 2022
In #14821 we fixed a panic that can happen if a leadership election happens in the middle of an upgrade. That fix checks that all servers are at the minimum version before initializing the keyring (which blocks evaluation processing during trhe upgrade). But the check we implemented is over the serf membership, which includes servers in any federated regions, which don't necessarily have the same upgrade cycle. Filter the version check by the leader's region.
tgross
added a commit
that referenced
this pull request
Oct 17, 2022
In #14821 we fixed a panic that can happen if a leadership election happens in the middle of an upgrade. That fix checks that all servers are at the minimum version before initializing the keyring (which blocks evaluation processing during trhe upgrade). But the check we implemented is over the serf membership, which includes servers in any federated regions, which don't necessarily have the same upgrade cycle. Filter the version check by the leader's region.
tgross
added a commit
that referenced
this pull request
Oct 17, 2022
In #14821 we fixed a panic that can happen if a leadership election happens in the middle of an upgrade. That fix checks that all servers are at the minimum version before initializing the keyring (which blocks evaluation processing during trhe upgrade). But the check we implemented is over the serf membership, which includes servers in any federated regions, which don't necessarily have the same upgrade cycle. Filter the version check by the leader's region. Also bump up log levels of major keyring operations
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #14819
During an upgrade to Nomad 1.4.0, if a server running 1.4.0 becomes the leader before one of the 1.3.x servers, the old server will crash because the keyring is initialized and writes a raft entry.
Wait until all members are on a version that supports the keyring before initializing it.