-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not upgrading servers across all regions to 1.4 causes "keyring has not been initialized yet" #14896
Comments
Hi @optiz0r! Thanks for bringing this discussion over from HangOps! When regions are federated they're joined in the same serf membership, but there's no raft replication between them. Unfortunately it looks like we're using the serf membership and not the raft cluster for the check so we're unnecessarily blocking keyring initialization even though the keys aren't actually replicated between regions. 🤦♂️ I'll take a look early next week as to the best way to get this fixed. |
First pass at a fix is here: #14901 |
#14901 has been merged, and we've identified a few other places where the server minimum version check is potentially wrong. This is a long-used pattern and the new keyring feature draws it out much more than some other features we've built in the last few major versions. Once we've got those patches in, we'll have a discussion about how soon we can ship this. Thanks for opening the issue @optiz0r! |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Main region: 1.4.1
Alternate region: 1.3.3
Issue
After upgrading 3x servers in the main region (which included the leader node) from 1.3.5 to 1.4.1, it was impossible to schedule any new jobs until the a second region containing a single server was also upgraded from 1.3.3 to 1.4.1. All evals failed with error "keyring has not been initialized yet". As soon as the alternate region server was upgraded, jobs in the main region started to be allocated once more.
Reproduction steps
Expected Result
Jobs to continue to be scheduled in the main cluster once all servers had been upgraded to 1.4.1
Actual Result
Cluster was in-operable until all servers across all regions were upgraded.
Possible remedies
The upgrade-specific instructions hint that there could be problems if the leader is allowed to rollback to an older cluster version, but not that servers across all regions need to be upgraded simultaneously in order to maintain cluster function.
Either the upgrade nodes should call out this requirement, or the leader should be able to handle servers in another region still running on an older version. Typically I would only upgrade one region at a time.
Nomad Server logs
The text was updated successfully, but these errors were encountered: