Eliminate potential race conditions while rebooting #631
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Several keys have been deprecated in favor of the same default values we were specifying before. By removing them and adding `version = 2` under licenses, we will maintain the same behavior but without the deprecation notices that we being printed.
The agent had a few conditions under which it can enter the MonitoringUpdate state before it successfully uncordons the node: * Brupop updates its state into Rebooted before the reboot terminates the process * After rebooting, an error ocurrs after updating the state but before performing the uncordon. We avoid this conditions by: * Exiting the agent process in cases where the desired state is Rebooted but we are not yet running the new version. * Always uncordoning the node before marking that we have successfully transitioned into the Rebooted state. * Defensively uncordoning the node once again when we enter the Monitoring state.
This was referenced Jun 25, 2024
vigh-m
approved these changes
Jun 26, 2024
yeazelm
approved these changes
Jun 26, 2024
This was referenced Jun 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
Closes #630
Description of changes:
This PR best left in commit-order, as it also performs several maintenance chores.
Testing done:
I built a custom Bottlerocket variant which always waits for a minute before performing a reboot, then used this to test these changes. In doing this, I was able to reliably trigger the race condition, and then also confirm that this new code does not misbehave when reboots occur slowly.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.