Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement backoff for k8s watcher retries #506

Merged
merged 3 commits into from
Aug 8, 2023
Merged

Implement backoff for k8s watcher retries #506

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
441bf95
Update kube to 0.85
cbgbt Aug 8, 2023
b042aa5
Add backoff to k8s watchers
cbgbt Aug 8, 2023
b229d35
controller: sleep longer if there are no nodes
cbgbt Aug 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions controller/src/controller.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ use tracing::{event, instrument, Level};
// Defines the length time after which the controller will take actions.
const ACTION_INTERVAL: Duration = Duration::from_secs(2);

// The interval between control loop polls if no nodes are detected.
const CANNOT_FIND_ANY_NODES_WAIT_INTERVAL: Duration = Duration::from_secs(10);

// Defines environment variable name used to fetch max concurrent update number.
const MAX_CONCURRENT_UPDATE_ENV_VAR: &str = "MAX_CONCURRENT_UPDATE";

Expand Down Expand Up @@ -278,6 +281,17 @@ impl<T: BottlerocketShadowClient> BrupopController<T> {
// On every iteration of the event loop, we reconstruct the state of the controller and determine its
// next actions. This is to ensure that the operator would behave consistently even if suddenly restarted.
loop {
if self.all_brss().is_empty() {
event!(
Level::INFO,
"Nothing to do: The bottlerocket-update-operator is not aware of any BottlerocketShadow objects. \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Is the bottlerocket-shadow CRD installed? Are nodes labelled so that the agent is deployed to them? \
See the project's README for more information.",
);
sleep(CANNOT_FIND_ANY_NODES_WAIT_INTERVAL).await;
continue;
}

let active_set = self.active_brs_set();
event!(Level::TRACE, ?active_set, "Found active set of nodes.");

Expand Down