update-agent: react to persistent deploy failure #286
Conversation
|
I haven't attempted to review the code in any detail, but the change SGTM. |
src/update_agent/mod.rs
Outdated
| /// is being abandoned. | ||
| fn deploy_attempted(&mut self, success: bool) -> Option<Release> { | ||
| // Maximum failed deploy attempts before declaring a persistent error. | ||
| const MAX_DEPLOY_ATTEMPTS: u8 = 60; |
There was a problem hiding this comment.
That seems like a lot - why not 3 or even 1?
There was a problem hiding this comment.
Good catch, I botched this. It was supposed to be 60 minutes. Retrying every 5 minutes, the watermark here should be at 12.
This cap is a bit arbitrary, anything strictly larger than 1 would indeed work.
A single attempt instead would remove the re-entering edge. With the current ticking logic, it could result in an infinite fast-spinning loop where the agent is oscillating between NoNewUpdate and UpdateAvailable as fast as it can.
lucab
changed the title
|
Ok, I manually tested this on top of the current |
This updates actor logic to use variant discriminants when comparing states.
|
Pushed a fixup commit to address the comments. |
This augments actor state-machine to prevent locking into a faulty `UpdateAvailable` state. In particular, rpm-ostree may not agree to deploy the currently detected update target. After a finite amount of failed attempts, Zincati now abandons its update target and goes back to the `NoNewUpdate` state, polling Cincinnati again for a new target.
This augments update-agent state-machine to prevent locking into a faulty
UpdateAvailablestate.In particular, rpm-ostree may not agree to deploy the currently
detected update target. After a finite amount of failed attempts,
Zincati now abandons its update target and goes back to the
NoNewUpdatestate, polling Cincinnati again for a new target.Ref: coreos/fedora-coreos-tracker#481
Closes: #288
New state-machine diagram:
