Better recovery #3709
Better recovery #3709
Commits on Jul 22, 2024
-
Flatten the structure of machine updates
The goal of this is to make recovering from individual machine update failures easier, so that the entire deployment can succeed
-
Add recovery to machine updates
The really cool part about these changes are it didn't actually take much change to how machine updates work. All I really do is call updateMachines, but with our original state instead of the state that we initially wanted to go to. I made a bug fix to lease clearing, since in some edge cases we weren't correctly clearing them. I made a bug fix to machine waits, since we were sometimes causing an infinite loop from not giving the Wait function time to set the waitErr, whoops
-
-
Update current state on every rollback attempt
This was dumb to not do before, since obviously the state wouldn't be current
-
-
Don't recursively try to rollback
We should only attempt the rollback functionality when we initially try to update machines, not on every rollback after that obviously
-
Disable deleting machines for now
We need to avoid deleting unmanaged machines
-
fix up the code to create and destroy machines
wasn't testing that path correctly. also, i added a quick optimization to waitForMachineState to avoid an unnecessary API call (just checking the machine state right then and there)
-
-
Check that configs are not equal before attempting an update
Just another saved API call
-
Small changes to focus on pushing forward deploys
The main change was just focusing on newAppState instead of the old. The rest is just cleaning up logging tbh
-
Fix waiting for multiple states
If any one /wait finishes, we should just return
-
-
Return on some unrecoverable errors + cache health check results
The more interesting bit is caching health checks. If we're trying to push forward,there's no point in wasting bandcwidth always retrying health checks. If a machine passes them once, they're good forever
-
-
-
-
-
-
-
-
-
-
-
-
-
move lease collection out of restartMachines
I prefer that the updateMachines function controls it
-
-
-
-
-
-
correctly catch unrecoverable errors
I wasn't correctly using errors.As. I also cleaned up catching context canceled errors
-
-
i forgot that lm isn't a pointer, so we need to check entry.leasableMachine for newly created machines
-
-
-
This covers most of the major functions and the major places there could be issues
-
-
-
ensure that we use machinedeployment flapsClient
also add some tests for updateorcreatemachine
-
-
-
md.warnAboutListenAddress requires this function
-
-
-
-
-
-
we can't start machines if we have a lease acquired
-
-
-
set default for deploy-retries to 0
i also added back the original deployment code, and use that if deploy-restries is set to 0. That way, we can more slowly roll this out without risking breaking user apps if there's some terrible bug
-
get deploy-retries from launchdarkly by default
Users can still set deploy-retries to whatever value they'd like, however.
