-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better recovery #3709
Better recovery #3709
Commits on Jul 22, 2024
-
Flatten the structure of machine updates
The goal of this is to make recovering from individual machine update failures easier, so that the entire deployment can succeed
Configuration menu - View commit details
-
Copy full SHA for 64db673 - Browse repository at this point
Copy the full SHA 64db673View commit details -
Add recovery to machine updates
The really cool part about these changes are it didn't actually take much change to how machine updates work. All I really do is call updateMachines, but with our original state instead of the state that we initially wanted to go to. I made a bug fix to lease clearing, since in some edge cases we weren't correctly clearing them. I made a bug fix to machine waits, since we were sometimes causing an infinite loop from not giving the Wait function time to set the waitErr, whoops
Configuration menu - View commit details
-
Copy full SHA for 6db4e43 - Browse repository at this point
Copy the full SHA 6db4e43View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5aa0cef - Browse repository at this point
Copy the full SHA 5aa0cefView commit details -
Update current state on every rollback attempt
This was dumb to not do before, since obviously the state wouldn't be current
Configuration menu - View commit details
-
Copy full SHA for e333394 - Browse repository at this point
Copy the full SHA e333394View commit details -
Configuration menu - View commit details
-
Copy full SHA for ef9acbe - Browse repository at this point
Copy the full SHA ef9acbeView commit details -
Don't recursively try to rollback
We should only attempt the rollback functionality when we initially try to update machines, not on every rollback after that obviously
Configuration menu - View commit details
-
Copy full SHA for 29c9a2b - Browse repository at this point
Copy the full SHA 29c9a2bView commit details -
Disable deleting machines for now
We need to avoid deleting unmanaged machines
Configuration menu - View commit details
-
Copy full SHA for 81c5bad - Browse repository at this point
Copy the full SHA 81c5badView commit details -
fix up the code to create and destroy machines
wasn't testing that path correctly. also, i added a quick optimization to waitForMachineState to avoid an unnecessary API call (just checking the machine state right then and there)
Configuration menu - View commit details
-
Copy full SHA for cb96bd1 - Browse repository at this point
Copy the full SHA cb96bd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1934ec3 - Browse repository at this point
Copy the full SHA 1934ec3View commit details -
Check that configs are not equal before attempting an update
Just another saved API call
Configuration menu - View commit details
-
Copy full SHA for af3beea - Browse repository at this point
Copy the full SHA af3beeaView commit details -
Configuration menu - View commit details
-
Copy full SHA for e649eff - Browse repository at this point
Copy the full SHA e649effView commit details -
Small changes to focus on pushing forward deploys
The main change was just focusing on newAppState instead of the old. The rest is just cleaning up logging tbh
Configuration menu - View commit details
-
Copy full SHA for 2df9f45 - Browse repository at this point
Copy the full SHA 2df9f45View commit details -
Fix waiting for multiple states
If any one /wait finishes, we should just return
Configuration menu - View commit details
-
Copy full SHA for 33d1571 - Browse repository at this point
Copy the full SHA 33d1571View commit details -
Configuration menu - View commit details
-
Copy full SHA for d0855b0 - Browse repository at this point
Copy the full SHA d0855b0View commit details -
Return on some unrecoverable errors + cache health check results
The more interesting bit is caching health checks. If we're trying to push forward,there's no point in wasting bandcwidth always retrying health checks. If a machine passes them once, they're good forever
Configuration menu - View commit details
-
Copy full SHA for 870bf35 - Browse repository at this point
Copy the full SHA 870bf35View commit details -
Configuration menu - View commit details
-
Copy full SHA for 393d9ff - Browse repository at this point
Copy the full SHA 393d9ffView commit details -
Configuration menu - View commit details
-
Copy full SHA for aedf269 - Browse repository at this point
Copy the full SHA aedf269View commit details -
Configuration menu - View commit details
-
Copy full SHA for b9605f0 - Browse repository at this point
Copy the full SHA b9605f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for eda2604 - Browse repository at this point
Copy the full SHA eda2604View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2826af9 - Browse repository at this point
Copy the full SHA 2826af9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2972b27 - Browse repository at this point
Copy the full SHA 2972b27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 76f7f3d - Browse repository at this point
Copy the full SHA 76f7f3dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 138e2d9 - Browse repository at this point
Copy the full SHA 138e2d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6b526af - Browse repository at this point
Copy the full SHA 6b526afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3148eda - Browse repository at this point
Copy the full SHA 3148edaView commit details -
Configuration menu - View commit details
-
Copy full SHA for ce1b30f - Browse repository at this point
Copy the full SHA ce1b30fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a484615 - Browse repository at this point
Copy the full SHA a484615View commit details -
move lease collection out of restartMachines
I prefer that the updateMachines function controls it
Configuration menu - View commit details
-
Copy full SHA for 14167c9 - Browse repository at this point
Copy the full SHA 14167c9View commit details -
Configuration menu - View commit details
-
Copy full SHA for ba45439 - Browse repository at this point
Copy the full SHA ba45439View commit details -
Configuration menu - View commit details
-
Copy full SHA for 66dd69b - Browse repository at this point
Copy the full SHA 66dd69bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 29c1dcd - Browse repository at this point
Copy the full SHA 29c1dcdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9dfd3e2 - Browse repository at this point
Copy the full SHA 9dfd3e2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 038a367 - Browse repository at this point
Copy the full SHA 038a367View commit details -
Configuration menu - View commit details
-
Copy full SHA for a526558 - Browse repository at this point
Copy the full SHA a526558View commit details -
correctly catch unrecoverable errors
I wasn't correctly using errors.As. I also cleaned up catching context canceled errors
Configuration menu - View commit details
-
Copy full SHA for 1e819d5 - Browse repository at this point
Copy the full SHA 1e819d5View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8e91d1 - Browse repository at this point
Copy the full SHA e8e91d1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 008780a - Browse repository at this point
Copy the full SHA 008780aView commit details -
i forgot that lm isn't a pointer, so we need to check entry.leasableMachine for newly created machines
Configuration menu - View commit details
-
Copy full SHA for 175aa53 - Browse repository at this point
Copy the full SHA 175aa53View commit details -
Configuration menu - View commit details
-
Copy full SHA for f4e03bc - Browse repository at this point
Copy the full SHA f4e03bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9f79bea - Browse repository at this point
Copy the full SHA 9f79beaView commit details -
This covers most of the major functions and the major places there could be issues
Configuration menu - View commit details
-
Copy full SHA for 9009e4d - Browse repository at this point
Copy the full SHA 9009e4dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1482fd6 - Browse repository at this point
Copy the full SHA 1482fd6View commit details -
Configuration menu - View commit details
-
Copy full SHA for caade93 - Browse repository at this point
Copy the full SHA caade93View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4abfdca - Browse repository at this point
Copy the full SHA 4abfdcaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 02c11dd - Browse repository at this point
Copy the full SHA 02c11ddView commit details -
ensure that we use machinedeployment flapsClient
also add some tests for updateorcreatemachine
Configuration menu - View commit details
-
Copy full SHA for f207dac - Browse repository at this point
Copy the full SHA f207dacView commit details -
Configuration menu - View commit details
-
Copy full SHA for 299fcac - Browse repository at this point
Copy the full SHA 299fcacView commit details -
Configuration menu - View commit details
-
Copy full SHA for 941b1f4 - Browse repository at this point
Copy the full SHA 941b1f4View commit details -
md.warnAboutListenAddress requires this function
Configuration menu - View commit details
-
Copy full SHA for 0f4bcb9 - Browse repository at this point
Copy the full SHA 0f4bcb9View commit details -
Configuration menu - View commit details
-
Copy full SHA for ee7829f - Browse repository at this point
Copy the full SHA ee7829fView commit details -
Configuration menu - View commit details
-
Copy full SHA for b27654d - Browse repository at this point
Copy the full SHA b27654dView commit details -
Configuration menu - View commit details
-
Copy full SHA for c0266ef - Browse repository at this point
Copy the full SHA c0266efView commit details -
Configuration menu - View commit details
-
Copy full SHA for da18ca2 - Browse repository at this point
Copy the full SHA da18ca2View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf82b9d - Browse repository at this point
Copy the full SHA bf82b9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 95a3c94 - Browse repository at this point
Copy the full SHA 95a3c94View commit details -
we can't start machines if we have a lease acquired
Configuration menu - View commit details
-
Copy full SHA for 42c1964 - Browse repository at this point
Copy the full SHA 42c1964View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4998e71 - Browse repository at this point
Copy the full SHA 4998e71View commit details -
Configuration menu - View commit details
-
Copy full SHA for b251e56 - Browse repository at this point
Copy the full SHA b251e56View commit details -
set default for deploy-retries to 0
i also added back the original deployment code, and use that if deploy-restries is set to 0. That way, we can more slowly roll this out without risking breaking user apps if there's some terrible bug
Configuration menu - View commit details
-
Copy full SHA for a717b49 - Browse repository at this point
Copy the full SHA a717b49View commit details -
get deploy-retries from launchdarkly by default
Users can still set deploy-retries to whatever value they'd like, however.
Configuration menu - View commit details
-
Copy full SHA for f5b3330 - Browse repository at this point
Copy the full SHA f5b3330View commit details