flaps: retry on certain failure modes, such as 504s
#3601
Conversation
These functions allow retrying machine operations like machine creation, though there's no logic yet for determining which errors are *actually* transient, so behavior is still the same
backoff modifies the Backoff state object while using it, so having one globally makes me uncomfortable
except for ephemeral machines - i'm cooking up a different strategy to do this so this is more of a check-in than anything
This reverts commit 49e38ba.
This reverts commit fb76c76.
Since Flaps has been abstracted to an Interface in flyctl, we can add a retry layer to it by just wrapping the existing client with *few* changes made elsewhere This is just a drop-in reliability improvement
|
Retrying POST/PATCH my yield unexpected results cause subsequent requests might to go to a different flaps and flyd among other issues. It's basically not Idempotent. https://flyio.discourse.team/t/flaps-what-status-codes-can-we-retry/5060/2 |
ah. that's a huge bummer. I'm going to try to see if I can get this working for GET requests then, and maybe afterwards come up with a different strategy for I wonder if some operations, like |
Going to have to follow up to see whether all of these are safe to retry, but I feel like they *should* be?
Change Summary
What and Why: All flaps operations will retry if the error is known to be transient. This should improve reliability - especially in situations like
fly deploywhere one hiccup can (currently) stop an entire deploy.How: Adds a wrapper in
flapsutilover*flaps.Clientthat implements retrying on certain failure modes, like 504.NewClientWithOptionsnow returns the genericFlapsClientso that we can return the wrapper type instead of a raw*flaps.Client.Documentation