You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a build is attempted, but a web/api instance throws a 5xx response, a number of strange behaviors can result:
Builds will get stuck in triggered state
Builds will fail randomly when updating the api fails
5xx responses will be returned from the web servers to users
Part of the resolution to this may be defensive protection around intermittent 5xx responses. This may take some operation changes as well though, as our load balancer should really assume a 5xx response is enough to dislodge the server from the lb group.
If that's the way to go, I suppose we could modify our slumber client API class to do this magic. It would be IO blocking, but I suppose that would be fine.
@agjohnson labeled as Operations so he is maybe thinking in another thing.
Yeah, my original take on this is that if a server is overloaded, we should be detecting this and limiting traffic to the server. It's probably a sign that our web servers are overloaded.
When a build is attempted, but a web/api instance throws a 5xx response, a number of strange behaviors can result:
Part of the resolution to this may be defensive protection around intermittent 5xx responses. This may take some operation changes as well though, as our load balancer should really assume a 5xx response is enough to dislodge the server from the lb group.
Raised from #2255
The text was updated successfully, but these errors were encountered: