-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Return HTTP 429 when performing too many requests from 1 IP on Consul agent #7527
Comments
This limit is non-visible and hurt us in production as well. I urge you to reflect that as @pierresouchay suggests. |
Hi @pierresouchay & @orarnon, I've added UX and enhancement labels to this for discussion. Thank you for the recommendation and clarity :) I will update here when I have more information. Best, |
@i0rek Made hashicorp/go-connlimit#6 and made a demo on how it could be integrated into Consul: pierresouchay@8399df9 (The last one is a WIP - not ready, just to prove it is doable) |
We were hurt by this as well - please consider this proposal |
Our testing team had this issue today in their tests (not in the infrastructure since we are aware of the issue). People not reading the changelog all the time, when they have this issue, are completely lost, it would really help users to have better diagnosis. |
Merged the library PR - thanks. @i0rek this should be easy enough to pull in and enable for Consul now. |
@i0rek @banks Basic implementation: #8221 More complex (but I am not convinced this is needed): pierresouchay@8399df9 |
@pierresouchay thank you for this change .can we know this default value 200 is for second or minute for consul client up address. Consul documents is not mentioned any details on timings.i am assuming this value for second. |
That's concurrent connection at a given time (so opened connections). |
IMPROVEMENTS: * Allow customization to sort nodes in service view in Consul-UI ([#71](#71)) NEW FEATURES: * Auto-detection of rate-limit on Consul Side (introduced in Consul 1.6.2+), this feature will work only Consul 1.9+ (see [Consul #7527](hashicorp/consul#7527)).
IMPROVEMENTS: * Allow customization to sort nodes in service view in Consul-UI ([criteo#71](criteo#71)) NEW FEATURES: * Auto-detection of rate-limit on Consul Side (introduced in Consul 1.6.2+), this feature will work only Consul 1.9+ (see [Consul #7527](hashicorp/consul#7527)).
With Consul 1.6.3+, a new limit http_max_conns_per_client has been added, but this limit is causing lots of issues (I have been contacted by 3 people for this issue and spent quite some time on it as well).
While the value has been upgraded to 200 in Consul 1.7.1, it still cause lots of issues because many SDKs and application don't deal well with just closing the connection, even with Vault or hashicorp/consul-template#1346.
Implementing this detection is also harder as I tried with consul-templaterb since it has to work with heuristics instead of having a clear message.
Simply closing the connection make the troubleshooting hard (I spent quite some time fighting with Curl to understand what was going on while testing Consul 1.6.3)
Could we consider returning HTTP 429 instead? It would help quite a lot SDK developpers as well as being easier to diagnose by operators.
The text was updated successfully, but these errors were encountered: