Return HTTP 429 when performing too many requests from 1 IP on Consul agent

[pierresouchay]

With Consul 1.6.3+, a new limit http_max_conns_per_client has been added, but this limit is causing lots of issues (I have been contacted by 3 people for this issue and spent quite some time on it as well).

While the value has been upgraded to 200 in Consul 1.7.1, it still cause lots of issues because many SDKs and application don't deal well with just closing the connection, even with Vault or hashicorp/consul-template#1346.

Implementing this detection is also harder as I tried with consul-templaterb since it has to work with heuristics instead of having a clear message.

Simply closing the connection make the troubleshooting hard (I spent quite some time fighting with Curl to understand what was going on while testing Consul 1.6.3)

Could we consider returning HTTP 429 instead? It would help quite a lot SDK developpers as well as being easier to diagnose by operators.

[orarnon]

This limit is non-visible and hurt us in production as well. I urge you to reflect that as @pierresouchay suggests.

[jsosulska]

Hi @pierresouchay & @orarnon,

I've added UX and enhancement labels to this for discussion. Thank you for the recommendation and clarity :)

I will update here when I have more information.

Best,
Jono

[pierresouchay]

@i0rek Made hashicorp/go-connlimit#6 and made a demo on how it could be integrated into Consul: pierresouchay@8399df9 (The last one is a WIP - not ready, just to prove it is doable)

[urjitbhatia]

We were hurt by this as well - please consider this proposal

[pierresouchay]

Our testing team had this issue today in their tests (not in the infrastructure since we are aware of the issue). People not reading the changelog all the time, when they have this issue, are completely lost, it would really help users to have better diagnosis.

[banks]

Merged the library PR - thanks. @i0rek this should be easy enough to pull in and enable for Consul now.

[pierresouchay]

@i0rek @banks Basic implementation: #8221

More complex (but I am not convinced this is needed): pierresouchay@8399df9

[Nagarajuwas]

@pierresouchay thank you for this change .can we know this default value 200 is for second or minute for consul client up address.

Consul documents is not mentioned any details on timings.i am assuming this value for second.

[pierresouchay]

That's concurrent connection at a given time (so opened connections).
So 200 means 1 IP can have 200 connections to the consul agent.
AFAIK, there is not timing involved.