Upgrade go-connlimit to v0.3.0 / return http 429 on too many connections

[pierresouchay]

This implementation is basic and returns HTTP 429 when too many connections are done as explained in #7527, and will:

• let SDKs or other applications to self regulate when Consul refuses connections
• operators to diagnose easily issues

In my experience, this is one of the biggest source of questions since the introduction of the feature in Consul 1.6.x

This implementation is simple as it uses the default implementation from hashicorp/go-connlimit#6, I made a proof of concept to handle it differently to avoid closing the connection in the main thread as quickly demonstrated here: pierresouchay@8399df9

I am not sure it deserves such complexity, however, but if you want to another implementation, it is still possible.

[hanshasselberg]

I want to highlight this and explain what I think the implications are and make sure we are aware:

• HTTPConnStateFunc closes the connection when it is beyond the limit. Close does not block.
• HTTPConnStateFuncWithDefault429Handler(10 * time.Millisecond) blocks until the following is done (worst case):
  1. conn.SetDeadline(10*time.Millisecond) so that
  2. conn.Write(429error) is guaranteed to timeout after 10ms, so that the http 429 can be written and
  3. conn.Close can happen

The implication of this change is that accepting any new connection is worst case delayed by 10ms. But only after a client reached the limit already.

[pierresouchay]

Yes, indeed, you are fully correct. From my test, even by forcing very high latency with the clients, there ate no measurable change since write/close is fast because probably buffered by the OS.

[hanshasselberg]

LGTM!