Backport of Return 429 response on HTTP max connection limit into release/1.3.x #14241
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This PR is auto-generated from #13621 to be assessed for backporting due to the inclusion of the label backport/1.3.x.
The below text is copied from the body of the original PR.
The
limits.http_max_conns_per_client
configuration option sets a limit on the max number of concurrent TCP connections per client IP address:nomad/website/content/docs/configuration/index.mdx
Lines 197 to 200 in 0b3ba5c
The current implementation behavior silently closes the TCP connection, resulting in opaque 'unexpected EOF' or 'connection reset' errors which many users fail to correctly diagnose, and is likely the root cause of various reported Nomad issues.
Instead of silently closing the connection, this PR returns a
429 Too Many Requests
HTTP response with a helpful error message to aid debugging when the connection limit is unintentionally reached. This should help diagnose/fix some instances of #12273 and #8718, where the fix is simply to increase the connection limit from the default of 100.The default of 100 is easily hit when doing anything with blocking queries, since each blocking query requires a separate HTTP client connection. One example from my use-case: the Nomad Autoscaler monitors each job status using a blocking query, which means that any autoscaler agent monitoring more than 100 jobs will exceed this default limit in normal operation and will need to be increased.
One tradeoff in this PR is that the connection is kept open slightly longer as the agent writes out the 429 response. The
HTTPConnStateFuncWithDefault429Handler
function accepts a write-deadline argument, which I've set as0 (no deadline)(updated:10 * time.millisecond
to match existing behavior in Consul) in this PR, this could alternatively be made configurable through an additional config option, if desired.