fix(node-http-handler): handle NodeHttp2Handler session failure #2289
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fix: detect errors on the NodeHttp2Handler, immediately destroy connections on unexpected error mode, and reconnect. Prior to this PR, if the server sent the client a GOAWAY frame, the session would not be removed from the connection pool and requests would fail indefinitely. This tries to avoid keeping streams(requests) on the Http2Session (tcp connection) from being stuck in an open state waiting for a gentle close even if there were unexpected protocol or connection errors during the request, assuming http2 errors are rare. (if a server or load balancer or network is misbehaving, close() might get stuck waiting for requests to finish, especially if requests and sessions don't have timeouts?) I'm only slightly familiar with http/2 client implementations from working on clients for Apple Push Notification Service. - In those, the client could rely on a session and request timeout existing, so close() would finish. In aws-sdk-js-v3, timeouts are optional. - I've seen some strange race conditions prior to node 12 in different client implementations for close and/or destroying Fixes aws#1525
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Codecov Report
@@ Coverage Diff @@
## main #2289 +/- ##
=======================================
Coverage ? 60.02%
=======================================
Files ? 470
Lines ? 24773
Branches ? 5866
=======================================
Hits ? 14870
Misses ? 9903
Partials ? 0 Continue to review full report at Codecov.
|
TysonAndre
commented
May 5, 2021
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
#1525
Description
detect errors on the NodeHttp2Handler, immediately destroy
connections on unexpected error mode, and reconnect.
Prior to this PR, if the server sent the client a GOAWAY frame,
the session would not be removed from the connection pool and requests
would fail indefinitely. This fixes that.
This tries to avoid keeping streams(requests) on the Http2Session
(tcp connection) from being stuck in an open state waiting for a gentle
close even if there were unexpected protocol or connection errors
during the request, assuming http2 errors are rare.
(if a server or load balancer or network is misbehaving,
close() might get stuck waiting for requests to finish,
especially if requests and sessions don't have timeouts?)
I'm only slightly familiar with http/2 client implementations from
working on clients for Apple Push Notification Service.
existing, so close() would finish. In aws-sdk-js-v3, timeouts are optional.
client implementations for (certain combinations of) close and/or destroying
Testing
Unit tests were added to confirm that disconnections initiated by the server
resulted in the connection being removed from the connection pool.
More thorough manual tests would be useful to test all possible networking edge cases,
e.g. completely losing a connection to aws load balancers, a connection closing without a goaway frame, etc,
testing this with a high volume of traffic to see if it recovers properly, etc.,
but I'm not familiar with how to simulate that accurately in node
Additional context
Add any other context about the PR here.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.