Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid incorrect maintained conn state #712

Merged
merged 9 commits into from
Jul 25, 2023

Conversation

mustard-mh
Copy link
Contributor

@mustard-mh mustard-mh commented Jul 6, 2023

This PR fixes the issue where the connection state maintained in lib was not identical to http2, resulting in no new connections being created. see also #683 (comment)

We use conn.destroyed and conn.closed since those properties are exposed in http2 core, and where they throw The session has been destroyed 1 are all checking with those properties.

And properties are compatible [http2/core v16.0] [connect-node min support v16.0]

Related Issues

Fix #680, #683

How to test

No idea, but we released a patch fork in npm and use it in our product [1], errors are gone

Footnotes

  1. search ERR_HTTP2_INVALID_SESSION in http2/core, they all check if (this.destroyed) { throw }

@CLAassistant
Copy link

CLAassistant commented Jul 6, 2023

CLA assistant check
All committers have signed the CLA.

@akosyakov
Copy link
Contributor

@timostamm I can confirm no more GOAWAY and session has been destroyed errors. Our SLO went up from 90 to 97%. We were running with this change for last 5 days.

Copy link
Member

@timostamm timostamm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mustard-mh, thank you for the PR. I'm glad to hear the patch brought down your error numbers, but I hope you understand we're hesitant to merge without understanding what is actually happening.

@@ -246,7 +246,7 @@ export class Http2SessionManager {

private async gotoReady() {
if (this.s.t == "ready") {
if (this.s.isShuttingDown()) {
if (this.s.conn.destroyed || this.s.conn.closed || this.s.isShuttingDown()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will avoid the problem found in #683 (comment). I think we can simply put all connections that received any GOAWAY frame into the "shutting down" state. We will need a test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed up a test in bf4d746. New requests open a new connection 👍

But the test framework reports an uncaught exception, see run log. That's because we remove event listeners for the connection here: https://github.com/bufbuild/connect-es/blob/4d1580fe8769a07be4aeb25f2eb4a411e90861f0/packages/connect-node/src/http2-session-manager.ts#L634-L644

Pushed up a15680d to use isShuttingDown for all GOAWAY frames. This means we are deferring the cleanup in onExitState, and pass the test without uncaught exception.

packages/connect-node/src/http2-session-manager.ts Outdated Show resolved Hide resolved
Copy link
Member

@timostamm timostamm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mustard-mh, pushed up a test and a change that moves the logic into isShuttingDown because to not leave uncaught errors. That way, you're still credited for your contribution.

@timostamm
Copy link
Member

CI failure is just browserstack, which does not have credentials in the fork. Since this does not affect any code tested in browserstack, we're good.

@timostamm timostamm merged commit 9f35c9f into connectrpc:main Jul 25, 2023
@timostamm timostamm changed the title Use built-in properties to avoid incorrect maintained conn state Avoid incorrect maintained conn state Jul 25, 2023
This was referenced Jul 25, 2023
@mustard-mh
Copy link
Contributor Author

Thank you @timostamm 🙏 , we will try new release on our side

@akosyakov
Copy link
Contributor

akosyakov commented Aug 9, 2023

@timostamm We tried and unfortunately had to revert back to our fork, i.e. original @mustard-mh changes. We still see The session has been destroyed internal errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New streams cannot be created after receiving a GOAWAY
5 participants