[internal] The session has been destroyed #683
Comments
|
I tried to recover and got following stack trace: |
|
so it is coming from here: https://github.com/bufbuild/connect-es/blob/76b17edbbc4c0d4c98d1de8eb754192835458054/packages/connect-node/src/node-universal-client.ts#L340 the rest of stack trace is wrapping an error in connect error |
|
That's good info. Since the stack does not include This means the connection owned by the ready state has been destroyed without the state noticing. Looking at the source, I don't see a smoking gun, we're maintaining event listeners on "close" and "error" all the time. It's also possible that the connection is already dead if we enter the "ready" state. We noticed some unexpected behavior with the |
|
fyi we upgrade to 0.10.1 but we still can see it happens for some users, so it does not seem to go away with #681 |
|
@timostamm We also still see |
|
@akosyakov, with #681, a new call will open a new connection if the current connection has received a GOAWAY. The error you see is unexpected. Will update #680 with the new information. |
|
I'm able to reproduce: $ node -v
v18.17.0
$ node repro-683.mjs
http2.connect
conn on connect
conn.request
req on finish
req on response 200
conn on goaway, code: 2
req on error Error [ERR_HTTP2_SESSION_ERROR]: Session closed with error code 2
req on close
connection after request closed: closed: false destroyed: true
conn on error Error [ERR_HTTP2_SESSION_ERROR]: Session closed with error code 2
conn on closerepro-683.mjsimport * as http2 from "http2";
// start a server that sends a GOAWAY with INTERNAL ERROR to every session
const PORT = 8088;
const server = http2.createServer()
.on("session", (sess) => {
setTimeout(() => {
sess.goaway(http2.constants.NGHTTP2_INTERNAL_ERROR)
}, 100);
})
.on("request", (req, res) => {
res.writeHead(200);
setTimeout(() => res.end(), 200);
});
await new Promise(resolve => {
server.listen(PORT, () => resolve());
});
// open a connection
const conn = await new Promise((resolve, reject) => {
console.log("http2.connect");
const conn = http2.connect(`http://localhost:${PORT}`);
conn.on("error", reject);
conn.on("connect", () => {
console.log("conn on connect");
resolve(conn);
});
conn.on("goaway", errorCode => {
console.log("conn on goaway, code:", errorCode);
});
conn.on("close", () => {
console.log("conn on close")
});
conn.on("error", e => {
console.log("conn on error", String(e));
});
});
// make a request
await new Promise((resolve) => {
console.log("conn.request");
const stream = conn.request({
":method": "POST",
":path": "/",
});
stream.on("response", h => {
console.log("req on response", h[":status"]);
});
stream.on("error", e => {
console.log("req on error", String(e));
});
stream.on("finish", () => {
console.log("req on finish");
});
stream.on("close", () => {
console.log("req on close");
resolve();
});
stream.on("close", resolve);
stream.end();
});
console.log("connection after request closed: closed:", conn.closed, "destroyed:", conn.destroyed);
await new Promise(resolve => setTimeout(resolve, 500))
process.exit(0);When the Since the request received the "error" and "close" events, it is complete, and user-code will continue. For example with a second request. But at this point, the session is destroyed ( |
|
v0.12.0 was just published with #712. We can reproduce the fix with the following script that uses the session manager: repro.mjsimport * as http2 from "http2";
import { createNodeHttpClient, Http2SessionManager } from "@bufbuild/connect-node";
import assert from "assert";
// start a server that sends a GOAWAY with INTERNAL ERROR to every session
const PORT = 8088;
const server = http2.createServer()
.on("session", (sess) => {
setTimeout(() => {
sess.goaway(http2.constants.NGHTTP2_INTERNAL_ERROR)
}, 100);
})
.on("request", (req, res) => {
res.writeHead(200);
setTimeout(() => res.end(), 200);
});
await new Promise(resolve => {
server.listen(PORT, () => resolve());
});
// create an h2 client
const sm = new Http2SessionManager(`http://localhost:${PORT}`);
const client = createNodeHttpClient({
httpVersion: "2",
sessionProvider: () => sm
});
// make requests
const res1 = await client({
url: `http://localhost:${PORT}`,
method: "POST",
header: new Headers(),
});
try {
for await (const chunk of res1.body) {
// consume response body to see error
}
} catch (e) {
assert(String(e) === "ConnectError: [internal] Session closed with error code 2");
}
console.log("res1 done")
const res2 = await client({
url: `http://localhost:${PORT}`,
method: "POST",
header: new Headers(),
});
console.log("res2 done")$ npm i @bufbuild/connect-node@0.11.0
$ node repro.mjs
res1 done
file:///Users/ts/Downloads/xrepro/node_modules/@bufbuild/connect/dist/esm/connect-error.js:70
return new ConnectError(reason.message, code);
^
ConnectError: [internal] The session has been destroyed
...$ npm i @bufbuild/connect-node@0.12.0
$ node repro.mjs
res1 done
res2 done
^CI'm going to close this issue as resolved. But in case you do still experience issues, let us know. Without a reproducible case, issues like this are impossible to fix while maintaining code quality though. |
|
@timostamm we released a new version of our extension with the connect library updated to |
|
Thanks for the confirmation, @akosyakov and @jeanp413. I was afraid that might happen. We discussed the issue and we are okay with applying a broad workaround. It will take use a couple of days. In the meantime, the recommended course is to pin to v0.11.0 if you are experiencing this issue. |
|
The new fix just shipped in v0.13.2. Let us know if this fixes the problem, @akosyakov and @jeanp413. |
version: 0.10.0
Unfortunately stack trace is minified on our side not sure which path lead to that, it seem to be an unary call. It seems somewhere stream is called when it is already destroyed.
The text was updated successfully, but these errors were encountered: