http: client keep-alive for UNIX domain sockets #13214

bengl · 2017-05-25T08:02:54Z

Makes Connection: keep-alive behave correctly when making client
connections to UNIX domain sockets.

Prior to this, connections would never be re-used, but the keep-alive
would cause the connections to stick around until they time out. This
would lead to an eventual EMFILE error due to all the connections
staying open. This was due to http.Agent not properly supporting UNIX
domain sockets.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

http, test

bengl · 2017-05-25T08:09:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/8296/

lpinca · 2017-05-25T08:57:27Z

semver-major? It seems that keep-alive was skipped on purpose when using UNIX domain sockets.

refack · 2017-05-25T10:45:17Z

lib/_http_client.js

@@ -274,7 +259,7 @@ function ClientRequest(options, cb) {
    this._last = true;
    this.shouldKeepAlive = false;
    if (typeof options.createConnection === 'function') {
-      newSocket = options.createConnection(options, oncreate);
+      var newSocket = options.createConnection(options, oncreate);


If it's initialized, never re-assigned, scope limited, and quacks like a duck; it's const

Good call! I left it as var as it had already been that on line 244 previously, but const is more appropriate here now.

refack · 2017-05-25T11:07:36Z

lib/_http_client.js

-  var newSocket;
-  if (this.socketPath) {
-    this._last = true;
-    this.shouldKeepAlive = false;


@bnoordhuis I tried to do a deep blame, seems like it's your code

node/lib/http.js

Line 1025 in bc7cfd7

self.shouldKeepAlive = false;

Is there a specific reason why POSIX sockets shouldKeepAlive = false?

jasnell · 2017-05-25T15:17:50Z

this could be viewed as semver-minor, not major, in that it's adding support for something that didn't work previously.

lpinca · 2017-05-25T16:04:58Z

req.socketPath should be removed from this condition if these changes are ok.

Refs: #10818

bengl · 2017-05-25T16:26:46Z

@jasnell @lpinca I'd be tempted to call this semver-patch, because prior to the change, it's not re-using the socket for keep-alive, and too many file descriptors are used, triggering an EMFILE. Seems like a bug to me.

jasnell · 2017-05-25T18:59:19Z

That works for me.

Trott · 2017-05-25T22:23:50Z

test/parallel/test-http-unix-socket-keep-alive.js

+  http.get({
+    socketPath: common.PIPE,
+    headers: {connection: 'keep-alive'}
+  }, (res) => res.on('data', () => {}).on('error', assert.fail).on('end', cb));


Nit: Instead of .on('data', () => {}), use .on('data', common.mustNotCall()) to make it clear that the expectation is that the handler will never execute in this test.

lpinca · 2017-05-26T05:57:15Z

test/parallel/test-http-unix-socket-keep-alive.js

+  asyncLoop(makeKeepAliveRequest, 10, () =>
+    server.getConnections(common.mustCall((err, conns) => {
+      assert.ifError(err);
+      assert.strictEqual(1, conns);


Nit: can you please swap the arguments?

lpinca · 2017-05-26T05:57:43Z

test/parallel/test-http-unix-socket-keep-alive.js

+
+function asyncLoop(fn, times, cb) {
+  fn(function handler() {
+    if (--times) {


~~Nit: if I'm not wrong this should be times--.~~

lpinca · 2017-05-26T08:43:07Z

lib/_http_agent.js

@@ -130,6 +130,9 @@ Agent.prototype.getName = function getName(options) {
  if (options.family === 4 || options.family === 6)
    name += ':' + options.family;

+  if (options.socketPath)
+    name += ':' + options.socketPath;


I think it makes sense to extend test/parallel/test-http-agent-getname.js and cover this.

Nit: the socketPath should not be set when a family, a port or a localAddress is set. Therefore it would be nicer branch wise to write

if (options.socketPath) { name += `:::${options.socketPath}`; } else { if (options.port) // ... if (options.localAddress) // ... // ... }

I think it is similar with servername down below.

lpinca · 2017-05-26T08:49:24Z

@jasnell @lpinca I'd be tempted to call this semver-patch, because prior to the change, it's not re-using the socket for keep-alive, and too many file descriptors are used, triggering an EMFILE. Seems like a bug to me.

Yes, I think you are right.

jasnell · 2017-06-02T03:30:36Z

test/parallel/test-http-unix-socket-keep-alive.js

+
+common.refreshTmpDir();
+
+server.listen(common.PIPE, () =>


common.mustCall() on these?

There's a common.mustCall() at the callback that's actually making assertions, and in order to get there, this callback needs to be called, so I think it's implied. (Or, is there a best-test-practice for this stating all callbacks should have a mustCall()?)

(Or, is there a best-test-practice for this stating all callbacks should have a mustCall()?)

Nah, it's a matter of judgment. There are certainly callbacks that should not be wrapped (such as callbacks in cluster workers, which will throw but be ignored by the cluster master, so having them wrapped gives a false sense of security that the callback is being executed).

I guess I'll put them in anyway to be super clear that these are all expected to run.

bengl · 2017-06-02T06:40:53Z

@lpinca @jasnell @Trott Cool I think your comments are all addressed. PTAL

@bnoordhuis As @refack mentioned, it would be great to get your input here. 😃

lpinca · 2017-06-02T19:23:49Z

New CI: https://ci.nodejs.org/job/node-test-pull-request/8446/

BridgeAR

Needs a rebase and I have a nit but it should not block this from landing. Otherwise LGTM

BridgeAR · 2017-08-27T00:34:22Z

lib/_http_agent.js

@@ -130,6 +130,9 @@ Agent.prototype.getName = function getName(options) {
  if (options.family === 4 || options.family === 6)
    name += ':' + options.family;

+  if (options.socketPath)
+    name += ':' + options.socketPath;


Nit: the socketPath should not be set when a family, a port or a localAddress is set. Therefore it would be nicer branch wise to write

if (options.socketPath) { name += `:::${options.socketPath}`; } else { if (options.port) // ... if (options.localAddress) // ... // ... }

I think it is similar with servername down below.

bengl · 2017-08-29T06:31:21Z

Hey folks! Sorry, I'm going to close this one for now. I'd tried rebasing it, and when I do, it exposes issues with tls that I'm resolving over here. Once that lands, I think it will be safe to re-open this one (which I'll add a tls test to).

BridgeAR · 2017-08-29T07:03:52Z

You could just add the blocked label and reopen now?

bengl · 2017-08-29T07:14:49Z

@BridgeAR Oh! That hadn't occurred to me. Thanks for pointing out that I could do that.

BridgeAR · 2017-09-22T23:07:58Z

As the other PR should land today, this should be rebased.

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets.

bengl · 2017-09-23T00:22:51Z

Rebased.

New CI: https://ci.nodejs.org/job/node-test-pull-request/10213/

bengl · 2017-09-23T02:05:52Z

Just a quick call-out to @nodejs/http and @bnoordhuis for review. I think unless there's any objections, I'll probably land this early next week.

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets. PR-URL: #13214 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

BridgeAR · 2017-09-28T05:29:29Z

Ping @bengl

bengl · 2017-09-28T05:57:05Z

Landed as 2043944.

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets. PR-URL: nodejs#13214 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

`Connection: keep-alive` is now properly supported when making client connections to UNIX domain sockets so `request.abort()` should not blindly destroy the underlying socket. Refs: nodejs#13214 (comment)

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets. PR-URL: #13214 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets. PR-URL: nodejs/node#13214 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

Makes `Connection: keep-alive` behave correctly when making client connections to UNIX domain sockets. Prior to this, connections would never be re-used, but the keep-alive would cause the connections to stick around until they time out. This would lead to an eventual EMFILE error due to all the connections staying open. This was due to http.Agent not properly supporting UNIX domain sockets. PR-URL: #13214 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins · 2017-10-17T19:00:42Z

Should this be backported to v6.x-staging? If yes please follow the guide and raise a backport PR, if not let me know or add the dont-land-on label.

bengl · 2017-10-17T19:05:38Z

@MylesBorins this depends on #14564, which is marked dont-land-on-v6.x so I'll add the label here as well.

`Connection: keep-alive` is now properly supported when making client connections to UNIX domain sockets so `request.abort()` should not blindly destroy the underlying socket. PR-URL: #15650 Refs: #13214 (comment) Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>

`Connection: keep-alive` is now properly supported when making client connections to UNIX domain sockets so `request.abort()` should not blindly destroy the underlying socket. PR-URL: nodejs/node#15650 Refs: nodejs/node#13214 (comment) Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>

nodejs-github-bot added the http Issues or PRs related to the http subsystem. label May 25, 2017

refack reviewed May 25, 2017

View reviewed changes

Trott reviewed May 25, 2017

View reviewed changes

lpinca reviewed May 26, 2017

View reviewed changes

lpinca approved these changes May 26, 2017

View reviewed changes

lpinca reviewed May 26, 2017

View reviewed changes

jasnell reviewed Jun 2, 2017

View reviewed changes

BridgeAR approved these changes Aug 27, 2017

View reviewed changes

lpinca requested a review from bnoordhuis August 27, 2017 13:14

jasnell approved these changes Aug 29, 2017

View reviewed changes

bengl closed this Aug 29, 2017

bengl added the blocked PRs that are blocked by other issues or PRs. label Aug 29, 2017

bengl reopened this Aug 29, 2017

bengl added 2 commits September 22, 2017 17:07

[squash] the nits

2da5453

bengl force-pushed the bengl/unixkeepalive branch from d41b431 to 2da5453 Compare September 23, 2017 00:18

BridgeAR removed the blocked PRs that are blocked by other issues or PRs. label Sep 23, 2017

BridgeAR approved these changes Sep 23, 2017

View reviewed changes

bengl closed this Sep 28, 2017

lpinca mentioned this pull request Sep 28, 2017

http: do not blindly destroy UNIX domain sockets #15650

Closed

3 tasks

MylesBorins mentioned this pull request Oct 3, 2017

v8.7.0 proposal #15762

Merged

MylesBorins added the backport-requested-v6.x label Oct 17, 2017

bengl added the dont-land-on-v6.x label Oct 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http: client keep-alive for UNIX domain sockets #13214

http: client keep-alive for UNIX domain sockets #13214

bengl commented May 25, 2017

bengl commented May 25, 2017

lpinca commented May 25, 2017

refack May 25, 2017

bengl May 25, 2017

refack May 25, 2017

jasnell commented May 25, 2017

lpinca commented May 25, 2017

bengl commented May 25, 2017

jasnell commented May 25, 2017

Trott May 25, 2017

lpinca May 26, 2017 •

edited

Loading

lpinca May 26, 2017 •

edited

Loading

lpinca May 26, 2017

BridgeAR Aug 27, 2017

lpinca commented May 26, 2017

jasnell Jun 2, 2017

bengl Jun 2, 2017

Trott Jun 2, 2017

bengl Jun 2, 2017

bengl commented Jun 2, 2017

lpinca commented Jun 2, 2017

BridgeAR left a comment

BridgeAR Aug 27, 2017

bengl commented Aug 29, 2017

BridgeAR commented Aug 29, 2017

bengl commented Aug 29, 2017

BridgeAR commented Sep 22, 2017

bengl commented Sep 23, 2017

bengl commented Sep 23, 2017

BridgeAR commented Sep 28, 2017

bengl commented Sep 28, 2017

MylesBorins commented Oct 17, 2017

bengl commented Oct 17, 2017


		common.refreshTmpDir();

		server.listen(common.PIPE, () =>

http: client keep-alive for UNIX domain sockets #13214

http: client keep-alive for UNIX domain sockets #13214

Conversation

bengl commented May 25, 2017

Checklist

Affected core subsystem(s)

bengl commented May 25, 2017

lpinca commented May 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell commented May 25, 2017

lpinca commented May 25, 2017

bengl commented May 25, 2017

jasnell commented May 25, 2017

Choose a reason for hiding this comment

lpinca May 26, 2017 • edited Loading

Choose a reason for hiding this comment

lpinca May 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lpinca commented May 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bengl commented Jun 2, 2017

lpinca commented Jun 2, 2017

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bengl commented Aug 29, 2017

BridgeAR commented Aug 29, 2017

bengl commented Aug 29, 2017

BridgeAR commented Sep 22, 2017

bengl commented Sep 23, 2017

bengl commented Sep 23, 2017

BridgeAR commented Sep 28, 2017

bengl commented Sep 28, 2017

MylesBorins commented Oct 17, 2017

bengl commented Oct 17, 2017

lpinca May 26, 2017 •

edited

Loading

lpinca May 26, 2017 •

edited

Loading