Issue with Node 4.x when receiving pipelined reqeusts

[DamianEdwards]

After updating to Node 4.x from 0.12.x, benchmarking by sending pipelining requests results in an initial spike of requests followed by no requests being processed on those connections, and those connections remaining open even after the load generator client finishes (connection leak). The same behavior does not occur when not pipelining requests, and it doesn't occur on 0.10.x or 0.12.x at all. The issue reproduces on both Linux and Windows.

I'm generating load using "wrk" (from a remote Linux machine) and the app is the benchmark app from the TechEmpower benchmarks.

[rmg]

@DamianEdwards which specific version? v4.2.0? v4.1.2?

[DamianEdwards]

I reproduced on both 4.1.2 and 4.2.0

[brendanashworth]

Does it occur on 4.1.1? There were a few changes in 4.1.2 (0504066, b3d9678) that might be causing this.

[DamianEdwards]

I'll drop back to 4.1.1 tomorrow and try it out.

[Fishrock123]

cc @indutny

[indutny]

@DamianEdwards does removing those lines help?

• https://github.com/nodejs/node/blob/master/lib/_http_server.js#L273-L274
• https://github.com/nodejs/node/blob/master/lib/_http_server.js#L459

[indutny]

Actually, latter one should be enough to verify it.

[DamianEdwards]

@indutny I'm assuming I have to build from source to try out removing that line?

[indutny]

@DamianEdwards yeah. Sorry, I forgot to mention it.

[DamianEdwards]

@indutny 😄
I'll see what I can do. That change was added in 4.1.2?

[indutny]

Yep.

[indutny]

Found one bug, already:

diff --git a/lib/_http_outgoing.js b/lib/_http_outgoing.js
index 6650c06..366c516 100644
--- a/lib/_http_outgoing.js
+++ b/lib/_http_outgoing.js
@@ -663,6 +663,9 @@ OutgoingMessage.prototype._flush = function() {
       this.output = [];
       this.outputEncodings = [];
       this.outputCallbacks = [];
+      if (this._onPendingData !== null)
+        this._onPendingData(-this.outputSize);
+      this.outputSize = 0;
     }

     if (this.finished) {

Looking for more.

[indutny]

Second part of issue is handling of paused parser in consumed socket. Working on fix.

[indutny]

Full fix:

diff --git a/lib/_http_outgoing.js b/lib/_http_outgoing.js
index 6650c06..366c516 100644
--- a/lib/_http_outgoing.js
+++ b/lib/_http_outgoing.js
@@ -663,6 +663,9 @@ OutgoingMessage.prototype._flush = function() {
       this.output = [];
       this.outputEncodings = [];
       this.outputCallbacks = [];
+      if (this._onPendingData !== null)
+        this._onPendingData(-this.outputSize);
+      this.outputSize = 0;
     }

     if (this.finished) {
diff --git a/lib/_http_server.js b/lib/_http_server.js
index c11d369..ffaa0b8 100644
--- a/lib/_http_server.js
+++ b/lib/_http_server.js
@@ -270,6 +270,7 @@ function connectionListener(socket) {
     // need to pause TCP socket/HTTP parser, and wait until the data will be
     // sent to the client.
     outgoingData += delta;
+    console.log(outgoingData);
     if (socket._paused && outgoingData < socket._writableState.highWaterMark)
       return socketOnDrain();
   }
@@ -405,6 +406,7 @@ function connectionListener(socket) {
     if (socket._paused) {
       // onIncoming paused the socket, we should pause the parser as well
       debug('pause parser');
+      parser.unconsume(socket._handle._externalStream);
       socket.parser.pause();
     }
   }

@DamianEdwards may I ask you to give it a try?

[indutny]

Full fix with test is here: #3342

[DamianEdwards]

I just confirmed the issue is not present in 4.1.1

I'll try building from source to test out the fix on our rig here.

[NawarA]

Hey guys, I believe we're seeing this issue in production for 4.2.1. I can share the error if that'd be helpful.

I believe @indutny fixed this, right? If so, why isn't this released? This should be considered a hotfix and get released as a 4.2.2 release ASAP

Just curious on whats going on with this given what we're seeing, given its fixed, and given the fix is unreleased

[jasnell]

@NawarA ... I'm likely going to be taking a look at spinning out a 4.2.2 early next week.

[NawarA]

@jasnell thanks. I had to roll back to iojs to keep production stable. Let me know if I can help with the release

[jasnell]

@NawarA ... smoke testing the v4.x branch would be helpful. If you pull the branch, do a fresh build, does it resolve the issue or are there specific commits we still need to land to address the issue?

[NawarA]

Will do. To confirm the error and ensure its related to this issue, here's what I'm seeing:

_stream_readable.js:480
  dest.on('unpipe', onunpipe);
       ^
TypeError: dest.on is not a function
    at Gzip.Readable.pipe (_stream_readable.js:480:8)
    at ChildProcess.onMessage (/home/admin_augur_io/Augur/LearningServer/routes/recon-master.js:60:10)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at handleMessage (internal/child_process.js:686:10)
    at Pipe.channel.onread (internal/child_process.js:440:11)

I'll pull the latest master branch, and see if it resolves the issue.

[indutny]

@NawarA this does not look related to this issue

[NawarA]

Good to know. I thought because it was as stream where piping was failing, and due to me seeing this issue after upgrading, that the issue was related. Thanks for sharing @indutny

[indutny]

Pipelining, not piping ;)

On Thursday, October 22, 2015, Nawar Alsafar notifications@github.com
wrote:

Good to know. I thought because it was as stream where piping was failing,
and due to me seeing this issue after upgrading, that the issue was
related. Thanks for sharing @indutny https://github.com/indutny

—
Reply to this email directly or view it on GitHub
#3332 (comment).