Parse Error: HPE_INVALID_CONSTANT on Transfer-Encoding response to HEAD request

[jeremyruppel]

I'm working with a service that sends back a "Transfer-Encoding: chunked" header in response to a HEAD request, which produces the following error:

{ [Error: Parse Error] bytesParsed: 110, code: 'HPE_INVALID_CONSTANT' }

I can reproduce this behavior with the following client-server setup:

// server.js
var http = require('http');

var server = http.createServer(function(req, res) {
  console.log(req.method, req.url);
  res.statusCode = 404;
  res.setHeader('transfer-encoding', 'chunked');
  res.end();
});

server.listen(3000, function() {
  console.log('server listening on port', 3000);
});

// client.js
var http = require('http');

var req = http.request({
  // method: 'GET',
  method: 'HEAD',
  host: 'localhost',
  port: 3000,
  path: '/whatever.txt',
  headers: {
    'Connection': 'close'
  }
}, function(res) {
  console.log('status:', res.statusCode);
  console.dir(res.headers);
  res.on('data', function(data) {
    console.log('data:', data);
  });
  res.on('end', function() {
    console.log('end');
  });
});
req.on('error', function(err) {
  console.warn('ERROR:', err);
});
req.end();

Changing the method to GET does not produce the error, so my expectation is that HEAD wouldn't either. I realize it's pointless for the Transfer-Encoding header to be present in a response without a body, but the RFC says that a response to a HEAD request should contain all of the headers that the GET would:

9.4 HEAD

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.

My node versions:

$ node -e 'console.dir(process.versions)'
{ http_parser: '1.0',
  node: '0.10.28',
  v8: '3.14.5.9',
  ares: '1.9.0-DEV',
  uv: '0.10.27',
  zlib: '1.2.3',
  modules: '11',
  openssl: '1.0.1g' }

[indutny]

Here is a section 3.3:

 The presence of a message body in a response depends on both the
   request method to which it is responding and the response status code
   (Section 3.1.2).  Responses to the HEAD request method (Section 4.3.2
   of [RFC7231]) never include a message body because the associated
   response header fields (e.g., Transfer-Encoding, Content-Length,
   etc.), if present, indicate only what their values would have been if
   the request method had been GET (Section 4.3.1 of [RFC7231]).

Going to fix it in a bit.

[indutny]

Should be fixed by #8365

[jeremyruppel]

@indutny thanks for the quick patch! But I'm sorry I wasn't clear: my issue is that the client produces the Parse Error when it gets the TE header back in the response to the HEAD request. Maybe I'm in the wrong place – does this sound like an issue with https://github.com/joyent/http-parser instead?

[indutny]

I think this patch should fix this problem.

[tjfontaine]

fixed in 1fddc1f

[delfuego]

@jeremyruppel Did you ever resolve this issue? I think I'm seeing the same thing — on the client side, seeing that HPE_INVALID_CONSTANT error be thrown specifically when consuming a chunked response from a specific server. (And it's worse, it's a load-balanced API endpoint and only some of their servers are causing the issue, and it's HTTPS-protected so it's a devil of a time actually dumping the wire content to see what's going on, etc.)

[jishi]

Was this actually ever fixed, or is this reintroduced? I'm still getting [Error: Parse Error] bytesParsed: 139, code: 'HPE_INVALID_CONSTANT when invoking a HEAD request against a server that responds with chunked transfer-encoding, in version 4.2.6. Here is a wireshark dump of the traffic:

HEAD /xml/device_description.xml HTTP/1.1
Host: 192.168.1.150:1400
Connection: close

HTTP/1.1 200 OK
Content-Type: text/xml
Transfer-Encoding: chunked
Server: Linux UPnP/1.0 Sonos/31.8-24090 (BR200)
Connection: close

0

Of course, one could argue the usefulness of a chunked transfer for a HEAD requests, but a regular browser handles this fine (Testing with Chrome and Postman to simulate the HEAD request).

[bnoordhuis]

@jishi I think http_parser has always considered that an error.

With Connection: close, the remainder can theoretically be ignored, although the response would technically still be invalid. Without Connection: close, it should always be an error because it's impossible otherwise to determine where the next response starts.

That's without going into whether it's actually desirable to support broken behavior.

[jishi]

@bnoordhuis Hm, I don't see what Connection: close has to do with this. It should be supported on both chunked and predetermined body length, it would only control what is supposed to happen after all content has been received.

After some research, I'm realize that it might be a protocol violation, although there seems to be some discrepancies on current implementations. Allegedly even Google servers did the same thing a while back. I guess the question is whether it should maximize compatibility or enforce protocols.

[jishi]

Clarification: The violation is in the empty chunk that is sent as part of the message-body. Not the combination of headers.

[bnoordhuis]

Hm, I don't see what Connection: close has to do with this. It should be supported on both chunked and predetermined body length, it would only control what is supposed to happen after all content has been received.

Connection: close means the current response is the last one that's going to be sent over the current connection; any trailing garbage can be ignored, it won't be the start of a new response.

I guess the question is whether it should maximize compatibility or enforce protocols.

The problem with the "be liberal in what you accept" philosophy is that it's a frequent source of bugs and security issues later on, even when the initial compatibility hack looked benign enough.

Point in case: next week's security release was prompted by being too lenient in a few places.

[jishi]

Alright, I'm with you now on the close debate. and fair enough, I get your point. This is no deal breaker for me either. I should probably reach out to Sonos and point out this protocol violation anyway.