HPE_UNEXPECTED_CONTENT_LENGTH

[gajus]

My logger includes hundreds of errors such as this:

{
  "code": "HPE_UNEXPECTED_CONTENT_LENGTH",
  "host": "[..]",
  "hostname": "[..]",
  "message": "Parse Error",
  "method": "GET",
  "name": "RequestError",
  "path": "[..]",
  "protocol": "https:",
  "stack": "RequestError: Parse Error\n    at ClientRequest.request.once.error (/srv/[..]/node_modules/@applaudience/cinema-data-scraper-http-client/node_modules/got/source/request-as-event-emitter.js:173:14)\n    at Object.onceWrapper (events.js:273:13)\n    at ClientRequest.emit (events.js:187:15)\n    at ClientRequest.origin.emit.args [as emit] (/srv/[..]/node_modules/@szmarczak/http-timer/source/index.js:36:11)\n    at TLSSocket.socketOnData (_http_client.js:438:9)\n    at TLSSocket.emit (events.js:182:13)\n    at addChunk (_stream_readable.js:283:12)\n    at readableAddChunk (_stream_readable.js:264:11)\n    at TLSSocket.Readable.push (_stream_readable.js:219:10)\n    at TLSWrap.onread (net.js:639:20)",
  "url": "https://[..]"
}

Stack trace refers to Got as the source of the issue:

RequestError: Parse Error
    at ClientRequest.request.once.error (/srv/[..]/node_modules/got/source/request-as-event-emitter.js:173:14)
    at Object.onceWrapper (events.js:273:13)
    at ClientRequest.emit (events.js:187:15)
    at ClientRequest.origin.emit.args [as emit] (/srv/[..]/node_modules/@szmarczak/http-timer/source/index.js:36:11)
    at TLSSocket.socketOnData (_http_client.js:438:9)
    at TLSSocket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
    at readableAddChunk (_stream_readable.js:264:11)
    at TLSSocket.Readable.push (_stream_readable.js:219:10)
    at TLSWrap.onread (net.js:639:20)

Is anyone familiar with error/ know whats the workaround?

[gajus]

Seems like I have already had a comparable issue with "request" module:

• Getting HPE_UNEXPECTED_CONTENT_LENGTH trying to proxy HTTPS request request/request#2091

A post in the same thread (request/request#2091 (comment)) suggests that the problem is due to server response including transfer-encoding: chunked and content-length headers, and that the solution (at a request initiator side) is to tolerate this behaviour. This behaviour can be tolerated using a custom HTTP parser:

process.binding('http_parser').HTTPParser = require('http-parser-js').HTTPParser;

I doubt if Got can do much about this issue.

Would be nice to have a test case at least so that we are aware of the issue.

[szmarczak]

According to https://github.com/nodejs/node/blob/8b4af64f50c5e41ce0155716f294c24ccdecad03/deps/http_parser/http_parser.c

The cause may be:

1. Double content-length header
2. Both headers present: chunked encoding & content length

[gajus]

I can confirm that the subject response indeed includes chunked encoding & content length headers.

-< HTTP/1.1 302 Found
-< cache-control: private
+< transfer-encoding: chunked
-< content-type: text/html; charset=utf-8
-< location: /[..]
-* Server Microsoft-IIS/8.5 is not blacklisted
-< server: Microsoft-IIS/8.5
-< x-aspnet-version: 4.0.30319
-* Added cookie AspxAutoDetectCookieSupport="1" for domain [..], path /, expire 0
-< set-cookie: AspxAutoDetectCookieSupport=1; path=/
-< x-powered-by: ASP.NET
-< date: Sat, 27 Oct 2018 10:54:51 GMT
-< connection: close
-< x-raygun-version: 4.17.1
-< x-raygun-request-id: 01CTTJK61183BHG2NN2WJXJA79
+< content-length: 228
-< x-rayman-checksum-sha256-hex: 110f5c9149e6c2fd977182943ab2ea5a16a4e913fff4d7539e17539dfaabbc31
-< x-rayman-cache-hit: false
-< x-rayman-forward-proxy: [..]

[szmarczak]

Unfortunately, there's nothing Got can do about that :/

[gajus]

It could use the custom parser, though thats probably excessive.

Either way, nice to have it documented in the issue thread.

The solution is to change the HTTP parser globally.

process.binding('http_parser').HTTPParser = require('http-parser-js').HTTPParser;

[gajus]

I can confirm that changing the parser fixes the issue.