zlib: decompression throw on truncated input #2595

jhamhader · 2015-08-28T11:58:57Z

Check for unexpected end-of-stream error when decompressing.
If the output buffer still has space after decompressing and deflate
returned Z_OK or Z_BUF_ERROR - that means unexpected end-of-stream.
Added test-zlib-truncated.js for the case of truncated input.
Fixed the zlib dictionary test to not end the inflate stream on a
truncated output (no crc) of deflate
See issue #2043

Fishrock123 · 2015-08-28T13:05:24Z

CI run: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/200/

Fishrock123 · 2015-08-28T13:06:12Z

test/parallel/test-zlib-truncated.js

+
+var common = require('../common');
+var assert = require('assert');
+var zlib = require ('zlib');


Please use const for values that won't be changed that you either added or modified in this commit. :)

Thanks for the input :)
Should I use 'let' for the scope-specific variables or is 'var' ok?

jhamhader · 2015-08-28T17:19:01Z

Committed 'let' and 'const' for 'test-zlib-dictionary.js' and 'test-zlib-truncated.js'

indutny · 2015-09-24T17:04:46Z

cc @nodejs/collaborators

trevnorris · 2015-09-24T17:24:15Z

Is this done elsewhere (as in, in other programs), and/or is it what the user would expect to happen?

jhamhader · 2015-09-27T16:12:23Z

gnu gunzip: fails with unexpected end of file
python2 gzip: fails with IOError: CRC check failed 0xca600aac != 0x6fe58c67L
python3 gzip: fails with EOFError: Compressed file ended before the end-of-stream marker was reached
ruby gzip: fails with Zlib::GzipFile::Error: unexpected end of file
perl gzip: fails with unexpected end of file at (eval 15)[/usr/share/perl5/core_perl/perl5db.pl:737] line 2.

trevnorris · 2015-09-27T17:09:30Z

Seems at the minimum we should be propagation an error. So will it throw for sync and emit error on async? (not familiar w/ much of the zlib api)

jhamhader · 2015-09-27T20:34:14Z

Throw for sync.
What do you mean by emit? The callback of the async function will be called with err on the first argument.

Check for unexpected end-of-file error when decompressing. If the output buffer still has space after decompressing and deflate returned Z_OK or Z_BUF_ERROR - that means unexpected end-of-file. Added test-zlib-truncated.js for the case of truncated input. Fixed the zlib dictionary test to not end the inflate stream on a truncated output (no crc) of deflate

trevnorris · 2015-09-28T17:59:26Z

The callback of the async function will be called with err on the first argument.

Nm me. Was thinking about using zlib w/ the streams API and how a pipe would propagate the error.

jhamhader · 2015-09-28T18:01:56Z

Sure :)
Anything else missing for the PR?

trevnorris · 2015-09-28T18:03:13Z

Not from me. Would like @indutny to sign off on this as well.

thefourtheye · 2015-09-29T01:54:54Z

I would prefer @chrisdickinson to sign off as well.

jhamhader · 2015-10-08T18:30:55Z

@indutny @chrisdickinson could you please take a look and confirm?
Thanks :)

trevnorris · 2015-10-08T21:54:58Z

@jhamhader ping me tomorrow if @indutny hasn't responded.

jhamhader · 2015-10-10T12:15:09Z

Pinging

trevnorris · 2015-10-10T18:11:10Z

Thanks. Let's go ahead and land this. I'll be back at my computer on Monday.

Also, I'm initially inclined to say this is semver-major. Even if it's not, don't know if this is something LTS would accept. We can check after it's in master. :-)

Fishrock123 · 2015-10-19T14:30:55Z

ping @trevnorris

trevnorris · 2015-10-19T21:07:03Z

Landed on 0fc0902. Thanks much!

Check for unexpected end-of-file error when decompressing. If the output buffer still has space after decompressing and deflate returned Z_OK or Z_BUF_ERROR - that means unexpected end-of-file. Added test-zlib-truncated.js for the case of truncated input. Fixed the zlib dictionary test to not end the inflate stream on a truncated output (no crc) of deflate Fixes: #2043 PR-URL: #2595 Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) nodejs#3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) nodejs#3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) nodejs#3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) nodejs#3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) nodejs#3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) nodejs#2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) nodejs#2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) nodejs#3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) nodejs#3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) nodejs#3407. * tls: - Added ALPN Support (Shigeki Ohtsu) nodejs#2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) nodejs#2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) nodejs#1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) nodejs#3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) nodejs#3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) nodejs#2595. PR-URL: nodejs#3466

Notable changes: * buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer, these have been deprecated for a long time (Sakthipriyan Vairamani) #2859. * console: (Breaking) Values reported by console.time() now have 3 decimals of accuracy added (Michaël Zasso) #3166. * fs: - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file descriptor as their first argument (Johannes Wüller) #3163. - (Breaking) In fs.readFile(), if an encoding is specified and the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3485. - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding, callback) form), if the internal toString() fails the error is no longer thrown but is passed to the callback (Evan Lucas) #3503. * http: - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342. - (Breaking) When parsing HTTP, don't add duplicates of the following headers: Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition to the following headers which already block duplicates: Content-Type, Content-Length, User-Agent, Referer, Host, Authorization, Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location, Max-Forwards (James M Snell) #3090. - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a function or a TypeError is thrown (James M Snell) #3090. - (Breaking) HTTP methods and header names must now conform to the RFC 2616 "token" rule, a list of allowed characters that excludes control characters and a number of separator characters. Specifically, methods and header names must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown (James M Snell) #2526. * node: - (Breaking) Deprecated the _linklist module (Rich Trott) #3078. - (Breaking) Removed require.paths and require.registerExtension(), both had been previously set to throw Error when accessed (Sakthipriyan Vairamani) #2922. * npm: Upgraded to version 3.3.6 from 2.14.7, see https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a major version bump for npm and it has seen a significant amount of change. Please see the original npm v3.0.0 release notes for a list of major changes (Rebecca Turner) #3310. * src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary due to the V8 upgrade. Native add-ons will need to be recompiled (Rod Vagg) #3400. * timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes a long-standing known issue where unrefed timers would perviously hold beforeExit open (Fedor Indutny) #3407. * tls: - Added ALPN Support (Shigeki Ohtsu) #2564. - TLS options can now be passed in an object to createSecurePair() (Коренберг Марк) #2441. - (Breaking) The default minimum DH key size for tls.connect() is now 1024 bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS option can be used to override the default. (Shigeki Ohtsu) #1831. * util: - (Breaking) util.p() was deprecated for years, and has now been removed (Wyatt Preul) #3432. - (Breaking) util.inherits() can now work with ES6 classes. This is considered a breaking change because of potential subtle side-effects caused by a change from directly reassigning the prototype of the constructor using `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })` to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)` (Michaël Zasso) #3455. * v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351. - Implements the spread operator, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator for further information. - Implements new.target, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target for further information. * zlib: Decompression now throws on truncated input (e.g. unexpected end of file) (Yuval Brik) #2595. PR-URL: #3466

targos added the zlib Issues and PRs related to the zlib subsystem. label Aug 28, 2015

Fishrock123 reviewed Aug 28, 2015
View reviewed changes

rvagg force-pushed the master branch from 11c25c2 to ba02bd0 Compare September 6, 2015 11:55

jhamhader force-pushed the zlib-truncated branch 2 times, most recently from 30b47df to 495039d Compare September 19, 2015 12:16

jhamhader force-pushed the zlib-truncated branch from 495039d to 5c796aa Compare September 27, 2015 16:41

jhamhader force-pushed the zlib-truncated branch from 5c796aa to aae310e Compare September 28, 2015 08:21

Fishrock123 added the semver-major PRs that contain breaking changes and should be released in the next major version. label Oct 10, 2015

Fishrock123 added this to the 5.0.0 milestone Oct 10, 2015

rvagg mentioned this pull request Oct 16, 2015

v5.0.0 Planning & Checklist #3397

Closed

13 tasks

trevnorris closed this Oct 19, 2015

rvagg mentioned this pull request Oct 21, 2015

Release proposal: v5.0.0 #3466

Merged

This was referenced Nov 25, 2015

Is there a proper way to send zlib partial files for unzipping? #4030

Closed

Error reading some .nii.gz files bids-standard/legacy-validator#52

Closed

jhamhader mentioned this pull request Mar 17, 2016

Gzip decoder too strict #5761

Closed

tomhughes mentioned this pull request Mar 31, 2016

Fix "incomplete" test fixture to be valid gzip of incomplete data mapbox/tilelive#167

Merged

conradoplg mentioned this pull request Oct 25, 2016

Zlib._handle.onerror unexpected end of file ttezel/twit#300

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zlib: decompression throw on truncated input #2595

zlib: decompression throw on truncated input #2595

jhamhader commented Aug 28, 2015

Fishrock123 commented Aug 28, 2015

Fishrock123 Aug 28, 2015

jhamhader Aug 28, 2015

jhamhader commented Aug 28, 2015

indutny commented Sep 24, 2015

trevnorris commented Sep 24, 2015

jhamhader commented Sep 27, 2015

trevnorris commented Sep 27, 2015

jhamhader commented Sep 27, 2015

trevnorris commented Sep 28, 2015

jhamhader commented Sep 28, 2015

trevnorris commented Sep 28, 2015

thefourtheye commented Sep 29, 2015

jhamhader commented Oct 8, 2015

trevnorris commented Oct 8, 2015

jhamhader commented Oct 10, 2015

trevnorris commented Oct 10, 2015

Fishrock123 commented Oct 19, 2015

trevnorris commented Oct 19, 2015

zlib: decompression throw on truncated input #2595

zlib: decompression throw on truncated input #2595

Conversation

jhamhader commented Aug 28, 2015

Fishrock123 commented Aug 28, 2015

Fishrock123 Aug 28, 2015

Choose a reason for hiding this comment

jhamhader Aug 28, 2015

Choose a reason for hiding this comment

jhamhader commented Aug 28, 2015

indutny commented Sep 24, 2015

trevnorris commented Sep 24, 2015

jhamhader commented Sep 27, 2015

trevnorris commented Sep 27, 2015

jhamhader commented Sep 27, 2015

trevnorris commented Sep 28, 2015

jhamhader commented Sep 28, 2015

trevnorris commented Sep 28, 2015

thefourtheye commented Sep 29, 2015

jhamhader commented Oct 8, 2015

trevnorris commented Oct 8, 2015

jhamhader commented Oct 10, 2015

trevnorris commented Oct 10, 2015

Fishrock123 commented Oct 19, 2015

trevnorris commented Oct 19, 2015