Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option compressionLevel, forwarded to zlib #85

Merged
merged 2 commits into from
Nov 2, 2024
Merged

add option compressionLevel, forwarded to zlib #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83c81ba
add option compressionLevel, forwarded to zlib
thejoshwolfe Nov 2, 2024
c3a613b
fix default compression method
thejoshwolfe Nov 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ After UTF-8 encoding, `metadataPath` must be at most `0xffff` bytes in length.
mtime: stats.mtime,
mode: stats.mode,
compress: true,
compressionLevel: 6,
forceZip64Format: false,
fileComment: "", // or a UTF-8 Buffer
}
Expand All @@ -77,6 +78,9 @@ yazl does not store group and user ids in the zip file.

If `compress` is `true`, the file data will be deflated (compression method 8).
If `compress` is `false`, the file data will be stored (compression method 0).
If `compressionLevel` is specified, it will be passed to [`zlib`](https://nodejs.org/api/zlib.html#class-options).
Specifying `compressionLevel: 0` is equivalent to `compress: false`.
If both `compress` and `compressionLevel` are given, asserts that they do not conflict, i.e. `!!compress === !!compressionLevel`.

If `forceZip64Format` is `true`, yazl will use ZIP64 format in this entry's Data Descriptor
and Central Directory Record even if not needed (this may be useful for testing.).
Expand Down Expand Up @@ -120,13 +124,14 @@ See `addFile()` for the meaning of the `metadataPath` parameter.
mtime: new Date(),
mode: 0o100664,
compress: true,
compressionLevel: 6,
forceZip64Format: false,
fileComment: "", // or a UTF-8 Buffer
size: 12345, // example value
}
```

See `addFile()` for the meaning of `mtime`, `mode`, `compress`, `forceZip64Format`, and `fileComment`.
See `addFile()` for the meaning of `mtime`, `mode`, `compress`, `compressionLevel`, `forceZip64Format`, and `fileComment`.
If `size` is given, it will be checked against the actual number of bytes in the `readStream`,
and an error will be emitted if there is a mismatch.

Expand Down Expand Up @@ -154,12 +159,13 @@ See `addFile()` for info about the `metadataPath` parameter.
mtime: new Date(),
mode: 0o100664,
compress: true,
compressionLevel: 6,
forceZip64Format: false,
fileComment: "", // or a UTF-8 Buffer
}
```

See `addFile()` for the meaning of `mtime`, `mode`, `compress`, `forceZip64Format`, and `fileComment`.
See `addFile()` for the meaning of `mtime`, `mode`, `compress`, `compressionLevel`, `forceZip64Format`, and `fileComment`.

This method has the unique property that General Purpose Bit `3` will not be used in the Local File Header.
This doesn't matter for unzip implementations that conform to the Zip File Spec.
Expand Down Expand Up @@ -251,10 +257,8 @@ and serving it without buffering the contents on disk or in ram.
`calculatedTotalSize` can become the `Content-Length` header before piping the `outputStream` as the response body.)

If `calculatedTotalSize` is `-1`, it means means the total size is too hard to guess before processing the input file data.
This will happen if and only if the `compress` option is `true` on any call to `addFile()`, `addReadStream()`, `addReadStreamLazy()`, `addBuffer()`, or `addEmptyDirectory()`,
or if `addReadStream()` or `addReadStreamLazy()` is called and the optional `size` option is not given.
In other words, clients should know whether they're going to get a `-1` or a real value
by looking at how they are using this library.
To ensure the final size is known, disable compression (set `compress: false` or `compressionLevel: 0`)
in every call to `addFile()`, `addReadStream()`, `addReadStreamLazy()`, and `addBuffer()`.

The call to `calculatedTotalSizeCallback` might be delayed if yazl is still waiting for `fs.Stats` for an `addFile()` entry.
If `addFile()` was never called, `calculatedTotalSizeCallback` will be called during the call to `end()`.
Expand Down
25 changes: 17 additions & 8 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,10 @@ ZipFile.prototype.addBuffer = function(buffer, metadataPath, options) {
entry.crc32 = crc32.unsigned(buffer);
entry.crcAndFileSizeKnown = true;
self.entries.push(entry);
if (!entry.compress) {
if (entry.compressionLevel === 0) {
setCompressedBuffer(buffer);
} else {
zlib.deflateRaw(buffer, function(err, compressedBuffer) {
zlib.deflateRaw(buffer, {level:1}, function(err, compressedBuffer) {
setCompressedBuffer(compressedBuffer);
});
}
Expand All @@ -121,6 +121,7 @@ ZipFile.prototype.addEmptyDirectory = function(metadataPath, options) {
if (options == null) options = {};
if (options.size != null) throw new Error("options.size not allowed");
if (options.compress != null) throw new Error("options.compress not allowed");
if (options.compressionLevel != null) throw new Error("options.compressionLevel not allowed");

if (shouldIgnoreAdding(self)) return;
var entry = new Entry(metadataPath, true, options);
Expand Down Expand Up @@ -171,7 +172,7 @@ function writeToOutputStream(self, buffer) {
function pumpFileDataReadStream(self, entry, readStream) {
var crc32Watcher = new Crc32Watcher();
var uncompressedSizeCounter = new ByteCounter();
var compressor = entry.compress ? new zlib.DeflateRaw() : new PassThrough();
var compressor = entry.compressionLevel !== 0 ? new zlib.DeflateRaw({level:entry.compressionLevel}) : new PassThrough();
var compressedSizeCounter = new ByteCounter();
readStream.pipe(crc32Watcher)
.pipe(uncompressedSizeCounter)
Expand All @@ -193,6 +194,15 @@ function pumpFileDataReadStream(self, entry, readStream) {
});
}

function determineCompressionLevel(options) {
if (options.compress != null && options.compressionLevel != null) {
if (!!options.compress !== !!options.compressionLevel) throw new Error("conflicting settings for compress and compressionLevel");
}
if (options.compressionLevel != null) return options.compressionLevel;
if (options.compress === false) return 0;
return 6;
}

function pumpEntries(self) {
if (self.allDone || self.errored) return;
// first check if calculatedTotalSize is finally known
Expand Down Expand Up @@ -245,7 +255,7 @@ function calculateTotalSize(self) {
for (var i = 0; i < self.entries.length; i++) {
var entry = self.entries[i];
// compression is too hard to predict
if (entry.compress) return -1;
if (entry.compressionLevel !== 0) return -1;
if (entry.state >= Entry.READY_TO_PUMP_FILE_DATA) {
// if addReadStream was called without providing the size, we can't predict the total size
if (entry.uncompressedSize == null) return -1;
Expand Down Expand Up @@ -436,10 +446,9 @@ function Entry(metadataPath, isDirectory, options) {
if (options.size != null) this.uncompressedSize = options.size;
}
if (isDirectory) {
this.compress = false;
this.compressionLevel = 0;
} else {
this.compress = true; // default
if (options.compress != null) this.compress = !!options.compress;
this.compressionLevel = determineCompressionLevel(options);
}
this.forceZip64Format = !!options.forceZip64Format;
if (options.fileComment) {
Expand Down Expand Up @@ -650,7 +659,7 @@ Entry.prototype.getCentralDirectoryRecord = function() {
Entry.prototype.getCompressionMethod = function() {
var NO_COMPRESSION = 0;
var DEFLATE_COMPRESSION = 8;
return this.compress ? DEFLATE_COMPRESSION : NO_COMPRESSION;
return this.compressionLevel === 0 ? NO_COMPRESSION : DEFLATE_COMPRESSION;
};

function dateToDosDateTime(jsDate) {
Expand Down
48 changes: 48 additions & 0 deletions test/test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ var BufferList = require("./bl-minimal.js");
// * extracting the zip file (via yauzl) gives the correct contents.
// * compress: false
// * specifying mode and mtime options, but not checking them.
// * verifying compression method defaults to true.
(function() {
var fileMetadata = {
mtime: new Date(),
Expand All @@ -28,6 +29,8 @@ var BufferList = require("./bl-minimal.js");
yauzl.fromBuffer(data, function(err, zipfile) {
if (err) throw err;
zipfile.on("entry", function(entry) {
var expectedCompressionMethod = entry.fileName === "without-compression.txt" ? 0 : 8;
if (entry.compressionMethod !== expectedCompressionMethod) throw new Error("expected " + entry.fileName + " compression method " + expectedCompressionMethod + ". found: " + entry.compressionMethod);
zipfile.openReadStream(entry, function(err, readStream) {
if (err) throw err;
readStream.pipe(new BufferList(function(err, data) {
Expand All @@ -42,6 +45,48 @@ var BufferList = require("./bl-minimal.js");
});
})();

// Test:
// * specifying compressionLevel varies the output size.
// * specifying compressionLevel:0 disables compression.
(function() {
var options = {
mtime: new Date(),
mode: 0o100664,
};
var zipfile = new yazl.ZipFile();
options.compressionLevel = 1;
zipfile.addFile(__filename, "level1.txt", options);
options.compressionLevel = 9;
zipfile.addFile(__filename, "level9.txt", options);
options.compressionLevel = 0;
zipfile.addFile(__filename, "level0.txt", options);
zipfile.end(function(calculatedTotalSize) {
if (calculatedTotalSize !== -1) throw new Error("calculatedTotalSize is impossible to know before compression");
zipfile.outputStream.pipe(new BufferList(function(err, data) {
if (err) throw err;
yauzl.fromBuffer(data, function(err, zipfile) {
if (err) throw err;

var fileNameToSize = {};
zipfile.on("entry", function(entry) {
fileNameToSize[entry.fileName] = entry.compressedSize;
var expectedCompressionMethod = entry.fileName === "level0.txt" ? 0 : 8;
if (entry.compressionMethod !== expectedCompressionMethod) throw new Error("expected " + entry.fileName + " compression method " + expectedCompressionMethod + ". found: " + entry.compressionMethod);
});
zipfile.on("end", function() {
var size0 = fileNameToSize["level0.txt"];
var size1 = fileNameToSize["level1.txt"];
var size9 = fileNameToSize["level9.txt"];
// Note: undefined coerces to NaN which always results in the comparison evaluating to `false`.
if (!(size0 >= size1)) throw new Error("Compression level 1 inflated size. expected: " + size0 + " >= " + size1);
if (!(size1 >= size9)) throw new Error("Compression level 9 inflated size. expected: " + size1 + " >= " + size9);
console.log("compressionLevel (" + size0 + " >= " + size1 + " >= " + size9 + "): pass");
});
});
}));
});
})();

// Test:
// * forceZip64Format for various subsets of entries.
// * specifying size for addReadStream.
Expand Down Expand Up @@ -118,6 +163,7 @@ var BufferList = require("./bl-minimal.js");
// Test:
// * just calling addBuffer() and no other add functions.
// * calculatedTotalSize should be known and correct for addBuffer with compress:false.
// * addBuffer with compress:false disables compression.
(function() {
var zipfile = new yazl.ZipFile();
zipfile.addBuffer(bufferFrom("hello"), "hello.txt", {compress: false});
Expand All @@ -134,6 +180,8 @@ var BufferList = require("./bl-minimal.js");
if (entry.fileName !== expectedName) {
throw new Error("unexpected entry fileName: " + entry.fileName + ", expected: " + expectedName);
}
var expectedCompressionMethod = 0;
if (entry.compressionMethod !== expectedCompressionMethod) throw new Error("expected " + entry.fileName + " compression method " + expectedCompressionMethod + ". found: " + entry.compressionMethod);
});
zipfile.on("end", function() {
if (entryNames.length === 0) console.log("justAddBuffer: pass");
Expand Down