Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add addReadStreamLazy() #80

Merged
merged 4 commits into from
Oct 19, 2024
Merged

add addReadStreamLazy() #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5f406e9
addReadStreamLazy
thejoshwolfe Oct 19, 2024
1efa513
re-implement what we need from bl
thejoshwolfe Oct 19, 2024
e649524
comments
thejoshwolfe Oct 19, 2024
331dc37
test for addReadStreamLazy callback error
thejoshwolfe Oct 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ zipfile.outputStream.pipe(fs.createWriteStream("output.zip")).on("close", functi
console.log("done");
});
// alternate apis for adding files:
zipfile.addReadStream(process.stdin, "stdin.txt");
zipfile.addBuffer(Buffer.from("hello"), "hello.txt");
zipfile.addReadStreamLazy("stdin.txt", cb => cb(null, process.stdin));
// call end() after all the files have been added
zipfile.end();
```
Expand Down Expand Up @@ -95,7 +95,24 @@ the number of simultaneous open files is `O(1)`, probably just 1 at a time.
#### addReadStream(readStream, metadataPath, [options])

Adds a file to the zip file whose content is read from `readStream`.
See `addFile()` for info about the `metadataPath` parameter.
This method is effectively implemented as `this.addReadStreamLazy(metadataPath, options, cb => cb(null, readStream))`.

In general, it is recommended to use `addReadStreamLazy` instead of this method
to avoid holding a large number of system resources open for a long time.
This method is provided for backward compatibility,
and for convenience in cases where the `readStream` doesn't require meaningful resources to hold open and waiting.

#### addReadStreamLazy(metadataPath[, options], getReadStreamFunction)

Adds a file to the zip file whose content is read from a read stream obtained by calling `getReadStreamFunction(cb)`.
`getReadStreamFunction(cb)` is called with a single callback function.
Your implementation of `getReadStreamFunction` should eventually call `cb(err, readStream)`
and give the `readStream` that provides the contents of the file to add to the zip file.
If `err` is given (if it is truthy), it will be emitted from this `ZipFile` object.
The return value from `cb` is unspecified.

See `addFile()` for the meaning of the `metadataPath` parameter.
`typeof getReadStreamFunction` must be `'function'`, which is used to determine when `options` has been omitted.
`options` may be omitted or null and has the following structure and default values:

```js
Expand All @@ -116,6 +133,15 @@ and an error will be emitted if there is a mismatch.
Note that yazl will `.pipe()` data from `readStream`, so be careful using `.on('data')`.
In certain versions of node, `.on('data')` makes `.pipe()` behave incorrectly.

Here's an example call to this method to illustrate the function callbacks:

```js
zipfile.addReadStreamLazy("path/in/archive.txt", function(cb) {
var readStream = getTheReadStreamSomehow();
cb(readStream);
});
```

#### addBuffer(buffer, metadataPath, [options])

Adds a file to the zip file whose content is `buffer`.
Expand All @@ -141,7 +167,7 @@ However, 7-Zip 9.20 has a known bug where General Purpose Bit `3` is declared an
(note that it really has nothing to do with the compression method.).
See [issue #11](https://github.com/thejoshwolfe/yazl/issues/11).
If you would like to create zip files that 7-Zip 9.20 can understand,
you must use `addBuffer()` instead of `addFile()` or `addReadStream()` for all entries in the zip file
you must use `addBuffer()` instead of `addFile()`, `addReadStream()`, or `addReadStreamLazy()` for all entries in the zip file
(and `addEmptyDirectory()` is fine too).

Note that even when yazl provides the file sizes in the Local File Header,
Expand Down Expand Up @@ -184,7 +210,7 @@ See `addFile()` for the meaning of `mtime` and `mode`.

#### end([options], [calculatedTotalSizeCallback])

Indicates that no more files will be added via `addFile()`, `addReadStream()`, or `addBuffer()`,
Indicates that no more files will be added via `addFile()`, `addReadStream()`, `addReadStreamLazy()`, `addBuffer()`, or `addEmptyDirectory()`,
and causes the eventual close of `outputStream`.

`options` may be omitted or null and has the following structure and default values:
Expand Down Expand Up @@ -225,8 +251,8 @@ and serving it without buffering the contents on disk or in ram.
`calculatedTotalSize` can become the `Content-Length` header before piping the `outputStream` as the response body.)

If `calculatedTotalSize` is `-1`, it means means the total size is too hard to guess before processing the input file data.
This will happen if and only if the `compress` option is `true` on any call to `addFile()`, `addReadStream()`, `addBuffer()`, or `addEmptyDirectory()`,
or if `addReadStream()` is called and the optional `size` option is not given.
This will happen if and only if the `compress` option is `true` on any call to `addFile()`, `addReadStream()`, `addReadStreamLazy()`, `addBuffer()`, or `addEmptyDirectory()`,
or if `addReadStream()` or `addReadStreamLazy()` is called and the optional `size` option is not given.
In other words, clients should know whether they're going to get a `-1` or a real value
by looking at how they are using this library.

Expand All @@ -243,7 +269,7 @@ It is typical to pipe this stream to a writable stream created from `fs.createWr
Internally, large amounts of file data are piped to `outputStream` using `pipe()`,
which means throttling happens appropriately when this stream is piped to a slow destination.

Data becomes available in this stream soon after calling one of `addFile()`, `addReadStream()`, `addBuffer()`, or `addEmptyDirectory()`.
Data becomes available in this stream soon after calling one of `addFile()`, `addReadStream()`, `addReadStreamLazy()`, `addBuffer()`, or `addEmptyDirectory()`.
Clients can call `pipe()` on this stream at any time,
such as immediately after getting a new `ZipFile` instance, or long after calling `end()`.

Expand Down
40 changes: 36 additions & 4 deletions index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ var PassThrough = require("stream").PassThrough;
var zlib = require("zlib");
var util = require("util");
var EventEmitter = require("events").EventEmitter;
var errorMonitor = require("events").errorMonitor;
var crc32 = require("buffer-crc32");

exports.ZipFile = ZipFile;
Expand All @@ -17,13 +18,18 @@ function ZipFile() {
this.ended = false; // .end() sets this
this.allDone = false; // set when we've written the last bytes
this.forceZip64Eocd = false; // configurable in .end()
this.errored = false;
this.on(errorMonitor, function() {
this.errored = true;
});
}

ZipFile.prototype.addFile = function(realPath, metadataPath, options) {
var self = this;
metadataPath = validateMetadataPath(metadataPath, false);
if (options == null) options = {};

if (shouldIgnoreAdding(self)) return;
var entry = new Entry(metadataPath, false, options);
self.entries.push(entry);
fs.stat(realPath, function(err, stats) {
Expand All @@ -45,14 +51,29 @@ ZipFile.prototype.addFile = function(realPath, metadataPath, options) {
};

ZipFile.prototype.addReadStream = function(readStream, metadataPath, options) {
this.addReadStreamLazy(metadataPath, options, function(cb) {
cb(null, readStream);
});
};

ZipFile.prototype.addReadStreamLazy = function(metadataPath, options, getReadStreamFunction) {
var self = this;
metadataPath = validateMetadataPath(metadataPath, false);
if (typeof options === "function") {
getReadStreamFunction = options;
options = null;
}
if (options == null) options = {};
metadataPath = validateMetadataPath(metadataPath, false);

if (shouldIgnoreAdding(self)) return;
var entry = new Entry(metadataPath, false, options);
self.entries.push(entry);
entry.setFileDataPumpFunction(function() {
entry.state = Entry.FILE_DATA_IN_PROGRESS;
pumpFileDataReadStream(self, entry, readStream);
getReadStreamFunction(function(err, readStream) {
if (err) return self.emit("error", err);
entry.state = Entry.FILE_DATA_IN_PROGRESS;
pumpFileDataReadStream(self, entry, readStream);
});
});
pumpEntries(self);
};
Expand All @@ -63,6 +84,8 @@ ZipFile.prototype.addBuffer = function(buffer, metadataPath, options) {
if (buffer.length > 0x3fffffff) throw new Error("buffer too large: " + buffer.length + " > " + 0x3fffffff);
if (options == null) options = {};
if (options.size != null) throw new Error("options.size not allowed");

if (shouldIgnoreAdding(self)) return;
var entry = new Entry(metadataPath, false, options);
entry.uncompressedSize = buffer.length;
entry.crc32 = crc32.unsigned(buffer);
Expand Down Expand Up @@ -98,6 +121,8 @@ ZipFile.prototype.addEmptyDirectory = function(metadataPath, options) {
if (options == null) options = {};
if (options.size != null) throw new Error("options.size not allowed");
if (options.compress != null) throw new Error("options.compress not allowed");

if (shouldIgnoreAdding(self)) return;
var entry = new Entry(metadataPath, true, options);
self.entries.push(entry);
entry.setFileDataPumpFunction(function() {
Expand All @@ -118,6 +143,7 @@ ZipFile.prototype.end = function(options, calculatedTotalSizeCallback) {
if (options == null) options = {};
if (this.ended) return;
this.ended = true;
if (this.errored) return;
this.calculatedTotalSizeCallback = calculatedTotalSizeCallback;
this.forceZip64Eocd = !!options.forceZip64Format;
if (options.comment) {
Expand Down Expand Up @@ -168,7 +194,7 @@ function pumpFileDataReadStream(self, entry, readStream) {
}

function pumpEntries(self) {
if (self.allDone) return;
if (self.allDone || self.errored) return;
// first check if calculatedTotalSize is finally known
if (self.ended && self.calculatedTotalSizeCallback != null) {
var calculatedTotalSize = calculateTotalSize(self);
Expand Down Expand Up @@ -260,6 +286,12 @@ function calculateTotalSize(self) {
return pretendOutputCursor + centralDirectorySize + endOfCentralDirectorySize;
}

function shouldIgnoreAdding(self) {
if (self.ended) throw new Error("cannot add entries after calling end()");
if (self.errored) return true;
return false;
}

var ZIP64_END_OF_CENTRAL_DIRECTORY_RECORD_SIZE = 56;
var ZIP64_END_OF_CENTRAL_DIRECTORY_LOCATOR_SIZE = 20;
var END_OF_CENTRAL_DIRECTORY_RECORD_SIZE = 22;
Expand Down
Loading