through2-batch

A stream that transforms chunks to batches form the stream.

A way to use a Node.JS Transform stream that batches chunks into an array (default is 10). Objects/chunks will still come in the same order, just in batched arrays.

Built using through2 and has the same API with the addition of a batchSize option.

Non-objectMode streams are supported for completeness.

Written by Nima Gardideh (halfmoon.ws) and used in production by Taplytics.

Install

npm install --save through2-batch

Examples

Process rows from a CSV in batches.

var through2Batch = require('through2-batch');

fs.createReadStream('data.csv')
  .pipe(csv2())
  .pipe(through2Batch.obj(
    {batchSize: 100},
    function (batch, enc, callback) {
      var self = this;
      console.log(batch.length); // 100
      someThingAsync(batch, function (newChunk) {
        self.push(newChunk);
      });
  }));

Don't specify a transform fn, and let the batches be processed by another stream

var through2Batch = require('through2-batch');

fs.createReadStream('data.csv')
  .pipe(csv2())
  .pipe(through2Batch.obj())
  .pipe(getSomeOtherStreamProcessingBatches());

Contributing

Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @ngardideh.