Skip to content

Commit

Permalink
doc: add documentation for brotli support
Browse files Browse the repository at this point in the history 
Backport-PR-URL: #27681
PR-URL: #24938
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Reviewed-By: Ali Ijaz Sheikh <ofrobots@google.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
  • Loading branch information
@addaleax @MylesBorins
addaleax authored and MylesBorins committed May 16, 2019
1 parent c670358 commit 93edf90
Show file tree
Hide file tree
Showing 3 changed files with 211 additions and 13 deletions.
10 changes: 5 additions & 5 deletions doc/api/errors.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -624,16 +624,16 @@ An attempt was made to register something that is not a function as an
The type of an asynchronous resource was invalid. Note that users are also able
to define their own types if using the public embedder API.

<a id="ERR_BROTLI_COMPRESSION_FAILED"></a>
### ERR_BROTLI_COMPRESSION_FAILED

Data passed to a Brotli stream was not successfully compressed.

<a id="ERR_BROTLI_INVALID_PARAM"></a>
### ERR_BROTLI_INVALID_PARAM

An invalid parameter key was passed during construction of a Brotli stream.

<a id="ERR_BROTLI_COMPRESSION_FAILED">
### ERR_BROTLI_COMPRESSION_FAILED

Data passed to a Brotli stream was not successfully compressed.

<a id="ERR_BUFFER_OUT_OF_BOUNDS"></a>
### ERR_BUFFER_OUT_OF_BOUNDS

Expand Down
212 changes: 204 additions & 8 deletions doc/api/zlib.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
> Stability: 2 - Stable
The `zlib` module provides compression functionality implemented using Gzip and
Deflate/Inflate. It can be accessed using:
Deflate/Inflate, as well as Brotli. It can be accessed using:

```js
const zlib = require('zlib');
Expand Down Expand Up @@ -54,8 +54,8 @@ and/or unrecoverable and catastrophic memory fragmentation.

## Compressing HTTP requests and responses

The `zlib` module can be used to implement support for the `gzip` and `deflate`
content-encoding mechanisms defined by
The `zlib` module can be used to implement support for the `gzip`, `deflate`
and `br` content-encoding mechanisms defined by
[HTTP](https://tools.ietf.org/html/rfc7230#section-4.2).

The HTTP [`Accept-Encoding`][] header is used within an http request to identify
Expand All @@ -76,12 +76,15 @@ const fs = require('fs');
const request = http.get({ host: 'example.com',
path: '/',
port: 80,
headers: { 'Accept-Encoding': 'gzip,deflate' } });
headers: { 'Accept-Encoding': 'br,gzip,deflate' } });
request.on('response', (response) => {
const output = fs.createWriteStream('example.com_index.html');

switch (response.headers['content-encoding']) {
// or, just use zlib.createUnzip() to handle both cases
case 'br':
response.pipe(zlib.createBrotliDecompress()).pipe(output);
break;
// Or, just use zlib.createUnzip() to handle both of the following cases:
case 'gzip':
response.pipe(zlib.createGunzip()).pipe(output);
break;
Expand Down Expand Up @@ -117,6 +120,9 @@ http.createServer((request, response) => {
} else if (/\bgzip\b/.test(acceptEncoding)) {
response.writeHead(200, { 'Content-Encoding': 'gzip' });
raw.pipe(zlib.createGzip()).pipe(response);
} else if (/\bbr\b/.test(acceptEncoding)) {
response.writeHead(200, { 'Content-Encoding': 'br' });
raw.pipe(zlib.createBrotliCompress()).pipe(response);
} else {
response.writeHead(200, {});
raw.pipe(response);
Expand All @@ -136,6 +142,7 @@ const buffer = Buffer.from('eJzT0yMA', 'base64');

zlib.unzip(
buffer,
// For Brotli, the equivalent is zlib.constants.BROTLI_OPERATION_FLUSH.
{ finishFlush: zlib.constants.Z_SYNC_FLUSH },
(err, buffer) => {
if (!err) {
Expand All @@ -156,6 +163,8 @@ decompressed result is valid.

<!--type=misc-->

### For zlib-based streams

From `zlib/zconf.h`, modified to Node.js's usage:

The memory requirements for deflate are (in bytes):
Expand Down Expand Up @@ -194,6 +203,16 @@ fewer calls to `zlib` because it will be able to process more data on
each `write` operation. So, this is another factor that affects the
speed, at the cost of memory usage.

### For Brotli-based streams

There are equivalents to the zlib options for Brotli-based streams, although
these options have different ranges than the zlib ones:

- zlib’s `level` option matches Brotli’s `BROTLI_PARAM_QUALITY` option.
- zlib’s `windowBits` option matches Brotli’s `BROTLI_PARAM_LGWIN` option.

See [below][Brotli parameters] for more details on Brotli-specific options.

## Flushing

Calling [`.flush()`][] on a compression stream will make `zlib` return as much
Expand Down Expand Up @@ -231,6 +250,8 @@ added: v0.5.8

<!--type=misc-->

### zlib constants

All of the constants defined in `zlib.h` are also defined on
`require('zlib').constants`. In the normal course of operations, it will not be
necessary to use these constants. They are documented so that their presence is
Expand Down Expand Up @@ -281,6 +302,77 @@ Compression strategy.
* `zlib.constants.Z_FIXED`
* `zlib.constants.Z_DEFAULT_STRATEGY`

### Brotli constants
<!-- YAML
added: REPLACEME
-->

There are several options and other constants available for Brotli-based
streams:

#### Flush operations

The following values are valid flush operations for Brotli-based streams:

* `zlib.constants.BROTLI_OPERATION_PROCESS` (default for all operations)
* `zlib.constants.BROTLI_OPERATION_FLUSH` (default when calling `.flush()`)
* `zlib.constants.BROTLI_OPERATION_FINISH` (default for the last chunk)
* `zlib.constants.BROTLI_OPERATION_EMIT_METADATA`
* This particular operation may be hard to use in a Node.js context,
as the streaming layer makes it hard to know which data will end up
in this frame. Also, there is currently no way to consume this data through
the Node.js API.

#### Compressor options

There are several options that can be set on Brotli encoders, affecting
compression efficiency and speed. Both the keys and the values can be accessed
as properties of the `zlib.constants` object.

The most important options are:

* `BROTLI_PARAM_MODE`
* `BROTLI_MODE_GENERIC` (default)
* `BROTLI_MODE_TEXT`, adjusted for UTF-8 text
* `BROTLI_MODE_FONT`, adjusted for WOFF 2.0 fonts
* `BROTLI_PARAM_QUALITY`
* Ranges from `BROTLI_MIN_QUALITY` to `BROTLI_MAX_QUALITY`,
with a default of `BROTLI_DEFAULT_QUALITY`.
* `BROTLI_PARAM_SIZE_HINT`
* Integer value representing the expected input size;
defaults to `0` for an unknown input size.

The following flags can be set for advanced control over the compression
algorithm and memory usage tuning:

* `BROTLI_PARAM_LGWIN`
* Ranges from `BROTLI_MIN_WINDOW_BITS` to `BROTLI_MAX_WINDOW_BITS`,
with a default of `BROTLI_DEFAULT_WINDOW`, or up to
`BROTLI_LARGE_MAX_WINDOW_BITS` if the `BROTLI_PARAM_LARGE_WINDOW` flag
is set.
* `BROTLI_PARAM_LGBLOCK`
* Ranges from `BROTLI_MIN_INPUT_BLOCK_BITS` to `BROTLI_MAX_INPUT_BLOCK_BITS`.
* `BROTLI_PARAM_DISABLE_LITERAL_CONTEXT_MODELING`
* Boolean flag that decreases compression ratio in favour of
decompression speed.
* `BROTLI_PARAM_LARGE_WINDOW`
* Boolean flag enabling “Large Window Brotli” mode (not compatible with the
Brotli format as standardized in [RFC 7932][]).
* `BROTLI_PARAM_NPOSTFIX`
* Ranges from `0` to `BROTLI_MAX_NPOSTFIX`.
* `BROTLI_PARAM_NDIRECT`
* Ranges from `0` to `15 << NPOSTFIX` in steps of `1 << NPOSTFIX`.

#### Decompressor options

These advanced options are available for controlling decompression:

* `BROTLI_DECODER_PARAM_DISABLE_RING_BUFFER_REALLOCATION`
* Boolean flag that affects internal memory allocation patterns.
* `BROTLI_DECODER_PARAM_LARGE_WINDOW`
* Boolean flag enabling “Large Window Brotli” mode (not compatible with the
Brotli format as standardized in [RFC 7932][]).

## Class: Options
<!-- YAML
added: v0.11.1
Expand All @@ -298,7 +390,7 @@ changes:

<!--type=misc-->

Each class takes an `options` object. All options are optional.
Each zlib-based class takes an `options` object. All options are optional.

Note that some options are only relevant when compressing, and are
ignored by the decompression classes.
Expand All @@ -317,6 +409,47 @@ ignored by the decompression classes.
See the description of `deflateInit2` and `inflateInit2` at
<https://zlib.net/manual.html#Advanced> for more information on these.

## Class: BrotliOptions
<!-- YAML
added: REPLACEME
-->

<!--type=misc-->

Each Brotli-based class takes an `options` object. All options are optional.

* `flush` {integer} **Default:** `zlib.constants.BROTLI_OPERATION_PROCESS`
* `finishFlush` {integer} **Default:** `zlib.constants.BROTLI_OPERATION_FINISH`
* `chunkSize` {integer} **Default:** `16 * 1024`
* `params` {Object} Key-value object containing indexed [Brotli parameters][].

For example:

```js
const stream = zlib.createBrotliCompress({
chunkSize: 32 * 1024,
params: {
[zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
[zlib.constants.BROTLI_PARAM_QUALITY]: 4,
[zlib.constants.BROTLI_PARAM_SIZE_HINT]: fs.statSync(inputFile).size
}
});
```

## Class: zlib.BrotliCompress
<!-- YAML
added: REPLACEME
-->

Compress data using the Brotli algorithm.

## Class: zlib.BrotliDecompress
<!-- YAML
added: REPLACEME
-->

Decompress data using the Brotli algorithm.

## Class: zlib.Deflate
<!-- YAML
added: v0.5.8
Expand Down Expand Up @@ -389,9 +522,13 @@ added: v0.5.8
Decompress either a Gzip- or Deflate-compressed stream by auto-detecting
the header.

## Class: zlib.Zlib
## Class: zlib.ZlibBase
<!-- YAML
added: v0.5.8
changes:
- version: REPLACEME
pr-url: https://github.com/nodejs/node/pull/24939
description: This class was renamed from `Zlib` to `ZlibBase`.
-->

Not exported by the `zlib` module. It is documented here because it is the base
Expand Down Expand Up @@ -440,7 +577,8 @@ Close the underlying handle.
added: v0.5.8
-->

* `kind` **Default:** `zlib.constants.Z_FULL_FLUSH`
* `kind` **Default:** `zlib.constants.Z_FULL_FLUSH` for zlib-based streams,
`zlib.constants.BROTLI_OPERATION_FLUSH` for Brotli-based streams.
* `callback` {Function}

Flush pending data. Don't call this frivolously, premature flushes negatively
Expand All @@ -460,6 +598,8 @@ added: v0.11.4
* `strategy` {integer}
* `callback` {Function}

This function is only available for zlib-based streams, i.e. not Brotli.

Dynamically update the compression level and compression strategy.
Only applicable to deflate algorithm.

Expand All @@ -478,6 +618,24 @@ added: v7.0.0

Provides an object enumerating Zlib-related constants.

## zlib.createBrotliCompress([options])
<!-- YAML
added: REPLACEME
-->

* `options` {brotli options}

Creates and returns a new [`BrotliCompress`][] object.

## zlib.createBrotliDecompress([options])
<!-- YAML
added: REPLACEME
-->

* `options` {brotli options}

Creates and returns a new [`BrotliDecompress`][] object.

## zlib.createDeflate([options])
<!-- YAML
added: v0.5.8
Expand Down Expand Up @@ -560,6 +718,40 @@ with `callback(error, result)`.
Every method has a `*Sync` counterpart, which accept the same arguments, but
without a callback.

### zlib.brotliCompress(buffer[, options], callback)
<!-- YAML
added: REPLACEME
-->
* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
* `options` {brotli options}
* `callback` {Function}

### zlib.brotliCompressSync(buffer[, options])
<!-- YAML
added: REPLACEME
-->
* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
* `options` {brotli options}

Compress a chunk of data with [`BrotliCompress`][].

### zlib.brotliDecompress(buffer[, options], callback)
<!-- YAML
added: REPLACEME
-->
* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
* `options` {brotli options}
* `callback` {Function}

### zlib.brotliDecompressSync(buffer[, options])
<!-- YAML
added: REPLACEME
-->
* `buffer` {Buffer|TypedArray|DataView|ArrayBuffer|string}
* `options` {brotli options}

Decompress a chunk of data with [`BrotliDecompress`][].

### zlib.deflate(buffer[, options], callback)
<!-- YAML
added: v0.6.0
Expand Down Expand Up @@ -832,6 +1024,8 @@ Decompress a chunk of data with [`Unzip`][].
[`.flush()`]: #zlib_zlib_flush_kind_callback
[`Accept-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
[`ArrayBuffer`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer
[`BrotliCompress`]: #zlib_class_zlib_brotlicompress
[`BrotliDecompress`]: #zlib_class_zlib_brotlidecompress
[`Buffer`]: buffer.html#buffer_class_buffer
[`Content-Encoding`]: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
[`DataView`]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView
Expand All @@ -845,6 +1039,8 @@ Decompress a chunk of data with [`Unzip`][].
[`Unzip`]: #zlib_class_zlib_unzip
[`stream.Transform`]: stream.html#stream_class_stream_transform
[`zlib.bytesWritten`]: #zlib_zlib_byteswritten
[Brotli parameters]: #zlib_brotli_constants
[Memory Usage Tuning]: #zlib_memory_usage_tuning
[RFC 7932]: https://www.rfc-editor.org/rfc/rfc7932.txt
[pool size]: cli.html#cli_uv_threadpool_size_size
[zlib documentation]: https://zlib.net/manual.html#Constants
2 changes: 2 additions & 0 deletions tools/doc/type-parser.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ const customTypesMap = {
'AsyncHook': 'async_hooks.html#async_hooks_async_hooks_createhook_callbacks',
'AsyncResource': 'async_hooks.html#async_hooks_class_asyncresource',

'brotli options': 'zlib.html#zlib_class_brotlioptions',

'Buffer': 'buffer.html#buffer_class_buffer',

'ChildProcess': 'child_process.html#child_process_class_childprocess',
Expand Down

0 comments on commit 93edf90

Please sign in to comment.