zlib: expose zlib.crc32() #52692

joyeecheung · 2024-04-25T17:53:00Z

This patch exposes the crc32() function from zlib to user-land.

It computes a 32-bit Cyclic Redundancy Check checksum of data. If
value is specified, it is used as the starting value of the checksum,
otherwise, 0 is used as the starting value.

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

To be consistent with other APIs, if the data is a string, it will
be encoded with UTF-8 before being used for computation. If users only
use Node.js to compute and match the checksums, this works well with
other APIs that uses the UTF-8 encoding by default.

Some third-party JavaScript libraries compute the checksum on a
string based on str.charCodeAt() so that it can be run in browsers.
If users want to match the checksum computed with this kind of library
in the browser, it's better to use the same library in Node.js
if it also runs in Node.js. If users have to use zlib.crc32() to
match the checksum produced by such a third-party library:

If the library accepts Uint8Array as input, use TextEncoder
in the browser to encode the string into a Uint8Array with UTF-8
encoding, and compute the checksum based on the UTF-8 encoded string
in the browser.
If the library only takes a string and compute the data based on
str.charCodeAt(), on the Node.js side, convert the string into
a buffer using Buffer.from(str, 'utf16le').

const zlib = require('node:zlib');
const { Buffer } = require('node:buffer');

let crc = zlib.crc32('hello');  // 907060870
crc = zlib.crc32('world', crc);  // 4192936109

crc = zlib.crc32(Buffer.from('hello', 'utf16le'));  // 1427272415
crc = zlib.crc32(Buffer.from('world', 'utf16le'), crc);  // 4150509955

NOTE: it's exposed in zlib because that's also what Python and Ruby do.

src/node_zlib.cc

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ```

doc/api/zlib.md

lpinca · 2024-04-25T19:33:08Z

Wouldn't it be better to expose it under crypto?

Edit: On second thought, maybe not.

joyeecheung · 2024-04-25T19:54:39Z

Wouldn't it be better to expose it under crypto?

It's not a cryptographic hash, but a checksum. I was thinking about copying the python documentation (python also puts it on zlib):

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

(Not sure if it's okay to copy other documentation, but if we have to write one from scratch it wouldn't be too different anyway?)

panva · 2024-04-26T09:49:30Z

Wouldn't it be better to expose it under crypto?

Edit: On second thought, maybe not.

I had the exact same thought process.

The algorithm is not cryptographically strong, and should not be used for authentication or digital signatures. Since the algorithm is designed for use as a checksum algorithm, it is not suitable for use as a general hash algorithm.

I agree this documentation bit or something to its effect is necessary to circumvent the same brain exercises myself and @lpinca went through :)

joyeecheung · 2024-04-26T09:51:26Z

The doc currently puts it as

The CRC algorithm is designed to compute checksums and to detect error
in data transmission. It's not suitable for cryptographic authentication.

Also, note that both Python and Ruby puts crc32 under a zlib module

github-actions · 2024-04-26T12:20:55Z

Failed to start CI

   ⚠  Something was pushed to the Pull Request branch since the last approving review.
   ✘  Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/8848132665

joyeecheung · 2024-04-26T12:24:33Z

huh, this is...amusing. I'll start from Jenkins.

nodejs-github-bot · 2024-04-26T12:24:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58716/

src/node_zlib.cc

Co-authored-by: Anna Henningsen <github@addaleax.net>

nodejs-github-bot · 2024-04-26T17:32:22Z

CI: https://ci.nodejs.org/job/node-test-pull-request/58721/

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: nodejs#52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: #52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 lib,src: * remove --experimental-policy (Rafael Gonzaga) #52583 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: TODO

Notable changes: cli: * (SEMVER-MINOR) allow running wasm in limited vmem with --disable-wasm-trap-handler (Joyee Cheung) #52766 doc: * add pimterry to collaborators (Tim Perry) #52874 fs: * (SEMVER-MINOR) allow 'withFileTypes' to be used with globs (Aviv Keller) #52837 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 lib,src: * remove --experimental-policy (Rafael Gonzaga) #52583 perf_hooks: * (SEMVER-MINOR) add `deliveryType` and `responseStatus` fields (Matthew Aitken) #51589 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #52971

nodejs/node#52692

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: #52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: TODO

Notable changes: doc: * add pimterry to collaborators (Tim Perry) #52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) #52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) #52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) #52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) #52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) #52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) #52692 PR-URL: #53486

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: nodejs#52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: cli: * (SEMVER-MINOR) allow running wasm in limited vmem with --disable-wasm-trap-handler (Joyee Cheung) nodejs#52766 doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 fs: * (SEMVER-MINOR) allow 'withFileTypes' to be used with globs (Aviv Keller) nodejs#52837 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 lib,src: * remove --experimental-policy (Rafael Gonzaga) nodejs#52583 perf_hooks: * (SEMVER-MINOR) add `deliveryType` and `responseStatus` fields (Matthew Aitken) nodejs#51589 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#52971

Notable changes: doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) nodejs#52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#53486

This patch exposes the crc32() function from zlib to user-land. It computes a 32-bit Cyclic Redundancy Check checksum of `data`. If `value` is specified, it is used as the starting value of the checksum, otherwise, 0 is used as the starting value. ```js const zlib = require('node:zlib'); const { Buffer } = require('node:buffer'); let crc = zlib.crc32('hello'); // 907060870 crc = zlib.crc32('world', crc); // 4192936109 crc = zlib.crc32(Buffer.from('hello')); // 907060870 crc = zlib.crc32(Buffer.from('world'), crc); // 4192936109 ``` PR-URL: nodejs#52692 Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes: cli: * (SEMVER-MINOR) allow running wasm in limited vmem with --disable-wasm-trap-handler (Joyee Cheung) nodejs#52766 doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 fs: * (SEMVER-MINOR) allow 'withFileTypes' to be used with globs (Aviv Keller) nodejs#52837 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 lib,src: * remove --experimental-policy (Rafael Gonzaga) nodejs#52583 perf_hooks: * (SEMVER-MINOR) add `deliveryType` and `responseStatus` fields (Matthew Aitken) nodejs#51589 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#52971

Notable changes: doc: * add pimterry to collaborators (Tim Perry) nodejs#52874 inspector: * (SEMVER-MINOR) introduce the `--inspect-wait` flag (Kohei Ueno) nodejs#52734 test_runner: * (SEMVER-MINOR) support test plans (Colin Ihrig) nodejs#52860 tools: * (SEMVER-MINOR) fix get_asan_state() in tools/test.py (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support max_virtual_memory test configuration (Joyee Cheung) nodejs#52766 * (SEMVER-MINOR) support != in test status files (Joyee Cheung) nodejs#52766 zlib: * (SEMVER-MINOR) expose zlib.crc32() (Joyee Cheung) nodejs#52692 PR-URL: nodejs#53486

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. zlib Issues and PRs related to the zlib subsystem. labels Apr 25, 2024

anonrig reviewed Apr 25, 2024

View reviewed changes

src/node_zlib.cc Show resolved Hide resolved

anonrig approved these changes Apr 25, 2024

View reviewed changes

joyeecheung force-pushed the crc32 branch 2 times, most recently from b2b9ece to 23223c6 Compare April 25, 2024 19:05

joyeecheung force-pushed the crc32 branch from 23223c6 to 53fb0e4 Compare April 25, 2024 19:16

joyeecheung commented Apr 25, 2024

View reviewed changes

doc/api/zlib.md Show resolved Hide resolved

joyeecheung added 2 commits April 25, 2024 22:01

fixup! zlib: expose zlib.crc32()

d9057fc

fixup! fixup! zlib: expose zlib.crc32()

d4fc5ca

joyeecheung force-pushed the crc32 branch from 278dfef to d4fc5ca Compare April 25, 2024 20:04

fixup! fixup! zlib: expose zlib.crc32()

b26c0c6

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Apr 26, 2024

github-actions bot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Apr 26, 2024

joyeecheung mentioned this pull request Apr 26, 2024

feat(ncu-ci)!: require --certify-safe flag nodejs/node-core-utils#798

Merged

lpinca approved these changes Apr 26, 2024

View reviewed changes

lpinca added the semver-minor PRs that contain new features and should be released in the next minor version. label Apr 26, 2024

addaleax approved these changes Apr 26, 2024

View reviewed changes

src/node_zlib.cc Outdated Show resolved Hide resolved

fixup! zlib: expose zlib.crc32()

905c481

Co-authored-by: Anna Henningsen <github@addaleax.net>

targos mentioned this pull request May 13, 2024

v22.2.0 release proposal #52971

Merged

scagood added a commit to eslint-community/eslint-plugin-n that referenced this pull request May 17, 2024

feat(node-builtin): Add node 22.2.0 support

15e090d

nodejs/node#52692

scagood mentioned this pull request May 17, 2024

feat(node-builtin): Add node 22.2.0 support eslint-community/eslint-plugin-n#282

Merged

aladdin-add pushed a commit to eslint-community/eslint-plugin-n that referenced this pull request May 17, 2024

feat(node-builtin): Add node 22.2.0 support (#282)

5221c40

nodejs/node#52692

marco-ippolito mentioned this pull request Jun 17, 2024

v20.15.0 proposal #53486

Merged

segfault-bilibili mentioned this pull request Jul 10, 2024

Incorporate suite of hash utilities into node #2444

Closed

trivikr mentioned this pull request Sep 23, 2024

chore: add Node.js CRC32 for comparison trivikr/benchmark-aws-crt-crc32#2

Merged

trivikr mentioned this pull request Nov 7, 2024

feat(middleware-flexible-checksums): use Node.js native CRC32 checksum API aws/aws-sdk-js-v3#6641

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zlib: expose zlib.crc32() #52692

zlib: expose zlib.crc32() #52692

joyeecheung commented Apr 25, 2024 •

edited

Loading

lpinca commented Apr 25, 2024 •

edited

Loading

joyeecheung commented Apr 25, 2024 •

edited

Loading

panva commented Apr 26, 2024

joyeecheung commented Apr 26, 2024 •

edited

Loading

github-actions bot commented Apr 26, 2024

joyeecheung commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

zlib: expose zlib.crc32() #52692

zlib: expose zlib.crc32() #52692

Conversation

joyeecheung commented Apr 25, 2024 • edited Loading

lpinca commented Apr 25, 2024 • edited Loading

joyeecheung commented Apr 25, 2024 • edited Loading

panva commented Apr 26, 2024

joyeecheung commented Apr 26, 2024 • edited Loading

github-actions bot commented Apr 26, 2024

joyeecheung commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

nodejs-github-bot commented Apr 26, 2024

joyeecheung commented Apr 25, 2024 •

edited

Loading

lpinca commented Apr 25, 2024 •

edited

Loading

joyeecheung commented Apr 25, 2024 •

edited

Loading

joyeecheung commented Apr 26, 2024 •

edited

Loading