crypto: implement crypto.hash()

[joyeecheung]

crypto: implement crypto.hash()

This patch introduces a helper crypto.hash() that computes
a digest from the input at one shot. This can be 1.2-2x faster
than the object-based createHash() for smaller inputs (<= 5MB)
that are readily available (not streamed) and incur less memory
overhead since no intermediate objects will be created.

Refs: nodejs/performance#136

From the benchmark CI https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1493/console

22:46:22                                                                           confidence improvement accuracy (*)    (**)   (***)
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='md5' length=1000             ***     76.69 %      ±11.25% ±15.02% ±19.65%
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='md5' length=100000            **     12.44 %       ±8.53% ±11.36% ±14.78%
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='sha1' length=1000            ***     72.44 %      ±11.25% ±15.01% ±19.61%
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='sha1' length=100000                  -1.19 %       ±6.72%  ±8.94% ±11.65%
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='sha256' length=1000          ***     65.75 %      ±11.03% ±14.71% ±19.22%
22:46:22 crypto/oneshot-hash.js n=1000 type='buffer' method='sha256' length=100000                -1.41 %       ±2.99%  ±3.99%  ±5.22%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='md5' length=1000             ***     98.77 %      ±15.43% ±20.67% ±27.18%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='md5' length=100000                    1.43 %       ±3.17%  ±4.21%  ±5.48%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='sha1' length=1000            ***    111.56 %      ±17.04% ±22.85% ±30.09%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='sha1' length=100000                   1.67 %       ±3.73%  ±4.96%  ±6.46%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='sha256' length=1000          ***     88.15 %      ±13.62% ±18.19% ±23.83%
22:46:22 crypto/oneshot-hash.js n=1000 type='string' method='sha256' length=100000                 2.00 %       ±2.05%  ±2.73%  ±3.56%
22:46:22 crypto/oneshot-hash.js n=100000 type='buffer' method='md5' length=1000           ***     39.52 %       ±6.05%  ±8.11% ±10.67%
22:46:22 crypto/oneshot-hash.js n=100000 type='buffer' method='sha1' length=1000          ***     41.96 %       ±7.36%  ±9.86% ±12.95%
22:46:22 crypto/oneshot-hash.js n=100000 type='buffer' method='sha256' length=1000        ***     23.13 %       ±3.97%  ±5.31%  ±6.96%
22:46:22 crypto/oneshot-hash.js n=100000 type='string' method='md5' length=1000           ***     23.50 %       ±3.04%  ±4.06%  ±5.30%
22:46:22 crypto/oneshot-hash.js n=100000 type='string' method='sha1' length=1000          ***     30.30 %       ±3.67%  ±4.89%  ±6.39%
22:46:22 crypto/oneshot-hash.js n=100000 type='string' method='sha256' length=1000        ***     18.63 %       ±3.12%  ±4.16%  ±5.43%

Benchmark results from macOS + M2

                                                                         confidence improvement accuracy (*)    (**)   (***)
crypto/node-digest.js n=1000 type='buffer' method='md5' length=1000             ***     24.64 %       ±6.24%  ±8.33% ±10.92%
crypto/node-digest.js n=1000 type='buffer' method='md5' length=100000                    1.82 %       ±1.87%  ±2.50%  ±3.26%
crypto/node-digest.js n=1000 type='buffer' method='sha1' length=1000            ***     46.77 %       ±4.13%  ±5.49%  ±7.15%
crypto/node-digest.js n=1000 type='buffer' method='sha1' length=100000                   1.99 %       ±3.24%  ±4.32%  ±5.65%
crypto/node-digest.js n=1000 type='buffer' method='sha256' length=1000          ***     46.38 %       ±3.38%  ±4.52%  ±5.93%
crypto/node-digest.js n=1000 type='buffer' method='sha256' length=100000                 1.81 %       ±2.71%  ±3.61%  ±4.70%
crypto/node-digest.js n=1000 type='string' method='md5' length=1000             ***     25.20 %       ±7.60% ±10.13% ±13.22%
crypto/node-digest.js n=1000 type='string' method='md5' length=100000             *      3.22 %       ±2.45%  ±3.28%  ±4.33%
crypto/node-digest.js n=1000 type='string' method='sha1' length=1000            ***     41.89 %       ±4.25%  ±5.67%  ±7.41%
crypto/node-digest.js n=1000 type='string' method='sha1' length=100000                   0.99 %       ±4.32%  ±5.77%  ±7.57%
crypto/node-digest.js n=1000 type='string' method='sha256' length=1000          ***     42.50 %       ±4.12%  ±5.50%  ±7.20%
crypto/node-digest.js n=1000 type='string' method='sha256' length=100000                -0.56 %       ±3.69%  ±4.96%  ±6.56%
crypto/node-digest.js n=100000 type='buffer' method='md5' length=1000           ***     17.91 %       ±4.82%  ±6.42%  ±8.35%
crypto/node-digest.js n=100000 type='buffer' method='sha1' length=1000          ***     47.75 %       ±2.83%  ±3.77%  ±4.92%
crypto/node-digest.js n=100000 type='buffer' method='sha256' length=1000        ***     56.42 %       ±2.75%  ±3.66%  ±4.79%
crypto/node-digest.js n=100000 type='string' method='md5' length=1000           ***     15.45 %       ±2.45%  ±3.28%  ±4.32%
crypto/node-digest.js n=100000 type='string' method='sha1' length=1000          ***     32.16 %       ±4.69%  ±6.27%  ±8.22%
crypto/node-digest.js n=100000 type='string' method='sha256' length=1000        ***     34.65 %       ±2.42%  ±3.23%  ±4.23%

Benchmark results from Ubuntu + Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz

                                                                         confidence improvement accuracy (*)   (**)  (***)
crypto/node-digest.js n=1000 type='buffer' method='md5' length=1000             ***     50.52 %       ±3.07% ±4.09% ±5.34%
crypto/node-digest.js n=1000 type='buffer' method='md5' length=100000            **      1.31 %       ±0.79% ±1.05% ±1.37%
crypto/node-digest.js n=1000 type='buffer' method='sha1' length=1000            ***     58.90 %       ±4.88% ±6.54% ±8.62%
crypto/node-digest.js n=1000 type='buffer' method='sha1' length=100000          ***      3.13 %       ±1.21% ±1.62% ±2.11%
crypto/node-digest.js n=1000 type='buffer' method='sha256' length=1000          ***     43.08 %       ±3.01% ±4.02% ±5.25%
crypto/node-digest.js n=1000 type='buffer' method='sha256' length=100000        ***      1.31 %       ±0.70% ±0.93% ±1.21%
crypto/node-digest.js n=1000 type='string' method='md5' length=1000             ***     45.39 %       ±3.28% ±4.40% ±5.78%
crypto/node-digest.js n=1000 type='string' method='md5' length=100000            **      1.17 %       ±0.77% ±1.02% ±1.33%
crypto/node-digest.js n=1000 type='string' method='sha1' length=1000            ***     52.71 %       ±2.93% ±3.93% ±5.18%
crypto/node-digest.js n=1000 type='string' method='sha1' length=100000           **      1.81 %       ±1.06% ±1.42% ±1.85%
crypto/node-digest.js n=1000 type='string' method='sha256' length=1000          ***     41.00 %       ±2.62% ±3.50% ±4.58%
crypto/node-digest.js n=1000 type='string' method='sha256' length=100000         **      0.90 %       ±0.54% ±0.72% ±0.93%
crypto/node-digest.js n=100000 type='buffer' method='md5' length=1000           ***     55.57 %       ±1.94% ±2.60% ±3.42%
crypto/node-digest.js n=100000 type='buffer' method='sha1' length=1000          ***     66.81 %       ±1.57% ±2.09% ±2.72%
crypto/node-digest.js n=100000 type='buffer' method='sha256' length=1000        ***     39.65 %       ±1.16% ±1.55% ±2.03%
crypto/node-digest.js n=100000 type='string' method='md5' length=1000           ***     38.61 %       ±1.55% ±2.07% ±2.71%
crypto/node-digest.js n=100000 type='string' method='sha1' length=1000          ***     43.45 %       ±1.20% ±1.59% ±2.07%
crypto/node-digest.js n=100000 type='string' method='sha256' length=1000        ***     31.46 %       ±1.33% ±1.77% ±2.30%

[nodejs-github-bot]

Review requested:

• @nodejs/crypto

[panva]

Interested, could the same be done for hmac?

[joyeecheung]

Results of comparing the one-shot digest implementation from this, the createHash() from #51034 and the main branch using a microbenchmark from nodejs/performance#136 (comment) on macOS + M2:

$ out/Release/node hash.js
612.720917
1000000 hashes, input length 1000, 1000000 results
$ ./node_51034 hash.js
930.777417
1000000 hashes, input length 1000, 1000000 results
$ ./node_main hash.js
1111.237292
1000000 hashes, input length 1000, 1000000 results

On Ubuntu + Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz

$ out/Release/node hash.js
1451.8053810000001
1000000 hashes, input length 1000, 1000000 results
$ ./node_51034 hash.js
2596.804682
1000000 hashes, input length 1000, 1000000 results
$ ./node_main hash.js
3014.865648
1000000 hashes, input length 1000, 1000000 results

[joyeecheung]

Interested, could the same be done for hmac?

Yes though I think for caching EVP_MAC for maximum speed we need to do the init-update-final ourselves because for HAMC there's only EVP_Q_mac which always fetches the implementation on its own.

[tniessen]

I suppose this supersedes #42233?

[joyeecheung]

I suppose this supersedes #42233?

If the goal is to "reducing as much unnecessary overhead as possible when computing a one-shot digest synchronously" then I think yes. With an implementation based on HashJob there would still be quite a lot of overhead, whereas not creating any BaseObject + explicit fetching via EVP_MD_fetch + caching EVP_MD* + mapping encoding/algorithm from string to ids + using EVP_Digest directly would shave off a lot more or get rid of most of the bottleneck that's still under Node.js's control.

I don't know if the current API is the best, however. But I think that might already be enough for maybe 80% of the use case (this assumes the string input is decoded as UTF8 - would the need of custom input decoding really be that common? With this API users can still decode their strings with other encodings into a buffer and use that as input if they really need a different encoding, though)

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/56773/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/56843/

[joyeecheung]

Added docs and renamed this to crypto.hash (because if we were to follow up with the same thing for HMAC later, crypto.digest() would be ambiguous). Can I get some review please? @nodejs/cpp-reviewers @nodejs/crypto

[joyeecheung]

Pinging @nodejs/cpp-reviewers @nodejs/crypto again..

[aduh95]

I can't really vouch for the C++ code, but JS code and benchmark LGTM 👍 Could you add a test that validates the correct error is returned when passing arguments with wrong types?

[H4ad]

Amazing work, I'm sure many libraries will see a good impact on performance with this new method.

@nodejs/performance Ping to get more attention/approvals.

[panva]

The default encoding being hex is kind of awkward and not inline with our other APIs. Should we pause inclusion of this in the upcoming release and reconsider/discuss to have the default be a buffer?

[tniessen]

We could at least mark this as experimental if it isn't already.

[joyeecheung]

From what I can tell, when people do one-shot hashing, most of the time they want the result to be encoded in a hex string - or from a quick search in popular packages, it seems rare for them to want the result to be returned in a buffer. What exactly makes this default awkward other than that it follows what most users want instead of what other APIs already do?

Personally it doesn't seem very necessary to me to return the hash in buffer by default in other APIs either, though they are already what they are. The hash is small and fixed-sized, and creating a buffer is not cheap. Encoding it to string isn't, either - for less than a megabyte of input, the buffer + string allocation are actually more expensive than the actual hashing (and the most common use case of this that I can see in top packages is hashing JS files, which are normally around this size).

[joyeecheung]

I agree we can mark it as 1.2, though, if users do think defaulting to buffer makes sense, we can switch to buffer before we make it 2.

[GauriSpears]

It looks like this commit broke my code. I use algorithm from external OpenSSL engine and now I get error:0308010C:digital envelope routines::unsupported
I suppose you added some check that algorithm exists but forgot that it may be loaded by crypto.setEngine and so it's not in the digests list of statically linked OpenSSL.

[joyeecheung]

@GauriSpears I think that should be caused by #51034 - can you open an issue with a reproduction?

[GauriSpears]

@joyeecheung , you are right. Not all of digest algos can be retrieved via EVP_MD_fetch and EVP_get_digestbyname should be used to get them. I've fixed the same problem in OpenSSL last year:
openssl/openssl#20651

[joyeecheung]

Not all of digest algos can be retrieved via EVP_MD_fetch and EVP_get_digestbyname should be used to get them.

The caching does EVP_get_digestbyname first and if EVP_MD_fetch doesn't return a result, it uses the one returned by EVP_get_digestbyname, otherwise it uses the one fetched by EVP_MD_fetch, so it should work as long as EVP_get_digestbyname provides valid results. We also have some tests for setEngine, but not a lot. If the caching somehow broke your use case, can you provide a reproduction?

[karlhorky]

This is great, thanks @joyeecheung and other contributors!! 🙌

For TypeScript users, @types/node will need a PR to DefinitelyTyped to add crypto.hash():

• https://github.com/DefinitelyTyped/DefinitelyTyped/blob/518b9ae4a518c621700a97be30ee128b66a5da78/types/node/crypto.d.ts#L231

TypeScript Playground

Error message:

Property 'hash' does not exist on type 'typeof import("node:crypto")'. Did you mean 'Hash'?(2551)

cc DT maintainers / contributors to @types/node crypto @Semigradsky @jhmaster2000

---

Edit: this has been added in @types/node@20.12.8 by this PR by @codershiba:

• [@types/node] Update type definitions for v20.12.2 DefinitelyTyped/DefinitelyTyped#69212

[GauriSpears]

The caching does EVP_get_digestbyname first and if EVP_MD_fetch doesn't return a result, it uses the one returned by EVP_get_digestbyname

I'm trying to dig into your code. From the first sight it seems really equivalent to what happens in OpenSSL. Could you explain: when the caching of algorithm list is done? And how it shoud work if setEngine loads engine with new algorithms?

[joyeecheung]

Could you explain: when the caching of algorithm list is done?

That was explained in #51034 (comment)

And how it shoud work if setEngine loads engine with new algorithms?

It's supposed to keep trying EVP_get_digestbyname when there's no cached entry for that name (i.e. when it's not explicitly fetchable via EVP_MD_fetch).

It seems a bit off topic for this PR. Can you open a separate issue for this? Ideally with a reproduction.