crypto: refactor crypto subsystem and introduce WebCrypto API

[jasnell]

@nodejs/tsc @nodejs/crypto ... This is a big one. It is still a work in progress. I'm opening the draft PR so that people can follow along with the work.

What does this do:

Three main things... lots of other little things

1. It refactors the Node.js src crypto internals so that they are more maintainable and organized. The existing node_crypto.cc has grown into a massive disorganized and unmaintainable mess that very few brave touch. This breaks that functionality up across multiple files in src/crypto that are organized by purpose/algorithm.

2. It makes a number of important fixes and improvements to the existing crypto internals. For example, previously, CryptoJob had no mechanism for tracking the memory associated with it. CryptoJob has been refactored into an AsyncWrap derived object, allowing and improving memory tracking.

3. It introduces an experimental Web Crypto API implementation as require('crypto').webcrypto. The initial intent of this API is to be standards compliant.

4. It introduces HKDF support for the legacy API. HKDF is required by the Web Crypto API implementation, so I decided to go ahead and add a variation for the legacy API also.

5. It introduces crypto.timingSafeEqual.bigint() for performing constant-time comparisons of bigint values. Pulled this due to some outstanding technical concerns that will need to be looked at later.

6. It introduces the ability to use ArrayBuffer for the existing legacy crypto APIs. Previously, the legacy API was restricted to Buffer, TypedArray, and DataView objects. Because the Web Crypto API makes use of ArrayBuffer also, I decided to go ahead and extend that capability to the legacy API. (Note: this is still a work in progress that will be completed before this PR moves out of draft status)

7. It introduces one extension to the Web Crypto API exportKey() and importKey() methods to allow converting back and forth between a Node.js KeyObject and a Web Crypto CryptoKey object. The CryptoKey is currently implemented as a wrapper around KeyObject.

My next task on this PR is to begin filling out the tests. This PR will remain in draft status until I have those ready. However, please feel free to begin reviewing the changes made so far.

This is a big PR that will take some time to review. I'm happy to jump on a zoom call with anyone to walk through it.

Outside of changes to error messages and codes (which have been made more consistent), there should be no backwards compatibility breaking changes to the existing legacy API. Any backwards breaking changes should be considered bugs.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[nodejs-github-bot]

Review requested:

• @nodejs/crypto

[jasnell]

For anyone curious about the performance difference that the SubtleCrypto APIs will have relative to the legacy apis... Here's an example comparing crypto.createHash() to subtle.digest() ...

Keep in mind, however, that this is tracking raw execution speed, and ignores factors such as event loop delay, event loop utilization and memory. The createHash variant is purely sync, while the subtle version uses the libuv threadpool. Significantly, the createHash variant is one operation at a time, while the subtle queues up n hash jobs concurrently.

crypto/webcrypto-digest.js n=1000 method="SHA-1" data=10 sync="createHash": 94,602.60002001791
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=10 sync="createHash": 20,070.058150585086
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=10 sync="createHash": 55,681.77686340876
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=10 sync="createHash": 43,733.5794076068
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=20 sync="createHash": 70,726.74488714593
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=20 sync="createHash": 52,796.95289777924
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=20 sync="createHash": 92,470.48873704823
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=20 sync="createHash": 36,330.60193812862
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=50 sync="createHash": 93,567.16391376476
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=50 sync="createHash": 95,433.87100776259
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=50 sync="createHash": 69,989.07610500153
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=50 sync="createHash": 86,632.92891638233
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=100 sync="createHash": 62,713.004720533296
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=100 sync="createHash": 92,227.69542356953
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=100 sync="createHash": 58,363.47758314272
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=100 sync="createHash": 91,339.97478468656
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=10 sync="subtle": 13,358.364593130998
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=10 sync="subtle": 32,006.11342371284
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=10 sync="subtle": 22,250.87690149597
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=10 sync="subtle": 20,624.592264701296
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=20 sync="subtle": 51,008.840087031276
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=20 sync="subtle": 14,754.004273644876
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=20 sync="subtle": 46,213.40955095308
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=20 sync="subtle": 40,880.32676789835
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=50 sync="subtle": 22,610.031492156264
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=50 sync="subtle": 33,920.458018446894
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=50 sync="subtle": 48,910.4416554189
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=50 sync="subtle": 12,468.43351102472
crypto/webcrypto-digest.js n=1000 method="SHA-1" data=100 sync="subtle": 21,238.01034673388
crypto/webcrypto-digest.js n=1000 method="SHA-256" data=100 sync="subtle": 25,608.43004933131
crypto/webcrypto-digest.js n=1000 method="SHA-384" data=100 sync="subtle": 47,109.37798850117
crypto/webcrypto-digest.js n=1000 method="SHA-512" data=100 sync="subtle": 22,149.473507014736

[panva]

@jasnell thank you for this work!

I assume the goal with the subtle interface is to 100% cover the specification, i also assume some Node.js specifics are going to be in place (e.g. passing Buffer instances, possibly also KeyObject instances to importKey?, dunno ...). I welcome the addition of built-in hkdf exposed outside of the subtle interfaces.

Altho the interface doesn't enable anything new algorithm-wise it's a great way to support modules such as jose to be isomorphic and web-compatible.

By far the biggest addition of this interface, from my point of view, is its implementation using the libuv threadpool. Being able to run both fast and slow algorithms using the same interface and not having the main thread blocked is finally here. Which brings me to obvious question - what about sign/verify/ecdh/etc operations using either key types or algorithms that are not supported by the webcrypto API specification?

Is there going to be a crypto/promises module with the existing sign/verify/publicEncrypt/privateDecrypt/createCipheriv/createDecipheriv interfaces ran using the libuv threadpool? It would be a shame if we could only get non-blocking crypto within the very limited algorithm scope of webcrypto api.

[jasnell]

Yes, the goal is 100% compatibility with the Web Crypto API. The CryptoKey object is a wrapper around the existing KeyObject. Allowing conversion between the two using importKey and exportKey is a good idea. Buffer will be supported but is treated as if it is any other TypedArray.

Once the basic algorithms required by the WebCrypto spec are implemented, support for the broader range of algorithms we support in the legacy API will be added, starting with a few we already support such as scrypt, DH (non-ECDH), all of the ciphers, hashes, and curves reported in the getHashes, getCiphers, and getCurves APIs, and all of the keygen types supported by the existing generateKeyPair API. This will take some effort as they will be integrated into the extensible framework already provided by WebCrypto. Anything that does not fit within the WebCrypto API will be exposed directly off the crypto/promises module and not the SubtleCrypto class.

After that, I have plans for a few new algorithms we do not currently support in either API. CMAC for instance. And possibly UUID and simple HOTP/TOTP based token generation. Once the changes here are completed, it will be far easier to incorporate such additions.

[targos]

Do you plan to integrate the WebCryptoAPI web platform tests?

[panva]

@jasnell amazing! Happy to hear about this roadmap.

Let me know if you need assistance with testing.

[panva]

This will take some effort as they will be integrated into the extensible framework already provided by WebCrypto

I would be very careful with extending WebCrypto with custom algorithms as the proprietary unregistered algs may clash and confuse developers thinking it's part of WebCrypto. I'd suggest to leave WebCrypto implementation inline with the current version of the spec and treat everything unregistered in WebCrypto like so 👇

Anything that does not fit within the WebCrypto API will be exposed directly off the crypto/promises module and not the SubtleCrypto class.

Ad CMAC > there's this PR that could use wrapping up. Altho from my testing it's not working as intended just yet.

[jasnell]

@targos:

Do you plan to integrate the WebCryptoAPI web platform tests?

Absolutely, but that will likely be in a follow on PR.

[jasnell]

@panva:

I would be very careful with extending WebCrypto with custom algorithms as the proprietary unregistered algs may clash and confuse developers thinking it's part of WebCrypto

Caution will definitely be waranted and any such extension will be in line with the extensibility guidelines of the spec (https://www.w3.org/TR/WebCryptoAPI/#extensibility). Specifically, any Node.js specific algorithms would be prefixed as such when used. For instance...

const { subtle } = require('crypto/promises');
const ec = new TextEncoder();

subtle.digest({ name: 'node.shake256', length: 100 }, ec.encode('hello'));

For well-known algorithms like Scrypt and CMAC, my intention would be to introduce those initially as node-prefixed algorithms e.g. {name: 'node-cmac' } and simultaneously open a proposal to have those added to the standard directly.

One alternative that I'm considering is exposing a separate NodeSubtleCrypto class off require('crypto/promises') that extends SubtleCrypto with the addition of the Node.js-specific extensions.

So, for instance, with this alternative, the following would fail:

const { subtle } = require('crypto/promises');
const ec = new TextEncoder();

subtle.digest({ name: 'node.shake256', length: 100 }, ec.encode('hello'));

But this would work:

const { nodeSubtle } = require('crypto/promises');
const ec = new TextEncoder();

nodeSubtle.digest({ name: 'node.shake256', length: 100 }, ec.encode('hello'));

In either case, developers will have to explicitly opt in to using the Node.js specific extensions.

[panva]

@jasnell sounds good