src: fix error handling for CryptoJob::ToResult

[tniessen]

The added test case crashes in recent versions of Node.js 15 due to a change in OpenSSL and improper error handling in node core.

The first problem is that ManagedEVPPKey::ToEncodedPublicKey and ManagedEVPPKey::ToEncodedPrivateKey returned Just(false) to indicate that an exception was thrown, but KeyPairGenTraits::EncodeKey only checks IsNothing() and not FromJust(). This leads to JavaScript handles being empty, and, consequently, to a crash in V8.

The next problem is that some code paths in ToResult throw exceptions whereas others leave error handling to CryptoJob::AfterThreadPoolWork. In the test case added here, the function WritePublicKey throws an exception because the curve does not have an OID.

I am not entirely sure why these functions use Maybe<bool> as their return type. Overall, I am pretty sure we need to rework substantial parts of the error handling logic in the crypto subsystem. Hopefully, this PR can still provide a reasonable workaround to avoid crashes for now.

I tried to keep the existing behavior of ignoring any results that are equal to Just(false), meaning that returning Just(false) from ToResult causes the operation to never complete or fail.

When ToResult returns Nothing<bool>(), the AfterThreadPoolWork function now assumes that an exception was thrown, and passes it to the job callback.

In synchronous mode, both Just(false) and Nothing<bool>() are ignored, which leads to exceptions being propagated to JavaScript correctly.

@jasnell I'd love to hear your opinion. I am happy to implement and test another solution. We should clarify the semantics of the tristate Maybe<bool>.

[jasnell]

Overall, I am pretty sure we need to rework substantial parts of the error handling logic in the crypto subsystem

Big +1 to this. It's been on my todo list for a while.

[jasnell]

This LGTM but you're 100% correct that the error handling in the entire subsystem needs a good audit

[tniessen]

CI is failing on ubuntu1804_sharedlibs_openssl111_x64, most likely because it uses an older OpenSSL 1.1.1 release that exports keys on unnamed curves without curve information. (This is not desirable behavior, but not a fault on our side, and fixed in more recent OpenSSL 1.1.1 releases.)

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/35753/ (Unrelated OSX failure.)

[richardlau]

CI is failing on ubuntu1804_sharedlibs_openssl111_x64, most likely because it uses an older OpenSSL 1.1.1 release that exports keys on unnamed curves without curve information. (This is not desirable behavior, but not a fault on our side, and fixed in more recent OpenSSL 1.1.1 releases.)

We're currently using OpenSSL 1.1.1g in the sharedlibs container in the CI: https://github.com/nodejs/build/blob/90e726898ac71c9c19690cd15b5b0027901c79cb/ansible/roles/docker/templates/ubuntu1804_sharedlibs.Dockerfile.j2#L51

[tniessen]

Thanks @richardlau, test should pass now.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/35756/

[addaleax]

We should clarify the semantics of the tristate Maybe<bool>.

It's the same as in V8: We use Maybe<bool> for methods that either throw or succeed and don't return any other information, because Maybe<void> is unfortunately not a thing (i.e. it's always either Just(true) or Nothing<bool>).

But it matters that we do, because this is the way that the V8 API and our code use to tell programmers "this is a method that may throw a JS exception", and a plain bool return type doesn't communicate that.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37088/

[tniessen]

@addaleax Thank you for the explanation and sorry about the delay. I tried to fix this based on your feedback :)

[addaleax]

LGTM :)

[tniessen]

CI is green, but GitHub is not picking it up.

[Trott]

Landed in feb60f8