Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node 11.9+ Exception on openssl digital envelope routines cipher parameter evp_lib.c #27213

Closed
MasterJames opened this issue Apr 13, 2019 · 21 comments
Labels
help wanted Issues that need assistance from volunteers or PRs that need help to proceed. tls Issues and PRs related to the tls subsystem.

Comments

@MasterJames
Copy link

MasterJames commented Apr 13, 2019

Version: 11.13 crashes but in 11.8 it works (fails/waiting since 11.9)
Platform: Linux 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Subsystem: ../deps/openssl/openssl/crypto/evp/evp_lib.c:81

Error:
error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81

It is a pretty cryptic error (ha ha no pun intended).
I guess it has to do with openssl encryption maybe there is something else I should update.
I pass some encrypted stuff over a socket to nodejs from Chrome and get this error message above, while in node 11.8 there is no problem.

This is via SSL only, and might be related to elliptic encryption.
Unrelated to node execution the keys are generated this way.
openssl ecparam -genkey -name secp384r1 -out server-key.pem

but it connects okay initially I guess it is more to do with sending over the socket.

(anonymous) | @ | VM81:2026
-- | -- | --
  | emit | @ | VM13 events.js:202
  | EventEmitter.emit | @ | VM146 domain.js:439
  | (anonymous) | @ | VM14 execution.js:128
  | TLSWRAP (async) |   |  
  | init | @ | VM34 inspector_async_hook.js:21
  | emitInitNative | @ | VM19 async_hooks.js:137
  | TLSSocket._wrapHandle | @ | VM126 _tls_wrap.js:427
  | TLSSocket | @ | VM126 _tls_wrap.js:327
  | tlsConnectionListener | @ | VM126 _tls_wrap.js:750
  | emit | @ | VM13 events.js:197
  | EventEmitter.emit | @ | VM146 domain.js:439
  | onconnection | @ | VM61 net.js:1501
  | TCPSERVERWRAP (async) |   |  
  | init | @ | VM34 inspector_async_hook.js:21
  | emitInitNative | @ | VM19 async_hooks.js:137
  | createServerHandle | @ | VM61 net.js:1180
  | setupListenHandle | @ | VM61 net.js:1236
  | listenInCluster | @ | VM61 net.js:1304
  | doListen | @ | VM61 net.js:1443
  | processTicksAndRejections | @ | VM30 next_tick.js:76
  | runNextTicks | @ | VM30 next_tick.js:51
  | Module.runMain | @ | VM46 loader.js:800
  | executeUserCode | @ | VM11 node.js:526
  | startMainThreadExecution | @ | VM11 node.js:439
  | TickObject (async) |   |  
  | init | @ | VM34 inspector_async_hook.js:21
  | emitInitNative | @ | VM19 async_hooks.js:137
  | emitInitScript | @ | VM19 async_hooks.js:336
  | TickObject | @ | VM30 next_tick.js:99
  | nextTick | @ | VM30 next_tick.js:130
  | lookup | @ | VM215 dns.js:132
  | lookupAndListen | @ | VM61 net.js:1438
  | Server.listen | @ | VM61 net.js:1388
  | listenUp | @ | VM81:2066
  | begin | @ | VM81:1993


@bnoordhuis bnoordhuis added tls Issues and PRs related to the tls subsystem. v11.x labels Apr 13, 2019
@bnoordhuis
Copy link
Member

When you say "crashes" you mean it throws an exception rather than abort the process outright?

Do you have an easy way to reproduce? What does the .opensslErrorStack property of the exception object contain?

@MasterJames
Copy link
Author

MasterJames commented Apr 14, 2019

Hmm? Line 170 in events.js at throw er has er in it is message stack and Symbol kEpandStackSymbol expands with:
name bound enhanceStackTrace

So nothing found like you asked for.

Going up the stack 'this' is a TLSSocket nothing called opensslErrorStack there either.
Maybe you could direct me further to what I'm looking for otherwise I guess that the answer is 'undefined'?

This is normal just a particular state?
authorizationError: "UNABLE_TO_GET_ISSUER_CERT"
encrypted is true

I am using the npm websocket library would be a step to reproduce.
https://www.npmjs.com/package/websocket
"websocket": "^1.0.28"
is
https://github.com/theturtle32/WebSocket-Node

@MasterJames MasterJames changed the title Node 11.13 Crashs on openssl digital envelope routines cipher parameter evp_lib.c Node 11.9+ Crashs on openssl digital envelope routines cipher parameter evp_lib.c Apr 14, 2019
@MasterJames MasterJames changed the title Node 11.9+ Crashs on openssl digital envelope routines cipher parameter evp_lib.c Node 11.9+ Exception on openssl digital envelope routines cipher parameter evp_lib.c Apr 14, 2019
@bnoordhuis
Copy link
Member

How can I reproduce that 'cipher parameter error' exception that you mention?

New enough v11.x releases attach that .opensslErrorStack property, see here. The documentation in master is more comprehensive but that aside.

UNABLE_TO_GET_ISSUER_CERT means Node.js cannot verify the peer's TLS certificate because one or more certificates in its certificate chain are untrusted (e.g., unknown or self-signed CA.)

@sam-github
Copy link
Contributor

sam-github commented Apr 15, 2019

attach that .opensslErrorStack property

To be exceedingly clear, they attach it to the Error object that is thrown (not the TLSSocket).

This would likely be easy to figure out if you posted a standalone reprodution. Without it, I'm not even clear from the bug report if this is a client or a server that is throwing. Are you sure this is v11.13.0? The first tls file:line ref in the report is to _tls_wrap.js:427: var handle;, which is not a call to EventEmitter.emit.

@MasterJames
Copy link
Author

MasterJames commented Apr 17, 2019

Okay thanks for the responses I will do my best to try and help get to the bottom of this.
Now I've switched to 11.14 and the problem still exists. Works in 11.8 fails in all versions after something changed there.
I have nodejs running in debug with the chrome inspector, and also the browser with it's dev tools open.

It fails first on the 'server' side as describe. Yes subsequently there's a problem on the browser side but it's a module load time type error as the server is frozen on the EventEmitter.prototype.emit = function emit(
as it's halted at the line 170 in events.js throw er; // unhandled 'error' event

I only mentioned the TLSSocket as I searched for opensslErrorStack in 'er' and it it is not there?

If I ask JSON.stringify(er) I get an empty object {}
type er in console it displays this
Error: 140559191078720:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:

opening the err in the Scope inspector under Block is
message as shown
stack prepends "Error:" to the message
also err, inspect, stringifiedEr are listed but all undefined.

Console trace after error displayed as

(anonymous) | @ | VM83:2025
-- | -- | --
  | emit | @ | VM16 events.js:198
  | EventEmitter.emit | @ | VM148 domain.js:481
  | (anonymous) | @ | VM41 execution.js:123
  | TLSWRAP (async) |   |  
  | init | @ | VM46 inspector_async_hook.js:21
  | emitInitNative | @ | VM25 async_hooks.js:132
  | TLSSocket._wrapHandle | @ | VM128 _tls_wrap.js:447
  | TLSSocket | @ | VM128 _tls_wrap.js:349
  | tlsConnectionListener | @ | VM128 _tls_wrap.js:786
  | emit | @ | VM16 events.js:193
  | EventEmitter.emit | @ | VM148 domain.js:481
  | onconnection | @ | VM63 net.js:1504
  | TCPSERVERWRAP (async) |   |  
  | init | @ | VM46 inspector_async_hook.js:21
  | emitInitNative | @ | VM25 async_hooks.js:132
  | createServerHandle | @ | VM63 net.js:1183
  | setupListenHandle | @ | VM63 net.js:1239
  | listenInCluster | @ | VM63 net.js:1307
  | doListen | @ | VM63 net.js:1446
  | processTicksAndRejections | @ | VM26 task_queues.js:81
  | runNextTicks | @ | VM26 task_queues.js:56
  | Module.runMain | @ | VM49 loader.js:880
  | (anonymous) | @ | VM43 run_main_module.js:21
  | TickObject (async) |   |  
  | init | @ | VM46 inspector_async_hook.js:21
  | emitInitNative | @ | VM25 async_hooks.js:132
  | emitInitScript | @ | VM25 async_hooks.js:330
  | TickObject | @ | VM26 task_queues.js:104
  | nextTick | @ | VM26 task_queues.js:135
  | lookup | @ | VM218 dns.js:132
  | lookupAndListen | @ | VM63 net.js:1441
  | Server.listen | @ | VM63 net.js:1391
  | listenUp | @ | VM83:2065
  | begin | @ | VM83:1992
  | boot | @ | VM58 hotbawx.js:360
  | (anonymous) | @ | VM58 hotbawx.js:383
  | Module._compile | @ | VM49 loader.js:813
  | Module._extensions..js | @ | VM49 loader.js:827
  | Module.load | @ | VM49 loader.js:685
  | Module._load | @ | VM49 loader.js:620
  | Module.runMain | @ | VM49 loader.js:877
  | (anonymous) | @ | VM43 run_main_module.js:21

I think the message UNABLE_TO_GET_ISSUER_CERT I foolishly mentioned is just a distraction because I didn't attach a client key.
The Browser shows the Certificate Chain is Valid and it normally works that way, so that's what the Server defaults to maybe instead of a client CERT_SIGNATURE_FAILURE or what I'm not sure let's please forget about that for now. It's normal with 11.8 for me to see that at this sock.encrypted = true and sock.authorized = false in all version.

I would like to try distilling 2 years work in an example but it's server code I am unaware of any fiddle for that. If there is a way let me know but I really doubt I will be able to upload code that is reproducible.

Maybe I need a suggestion or an example of how to use sockets without this npm package 'websockets' as I guess your recent changes since 11.8 are no longer working with it I'm supposing.

Maybe this is a good question to ask?
Where is /deps/openssl/openssl/crypto/evp/evp_lib.c:81: and what changes occurred in 11.9?

If I look further into 'er' there is proto within proto but no opensslErrorStack to be found after extensive searching.
There is also Symbol kExpandStackSymbol but it is equally missing anything useful.

I can inspect that Symbol and it shows er["Symbol(kExpandStackSymbol)"] is a function but executing it fails and it's otherwise undefined so I guess it's crashed at that point.

As I move up the stack the next item is line 481 in domain.js
return Reflect.apply(eventEmit, this, args);
there 'this' is the TLSSocket

see next post ...

@MasterJames
Copy link
Author

MasterJames commented Apr 17, 2019

JSON.stringify(this) minus data buffers 'this' is the TLSSocket
{"_tlsOptions":{"secureContext":{"context":{},"alpnBuffer":{"type":"Buffer","data":[]}},"isServer":true,"server":{"_contexts":[],"requestCert":true,"rejectUnauthorized":false,"key":{"type":"Buffer","data":[]},"sessionIdContext":"1a3e89f76acfef2465334dbe8b8d7a27","_sharedCreds":{"context":{},"alpnBuffer":{"type":"Buffer","data":[]}},"_events":{},"_eventsCount":5,"_connections":7,"_handle":{"reading":false},"_usingWorkers":false,"_workers":[],"_unref":false,"allowHalfOpen":false,"pauseOnConnect":false,"httpAllowHalfOpen":false,"timeout":120000,"keepAliveTimeout":5000,"maxHeadersCount":null,"headersTimeout":40000,"_connectionKey":"6:2001:569:7561:3800:a00:28dd:fe2f:38ee:443"}},"_secureContext":{"context":{},"alpnBuffer":{"type":"Buffer","data":[8,104,116,116,112,47,49,46,49]}},"reading":true,"lastHandshakeTime":21029,"handshakes":0,"_consumed":true},"_requestCert":true,"_rejectUnauthorized":false,"timeout":0,"parser":null,"_paused":false,"_peername":{"address":"2001:569:7561:3900:202b:7822:164f:8482","family":"IPv6","port":50692},"aGnt":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36","mobile":false,"resourceURL":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/register","path":"/register","href":"/register"},"ip":"2001:569:7561:3900:202b:7822:164f:8482"}

Not that this would be much help I'm just showing you what information is available next in the stack trace.

from the doc you referenced https://nodejs.org/docs/v11.14.0/api/errors.html#errors_class_error

For crypto only, Error objects will include the OpenSSL error stack in a separate property called opensslErrorStack **if** it is available when the error is thrown.
I guess it is not available at this time for some reason.

@MasterJames
Copy link
Author

Newly released Version 12 of nodejs also not working.

@sam-github
Copy link
Contributor

Where is /deps/openssl/openssl/crypto/evp/evp_lib.c:81: and what changes occurred in 11.9

It is the node src, download and untar it, or look on github, the exact file path is in your question.

Between 11.8 and 11.9, OpenSSL was updated from 1.1.0 to 1.1.1a, you could look at the openssl changelog to see the differences, though I'd be surprised if it was illuminating.

You don't have a reproduction we can run, and I'm not really clear on your situation.

Server is node, using the websocket package.

Client is a browser? Using websockets from where?

And this happens during websocket connection? Or https? Or after connection, during data exchange?

Also, the error message suggests problems verifying a peer cert, so are you doing client auth, that is, the client has a cert?

I'd suggest trying to strip this down. I appreciate that your app is likely large, but building a tiny server that offers a websocket connection, and then seeing if it has the same problem when your browser client connects would be the first step in problem isolation. The second would be replacing the browser with a node client. Somewhere along there, you might get a working system, at which point you could ask yourself how its different from your large system where its not working.

Or, if your system doesn't work, you will end up with a repro we can run.

#27376 would probably help diagnose this, but it hasn't landed yet, much less gotten released.

@MasterJames
Copy link
Author

Thanks for the input Sam.
The websocket is from npm by that name which is https://github.com/theturtle32/WebSocket-Node
I cross posted this issue there.
theturtle32/WebSocket-Node#352
The last release of that is from Sept 2018 v1.0.28 and no response to this issue.
I was thinking it might be wise to replace that part if node team is confident it's something else, and there's no response there.

My Ubuntu version shows it's openssl version is OpenSSL 1.1.0g 2 Nov 2017
I'll check the system for updates/upgrades... 32? Now there's not enough room to upgrade on 18.04
I will have to rebuild the entire server from scratch too, to Ubuntu latest 19.04.
This one will have OpenSSL 1.1.1b Feb 2019 I'll report back after some time for rebuilding and testing.

@sam-github
Copy link
Contributor

My Ubuntu version shows it's openssl version is OpenSSL 1.1.0g 2 Nov 2017

This should not be relevant. node has its own builtin version of openssl, check that version with node -p process.versions. The result will show if you are using an ubuntu distro build of node that uses their shared openssl. If that is the case, then please reproduce the problem using node from nodejs.org downloads.

@MasterJames
Copy link
Author

-p process.version is showing the same as --version either 10.8 works or anything after that with the internal openssl 1.1.1 does not.
I figured I might have to rebuild all server/client keys with 1.1.1
The client key is requested by the server but in all cases when working or not it is not provided by the browser, so again that (I think you wondered about that in previous message), is not a factor at this stage.
I still believe I need to rebuild moving forward with ubuntu 19+ and openssl 1.1.1 (as this has been brought to my awareness) and the OS has changed enough to be a problem with my older build scripts so it seems now something called netplan in yaml is working for resolv.conf etc.
Anyway I may in the end while trying to build the core example a reproducible isolated proof of the 2 way websocket linking stuff you need I will probably replace the client websocket and the problem will go away then hopefully.

@MasterJames
Copy link
Author

I managed to get Ubuntu 19 working despite a major fault with running BIND locally and some new netplan using 127.0.0.53 instead of 127.0.0.1 etc. (but that's another story).
I also upgraded/built openssl 1.1.1b on the previous Ubuntu 18, my problem there was how the drive was filled with nodejs versions caused by 'n', (but that's another story too).

Anyway I have also got the latest version of nodejs via 'n' so 12.2.0 is also not working and more importantly the openssl version has nothing to do with this problem, which remains.

2019:05:15 02:22:34.304 •1[+2> process uncaughtException:ERR_SSL_CIPHER_PARAMETER_ERROR
2019:05:15 02:22:34.305 •1[+2> uncaughtException: [Error: 140220051621696:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:
] {
  library: 'digital envelope routines',
  function: 'EVP_CIPHER_asn1_to_param',
  reason: 'cipher parameter error',
  code: 'ERR_SSL_CIPHER_PARAMETER_ERROR'
}

@sam-github
Copy link
Contributor

A trace with --trace-tls from #27497 might reveal the cause of this without requiring a standalone repro.

@MasterJames
Copy link
Author

MasterJames commented May 19, 2019

Oh okay cool --trace-tls seems like a helpful thing thanks for the input.

2019:05:18 23:52:24.917 •1[+2> process uncaughtException:ERR_SSL_CIPHER_PARAMETER_ERROR
2019:05:18 23:52:24.918 •1[+2> uncaughtException: [Error: 140366572808000:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:
] {
  library: 'digital envelope routines',
  function: 'EVP_CIPHER_asn1_to_param',
  reason: 'cipher parameter error',
  code: 'ERR_SSL_CIPHER_PARAMETER_ERROR'
}

It sure reveals all kinds of extra output.

Here is the bit before the error...

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ChangeCipherSpec (20)
  Length = 1
Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ApplicationData (23)
  Length = 69
  Inner Content Type = Handshake (22)
    Finished, Length=48
      verify_data (len=48): C2D4265408A....1338EF16158C....78D698D7700

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ApplicationData (23)
  Length = 553
  Inner Content Type = ApplicationData (23)
Sent Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ApplicationData (23)
  Length = 204
  Inner Content Type = ApplicationData (23)

I have moved to Ubuntu 19.04 despite various issues with that in progress version. One being updated to openssl 1.1.1b which I've done on this Ubuntu 18 as well.
This output came from the error running nodejs 12.2.0

It seems like I should switch to the internal TLS tools and dump turtle's websocket library, so it's a breaking change that was inevitable I suppose.
It will be as the new documentation suggests best with TLS 1.3 ECDHE or what exactly will work with the browser at this time etc. I will have to struggle again to see. Hopefully it goes more easily.
https://nodejs.org/api/tls.html#tls_perfect_forward_secrecy

I think it might be something to do with the particular cipher and as it is complicated to get all the bit's and pieces working together hard to track down what breaks when other then it crashes on a version change. I believe this is what it is using in this case cipher_suite {0x13, 0x02} TLS_AES_256_GCM_SHA384

@MasterJames
Copy link
Author

MasterJames commented May 23, 2019

I've tried without success to use 'tls' sockets to repair this issue and as of node 12.3.1 this is still not working.
I thought it might be useful to say that with 11.8 Chrome shows this as the security connection that works with it that's been established.

The connection to this site is encrypted and authenticated using TLS 1.2, ECDHE_ECDSA with X25519, and AES_128_GCM.

Based on the output from the new trace-tls flag it would seem TLS 1.3 is not being negotiated, but the same keys can make a 'tls' connection with 1.3 it's just the socket is nothing like how 'https' works.

See here for that attempt...
https://stackoverflow.com/questions/56252894/nodejs-12-2-tls-compatibility-with-https

@targos targos added v12.x and removed v11.x labels Jul 20, 2019
@MasterJames
Copy link
Author

Still errors in 12.10

2019:09:08 00:42:31.513 •1[+2> process uncaughtException:ERR_SSL_CIPHER_PARAMETER_ERROR
2019:09:08 00:42:31.514 •1[+2> uncaughtException: [Error: 139657234708352:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:
] {
  library: 'digital envelope routines',
  function: 'EVP_CIPHER_asn1_to_param',
  reason: 'cipher parameter error',
  code: 'ERR_SSL_CIPHER_PARAMETER_ERROR'
}

@MasterJames
Copy link
Author

Just tried 13.1.0 and unfortunately it still errors in this way.

uncaughtException: [Error: 140172522628992:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:
] {
  library: 'digital envelope routines',
  function: 'EVP_CIPHER_asn1_to_param',
  reason: 'cipher parameter error',
  code: 'ERR_SSL_CIPHER_PARAMETER_ERROR'
}

@MasterJames
Copy link
Author

MasterJames commented Dec 9, 2019

Same problem in 13.3.0

uncaughtException: Error: 139737098114944:error:060CC07A:digital envelope routines:EVP_CIPHER_asn1_to_param:cipher parameter error:../deps/openssl/openssl/crypto/evp/evp_lib.c:81:

@targos targos added the v13.x label Jan 14, 2020
@bnoordhuis
Copy link
Member

I believe I've asked about it before but how can I reproduce (easily)? A small test case using only builtin modules is preferred; anything that depends on 100s or 1,000s of lines of third-party code is unlikely to get looked at.

@BridgeAR BridgeAR removed the v13.x label May 1, 2020
@MasterJames
Copy link
Author

MasterJames commented May 6, 2020

Hi I was just checking what's happening here. It still has not been magically fixed. Just tested versions 12.16.3 and 14.2.0 with the same results.
I figure after years or now I've forgotten how long I spend getting things working before.
It certainly is not trivial or easy to reproduce or strip down here.

I can say I am using this npm websocket package and thought maybe that is part of the problem.
https://www.npmjs.com/package/websocket
I thought it might be related to that package, since it is an integral part of my system and how those parts all work together.
I tried to report the problem there, and nothing happened for a long time. I see newer versions there now but the problem remains the same.
One could try to see if that package and it's basic demo work via instructions from it's npm site.
I will report back when I know more.

@jasnell jasnell added the help wanted Issues that need assistance from volunteers or PRs that need help to proceed. label Jun 26, 2020
@targos targos removed the v12.x label Apr 8, 2022
@bnoordhuis
Copy link
Member

Seeing there's been no follow-up in almost two years, can I suggest we close this issue?

@targos targos closed this as completed Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Issues that need assistance from volunteers or PRs that need help to proceed. tls Issues and PRs related to the tls subsystem.
Projects
None yet
Development

No branches or pull requests

6 participants