Upgrade to OpenSSL-1.1.0h #19794

shigeki · 2018-04-04T13:28:26Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

This has very big patches due to the source updates from OpenSSL-1.0.2o to 1.1.0h and generated asm files which are not necessary to be reviewed in this PR.

In order to see the differences easily for reviewers, I made two branches in which those changes are removed. Please refer the branch diffs as below in reviewing for it has just several thousands kb diffs.

shigeki/node@raw_upgrade_openssl110h...shigeki:no_archfiles_upgrade_openssl110h

Note that this has a new build requirement of assembler for asm support. Especially nasm is needed to build Windows that is required by OpenSSL. Please refer 9e38498 for details.

CC @nodejs/crypto @nodejs/tsc

The following is the description written in deps/openssl/README.md

This has a new binding scheme in builing OpenSSL-1.1.0 library with
Node.js. OpenSSL-1.1.0 uses a new build system with perl for various
supported platforms. See openssl/Configurations/README and
openssl/Configurations/README.design in the OpenSSL source for
details.

In order to build OpenSSL library without perl in the build of Node.js
for various supported platforms, platform dependent files (e.g. asm
and header files ) are pre-generated and stored into the
config/archs directory.

config/Makefile and config/generate_gypi.pl

Makefile has supported platform list and generates and copies
platform dependent files (e.g. asm files) into arch directory with
generate.pl. Platform dependent gypi files also created obtaining
build information from configdata.pm that is generated with
Configure in the OpenSSL build system.

For Windows, Configure generates makefile that is only available to
nmake command. config/Makefile_VC-WIN32 and
config/Makefile_VC-WIN64A are made created by hand for the use of
GNU make. If make rules or targets are changed in the version up of
OpenSSL, they should be also updated.
gyp and gypi files (openssl*.{gyp,gypi})

openssl.gyp has two targets of openssl and openssl-cli referred
from node.gyp. They includes asm and no_asm gypi files with arch
dependent gypi according to its build options and platforms . The
gyp data which is common with asm and no_asm are stored in
openssl_common.gypi.
header files (config/*.{h,h.tmpl})

bn_conf.h, dso_conf.h and opensslconf.h are platform dependent
in the OpenSSL sources. They are replaced with config/*.h.tmpl
files to include the file in the ../../../config/ and referred to
each arch files that depends on asm and no-asm option.

Supported architectures for use of ASM

Here is a list of supported architectures for use of ASM in OpenSSL.

--dest-os	--dest-cpu	OpenSSL target arch	CI
aix	ppc	aix-gcc	o
aix	ppc64	aix64-gcc	o
linux	ia32	linux-elf	o
linux	x32	linux-x32	-
linux	x64	linux-x86_64	o
linux	arm	linux-armv4	o
linux	arm64	linux-aarch64	o
linux	ppc	linux-ppc	o
linux	ppc64	linux-ppc64	o
linux	ppc64	linux-ppc64le	o
linux	s390	linux32-s390x	o
linux	s390x	linux64-s390x	o
mac	ia32	darwin-i386-cc	-
mac	x64	darwin64-x86-cc	o
win	ia32	VC-WIN32	-
win	x64	VC-WIN64A	o
solaris	ia32	solaris-x86-gcc	o
solaris	x64	solaris64-x86_64-gcc	o
freebsd	ia32	BSD-x86	-
freebsd	x64	BSD-x86_64	o
openbsd	ia32	BSD-x86	-
openbsd	x64	BSD-x86_64	-
others	others	linux-elf	-

These are listed in config/Makefile.
Please refer config/opensslconf_asm.h for details.

Upgrading OpenSSL

Please refer config/README.md .

vsemozhetbyt · 2018-04-04T13:59:19Z

doc/api/tls.md

-[SSL_CTX_set_timeout]: https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_timeout.html
-[SSL_METHODS]: https://www.openssl.org/docs/man1.0.2/ssl/ssl.html#DEALING-WITH-PROTOCOL-METHODS
+[SSL_CTX_set_timeout]: https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_timeout.html
+[SSL_METHODS]: https://www.openssl.org/docs/man1.1.0/ssl/ssl.html#DEALING-WITH-PROTOCOL-METHODS


It seems the new hash needs to be #Dealing-with-Protocol-Methods

Fixed the hash link. I resolved some conflicts but it needs a more fix . Fixed in 9cf8473b2c75f043f81cf82785d43b4e03dda967

vsemozhetbyt · 2018-04-04T14:04:51Z

BUILDING.md

@@ -88,6 +88,23 @@ Depending on host platform, the selection of toolchains may vary.

 * Visual Studio 2017 or the Build Tools thereof

+#### OpenSSL asm support
+
+OpenSSL-1.1.0 requires the following asssember version for use of asm


asssember -> assembler

Fixed in 07d5ac52ee66ada4e4a716edb0d4880360b122d6.

vsemozhetbyt · 2018-04-04T14:07:29Z

BUILDING.md

+
+Otherwise, `--openssl-no-asm` is added with warning in configure.
+
+*Note:* The forthcoming OpenSSL-1.1.1 will require more higher


more higher -> higher

vsemozhetbyt · 2018-04-04T14:14:14Z

deps/openssl/README.md

@@ -0,0 +1,76 @@
+This has a new binding scheme in builing OpenSSL-1.1.0 library with


builing -> building

Fixed in cb6aee5a3d8201fb656a8e971b3741d85976a9d0.

vsemozhetbyt · 2018-04-04T14:14:44Z

deps/openssl/README.md

@@ -0,0 +1,76 @@
+This has a new binding scheme in builing OpenSSL-1.1.0 library with
+Node.js. OpenSSL-1.1.0 uses a new build system with perl for various


perl -> Perl?

vsemozhetbyt · 2018-04-04T14:15:35Z

deps/openssl/README.md

+`openssl/Configurations/README.design` in the OpenSSL source for
+details.
+
+In order to build OpenSSL library without perl in the build of Node.js


perl -> Perl?

vsemozhetbyt · 2018-04-04T14:29:58Z

deps/openssl/README.md

+
+  `openssl.gyp` has two targets of openssl and openssl-cli referred
+  from `node.gyp`. They includes asm and no_asm gypi files with arch
+  dependent gypi according to its build options and platforms . The


platforms . -> platforms.

vsemozhetbyt · 2018-04-04T14:30:33Z

deps/openssl/README.md

+  from `node.gyp`. They includes asm and no_asm gypi files with arch
+  dependent gypi according to its build options and platforms . The
+  gyp data which is common with asm and no_asm are stored in
+  openssl_common.gypi.


openssl_common.gypi -> `openssl_common.gypi`?

vsemozhetbyt · 2018-04-04T14:32:47Z

deps/openssl/README.md

+  `bn_conf.h`, `dso_conf.h` and `opensslconf.h` are platform dependent
+  in the OpenSSL sources. They are replaced with `config/*.h.tmpl`
+  files to include the file in the `../../../config/` and referred to
+  each arch files that depends on asm and no-asm option.


each arch files -> each arch file?

vsemozhetbyt · 2018-04-04T14:38:18Z

deps/openssl/README.md

+  | linux     | arm64      | linux-aarch64        | o   |
+  | linux     | ppc        | linux-ppc            | o   |
+  | linux     | ppc64      | linux-ppc64          | o   |
+  | linux     | ppc64      | linux-ppc64le        | o   | (node_byteoder: little)


I do not see (node_byteoder: little) in the rendered table. Should it be there? Is it parsed as a hidden comment?

This is a note. I fixed it as a footnote.

vsemozhetbyt · 2018-04-04T14:41:37Z

deps/openssl/README.md

+
+### Upgrading OpenSSL
+
+Please refer [config/README.md](config/README.md) .


) . -> ).

vsemozhetbyt · 2018-04-04T14:52:53Z

Are there any more docs to review except doc/api/crypto.md, doc/api/tls.md, BUILDING.md, and deps/openssl/README.md?

Should deps/openssl/doc/UPGRADING.md and deps/openssl/config/README.md be reviewed or are they upstream docs?

shigeki · 2018-04-04T15:00:01Z

@vsemozhetbyt Thanks for fixing my English. Please review deps/openssl/config/README.md . deps/openssl/doc/UPGRADING.md was removed.

shigeki · 2018-04-04T15:23:13Z

CI of https://ci.nodejs.org/job/node-test-pull-request/14044/ will be fine except ubuntu1604_sharedlibs_openssl102_x64. It is to be fixed in nodejs/build#1210.

vsemozhetbyt · 2018-04-04T15:38:31Z

deps/openssl/README.md

@@ -53,7 +53,7 @@ Here is a list of supported architectures for use of ASM in OpenSSL.
  | linux     | arm64      | linux-aarch64        | o   |
  | linux     | ppc        | linux-ppc            | o   |
  | linux     | ppc64      | linux-ppc64          | o   |
-  | linux     | ppc64      | linux-ppc64le        | o   | (node_byteoder: little)
+  | linux     | ppc64      | linux-ppc64le        | o   |(*1)


It is not in the table still:

https://github.com/shigeki/node/blob/cb6aee5a3d8201fb656a8e971b3741d85976a9d0/deps/openssl/README.md#supported-architectures-for-use-of-asm

Maybe it should be placed in a cell or its own column should be added?

Sorry, I missed to view and check markdown. Fixed to include it in the cell as f8cdc0f.

vsemozhetbyt

Sorry for nits)

vsemozhetbyt · 2018-04-04T16:13:24Z

deps/openssl/config/README.md

+## Upgrading OpenSSL-1.1.0
+
+### Requirements
+- Linux enviroment (Only CentOS7.1 and Ubuntu16 are tested)


enviroment -> environment

vsemozhetbyt · 2018-04-04T16:14:40Z

deps/openssl/config/README.md

+- Linux enviroment (Only CentOS7.1 and Ubuntu16 are tested)
+- perl5
+- nasm (http://www.nasm.us/)  The version of 2.11 or higher is needed.
+- GNU as in binutils. The version of 2.26 or higher is needed.


as -> `as`

vsemozhetbyt · 2018-04-04T16:18:57Z

deps/openssl/config/README.md

+$ nasm -v
+NASM version 2.11.08
+```
+### 1. Obtain and extract new OpenSS sources.


sources. -> sources for consistency with other headings?

vsemozhetbyt · 2018-04-04T16:24:03Z

deps/openssl/config/README.md

+```
+### 1. Obtain and extract new OpenSS sources.
+
+Get a new source from  https://www.openssl.org/source/ and extract


Get a new source... and extract them

-> Get a new source... and extract all files?
or
-> Get new source files... and extract them?

Fixed. My choice is the former.

vsemozhetbyt · 2018-04-04T16:26:50Z

deps/openssl/config/README.md

+$ nasm -v
+NASM version 2.11.08
+```
+### 1. Obtain and extract new OpenSS sources.


OpenSS -> OpenSSL

vsemozhetbyt · 2018-04-04T16:46:26Z

deps/openssl/config/README.md

+*Note*: On Windows, OpenSSL Configure generate `makefile` that can be
+ used for `nmake` command. The `make` command in the step 2 above uses
+  `Makefile_VC-WIN64A` and `Makefile_VC-WIN32` that is manually
+ created. When sources files or build options are updated in Windows,


sources files -> source files?

vsemozhetbyt · 2018-04-04T16:47:07Z

deps/openssl/config/README.md

+ used for `nmake` command. The `make` command in the step 2 above uses
+  `Makefile_VC-WIN64A` and `Makefile_VC-WIN32` that is manually
+ created. When sources files or build options are updated in Windows,
+ it needs to change the these two Makefiles by hand. If you are not


the these -> these

vsemozhetbyt · 2018-04-04T16:48:31Z

deps/openssl/config/README.md

+ created. When sources files or build options are updated in Windows,
+ it needs to change the these two Makefiles by hand. If you are not
+ sure, please ask @shigeki for details.
+### 4. Commits and make test


4. -> 5.

Commits -> Commit?

vsemozhetbyt · 2018-04-04T16:50:06Z

deps/openssl/config/README.md

+ sure, please ask @shigeki for details.
+### 4. Commits and make test
+
+Updates all architecture dependent files. Do not forget to git add or remove


Updates -> Update or This updates?

Fixed. My choice is the former.

vsemozhetbyt · 2018-04-04T16:52:05Z

deps/openssl/config/README.md

+     deps: update archs files for OpenSSL-1.1.0
+```
+
+Finally, build Node and run test it.


run test it -> run tests or test it?

Fixed. My choice is the former.

jasnell

rubber-stamp LGTM

rvagg · 2018-04-05T01:09:02Z

@shigeki so I take it from nodejs/build#1210 that you're not able to maintain backward compatibility with 1.0.2? As per nodejs/TSC#479 the hope was that we would maintain the ability to still compile against 1.0.2 in the same way that Node 8/9 can compile against 1.1.0 now. The problem is going to be with Linux distros that insist on dynamically compiling OpenSSL against what they ship, so if they don't ship OpenSSL 1.1.0 then they won't be able to ship Node 10. I'm not sure which distros, if any, that is actually going to impact, however.

shigeki · 2018-04-05T02:30:58Z

@rvagg It can be possible to be compatible between 1.0.2 and 1.1.0 at this moment. But I am pessimistic to maintain it until the EOLS of 1.0.2 at the end of 2019 unless we keep freezing new features of OpenSSL-1.1.x. Node8/9 could do it since we did not add no new crypto/tls features specific to 1.1.0. I'm fearing that we lose a chance to remove the support of 1.0.2 in Node10 in the future.

If it needs a large discussion, I can make back compatibilities with 1.0.2 in this PR and submit a separated PR or issues to remove 1.0.2 support.

rvagg · 2018-04-05T03:04:30Z

If it's not too much work I'd like to see 1.0.2 support maintained @shigeki. I think liberal use of "this feature is not supported by 1.0.2" runtime errors is fine so we can move forward embracing 1.1.x features and just make them not available to builds against 1.0.2, similar to how we've handled FIPS and how others have handled LibreSSL support.

@bnoordhuis @indutny could either of you weigh in on this? Perhaps it's just not worth it and we should make a clean break?

@kapouer are you available to offer an opinion here since you have a foot in the dynamic linking camp? If Node 10 goes out with no ability to compile against 1.0.2 how many yelps are we going to hear?

shigeki

@vsemozhetbyt Thanks for reviewing my English. I fixed in 9a62a0c.

vsemozhetbyt

Docs LGTM)

jasnell · 2018-04-05T15:04:27Z

Btw, if the plan is for this to go in to 10.0.0, it should land no later than April 10th. After that, I'll only pull in tsc approved semver-majors

rvagg · 2018-04-06T03:45:37Z

OK, no response to my 1.0.2 proposition so how about this: let's just land pure 1.1.0 support and ditch 1.0.2 support. It's something that could be added in afterward, during 10.x Current if enough people yelp.

I'll go ahead and pull 1.0.2 out of CI for 10+ and I'll update nodejs/TSC#479 to change the plan and get that in front of the TSC.

@shigeki anything else we need to get this landed?

rvagg · 2018-04-06T04:14:24Z

configure

-  o['variables']['node_use_openssl'] = b(not options.without_ssl)
-  o['variables']['node_shared_openssl'] = b(options.shared_openssl)
-  o['variables']['openssl_no_asm'] = 1 if options.openssl_no_asm else 0
+  variables = o['variables'];


semicolon probably should go

Fixed. Thanks.

rvagg · 2018-04-06T05:22:06Z

a near perfect CI run @ https://ci.nodejs.org/job/node-test-commit/17482/, one failure, a known Raspberry Pi problem unrelated to this 👍 great work @shigeki

shigeki · 2018-04-06T14:45:39Z

I will wait for anyone's reviews until the next Monday night in JST.

Notable changes: * deps: add s390 asm rules for OpenSSL-1.1.1 (Shigeki Ohtsu) [#19794](#19794) * src: add .code and SSL specific error properties (Sam Roberts) [#25093](#25093) * tls: * add --tls-min-v1.2 CLI switch (Sam Roberts) [#26951](#26951) * supported shared openssl 1.1.0 (Sam Roberts) [#26951](#26951) * revert default max toTLSv1.2 (Sam Roberts) [#26951](#26951) * revert change to invalid protocol error type (Sam Roberts) [#26951](#26951) * support TLSv1.3 (Sam Roberts) [#26209](#26209) * add code for ERR\_TLS\_INVALID\_PROTOCOL\_METHOD (Sam Roberts) [#24729](#24729) PR-URL: #27314

Ported from OpenSUSE:nodejs8-8.17.0-lp152.147.1:openssl_1_1_1.patch Original commit message: Backport OpenSSL 1.1.1 support, mostly be disabling TLS 1.3 Upstream commits: commit 8dd8033 Author: Shigeki Ohtsu <ohtsu@ohtsu.org> Date: Wed Sep 12 17:34:24 2018 +0900 tls: workaround handshakedone in renegotiation `SSL_CB_HANDSHAKE_START` and `SSL_CB_HANDSHAKE_DONE` are called sending HelloRequest in OpenSSL-1.1.1. We need to check whether this is in a renegotiation state or not. Backport-PR-URL: nodejs#26270 PR-URL: nodejs#25381 Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Shigeki Ohtsu <ohtsu@ohtsu.org> commit 161dca7 Author: Sam Roberts <vieuxtech@gmail.com> Date: Wed Nov 28 14:11:18 2018 -0800 tls: re-define max supported version as 1.2 Several secureProtocol strings allow any supported TLS version as the maximum, but our maximum supported protocol version is TLSv1.2 even if someone configures a build against an OpenSSL that supports TLSv1.3. Fixes: nodejs#24658 PR-URL: nodejs#25024 Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Partial port, remain compatible with 1.0.2: commit 970ce14 Author: Shigeki Ohtsu <ohtsu@ohtsu.org> Date: Wed Mar 14 14:26:55 2018 +0900 crypto: remove deperecated methods of TLS version All version-specific methods were deprecated in OpenSSL 1.1.0 and min/max versions explicitly need to be set. This still keeps comptatible with JS and OpenSSL-1.0.2 APIs for now. crypto, constants: add constant of OpenSSL-1.1.0 Several constants for OpenSSL-1.1.0 engine were removed and renamed in OpenSSL-1.1.0. This added one renamed constant in order to have a compatible feature with that of OpenSSL-1.0.2. Other missed or new constants in OpenSSL-1.1.0 are not yet added. crypto,tls,constants: remove OpenSSL1.0.2 support This is semver-majar change so that we need not to have compatibilities with older versions. Fixes: nodejs#4270 PR-URL: nodejs#19794 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Signed-off-by: Su Baocheng <baocheng.su@siemens.com>

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: nodejs#1529 Refs: nodejs#10859 Refs: nodejs#19794 Refs: nodejs#38116

This function was introduced in 2684c90 as an internal helper function. The C++ implementation became a no-op in a57e2f2 when building against OpenSSL 1.1.0 (instead of OpenSSL 1.0.2), and eventually became a no-op in all supported OpenSSL versions in 970ce14. Finally, eb20447 removed the only call site of setFreeListLength (which was already a no-op at that point). Refs: #1529 Refs: #10859 Refs: #19794 Refs: #38116 PR-URL: #44300 Reviewed-By: Feng Yu <F3n67u@outlook.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

These constants have not existed since OpenSSL 1.1.0 reached EOL a few years ago. Refs: nodejs#19794