Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Platform requirements for Node.js 18 #2815

Closed
2 of 4 tasks
Tracked by #42139
richardlau opened this issue Nov 29, 2021 · 30 comments
Closed
2 of 4 tasks
Tracked by #42139

Platform requirements for Node.js 18 #2815

richardlau opened this issue Nov 29, 2021 · 30 comments

Comments

@richardlau
Copy link
Member

richardlau commented Nov 29, 2021

Node.js 18 is due out in April 2022 and is planned to last until April 2025 (https://github.com/nodejs/Release/blob/main/schedule.json). We need to plan out any required toolchain/hardware changes we want/need to make.

cc @nodejs/build

What we build and test on

From https://github.com/nodejs/node/blob/master/BUILDING.md#official-binary-platforms-and-toolchains

Binary package Platform and Toolchain Decision done
aix-ppc64 AIX 7.2 TL04 on PPC64BE with GCC 8 move minimum hardware to Power 8
darwin-x64 macOS 10.15, Xcode Command Line Tools 11 with -mmacosx-version-min=10.13
darwin-arm64 (and .pkg) macOS 11 (arm64), Xcode Command Line Tools 12 with -mmacosx-version-min=10.13
linux-arm64 CentOS 7 with devtoolset-8 / GCC 8 Need to migrate (#2741)
linux-armv7l Cross-compiled on Ubuntu 18.04 x64 with custom GCC toolchain
linux-ppc64le CentOS 7 with devtoolset-8 / GCC 8 Need to migrate (#2741) ✔️
linux-s390x RHEL 7 with devtoolset-8 / GCC 8 Need to migrate (#2741) ✔️
linux-x64 CentOS 7 with devtoolset-8 / GCC 8 Need to migrate (#2741) ✔️
win-x64 and win-x86 Windows 2012 R2 (x64) with Visual Studio 2019

Compiler levels

Operating System Compiler Versions
Linux GCC >= 8.3
Windows Visual Studio >= 2019 with the Windows 10 SDK on a 64-bit host
macOS Xcode >= 11 (Apple LLVM >= 11)

Actions

@rvagg
Copy link
Member

rvagg commented Nov 29, 2021

Any reason to move the minimum deployment targets for macOS on from macOS 10.13?

No really good reason, but I'm wondering if we should just adopt a practice of incrementing this by 1 at this time every year. Then we don't have to do big jumps like we've done a couple of times. Being a few major versions behind seems reasonable enough I think and I don't recall any complaints (ever) from users about macOS compatibility—in fact the only complaint I recall was a Node core dev who insisted on running a couple of versions behind the latest macOS, but that was within the value we set for this.

RHEL 8

What are our IaaS options for this? Do our providers have RHEL options, or can we switch a CentOS machine into RHEL mode somehow? Or should we be considering containierising this build?

@targos
Copy link
Member

targos commented Nov 30, 2021

It might be good to use a higher Windows server version even if it's not based on Windows 11. 2012 R2 is quite old.

@richardlau
Copy link
Member Author

RHEL 8

What are our IaaS options for this? Do our providers have RHEL options, or can we switch a CentOS machine into RHEL mode somehow? Or should we be considering containierising this build?

Not ideal. Unsurprisingly IBM Cloud offer RHEL options but neither Digital Ocean nor Rackspace currently do (#2741 (comment)) although they do provide Alma/Rocky. Red Hat do provide a tool for switching (https://www.redhat.com/en/blog/introduction-convert2rhel-now-officially-supported-convert-rhel-systems-rhel) which I've not tested yet. Containerization or custom VM images are other things on my list of things to investigate.

I have an action to obtain a RHEL subscription for the Node.js project (possibly via https://www.redhat.com/en/blog/extending-no-cost-red-hat-enterprise-linux-open-source-organizations) and see if we can make it work for us (i.e. the build wg). Having a subscription would immediately simplify set up for the existing RHEL7 LinuxOne machines (at the moment we have to manually copy rpms over to set up the compiler toolchain).

@AshCripps
Copy link
Member

No really good reason, but I'm wondering if we should just adopt a practice of incrementing this by 1 at this time every year.

+1 on this as apple can be quite aggressive on dropping support for its OSes.

@sxa
Copy link
Member

sxa commented Nov 30, 2021

Particularly with the new Ampere Altra systems being available, we should also consider whether we need to continue to cross-compile for arm32 or whether to use an arm32 docker container on one of the Altrea aarch64 hosts for building to reduce the complexity and increase commonality with how other platforms are built.

@mhdawson
Copy link
Member

mhdawson commented Dec 3, 2021

To add to @richardlau's comment above our providers that don't have RHEL8 images built in do allow you to bring your own image(I think that is what @richardlau meant by customer VM images). I think that should be relatively easy to do and we should be able to get the images under the program to let Open Source projects use RHEL.

@mhdawson
Copy link
Member

@joaocgreis thoughts on the Windows levels?

@rvagg
Copy link
Member

rvagg commented Dec 14, 2021

From the trusty https://en.wikipedia.org/wiki/Glibc#Version_history table, switching to RHEL8 would give us a minimum glibc of 2.28 which ships with Ubuntu 18.10, Debian 10 (Buster), and Fedora 29. The only concern there is that it's later than 18.04, so we're ruling out those systems. But I think that's reasonable for a new release of Node.js - 18.04 users will either need to stick to older Node.js or compile themselves using a newer toolchain from ppa.

@rvagg
Copy link
Member

rvagg commented Dec 14, 2021

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

@targos
Copy link
Member

targos commented Feb 23, 2022

Here are some end of support dates for Windows and Visual Studio. Found on https://docs.microsoft.com/en-us/lifecycle/products/

I did not count "Extended Security Update Year X", as the documentation says:

The Extended Security Update (ESU) program is a last resort option for customers who need to run certain legacy Microsoft products past the end of support.

@targos
Copy link
Member

targos commented Feb 24, 2022

Here's my suggestion based on ^

nodejs/node#42105

@targos
Copy link
Member

targos commented Feb 26, 2022

We found out in nodejs/node-v8#220 that V8 now uses a kernel feature that is only available in Linux 3.17 (glibc 2.27).

@rvagg
Copy link
Member

rvagg commented Feb 27, 2022

@targos have we committed to that version of V8 for Node 18?

If so, that messes with our Linux strategy quite a bit, we'll have to bump minimum build requirements to:

  • Ubuntu 18+
  • RHEL / CentOS 8+
  • Debian 10+

Maybe devtoolset gets us out of some trouble for runtime requirements for our shipped binaries, but I suppose we need to test that they're compiling—do we have any runs of this through the release pipeline yet for v8-canary builds, or are we still at the integration phase?

@richardlau
Copy link
Member Author

@targos have we committed to that version of V8 for Node 18?

If so, that messes with our Linux strategy quite a bit, we'll have to bump minimum build requirements to:

* Ubuntu 18+

* RHEL / CentOS 8+

* Debian 10+

Maybe devtoolset gets us out of some trouble for runtime requirements for our shipped binaries, but I suppose we need to test that they're compiling—do we have any runs of this through the release pipeline yet for v8-canary builds, or are we still at the integration phase?

For that particular V8: https://ci.nodejs.org/job/node-test-commit/52045/
Basically any CentOS 7/RHEL 7 Linux failed but the other Linux distributions built/tested successfully. Ubuntu 18.04 uses glibc 2.27 (where the API being used by V8 was introduced) and passed.

I'm planning to get some RHEL8 x64 instances set up for the CI this coming week to get some test binaries to try out on the various other distributions. I'm fairly certain though that we'll need at least glibc 2.28 to run the binaries, which as noted by you before (#2815 (comment)) would look to rule out Ubuntu 18.04. AFAIK we don't have glibc checks at build time though, so anyone on those could still attempt to build from source.

We're going to have to move off CentOS 7 for Node.js 18 unless we want to be running on an unsupported OS at the tail end of Node.js 18's expected lifespan, so a glibc bump is going to happen (CentOS 7/RHEL 7 are glibc 2.17).

@targos
Copy link
Member

targos commented Feb 28, 2022

have we committed to that version of V8 for Node 18?

We have always tried to include the most recent possible V8 version available on release day.

do we have any runs of this through the release pipeline yet for v8-canary builds

Latest canary release build has this: https://ci-release.nodejs.org/job/iojs+release/8275/
Integration is done with GitHub actions, which are able to build it.

@mhdawson
Copy link
Member

@rvagg I think we were already planning on moving up to RHEL 8 for 18.x due to the EOL for REHL 7.

richardlau added a commit that referenced this issue Mar 1, 2022
Extend Ansible and Jenkins scripts for Red Hat Enterprise Linux 8.
Also add new `release-builder` role, for setting up ssh config and
keys to upload to the staging server, and changes to make the
playbook idempotent.

Refs: #2741
Refs: #2815
richardlau added a commit that referenced this issue Mar 8, 2022
Add RHEL 8 x64 instances, including repurposing some obsolete CentOS 6
instances which are no longer used by any CI jobs.

Add:
- release-digitalocean-rhel8-x64-1
- release-ibm-rhel8-x64-1
- release-ibm-rhel8-x64-2
- test-digitalocean-rhel8-x64-1
- test-ibm-rhel8-x64-1
- test-ibm-rhel8-x64-2
- test-ibm-rhel8-x64-3

Remove:
- release-digitalocean-centos6-x86-1
- release-softlayer-centos6-x64-1
- test-digitalocean-ubuntu1404-x86-1
- test-softlayer-centos6-x64-1
- test-softlayer-centos6-x64-2

Refs: #2741
Refs: #2815
richardlau added a commit that referenced this issue Mar 10, 2022
Add new ppc64le RHEL 8 instances. Repurpose test-osuosl-centos7-ppc64_le-4
as an additional RHEL 8 instance.

Also ensure time synchronization via `chrony` on RHEL 8, and add a handler
to run when packages are updated to sync with Red Hat Subscription Manager.

Refs: #2741
Refs: #2815
@richardlau
Copy link
Member Author

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

After April 2023 Ubuntu 18.04 requires an Ubuntu Advantage subscription to get further updates, which is two years before Node.js 18 goes End-of-Life: https://ubuntu.com/about/release-cycle
Do we want to move to something else? April 2023 is also a year before Node.js 16 goes End-of-Life, so we're potentially exposed there.

@richardlau
Copy link
Member Author

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

After April 2023 Ubuntu 18.04 requires an Ubuntu Advantage subscription to get further updates, which is two years before Node.js 18 goes End-of-Life: https://ubuntu.com/about/release-cycle Do we want to move to something else? April 2023 is also a year before Node.js 16 goes End-of-Life, so we're potentially exposed there.

Speaking with @sxa it sounds like it might not matter so much which OS is being used and it's more the cross compiler, with the GLIBC version being linked against coming from the cross compiler in https://github.com/rvagg/rpi-newer-crosstools.

@richardlau
Copy link
Member Author

Having said that, https://ci.nodejs.org/job/node-cross-compile/37193/nodes=cross-compiler-ubuntu1804-armv7-gcc-8/ failed for #2815 (comment) so our cross compiler is linking against a glibc older than 2.27. Although the particular commit was patched out (nodejs/node@ec77f2d) I have no idea if we'll just get another down the line. V8 use clang themselves so I don't think they have a notion of minimum glibc support.

@rvagg
Copy link
Member

rvagg commented Mar 15, 2022

btw, I do have a commitment from Canonical to sponsor us with some ESM licenses for our infra, I've just failed to follow up on that due to being very occupied with other things! it was mainly interesting for some of our infra that's a bit stuck on older versions and are painful to upgrade but we can use it in our testing infra too.

@targos
Copy link
Member

targos commented Mar 17, 2022

V8 just landed v8/v8@4e81f25, which reverts the glibc bump requirement on that side.

@targos
Copy link
Member

targos commented Mar 18, 2022

Since it's a runtime check, we would still have to compile using a newer version of glibc:

https://ci.nodejs.org/job/node-cross-compile/37428/nodes=cross-compiler-ubuntu1804-armv7-gcc-8/consoleFull

Is it going to happen before the release of v18.0.0 or I should reintroduce the workaround?

@richardlau
Copy link
Member Author

I don't know what we're going to do about the cross-compiler for armv7 -- the glibc compiled against is the one in the cross compiler and not the host (ubuntu 18.04 in this case).
arm64 will be moving (it's dependent on getting another release machine from Equinix which is in progress).

@richardlau
Copy link
Member Author

I don't know what we're going to do about the cross-compiler for armv7 -- the glibc compiled against is the one in the cross compiler and not the host (ubuntu 18.04 in this case).

I had a chat with @sxa and he suggested we could try building armv7 in an armv7 container on the arm64 machine(s) and avoid cross-compilation altogether.

nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Mar 18, 2022
Update the macOS deployment target for Node.js 18 onwards.

PR-URL: #42292
Refs: nodejs/build#2815
Refs: libuv/libuv#3406
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Ash Cripps <acripps@redhat.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
richardlau added a commit that referenced this issue Mar 22, 2022
We're starting to get upstream changes that require later glibc
than is available on CentOS 7. In addition, our nightly (and
eventually Node.js 18 onwards releases) binaries require later
glibc (2.28) which will not run on CentOS 7.

Refs: #2815
@rvagg
Copy link
Member

rvagg commented Mar 28, 2022

Not cross-compiling is certainly an option, if you want to pursue that then maybe that'll be a decent solution. But, because it's possible, I went ahead and made a copy of the gcc 8.3.0 cross-compiler but with glibc 2.27 rather than 2.24. I don't have time to try it out right now but if someone does, it's here on a branch: rvagg/rpi-newer-crosstools#1

Unfortunately it's a bit involved to wire it up -- if it's still working like last time I touched it, something like:

  • Edit ansible/roles/docker/files/cc-selector.sh to add it as a new option
  • Add a new label to the cross-compiling nodes in Jenkins (ci and ci-release both use labels for selection)
  • Edit jenkins/scripts/VersionSelectorScript.groovy to shunt the right Node.js version to the new label
  • (edited): Re-deploy the cc-selector.sh and the rpi-newer-crosstools onto the cross compiling machines -- since I think we do this in Docker contains in ci (don't recall ci-release, maybe the same), this might be fiddly, and maybe it'd be easier if I merged that PR so it's master, let me know.

The node running the build extracts the label that was chosen ($label) and matches that to the right compiler, this is all done in cc-selector.sh.

We probably shouldn't touch existing 8.3.0 use, bumping glibc would be a breaking change for some users.

@richardlau
Copy link
Member Author

@rvagg I tried rvagg/rpi-newer-crosstools#1 locally in a container and it appears to work. I'm happy to do the wiring on the deploy to the cross-compiling machines/Jenkins side.

What do you think about bumping to glibc 2.28 to be consistent with all of the other Linux archs that we are building on RHEL 8? FWIW I put together a Linux distribution->glibc map -- while glibc 2.28 would rule out Ubuntu 18.04, Debian 10 is glibc 2.28: https://gist.github.com/richardlau/6a01d7829cc33ddab35269dacc127680

Also FWIW the gcc 8.3 cross compilation toolchain from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads is glibc 2.28.

We probably shouldn't touch existing 8.3.0 use, bumping glibc would be a breaking change for some users.

100% agree.

@rvagg
Copy link
Member

rvagg commented Mar 29, 2022

@richardlau yep, pushed a new version that uses 2.28. crosstools-ng is a bit out of date but it does at least have a good range of glibc that we are about that it'll deal with without too much hassle.

@richardlau
Copy link
Member Author

@rvagg I had a go at the cc-selector/Jenkins changes required: #2912

The cross compilation looks like it's working but unfortunately I'd forgotten that although the Pi's are running Debian 10 the tests are being run in a Debian 9 container which only has glibc 2.24. I think I've made the required Ansible changes to add a Debian Buster container, but ran into trouble trying to run the playbook against the Pi's (the first one I picked was one of the offline ones and the second one failed at the package upgrade step).

richardlau added a commit that referenced this issue Apr 8, 2022
Add a new RHEL 8 based container for cross compilation of armv7l
binaries for Node.js 18 using the recently added cross compiler
toolchain based on gcc 8 with glibc 2.28.

Adds a Debian 10 (Buster) container to the Pi's -- even though the Pi's
are running Debian 10 the tests are run in containers and prior to this
commit were running in Debian 9, which is on glibc 2.24.

Add a Debian 10 based armv7l container to be used as an extra way
to test the armv7l cross compiled binaries in addition to the Pi's.

Refs: #2815
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Apr 13, 2022
Starting with Node.js 18 we have moved building the release binaries
from being built on CentOS 7/RHEL 7 to being built on RHEL 8 as the
End-of-Life date of CentOS 7/RHEL 7 is before the End-of-Life date
of Node.js 18 and we do not want to have to make a disruptive change
towards the end of Node.js 18's life cycle.

Moving to RHEL 8 has raised the glibc version being linked against
(2.28). The official Node.js Linux release binaries will only run on
Linux distributions with a matching or higher version of glibc.

PR-URL: #42659
Refs: nodejs/build#2815
Refs: nodejs/build#2741
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Stewart X Addison <sxa@redhat.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
@targos
Copy link
Member

targos commented Apr 13, 2022

nodejs/node#42659 just landed. Are we done?

@richardlau
Copy link
Member Author

richardlau commented Apr 13, 2022

nodejs/node#42659 just landed. Are we done?

The only thing that hasn't been done is updating the Pi's (#2914 (comment)) but I think we can release Node.js 18 without them (we have some armv7l testing run in containers) -- I have temporarily stopped https://ci.nodejs.org/job/node-test-binary-arm-12+/ running for Node.js 18 and later so the CI will pass (#2912 (comment)).

vmoroz pushed a commit to vmoroz/node that referenced this issue Apr 13, 2022
Starting with Node.js 18 we have moved building the release binaries
from being built on CentOS 7/RHEL 7 to being built on RHEL 8 as the
End-of-Life date of CentOS 7/RHEL 7 is before the End-of-Life date
of Node.js 18 and we do not want to have to make a disruptive change
towards the end of Node.js 18's life cycle.

Moving to RHEL 8 has raised the glibc version being linked against
(2.28). The official Node.js Linux release binaries will only run on
Linux distributions with a matching or higher version of glibc.

PR-URL: nodejs#42659
Refs: nodejs/build#2815
Refs: nodejs/build#2741
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Stewart X Addison <sxa@redhat.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
xtx1130 pushed a commit to xtx1130/node that referenced this issue Apr 25, 2022
Update the macOS deployment target for Node.js 18 onwards.

PR-URL: nodejs#42292
Refs: nodejs/build#2815
Refs: libuv/libuv#3406
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Ash Cripps <acripps@redhat.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
xtx1130 pushed a commit to xtx1130/node that referenced this issue Apr 25, 2022
Starting with Node.js 18 we have moved building the release binaries
from being built on CentOS 7/RHEL 7 to being built on RHEL 8 as the
End-of-Life date of CentOS 7/RHEL 7 is before the End-of-Life date
of Node.js 18 and we do not want to have to make a disruptive change
towards the end of Node.js 18's life cycle.

Moving to RHEL 8 has raised the glibc version being linked against
(2.28). The official Node.js Linux release binaries will only run on
Linux distributions with a matching or higher version of glibc.

PR-URL: nodejs#42659
Refs: nodejs/build#2815
Refs: nodejs/build#2741
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Stewart X Addison <sxa@redhat.com>
Reviewed-By: Beth Griggs <bgriggs@redhat.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants