Linux build distro strategy (CentOS 5 retired)

[rvagg]

So, CentOS 5, which we use to build our Linux x64 and 86 binaries, ceased support at the end of March. You can't even do a yum update now on CentOS 5 and we need to move on.

In terms of getting the oldest reasonably supported libc to compile against, the next oldest I believe is actually RHEL 6, therefore CentOS 6, with version 2.12, the next from that is Debian 7 (Wheezy) with 2.13 (CentOS 5 gives us 2.5 FYI).

So, we could switch to CentOS 6 for building our binaries, which would be fine for Node 8 because it's a semver-major and easily justifiable. But what about Argon and Boron? Is it going to be acceptable to just switch the binaries to a new base libc mid-stream or do we need to keep the unsupported CentOS 5 boxes running just for these?

/cc @nodejs/lts

[gibfahn]

If nodejs/node#12672 lands then we're good to update mid-stream.

EDIT: It landed

[jbergstroem]

Relevant discussion in here too: #688 (comment)

[mhdawson]

I think technically it should be semver-major to switch the libc base as I could see that there would be a reasonable expectation that unless you upgrade to a new major Node.js version you don't have to upgrade the OS that you are running your application on.

Going forward we should probably look at the EOL of the platforms we build on and make sure the minimum level specified for an LTS release will live until the end of its lifespan.

In this specific case, it would be useful to know how many customers might still be using older versions to know what the impact of switching would be. Note sure how we'll get the data though.

[jasnell]

Going forward we should probably look at the EOL of the platforms we build on and make sure the minimum level specified for an LTS release will live until the end of its lifespan.

This is definitely a good policy in general. It would require being a bit more proactive about finding and documenting the expected EOL dates for supported platforms. At the moment we're fairly loose about that.

[jasnell]

Just to clarify, what would be the specific impact for an average user if we switch to building on CentOS 6?

[jbergstroem]

What lifespan though? There's Rhel, Centos or even Oracle and Unbreakable. Some extends EOL, some doesn't. I think an interesting add to this mix is the fact that although we provide packages it is not always our responsibility to cater for every permutation. Thats why some distributions gets paid and some doesn't.

Personally, I'd be happy with following the "majority oss upstream" distributions and just drop at their EOL.

[bnoordhuis]

I could see that there would be a reasonable expectation that unless you upgrade to a new major Node.js version you don't have to upgrade the OS that you are running your application on

False equivalence, IMO. If you willfully stay at an OS version that its vendor no longer cares to support, you have no reasonable expectation that third-party software keeps working.

[gibfahn]

Just to clarify, what would be the specific impact for an average user if we switch to building on CentOS 6?

You might be unable to run the binary on a machine with an earlier libc (or kernel version) than the machine we build on (AIUI).

[gibfahn]

Going forward we should probably look at the EOL of the platforms we build on and make sure the minimum level specified for an LTS release will live until the end of its lifespan.

@jasnell @mhdawson as I said in nodejs/node#12672, that means dropping support for Ubuntu 14.04 in Node 8. I don't think that's a reasonable thing to do. Basically we have to drop support for platforms 2½ years before their maintainers do.

[mhdawson]

@gibfahn that specific example makes a good point. Maybe I should have said that we should take it into consideration. It is also an argument for the "its supported as long as the OS is". Personally I'm ok with that, although if an OS is going to go EOL the month after the release then maybe just not supporting it at all makes sense.

[joyeecheung]

IMO if there is no significant problem building Argon and Boron on CentOS5, then keep building them on CentOS5 will at least give people enough time to upgrade to a newer OS without having to build a binary themselves / deal with libc upgrades. It doesn't even need to be supported on these old platforms, but it should at least be able to run there when no additional compatibility work is needed.

[gibfahn]

You can't even do a yum update now on CentOS 5

if there is no significant problem building Argon and Boron on CentOS5, then keep building them

The fact that there are no more updates means that we won't get security patches, which isn't really acceptable for our main build machine. RHEL 7 has been out for almost 3 years, and RHEL 6 for 6 years.

People can continue to use Node 6.10.3 for as long as they like, but if they don't get updates for their OS then they probably don't update their Node instance. The supported platforms lists the lowest kernel level as glibc >= 2.19, so people should really have been upgrading after Node 0.12 went EoL.

deal with libc upgrades

Updating the libc on a machine isn't really possible.

It doesn't even need to be supported on these old platforms

The problem is that maintaining machines is a constant cost. I understand the argument of "why stop using a working platform", but the problem is that when the next thing breaks we won't be able to stop all releases while we transition over to CentOS 6, that's something we need to be looking at in advance.

[joyeecheung]

People can continue to use Node 6.10.3 for as long as they like, but if they don't get updates for their OS then they probably don't update their Node instance

From my experience this is not usually true in an enterprise if the OS is managed by one team and the application runtime is managed by another team, as you don't have to use the global node in PATH to run an application so they could be updated separately, and it is common to bundle the application with the Node instance together as a standalone unit. (Same goes for desktop OS where people are using Node to build GUI applications, although that's usually Windows).

Updating the libc on a machine isn't really possible.

AFAIK not safely for sure, but it is possible, with hacks..which is why that's not what people usually want to do.

The fact that there are no more updates means that we won't get security patches, which isn't really acceptable for our main build machine. RHEL 7 has been out for almost 3 years, and RHEL 6 for 6 years.

I understand that, but is it possible to provide an distro built on CentOS5 alongside the ones built on a newer OS? If that's still too much hassle then nevermind, people can still get a binary somewhere other than the official website, although the official website is where people trust the most.

[bnoordhuis]

The issue I have with saying "yes, we'll keep on supporting centos 5" is that keeping the code base working with obsolete technology often requires a disproportionate amount of effort.

Case in point: the fix we had to develop and the buildbot we had to bring up to work around bugs in clang 3.4.1 that had been fixed in 3.4.2.

The case of centos 5 is doubly aggravating because anyone still using it today had 6 years to upgrade to centos 6 but chose not to for whatever reason.

That's irresponsible bordering on negligent and not something that should be condoned or encouraged by committing to its continued support.

[gibfahn]

From my experience this is not usually true in an enterprise if the OS is managed by one team and the application runtime is managed by another team

sure, but from my experience if the devops team managing the OS haven't seen the need to upgrade the system in the last 6 years, they probably won't do it until they have to, which will be when the next Node binary stops working and the applications team start pestering them.

[rvagg]

See #740, added 2 new hosts and some Jenkins config to make this >=8.0.0, still testing but if all goes well and there are no objections, 8.0.0 should be libc 2.12 instead of 2.5. Built using devtoolset-2 so same gcc as we are using on CentOS 5 still.

[rvagg]

Existing builds:

$ $ objdump -T /usr/local/bin/node | awk '{ print $5}' | grep GLIBC | sort | uniq
GLIBC_2.2.5
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.4
GLIBCXX_3.4

New builds for Node 8:

$ objdump -T /usr/local/bin/node | awk '{ print $5}' | grep GLIBC | sort | uniq
GLIBCXX_3.4
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.9
GLIBC_2.2.5
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.7

So technically I think that means we maintain Debian 5 (Lenny) & Ubuntu 8.04 support (yay?) since we're not even reaching up to 2.12.

There's a test build @ https://nodejs.org/download/test/v8.0.0-test201705296aa5896d1b14eb59d4d2c5b1a725b8eed1abfa64/ if anyone else wants to poke at it.

[rvagg]

Removed ctc-review, we've pretty much locked this in now. Node.js 4 & 6 LTS will continue to be compiled on CentOS 5 (somehow we'll have to make this work!), 8+ are on CentOS 6.

Since this is also reflected in Node's BUILDING.md I'm closing this. If anyone has further discussion points on this feel free to reopen.