Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Composite Actions using Node 20 don't work with CentOS 7 #2906

Open
ajschmidt8 opened this issue Oct 4, 2023 · 13 comments
Open

Composite Actions using Node 20 don't work with CentOS 7 #2906

ajschmidt8 opened this issue Oct 4, 2023 · 13 comments
Labels
bug Something isn't working

Comments

@ajschmidt8
Copy link
Contributor

Problem

As mentioned in the GitHub blog post below, Actions is currently transitioning from Node 16 to Node 20.

However, Node 20 seems to be incompatible with CentOS 7, which is listed as a compatible OS for self-hosted runners below:

This incompatibility seems to stem from the fact that NodeJS version 20 supports glibc >=2.28 (as seen in the link below), but CentOS 7 only has glibc==2.17.

The consequence of this is that composite actions that have been updated to use Node 20 no longer work with CentOS 7 host machines or containers.

Here is an example of the error message that results from trying to run a composite action using Node 20 on a CentOS 7 container:

As a result of this incompatibility, we've had to intentionally downgrade our composite actions to a version that uses Node 16 so that our workflows continue to work.

To Reproduce

Run the following workflow to exhibit the error message

name: centos7

on:
  push:

jobs:
  run:
    runs-on: ubuntu-latest
    container: centos:7
    steps:
      - uses: actions/checkout@v4
      - run: echo "hello, world"

image

Expected behavior

Since the documentation below states that self-hosted runners support CentOS 7, I would expect the above workflow to run to completion without any errors

Runner Version and Platform

Ubuntu 22.04, runner version 2.309.0

@firewave
Copy link

firewave commented Jan 28, 2024

This also affects

ubuntu:16:04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809147?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

ubuntu:18.04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809224?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

@TomasNovo
Copy link

TomasNovo commented Feb 26, 2024

hey @ajschmidt8 ,
I'm currently facing this problem. I've tried several ways to use Node 16 but seems that the action always calls Node 20

/lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /opt/actions-runner/externals/node20/bin/node)

how did you effectively managed to downgrade your composite actions to a version that uses Node 16 so that the workflows continue to work?

@troyxmccall
Copy link

@TomasNovo you'll need to audit all of your workflows and look up Node v16 compatible versions

ie:

FranzDiebold/github-env-vars-action@v2

would need to be rolled back to a specific version of

FranzDiebold/github-env-vars-action@v2.7.0

per https://github.com/FranzDiebold/github-env-vars-action/releases/tag/v2.7.0

@luciansmith
Copy link

I feel like this isn't getting the attention it needs: anyone using github actions to create manylinux2014 builds will no longer be able to build on that platform at all, if the proposed Node16->Node20 change becomes required. Is there any proposed mitigation of this? Am I incorrect in my assessment?

@alex
Copy link

alex commented May 11, 2024

Your assessment is correct. As a result, this change will have substantial negative impact on the Python community.

#3128 sketches out a direction Github could take to resolve this, but there has been no response thus far.

@dixyes
Copy link

dixyes commented May 14, 2024

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

  1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
  2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
  3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

@dreads
Copy link

dreads commented Jul 3, 2024

As much as I see this issue reported, I can find no solution to it or even instructions or tips on how to work around it.

@Y--
Copy link

Y-- commented Jul 3, 2024

We were able to keep using Node 16 with:

   env:
      ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
      ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

(from this article )

But that is likely to break very soon indeed. We need a longer term solution.

@sethrj
Copy link

sethrj commented Jul 3, 2024

@Y-- the ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16 seems not to be needed: at least, not for the case where our failure was due to running on a CentOS7 image.

@danielefranceschi
Copy link

We handled this notorious issue using the appropriate node16-compatible actions versions in centos7 containers (for instance, actions/checkout@v3 instead of @v4).

With a curious coincidence with the Centos7 EOL, everything WAS fine until some days ago, but now node20 is used anyway even if action.yml specifies runs.using=node16.

FTR: self hosted runners 2.317.0.

Solved putting the suggested environment variable in the job:

generate-RH7:
  runs-on: [self-hosted, linux, docker]
  env:
    ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: 'true'
  container:
    image: ghcr.io/xxxxx/yyyy:5-centos7

@solardiz
Copy link

solardiz commented Jul 5, 2024

FWIW, in lkrg-org/lkrg#344 we've just dealt with both CentOS 7 EOL and then this issue with:

+++ b/.github/workflows/builds.yml
@@ -4,12 +4,17 @@ on: [push, pull_request]
 jobs:
     build:
         runs-on: ubuntu-22.04
+        env:
+            ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
+            ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
         strategy:
             fail-fast: false
             matrix:
                 include:
                     - image: centos:7
-                      run: yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
+                      run: |
+                          sed -i 's,^mirrorlist=,#,; s,^#baseurl=http://mirror\.centos\.org/centos/$releasever,baseurl=https://vault.centos.org/7.9.2009,' /etc/yum.repos.d/CentOS-Base.repo
+                          yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
                     - image: almalinux:8
                       run: yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
                     - image: rockylinux:8

This is with actions/checkout@v3. I hope (and ask that) this remains supported for a few years more, since there are now Enterprise Linux 7 prolonged support distros so we are not dropping our EL7 support, which we continue to test on CentOS 7.

olliecheng added a commit to DavidsonGroup/flexiplex that referenced this issue Jul 11, 2024
acelyc111 added a commit to apache/incubator-pegasus that referenced this issue Jul 11, 2024
ruojieranyishen pushed a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jul 17, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
amadio added a commit to amadio/xrootd that referenced this issue Jul 25, 2024
@klepiz
Copy link

klepiz commented Aug 12, 2024

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

  1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
  2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
  3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

Hello! this workaround worked for me for GH runner version 2.317 and 2.318 on a C7, but for 2.319 this is not working anymore and I am getting the issue

externals/node20/bin/node --version
externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by externals/node20/bin/node) 

Is there another way to make the latest GH runner version work on C7 + node20?

rudymatela added a commit to rudymatela/hello-haskell that referenced this issue Aug 26, 2024
... the Haskell image of it does not support running node20:

	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
	/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

I could try the following on GitHub actions:

    env:
        ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
        ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

as noted here: actions/runner#2906

However, it seems support for this was removed this spring anyway:

	https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

... or will be soon.
So I might as well stop running CI on older GHCs already...
@olsen232
Copy link

olsen232 commented Sep 5, 2024

+1 to fixing this issue
I'm using the Python manylinux2014 image but that still has the same issue. AFAICT there is no newer manylinux image than that, ie, Python best practice is still to build wheels on a linux image from 2014 that has an old GLIBC, but the builtin GH actions - due to using dynamically linked up-to-date nodejs - don't work with an old GLIBC. This is a pain and will surely only get worse as support is dropped for older node versions - this has already broken our build more than once as various actions we depend on switch to newer nodes, and each time I have to figure out how to pin whatever has gone wrong, which will only work as long as support is maintained for that old version.

I think using a statically linked node that just works for the built-in actions would be the way to go - as demonstrated here: #3128
Or - allow people to provide their own node that works for the container. I made sure that node was installed and working before using actions/checkout - but actions/checkout doesn't care, it installs a broken version of node anyway and uses that.

Lots of people have hit this already:
#337
#801
#2115
#2906
#3373

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests