node20 (and other version) binary used for running actions should be fully statically linked (including libc)

[alex]

Description

Currently the node binary that's used to run external actions dynamically links against libc. This has the effect that when used with containers, actions break because of libc symbols not being available:

• https://github.com/pyca/bcrypt/actions/runs/7221890745/job/19677826890 Error relocating /__e/node20/bin/node: fcntl64: symbol not found
• https://github.com/pyca/bcrypt/actions/runs/7221730927/job/19677334188 /__e/node20/bin/node: /lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /__e/node20/bin/node), etc.

In order to insulate the node binary from containers it's running in, it should be fully statically linked.

This can be accomplished with --fully-static argument to configure when building node (https://github.com/nodejs/node/blob/main/configure.py#L155-L160)

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Current runner version: '2.311.0'

Is it regression?

No

Expected behavior

node binary can be invoked inside of alpine containers or other environments with different libc

Actual behavior

Crashes with various dynamic linking errors

Repro steps

1. Set up a self-hosted arm64 runner
2. Run an image such as alpine or pyca/cryptography-manylinux2014 that has a different libc (musl and old, respectively)
3. Attempt to use any node20 action, such as actions/checkout@v4
4. See it fail

[mikhailkoliada]

Hey, we do not build static binaries and have no plans to do so (we do not build anything from source code at all), if you are using alternative libc we are unlikely to offer any help, because musl is not prioritised across major linux distributions and all the builds are gibc-centric, as far as I am aware, there are no official node ports to support musl even

[mikhailkoliada]

transferring to setup-node, might be they have anything to add, but I guess not much.

[alex]

While I recognize that in many respects alpine is the "odd distro out", it's extremely widely used for building containers. Similarly, older libc from distributions like Red Hat are still widely used in the enterprise.

As an open source maintainer, I need to support a wide variety of platforms for my software.

The reason I suggest static linking, and not official support for these platforms, is precisely to minimize the ongoing work for for the Actions team.

[marko-zivic-93]

Hello @alex
Thank you for creating this issue. We will investigate it and come back to you with our findings.

[alex]

Thanks. Please let me know if there's any other questions I can answer.

…

On Mon, Dec 18, 2023, 10:13 AM Marko Zivic ***@***.***> wrote: Hello @alex <https://github.com/alex> Thank you for creating this issue. We will investigate it and come back to you with our findings. — Reply to this email directly, view it on GitHub <#922 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBBTTGR53WI3ZT3MGCLYKBMSXAVCNFSM6AAAAABAXUCAE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRQG43TKNJUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[priyagupta108]

Hello @alex ! Thank you for reporting. The actions/setup-node action, as detailed in the README.md, provides the functionality to setup a Node.js environment in a GitHub Action workflow environment. It doesn't handle the building or linking of Node.js. Instead, it downloads pre-built Node.js binaries from the official Node.js distribution site.
Statically linking the Node.js binary, including libc, depends primarily on how Node.js itself is built, not something that can be controlled or modified by the actions/setup-node action. Therefore, to have a fully statically linked Node.js binary, changes would have to be made at the level of Node.js build process, which is outside the scope of the setup-node repository.

Hope this clarifies! :)

[alex]

I understand the role that actions/setup-node plays.

I originally filed this bug against the runner image, because this impacts the use of any node20 action. It seems like what you are saying is that in order to resolve this, the runner image would need to stop using setup-node, and build its own version of node for use running actions themselves, is that correct?

[dominykas]

Node.js project provides an official Alpine container, source: https://github.com/nodejs/docker-node

That official Alpine container, ironically, contains an unofficial build of Node.js build inside, source: https://github.com/nodejs/unofficial-builds

We've been using the official Node.js Alpine containers in production for many years now.

Perhaps these unofficial builds could be downloaded and used for the Alpine scenarios that you need, although I wonder if it's up to actions/setup-node to provide that.

[priyagupta108]

Hello @alex. I've tried to investigate the issue deeper and it seems like the errors encountered are due to the fact that the Node.js binary built for Actions is linked dynamically against libc. When this binary is used in environments with a different libc or in Alpine Linux containers (which use musl libc), the symbols can't be resolved, causing errors.
One way to solve this issue could be to use an unofficial Node.js binary that's statically linked against musl instead of glibc.

As @dominykas mentioned for alpine images unofficial-builds.nodejs.org builds are used or nodejs is built from source code.

[alex]

I think we're talking past each other: I cannot control the binary that's used for running an action via the uses directive, only the Github Actions Runner controls that. If GHA provided me a way to "bring my own node.js" I could do that, but such an option does not exist.

[dominykas]

setup-node is just an action - it does things that other actions (incl. Your own) can do. And it downloads node for you. There might be some caching involved and while I haven't tried it, I'm fairly certain you can bring your own node to your workflow?

Ref:

setup-node/src/distributions/base-distribution.ts

Line 130 in d86ebcd

`Acquiring ${info.resolvedVersion} - ${info.arch} from ${info.downloadUrl}`

You could also execute whatever you need executing using docker inside an Alpine based image - while it wouldn't be as tidy as a simple setup-node in YAML, this is not a limitation that can't be worked around?

[alex]

I think it would be helpful for me to restate the problem I'm trying to solve from the top, as there appears to be some confusion:

• I was requesting static linking for the node binary used for running actions, that is to say those that specify runs: using: node20 in their action.yml (e.g. https://github.com/actions/checkout/blob/main/action.yml#L97C1-L99C1)
• When such actions are used in a container, Github Actions will automatically mount a volume which injects a node binary into the container (at /__e/node20/bin/node).
• Because this binary dynamically links libc, it can encounter issues in situations such as when the system glibc is older than expected, or when the system libc is musl.
• Because Github Actions itself injects the node binary via a mounted volume, I (as a workflow author) have no control over it.

This is why I originally filed this issue on the runner workflow. It was moved here because the node binary is apparently the same one used by setup-node, however I do not interact with setup-node itself ever in this process.

Hopefully this alleviates some of the confusion, I am not a user of setup-node, the node binary that is impacting me (and which I'm requesting statically link libc), is the one provided by the actions runner at /__e/node20/bin/node).

[alex]

On deeper review, I'm not convinced setup-node is actually involved in this process, and I think it'd be more appropriate to move this issue to the actions/runner repo.

[norbusan]

We are facing the very same problem: We have to build binaries against old centos, against musl, ... and the provided node fails to run in these containers.
While of course there is the work-around to write your own action with docker in docker, it makes writing workflows very inconvenient and convoluted.
The checkout action past v1 cannot be used, and artifact uploading cannot be used at all since it uses node.

I strongly suggest either one of the two options being implemented:

• link node statically (as suggested above)
• allow for a way to "bring your own node" that will be used for further runners

Thanks for consideration

[norbusan]

Just to give you an idea: We are trying to build TeX Live binaries for various architecture/os combinations (we support about 15 of those). To make binaries run on all kind of systems, we need to compile on as old systems as possible (depending on the required GCC compiler).

That leads to the following errors:

• linuxmusl based on image alpine:3.5, error: Error relocating /__e/node20_alpine/bin/node: getentropy: symbol not found
• i386 based on image i386/ubuntu:xenial, error: OCI runtime exec failed: exec failed: unable to start container process: exec /__e/node20/bin/node: no such file or directory: unknown
• amd64 based on image centos:7, error: /__e/node20/bin/node: /lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /__e/node20/bin/node) ...

So even building for old systems does not work, which makes the container approach deeply flawed.

[priyagupta108]

Hello everyone. Thank you all for the insightful discussion on this issue. We've thoroughly examined all the given inputs, but it's clear that the request falls outside of the scope of the setup-node action.
For now I'm going to close the issue. If you have any concerns feel free to ping us.
We appreciate your time and cooperation :)

[alex]

Would it be possible to move this to the actions/runner repo? That's the appropriate place for it, and it'd be good to preserve the discussion here, rather than refiling it.

[janeklb]

@priyagupta108 would it be possible to move this to the actions/runner repo (or wherever is appropriate)?

[nicoddemus]

There is already a similar issue here: actions/runner#2906