Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] npm exits silently with the code 254 and read only filesystem #4838

Closed
1 of 2 tasks
svsool opened this issue May 2, 2022 · 9 comments · Fixed by #5197
Closed
1 of 2 tasks

[BUG] npm exits silently with the code 254 and read only filesystem #4838

svsool opened this issue May 2, 2022 · 9 comments · Fixed by #5197
Assignees
Labels
Bug thing that needs fixing Priority 1 high priority issue Release 8.x work is associated with a specific npm 8 release

Comments

@svsool
Copy link

svsool commented May 2, 2022

Is there an existing issue for this?

  • I have searched the existing issues

This issue exists in the latest npm version

  • I am using the latest npm

Current Behavior

I use npm 8.6.0 and when I exec into Kubernetes pod and run npm --version npm exits silently. Then running echo $? prints 254. It used to work fine with npm 8.5.5, read only file system and mentioned below security context config.

image

Expected Behavior

I expect that npm --version will run fine.

Steps To Reproduce

  1. Create a static pod in the Kubernetes cluster using the following yaml:
apiVersion: v1
kind: Pod
metadata:
  name: node-test
spec:
  containers:
  - name: node-test-pod
    image: node:18.0.0-alpine
    command:
    - tail
    - "-f"
    - /dev/null
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - AUDIT_WRITE
          - CHOWN
          - DAC_OVERRIDE
          - FOWNER
          - FSETID
          - KILL
          - MKNOD
          - NET_BIND_SERVICE
          - NET_RAW
          - SETFCAP
          - SETGID
          - SETPCAP
          - SETUID
          - SYS_CHROOT
      privileged: false
      readOnlyRootFilesystem: true <------ this is important to reproduce a bug
      runAsNonRoot: true
      runAsUser: 1000
  1. Run kubectl exec -it <pod_name> -- /bin/sh
  2. Run npm --version
  3. It should fail silently

Environment

  • npm: 8.6.0
  • Node.js: v18.0.0
  • OS Name: Alpine Linux v3.15
  • System Model Name:
  • npm config:
; node bin location = /usr/local/bin/node
; cwd = /app
; HOME = /home/node
; Run `npm config ls -l` to show all defaults.
@svsool svsool added Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release labels May 2, 2022
@svsool svsool changed the title [BUG] npm refuses to run with read only root filesystem and exits silently with the code 254 [BUG] npm exits silently with the code 254 and read only filesystem May 2, 2022
@wraithgar
Copy link
Member

npm logging was changed recently to open the logfile early and stream to it, rather than wait for errors. To disable this completely use the flag --max-logs=0. Please try this and see if it solves this issue.

The logging code is supposed to gracefully handle this situation but it appears to not be doing so in your case. Can you also try running with --loglevel silly to see what happens? Usually a WARN message should happen:

$ npm -v 
npm WARN logfile could not be created: Error: EACCES: permission denied, open '/Users/wraithgar/.npm/_logs/2022-05-03T20_27_46_794Z-debug-0.log'
8.8.0

@wraithgar wraithgar added the Needs Discussion is pending a discussion label May 3, 2022
@XhmikosR
Copy link
Contributor

XhmikosR commented Jun 7, 2022

I'm also hitting this. Indeed, the issue happens only in docker.

I had to pin down the Node.js image to v16.15.0 which ships with npm v8.5.5, but using Node.js v16.15.1 with npm v8.5.5 also worked fine. I also tried npm v8.12.1 and the same error happens.

If someone could look into it, I'd really appreciate it!

@lukekarrys lukekarrys self-assigned this Jun 7, 2022
@lukekarrys lukekarrys added Priority 1 high priority issue and removed Needs Discussion is pending a discussion Needs Triage needs review for next steps labels Jun 7, 2022
@darcyclarke
Copy link
Contributor

@XhmikosR did you try @wraithgar's suggested fix?

@whs
Copy link

whs commented Jun 8, 2022

Doesn't seems to do anything

node@pod:/usr/local/app$ npm --max-logs=0 --loglevel silly

node@pod:/usr/local/app$ echo $?
254

@jasonwilliams
Copy link

jasonwilliams commented Jun 8, 2022

@wraithgar's fix didn't work for me I still get 254 just like @whs does. Node 18.1.0, npm 8.8.0

@XhmikosR
Copy link
Contributor

XhmikosR commented Jun 8, 2022

OK, so this very weird. I just tried it today and it works in most projects except for the ones we have npm install warnings which seem to be now errors with the new version.

I'll give it a go again, but I could reproduce the failure in 4 separate machines yesterday. I'll try to pinpoint the failure and see if the npm peer dependencies issues are now errors instead of warnings (the repos are Angular apps which have the issue #3666).

@mhamann
Copy link

mhamann commented Jun 28, 2022

We just hit this as well. Had to pin Node.js to v16.15.0 so we get npm v8.5.5. This is a critical issue given that it's strongly recommended to run production workloads on a read-only filesystem.

@bkatiemills
Copy link

Fixing to npm 8.5.5 is a no-go for us in light of CVE-2022-29244.

lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 20, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
lukekarrys added a commit that referenced this issue Jul 21, 2022
This also changes all the log messages about not being able to create
initial directories and files to `log.verbose` since we know run those
commands on init. There are a lot of valid reasons why those might fail,
and we don't want to show a warning for them every time.

Fixes: #4769
Fixes: #4838
Fixes: #4996
@thw0rted
Copy link

Just to close the loop for anyone who was watching this, the root cause was

#4996 -- failed to output a useful message when denied permission to open a log

and the reason people encountered it in Docker was

nodejs/docker-node#1734 -- image runs as root by default, and a recent change to npm behavior causes it to run as the owner of the working directory when launched by root, but the host-OS user ID does not exist in the container, resulting in a read failure (triggering the above)

Buzu added a commit to Buzu/Dockerfiles that referenced this issue Jun 10, 2023
Running npm (ex npm test, or npx {script}) resulted in an error about
not being able to write to log files. Apparently this was introduced at
some point by an npm update. The reson for the error was that the
default location for the loga files is ~/.npm, but since the pptr user
did not have a home directory, the log files could not be created. To
fix this, the pptr user now has a home directory.

npm/cli#4838 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing Priority 1 high priority issue Release 8.x work is associated with a specific npm 8 release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants