Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] EPERM issues since v6.11.0 #476

Closed
rowanoulton opened this issue Nov 13, 2019 · 18 comments
Closed

[BUG] EPERM issues since v6.11.0 #476

rowanoulton opened this issue Nov 13, 2019 · 18 comments
Assignees
Labels
Bug thing that needs fixing Release 6.x work is associated with a specific npm 6 release

Comments

@rowanoulton
Copy link

rowanoulton commented Nov 13, 2019

What / Why

We recently upgraded from npm 6.0.1 to 6.12.0 and several of our engineers are encountering persistent EPERM issues when they run npm install.

From all angles the folders in question have the right permissions set, and we've ruled out nvm as a possible reason for this failure too. No combination of uninstalling and reinstalling or cache clearing has resolved it.

Whenever they run npm install, they get output like this:

code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at Object.spawn (child_process.js:540:9)
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at Object.spawn (child_process.js:540:9)\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'webapp'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.
npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/REDACTED/.npm/_logs/2019-11-12T21_45_00_580Z-debug.log

I walked back the version of npm until 6.10.3, where installs work again. The mention of pacote in the above logs seems suspicious when you compare it to the release notes for 6.11.0:

dc8f9e5 pacote@9.5.7: Infer the ownership of all unpacked files in node_modules, so that we never have user-owned files in root-owned folders, or root-owned files in user-owned folders. (@isaacs)

Note that upgrading to 6.13.0 does not fix the problem.

Also note that we have some private git+ssh urls in our package.json

Where

npm public registry

How

Current Behavior

Install fails with above error output, EPERM

Steps to Reproduce

  • npm i -g npm@6.11.0 && npm install

Expected Behavior

Install works as before

@isaacs
Copy link
Contributor

isaacs commented Nov 13, 2019

It looks like the issue is involving a git repo dependency that has a prepare script of some kind.

Can you share some details about your dependencies? I know that they're private git deps, but how are you referencing them? What are they doing?

@rowanoulton
Copy link
Author

Hi @isaacs, thanks for the quick response.

In addition to public npm registry dependencies, we have two types of private modules:

  1. Internal modules we publish to our Artifactory registry. These are namespaced and configured with .npmrc. They take the format of @company-name/module
  2. Dependencies on repositories from our Github Enterprise instance, of the format git+ssh://git@company-url.com:company/repo-name.git#<full sha1 commit hash>

Does that answer your question?

@doochik
Copy link

doochik commented Nov 14, 2019

I think this issue is related to #286

Since v6.11 npm calls git with the directory owner user
https://github.com/npm/pacote/blob/v9.5.8/lib/util/git.js#L216

Examples:
I'm user1 and group1. Run npm install in directory /workdir which is owned by user2 and group2. npm runs git with uid=user2 and gid=group2 and fails to start with Error: spawn EPERM. Because user1 hasn't rights to start process for user2.

Another example is for docker. I have docker container for build and mount /workdir inside. Inside container /workdir is owned by unknown user. And we have error

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.

@isaacs
Copy link
Contributor

isaacs commented Nov 14, 2019

Ok, thanks. So this isn't about prepare scripts, it's just failing to run git because the uid/gid is getting set to the owner of the folder that it's unpacking to. The private-ness of the git repo is a red herring in this case, I think, because what you're describing seems to be about spawning git in the first place, regardless of the repo you're fetching from.

I think the problem here is that it's mixing "folder owner uid" with "euid of child process" under the same uid/gid config param.

We'll need to dig into it a bit more, because if we just roll that back, we'll be back to the state of checking out root-owned files in user-owned locations. Setting an effective uid on the spawned process does make the files default to being owned by that user, which is nice, but we should be skipping that unless the current user is root.

So, it might still fail if the user1 doesn't have permission to write to the folders owned by user2, but I think that would've been a problem before 6.11 as well. I'm assuming that /workdir has a mode of 775 rather than 755?

@isaacs
Copy link
Contributor

isaacs commented Nov 14, 2019

Note (for myself, mostly) that means you'll end up with files being owned by user2 in a folder owned by user1, but since the files checked out are owned by user2, then user2 should have permission to chown them over to user1.

@CNSKnight
Copy link

+1 npm 10.-12.

$ npm install --save https://github.com/CNSKnight/mifterialize-tree.git
npm ERR! code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at spawn (child_process.js:540:9)
npm ERR!       at execFile (child_process.js:224:17)
npm ERR!       at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!       at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!       at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!       at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at spawn (child_process.js:540:9)\n' +
npm ERR!     '    at execFile (child_process.js:224:17)\n' +
npm ERR!     '    at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)\n' +
npm ERR!     '    at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)\n' +
npm ERR!     '    at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)\n' +
npm ERR!     '    at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'ANCILLARIES'
npm ERR! }

@doochik
Copy link

doochik commented Nov 15, 2019

I'm assuming that /workdir has a mode of 775 rather than 755?

Yes, 775. I have write access to workdir.
In docker I’m root and have access to whole fs

@bertrama
Copy link

I believe this is also related to issue #296 .

@darcyclarke darcyclarke added Bug thing that needs fixing Epic a larger scope of work that will be tracked in a Project Board labels Nov 19, 2019
@sppatel
Copy link

sppatel commented Nov 21, 2019

I am seeing similar issues too. What is strange is that if I run npm install w/ the package locks removed, npm install succeeds with 6.12.0. At that point I will have fresh package-lock files. I then wipe node_modules and re-run npm install and it fails with the EPERM error.

If I roll back npm versions v10.16.2 (the version I was on previously) - no issues.

@sppatel
Copy link

sppatel commented Nov 21, 2019

Two questions on the above comment that I don't quite understand.

(1) Why is this not seen on an npm install w/o a package lock file.
(2) Why is this not more widespread and seen consistently between users? I could not get others to repro.

@mikemimik mikemimik added the Release 6.x work is associated with a specific npm 6 release label Nov 26, 2019
@mikemimik mikemimik added this to the Release 6.14.0 milestone Nov 26, 2019
@SebCorbin
Copy link

SebCorbin commented Nov 28, 2019

I'm encountering this issue too in a directory created by docker (I think, it's been a while).

I've fixed that by changing the group of the directory to my default user group, all runs fine now

@bertrama
Copy link

bertrama commented Dec 2, 2019

One example way to reproduce:

  • Installing a dependency from a git repository
  • Running in linux or osx.
  • The package directory’s group is not the user’s default group.
  • The package directory has the setgid bit turned on (i.e. chmod g+s or 2775) such that the not-default-group is preserved when npm install creates new directories.

In my ubuntu environment, my default username/group might be bertrama/bertrama and I’m a member of a group cdrom. Steps for me to reproduce are:

mkdir -m 2775 eperm-test
chgrp cdrom eperm-test
cd eperm-test
npm init -y
npm add left-pad@git://github.com/left-pad/left-pad

I haven't tested, but you may be able to substitute chgrp cdrom eperm-test with chgrp "$(groups | awk '{print $2}')" eperm-test.

@mikemimik mikemimik added the Priority Backlog a "backlogged" item that will be tracked in a Project Board label Dec 3, 2019
@mikemimik mikemimik removed this from the Release 6.13.2 milestone Dec 3, 2019
@darcyclarke darcyclarke removed Priority Backlog a "backlogged" item that will be tracked in a Project Board Epic a larger scope of work that will be tracked in a Project Board labels Dec 4, 2019
isaacs added a commit to npm/pacote that referenced this issue Dec 4, 2019
isaacs added a commit to npm/pacote that referenced this issue Dec 4, 2019
@isaacs
Copy link
Contributor

isaacs commented Dec 4, 2019

Fixed in pacote 9.5.10, will be included in the next npm v6 release.

@sppatel
Copy link

sppatel commented Dec 5, 2019

Really appreciate the quick turn-around on this. Thank you. Is there any rough (and I mean rough 😄 ) estimate as to when the next release is targeted for?

@isaacs
Copy link
Contributor

isaacs commented Dec 10, 2019

in about an hour

@sppatel
Copy link

sppatel commented Dec 10, 2019

Confirmed fixed! TY!

@dmikalova
Copy link

dmikalova commented May 6, 2020

I'm still getting this issue on NPM 6.14.4 with a container volume mount in Jenkins. Jenkins docker runs with a user that doesn't exist on the container, but the volume mounts are owned by that user, and I end up using root for unrelated reasons. NPM then fails because the owner of the dir doesn't exist. In brief, where user 11028 doesn't exist:

docker run -t -d -u 11028:100 --user root:root -w /tmp/jenkins-ea2356b0/workspace/x -v /tmp/jenkins-ea2356b0/workspace/x:/tmp/jenkins-ea2356b0/workspace/x:rw,z x
...
npm install
...
npm ERR! No user exists for uid 11028

NPM is the only thing that ends up having this behavior. Copying the dir to a temp dir and running let's me continue with the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug thing that needs fixing Release 6.x work is associated with a specific npm 6 release
Projects
None yet
Development

No branches or pull requests

10 participants