Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Failed to install npm package from git in docker since v6.11.0 #624

Closed
doochik opened this issue Dec 23, 2019 · 32 comments · Fixed by #673
Closed

[BUG] Failed to install npm package from git in docker since v6.11.0 #624

doochik opened this issue Dec 23, 2019 · 32 comments · Fixed by #673

Comments

@doochik
Copy link

doochik commented Dec 23, 2019

What / Why

This bug is related to #476 (comment) and #514

When

npm install packages from git in docker

Where

  • packages from public git repositories

How

Current Behavior

# npm i
npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-12-23T09_49_05_990Z-debug.log

Steps to Reproduce

  • Create simple package.json with git dependency
{
  "name": "test-npm",
  "dependencies": {
    "mockdate": "git://github.com/boblauer/MockDate.git#v2.0.5"
  }
}
  • run docker run --network=host -it -v $(pwd):/var/work/test -w /var/work/test node:12.14.0 /bin/bash
  • ls -l shows that files are owned by unknown user and unknown group (this is typical behaviour on build agents like teamcity, teamcity clones git repo and mounts it inside docker container)
# ls -l
-rw-r--r-- 1 20857 115755  114 Dec 23 09:46 package.json
  • run npm install
# Depends on environment you can get

npm ERR! code 128
npm ERR! Command failed: git clone --depth=1 -q -b v2.0.5 git://github.com/boblauer/MockDate.git /root/.npm/_cacache/tmp/git-clone-c9699b7f
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-c9699b7f': Permission denied
npm ERR! 

# or

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.
  • run npm i -g npm@6.10 (downgrade to npm 6.10.X)
  • run npm install
# all works well!

updated 1 package and audited 1 package in 1.355s
found 0 vulnerabilities

Expected Behavior

  • npm install / npm ci works well like npm@6.10
@SabatierBoris
Copy link

I have the same issue with npm 6.13.4.
It's working with npm 6.4.1

@doochik
Copy link
Author

doochik commented Dec 28, 2019

@isaacs please, take a look on this

@isaacs
Copy link
Contributor

isaacs commented Dec 30, 2019

Is the npm command being run as root, or some other user?

I'd like to try to zoom in on the problem and leave the docker aspect out of it.

If I understand this correctly:

  • The user running npm install is root
  • The $HOME directory is /root or some other root owned directory
  • Thus the cache folder in $HOME/.npm is root-owned.
  • But pacote is executing a git command based on the user of the ultimate target of the extraction, not the cwd where the git action is taking place, and it fails.

Simpler reproduction, sans docker:

$ sudo su
# echo $HOME
/var/root
# npm i git://github.com/isaacs/abbrev-js
npm ERR! code 128
npm ERR! Command failed: git clone --mirror -q git://github.com/isaacs/abbrev-js.git /var/root/.npm/_cacache/tmp/git-clone-35b93572/.git
npm ERR! fatal: could not create leading directories of '/var/root/.npm/_cacache/tmp/git-clone-35b93572/.git'
npm ERR!

npm ERR! A complete log of this run can be found in:
npm ERR!     /var/root/.npm/_logs/2019-12-30T20_15_05_627Z-debug.log

I will take a look today and see how hard this'll be to fix.

@isaacs
Copy link
Contributor

isaacs commented Dec 30, 2019

Note for myself and @npm/cli-team: this problem does not occur with pacote v10, so it's strictly a pacote v9 issue.

To verify this, I applied this diff to pacote:

diff --git a/lib/git.js b/lib/git.js
index ade670b..a79a0e1 100644
--- a/lib/git.js
+++ b/lib/git.js
@@ -36,6 +36,7 @@ class GitFetcher extends Fetcher {
   constructor (spec, opts) {
     super(spec, opts)
     this.resolvedRef = null
+    this.spec.hosted = null
     if (this.spec.hosted)
       this.from = this.spec.hosted.shortcut({ noCommittish: false })
 

(This is required because otherwise pacote v10 will try to fetch github repos from their static tarball endpoint, as that's usually much faster.)

Then ran:

# pacote extract git+ssh://git@github.com/isaacs/abbrev-js abbrev-from-git
{
  resolved: 'git+ssh://git@github.com/isaacs/abbrev-js#b8f3a2fc0c3bb8ffd8b0d0072cc6b5a3667e963c',
  integrity: 'sha512-NFzZKD4tPyRbi5lxIbIhczbuAaIlTCMCwR9GGG9dqptIOM5G7e9w80NOInG5Ny6yUS/QcCp0XHxVPOKrItUUIg==',
  from: 'git+ssh://git@github.com/isaacs/abbrev-js.git'
}
# ls -laF abbrev-from-git/
total 32
drwxr-xr-x    6 isaacs  staff   192 Dec 30 12:22 ./
drwxr-xr-x  133 isaacs  staff  4256 Dec 30 12:21 ../
-rw-r--r--    1 isaacs  staff  2011 Oct 26  1985 LICENSE
-rw-r--r--    1 isaacs  staff   499 Oct 26  1985 README.md
-rw-r--r--    1 isaacs  staff  1763 Oct 26  1985 abbrev.js
-rw-r--r--    1 isaacs  staff   529 Oct 26  1985 package.json

isaacs added a commit to npm/pacote that referenced this issue Dec 30, 2019
Fix: npm/cli#624

Infer the ownership of a git command invocation based on the cwd, if one is
specified.
isaacs added a commit to npm/pacote that referenced this issue Dec 30, 2019
Fix: npm/cli#624

Infer the ownership of a git command invocation based on the cwd, if one is
specified.
@isaacs
Copy link
Contributor

isaacs commented Dec 30, 2019

Looks like npm/pacote#26 will fix this problem, assuming that I am understanding the root cause. (At the very least, it fixes a problem, which is that running npm install of a git dep as a root user with a root-owned npm cache will not fail, and it seems like that's the thing happening here.)

isaacs added a commit to npm/pacote that referenced this issue Dec 30, 2019
Fix: npm/cli#624
Fix: npm/cli#642
Fix: npm/cli#514

Infer the ownership of a git command invocation based on the cwd, if one is
specified.
@doochik
Copy link
Author

doochik commented Dec 31, 2019

Yes, npm install is being run as root.

Thanks for the fix!

@simonkotwicz
Copy link

simonkotwicz commented Feb 19, 2020

@isaacs Installing npm packages from git private repositories in docker still fails (since v6.11.0 and still fails with 6.13.6) when using a ~/.netrc file or ~/.git-credentials file

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/some-private-repo
npm ERR!
npm ERR! remote: Password authentication is not available for Git operations.
npm ERR! remote: You must use a personal access token or SSH key.
npm ERR! remote: See https://github.com/settings/tokens or https://github.com/settings/ssh
npm ERR! fatal: unable to access 'https://github.com/some-private-repo/': The requested URL returned error: 403
npm ERR!
npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-02-19T02_07_01_401Z-debug.log

@isaacs
Copy link
Contributor

isaacs commented Feb 26, 2020

@simonkotwicz That's a different issue. Can you please post a new issue describing the situation, ideally with a more fleshed-out reproduction case?

@simonkotwicz
Copy link

opened #950

@augnustin
Copy link

Using npm in a docker, the issue still occurs to me.

With:

npm install
npm ERR! code 128
npm ERR! command failed
npm ERR! command git clone https://repo:token@my-own-gitlab.com/ui/myy-repo.git /root/.npm/_cacache/tmp/git-clone-4af321f1 --recurse-submodules
npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-4af321f1': Permission denied

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-03-23T10_34_17_446Z-debug.log

npm --version
7.6.3

node --version
v15.12.0

Any idea?

@kowalk
Copy link

kowalk commented Mar 29, 2021

same here ;/

@kowalk
Copy link

kowalk commented Mar 29, 2021

@augnustin after downgrade to version 6.10 all is working fine...

@yuiseki
Copy link

yuiseki commented Mar 31, 2021

I'm facing same problem at npm@v7.7.0 on docker

@Coding-Kiwi
Copy link

Coding-Kiwi commented May 10, 2021

I also face this issue with node 7 (every subversion)
Works for me on 6.14.13

npm i -g npm@6.14.13

@dmercertaylor
Copy link

dmercertaylor commented May 14, 2021

I ran into this issue on several subversions, including 7.13.0, on WSL Debian. Interestingly, the project was installed in /home/<non-root user> a user, and running npm i as that user works fine, only root seems to have this problem. Install as root also works fine if the project is installed outside of /home. Honestly, I assumed this was a user error with permissions in my case, I'm surprised to see other people get this.

@MrJuliuss
Copy link

Hi, same issue with Npm 7. It works fine with 6.14.*

@ljharb
Copy link
Contributor

ljharb commented Jun 3, 2021

A fresh issue describing the failure on npm 7 would be helpful.

@codegod100
Copy link

broken for me on 7

@MrJuliuss
Copy link

MrJuliuss commented Jun 9, 2021

I tried something : use node user instead of root user, and it works.

@molinto
Copy link

molinto commented Jul 8, 2021

Broken for me too on:
node: v14.17.3
npm: v6.14.13

@mrzealot
Copy link

Broken for me as well (with a package having github dependencies).
node: v16.5.0
npm: v7.20.0

Any info on a fix?

@nettybun
Copy link

nettybun commented Aug 8, 2021

None of these install methods work:

❯ npm i -D "github:antfu/esbuild-node-loader#0.2.0"
❯ npm i -D "git+https://github.com/antfu/esbuild-node-loader#0.2.0" 
❯ npm i -D "git+https://github.com/antfu/esbuild-node-loader.git#0.2.0"
❯ npm i -D "git://github.com/antfu/esbuild-node-loader#0.2.0"
❯ npm i -D "https://github.com/antfu/esbuild-node-loader.git#0.2.0"

They all throw:

npm ERR! code 128
npm ERR! command failed
npm ERR! command git --no-replace-objects clone --mirror -q ssh://git@github.com/antfu/esbuild-node-loader.git /home/today/.npm/_cacache/tmp/git-clone-e55e766c/.git
npm ERR! fatal: destination path '/home/today/.npm/_cacache/tmp/git-clone-e55e766c/.git' already exists and is not an empty directory.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/today/.npm/_logs/2021-08-08T01_12_10_223Z-debug.log

I have rm -rf the _cacache/tmp/ directory but it doesn't help.

@androidfans
Copy link

same here

@eisenyang
Copy link

I run into this error, Then I tried yarn, it worked.

@goncalves-diogo
Copy link

Hi, I've also come across the same error, If you look deeper into the logs you can see that the issue lies on the _cacache directory. NPM doesn't have permission to write into it. In my case, It was trying to write to the folder on /root/.npm/_cacache which isn't really a good idea because it's root.
That being I think there are two possible temporary solutions to this problem:

  1. Give write permission give something like sudo chown -R 1000:1000 /current/cache/folder (If this folder is root, don't do it)
  2. Change the npm cache directory npm config set cache ~/random/place --global. (NOTE: It might be required to do step 1 with the new repository).

@eMPee584
Copy link

2. Change the npm cache directory `npm config set cache ~/random/place --global`. (NOTE: It might be required to do step 1 with the new repository).

Good idea, but with /tmp/npm.cache at 777 permissions, it will still throw the same problem..

npm ERR! Cloning into '/tmp/npm.cache/_cacache/tmp/git-clone-596d9ec4'...
npm ERR! /tmp/npm.cache/_cacache/tmp/git-clone-596d9ec4/.git: Permission denied

Also, because I was running under sudo, I tried prepending env -i to not forward SUDO_UID variable, but that doesn't fix it either.

@likemusic
Copy link

likemusic commented Oct 30, 2021

0 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'install' ]
1 info using npm@8.1.2
2 info using node@v17.0.1
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 5ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 4ms
6 timing config:load:builtin Completed in 5ms
7 timing config:load:cli Completed in 3ms
8 timing config:load:env Completed in 0ms
9 timing config:load:file:/vue2-datepicker/.npmrc Completed in 1ms
10 timing config:load:project Completed in 3ms
11 timing config:load:file:/root/.npmrc Completed in 1ms
12 timing config:load:user Completed in 1ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 2ms
14 timing config:load:global Completed in 7ms
15 timing config:load:validate Completed in 1ms
16 timing config:load:credentials Completed in 3ms
17 timing config:load:setEnvs Completed in 3ms
18 timing config:load Completed in 32ms
19 timing npm:load:configload Completed in 32ms
20 timing npm:load:setTitle Completed in 0ms
21 timing npm:load:setupLog Completed in 2ms
22 timing config:load:flatten Completed in 10ms
23 timing npm:load:cleanupLog Completed in 6ms
24 timing npm:load:configScope Completed in 0ms
25 timing npm:load:projectScope Completed in 1ms
26 timing npm:load Completed in 56ms
27 timing arborist:ctor Completed in 2ms
28 timing idealTree:init Completed in 3386ms
29 timing idealTree:userRequests Completed in 0ms
30 silly idealTree buildDeps
31 silly fetch manifest date-parse-multiformats@git+https://git@github.com/likemusic/date-parse-multiformats.git
32 timing arborist:ctor Completed in 1ms
33 silly placeDep ROOT date-parse-multiformats@ OK for: vue2-datepicker@3.10.2 want: git+https://git@github.com/likemusic/date-parse-multiformats.git
34 timing idealTree:#root Completed in 3131ms
35 timing idealTree:node_modules/date-parse-multiformats Completed in 0ms
36 timing idealTree:buildDeps Completed in 3135ms
37 timing idealTree:fixDepFlags Completed in 39ms
38 timing idealTree Completed in 6565ms
39 timing command:install Completed in 6611ms
40 verbose stack Error: An unknown git error occurred
40 verbose stack     at makeError (/usr/local/lib/node_modules/npm/node_modules/@npmcli/git/lib/make-error.js:28:13)
40 verbose stack     at /usr/local/lib/node_modules/npm/node_modules/@npmcli/git/lib/spawn.js:36:26
40 verbose stack     at async Object.withTempDir (/usr/local/lib/node_modules/npm/node_modules/@npmcli/fs/lib/with-temp-dir.js:23:14)
40 verbose stack     at async Arborist.[nodeFromEdge] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:1061:19)
40 verbose stack     at async Arborist.[buildDepStep] (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:930:11)
40 verbose stack     at async Arborist.buildIdealTree (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:216:7)
40 verbose stack     at async Promise.all (index 1)
40 verbose stack     at async Arborist.reify (/usr/local/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:149:5)
40 verbose stack     at async Install.install (/usr/local/lib/node_modules/npm/lib/install.js:170:5)
41 verbose cwd /vue2-datepicker
42 verbose Linux 5.10.16.3-microsoft-standard-WSL2
43 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
44 verbose node v17.0.1
45 verbose npm  v8.1.2
46 error code 128
47 error An unknown git error occurred
48 error command git --no-replace-objects clone https://git@github.com/likemusic/date-parse-multiformats.git /root/.npm/_cacache/tmp/git-cloneMplQnl --recurse-submodules --depth=1
49 error fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-cloneMplQnl': Permission denied
50 verbose exit 128

If run manually run git --no-replace-objects clone https://git@github.com/likemusic/date-parse-multiformats.git /root/.npm/_cacache/tmp/git-cloneMplQnl --recurse-submodules --depth=1 it works as expected without any errors.

garethsb added a commit to garethsb/nmos-cpp that referenced this issue Dec 7, 2021
garethsb added a commit to garethsb/nmos-cpp that referenced this issue Dec 7, 2021
Explicitly pecify git+https://git@github.com/{user}/{repo}.git to workaround npm "feature" of rewriting GitHub package specs (npm/cli#2610), which applies on GitHub Actions after the Ubuntu 20.04 image 20211129.1 update (actions/runner-images#4625) upgraded to Node v16.13.0 (npm v8.1.0)

Finally, stop using sudo in order to avoid permissions issue on the npm cache (npm/cli#624)
jonathan-r-thorpe pushed a commit to sony/nmos-cpp that referenced this issue Dec 7, 2021
Explicitly pecify git+https://git@github.com/{user}/{repo}.git to workaround npm "feature" of rewriting GitHub package specs (npm/cli#2610), which applies on GitHub Actions after the Ubuntu 20.04 image 20211129.1 update (actions/runner-images#4625) upgraded to Node v16.13.0 (npm v8.1.0)

Finally, stop using sudo in order to avoid permissions issue on the npm cache (npm/cli#624)
@wongjiahau
Copy link

I'm facing this issue when trying to install from a Gitlab private repo using HTTPS with node@16.13.1 and npm@8.3.0. Things worked when I downgraded npm as per #624 (comment).

@likemusic
Copy link

When I previously post log for issue I investigated the problem.

This happens bacuase npm tries to clone (git clone subprocess) in temp directory for current user but it runs as current directory owner user where npm runs.

In my case the owner of directory was node but I run npm as root user. So npm start process as root-user (because I run it as root user), but when git clone supbrocess runs it tries clone in temp directory for root user as node user whose haven't enough rigth to create subdirectories during git clone.

To prevent this issue (untill it would be fixed) you must run npm command as user who is current directory owner, for example:

runuser -u node -- npm install

Or change owner for current directory:

chown $(whoami) .

@wongjiahau
Copy link

@likemusic For my case the directory that contains package.json is in fact not owned by root, and after doing chown root:root . the problem is fixed indeed. Thanks!

@murtycodes
Copy link

murtycodes commented Jun 14, 2022

This worked for me: RUN apk add --no-cache git

https://stackoverflow.com/a/63033749/6884069

rish9511 added a commit to ElucidataInc/node-s3-client that referenced this issue Jun 30, 2022
Before this commit it was required to install the package
using Github. However when using NPM version 8.5, the installation
used to fail with the following error :
"... git --no-replace-objects clone https://git@github.com/ElucidataInc/node-s3-client.git
 /root/.npm/_cacache/tmp/git-cloneNZtpAU --recurse-submodules
npm ERR! fatal: could not create leading directories of
'/root/.npm/_cacache/tmp/git-cloneNZtpAU': Permission denied"

The following Github threads were reported with similar issues
- npm/cli#624
- npm/cli#642

As the suggested methods in the above threads, like shifting to older
version of NPM, did not satisfy our needs, we are publishing the package
to NPM public registry
@georgisoft2020
Copy link

When there is a reference to a git library in package json, like : git:// it really shows the error : npm ERR! No user exists for uid 1000
That's why for me a workaround was to use the command npm install with other user inside docker container.
First check what users do you have on that docker with the help of cat /etc/passwd , then use an user different from root.
runuser -u otheruser -- npm install

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.