Dependencies install fails in NPM version 7.5.2 due to Appium

[fluiddot]

Description

Running npm install with NPM version 7.5.2 produces the following error:

npm ERR! code ENOENT
npm ERR! syscall chmod
npm ERR! path <WORKSPACE_DIR>/gutenberg/node_modules/appium/node_modules/.bin/authorize-ios
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, chmod '<WORKSPACE_DIR>/gutenberg/node_modules/appium/node_modules/.bin/authorize-ios'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     <HOME_DIR>/.npm/_logs/2021-02-08T11_26_48_383Z-debug.log

Step-by-step reproduction instructions

1. Install latest NPM version 7.5.2 (with NVM: nvm install --latest-npm)
2. Run npm install

Expected behaviour

The dependencies installation should succeed.

Actual behaviour

The dependencies install fails with the error described above.

Screenshots or screen recording (optional)

N/A

WordPress information

• WordPress version: N/A
• Gutenberg version: 9.9.1
• Are all plugins except Gutenberg deactivated? N/A
• Are you using a default theme (e.g. Twenty Twenty-One)? N/A

Device information

• Device: N/A
• Operating system: N/A
• WordPress app version: N/A

[fluiddot]

This issue is related with the following Appium's issue: appium/appium#14901.
It was fixed and released in Appium version 1.19.2 so bumping this dependency to at least that version should fix it 🤞 .

[fluiddot]

Bumping Appium to latest version 1.20.2 fixes the error but produces a new one (quite similar but for a different file):

npm ERR! code ENOENT
npm ERR! syscall chmod
npm ERR! path <WORKSPACE_DIR>/gutenberg/node_modules/appium/build/lib/main.js
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, chmod '<WORKSPACE_DIR>/gutenberg/node_modules/appium/build/lib/main.js'
npm ERR! enoent This is related to npm not being able to find a file.

I'll keep investigating but I tried to install this dependency in an empty project and it works so it might be related to something specific to the project or caching.

[ockham]

Thanks @fluiddot! I'm seeing the same issue here (also tried upgrading to 1.19.1 and 1.20.2, and ran into the same issue as you did).

I'm a bit surprised we're not seeing this more widely reported in Slack 🤔 Maybe it's platform specific? FWIW, I'm on Linux.

[ockham]

(FWIW, my local workaround for now is to use node 12 and npm 6.14.10. Since I'm using nvm, that's fairly easy to do: nvm use 12.)

[fluiddot]

I'm a bit surprised we're not seeing this more widely reported in Slack 🤔 Maybe it's platform specific? FWIW, I'm on Linux.

Yeah, actually some people already noticed the issue but it was bypassed by downgrading to npm@6. The problem is that now installing the latest LTS version of node (v14.15.4) includes the NPM version 7.5.2.

[ItsJonQ]

👋 Haiii!

I ran into this last week when I was setting up my new computer.

At first, I thought it may have been my computer (it's an M1 MacBook). Because of the M1 chip. I thought perhaps development things were still weird.

I tried a bunch of stuff, including clearing npm cache, forcing specific versions of appium, globalling installing appium, etc...

I was (finally) able to resolve it by downgrading to npm@6.

---

Update: That being said... I was able to get non Gutenberg development projects working with npm@7.

[fluiddot]

👋 Haiii!

I ran into this last week when I was setting up my new computer.

At first, I thought it may have been my computer (it's an M1 MacBook). Because of the M1 chip. I thought perhaps development things were still weird.

I tried a bunch of stuff, including clearing npm cache, forcing specific versions of appium, globalling installing appium, etc...

I was (finally) able to resolve it by downgrading to npm@6.

Update: That being said... I was able to get non Gutenberg development projects working with npm@7.

Yeah I solved it by downgrading too.

[rafaelgallani]

Hi!
I was following the conversation in Slack, and it seems that even after upgrading appium to the latest version, there are still errors.
According to appium/appium#14901 (comment), this is happening due to the npm@7 way of symlinking the references in bin section of package.json. They have removed the authorize-ios file from the bin section, but it seems that the referred main.js file is still there:

https://github.com/appium/appium/blob/a539a5613099523ad9b7dd3c5d98144927ff38ed/package.json#L29-L31

I couldn't reproduce it when testing locally on a new project, but I'll try to do it and work on this issue.

---

Update:
I was able to reproduce it locally in a new project - if you run npm i followed by npm ci, it fails.
This is not particular to appium, as some other packages are having this error as well.
There's also an open issue for this on the NPM repository: npm/cli#2251

[fluiddot]

Hello @rafaelgalani 👋 and thanks for the help!

According to appium/appium#14901 (comment), this is happening due to the npm@7 way of symlinking the references in bin section of package.json. They have removed the authorize-ios file from the bin section, but it seems that the referred main.js file is still there:

Yes, even upgrading appium the issue is not solved. I also got the error related to main.js file when running npm install with NPM 7.5.2.

I managed to fix it by downgrading to NPM 6, install the latest version of appium and then switch back to NPM 7. Additionally I also tried to remove the node_modules folder and install the dependencies again and it didn't fail, I'm not sure if it has to do with the new package-lock.json format or another thing I might missing.

Update:
I was able to reproduce it locally in a new project - if you run npm i followed by npm ci, it fails.
This is not particular to appium, as some other packages are having this error as well.
There's also an open issue for this on the NPM repository: npm/cli#2251

Thanks for sharing it! Looks like NPM 7 introduced some critical changes, probably we should hold back using this version until these issues are solved.

[gziolo]

It'd be great to report back your findings to the upstream issues you referenced to increase the chances it gets resolved pretty quickly.

[fluiddot]

This issue will be addressed with the following PRs:

• Update Appium dependency version to 1.20.2 Update Appium dependency version to 1.20.2 #28890
• Update lock file format version to version 2 (NPM 7) Update lock file format version to version 2 (NPM 7) #28886

Here is the status of the different issues found so far:

CI failures due to not having access to some repositories ✅

This has been fixed in NPM version 7.5.3 and by updating the references of the forked dependencies from react-native-editor package.

Dependencies install error 🔴

The original error has been fixed by upgrading appium dependency, however a new error is now thrown quite similar but referencing a new file:

npm ERR! code ENOENT
npm ERR! syscall chmod
npm ERR! path /home/runner/work/gutenberg/gutenberg/node_modules/appium/build/lib/main.js
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, chmod '/home/runner/work/gutenberg/gutenberg/node_modules/appium/build/lib/main.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

I've created an issue in Appium repository just in case they can fix it, although this issue is already reported in npm/cli.

Incompatible license ✅

Unfortunately in the appium upgrade, one of its dependencies has changed its license and now it's Python-2.0 which is incompatible. Here is the error from check-licenses script:
Module argparse has an incompatible license 'Python-2.0'.

I'm not sure what's the solution for this, what's done in this case?

EDIT: The license has been added in #28968.

[gziolo]

According to https://www.gnu.org/licenses/license-list.en.html license of Python 2.0.1, 2.1.1, and newer versions are GPL compatible:

This is a free software license and is compatible with the GNU GPL. Please note, however, that intermediate versions of Python (1.6b1, through 2.0 and 2.1) are under a different license

However, license of Python 1.6b1 through 2.0 and 2.1 are NOT compatible:

This is a free software license but is incompatible with the GNU GPL. The primary incompatibility is that this Python license is governed by the laws of the State of Virginia, in the USA, and the GPL does not permit this.

[gziolo]

As far as I know Apache 2.0 license that appium uses isn't compatible with the license that Gutenberg uses but only for the purposes of code distribution. It's fine to use less permissive licenses for development. The question is whether we should add Python 2.0 to the list of exceptions.

[gziolo]

I asked for confirmation regarding the license in the WordPress Slack in the #core channel (link requires registration at https://make.wordpress.org/chat/):

https://wordpress.slack.com/archives/C02RQBWTW/p1612944016311700

[fluiddot]

Thank you very much @gziolo for checking this! I'll wait for the confirmation from WordPress Slack in the #core channel.

[fluiddot]

While I was testing the check-licenses script I spotted that it's not working properly on NPM 7, I've just created a PR with a fix.

I'm a bit concerned about the potential break changes that NPM 7 could introduce, I checked the changes from the release post and looks like that apart from the use of npm ls in the check-licenses script, there's nothing else that could affect the project.

I'd appreciate if someone else could take a look just in case I'm missing something, thanks!

[fluiddot]

Regarding the error when installing the dependencies, I saw that the issue in npm/cli has been prioritised in a milestone which due date is February 23, 2021. I'm not sure if this is critical enough to fix it before a new version of npm/cli is released, one option we could explore is to fork appium and apply a fix but maybe it's not worth it.

[ockham]

According to https://www.gnu.org/licenses/license-list.en.html license of Python 2.0.1, 2.1.1, and newer versions are GPL compatible:

This is a free software license and is compatible with the GNU GPL. Please note, however, that intermediate versions of Python (1.6b1, through 2.0 and 2.1) are under a different license

However, license of Python 1.6b1 through 2.0 and 2.1 are NOT compatible:

This is a free software license but is incompatible with the GNU GPL. The primary incompatibility is that this Python license is governed by the laws of the State of Virginia, in the USA, and the GPL does not permit this.

Thanks for researching this, @gziolo! We should maybe try and bring this up with argparse maintainers -- this seems like such a technicality that I don't think they'd be opposed to bumping their license to v2.0.1.

[ockham]

I'll file an issue against argparse. (However, it'll probably take a while until that change would make its way up the dependency chain, so ideally, we find a different fix in the meantime.)

[gziolo]

My bet is that it should be acceptable to add this license to the allowed list.

[ockham]

I'll file an issue against argparse. (However, it'll probably take a while until that change would make its way up the dependency chain, so ideally, we find a different fix in the meantime.)

nodeca/argparse#160

[ockham]

While I was testing the check-licenses script I spotted that it's not working properly on NPM 7, I've just created a PR with a fix.

Awesome, thanks a lot!

I'm a bit concerned about the potential break changes that NPM 7 could introduce, I checked the changes from the release post and looks like that apart from the use of npm ls in the check-licenses script, there's nothing else that could affect the project.

Yeah, given the obstacles we've encountered so far, I think there's certainly potential for breakage caused by NPM 7. My main strategy here would be to de-risk the upgrade as much as possible (as discussed earlier): Get all the required fixes (such as #28890) merged before we attempt the actual upgrade -- which we can then isolate in a small PR, and if necessary revert.

I'd appreciate if someone else could take a look just in case I'm missing something, thanks!

I'm a bit busy with other things currently, but I'll try to give it a look later!

[ockham]

Regarding the error when installing the dependencies, I saw that the issue in npm/cli has been prioritised in a milestone which due date is February 23, 2021. I'm not sure if this is critical enough to fix it before a new version of npm/cli is released, one option we could explore is to fork appium and apply a fix but maybe it's not worth it.

IMO: Totally not worth bothering with a fork (and later going back to using the 'original' project), given that that date is less than two weeks from now.

[fluiddot]

Thanks @ockham for creating the issue on argparse!

IMO: Totally not worth bothering with a fork (and later going back to using the 'original' project), given that that date is less than two weeks from now.

I agree, for now let's put it on hold until they release the fix.

[gziolo]

Can we pin npm 6 in the repo until it works with npm 7? Well, I'm not sure how much work it would be and how we could let contributors know so they avoid all the hassle. Just an idea to consider if that would be a quick change.

[fluiddot]

Can we pin npm 6 in the repo until it works with npm 7? Well, I'm not sure how much work it would be and how we could let contributors know so they avoid all the hassle. Just an idea to consider if that would be a quick change.

Do you mean to update the NPM version on the package.json file?
The current value for NPM is: "npm": ">=6.9.0", we could change it to something like "npm": ">=6.9.0 <7.0".

[ockham]

Can we pin npm 6 in the repo until it works with npm 7? Well, I'm not sure how much work it would be and how we could let contributors know so they avoid all the hassle. Just an idea to consider if that would be a quick change.

Do you mean to update the NPM version on the package.json file?
The current value for NPM is: "npm": ">=6.9.0", we could change it to something like "npm": ">=6.9.0 <7.0".

That might be a good stop-gap fix for the time being.

[gziolo]

If it works then sure 😃

[fluiddot]

Ok, I'll create a quick PR with that change.

[davidshq]

Appium suggested downgrading to npm 6 (as noted earlier in this thread) by using npm i -g npm@latest but this no longer works because npm@latest installs npm 7. You can still do the downgrade, just use npm i -g npm@6 instead of npm i -g npm@latest.

[fluiddot]

Ok, I'll create a quick PR with that change.

I've just created the PR with the change. I've tested locally and works fine.

[fluiddot]

👋 Looks like the npm/cli bug related to the dependencies install error is already solved and aims to be released on NPM 7.5.4 NPM 7.5.5 🤞 .

EDIT: Sorry, I pointed the wrong NPM version, the version that most likely will be included is 7.5.5 which is not released yet 🙏 .

[ockham]

Looks like the npm/cli bug related to the dependencies install error is already solved and aims to be released on NPM 7.5.4 .

Great! I'll try rebasing #29097.

[fluiddot]

Looks like the npm/cli bug related to the dependencies install error is already solved and aims to be released on NPM 7.5.4 .

Great! I'll try rebasing #29097.

@ockham sorry for the mistake on my previous comment, the NPM version which looks like it will be included is 7.5.5 not 7.5.4 but it's not released yet.

[fluiddot]

There's a new release of npm (7.5.6) that should fix the problems related to the dependencies install mentioned in previous comments so I ran some tests to verify if it works with the latest version in this PR.

Test results

chmod issue produced by the Appium dependency 🟢

Fortunately this is already fixed in the new npm version but new problems have arise.

Check license script fails 🔴

Not sure why but running the command check-license fails in the new npm version, looks like the error is related to the command npm ls that is used for fetching all the dependencies data.

I tried to run npm ls --json --all to check the output of that command and it's giving a lot of error like the following ones:

npm ERR! code ELSPROBLEMS
npm ERR! missing: sveltedoc-parser@^3.0.4, required by @storybook/addon-docs@6.1.11
npm ERR! missing: vue@^2.6.10, required by @storybook/addon-docs@6.1.11
npm ERR! missing: canvas@^2.5.0, required by jsdom@16.4.0
npm ERR! invalid: react-refresh@0.4.2 /Users/fluiddot/workspace/gutenberg/node_modules/react-refresh
...

Dependencies installation fails on MacOS machine 🔴

Now the jobs executed on MacOS machines fail after installing the latest npm version and running npm ci, it produces the following error:

npm ERR! code ENOTFOUND
npm ERR! syscall getaddrinfo
npm ERR! errno ENOTFOUND
npm ERR! network request to https://codeload.github.com/wordpress-mobile/react-native-reanimated/tar.gz/ed48f510fba751cd75da7629e92276166766be91 failed, reason: getaddrinfo ENOTFOUND codeload.github.com
npm ERR! network This is a problem related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settings.
npm ERR! network 
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

I tried to re-run the jobs multiple times just in case was a network issue but keeps throwing the error.

Scripts that use master branch fail 🔴

In the workflows Compressed Size / Check (pull_request) and Performances Tests / Run performance tests (pull_request), we checkout master but due to the recent changes limiting the version of npm to v6, the processes fail because of this.

In the future once this PR is ready we should first revert those changes and then verify that these PR checks pass.

[ockham]

Thanks a lot for this latest report, @fluiddot!

How about the following strategy:

• We keep a long-lived branch and accompanying PR with just the package-lock.json v2 fix (akin to your Update lock file format version to version 2 (NPM 7) #28886 -- but without the explicit npm install -g npms in GH workflows, or Noah's Update package-lock to v2 #29097). We comment on that branch to report remaining issues/blockers. The only other thing that that branch should include is an update for the version range in package.json's engine.npm field. (We should keep the logic in check-latest-npm that was added by Limit the npm version to be 6 #29204 IMO -- I find it overall a good addition, and it won't impede the npm update.)
• We file individual issues for the problems we encounter -- starting with your list -- and address them with individual PRs. This will allow us discussing the individual problems better. (For example, maybe the license check problem just requires updating the script to ignore those messages coming from npm. Or maybe those messages hint at a deeper problem with our dependency graph. Individual issues/PRs will allow us focussing on those problems, and finding solutions to them.)
• As we land those PRs, we rebase the long-lived package-lock.json upgrade branch, to see if there are any issues remaining, or new ones cropping up.

[ockham]

cc/ @griffbrad @jsnajdr @sarayourfriend @scinos @flootr @tyxla -- Do any of the above issues ring a bell? Any other experience you can share from attempting to upgrade Calypso to npm 7? Any recommendations with regard to strategy? 😊

[fluiddot]

Thanks @ockham for putting this together and planning the strategy!

How about the following strategy:

• We keep a long-lived branch and accompanying PR with just the package-lock.json v2 fix (akin to your Update lock file format version to version 2 (NPM 7) #28886 -- but without the explicit npm install -g npms in GH workflows, or Noah's Update package-lock to v2 #29097). We comment on that branch to report remaining issues/blockers. The only other thing that that branch should include is an update for the version range in package.json's engine.npm field. (We should keep the logic in check-latest-npm that was added by Limit the npm version to be 6 #29204 IMO -- I find it overall a good addition, and it won't impede the npm update.)

No problem, we can use any of both.

Regarding my PR, I added a step for running the command npm install -g npm because I wanted the CI jobs to use the latest npm version, currently they're using npm 6.14.11. My idea was to keep that step until we assure that is working for that npm version and then remove it.

• We file individual issues for the problems we encounter -- starting with your list -- and address them with individual PRs. This will allow us discussing the individual problems better. (For example, maybe the license check problem just requires updating the script to ignore those messages coming from npm. Or maybe those messages hint at a deeper problem with our dependency graph. Individual issues/PRs will allow us focussing on those problems, and finding solutions to them.)

I totally agree, I'll file the different issues next week to address them individually. In fact since the problem with Appium looks already solved in the latest npm version we could close this issue.

• As we land those PRs, we rebase the long-lived package-lock.json upgrade branch, to see if there are any issues remaining, or new ones cropping up.

Good idea!

[fluiddot]

👋 I've just created the separated tasks as described in the previous comment:

• Check license script fails in npm version 7.6.0 #29524
• Dependencies installation fails on MacOS machine (GitHub actions) with npm version 7.6.0 #29527
• GitHub actions workflows that use trunk branch fail when using npm version 7.6.0 #29528

If you agree I'm going to close this issue since the problem with Appium looks already solved in the latest npm version. We can continue discussing the issues related to npm version 7 in the issues listed above.

[ockham]

I've just created the separated tasks as described in the previous comment:

Awesome, thanks a lot!

If you agree I'm going to close this issue since the problem with Appium looks already solved in the latest npm version. We can continue discussing the issues related to npm version 7 in the issues listed above.

Yes, I think that's a great plan 👍