npm v7 support

[JamieMagee]

What would you like Renovate to be able to do?

Support the newly released npm v7. The blog post Presenting v7.0.0 of the npm CLI goes over some of the new features and breaking changes.

I'd like to use this issue as a mega-issue to discuss and co-ordinate npm v7 related changes to Renovate.

Did you already have any implementation ideas?

Some of the key features that will require Renovate changes:

• package-lock.json v2

  I can't yet find a schema for this, but here's a good example

• Workspaces

  See RFC 0026

• npx

  npx has been completely rewritten to use the npm exec command. There are various changes in functionality, most noticeable being a prompt if the module you are trying to run is not yet installed.

  There are a few places we use npx, and we should make sure that we are calling npx non-interatively.

• Peer dependencies

  Not sure if this one affects us

  Automatically installing peer dependencies (while this feature is something we think is desirable new behavior, it does potentially break certain workflows).

This list may not be complete.

The new library, @npmcli/arborist is also quite interesting, and should make interacting with the node_modules tree easier in future.

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Switch to yarn 😂

Is this a feature you'd be interested in implementing yourself?

Yes, with assistance

[mccxiv]

I'm here to express my confusion because we're already receiving PRs from renovate with "lockfileVersion": 2.
Is this already implemented?

[rarkins]

@mccxiv it's not intended. Can you reproduce the problem in a public repo? Does your engines allow node 15?

[mccxiv]

I don't have a public repro for now

We use a monorepo on Gitlab. The top level package.json has:

"engines": {
    "node": ">= 10.0.0",
    "npm": ">= 5.6.0"
  },

The individual packages do not have an engines: field

The odd thing is some PRs don't use lockfile v2, and some do (see line diffs):

[rarkins]

I can reproduce it. If you have an engines that allows node 15, and no npm constraint limiting <7, then Renovate will run npm install using Node 15 which defaults to npm 7. Example log:

DEBUG: Found compatible npm version (repository=renovate-tests/npm42, branch=renovate/chalk-2.x)
       "constraint": ">=12",
       "version": "15.0.1"
DEBUG: Resolved tag constraint (repository=renovate-tests/npm42, branch=renovate/chalk-2.x)
       "image": "docker.io/renovate/node",
       "tagConstraint": ">=12",
       "tagVersioning": "npm",
       "tag": "15.0.1"

[rarkins]

I've experimented with using an npm constraint (e.g. npm i npm@^6.0.0) but it doesn't seem to downgrade the npm version in node 15.

[rarkins]

If you add the following to your renovate.json then it succeeds in getting Renovate to use node 14:

  "constraints": {
    "node": "< 15.0.0"
  }

Unfortunately it's only succeeding due to what seems like a bug, but it can work for now and be forwards compatible for anyone who wants to support node 15 in their engines without running it during Renovate PRs.

[rarkins]

Actually, the required config will be as following once #7561 is merged and live:

{
  "force": {
    "constraints": {
      "node": "< 15.0.0"
    }
  }
}

The force wrapper is necessary because otherwise the detected engines node constraint takes precedence.

[ylemkimon]

@rarkins

I've experimented with using an npm constraint (e.g. npm i npm@^6.0.0) but it doesn't seem to downgrade the npm version in node 15.

Do you mean by setting

{
  "constraints": {
    "npm": "^6.0.0"
  }
}

?

In that case, it would haven't worked due to the cached location of npm after installing npm:

renovate/lib/manager/npm/post-update/npm.ts

Lines 30 to 33 in 367e591

	const npmCompatibility = config.constraints?.npm;
	if (validRange(npmCompatibility)) {
	installNpm += `@${quote(npmCompatibility)}`;
	}

I opened a PR to fix the issue including automatically setting the constraints in #7700.

• package-lock.json v2
  I can't yet find a schema for this, but here's a good example

[ylemkimon]

• Workspaces
  See RFC 0026

Workspaces seem to work out of the box: https://github.com/ylemkimon/npm-workspace.

• npx

  npx has been completely rewritten to use the npm exec command. There are various changes in functionality, most noticeable being a prompt if the module you are trying to run is not yet installed.

  There are a few places we use npx, and we should make sure that we are calling npx non-interatively.

The only place npx is used is:

renovate/lib/manager/npm/post-update/yarn.ts

Lines 145 to 156 in 7573780

	logger.debug('Performing yarn dedupe fewer');
	commands.push('npx yarn-deduplicate --strategy fewer');
	// Run yarn again in case any changes are necessary
	commands.push(`yarn install ${cmdOptions}`.trim());
	}
	if (
	(isYarn1 || isYarnDedupeAvailable) &&
	config.postUpdateOptions?.includes('yarnDedupeHighest')
	) {
	logger.debug('Performing yarn dedupe highest');
	if (isYarn1) {
	commands.push('npx yarn-deduplicate --strategy highest');

[ylemkimon]

npx will also work out of the box, i.e., non-interactively, because we set CI=true:

renovate/lib/manager/npm/post-update/yarn.ts

Lines 64 to 67 in 7573780

	const extraEnv: ExecOptions['extraEnv'] = {
	NPM_CONFIG_CACHE: env.NPM_CONFIG_CACHE,
	npm_config_store: env.npm_config_store,
	CI: 'true',

and npm 7 detects it: https://github.com/npm/cli/blob/a28aff769a77f127f371c31afcb9e9814722e5cd/lib/exec.js#L190-L191

I think we can say Renovate has full support for npm 7.

[stephenwade]

#7700 adds support for npm 7 in Docker, but npm 6 is still used when run locally (npm i -g renovate). Is there any way to get Renovate to use npm 7 (or get it to use my globally installed npm instead of the bundled npm) when running locally?

EDIT: npm 6 is only used when in local development. Globally installed renovate will properly use your globally installed npm version.

[viceice]

Looks like this is a technical limitation, as yarn / npm always prefer local bin before global.

Maybe we need to migrate away from referencing the full npm package.

[stephenwade]

Renovate doesn't reference npm directly, but semantic-release does so it still ends up in the local bin folder.

[stephenwade]

Are there any config options to specify the path to npm?

[viceice]

Semantic release is a dev dependency and shouldn't be installed on production. 🤔

[viceice]

Are there any config options to specify the path to npm?

Nope

[stephenwade]

Okay, I found my issue. I was testing with a local dev build which is why I had npm@6 in that folder. When I run the globally installed renovate, it uses my globally installed npm version.

[ylemkimon]

@stephenwade Try using binarySource: 'docker' (--binary-source=docker).

[viceice]

I think we properly support npm v7 now. 🤔

I'll close this, feel free to reopen or open a new issue / discussion if there are any problems.

[rarkins]

I think we're missing workspaces, but let's handle that in a separate issue and get a reproduction for it

[abernix]

🤔 It seems that, despite having engines.npm set to a 6.x range, that the attempt to update is still happening with npm 7:

From apollographql/apollo-server#4998 (comment):

npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: undefined,
npm WARN EBADENGINE   required: { node: '>=6 <15', npm: '^6.14.11' },
npm WARN EBADENGINE   current: { node: 'v14.16.0', npm: '7.6.1' }
npm WARN EBADENGINE }

Is there another way that this is meant to be configured/conveyed?

[rarkins]

@abernix please create a new bug report issue and locate the relevant job logs from app.renovatebot.com. We'd want to see any mentions of constraints in the logs as well as see the debugging prior to the docker run command being executed.

[rarkins]

Can you try again now? I restarted the API process and I think that's cleared it

[gremo]

Hi, for me "force" seems not to work.

Renovate is still creating PR with NPM version 7 (see gremo/nest-winston@4f8ddb8). Any help is much appreciated!

{
  "extends": [
    "config:base",
    ":disablePeerDependencies"
  ],
  "packageRules": [
    {
      "depTypeList": [
        "devDependencies"
      ],
      "rangeStrategy": "bump"
    }
  ],
  "force": {
    "constraints": {
      "node": "< 15.0.0"
    }
  }
}

[JamieMagee]

@gremo Can you please open a new issue, or comment on an existing open issue. @abernix's issue #9036 appears to be relevant.

[gremo]

@JamieMagee thanks created a new issue #9103. Hope I find where the problem is.

[rarkins]

It's the same scenario as for @abernix - forced constraints with node only now don't stop npm@latest from being installed.

[gremo]

Thanks @rarkins, so... what's the proposed / working solution?

[viceice]

@gremo Do not use the force constraints and let the renovate's magic work.

Als long there is a v1 lockfile, renovate will force node v14 and npm v6.