[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung · 2024-10-01T16:42:48Z

This backports the following PRs that are related to the unflagging of --experimental-require-module:

Note that I don't think this backport PR is strictly required - the PRs seem to land cleanly on v22.x except the unflagging commit has a small conflict in the YAML changelog description. This PR is mostly opened to look for regressions in v22 with CITGM.

Refs: #52697

nodejs-github-bot · 2024-10-01T16:42:54Z

Review requested:

@nodejs/loaders

joyeecheung · 2024-10-01T16:46:41Z

CITGM for v22.x-staging: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3481/
CITGM for this PR: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3482/

joyeecheung · 2024-10-02T13:52:39Z

Some regressions found:

yarn is expecting ERR_REQUIRE_ESM in its tests, probably to check that they didn't do anything to alter Node.js module loading https://github.com/yarnpkg/berry/blob/33f249b6cc06db1803f03eb5f17306a23d11115c/packages/acceptance-tests/pkg-tests-specs/sources/pnp-esm.test.ts#L617
Gulp tests are breaking because they don't expect the experimental warning coming out of Node.js https://github.com/gulpjs/gulp/blob/2fa4981a910d7bdedb758bd09868620c9bb21d54/test/index.test.js#L90 (I am not sure how they load the .mjs configs, though it seems they are also doing a try-catch of require(config) somewhere)
2 also happened to Jest.
In local testing, I also noticed that the debug package (used by npm, etc.) has a try { require(esm) } catch { /* fallback */ } pattern here, which only gets improvements out of require(esm) support after the unflagging, but then the experimental warning would also break the output.

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

joyeecheung · 2024-10-02T14:58:59Z

As for esm, from what I can tell in standard-things/esm#883 it seems to have been broken in different ways since many major releases of Node.js ago. The regression that predated unflagging of require(esm) I found in #55085 (comment) also seem to come from some TDZ issue in the way the package is written (it relies on access to Node.js internals and seem to expect certain internals to be invoked in certain order). It doesn't seem to be very cost-effective to do anything about it on v22.

richardlau · 2024-10-02T15:27:11Z

I think for backporting to v22.x we should probably do some changes:

a. Provides process.features.require_module for feature detection
b. Do not emit the experimental warning if it's coming out of node_modules

b would not fix the test regressions 2-4, the packages still need to update their tests which is to be expected. But at least it would not break the CLIs for end users.

These SGTM

joyeecheung · 2024-10-02T16:33:21Z

Opened #55241 for feature detection.

joyeecheung · 2024-10-04T10:39:20Z

Added a commit from #55243 and a commit for v22.x specifically (and maybe v20.x in the future) to disable the warning when require() comes from node_modules.

targos · 2024-10-04T11:58:50Z

I just updated v22.x-staging so this now has conflicts because of duplicated commits I guess.

joyeecheung · 2024-10-07T15:49:07Z

Rebased to drop the duplicate commits. Also cherry-picked #55243 and #55241 from the main branch. I think for backport to v22.x it is ready, note that the last commit (silence the experimental warning when it comes from node_modules) is only targeting v22.x, as proposed in #55217 (comment) and agreed in the last TSC meeting.

nodejs-github-bot · 2024-10-07T15:56:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62972/

aduh95 · 2024-10-08T10:47:39Z

Can you rebase please?

mcollina

lgtm

Some packages have been using try-catch to load require(esm) in environments that are available. In 23, where require(esm) is unflagged, we emit an experimental warning when require() is used to load ESM. To backport require(esm) to older LTS releases, however, this could break existing CLI output. To reduce the disruption for LTS, on older release lines, this commit is applied to skip the warning if require(esm) comes from node_modules. This warning will be eventually removed when require(esm) becomes stable. PR-URL: nodejs#55217 Refs: nodejs#52697 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

When a ESM module cannot be loaded by require due to the presence of TLA, its module status would be stopped at kInstantiated. In this case, when it's imported again, we should allow it to be evaluated asynchronously, as it's also a common pattern for users to retry with dynamic import when require fails. PR-URL: nodejs#55502 Fixes: nodejs#55500 Refs: nodejs#52697 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il>

Trim off irrelevant internal stack frames for require(esm) warnings so it's easier to locate where the call comes from when --trace-warnings is used. PR-URL: nodejs#55496 Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Paolo Insogna <paolo@cowtech.it>

github-actions · 2024-11-12T01:41:16Z

Failed to start CI

   ⚠  Something was pushed to the Pull Request branch since the last approving review.
   ✘  Refusing to run CI on potentially unsafe PR

https://github.com/nodejs/node/actions/runs/11789132997

nodejs-github-bot · 2024-11-15T07:31:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63551/

nodejs-github-bot · 2024-11-15T11:49:25Z

CI: https://ci.nodejs.org/job/node-test-pull-request/63552/

joyeecheung · 2024-11-18T23:31:13Z

Last Jenkins CI was green, FWIW. Coverage is failing due to a pre-existing issue #55510

joyeecheung · 2024-11-18T23:32:39Z

Running CITGM again:

v22.x-staging: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3509/
This PR: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3510/

mcollina

RSLGTM

targos · 2024-11-19T09:17:40Z

I'm generally in favor of landing this, but are we ready for all the issues that will be opened because of debug-js/debug#975 / npm/cli#7857 ?

nicolo-ribaudo · 2024-11-19T11:44:19Z

Isn't it fixed by not warning inside node_modules?

targos · 2024-11-19T12:15:33Z

Ah, maybe? Since the warning is still here with the latest v23.2.0, is it a patch that's only going to be pushed to v22?

joyeecheung · 2024-11-19T12:29:25Z

Yes, the patch for disabling warning from node_modules here is only targeting 22.x.

We could consider landing it in 23, if necessary, but then that's 23.

slorber · 2024-11-19T12:56:18Z

A warning is not a big deal, but it would be great to port this warning patch to v23 indeed.

Otherwise, many OSS lib authors would get issues opened asking to fix those warnings while we can't easily do anything about it. It happened for me with things like punycode, and will likely happen again. Other example issues mentioning these warnings in v23:

joyeecheung · 2024-11-20T13:36:27Z

I think we have a few options that would need input from @nodejs/releasers and/or @nodejs/tsc

Hide the warning in require() from node_modules on both v22 and v23. However if the patch is also promoted to v23, we may need to adhere to the LTS policy and have to further delay the unflagging on v22. The next v23 release is Dec 3, which means the following v22 release is Dec 24. I am a bit uncomfortable about releasing the unflagging on Dec 24, though...(or having a release on Dec 24 just sounds risky in general, not sure if the release plan in Release plan - v22.x Active LTS Release#1001 is just tentative).
Hide the warning from node_modules on v22, leave it as-is on v23. That means the unflagging on 22 may be able to make it to the next 22 release, scheduled Nov 26.
Or..YOLO, we can just release the node_modules hiding on both next 23 and 22 releases, with the 22 coming out first.
Or we can squeeze the node_modules hiding into 23 first, either as 2024-11-20, Version 23.3.0 (Current), @RafaelGSS #55921 or as a patch release, and then get it out in the next 22 release without waiting for a full 2 weeks.

The patch in question is 14170f8?w=1 which is not that complex.

richardlau · 2024-11-20T13:39:40Z

(or having a release on Dec 24 just sounds risky in general, not sure if the release plan in Release plan - v22.x Active LTS Release#1001 is just tentative).

FYI I think tentative -- the table was updated with generated dates by the tool added in nodejs/Release#1058.

ruyadorno · 2024-11-20T14:43:36Z

iirc there's precedent for option 3 and in this case it does not sound unreasonable.

ruyadorno · 2024-11-22T04:48:55Z

The patch in question is 14170f8?w=1 which is not that complex.

Did this patch ever landed on main ? I can't seem to find it there.

joyeecheung · 2024-11-22T15:36:35Z

No, it was intended for v22.x (see https://github.com/nodejs/TSC/blob/main/meetings/2024-10-02.md), though now there is request to get it to v23.x as well. Not sure if we should just land it on main, though that might just make everything further and we'll have to wait for the next year to release it...

ruyadorno · 2024-11-22T16:06:39Z

No, it was intended for v22.x (see https://github.com/nodejs/TSC/blob/main/meetings/2024-10-02.md), though now there is request to get it to v23.x as well. Not sure if we should just land it on main, though that might just make everything further and we'll have to wait for the next year to release it...

oh yeah, I think we should def land it on main first then, can we try to make a group effort to fast-track it? I don't mind delaying the release a couple of days or an extra week to get that one in as I was already considering delaying it so that I get a few extra days to continue the cherry-pick automation for releases that I'm working on in parallel.

joyeecheung · 2024-11-22T23:07:20Z

I split out the patch in #55960

As part of the standard experimental feature graduation policy, when we unflagged require(esm) we moved the experimental warning to be emitted when require() is actually used to load ESM, which previously was an error. However, some packages in the ecosystem have already being using try-catch to load require(esm) to e.g. resolve optional dependency, and emitting warning from there instead of throwing directly could break the CLI output. To reduce the disruption for releases, as a compromise, this patch skips the warning if require(esm) comes from node_modules, where users typically don't have much control over the code. This warning will be eventually removed when require(esm) becomes stable. This patch was originally intended for the LTS releases, though it seems there's appetite for it on v23.x as well so it's re-targeted to the main branch. PR-URL: #55960 Refs: #55217 Refs: #52697 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me>

aduh95 · 2024-11-23T18:44:15Z

doc/api/modules.md

+added:
+  - v22.0.0
+  - v20.17.0


~~This change looks unrelated~~ (EDIT: I see that it was in my original PR, so "unrelated" is an unfair characterization, but still it should not be included in the backport PR), also on release branches we only keep track of the changes for that release line.

Suggested change

added:

- v22.0.0

- v20.17.0

added: v22.0.0

aduh95 · 2024-11-23T18:47:20Z

14170f8 message should be amended to correspond to 1919560

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. v22.x v22.x Issues that can be reproduced on v22.x or PRs targeting the v22.x-staging branch. labels Oct 1, 2024

RedYetiDev added module Issues and PRs related to the module subsystem. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. labels Oct 1, 2024

RedYetiDev added the needs-citgm PRs that need a CITGM CI run. label Oct 1, 2024

tats-u mentioned this pull request Oct 4, 2024

Enable consuming of ES modules in extensions microsoft/vscode#130367

Open

targos mentioned this pull request Oct 5, 2024

doc: fix typos #55066

Merged

This was referenced Oct 5, 2024

src: implement IsInsideNodeModules() in C++ #55286

Merged

Tracking issue: require(esm) #52697

Open

joyeecheung force-pushed the backport-require-esm-2 branch from dda6fc3 to 9c282c3 Compare October 7, 2024 15:45

joyeecheung marked this pull request as ready for review October 7, 2024 15:47

joyeecheung force-pushed the backport-require-esm-2 branch from 9c282c3 to 3d0afe0 Compare October 7, 2024 15:52

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Oct 7, 2024

joyeecheung force-pushed the backport-require-esm-2 branch 2 times, most recently from 5ed6462 to 898396a Compare October 8, 2024 19:00

mcollina approved these changes Oct 9, 2024

View reviewed changes

joyeecheung force-pushed the backport-require-esm-2 branch from 898396a to 5ea60dc Compare October 9, 2024 17:08

joyeecheung added 3 commits November 12, 2024 09:31

joyeecheung force-pushed the backport-require-esm-2 branch from 274b3bb to f7bdcee Compare November 12, 2024 01:31

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 12, 2024

github-actions bot added the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Nov 12, 2024

panva removed the request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. label Nov 14, 2024

mcollina approved these changes Nov 19, 2024

View reviewed changes

backflip mentioned this pull request Nov 20, 2024

debug causes npm to always issue an experimental ESM warning in node 23 debug-js/debug#975

Open

joyeecheung mentioned this pull request Nov 22, 2024

module: do not warn when require(esm) comes from node_modules #55960

Merged

aduh95 reviewed Nov 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

joyeecheung commented Oct 1, 2024 •

edited

Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

github-actions bot commented Nov 12, 2024

nodejs-github-bot commented Nov 15, 2024

nodejs-github-bot commented Nov 15, 2024

joyeecheung commented Nov 18, 2024

joyeecheung commented Nov 18, 2024

mcollina left a comment

targos commented Nov 19, 2024

nicolo-ribaudo commented Nov 19, 2024

targos commented Nov 19, 2024

joyeecheung commented Nov 19, 2024

slorber commented Nov 19, 2024 •

edited

Loading

joyeecheung commented Nov 20, 2024 •

edited

Loading

richardlau commented Nov 20, 2024

ruyadorno commented Nov 20, 2024

ruyadorno commented Nov 22, 2024

joyeecheung commented Nov 22, 2024

ruyadorno commented Nov 22, 2024

joyeecheung commented Nov 22, 2024

aduh95 Nov 23, 2024 •

edited

Loading

aduh95 commented Nov 23, 2024

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

Are you sure you want to change the base?

[v22.x] backport unflagging of --experimental-require-module and related fixes/refactorings #55217

Conversation

joyeecheung commented Oct 1, 2024 • edited Loading

nodejs-github-bot commented Oct 1, 2024

joyeecheung commented Oct 1, 2024

joyeecheung commented Oct 2, 2024 • edited Loading

joyeecheung commented Oct 2, 2024 • edited Loading

richardlau commented Oct 2, 2024

joyeecheung commented Oct 2, 2024

joyeecheung commented Oct 4, 2024

targos commented Oct 4, 2024

joyeecheung commented Oct 7, 2024

nodejs-github-bot commented Oct 7, 2024

aduh95 commented Oct 8, 2024

mcollina left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 12, 2024

nodejs-github-bot commented Nov 15, 2024

nodejs-github-bot commented Nov 15, 2024

joyeecheung commented Nov 18, 2024

joyeecheung commented Nov 18, 2024

mcollina left a comment

Choose a reason for hiding this comment

targos commented Nov 19, 2024

nicolo-ribaudo commented Nov 19, 2024

targos commented Nov 19, 2024

joyeecheung commented Nov 19, 2024

slorber commented Nov 19, 2024 • edited Loading

joyeecheung commented Nov 20, 2024 • edited Loading

richardlau commented Nov 20, 2024

ruyadorno commented Nov 20, 2024

ruyadorno commented Nov 22, 2024

joyeecheung commented Nov 22, 2024

ruyadorno commented Nov 22, 2024

joyeecheung commented Nov 22, 2024

aduh95 Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

aduh95 commented Nov 23, 2024

joyeecheung commented Oct 1, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

joyeecheung commented Oct 2, 2024 •

edited

Loading

slorber commented Nov 19, 2024 •

edited

Loading

joyeecheung commented Nov 20, 2024 •

edited

Loading

aduh95 Nov 23, 2024 •

edited

Loading