ESM loader in v11 does not work with got package, after installing the latest import-in-the-middle

[corevo]

Description

Attempting to use v11 in an ESM project that includes got causes the node process to crash.

In Node v18 you will get a silent exit code 13. In Node v20, it will throw a SyntaxError before exit code 1.

app-1  | (node:8) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
app-1  | --import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("newrelic/esm-loader.mjs", pathToFileURL("./"));'
app-1  | (Use `node --trace-warnings ...` to show where the warning was created)
app-1  | file:///app/node_modules/@payzen/got-commons/node_modules/got/dist/source/index.js?iitm=true:39
app-1  |     let $default = _.default
app-1  |         ^
app-1  |
app-1  | SyntaxError: Identifier '$default' has already been declared
app-1  |     at ModuleLoader.moduleStrategy (node:internal/modules/esm/translators:155:18)
app-1  |     at callTranslator (node:internal/modules/esm/loader:285:14)
app-1  |     at ModuleLoader.moduleProvider (node:internal/modules/esm/loader:291:30)
app-1  |
app-1  | Node.js v20.10.0
app-1 exited with code 1

Expected Behavior

Running node --loader newrelic/esm-loader.mjs should work without crashing node.
NOTE: # ( Tell us what you expected to happen. )

Troubleshooting or NR Diag results

Steps to Reproduce

1. run npm init -y
2. run npm i --save got newrelic
3. create the file index.mjs with the contents

import got from 'got'

4. run node --loader newrelic/esm-loader.mjs index.mjs
5. crash

Your Environment

• ex: Browser name and version: not relevant
• ex: Node version: 20.10.0
• ex: Operating System and version: Linux

Additional context

[workato-integration]

https://new-relic.atlassian.net/browse/NR-199684

[jsumners-nr]

What version of the agent are you using? What version of import-in-the-middle is being used? The output of npm ls newrelic import-in-the-middle got should help us proceed.

[zintus]

zintus/newrelic-node-examples@c209a0d
I believe this repro is related. Please advise

npm run start fails
running node index.js succeeds

Node v21.3.0

[corevo]

  └─┬ newrelic@11.7.0
    └── import-in-the-middle@1.7.2

[jsumners-nr]

Thank you. I have narrowed down the issue and submitted a PR upstream (nodejs/import-in-the-middle#53).

[ljharb]

You can't export default multiple times, so it's possible this is a node loader bug.

[jsumners-nr]

Thank you @ljharb. It's a problem with IITM. I wrote a test for core and it passes:

Core Test

diff --git a/test/es-module/test-esm-initialization.mjs b/test/es-module/test-esm-initialization.mjs
index f03a47d5d3..e7882e0839 100644
--- a/test/es-module/test-esm-initialization.mjs
+++ b/test/es-module/test-esm-initialization.mjs
@@ -28,3 +28,19 @@ describe('ESM: ensure initialization happens only once', { concurrency: true },
     assert.strictEqual(code, 0);
   });
 });
+
+describe('ESM: nested default exports are handled correctly', () => {
+  it(async() => {
+    const { code, stderr, stdout } = await spawnPromisified(execPath, [
+      '--experimental-import-meta-resolve',
+      '--loader',
+      fixtures.fileURL('es-module-loaders', 'loader-resolve-passthru.mjs'),
+      '--no-warnings',
+      fixtures.path('es-modules', 'nested-default', 'index.mjs'),
+    ]);
+
+    assert.strictEqual(stderr, '');
+    assert.strictEqual(stdout.match(/resolve passthru/g)?.length, 3);
+    assert.strictEqual(code, 0);
+  });
+});
diff --git a/test/fixtures/es-modules/nested-default/a.mjs b/test/fixtures/es-modules/nested-default/a.mjs
new file mode 100644
index 0000000000..dc07e2033d
--- /dev/null
+++ b/test/fixtures/es-modules/nested-default/a.mjs
@@ -0,0 +1,3 @@
+const a = 'a'
+export default a
+export * from './b.mjs'
diff --git a/test/fixtures/es-modules/nested-default/b.mjs b/test/fixtures/es-modules/nested-default/b.mjs
new file mode 100644
index 0000000000..7f5dadf656
--- /dev/null
+++ b/test/fixtures/es-modules/nested-default/b.mjs
@@ -0,0 +1,2 @@
+const b = 'b'
+export default b
diff --git a/test/fixtures/es-modules/nested-default/index.mjs b/test/fixtures/es-modules/nested-default/index.mjs
new file mode 100644
index 0000000000..755d201fc8
--- /dev/null
+++ b/test/fixtures/es-modules/nested-default/index.mjs
@@ -0,0 +1 @@
+export * from './a.mjs'

[zintus]

I can confirm your fix helps with my repro too. Thanks for the investigation @jsumners-nr

[corevo]

I've tried your fix and the error is now different, it looks like the module loading is okay, but not all the exports are re-exported correctly, it looks like the exported object is got's Request rather than the got object itself, I'm using got@14.0.0 which is a native ESM module.

TypeError: got.extend is not a function
    at file:///app/node_modules/@payzen/got-commons/build/got-commons.js:185:29
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:120:12)

Printing the default exports yields

[class Request extends Duplex]

[jsumners-nr]

Thank you for the update. Figuring out the full solution given that information will be tricky indeed.

[jsumners-nr]

I have determined the underlying problem. It is going to require some significant work in IITM.

[jsumners-nr]

The issue is that in https://github.com/DataDog/import-in-the-middle/blob/ae292fb72d2a3444f3c4fa77439be81df49ff8f9/hook.js#L205-L212 we merge all transitive exports into a singular namespace. Whatever was the last "default" export in derived namespaces ends up being the default exported from the shim module. Which means we need to figure out how to provide all of the required exports from the shim module while also picking the primary default and renaming the others according to their module name (e.g. './foo-bar.js' default export should be renamed and exported as fooBar).

I'm at a loss for how to accomplish this. I welcome any ideas.

[corevo]

Sorry but I think I don't fully understand the issue here.

Using the keyword default more than once is illegal, it will return a parsing error of duplicate definition of .default on the module object, are you referring to a module like this

/* first.mjs */
export default 1

/* second.mjs */
export default 2

/* index.mjs */
export * from './first.mjs'
export * from './second.mjs'

Because if so then in index.mjs the .default is not defined because the default keyword was not strictly used and the exported module will be {}, unless you want the default and then you have to rename it if imported from more than one module.

[ljharb]

export * from doesn’t bring the default export, so there shouldn’t ever be more than one.

[jsumners-nr]

After my last update I had a brainwave and seem to have a viable solution. I'm currently working through cleaning it up and expanding my test to try and cover all bases.

[jsumners-nr]

I believe commit nodejs/import-in-the-middle@1abc010 solves the problem.

[jsumners-nr]

I think we should see a release soon, but in the meantime, if you'd like to try the latest main branch of import-in-the-middle it should solve the issue.

[alexey-yarmosh]

Hey, thanks for the fix. I've just checked with the new version of import-in-the-middle. It fixes got issue.

In the meantime it still fails with another import in our project. When I try to:

import { defineScript } from 'redis';

I get:

node --loader newrelic/esm-loader.mjs test.mjs
(node:97636) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("newrelic/esm-loader.mjs", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
import { defineScript } from 'redis';
         ^^^^^^^^^^^^
SyntaxError: The requested module 'redis' does not provide an export named 'defineScript'
    at ModuleJob._instantiate (node:internal/modules/esm/module_job:132:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:214:5)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12)

Node.js v20.10.0

Also seems problem is specificly with defineScript, cause import { createClient } from 'redis'; works fine.

[barryhagan]

In the meantime it still fails with another import in our project. When I try to:
...
Also seems problem is specificly with defineScript, cause import { createClient } from 'redis'; works fine.

This is because of the re-export issue. Only defineScript is a re-export, not createClient.

We've been waiting for some time for this to be able to move to version 11 and it looks like they finally merged the 6 month old PR.

[jsumners-nr]

IITM 1.7.3 has been published and resolves the original issue described here. Please update your dependencies to get the fix. If there are other problems, please open a new issue.

[alexey-yarmosh]

Hey, do you have plans/approximate dates to update iitm version in node-newrelic itself?

[jsumners-nr]

The update is covered by the semver range qualifier specified in the package.json:

node-newrelic/package.json

Line 203 in f414564

"import-in-the-middle": "^1.6.0",

Updating your dependencies will net the fix.

[alexey-yarmosh]

Thanks!

[zintus]

Thanks for working on this! @jsumners-nr

My repro i've linked before still fails with same steps. Please advise
newrelic/newrelic-node-examples@main...zintus:newrelic-node-examples:main

> esm-app@1.0.0 start
> node --experimental-loader newrelic/esm-loader.mjs -r newrelic index.js

(node:36337) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("newrelic/esm-loader.mjs", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
file:///Users/zintus/workspace/newrelic-node-examples/esm-app/node_modules/openai/resources/index.mjs?iitm=true:55
    let $Completions = namespace.Completions
        ^

SyntaxError: Identifier '$Completions' has already been declared
    at ModuleLoader.moduleStrategy (node:internal/modules/esm/translators:167:18)
    at callTranslator (node:internal/modules/esm/loader:285:14)
    at ModuleLoader.moduleProvider (node:internal/modules/esm/loader:291:30)

[jsumners-nr]

There have been a few new releases of import-in-the-middle. Does your issue still persist, @zintus ?