Pure ES Modules break the Mocha tests #3779

eatyourgreens · 2022-10-14T16:29:51Z

Updated 2024 - List of third-party packages this applies to:

(Dependabot has been told to ignore the major version upgrades)

Originally posted by @shaunanoordin in #3361 (comment) The tests for app-project and lib-classifier are failing on:

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/shaun/projects/front-end-monorepo/node_modules/d3/src/index.js from /Users/shaun/projects/front-end-monorepo/packages/app-project/src/shared/components/ProjectStatistics/components/CompletionBar/CompletionBar.js not supported.
Instead change the require of index.js in /Users/shaun/projects/front-end-monorepo/packages/app-project/src/shared/components/ProjectStatistics/components/CompletionBar/CompletionBar.js to a dynamic import() which is available in all CommonJS modules.

and

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/shaun/projects/front-end-monorepo/node_modules/d3/src/index.js from /Users/shaun/projects/front-end-monorepo/packages/lib-classifier/src/components/Classifier/components/Feedback/components/Graph2dRangeFeedback.js not supported.
Instead change the require of index.js in /Users/shaun/projects/front-end-monorepo/packages/lib-classifier/src/components/Classifier/components/Feedback/components/Graph2dRangeFeedback.js to a dynamic import() which is available in all CommonJS modules.

and etc

OK, so upon review:

D3 now ships as pure ES modules and requires Node.js 12 or higher. - this IS a problem.

Additional reading:

The awkward valley to ESM: Node.js, Victory, and D3 - 15 Jun 2022, formidable.com - TL;DR: as more libraries (e.g. d3.js) drop CommonJS support and go 100% ESM (ECMAScript Modules), a lot of upstream/downstream dependencies that use CommonJS get borked. (ESM setups can read ESM + CommonJS modules, but CommonJS setups can't read ESModules.) Awkward transpiling workarounds are required.
Importing Issues - Error [ERR_REQUIRE_ESM]: Must use import to load ES Module - 29 Jun 2021, D3 repo - TL;DR: lol good luck
import conflict with 3.0.0 update - 25 Apr 2021, D3 repo - TL;DR: lol not a d3 problem, fix your NextJS
- Yeah, but lib-classifier has the same error, and it's not running Next.JS. Also, I think our tests aren't running Next.JS?

Status

🤷 throw_hands_up_emoji

Good gravy, I need to check with the other devs to see if anybody else has encountered the ESM import issues. I might have missed something someone on the team is already aware of, since this doesn't seem to be a niche problem.

The text was updated successfully, but these errors were encountered:

eatyourgreens · 2022-10-14T17:35:52Z

Why our Mocha tests are using CommonJS to require modules, I do not know. Some Babel config rewriting import as require?

eatyourgreens · 2022-10-14T17:46:26Z

We could try adding regenerator-runtime/runtime to the Mocha config.
https://stackoverflow.com/a/60522428

eatyourgreens · 2022-10-14T17:49:29Z

We could also try publishing the libraries as ES Modules, but need to be mindful of not breaking Zoo Notes or Mapping Viz.

eatyourgreens · 2022-10-15T09:58:16Z

Mocha Babel examples might be useful.

eatyourgreens · 2022-11-28T23:11:57Z

#3900 adds ES modules to @zooniverse/classifier and @zooniverse/react-components, alongside the UMD modules built by webpack. Builds and deploys use the ES modules. Mocha tests require the UMD modules. Everything works but we need webpack builds, which are pretty big and slow, in order to run the tests.

#3920 converts @zooniverse/async-states, which is a single JS file, from CommonJS to ES6. Builds that use it run fine, but tests fail for both NextJS apps and the classifier, when they try to require the ES module.

eatyourgreens · 2023-01-17T19:23:12Z

#4064 removed the webpack UMD builds. The classifier and Zooniverse React Components are published as ESM, for the NextJS builds, and CommonJS for the Mocha tests.

goplayoutside3 · 2023-06-01T16:58:30Z

Going to close #3781 for now, but linking here because of relation to ESM

eatyourgreens · 2023-09-11T22:52:34Z

@zooniverse/panoptes-js also uses require and module.exports (CommonJS.)

eatyourgreens · 2023-09-12T14:48:36Z

Shaun's ESM notes here are very useful.

Change to ESM (ECMAScript module) community-catalog#165

eatyourgreens · 2023-09-12T14:52:54Z

next-i18next breaks if you convert the config file to ES6.

Usage example for ESM i18next/next-i18next#2185

There may be a workaround. See the conversation on that issue.

eatyourgreens · 2023-09-12T14:55:08Z

next-i18next also breaks if you rename the config to next-i18next.config.cjs.

eatyourgreens · 2023-09-14T15:54:50Z

Bun has a good explanation of Node module resolution and the differences between CommonJS and ESM.

https://bun.sh/docs/runtime/modules

eatyourgreens · 2024-03-19T09:48:34Z

Node 21 now has experimental support for require(esm).

https://joyeecheung.github.io/blog/2024/03/18/require-esm-in-node-js/

eatyourgreens · 2024-10-20T07:53:53Z

require(esm) is now default behaviour in Node 23. 🎉

https://nodejs.org/en/blog/release/v23.0.0

I think you can enable it with a flag in Node 22, which will enter LTS next week.

Top level await is still restricted to modules.

With this feature enabled, Node.js will no longer throw ERR_REQUIRE_ESM if require() is used to load a ES module. It can, however, throw ERR_REQUIRE_ASYNC_MODULE if the ES module being loaded or its dependencies contain top-level await. When the ES module is loaded successfully by require(), the returned object will be a ES module namespace object similar to what's returned by import(), and can be checked using util.isModuleNamespaceObject().

eatyourgreens · 2024-10-23T09:07:12Z

How to convert CommonJS to ESM, just published yesterday.

eatyourgreens · 2024-10-25T14:55:37Z

If you switch to Node 22, then yarn bootstrap and yarn test both fail 😞. I discovered that by accident today, when I forgot to switch back from Node 22 to Node 20 while building the monorepo.

It also looks like module loading has changed in Node 22. This warning is from lib-content, but it applies to all the monorepo packages without type declarations:

@zooniverse/content:test
$ mocha --config ./test/.mocharc.json ./.storybook/specConfig.js "./src/**/*.spec.js"
      (node:69592) [MODULE_TYPELESS_PACKAGE_JSON] Warning: Module type of file:///Users/jimodonnell/zooniverse/front-end-monorepo/packages/lib-content/test/setup.js is not specified and it doesn't parse as CommonJS.
      Reparsing as ES module because module syntax was detected. This incurs a performance overhead.
      To eliminate this warning, add "type": "module" to /Users/jimodonnell/zooniverse/front-end-monorepo/packages/lib-content/package.json.
      (Use `node --trace-warnings ...` to show where the warning was created)

eatyourgreens · 2024-10-25T18:20:16Z

require(esm) is also available in Node 20.17, behind the --experimental-require-module flag. That should allow you to upgrade packages without having to upgrade Node.

https://nodejs.org/en/blog/release/v20.17.0

goplayoutside3 · 2024-12-06T17:04:21Z

I was investigating this mocha config x ESM issue yesterday and want steer this Github discussion toward potential solutions for the monorepo, ideally without completely change our testing architecture.

Please correct me if I'm wrong, but in summary:

FEM's mocha config uses @babel/register to transpile. The purpose of babel is to transpile your js code to an understandable version of js for the given tool you're using. Mocha does not understand JSX, so in FEM babel is a must.
Third-party package maintainers are moving toward ESM-only, which is fine for building and running the Next.js apps, or Storybook in the browser, but throws Error [ERR_REQUIRE_ESM]: require() of ES Module [third-party package] from [FEM component] not supported in a testing environment. For example: mime v4 in lib-content's unit tests.
Mocha can work with ESM-only. This requires type: module in the package.json.
- I experimented with adding type: module to lib-content, ensuring no require() syntax in the library code, and even excluding its jsdom setup file. However, this config results in the problem documented here where I saw Unexpected token '<' thrown on the first .spec.js file. babel/register does not support compiling native Node.js ES modules on the fly (bottom of their docs here).
- At this point, I'm not sure if this shortcoming of @babel/register is a complete blocker to FEM's architecture of React + Babel + Mocha.

@eatyourgreens you've left a few comments with potential solutions. Did you try any of these combos?

The --experimental-require-module flag with Node v20
Node v22 with type: module in a library?

eatyourgreens · 2024-12-06T17:15:29Z

I just ran a very quick experiment with Node 22:

nvm use 22
yarn install --ignore-scripts
yarn bootstrap

The project app build fails with:

unhandledRejection TypeError: Cannot set properties of undefined (setting 'eio')
    at /Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/panoptes-client/lib/SugarClient/primus.js:3805:9
    at Array.<anonymous> (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/panoptes-client/lib/SugarClient/primus.js:3806:3)
    at UMDish (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/panoptes-client/lib/SugarClient/primus.js:4:15)
    at Object.<anonymous> (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/panoptes-client/lib/SugarClient/primus.js:11:3)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Object..js (node:internal/modules/cjs/loader:1689:10)
    at Module.load (node:internal/modules/cjs/loader:1318:32)
    at Function._load (node:internal/modules/cjs/loader:1128:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:218:24) {
  type: 'TypeError'
}

goplayoutside3 · 2024-12-06T17:18:35Z

cannot set properties of undefined (setting 'eio')

This looks similar to the hexoid error previously thrown by app-root that was fixed by using alias in the webpack config object in next.config.js.

eatyourgreens · 2024-12-06T17:20:22Z

One problem the monorepo has is that both JS and JSX files use the .js extension, so build tools would have to assume that every .js file is a JSX file and transpile to JS.

If you look at PFE, for example, I think JSX files use .jsx there. Changing all the file extensions for the monorepo doesn't seem practical, though.

eatyourgreens · 2024-12-06T17:23:31Z

Oh, looks like I already tried Node 22 back in October: #3779 (comment)

goplayoutside3 · 2024-12-06T17:24:21Z

Oh, looks like I already tried Node 22 back in October:

Right, that's why I was curious if you'd tried Node v22 while also adjusting type: module in package.json files.

eatyourgreens · 2024-12-06T17:30:29Z

Adding type: module to lib-content, then running the tests, errors, because the tests set window.navigator, which is read-only.

@zooniverse/content:test
$ mocha --config ./test/.mocharc.json ./.storybook/specConfig.js "./src/**/*.spec.js"
      
       Exception during run: TypeError: Cannot set property navigator of #<Object> which has only a getter
          at file:///Users/jimodonnell/zooniverse/front-end-monorepo/packages/lib-content/test/setup.js:43:18
          at ModuleJob.run (node:internal/modules/esm/module_job:268:25)
          at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:543:26)
          at async formattedImport (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/mocha/lib/nodejs/esm-utils.js:9:14)
          at async exports.requireOrImport (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/mocha/lib/nodejs/esm-utils.js:42:28)
          at async exports.loadFilesAsync (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/mocha/lib/nodejs/esm-utils.js:100:20)
          at async singleRun (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/mocha/lib/cli/run-helpers.js:162:3)
          at async exports.handler (/Users/jimodonnell/zooniverse/front-end-monorepo/node_modules/mocha/lib/cli/run.js:375:5)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Modules run in strict mode, so that's probably the source of that error.

eatyourgreens · 2024-12-06T17:33:04Z

Removing the offending line, then running the lib-content tests again, gets us back to unexpected token < because JSX isn't JS.

@zooniverse/content:test
$ mocha --config ./test/.mocharc.json ./.storybook/specConfig.js "./src/**/*.spec.js"
      
       Exception during run: SyntaxError[ @/Users/jimodonnell/zooniverse/front-end-monorepo/packages/lib-content/.storybook/specConfig.js ]: Unexpected token '<'
          at compileSourceTextModule (node:internal/modules/esm/utils:340:16)
          at ModuleLoader.moduleStrategy (node:internal/modules/esm/translators:102:18)
          at #translate (node:internal/modules/esm/loader:433:12)
          at ModuleLoader.loadAndTranslate (node:internal/modules/esm/loader:480:27)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

eatyourgreens · 2024-12-06T17:36:01Z

Mocha tests containing JSX with ESM suggests running Mocha with a custom Node loader, but I don't know how old that is or which versions of Node it was written for.

@node-loader/babel looks like it's still maintained, so could be worth trying.

eatyourgreens · 2024-12-06T17:41:04Z

I have to confess that currently I use Storybook with Playwright to run tests for JSX components in a pure ESM project, and it just works out-of-the-box.

https://github.com/nismod/irv-jamaica/blob/83a31ea2e4a20b4a7ab42555b262b4af8894dfcc/.github/workflows/storybook-tests.yml#L19-L29

eatyourgreens · 2024-12-06T18:47:09Z

I was investigating this mocha config x ESM issue yesterday and want steer this Github discussion toward potential solutions for the monorepo, ideally without completely change our testing architecture.

This seems like an excellent summary to me. I'd add a couple of points:

Older packages like lib-panoptes-js were written before Node supported import/export, so are still using require and module.exports. This leads to CI errors in Panoptes.js: add lib-panoptes-js dev server, add experimental auth #6375, where browser code (import/export) exists alongside older Node code(require/module exports). Now that Node supports import and export, it might be worth taking some time to remove require and module exports= from all packages. That might require converting them from type: commonjs to type: module.
Up until Node 20, type: commonjs is the default for any package that doesn't explicitly specify a type. The Node 22 module loader seems to dynamically determine a type, by inspecting a file for import/export syntax. This can lead to errors if a file uses ES6 syntax but doesn't run in strict mode. I'd suggest explicitly setting type: commonjs on all packages, until they've been converted to ESM. You can also test if files work in strict mode by adding the use strict directive at the top of a file.

eatyourgreens · 2024-12-07T08:33:45Z

This looks similar to the hexoid error previously thrown by app-root that was fixed by using alias in the webpack config object in next.config.js.

The Sugar client is crashing here, when it adds engine-io to the window (or to the Primus client instance).

https://github.com/zooniverse/panoptes-javascript-client/blob/721b696dccecf86a35614244ab29d19d1a62c3b7/lib/SugarClient/primus.js#L3795-L3806

The Sugar client, and in fact the Panoptes JS Client in general, only runs in browsers, but monorepo code runs in Node and in browsers. So that might be causing problems when window is undefined.

Something to be aware of in #6375 too. That PR is adding browser-only code to a library, @zooniverse/panoptes-js, that's designed to be isomorphic. I still recommend taking a look at the new, isomorphic Panoptes auth client that Roger started writing for the monorepo (#6376), if only to get an idea of what problems you might run into adapting auth to run in Next.js.

eatyourgreens · 2024-12-07T22:31:21Z

#3920 converted lib-async-states to ESM. It might be worth revisiting that with Node 20 or Node 22.

goplayoutside3 · 2024-12-10T23:49:12Z

I've started to track the third-party libraries that cause tests to fail in the top level description of this Issue.

eatyourgreens · 2024-12-11T10:44:37Z

I tagged Dependabot PRs with ESM if they break the tests. I don’t know if that still happens. Labels are useful if you want to track a dynamic list of PRs.

https://github.com/zooniverse/front-end-monorepo/issues?q=label%3AESM

eatyourgreens mentioned this issue Oct 14, 2022

[Security] Bump d3 from 6.7.0 to 7.6.1 #3781

Closed

10 tasks

eatyourgreens mentioned this issue Nov 18, 2022

Add ES6 builds to libraries #3900

Merged

11 tasks

eatyourgreens mentioned this issue Nov 28, 2022

Convert @zooniverse/async-states to ES6 #3920

Closed

11 tasks

eatyourgreens mentioned this issue Sep 11, 2023

Upgrade Remark #2607

Closed

eatyourgreens mentioned this issue Sep 12, 2023

Deprecate (and remove) the TESS light curve viewer #5324

Open

6 tasks

eatyourgreens mentioned this issue Sep 18, 2023

Markdownz: replace remark with markdownz #5352

Merged

11 tasks

mcbouslog mentioned this issue Oct 23, 2023

lib-user: add Grommet, zooTheme, storybook, and testing setup #5475

Merged

5 tasks

goplayoutside3 mentioned this issue Dec 15, 2023

build(deps): bump mime from 3.0.0 to 4.0.0 #5711

Closed

goplayoutside3 added this to Maintenance Aug 7, 2024

goplayoutside3 moved this to To do in Maintenance Aug 7, 2024

goplayoutside3 mentioned this issue Oct 19, 2024

fix(app-root): wait for login to complete before fetching authenticated user data #6345

Merged

12 tasks

eatyourgreens mentioned this issue Dec 8, 2024

build: use a base image for Docker builds #6515

Open

11 tasks

This was referenced Dec 10, 2024

build(deps): bump react-player from 2.13.0 to 2.16.0 #6436

Closed

build(deps): bump mime from 3.0.0 to 4.0.4 #6265

Closed

goplayoutside3 added the ESM label Dec 10, 2024

goplayoutside3 mentioned this issue Dec 10, 2024

build(deps): bump graphql-request from 6.1.0 to 7.1.2 #6427

Closed

goplayoutside3 mentioned this issue Dec 10, 2024

build(deps-dev): bump sinon-chai from 3.7.0 to 4.0.0 #6543

Closed

goplayoutside3 mentioned this issue Dec 18, 2024

build(deps): bump mime from 3.0.0 to 4.0.6 #6584

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pure ES Modules break the Mocha tests #3779

Pure ES Modules break the Mocha tests #3779

eatyourgreens commented Oct 14, 2022 •

edited by goplayoutside3

Loading

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 15, 2022

eatyourgreens commented Nov 28, 2022 •

edited

Loading

eatyourgreens commented Jan 17, 2023

goplayoutside3 commented Jun 1, 2023

eatyourgreens commented Sep 11, 2023

eatyourgreens commented Sep 12, 2023 •

edited

Loading

eatyourgreens commented Sep 12, 2023

eatyourgreens commented Sep 12, 2023

eatyourgreens commented Sep 14, 2023

eatyourgreens commented Mar 19, 2024 •

edited

Loading

eatyourgreens commented Oct 20, 2024 •

edited

Loading

eatyourgreens commented Oct 23, 2024

eatyourgreens commented Oct 25, 2024 •

edited

Loading

eatyourgreens commented Oct 25, 2024 •

edited

Loading

goplayoutside3 commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024

goplayoutside3 commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024

goplayoutside3 commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 7, 2024 •

edited

Loading

eatyourgreens commented Dec 7, 2024

goplayoutside3 commented Dec 10, 2024

eatyourgreens commented Dec 11, 2024 •

edited

Loading

Pure ES Modules break the Mocha tests #3779

Pure ES Modules break the Mocha tests #3779

Comments

eatyourgreens commented Oct 14, 2022 • edited by goplayoutside3 Loading

Status

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 14, 2022

eatyourgreens commented Oct 15, 2022

eatyourgreens commented Nov 28, 2022 • edited Loading

eatyourgreens commented Jan 17, 2023

goplayoutside3 commented Jun 1, 2023

eatyourgreens commented Sep 11, 2023

eatyourgreens commented Sep 12, 2023 • edited Loading

eatyourgreens commented Sep 12, 2023

eatyourgreens commented Sep 12, 2023

eatyourgreens commented Sep 14, 2023

eatyourgreens commented Mar 19, 2024 • edited Loading

eatyourgreens commented Oct 20, 2024 • edited Loading

eatyourgreens commented Oct 23, 2024

eatyourgreens commented Oct 25, 2024 • edited Loading

eatyourgreens commented Oct 25, 2024 • edited Loading

goplayoutside3 commented Dec 6, 2024 • edited Loading

eatyourgreens commented Dec 6, 2024

goplayoutside3 commented Dec 6, 2024 • edited Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024

goplayoutside3 commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 • edited Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 • edited Loading

eatyourgreens commented Dec 6, 2024

eatyourgreens commented Dec 6, 2024 • edited Loading

eatyourgreens commented Dec 7, 2024 • edited Loading

eatyourgreens commented Dec 7, 2024

goplayoutside3 commented Dec 10, 2024

eatyourgreens commented Dec 11, 2024 • edited Loading

eatyourgreens commented Oct 14, 2022 •

edited by goplayoutside3

Loading

eatyourgreens commented Nov 28, 2022 •

edited

Loading

eatyourgreens commented Sep 12, 2023 •

edited

Loading

eatyourgreens commented Mar 19, 2024 •

edited

Loading

eatyourgreens commented Oct 20, 2024 •

edited

Loading

eatyourgreens commented Oct 25, 2024 •

edited

Loading

eatyourgreens commented Oct 25, 2024 •

edited

Loading

goplayoutside3 commented Dec 6, 2024 •

edited

Loading

goplayoutside3 commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 6, 2024 •

edited

Loading

eatyourgreens commented Dec 7, 2024 •

edited

Loading

eatyourgreens commented Dec 11, 2024 •

edited

Loading