ESM Modules not supported #16

tungnnt309 · 2024-01-05T09:27:08Z

What version of pkg are you using?

5.11.1

What version of Node.js are you using?

16.19.1

What operating system are you using?

macOS

What CPU architecture are you using?

x86_64

What Node versions, OSs and CPU architectures are you building for?

node16-macos-x64

Describe the Bug

This issue is mentioned in the original package, but the package is archived, then I found a solution in issue's comment:
vercel#1936

"pkg": {
   "scripts": [
     "node_modules/axios/dist/node/*",
   	[...]
   ]

But it still does not work

Expected Behavior

Compile app without axios (lastest version) error

To Reproduce

A Project that include the lastest version of axios (e.g: v1.6.4)

The text was updated successfully, but these errors were encountered:

robertsLando · 2024-01-08T08:13:19Z

What's the error you see on console when you run the app? I have axios too in my project and using that solution works for me

manast · 2024-01-18T22:11:20Z

In my case I get these errors to begin with:

> Warning Non-javascript file is specified in 'scripts'.
  Pkg will probably fail to parse. Specify *.js in glob.
  /Users/manuelastudillo/Dev/myapp/node_modules/axios/dist/node/axios.cjs.map
> Warning Babel parse has failed: Missing semicolon. (1:10)

then it continues with the same errors as before:

Warning Failed to make bytecode node18-x64 for file /snapshot/myapp/node_modules/axios/dist/node/axios.cjs.map
> Warning Failed to make bytecode node18-x64 for file /snapshot/myapp/node_modules/axios/index.js
> Warning Failed to make bytecode node18-x64 for file /snapshot/myapp/node_modules/axios/lib/axios.js

and so on...

robertsLando · 2024-01-19T07:18:23Z

I suggest to try pre-building your application using es-build and then pkg against the single file bundle. Otherwise if you are able to spot the error feel free to submit a PR

robertsLando · 2024-02-15T07:14:02Z

Let's use this issue for all the others. Seems that starting from nodejs 18.19.0 and for all nodejs 20+ versions some modules that support both esm and cjs are not bundled correctly. This is IMO due to resolve package that is not able to correctly parse package.json exports fields (see issue).

Let's use this issue as a tracker for such issues instead of having multiple open ones.

The error will look like:

Error [ERR_REQUIRE_ESM]: require() of ES Module C:\snapshot\my-yao-pkg\node_modules\@apollo\server\dist\cjs\index.js from C:\snapshot\my-yao-pkg\main.cjs not supported.
index.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename index.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in C:\snapshot\my-yao-pkg\node_modules\@apollo\server\package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Module.require (pkg/prelude/bootstrap.js:1851:31)
    at Object.<anonymous> (C:\snapshot\my-yao-pkg\main.cjs)
    at Module._compile (pkg/prelude/bootstrap.js:1930:22) {
  code: 'ERR_REQUIRE_ESM'
}

Alternative solutions

Below some solutions I tried and other possible working ones

Using `esbuild`

Actually a possible solution is to use esbuild to pre-compile the bundle in cjs and then use pkg on the output bundle file. I have a working example of this in this repo: https://github.com/zwave-js/zwave-js-ui/blob/master/esbuild.js

Note that this solution doesn't only solve this issue but also produces very small binaries. That example also handles .node native addons, them require a little monkey patch on the file that is described in the link above. Feel free to add some more context to this.

Using `bpkg`

Other possible solution is to use bpkg tool instead of esbuild. I still never tried it so if someone does please give me some feedbacks. It should handle node native addons out of the box better then esbuild as it encodes them in base64 into the bundle.

Use Nodejs Single executable

Docs: https://nodejs.org/api/single-executable-applications.html.

I belive a lot in this feature and I think it will be the future of packaging nodejs applications. Anyway it's still experimental, if you would like to give it a try please add your feedback here.

manast · 2024-02-15T22:14:45Z

The only approach that finally worked for me was to use esbuild, but it was still a huge undertaking, as my project is quite large an complex...

robertsLando · 2024-02-16T08:25:44Z

@manast tried bpkg ? It should work too and should be easier. If I make it work like I want I could consider adding it to pkg behind an option

robertsLando · 2024-03-22T14:32:50Z

This may fix all those issues: nodejs/node#51977 (comment)

Ref: https://joyeecheung.github.io/blog/2024/03/18/require-esm-in-node-js/

luckyyyyy · 2024-03-22T18:34:05Z

This may fix all those issues: nodejs/node#51977 (comment)

Ref: https://joyeecheung.github.io/blog/2024/03/18/require-esm-in-node-js/

That's really exciting news, but I've already given up on using pkg with ESM because I wanted to support the import syntax in the VM and use the latest packages. It looks like I can go back to pkg now?

robertsLando · 2024-03-23T06:24:00Z

@luckyyyyy not yet there is still not an available nodejs lts version with that and I need to play with it a bit

Avivbens · 2024-05-04T02:27:15Z

Let's use this issue for all the others. Seems that starting from nodejs 18.19.0 and for all nodejs 20+ versions some modules that support both esm and cjs are not bundled correctly. This is IMO due to resolve package that is not able to correctly parse package.json exports fields (see issue).

Let's use this issue as a tracker for such issues instead of having multiple open ones.

The error will look like:
Error [ERR_REQUIRE_ESM]: require() of ES Module C:\snapshot\my-yao-pkg\node_modules\@apollo\server\dist\cjs\index.js from C:\snapshot\my-yao-pkg\main.cjs not supported.
index.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename index.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in C:\snapshot\my-yao-pkg\node_modules\@apollo\server\package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Module.require (pkg/prelude/bootstrap.js:1851:31)
    at Object.<anonymous> (C:\snapshot\my-yao-pkg\main.cjs)
    at Module._compile (pkg/prelude/bootstrap.js:1930:22) {
  code: 'ERR_REQUIRE_ESM'
}
Alternative solutions

Below some solutions I tried and other possible working ones

Using esbuild

Actually a possible solution is to use esbuild to pre-compile the bundle in cjs and then use pkg on the output bundle file. I have a working example of this in this repo: https://github.com/zwave-js/zwave-js-ui/blob/master/esbuild.js

Note that this solution doesn't only solve this issue but also produces very small binaries. That example also handles .node native addons, them require a little monkey patch on the file that is described in the link above. Feel free to add some more context to this.

Using bpkg

Other possible solution is to use bpkg tool instead of esbuild. I still never tried it so if someone does please give me some feedbacks. It should handle node native addons out of the box better then esbuild as it encodes them in base64 into the bundle.

Use Nodejs Single executable

Docs: https://nodejs.org/api/single-executable-applications.html.

I belive a lot in this feature and I think it will be the future of packaging nodejs applications. Anyway it's still experimental, if you would like to give it a try please add your feedback here.

My issue

I've been trying for a while to resolve this issue, as I'm stuck behind without the option to add some modules that shipped with the esm format (many packages push versions with this format only).
I've tried to use ESBuild, but seems to have just the same problem 🥲

My `esbuild.ts`

import { PluginBuild, build } from 'esbuild'
import { cwd } from 'node:process'
import { esbuildDecorators } from '@anatine/esbuild-decorators'

const external = [
    // ...Object.keys(devDependencies || {}),
    '@nestjs/microservices',
    '@nestjs/platform-express',
    '@nestjs/websockets/socket-module',
    '@nestjs/microservices/microservices-module',
    '@nestjs/microservices',
    'class-transformer',
    'class-validator',
]

// from https://github.com/evanw/esbuild/issues/1051#issuecomment-806325487
const nativeNodeModulesPlugin = {
    name: 'native-node-modules',
    setup(build: PluginBuild) {
        // If a ".node" file is imported within a module in the "file" namespace, resolve
        // it to an absolute path and put it into the "node-file" virtual namespace.
        build.onResolve({ filter: /\.node$/, namespace: 'file' }, (args) => ({
            path: require.resolve(args.path, { paths: [args.resolveDir] }),
            namespace: 'node-file',
        }))

        // Files in the "node-file" virtual namespace call "require()" on the
        // path from esbuild of the ".node" file in the output directory.
        build.onLoad({ filter: /.*/, namespace: 'node-file' }, (args) => ({
            contents: `
          import path from ${JSON.stringify(args.path)}
          try { module.exports = require(path) }
          catch {}
        `,
        }))

        // If a ".node" file is imported within a module in the "node-file" namespace, put
        // it in the "file" namespace where esbuild's default loading behavior will handle
        // it. It is already an absolute path since we resolved it to one above.
        build.onResolve({ filter: /\.node$/, namespace: 'node-file' }, (args) => ({
            path: args.path,
            namespace: 'file',
        }))

        // Tell esbuild's default loading behavior to use the "file" loader for
        // these ".node" files.
        let opts = build.initialOptions
        opts.loader = opts.loader || {}
        opts.loader['.node'] = 'file'
    },
}

async function myBuilder(tsconfig: string, entryPoints: string[]) {
    const commonjsPlugin: any = await import('@chialab/esbuild-plugin-commonjs')

    const CWD = cwd()

    const buildResult = await build({
        platform: 'node',
        target: 'node20',
        format: 'cjs',
        bundle: true,
        sourcemap: 'external',
        plugins: [
            esbuildDecorators({
                tsconfig,
                cwd: CWD,
            }),
            commonjsPlugin.default(),
            nativeNodeModulesPlugin,
        ],
        tsconfig,
        entryPoints,
        outdir: 'dist',
        external,
        resolveExtensions: ['.ts', '.js'],
        treeShaking: true,
        allowOverwrite: true,
    })
}

;(async () => {
    await myBuilder('tsconfig.json', ['src/main.ts'])
})()

Errors

DanielCaspers · 2024-05-06T22:04:04Z

Let's use this issue for all the others. Seems that starting from nodejs 18.19.0 and for all nodejs 20+ versions some modules that support both esm and cjs are not bundled correctly. This is IMO due to resolve package that is not able to correctly parse package.json exports fields (see issue).

Let's use this issue as a tracker for such issues instead of having multiple open ones.

The error will look like:
Error [ERR_REQUIRE_ESM]: require() of ES Module C:\snapshot\my-yao-pkg\node_modules\@apollo\server\dist\cjs\index.js from C:\snapshot\my-yao-pkg\main.cjs not supported.
index.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead either rename index.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in C:\snapshot\my-yao-pkg\node_modules\@apollo\server\package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Module.require (pkg/prelude/bootstrap.js:1851:31)
    at Object.<anonymous> (C:\snapshot\my-yao-pkg\main.cjs)
    at Module._compile (pkg/prelude/bootstrap.js:1930:22) {
  code: 'ERR_REQUIRE_ESM'
}
Alternative solutions

Below some solutions I tried and other possible working ones

Using esbuild

Actually a possible solution is to use esbuild to pre-compile the bundle in cjs and then use pkg on the output bundle file. I have a working example of this in this repo: https://github.com/zwave-js/zwave-js-ui/blob/master/esbuild.js

Note that this solution doesn't only solve this issue but also produces very small binaries. That example also handles .node native addons, them require a little monkey patch on the file that is described in the link above. Feel free to add some more context to this.

Using bpkg

Other possible solution is to use bpkg tool instead of esbuild. I still never tried it so if someone does please give me some feedbacks. It should handle node native addons out of the box better then esbuild as it encodes them in base64 into the bundle.

Use Nodejs Single executable

Docs: https://nodejs.org/api/single-executable-applications.html.

I belive a lot in this feature and I think it will be the future of packaging nodejs applications. Anyway it's still experimental, if you would like to give it a try please add your feedback here.

While still not an ideal solution, I had similar issues upgrading from Node 18 to Node 20 and was able to get basic support for a pkg built executable with native node modules, embedded C++ binaries, and other JS dist assets bundled and functioning with minimal effort using #10 (comment)

I still agree that for existing known mitigations, refactoring to use ESBuild may prove to be a better long term tactic.

CMCDragonkai · 2024-05-10T10:57:27Z

Just wanted to let you all know that we have a successful usage of esbuild with PKG and node 20 in https://github.com/MatrixAI/Polykey-CLI - it even has native binaries involved and nix involved (all in flake.nix).

Avivbens · 2024-05-10T15:03:33Z

Just wanted to let you all know that we have a successful usage of esbuild with PKG and node 20 in https://github.com/MatrixAI/Polykey-CLI - it even has native binaries involved and nix involved (all in flake.nix).

Thanks for sharing 🙏

Any description of how?

Mikescops · 2024-05-15T07:24:13Z

Hey there, following the post of @CMCDragonkai I also tried to move our CLI to use ESM with esbuild (on node18).

I could manage to get the full flow working, including with native librairies.

The build script is pretty similar to the one in Polykey: https://github.com/Dashlane/dashlane-cli/blob/master/scripts/build.js
I had to do a few changes to make it work well.

I could use got and chai with ESM without problems.

One caveat I encountered is that esbuild is not transpiling the dynamic imports await import into require when bundling to CJS so you have to force the usage of require() in your code if you need this. In my case I have platform specific dependencies to manage so it's important.

If you have time to take a look, I'm really looking for feedback :)

CMCDragonkai · 2024-05-16T05:22:41Z

Hey there, following the post of @CMCDragonkai I also tried to move our CLI to use ESM with esbuild (on node18).

I could manage to get the full flow working, including with native librairies.

The build script is pretty similar to the one in Polykey: https://github.com/Dashlane/dashlane-cli/blob/master/scripts/build.js I had to do a few changes to make it work well.

I could use got and chai with ESM without problems.

One caveat I encountered is that esbuild is not transpiling the dynamic imports await import into require when bundling to CJS so you have to force the usage of require() in your code if you need this. In my case I have platform specific dependencies to manage so it's important.

If you have time to take a look, I'm really looking for feedback :)

Check this evanw/esbuild#2651.

CMCDragonkai · 2024-05-16T05:25:29Z

And this evanw/esbuild#700 regarding dynamic imports.

tuxedoder · 2024-08-20T15:39:18Z

This may fix all those issues: nodejs/node#51977 (comment)

@luckyyyyy not yet there is still not an available nodejs lts version with that and I need to play with it a bit

@robertsLando nodejs/node#51977 mentions the lts pr nodejs/node#54447 and it seems to soon be in v20.17.0, which is the next lts version.

tuxedoder · 2024-08-22T15:44:48Z

v20.17.0 is now released and has --experimental-require-module.

SagePacheco · 2024-08-30T15:36:28Z

@robertsLando, how much of a lift would it be to make the v20.17.0 LTS available to pkg-fetch? I would love to start testing this flag within PKG's context.

robertsLando · 2024-09-02T07:56:42Z

@SagePacheco long story, when I started this fork I know it would have been a pain but didn't thought like this (reason why most of people maintaining this project have give up at a certain point)

I spent weeks on this PR in order to fix compatibility with node 20.14 and ended up being able to make everything working except for windows that actually builds but fails on runtime (check conversation if you are curious).

After many thoughts on this I decided to just merge the PR so users are able to use latest nodejs versions with linux and mac (hope someone will help to fix windows one once issues start) then I discovered that GH removed macos-11 from runners, so I bumped macos runners to use macos-13 and guess what? now mac build are not working and the sad thing is that they just hang without any output.

I'm a bit stuck now as the free time I have to invest in this is very limited and the tests here require long waiting time. I hope someone could help with this

robertsLando · 2024-09-06T06:20:44Z

Just a quick update here. I finally managed to make all build working and new release is coming with latest nodejs 18 and 20.17.0 support 🎉

Just need to wait for this to finish now

Please consider sending some support here: https://github.com/sponsors/robertsLando 🙏🏼 ❤️

robertsLando · 2024-09-06T11:59:03Z

pkg 5.13.0 is ou now with nodejs 20.17.0 and 18.20.4 support 🎉

SagePacheco · 2024-09-06T17:20:20Z

Cheers, @robertsLando! Thanks for working that out. It is much appreciated for you to keep this library alive after Vercel. PKG is hugely important to my organization and will continue to be that way until SEA is worth anything, so let me see what I can talk my leadership into for sponsorship.

In regard to the --experimental-require-module flag... I've just done some extensive testing and there seems to be some good and bad.

The good:
If you require a pure esm package of a single depth (no dependencies), everything will work as expected.

The bad:
If the esm package you're requiring has any underlying dependencies that it brings in using the import syntax... you will be met with an 'ERR_MODULE_NOT_FOUND' message at runtime on those underlying dependencies.

I have even tried to bundle these "missing files" under assets / scripts to force inclusion in the virtual file system and then validated its existence at the expected path using --debug and the DEBUG_PKG env variable. The file will be there in the virtual file system, but the runtime says that it isn't, so I can only assume it's because PKG isn't yet hooked into the import syntax yet.

robertsLando · 2024-09-06T17:27:10Z

@SagePacheco thanks for looking at that, was on my TODOs list but no idea when and if I would ever found the time to look at it. ATM the best way (it's what I do on my projects) is to pre-compile everything with esbuild and I have to say that's very powerful as you also reduce binary size a lot

paulish · 2024-09-20T11:41:35Z

Seems the majority of problems solved with experimental-require-module

Here is a working docker file which I use for my project:

FROM node:20.17.0-alpine as build

# prepare
RUN mkdir -p app
WORKDIR /app

# install dependencies
RUN npm i -g @yao-pkg/pkg@5.15.0
COPY . .
RUN npm i
# build binary
RUN pkg -C GZip --options experimental-require-module --targets node20-alpine -o app-alpine .

FROM node:20.17.0-alpine as release

# for alpine
RUN apk --no-cache add ca-certificates && apk --update --no-cache add curl tzdata bash

WORKDIR /app
COPY --from=build /app/app-alpine ./

# run
# CMD ["bash", "-c", "./app-alpine"]
ENTRYPOINT ["/app/app-alpine"]

robertsLando · 2024-09-20T11:46:49Z

Thanks for the report @paulish !

pmosconi · 2024-10-10T14:26:09Z

I have a CommonJS application that depends on axios and @inquirer/prompts and I have been able to package it like this:

esbuild index.js --bundle --platform=node --target=node20 --outfile=my-app.js
pkg my-app.js --targets node20-win-x64 -o my-app.exe

robertsLando · 2024-10-22T13:07:24Z

Someone wants to give this a try? #110 🚀

un0tec · 2024-11-16T20:46:26Z

@robertsLando is there any workaround to precompile esm modules with top-level-await and then use pgk?
using esm with babel before using pkg works fine, but if I use top-level-await I am forced to put all the code inside (async () => { ...code...})(); Thanks!

luckyyyyy · 2024-11-18T00:03:30Z

Someone wants to give this a try? #110 🚀

Has this PR been completed?

I have a really big project that includes various native addons, which have been obfuscated because ESM isn't supported. I can give it a try.

robertsLando · 2024-11-18T08:56:10Z

is there any workaround to precompile esm modules with top-level-await and then use pgk?

@un0tec If you are using esbuild you can try using --banner:js='(async () => {' --footer:js='})()' that does the same without adding monkey patches

Has this PR been completed?

@luckyyyyy Yes it's completed but in order to work with native addons you may need to patch your code and provide the addons in separated files

robertsLando changed the title ~~Axios build error~~ ESM Modules not supported Feb 15, 2024

robertsLando pinned this issue Feb 15, 2024

robertsLando linked a pull request Feb 16, 2024 that will close this issue

feat: esm support #33

Draft

robertsLando added the enhancement New feature or request label Feb 16, 2024

robertsLando mentioned this issue Apr 18, 2024

pkg fails to respect the exports fields #56

Closed

DanielRuf mentioned this issue Jul 20, 2024

v3: compiled CLI via pkg not working highcharts/node-export-server#543

Open

ANGkeith mentioned this issue Sep 7, 2024

Refactor/esm firecow/gitlab-ci-local#1331

Merged

robertsLando mentioned this issue Sep 30, 2024

Using issue on mac 15.0 (24A335) with node 20 #96

Closed

avvertix mentioned this issue Oct 13, 2024

Update Node and dependencies avvertix/franc-bin#18

Open

ESM Modules not supported #16

ESM Modules not supported #16

Comments

tungnnt309 commented Jan 5, 2024 • edited Loading

What version of pkg are you using?

What version of Node.js are you using?

What operating system are you using?

What CPU architecture are you using?

What Node versions, OSs and CPU architectures are you building for?

Describe the Bug

Expected Behavior

To Reproduce

robertsLando commented Jan 8, 2024

manast commented Jan 18, 2024

robertsLando commented Jan 19, 2024

robertsLando commented Feb 15, 2024 • edited Loading

Alternative solutions

Using esbuild

Using bpkg

Use Nodejs Single executable

manast commented Feb 15, 2024

robertsLando commented Feb 16, 2024

robertsLando commented Mar 22, 2024 • edited Loading

luckyyyyy commented Mar 22, 2024

robertsLando commented Mar 23, 2024

Avivbens commented May 4, 2024

Alternative solutions

Using esbuild

Using bpkg

Use Nodejs Single executable

My issue

My esbuild.ts

Errors

DanielCaspers commented May 6, 2024

Alternative solutions

Using esbuild

Using bpkg

Use Nodejs Single executable

CMCDragonkai commented May 10, 2024 • edited Loading

Avivbens commented May 10, 2024

Mikescops commented May 15, 2024

CMCDragonkai commented May 16, 2024

CMCDragonkai commented May 16, 2024

tuxedoder commented Aug 20, 2024

tuxedoder commented Aug 22, 2024

SagePacheco commented Aug 30, 2024

robertsLando commented Sep 2, 2024 • edited Loading

robertsLando commented Sep 6, 2024 • edited Loading

robertsLando commented Sep 6, 2024

SagePacheco commented Sep 6, 2024

robertsLando commented Sep 6, 2024

paulish commented Sep 20, 2024

robertsLando commented Sep 20, 2024

pmosconi commented Oct 10, 2024

robertsLando commented Oct 22, 2024

un0tec commented Nov 16, 2024

luckyyyyy commented Nov 18, 2024

robertsLando commented Nov 18, 2024

tungnnt309 commented Jan 5, 2024 •

edited

Loading

robertsLando commented Feb 15, 2024 •

edited

Loading

Using `esbuild`

Using `bpkg`

robertsLando commented Mar 22, 2024 •

edited

Loading

Using `esbuild`

Using `bpkg`

My `esbuild.ts`

Using `esbuild`

Using `bpkg`

CMCDragonkai commented May 10, 2024 •

edited

Loading

robertsLando commented Sep 2, 2024 •

edited

Loading

robertsLando commented Sep 6, 2024 •

edited

Loading