Feature: ESM in executable files

[WebReflection]

Coming from this comment: nodejs/modules#151 (comment)

It is now possible to create executable, extension-less, files:

#!/usr/bin/env node
console.log(__filename);

But it's not possible to enable ESM as parsing goal.

#!/usr/bin/env node -m
console.log(import.meta.url);

Even if there are OS incapable to parse the whole shebang up to the -m flag (or whatever flag will land in nodejs), there is no way to even define an executable that would like to parse the source file without any extension.

Example

Given the following esm file, reachable through /usr/local/bin or similar OS folder:

#!/usr/bin/env bash
node --module $1

And given the following executable:

#!/usr/bin/env esm
console.log(import.meta.url);

It should be possible to have extensions-less files parsable as ESM.

Update

There is a solution to the single file problem that would still require --module hook to bootstrap the file as ESM parse goal.

[bmeck]

There are a variety of solutions we should probably evaluate and see if they are or are not feasible that I can think of.

Right now we can point executables without extensions to other files using symlinks. I'm not curious about what file extension maps to what format, but I am curious about if that disambiguation and/or indirection method is infeasible. I'd be curious about places that cannot support that behavior. Most installations put the .cmd/Symlink as an indirection layer to a folder and the main file for "bin" installations through package.json so this probably falls on if the file extension can be used, whatever it maps to.

In addition the executable could stay CJS and require/import() to another file to change the mode. This also is a method of indirection, but is not relying on file extension and instead relying on w/e disambiguation method is done at runtime, even if it uses the file extension it isn't mandating such.

I also am concerned about hashbang being unusable not just on linux shells but on windows. I agree it is a design space that we can look into but I am not sure how feasible it is. I do not think loaders can solve this problem using a hashbang either because they currently are not runtime configurable, nor am I convinced they should become runtime configurable due to this single issue. If we were to use a loader configuration it would need to be done outside of the file using CLI/ENV/package.json/etc. That makes me think we might want to look at indirection so that we can get that out of band data somehow.

I do have scaling concerns for having multiple executables because things like BinaryAST / WASM / WebPackage are other goals I'd like to support in the future and they would explode the number of executables that node ships with. I think however this format is configured and sent to node it should be planned around adding at least those 2 formats as well.

I do not think wrapping processes are a way to solve this problem either. Notably, Windows does not have an exec capability like unix/shells that can replace the same process ID, and using child_process would create wrappers that might be problematic for various process ID tracking service manager.

[bmeck]

Forgot to add, that even though Loaders are not configurable, they could be done in a per package manner for things that are not installed using "bin" from package.json approaches.

[WebReflection]

Not sure I follow, but I've used Linux in pretty much every dev/production env I know and written dunno how many binary files based on the hashbang #!/usr/bin/env node.

I also am concerned about hashbang being unusable not just on linux shells but on windows.

NodeJS ignores hashbang too since ever because it's been a valid use case for long time so it shouldn't probably be under discussion the ability to create nodejs executable based on ESM, but I hope I've misunderstood your reply.

[bmeck]

@WebReflection that is executing the node executable as the shell that invoked the file parsed the hashbang (well env is invoked really). My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable. Even then, the declaration of the executable through the hashbang is picked up by the shell and not node itself.

[WebReflection]

My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable.

then I misunderstood, and indeed since it doesn't work in the only env I've used it (Linux) I guess we would need a better way.

I don't have ideas for now but I'll keep thinking about it.

[xtuc]

If we want to avoid passing parameters in the shebang, we could create an excutable wrapper that passes them (nodem, node-m).

But from my point of view that would be confusing for people used to run node.

[guybedford]

This is one of the benefits of the package.json "mode" proposal in that it can allow this scenario to be handled clearly (when executing the "bin", the package.json is loaded and used as source-of-truth of the format).

@xtuc's suggestion is an interesting alternative here too, although agreed being weary of adding to confusion.

[GeoffreyBooth]

Don’t forget the case where the extensionless file is outside the package folder, e.g. /usr/local/bin. For example, when CoffeeScript is installed globally, e.g. npm install --global coffeescript, it puts a symlink /usr/local/bin/coffee that points to /usr/local/lib/node_modules/coffeescript/bin/coffee (on Mac/Linux systems, anyway). Maybe in the case of followed symlinks Node can look for a package.json in the folder where the symlink resolves.

[devsnek]

node by default uses the extension of the resolved file, not the symlink e.g. file.json -> file.js is seen as cjs not json

[WebReflection]

FWIW @xtuc suggestion is exact equivalent of my ./esm binary with content:

#!/usr/bin/env bash
node --module $1

This is a pragmatic solution to the shebang gotcha but it feels future hostile needing that in order to have an executable based on modern code so, unfortunately, we might need a better idea than just an indirection 😢

[bmeck]

@WebReflection whats wrong with the symlink indirection that most things do today?

[WebReflection]

@bmeck if I understand correctly that would require a package.json somewhere else, right? In AUR (or other packaging formats different from npm) that might not be easy/straight forward to implement.

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

[bmeck]

@WebReflection it would rely on some file that it points to being unambiguous, yea. Not necessarily using package.json, could be a file extension, etc.

[bmeck]

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

I think this is where all of these are breaking down. Perhaps, the way that things are done today doesn't scale well, even at 2 possibilities we are seeing problems unless we define a different mechanism than today. Wait for BinaryAST and WASM entrypoints and we have 4 possibilities.

[WebReflection]

@bmeck I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?

Anyway, I forgot I have some hackery to start GJS so that this would be my solution:

#!/usr/bin/env bash
Function=Function//; node -m "$0" "$@"; exit

console.log('ESM');

Explanation

• the shebang is the most widely compatible one
• in bash, Function has no meaning but you can assign it to anything, including slashes //
• the ; after slashes ends the variable assignment and bootstrap node -m "$0"
• the rest of the arguments is passed along via "$@" and the bash program exits (after node exits too so nothing else will be executed / parsed / interpreted from bash)
• the node will execute the file as module, it will ignore the shebang and it will also ignore the assignment of the Function to itself, or better, there won't be any side effect, and the comment will nullify bash.

All it's missing now, is this mechanism to bypass the file extension and enable the --module like parser so that import.meta.url or any other ESM related syntax would be valid.

[bmeck]

I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?

I think you are over focusing on a hashbang based solution. Hashbang doesn't work for a variety of cases or at all on some deployment targets like windows. I'd be wary of things that can vary per shell and aren't even in some environments.

[WebReflection]

windows has numerous way to bring in bash though, but yeah, I'm focusing on developers, those that actually use NodeJS the most, and on Servers, those by stats dominated by Linux.

Focusing on windows without Linux subsystem or chocolate bash, aka focusing on non developers, looks like not so important. Windows has many ways to ship binaries, including ways compatible, or based, on shebangs.

npm would also solve that case via bin, but we miss a way to #!/usr/bin/env node which has been historically both possible and very much used for server side development.

[WebReflection]

npm would also solve that case via bin

even better, you could use my technique both via npm and as a stand alone file/application, so I don't see any reason to ignore it as use case/solution.

[MylesBorins]

Is this feature documented? can this be closed?

[GeoffreyBooth]

Yes, it’s in the features list in the README: https://github.com/nodejs/modules#existing-nodejs-utility-features

[MylesBorins]

Closing based on above comment. please re-open (or ask me to) if this was a mistake

[curiousdannii]

I don't think this should've been closed until there's a reliable way to do this cross platform without relying on bash hacks.

FWIW, the unambiguous grammar proposal would've easily solved this. :(

[WebReflection]

FWIW, the unambiguous grammar proposal would've easily solved this.

FWIW, I think that's the only way to solve this (without bash hackery)

[GeoffreyBooth]

FWIW, I think that’s the only way to solve this (without bash hackery)

Or the shebang suggestion, like starting your file with #!/usr/bin/env node --mode=esm, but people pointed out that that had various issues such as Windows compatibility.

I had suggested elsewhere that unambiguous grammar could be treated as a fallback, like if the mode wasn’t explicitly defined via something in package.json or a CLI flag, then Node would try to figure it out by parsing the entrypoint (and then all subsequent imported files would need an explicit marker like an .mjs filename or package.json field). People didn’t like this approach, but I’m still not persuaded why simply halting on an import statement is better, especially if the behavior is limited to extensionless files. I understand the concern where once unambiguous grammar is in the runtime at all, people will take advantage of it as a way to avoid needing to specify the module mode any other way, and then we need to deal with the complexity of adding new syntax in the future, like import(), while avoiding breaking old scripts that depend on some older unambiguous grammar algorithm.

The minimal kernel currently in progress inherits the assumption from --experimental-modules that module mode is determined via metadata in the script or module, e.g. file extension or package.json property—a.k.a. author-driven disambiguation. This is at the same time that the minimal kernel (and Node core) just adopted createRequireFromPath, which essentially enables the NPM implementation’s approach of consumer-driven disambiguation where require is used to import CommonJS and import is used to import ESM, and the metadata from file extension or package.json isn’t used (yet). I understand the arguments against consumer-driven disambiguation, not least that it’s way more frustrating for users who would then need to figure out what type of module they’re working with in order to use it. But author-driven disambiguation could be achieved through unambiguous grammar, if we could come up with a bulletproof unambiguous grammar algorithm that worked and we were convinced would be future-proof. Of course, good luck with that.

Even if that could be designed, though, relying on unambiguous grammar for determining module mode has its own issues as listed succinctly in nodejs/modules#150 (comment). Part of me still would love for someone to write up a proposal making the case for author-driven disambiguation via unambiguous grammar; @bmeck did a lot of work on unambiguous grammar years ago, but basically walked away from it as he didn’t think it could be made to work. But if someone wants to give it a shot, I’d love to see another attempt 😄even if we come to the same conclusion. If nothing else, it would help persuade the mob that every attempt was made to try to come up with a more developer-friendly UX than whatever we end up shipping with. Or if a workable unambiguous syntax algorithm could be designed, well, that would be very interesting. We’d still have the other issues to address, perhaps by shipping unambiguous grammar along with metadata flags like file extension and package.json fields, but we’d have more options.

[WebReflection]

#!/usr/bin/env node --mode=esm

Unfortunately the issue there is not just Windows, but Linux too, where that wouldn't work, you need my suggested hacky bootstrap.

But executables with shebangs are extremely common in Linux (servers), so anything that wouldn't work in there makes little sense, in terms of wasted time, IMO.

[ljharb]

@GeoffreyBooth it’s not really worth another attempt - it would require TC39 to change the spec, and a number of members of the committee have expressed legitimate opposition such that (unfortunately) it basically can never happen.

[GeoffreyBooth]

it’s not really worth another attempt - it would require TC39 to change the spec

We thought the same thing about named exports, but they appear to have moved on that, haven’t they?

Again I don’t see how a proposal could resolve all or most of the issues raised in nodejs/modules#150 (comment), so like you I’m not optimistic that an unambiguous grammar solution is possible. But I would still love to see someone take a crack at persuading me (and perhaps the TC39) otherwise. All it takes is a gist explaining in pseudocode what the proposed unambiguous grammar algorithm would be (perhaps based on @bmeck’s prior work) and how it would be future-proof, and how to address the other issues raised by nodejs/modules#150 (comment).

[ljharb]

@GeoffreyBooth i don't think you have an accurate understanding; it's more that the committee has always wanted to help node, but previous attempts weren't tenable. The recent attempt, however, was tenable. As it relates to unambiguous parsing, the committee as a whole actively does not want unambiguous parsing, no matter the solution - many members explicitly want it to be ambiguous, and as such, it's not worth any attempt, because conceptually it's doomed from the start (as dynamic named exports were not).

[ljharb]

@GeoffreyBooth the context hasn’t changed; mjs and mimes were the plan during the entire unambiguous grammar process with TC39 - it wasn’t as long ago as you think.

Certainly node can decide to require unambiguous grammar - but we’d be doing that by restricting ourselves to a subset of JavaScript programs, which doesn’t seem like a good idea for node dot JS to be doing. As it relates to TC39, there’s really no further debate to be had. If you want to debate doing our own thing, we can, but i don’t see how that’s legitimate or productive.

[curiousdannii]

@devsnek Apologies for not actually checking that an executable .mjs script would just work. With all the trouble I had for extensionless scripts I'd assumed that at some time I must have also tried it with the extension, but I guess I never had (even though my solution was to have the extension and then make a sh launcher script!) In my defence the docs don't say that it's possible or that this exception to needing --experimental-modules exists :).

[stuartpb]

Pardon if I'm misunderstanding the documentation, but can't a module-based extensionless script work as ESM by putting itself in a package's bin directory and having "type": "module" in package.json?

Is this a new development since this issue was last commented on, or is that was meant when people mentioned NPM being able to handle it above?

[stuartpb]

Oh, the problem is that that only applies if --experimental-modules is set, isn't it? And it's not possible to set multiple flags for a shebang?

What's the roadmap for when ESM will be enabled without a flag? Sometime in 2020?

[GeoffreyBooth]

This works now, and you’re right the --experimental-modules makes it awkward:

 ✦ cd /tmp && mkdir test && cd test
 ✦ echo '{"type": "module"}' > package.json
 ✦ echo '#!/usr/bin/env node\nimport { version } from "process";\nconsole.log(version);' \
   > executable
 ✦ chmod +x ./executable
 ✦ export NODE_OPTIONS='--experimental-modules'
 ✦ ./executable
(node:54196) ExperimentalWarning: The ESM module loader is experimental.
v12.6.0

You can use clever shell scripting to avoid the need for package.json: https://gist.github.com/WebReflection/8840ec29d296f2fa98d8be0102f08590 (via nodejs/modules#318).

The plan is to unflag when Node 12 goes LTS in October 2019.

[talentlessguy]

With unflagged modules in 15.5 I'm still getting the error:

node:internal/process/esm_loader:74
    internalBinding('errors').triggerUncaughtException(
                              ^

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ""

package.json:

"type": "module",
  "module": "index.js",
  "bin": "simple-ddos",

[ljharb]

You also don’t ever need type module; you can name your file with .mjs instead.

I’m not sure extensionless files work with ESM at all, though, unless they’re the node entry point and you pass the input-type flag.

[talentlessguy]

@DerekNonGeneric thanks for the tip but it drops the same error :/

@ljharb I tried to do #!/usr/bin/env node --input-type="module" but it gets stuck this way and doesn't run the script:

#!/usr/bin/env node --input-type="module"

UPDATE

the solution I came up with is to create a shell file that launches the js script, e.g. this

├── bin
│  └── script
├── cli.js

script's contents:

#!/usr/bin/env sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

node $DIR/cli.js "$@"

[WebReflection]

you can name your file with .mjs instead

that's half a solution, but also, imagine you had to launch executable files like Chrome.cpp or terminal.gjs and so on 😅

I really hope soon there will be a way to have an ESM only executable files, that also force-load everything as ESM without needing a package.json around.

[ljharb]

@WebReflection i totally agree there should be some way to make it work without an extension - but i think it’s very important that “type module” doesn’t get thrown around and misunderstood as a solution to anything beyond the one tiny thing it does :-)

[WebReflection]

@ljharb it's not so tiny though ... and until there is a --default-input-type="module" flag around, people will abuse the package.json hack here and there, so the sooner we can use that flag, the better for anyone willing to have ESM only executable files.

[ljharb]

It has nothing to do with extensionless files, though, which is the only relevant part of this thread.

[WebReflection]

it does! the day that works one can write:

#!/usr/bin/env sh
node --default-input-type="module" anyfile.js "$@"

and call it a day

edit
but yeah, it's not a full solution for extensionless that would like to use node as env, but there are workaround for that, although these need to hack the package.json here and there, which is undesired.
I've already written about solutions for extensionless a while ago, not going to re-iterate there, but mine is a hack, it'd be great to have something native, including node-esm distributed file (these days deno kinda does that)

[ljharb]

Right, the goal is avoiding a wrapper like that. type module is utterly unrelated - your wrapper could point to an mjs file just as easily.

[mshiltonj]

Being able to signal a short cli script to use esm over cjs is important. For small, general purpose, bash-like, copy-paste-able, executable scripts like, for example:

#!/usr/bin/env node
import http from 'http'
const host = "0.0.0.0"
const port = 8000

const requestListener = function(req, res){
  res.writeHead(200)
  res.end("Hello World")
}

const server = http.createServer(requestListener)
server.listen(port, host, () => {
  console.log(`server is listening on port ${port}`)
})

I'd like to name this script 'hello-world' instead of 'hello-world.mjs` for it to run without error.

[ljharb]

@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.

[sdesalas]

I'm a long term user of Node.js (since v0.4.0) and totally happy with require('module') syntax. It does the job fine, probably even better than ESM modules from a practical standpoint.

I do however DISAGREE with your comment @ljharb. By not supporting ESM module syntax in extensionless shebang executables you are not only forcing programmers to use the CommonJS require() syntax inside the bin file, BUT ON EVERY SINGLE DEPENDENCY from that file up the module tree.

If the problem was with a single file it would really be (as you point out) "a few users of a niche use case", but by making every dependency use CommonJS you are dooming extensionless bin files as a whole which are part of a lot of major Node.js libraries.

Just to pick a few examples: mocha, tape, semver, uuid, he, ncp.

Quoting NPM:

A lot of packages have one or more executable files that they'd like to install into the PATH. npm makes this pretty easy (in fact, it uses this feature to install the "npm" executable). To use this, supply a bin field in your package.json which is a map of command name to local file name. When this package is installed globally, that file will be linked where global bins go so it is available to run by name.

https://docs.npmjs.com/cli/v8/configuring-npm/package-json#bin

[ljharb]

No? You can, in a CJS bin, async import any ESM module.

[sdesalas]

Extensionless bin files I mean. If you add the extension .mjs you can start using import (or even the .js extension with "type": "module" in the package.json)

[ljharb]

Sure. But in CJS, you can always use import().

[sdesalas]

await import('module') syntax inside an extensionless bin file still produces the same issue.

Without "type": "module" it kicks off an error in the dependency chain.

import axios from 'axios';
^^^^^^

SyntaxError: Cannot use import statement outside a module

Using "type": "module" there is still a problem with the resolver.

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /path/to/binfile

Using import or import() or require() inside the dependencies is something I cant always control. So back to square one... 🤔 unless I understood you wrong.. and if that's the case please feel free to explain further.

On the other hand, it seems like @bmeck is still dedicating brain time to this (#42301).

[ljharb]

@sdesalas type module only determines whether your own .js files are ESM or not (it has no effect on things in node_modules). You can still use .mjs for ESM (and should).

[WebReflection]

@sdesalas CommonJS doesn't have top-level await, so instead of this ext.mjs content:

#!/usr/bin/env node

const {default: axios} = await import('axios');

console.log(axios);

You should write something like this in a noext file:

#!/usr/bin/env node

(async () => {
  const {default: axios} = await import('axios');

  console.log(axios);
})();

if you have these excusable files in the same folder where node_modules/axios is installed, you won't have any issue.

[sdesalas]

@WebReflection I used an async IIFE for the await part.

The problem was not with axios on the extensionless shebanged file, but on the dependency chain from its required modules. I'll write up a working example when i have a mo.

[ljharb]

@sdesalas the dependency chain shouldn't have any impact - altho CJS can't require ESM, it can dynamically import it, so any module format can be accessed from any module format, just not always synchronously.

[rmclaughlin-nelnet]

Hey all, I am new to this thread, but I am in the process of upgrading from Node 12 to Node 14 and thought it would be good to convert everything to ESM. However I am running into a bunch of problems, it seems like ESM is not fully baked.

One of our major problems is getting our .bin executables to run with ESM. I have tried using .mjs I have tried type: "module" but it always seems to have some problem. Is this just not possible currently? Is the current guideline to only use CommonJS in executables? If it is possible is there a fully baked example to refer to? For example, in the package.json do I need to specify the .mjs extension in the key? The examples in the doc do not specify.

@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.