Consider additive systems for importing ES Modules

[jamesplease]

I think people want this

[jkrems]

I'm sure you mean well but there's a lot of context behind the decisions to go for .mjs that go beyond "someone just wanted a file extension". So with the current description this issue looks a bit unproductive since it might invite fruitless discussions that have been had a lot of times.

Removing .mjs would always mean also removing some of the features associated with it (e.g. the ability to import a CommonJS file). If you have a counter-proposal with different trade-offs, this issue might look a lot better. :)

[tbranyen]

@jkrems To counter your point, there's also context for why many developers feel this is a poor design decision. I personally feel that Node loses empathy, by continually dismissing counter-points to this effect. If you need proof that it's possible to avoid MJS with ESM, look no further than Google Chrome or @jdalton's @std/esm, as both have already proved it is possible to support both without .mjs.

I find it disappointing that we can't even have this conversation here, in a fork, designed to invite empathy and community. I implore you to rethink how you address this issue with this in mind.

It is fundamentally wrong and irritating to hear that .mjs is required to load CJS from ESM. It is as if import { require } from 'module'; is an impossible ask or something the community wouldn't want.

My question to Node and to you @jkrems is if you've asked developers what they want? Do they want fragmentation or do they want unity at the expense of edge case issues?

[jamesplease]

It's a little tongue-in-cheek after a discussion at work, but at the same time, my naive impression is that there may be alternative solutions worth considering that mitigate some of the issues with .mjs. For instance, .m.js is unquestionably uglier, but it is less likely to is break tooling that has hardcoded .js extension behavior, such as macOS' Spotlight search, which won't show the text contents of .mjs, but will show it for .js, or text editor highlighting, etc.

I wasn't opening this with any expectation that you would change anything here, and, you're right, this discussion has been discussed ad nauseam on the Node.js project. I don't know all of the goals of Ayo, but I bet if some smart people got together to discuss an alternative to .mjs, we could come up with something interesting.

One thing I think would be cool would be to consider what it'd look like to begin planning for a breaking change where .js defaults to ES Modules. Maybe the discussion has already been had (I definitely haven't read every last issue there is on the subject), but, shrug, maybe that's worth considering.

Feel free to close this if you'd like @jkrems, but I do think Ayo provides a compelling place for an .mjs alternative to be explored. It seems clear to me that a lot of the JS community doesn't feel that it's the right decision.

And, who knows, maybe a really good .mjs alternative in Ayo would compel more people to adopt it.

[jkrems]

I do think Ayo provides a compelling place for an .mjs alternative to be explored.

I agree (even though I hardly speak for ayo)! :) But to paraphrase my favorite Hearthstone streamer: "If you suggest to add a card, you need to suggest to remove a card". Otherwise every decisions seems way easier than it actually is.

There definitely are ways to get rid of .mjs and @tbranyen listed some:

1. Go the Chrome route and don't allow import of CJS modules. Maybe even don't support file searching at all and be 100% compatible with browser resolution. Obvious downside: Porting code incrementally becomes a lot harder unless there's an easy to get a require inside of the module. Possible solution would some sort of import.meta.require - but that depends on APIs that are pretty far from landing in V8. It might force us into using a non-standard (actually explicitly rejected) hack like import { require } from 'js:context'.

2. @std/esm doesn't allow import w/o mjs by default afaik. But I assume you're talking about the unlockables. There it does use a barely-even-proposed feature to decide if a file should be interpreted as ES6 of CJS. This implies that some files that would be interpreted as ES6 by browsers wouldn't be by node, leading to awkward incompatibilities. It also optionally allows require of ES6 modules which breaks async loading of file sources and would lock us out of future optimizations enabled by ES6 modules.

I'd personally be fine with dropping all module resolution, dropping support for import of CJS, and aligning with what browsers are doing. But suggesting it without calling out the downsides isn't quite fair. :)

[ljharb]

.mjs is the only solution that solves every problem except "but i like dot js".

That's a totally valid consideration, but "ew, a new file extension" isn't as bad as "i can't use modules because i don't have a package.json".

[tbranyen]

@ljharb a Node flag is completely out of the question because....? node --module index.js. If you want CJS, import { require } from 'module'; or something similar. I'm not sure if I'm being dense here or what, but this doesn't seem complicated to add in today.

[tbranyen]

@ljharb I'm already having to write ridiculous scripts to maintain parity with .mjs. I want to see it as temporary. Much like AMD was temporary. We're so close to a solid standard, and you're okay with fragmentation? Causing all build tool authors to consider mjs and js. Just look at the babel issue tracker. It's only the start.

[jdalton]

@jkrems

@std/esm doesn't allow import w/o mjs by default afaik. But I assume you're talking about the unlockables...

I think @std/esm is pretty great (totally biased). It's implemented more of Node's way (by default) than even Node has yet and allows folks with strong preferences (in either direction) to have their 🍰 and eat it too.

Watching things unfold over the last 2 years I was struck by how many issues come down to an unwillingness to budge on the part of __everyone__ (Node, TC39, browser reps, JS tooling, etc.). Meanwhile, I'm over here like...

[addaleax]

I don’t think Ayo should get rid of .mjs if that’s what Node is going with, but I don’t see why we can’t discuss alternative purely additional options?

[jkrems]

@jdalton DIdn't mean to sounds as if I'm not appreciating your work on @std/esm. It'll help a lot in making sure to lift some of these discussions from "in theory a year from now someone could X" to "let's try it - now - and see".

But I do disagree that it allows people to "have their 🍰 and eat it too". I'd maybe call it "allow people to pick a different flavor of 🍰 and eat that". It's not like the unlockables are just pure additions w/o any downsides and some of them do exploit the fact that you're not bound to the existing module spec & implementations. E.g. exposing require/__filename is easy if you assume that at some point magic per-module quasi-global will make it into engines (I'd call that questionable) or that node's wrapper-function-hack will stay forever. I do understand why that trade-off is worth it for some people and they'd rather pick that one. But it's still a trade-off and not just a purely positive addition w/o any downsides whatsoever. :)

[jdalton]

@addaleax

I don’t think Ayo should get rid of .mjs if that’s what Node is going with, but I don’t see why we can’t discuss alternative purely additional options?

Coming from my side, userland package solution, I was surprised to see usage like

node -r @std/esm app.js

be so popular. That makes me think a Node/Ayo flag might be an acceptable route. One of the ways @std/esm was able to side-step a lot of the gridlock of implementation is that its non-standard things are off by default (this avoids or simplifies getting buy-in).

For example, take one of the unlockable @jkrems was picking at, source type pragmas (i.e. "use module"). Being able to specify how you want your source code parsed is a good thing. It takes the guess work out of tooling and loading of files, but it winds up gridlocked with stakeholders unbudging. Having that opt-in/unlockable empowers devs while avoiding the on-by-default mass impact concern (since devs have to flip a switch).

@jkrems

But I do disagree that it allows people to "have their cake and eat it too". I'd maybe call it "allow people to pick a different flavor of cake and eat that".

It's still 🍰.

Opting-in is the dev saying "Yes, I understand this thing may not be perfect, but it's good enough for my needs and I'm willing to accept any gotchas that might pop up."

[Qard]

Wouldn't needing to do import { require } from 'module'; break literally everything in userland? 🤔

[tbranyen]

@Qard, what I proposed was a --module option which would infer that all code is to be interpreted with ESM semantics. This distinction is identical to <script type="module">. Notice this is opt-in, it would be backwards compatible and break nothing in userland.

Under --module if you wanted to execute CJS code, say from Node core, or the NPM registry, you would import the require function and use that. This seems to make so much more sense than trying adapt CJS to ESM. Did you know that you cannot do import { readFile } from 'fs';? How would you know that given the fact that it appears to be an ES module? This would disambiguate in a way that makes more sense, to me, than a file extension.

Once a script gets executed with require, regardless of --module flag, it and all scripts imported from it will be treated as CJS.

I'd love for someone to refute why this is not an acceptable solution.

[Qard]

Ah, missed that. So that require would create a Script type rather than Module type, therefore no access to es module syntax inside those? Seems reasonable.

[jamesplease]

Imagine something like @tbranyen's solution, but with this additional roadmap:

1st major release: the --module flag is added, as well as the --cjs flag. The default behavior remains --cjs. The Node/Ayo team communicates their intention to eventually switch the default flag, and encourages developers to use --cjs for their CJS modules.
2nd major release: no changes
3rd major release: not specifying a flag is now deprecated, and warns you when you boot up Node.
4th major release: no flag now defaults to --module

I'd love to see modules become the default sometime, and I think a slow rollout like this makes it seem realistic. As a developer looking at this roadmap, I'd know how to plan and prepare for the eventual breaking change, well in advance.

Ecosystem tools, like npm, might be able to do some things to ease the transition. The engines field in package.json could possibly be leveraged to mitigate the challenges with determining if the lib you're importing will work as you'd expect.

Perhaps something like:

[npm WARN]: blahblah@2.0.0 specifies `node >= 12.0` in `engines.json`.
  Please specify `node >= 12.0` in your `engines` if this package supports
  the breaking changes in Node 12.

I'm definitely not an expert on this subject, though, so there could be some issues with this suggestion.

[tbranyen]

Note with this approach the Node core would have to remain CJS (in order to allow --cjs code to still import Node's standard library). It has already been brought up as non-trivial change to make the core ESM so this aligns well with that concern and offers further justification for keeping it as-is.

[tbranyen]

For fun, I'm going to try and implement my suggestion in Ayo. I've wanted a reason to explore the Node core and this is an interesting alternative to implement. I've already got some of this working (the code is so good this is straightforward to piece together). Not sure if I'll PR, but I'd like to try this out for a while to see how it breaks down.

One change I'm making is --esm instead of --modules to align with the --cjs flag.

[Fishrock123]

I mean, Ayo really isn't constrained with user breakage they way Node is, so... migrating to module mode as a default is probably possible.

It probably won't be nice either though - you'll end up running into code that refuses to run correctly just by using npm modules.

---

As for flags, I'd suggest taking a look at nodejs/node-eps#62 - which will probably be pulled into Ayo at some point anyways once it lands in node, and would allow us to re-use some familiar options.

Also do consider that in the future, loading WASM/WAST modules will likely be a thing, and the same issue will occur there. Node is very likely to extension-detect .wasm.

[bmeck]

There are also discussions of loader hooks ala nodejs/node#15445 ; I would want strong discussions to take place around use cases since as @jdalton has stated, no one party seems to be satisfied by any of the approaches.

As I have stated in nodejs/node-eps#60 , additive changes are welcome in Node as well. If either environment wishes to make breaking changes wrt the other or the web I would be hesitant to adopt things without very thorough discussions.

[bmeck]

@jmeas I would be hesitant to encourage such a broad change, almost everything on npm doesn't work like real ESM even if it uses the grammar and such breakage means all CLIs could stop working when you upgrade node/ayo (since they default to CJS)

[bmeck]

got a discord channel if people want to chat in there: https://discordapp.com/channels/350004786915180548/359451835779514378

[tbranyen]

@bmeck I tried accessing that url, but it just loads a blank page in discord asking me to add friends, do I need special access?

[TimothyGu]

@tbranyen How about https://discord.gg/QhR4JyZ?